Layer7 API Management

Expand all | Collapse all

How to completely disable all logging for a policy?

  • 1.  How to completely disable all logging for a policy?

    Posted Jul 12, 2017 08:19 AM

    Currently we are using loglevel INFO on our QA. We are using loglevel WARNING on Production.
    However, we have a customized healthservice for our loadbalancer that gets called a lot on all environments.
    This generates a huge amount of audit logging.

     

    I want to disable all logging for this policy completely. By setting it to FINEST for example.
    Sadly this seems to be impossible, or i haven't found how yet.
    I've tried setting the context variable auditLevel to FINEST. But it has no effect.



  • 2.  Re: How to completely disable all logging for a policy?

    Posted Jul 14, 2017 12:54 AM

    Hi Dimitri,

     

            I hope its not possible to disable the logs of particular API/Service. Because mostly Layer 7 provide the security and reliability to capture the errors where the issue is so i am sure its not possible.

     

    Please can someone rectify me if i am wrong. As far as i know its not possible to disable for the particular Policy logs.

     

    If i am correct we can add this in idea page.

     

    Regards

    Rajasekhar



  • 3.  Re: How to completely disable all logging for a policy?

    Posted Jul 14, 2017 06:40 AM

    Usually i would agree, but i've learned there are always good reasons for exceptions
    We are not really interested in logging every 20seconds for each VIP from our loadbalancer.

    If it would just be possible to change the loglevel to FINEST for an entire policy for example, it would be solved.


    You can switch a policy between INFO and WARNING with the assertion "Audit Messages in Policy". We already use this to reduce logging on production, by putting the cluster minimum Loglevel on Production to WARNING and on QA to INFO. IF we need to log extra for the policy, we change that assertion to WARNING on production for one specific policy.

    But for this Health-service i would need more control, another level basically. If i can set this one to at least FINE, then it won't be in the audit logging on QA and PROD, which is what we want here.

     

    So the essential complaint is, you cannot use any loglevels other than INFO and WARNING for a policy, which is an unnecessary restriction, and one artificially probably because the gui only supports these two, while the logging system does support all of them.



  • 4.  Re: How to completely disable all logging for a policy?

    Posted Jul 14, 2017 09:47 AM

    Are you using separate webservices published on gateway to monitor health of gateway's ?

     

    Thanks,

    Ankush



  • 5.  Re: How to completely disable all logging for a policy?

    Posted Jul 17, 2017 10:44 AM

    Yes we are. We have built a policy that actually calls the built in healt service, using a http route to 127.0.0.1
    We have restricted the in health service is available only on the loopback listener on 127.0.0.1, so it is not available to the internet.

     

    Our health policy also has additional logic that is required for us:
    It has ip filtering, allowing only the loadbalancer ip's to call it. All other clients will get a 404 denied without any payload.
    We also have 3 states in our health service:

    1. Online (downstream from CA healtservice)
    2. Maintenance (custom KBC logic)
    3. Offline (downstream from CA healtservice)

     

    We can put one specific apigateway in a cluster in maintenance, by changing a clusterwide property. This will cause our loadbalancer to gracefully drain one gateway before we do any maintenance work. We use this for monthly patches for example, which we implement every third Tuesday of the month on PROD.

     

     

    Sadly, this thing gets called for each listener we have. On each NIC we have.
    Which is currently very often.



  • 6.  Re: How to completely disable all logging for a policy?

    Posted Jul 18, 2017 10:18 AM

    If you can configure " separate log file for services used for monitoring health " , ca api gateway has features to configure separate log file for policies, you might be able to achieve what you are looking for.

    Try this :  Working with Log Sinks and Debug Logs - CA API Gateway - 9.2 - CA Technologies Documentation 

     

    Thanks,

    Ankush



  • 7.  Re: How to completely disable all logging for a policy?

    Posted Jul 18, 2017 11:21 AM

    That's an include filter not an exclude filter and sending events from one service to another log will not prevent it from going to the other, which means that this would not prevent thier healthcheck traffic from showing in the main log (or the audits, which I'm certain is what they mean when they say 'logs' based on the cluster properties and event levels).



  • 8.  Re: How to completely disable all logging for a policy?

    Posted Jul 17, 2017 06:39 PM

    Take a look at the audit details of your healthservice, there are some events there.  If you set the level of those events below the level of your audit.messageThreshold then it will stop recording the message because auditing is triggered by the level of the message which is the highest level of detail event that occured durring reqeust processing.  This is an environment wide setting so the other services may also stop appearing (when debug is turned off for them).

    You can set the level of specific audit details to a certail level using cluster wide properties:

    audit.setDetailLevel.FINEST

    audit.setDetailLevel.FINER

    audit.setDetailLevel.FINE

    audit.setDetailLevel.INFO

    audit.setDetailLevel.WARNING

    audit.setDetailLevel.ERROR

    audit.setDetailLevel.SEVERE

    value is a space separated list of the audit detail codes (i.e. 7101 7103 7104 7100 4700 4716 4715 9648)



  • 9.  Re: How to completely disable all logging for a policy?

    Posted Jul 19, 2017 03:30 AM

    Ben,

     

    This would work, and is my prefered way of dealing with it. But it currently also impacts everything else. As mentioned earlier our AuditLevel in QA is INFO, the AuditLevel in PROD is WARNING. The "Audit messages in policy" assertion only supports these two audit levels anyway, so all the other levels we can set globally are kinda useless because of this.

     

    If we want to log "message received information" and "message payloads" using that assertion, we can only choose Warning and Info. Because of this every policy will be logging their main logging in either Warning or Info, this cannot be changed, it must be either of these two. If i set the Global Loglevel to FINE for example, no policy will log message payloads, and there is no option to have the policy log this information in FINE. Reverse, if i set the LogLevel to ERROR, all policies will log payload, there is no option to set this logging in a policy to anything lower than WARNING.

     

    Because of this, the only loglevels i can utilize for this now, are Warning and Info.
    It seems to me to perhaps be a bug in the "audit messages in policy" assertion, which only supports WARNING/INFO and in this way restricts the functionality of the different loglevels



  • 10.  Re: How to completely disable all logging for a policy?

    Posted Jul 19, 2017 02:02 PM

    I'm guessing that you have put the audit messages in policy assertion into one of the global policy fragments, otherwise it wouldn't be affecting all services.  You could set the message level to INFO and with your audit level where it is the messages would be captured in prod only when auditing occurs, which is calculated (and controlled) as described in my other comment.  In QA you're still auditing everything at level INFO, so to not capture audits for that one service you would have to not invoke the audit messages in policy assertion during processing of those requests, however you could at least not capture the messages by overriding the settings by calling the audit messages in policy assertion again in that service...  Or set audit level in QA to warning and the message level to warning, then override message level to INFO for that one service, ya that should work...

     

    Or you can abandon the audit messages in policy assertion and instead use the add audit details of ${request.mainpart) and ${response.mainpart}, and if your error handling is well crafted then you can prevent capturing the messages for failure modes where they would not be relevant.  I prefer this in part because the audit message assertion will capture the message sent to the client but ${resonse.mainpart} is what was returned to the gateway from the backend server.



  • 11.  Re: How to completely disable all logging for a policy?

    Posted Jul 24, 2017 05:22 AM

    I was a bit unclear. It seems every policy logs message processed sucessfully logging. This is without any logging code inside your policy, nothing was added to produce this log. This does affect all services, and cannot be turned of as far as i know:

    Node : Gateway1
    Time : 20170724 11:08:30.928
    Severity : INFO
    Request Id : 0000015d17250f4e-35027d
    Message : Message processed successfully
    Audit Record ID: 919c8a73b9b9d42b2ff4174106fd0bba

    Event Type : Message Summary
    Client IP : ------
    Service : HealthCheck [/healthcheck]
    Operation : null
    Rqst Length : <Not Saved>
    Resp Length : <Not Saved>
    Resp Status : 200
    Resp Time : 3ms
    User ID : <No ID>
    User Name : null

     

     

    Additionally the policy also logs specific audit lines for some assertions in the policy. This is always INFO for the regex compare one, and cannot be changed.
    For example:

    20170724 11:16:20.885INFO7101Comparison did not match: ${lookup.output} is equal to true

     

    This global "Message processed successfully" logging can be added to your policy, and changed to WARNING.
    It only allows these two options:

    When you do that, it will log like this:

    Node : Gateway1
    Time : 20170724 11:15:06.849
    Severity : WARNING
    Request Id : 0000015d17250f4e-350615
    Message : Message processed successfully
    Audit Record ID: 919c8a73b9b9d42b2ff4174106fd14c9

    Event Type : Message Summary
    Client IP : ----
    Service : HealthCheck [/healthcheck]
    Operation : null
    Rqst Length : <Not Saved>
    Resp Length : <Not Saved>
    Resp Status : 200
    Resp Time : 3ms
    User ID : <No ID>
    User Name : null



  • 12.  Re: How to completely disable all logging for a policy?

    Posted Jul 24, 2017 05:34 AM

    I did some more research. And the default logging of messages is INFO, while the default loglevel is WARNING.
    So in a default factory gateway, no policy will log anything below WARNING, unless you enable it to log. 


    Our environment is different, because we enabled all logging on QA, by setting the global threshold to INFO.
    Because we want all logging for most policies, but not for all.



  • 13.  Re: How to completely disable all logging for a policy?
    Best Answer

    Posted Jul 24, 2017 12:30 PM

    Ok, here's what you do.

    1. Wherever you put the 'audit messages in policy' assertion, nest it inside an 'at least one must be true' and just ahead of it use a 'compare' to see if the request.url.path is your health check service.  This way you prevent flagging the health check messages for audit.
    2. Set the level of the 'audit messages in policy' assertion to WARNING
    3. Set audit.messageThreshold back to WARNING

    This way everything except the health check service will be treated as WARNING and therefore everything except the health check will get audited.



  • 14.  Re: How to completely disable all logging for a policy?

    Posted Jul 27, 2017 08:42 AM

    We did something similar.

    • I added an exported context-variable kbcLogWarning in the pre-service global policy.
      • This flag is set to 1 on QA.
      • This flag is set to 0 on PROD
    • I set this flag kbcLogWarning inside the healthCheck policy code to 0 on QA and PROD
    • I added an audit messages in policy in the post-service global policy.
      • If kbcLogWarning is 1, the audit messages in policy changes the logging to warning
      • if KbcLogWarning is 0, the audit messages in policy is not changed/set

     

    This has the expected result. All the policies in QA log into the audit log for every message received. 
    And a policy, like our HealthCheck policy can set the flag to 0 to disable its logging in QA.
    All of our policy's are built to log their audit events as INFO, it is only overruled globally on QA via the flag and post-service logic.

     

    Still, i think it would be easier if CA could allow usage of more log-levels here