Layer7 API Management

  • 1.  Traffic Logger & Audit sink

    Posted Oct 15, 2015 10:34 AM

    Hello All,

     

    I am currently using the SSG logs to log all details pertaining to transactions done by the published services. It is becoming tedious to troubleshoot issues pertaining to a single transaction using this SSG logs since the info regarding a particular transaction can't be found sequentially as all parellel transactions happening at that time is also logged there. Hence I thought of implementing traffic logger and configuring it in such a way that it can log info regarding the request ID, request method, request elapsed time, response HTTP status, request message and response message etc. in a single line and can be found easily for troubleshooting issues and it can also be easily parsed by external log management tools such as Splunk, IBM QRadar etc.,.

     

    In addition I would also like to log exceptions in the Audit table in the internal SSG DB. I am planning to set the level to "Warning" so that only transactions with exceptions/failures would be audited.

     

    My Questions are,

     

    1. Will there be any performance issue due to the traffic logger since it has to wait for the entire transaction to complete before logging the details?
    2. Will I be able to configure traffic log as a syslog and send to external log management tools using UDP or TCP?
    3. Is it a better approach to use traffic logger instead of SSG logs in production?
    4. Is it mandatory to have Audit sink configured in production? or Can I just save the audit records to internal DB and purge it at a regular interval? (We are not expecting too much production traffic and it will be a fraction of the peak load suggested by CA)

     

    Kindly clarify the above questions.

     

    Thank You!



  • 2.  Re: Traffic Logger & Audit sink

    Broadcom Employee
    Posted Oct 19, 2015 04:18 PM

    I can speak to the use of a Traffic Logger and answer Question #2 and #3.

     

    Traffic Log Feature:

    We have a great Out of the Box feature for creating a Traffic Log. You basically choose this option as a Category under the Filters section of the Log Sink Properties dialogue window when you choose to Create a new log (from originally choosing Tasks > Manage Log/Audit Sinks). You then have the ability to set the content captured by this log in a Cluster-Wide property named trafficlogger.detail.

     

    Steps:

    1. Tasks > Manage Log/Audit Sinks

    2. Click Create on the right side of the window

    3. Name the log accordingly, and check the Enabled checkbox

    4. Add description to the log

    5. Set the Type to File or Syslog, depending on your desire

    6. Choose Severity Threshold as Info, unless you are familiar with our logging levels and want to change. Most events will happen from Info - Warning, so recommend to keep at Info

    7. In the Filters section, click Add, and choose Traffic Log under the first filter dropdown called Category

    8. Click on the File Settings tab at the top of the window, and choose values accordingly

    9. Click on the Syslog Settings tab if applicable and choose desired settings. You can close this window as the log has been created

    10. At the top of the Policy Manager go to Tasks > Manage Cluster-Wide Properties

    11. Click Add on the right hand side of the window

    12. Click the Key dropdown menu and scroll to the bottom to find trafficlogger.detail

    13. In the Value section, add the context variables and format you would like to log them in

    a) Ex: RemoteIP:${request.tcp.remoteIP}|NodeIP:${ssgnode.ip}|Time:${request.time}|Method:${request.http.method}|URL:${request.url}|Status:${response.http.status}|User:${request.username}|ClientID:${request.clientid}

    b) You can reference any variable to your heart's content. Just note that any variables created w/ a Set Context Variable assertion in a policy will be a localized variable and may not apply to policies; we reccomend only using default context variables such as ones in the above example.

    14. You click OK to close this window as the Traffic Log has been fully set up. Note that at any time you can edit this Cluster-Wide property to have an immediate effect on the log

     

    Questions:

    2. Will I be able to configure traffic log as a syslog and send to external log management tools using UDP or TCP?

    Yes in the Log properties you can do:

        a) Create Traffic Log locally only

        b) Create Traffic Log via Syslog only

        c) Create two logs: Local Traffic Log and Syslog Traffic log

     

    3. Is it a better approach to use traffic logger instead of SSG logs in production?

    From my experience, clients often use both/multiple logs. They have ssg as a catch-all log and then opt to create specific logs for certain circumstances, in your case, a separate Traffic Log.



  • 3.  Re: Traffic Logger & Audit sink

    Posted Oct 20, 2015 07:19 AM


    Hello bluna01,

     

    Thanks a lot for the detailed steps.

     

    I still would like to understand more on Audits and would be grately helpful if I can get answers for questions 1 & 4.



  • 4.  Re: Traffic Logger & Audit sink

    Broadcom Employee
    Posted Oct 20, 2015 11:37 AM

    I'll try and answer these to the best of my knowledge and experience.

     

    1. Will there be any performance issue due to the traffic logger since it has to wait for the entire transaction to complete before logging the details?

    In essence, if you use predefined context variables and not custom ones set in a policy, you should see no performance hit to your Gateway. Basically all these values are being set behind the scenes anyways when you execute policies, so all your doing is calling the variable. Again, if you you were to call a Set Context Variable that was never actually set, the Gateway does take a small performance hit when that value is blank, so that is why it's recommended to ONLY include default, predefined context variables. Other than that, the log will not produce results until the policy has finished--so lets say you had a 1000 line policy, of course it would take a little time for the policy to run through before it spat it out to logs, but that's not really a performance hit on the Gateway itself rather than a limitation by the policy.

     

    4. Is it mandatory to have Audit sink configured in production? or Can I just save the audit records to internal DB and purge it at a regular interval? (We are not expecting too much production traffic and it will be a fraction of the peak load suggested by CA)

    Ideally, we of course recommend to "off-box" audits with an Audit Sink policy due to performance read/writes happening to the audit record tables in the application database. If you are foreseeing very little TPS and service usage on the Gateway I could understand the motive to just leave audits locally for now. It's not recommended, but it can be done.


    The only caveat to doing that is that we recommend you use Logs RATHER than audits to do any type of "auditing" in-policy as to limit the usage of read/writes to the database. So when you use an Add Audit Details Assertion, set it to send to log, etc. Maybe even look to change the audit.messageThreshold level to WARNING after testing of polices/services has been completed to lighten to number of records being written to the database (this is done via the Cluster-Wide property audit.messageThreshold).



  • 5.  Re: Traffic Logger & Audit sink

    Broadcom Employee
    Posted Oct 21, 2015 11:04 PM

    In addition to Nathan's comment, here are few things about performance:

    # the gateway before 8.2 will wait for auditing.

    # using database for auditing is not recommended for a busy system -- the database is apparently the bottleneck. The performance even worst if DB replication enabled, that's the cost of HA. (Well, for a small system, regular purge should be fine)

     

    So, we recommend to use syslog to push the audits to a central server. Some complex implementation even use message queue to async the auditing.