Layer7 API Management

  • 1.  Using response.mainpart.size variable causes message size limit error

    Posted Jan 30, 2019 11:12 AM

    Hi there,

    we found a very strange behavior with the following outcome/result.

    We have an API, which works with simple small GET-requests, but the response is a very large json-object. As of now this was working fine without any issues or error messages. Now we were optimizing our policies and included some additional variables for external logging. One of these variables is "response.mainpart.size". But as soon as this "Set Context Variable"-assertion is active, ALL requests are ending in the "Unable to read stream: the specified maximum data size limit would be exceeded" error message. During further analysis and debugging we found out, that not only this system variable is causing any additional internal checks, but also the debugger in the Policy Manager itself as well. Means, even with disabled "Set Context Variable"-assertion, which normally works fine again without any error, we still get above mentioned error, as soon as we are starting the debugger.

    So does anyone has any idea, what's happing in the background and causing this error? And even more important, is there any solution for this or at least some kind of workaround?

    If you need any further Information, please let me know.

    We are currently running on 9.1.01

    Thank you!

     

    Ciao Stefan



  • 2.  Re: Using response.mainpart.size variable causes message size limit error
    Best Answer

    Broadcom Employee
    Posted Jan 30, 2019 12:42 PM

    Stefan,

     

    Good evening. Depending how the policy is configured the response message will be streamed back to the client so it will not cause the size limitation to be enacted. When you enable the debugger and touch the response message in some way will pull the message into memory causing this to occur. If you are looking to analyze the size of the response from the backend you should be able to use the variable ${response.http.header.content-length} without forcing the response into memory.

     

    Sincerely,

     

    Stephen Hughes

    Broadcom Support



  • 3.  Re: Using response.mainpart.size variable causes message size limit error

    Broadcom Employee
    Posted Jan 30, 2019 04:52 PM

    Hello StefanKlotz ,

    It's usually due to the response is zipped (usually we have header in the request to accept gzip, etc. so the response usually be zipped), once you use some response variables, or scan response body, the gateway will unzip it.

     

    Without unzip, it may under the size limit, but after unzip, it may violate the size limit -- you may need to set a bigger size limit according to the original response size.

     

    Regards,

    Mark



  • 4.  Re: Using response.mainpart.size variable causes message size limit error

    Posted Jan 31, 2019 04:40 AM

    Hi Stephen and Mark,

    thank you both for your quick response and it seems going in the right direction.

    First of all, I was not aware that using request- or response-variables, that this cause loading the request/response into memory.

    And second, it seems that the response is really compressed. At least I see the accept-encoding header in the request. But if the response is zipped, then the content-length header is not available. So is there any other chance to get the size of the response for our logging? If it would be zipped, I would be happy with the zipped size, because we are only interested in the amount of traffic going through the gateway. The unzipped size is not interesting for us.

    And third, it still breaks when the debugger is running. Here as well, is there any chance to get this working without loading the request into memory? Or what is the reason, why a request is loaded into memory when using the debugger? Is this depending on any specific assertions or will this ALWAYS be the case and can't be adjusted?
    Thank you!

     

    Ciao Stefan



  • 5.  Re: Using response.mainpart.size variable causes message size limit error

    Posted Jan 31, 2019 05:49 AM

    Hi again,

    I made some additional tests and put the Content-Length header value into the logs. It shows the same size (11.373.620) as the locally save json-file. So there is NO compression in use.

    So as a conclusion, the message size limit is only affective, if the request is somehow loaded into memory?

    Or are there any differences in behavior between requests and responses? Is the ${request.mainpart.size} also causing the request to be loaded into memory? So should we use the Content-Length header here as well (if available)?

    And is there any overview available, which assertions are causing a request/response to be loaded into memory?

    This would be generally interesting in policy design and which assertions to use, because using memory for traffic handling can have dramatically impact on hardware resources, in this case available RAM.

    Thank you!

     

    Ciao Stefan



  • 6.  Re: Using response.mainpart.size variable causes message size limit error

    Broadcom Employee
    Posted Jan 31, 2019 12:10 PM

    Stefan,

     

    I would use the request.http.header.*** values where possible to avoid the message from not streaming through. As for the debugger, as this stops all streaming then it will trigger the same behavior. Lastly, any time that the body of the message is act against it will have to pull it into memory so XML activity (Schema validation, XPath,etc), auditing the body of the response to name a few.

     

    Sincerely,

     

    Stephen Hughes

    Broadcom Support



  • 7.  Re: Using response.mainpart.size variable causes message size limit error

    Posted Feb 05, 2019 09:58 AM

    One hopefully last question here.

    I made some tests with the Content-Length header, but I noticed that this is NOT included if the response is chunked.

    So is there any other possibility to get the size of the response without using the mainpart.size variable?

    And what about the Content-Length header for requests? Is this always available as soon as the request contains a body/payload? Or are there also special scenarios where a body/payload is included in the request, but there is no Content-Length header?

    Thank you!

     

    Ciao Stefan )



  • 8.  Re: Using response.mainpart.size variable causes message size limit error

    Posted Feb 05, 2019 12:09 PM

    Hi again,

    during lots of reading here in the forum (also for other topics), I found the request.size and response.size variables.

    But it seems using these variables that the request/response will again loaded into memory.

    So is the Content-Length really the only option?
    Thank you!

     

    Ciao Stefan



  • 9.  Re: Using response.mainpart.size variable causes message size limit error

    Broadcom Employee
    Posted Feb 05, 2019 07:03 PM

    The content-length header of response is set by the backend server, if the body is compressed, it should be the size of compressed body

    (ie. you may need to check the backend server if content-length header is not found, or not correct)

    for requests, it's set by the requestor.

    here is the specification,

    RFC 2616 - Hypertext Transfer Protocol -- HTTP/1.1