Arvind,
The usage of the limit size assertion should not introduce any latency as it only views the content size header and evaluates the size of the payload. What could be impacting the flow is where you are streaming or not streaming in the request or back the response. For inbound there is an assertion Configure message streaming (only for inbound requests) and as long as the payload is not modified then it will stream through or for responses if you include anything after the HTTP Routing assertion that inspects or changes the payload then it will not stream back.
Sincerely,
Stephen Hughes
Director, CA Support