Some customers have been curious as to why SOAP services and REST services provide different errors when message size limits are exceeded. This leads to further questions about the relationship between io.xmlPartMaxBytes cluster-wide property and the "Limit Message Size" assertion.
The difference in behavior is due to a setting in the Service Properties called "Perform WS-Security processing for this service". When that checkbox is checked, it results in the message being buffered and some processing of the message is performed before reaching the service policy. In that instance, the message is checked against the cluster-wide property first before it reaches the service policy. If the message passes the cluster-wide property, then it makes it to the policy and is checked against the "Limit Message Size" assertion if present. If the message fails the check against the cluster-wide property to begin with, it never makes it to policy regardless of the value of the "Limit Message Size" assertion.
If the "Perform WS-Security processing for this service" checkbox is unchecked, then the message does not get buffered and goes to the service policy directly. The message is then compared against the "Limit Message Size" assertion if present. If the assertion isn't present in the policy, then it will be compared to the cluster-wide property value.
The difference between REST vs SOAP in this instance is that SOAP services by default have this checkbox checked, whereas REST services have this checkbox unchecked. If you uncheck this value from a SOAP service, it will behave like a default REST service. If you check this value in a REST service, it will behave like a default SOAP service.
So to simplify it, a request to a REST service or SOAP service with WS-Security processing enabled is buffered and therefore checked against the cluster-wide property first. If it exceeds the limit, you receive:
Error: Unable to read stream: the specified maximum data size limit would be exceeded
If the message size is under the limit defined, the message continues to the policy, and processed accordingly.
The REST service and SOAP service with WS-Security processing disabled goes to the policy first, and ONLY adheres to the "Limit Message Size" assertion if present and ignores the cluster-wide property. If the message size exceeds the limit of the assertion, then you will receive:
<l7:policyResult status="Assertion Falsified" xmlns:l7="http://www.layer7tech.com/ws/policy/fault"/>
If the Limit Message Size assertion isn't present in the policy, then it adheres to the limit defined in the cluster-wide property.