I had a similar issue i.e. when the number of groups was huge, policy server as part of response read the group info from user store along with other headers (email address, firstname, lastname). I was able to see in the policy server trace logs, the entire list of groups. However on the webagent trace logs, only single values response attributes like email address, firstname, lastname was visible. The group header was never received by the webagent.
I was supposed to raise a Support Case, but haven't the time to do that.
The workaround was to trim the DN's of the group and return only GroupNames.
https://communities.ca.com/thread/241781903-can-you-parse-smusergroups-to-only-return-just-the-cn-of-a-group-instead-of-the-full-dns-of-each-group
We can further cut down the value by using FILTER from expressions.
The workaround may be considered as a Solution. However the question still remains, what if the number of group names returned hit that limit over a period of time even after writing a expression. We really want to understand is there a fundamental limit / size of header that is allowed between the Policy Server and WebAgent. I know we have properties files to control the size of Assertion. But never seen one for header responses. The curious factor is we cannot even state that it is the network / packet size limit, because between the UserStore and Policy Server, the Policy Server was able to read the entire list and print the list in smtracedefault.log. However between the Policy Server and WebAgent, there was some size limit encountered which prevented only the group header from passing through. Would need further investigation.