Hi HubertDennis
I've been working with this a bit more. I have determined that my front end apache is refusing connections. The backend and tomcat (Underlying user directories, policy server, policy store) seem to be holding up okay. I can see in the apache logs saying that the thread limit reached, consider increasing the MaxThreadsPerChild setting.
When I try to increase MaxThreadsPerChild, it gives me a warn during startup in apache error log that MaxThreadsPerChild is set higher than ThreadLimit of 1920 and it reduces the MaxThreadsPerChild to 1920.
Is there a hard limit of 1920 on the apache threads? Can this be increased? the mpm_winnt has a limit of 15000, is it possible to set the SPS value closer to this number?
I have 14 SPS instances running to serve a load of 70,000 WS-FED transactions within a 10 minute period. So that is about a combined concurrent requests of 25,000 or so.