Vivek,
While you might be able to get this working on the OS that your working with, it's not a supported use case.
https://support.ca.com/phpdocs/7/5262/5262_SiteMinder_12_52_SP1_Platform_Support.pdf
The supported Windows OS's would be...
The 500 is occurring do the agent not starting properly.
The agent not starting could be a number of issue even if we rule out the OS Release.
What could try is to attempt to start the agent via command line.
From the agent install, naviagte to the /CA/webagent/win64/bin
Run LLAWP without any switches.
You will see the command requirements.
>llawp
Usage:
llawp <configfile> -<Web Platform> <-a> <-c>
<configfile> : Path to valid WebAgent.conf file.
<Web Platform>: -APACHE20, -APACHE22, -APACHE24, -ISAPI60, -IIS70, -IIS75, -IIS80, -IIS100 -SPS60, -SPAGENT, -SUNONE.
<-a> : Automatic shutdown - Shutdown when the LLAWP clientlist goes to 0.
<-c> : Cache Client - LLAWP attaches to caches without attempting to create them
<-shutdown> : Shutdown the running worker process associated with a WebAgent.conf file.
NOTE: Configuration filenames and version strings that contain spaces
should be surrounded by quotes. Example: "value with spaces"
So you command would something like ...
llawp %PathTo%\webagent.conf -IIS70.
If the prompt generates a 'rolling' message, the agent started.
If you configured logs in the ACO, those should be generated.
If the command, just drops to the next line, then the agent didn't start.
If it does start and there are logs, then the issue is within the configuration or capability of the release of IIS.
If it doesnn't start at all, then the issue might be with the installation, general configuration of the agent or the overall support of the OS.
Even if you go through all that and get it to work, then you will still not be in a supported configuration and unable to work with SSO Support going forward.
Therefore, let's start with a support OS and work forward from there.
If you still have the same issue, we would need to review the event viewer log and the SMPS Logs to see if we can get a better take on the issue.
You might want to create a formal case with Broadcom SSO Support if needed.
-James