Symantec Access Management

We are doing upgrade from 12.52 to 12.72. We have installed/configured policy server, SPS and Fed components on separate VMs. Now our old Components are co-existing with new components on Win 2012R2. All Policy Servers are referring to single Policy/Key Store. Policy Servers on all the Web Agents SMHost.config files have been updated to point to new policy servers. We have a custom Form based Login Gateway app.  When we try to access individual application URL, we are redirected to Login gateway screen, after authentication, SMSession is generated and passed back to the original web app but the original application Web Agent cannot validate the SMSession and redirects back to the Custom Login page. I do see following error in the web app WALog.txt file:

[CSmHttpPlugin.cpp:2223][WARNING][sm-HTTPAgent-00190] Unable to process SMSESSION cookie.

Any help/guidance is appreciated!

AroraS

Let's start here. Please check the corresponding transaction in WebAgent Trace log. That will give a better understand of why the SMSESSION was not processed.

[CSmHttpPlugin.cpp:2223][WARNING][sm-HTTPAgent-00190] Unable to process SMSESSION cookie.

Possible scenario's.....

• Often this happens when there are scenarios like multiple KEYS markers in the Key Store and the WebAgent is unable to decode the SMSESSION, even though everything is pointing to the same Policy / KeyStore.
• Time is off sync between different WebAgents, whereby the SMESSION is now considered to be expired.

Regards

Hubert

.- Correct all the 4 Policy Servers (2 Old Policy Servers and 2 new Policy Servers are referring to the same Policy/Key Store. I ran smkeyexport command on all the policy servers and keys match.

- MaxTimeDeltaBetweenServers is set to 30 sec and all the servers have their time synced too.

AroraS

Please note that only one Policy Server can be the Key Generator amongst all the 4. 

Nevertheless, I need us to look at the WebAgentTrace.log before we discuss anything else, as that will pinpoint the source of the issue.

Correct., we have only one Policy Server as Key generator. Here are the WebAgent Logs.

-----------------------------------

11/13/2018][15:48:12][4940][3348][LLAWPLogQ.cpp:693][LogWorkerFunc][][][][][][][Tracing initialized.]
[11/13/2018][15:48:15][4940][3160][LLAWPMsgBus.cpp:181][ProcessMessage][][][][][][][Open message received from client '3896.1332']
[Date][Time][Pid][Tid][SrcFile][Function][TransactionID][IPAddr][IPPort][AgentName][Resource][User][Message]
[====][====][===][===][=======][========][=============][======][======][=========][========][====][=======]
[11/13/2018][15:48:15][3896][1332][CSmLowLevelAgent.cpp:5241][ConfigureMP][][][][][][][Tracing initialized.]
[Date][Time][Pid][Tid][SrcFile][Function][TransactionID][IPAddr][IPPort][AgentName][Resource][User][Message]
[====][====][===][===][=======][========][=============][======][======][=========][========][====][=======]
[11/13/2018][15:48:15][4940][3160][LLAWPMsgBus.cpp:254][ProcessMessage][][][][][][][Delivering response to Manage query received from client '3896.1332']
[11/13/2018][15:48:15][3896][1332][CSmAdminManager.cpp:841][ManageAgent][][][][][][][Received message KEY_UPDATE_PERSISTENT.]
[11/13/2018][15:48:15][3896][1332][CSmAdminManager.cpp:792][ManageAgent][][][][][][][Received message KEY_UPDATE_LAST.]
[11/13/2018][15:48:15][3896][1332][CSmAdminManager.cpp:816][ManageAgent][][][][][][][Received message KEY_UPDATE_CURRENT.]
[11/13/2018][15:48:15][3896][1332][CSmAdminManager.cpp:766][ManageAgent][][][][][][][Received message KEY_UPDATE_NEXT.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Resource Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Session Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Response Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Session Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmProtectionManager.cpp:90][CSmProtectionManager::Initialize][][][][][][][ProtectionManager initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Credential Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Challenge Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Response Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Session Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmAuthenticationManager.cpp:87][CSmAuthenticationManager::Initialize][][][][][][][AuthenticationManager initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Response Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmManager.cpp:67][Variable Manager][][][][][][][Initialized.]
[11/13/2018][15:48:15][3896][1332][CSmAuthorizationManager.cpp:89][CSmAuthorizationManager::Initialize][][][][][][][AuthorizationManager initialized.]
[11/13/2018][15:48:15][3896][1332][CSmHighLevelAgent.cpp:190][Initialize][][][][][][][High Level Agent Initialized.]
[11/13/2018][15:48:15][3896][1332][SmIIS60Filter.cpp:473][CSmIIS60Filter::Init][][][][][][][SiteMinder IIS 6.0 Filter starting...]
[11/13/2018][15:48:15][3896][1332][SmIIS60Filter.cpp:474][CSmIIS60Filter::Init][][][][][][][Opened registry key 'SOFTWARE\Netegrity\SiteMinder Web Agent\Microsoft IIS'.]
[11/13/2018][15:48:15][3896][1332][SmIIS60Filter.cpp:475][CSmIIS60Filter::Init][][][][][][][Accessed registry value 'configfile' = 'D:\Program Files\CA\webagent\win64\bin\IIS\WebAgent.conf']
[11/13/2018][15:48:15][3896][1332][SmIIS60Filter.cpp:489][CSmIIS60Filter::Init][][][][][][][Opened registry key 'SYSTEM\CurrentControlSet\Services\W3SVC\Parameters']
[11/13/2018][15:48:15][3896][1332][SmIIS60Filter.cpp:494][CSmIIS60Filter::Init][][][][][][][Accessed registry value 'MajorVersion' = '8']
[11/13/2018][15:48:15][3896][1332][SmIIS60Filter.cpp:505][CSmIIS60Filter::Init][][][][][][][Siteminder IIS 6.0 Filter initialized in ISAPI Extension mode.]
[11/13/2018][15:48:15][3896][1332][CSmHighLevelAgent.cpp:321][ProcessRequest][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][][][][][][Start new request.]
[11/13/2018][15:48:15][3896][1332][CSmResourceManager.cpp:75][CSmResourceManager::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][][][][][][Calling SM_WAF_HTTP_PLUGIN->ProcessResource.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:399][CSmHttpPlugin::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][][][][][][Resolved HTTP_HOST: 'mdlappwf.company.com'.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:5249][Entered CSmHttpPlugin::ResolveFQServerName sHost: ][][][][][][][mdlappwf.company.com]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:490][CSmHttpPlugin::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][][][][][][Resolved hostname: 'mdlappwf.company.com'.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:509][CSmHttpPlugin::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][][][][][][Resolved agentname: 'CAR1252SP1-Agent-mdlappwfweb1-appwf'.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:5602][CSmHttpPlugin::ResolveClientIp][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][][][CAR1252SP1-Agent-mdlappwfweb1-appwf][][][Resolved Client IP address '10.11.75.102'.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:657][CSmHttpPlugin::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][][][Resolved URL: '/webworkflow/appwf/default.aspx'.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:781][CSmHttpPlugin::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Resolved METHOD: 'GET'.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:834][CSmHttpPlugin::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Resolved cookie domain: '.company.com'.]
[11/13/2018][15:48:15][3896][1332][CSmResourceManager.cpp:112][CSmResourceManager::ProcessResource][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][SM_WAF_HTTP_PLUGIN->ProcessResource returned SmSuccess.]
[11/13/2018][15:48:15][3896][1332][CSmSessionManager.cpp:82][CSmSessionManager::EstablishSession][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Calling SM_WAF_HTTP_PLUGIN->EstablishSession.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:6696][CSmHttpPlugin::ProcessSessionCookie][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Unable to decode SMSESSION cookie.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:2224][CSmHttpPlugin::EstablishSession][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Unable to process SMSESSION cookie.]
[11/13/2018][15:48:15][3896][1332][CSmSessionManager.cpp:126][CSmSessionManager::EstablishSession][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][SM_WAF_HTTP_PLUGIN->EstablishSession returned SmNoAction.]
[11/13/2018][15:48:15][3896][1332][CSmLowLevelAgent.cpp:503][IsResourceProtected][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Resource is protected from Policy Server.]
[11/13/2018][15:48:15][3896][1332][CSmResponseManager.cpp:193][ProcessResponses][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Calling SM_WAF_HTTP_PLUGIN->ProcessResponses.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:2824][CSmHttpPlugin::ProcessResponses][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Processing IsProtected responses.]
[11/13/2018][15:48:15][3896][1332][CSmResponseManager.cpp:231][ProcessResponses][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][SM_WAF_HTTP_PLUGIN->ProcessResponses returned SmSuccess.]
[11/13/2018][15:48:15][3896][1332][CSmCredentialManager.cpp:132][CSmCredentialManager::GatherCredentials][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Calling SM_WAF_HTTP_PLUGIN->ProcessCredentials.]
[11/13/2018][15:48:15][3896][1332][CSmCredentialManager.cpp:176][CSmCredentialManager::GatherCredentials][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][SM_WAF_HTTP_PLUGIN->ProcessCredentials returned SmNoAction.]
[11/13/2018][15:48:15][3896][1332][CSmHighLevelAgent.cpp:583][ProcessRequest][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][CredentialManager returned SmNo or SmNoAction, calling ChallengeManager.]
[11/13/2018][15:48:15][3896][1332][CSmChallengeManager.cpp:105][CSmChallengeManager::DoChallenge][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Calling SM_WAF_HTTP_PLUGIN->ProcessChallenge.]
[11/13/2018][15:48:15][3896][1332][CSmHttpCredCore.cpp:1680][CSmHttpCredCore::DoFormsChallenge][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Executing forms challenge.]
[11/13/2018][15:48:15][3896][1332][CSmHttpCredCore.cpp:1973][CSmHttpCredCore::DoFormsChallenge][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Redirecting to credential collector 'https://securemdl.company.com/publicsite/default.asp?TYPE=33554433&REALMOID=06-e5f5be59-dade-4f55-a9bc-2f141db641f1&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=-SM-pstcxquGRLsAOPwfZWCkio5qL1LN4En%2fxwSpDMj2xXl51j49G4u%2fKwb8goEqGU5iHqRcsBNP38wkvVGGHntGkJNXHtS4PbPW&TARGET=-SM-http%3a%2f%2fmdlappwf%2ecompany%2ecom%2fwebworkflow%2fappwf%2fdefault%2easpx'.]
[11/13/2018][15:48:15][3896][1332][SmPluginUtilities.cpp:405][HandleCredCollectorChallenge][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Redirecting for credentials 'https://securemdl.company.com/publicsite/default.asp?TYPE=33554433&REALMOID=06-e5f5be59-dade-4f55-a9bc-2f141db641f1&GUID=&SMAUTHREASON=0&METHOD=GET&SMAGENTNAME=-SM-pstcxquGRLsAOPwfZWCkio5qL1LN4En%2fxwSpDMj2xXl51j49G4u%2fKwb8goEqGU5iHqRcsBNP38wkvVGGHntGkJNXHtS4PbPW&TARGET=-SM-http%3a%2f%2fmdlappwf%2ecompany%2ecom%2fwebworkflow%2fappwf%2fdefault%2easpx'.]
[11/13/2018][15:48:15][3896][1332][CSmChallengeManager.cpp:124][CSmChallengeManager::DoChallenge][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][SM_WAF_HTTP_PLUGIN->ProcessChallenge returned SmExit.]
[11/13/2018][15:48:15][3896][1332][CSmHighLevelAgent.cpp:607][ProcessRequest][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Challenge Manager returned SmExit, end new request.]

AroraSAroraS

As I suspected we need to investigate the KEYS in KStore. Please take an export of the KEYS from KStore using smkeyexport.sh and see how many keys are present. We should have only 4 Agent Key Markers.

[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:6696][CSmHttpPlugin::ProcessSessionCookie][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Unable to decode SMSESSION cookie.]
[11/13/2018][15:48:15][3896][1332][CSmHttpPlugin.cpp:2224][CSmHttpPlugin::EstablishSession][00000000000000000000000001000000-0f38-5beb469f-0534-01af07d3][*10.11.75.102][][CAR1252SP1-Agent-mdlappwfweb1-appwf][/webworkflow/appwf/default.aspx][][Unable to process SMSESSION cookie.]

Reference Links :

Tech Tip : CA Single Sign-On : Data Protection, Key Management,Configuration & Common Issues 

How to Clean up a SiteMinder Key Store? - CA Knowledge 

Tech Tip : CA Single Sign-On:: Policy Server : Best practice on importing Agent Keys 

• Are we using STATIC Keys OR Dynamic Keys ?
• Do we know which WebAgent generated SMSESSION and which WebAgent is try to consume (SSO into) that SMSession ? It would be also beneficial to identify which Policy Server Infrastructure (i.e. Old OR New), these WebAgents are connected to.

1. How many Keys are present

--- Correct, 4 Agent Key Markers are in the keystore

2. Are we using STATIC Keys OR Dynamic Keys ?

---DYNAMIC

3. Do we know which WebAgent generated SMSESSION and which WebAgent is try to consume (SSO into) that SMSession ? It would be also beneficial to identify which Policy Server Infrastructure (i.e. Old OR New), these WebAgents are connected to.

---Yes, SMSESSION is generated by securemdl.company.com web site and is being consumed by mdlappwf.company.com. All the web agents have new policy servers in their SMHost and also the existing HCO object in SMHost files have new policy servers.

Also the Policy Store is not yet upgraded and new Policy Servers are still running in compatibility Mode while connecting to Policy/Key Store. Do you think updating the Policy/Key Store as part of 12.72 upgrade is required and will resolve the issue?