Symantec Access Management

Hi All, 

    We recently configured SiteMinder 12.5 on RHEL 6.6 machine (after OS upgrade from RHEL 5.11 to RHEL 6.6) and interestingly there are no errors getting logged in smps and smtracedefault logs after the configuration for the recent 2 servers.

  We have been configuring the same SiteMinder version on many OS upgraded RHEL 6.6 machines in past 2 months having identical configuration. We always used to observe the same generic errors logged even after the fresh install.

For the recent 2 machines, smps.log logged the smpolicysrv process start and after then it is no more logging anything (file size is 0 everyday) and smtracedefault not logging any error at all. No auto-restarts logged ib smexec log either. This situation looks too good to be of any use and I suspect there could be an underlying issue. Wondering if anyone had incurred this situation ever.  

Hi Achyut ,

What is your log rollover policy like ?

Can you provide the content of all entries under following registry :

HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\LogConfig

Regards,

Ujwol

Hi Ujwol,

     Kindly provide the the path for that file in linux installation please.

Thanks,

Achut

<ps_install_path>/siteminder/registry/sm.registry

I hate when I type too fast...

<ps_install_path>/ps/registry/sm.registry

my apologies

Hi Samat,

Thank you. My bad ! I saw that windows path and didn't notice the registry being mentioned. After configuring SiteMinder, we used to copy this sm.registry file from the already properly configured machine before starting the smpolicysrv process. I assure this registry file is correct and properly configured.

Please find it's contents below:

Thanks,

Achut

No placed to start except at the beginning...

You are starting policy server with 'start-all' and as user 'smuser' (or some other non-prov account)?

Can you provide output from:

ls -l /opt/ca/siteminder/start-all

Also verify that entire path to /opt/ca/siteminder/log is either owned by smuser (or equivalent) or world permissions allow write.

Hi Samat,

Sorry for the delayed response. To clarify, smps.log is getting generated everyday as expected but not getting updated with anything else other than the expected logging of process stop or start logs, whenever there is a smpolicysrv process stop or start.

Yes, we start policy server with 'start-all' and as 'smuser12' (belongs to smuser group), which is same ID we use to start in other functional systems as well, having the required privilges.

$ ls -l /opt/ca/siteminder/start-all

-rwxr-xr-x 1 smuser12 smuser 2225 Jun  7 14:01 /opt/ca/siteminder/start-all

The content is verified and is same as that of the functional machine.

  And yes, the entire path to /logs/ca/smpolicy/logs is owned by smuser12 (smuser equivalent). We have validated the application functionality with this machine and everything seems to be working fine, but still there is unprecedented drastic consequences once we deploy production VMs using this machine as template and being exposed to live production (user) traffic. This is just been never observed before and smps.log used to be never empty.

Thanks,

Achut

OK.  Next I would do an lsof on the smpolicysrv process to verify that the smps.log file is opened by the process.  To do this, get the PID of the smpolicysrv process and enter this command on the server:

lsof -p <PID>

You will get a list of all open files for that process.  Yet again another sanity check.

Hi Samat,

Please find the output for lsof -p <PID of smpolicysrv process>:

Thanks,

Achut

So I see you are logging to /logs/ca/smpolicy/smps.log but you've defined in the registry the path to be /opt/ca/siteminder/log/smps.log.  Assuming a symbolic link somewhere.  Verify the path for /opt as well as /logs is owned and writable by smuser12.  Verify the actual symbolic link is writable by smuser12.

Hi Samat,

Yes, we have softlink from: /opt/ca/siteminder/log  to  /logs/ca/smpolicy/ , to ensure all logs are collected in /logs path on our servers.

And in both paths, they are under smuser12 ownership and appropriate permissions.

Thanks,

Achut

  Also it's way too weird that there are no logging of any handshake errors at all on this new machine, while the old one we used as a reference to configure everything on this machine has handshake errors getting logged continuously.

Another newly configured machine in lower environment has been continuously logging handshake errors while there are no webagent requests at all in smaccess.log . This is even more spookier as well. Getting no conclusion regarding from where these handshake errors are arising from and tried to check from CA discussions but couldn't reach to RCA. Guidance requested for troubleshooting.

[14237/4087053168][Mon Jun 20 2016 15:42:59][CServer.cpp:2138][ERROR][sm-Server-01070] Failed handshake with 139.71.220.5:55651

[14237/4045093744][Mon Jun 20 2016 15:43:02][CServer.cpp:1966][ERROR][sm-Tunnel-00010] Bad security handshake attempt. Handshake error: 3159

[14237/4045093744][Mon Jun 20 2016 15:43:02][CServer.cpp:1971][ERROR][sm-Tunnel-00020] Handshake error: Failed to receive client hello. Client disconnected

Thanks,
Achut

Hi Achut,

I think at this point in time , this issue needs some advanced troubleshooting so would be good if you could create a CA support case.

If you are available right now, I can probably have a remote session and assist you ?

Regards,

Ujwol Shrestha

Hello Ujwol,

Sounds Perfect ! I will get a case assigned right away. Yes, I would be available for few more hours for the troubleshooting session.

Thanks,
Achut

Hi Ujwol,

  Have got CA case# 436963 assigned for the mentioned issues. Kindly forward the remote session details to my contact email address mentioned in the case, if currently feasible.

Thanks,

Achut

Hi Achut,

Thank you for your time today to troubleshoot the case, here is the summary of action we performed to confirm that the Policy server is working as expected.

-We first restarted Policy server and ensured that the Policy server startup message is logged properly in the smps.log.

This includes messages about Policy Server Version, TIme Zones, Ports, Policy Store details etc and the fact that how many Policy store objects are loaded from the policy store.

As, all of these were logged , there didn't seem to be any problem logging the startup messages in the smps.log

- Next, we tested whether Policy server is able to print the statistics message in the smps.log. For this, we first sourced the Policy server environment script (ca_ps_env.ksh) and then from the policy server bin folder executed the command :

./smpolicysrv -stats

When this command was executed, it updated the smps.log with the Policy server statistics information as expected.

- As this was a production environment , we were not able to perform any negative scenario testing to see if the Policy server is able to log error condition in the smps.log, but given the fact that Informational messages are logged with no issue, I don't see any reason why it won't be able to run the error messages.

-For your information, Policy server log (smps.log) is designed to log only messages related to Policy server status (e.g startup, statistics, or any error condition specific to it being unable to connect to any components like policy store/user store/webagents etc.) It is not supposed to log any error messages related to user transaction like user not being authenticated/not authorized etc.

-Next ,we verified the policy server trace log (smtracedefault.log) and it was logging in the all the user/agent transacations like IsProtected/IsAuthenticated/IsAuthorized etc. So this also didn't seem to have any issue logging in the messages. As this Policy server is currently configured with only one webagent/webserver, there were not that many request coming through. But once this is configured in your live environment and with multiple web agents, you are expected to see some failed user transactions like authentication/authorization failures etc. Bug again, as this was your production environment we weren't' able to perform any negative scenario testing.

So ,in conclusion, as per the testing we performed during the remote session , I do not see any problem related to logging in this particular setup.

As per your consent, we then finally agreed to mark this case/thread as resolved.

Cheers,

Ujwol Shrestha

Ujwol's Single Sign-On Blog

Hi Ujwol,

Thanks a ton for your time and quick resolution for all the posted the queries.

Thanks,
Achut

Hello,

You can try to run a strace on the PS process to track system call and check why no logs are generated. Maybe not enough permissions... Are you running PS as root ? With the user that you installed with ?

You can view the /var/log/messages if there is anything recorded. Do you have SE linux enabled ?

Hope it helps,

Julien.

Hi Julein,

  Our current server administrator is unaware of running strace and we do not have the application expert who can analyze strace output. We are running PS using privileged service ID that has privileges lesser than root user. It's the same way configured on all other functional systems. We have SE Linux disabled on our machines. Currently trying to get the messages file from our server administrator to check if anything got logged.

Thanks,

Achut