Symantec Access Management

Hey,

Have a tough question.  We recently upgraded siteminder from 12.51 to:

FullVersion=12.52.105.2112

This is running on a Redhat linux server.  CA Directory is our policy store, MS-SQL is our user store.  Most things run fine.  We a custom policy written in java that is used for certain Realms.  If one of those realms is called, the policy server thread will die and get restarted.  Below is what we get at the end of the smtracedefault.log.  We have sent a core dump to CA support and they are looking into it.

It is like the thread dies each time it tries to spawn off a new JVM thread.  Our java code is unchanged and works fine on our other 12.51 server.

Anyone have any clue?

thanks

chad 

[05/26/2016][15:53:33.590][15:53:33][17749][4005206896][SmAuthorization.cpp:809][CSmAz::TestPolicy][][][][][][][AAFP][][AAFP Member/AAFP Staff][][][][][][][][][][][][Evaluating policy...]

[05/26/2016][15:53:33.590][15:53:33][17749][4005206896][SmAuthorization.cpp:2218][CSmAz::ProcessActiveExpression][][][][][][][][][][][][][][][][][][][][][Enter function CSmAz::ProcessActiveExpression]

[05/26/2016][15:53:33.590][15:53:33][17749][4005206896][SmActiveExpr.cpp:501][CSmActiveExpr::GetActiveValue][][][][][][][][][][][][][][][][][][][][][Enter function CSmActiveExpr::GetActiveValue]

[05/26/2016][15:53:33.590][15:53:33][17749][4005206896][SmActiveExpr.cpp:382][CSmActiveExprLibrary::Lookup][][][][][][][][][][][][][][][][][][][][][Enter function CSmActiveExprLibrary::Lookup]

[05/26/2016][15:53:33.590][15:53:33][17749][4005206896][SmActiveExpr.cpp:247][CSmActiveExprLibrary::Init][][][][][][][][][][][][][][][][][][][][][Enter function CSmActiveExprLibrary::Init]

[05/26/2016][15:53:33.591][15:53:33][17749][4005206896][SmJVMSupport.cpp:251][GetJVMEnv][][][][][][][][][][][][][][][][][][][][][SmJVMSupport: JVM library loaded sucessfully]

[05/26/2016][15:53:33.592][15:53:33][17749][4005206896][SmJVMSupport.cpp:255][][][][][][][][][][][][][][][][][][][][][][LogMessage:INFO:[sm-JavaApi-01030] SmJVMSupport: Using the following JRE: /usr/java/jdk1.7.0_79/jre]

[05/26/2016][15:53:33.592][15:53:33][17749][4005206896][SmJVMSupport.cpp:260][][][][][][][][][][][][][][][][][][][][][][LogMessage:INFO:[sm-JavaApi-01040] SmJVMSupport: Loaded the following JVM library: /usr/java/jdk1.7.0_79/jre/lib/i386/server/libjvm.so]

[05/26/2016][15:53:33.592][15:53:33][17749][4005206896][SmJVMSupport.cpp:287][GetJVMEnv][][][][][][][][][][][][][][][][][][][][][SmJVMSupport: Attempting to retrieve existing JVM instance]

[05/26/2016][15:53:33.592][15:53:33][17749][4005206896][SmJVMSupport.cpp:300][GetJVMEnv][][][][][][][][][][][][][][][][][][][][][SmJVMSupport: Attempting to create new JVM instance]

Hi Philips,

This looks very similar to one of the issue I was working with very recently.

Can you please confirm if you have an integration with CAWily Agent  (Introscope)?

One easiest way to identify the problem with JVM loading is , start Policy server manually by executing "smpolicysrv" executable from the bin folder.

If it encounters any error during initialization ,the same would be printed in the std out.

However, please note Policy server does NOT initialize JVM until it receives a request for any custom java code (e.g custom authscheme/expression/policy etc).

Regards,

Ujwol

Can you also provide the ca support ticket # that you raised for this issue ? I can have a quick look at the logs and see if matches the issue that I was working.

Thank you for the reply.  I don't believe we have an integration with CAWily Agent.  (what is the best way to check?)

I started the policy server manually.  As soon as I ran a test that would call the custom java code, it died.  The output is below.

[smuser@SM1-DEV bin]$ ./smpolicysrv

Segmentation fault (core dumped)

I will get the number of the CA support ticket, we sent a copy of the core dump to CA support.  (I am working with a consulting company and the ticket is through them, so I don't have the number myself).

thanks

chad

Hi,

I think I am seeing the exact same behavior with my new setup for r12.52 SP1 CR5 on REHL 6.  It appears that as soon as we make a SAML federation service request or any other request that requires loading of the JVM, the policy server will then load/initialize the Java JVM and will then crash/terminate the policy server thread and it will then restart itself.  When this happens the agent.log for all of the web agents will showed that it had lost connection to the policy server.  The only thing that shows up in they smps.log and smtracedefault.log is the policy server's last task, which is loading the Java JVM:

smps.log:

[SmJVMSupport.cpp:255][INFO][sm-JavaApi-01030] SmJVMSupport: Using the following JRE: /usr/pservices/ca/jdk1.7.0_79/jre

[SmJVMSupport.cpp:260][INFO][sm-JavaApi-01040] SmJVMSupport: Loaded the following JVM library: /usr/pservices/ca/jdk1.7.0_79/jre/lib/i386/server/libjvm.so

smtracedefault.log:

[SmJVMSupport: JVM library loaded sucessfully][][]

[LogMessage:INFO:[sm-JavaApi-01030] SmJVMSupport: Using the following JRE: /usr/pservices/ca/jdk1.7.0_79/jre][][]

[LogMessage:INFO:[sm-JavaApi-01040] SmJVMSupport: Loaded the following JVM library: /usr/pservices/ca/jdk1.7.0_79/jre/lib/i386/server/libjvm.so][][]

[SmJVMSupport: Attempting to retrieve existing JVM instance][][]

[SmJVMSupport: Attempting to create new JVM instance][][]

Here's the thread to my other CA Community post on this issue so hoping we can figure this out:

smps.log on newly installed r12.52 SP1 CR05 policy servers indicates that policy server is restarting by itself periodically

Thank you!

Since the description note says "Our java code is unchanged and works fine on our other 12.51 server.", which means you have not re-compiled the java code using new upgrade SDK for version 12.52.105.2112.

API changes from version to version, if I recall, we added a note to support matrix, specifically encourage SDK users to upgrade their SDK kit and recompile program to match with policy server version, to avoid unexpected problems, sometimes performance related, other times crashing too.

It is very possible due to api logic change, whenever a function was called, now no longer working and resulted program crash.

Another aspect is looking for any realm object corruption related to store object changes, double check if same java program works in other domain/realms. What is different in this particular realm comparing with others.

Thanks,

Hongxu