Symantec Access Management

Hi,

I have r12.52 SP1 PS and CA Directory r12 SP14 as the policy store, both running on the same machine on Red Hat Enterprise Linux

Everything is humming along nicely, but then suddenly I see these errors in the smps.log

[SmObjProvider.cpp:187][ERROR][sm-Server-03090] Policy store failed operation 'Search'  for object type 'AgentCommand' . LDAP Error Doing AgentCommand_Search: 82: Local error

[SmObjProvider.cpp:187][ERROR][sm-Server-03090] Policy store failed operation 'Search'  for object type 'Property' . LDAP Error Doing Property_Search: 82: Local error

[SmObjProvider.cpp:187][ERROR][sm-Server-03090] Policy store failed operation 'Search'  for object type 'ServerCommand' . LDAP Error Doing ServerCommand_Search: 82: Local error

[SmObjStore.cpp:857][ERROR][sm-log-00000] Failure Returned in calling CSmObjStore::Search() while processing policy store journal commands.

[SmPolicyServer.cpp:1759][ERROR][sm-Server-00620] Exception in JournalThread. Text: Policy store failed operation 'CleanServerCmds' for object type 'Policy store provider'. LDAP Error Doing ServerCommand_Search: 82: Local error

[XPSLDAP.cpp:1651][ReadOneParameter][ERROR][sm-xpsxps-01080] Error occurred during "index search" for "xpsParameter=CA.SM::$AgentConnectionMaxLifetime,ou=XPS,ou=policysvr4,ou=siteminder,ou=netegrity,ou=smpolicystore,o=spe,c=US", text: Local error

[SmObjProvider.cpp:187][ERROR][sm-Server-03090] Policy store failed operation 'Search'  for object type 'Response' . LDAP Error Doing Response_Search: 82: Local error

[SmObjProvider.cpp:187][ERROR][sm-Server-03090] Policy store failed operation 'Search'  for object type 'ResponseGroup' . LDAP Error Doing ResponseGroup_Search: 82: Local error

[SmAdapterObject.cpp:1265][Create][ERROR][sm-xadobj-00110] CA.SM::Response@07-000428b0-7ae5-14ee-9a35-c414adfbb05d: Create failed. (Unknown Failure)

[SmStore.cpp:326][Create][ERROR][sm-xobsm-00960] Failed to create object. (CA.SM::Response@07-000428b0-7ae5-14ee-9a35-c414adfbb05d(ldap response))

[XPSIO.cpp:1578][CreateObject][ERROR][sm-xpsxps-00540] Previous error occurred on object "CA.SM::Response@07-000428b0-7ae5-14ee-9a35-c414adfbb05d(ldap response)"

[XPSPolicyData.cpp:1238][CommitOrTestRollback][ERROR][sm-xpsxps-00740] XPS Transaction COMMIT has failed.

[XPSPolicyData.cpp:409][CreateOrUpdateImpl][ERROR][CA-SM-Assert] Assert failed: Commit()

[XPSSvcHandlerManageObjects.cpp:249][ProcessRequest][ERROR][sm-xpssvc-00650] Failed Create Operation

[XPSSvc.cpp:158][InvokeHandler][ERROR][sm-xpssvc-00020] Cannot Process the Request.

[XPSLDAP.cpp:1651][ReadOneParameter][ERROR][sm-xpsxps-01080] Error occurred during "index search" for "xpsParameter=CA.SM::$AgentConnectionMaxLifetime,ou=XPS,ou=policysvr4,ou=s

It seems as though suddenly it can't access the policy store? At this point some web agents start failing and throw up the internal server error.

The policy store is running on the same box as the policy server, so it's not like there is a network link that would fail.

It either resolves itself or I have to restart the policy server to resolve the issue. But, when I launch a test connection through the smconsole, the test connection is always successful.

Has anyone seen this before?

Regards,

Anand.

Anand anand3g

Is this a brand new setup? What is the number of Objects in the Policy Store? Am wondering is this probably an indexing OR too many objects issue.

Did you check the corresponding TRANSACTIONAL log on DSA side to see what errors have been recorded.

There are a few tuning parameters in CA Directory which we could play with. However unless we know what is the core issue; we'd just be shooting in dark by flipping random parameters.

• Indexing.
• Max Search Entries : set max-op-size = 100;  [Returns only 100 entries if search result is more than 100 and then throws up an error in CA Directory logs].
• Max Search Time : set max-op-time = 60;

Regards

Hubert

Yes it's a relatively new setup. There about 5000 objects in the directory. At least that's what the policy server says when it does a bulk fetch upon starting.

Are there any ideal settings for the op size and look through limits?

Regards,

Anand.

Sent from my HTC

If this environment is non-production then try setting the values as below and test. Again as I am re-iterating; there are parameters we could play with. However without finding the right reasoning from logs and correlating; it would a blind guestimate and hope.

set max-op-size = 2000;

set max-op-time = 60;

Restart your DSA Server and try.

Regards

Hubert

Hi Dennis,

We are also getting the same Local error 82 with CA Directory as policy store.

I have checked the above mentioned parameters in our setup, they seem to be good to me as per the recommendations given.  Still the error persists.

Any other inputs / thoughts would be appreciated.

Regards

Sandeep

Sandeep SKhurana

I would typically start debugging by comparing Policy Server Trace Logs and DSA Transaction Logs, this is the first place to start comparing both product logs.

Meanwhile a quick search yeilded these results.

• LDAP Error Messages
• RE: LDAP Error 82: local Error

Could we check if both Policy Server and DSA are in time sync.

Could we check what are you using to connect to DSA (using FQDN or IP Address in Policy Server side configuration? If using FQDN, could we flip to IP Address and test).

Regards

Hubert.

After digging around a little more, I found these notes. However these are for ODBC calls; nevertheless what caught my eye is the error itself. May be the error is common, however the representation in LDAP and ODBC may different. So another avenue to investigate.

• Unexpected local error code -1063 while making database call (Legacy_Onyx KB Id: 182687)
• Policy store failed operation 'CleanServerCmds' Error code -1063.
• SM6 Policy Server: Unexpected local error code -1063 while making database call. Error code -1063 (Legacy_Onyx KB Id: 19…

All of the above point to several things in common.

1. Policy Store error and error definition looks similar.
2. They all speak about ServerCommandTimeDelay.
   • Re: Using ServerCommandTimeDelay and/or ServerCmdMsec in R12.5

Could we check if all of your server's participating in the Solution are in time sync (be it different / multiple Policy Server's OR Policy Store's).

Regards

Hubert

The times are in sync. The policy store is on the same machine as the

policy server.

One thing I notice is that I have white hat scanning a lot of my web agents

at that time.

White hat I assume makes a request by manipulating the agentname parameter

in the URL.

Would a non existent agent name query to the policy store cause these error

messages?

Regards,

Anand.

On Mon, Mar 2, 2015 at 10:41 AM, HubertDennis <

Ideally a non existent agent is like an untrusted host. You should see bad handshake errors. I have never seen non existant webagents polls causing directory issues. They just fill up the smps.log with bad handshake errors, because the Policy server does not trust the incoming request.

Back to my very first question, Have we checked the corresponding DSA Transactional and Error Logs to see what the DSA is doing at the very moment?

Regards

Hubert.

anand3g - Hello, by any chance, did you get to the bottom of these errors?

Any root cause on this issue ? I am seeing similar errors

I encountered similar issue with policy server version  R12.52 sp1 cr6 and policy store is Sun-Directory-Server/11.1.1.7.2 B2014.1030.2115(ODSEE 11gr1 ), Did re-index of policy store as well , any leads ?

[5291/-143206688][Tue Jan 03 2017 09:51:41][XPSLDAP.cpp:799][CreateRoot][INFO][sm-xpsxps-01160] LDAP Provider Info String = Sun-Directory-Server/11.1.1.7.2 B2014.1030.2115
[5291/-143206688][Tue Jan 03 2017 09:51:41][XPSIO.cpp:465][InitialLoad][INFO][sm-xpsxps-00560] Database Transactions are 0.
[5291/-143206688][Tue Jan 03 2017 09:51:41][XPSIO.cpp:492][InitialLoad][INFO][sm-xpsxps-00300] 1 Parameter(s) loaded from Policy Store, 1 total.
[5291/-143206688][Tue Jan 03 2017 09:51:41][XPSIO.cpp:497][InitialLoad][INFO][sm-xpsxps-00330] Caching Policy Data...
[5291/-143206688][Tue Jan 03 2017 09:51:42][Reference.cpp:441][ReferenceImpl][ERROR][CA-SM-Assert] Assert failed: String && *String
[5291/-143206688][Tue Jan 03 2017 09:51:42][Reference.cpp:441][ReferenceImpl][ERROR][CA-SM-Assert] Assert failed: String && *String
[5291/-143206688][Tue Jan 03 2017 09:51:42][Reference.cpp:441][ReferenceImpl][ERROR][CA-SM-Assert] Assert failed: String && *String

For anyone stubmling upon this one, this is a known netowork corruption issue with CA directory 12.0.14 until 12.0.17 

This issue is fixed in CA Directory 12.0.17.CR1

More here :

Seeing many "Policy store failed operation 'MultipleSearch' errors in the SMPS.log with R12.52 SP1 Policy Server. 

Symptoms:
Policy Server reports Error 82 during cache rebuilds:

[17987/4011699056][Fri Oct 16 2015 08:45:01][SmObjProvider.cpp:187][ERROR][sm-Server-03090] Policy store failed operation 'MultipleSearch' for object type 'UserDirectory' . LDAP Error Doing UserDirectory_Fetch: 82: Local error

These errors are encountered on other object types as well such as "UserDirectory", "TrustedHost", "PropertyCollection", and "ServerCommand" to name a few.

Environment:

• SiteMinder Policy Server : R12.52 SP1 
• Policy Store : CA Directory version >= R12.0.14 and < R12.0.17 CR1

Cause:

CA directory has setting dxgrid-queue and the issue may occur when this setting set to true. Pre-SP14, it was set to ‘false’ by default, Post-SP14, it is set to ‘true’ by default. 

These failed searches are the result of a packet/memory corruption issue in CA Directory R12.0.14 through R12.0.17 with 'set dxgrid-queue=true' (default).

This error is reported by LDAP SDK on the Policy Server side due to malformed packet received from CA Directory.

Resolution:

This issue is resolved in CA Directory R12.0.17 CR-01.  So, to resolve this issue and still be able to use the dxgrid-queue upgrade CA Directory to version R12.0.17 CR-01 or later.  

Workaround:

Disable dxgrid-queue by running following command:

set dxgrid-queue=false

However, disabling the dxgrid-queue comes with the penalty of loosing the following benefits which comes with the dxgrid-queue :

• Improves performance of concurrent search and update requests.
• Allows abandoning of searches that are not performed yet (due to reasons such as client disconnect or timeout).
• Increases thread utilization, thus allowing better throughput.
• Allows the set interrupt-searches = true|false; command to be used to prevent long running searches blocking updates. See the set interrupt-searches command for more details.

And, from CA Directory release notes fro 12.0.17 CR1:

Response Network Packet Corruption

Symptom:

In the following instances, response network packet corruption may occur:

1. When the DSA sends a result that is blocked or partially sent, a trace message "comms_send: combining buffer" is present along with the result.
2. When handling multiple search requests from the same LDAP client, the search response may be corrupted when one of the search results contains more than 200 entries (depending on size of information being returned).

 Both these cases will cause the LDAP client to reject the response as invalid ASN.1.

Solution:

This issue is fixed.