Hello ,
We are facing lot of issues with our CA IDM setup. We have situation where 50% of the task that are submitted are going in progress.
Please find below the steps that we have followed in order to resolve them:-
1. We tried to restart the server to see if the task start to process themselves, but it did not help.
2. We cleared our notify database/queue.
3. We have replaced the Task persistence database.
All these steps did not help us in resolving the issue.
We took thread dump from the server and it looks like below :-
Thread 242: (state = IN_NATIVE)
- java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
- java.net.SocketInputStream.read(byte[], int, int, int) @bci=87, line=152 (Compiled frame)
- java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=122 (Compiled frame)
- com.microsoft.sqlserver.jdbc.TDSChannel.read(byte[], int, int) @bci=7 (Compiled frame)
- com.microsoft.sqlserver.jdbc.TDSReader.readPacket(int) @bci=107 (Compiled frame)
- com.microsoft.sqlserver.jdbc.TDSReader.readPacket() @bci=2 (Compiled frame)
- com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(boolean) @bci=229 (Compiled frame)
- com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd) @bci=69 (Compiled frame)
- com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute() @bci=5 (Compiled frame)
- com.microsoft.sqlserver.jdbc.TDSCommand.execute(com.microsoft.sqlserver.jdbc.TDSWriter, com.microsoft.sqlserver.jdbc.TDSReader) @bci=11 (Compiled frame)
- com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(com.microsoft.sqlserver.jdbc.TDSCommand) @bci=42 (Compiled frame)
- com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(com.microsoft.sqlserver.jdbc.TDSCommand) @bci=32 (Compiled frame)
- com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(com.microsoft.sqlserver.jdbc.TDSCommand) @bci=19 (Compiled frame)
- com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeQuery() @bci=15 (Compiled frame)
- org.jboss.jca.adapters.jdbc.WrappedPreparedStatement.executeQuery() @bci=37, line=462 (Compiled frame)
- com.netegrity.ims.persistence.DBDAOFactory.fetchObjectById(java.sql.Connection, java.lang.String, int, boolean) @bci=129, line=115 (Interpreted frame)
- com.netegrity.ims.persistence.DBDAOFactory.getObjectById2(java.sql.Connection, java.lang.String, int, boolean) @bci=25, line=158 (Interpreted frame)
- com.netegrity.ims.persistence.PersistenceProvider.getTaskSessionFromResultSet(java.sql.Connection, java.sql.ResultSet) @bci=85, line=3470 (Interpreted frame)
- com.netegrity.ims.persistence.PersistenceProvider.getTaskSession(java.lang.String) @bci=95, line=312 (Interpreted frame)
- com.netegrity.ims.IMSEvent.getTaskSession() @bci=16, line=441 (Interpreted frame)
- com.netegrity.ims.events.IMTaskEvent.getTaskSession() @bci=1, line=32 (Interpreted frame)
- com.ca.identitymanager.policyxpress.plugins.actions.PxActionUserAttribute.getModifyUserEvent(com.netegrity.imapi.EventContext, java.lang.String, java.util.List) @bci=235, line=220 (Interpreted frame)
- com.ca.identitymanager.policyxpress.plugins.actions.PxActionUserAttribute.getActionEvent() @bci=107, line=169 (Interpreted frame)
- com.ca.identitymanager.policyxpress.engine.flow.FlowAction.event(com.ca.identitymanager.policyxpress.context.PxContext, com.ca.identitymanager.policyxpress.plugins.actions.PxActionHandler, com.ca.identitymanager.policyxpress.engine.flow.PxFlowManager) @bci=1, line=70 (Interpreted frame)
- com.ca.identitymanager.policyxpress.engine.flow.FlowAction.runTask(com.ca.identitymanager.policyxpress.context.PxContext, com.ca.identitymanager.policyxpress.engine.flow.PxFlowManager) @bci=76, line=48 (Interpreted frame)
- com.ca.identitymanager.policyxpress.engine.PxEventController.executeEvent(com.ca.identitymanager.policyxpress.context.PxContext, com.ca.identitymanager.policyxpress.engine.flow.PxFlowManager) @bci=32, line=140 (Interpreted frame)
- com.ca.identitymanager.policyxpress.engine.PxEventController.processEvent(com.ca.identitymanager.policyxpress.context.PxContext, com.ca.identitymanager.policyxpress.engine.PxProcessingType) @bci=174, line=117 (Interpreted frame)
- com.ca.identitymanager.policyxpress.listeners.PxStateListener.execute(com.netegrity.ims.IMSEvent, com.ca.identitymanager.policyxpress.listeners.PxStepDefinition) @bci=182, line=137 (Interpreted frame)
- com.ca.identitymanager.policyxpress.listeners.PxStateListener.post(com.netegrity.ims.IMSEvent) @bci=7, line=96 (Interpreted frame)
- com.netegrity.ims.bootstrap.IMSMessageListener.processEventToNextSavePoint(com.netegrity.ims.bootstrap.EventMessageWrapper) @bci=488, line=420 (Interpreted frame)
- com.netegrity.ims.bootstrap.IMSMessageListener.handleMessage(javax.jms.Message) @bci=811, line=192 (Interpreted frame)
- com.netegrity.ims.bootstrap.IMSMessageListener.onMessage(javax.jms.Message) @bci=52, line=103 (Interpreted frame)
Also our DBA has reported that there are locks on task persistence database and it looks like below :-
(@P0 varbinary(8000),@P1 nvarchar(4000),@P2 int)update object12_5 WITH (ROWLOCK) set object=@P0 where objectid=@P1 and objecttype=@P2
(@P0 varbinary(8000),@P1 nvarchar(4000),@P2 int)update object12_5 WITH (ROWLOCK) set object=@P0 where objectid=@P1 and objecttype=@P2
Stored procedure whoisactive with locks parameter gives us below information :-
<Database name="db_taskpersistence">
<Locks>
<Lock request_mode="S" request_status="GRANT" request_count="1" />
</Locks>
<Objects>
<Object name="object12_5" schema_name="dbo">
<Locks>
<Lock resource_type="KEY" index_name="PK_object12_5" request_mode="X" request_status="GRANT" request_count="2" />
<Lock resource_type="OBJECT" request_mode="IX" request_status="GRANT" request_count="1" />
<Lock resource_type="PAGE" page_type="*" index_name="PK_object12_5" request_mode="IX" request_status="GRANT" request_count="3" />
<Lock resource_type="RID" page_type="*" index_name="PK_object12_5" request_mode="X" request_status="GRANT" request_count="1" />
</Locks>
</Object>
<Object name="tasksession12_5" schema_name="dbo">
<Locks>
<Lock resource_type="KEY" index_name="PK_tasksession12_5" request_mode="X" request_status="GRANT" request_count="1" />
<Lock resource_type="OBJECT" request_mode="IX" request_status="GRANT" request_count="1" />
<Lock resource_type="PAGE" page_type="*" index_name="PK_tasksession12_5" request_mode="IX" request_status="GRANT" request_count="1" />
</Locks>
</Object>
</Objects>
</Database>
We have already opened a case with CA support. But if anyone can provide some more information/tips ,we would really appreciate this.
Can anyone help us in here ?
Regards,
Shashank