AnsweredAssumed Answered

Clarity DB Connectivity Issue

Question asked by sandhar on Jun 12, 2014
Latest reply on Jun 23, 2014 by Jeanne_Gaskill_CA_Clarity_Support

We had been facing DB connectivity issue with Clarity lately. App and BG run on seperate servers. A couple of weeks ago, it was the bg service which stopped processing the jobs and last evenning it was the Clarity app which stopped responding. The bg and app windows services itself were running, although it stopped processing anything.

Looking at the logs for both app and bg, I see the following errors:

From app logs:

ERROR 2014-06-11 19:27:36,878 [Niku Cache Monitor Thread] persistence.PersistenceController (clarity:none:none:none) Exception due to DB Connection failure

ERROR 2014-06-11 19:27:36,878 [Niku Cache Monitor Thread] (clarity:none:none:none) AuthenticationService.timeoutAction:Unable to clear session on timeout:8853468__05FD92E9-446A-4FF1-B962-CEA6735430CF Database error while removing user session
FATAL 2014-06-11 19:28:00,441 [http-80-13] union.persistence (clarity:<removedusername> 
java.sql.SQLNonTransientConnectionException: [CA Clarity][SQLServer JDBC Driver]A problem occurred when attempting to contact the server (Server returned: Software caused connection abort: recv failed). Please ensure that the server parameters passed to the driver are correct and that the server is running. Also ensure that the maximum number of connections have not been exceeded for this server.
From bg service:
FATAL 2014-06-11 19:29:46,817 [Process Monitor (tenant=clarity)] union.persistence (clarity:process_admin:8687024__BC926540-32A0-401D-A73B-4C31606589CD:none) Failed to retrieve database version
java.lang.RuntimeException: Unexpected invocation exception: [CA Clarity][SQLServer JDBC Driver]Object has been closed.
The error showed up in both app and bg logs; however one day the bg was hung and the other day app was hung.The network team is confident that there has not been network related errors. I checked the SQL Server logs and I do not see an issue there. 

Has anyone faced similar issues before? Is network the only cause of the error? I have checked Clarity processes and jobs and there has not been any long running XOG actions or jobs which could potentially cause this. On top of this I see a lot of following errors in the logs:

ERROR 2014-06-11 17:11:31,596 [http-80-212] out.service (clarity:none:none:none) A Checked Exception encounted.
ClientAbortException: Connection reset by peer: socket write error
I know the ca article regarding the above error where it is recommened "Change network setting on the switch from Automatic to Full Duplex". I have also verified this and it is set to full Duplex in the environment.
Except for network error, what could be other causes of the errors? It gets more interesting that one day my app service was able to recover from the error and the other day it was the bg service although both faced the same error at the same time. So, which service will be hung is intermittent!