Unable to start RegistryService - DevTest 8.0

Discussion created by MANISHBHATNAGAR75811191 on Mar 21, 2015
Latest reply on Feb 12, 2016 by Tushar Ranjan

Like • Show 0 Likes0
Comment • 20

Hi,

I am unable to start the resistryService for DevTest 8.0. The registry, VSE, Simulator and Coordinator were all running and working fine until a week ago on the same Linux box. We are still getting used to DevTest and due to slow adotpion rate, the deployed services are not really utilized everyday, so I am not 100% sure when exactly it went down. However, I amunable to start it again. The registry log shows some Database Accessor exception. Please see below for exact exception. We are using Oracle DB with the approriate ojdbc.

Any suggestions/ideas?

Thanks

Manish Bhatnagar

2015-03-20 22:06:07,855Z (18:06) [ServerRequestResponder 6739] WARN com.itko.lisa.net.RemoteMethodExec - InvocationTargetException on invoke sendMetricsToDradis

java.lang.reflect.InvocationTargetException

at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

at java.lang.reflect.Method.invoke(Unknown Source)

at com.itko.lisa.net.RemoteMethodExec.execute(RemoteMethodExec.java:57)

at com.itko.lisa.net.ServerRequestHandler.processRMIMessage(ServerRequestHandler.java:432)

at com.itko.lisa.net.ServerRequestHandler.access$300(ServerRequestHandler.java:67)

at com.itko.lisa.net.ServerRequestHandler$4.run(ServerRequestHandler.java:355)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Caused by: Exception [EclipseLink-4005] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException

Exception Description: DatabaseAccessor not connected.

at org.eclipse.persistence.exceptions.DatabaseException.databaseAccessorNotConnected(DatabaseException.java:116)

at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:309)

at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.beginTransaction(DatasourceAccessor.java:238)

at org.eclipse.persistence.internal.sessions.AbstractSession.basicBeginTransaction(AbstractSession.java:485)

at org.eclipse.persistence.sessions.server.ClientSession.addWriteConnection(ClientSession.java:646)

at org.eclipse.persistence.sessions.server.ServerSession.acquireClientConnection(ServerSession.java:246)

at org.eclipse.persistence.sessions.server.ClientSession.getAccessor(ClientSession.java:300)

at org.eclipse.persistence.internal.sequencing.SequencingManager$Preallocation_Transaction_NoAccessor_State.getNextValue(SequencingManager.java:463)

at org.eclipse.persistence.internal.sequencing.SequencingManager.getNextValue(SequencingManager.java:1067)

at org.eclipse.persistence.internal.sequencing.ClientSessionSequencing.getNextValue(ClientSessionSequencing.java:70)

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:349)

at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:308)

at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:465)

at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:450)

at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.cloneAndRegisterNewObject(RepeatableWriteUnitOfWork.java:593)

at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalRegisterObject(UnitOfWorkImpl.java:2912)

at org.eclipse.persistence.internal.sessions.MergeManager.registerObjectForMergeCloneIntoWorkingCopy(MergeManager.java:936)

at org.eclipse.persistence.internal.sessions.MergeManager.mergeChangesOfCloneIntoWorkingCopy(MergeManager.java:494)

at org.eclipse.persistence.internal.sessions.MergeManager.mergeChanges(MergeManager.java:271)

at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeCloneWithReferences(UnitOfWorkImpl.java:3495)

at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.mergeCloneWithReferences(RepeatableWriteUnitOfWork.java:378)

at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeCloneWithReferences(UnitOfWorkImpl.java:3455)

at org.eclipse.persistence.internal.jpa.EntityManagerImpl.mergeInternal(EntityManagerImpl.java:486)

at org.eclipse.persistence.internal.jpa.EntityManagerImpl.merge(EntityManagerImpl.java:463)

at com.itko.lisa.utils.JpaUtil.addObjectToDradisCache(JpaUtil.java:1751)

at com.itko.lisa.coordinator.TestRegistryImpl.saveObjectToDradisCache(TestRegistryImpl.java:4441)

at com.itko.lisa.coordinator.TestRegistryImpl.sendMetricsToDradis(TestRegistryImpl.java:4307)

... 12 more

2015-03-20 22:06:07,860Z (18:06) [ServerRequestResponder 6739] WARN org.eclipse.persistence - org.eclipse.persistence WARNING: Local Exception Stack: Exception [EclipseLink-4005] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException Exception Description: DatabaseAccessor not connected. at org.eclipse.persistence.exceptions.DatabaseException.databaseAccessorNotConnected(DatabaseException.java:116) at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:309) at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.beginTransaction(DatasourceAccessor.java:238) at org.eclipse.persistence.internal.sessions.AbstractSession.basicBeginTransaction(AbstractSession.java:485) at org.eclipse.persistence.sessions.server.ClientSession.addWriteConnection(ClientSession.java:646) at org.eclipse.persistence.sessions.server.ServerSession.acquireClientConnection(ServerSession.java:246) at org.eclipse.persistence.sessions.server.ClientSession.getAccessor(ClientSession.java:300) at org.eclipse.persistence.internal.sequencing.SequencingManager$Preallocation_Transaction_NoAccessor_State.getNextValue(SequencingManager.java:463) at org.eclipse.persistence.internal.sequencing.SequencingManager.getNextValue(SequencingManager.java:1067) at org.eclipse.persistence.internal.sequencing.ClientSessionSequencing.getNextValue(ClientSessionSequencing.java:70) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:349) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:308) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:465) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:450) at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.cloneAndRegisterNewObject(RepeatableWriteUnitOfWork.java:593) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalRegisterObject(UnitOfWorkImpl.java:2912) at org.eclipse.persistence.internal.sessions.MergeManager.registerObjectForMergeCloneIntoWorkingCopy(MergeManager.java:936) at org.eclipse.persistence.internal.sessions.MergeManager.mergeChangesOfCloneIntoWorkingCopy(MergeManager.java:494) at org.eclipse.persistence.internal.sessions.MergeManager.mergeChanges(MergeManager.java:271) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeCloneWithReferences(UnitOfWorkImpl.java:3495) at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.mergeCloneWithReferences(RepeatableWriteUnitOfWork.java:378) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeCloneWithReferences(UnitOfWorkImpl.java:3455) at org.eclipse.persistence.internal.jpa.EntityManagerImpl.mergeInternal(EntityManagerImpl.java:486) at org.eclipse.persistence.internal.jpa.EntityManagerImpl.merge(EntityManagerImpl.java:463) at com.itko.lisa.utils.JpaUtil.addObjectToDradisCache(JpaUtil.java:1751) at com.itko.lisa.coordinator.TestRegistryImpl.saveObjectToDradisCache(TestRegistryImpl.java:4441) at com.itko.lisa.coordinator.TestRegistryImpl.sendMetricsToDradis(TestRegistryImpl.java:4307) at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.itko.lisa.net.RemoteMethodExec.execute(RemoteMethodExec.java:57) at com.itko.lisa.net.ServerRequestHandler.processRMIMessage(ServerRequestHandler.java:432) at com.itko.lisa.net.ServerRequestHandler.access$300(ServerRequestHandler.java:67) at com.itko.lisa.net.ServerRequestHandler$4.run(ServerRequestHandler.java:355) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)

2015-03-20 22:06:07,861Z (18:06) [ServerRequestResponder 6739] WARN org.eclipse.persistence - org.eclipse.persistence WARNING: Local Exception Stack: Exception [EclipseLink-4005] (Eclipse Persistence Services - 2.3.2.v20111125-r10461): org.eclipse.persistence.exceptions.DatabaseException Exception Description: DatabaseAccessor not connected. at org.eclipse.persistence.exceptions.DatabaseException.databaseAccessorNotConnected(DatabaseException.java:116) at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.incrementCallCount(DatasourceAccessor.java:309) at org.eclipse.persistence.internal.databaseaccess.DatasourceAccessor.rollbackTransaction(DatasourceAccessor.java:671) at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.rollbackTransaction(DatabaseAccessor.java:1569) at org.eclipse.persistence.internal.sessions.AbstractSession.basicRollbackTransaction(AbstractSession.java:588) at org.eclipse.persistence.sessions.server.ClientSession.basicRollbackTransaction(ClientSession.java:153) at org.eclipse.persistence.internal.sessions.AbstractSession.rollbackTransaction(AbstractSession.java:3415) at org.eclipse.persistence.internal.sequencing.SequencingManager$Preallocation_Transaction_NoAccessor_State.getNextValue(SequencingManager.java:478) at org.eclipse.persistence.internal.sequencing.SequencingManager.getNextValue(SequencingManager.java:1067) at org.eclipse.persistence.internal.sequencing.ClientSessionSequencing.getNextValue(ClientSessionSequencing.java:70) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:349) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.assignSequenceNumber(ObjectBuilder.java:308) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:465) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.assignSequenceNumber(UnitOfWorkImpl.java:450)at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.cloneAndRegisterNewObject(RepeatableWriteUnitOfWork.java:593) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalRegisterObject(UnitOfWorkImpl.java:2912) at org.eclipse.persistence.internal.sessions.MergeManager.registerObjectForMergeCloneIntoWorkingCopy(MergeManager.java:936) at org.eclipse.persistence.internal.sessions.MergeManager.mergeChangesOfCloneIntoWorkingCopy(MergeManager.java:494) at org.eclipse.persistence.internal.sessions.MergeManager.mergeChanges(MergeManager.java:271) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeCloneWithReferences(UnitOfWorkImpl.java:3495) at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.mergeCloneWithReferences(RepeatableWriteUnitOfWork.java:378) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeCloneWithReferences(UnitOfWorkImpl.java:3455) at org.eclipse.persistence.internal.jpa.EntityManagerImpl.mergeInternal(EntityManagerImpl.java:486) at org.eclipse.persistence.internal.jpa.EntityManagerImpl.merge(EntityManagerImpl.java:463) at com.itko.lisa.utils.JpaUtil.addObjectToDradisCache(JpaUtil.java:1751) at com.itko.lisa.coordinator.TestRegistryImpl.saveObjectToDradisCache(TestRegistryImpl.java:4441) at com.itko.lisa.coordinator.TestRegistryImpl.sendMetricsToDradis(TestRegistryImpl.java:4307) at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.itko.lisa.net.RemoteMethodExec.execute(RemoteMethodExec.java:57) at com.itko.lisa.net.ServerRequestHandler.processRMIMessage(ServerRequestHandler.java:432) at com.itko.lisa.net.ServerRequestHandler.access$300(ServerRequestHandler.java:67) at com.itko.lisa.net.ServerRequestHandler$4.run(ServerRequestHandler.java:355) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source)

Outcomes

20 Replies

KevinNg Mar 23, 2015 1:48 PM
Hi Manish,

The "DatabaseAccessor not connected" error is most likely related to the oracle database itself. If you haven't done so already, check if any changes have been made to the database. Are you able to get the enterprise dashboard up and running?

Kevin
Like • Show 1 Like1
Actions
- Koustubh.Warty @ KevinNg on Mar 23, 2015 4:50 PM
  Can you save all the log files and then restart the ED and then the Registry and check if it works for you? If it does, we can then look at the logs to see the cause of this.
  
  BTW....we already have 8.0.1 available for over a month now.
  Like • Show 0 Likes0
  Actions
- MANISHBHATNAGAR75811191 @ KevinNg on Mar 23, 2015 5:35 PM
  Kevin, Koustubh
  
  Correct me if I am wrong but I thought Registry has to be up and running in order for ED to come up. Since I am unable to bring up the registry, I can't bring up the ED either.
  
  There were no changes made to the DB. I am able to Ping, telnet and login to the DB using the same credentials and connection string as specified in site.properties and local.properties.
  
  -Manish
  Like • Show 0 Likes0
  Actions
  - Koustubh.Warty @ MANISHBHATNAGAR75811191 on Mar 23, 2015 5:40 PM
    It is in fact the other way round. The ED has to be up and running first and then the Registry and the other components follow...
    Like • Show 0 Likes0
    Actions
  - KevinNg @ MANISHBHATNAGAR75811191 on Mar 23, 2015 5:44 PM
    Hi Manish,
    
    The relationship between the registry and the ED is the other way around, so the order of startup would be ED -> Registry->other server components.
    
    Try Koustubh's suggestion and restart it in the order of ED -> Registry. It seems the error is happening when attempting to send information to the ED.
    
    Thanks,
    Kevin
    Like • Show 0 Likes0
    Actions
    - MANISHBHATNAGAR75811191 @ KevinNg on Mar 23, 2015 6:01 PM
      Thanks for correcting me Kevin and Koustubh. I tried starting the ED and no luck! Surprisingly, when I start the service, the ED log is not even getting updated for some reason.
      
      Is it ok to delete enterprisedashboard.log just to verify that it is really not updating the log? It should recreate the log file automatically the nect time I start the service, correct?
      
      -Manish
      Like • Show 0 Likes0
      Actions
MANISHBHATNAGAR75811191 Mar 23, 2015 6:30 PM
I deleted "Enterprisedashboard.log" and tried to restart the service. Neither the service nor the log file is coming up. Just to clarify, when I start the service, a new "Enterprisedashboard.log" is not created.

-Manish
Like • Show 0 Likes0
Actions
- Koustubh.Warty @ MANISHBHATNAGAR75811191 on Mar 23, 2015 6:57 PM
  Weird....can you verify a couple of things related to the database -
  
  - Can you verify if both Registry and ED both are on the same database but different schema?
  - Can you verify that you can connect to their databases outside of DevTest without any issues?
  
  Thanks!
  Like • Show 0 Likes0
  Actions
- RichieB @ MANISHBHATNAGAR75811191 on Mar 25, 2015 11:56 AM
  The log file should still be created even if the port is in use. The only time I've found log files not created is when the directory they are created in is inaccessible to the account that the service is running under because of insufficient permissions.
  Like • Show 1 Like1
  Actions
  - Koustubh.Warty @ RichieB on Mar 25, 2015 1:16 PM
    Good point here...I've seen that behavior on Windows as well but then it is a matter of permissions which is simple enough to resolve!
    Like • Show 0 Likes0
    Actions
MANISHBHATNAGAR75811191 Mar 23, 2015 7:04 PM
Koustubh,

Both the things are verified.

ED and Registry have their own dedicated Schemas in the same DB. This is how we had done the original set-up.
I have also validated that the credentials and connection string for both ED and Registry are correct and accesible outside of DevTets.

-Manish
Like • Show 0 Likes0
Actions
MANISHBHATNAGAR75811191 Mar 23, 2015 7:19 PM
I tired running just the ./EnterpriseDashboard and it complains about port 2003 already being in use. Please see below. I have also run the netstat with and without a grep on '2003' but didn't find anything there either.

lisaadmin@sandstgccplisa02:~/CA/Lisa/8.0/bin> ./EnterpriseDashboard
Locale: en_US
Setting System.out and System.err to: UTF-8
LISA_HOME set to /home/lisaadmin/CA/Lisa/8.0/
DevTest temporary directory is /home/lisaadmin/lisatmp_8.0.0
Configuring logging from '/home/lisaadmin/CA/Lisa/8.0/logging.properties'
DevTest Solutions Version :: 8.0.0 (8.0.0.1615)

Starting Enterprise Dashboard, service=tcp://sandstgccplisa02.ga.adp.com:2003/EnterpriseDashboard
WARNING: Failed to start a server listener at port 2,003
WARNING: Reason: class java.io.IOException Failed to bind to server socket: tcp://0.0.0.0:2003/EnterpriseDashboard due to: java.net.BindException: Address already in use
WARNING: Almost certainly there is another server running on this machine with the same port number
WARNING: Change the name by setting the appropriate lisa property (lisa.registryName, lisa.coordName,
WARNING: lisa.simulatorName or lisa.vseName) to a fully qualifed name such as tcp://host:4567/MyVSE
WARNING: or use "-n tcp://host:4567/MyVSE" on the command line or set the environment variable
WARNING: LISA_MORE_VM_PROPS=-Dlisa.vseName=tcp://host:4567/MyVSE
Like • Show 0 Likes0
Actions
- Koustubh.Warty @ MANISHBHATNAGAR75811191 on Mar 23, 2015 7:28 PM
  Don't want to sound like 'windows' but can you try restarting the machine?
  Like • Show 0 Likes0
  Actions
MANISHBHATNAGAR75811191 Mar 23, 2015 7:29 PM
Here are the netstat results:

lisaadmin@sandstgccplisa02:~/CA/Lisa/8.0/bin> netstat -ln | grep ':2003' | grep 'LISTEN'
tcp 0 0 0.0.0.0:2003 0.0.0.0:* LISTEN
lisaadmin@sandstgccplisa02:~/CA/Lisa/8.0/bin>

-Manish
Like • Show 0 Likes0
Actions
- Rick.Brown @ MANISHBHATNAGAR75811191 on Mar 23, 2015 8:05 PM
  what's the output of the command:
       lsof -i tcp:2003
  This should give you process details of what's holding that port open. My output is:
  COMMAND   PID   USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
  java    22895 rbrown 524u IPv4 0x6d83ced0300ab04f      0t0 TCP *:2003 (LISTEN)
  
  If you then do
       ps -ef | grep 22895
  where 22895 is my PID number from the lsof command
  you should get the process. Mine is Enterprise Dashboard, which reports this:
  501 22919   356   0 7:45pm ttys000    0:00.00 grep 22895
  501 22895 22394   0 7:39pm ttys001    0:18.48 /Applications/DevTest801/.install4j/jre.bundle/Contents/Home/jre/bin/java -server -Dinstall4j.jvmDir=/Applications/DevTest801/.install4j/jre.bundle/Contents/Home/jre -Dexe4j.moduleName=/Applications/DevTest801/bin/EnterpriseDashboard -XX:+Hea<SNIPPED for clarity>
  
  Determine from there whether you can kill or restart that process.
  Like • Show 0 Likes0
  Actions
  - MANISHBHATNAGAR75811191 @ Rick.Brown on Mar 23, 2015 8:32 PM
    I get nothing back with: lsof -i tcp:2003 which leads me to believe that there is nothing holding that port open, however the ED logs suggest otherwise. Wierd..I have also recycled the Linux box and still have the same issue.
    
    -Manish
    Like • Show 0 Likes0
    Actions
    - Koustubh.Warty @ MANISHBHATNAGAR75811191 on Mar 23, 2015 8:36 PM
      Manish - Weird scenario indeed. I'd recommend to call CA Support at 1-800-CALL-CAI (225-5224) or open a ticket with them at http://support.ca.com if you have the access.They should be able to assist you on this further in greater details.
      Like • Show 0 Likes0
      Actions
MANISHBHATNAGAR75811191 Mar 24, 2015 1:05 AM
Ok, it is solved! I am not very excited to share the solution though because I had to recycle the Linux box and it still doesn't make much sense. But here it is:
1. Recycled - Still unable to bring up the service
2. Launched the Program by itself ./Enterprisedashboard and stopped it
3. Tried to bring up the service after some time and it came up!
4. 1 thing is for sure, ./Enterprisedashboard as well as ./EnterprisedashboardService take some time to really shut down and release the port. I had to wait good 5-10 minutes before using the same port or else it kep complaining about the "Port already being in use".
Thanks for your time and help on this!

-Manish
Like • Show 2 Likes2
Actions
- Koustubh.Warty @ MANISHBHATNAGAR75811191 on Mar 25, 2015 1:15 PM
  Good to know!
  Like • Show 0 Likes0
  Actions
- Tushar Ranjan @ MANISHBHATNAGAR75811191 on Feb 12, 2016 9:53 PM
  You may try to add below in your local.properties file. Also make sure you do not have any other versions of Lisa running on the same machine.
  lisa.enterprisedashboard.service.url=tcp://<<IP Address>>:2003/EnterpriseDashboard
  lisa.net.bindToAddress=<<IP Address>>
  Like • Show 0 Likes0
  Actions

Unable to start RegistryService - DevTest 8.0

Outcomes

Related Content

Recommended Content