Service Virtualization

we have a coordinator server that keeps failing for out of Java Heap after running for a month or more. 

there are tests run every few minutes. we have increased the jvm memory, and the time elongates. 

how can we discover what is consuming the heap

Are you getting any hprof files getting created?

No..  

Hi Sam,

Could you post the exception/error you are getting in the coordinator log files when the Coordinator fails?

Thank you,

Heloisa

1st error in the coordinator.log

2016-12-14 03:53:03,888Z (22:53) [Event Sink Thread Pool Thread 4] INFO com.itko.lisa.stats.MetricControllerImpl - Error retrieving metric
java.lang.IllegalStateException: Could not put anything new on the event queue
at com.itko.lisa.simulator.EventHandler.testEvent(EventHandler.java:242)
at com.itko.lisa.stats.MetricControllerImpl.fireTestEvent(MetricControllerImpl.java:571)
at com.itko.lisa.stats.MetricControllerImpl.eventReceipt(MetricControllerImpl.java:514)
at com.itko.util.EventThread.execEvents(ThreadedEventSink.java:79)
at com.itko.util.EventThread.run(ThreadedEventSink.java:59)
Caused by: java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220)
at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335)
at java.util.concurrent.LinkedBlockingQueue.offer(LinkedBlockingQueue.java:380)
at com.itko.lisa.simulator.EventHandler.testEvent(EventHandler.java:236)
... 4 more
2016-12-14 03:53:11,000Z (22:53) [Event Sink Thread Pool Thread 3] ERROR com.itko.util.Pulse - Error calling object pulse()
java.lang.NullPointerException

1st notice of heap space, there are a bunch of errors between 1st and this error

2016-12-14 03:54:58,817Z (22:54) [ServerRequestResponder 10083] ERROR com.itko.lisa.net.ServerRequestHandler - Could not process message: Unexpected reference to target tcp://152.144.6.112:2011/AA65EC44C1B011E68E49005056AA0A0C lost making call endTest known soft targets: [PERF-Coordinator, DCE356EFC1B011E68E49005056AA0A0C, metricsAA637B43C1B011E68E49005056AA0A0C, AA637B43C1B011E68E49005056AA0A0C, brokerServices, DA2D379EC1B011E68E49005056AA0A0C] dying targets: [metrics86A1476AC1B011E68E49005056AA0A0C, metricsAA613152C1B011E68E49005056AA0A0C, metrics869EFD79C1B011E68E49005056AA0A0C, metricsAA65EC44C1B011E68E49005056AA0A0C] hard targets: [PERF-Coordinator, DCE356EFC1B011E68E49005056AA0A0C, metricsAA637B43C1B011E68E49005056AA0A0C, metrics869EFD79C1B011E68E49005056AA0A0C, metricsAA65EC44C1B011E68E49005056AA0A0C, AA637B43C1B011E68E49005056AA0A0C, brokerServices, DA2D379EC1B011E68E49005056AA0A0C, metrics86A1476AC1B011E68E49005056AA0A0C, metricsAA613152C1B011E68E49005056AA0A0C] colocated: [metrics86A1476AC1B011E68E49005056AA0A0C, metricsAA65EC44C1B011E68E49005056AA0A0C, brokerServices, DA2D379EC1B011E68E49005056AA0A0C, metricsAA637B43C1B011E68E49005056AA0A0C, PERF-Coordinator, metrics869EFD79C1B011E68E49005056AA0A0C, DCE356EFC1B011E68E49005056AA0A0C, AA637B43C1B011E68E49005056AA0A0C, metricsAA613152C1B011E68E49005056AA0A0C]
java.rmi.RemoteException: Unexpected reference to target tcp://152.144.6.112:2011/AA65EC44C1B011E68E49005056AA0A0C lost making call endTest known soft targets: [PERF-Coordinator, DCE356EFC1B011E68E49005056AA0A0C, metricsAA637B43C1B011E68E49005056AA0A0C, AA637B43C1B011E68E49005056AA0A0C, brokerServices, DA2D379EC1B011E68E49005056AA0A0C] dying targets: [metrics86A1476AC1B011E68E49005056AA0A0C, metricsAA613152C1B011E68E49005056AA0A0C, metrics869EFD79C1B011E68E49005056AA0A0C, metricsAA65EC44C1B011E68E49005056AA0A0C] hard targets: [PERF-Coordinator, DCE356EFC1B011E68E49005056AA0A0C, metricsAA637B43C1B011E68E49005056AA0A0C, metrics869EFD79C1B011E68E49005056AA0A0C, metricsAA65EC44C1B011E68E49005056AA0A0C, AA637B43C1B011E68E49005056AA0A0C, brokerServices, DA2D379EC1B011E68E49005056AA0A0C, metrics86A1476AC1B011E68E49005056AA0A0C, metricsAA613152C1B011E68E49005056AA0A0C] colocated: [metrics86A1476AC1B011E68E49005056AA0A0C, metricsAA65EC44C1B011E68E49005056AA0A0C, brokerServices, DA2D379EC1B011E68E49005056AA0A0C, metricsAA637B43C1B011E68E49005056AA0A0C, PERF-Coordinator, metrics869EFD79C1B011E68E49005056AA0A0C, DCE356EFC1B011E68E49005056AA0A0C, AA637B43C1B011E68E49005056AA0A0C, metricsAA613152C1B011E68E49005056AA0A0C]
at com.itko.lisa.net.ServerRequestHandler.locateTarget(ServerRequestHandler.java:518)
at com.itko.lisa.net.ServerRequestHandler.processRMIMessage(ServerRequestHandler.java:427)
at com.itko.lisa.net.ServerRequestHandler.access$300(ServerRequestHandler.java:67)
at com.itko.lisa.net.ServerRequestHandler$4.run(ServerRequestHandler.java:357)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-12-14 03:54:52,331Z (22:54) [Event Sink Thread Pool Thread 2] INFO com.itko.lisa.history.AmqDbWriter - queue name reporting_14661 queued: 68 dequeued: 68 (100%)
2016-12-14 03:56:28,080Z (22:56) [C3P0PooledConnectionPoolManager[identityToken->1c01r7lojrsua1vj2j2c|6690f860]-AdminTaskTimer] ERROR System.err - Exception in thread "C3P0PooledConnectionPoolManager[identityToken->1c01r7lojrsua1vj2j2c|6690f860]-AdminTaskTimer" java.lang.OutOfMemoryError: Java heap space
2016-12-14 03:56:28,077Z (22:56) [pool-3-thread-1] INFO com.itko.lisa.net.ServerRequestHandler

What is the database and version you are using for your Enterprise Dashboard and Registry?

Oracle 11.2

Do you have tried to increase the heap size on your Coordinator Service?

yes, the CoordinatorServer.vmoptions

-Xmx4096m

but ps -ef shows different

root     26334 24860 26 Jan10 pts/2    05:04:45 /opt/CA/DevTest_9.1/jre/bin/java -server -Dinstall4j.jvmDir=/opt/CA/DevTest_9.1/jre -Dexe4j.moduleName=/opt/CA/DevTest_9.1/bin/CoordinatorServer -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/CA/DevTest_9.1/bin//../tmp -Djava.endorsed.dirs=/opt/CA/DevTest_9.1/bin//../lib/endorsed -DLISA_HOME=/opt/CA/DevTest_9.1/bin//../ -DLISA_LOG=coordinator.log -Xms24m -Xmx768m -Dfile.encoding=UTF-8 -Dcom.sun.management.jmxremote -Djava.security.policy=/opt/CA/DevTest_9.1/bin//../lisa.permissions -Djava.net.preferIPv4Stack=true -DdummyArg -Dinstall4j.launcherId=75 -Dinstall4j.swt=false -Di4jv=0 -Di4jv=0 -Di4jv=0 -Di4jv=0 -Di4jv=0 -Xmx4096m -Di4j.vpt=true -classpath /opt/CA/DevTest_9.1/.install4j/i4jruntime.jar:/opt/CA/DevTest_9.1/hotDeploy/xmlunit-core-2.0.0-alpha-03.jar:/opt/CA/DevTest_9.1/hotDeploy/xmlunit-legacy-2.0.0-alpha-03.jar:/opt/CA/DevTest_9.1/hotDeploy/xmlunit-matchers-2.0.0-alpha-03.jar:/opt/CA/DevTest_9.1/lib/core/agent-zip-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/atk-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/desensitize-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/devtest-security-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/dradis-cic-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-acl-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-acl-api-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-agent.jar:/opt/CA/DevTest_9.1/lib/core/lisa-annotations-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-baseline-1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-cloud-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-core-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-esb-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-glass-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-httpweb-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-installer-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-int-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-invoke2-api-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-invoke2-client-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-j2ee-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-jdbc-sim-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-jdbridge-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-jmx-support-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-json-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-mobile-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-modules-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-netbeans-beanshell-editor-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-netbeans-editor-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-netbeans-patches-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-netbeans-stateparser-api-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-netbeans-xmldiff-viewer-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-pf-adapter-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-pf-chromeextension-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-pf-core-bridge-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-pf-flex-agent-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-pf-lightagent-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-reporting-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-test-byop-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-tests-common-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-util-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-virtualize-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/lisa-virtualize-byop-9.1.0.jar:/opt/CA/DevTest_9.1/lib/core/

the question is, what is causing the leak. how do we debug this?  I see some old heapdump tools, but they don't work now. 

Sam, I would open a support case on this so we can debug further.

Done, case opened

Thank you Sam.  We will handle this through the support case.

thanks to the support team.. this appears to have been caused by a database server outage, affecting the registry database that the coordinator needed to send stats to.. (altho we don't use the stats at all).. 

we are now looking into monitoring and alerting process to insure we know about these problems..

also looking for ways to reduce/eliminate the stats recording feature, as we don't use it