AnsweredAssumed Answered

WTG Agent Keeps Crashing

Question asked by RazMan on Feb 12, 2014
Latest reply on Jan 28, 2016 by Hiko_Davis

Hello Community!

I was hoping someone had insight into the problem we are having: our WTG agents keep crashing unexpectedly.  They will run for 8 to 12 hours and then just stop. 

running wtgagent status <AGENT> will show that is it not running.   Logs show it is an error with the ports being in use; however, to our knowledge, only WTG should be using those ports.  Additionally, we have changed the ports completely and the issue still comes up. 

 

Thank you in advance for any help. 

 

WTGVER is 9.1.0 installed on 64 bit RHEL.  RTM Server and Agent installed on same machine.

Looking at the WTGTrace_***.log we see the following just before a problem:

02/12/2014 05:45:31 [D]-4158568144- WilyConnectThread  : SendMsg - Connected, sending msg.
02/12/2014 05:45:31 [D]-4145863568- WilyConnectionNDA  : SendMsg: About to send 7 metrics using SendMsg.
02/12/2014 05:45:31 [D]-4145863568- WilyConnectionNDA  : SendMsg: Call to pIntroscopeSendCounterArray successful.
02/12/2014 05:45:31 [D]-4158568144- WilyConnectThread  : SendMsg - msg sent successfully
02/12/2014 05:45:31 [D]-4158568144- CAgentUtilityThread: IScopeHeartBeat sent 0.
02/12/2014 05:45:41 [D]-4158568144- WilyConnectThread  : SendMsg - Connected, sending msg.
02/12/2014 05:45:41 [D]-4145863568- WilyConnectionNDA  : SendMsg: About to send 7 metrics using SendMsg.
02/12/2014 05:45:41 [D]-4145863568- WilyConnectionNDA  : SendMsg: Call to pIntroscopeSendCounterArray successful.
02/12/2014 05:45:41 [D]-4158568144- WilyConnectThread  : SendMsg - msg sent successfully
02/12/2014 05:45:41 [D]-4158568144- CAgentUtilityThread: IScopeHeartBeat sent 1.
02/12/2014 05:45:43 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:45:43 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:45:44 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:45:51 [D]-4158568144- WilyConnectThread  : SendMsg - Connected, sending msg.
02/12/2014 05:45:51 [D]-4145863568- WilyConnectionNDA  : SendMsg: About to send 7 metrics using SendMsg.
02/12/2014 05:45:51 [D]-4145863568- WilyConnectionNDA  : SendMsg: Call to pIntroscopeSendCounterArray successful.
02/12/2014 05:45:51 [D]-4158568144- WilyConnectThread  : SendMsg - msg sent successfully
02/12/2014 05:45:51 [D]-4158568144- CAgentUtilityThread: IScopeHeartBeat sent 0.
02/12/2014 05:45:55 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:45:55 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:45:57 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:46:01 [D]-4158568144- WilyConnectThread  : SendMsg - Connected, sending msg.
02/12/2014 05:46:01 [D]-4145863568- WilyConnectionNDA  : SendMsg: About to send 7 metrics using SendMsg.
02/12/2014 05:46:01 [D]-4145863568- WilyConnectionNDA  : SendMsg: Call to pIntroscopeSendCounterArray successful.
02/12/2014 05:46:01 [D]-4158568144- WilyConnectThread  : SendMsg - msg sent successfully
02/12/2014 05:46:01 [D]-4158568144- CAgentUtilityThread: IScopeHeartBeat sent 1.
02/12/2014 05:46:02 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:46:08 [C]-4124461968- WebConnection      : Re-Connect to RTMCServer - [localhost]:7301 failed; code=111
02/12/2014 05:46:11 [D]-4158568144- WilyConnectThread  : SendMsg - Connected, sending msg.
02/12/2014 05:46:11 [D]-4145863568- WilyConnectionNDA  : SendMsg: About to send 7 metrics using SendMsg.
02/12/2014 05:46:11 [D]-4145863568- WilyConnectionNDA  : SendMsg: Call to pIntroscopeSendCounterArray successful.
02/12/2014 05:46:11 [D]-4158568144- WilyConnectThread  : SendMsg - msg sent successfully
02/12/2014 05:46:11 [D]-4158568144- CAgentUtilityThread: IScopeHeartBeat sent 0.
02/12/2014 05:46:16 [I]-4158568144- CRTMServerControl  : Starting : RTMCServer
02/12/2014 05:46:16 [I]-4158568144- CRTMServerControl  : Launching: /opt/SA/APMTransactionGenerator/bin/RTMCServer -p 7301
02/12/2014 05:46:16 [A]-4158568144- ResponseAgent      : Event signalled - by stop command
02/12/2014 05:46:16 [A]-4158568144- ResponseAgent      : Shutdown Requested - WTGAgent stop FileNet command
02/12/2014 05:46:16 [D]-4158568144- WilyConnectThread  : SendMsg - Connected, sending msg.
02/12/2014 05:46:16 [D]-4145863568- WilyConnectionNDA  : SendMsg: About to send 1 metrics using SendMsg.
02/12/2014 05:46:16 [D]-4145863568- WilyConnectionNDA  : SendMsg: Call to pIntroscopeSendCounterArray successful.
02/12/2014 05:46:16 [D]-4158568144- WilyConnectThread  : SendMsg - msg sent successfully
02/12/2014 05:46:16 [A]-4158568144- ResponseAgent      : Saving data before shutdown
02/12/2014 05:46:16 [I]-4158568144- CRTMServerControl  : RTMCServer PID = 0
02/12/2014 05:46:16 [A]-4158568144- ResponseAgent      : Stopping Threads
02/12/2014 05:46:16 [A]-4158568144- ResponseAgent      : Stopping RTMCServer
02/12/2014 05:46:16 [D]-4145863568- WilyConnectionNDA  : Entering Disconnect() method.
02/12/2014 05:46:21 [C]-4158568144- ConnectionThread   : CheckForNoResponse() Unable to connect to RTMCServer
02/12/2014 05:46:21 [A]-4124461968- WebConnection      : Re-Connected to RTMCServer - [localhost]:7301
02/12/2014 05:46:21 [C]-4158568144- CObjects           : ResetEntryList() Rescheduled all entries
02/12/2014 05:46:21 [I]-4158568144- CRTMServerControl  : Sent Stop to RTMCServer at localhost:7301
02/12/2014 05:46:21 [A]-4158568144- ResponseAgent      : Waiting for threads to stop...
02/12/2014 05:46:23 [A]-4158568144- ResponseAgent      : WTGAgent Stopped...

Looking at the RTMCxxx.log we will see the following

02/12/2014 05:46:16 : Started RTMCServer version 9.5
02/12/2014 05:46:16 :
Opening RTMCServer listening ports for IPv4/IPv6...
02/12/2014 05:46:16 : ERROR: Port 7301 for IPv6 already in use
02/12/2014 05:46:16 : T-157000816 Connected to 127.0.0.1 on port 6613
02/12/2014 05:46:16 : T-157000816 Received STOP Command. Stopping RTMCServer...
02/12/2014 05:46:16 : T-157000816 Received STOP Command. STOP Event is set.
02/12/2014 05:46:16 : T-157000816 CRTMConnection::Disconnected(): 127.0.0.1  client disconnected
02/12/2014 05:46:16 : RTMCServer (IPv4) Listening on port 7301
02/12/2014 05:46:16 : RTMCServer: main() receive stop event. Program terminating..
02/12/2014 05:46:16 : RTMCServer: main(): calling cRTMServer.ShutAlldown()..
02/12/2014 05:46:21 : T-146510960 Connected to 127.0.0.1 on port 6615
02/12/2014 05:46:21 : T-146510960 CRTMConnection::Disconnected(): 127.0.0.1  client disconnected
02/12/2014 05:46:21 : T-177980528 Connected to 127.0.0.1 on port 6615
02/12/2014 05:46:21 : T-177980528 Hello from <SERVER>_FileNet

 

 

 

Outcomes