AnsweredAssumed Answered

Weblogic 10.3.x wily agent startup failure

Question asked by ali.paracha on Jun 14, 2017
Latest reply on Sep 5, 2017 by Lynn_Williams

Hi,

 

We are trying to use wily APM agent 10.3 with weblogic 10.3.x using JDK 1.6 IBM on AIX. The issue has been persistently around "Agent not available". Now we use node manager to start/stop the applications, startup script is not used, so we used server administration to do following:

 

1. added startup class 

2. added WebAppSupport.jar in classpath arguments of the application -classpath /bea/agent/wily/WebAppSupport.jar

3. added java agent arguments -javaagent:/bea/agent/wily/Agent.jar

 

restarted the server and got following exception:

 

<Jun 8, 2017 4:02:36 PM> <INFO> <NodeManager> <Server output log file is '/bea/bea/user_projects/domains/CCSDomain/servers/CCSManaged/logs/CCSManaged.out'>
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent Release 10.3.0.15 (Build 990015)
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Using Java VM version "IBM J9 VM 1.6.0" from IBM Corporation
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Trying to load agent profile based on system property "com.wily.introscope.agentProfile"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Unable to load agent profile because the system property could not be found
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Trying to load agent profile based on system property "agentProfile"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Trying to load file from /bea/agent/wily/core/config/IntroscopeAgent.profile
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Loaded file from /bea/agent/wily/core/config/IntroscopeAgent.profile
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent using old configuration (introscope.agent.configuration.old) set to default value of "false"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Canonical Key Map Thread Local (introscope.agent.reduceAgentMemoryOverhead.usecanonicalkeysthreadlocal) set to default value of "false"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Thread priority (introscope.agent.thread.all.priority) set to default value of "5"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent Transaction Trace Limit (introscope.agent.ttClamp) set to "50"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Configuration property (introscope.agent.tracer.sampling.maxrate) set to default value of "10"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Configuration property (introscope.agent.tracer.sampling.initial.period) set to default value of "100"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Configuration property (introscope.agent.tracer.sampling.reset.period) set to default value of "10000"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Configuration property (introscope.agent.transactiontracer.tailfilterPropagate.enable) set to "true"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Configuration property (introscope.agent.transactiontracer.parameter.capture.sessionid) set to default value of "true"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent Metric Aging (introscope.agent.metricAging.turnOn) set to "true"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent Metric Aging Interval (introscope.agent.metricAging.heartbeatInterval) set to default value of "86400" seconds
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent Metric Aging Data Chunk (introscope.agent.metricAging.dataChunk) set to default value of "500"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent Metric Aging Exclusion List (introscope.agent.metricAging.metricExclude.ignore) set to "[Threads*, ChangeDetector.AgentID, File System*]"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Agent Metric Agent Timeslice Intervals (introscope.agent.metricAging.numberTimeslices) set to default value of "180000"
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] Looking for agent profile property "introscope.agent.extensions.directory" to locate the extensions directory.
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] The agent extensions directory /bea/agent/wily/core/ext was successfully located
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] EagerClassLoader(ProbeBuilder.jar): Pre-loaded 701 classes.
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] EagerClassLoader(DynInstrSupport15.jar): Pre-loaded 133 classes.
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] EagerClassLoader(Java15DynamicInstrumentation.jar): Pre-loaded 99 classes.
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] EagerClassLoader(DynInstrBootstrap.jar): Pre-loaded 69 classes.
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] EagerClassLoader(Inheritance.jar): Pre-loaded 22 classes.
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] EagerClassLoader(BasicDirectiveLoader.jar): Pre-loaded 9 classes.
6/08/17 04:02:41 PM PKT [INFO] [IntroscopeAgent.Agent] EagerClassLoader(IntelligentInstrumentation.jar): Pre-loaded 234 classes.
6/08/17 04:02:41 PM PKT [ERROR] [IntroscopeAgent] A problem occurred while attempting to create the delegate agent
6/08/17 04:02:41 PM PKT [ERROR] [IntroscopeAgent] Agent Unavailable
<Jun 8, 2017 4:02:41 PM PKT> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
/bea/bea/user_projects/domains/CCSDomain/lib/WebAppSupport.jar>
<Jun 8, 2017 4:02:41 PM PKT> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with IBM J9 VM Version 2.4 from IBM Corporation>
<Jun 8, 2017 4:02:42 PM PKT> <Info> <Management> <BEA-141107> <Version: WebLogic Server 10.3.3.0 Fri Apr 9 00:05:28 PDT 2010 1321401 >
<Jun 8, 2017 4:02:44 PM PKT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<Jun 8, 2017 4:02:44 PM PKT> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<Jun 8, 2017 4:02:44 PM PKT> <Notice> <Log Management> <BEA-170019> <The server log file /bea/bea/user_projects/domains/CCSDomain/servers/CCSManaged/logs/CCSManaged.log is opened. All server side log events will be written to this file.>
<Jun 8, 2017 4:02:47 PM PKT> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>
<Jun 8, 2017 4:02:48 PM PKT> <Critical> <WebLogicServer> <BEA-000286> <Failed to invoke startup class "APM-StartupClass", com.wily.introscope.agent.AgentNotAvailableException
com.wily.introscope.agent.AgentNotAvailableException
at com.wily.introscope.agent.AgentShim.IAgentShim_getAgent(AgentShim.java:696)
at com.wily.introscope.api.weblogic.IntroscopeStartupClass.startup(IntroscopeStartupClass.java:70)
at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeStartup(ClassDeploymentManager.java:287)
at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeClass(ClassDeploymentManager.java:265)
at weblogic.management.deploy.classdeployment.ClassDeploymentManager.access$000(ClassDeploymentManager.java:54)
Truncated. see log file for complete stacktrace
>
<Jun 8, 2017 4:02:49 PM PKT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY>
<Jun 8, 2017 4:02:49 PM PKT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>

 

We have not received ClassNotFound exception, which we get if WebAppSupport.jar is not at the specified location, can someone help us figuring out by looking at the exception that what could be the reason?

 

Startup section logs is below:

 

JAVA Memory arguments:
.
WLS Start Mode=Production
.
CLASSPATH=/bea/bea/patch_wls1033/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/bea/bea/patch_ocp353/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java6_64/lib/tools.jar:/bea/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/bea/bea/wlserver_10.3/server/lib/weblogic.jar:/bea/bea/modules/features/weblogic.server.modules_10.3.3.0.jar:/bea/bea/wlserver_10.3/server/lib/webservices.jar:/bea/bea/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/bea/bea/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/bea/bea/wlserver_10.3/common/derby/lib/derbyclient.jar:/bea/bea/wlserver_10.3/server/lib/xqrl.jar:/bea/agent/wily/WebAppSupport.jar /bea/bea/wlserver_10.3/server/lib/weblogic.jar
.
PATH=/bea/bea/wlserver_10.3/server/bin:/bea/bea/modules/org.apache.ant_1.7.1/bin:/usr/java6_64/jre/bin:/usr/java6_64/bin:/bea/bea/wlserver_10.3/server/bin:/bea/bea/modules/org.apache.ant_1.7.1/bin:/usr/java6_64/jre/bin:/usr/java6_64/bin:/bea/bea/wlserver_10.3/server/bin:/bea/bea/modules/org.apache.ant_1.7.1/bin:/usr/java6_64/jre/bin:/usr/java6_64/bin:/usr/java6_64/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/bea/bin:/usr/bin/X11:/sbin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java6_64/jre/bin:/usr/java6_64/bin:/usr/java6_64/bin/:/opt/CA/SharedComponents/bin:/opt/CA/SharedComponents/ccs/cam/bin.:.:/usr/java6_64/bin
.
***************************************************
* To start WebLogic Server, use a username and *
* password assigned to an admin-level user. For *
* server administration, use the WebLogic Server *
* console at http://hostname:port/console *
***************************************************
starting weblogic with Java version:
java version "1.6.0"
Java(TM) SE Runtime Environment (build pap6460sr12-20121025_01(SR12))
IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr12-20121024_126067 (JIT enabled, AOT enabled)
J9VM - 20121024_126067
JIT - r9_20120914_26057
GC - 20120928_AA)
JCL - 20121014_01
Starting WLS with line:
/usr/java6_64/bin/java -Dweblogic.Name=CCSManaged -Djava.security.policy=/bea/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -javaagent:/bea/agent/wily/Agent.jar -Dcom.wily.introscope.agentProfile=/bea/agent/wily/core/config/IntroscopeAgent.profile -da -Dplatform.home=/bea/bea/wlserver_10.3 -Dwls.home=/bea/bea/wlserver_10.3/server -Dweblogic.home=/bea/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=http://10.254.166.125:7102 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/bea/bea/patch_wls1033/profiles/default/sysext_manifest_classpath:/bea/bea/patch_ocp353/profiles/default/sysext_manifest_classpath weblogic.Server

 

thanks

 

K

Outcomes