Alan Baugher

Monitoring the Identity Suite with CA APM (Wily)

Discussion created by Alan Baugher Employee on Apr 9, 2018
Latest reply on Apr 9, 2018 by Alan Baugher

Hello Team,

 

Jim DeZego  James.DeZego & I were reviewing CA APM value for the CA Identity Suite solution.

 

The CA Identity Suite Virtual Appliance and standard deployments typically leverage Wildfly as the J2EE.     Standalone deployments (non-vApp) may also choose RHEL JBOSS EAP (commercial version), IBM WebSphere, or Oracle Weblogic.

 

Lets focus on the Wildfly instances for Identity Manager, Identity Governance, and Identity Portal.

 

 

 

 

Documentation for CA APM with Wildfly as the J2EE shows the following JVM options to be leveraged.

JBoss and WildFly - CA Application Performance Management - 10.7 - CA Technologies Documentation 

 

"For JBoss 7 and above (including WildFly 8, 9, and 10): Add -Djboss.modules.system.pkgs=org.jboss.byteman,com.wily,com.wily.* -javaagent:<PathToAgentJar> -DagentProfile=<PathToAgentProfile>."

 

 

 

 

 

However, during our testing with the Identity Suite vApp and standalone deployments, there were a few additional switches required to support the additional logging features that CA APM required for Wildfly.

 

 

JVM switches required for CA APM agent for wildfly:

 

-Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager 

-javaagent:/opt/CA/VirtualAppliance/custom/apm/wily/Agent.jar

-Dcom.wily.introscope.agentProfile=/opt/CA/VirtualAppliance/custom/apm/wily/core/config/IntroscopeAgent.profile

 

JVM switch, to "rename" without updating the "profile" file:

-Dcom.wily.introscope.agent.agentName=IM_Wildfly_AWS_Node2

 

JVM switches required for Wildfly logging that CA APM required.

 

-Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager 

-Djava.util.logging.manager=org.jboss.logmanager.LogManager

-Xbootclasspath/p:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar

 

[Note:   Used   find /opt/CA/wildfly-idm   -name "*log*.jar"    to locate ALL the log4j modules under "org.jboss"]

 

 

 

 

Example of the IM JVM OPTIONS:

 

JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager  -javaagent:/opt/CA/VirtualAppliance/custom/apm/wily/Agent.jar -Dcom.wily.introscope.agent.agentName=IM_Wildfly_Alan_Laptop -Dcom.wily.introscope.agentProfile=/opt/CA/VirtualAppliance/custom/apm/wily/core/config/IntroscopeAgent.profile -Xbootclasspath/p:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:/opt/CA/wildfly-idm/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar

 

 

 

Example of the IG JVM OPTIONS:

 

JAVA_OPTS=-Xms512m -Xmx2048m -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager -javaagent:/opt/CA/VirtualAppliance/custom/apm/wily/Agent.jar -Dcom.wily.introscope.agent.agentName=IG_Wildfly_Alan_Laptop -Dcom.wily.introscope.agentProfile=/opt/CA/VirtualAppliance/custom/apm/wily/core/config/IntroscopeAgent.profile -Xbootclasspath/p:/opt/CA/wildfly-ig/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:/opt/CA/wildfly-ig/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:/opt/CA/wildfly-ig/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:/opt/CA/wildfly-ig/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar

 

 

 

Three (3) common error messages:

 

1) Memory issues due to use of -XX:+UseLargePages JVM Option with CA APM   [wildfly-console.log]

ERROR:   Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory (errno = 1).

RESOLUTION:   Remove JVM option

 

2) JBoss/Wildfly Log Manager required by CA APM.     [wildfly-console.log]

ERROR:  Could not load Logmanager "org.jboss.logmanager.LogManager"
java.lang.ClassNotFoundException: org.jboss.logmanager.LogManager
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

RESOLUTION:   Add in the JVM switches to declare the LogManager modules & class files (jar)

 

3) -Djboss.modules.system.pkgs=com.wily,com.wily.*   switch is NOT defined or overwritten on the vApp.

ERROR: Exception in thread "main" java.lang.NoClassDefFoundError: com/wily/introscope/agent/AgentShim
        at org.jboss.as.server.Main.main(Main.java)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

RESOLUTION:    Update the JVM swtich for system modules to include com.wily,com.wily.*

 

 

After the above has been configured, ensure that TCP 5001  (both incoming/outgoing) is enabled.

Execute a netstat -an | findstr 5001  [Win OS]    or    netstat -an | grep 5001    [Linux OS]

Ensure that a network connection is ESTABLISHED between the agent(s) and the CA APM Mgmt Server.

 

 

 

Example of the CA APM Management token in the "profile".

agentManager.url.1=54.152.177.5:5001

 

 

Example of the CA APM Agent response within CA APM Management UI.

 

 

 

 

Cheers,

 

Alan Alan_Baugher & Jim DeZego  James.DeZego

 

 

 

 

 

 

Edit:   4/9/2018

 

If you wish to use business friendly names for your CA APM agents, you may wish to use the above JVM switch, -Dcom.wily.introscope.agent.agentName=IM_Wildfly_AWS_Node2 ,      This may be important to clarify when more than one Wildfly J2EE instance exists on a shared server where only one (1) CA APM profile is required; useful for dev-ops deployments.

 

 

However, if you wish to take advantage of "auto-naming", there is an additional JAR & configuration that may be used.

- May also be used for Wildfly 8.x

 

Java Agent Naming - CA Application Performance Management - 10.7 - CA Technologies Documentation 

JBoss and WildFly - CA Application Performance Management - 10.7 - CA Technologies Documentation 

 

Configure JBoss 7 for Autonaming

The Automatic Agent Naming feature works with JBoss 7 only if you copy the webappsupport.jar file to the appserver deployment folder. The agent name is based on the configuration name that is specified when the server is started.

Example

For a standalone server, copy webappsupport.jar to the <JBoss7_Home>/standalone/deployments/ folder

Enable the Autonaming feature for JBoss 7.1 application servers.

Follow these steps:

  1. Navigate to the <Agent_Home>\core\ext directory of your Java agent installation.
  2. Select and move the WebAppSupport.jar file into the appserver(standalone) deployments folder.
  3. Add the following line to the IntroscopeAgent.profile:

    com.wily.use.platform.mbeanserver=true

  4. Add the following line to the standalone.xml file:

    <server name="jboss7" xmlns="urn:jboss:Domain:1.2">

  5. Verify that the name specified displays in the Investigator.

 

 

 

For the Identity Suite vApp, to update the standalone-full-ha.xml file, will required use of the JBOSS_HOME/bin/jboss-cli.sh process, to allow secure access to the vApp's Wildfly configuration files.  (JBOSS_HOME/standalone/configuration/standalone-*.xml)

 

 

 

Edit: 4/11/18  -   Attached examples of Wildfly 8.2 standalone.conf for IM and IG; as well as the jvm-args.conf for the IM vApp Wildfly example; and jvm_options.conf for the IAMCS (jcs) on vApp

 

 

Edit: 4/11/18  Added a few extra screen shots of CA APM for IM/IP/IG & IAMCS with JMX metrics as well.

 

 

 

 

####   ####

 

Note1:    Most CA APM settings may be updated in the CA_APM_HOME/wily/core/config/Introscope.profile  configuration.

 

However, it is also possible to define these CA APM key:value pairs as a JVM option.   This method may assist you with your testing cycle, and keep a "generic" Introscope.profile for your various JVM components.

 

Note 2:   If using CA APM for multiple JVM on the same server, ensure that separate CA APM agent folder are used for each, to avoid collision of logs and updates from the CA APM management server back to the CA APM agent folder.

 

Example:   

 

1) IAMCS (jcs) enabled with CA APM JVM (default + JMX) via   /opt/CA/IdentityManager/ConnectorServer/data/jvm_options.conf

 

-Xms1G -Xmx2G -Dcom.sun.net.ssl.enableECC=false -Dcom.wily.introscope.agent.agentName=IAMCS_JCS_Node01 -Dcom.wily.introscope.agent.agentManager.url.1=54.152.177.5:5001 -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom  -Djboss.modules.system.pkgs=com.wily,com.wily.*  -javaagent:/opt/CA/VirtualAppliance/custom/apm/wily_jcs/Agent.jar -Dcom.wily.introscope.agentProfile=/opt/CA/VirtualAppliance/custom/apm/wily_jcs/core/config/IntroscopeAgent.profile  -Dcom.wily.introscope.agent.jmx.enable=true  -Dcom.wily.introcope.agent.jmx.syncWithAllMbeanServersForcibly=true  -Dcom.wily.use.platform.mbeanserver=true

 

2) The IM/IG/IP Wildfly instances for the vApp; CA APM default agent & JMX monitoring:

 

# IM jvm-args.conf with JMX monitoring [Copy AGENT_HOME/common/WebAppSupport.jar to AGENT_HOME/core/ext]
JAVA_OPTS=-Xms1G -Xmx4G -Dcom.wily.introscope.agent.agentName=IM_Wildfly_Node01 -Dcom.wily.introscope.agent.agentManager.url.1=54.152.177.5:5001 -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager  -javaagent:/opt/CA/VirtualAppliance/custom/apm/wily_im/Agent.jar -Dcom.wily.introscope.agentProfile=/opt/CA/VirtualAppliance/custom/apm/wily_im/core/config/IntroscopeAgent.profile -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar  -Dcom.wily.introscope.agent.jmx.enable=true  -Dcom.wily.introcope.agent.jmx.syncWithAllMbeanServersForcibly=true  -Dcom.wily.use.platform.mbeanserver=true

# IG jvm-args.conf with JMX monitoring [Copy AGENT_HOME/common/WebAppSupport.jar to AGENT_HOME/core/ext]
JAVA_OPTS=-Xms2G -Xmx8G -Dcom.wily.introscope.agent.agentName=IG_Wildfly_Node01 -Dcom.wily.introscope.agent.agentManager.url.1=54.152.177.5:5001 -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager  -javaagent:/opt/CA/VirtualAppliance/custom/apm/wily_ig/Agent.jar -Dcom.wily.introscope.agentProfile=/opt/CA/VirtualAppliance/custom/apm/wily_ig/core/config/IntroscopeAgent.profile -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar  -Dcom.wily.introscope.agent.jmx.enable=true  -Dcom.wily.introcope.agent.jmx.syncWithAllMbeanServersForcibly=true  -Dcom.wily.use.platform.mbeanserver=true

# IP jvm-args.conf with JMX monitoring [Copy AGENT_HOME/common/WebAppSupport.jar to AGENT_HOME/core/ext]
JAVA_OPTS=-Xms1G -Xmx2G -Dcom.wily.introscope.agent.agentName=IP_Wildfly_Node01 -Dcom.wily.introscope.agent.agentManager.url.1=54.152.177.5:5001 -XX:+UseG1GC -XX:+UseStringDeduplication -XX:+UseCompressedOops -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true  -Djava.security.egd=file:/dev/./urandom -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Djboss.modules.system.pkgs=com.wily,com.wily.*,org.jboss.byteman,org.jboss.logmanager  -javaagent:/opt/CA/VirtualAppliance/custom/apm/wily_ip/Agent.jar -Dcom.wily.introscope.agentProfile=/opt/CA/VirtualAppliance/custom/apm/wily_ip/core/config/IntroscopeAgent.profile -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logging/main/jboss-logging-3.1.4.GA.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/log4j/logmanager/main/log4j-jboss-logmanager-1.1.0.Final.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/as/logging/main/wildfly-logging-8.2.0.Final.jar:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-1.5.2.Final.jar  -Dcom.wily.introscope.agent.jmx.enable=true  -Dcom.wily.introcope.agent.jmx.syncWithAllMbeanServersForcibly=true  -Dcom.wily.use.platform.mbeanserver=true

Outcomes