Alan Baugher

Smart Debugging: Update JBOSS/Wildfly Log4j without restart for Identity Suite (and vApp)

Discussion created by Alan Baugher Employee on Oct 13, 2017

Team,

 

While assisting a customer with their documentation efforts of their current production IM business logic, we typically use the ims.policyxpress=DEBUG logger, using the dynamic logger tool:  logging.jsp.

 

 

However, there are times when you would like to leave these loggers in place. for test/dev/staging environments; or to use in production without restarting the solution.

 

Fortunately, if you are using JBOSS/Wildfly, this functionality does exist, within the JBOSS_HOME/bin/jboss-cli.sh (.bat)

 

2018-08-25 - Update.  Added text download versions to add/remove the new loggers/appender file.

2018-09-10 - Update.  Added new versions that include IP/nodename as part of the appender file, to assist with RCA for cluster logging.

2018-10-01 - Update.  One caution note for the vApp, ensure you are using the "batch" mode during removal of objects to prevent a possible error.  Otherwise, if you remove a dependency before another object, you may impact startup of Wildfly.  On standalone deployments, this is not an issue, but the vApp's secured deployments permissions would not allow the 'config' userID to update the standalone-full-ha-***.xml configuration file, that is owned by "wildfly" userID.

 

 

 

 Using select loggers will only add add < 10% delay to processing time.

 

Below is an example to add the select loggers, with a NEW file appender/handler to allow business logic to be separated from the default J2EE server.log

 

Step 1:  Create a batch file to add the loggers:  jbosslog.cli

 

connect

batch

/subsystem=logging/periodic-rotating-file-handler=im_business_logic:add(file={"path"=>"im_business_logic_debug.log", "relative-to"=>"jboss.server.log.dir"}, suffix=".yyyy.MM.dd", level=DEBUG, append=true, autoflush=false, suffix=".yyyy-MM-dd-HH", formatter="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n")

/subsystem=logging/logger=ims.tasktrack.custom:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

/subsystem=logging/logger=im.feeder:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

/subsystem=logging/logger=ims.policyxpress:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

/subsystem=logging/logger=ims.llsdk.role.azengine:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

/subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

/subsystem=logging/logger=ims.tmt.submit.validation.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

/subsystem=logging/logger=ims.tmt.submit.execute.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)

run-batch

 

Step 2:  Execute this batch file by navigating to the JBOSS_HOME\bin\ folder; and execute  .

./jboss-cli.sh --file=jbosslog.cli

 

The logger will now be active, and you will notice a new FILE handler under:

JBOSS_HOME/standalone/log/im_business_logic_debug.log

 

 

If you wish to remove these loggers and FILE handler

 

 

Step 3:   Create a 2nd batch file:   jbosslogremove.cli

 

connect

batch

/subsystem=logging/periodic-rotating-file-handler=im_business_logic:remove

/subsystem=logging/logger=ims.tasktrack.custom:remove

/subsystem=logging/logger=im.feeder:remove

/subsystem=logging/logger=ims.policyxpress:remove

/subsystem=logging/logger=ims.llsdk.role.azengine:remove

/subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:remove

/subsystem=logging/logger=ims.tmt.submit.validation.blth:remove

/subsystem=logging/logger=ims.tmt.submit.execute.blth:remove

run-batch

 

 

Step 4:  Execute this batch file by navigating to the JBOSS_HOME\bin\ folder; and execute  .

./jboss-cli.sh --file=jbosslogremove.cli

 

 

 

 

 

Optional Notes:   To execute these steps on the secure Identity Suite Virtual Appliance, it is necessary to create a local JBOSS/Wildfly account.   Fortunately, this batch "add-user.sh" is part of the sudoer list for the "config" userID.

 

Ref: Administrating CA Identity Suite Virtual Appliance (vApp) - CA Identity Suite - 14.0 - CA Technologies Documentation 

 

 

 

Step A:   Execute the following command:

sudo /opt/CA/wildfly-idm/bin/add-user.sh

 

Add an "administrator" user, with a good password.

 

Step B:  Execute the IDM Wildfly command, jboss-cli.sh, with a redirect to the batch file

/opt/CA/wildfly-idm/bin/jboss-cli.sh  --file=/tmp/jbosslog.cli

 

This batch information will now be added to JBOSS_HOME/standalone/configuration/ca-standalone-full-ha.xml & be available immediately for use (and upon reboot of the system or restart of Wildfly).

 

 

 

Let me know if you find this of value for your own debugging or reverse-engineering documentation purposes.

 

 

Cheers,

 

A.

 

 

Attaching a PDF note that explains the value of the loggers that I have selected.

 

 

Edit:  9/10/2018     Two (2) items:

 

 1)  Update logger/appender's formatter pattern to have IP address/NodeName within the log file, so when consolidation is utilized, the resource will be able to identify where the cluster logs originated from.

2)   Added space for the %s%E pattern &  %l to assist with RCA .

 

im_business_logic.cli.txt

 

connect
batch
/subsystem=logging/periodic-rotating-file-handler=im_business_logic:add(file={"path"=>"im_business_logic_debug.log", "relative-to"=>"jboss.server.log.dir"}, suffix=".yyyy.MM.dd", level=DEBUG, append=true, autoflush=false, suffix=".yyyy-MM-dd-HH", formatter="%d{HH:mm:ss,SSS} [${jboss.bind.address}] [${jboss.node.name}] %-5p [%c] (%t) %-120s%E {%l}%n")
/subsystem=logging/logger=ims.tasktrack.custom:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
/subsystem=logging/logger=im.feeder:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
/subsystem=logging/logger=ims.policyxpress:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
/subsystem=logging/logger=ims.llsdk.role.azengine:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
/subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
/subsystem=logging/logger=ims.tmt.submit.validation.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
/subsystem=logging/logger=ims.tmt.submit.execute.blth:add(handlers=["im_business_logic"], level=DEBUG, use-parent-handlers=false)
run-batch

 

 

 

im_business_logic_remove.cli.txt

 

connect
batch
/subsystem=logging/periodic-rotating-file-handler=im_business_logic:remove
/subsystem=logging/logger=ims.tasktrack.custom:remove
/subsystem=logging/logger=im.feeder:remove
/subsystem=logging/logger=ims.policyxpress:remove
/subsystem=logging/logger=ims.llsdk.role.azengine:remove
/subsystem=logging/logger=ims.llsdk.role.azcache.ridiculouslydetailed:remove
/subsystem=logging/logger=ims.tmt.submit.validation.blth:remove
/subsystem=logging/logger=ims.tmt.submit.execute.blth:remove
run-batch

 

 

im_full_business_logic.cli.txt

 

connect
batch
/subsystem=logging/periodic-rotating-file-handler=im_all_business_logic:add(file={"path"=>"im_all_business_logic_debug.log", "relative-to"=>"jboss.server.log.dir"}, suffix=".yyyy.MM.dd", level=DEBUG, append=true, autoflush=false, suffix=".yyyy-MM-dd-HH", formatter="%d{HH:mm:ss,SSS} [${jboss.bind.address}] [${jboss.node.name}] %-5p [%c] (%t) %-120s%E {%l}%n")
/subsystem=logging/logger=im:add(handlers=["im_all_business_logic"], level=DEBUG, use-parent-handlers=false)
/subsystem=logging/logger=ims:add(handlers=["im_all_business_logic"], level=DEBUG, use-parent-handlers=false)
run-batch

 

 

im_full_business_logic_remove.cli.txt

 

connect
batch
/subsystem=logging/periodic-rotating-file-handler=im_all_business_logic:remove
/subsystem=logging/logger=im:remove
/subsystem=logging/logger=ims:remove
run-batch

 

 

 

Log4J Formatter Syntax Reference:    

https://access.redhat.com/documentation/en-us/jboss_enterprise_application_platform/6/html/administration_and_configuration_guide/chap-the_logging_subsystem#Log_Formatter_Syntax1

 

 

 

 

Output example:

 

15:51:07,804 [192.168.242.146] [iamnode1]  WARN  [im.plugins.IMEnvironmentInitializerPlugin] (MSC service thread 1-1) http://192.168.242.146:8080/iam/im/ETACALLBACK/?env=identityEnv is already registered as a inbound URI on the Provisioning Server. Just updating the shared secret. {com.ca.identitymanager.plugins.IMEnvironmentInitializerPlugin.startEnvironment(IMEnvironmentInitializerPlugin.java:209)}

15:51:09,901 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) * Started environment: identityEnv                                                                                       {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}

15:51:10,088 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) ** FIPS mode enabled : false                                                                                             {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}

15:51:10,338 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) * Startup Step 30 : Attempting to start ApplicationContextInitializer plug-ins                                           {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}

15:51:10,388 [192.168.242.146] [iamnode1]  INFO  [ims.Main] (MSC service thread 1-1) ---- CA IAM FW Startup Sequence Complete. ----                                                                           {com.netegrity.llsdk6.imsimpl.utility.log.Log4JLogImpl.logInfo(Log4JLogImpl.java:66)}

Outcomes