Release Automation

Environment

• Linux (Fedora was used while writing this article, thus recommended commands may very slightly for other Linux versions.)
• CA LISA Release Automation version 5.0.1.
• Note: A similar article exists for AIX environments: https://communities.ca.com/thread/106472795

Problem

As most sysadmins know, Java updates are many and frequent. This can cause a headache for the novice to CA Release Automation, as updating Java on an agent machine will break the agent so that the service can longer start, resulting in it appearing offline in Automation Studio.

The unfortunate user experiencing this problem will see the following error when attempting to run the agent:

cp: cannot stat ‘(Agent Root)/jre/bin/java’: No such file or directory

Starting Nolio Deployer Agent Service...

Waiting for Nolio Deployer Agent Service...........................................

WARNING: Nolio Deployer Agent Service may have failed to start.

Cause

• When the agent is first installed, symbolic links are setup in the agent's /jre/lib/ directory and /jre/bin/ directory that point to the same respective folders within the location of the JRE installation at the time of the agent's installation.
• If Java is updated, a new installation directory is created for it, changing the location of the /lib and /bin directories. This breaks the agent's symbolic link since it is no longer pointing to the current version's directories.

This can best be illustrated with an example: Assume that at the time of the agent's original installation, Java OpenJDK version 1.7.0.60 was running on the machine.  Java is later updated to version 1.7.0.65. After this update, the agent fails to start and the error in question appears. The agent cannot start now because it is still configured with its original symbolic links pointing to the now outdated 1.7.0.60 version.

Solution

1. Navigate to (Agent Root)/jre/lib.
2. Check where the symbolic link "ext" is pointing to (ls -lrt), which is likely the location of the previously installed JRE version.
3. Navigate to (Agent Root)/jre/bin.
4. Check the symbolic link "NolioAgent" using the same command (ls -lrt).
5. Run "whereis jvm" to find where the current JVM is located. It may return more than one location, but typically the /usr/lib/jvm location is the one of interest.
6. Navigate to the location returned by the "whereis" command then drill down to locate the JRE's /lib and /bin directories. Typically these directories are under /usr/lib/jvm/(java version)/jre.
7. Return to (Agent Root)/jre/lib.
8. Update the symbolic link to point to the new JRE version's /lib/ext location. Example: "ln -nsf /usr/lib/jvm/(java version)/jre/lib/ext ext" (where "ext" is the name of the symbolic link identified in Step 2).
9. Run "ls -lrt" again to verify the new symbolic link is now correctly updated.
10. Return to (Agent Root)/jre/bin.
11. Update the symbolic link to point to the new JRE version's /bin/java location. Example: "ln -nsf /usr/lib/jvm/(java version)/jre/bin/java NolioAgent" (where "NolioAgent' is the name of the symbolic link identified in Step 4).
12. Verify the change by running "ls -lrt" again.
13. Start the agent. It should now start successfully then appear online in Automation Studio.

Let's clarify the above steps by applying them to the previously mentioned example:

• The command "whereis jvm" reveals that the JVM directory can be found at /usr/lib/jvm and /usr/share/jvm.  The latter can be ignored; we are only concerned with the first /lib/jvm location.
• Java was updated so that the new JRE's location is found by drilling down from the above JVM location, which is discovered to reside at /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65-2.5.1.3.fc20.x86_64/jre.
• The above /jre folder contains the new /lib/ext and /bin/java locations that we need to point to.
• We then navigate to (Agent Root)/jre/lib and run "ls -lrt". We see the outdated symbolic link: "ext -> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.5.0.1.fc20.x86_64/jre/lib/ext" (It still points to 1.7.0.60).
• Repeating the same in (Agent Root0/jre/bin reveals the same problem: "NolioAgent -> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.60-2.5.0.1.fc20.x86_64/jre/bin/java".
• Time to update the links!  Back in the agent's /lib directory, we update the link accordingly: "ln -nsf /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65-2.5.1.3.fc20.x86_64/jre/lib/ext ext".
• Repeat this for the link in the agent's /bin directory: "ln -nsf /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65-2.5.1.3.fc20.x86_64/jre/bin/java NolioAgent".
• Run "ls -lrt" in each of the two directories to verify the links now point to the newly installed 1.7.0.65 version.
• With the symbolic links updated, we start the agent again, and now it runs!  After a few moments, it appears online once again in Automation Studio.

Simply repeat this anytime Java is updated on the machine.