JMertin

Docker - using a passive container as file-system volume

Blog Post created by JMertin Employee on Aug 6, 2018

Introduction

It happens very often that on micro-services environment, one wants to add a special application for a dedicated purpose.
Imagine in the APM environment, one deploys an application spawning 30 containers (micro-services), in 20 of these, a java-application is started, and these are to be monitored.

The usual approach would be to add the java-agent code into each and every image requiring the agent to be ran on.

The alternative approach would be to provide a file system with the Agent code as "external partition" to be mounted by the image (passive image, means it is an Image in terms of Docker, it will however have no ENTRYPOINT per se, except a /bin/true to validate the docker build process.), 
This also provides the big advantage that in case the Java Agent code/binaries are changing, one only needs to update one image and restart the containers, and the binary content can be shared among as many running containers as wanted.
One thing however needs to be taken into account: One needs to export the Configuration file and Log-Directory to a different location.

Loading the file-system from that image will however require the target image to be modified to actually use the Agent!

Building a passive docker image

Required files

The building blocks for such a passive image will be the Dockerfile and the actually Agent Binary package. For the following example, we will have the following files in our wily_agent_unix directory:

Dockerfile
IntroscopeAgentFiles-NoInstaller10.7.0.94default.unix.tar
  • The IntroscopeAgentFiles... are downloaded from the Official CA Repositories.
  • The Dockerfile will look like this:
# Create the Docker-Container from the smallest possible source.
FROM jeanblanchard/alpine-glibc:latest
# Set the Agent Version into a Var. This makes it easier to use it at different locations
ENV INTROSCOPE_VERSION=10.7.0.94
# Set the Label. Information for which version of the Agent is running
LABEL agent.version=${INTROSCOPE_VERSION}
# Create the Agent target directory
RUN mkdir -p /opt/Agent
# Add the Tar-File to the Target Directory (*)
ADD IntroscopeAgentFiles-NoInstaller${INTROSCOPE_VERSION}default.unix.tar /opt/Agent
# Make sure the group is able to write the files (required for OpenShift/Kubernetes)
RUN chmod g+w -R /opt/Agent
# This would be the Entrypoint - which returns a true statement
CMD /bin/true

*) As Docker-Filesystems are all tar-archives on the OS, Adding a "tar" file is the same as Un-taring it on the target location!

Building the image

It is always best to build images and append the build-tag/version straight away. This - to make sure the ":latest" (appended by default) is not overwriting an existing running image.

Build the image with:

~# docker build -t wily/agent-unix:10.7.0.94 .
Sending build context to Docker daemon  182.5MB
Step 1/7 : FROM jeanblanchard/alpine-glibc:latest
---> 273a23f56cf7
Step 2/7 : ENV INTROSCOPE_VERSION=10.7.0.94
---> Using cache
---> f07bfb0ce5a3
Step 3/7 : LABEL agent.version=${INTROSCOPE_VERSION}
---> Using cache
---> 3fd92296c1ad
Step 4/7 : RUN mkdir -p /opt/Agent
---> Using cache
---> 8b8a3e0aa716
Step 5/7 : ADD IntroscopeAgentFiles-NoInstaller${INTROSCOPE_VERSION}default.unix.tar /opt/Agent
---> Using cache
---> ac61ca7335df
Step 6/7 : RUN chmod g+w -R /opt/Agent
---> Using cache
---> 03a77bb1719d
Step 7/7 : CMD /bin/true
---> Using cache
---> 3e37f210a933
Successfully built 3e37f210a933
Successfully tagged wily/agent-unix:10.7.0.94

If one checks the images, the just created image should show up:

[wily_agent_unix]$ docker images | grep "10.7.0.94"
wily/agent-unix                                          10.7.0.94                3e37f210a933        2 minutes ago         107MB

To be able to use this image afterwards, we have to start its content into a container.

~# docker create -v /opt/Agent --name agent-unix wily/agent-unix:10.7.0.94 /bin/true

This will create a container called "agent-unix" that can be used by other running containers.
Note that in a complex environment, make sure the agent version is appended to the name, or it will become a mess pretty fast. 
For this example, we just use "agent-unix".

Inserting the passive image file-system into another container

Inserting the passive image file-system into another container involves an existing application already known to be running.

Note that to run the Agent inside of the new container, the ENTRYPOINT (start scripts) need to be adapted so the application is started with the Agent. To prepare the application, a simple trick can be used, notably start the container by "hijacking" the entrypoint and drop one to a shell.

The actual link is done using the "--volumes-from <container-name>" directive. The file system will be mounted where the creation process placed it (here: "create -v /opt/Agent").

[ indexer]$ docker run -ti --entrypoint=/bin/sh --volumes-from agent-unix mertin/ingestion:latest
sh-4.2$ cd /opt/Agent/
sh-4.2$ pwd
/opt/Agent
sh-4.2$ ls -l
total 696
-rw-rw-r-- 1 root root 706032 Jun 19 19:58 APM_ThirdPartyLicensingDoc.txt
drwxrwxr-x 9 root root   4096 Aug  2 13:32 wily
sh-4.2$ cd wily/
sh-4.2$ ls -l
total 4924
-rw-rw-r-- 1 root root 5009453 Jun 19 19:58 Agent.jar
drwxrwxr-x 2 root root    4096 Aug  2 13:32 common
drwxrwxr-x 2 root root    4096 Aug  2 13:32 connectors
drwxrwxr-x 4 root root    4096 Aug  2 13:32 core
drwxrwxr-x 8 root root    4096 Aug  2 13:32 examples
drwxrwxr-x 3 root root    4096 Aug  2 13:32 extensions
drwxrwxr-x 2 root root    4096 Jun 19 19:58 logs
drwxrwxr-x 2 root root    4096 Aug  2 13:32 tools

 

In the current example, the indexer is actually ran from the "/opt/ca/indexer/startIndexer.sh", so all that needs to be done is the following.

Copy the Agent configuration file to a new location, and adapt it to the local environment, and make sure there is another log-target directory.

sh-4.2$ mkdir -p /opt/ca/indexer/wilyAgent/core/config /opt/ca/indexer/wilyAgent/logs
sh-4.2$ cp -r /opt/Agent/wily/core/config/IntroscopeAgent.profile /opt/ca/indexer/wilyAgent/core/config/

Configure the Collector into the configuration-file (the sed statement is all on one line).

sh-4.2$ sed -i "s/agentManager.url.1=localhost:5001/agentManager.url.1=${APM_EM_PORT_5001_TCP_ADDR}:${APM_EM_PORT_5001_TCP_PORT}/g" /opt/ca/indexer/wilyAgent/core/config/IntroscopeAgent.profile

Of course, all that need to be made persistent into the ENTRYPOINT script, which in this case is "/opt/ca/indexer/startIndexer.sh".

The relevant parts of the script, with the applied changes could look like this:

## We want to have the Agent - or else we wouldn't have started this container
WILY_AGENT="-javaagent:/opt/Agent/wily/Agent.jar -DagentProfile=/opt/ca/indexer/wilyAgent/core/config/IntroscopeAgent.profile"
## We need to copy some files around
mkdir -p /opt/ca/indexer/wilyAgent/core/config /opt/ca/indexer/wilyAgent/logs
cp -r /opt/Agent/wily/core/config/IntroscopeAgent.profile /opt/ca/indexer/wilyAgent/core/config/
sed -i "s/agentManager.url.1=localhost:5001/agentManager.url.1=${APM_EM_PORT_5001_TCP_ADDR}:${APM_EM_PORT_5001_TCP_PORT}/g" /opt/ca/indexer/wilyAgent/core/config/IntroscopeAgent.profile
# Tell the world we have the java agent enabled.
echo "Java Agent enabled, reporting to ${EM APM_EM_PORT_5001_TCP_ADDR}:${APM_EM_PORT_5001_TCP_PORT}"
# Actual Java process start (With the Wily Agent code)
java $WILY_AGENT -Xms$min_mem_limit -Xmx$max_mem_limit -Dlog4j.configurationFile="file:/opt/ca/indexer/config/indexer.log4j.properties" -jar indexer.jar

The image (in this case "indexer") needs to be rebuilt with the modified ENTRYPOINT script. Of course, it would be way "cooler" to have the inclusion of the Java Agent be dependent of a flag defined in the startup environment.
For example, one could use "JAVA_AGENT=on/off" to have it triggered. Even if the environment / code would be available, the Agent would not be enabled. But in the events it needs to be enabled, editing the Tag and restarting the container would enable it on the fly and give valuable information on teh running java application.

Things to consider

A very good candidate for this could also be the java version to be deployed. It would reduce the number of images to be actively rebuild, as only the passive container volume for java would need to be rebuild. The tech would then however need to figure out a way to provide the correct Java version/java home directories to be usable.

Using passive container volumes may ease things when upgrading the Agent release or java version. One however will need to take into account the complexity of the setup, as every volume, Agent/Java version will add a branch to the complexity tree. So good documentation is a must/requirement when working with this type of setup.

Outcomes