DX Application Performance Management

The most common reason a set of dependent docker containers do not start up correctly, is because one of the containers did not come up correctly.

Analyzing many docker-containers, I have noticed very often "developers" to add a startup script that looks like the below (I admit, this is one of the worst versions I have seen):

java -jar ${javajunk} &
sleep 6;
java -jar ${javajunk2} &
sleep 10;
java -jar ${javajunk3}

That is the portion of a script that controls 3 java processes to be started.

We have in that example several issues, things that should definitely not be done!

1. Docker containers are supposed to handle micro-services. Here, we start 3 java programs each using at least 2GB of Ram etc. Micro-Service is the wrong name here, it should be called Macro-Service.
   The only understandable reason to make a docker-container out of this is to have it all packaged and pre-configured to run.
2. Programs started into the background. The Docker-Server will not be able to know if the application started or not.
3. Depending on the host-load, 6 seconds may not be enough. The best part is 10 seconds for the 3rd app to start. Depending on the induced load by app 1 and 2, it may be too short. So if application 3 really depends on application 1 and 2 being started, this can fail. Especially if installed in a customer environment where the host already hosts some other applications causing high CPU load.
4. Dependencies cannot really be handled here. Means, java app 2 needs to be started after java app 1 is up.

So - what is the solution here?

1. Start one container per application. If that is not possible, an internal job-handler needs to be written (shell-script is well possible).
2. Never start a program in the background. When starting the program through a shell-script, prepend "exec" to the call to replace the current shell with that program. In that case, if the application crashes, the docker-server will know.
   Note that this will only tell the docker-server that the application has been started. It will not know if it is ready.
3. using a. and b., start one container per application and use docker-compose to start the containers in order.
   If that is not possible. Implement inside the applications a "ready" status indicator that can be passed on to other scripts or polled - so they know it is ready and the current app can be started. In the regular linux world, using mysql one can issue a regular mysql call to the mysql-command and check the status of the mysql-server.
   

   mysqladmin --host=db1.example.org --user=monitor status
   Uptime: 884637  Threads: 1  Questions: 5534424  Slow queries: 144  Opens: .....

    If the status returns a valid entry (known), the next container can be started.
   This will make the order and in time startup of the applications doable. All other implementations, especially java applications which tend to take ages to start, is just a wild guess.
4. all of the above together.

The main issue will be, if as stated before, one app depends on another one and there is no way to tell when that app is ready. Without an internal "ready" status, this will not really be possible and lead to ugly hacks as "sleep 10" ...