DX Application Performance Management

Discussion:

What is your source control and deployment management for APM artifacts?

Details:

There is more than a handful of files that are modified or customized within Application Performance Management installation, use, and administration.

To name a few:

   EM - Configuration

      IntroscopeEnterpriseManager.profile

      Domain.xml

      users.xml

      WebView/Introscope.lax files

      Management Modules

      custom email shell script

   EM - Java Script Calculators

   EP Agent (variant depending on host/OS)

      plugins

      configuration

      start/stop scripts

   Application Agent (java)

      .pbl

      .pbd

      IntroscopeAgent.profile

Database (postgresql)

      configuration/pg_hba.conf

With each of these artifacts, there, hopefully, should be a deployment plan along with some sort of test/verification plan to get the changes through some sort of software life cycle within multiple environments (test, pre-production, production).

Currently we have the majority of our artifacts on a network directory with no check-in / check-out control.  Our deployments are manual and custom per server.

Certainly, you could take that the APM, monitoring as not being a production impact and an acceptable loss if APM were to shutdown for some reason.  But if you did, once in the past, deploy a pbd that caused issues on the production environment, the first questions usually are, was the changes tested, what is your roll-back plan, how long will production have to be down, what changed?

So, what is your APM source control and deployment plans?

There was mention of two configuration tools on the community board as of late:

APM Studio

https://communities.ca.com/docs/DOC-231152665

Looks like the APM Studio might be able to help with the agent pbd/pbl into source control and deployment since it is a Eclipse plugin, but what about the other agents: epagent, IBM mq, webserver, browser agent?

Our environment performance agents help provide for some infrastructure performance monitoring (cpu, disk, memory, processes) and with that we have added quite a few Perl script plugins.    To make our scripts portable, we have moved all host specific configuration to the APM agent user home directory and pushed the IntroscopeEPAgent.properties to the APM agent user home also.  With this, we changed the EPACtrl.sh script to look for the ~/apmuser/config/IntroscopeEPAgent.properties file that has the specific plugins for that host within it.

EM Config Utility - 10.5 - command line tool

https://communities.ca.com/docs/DOC-231171532

There are a few options such as listing changes/customization that would be very helpful during an upgrade.  But to move this up a notch, would need an Eclipse plug-in in order to get the source control and hopefully deployment abilities.  I didn't see mention on the features of management of the non-configuration files such as JavaScript calculators, Management Modules, custom scripts (email),

There is plenty of elements within APM with the custom host metrics to be able to really have a full featured EM control GUI that would include a graphical smartstor management.  Hopefully smartstor is on the short list of being replaced with something else, but till then, the direction of being able to administrate an APM cluster enterprise managers would be very useful.

Thanks Billy for mentioning the ConfigUtil Doc. It is also documented in Knowledge Base Articles . Your feedback on it is appreciated. 

So, blazing a trail here...construction ahead...

In prep for a 10.0 to 10.5 upgrade started to look at the pain points at trying to keep the changed, custom, one off files in view.  At one point we had nine different epagents which was driving our system admins a bit batty.  So, I devised a way to combine all of the agents and have a more configuration driven epagent, so that we have a single epagent.

IntroscopeEPA.properties use of Environment Variables 

This does help start to frame the source control a bit better, just starting the fire with the epagent.

epagent

    <environment performance agent> This would be a full directory structure agent with all the required files.  

                lib/agent.jar

                bin/EPACtrl.sh

                logs

                plugin_one.pl

                plugin_two.sh

                plugin_three.java

               IntroscopeEPAgent.properties.aix

               IntroscopeEPAgent.properties.sles

               IntroscopeEPAgent.properties.rhel

            <Host>

                  syshome/wilyapm/config/IntroscopeEPAgent.properties

                  syshome/wily.apm/config/caapm.conf

                  syshome/wily.apm/config/plugin_one.properties

With a single epagent directory and all the environment or custom configuration being located within the /syshome/wilyapm/config directory, we can mass deploy the single epagent to all servers and all environments.

We would tag the tree when there was a change and then when we winscp the contents (being aware of binary for jar/so and text for basically everything else) we create a zip/tar/gzip with the APM release and tag value

     epagent_10.0.0.12_20170124.zip

So this is the epagent structure that I am planning on using within source control to manage the sea of custom and changed files.

So far, this structure has cut down on the variants and explaining what goes where such as the AIX epagent goes under epagent/production/AIX/host and that one is RedHat so it goes into the epagnet/production/RHEL/Host.  But the only real difference is the existence of different plugins and the introscopeEPAgent.properties file.

When I get around to the Application agents (websphere, weblogic, .NET) and have to deal with pbl/pbd and .profile I'll try to post what I am planning on doing.

      Cheers,

Billy

Thanks Hiko,

Wished I had enough permission to run an installation script on a server, but sadly, the APM admin typically gets a sandbox directory that the agent is copied into and we can make changes to things inside the directory.

If we provide a single agent directory (epagent, websphere1.6, webphere1.7, weblogic1.7) and provide simple instructions to put the contents of the zip/tar/gzip into a specific directory, then a few root command like chown, chmod, adding the EPACtrl.sh stop/start to the server start and stop, things go rather well.

Currently we have to instruct how to change the MOM host name in the application agents, but might look at externalizing that also since we already have a server conf file (/syshome/wilyapm/config/caapm_env.conf) that has a single variable with value of "PROD, NON_PROD, TEST".  Gotta go see if there is a way of injecting the environment on the JVM.   

Once we get into the system admins or WebSphere admins running an install on each server (+1500) now we have more effort and time.  With basically using the system admin's package deployment software, we can throw an epagent on a server, have it start and we are done.

For the application agents, typically the websphere admin will script all of the agent's elements so when a server is created everything is already in place ready to go.

Another issue we will be facing that I'm still noddling, is how to inject custom pbd within an agent dependent on which application server software and which applications are present.  Was thinking on having all of the 1.6 and 1.7 custom pbd defined and included for all instances.  If the specific class inclusion or exclusion is not within the JVM, what harm would it cause?  I'll investigate that and see if I can come up with something.

Thank you,

Billy