bwcole

Adventure in Installing 10.5.2 sp1 on RHEL

Blog Post created by bwcole on Sep 15, 2017

We have had APM versions 9.0.5.6, 9.1.1.1, 9.6 and 10.0 all on SLES and we are moving to RedHat this time and instead of trying to move everything over to RHEL then upgrading, we are doing a clean install 10.5.2 sp1.

 

There are a number of things that, in the past, we had issues and made changes:

 

1. Uncomment concurrency pool max in IntroscopeEnterpriseManager.properties on the MOM

   transport.override.isengard.high.concurrency.pool.max.size=10

 

2.  Remove "console" from IntroscopeEnterpriseManager.properties since the EMCtrl.sh will pipe all console messages to the em.log, and the IntroscopeEnterpriseManager.log.

 

3.  Route all of the logfile to logfile1 in the EMCtrl.sh shell script so all of the log messages are directed to a managed (by log4j) for size and rotation.

 

4.  Change the log file number and size to something reasonable.  Default it is set to 4 x 200 MB.  I change this to 10 x 20 MB and typically get several months of logging in a single log.

 

5.  In the EMCtrl.sh, basically hard code the EM path since the EMCtrl.sh could be called from a different directory and the logic at the top of the script will take the current directory, pop up one directory and call that the EM Home.

 

6.  Found that I had to set a java.io.tmpdir in the Introscope_Enterprise_Manager.lax file.  We pushed all of the CA APM installs into the /opt/CA/APM structure with the EM having a tmp directory then on the lax.nl.java.option.additional, we appended:    -Djava.io.tmpdir=/opt/CA/APM/EM/tmp

 

7.  Create multiple mount points for the different data sets

   /opt/CA/APM/EM   - Enterprise Manager install directory

   /opt/CA/APM/EM/data  - smart stor data

   /opt/CA/APM/EM/trace - trace storage

   /opt/CA/APM/EM/threaddump - Thread Dump data so it won't over run the others

 

8.  You might be tempted to just copy your previous version configuration files to the new installed version.  I didn't.  I did a text compare on every file I knew about and then did a line by line change to the new files.

We had changes to the following files or they had passwords that were text till the EM was started.  Highly suggest before starting the EM for the first time, copy the <em_home>/config files (.xml / .properties) and use source control to keep track of your changes.  Take the time, each and every change, document, tag, add a version header so you can find the changes.  Copy the line, comment the original out and change the copy line.  Add a comment above with support case numbers or a brief description why you are doing the change.

   apm-event-thresholds-config.xml

   domains.xml

   IntroscopeEnterpriseManager.properties

   IntroscopeWebView.properties

   realms.xml

   server.xml

   tess-db-cfg.xml

   users.xml

 

9.  Source Control

We didn't have source control on the first couple of versions of APM and by the time we got our source control setup, we didn't have the original out of the box files to use as a baseline.  Yes, it does add about double the time but NEVER edit a file on the server.  Pull the file down, get it checked into source control, make the change, get it checked in, then deploy the file to the server.  

 

Our source control structure is CA APM/Enterprise Manager/<server name>/opt/CA/APM/EM/[config | bin] so the file in source control is in the same directory structure as the deploy target.

 

10.  Don't forget your Postgres IP white list, add your enterprise managers and if you access postgres through a SQL tool, add your workstation static IP addresses.  Restart postgres after your changes

      /<apm db>/database/data/pg_hba.conf

 

11.  Your first dashboards should be the MOM_Infrastructure and Collector management modules (in the <em_home>/examples.  Take the time to set them up for your environment.  Found that I didn't request enough drive space in the EM drive.

 

12.  If you based any of your management modules on out of the box examples, deploy the out of the box management modules and compare with the previous version.  It may have some changes that might be useful.

 

13.  Deploy the WebView module and get it configured

Don't freak if you deploy the Webview management module and nothing is appearing.  you have to access WebView for the agent to generate some of the metrics.

 

14.  Don't assume you will remember anything you did during the install process.  Screen shot, document, or video the screen while you are installing.  Sure, you can go in after the fact and alter/correct the configuration but if you walk through the install, and document, the next EM you install will be easier.

 

15.  Know your resources

   Plan out how much RAM to allocate to the EM, WebView.  Your mount points.  User names for the various servers.  Server name.  

 

16.  If you don't have a known reason to change a default port, don't change it.  Lots of times the default ports are assumed in documentation and agent installations.

 

17.  Secondary artifacts:

   custom email script - really good time to dust it off and make sure it is being used, doing it's job

   JavaScript calculators - really good time to check the agent expressions to make sure the agent targets still exist

   Management Modules - Good time to do some house cleaning

   Server  stop/start scripts - If the servers are stopped/started want to check to insure the new version has been tested with the process

 

18.  Test everything, don't assume that it will just work

Review access - webview (chrome/ie/firefox), workstation

Review dashboards are rendering correctly

Alerts are able to get to the email server

The MOM and collectors are able to get to the APM DB

 

   CHECK the logs!

Outcomes