DX Application Performance Management

Author: Mary_Clark

Document ID:  TEC596118

Last Modified Date:  7/25/2013

• Products
  • CA Application Performance Management
• Components
  • INTROSCOPE

Description:Seeing the following messages in the EM (MOM or Tim Collector) logs:[PO:AppMapAgentPO Mailman 1] [Apm.Data.Model] [ Method : saveAllEdges, RootCause : A ResourcePool could not acquire a resource from its primary factory or source., Error Message : Could not open Hibernate Session for transaction org.hibernate.exception.GenericJDBCException: Cannot open connection][ERROR] [WrapperSimpleAppMain] [Apm.Data.Model] APM Database is not reachable. Please check the connectivity…org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: Cannot open connection; uncategorized SQLException for SQL [???]; SQL state [null]; error code [0]; Connections could not be acquired from the underlying database!; nested exception is java.sql.SQLException: Connections could not be acquired from the underlying database!These errors are indicating connection to the APM Database is not available.Solution:There is only one APM Database in a cluster. It may be installed locally to the MoM or a Collector, or it may be on a remote server.APM supports either Oracle or Postgres databases. This article focuses on problems you may experience when attempting to connect to an Oracle APMDB database and discusses what to do about them.Problem: EM's are unable to connect to the APM Database with the following exception:
java.sql.SQLException:Connections could not be acquired from the underlying database!Enlist help from your Oracle DBA as necessary to perform the following tests.Restart the APMDB Oracle database listener.
Restart the APMDB Oracle database instance.If you are able to use sqlplus or SQLDeveloper to access the database, check the version of the APMDB to be sure it is the same as the version of the APM software you are using by getting the output of:
SELECT ts_db_versions FROM ts_domains;You can delete the OSGI cache to attempt a clean connection as follows:
delete all org.* folders present in the <EM_Home>/product/enterprisemanager/configuration folder and restart all EMs (MoM and Collectors).

1. Test if you can connect to an APMDB on Oracle using SQLDeveloper or sqlplus by login using the credentials in tess-db-cfg.xml. Ideally, this should be done from the MOM server to confirm that the MOM can reach Oracle across the network using these credentials. If you cannot install sqlplus or SQLDeveloper on the MOM server, then doing this test from somewhere else will at least confirm that Oracle is up and the Oracle TNSLSNR (TNS Listener) is running for the APMDB instance.
   
   
2. Be sure the login account is not locked and that this account is the APMDB schema owner in Oracle.
   
   
3. If the account is locked, double check the password saved in tess-db-cfg.xml on all EMs since attempting to connect with the wrong password will again lock the account if the wrong password is used repeatedly.
   
   
4. Ping or netstat port 1521 (the Oracle default communications port) on the Oracle box. The listener should be attached. Do the same from the MOM box back to the Oracle box on port 1521 to be sure you have open communications on the Oracle port in both directions.
   
   netstat -vatn | grep 1521 (this is the netstat command for Linux. It may have different arguments for Windows)
   
   
5. Examine or upload to CA Support the Oracle listener log from the Oracle server.
   
   Check the Oracle logs for ORA-#### errors. Oracle errors are well-known and can be "Googled".
   
   
6. If there are not enough connections allocated by Oracle (there will be an ORA-#### error indicative of this), ask your DBA to increase the shared pool size for the Oracle database.
   
   
7. Check the Windows system timezone on each EM (MOM & Collector) server and be sure it is the same as the timezone on the Oracle server.
   
   ORA-00604: error occurred at recursive SQL level 1
   ORA-01882: timezone region not found
   
   If the system timezone is different for the Oracle server than for the EMs, add the following JVM argument to each EM's IntroscopeEnterpriseManager.lax file to override the default system timezone on the APM server:
   -Duser.timezone=UTC (where UTC represents the timezone used by Oracle)
   
   If you are still unable to connect, put the MOM in DEBUG logging mode, restart all EMs in the cluster, attempt to view items in the AppMap in the APM Workstation, and upload the debug level log and the tess-db-cfg.xml file from the MOM to CA Support for analysis.
   
   
   1. To turn on DEBUG on the EM (or MOM If this is a cluster) note that this is a hot config property and does not require a restart of the EMs.
      
      
   2. Edit the <EM home>/config/IntroscopeEnterpriseManager.properties file
      
      
   3. Locate the line log4j.logger.Manager=INFO, logfile, console
      
      
   4. Modify this line so that it says log4j.logger.Manager=DEBUG, logfile, console
      
      
   5. Save and exit the file.
      
      
8. Attempt to access the Triage Map in the APM Workstation, then upload the following files to CA Support:
   
   
   1. <EM home>/config/IntroscopeEnterpriseManager.properties file
      
      
   2. <EM home>/config/tess-db-cfg.xml
      
      
   3. Entire zipped contents of <EM home>/logs
      
      
   4. Entire zipped contents of <EM home>/install/*.log

Search the Entire CA APM Knowledge Base