We are not happy with the performance of CA APM.
It's almost unusable in our environment.
I opened a support case to CA (00151073) but I received a list of things to check.
We have done those steps 8 months ago and nothing changed.
CPU usage is %1, 8 GB of free memory, no disk IO and CA APM response time is more than 3 minutes.
Why?
Hi Sergio,
Thank you for the reply.
I don't know this product so I need detailed instructions.
- What is verbose mode and how will I enable it?
- if possible restart the MoM with a refresh new smartstor db / How will I do this? Do you mean delete all data?
How will I restart MOM? By the weay, what is MOM?
- Make sure clocks of all EMs are in synch, you must configure a NTP server. // MY colleague says things get worse when NTP is enabled. He has a script to sync time.
Since I do not know the product, it is not possible to follow your instructions.
Is EM the same thing as Collector?
What is MOM?
Hi:
It may be helpful to review the APM Overview document (part of official documentation.) which contains an architectural overview, glossary and more.
Thanks
Hal German
APM Support
These are just copy/paste instructions which do not have anything towards a solution.
I've received all this info many times but the result is the same: poor performance, unusable product.
You wrote:
-[WARN] [PO:client_main Mailman 1][Manager.AsyncQueryResultStateMachine] Received tardy historical data from slow collector
This other error indicates a performance issue in the cluster (MOM and collector communication)
Yes, I'm aware of the performance problem and that's the reason I opened a case.
Hi:
I mark as assumed answered since a case is opened .If you are asking what is a MOM and common terms, this is covered in the APM Glossary. At this point please work through the case for further assistance.
Thanks
Hal German
APM Support
Hi,
I've also looked at your case, you have a MOM and two collectors, they are all Enterprise Managers but they have different roles.
The MOM (manager of managers) is the server that you connect to to see data, this Is where you load your dashboards,
You have two collectors that are the ones that the agents send the data to, the MOM then connects to the collectors so you get to see all data from your collectors in one place.
I noticed in the MOM configuration that the live metric limit for the MOM was raised to 1 million and the historical metric limit for the MOM was raised to 2.5 million.
The default values are 500,000 for live data and 1.2 million for historical data, so they have been doubled from default values.
The more historical data you have, generally the slower will be the response from the product - I'm mentioning his because someone clearly decided to raise those values in the apm-events-thresholds-config.xml file in the config folder of the Enterprise Manager.
We can guess that you managed to hit a metric clamp on the server, when that happens you would see data missing in the system and data looking greyed out, so it is normal to raise the values.
However it is vitally important in those scenarios that you look at the data coming into the system as well - if your agents are reporting a large amount of metrics, not only does that give you a large amount of data, generally agents that report a large amount of metrics also affect the performance of the monitored application.
As Sergio said, we need to see the logs from the collectors and the other data he has requested, there is a very old saying in our product that your APM cluster (you have a cluster because you have a MOM and collectors) is only as good as your worst performing collector.
Based on the data that Sergio has requested, it would be much easier to suggest something concrete to improve the performance of the system.
Thanks,
David
Can you tell me which parameter should I correct?
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<apmEvents xsi:schemaLocation="http://www.ca.com/schema/apm/monitor apm-events-thresholds-9.1.xsd" xmlns="http://www.ca.com/schema/apm/events" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<clamps>
<description>
APM Clamps and Important Events configuration file. Please refer apm-evets-thresholds-config.xsd for schema file.
This configuration configures the pre-defined the clamps and important events. Each clamp/important event
threshold value will be applied for each agent or entire EM depends configuration.
</description>
<clamp id="introscope.enterprisemanager.agent.metrics.limit">
<description>
Most properties here put limits on # of metrics.
The last limits metric data. The metric clamping properties support hot config.
Per Agent limit. Takes into account live and historical metrics.
</description>
<threshold value="50000"/>
</clamp>
<clamp id="introscope.enterprisemanager.transactionevents.storage.max.disk.usage">
<description>
The maximum desired disk usage in MB for the trace storage. If this maximum is exceeded,
then the daily historical trace stores will be deleted starting with the oldest first
until the total historical trace storage size is below this value.
The current days trace store actively storing traces will not be deleted even if
its size exceeds this property value. The size of the trace index is not considered
when determining what historical trace stores to delete.
</description>
<threshold value="1024"/>
</clamp>
<clamp id="introscope.enterprisemanager.metrics.live.limit">
<description>
Per EM limit. Takes into account only live metrics (i.e. currently reporting from Agents)
</description>
<threshold value="1000000"/>
</clamp>
<clamp id="introscope.enterprisemanager.metrics.historical.limit">
<description>
Per EM limit. Takes into account metrics with Smartstor data (i.e. live and historical metrics)
</description>
<threshold value="2500000"/>
</clamp>
<clamp id="introscope.enterprisemanager.agent.connection.limit">
<description>
Limits the # of agent connections
</description>
<threshold value="400"/>
</clamp>
<clamp id="introscope.enterprisemanager.disconnected.historical.agent.limit">
<description>
Limits the # of dead/disconnected agents that are mounted state to minimize
the runtime memory of EM.
</description>
<threshold value="400"/>
</clamp>
<clamp id="introscope.enterprisemanager.events.limit">
<description>
Limits # of Events Processed by EM Per Interval
</description>
<threshold value="1250"/>
</clamp>
<clamp id="introscope.enterprisemanager.agent.trace.limit">
<description>
Limits # of Transaction Events Processed Per Interval(Per Agent)
</description>
<threshold value="1000"/>
</clamp>
<clamp id="introscope.enterprisemanager.agent.error.limit">
<description>
Limits # of Error Events Per Interval
</description>
<threshold value="10"/>
</clamp>
<clamp id="introscope.clw.max.users">
<description>
Limits the number of clw users
</description>
<threshold value="40"/>
</clamp>
<clamp id="introscope.workstation.max.users">
<description>
Limits the number of workstation users
</description>
<threshold value="40"/>
</clamp>
<clamp id="introscope.enterprisemanager.collector.cdv.max">
<description>
Limits the number of CDV connections
</description>
<threshold value="5"/>
</clamp>
<clamp id="introscope.enterprisemanager.transaction.discovery.max.nonidentifying.components">
<description>
Transaction Discovery Processor (autogen) Maximum Number of Non-Identifying Components
</description>
<threshold value="50"/>
</clamp>
<clamp id="introscope.enterprisemanager.max.number.domain.configuration.changes">
<description>
Maximum number of domain configuration changes returned by query. 0 is unlimited.
</description>
<threshold value="0"/>
</clamp>
<clamp id="introscope.enterprisemanager.max.transaction.user.groups">
<description>Maximum number of transaction user groups.</description>
<threshold value="5000"/>
</clamp>
<clamp id="introscope.enterprisemanager.max.application.user.rows">
<description>Maximum number of application users retrieved.</description>
<threshold value="1000"/>
</clamp>
</clamps>
<importantEvents>
<importantEvent id="introscope.enterprisemanager.harvest.duration">
<description>
EM Harvest Cycle(ms).
</description>
<threshold value="3500"/>
</importantEvent>
<importantEvent id="introscope.enterprisemanager.cpu.usage">
<description>
EM CPU Usage(%)
</description>
<threshold value="60"/>
</importantEvent>
<importantEvent id="introscope.enterprisemanager.smartstor.duration">
<description>
EM Smart StorTime(ms)
</description>
<threshold value="3500"/>
</importantEvent>
<importantEvent id="apm.events.thresholds.config.error">
<description>
Important Event to notify the Introscope admin users of apm-events-thresholds-config.xml file's parsing
errors.
</description>
</importantEvent>
<importantEvent id="introscope.enterprisemanager.database.connectivity">
<description>Database Connectivity</description>
</importantEvent>
<importantEvent id="introscope.enterprisemanager.harvest.cycle.diff">
<description>Collector considered as slow if number of timesliceindex behind collector is relative to MOM
Harvest.
</description>
<threshold value="4"/>
</importantEvent>
</importantEvents>
</apmEvents>
If someone can show me what parameter should be changed, then it will be great
########################################################################
#
# Introscope Enterprise Manager Preferences
#
# CA Wily Introscope(R) Version 9.1 Release 9.1.5.0
# Copyright (c) 2013 CA. All Rights Reserved.
# Introscope(R) is a registered trademark of CA.
#######################################################################
#######################
# Enterprise Manager Name
#
# ================
# Set the name of this EM.
# Default is: Introscope Enterprise Manager
#introscope.enterprisemanager.name=
#######################
# Available Processors
#
# ================
# Set the number of processors available to this EM.
# Use this property if there is more than one EM running on the host.
# Default is to use all available processors.
#introscope.enterprisemanager.availableprocessors=
#######################
# IP Address Binding
#
# ================
# The property for binding all EM communication channels (including the embedded
# web server) to a specific local IP address. When not configured, the EM will
# accept incoming connections on all local addresses (the wildcard address).
#
# For IPV6 environments this must be an IPV6 formatted address.
# introscope.enterprisemanager.ipaddress=(set to a valid IP address on EM machine)
#######################
# Port Settings
#
# ================
# The EM is configured with several communication channels.
# To enable a communication channel, add the channel id to the
# comma separated list of enabled channels. Likewise, a communication
# channel may be disabled by removing it from that same list.
#
# All communication to the EM is obfuscated, and passwords are always encrypted.
# The enabled communication channels.
introscope.enterprisemanager.enabled.channels=channel1
# Use this to enable both the default channel and secure channel.
#introscope.enterprisemanager.enabled.channels=channel1,channel2
# The default communication channel.
introscope.enterprisemanager.serversocketfactory.channel1=com.wily.isengard.postofficehub.link.net.server.DefaultServerSocketFactory
introscope.enterprisemanager.port.channel1=5001
# The secure (SSL) communication channel.
introscope.enterprisemanager.serversocketfactory.channel2=com.wily.isengard.postofficehub.link.net.server.SSLServerSocketFactory
introscope.enterprisemanager.port.channel2=5443
# Location of a keystore containing certificates for authenticating the EM to clients.
# Either an absolute path or a path relative to the config directory.
# On Windows, backslashes must be escaped. For example:
# introscope.enterprisemanager.keystore.channel2=C:\\Introscope\\config\\internal\\server\\keystore
introscope.enterprisemanager.keystore.channel2=internal/server/keystore
# Set this property to true to enable encryption of introscope.enterprisemanager.keypassword.channel2
# Once introscope.enterprisemanager.keypassword.channel2 is encrypted,
# this property is automatically set to false.
#
# To change the existing password, enter the new password and set this property to true.
# Note: If this property is set to true and the password is not changed, the existing encrypted password will be encrypted again.
# If password field for a new channel is configured, add the corresponding
# plaintextpassword field and set it to true to enable encryption.
introscope.enterprisemanager.keypassword.channel2.plaintextpassword=true
# The password for the keystore.
introscope.enterprisemanager.keypassword.channel2=password
# Location of a truststore containing trusted client certificates.
# Either an absolute path or a path relative to the config directory.
# On Windows, backslashes must be escaped. For example:
# introscope.enterprisemanager.truststore.channel2=C:\\Introscope\\config\\internal\\server\\keystore
# The truststore is optional. It is needed only if client authentication is required.
# If no truststore is specified, the EM trusts all client certificates.
#introscope.enterprisemanager.truststore.channel2=internal/server/keystore
# Set this property to true to enable encryption of introscope.enterprisemanager.trustpassword.channel2
# Once introscope.enterprisemanager.trustpassword.channel2 is encrypted,
# this property is automatically set to false.
#
# To change the existing password, enter the new password and set this property to true.
# Note: If this property is set to true and the password is not changed, the existing encrypted password will be encrypted again.
# If password field for a new channel is configured, add the corresponding
# plaintextpassword field and set it to true to enable encryption.
# introscope.enterprisemanager.trustpassword.channel2.plaintextpassword=false
# The password for the truststore
#introscope.enterprisemanager.trustpassword.channel2=password
# Set to true to require clients to authenticate.
# If true, clients must be configured with a keystore containing a certificate trusted by the EM.
# Default is false
introscope.enterprisemanager.needclientauth.channel2=false
# Set the enabled cipher suites.
# A comma-separated list of cipher suites.
# If not specified, use the default enabled cipher suites.
#introscope.enterprisemanager.ciphersuites.channel2=
# This property is used for Workstations launched via Java Web Start, to set
# the communication port used for communicating with the Enterprise Manager.
introscope.enterprisemanager.workstation.connection.channel=channel1
# When this property is set to true, the Time Range and Resolution widgets are
# enabled in the workstation dashboard in the "live" mode.
# This allows the user to enter custom time range and resolution for the live mode
# instead of the default time range of 8 minutes and resolution of 15 seconds.
# Please note that a time range greater than 8 minutes could potentially impact
# performance in the EM (due to I/O operations to fetch data from smartstor)
introscope.enterprisemanager.workstation.extendedLiveQuery=true
#################################
# Workstation Login Parameters Auto Fill
#
# ================
# Workstation remembers login parameters (Enterprise Manager host, port, and UserId) for webstart.
# Default is TRUE. If set to FALSE, Workstation will not remember login parameters.
introscope.enterprisemanager.workstation.rememberLogin=true
# Set the following property to true to allow a communication channel
# server socket to rebind to a local port that is stuck in a FIN_WAIT or
# TIME_WAIT state (SO_REUSEADRR). Once the EM starts up again, this property
# should be disabled. In general it is not a good idea to enable this property
# by default since there may be a valid reason that the port is already in use.
#introscope.enterprisemanager.serversockets.reuseaddr=
#######################
# EM Web Server Settings
#
# ================
# Controls configuration of the embedded web server;
# The port that the web server listens on
introscope.enterprisemanager.webserver.port=8081
# The directory for deployment of web application files
introscope.enterprisemanager.webserver.dir=webapps
# How often to hot-deploy web applications (in seconds)
introscope.enterprisemanager.webserver.refresh=60
# The property to keep the EM from starting if the web server port is already in use
introscope.enterprisemanager.webserver.portinuse.fail=true
# The maximum number of listener threads in the web server. Defaults to 100.
#
# DO NOT CONFIGURE THIS PROPERTY unless more than 80 HTTP tunneling agents
# are connecting to this EM. In this case we recommend setting the property
# value to 20 more than the maximum number of connecting HTTP tunneling agents.
#
#introscope.enterprisemanager.webserver.max.threads=100
# Path to a Jetty 6.1.26 XML configuration file. Either an absolute path, or a path relative to the config directory.
# For advanced Jetty configuration, uncomment this property and modify the settings in the Jetty configuration file.
# Default is em-jetty-config.xml. This file creates an SSL listener on port 8444 for https traffic.
# If uncommented, the http listener specified by the introscope.enterprisemanager.webserver.port property is not created,
# but if desired it can be configured in the Jetty XML file.
#introscope.enterprisemanager.webserver.jetty.configurationFile=em-jetty-config.xml
#################################
# Logging Configuration
# ================
#
# Logging properties support hot config.
#
# The following property, log4j.logger.Manager,
# controls both the amount of detail
# that is logged and the output location.
# Replace the text 'INFO' with the much longer text shown below -
# 'VERBOSE#com.wily.util.feedback.Log4JSeverityLevel'
# to increase the level of detail.
# Replace 'console' with 'logfile' (without the quotes)
# to send the output to a log file instead of the console.
log4j.logger.Manager=INFO, console, logfile
log4j.logger.Manager.Support=INFO, supportlogfile
log4j.logger.com.wily=INFO, console, logfile
log4j.logger.com.timestock=INFO,console,logfile
log4j.logger.com.ca.wily=INFO,console,logfile
log4j.logger.Apm.Data.Model=INFO,console,logfile
log4j.logger.apm.events=INFO,console,logfile
log4j.additivity.Manager.Support=false
log4j.additivity.com.wily=false
log4j.additivity.com.timestock=false
log4j.additivity.com.ca.wily=false
log4j.additivity.Apm.Data.Model=false
log4j.additivity.apm.events=false
log4j.logger.org.mortbay=ERROR
log4j.logger.org.quartz=ERROR, console, logfile
log4j.logger.org.apache.myfaces=ERROR,console
log4j.logger.org.apache.jasper.compiler.TagLibraryInfoImpl=ERROR
#log4j.logger.org.springframework.transaction=DEBUG,console
#Turn off the info logging for hibernate and spring
log4j.logger.org.hibernate=ERROR,console,logfile
#This is to avoid logging c3p0 data source closing as errors during EM shutdown.
log4j.logger.org.hibernate.util.JDBCExceptionReporter=FATAL
log4j.logger.org.hibernate.event.def.AbstractFlushingEventListener=FATAL
# suppress hibernate logging of LIEs (let CEM log these)
log4j.logger.org.hibernate.LazyInitializationException=FATAL
# sql logging
log4j.logger.org.hibernate.SQL=ERROR,console,logfile
# c3p0 logging
log4j.logger.com.mchange.v2.c3p0=ERROR,console,logfile
log4j.logger.com.mchange.v2.resourcepool=ERROR,console,logfile
# Other third party libraries Logging
log4j.logger.net.sf.navigator.taglib.DisplayMenuTag=ERROR,console,logfile
log4j.logger.net.sf.ehcache=ERROR,console,logfile
log4j.logger.org.springframework=ERROR,console,logfile
log4j.logger.org.acegisecurity=ERROR,console,logfile
log4j.logger.org.displaytag=ERROR,console,logfile
log4j.logger.org.jasypt=ERROR,console,logfile
# If logfile is specified above, the location of the log file
# is configured with the following property,
# log4j.appender.logfile.File. Full paths can
# be used if desired.
log4j.appender.logfile.File=logs/IntroscopeEnterpriseManager.log
log4j.appender.supportlogfile.File=logs/IntroscopeEnterpriseManagerSupport.log
########## See Warning below ##########
# Warning: The following properties should not be modified for normal use.
log4j.additivity.Manager=false
log4j.appender.console=com.wily.org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss.SSS a z} [%-3p] [%t] [%c] %m%n
log4j.appender.logfile=com.wily.org.apache.log4j.RollingFileAppender
log4j.appender.logfile.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.logfile.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss.SSS a z} [%-3p] [%t] [%c] %m%n
log4j.appender.logfile.MaxBackupIndex=4
log4j.appender.logfile.MaxFileSize=200MB
log4j.appender.supportlogfile=com.wily.org.apache.log4j.RollingFileAppender
log4j.appender.supportlogfile.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.supportlogfile.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z} [%-3p] [%c] %m%n
log4j.appender.supportlogfile.MaxBackupIndex=4
log4j.appender.supportlogfile.MaxFileSize=20MB
introscope.enterprisemanager.performance.compressed=true
log4j.additivity.Manager.Performance=false
log4j.logger.Manager.Performance=DEBUG, performance
log4j.appender.performance=com.wily.introscope.server.enterprise.monitor.PerformanceLogFileAppender
log4j.appender.performance.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.performance.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z},%m%n
log4j.appender.performance.File=logs/perflog.txt
log4j.appender.performance.MaxSize=4000000
### Logging for Tess performance log ###
log4j.additivity.com.wily.apm.tess.supportability.TessPerformanceCollector=false
log4j.logger.com.wily.apm.tess.supportability.TessPerformanceCollector=DEBUG, tessperformance
log4j.appender.tessperformance=com.wily.apm.tess.supportability.TessPerformanceLogFileAppender
log4j.appender.tessperformance.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.tessperformance.layout.ConversionPattern=%d{M/dd/yy hh:mm:ss a z},%m%n
log4j.appender.tessperformance.File=logs/tessperflog.txt
log4j.appender.tessperformance.MaxSize=4000000
# Configure HTTP Tunneling Transport logging
log4j.logger.org.apache.axis=OFF
log4j.additivity.Manager.QueryLog=false
log4j.logger.Manager.QueryLog=INFO, querylog
log4j.appender.querylog=com.wily.org.apache.log4j.RollingFileAppender
log4j.appender.querylog.layout=com.wily.org.apache.log4j.PatternLayout
log4j.appender.querylog.layout.ConversionPattern=%m%n
log4j.appender.querylog.File=logs/querylog.txt
log4j.appender.querylog.MaxBackupIndex=1
log4j.appender.querylog.MaxFileSize=4MB
# Configuring CLWCommandLog
# =========================
# The property log4j.logger.Manager.CLW is set to DEBUG and the property log4j.additivity.Manager.CLW
# is set to true to log the CLW commands in the log file and on the EM Console
log4j.additivity.Manager.CLW=false
log4j.logger.Manager.CLW=INFO
#####################################################################
# EM Metric Clamps are moved to new XML configuration file apm-events-thresholds-config.xml file
# under the same directory as this file.
#
# Important Note: Please note that the clamps' threshold configured in this file will not be affective.
# Modify apm-events-thresholds-config.xml configuration file for configuring the new thresholds.
############################################################################################
#######################
# SmartStor Settings
# ================
# The directory that the EM will use for data storage
introscope.enterprisemanager.smartstor.directory=E:/data
introscope.enterprisemanager.smartstor.directory.archive=E:/data/archive
introscope.enterprisemanager.smartstor.dedicatedcontroller=true
# For the next three sets of properties, the storage
# frequency must be a multiple of 15 seconds and the
# data age is specified in days. The age must not be
# zero. The frequency must not be greater than
# 1800 seconds (=30 minutes). Each tier's frequency must
# be an even multiple of the previous tier's frequency (this
# multiple can be one; each tier can have the same
# frequency). All three tiers must be defined, and you
# cannot define more than three tiers.
introscope.enterprisemanager.smartstor.tier1.frequency=15
introscope.enterprisemanager.smartstor.tier1.age=7
introscope.enterprisemanager.smartstor.tier2.frequency=60
introscope.enterprisemanager.smartstor.tier2.age=23
introscope.enterprisemanager.smartstor.tier3.frequency=900
introscope.enterprisemanager.smartstor.tier3.age=335
# Time offset for data tiering (hour of day between 0-23)
# For example, if this offset is set to 2, the data tiering will start at 2am
# The default value is 0 (12am or midnight).
#introscope.enterprisemanager.smartstor.reperiodizationOffsetHour=0
# Time offset for spool converting to data (minute of hour between 0-59)
# For example, if this offset is set to 10, the conversion will start at xx:10
# The default value is 0 (xx:00).
#introscope.enterprisemanager.smartstor.conversionOffsetMinute=0
#######################
# Transaction Event Database Settings
# ================
# The Enterprise Manager stores captured Transaction Events
# locally on its file system for a fixed length of time or
# until a maximum disk usage is exceeded.
# The following three properties allow you to control where
# to store these events and when they should be deleted.
# The directory that the EM will use for trace storage
introscope.enterprisemanager.transactionevents.storage.dir=traces
# How many days to store traces
introscope.enterprisemanager.transactionevents.storage.max.data.age=14
# Specifies how often (frequency in hours) trace storage should be optimized by cleaning
# up aged data.
# The default value is 24 hours.
introscope.enterprisemanager.transactionevents.storage.optimize.frequency=24
# Time offset for storage optimization (hour of day between 00-23)
# For example, if time offset is set to 02 and optimize frequency is set to 24 hours,
# optimization will be run every 24 hours offset from time 02:00.
# Or for example, if time offset is set to 03 and optimize frequency is set to 6 hours,
# optimization will be run every 6 hours offset from time 03:00.
# The default value is 02 (2 am).
introscope.enterprisemanager.transactionevents.storage.optimize.timeoffsethour=02
#######################
# EM Object Database Settings
# ================
# The Enterprise Manager stores baselines and other system data in an
# embedded Object Database.
# The following property allows you to control its location.
introscope.enterprisemanager.dbfile=data/baselines.db
#######################
# EM Config Directory
# ================
# The directory the EM will use for configuration
# Either an absolute path or a path relative to the Introscope installation directory
# On Windows, backslashes must be escaped. For example:
# introscope.enterprisemanager.directory.config=C:\\Introscope\\em1\\config
# Default is config directory in Introscope installation
#introscope.enterprisemanager.directory.config=config
#######################
# Auto-Unmount of Disconnected Agents
# (Known as "Auto-purging" in previous Introscope releases)
# ================
# Once an agent has disappeared, the Enterprise Manager continues to track it
# (in case the interruption is temporary). This delay allows you to control the
# amount of time that will pass after the disconnect before the Enterprise Manager
# releases the resources associated with tracking the agent. Units are minutes.
introscope.enterprisemanager.autoUnmountDelayInMinutes=60
#######################
# SQL Database Settings
#
# ================
# Introscope records traced data to a relational database as
# specified here. Comment out these properties
# if you want to turn off database recording
#
# To change introscope.enterprisemanager.db.password,
# specify the new password and set introscope.enterprisemanager.db.password.plaintextpassword to true.
# Note: If introscope.enterprisemanager.db.password.plaintextpassword property is set to true
# and the password is not changed, the existing encrypted password will be encrypted again.
# Once introscope.enterprisemanager.db.password is encrypted,
# introscope.enterprisemanager.db.password.plaintextpassword is automatically set to false.
#introscope.enterprisemanager.db.driver=oracle.jdbc.driver.OracleDriver
#introscope.enterprisemanager.db.url=jdbc:oracle:thin:@myDatabase:myPortID:SID
#introscope.enterprisemanager.db.username=myUser
#introscope.enterprisemanager.db.password.plaintextpassword=false
#introscope.enterprisemanager.db.password=myPassword
# Sets the interval in seconds between automatic attempts to reconnect to the
# database. Comment out this property to disable database reconnection.
introscope.enterprisemanager.db.reconnect.intervalInSeconds=30
# Sets the maximum number of metric data points in the database queue.
# Default is 150K.
# This property supports hot config.
introscope.enterprisemanager.database.recordingQueue.upperLimit.value=153600
# This enables batch insertion of metric values into the database.
# Batch inserts are a JDBC 2.0 feature; if your JDBC driver supports
# them, enabling this switch will increase the performance of value
# inserts significantly.
#introscope.enterprisemanager.db.useBatchInserts=true
# This controls the size (in row insertion count) of batch inserts, if they are enabled.
#introscope.enterprisemanager.db.batchInsertSize=100
# Sets the amount of time to wait for a query to complete before
# giving up. The value is the number of seconds to wait. A value
# of zero (the default value) means no timeout for queries.
# Not all databases and JDBC drivers support query timeouts.
# This property will only be used when both the JDBC driver
# and the database support query timeouts.
#introscope.enterprisemanager.db.queryTimeoutInSeconds=0
# This specifies the table qualifier to explicitly fully qualify table names.
# This qualifier will be prepended (or appended as appropriate) directly to
# the table name so be sure to include any separator required by the database.
#introscope.enterprisemanager.db.tablequalifier=myExplicitTableQualifier
# Sets the size of the internal caches which are used to improve performance.
# A value of -1 will allow the caches to grow as much as necessary and is
# recommended unless the limits of system memory are being reached. The
# value is the number of cache entries per database table so it will scale
# linearly with memory usage.
#introscope.enterprisemanager.db.maximumTableCacheSize=-1
# Sets the number of database connections to use for running database queries.
# A larger number of connections will consume more system resources but allow
# more simultaneous database queries to be processed. Each simultaneous
# connection needs the use of a temporary table in the database. The provided
# database schema only contains five temporary tables and therefore only
# supports five simultaneous query connections. If additional connections
# are desired, additional temporary database tables must be added.
#introscope.enterprisemanager.db.queryConnections=1
#######################
# Database Persistent Collection Settings
#
# ================
# Persistent collections define metric data written to
# an external relational database.
#
# Persistent collections require three properties:
# agentExpression: a Perl5 regular expression to match agents
# metricExpression: a Perl5 regular expression to match metrics
# frequencyinseconds: how often metric data is aggregated and written to the database
#
# Backslashes must be escaped.
#
# Persistent collections are identified by a unique id appended to each of the
# introscope.enterprisemanager.database.collection properties
#
# Persistent collection properties support hot config.
#
# Examples:
#introscope.enterprisemanager.database.collection1.agentExpression=rhart-dt1\\|EPAgentProcess1\\|EPAgent5
#introscope.enterprisemanager.database.collection1.metricExpression=Servlets\\|MyServlet:Average Response Time \\(ms\\)
#introscope.enterprisemanager.database.collection1.frequencyinseconds=3600
#introscope.enterprisemanager.database.collection2.agentExpression=(.*)
#introscope.enterprisemanager.database.collection2.metricExpression=EJB\\|Session\\|MyBean\\|service:Stalled Method Count
#introscope.enterprisemanager.database.collection2.frequencyinseconds=60
#######################
# Flat File Recording
#
# ================
# Introscope can record traced data to comma separated value (.csv)
# flat files in a specified directory. Uncomment this line
# to turn on flat file recording, and change the value to specify
# the directory that hosts the flat files
introscope.enterprisemanager.flatfile.homedir=flatfiles
# Sets the maximum size of a flat file in megabytes. When exceeded,
# Introscope will archive the file to a zip file, compressing
# by approximately 95%. The default maximum size is 100 megabytes.
introscope.enterprisemanager.flatfile.maxFileLength=100
# Sets the amount of time that will be stored in a single flat file
# before archiving. Can be used along with a maximum size. Archiving
# based on time is disabled if the property is commented out.
#introscope.enterprisemanager.flatfile.maxTimeInHours=24
# Can be set to true to disable compression when archiving flat files.
# If disabled, flat files will simply be moved into the archive directory
# with the extension .full.csv.
introscope.enterprisemanager.flatfile.disableCompression=false
#######################
# Flat File Persistent Collection Settings
#
# ================
# Introscope records metric data to flat files as specified here.
# Each persistent collection entry consists of an
# agent expression, a metric expression and a frequency in seconds. The
# metric data for the matching metrics will be written to a flat file
# at the specified frequency. Each entry starts with the key prefix
# introscope.enterprisemanager.flatfile.collection and should be numbered
# sequentially starting with 1.
#introscope.enterprisemanager.flatfile.collection1.agentExpression=.*
#introscope.enterprisemanager.flatfile.collection1.metricExpression=.*
#introscope.enterprisemanager.flatfile.collection1.frequencyinseconds=60
#introscope.enterprisemanager.flatfile.collection2.agentExpression=.*
#introscope.enterprisemanager.flatfile.collection2.metricExpression=.*
#introscope.enterprisemanager.flatfile.collection2.frequencyinseconds=3600
#######################
# Interactive Mode
#
# ================
# Set to false to turn on interactive mode
introscope.enterprisemanager.disableInteractiveMode=true
#######################
# SNMP Adapter
#
# ================
# Set to true to enable the SNMP Adapter
# Default is false
introscope.enterprisemanager.snmp.enable=false
# The port used by the SNMP Agent. Defaults to 161.
introscope.enterprisemanager.snmp.agent.port=161
# This property supports hot config
# Set to true to enable Notifications (Traps).
# Default is true.
introscope.enterprisemanager.snmp.notification.enable=true
# The target port for Notifications.
# Default is 162.
introscope.agent.enterprisemanager.snmp.target.port=162
# The target host (hostname or IP address) for Notifications.
# Default is localhost.
introscope.agent.enterprisemanager.snmp.target.host=localhost
# The target SNMPv2 community for Notifications.
# Default is public.
introscope.agent.enterprisemanager.snmp.target.community=public
#######################
# WebView Configuration
#
# ================
# Set to true to prevent WebView from connecting to the
# Enterprise Manager. Defaults to false.
introscope.enterprisemanager.webview.disableLogin=false
#######################
# Management Module Bootstrapping
#
# ================
# Set to true to stop the Enterprise Manager from starting
# with management modules containing invalid references.
# If false, invalid references are logged as warnings. Defaults to false.
introscope.enterprisemanager.managementModules.invalid.references.as.errors=false
#######################
# Management Module Deployment
#
# ================
# Magic directory to watch
introscope.enterprisemanager.hotdeploy.directory=deploy
# How often to check for new MM files to deploy
introscope.enterprisemanager.hotdeploy.checkFrequencyInSeconds=60
#######################
# Alert State Metrics
#
# ================
# Metric name prefix of the alert-states metrics
introscope.enterprisemanager.alertstatemetric.prefix=Alerts
#######################
# Hot Config
#
# ================
# The hot config properties support hot config
#
# Set to false to disable hot config.
# Default is true.
introscope.enterprisemanager.hotconfig.enable=true
# How often to check this file for changes (in seconds).
# Default is 60 seconds.
introscope.enterprisemanager.hotconfig.pollingInterval=60
#######################
# Application Heuristics
#
# Setting this to false improves the performance of the EM but disables the EM's assessment of JVM, application, and backend health
#
# ================
introscope.enterprisemanager.application.overview.baselines=true
######################
# Agent Config
# (pushed to Agents)
#
# Sampling Rate
# NOTE: The agent defaults to these values. Extreme care should be taken
# when overriding these defaults because of the potential for higher
# load in the EM when a high sampling rate is used
#
#introscope.agent.transactiontracer.sampling.perinterval.count=1
#introscope.agent.transactiontracer.sampling.interval.seconds=120
#######################
# Hot Failover
#
# ================
#Settings for Hot Failover.
#
# EMs sharing the same installation directory can be configured for Hot Failover.
# Set to true to enable hot failover.
# Default is false.
introscope.enterprisemanager.failover.enable=false
# Set to hostname or IP address of the primary EM.
# To configure multiple EMs as primary,
# set to a comma-separated list of hostnames or IP addresses.
# To configure for failover to work in a single host set this property as localhost(refer documentation for more information)
introscope.enterprisemanager.failover.primary=
# Set to hostname or IP address of the secondary EM.
# To configure for failover to work in a single host leave this property blank (refer documentation for more information)
introscope.enterprisemanager.failover.secondary=
# How often a running Secondary EM checks to see if a Primary EM is waiting to start (seconds).
# This property supports hot config.
# Default is 120 seconds (2 minutes).
introscope.enterprisemanager.failover.interval=120
###################################
# Enterprise Manager Clustering Mode
#
# ================
# Clustering mode in which the Enterprise Manager would run
#
# Valid values (case insensitive) are:
#
# StandAlone - Enterprise Manager runs in the stand alone mode
# CDV - Enterprise Manager runs as a Cross-cluster Data Viewer
# Collector - Enterprise Manager runs as a Collector
# MOM - Enterprise Manager runs as a Manager
#
####################################
introscope.enterprisemanager.clustering.mode=MOM
#######################
# Clustering Settings
#
# ================
# Controls clustering of Enterprise Managers.
#
# Settings for Cluster Collector role.
#
# An arbitrary identifier appended to the name of Custom Metric Agents.
# For example, if the identifier is em1, then the agent name is:
# Custom Metric Agent (Virtual) (em1).
# The purpose of the identifier is to uniquely name Custom Metric Agents across the cluster.
# Use a different identifier for each Collector.
# If no identifier is specified, the default identifier of host@port is used,
# where host is the EM's host name, and port is the EM's connection port.
#introscope.enterprisemanager.clustering.collector.identifier=
#
# Settings for Cluster Manager (MOM) role.
#
# Threshold for when a collector is considered slow (in milliseconds)
introscope.enterprisemanager.clustering.manager.slowcollectorthreshold=10000
# Seconds a MOM waits for a Collector to respond before it disconnects it as hung.
# Default is 60 seconds. Setting to zero (or negative) disables the check (not recommended).
# The MOM checks collectors every 7.5 seconds, so small changes don't affect behavior.
introscope.enterprisemanager.clustering.manager.slowcollectordisconnectthresholdseconds=60
# Maximum simultaneous output connections. A higher number does not directly increase
# response time, but will keep isolated slow connections from crippling the whole manager.
# The default, 5, is suitable for most standalone EMs and Collectors. MOMs should have
# one thread per Collector, plus about 1 for every 10 workstations. Do not oversize, as
# additional thread overhead may cause OutOfMemory exceptions - especially on 32 bit JVMs.
# transport.override.isengard.high.concurrency.pool.max.size=10 ## Uncomment for MOMs
# This is the private key used to authenticate a MOM to a collector.
# Either an absolute path or a path relative to the config directory.
introscope.enterprisemanager.collector.privatekey=internal/server/EM.private
# Define MOM Load Balancing properties.
#
# Load Balancing properties support hot config.
#
# The tolerance of the Load Balancer for imbalance.
# The number of metrics in a Collector must differ from the weight-adjusted
# cluster average (i.e. the average for all Collectors) by at least the
# threshold before the Load Balancer rebalances the Cluster.
#
# Rebalancing means redistributing Agents among Collectors
# to equalize the weight-adjusted metric load.
# See the connection properties about setting weights.
#
# Only version 8.0 or later agents that connected to the MOM
# are eligible for rebalancing.
#
# Default is 20000 metrics.
introscope.enterprisemanager.loadbalancing.threshold=20000
# How often the Load Balancer rebalances (seconds).
# Minimum is 120 seconds (2 minutes).
# Default is 600 seconds (10 minutes).
introscope.enterprisemanager.loadbalancing.interval=600
# Define Agent Connection Control properties
#
# Agent Connection Control properties hot config
# All agent control properties values of MOM overrides the respective agent control properties values at Collector.
# At Collector these property values are not considered, if any MOM has connected to the Collector ever.
# For agent(s) that are not specifically matching any regular expression in loadbalancing.xml,
# this decides whether to allow or disallow those agent(s).
introscope.apm.agentcontrol.agent.allowed=true
# Agent utilizes the list if both the MOM and the initial Collector appear to be non-responsive
# Agent utilizes the EM list given by cluster for self lookup in case the Collector the agent connected to goes down
# and the MOM is also not available.
# Agent is also provided with the EM list by StandAlone.
introscope.apm.agentcontrol.agent.emlistlookup.enable=true
# Maximum number of disallowed agents.
# This property value is not overriden by MOM.
# default is 0 (unlimited)
introscope.enterprisemanager.agent.disallowed.connection.limit=0
# This property is to instruct agent interval duration to wait before reattempting the connection to EM.
# EM will randomly add one of these values [-30, -15, 0, +15, +30] to wait. This is done to distribute the simultaneous
# multiple agents connection across time. Agent will wait for this time before reattempting the connection to EM.
# Default is 45 seconds. If the calculated wait value is less than 15 then default value 45 is used
introscope.apm.agentcontrol.agent.reconnect.wait=45
# Define connection properties for clustered EMs.
# Connection properties support hot config.
# Each clustered EM requires three properties:
# host: hostname or ip address of the collector
# port: connecting port of the collector
# publickey: the public key to authenticate the MOM to the collector -
# either an absolute path or a path relative to the config directory
#
# There is an optional weight property to control the relative load of the collector:
# weight: the load balancing weight for this collector EM
#
# All other things being equal, the percentage of metric load assigned to the collector
# is the weight divided by the total weight for the cluster.
# For example, if the weight of a collector is 50 and the sum of the weights of all collectors is
# 250, the collector should get 20% of the cluster's metric load.
#
# The weight should be a positive integer.
# Default is the average weight for the cluster.
#
# Sample properties for a clustered EM.
# Note: em1 is an arbitrary identifier. Use a different identifier for each set of connection properties.
# Avoid use of the value "localhost" as the host.
#
#introscope.enterprisemanager.clustering.login.em1.host=hostname
#introscope.enterprisemanager.clustering.login.em1.port=5001
#introscope.enterprisemanager.clustering.login.em1.publickey=internal/server/EM.public
introscope.enterprisemanager.clustering.login.em1.host=10.0.6.122
introscope.enterprisemanager.clustering.login.em1.port=5001
introscope.enterprisemanager.clustering.login.em1.publickey=config/internal/server/EM.public
introscope.enterprisemanager.clustering.login.em2.host=10.0.6.123
introscope.enterprisemanager.clustering.login.em2.port=5001
introscope.enterprisemanager.clustering.login.em2.publickey=config/internal/server/EM.public
#introscope.enterprisemanager.clustering.login.em1.weight=
######################
# Map Agent Config
#
# ================
# In Introscope 9.x, agents can detect when monitored applications communicate
# with external components, and report this activity to the EM. The activity data
# is then used to draw application maps in the Workstation.
#
# To support this task, agents load an important configuration property from the EM:
# the agingTime property. This property controls how often the agent will report
# dependency data to the EM.
#
# The property works like this:
# 1) When the agent observes a new application dependency, it reports and then caches it.
# 2) Cached activity is not reported.
# 3) After the agingTime interval has elapsed, the agent removes the dependency from its cache.
# 4) The next occurrence of the dependency will be treated as per step 1).
#
# In this way, the agingTime property affects both the "freshness" of the map displays,
# and the resulting performance overhead on the Agents and EM.
#
# If the property is set to:
# - ...a lower value, maps will show a more real-time picture of application activity,
# while incurring more Agent and EM overhead.
# - ...a higher value, maps will show a less up-to-date picture of application activity,
# and make fewer demands on Agent and EM performance.
#
# For more information, see the Introscope Sizing Guide.
#
# The property is not hot configurable, and if changed the connected agents should be
# re-started to use the new aging value set in this file.
# The agingTime property defaults to 86400000 ms, or 1 day.
introscope.apm.data.agingTime=1 DAY
# Data Time Window is a period that would used to get dependency map.
# For Example, if Data time window is 3 days,
# The workstation will pull the dependency map for 3 days
introscope.apm.data.timeWindow=3 DAYS
###################################
# Map Data Pruning Config
#
# ================
# Application dependency data is pruned from the database
# according to these properties.
#
# WARNING: If the preservation age is set too low (to SECONDS),
# data might be deleted as it arrives.
#
# Default values for these properties are:
# introscope.apm.pruning.enabled=true
# introscope.apm.data.preserving.time=365 DAYS
# introscope.apm.pruning.cron.trigger.expression=0 0 6 * * ?
#
###################################
# Set to true to enable pruning
# Set to false to disable pruning and keep all data indefinitely
# This property should always be disabled in collector mode but needs to be
# enabled in standalone mode. ApmPropertyService class
# sets this property to false when collector is initializing.
introscope.apm.pruning.enabled=true
# This property determine the maximum preservation age for
# application dependency data. Data older than the preservation age will be pruned.
# Number of time units to use when calculating the preservation age
introscope.apm.data.preserving.time=365 DAYS
# This property controls how often the database is pruned of old data.
# It is formatted as an opensymphony quartz scheduler expression.
# This example prunes the database every minute, every hour:
#introscope.apm.pruning.cron.trigger.expression=0 0/1 0-23 * * ?
# By default, the prune task runs every day at 6 AM:
introscope.apm.pruning.cron.trigger.expression=0 0 6 * * ?
###################################
# Home Dashboard Config
#
# ================
# Default home dashboard loaded by the workstation is set by
# these properties. If this dashboard cannot be loaded the first
# available dashboard is loaded.
#
# Each individual user may override and set their own individual
# home dashboard in their user preferences.
#
# Default values for these properties are:
# introscope.workstation.dashboard.home.domain=
# introscope.workstation.dashboard.home.module=Sample
# introscope.workstation.dashboard.home.dashboard=An Intro to Introscope
#
# NOTE: If the dashboard has been translated to multiple languages, specify
# the dashboard name using the default language.
###################################
# The path of the domain containing the default home dashboard
# No domain path means dashboard is in SuperDomain
introscope.workstation.dashboard.home.domain=
# The name of the Management Module containing the default home dashboard
introscope.workstation.dashboard.home.module=Sample
# The name of the default home dashboard
introscope.workstation.dashboard.home.dashboard=An Intro to CA Introscope
####################################
# CEM incident web services
#
# Maximum number of incidents that may be returned by the CEM incident operations
# defined in the APM Model Web Services. Default value 500.
introscope.enterprisemanager.ws.max.incidents=500
######################################################
#APM SaaS dummy password - to be used in APMSaaS AppLogic Grid template by new WebView
com.ca.apm.webview.saas.dummyuserpassword=@@password@@
######################################################
# Properties required for enabling smartcard
# authentication in APM. Smartcard authentication
# is done in APM with the help of SCARVES server which is
# Smart Card Authentication and REvocation Service
######################################################
# whether to enable or disable smart card authentication.
#introscope.enterprisemanager.webserver.scauth.enable=true
# hostname and port of SCARVES server which validates
# and authenticates client cert from smart card
#introscope.enterprisemanager.webserver.scauth.hostname=localhost
#introscope.enterprisemanager.webserver.scauth.port=9998
# keystore details which contains the SCARVES server certificate
# to enable communication between EM and SCARVES server
#introscope.enterprisemanager.webserver.scauth.keystore=config/internal/server/daemoncert
#introscope.enterprisemanager.webserver.scauth.keypass=password
#########################################
#
# Thread dump feature properties
#
###########################################
# The property is used to enable or disable the thread dump collection on
# the Enterprise Manager i.e this disables the collection of thread dump on any agent connected
# to this Enterprise Manager.
introscope.enterprisemanager.threaddump.enable=true
# Specifies the location of the folder where the Thread dumps
# will be stored on the Enterprise Manager
introscope.enterprisemanager.threaddump.storage.dir=threaddumps
# The Thread dump files purger will clean files older than the specified days.
# specify in days
introscope.enterprisemanager.threaddump.storage.clean.disk.olderthan.days=30
# The frequency of purging the Thread dump files.
# specify in days
introscope.enterprisemanager.threaddump.storage.clean.disk.freq.days=1
# The maximum thread dump files that can be stored on the Enterprise Manager
introscope.enterprisemanager.threaddump.storage.max.disk.usage=5000
#######################################################################
# SNMP Configuration for Triage Map Alerts for SSA/Catalyst
#
# =====================================================================
# This needs to be set to the ip address of the SSA/Catalyst connector
# where the the triage map SNMP traps will be sent.
introscope.apm.catalyst.triagemapalert.snmp.destination.host.ip=
# This needs to be set to the SNMP port where the the triage map SNMP traps
# will be sent. Default value is 162.
introscope.apm.catalyst.triagemapalert.snmp.destination.trap.port=162
# This needs to be set to the SNMP community string. Default value is public.
introscope.apm.catalyst.triagemapalert.snmp.community=public
# This needs to be set to the trigger type desired for SNMP trap notifications. Default
# value is numeric equivalent of trigger Whenever Severity Changes.
introscope.apm.catalyst.triagemapalert.snmp.trigger=3
# Compression index for Transaction Tracer Optimization.
# Default index needs to be set to 3. Lowering it will cause Out Of Memory problems.
introscope.enterprisemanager.lucene.maxNumSegments=3
###############################################################################
# Change Detector Settings
#
# ==========================
# By default Change Detector is turned off on the EM. To enable change detector
# on EM change this property value to false. Its not a hot property.
#introscope.changeDetector.disable=true
###############################################################################
# Gokhan Kocak 19.12.2014
transport.outgoingMessageQueueSize=6000
transport.override.isengard.high.concurrency.pool.min.size=10
transport.override.isengard.high.concurrency.pool.max.size=10
# Gokhan Kocak 19.12.2014
com.wily.introscope.soa.deviation.enable=false
Hello,
As most of the people already suggested, I would like to add few more points since we faced the same type of issues on our environment.
1) Please go through APM overview guide like Hal suggested. If you have a short time, please understand MOM, collector, smartstor, workstation, APM DB, agents and how these are interacting with each other.
2) Once you are well good about APM, please do end to end health check. Don't apply any config changes unless you understand the behavior of your environment. Below link will help you to do a comprehensive health check.
Cookbook - EM HealthCheck v20.pdf
3) Capture the observations/behavior of your environment once you have finished the health check. By this time,you will be able to determine the cause of slow performance
4) Now apply the config changes or do necessary steps based on the pattern you noticed.
We faced the similar issue and noticed there were lot of agents added which increased the capacity and collectors were not able to handle the load. We had to add new collector to the cluster to resolve performance issues.
Hope this helps.
Thanks,
Karthik
Hi Gökhan,
Nice to catch up with you. And I'm indeed sorry to hear that you're still having Performance issues.
Aside the advise you have been given (I'm sorry I have been on vacation so I have not had the opportunity to step in lately) on this thread as well in the support case I would like to take a step back to our previous collaboration or resolving your problem.
As you know heap memory was increased on your EMs and you reported back that that did in fact improve performance.
However, as we indeed agreed during one of our WebEx sessions you are having way too many database metrics (in Instroscope lingo known as "metric explosion"). These metrics originate from
1) data exchange using Excel spreadsheets as data sources (which all have different names - hence the explosion); and
2) from your very high number of database queries using distinct names
As we talked about the number of these metrics can be vastly reduced by "normalization" (i.e. aggregating related metrics thereby eliminating individual metrics in favor of aggregated metrics). You mentioned that
1) You would simply want to get rid of these metrics all together.
My offer of assisting you achieve this of course still stands: If you can please from your developers get information of how your application is connecting to the Excel spreadsheets we can eliminate these metrics.
2) You would like to aggregate these metrics as well utilizing normalization as well. However, you were not sure what aggregation would make sense. You would investigate this with your peers and come back.
My offer of assisting you achieve this of course still stands: If you can please with your peers decide the normalization that would make sense we can resolve this too.
Getting these metrics numbers vastly reduced should have a noticeable positive impact on your performance of retrieving data which is your problem.
This is indeed the preferred approach before eventually adding EMs to your cluster.
I'm looking forward to work with in resolving your performance issue.
Please feel free to reach out to me directly via e-mail.
Regards
Henrik
Hi Henrik,
Excel files are used when loading/saving some data and they are deleted.
Since we are using stored procedures, how will you apply normalization?
Normalization docs talk about ad-hoc queries, but there are only SPs in our applications.
Your proposals did not sound feasible to me.
Regards,
// Gokhan
Hi Gokhan,
please continue to work with CA Support on the ticket and with ravhe01. In parallel you should follow nkarthik's advice from above: check the EM supportability metrics. From your description above I would primarily check Historical Metric Count (should not be visibly increasing in 30 day view) and Harvest duration (should be well below 3.5s, ideally below 1s during normal operation, e.g. 6hr taken at late afternoon).
What exactly do you mean by "CA APM response time is more than 3 minutes"? What are you trying to do? Open the workstation, open webview, display a certain dashboard?
Ciao,
Guenter
Hello,
I can see the case, to investigate further support would need the below information, please attach it to the case:
- Copy of the logs directory from all the EMs (Mom and collectors), they will need VERBOSE logs.
- Screenshot of the "Custom Metric Host (Virtual) | Custom Metric Process (Virtual) | Custom Metric Agent (Virtual) | Enterprise Manager | Data Store | Smartstor | Metadata | Metrics with Data” supportability metric from all Collectors.
- Screenshot of MOM > “Status console”
- Collect a series of threadump from MOM and collector to find out the root cause.
Confirm if you are using a physical or virtual environment.
I have checked quickly the logs, you can find below some observations:
From the MoM perflog, I Performance.MetricDataManager.QueryMemory is ~ 27,151,588
From the Mom log, I see:
- a huge amount of these error: "data seems to be OK - not sure what to do with append inputs",
if possible restart the MoM with a refresh new smartstor db
-[WARN] [PO:client_main Mailman 1][Manager.AsyncQueryResultStateMachine] Received tardy historical data from slow collector
This other error indicates a performance issue in the cluster (MOM and collector communication)
-[WARN] [Collector ***@5001] [Manager.Cluster] Collector clock is too far skewed from MOM. Collector clock is skewed from MOM clock by 56,085 ms. The maximum allowed skew is 3,000 ms. Please change the system clock on the collector EM.
Make sure clocks of all EMs are in synch, you must configure a NTP server.
- [Collector xxxxx@5001] [Manager.Cluster] The Introscope Enterprise Manager will continue to attempt to re-connect to the Introscope Enterprise Manager at xxxx@5001. Further failures will not be logged
- [WARN] [pool-1-thread-1] [Manager.Cluster] The Collector 10.0.6.122@5001 is responding slower than 10000ms and may be hun
The above indicates that the Mom keep disconnecting from collectors.
Check how the individual collectors are performing, check the logs, key words to search: outgoing, capacity, WARN, ERROR, reached, CancelledKeyException, java.io.IOException, outofmemory,
Also, I noticed you are using 914, if possible I suggest you to upgrade to a latest release as many issues affecting the EM, clustering, loadbalancing mechanism, etc has been fixed in latest releases. Below link to the master list
http://www.ca.com/us/support/ca-support-online/product-content/knowledgebase-articles/tec1075326.aspx?intcmp=searchresultclick&resultnum=4
I hope this helps,
Regards,
Sergio