DX Application Performance Management

In 10.5 we are facing regularly full memory utilisation.Like, we cleared the spaces(logs n traces) whenever we received.But, what is the permanenet solution of it.

$ free -g
         total used free shared buffers cached
Mem: 7       7       0       0          0          4
-/+ buffers/cache: 2 5
Swap: 1 0 1
$

Hello Amritesh,

From your comments about logs & traces you are probably referring to an APM Enterprise Manager memory usage?

The memory usage is limited by the max Java heap set in the Introscope_Enterprise_Manager.lax file via line 
lax.nl.java.option.additional JVM paremeter "-Xmx"

Is your free -g output actually showing a problem with used/free memory?

It does not appear to be because once you factor in the 2nd line data for buffers/cache the used and free numbers (2G & 5G) do not appear to show a problem i.e. the Linux OS is just using the available RAM for caching. Here is a useful reference:

linux - Meaning of the buffers/cache line in the output of free - Server Fault 

To actually understand what the numbers mean, you need a bit of background about the virtual memory (VM) subsystem in Linux. Just a short version: Linux (like most modern OS) will always try to use free RAM for caching stuff, so Mem: free will almost always be very low. Therefore the line -/+ buffers/cache: is shown, because it shows how much memory is free when ignoring caches; caches will be freed automatically if memory gets scarce, so they do not really matter.

A Linux system is really low on memory if the free value in -/+ buffers/cache: line gets low.

Hope that helps

Regards,

Lynn

Hi William,

XMX value is 1024m. Iz there any possibility to check where full spaces are occupied n which one is taking more space n can we cleared that specifically or not.

You cannot hope to run an EM with only 1GiB RAM unless it's just a test system. You will need at least 1.5GiB, depending on the metric load, to run a healthy system. You'll want to look at the sizing guide and calculators to determine how much heap you will need. You'll likely need between 4-10GiB RAM with 2-4 CPUs for a production system.

Hi Amritesh,

Per my earlier reply from the "free -g" output you don't appear to have a memory problem because only 2G is actually being used by processes and Linux is using the remaining free RAM for caching (which is why the first line shows 7G as used). Hope that makes sense.

Hiko_Davis has advised on the EM heap minimum size.

Thanks

Lynn

Hi William,

You can see the lax file details below :-

MOM :-

# LaunchAnywhere (tm) Executable Properties File - Flexera Software LLC

# LAX.APPLICATION.NAME
# --------------------
# the default name of this executable -- do not edit

lax.application.name=Introscope_Enterprise_Manager

# LAX.CLASS.PATH
# --------------
# the Java classpath necessary to run this application
# Can be separated by colons (Mac OS/Unix) or semicolons (Windows)

lax.class.path=launcher.jar:ext:lax.jar

# LAX.COMMAND.LINE.ARGS
# ---------------------
# what will be passed to the main method -- be sure to quote arguments with spaces in them

lax.command.line.args=$CMD_LINE_ARGUMENTS$ -consolelog -noExit -product com.wily.introscope.em.product -install "./product/enterprisemanager" -configuration "./product/enterprisemanager/configuration"

# LAX.DIR
# -------
# path to the directory holding LaunchAnywhere's native launcher

lax.dir=./

# LAX.MAIN.CLASS
# --------------
# the class that contains the main method for the application

lax.main.class=org.eclipse.core.launcher.Main

# LAX.MAIN.METHOD
# ---------------
# the method in the main class that will be invoked

lax.main.method=main

# LAX.NL.CURRENT.VM
# -----------------
# the VM to use for the next launch

lax.nl.current.vm=jre/bin/java

# LAX.NL.JAVA.LAUNCHER.MAIN.CLASS
# -------------------------------
# main class of LaunchAnywhere's java launcher -- do not adjust

lax.nl.java.launcher.main.class=com.zerog.lax.LAX

# LAX.NL.JAVA.LAUNCHER.MAIN.METHOD
# --------------------------------
# main method of LaunchAnywhere's java launcher -- do not adjust

lax.nl.java.launcher.main.method=main

# LAX.NL.JAVA.OPTION.ADDITIONAL
# -----------------------------
# place any optional Java arguments (such as heap size) here.
# The following switches are recommended for 32-bit Sun JVMs:
# -showversion -XX:CMSInitiatingOccupancyFraction=50 -Xss256k
# The following switches are recommended for 64-bit Sun JVMs:
# -Xss512k
# If you are using both Hot Failover and the CEM UI,
# add the following switch when starting the Secondary Introscope Enterprise Manager (or second Primary),
# -Dosgi.clean=false
# If you are installing on AIX (with JRE7 or earlier), add the following switch
# -XX:MaxPermSize=256m
# Set the lax java option additional values like: -Xms1024m -Xmx1024m -Djava.awt.headless=false -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

#lax.nl.java.option.additional=-Xms1024m -Xmx1024m -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xss512k

lax.nl.java.option.additional=-Xms8192m -Xmx8192m -Djava.awt.headless=true -Dmail.mime.charset=UTF-8 -Dorg.owasp.esapi.resources=./config/esapi -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Xss512k

# LAX.NL.MESSAGE.VM.NOT.LOADED
# ----------------------------
# message displayed in a user dialog if NO VM from the lax.nl.valid.vm.list can be found. Note: This property is internal to the InstallAnywhere launcher. Introscope has separate Java requirements. For more information, consult the "System Requirements" section of Introscope's user documentation.

lax.nl.message.vm.not.loaded=LaunchAnywhere either could not find a Java VM, or the Java VM on this system is too old. LaunchAnywhere requires a Java 2 VM in order to launch Introscope.

# LAX.NL.VALID.VM.LIST
# --------------------
# a string containing one or more of [ ALL JDK JRE J1 J2 JRE_J1 JDK_J1 JRE_J2 JDK_J2 MSJ ] delimited by spaces or commas.
# if the native launcher cannot find the current vm,
# it will search for ones in this list

lax.nl.valid.vm.list=J2

# LAX.NL.WIN32.MICROSOFTVM.MIN.VERSION
# ------------------------------------
# The minimum version of Microsoft's VM this launcher will run with

lax.nl.win32.microsoftvm.min.version=2750

# LAX.ROOT.INSTALL.DIR
# --------------------
# path to the installdir magic folder

lax.root.install.dir=.

# LAX.STDERR.REDIRECT
# -------------------
# leave blank for no output, "console" to send to a console window,
# and any path to a file to save to the file

lax.stderr.redirect=console

# LAX.STDIN.REDIRECT
# ------------------
# leave blank for no input, "console" to read from the console window,
# and any path to a file to read from that file. Note: If you are running
# the Enterprise Manager in nohup mode for Unix platforms, do NOT
# change this setting, or the EM may fail to start. Should you choose
# to change this setting to blank, you MUST disable Interactive Mode
# in the IntroscopeEnterpriseManager.properties file.

lax.stdin.redirect=console

# LAX.STDOUT.REDIRECT
# -------------------
# leave blank for no output, "console" to send to a console window,
# and any path to a file to save to the file

lax.stdout.redirect=console

# LAX.USER.DIR
# ------------
# left blank, this property will cause the native launcher to not
# alter the platform default behavior for setting the user dir.
# To override this you may set this property to a relative or absolute path.
# Relative paths are relative to the launcher.

lax.user.dir=./

# LAX.VERSION
# -----------
# version of LaunchAnywhere that created this properties file

lax.version=16.5

Hi Amritesh,

We fought with memory for years, when we started with APM.  Mainly since we "Right Size" our resources and have to show need before we can get allocation.  All of our APM servers are virtual (ESX).  So we had to burn through our resources, show that we used our allocation and need to have more added.

Currently our MOM host has 16 GB of RAM, EM has 10 GB, 3 GB for WebView and then 3 GB for system

Our collectors have either 8 to 10 GB, with our collector EM having 6 or 8 GB depending on where the CEM processes land.  Each host, we have two CPU cores.

This is our production resources and this handles our agent and user load fairly well. 

Is there a permanent answer to the APM memory and CPU issues?  sorry, nope.

The more agents, the more traffic, the more traces the more end users requesting 30 days worth of data across a dozen so server/applications, then your enterprise managers will shift what they need.  So, add environment performance agents and a bunch of system level PERL scripts/plugins to pull back the CPU, memory, and network.  If at all possible also get disk I/O, since the collectors write very small packets but thousands and thousands of them to the disk every 7.5 seconds (smartstor/spool/traces).  With that, watch out with the disk caching at the OS level, since a delay in getting the metrics down to the SmartStor, things back up very quickly and the enterprise managers will start to merge time slices.

So once you have OS level metrics and have some basic alerts setup, implement the MOM Infrastructure management modules.  For us, it was very helpful in tracking down which resource to go begging for more allocation.

If you give an EM too much memory, then your garbage collection becomes too long, and you get increased CPU.  If you don't give your EM enough memory, then the CPU will go really high as the JVM is hunting for free memory.  So when adjusting the resources, take a look at the garbage collection of the EM JVM to verify that you are not seeing long or a lot of garbage collection cycles.

If the memory looks good, but the CPU is staying above 90% for periods of time, then you may need to allocate additional CPU.  EMs will have different cycles to write memory to spool then spool to SmartStor, check your IntroscopeEnterpriseManager.properties file and set your various periods so you don't have all of your enterprise managers attempting to write a novel to disk at the same time.

The baselines in the documentation are good starting points, guidance, but the real nature of how much memory and CPU to allocate to your APM instance really depends on the volume of metrics,  the volume of transaction traces, the amount and size of the error traces and each of these really depend on the number of applications and which agent extensions you have active.

The best advice I could give is to separate your production and non-production.  Your non-production typically won't have the transaction trace volume but will have a much larger number of metrics.  This will free your production instance to run the much larger number of transaction traces across a smaller population of metrics/agents.

Hope this helps,

Billy