CA Service Management

  • 1.  Fine Tuning Tomcat for better performance

    Posted Mar 09, 2016 06:23 AM

    Dear all,

     

    I starting this tread as proposed by Kyle for us to share our recommendation and experience on troubleshooting/fine tuning Tomcat for performance and eventually crash issue as an essential component of the application.

    The idea is to not come too much in details and technical but keep it simple for everyone to benefit.

     

    There is also a idea open hereIIS web server available for all SDM functionality. to move functionality from Tomcat to IIS to reduce that dependency but may you prefer continue to use Tomcat and until then I listed some bullet point already below.

     

    The order of displaying the information below is intentional and you must start from the top at your first step and fall down to the end may this be needed.

     

    Heap Memory

     

    Out of the box configuration the heap memory to be consumed by tomcat in set relatively low and you may want to increase base on your need and size of system RAM.

    This settings can be modified in the nx.env and will require a restart of the tomcat container

     

    @NX_JAVA_OPTIONS=-Djava.net.preferIPv4Stack=false -Xms64M -Xmx1024M

     

    the -Xms switch determine the minimum memory size used at the started of the container

    the -Xmx switch determine the maximum size that can be used by the container.

     

    I often recommend to put the size of the -Xmx to 50% of you available RAM but if you run multiple application on that server you may want to go to a lower number.

    You can configure those parameters for each of your tomcat container (REST, SA, Visualizer) but the total of those must not exceed more that 60% of your total system memory available to avoid impacting others components of your system.

     

    If you have enough memory available, you can set the minimum and maximum heap to the same size to avoid garbage collection to happen too frequently. Garbage collection is an intensive consumer of CPU resource and much you can avoid it better it will be.

    Please consider that this is a costly memory process as those memory will be reserve to Tomcat and can't be used by others application. Then size your RAM accordingly or try to compromise the ratio between the Xms and XmX that better suite your system

     

    The example below will allocate and reserve 2gb of memory and prevent garbage collection until then

     

    @NX_JAVA_OPTIONS=-Djava.net.preferIPv4Stack=false -Xms2048M -Xmx2048M

     

     

    Note that this is just example and you need to fine tune to suit your needs.

    still need to write here some steps to determine the size according to consumption

     

    Threads

     

    Thread settings are defined as part of your HTTP connector in your ./bopcfg/www/CATALINA_BASE/\conf /server.xml

    Tread determine the number of "action" that Tomcat can perform in parallel for this connector. When the maximum number of thread is reached then the pending action have to wait for another to terminate.

     

    minSpareThreads determine the number on thread to be created at startup

    maxThreads determine the maximum number of Thread that the connector will handle simultaneously

    acceptCount determine the number or request that will be queued if the maxThreads  number is reached before the connector simply refuse the connection

     

     

    Note that the reach of the max thread may be due to the application not releasing them  because for example database queries are returned to slow. Then prior to increase this value you may want to check if the problem is not residing there vs. Tomcat. Increasing this number is consuming an important amount of the memory that we defined above.

    There is generally no reason to increase those settings if you don't receive specific errors about this in your Tomcat log.

    I will not recommend to go over 500 in any case

     

    <Connector SSLEnabled="true" clientAuth="false" keystoreFile="E:\SSL\.keystore.pfx" keystorePass="password"  maxThreads="250" minSpareThreads="10" port="443" protocol="org.apache.coyote.http11.Http11Protocol" scheme="https" secure="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1,TLSv1.1,TLSv1.2">

     

     

    AJP Connector

    This connector like the HTTP is configured in your ./bopcfg/www/CATALINA_BASE/\conf /server.xml

    This connector is only needed if you use Tomcat in combination with Apache. if this is not the case you can disable this connector to save some resource by commenting it out

    <!-- <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" /> -->

     

    to continue here

     

    Hope this help

     

    /J



  • 2.  Re: Fine Tuning Tomcat for better performance

     
    Posted Mar 09, 2016 11:14 AM

    Thanks for sharing this with the community jmayer!

    Fine Tuning Tomcat for better performance



  • 3.  Re: Fine Tuning Tomcat for better performance

    Posted Mar 10, 2016 12:27 AM

    Excellent!

     

    Thanks, Jerome.

     

    All feedback on this thread is welcome.

     

    • What do you setup for Tomcat?

     

    • What is right about this post?

     

    • What could be expanded on?

     

     

    If we get enough verification of site usage, I'll put this into a TEC document. Otherwise, please have at it on this thread, and it will be a useful resource point for anyone tuning Tomcat.

     

    Thanks again, Jerome.

     

    Kyle_R.



  • 4.  Re: Fine Tuning Tomcat for better performance

    Posted Mar 11, 2016 04:50 PM

    Hi jmayer,

     

    Thank you for sharing!  This is very helpful.  I'm working on updating my dev/test environments with the suggestions from this document.  I just finished the changes in my dev environment and was expecting the server memory to jump up considerably.  The primary service desk tomcat container, rest container, and vizualizer container have all been updated so that the xms/xmx values are the same.  During startup I can see the memory is being assigned correctly in the stdlogs.  My question is this, is the server supposed to allocate the entire xms value during startup or does it slowly utilize more memory until it reaches the xmx value?

     

    Here are the memory settings in my nx.env file.

    @NX_JAVA_OPTIONS=-Djava.net.preferIPv4Stack=false -Xms2048M -Xmx2048M
    @NX_JAVA_OPTIONS_REST=-Djava.net.preferIPv4Stack=false -Xms2048M -Xmx2048M -Djavax.servlet.request.encoding=UTF-8 -Djavax.servlet.response.encoding=UTF-8 -Dfile.encoding=UTF-8
    @NX_JAVA_OPTIONS_FS=-Djava.net.preferIPv4Stack=false -Xms64M -Xmx512M -Djavax.servlet.request.encoding=UTF-8 -Djavax.servlet.response.encoding=UTF-8 -Dfile.encoding=UTF-8
    

     

    Service Desk started successfully with the memory configurations shown and the server is currently using 2.2GB of 8GB ram.  I just want to clarify that is the expected behavior.  My work is very strict with VM memory, so if I can't prove that it needs the entire 8GB they will take some of it back.



  • 5.  Re: Fine Tuning Tomcat for better performance

    Posted Mar 12, 2016 02:51 AM

    Hi Grant,

     

    The Xms is defining the minimun as the startup of the container so that will be the minimum memory to be used.

    as per the config you show above you supposed to use 4160 so little less that 4GB vs. 2.2GB.

     

    However some point to take in consideration:

     

    - Either for the minimum tomcat will allocate the memory progressively, so if you look directly at the time your start all may not be loaded yet.

    Wait for few minutes after you start to have correct consumption. (tips look in your task manager for the javaw process) and you must see the mem growing there until reaching at less the minimum configured.

    - if no application is deployed in your container no thread will be loaded. So my question here will be if you have deployed your REST in the container and if you use the Federated search engine as the container will not be started if not configured.

    - I don't see visualizer in your config

    - There is no need to use 2048 in -Xms if you have constrains on VM memory and don't need it. The Xmx value is the more important to avoid crash and/or slow performance. Xms and Xmx been identic will increase performance for sure but at the cost of this memory.You really need to check what is the real consumption of your server to fine tune it based on your usage.

    Hope this help

    /J



  • 6.  Re: Fine Tuning Tomcat for better performance

    Posted Mar 14, 2016 10:29 AM

    Thank you for the reply!

     

    You mention that tomcat will allocate memory progressively.  So does that mean that it will not use the whole XMS value at startup?  We don't have anything deployed in the rest container but have a few applications which auto generate incidents.

     

    Our production systems have started to slow the last few months and I noticed they were starved for memory.  So this post came at a good time as I was already looking at ways to improve performance.

     

    Here's a screenshot of event viewer with Service Desk running, followed by our tomcat config.  If tomcat does allocate memory progressively then this may be normal.

     

     

    # Tomcat 
    @NX_JAVA_OPTIONS=-Djava.net.preferIPv4Stack=false -Xms2048M -Xmx2048M
    @NX_JAVA_OPTIONS_REST=-Djava.net.preferIPv4Stack=false -Xms2048M -Xmx2048M -Djavax.servlet.request.encoding=UTF-8 -Djavax.servlet.response.encoding=UTF-8 -Dfile.encoding=UTF-8
    @NX_JAVA_OPTIONS_FS=-Djava.net.preferIPv4Stack=false -Xms64M -Xmx512M -Djavax.servlet.request.encoding=UTF-8 -Djavax.servlet.response.encoding=UTF-8 -Dfile.encoding=UTF-8
    @NX_JAVA_OPTIONS_VIZ=-Djava.net.preferIPv4Stack=false -Xms1024M -Xmx1024M
    


  • 7.  Re: Fine Tuning Tomcat for better performance

    Posted Mar 14, 2016 02:24 PM

    no that mean the Xms growth up slowly at the container startup to reach the defined value but must be normally there 1-2 minutes after startup.

    Did you look in the process tab to see the memory consumption for your respective javaw.exe?

    I typically use Process explorer for that as this give me more info that the task manager.

     

    Also I may need to clarify by default windows task manager display working set vs. private bytes where the Xms value are private bytes

    Mem is also display per kb vs Mb so their is little conversion there too

    See below after a quick change and restart where task Process explorer show both the private bytes allocated as per the NX.env and the working set managed by the OS and reported in task manager

    Hope this clarify

    /J

    javaw_ mem.PNG



  • 8.  Re: Fine Tuning Tomcat for better performance

    Posted Mar 14, 2016 02:52 PM

    Thank you for pointing this out!  I forgot about Process Explorer.  I can now see the memory is being allocated during startup.  One more thing to point out, Service Desk Manager is currently only compatible with 32 bit JREs.  The first config I posted didn't work as it exceeded the amount of memory compatible with the 32 bit JRE.



  • 9.  Re: Fine Tuning Tomcat for better performance

    Posted May 09, 2017 06:47 PM

    What are the ramifications of increasing the max thread if you are not seeing any specific errors in the tomcat log.



  • 10.  Re: Fine Tuning Tomcat for better performance

    Posted May 10, 2017 09:42 AM

    The main one will be the hardware CPU and memory resource available for your tomcat.

    Going over 300 mostly require to have tomcat in a cluster mode.

    If you don't have the max tread error in your log mean to never reach it so need to increase.

    Main principle must be that if your app is working well don't touch !

    My 2 cents

    /J