DX Unified Infrastructure Management

Expand all | Collapse all

UIM: UMP java.lang.OutOfMemoryError: GC overhead limit exceeded

  • 1.  UIM: UMP java.lang.OutOfMemoryError: GC overhead limit exceeded

    Broadcom Employee
    Posted Dec 06, 2016 10:54 AM

    Dear community,

     

    one of our customers started seeing the following error in UMP:

    java.lang.OutOfMemoryError: GC overhead limit exceeded

     

    The same error can be seen in wasp.log and portal.log files.

     

    Dec 02 09:36:17:052 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase] run() Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[wasp-engine]]
    Dec 02 09:36:22:768 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase] java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.StringCoding.safeTrim(Unknown Source)
    at java.lang.StringCoding.access$300(Unknown Source)
    at java.lang.StringCoding$StringEncoder.encode(Unknown Source)
    at java.lang.StringCoding.encode(Unknown Source)
    at java.lang.String.getBytes(Unknown Source)
    at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
    at java.io.File.lastModified(Unknown Source)
    at org.apache.naming.resources.FileDirContext$FileResourceAttributes.getLastModified(FileDirContext.java:1008)
    at org.apache.naming.resources.FileDirContext$FileResourceAttributes.getCreation(FileDirContext.java:980)
    at org.apache.naming.resources.FileDirContext$FileResourceAttributes.<init>(FileDirContext.java:925)
    at org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirContext.java:403)
    at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1157)
    at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1110)
    at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:882)
    at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:1026)
    at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:500)
    at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:420)
    at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1345)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1546)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1556)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1556)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1524)
    at java.lang.Thread.run(Unknown Source)

    Dec 02 09:37:08:604 ERROR [http-bio-80-exec-180, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/nisapi].[resteasy-servlet]] invoke() Servlet.service() for servlet [resteasy-servlet] in context with path [/nisapi] threw exception
    Dec 02 09:37:08:605 ERROR [http-bio-80-exec-180, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/nisapi].[resteasy-servlet]] org.jboss.resteasy.spi.UnhandledException: java.lang.OutOfMemoryError: GC overhead limit exceeded
    at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
    at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
    at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)
    Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

    Dec 02 11:16:45:875 ERROR [http-bio-80-exec-207, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/].[jsp]] invoke() Servlet.service() for servlet jsp threw exception
    Dec 02 11:16:45:875 ERROR [http-bio-80-exec-207, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/].[jsp]] java.lang.OutOfMemoryError: GC overhead limit exceeded

    02 Dec 2016 11:33:23,861 ERROR [PortalInitAction:111] Exception in thread "http-bio-80-exec-207"
    02 Dec 2016 11:33:23,861 ERROR [PortalInitAction:111] java.lang.OutOfMemoryError: GC overhead limit exceeded

    The UMP server is a dedicated VM with the following specs:

    lscpu
    Architecture: x86_64
    CPU op-mode(s): 32-bit, 64-bit
    Byte Order: Little Endian
    CPU(s): 4
    On-line CPU(s) list: 0-3
    Thread(s) per core: 1
    Core(s) per socket: 4
    Socket(s): 1
    NUMA node(s): 1
    Vendor ID: GenuineIntel
    CPU family: 6
    Model: 63
    Stepping: 2
    CPU MHz: 2299.998
    BogoMIPS: 4599.99
    Hypervisor vendor: VMware
    Virtualization type: full
    L1d cache: 32K
    L1i cache: 32K
    L2 cache: 256K
    L3 cache: 25600K
    NUMA node0 CPU(s): 0-3

    free -m
    total used free shared buffers cached
    Mem: 11912 5856 6056 0 258 3353
    -/+ buffers/cache: 2244 9668
    Swap: 16383 0 16383

    df -h
    Filesystem Size Used Avail Use% Mounted on
    /dev/sda2 24G 6.2G 17G 28% /
    tmpfs 5.9G 0 5.9G 0% /dev/shm
    /dev/sda1 339M 71M 251M 22% /boot
    /dev/sda5 59G 52M 56G 1% /usr/people

    At a first glance it seems like a performance issue.

     

    I'm looking for suggestions what to check next

     

    Many thanks

     

    Mike



  • 2.  Re: UIM: UMP java.lang.OutOfMemoryError: GC overhead limit exceeded
    Best Answer

    Posted Dec 06, 2016 11:50 AM

    open the wasp probe raw configure > startup > opt > java_mem_max

    compare that to how much memory wasp is using

    it can be increased as needed provided the system can support it.

    deactivate activate after making a change

     

    a common cause for wasp using a whole lot of memory is a whole lot of alarms, especially open alarms.