AnsweredAssumed Answered

UIM: UMP java.lang.OutOfMemoryError: GC overhead limit exceeded

Question asked by krimi03 Employee on Dec 6, 2016
Latest reply on Dec 6, 2016 by DavidM

Dear community,

 

one of our customers started seeing the following error in UMP:

java.lang.OutOfMemoryError: GC overhead limit exceeded

 

The same error can be seen in wasp.log and portal.log files.

 

Dec 02 09:36:17:052 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase] run() Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[wasp-engine]]
Dec 02 09:36:22:768 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase] java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.StringCoding.safeTrim(Unknown Source)
at java.lang.StringCoding.access$300(Unknown Source)
at java.lang.StringCoding$StringEncoder.encode(Unknown Source)
at java.lang.StringCoding.encode(Unknown Source)
at java.lang.String.getBytes(Unknown Source)
at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
at java.io.File.lastModified(Unknown Source)
at org.apache.naming.resources.FileDirContext$FileResourceAttributes.getLastModified(FileDirContext.java:1008)
at org.apache.naming.resources.FileDirContext$FileResourceAttributes.getCreation(FileDirContext.java:980)
at org.apache.naming.resources.FileDirContext$FileResourceAttributes.<init>(FileDirContext.java:925)
at org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirContext.java:403)
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1157)
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1110)
at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:882)
at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:1026)
at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:500)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:420)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1345)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1546)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1556)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1556)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1524)
at java.lang.Thread.run(Unknown Source)

Dec 02 09:37:08:604 ERROR [http-bio-80-exec-180, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/nisapi].[resteasy-servlet]] invoke() Servlet.service() for servlet [resteasy-servlet] in context with path [/nisapi] threw exception
Dec 02 09:37:08:605 ERROR [http-bio-80-exec-180, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/nisapi].[resteasy-servlet]] org.jboss.resteasy.spi.UnhandledException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

Dec 02 11:16:45:875 ERROR [http-bio-80-exec-207, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/].[jsp]] invoke() Servlet.service() for servlet jsp threw exception
Dec 02 11:16:45:875 ERROR [http-bio-80-exec-207, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/].[jsp]] java.lang.OutOfMemoryError: GC overhead limit exceeded

02 Dec 2016 11:33:23,861 ERROR [PortalInitAction:111] Exception in thread "http-bio-80-exec-207"
02 Dec 2016 11:33:23,861 ERROR [PortalInitAction:111] java.lang.OutOfMemoryError: GC overhead limit exceeded

The UMP server is a dedicated VM with the following specs:

lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Stepping: 2
CPU MHz: 2299.998
BogoMIPS: 4599.99
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-3

free -m
total used free shared buffers cached
Mem: 11912 5856 6056 0 258 3353
-/+ buffers/cache: 2244 9668
Swap: 16383 0 16383

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 24G 6.2G 17G 28% /
tmpfs 5.9G 0 5.9G 0% /dev/shm
/dev/sda1 339M 71M 251M 22% /boot
/dev/sda5 59G 52M 56G 1% /usr/people

At a first glance it seems like a performance issue.

 

I'm looking for suggestions what to check next

 

Many thanks

 

Mike

Outcomes