Dear community,
one of our customers started seeing the following error in UMP:
java.lang.OutOfMemoryError: GC overhead limit exceeded
The same error can be seen in wasp.log and portal.log files.
Dec 02 09:36:17:052 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase] run() Unexpected death of background thread ContainerBackgroundProcessor[StandardEngine[wasp-engine]]
Dec 02 09:36:22:768 ERROR [ContainerBackgroundProcessor[StandardEngine[wasp-engine]], org.apache.catalina.core.ContainerBase] java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.StringCoding.safeTrim(Unknown Source)
at java.lang.StringCoding.access$300(Unknown Source)
at java.lang.StringCoding$StringEncoder.encode(Unknown Source)
at java.lang.StringCoding.encode(Unknown Source)
at java.lang.String.getBytes(Unknown Source)
at java.io.UnixFileSystem.getLastModifiedTime(Native Method)
at java.io.File.lastModified(Unknown Source)
at org.apache.naming.resources.FileDirContext$FileResourceAttributes.getLastModified(FileDirContext.java:1008)
at org.apache.naming.resources.FileDirContext$FileResourceAttributes.getCreation(FileDirContext.java:980)
at org.apache.naming.resources.FileDirContext$FileResourceAttributes.<init>(FileDirContext.java:925)
at org.apache.naming.resources.FileDirContext.doGetAttributes(FileDirContext.java:403)
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1157)
at org.apache.naming.resources.BaseDirContext.getAttributes(BaseDirContext.java:1110)
at org.apache.naming.resources.ProxyDirContext.getAttributes(ProxyDirContext.java:882)
at org.apache.catalina.loader.WebappClassLoader.modified(WebappClassLoader.java:1026)
at org.apache.catalina.loader.WebappLoader.modified(WebappLoader.java:500)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:420)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1345)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1546)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1556)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1556)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1524)
at java.lang.Thread.run(Unknown Source)
Dec 02 09:37:08:604 ERROR [http-bio-80-exec-180, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/nisapi].[resteasy-servlet]] invoke() Servlet.service() for servlet [resteasy-servlet] in context with path [/nisapi] threw exception
Dec 02 09:37:08:605 ERROR [http-bio-80-exec-180, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/nisapi].[resteasy-servlet]] org.jboss.resteasy.spi.UnhandledException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
at org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
at org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:149)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:372)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Dec 02 11:16:45:875 ERROR [http-bio-80-exec-207, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/].[jsp]] invoke() Servlet.service() for servlet jsp threw exception
Dec 02 11:16:45:875 ERROR [http-bio-80-exec-207, org.apache.catalina.core.ContainerBase.[wasp-engine].[localhost].[/].[jsp]] java.lang.OutOfMemoryError: GC overhead limit exceeded
02 Dec 2016 11:33:23,861 ERROR [PortalInitAction:111] Exception in thread "http-bio-80-exec-207"
02 Dec 2016 11:33:23,861 ERROR [PortalInitAction:111] java.lang.OutOfMemoryError: GC overhead limit exceeded
The UMP server is a dedicated VM with the following specs:
lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Stepping: 2
CPU MHz: 2299.998
BogoMIPS: 4599.99
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-3
free -m
total used free shared buffers cached
Mem: 11912 5856 6056 0 258 3353
-/+ buffers/cache: 2244 9668
Swap: 16383 0 16383
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 24G 6.2G 17G 28% /
tmpfs 5.9G 0 5.9G 0% /dev/shm
/dev/sda1 339M 71M 251M 22% /boot
/dev/sda5 59G 52M 56G 1% /usr/people
At a first glance it seems like a performance issue.
I'm looking for suggestions what to check next
Many thanks
Mike