AutoSys Workload Automation

We had a CA Health Check towards the end of December of 17 and have completed most of the recommendations they made. We stood up new Windows servers for WCC and EEM. WCC now sits on its own server and is no longer using the Derby database and instead going to an Oracle database on another host. EEM now sits on its own server as well.

We constantly are running into a problem where the login screen for WCC comes up just fine but once a user enters their credentials, it sits and spins and then times out. We have monitoring around this using DynaTrace which see's the issue and points right to tomcat_*.

Threads resources exhausted

Threads exhausted on tomcat_*

Looking through logs we find...

WARNING: The web application [wcc#quickview] appears to have started a thread named [pool-5-thread-3] but has failed to stop it. This is very likely to create a memory leak. 

We are using WCC Release: 11.4.5.20170424-b146

Its on: Microsoft Windows Server 2012 R2 Standard

Version: 6.3.9600 Build 9600

Memory: 8GB

CPU: 4 Intel(R) Xeon(R) CPU E5-2670 0 @ 3.60GHz

Tomcat 32 wrapper.conf is as follows:

#********************************************************************
# Wrapper Properties
#********************************************************************
# Java Application
wrapper.java.command=../../jre_32/bin/java
wrapper.debug=TRUE
wrapper.java.initmemory=512
wrapper.java.maxmemory=1024
wrapper.ignore_sequence_gaps=TRUE

Tomcat wrapper.conf is as follows:

#********************************************************************
# Wrapper Properties
#********************************************************************
# Java Application
wrapper.java.command=../../jre/bin/java
wrapper.debug=false
wrapper.java.initmemory=2048
wrapper.java.maxmemory=4096
wrapper.ignore_sequence_gaps=TRUE

Here is a bit more information from the logs. We see this each time where the collector seems to bomb but we can not tell if this is the actual issue causing the threat exhaustion or is is a symptom of whatever is actually causing the thread exhaustion.

We see this in the collector.log

2018-01-16 13:47:32,879 @reporting-collector <reporting-collectors-loop> [] INFO #AutoSysJobRunExtractor # getLatestEventDateTime(SIT_AutoSys) returns Wed Jan 17 22:00:00 EST 2018
2018-01-16 13:52:33,392 @reporting-collector <reporting-collectors-loop> [] WARN #ConfigHttpInvokerRequestExecutor # Waiting for full startup of configuration services: Read timed out
2018-01-16 13:52:33,392 @reporting-collector <reporting-collectors-loop> [] ERROR #ConfigHttpInvokerRequestExecutor # Configuration Remote Interface: SocketTimeoutException encountered: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1569)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1474)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.validateResponse(SimpleHttpInvokerRequestExecutor.java:178)
at org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.doExecuteRequest(SimpleHttpInvokerRequestExecutor.java:92)
at com.ca.wcc.config.remote.httpinvoker.ConfigHttpInvokerRequestExecutor.doExecuteRequest(ConfigHttpInvokerRequestExecutor.java:75)
at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy104.getPreference(Unknown Source)
at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.ca.wcc.advice.CacheAdvice.get(CacheAdvice.java:67)
at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy115.getPreference(Unknown Source)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getMarginForMaxEventAge(AutoSysJobRunExtractorWithNewFilter.java:303)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getFilter(AutoSysJobRunExtractorWithNewFilter.java:212)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractor.extract(AutoSysJobRunExtractor.java:179)
at com.ca.uejm.datawarehouse.wcc.AutoSysJobRunExtractorWrapper.instanciateAndRun(AutoSysJobRunExtractorWrapper.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapperWithServerLoop.instanciateAndRun(BaseWrapperWithServerLoop.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapper.run(BaseWrapper.java:76)
at com.ca.uejm.datawarehouse.wcc.AutosysCollector.run(AutosysCollector.java:71)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.collect(ReportingCollectorsLoopService.java:380)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.doWork(ReportingCollectorsLoopService.java:231)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.work(BaseLoopImpl.java:230)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.workNoLock(LoopWithLock.java:71)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.work(LoopWithLock.java:99)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.runLoop(BaseLoopImpl.java:193)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.runLoop(ReportingCollectorsLoopService.java:199)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl$1.run(BaseLoopImpl.java:141)
at java.lang.Thread.run(Thread.java:745)
2018-01-16 13:52:33,408 @reporting-collector <reporting-collectors-loop> [] ERROR #BaseWrapperWithServerLoop # for server SIT_AutoSys: Could not access HTTP invoker remote service at [http://localhost:10132/wcc-configuration/cfmremote/remoteGlobalPrefService]; nested exception is com.ca.wcc.config.api.exceptions.WCCException: Read timed out
org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [http://localhost:10132/wcc-configuration/cfmremote/remoteGlobalPrefService]; nested exception is com.ca.wcc.config.api.exceptions.WCCException: Read timed out
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:213)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:145)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy104.getPreference(Unknown Source)
at sun.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at com.ca.wcc.advice.CacheAdvice.get(CacheAdvice.java:67)
at sun.reflect.GeneratedMethodAccessor221.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy115.getPreference(Unknown Source)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getMarginForMaxEventAge(AutoSysJobRunExtractorWithNewFilter.java:303)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractorWithNewFilter.getFilter(AutoSysJobRunExtractorWithNewFilter.java:212)
at com.ca.uejm.datawarehouse.AutoSysJobRunExtractor.extract(AutoSysJobRunExtractor.java:179)
at com.ca.uejm.datawarehouse.wcc.AutoSysJobRunExtractorWrapper.instanciateAndRun(AutoSysJobRunExtractorWrapper.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapperWithServerLoop.instanciateAndRun(BaseWrapperWithServerLoop.java:56)
at com.ca.uejm.datawarehouse.wcc.BaseWrapper.run(BaseWrapper.java:76)
at com.ca.uejm.datawarehouse.wcc.AutosysCollector.run(AutosysCollector.java:71)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.collect(ReportingCollectorsLoopService.java:380)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.doWork(ReportingCollectorsLoopService.java:231)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.work(BaseLoopImpl.java:230)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.workNoLock(LoopWithLock.java:71)
at com.ca.uejm.datawarehouse.wcc.LoopWithLock.work(LoopWithLock.java:99)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl.runLoop(BaseLoopImpl.java:193)
at com.ca.uejm.datawarehouse.wcc.ReportingCollectorsLoopService.runLoop(ReportingCollectorsLoopService.java:199)
at com.ca.uejm.datawarehouse.wcc.BaseLoopImpl$1.run(BaseLoopImpl.java:141)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.ca.wcc.config.api.exceptions.WCCException: Read timed out
at com.ca.wcc.config.remote.httpinvoker.ConfigHttpInvokerRequestExecutor.doExecuteRequest(ConfigHttpInvokerRequestExecutor.java:108)
at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174)
at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142)
... 38 more

This then just keeps repeating in the log over and over until we conduct the restart of WCC.

We see this in the config-services.log...

2018-01-16 14:15:36,991 @configservices <clusterHealthChecker> [] ERROR #ClusterHealtAnalyzer # Error checking cluster
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1439)
at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
at com.ca.wcc.health.ClusterHealtAnalyzer$ClusterSanityAnalyzerTask.run(ClusterHealtAnalyzer.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-01-16 14:16:36,981 @configservices <clusterHealthChecker> [] ERROR #ClusterHealtAnalyzer # Error checking cluster
java.util.NoSuchElementException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1439)
at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
at com.ca.wcc.health.ClusterHealtAnalyzer$ClusterSanityAnalyzerTask.run(ClusterHealtAnalyzer.java:68)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2018-01-16 14:17:32,560 @configservices <http-nio-127.0.0.1-10132-exec-15> [] ERROR #DefaultUserPreferenceService # Exception caught setting user preference.
User[vqq4r], Key[config.user.timezone.offset.raw], Value[-18000000]
Exception: Hazelcast Instance is not active!

Again, it repeats over and over until we restart WCC.

We see this in the remoteservices.log...

2018-01-16 13:38:32,666 @remote <http-nio-127.0.0.1-10132-exec-14> [] ERROR #ASAPI # The AutoSys application server timed out.
com.ca.autosys.services.AsTimeoutException:
at com.ca.autosys.services.JResponseQ.dequeue(Native Method)
at com.ca.autosys.services.response.FilterRspSet.dequeueResponse(FilterRspSet.java:88)
at com.ca.autosys.services.response.ApiResponseSet.hasNext(ApiResponseSet.java:70)
at com.ca.uejm.adapter.autosys.command.ASAPI.getHistoryJobRuns(ASAPI.java:5650)
at com.ca.uejm.adapter.autosys.command.ASAPI.getHistoryJobRuns(ASAPI.java:5596)
at com.ca.wcc.service.adapter.AutoSysServiceImpl.getJobRuns(AutoSysServiceImpl.java:804)
at sun.reflect.GeneratedMethodAccessor352.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy36.getJobRuns(Unknown Source)
at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)
at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78)
at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114)
at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:78)
at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:49)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at com.ca.wcc.context.LazyWrappingServlet.service(LazyWrappingServlet.java:109)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:783)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:798)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1434)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

CA Support has had us adjust these java memory numbers numerous times. All it has done is slowed down how often the problem takes place. We went from 3 times a day to about once a week now. The only way to resolve it is to restart. We are really at a loss as to why we keep running into this issue and I am hoping somebody else out there has or is running into this same thing.

Other basic info

We do have the reporting enabled but not really used much at all in WCC which again now uses an Oracle database. EEM appears to be just fine and never gives us any issues. Same with Autosys. 

Any help would be greatly appreciated!!!! Thank you

I would also like to know if anyone else is also experiencing this at all?

Hello Roberts

I think there is not enough RAM on the machine. I have found your ticket in our database and the RAM is 4GB.

May be this information is not accurate or since then you upgraded the real memory but our hardware recommendations for a WCC server are described HERE

In short:

The requirements for a Windows or Linux x86 computer are as follows:

• 2 GHz or higher processor recommended.
• Two CPUs minimum, four CPUs recommended for large enterprises.
• At least 6 GB of RAM; 8 GB recommended.
• 2 GB of hard disk space for the installation, including CA WCC and the required common components.
• Additionally, the fastest processor recommended for any single component should be used.

In addition, as a best practice, you need to set page file "1.5 times" the RAM or memory available on any Windows Servers.

If you are still getting such problems after the implementation of these requirements and best practices, please open a new case at CA support to continue the investigations

Regards

Jean Paul

Please see above post again in detail.

Memory: 8GB

CPU: 4 Intel(R) Xeon(R) CPU E5-2670 0 @ 3.60GHz

You are correct, that is when we first started to deal with this. Since that time the memory was raised to 8GB on each WCC instance so memory should not be the issue. 

Also, the official CA documentation you linked that I have also looked at many times states...

Note: The requirements that follow are for a typical enterprise with up to two CA Workload Automation AE instances and up to 15,000 jobs in the database of each CA Workload Automation AE instance.

We only have 4,500 jobs in our database so technically if I am going by this.... Our specs should be overkill. I would really like to know if anyone else has run into this where they are finding they need to restart WCC a bit too often. 

The newer version of WCC is more efficient with the number of Tomcat engines being used.

Tho' IMHO, I would bump up the 8 GB to double that.

Two factors impact the usage of WCC, the number of jobs and the number of queries thrown at it.

Perhaps, WCC is being heavily used?

Are there any error messages in the catalina.out file or the system.out/system.err files?

[With the older version of WCC, we had to recycle the services daily - Steve C, will remember this]

The other thing I noticed is that a WIN platform is more picky, I do not have hard facts to back up the statement, it is only an observation.

As with all things there is a caveat: I am no WCC and/or java expert and the comments are best effort.

Cheers,

Chris  <CJ>

The newer version of WCC is more efficient with the number of Tomcat engines being used.

Tho' IMHO, I would bump up the 8 GB to double that.

Two factors impact the usage of WCC, the number of jobs and the number of queries thrown at it.

Perhaps, WCC is being heavily used?

I too has this thought but we only have 4,500 jobs and about 5 to 6 users and any given time on WCC.

Are there any error messages in the catalina.out file or the system.out/system.err files?

None that I have been unable to find. The next time I encounter the lockup again, I will go right there and look.

[With the older version of WCC, we had to recycle the services daily - Steve C, will remember this]

This is exactly why we are becoming very frustrated with CA in general. We had these types of issues with Service Desk years ago and it took over 6 months of weekly calls and trying to prove to them there was a real issue with the product and that restarting the services is not a "fix". Eventual after much pressure, CA fixed it. Now with WCC, we are right back at it again. 

Thank you! I really do appreciate the assistance.

Hello Roberts,

Some suggestions if you are able to test...

1) If you are not using WCC Reports, then disable the Reporting Collector in the Configuration. You'll save a few threads.

2) If DynaTrace is pointing to the Tomcat "worker" threads, then increase (from 150 to 300) the maxThreads in the %CA_WCC_INSTALL_LOCATION%\tomcat_32\conf\server.xml file, like so:

                <Connector acceptCount="100" clientAuth="false"
                        disableUploadTimeout="true" enableLookups="false"
                        maxThreads="300" minSpareThreads="25" port="10132" scheme="http"
                        protocol="HTTP/1.1" URIEncoding="UTF-8" compression="on"
                        compressableMimeType="text/html,text/xml,text/plain,text/javascript,text/css,application/x-javascript"
                        server="WCC" />

With that we are allowing 150 more worker threads to be spawned by Tomcat.

A high level flow of the Tomcat is this. The acceptor thread accepts connections and checks if there are any free "worker" threads to process the request. If there aren't any, then create a worker thread provided the maxThreads is not exceeding. Else,  wait for a worker thread to become available. When it finds a free worker thread, it hands off the connection to that thread and goes back to listen for new connections. With the aforementioned config, there will be 100 acceptor threads, 300 worker threads.

It may boil down a some worker threads not being freed or timed out and so all worker threads gets busy processing and Tomcat is not able spawn anymore (Too high a number of threads isn't good either as it might result in CPU starvation).

3) Apply WCC 11.4 SP5 Incr1 fix RO99200 if not already done. Monitoring collector has been tuned to handle performance issues.

4) If you are using WAAE 11.3.6 SP6, then apply the Cum1 fix which addresses a known memory leak issue with SDKs (WCC uses WAAE SDKs extensively)

On a side note, Tomcat thread exhaustion has little to do with OutOfMemory as the threads do not use JVM heap size. Threads rather use the stack which is besides the JVM heap.  The stack size can be configured using JVM -Xss option, but the default is usually large enough. In case one runs out of stack, then StackOverFlow exception will be thrown.

Thanks,

Chandru

If you are not using WCC Reports, then disable the Reporting Collector in the Configuration. You'll save a few threads.

That is a step we have been taking as well and it does help. It's a situation where it feels like we are paying for a product that clearly has some issues with it's options when enabled.

If DynaTrace is pointing to the Tomcat "worker" threads, then increase (from 150 to 300) the maxThreads in the %CA_WCC_INSTALL_LOCATION%\tomcat_32\conf\server.xml file, like so:

                <Connector acceptCount="100" clientAuth="false"
                        disableUploadTimeout="true" enableLookups="false"
                        maxThreads="300" minSpareThreads="25" port="10132" scheme="http"
                        protocol="HTTP/1.1" URIEncoding="UTF-8" compression="on"
                        compressableMimeType="text/html,text/xml,text/plain,text/javascript,text/css,application/x-javascript"
                        server="WCC" />

 

With that we are allowing 150 more worker threads to be spawned by Tomcat.

This is good info!!! I will give this a try and see if it helps.

It may boil down a some worker threads not being freed or timed out and so all worker threads gets busy processing and Tomcat is not able spawn anymore (Too high a number of threads isn't good either as it might result in CPU starvation).

This is exactly what we suspect is taking place. Threads are not being freed for some reason and then through time depending on thread settings, like a chain reaction, WCC chokes. I have been unable to find a way to prove this or get CA to even look that direction. 

If you are using WAAE 11.3.6 SP6, then apply the Cum1 fix which addresses a known memory leak issue with SDKs (WCC uses WAAE SDKs extensively)

We are on 11.4 SP5 (release 11.4.5.20170424-b146)

Thank you for the assistance and good info! I really appreciate it!!!

Apply WCC 11.4 SP5 Incr1 fix RO99200 if not already done. Monitoring collector has been tuned to handle performance issues.

Had no clue there even was an R099200. Good stuff!

I really wish CA would start adding the fixes to the SP lists like here.