AnsweredAssumed Answered

App service is gone!

Question asked by L00421 on Oct 16, 2012
Latest reply on Jan 10, 2013 by another_martink
Hi team,
we are running Clarity v13.0.1 on two servers and I have just been reported that Clarity was down. I checked on the primary server and all services were up. I stopped them and restarted them to see if that helped bring Clarity up again. App service didn't start again, so I went into Control Panel/Services to try and restart it from there. I got a message saying that the service was marked for deletion, and a few seconds after, it was gone. I was able to bring Clarity up by restarting the services on the secondary server, however my primary server is not working, only beacon, nsa and bg services are up. I don't know how to re-create app service there, especially since I don't know how it was deleted :(
I found a post in which they say that even a single user can boost the Java memory usage to its maximum and make it crash, with error java.lang.OutOfMemoryError: Java heap space (which is what we have)
16379798
This is the last registered in the app-ca.log:

ERROR 2012-10-16 13:42:48,836 [http-443-14] performance.PerformanceMonitor (clarity:B91142:5426526__D8E4CF38-B10A-451F-9692-563EB65927FE:projmgr.capacity) Mismatch in timing element stack, where titles do not match ('webRequest' != 'action') or ids do not match ('projmgr.capacity' != 'projmgr.capacity')
ERROR 2012-10-16 13:42:48,858 [http-443-7] union.service (clarity:B91142:5426526__D8E4CF38-B10A-451F-9692-563EB65927FE:npt.gridUserLayoutEntry) rollback exception
java.lang.OutOfMemoryError:
ERROR 2012-10-16 13:42:58,566 [http-443-7] web.WebActionController (clarity:B91142:5426526__D8E4CF38-B10A-451F-9692-563EB65927FE:npt.gridUserLayoutEntry)
com.niku.union.web.WebException: Service call failed for portlet: npt.portletToc
at com.niku.union.web.PortletController.processAuthorizedPortlet(PortletController.java:906)
at com.niku.union.web.PortletController.processPageActionRequest(PortletController.java:394)
at com.niku.union.web.PortletController.processRequest(PortletController.java:163)
at com.niku.union.web.WebActionController.processAction(WebActionController.java:1124)
at com.niku.union.web.WebActionController.processRequest(WebActionController.java:163)
at com.ca.clarity.uif.service.vxml.VXMLService.processRequest(VXMLService.java:403)
at com.ca.clarity.uif.UIServlet.service(UIServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.niku.union.web.filter.GZIPResponseFilter.doFilter(GZIPResponseFilter.java:130)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.niku.union.web.filter.CharsetFilter.doFilter(CharsetFilter.java:56)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:619)
Caused by: com.niku.union.web.WebException: WebServiceAdapter::processRequest -> failed to process service request
at com.niku.union.web.WebServiceAdapter.processRequest(WebServiceAdapter.java:198)
at com.niku.union.web.PortletController.processServiceRequest(PortletController.java:1393)
at com.niku.union.web.PortletController.processAuthorizedPortlet(PortletController.java:876)
... 26 more
Caused by: com.niku.union.service.ServiceException: java.lang.OutOfMemoryError: Java heap space
at com.niku.xql2.binding.ServiceControlBindingXql.processRequest(ServiceControlBindingXql.java:198)
at com.niku.union.service.ServiceControlImpl.processRequest(ServiceControlImpl.java:749)
at com.niku.union.service.ServiceControlImpl.processRequest(ServiceControlImpl.java:211)
at com.niku.union.web.WebServiceAdapter.processRequest(WebServiceAdapter.java:194)
... 28 more

Now my secondary server is down too, and the last entry on the app-ca.log is similar
FATAL 2012-10-16 14:44:03,142 [http-443-9] web.WebControlServlet (clarity:B84967:5428379__7E82D5DF-117C-460B-A61B-48FB69AF386E:npt.gridExcelExport) Action request error for request. Could not render standard error page
java.lang.OutOfMemoryError: Java heap space
at com.ca.clarity.jdbc.sqlserver.tds.ddr.a(Unknown Source)
at com.ca.clarity.jdbc.sqlserver.tds.ddr.a(Unknown Source)
at com.ca.clarity.jdbc.sqlserver.ddh.a(Unknown Source)
at com.ca.clarity.jdbc.sqlserverbase.dddk.getObject(Unknown Source)
at com.niku.union.persistence.jdbc.SQLTraceResultSet.getObject(SQLTraceResultSet.java:372)
at com.niku.union.persistence.utils.SQLUtils.extractValue(SQLUtils.java:108)
at com.niku.union.persistence.utils.SQLUtils.extractValueAsObject(SQLUtils.java:201)
at com.niku.union.persistence.results.DBRow.<init>(DBRow.java:36)
at com.niku.union.persistence.results.DBResultResponse.createRow(DBResultResponse.java:111)
at com.niku.union.persistence.PersistenceResponse.sliceUsingAnalyticFunctions(PersistenceResponse.java:287)
at com.niku.union.persistence.PersistenceResponse.extractSlicedResultSet(PersistenceResponse.java:326)
at com.niku.union.persistence.PersistenceController.extractResultSet(PersistenceController.java:1802)
at com.niku.union.persistence.PersistenceController.extractResultSets(PersistenceController.java:1856)
at com.niku.union.persistence.PersistenceController.processSql(PersistenceController.java:2596)
at com.niku.union.persistence.PersistenceController.processStatement(PersistenceController.java:851)
at com.niku.union.persistence.PersistenceController.processStatements(PersistenceController.java:751)
at com.niku.union.persistence.PersistenceController.doProcessRequest(PersistenceController.java:559)
at com.niku.union.persistence.PersistenceController.processRequest(PersistenceController.java:289)
at com.niku.union.persistence.nsql.NSQLQuery._execute(NSQLQuery.java:584)
at com.niku.union.persistence.nsql.NSQLQuery.execute(NSQLQuery.java:467)
at com.niku.odf.griphendor.dal.NSQLClient.getCube(NSQLClient.java:132)
at com.niku.odf.griphendor.DefaultGriphendorRequest.getCube(DefaultGriphendorRequest.java:663)
at com.niku.odf.griphendor.grid.GridKernel.processRequest(GridKernel.java:380)
at com.niku.odf.griphendor.GriphendorEngine.processGridRequest(GriphendorEngine.java:133)
at com.niku.odf.griphendor.GriphendorEngine.processRequest(GriphendorEngine.java:82)
at com.niku.npt.griphendor.GridServiceAdapter.processRequest(GridServiceAdapter.java:95)
at com.niku.union.service.binding.ServiceControlBindingJava.processRequest(ServiceControlBindingJava.java:58)
at com.niku.union.service.ServiceControlImpl.processRequest(ServiceControlImpl.java:749)
at com.niku.union.service.ServiceControlImpl.processRequest(ServiceControlImpl.java:211)
at com.niku.union.web.WebServiceAdapter.processRequest(WebServiceAdapter.java:194)
at com.niku.union.web.PortletController.processServiceRequest(PortletController.java:1393)
at com.niku.union.web.PortletController.processAuthorizedPortlet(PortletController.java:876)
ERROR 2012-10-16 14:44:03,142 [http-443-9] performance.PerformanceMonitor (clarity:B84967:5428379__7E82D5DF-117C-460B-A61B-48FB69AF386E:npt.gridExcelExport) Mismatch in timing element stack, where titles do not match ('webRequest' != 'action') or ids do not match ('npt.gridExcelExport' != 'npt.gridExcelExport')
WARN 2012-10-16 14:44:13,152 [http-443-22] security.DefaultSecurityIdentifier (clarity:B84967:5440036__807E4F37-A694-4ABA-8A7C-4CA286B566B2:security.loginAction) This call has no effect, as SQL Tracing is now controlled through logger categories.

My questions are:
- how can I make the App service appear again on my primary server?
- do you think the java error caused it? how can I solve it?

Thank you SO much!
jorgie

Outcomes