AnsweredAssumed Answered

ssg process restarting frequently

Question asked by udareddy on Aug 4, 2017
Latest reply on Aug 4, 2017 by Charles_Lilienkamp

Hi All,

In few nodes the ssg process getting restarted suddenly and it's happening frequently, before and after ssg process getting restart the node load is less and https connections ( ESTABLISHED,CLOSE_WAIT,TIME_WAIT) are less than 100 and memory and disk utilization also less but in logs we can able to see IO errors ( please find below detailed error), is it causing ssg restart ? or what might be the valid reasons behind ssg process restart.?

 

error  logs:

------------------

2017-08-03T21:47:35.725-0400 WARNING 1585 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: size (Stream timeout)
com.l7tech.server.transport.http.TimeoutInputStream$TimeoutIOException: Stream timeout
at com.l7tech.server.transport.http.TimeoutInputStream.d(Unknown Source)
at com.l7tech.server.transport.http.TimeoutInputStream.exitBlocking(Unknown Source)
at com.l7tech.server.transport.http.TimeoutInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.io.PushbackInputStream.read(PushbackInputStream.java:186)
at com.l7tech.common.io.ByteLimitInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at com.l7tech.common.mime.HybridStashManager.stash(Unknown Source)
at com.l7tech.common.mime.h.a(Unknown Source)
at com.l7tech.common.mime.h.stashAndRecall(Unknown Source)
at com.l7tech.common.mime.i.getInputStream(Unknown Source)
at com.l7tech.common.mime.MimeBody.readAndStashEntireMessage(Unknown Source)
at com.l7tech.common.mime.MimeBody.getEntireMessageBodyLength(Unknown Source)
at com.l7tech.message.j.getContentLength(Unknown Source)
at com.l7tech.server.policy.variable.bg.select(Unknown Source)
at com.l7tech.server.policy.variable.MessageSelector.select(Unknown Source)
at com.l7tech.server.policy.variable.MessageSelector.select(Unknown Source)
at com.l7tech.server.policy.variable.ExpandVariables.a(Unknown Source)
at com.l7tech.server.policy.variable.ExpandVariables.a(Unknown Source)
at com.l7tech.server.policy.variable.ExpandVariables.processSingleVariableAsObject(Unknown Source)
at com.l7tech.server.policy.variable.ExpandVariables.processSingleVariableAsObject(Unknown Source)
at com.l7tech.server.policy.variable.we.get(Unknown Source)
at com.l7tech.server.policy.variable.og.a(Unknown Source)
at com.l7tech.server.policy.variable.ServerVariables.get(Unknown Source)
at com.l7tech.server.message.b.getBuiltinVariable(Unknown Source)
at com.l7tech.server.message.b.a(Unknown Source)
at com.l7tech.server.message.b.getVariableMap(Unknown Source)
at com.l7tech.external.assertions.comparison.server.ServerComparisonAssertion.checkRequest(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerOneOrMoreAssertion.checkRequest(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerCompositeAssertion.iterateChildren(Unknown Source)
at com.l7tech.server.policy.assertion.composite.ServerAllAssertion.checkRequest(Unknown Source)
at com.l7tech.server.policy.ServerPolicy.checkRequest(Unknown Source)
at com.l7tech.server.policy.z.call(Unknown Source)
at com.l7tech.server.policy.z.call(Unknown Source)
at com.l7tech.common.log.HybridDiagnosticContext.doInContext(Unknown Source)
at com.l7tech.server.policy.ServerPolicyHandle.checkRequest(Unknown Source)
at com.l7tech.server.mb.b(Unknown Source)
at com.l7tech.server.mb.a(Unknown Source)
at com.l7tech.server.mb.access$700(Unknown Source)
at com.l7tech.server.MessageProcessor.a(Unknown Source)
at com.l7tech.server.MessageProcessor.access$200(Unknown Source)
at com.l7tech.server.kb.call(Unknown Source)
at com.l7tech.server.kb.call(Unknown Source)
at com.l7tech.server.audit.AuditContextFactory.doWithNewAuditContext(Unknown Source)
at com.l7tech.server.MessageProcessor.processMessage(Unknown Source)
at com.l7tech.server.SoapMessageProcessingServlet.service(Unknown Source)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:436)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:342)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:302)
at com.l7tech.server.transport.http.HttpNamespaceFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.WsdlFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.transport.http.ConnectionIdFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.transport.http.InputTimeoutFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.l7tech.server.log.HybridDiagnosticContextServletFilter.doFilter(Unknown Source)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at com.l7tech.server.tomcat.ResponseKillerValve.invoke(Unknown Source)
at com.l7tech.server.tomcat.ConnectionIdValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:295)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:396)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
2017-08-03T21:47:35.726-0400 WARNING 1585 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: size on com.l7tech.message.Message
2017-08-03T21:47:35.726-0400 WARNING 1585 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: request.size
2017-08-03T21:47:35.727-0400 WARNING 1585 com.l7tech.server.SoapMessageProcessingServlet: I/O error while processing message: Stream timeout
2017-08-03T21:47:35.821-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
2017-08-03T21:47:35.821-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
2017-08-03T21:47:35.822-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
2017-08-03T21:47:35.822-0400 WARNING 1702 com.l7tech.message.JsonFacet: Did not need to create json data
2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: now for date time. Illegal pattern character 'n'
2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: gateway.time.now
2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: now for date time. Illegal pattern character 'n'
2017-08-03T21:47:35.824-0400 WARNING 1702 com.l7tech.server.policy.variable.ServerVariables: Unsupported variable: gateway.time.now

 

gateway version is 8.1.0

 

Thanks in advance for your inputs and help.

 

--Uday

Outcomes