CA Service Management

Expand all | Collapse all

CA servicedesk Tomcat issue

  • 1.  CA servicedesk Tomcat issue

    Posted Dec 21, 2016 02:40 AM

    Dears, I am facing issue with Tomcat. It is getting down frequently. I can see following error messages before it is getting down. Any idea on this issue?

     

    "ERROR] CommonsHTTPTransportSender - null <org.apache.axis2.AxisFault>org.apache.axis2.AxisFault
    at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
    at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
    at org.apache.axis2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:364)
    at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:241)
    at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:443)
    at org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:43)
    at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:114)
    at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:181)
    at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
    at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)
    Caused by: com.ctc.wstx.exc.WstxIOException: null
    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)
    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
    at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:222)
    at org.apache.axiom.om.impl.llom.OMSerializableImpl.serializeAndConsume(OMSerializableImpl.java:192)
    at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
    ... 30 more
    Caused by: ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:367)
    at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331)
    at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
    at java.io.BufferedOutputStream.flush(Unknown Source)
    at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
    at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
    at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1690)
    ... 35 more
    Caused by: java.net.SocketException: Software caused connection abort: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(Unknown Source)
    at java.net.SocketOutputStream.write(Unknown Source)
    at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
    at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
    at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:800)
    at org.apache.coyote.Response.action(Response.java:172)
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363)
    ... 42 more

    [ERROR] CommonsHTTPTransportSender - null <org.apache.axis2.AxisFault>org.apache.axis2.AxisFault
    at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
    at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
    at org.apache.axis2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:364)
    at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:241)
    at org.apache.axis2.engine.AxisEngine.sendFault(AxisEngine.java:526)
    at org.apache.axis2.transport.http.AxisServlet.handleFault(AxisServlet.java:433)
    at org.apache.axis2.transport.http.AxisServlet.processAxisFault(AxisServlet.java:398)
    at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:188)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)
    Caused by: com.ctc.wstx.exc.WstxIOException: null
    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)
    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
    at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:222)
    at org.apache.axiom.om.impl.llom.OMSerializableImpl.serializeAndConsume(OMSerializableImpl.java:192)
    at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
    ... 28 more
    Caused by: ClientAbortException: java.net.SocketException: Software caused connection abort: socket write error
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:367)
    at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:331)
    at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
    at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
    at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
    at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1690)
    ... 33 more
    Caused by: java.net.SocketException: Software caused connection abort: socket write error
    at java.net.SocketOutputStream.socketWrite0(Native Method)
    at java.net.SocketOutputStream.socketWrite(Unknown Source)
    at java.net.SocketOutputStream.write(Unknown Source)
    at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
    at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
    at org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:119)
    at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:800)
    at org.apache.coyote.Response.action(Response.java:172)
    at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:363)
    ... 39 more

    [INFO] AxisServlet - org.apache.axis2.AxisFault"



  • 2.  Re: CA servicedesk Tomcat issue

    Posted Dec 21, 2016 04:33 AM

    Just to add in to this, since this issue occurred i am getting this files  "hs_err_pid2392.mdmp", "hs_err_pid2392.log" files  in the "D:\Program Files (x86)\CA\Service Desk Manager\bin" directory with following messages in log files.

     


    # A fatal error has been detected by the Java Runtime Environment:
    #
    # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x71342dac, pid=2392, tid=7292
    #
    # JRE version: 7.0_10-b18
    # Java VM: Java HotSpot(TM) Client VM (23.6-b04 mixed mode windows-x86 )
    # Problematic frame:
    # C [sda65.dll+0x142dac] Java_usdjws_usdsda_onIdle+0xe764c
    #
    # Core dump written. Default location: [thread 7580 also had an error]
    #
    # If you would like to submit a bug report, please visit:
    # http://bugreport.sun.com/bugreport/crash.jsp
    # The crash happened outside the Java Virtual Machine in native code.
    # See problematic frame for where to report the bug.
    #

    --------------- T H R E A D ---------------

    Current thread (0x476bb400): JavaThread "http-bio-8080-exec-12" daemon [_thread_in_native, id=7292, stack(0x4b160000,0x4b1b0000)]

    siginfo: ExceptionCode=0xc0000005, reading address 0xdddddf0d

    Registers:
    EAX=0xdddddf09, EBX=0x4459f2d0, ECX=0xdddddf09, EDX=0x00000004
    ESP=0x4b1adebc, EBP=0x4b1adec4, ESI=0x4459f2d0, EDI=0x476bb400
    EIP=0x71342dac, EFLAGS=0x00010296

    Top of Stack: (sp=0x4b1adebc)
    0x4b1adebc: ffffffff dddddf09 4b1aded4 7122c857
    0x4b1adecc: dddddddd 00000000 4b1adefc 7122dab1
    0x4b1adedc: 0000001e 0000000a 035a30f0 035a31b8
    0x4b1adeec: 477ae1b6 4b1af5dc 7135c455 dddddddd
    0x4b1adefc: 4b1adf18 7120d18f 00000004 4b1adf10
    0x4b1adf0c: 4b98d520 00000000 dddddddd 4b1ae214
    0x4b1adf1c: 7121c745 4f27df85 00000004 4b1ae088
    0x4b1adf2c: 014d5300 035a31b8 00000005 035a31d0

    Instructions: (pc=0x71342dac)
    0x71342d8c: 7c 11 37 71 8b e5 5d c3 cc cc cc cc cc cc cc cc
    0x71342d9c: cc cc cc cc 55 8b ec 51 89 4d fc 6a ff 8b 45 fc
    0x71342dac: 8b 48 04 51 ff 15 54 12 37 71 8b 55 fc c6 42 08
    0x71342dbc: 01 8b e5 5d c3 cc cc cc cc cc cc cc cc cc cc cc


    Register to memory mapping:

    EAX=0xdddddf09 is an unknown value
    EBX=0x4459f2d0 is an oop
    {method}
    - klass: {other class}
    ECX=0xdddddf09 is an unknown value
    EDX=0x00000004 is an unknown value
    ESP=0x4b1adebc is pointing into the stack for thread: 0x476bb400
    EBP=0x4b1adec4 is pointing into the stack for thread: 0x476bb400
    ESI=0x4459f2d0 is an oop
    {method}
    - klass: {other class}
    EDI=0x476bb400 is a thread

     

    Regards,

    Sushant 



  • 3.  Re: CA servicedesk Tomcat issue

    Broadcom Employee
    Posted Dec 21, 2016 01:37 PM

    It looks like the Java process tied to Tomcat is crashing for some reason.

     

    Is this server heavily used in respect to Tomcat functions (i.e. Export, Attachments, Web Services)?

     

    What are the hardware specs on the server?

     

    Has Java been recently updated by chance



  • 4.  Re: CA servicedesk Tomcat issue

    Posted Dec 22, 2016 12:35 AM

    Hello,

     

    We are having integrations(webservices) from other environment which

    interchange activities to open & track the ticket status.This is working

    since long time however we are getting this issue recently.

     

    Hardware specification is:-

     

    Processor:-Intel-xeon CPU E5-2680 @ 2.70GHZ

    RAM:-24 GB

    System type:-64-bit operating system

     

    JAVA version:-java version "1.7.0_25"

    Webengines:-4 webengines

     

    We have already raised support ticket with CA on this but unfortunately we

    didn't got any solution for this. We have moved attachment servlet path

    from this server to primary server to decrease the load still this server

    is having tomcat issue.

     

     

     

     

     

     

    Regards,

    Sushant Bansode

    Givaudan GSD Support

     

    On Wed, Dec 21, 2016 at 7:39 PM, Paul_Coccimiglio <



  • 5.  Re: CA servicedesk Tomcat issue

    Broadcom Employee
    Posted Dec 22, 2016 02:10 AM

    I am assuming this is 14.1.  Either way would you be able to clarify the release + patch of SDM in question here ?

     

    I hoped to see couple of lines more in the hspid log file you posted. But looks like it only printed one line. Usually that stack dump gives good clues for us.

     

    _R



  • 6.  Re: CA servicedesk Tomcat issue

    Posted Dec 22, 2016 02:22 AM

    Hello,

     

    CA SD Version is 12.9

    Following are the patches applied :-

    T5U3257 (USRD)

    T57I015 (USRD)

    T51R087 (USRD)

    T56H024 (USRD)

    RO72246 (USRD)

    RO72249 (USRD)

    RO72247 (USRD)

    RO72248

    RO72250

    RO72251

    RO72252

    RO72254

    RO72256

    RO72257

    RO72258

    T6D3031 (USRD)

    T56H103 (USRD)



  • 7.  Re: CA servicedesk Tomcat issue

    Posted Dec 22, 2016 03:28 AM
    hs_err_pid8068.log description:-
    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6c1b2dac, pid=8068, tid=4300
    #
    # JRE version: 7.0_10-b18
    # Java VM: Java HotSpot(TM) Client VM (23.6-b04 mixed mode windows-x86 )
    # Problematic frame:
    # C  [sda65.dll+0x142dac]  Java_usdjws_usdsda_onIdle+0xe764c
    #
    # Core dump written. Default location: [thread 7468 also had an error]
    #
    # If you would like to submit a bug report, please visit:
    # The crash happened outside the Java Virtual Machine in native code.
    # See problematic frame for where to report the bug.
    #
    ---------------  T H R E A D  ---------------
    Current thread (0x49aef000):  JavaThread "http-bio-8080-exec-10" daemon [_thread_in_native, id=4300, stack(0x4bdc0000,0x4be10000)]
    siginfo: ExceptionCode=0xc0000005, reading address 0xdddddf0d
    Registers:
    EAX=0xdddddf09, EBX=0x4425f820, ECX=0xdddddf09, EDX=0x00000006
    ESP=0x4be0daec, EBP=0x4be0daf4, ESI=0x4425f820, EDI=0x49aef000
    EIP=0x6c1b2dac, EFLAGS=0x00010296
    Top of Stack: (sp=0x4be0daec)
    0x4be0daec:   ffffffff dddddf09 4be0db04 6c09c857
    0x4be0dafc:   dddddddd 43ff7d40 4be0db2c 6c09dab1
    0x4be0db0c:   49aef000 4be0db30 4be0db2c 4be0db48
    0x4be0db1c:   e3a89851 4be0f51c 6c1cc455 dddddddd
    0x4be0db2c:   4be0db48 6c07d18f 00000006 4be0db40
    0x4be0db3c:   4bf033c0 00000000 dddddddd 4be0e128
    0x4be0db4c:   6c0966a5 06f2d556 00000006 4be0dcd8
    0x4be0db5c:   49aef000 4be0e198 49aef000 49aef000 
    Instructions: (pc=0x6c1b2dac)
    0x6c1b2d8c:   7c 11 1e 6c 8b e5 5d c3 cc cc cc cc cc cc cc cc
    0x6c1b2d9c:   cc cc cc cc 55 8b ec 51 89 4d fc 6a ff 8b 45 fc
    0x6c1b2dac:   8b 48 04 51 ff 15 54 12 1e 6c 8b 55 fc c6 42 08
    0x6c1b2dbc:   01 8b e5 5d c3 cc cc cc cc cc cc cc cc cc cc cc 
    Register to memory mapping:
    EAX=0xdddddf09 is an unknown value
    EBX=0x4425f820 is an oop
    {method} 
     - klass: {other class}
    ECX=0xdddddf09 is an unknown value
    EDX=0x00000006 is an unknown value
    ESP=0x4be0daec is pointing into the stack for thread: 0x49aef000
    EBP=0x4be0daf4 is pointing into the stack for thread: 0x49aef000
    ESI=0x4425f820 is an oop
    {method} 
     - klass: {other class}
    EDI=0x49aef000 is a thread
    Stack: [0x4bdc0000,0x4be10000],  sp=0x4be0daec,  free space=310k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    C  [sda65.dll+0x142dac]  Java_usdjws_usdsda_onIdle+0xe764c
    C  [sda65.dll+0x2c857]  CallSpellMethod+0x1e07
    C  [sda65.dll+0x2dab1]  CallSpellMethod+0x3061
    C  [sda65.dll+0xd18f]
    C  [sda65.dll+0x266a5]  DoSelect+0x25
    C  [sda65.dll+0x5592b]  Java_usdjws65_usdsda_doSelect+0x6b
    j  usdjws65.usdsda.doSelect(ILjava/lang/String;Ljava/lang/String;I[Ljava/lang/String;ILjavax/xml/rpc/holders/IntHolder;)I+0
    j  usdjws65.USDImpl.doSelect(ILjava/lang/String;Ljava/lang/String;ILusdjws65/ArrayOfString;)Ljava/lang/String;+200
    j  usdjws65.USD_WebServiceSoapImpl.doSelect(ILjava/lang/String;Ljava/lang/String;ILusdjws65/ArrayOfString;)Ljava/lang/String;+11
    j  usdjws65.USD_WebServiceSoapSkeleton.doSelect(ILjava/lang/String;Ljava/lang/String;ILusdjws65/ArrayOfString;)Ljava/lang/String;+11
    j  sun.reflect.GeneratedMethodAccessor29.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+220
    J  sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
    j  org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4
    j  java.lang.Thread.run()V+11
    v  ~StubRoutines::call_stub
    V  [jvm.dll+0x12a3ba]
    V  [jvm.dll+0x1d97de]
    V  [jvm.dll+0x12a5a3]
    V  [jvm.dll+0x12a607]
    V  [jvm.dll+0xd315f]
    V  [jvm.dll+0x14a6b7]
    V  [jvm.dll+0x14a820]
    V  [jvm.dll+0x17f009]
    C  [msvcr100.dll+0x10fac]  freefls+0x130
    C  [msvcr100.dll+0x110b1]  __fls_getvalue+0x58
    C  [kernel32.dll+0x1336a]  BaseThreadInitThunk+0x12
    C  [ntdll.dll+0x39902]  RtlInitializeExceptionChain+0x63
    C  [ntdll.dll+0x398d5]  RtlInitializeExceptionChain+0x36
    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    j  usdjws65.usdsda.doSelect(ILjava/lang/String;Ljava/lang/String;I[Ljava/lang/String;ILjavax/xml/rpc/holders/IntHolder;)I+0
    j  usdjws65.USDImpl.doSelect(ILjava/lang/String;Ljava/lang/String;ILusdjws65/ArrayOfString;)Ljava/lang/String;+200
    j  usdjws65.USD_WebServiceSoapImpl.doSelect(ILjava/lang/String;Ljava/lang/String;ILusdjws65/ArrayOfString;)Ljava/lang/String;+11
    j  usdjws65.USD_WebServiceSoapSkeleton.doSelect(ILjava/lang/String;Ljava/lang/String;ILusdjws65/ArrayOfString;)Ljava/lang/String;+11
    j  sun.reflect.GeneratedMethodAccessor29.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;+220
    J  sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
    J  java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
    J  org.apache.axis.providers.java.RPCProvider.processMessage(Lorg/apache/axis/MessageContext;Lorg/apache/axis/message/SOAPEnvelope;Lorg/apache/axis/message/SOAPEnvelope;Ljava/lang/Object;)V
    J  org.apache.axis.providers.java.JavaProvider.invoke(Lorg/apache/axis/MessageContext;)V
    J  org.apache.axis.strategies.InvocationStrategy.visit(Lorg/apache/axis/Handler;Lorg/apache/axis/MessageContext;)V
    J  org.apache.axis.SimpleChain.doVisiting(Lorg/apache/axis/MessageContext;Lorg/apache/axis/HandlerIterationStrategy;)V
    J  org.apache.axis.SimpleChain.invoke(Lorg/apache/axis/MessageContext;)V
    J  org.apache.axis.handlers.soap.SOAPService.invoke(Lorg/apache/axis/MessageContext;)V
    J  org.apache.axis.server.AxisServer.invoke(Lorg/apache/axis/MessageContext;)V
    J  org.apache.axis.transport.http.AxisServlet.doPost(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
    J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
    J  org.apache.axis.transport.http.AxisServletBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V
    J  javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
    J  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
    J  org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
    J  org.apache.tomcat.websocket.server.WsFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V
    J  org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
    J  org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V
    J  org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
    J  org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
    J  org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
    J  org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
    J  org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
    J  org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
    J  org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V
    J  org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V
    J  org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
    J  org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState;
    J  org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run()V
    J  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
    j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
    j  org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4
    j  java.lang.Thread.run()V+11
    v  ~StubRoutines::call_stub
    ---------------  P R O C E S S  ---------------
    Java Threads: ( => current thread )
      0x48d06400 JavaThread "http-bio-8080-exec-75" daemon [_thread_in_native, id=7240, stack(0x4dea0000,0x4def0000)]
      0x48c46800 JavaThread "http-bio-8080-exec-74" daemon [_thread_in_native, id=4036, stack(0x4e700000,0x4e750000)]
      0x48c46400 JavaThread "http-bio-8080-exec-73" daemon [_thread_in_native, id=5948, stack(0x4e610000,0x4e660000)]
      0x48c45c00 JavaThread "http-bio-8080-exec-72" daemon [_thread_in_native, id=4864, stack(0x4e330000,0x4e380000)]
      0x48c45800 JavaThread "http-bio-8080-exec-71" daemon [_thread_blocked, id=6924, stack(0x4e470000,0x4e4c0000)]
      0x48c45000 JavaThread "http-bio-8080-exec-70" daemon [_thread_in_native, id=2584, stack(0x4e140000,0x4e190000)]
      0x48c44800 JavaThread "http-bio-8080-exec-69" daemon [_thread_in_native, id=7516, stack(0x4e0f0000,0x4e140000)]
      0x48c44400 JavaThread "http-bio-8080-exec-68" daemon [_thread_in_native, id=7944, stack(0x4df70000,0x4dfc0000)]
      0x48c43c00 JavaThread "http-bio-8080-exec-66" daemon [_thread_in_native, id=6132, stack(0x4e190000,0x4e1e0000)]
      0x48c43800 JavaThread "http-bio-8080-exec-64" daemon [_thread_blocked, id=6228, stack(0x4e0a0000,0x4e0f0000)]
      0x48c43000 JavaThread "http-bio-8080-exec-63" daemon [_thread_in_native, id=6300, stack(0x4de10000,0x4de60000)]
      0x49c14800 JavaThread "http-bio-8080-exec-61" daemon [_thread_in_native, id=2744, stack(0x4daf0000,0x4db40000)]
      0x49c15c00 JavaThread "http-bio-8080-exec-60" daemon [_thread_in_native, id=7968, stack(0x4dfc0000,0x4e010000)]
      0x49c15400 JavaThread "http-bio-8080-exec-59" daemon [_thread_in_native, id=5688, stack(0x4e020000,0x4e070000)]
      0x49c15000 JavaThread "http-bio-8080-exec-58" daemon [_thread_in_native, id=6676, stack(0x4df20000,0x4df70000)]
      0x49c14400 JavaThread "http-bio-8080-exec-56" daemon [_thread_in_native, id=6148, stack(0x4dc20000,0x4dc70000)]
      0x49c13c00 JavaThread "http-bio-8080-exec-55" daemon [_thread_in_native, id=5540, stack(0x4d9d0000,0x4da20000)]
      0x49c13400 JavaThread "http-bio-8080-exec-54" daemon [_thread_in_native, id=5812, stack(0x4dbd0000,0x4dc20000)]
      0x49c13000 JavaThread "http-bio-8080-exec-53" daemon [_thread_in_native, id=8160, stack(0x4d760000,0x4d7b0000)]
      0x49c12800 JavaThread "http-bio-8080-exec-52" daemon [_thread_in_native, id=7864, stack(0x4d600000,0x4d650000)]
      0x48d06c00 JavaThread "http-bio-8080-exec-51" daemon [_thread_in_native, id=3184, stack(0x4daa0000,0x4daf0000)]
      0x48d07800 JavaThread "http-bio-8080-exec-50" daemon [_thread_in_native, id=7468, stack(0x4d820000,0x4d870000)]
      0x48bb8800 JavaThread "http-bio-8080-exec-49" daemon [_thread_blocked, id=3564, stack(0x4d880000,0x4d8d0000)]
      0x48bb8000 JavaThread "http-bio-8080-exec-48" daemon [_thread_in_native, id=6288, stack(0x4d540000,0x4d590000)]
      0x48bb7c00 JavaThread "http-bio-8080-exec-47" daemon [_thread_in_native, id=6644, stack(0x4d6a0000,0x4d6f0000)]
      0x48bb7400 JavaThread "http-bio-8080-exec-46" daemon [_thread_in_native, id=3520, stack(0x4cea0000,0x4cef0000)]
      0x48bb6c00 JavaThread "http-bio-8080-exec-45" daemon [_thread_in_native, id=6856, stack(0x496d0000,0x49720000)]
      0x48bb6000 JavaThread "http-bio-8080-exec-44" daemon [_thread_in_native, id=8080, stack(0x4d0b0000,0x4d100000)]
      0x48bb6800 JavaThread "http-bio-8080-exec-43" daemon [_thread_blocked, id=7940, stack(0x4cbb0000,0x4cc00000)]
      0x49af6000 JavaThread "http-bio-8080-exec-42" daemon [_thread_in_native, id=5532, stack(0x4d2e0000,0x4d330000)]
      0x48bb5c00 JavaThread "http-bio-8080-exec-40" daemon [_thread_in_native, id=7852, stack(0x4d350000,0x4d3a0000)]
      0x48bb5400 JavaThread "http-bio-8080-exec-39" daemon [_thread_blocked, id=5076, stack(0x4d240000,0x4d290000)]
      0x48bb5000 JavaThread "http-bio-8080-exec-37" daemon [_thread_blocked, id=5372, stack(0x4d1b0000,0x4d200000)]
      0x49af5c00 JavaThread "http-bio-8080-exec-33" daemon [_thread_in_native, id=5884, stack(0x4cfb0000,0x4d000000)]
      0x49af5000 JavaThread "http-bio-8080-exec-32" daemon [_thread_in_native, id=2568, stack(0x4c280000,0x4c2d0000)]
      0x49af5400 JavaThread "http-bio-8080-exec-31" daemon [_thread_blocked, id=5216, stack(0x4ccd0000,0x4cd20000)]
      0x49af3c00 JavaThread "http-bio-8080-exec-30" daemon [_thread_in_native, id=7192, stack(0x4cc30000,0x4cc80000)]
      0x49af4800 JavaThread "http-bio-8080-exec-28" daemon [_thread_blocked, id=6580, stack(0x4cb60000,0x4cbb0000)]
      0x49af4000 JavaThread "http-bio-8080-exec-26" daemon [_thread_blocked, id=4480, stack(0x4cd20000,0x4cd70000)]
      0x49af3000 JavaThread "http-bio-8080-exec-25" daemon [_thread_blocked, id=5612, stack(0x482e0000,0x48330000)]
      0x49af3400 JavaThread "http-bio-8080-exec-23" daemon [_thread_blocked, id=7560, stack(0x4cda0000,0x4cdf0000)]
      0x49af1000 JavaThread "http-bio-8080-exec-22" daemon [_thread_blocked, id=7096, stack(0x4c890000,0x4c8e0000)]
      0x49af2400 JavaThread "http-bio-8080-exec-20" daemon [_thread_blocked, id=8028, stack(0x4ae10000,0x4ae60000)]
      0x49af2800 JavaThread "http-bio-8080-exec-18" daemon [_thread_blocked, id=6140, stack(0x4c980000,0x4c9d0000)]
      0x49af1400 JavaThread "http-bio-8080-exec-15" daemon [_thread_blocked, id=7728, stack(0x4ca50000,0x4caa0000)]
      0x49af1c00 JavaThread "http-bio-8080-exec-14" daemon [_thread_blocked, id=7708, stack(0x4c9f0000,0x4ca40000)]
      0x49af0800 JavaThread "TCP_port-Write:web:gvecsm0140ap.ds.givaudan.com:3" daemon [_thread_blocked, id=6540, stack(0x4c7f0000,0x4c840000)]
      0x49af0400 JavaThread "TCP_port-Write:web:gvecsm0140ap.ds.givaudan.com:4" daemon [_thread_blocked, id=7564, stack(0x4c230000,0x4c280000)]
      0x49aefc00 JavaThread "TCP_port-Write:web:gvecsm0140ap.ds.givaudan.com:2" daemon [_thread_blocked, id=6768, stack(0x4c760000,0x4c7b0000)]
      0x49aef800 JavaThread "TCP_port-Write:web:wsp" daemon [_thread_blocked, id=6524, stack(0x4c0f0000,0x4c140000)]
    =>0x49aef000 JavaThread "http-bio-8080-exec-10" daemon [_thread_in_native, id=4300, stack(0x4bdc0000,0x4be10000)]
      0x49aee800 JavaThread "http-bio-8080-exec-9" daemon [_thread_blocked, id=2776, stack(0x4b7b0000,0x4b800000)]
      0x48d08400 JavaThread "TCP_port-Write:web:local" daemon [_thread_blocked, id=7536, stack(0x4b880000,0x4b8d0000)]
      0x48d0b800 JavaThread "TCP_port-Read:web:gvecsm0140ap.ds.givaudan.com:3" daemon [_thread_in_native, id=4920, stack(0x4b720000,0x4b770000)]
      0x48d0b000 JavaThread "TCP_port-Read:web:local" daemon [_thread_in_native, id=7500, stack(0x4b810000,0x4b860000)]
      0x48d0ac00 JavaThread "TCP_port-Read:web:gvecsm0140ap.ds.givaudan.com:4" daemon [_thread_in_native, id=2104, stack(0x4b660000,0x4b6b0000)]
      0x48d0a400 JavaThread "TCP_port-Read:web:gvecsm0140ap.ds.givaudan.com:2" daemon [_thread_in_native, id=1916, stack(0x4b610000,0x4b660000)]
      0x48d0a000 JavaThread "TCP_port-Read:web:wsp" daemon [_thread_in_native, id=5628, stack(0x4b490000,0x4b4e0000)]
      0x48d09800 JavaThread "PortHeartBeat" daemon [_thread_blocked, id=2436, stack(0x4b050000,0x4b0a0000)]
      0x48d09000 JavaThread "TCP_port-Read:Slump_nxd" daemon [_thread_in_native, id=5708, stack(0x4b2e0000,0x4b330000)]
      0x48d08c00 JavaThread "TCP_port-Write:Slump_nxd" daemon [_thread_blocked, id=5348, stack(0x4af20000,0x4af70000)]
      0x48d08000 JavaThread "http-bio-8080-exec-8" daemon [_thread_in_native, id=7828, stack(0x4b0f0000,0x4b140000)]
      0x48d07400 JavaThread "http-bio-8080-exec-6" daemon [_thread_blocked, id=4588, stack(0x4aeb0000,0x4af00000)]
      0x48d06000 JavaThread "http-bio-8080-exec-3" daemon [_thread_blocked, id=3360, stack(0x49870000,0x498c0000)]
      0x48d05800 JavaThread "http-bio-8080-exec-2" daemon [_thread_in_native, id=4732, stack(0x497d0000,0x49820000)]
      0x48d05400 JavaThread "http-bio-8080-exec-1" daemon [_thread_in_native, id=7812, stack(0x49820000,0x49870000)]
      0x48d04c00 JavaThread "http-bio-8080-AsyncTimeout" daemon [_thread_blocked, id=5412, stack(0x49270000,0x492c0000)]
      0x48d04800 JavaThread "http-bio-8080-Acceptor-0" daemon [_thread_in_native, id=6724, stack(0x47970000,0x479c0000)]
      0x48d04000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=2736, stack(0x49730000,0x49780000)]
      0x48ccd400 JavaThread "PortHeartBeat" daemon [_thread_blocked, id=7100, stack(0x49550000,0x495a0000)]
      0x48c1c000 JavaThread "TCP_port-Read:Slump_nxd" daemon [_thread_in_native, id=7316, stack(0x495e0000,0x49630000)]
      0x486e9000 JavaThread "TCP_port-Write:Slump_nxd" daemon [_thread_blocked, id=7540, stack(0x49450000,0x494a0000)]
      0x486e8800 JavaThread "Thread-6" daemon [_thread_blocked, id=6500, stack(0x494a0000,0x494f0000)]
      0x48afa000 JavaThread "Thread-5" daemon [_thread_blocked, id=6564, stack(0x00a70000,0x00ac0000)]
      0x48717400 JavaThread "Thread-4" daemon [_thread_in_native, id=5208, stack(0x49370000,0x493c0000)]
      0x48a09c00 JavaThread "Timer-1" daemon [_thread_blocked, id=4892, stack(0x49190000,0x491e0000)]
      0x4801ec00 JavaThread "Timer-0" daemon [_thread_blocked, id=328, stack(0x484e0000,0x48530000)]
      0x48096400 JavaThread "GC Daemon" daemon [_thread_blocked, id=3980, stack(0x48360000,0x483b0000)]
      0x4756dc00 JavaThread "Service Thread" daemon [_thread_blocked, id=7892, stack(0x47b10000,0x47b60000)]
      0x47566c00 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=1572, stack(0x47a70000,0x47ac0000)]
      0x47565800 JavaThread "Attach Listener" daemon [_thread_blocked, id=4380, stack(0x478d0000,0x47920000)]
      0x47501000 JavaThread "Finalizer" daemon [_thread_blocked, id=2836, stack(0x47800000,0x47850000)]
      0x474f9c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=6928, stack(0x476f0000,0x47740000)]
      0x00f3d800 JavaThread "main" [_thread_in_native, id=3092, stack(0x010a0000,0x010f0000)]
    Other Threads:
      0x474f4800 VMThread [stack: 0x477b0000,0x47800000] [id=3780]
      0x47582c00 WatcherThread [stack: 0x47bc0000,0x47c10000] [id=6372]
    VM state:not at safepoint (normal execution)
    VM Mutex/Monitor currently owned by a thread: None
    Heap
     def new generation   total 40000K, used 12460K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,  35% used [0x03140000, 0x03d6b0f8, 0x05400000)
      from space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
      to   space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
     tenured generation   total 88796K, used 31700K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  35% used [0x18690000, 0x1a5852e8, 0x1a585400, 0x1dd47000)
     compacting perm gen  total 21248K, used 21126K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445e1b78, 0x445e1c00, 0x44600000)
    No shared spaces configured.
    Card table byte_map: [0x47140000,0x47370000] byte_map_base: 0x47127600
    Polling page: 0x00180000
    Code Cache  [0x01140000, 0x01938000, 0x03140000)
     total_blobs=3693 nmethods=3520 adapters=106 free_code_cache=24627Kb largest_free_block=25212480
    Compilation events (10 events):
    Event: 10948.547 Thread 0x47566c00 3569             org.apache.axis.wsdl.fromJava.Emitter::createMessageName (69 bytes)
    Event: 10948.548 Thread 0x47566c00 nmethod 3569 0x01931148 code [0x019312e0, 0x01931ba4]
    Event: 10948.548 Thread 0x47566c00 3570             com.ibm.wsdl.DefinitionImpl::getMessage (35 bytes)
    Event: 10948.549 Thread 0x47566c00 nmethod 3570 0x019321c8 code [0x019322f0, 0x01932468]
    Event: 10948.549 Thread 0x47566c00 3571             org.apache.axis.wsdl.fromJava.Emitter::getResponseQName (80 bytes)
    Event: 10948.550 Thread 0x47566c00 nmethod 3571 0x01932548 code [0x01932700, 0x0193345c]
    Event: 10948.550 Thread 0x47566c00 3572             org.apache.axis.wsdl.fromJava.Types::writeWrapperElement (107 bytes)
    Event: 10948.550 Thread 0x47566c00 nmethod 3572 0x01933ac8 code [0x01933c50, 0x01933e3c]
    Event: 10948.550 Thread 0x47566c00 3573             org.apache.axis.wsdl.fromJava.Types::writeSchemaElementDecl (55 bytes)
    Event: 10948.551 Thread 0x47566c00 nmethod 3573 0x01934008 code [0x01934160, 0x01934408]
    GC Heap History (10 events):
    Event: 9601.565 GC heap before
    {Heap before GC invocations=1216 (full 1136):
     def new generation   total 40000K, used 8701K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,  24% used [0x03140000, 0x039bf418, 0x05400000)
      from space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
      to   space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
     tenured generation   total 88796K, used 26036K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  29% used [0x18690000, 0x19ffd0d0, 0x19ffd200, 0x1dd47000)
     compacting perm gen  total 21248K, used 21082K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d6bc8, 0x445d6c00, 0x44600000)
    No shared spaces configured.
    Event: 9601.718 GC heap after
    Heap after GC invocations=1217 (full 1137):
     def new generation   total 40000K, used 0K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,   0% used [0x03140000, 0x03140000, 0x05400000)
      from space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
      to   space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
     tenured generation   total 88796K, used 27083K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  30% used [0x18690000, 0x1a102c88, 0x1a102e00, 0x1dd47000)
     compacting perm gen  total 21248K, used 21082K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d6bc8, 0x445d6c00, 0x44600000)
    No shared spaces configured.
    }
    Event: 9609.094 GC heap before
    {Heap before GC invocations=1217 (full 1137):
     def new generation   total 40000K, used 7856K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,  22% used [0x03140000, 0x038ec330, 0x05400000)
      from space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
      to   space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
     tenured generation   total 88796K, used 27083K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  30% used [0x18690000, 0x1a102c88, 0x1a102e00, 0x1dd47000)
     compacting perm gen  total 21248K, used 21082K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d6bc8, 0x445d6c00, 0x44600000)
    No shared spaces configured.
    Event: 9609.179 GC heap after
    Heap after GC invocations=1218 (full 1138):
     def new generation   total 40000K, used 0K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,   0% used [0x03140000, 0x03140000, 0x05400000)
      from space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
      to   space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
     tenured generation   total 88796K, used 28073K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  31% used [0x18690000, 0x1a1fa688, 0x1a1fa800, 0x1dd47000)
     compacting perm gen  total 21248K, used 21082K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d6bc8, 0x445d6c00, 0x44600000)
    No shared spaces configured.
    }
    Event: 10173.476 GC heap before
    {Heap before GC invocations=1218 (full 1138):
     def new generation   total 40000K, used 35584K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K, 100% used [0x03140000, 0x05400000, 0x05400000)
      from space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
      to   space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
     tenured generation   total 88796K, used 28073K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  31% used [0x18690000, 0x1a1fa688, 0x1a1fa800, 0x1dd47000)
     compacting perm gen  total 21248K, used 21082K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d6be8, 0x445d6c00, 0x44600000)
    No shared spaces configured.
    Event: 10173.491 GC heap after
    Heap after GC invocations=1219 (full 1138):
     def new generation   total 40000K, used 4416K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,   0% used [0x03140000, 0x03140000, 0x05400000)
      from space 4416K, 100% used [0x05850000, 0x05ca0000, 0x05ca0000)
      to   space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
     tenured generation   total 88796K, used 29779K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  33% used [0x18690000, 0x1a3a4eb0, 0x1a3a5000, 0x1dd47000)
     compacting perm gen  total 21248K, used 21082K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d6be8, 0x445d6c00, 0x44600000)
    No shared spaces configured.
    }
    Event: 10466.882 GC heap before
    {Heap before GC invocations=1219 (full 1138):
     def new generation   total 40000K, used 21283K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,  47% used [0x03140000, 0x041b8d10, 0x05400000)
      from space 4416K, 100% used [0x05850000, 0x05ca0000, 0x05ca0000)
      to   space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
     tenured generation   total 88796K, used 29779K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  33% used [0x18690000, 0x1a3a4eb0, 0x1a3a5000, 0x1dd47000)
     compacting perm gen  total 21248K, used 21086K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d7a88, 0x445d7c00, 0x44600000)
    No shared spaces configured.
    Event: 10467.013 GC heap after
    Heap after GC invocations=1220 (full 1139):
     def new generation   total 40000K, used 0K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,   0% used [0x03140000, 0x03140000, 0x05400000)
      from space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
      to   space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
     tenured generation   total 88796K, used 33101K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  37% used [0x18690000, 0x1a6e3618, 0x1a6e3800, 0x1dd47000)
     compacting perm gen  total 21248K, used 21086K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d7a88, 0x445d7c00, 0x44600000)
    No shared spaces configured.
    }
    Event: 10817.556 GC heap before
    {Heap before GC invocations=1220 (full 1139):
     def new generation   total 40000K, used 23614K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,  66% used [0x03140000, 0x0484fb28, 0x05400000)
      from space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
      to   space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
     tenured generation   total 88796K, used 33101K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  37% used [0x18690000, 0x1a6e3618, 0x1a6e3800, 0x1dd47000)
     compacting perm gen  total 21248K, used 21092K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d90c0, 0x445d9200, 0x44600000)
    No shared spaces configured.
    Event: 10817.730 GC heap after
    Heap after GC invocations=1221 (full 1140):
     def new generation   total 40000K, used 0K [0x03140000, 0x05ca0000, 0x18690000)
      eden space 35584K,   0% used [0x03140000, 0x03140000, 0x05400000)
      from space 4416K,   0% used [0x05850000, 0x05850000, 0x05ca0000)
      to   space 4416K,   0% used [0x05400000, 0x05400000, 0x05850000)
     tenured generation   total 88796K, used 31700K [0x18690000, 0x1dd47000, 0x43140000)
       the space 88796K,  35% used [0x18690000, 0x1a5852e8, 0x1a585400, 0x1dd47000)
     compacting perm gen  total 21248K, used 21092K [0x43140000, 0x44600000, 0x47140000)
       the space 21248K,  99% used [0x43140000, 0x445d9070, 0x445d9200, 0x44600000)
    No shared spaces configured.
    }
    Deoptimization events (0 events):
    No events
    Internal exceptions (10 events):
    Event: 10959.942 Thread 0x48d06400 Threw 0x03af39c8 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jvm.cpp:1166
    Event: 10959.945 Thread 0x48d06400 Threw 0x03af9bd0 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jvm.cpp:1166
    Event: 10959.947 Thread 0x48d06400 Threw 0x03aff5f0 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jvm.cpp:1166
    Event: 10959.951 Thread 0x48d06400 Threw 0x03b07260 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jni.cpp:742
    Event: 10959.951 Thread 0x48d06400 Threw 0x03b07930 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jni.cpp:742
    Event: 10959.952 Thread 0x48d06400 Threw 0x03b1dd00 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jni.cpp:742
    Event: 10968.584 Thread 0x48c46800 Threw 0x03a63910 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jni.cpp:742
    Event: 11019.262 Thread 0x48c46800 Threw 0x03b9bc60 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jvm.cpp:1166
    Event: 11019.265 Thread 0x48c46800 Threw 0x03bb4000 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jvm.cpp:1166
    Event: 11019.265 Thread 0x48c46800 Threw 0x03bbc788 at C:\jdk7u2_32P\jdk7u10\hotspot\src\share\vm\prims\jvm.cpp:1166
    Events (10 events):
    Event: 11019.264 loading class 0x486be698
    Event: 11019.264 loading class 0x486be698 done
    Event: 11019.265 loading class 0x48019038
    Event: 11019.265 loading class 0x48019038 done
    Event: 11019.955 Thread 0x48d06400 Thread exited: 0x48d06400
    Event: 11019.962 Thread 0x48d06400 Thread added: 0x48d06400
    Event: 11019.962 Executing VM operation: RevokeBias
    Event: 11019.963 Executing VM operation: RevokeBias done
    Event: 11019.964 Executing VM operation: RevokeBias
    Event: 11019.964 Executing VM operation: RevokeBias done
    Dynamic libraries:
    0x01110000 - 0x0113f000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\javaw.exe
    0x77c10000 - 0x77d90000 C:\Windows\SysWOW64\ntdll.dll
    0x75cb0000 - 0x75dc0000 C:\Windows\syswow64\kernel32.dll
    0x76400000 - 0x76447000 C:\Windows\syswow64\KERNELBASE.dll
    0x76450000 - 0x764f1000 C:\Windows\syswow64\ADVAPI32.dll
    0x761b0000 - 0x7625c000 C:\Windows\syswow64\msvcrt.dll
    0x76600000 - 0x76619000 C:\Windows\SysWOW64\sechost.dll
    0x760c0000 - 0x761b0000 C:\Windows\syswow64\RPCRT4.dll
    0x752a0000 - 0x75300000 C:\Windows\syswow64\SspiCli.dll
    0x75290000 - 0x7529c000 C:\Windows\syswow64\CRYPTBASE.dll
    0x76500000 - 0x76600000 C:\Windows\syswow64\USER32.dll
    0x75b90000 - 0x75c20000 C:\Windows\syswow64\GDI32.dll
    0x75850000 - 0x7585a000 C:\Windows\syswow64\LPK.dll
    0x75de0000 - 0x75e7d000 C:\Windows\syswow64\USP10.dll
    0x6e4f0000 - 0x6e68e000 C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.7601.18837_none_41e855142bd5705d\COMCTL32.dll
    0x75fa0000 - 0x75ff7000 C:\Windows\syswow64\SHLWAPI.dll
    0x75460000 - 0x754c0000 C:\Windows\system32\IMM32.DLL
    0x75900000 - 0x759cc000 C:\Windows\syswow64\MSCTF.dll
    0x71a40000 - 0x71afd000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\msvcr100.dll
    0x71590000 - 0x718dc000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\client\jvm.dll
    0x736c0000 - 0x736c7000 C:\Windows\system32\WSOCK32.dll
    0x766a0000 - 0x766d5000 C:\Windows\syswow64\WS2_32.dll
    0x75b80000 - 0x75b86000 C:\Windows\syswow64\NSI.dll
    0x72e50000 - 0x72e82000 C:\Windows\system32\WINMM.dll
    0x77be0000 - 0x77be5000 C:\Windows\syswow64\PSAPI.DLL
    0x73e10000 - 0x73e1c000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\verify.dll
    0x73d70000 - 0x73d90000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\java.dll
    0x73d90000 - 0x73da3000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\zip.dll
    0x73db0000 - 0x73dba000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\management.dll
    0x73860000 - 0x73874000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\net.dll
    0x73680000 - 0x736bc000 C:\Windows\system32\mswsock.dll
    0x72fc0000 - 0x72fc6000 C:\Windows\System32\wship6.dll
    0x73b00000 - 0x73b0f000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\nio.dll
    0x73d60000 - 0x73d69000 D:\Program Files (x86)\CA\SC\JRE\1.7.0_10\bin\sunmscapi.dll
    0x75520000 - 0x75641000 C:\Windows\syswow64\CRYPT32.dll
    0x75840000 - 0x7584c000 C:\Windows\syswow64\MSASN1.dll
    0x732e0000 - 0x732f7000 C:\Windows\system32\CRYPTSP.dll
    0x73050000 - 0x7308b000 C:\Windows\system32\rsaenh.dll
    0x6c7f0000 - 0x6c807000 C:\Windows\system32\USERENV.dll
    0x73040000 - 0x7304b000 C:\Windows\system32\profapi.dll
    0x72de0000 - 0x72df0000 C:\Windows\system32\NLAapi.dll
    0x72d80000 - 0x72d90000 C:\Windows\system32\napinsp.dll
    0x72d30000 - 0x72d74000 C:\Windows\system32\DNSAPI.dll
    0x72d10000 - 0x72d18000 C:\Windows\System32\winrnr.dll
    0x73670000 - 0x73675000 C:\Windows\System32\wshtcpip.dll
    0x738e0000 - 0x738fc000 C:\Windows\system32\IPHLPAPI.DLL
    0x738d0000 - 0x738d7000 C:\Windows\system32\WINNSI.DLL
    0x72c60000 - 0x72c66000 C:\Windows\system32\rasadhlp.dll
    0x72c70000 - 0x72ca8000 C:\Windows\System32\fwpuclnt.dll
    0x6c070000 - 0x6c285000 D:\Program Files (x86)\CA\Service Desk Manager\bin\sda65.dll
    0x73cc0000 - 0x73ccf000 D:\PROGRA~1\CA\SERVIC~1\bin\capkiSD.dll
    0x718e0000 - 0x719cb000 C:\Windows\system32\dbghelp.dll
    0x4b8d0000 - 0x4b972000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\libcapki.dll
    0x71b40000 - 0x71bdb000 C:\Windows\WinSxS\x86_microsoft.vc80.crt_1fc8b3b9a1e18e3b_8.0.50727.6195_none_d09154e044272b9a\MSVCR80.dll
    0x4baa0000 - 0x4bb99000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\libcaopenssl_crypto.dll
    0x49150000 - 0x4918c000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\libcaopenssl_ssl.dll
    0x00640000 - 0x00647000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\libcapki_thread.dll
    0x00b00000 - 0x00b07000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\libcapki_ipthread.dll
    0x495a0000 - 0x495da000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\cryptocme.dll
    0x4b140000 - 0x4b191000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\ccme_base.dll
    0x49980000 - 0x499b7000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\ccme_asym.dll
    0x4bba0000 - 0x4bc22000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\ccme_ecc.dll
    0x4b6b0000 - 0x4b719000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\ccme_ecdrbg.dll
    0x4bc30000 - 0x4bcf9000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\ccme_ecc_accel_fips.dll
    0x476d0000 - 0x476d7000 D:\Program Files (x86)\CA\SC\eTPKI\CAPKI\Windows\x86\32\lib\ccme_error_info.dll
    0x759d0000 - 0x75aeb000 C:\Windows\syswow64\WININET.DLL
    0x75510000 - 0x75513000 C:\Windows\syswow64\Normaliz.dll
    0x75680000 - 0x7583b000 C:\Windows\syswow64\iertutil.dll
    0x75e80000 - 0x75f9a000 C:\Windows\syswow64\urlmon.dll
    0x75300000 - 0x7545d000 C:\Windows\syswow64\ole32.dll
    0x75860000 - 0x758f1000 C:\Windows\syswow64\OLEAUT32.dll
    VM Arguments:
    jvm_args: -Xrs -Djava.net.preferIPv4Stack=false -Xms64M -Xmx1024M -Djaas.config=D:\PROGRA~1\CA\SERVIC~1\add-ons\caflow\jaas.config -Djava.security.auth.login.config=D:\PROGRA~1\CA\SERVIC~1\add-ons\caflow\jaas.config -Djdbc.baseDriver=com.microsoft.sqlserver.jdbc.SQLServerDriver -Djavax.xml.transform.TransformerFactory=com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl -Dsun.io.useCanonCaches=false -Djava.awt.headless=true -Djava.endorsed.dirs=D:\Program Files (x86)\CA\SC\tomcat\7.0.55\common\endorsed -Dcatalina.base=D:\PROGRA~1\CA\SERVIC~1\bopcfg\www\CATALINA_BASE -Dcatalina.home=D:\Program Files (x86)\CA\SC\tomcat\7.0.55 -Djava.io.tmpdir=D:\PROGRA~1\CA\SERVIC~1\bopcfg\www\CATALINA_BASE\temp 
    java_command: org.apache.catalina.startup.Bootstrap start
    Launcher Type: SUN_STANDARD
    Environment Variables:
    PATH=D:\PROGRA~1\CA\SERVIC~1\bin;D:\PROGRA~1\CA\SERVIC~1\bin;;D:\PROGRA~1\CA\SERVIC~1\bin;;D:\Program Files (x86)\CA\Service Desk Manager\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\BATCH;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\System Center Operations Manager 2007\;D:\Program Files (x86)\CA\Service Desk Manager\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\BATCH;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\System Center Operations Manager 2007\;D:\Program Files (x86)\CA\Service Desk Manager\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\BATCH;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\VSShell\Common7\IDE\;C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files\Microsoft SQL Server\100\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\;C:\Program Files\System Center Operations Manager 2007\
    USERNAME=GSDDomainUser
    OS=Windows_NT
    PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 26 Stepping 4, GenuineIntel
    ---------------  S Y S T E M  ---------------
    OS: Windows NT 6.1 , 64 bit Build 7601 Service Pack 1
    CPU:total 8 (8 cores per cpu, 1 threads per core) family 6 model 26 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, tsc, tscinvbit, tscinv
    Memory: 4k page, physical 25165304k(14975384k free), swap 50328748k(40659756k free)
    vm_info: Java HotSpot(TM) Client VM (23.6-b04) for windows-x86 JRE (1.7.0_10-b18), built on Nov 28 2012 08:34:29 by "java_re" with unknown MS VC++:1600
    time: Thu Dec 22 00:10:32 2016
    elapsed time: 11020 seconds



  • 8.  Re: CA servicedesk Tomcat issue

    Broadcom Employee
    Posted Dec 22, 2016 09:17 AM

    I have found your Support case for this item.

     

    I believe I might have identified a possible testfix for this one, I'll update you via the case to see how it goes.

     

    _R



  • 9.  Re: CA servicedesk Tomcat issue

    Posted Dec 23, 2016 01:38 AM

    Thanks Raghu!!!

     

    Regards,

    Sushant Bansode

     

     

    On Thu, Dec 22, 2016 at 3:17 PM, Raghu.Rudraraju <



  • 10.  Re: CA servicedesk Tomcat issue

    Broadcom Employee
    Posted Dec 21, 2016 02:43 PM

    Your Tomcat/javaw.exe crashed and that was the crash log.

    There could be a lot of reasons why Tomcat crashes...a WS call with some special char, a huge attachment to an SDM ticket,

    memory leak, etc. I think the best way to get to the bottom of this is to open a Support case to investigate.



  • 11.  Re: CA servicedesk Tomcat issue

    Posted Dec 22, 2016 03:28 AM

    HI,

     from the logs you published here we can see that this seems to be caused by an AXIS call meaning your web services . so I will investigate any change recently to your integration and look if for some reason you have more calls that before that can't be supported by your current infra or any change in the syntax.

     

    I can also see in this tread that you mention you have jre 1.7.0_25 but your log show 1.7.0_18. did you make any update to your java recently or have install any new apps on your server?

     

    This type of error  EXCEPTION_ACCESS_VIOLATION (0xc0000005) is generally caused by a NULL pointer exception so I will suspect either a non supported JRE to be used or a bug in CA code that don't address such correctly.

     

    That say this can also be caused by not having enough space for your JAVA VM to process the garbage collection or others.

    You may want to look at the article below to try to fine tune and more specifically to increase your heap size (Xms,Xmx)

    https://communities.ca.com/thread/241749877

     

    Definitively a case for CA support

    My 2 cents

    /J



  • 12.  Re: CA servicedesk Tomcat issue

    Posted Dec 22, 2016 09:57 AM

    You say that files are being written to "D:\Program Files (x86)\CA\Service Desk Manager\bin"?  Nothing should be writing to bin as a log or dump file.  Is there a chance that the an environmental variable for the Tomcat or JRE installation is wrong?

     

    J.W.



  • 13.  Re: CA servicedesk Tomcat issue

    Broadcom Employee
    Posted Dec 22, 2016 10:01 AM

    J.W.

    Unfortunately Java's crash dumps do end up getting written to that location in SDM.   They're of the format   hs***.log

     

    _R



  • 14.  Re: CA servicedesk Tomcat issue

    Posted Dec 22, 2016 10:05 AM

    Wow.  I didn't know that.  Not writing to bin was basic programming back in the day.  SMH.

    thanks,

    J.W.



  • 15.  Re: CA servicedesk Tomcat issue

    Posted Feb 15, 2017 01:12 PM

    Hi SUHSB,

    Did you finally get a fix for your issue? 

    Do the root cause have been identified?

    Thanks to share here for your experience to  benefits to others

    /J