Service Virtualization

  • 1.  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Posted Sep 26, 2017 01:56 AM

    please help

    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (malloc) failed to allocate 34816 bytes for Chunk::new
    # Possible reasons:
    #   The system is out of physical RAM or swap space
    #   In 32 bit mode, the process size limit was hit
    # Possible solutions:
    #   Reduce memory load on the system
    #   Increase physical memory or swap space
    #   Check if swap backing store is full
    #   Use 64 bit Java on a 64 bit OS
    #   Decrease Java heap size (-Xmx/-Xms)
    #   Decrease number of Java threads
    #   Decrease Java thread stack sizes (-Xss)
    #   Set larger code cache with -XX:ReservedCodeCacheSize=
    # This output file may be truncated or incomplete.
    #
    #  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208
    #
    # JRE version: Java(TM) SE Runtime Environment (8.0_102-b14) (build 1.8.0_102-b14)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode windows-amd64 )
    # Failed to write core dump.
    #

    ---------------  T H R E A D  ---------------

    Current thread (0x0000000034407000):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=29192, stack(0x000000003a850000,0x000000003a950000)]

    Stack: [0x000000003a850000,0x000000003a950000]
    [error occurred during error reporting (printing stack bounds), id 0xc0000005]

    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)


    Current CompileTask:
    C2: 100700 12303       4       sun.nio.fs.WindowsPath::hashCode (52 bytes)


    ---------------  P R O C E S S  ---------------

    Java Threads: ( => current thread )
      0x0000001577749000 JavaThread "elasticsearch[Dr. Otto Octavius][management][T#2]" daemon [_thread_blocked, id=26476, stack(0x00000015b6da0000,0x00000015b6ea0000)]
      0x0000001577748800 JavaThread "ServoPollScheduler-9" daemon [_thread_blocked, id=12804, stack(0x00000015b6ca0000,0x00000015b6da0000)]
      0x0000001577744800 JavaThread "ServoPollScheduler-8" daemon [_thread_blocked, id=11196, stack(0x00000015b6ba0000,0x00000015b6ca0000)]
      0x0000001577747800 JavaThread "ServoPollScheduler-7" daemon [_thread_blocked, id=9088, stack(0x00000015b69a0000,0x00000015b6aa0000)]
      0x0000001577746000 JavaThread "ServoPollScheduler-6" daemon [_thread_blocked, id=20284, stack(0x00000015b70a0000,0x00000015b71a0000)]
      0x0000001577745800 JavaThread "qtp1879034789-198" [_thread_blocked, id=4788, stack(0x00000015b64f0000,0x00000015b65f0000)]
      0x000000157773c800 JavaThread "org.eclipse.jetty.server.session.HashSessionManager@3847fe81Timer" daemon [_thread_blocked, id=25596, stack(0x00000015b67a0000,0x00000015b68a0000)]
      0x000000157773f800 JavaThread "threadPoolExecutor-6" daemon [_thread_blocked, id=26556, stack(0x00000015b66a0000,0x00000015b67a0000)]
      0x000000157773e800 JavaThread "threadPoolExecutor-5" daemon [_thread_blocked, id=10588, stack(0x00000015b61f0000,0x00000015b62f0000)]
      0x000000157773e000 JavaThread "threadPoolExecutor-4" daemon [_thread_blocked, id=28880, stack(0x00000015b60f0000,0x00000015b61f0000)]
      0x0000001577738800 JavaThread "ServoPollScheduler-5" daemon [_thread_blocked, id=15944, stack(0x00000015b5ff0000,0x00000015b60f0000)]
      0x000000157773b800 JavaThread "threadPoolExecutor-3" daemon [_thread_blocked, id=28872, stack(0x00000015b5d80000,0x00000015b5e80000)]
      0x0000001577738000 JavaThread "threadPoolExecutor-2" daemon [_thread_blocked, id=20040, stack(0x00000015b5c80000,0x00000015b5d80000)]
      0x0000001577739800 JavaThread "threadPoolExecutor-1" daemon [_thread_blocked, id=9428, stack(0x00000015b5980000,0x00000015b5a80000)]
      0x000000157773b000 JavaThread "ServoPollScheduler-4" daemon [_thread_blocked, id=15984, stack(0x00000015b5880000,0x00000015b5980000)]
      0x0000001577737000 JavaThread "elasticsearch[Dr. Otto Octavius][listener][T#3]" daemon [_thread_blocked, id=14400, stack(0x00000015b5780000,0x00000015b5880000)]
      0x0000001577733800 JavaThread "elasticsearch[Dr. Otto Octavius][listener][T#2]" daemon [_thread_blocked, id=21196, stack(0x00000015b5480000,0x00000015b5580000)]
      0x0000001577735000 JavaThread "elasticsearch[Dr. Otto Octavius][listener][T#1]" daemon [_thread_blocked, id=22876, stack(0x00000015b5380000,0x00000015b5480000)]
      0x0000001577734000 JavaThread "elasticsearch[Dr. Otto Octavius][warmer][T#1]" daemon [_thread_blocked, id=27088, stack(0x00000015b5030000,0x00000015b5130000)]
      0x0000001577735800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_boss][T#1]{New I/O server boss #65}" daemon [_thread_in_native, id=28628, stack(0x00000015b4d30000,0x00000015b4e30000)]
      0x000000157772f800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#64]{New I/O worker #64}" daemon [_thread_in_native, id=22748, stack(0x00000015b4c30000,0x00000015b4d30000)]
      0x0000001577730800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#63]{New I/O worker #63}" daemon [_thread_in_native, id=24424, stack(0x00000015b4b30000,0x00000015b4c30000)]
      0x0000001577732800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#62]{New I/O worker #62}" daemon [_thread_in_native, id=27944, stack(0x00000015b4a30000,0x00000015b4b30000)]
      0x000000157772f000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#61]{New I/O worker #61}" daemon [_thread_in_native, id=22348, stack(0x00000015b4930000,0x00000015b4a30000)]
      0x0000001577732000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#60]{New I/O worker #60}" daemon [_thread_in_native, id=24588, stack(0x00000015b4830000,0x00000015b4930000)]
      0x0000001577731000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#59]{New I/O worker #59}" daemon [_thread_in_native, id=24500, stack(0x00000015b4730000,0x00000015b4830000)]
      0x000000157772e000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#58]{New I/O worker #58}" daemon [_thread_in_native, id=13032, stack(0x00000015b4630000,0x00000015b4730000)]
      0x000000157772a800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#57]{New I/O worker #57}" daemon [_thread_in_native, id=22020, stack(0x00000015b4530000,0x00000015b4630000)]
      0x000000157772c800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#56]{New I/O worker #56}" daemon [_thread_in_native, id=7188, stack(0x00000015b4430000,0x00000015b4530000)]
      0x000000157772c000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#55]{New I/O worker #55}" daemon [_thread_in_native, id=16000, stack(0x00000015b4330000,0x00000015b4430000)]
      0x000000157772b000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#54]{New I/O worker #54}" daemon [_thread_in_native, id=9936, stack(0x00000015b4230000,0x00000015b4330000)]
      0x000000157772d800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#53]{New I/O worker #53}" daemon [_thread_in_native, id=15308, stack(0x00000015b4130000,0x00000015b4230000)]
      0x0000001577858800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#52]{New I/O worker #52}" daemon [_thread_in_native, id=22120, stack(0x00000015b4030000,0x00000015b4130000)]
      0x0000001577858000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#51]{New I/O worker #51}" daemon [_thread_in_native, id=12560, stack(0x00000015b3f30000,0x00000015b4030000)]
      0x0000001577859800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#50]{New I/O worker #50}" daemon [_thread_in_native, id=14828, stack(0x00000015b3e30000,0x00000015b3f30000)]
      0x0000001577855800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#49]{New I/O worker #49}" daemon [_thread_in_native, id=29516, stack(0x00000015b3d30000,0x00000015b3e30000)]
      0x0000001577855000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#48]{New I/O worker #48}" daemon [_thread_in_native, id=27228, stack(0x00000015b3c30000,0x00000015b3d30000)]
      0x0000001577857000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#47]{New I/O worker #47}" daemon [_thread_in_native, id=25140, stack(0x00000015b3b30000,0x00000015b3c30000)]
      0x0000001577856800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#46]{New I/O worker #46}" daemon [_thread_in_native, id=27876, stack(0x00000015b3a30000,0x00000015b3b30000)]
      0x0000001577852000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#45]{New I/O worker #45}" daemon [_thread_in_native, id=20964, stack(0x00000015b3930000,0x00000015b3a30000)]
      0x0000001577854000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#44]{New I/O worker #44}" daemon [_thread_in_native, id=22292, stack(0x00000015b3830000,0x00000015b3930000)]
      0x0000001577850800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#43]{New I/O worker #43}" daemon [_thread_in_native, id=28524, stack(0x00000015b3730000,0x00000015b3830000)]
      0x0000001577852800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#42]{New I/O worker #42}" daemon [_thread_in_native, id=12516, stack(0x00000015b3630000,0x00000015b3730000)]
      0x0000001577853800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#41]{New I/O worker #41}" daemon [_thread_in_native, id=29688, stack(0x00000015b3530000,0x00000015b3630000)]
      0x0000001577851000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#40]{New I/O worker #40}" daemon [_thread_in_native, id=27164, stack(0x00000015b3430000,0x00000015b3530000)]
      0x000000157784f800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#39]{New I/O worker #39}" daemon [_thread_in_native, id=428, stack(0x00000015b3330000,0x00000015b3430000)]
      0x000000157784c000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#38]{New I/O worker #38}" daemon [_thread_in_native, id=26536, stack(0x00000015b3230000,0x00000015b3330000)]
      0x000000157784f000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#37]{New I/O worker #37}" daemon [_thread_in_native, id=15284, stack(0x00000015b3130000,0x00000015b3230000)]
      0x000000157784e000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#36]{New I/O worker #36}" daemon [_thread_in_native, id=19896, stack(0x00000015b3030000,0x00000015b3130000)]
      0x000000157784c800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#35]{New I/O worker #35}" daemon [_thread_in_native, id=22708, stack(0x00000015b2f30000,0x00000015b3030000)]
      0x000000157784b000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#34]{New I/O worker #34}" daemon [_thread_in_native, id=948, stack(0x00000015b2e30000,0x00000015b2f30000)]
      0x000000157784a800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#33]{New I/O worker #33}" daemon [_thread_in_native, id=29432, stack(0x00000015b2d30000,0x00000015b2e30000)]
      0x000000157784d800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#32]{New I/O worker #32}" daemon [_thread_in_native, id=17848, stack(0x00000015b2c30000,0x00000015b2d30000)]
      0x000000003e322000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#31]{New I/O worker #31}" daemon [_thread_in_native, id=18408, stack(0x00000015b2b30000,0x00000015b2c30000)]
      0x000000003e325000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#30]{New I/O worker #30}" daemon [_thread_in_native, id=27776, stack(0x00000015b2a30000,0x00000015b2b30000)]
      0x000000003e323800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#29]{New I/O worker #29}" daemon [_thread_in_native, id=16720, stack(0x00000015b2930000,0x00000015b2a30000)]
      0x000000003e324800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#28]{New I/O worker #28}" daemon [_thread_in_native, id=17288, stack(0x00000015b2830000,0x00000015b2930000)]
      0x000000003e323000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#27]{New I/O worker #27}" daemon [_thread_in_native, id=6392, stack(0x00000015b2730000,0x00000015b2830000)]
      0x000000003e31f000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#26]{New I/O worker #26}" daemon [_thread_in_native, id=24584, stack(0x00000015b2630000,0x00000015b2730000)]
      0x000000003e31e800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#25]{New I/O worker #25}" daemon [_thread_in_native, id=14360, stack(0x00000015b2530000,0x00000015b2630000)]
      0x000000003e320800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#24]{New I/O worker #24}" daemon [_thread_in_native, id=13636, stack(0x00000015b2430000,0x00000015b2530000)]
      0x000000003e320000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#23]{New I/O worker #23}" daemon [_thread_in_native, id=17912, stack(0x00000015b2330000,0x00000015b2430000)]
      0x000000003e321800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#22]{New I/O worker #22}" daemon [_thread_in_native, id=5280, stack(0x00000015b2230000,0x00000015b2330000)]
      0x000000156d301800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#21]{New I/O worker #21}" daemon [_thread_in_native, id=28016, stack(0x00000015b2130000,0x00000015b2230000)]
      0x000000156d302800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#20]{New I/O worker #20}" daemon [_thread_in_native, id=27652, stack(0x00000015b2030000,0x00000015b2130000)]
      0x000000156d301000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#19]{New I/O worker #19}" daemon [_thread_in_native, id=21556, stack(0x00000015b1f30000,0x00000015b2030000)]
      0x000000156d2fd000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#18]{New I/O worker #18}" daemon [_thread_in_native, id=20412, stack(0x00000015b1e30000,0x00000015b1f30000)]
      0x000000156d300000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#17]{New I/O worker #17}" daemon [_thread_in_native, id=20364, stack(0x00000015b1d30000,0x00000015b1e30000)]
      0x000000156d2ff800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#16]{New I/O worker #16}" daemon [_thread_in_native, id=20672, stack(0x00000015b1c30000,0x00000015b1d30000)]
      0x000000156d2fc800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#15]{New I/O worker #15}" daemon [_thread_in_native, id=5932, stack(0x00000015b1b30000,0x00000015b1c30000)]
      0x000000156d2fe800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#14]{New I/O worker #14}" daemon [_thread_in_native, id=8384, stack(0x00000015b1a30000,0x00000015b1b30000)]
      0x000000156d2fb800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#13]{New I/O worker #13}" daemon [_thread_in_native, id=29220, stack(0x00000015b1930000,0x00000015b1a30000)]
      0x000000156d2fe000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#12]{New I/O worker #12}" daemon [_thread_in_native, id=26952, stack(0x00000015b1830000,0x00000015b1930000)]
      0x000000156d2fa000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#11]{New I/O worker #11}" daemon [_thread_in_native, id=26948, stack(0x00000015b1730000,0x00000015b1830000)]
      0x000000156d2fb000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#10]{New I/O worker #10}" daemon [_thread_in_native, id=1228, stack(0x00000015b1630000,0x00000015b1730000)]
      0x000000156d2f8800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#9]{New I/O worker #9}" daemon [_thread_in_native, id=7164, stack(0x00000015b1530000,0x00000015b1630000)]
      0x000000156d2f8000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#8]{New I/O worker #8}" daemon [_thread_in_native, id=2356, stack(0x00000015b1430000,0x00000015b1530000)]
      0x000000156d2f9800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#7]{New I/O worker #7}" daemon [_thread_in_native, id=21944, stack(0x00000015b1330000,0x00000015b1430000)]
      0x000000156d2f4000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#6]{New I/O worker #6}" daemon [_thread_in_native, id=28732, stack(0x00000015b1230000,0x00000015b1330000)]
      0x000000156d2f7000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#5]{New I/O worker #5}" daemon [_thread_in_native, id=11148, stack(0x00000015b1130000,0x00000015b1230000)]
      0x000000156d2f3800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#4]{New I/O worker #4}" daemon [_thread_in_native, id=22080, stack(0x00000015b1030000,0x00000015b1130000)]
      0x000000156d2f5000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#3]{New I/O worker #3}" daemon [_thread_in_native, id=28120, stack(0x00000015b0f30000,0x00000015b1030000)]
      0x000000156d2f6800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#2]{New I/O worker #2}" daemon [_thread_in_native, id=13080, stack(0x00000015b0e30000,0x00000015b0f30000)]
      0x000000156d2f5800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#1]{New I/O worker #1}" daemon [_thread_in_native, id=7908, stack(0x00000015b0d30000,0x00000015b0e30000)]
      0x000000158f52b000 JavaThread "elasticsearch[Dr. Otto Octavius][management][T#1]" daemon [_thread_blocked, id=5920, stack(0x00000015b0c30000,0x00000015b0d30000)]
      0x00000015a676c000 JavaThread "elasticsearch[Dr. Otto Octavius][riverClusterService#updateTask][T#1]" daemon [_thread_blocked, id=23628, stack(0x00000015b0b30000,0x00000015b0c30000)]
      0x00000015a676b000 JavaThread "elasticsearch[Dr. Otto Octavius][generic][T#1]" daemon [_thread_blocked, id=13320, stack(0x00000015b0830000,0x00000015b0930000)]
      0x00000015a676a800 JavaThread "elasticsearch[Dr. Otto Octavius][clusterService#updateTask][T#1]" daemon [_thread_blocked, id=21312, stack(0x00000015b0730000,0x00000015b0830000)]
      0x0000001593a1f800 JavaThread "elasticsearch[Dr. Otto Octavius][[ttl_expire]]" daemon [_thread_blocked, id=22972, stack(0x00000015b0630000,0x00000015b0730000)]
      0x0000001593a1e800 JavaThread "elasticsearch[Dr. Otto Octavius][master_mapping_updater]" [_thread_blocked, id=10860, stack(0x00000015b0530000,0x00000015b0630000)]
      0x0000001593a1b800 JavaThread "elasticsearch[Dr. Otto Octavius][scheduler][T#1]" daemon [_thread_blocked, id=12792, stack(0x00000015b0430000,0x00000015b0530000)]
      0x0000001593a1d000 JavaThread "ServoPollScheduler-3" daemon [_thread_blocked, id=28020, stack(0x00000015af280000,0x00000015af380000)]
      0x0000001593a1e000 JavaThread "elasticsearch[Dr. Otto Octavius][[timer]]" daemon [_thread_blocked, id=4556, stack(0x000000003d020000,0x000000003d120000)]
      0x0000001593a1c800 JavaThread "Actor Thread 1" daemon [_thread_blocked, id=7096, stack(0x0000000075110000,0x0000000075210000)]
      0x0000001593a17000 JavaThread "ServoPollScheduler-2" daemon [_thread_blocked, id=20556, stack(0x0000000071070000,0x0000000071170000)]
      0x0000001593a15800 JavaThread "ServoPollScheduler-1" daemon [_thread_blocked, id=24300, stack(0x00000015a8c20000,0x00000015a8d20000)]
      0x0000001593a12000 JavaThread "ServoPollScheduler-0" daemon [_thread_blocked, id=29672, stack(0x000000007fea0000,0x000000007ffa0000)]
      0x0000001593a15000 JavaThread "SimpleHostRoutingFilter.CONNECTION_MANAGER_TIMER" daemon [_thread_blocked, id=29236, stack(0x00000000582f0000,0x00000000583f0000)]
      0x000000158f526800 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=26280, stack(0x000000006bfe0000,0x000000006c0e0000)]
      0x000000158f528000 JavaThread "Tomcat JDBC Pool Cleaner[724125922:1506335175053]" daemon [_thread_blocked, id=4456, stack(0x0000000054290000,0x0000000054390000)]
      0x00000015a6760000 JavaThread "FileWatchdog" daemon [_thread_blocked, id=24328, stack(0x00000000423f0000,0x00000000424f0000)]
      0x000000003e13f000 JavaThread "qtp1879034789-94" [_thread_blocked, id=12484, stack(0x00000000424f0000,0x00000000425f0000)]
      0x000000003e13b000 JavaThread "qtp1879034789-92" [_thread_blocked, id=568, stack(0x00000000422f0000,0x00000000423f0000)]
      0x000000003e13e000 JavaThread "qtp1879034789-91" [_thread_blocked, id=17244, stack(0x00000000421f0000,0x00000000422f0000)]
      0x000000003e13a800 JavaThread "qtp1879034789-90" [_thread_blocked, id=25836, stack(0x00000000420f0000,0x00000000421f0000)]
      0x000000003e13d800 JavaThread "qtp1879034789-89" [_thread_blocked, id=12752, stack(0x0000000041ff0000,0x00000000420f0000)]
      0x000000003e139800 JavaThread "qtp1879034789-88" [_thread_blocked, id=29152, stack(0x0000000041ef0000,0x0000000041ff0000)]
      0x000000003e13c800 JavaThread "qtp1879034789-87" [_thread_blocked, id=6348, stack(0x0000000041df0000,0x0000000041ef0000)]
      0x000000003e139000 JavaThread "qtp1879034789-86" [_thread_blocked, id=17000, stack(0x0000000041cf0000,0x0000000041df0000)]
      0x000000003e13c000 JavaThread "qtp1879034789-85" [_thread_blocked, id=21816, stack(0x0000000041bf0000,0x0000000041cf0000)]
      0x000000003e138000 JavaThread "qtp1879034789-84" [_thread_blocked, id=24008, stack(0x00000000418f0000,0x00000000419f0000)]
      0x000000003e135000 JavaThread "qtp1879034789-83" [_thread_blocked, id=14128, stack(0x00000000417f0000,0x00000000418f0000)]
      0x000000003e134800 JavaThread "qtp1879034789-82" [_thread_blocked, id=27428, stack(0x00000000416f0000,0x00000000417f0000)]
      0x000000003e137800 JavaThread "qtp1879034789-81" [_thread_blocked, id=20996, stack(0x00000000415f0000,0x00000000416f0000)]
      0x000000003e136000 JavaThread "qtp1879034789-80" [_thread_blocked, id=28476, stack(0x00000000414f0000,0x00000000415f0000)]
      0x000000003e133800 JavaThread "qtp1879034789-79" [_thread_blocked, id=16964, stack(0x00000000413f0000,0x00000000414f0000)]
      0x000000003e136800 JavaThread "qtp1879034789-78" [_thread_blocked, id=3672, stack(0x00000000412f0000,0x00000000413f0000)]
      0x000000003e0c7000 JavaThread "qtp1879034789-77" [_thread_blocked, id=24628, stack(0x00000000411f0000,0x00000000412f0000)]
      0x000000003e0c6800 JavaThread "qtp1879034789-76" [_thread_blocked, id=11804, stack(0x00000000410f0000,0x00000000411f0000)]
      0x000000003e0c5800 JavaThread "qtp1879034789-75" [_thread_blocked, id=28996, stack(0x0000000040ff0000,0x00000000410f0000)]
      0x000000003e0c5000 JavaThread "qtp1879034789-74" [_thread_blocked, id=7240, stack(0x0000000040ef0000,0x0000000040ff0000)]
      0x000000003e0c3800 JavaThread "qtp1879034789-73" [_thread_blocked, id=23144, stack(0x0000000040df0000,0x0000000040ef0000)]
      0x000000003e0c2800 JavaThread "qtp1879034789-72" [_thread_blocked, id=13248, stack(0x0000000040cf0000,0x0000000040df0000)]
      0x000000003e0c4000 JavaThread "qtp1879034789-71" [_thread_blocked, id=11736, stack(0x0000000040bf0000,0x0000000040cf0000)]
      0x000000003e0bf000 JavaThread "qtp1879034789-70" [_thread_blocked, id=15844, stack(0x0000000040af0000,0x0000000040bf0000)]
      0x000000003e0c2000 JavaThread "qtp1879034789-69" [_thread_blocked, id=18096, stack(0x00000000409f0000,0x0000000040af0000)]
      0x000000003e0bf800 JavaThread "qtp1879034789-68" [_thread_blocked, id=18524, stack(0x00000000408f0000,0x00000000409f0000)]
      0x000000003e0be000 JavaThread "qtp1879034789-67" [_thread_blocked, id=21560, stack(0x00000000407f0000,0x00000000408f0000)]
      0x000000003e0c1000 JavaThread "qtp1879034789-66" [_thread_blocked, id=3608, stack(0x00000000406f0000,0x00000000407f0000)]
      0x000000003e0c0800 JavaThread "qtp1879034789-65" [_thread_blocked, id=7048, stack(0x00000000405f0000,0x00000000406f0000)]
      0x000000003e0bd800 JavaThread "qtp1879034789-64" [_thread_blocked, id=24096, stack(0x00000000404f0000,0x00000000405f0000)]
      0x000000003e0b9800 JavaThread "qtp1879034789-63" [_thread_blocked, id=25260, stack(0x00000000403f0000,0x00000000404f0000)]
      0x000000003e0bc800 JavaThread "qtp1879034789-62" [_thread_blocked, id=27012, stack(0x00000000402f0000,0x00000000403f0000)]
      0x000000003e0ba800 JavaThread "qtp1879034789-61" [_thread_blocked, id=18200, stack(0x00000000401f0000,0x00000000402f0000)]
      0x000000003e0bb000 JavaThread "qtp1879034789-60" [_thread_blocked, id=26020, stack(0x00000000400f0000,0x00000000401f0000)]
      0x000000003e0bc000 JavaThread "qtp1879034789-59" [_thread_blocked, id=23656, stack(0x000000003fff0000,0x00000000400f0000)]
      0x000000003e0b9000 JavaThread "qtp1879034789-58" [_thread_blocked, id=19780, stack(0x000000003fef0000,0x000000003fff0000)]
      0x000000003e0b8000 JavaThread "qtp1879034789-57" [_thread_blocked, id=4460, stack(0x000000003fdf0000,0x000000003fef0000)]
      0x000000003be32000 JavaThread "qtp1879034789-56" [_thread_blocked, id=10736, stack(0x000000003fcf0000,0x000000003fdf0000)]
      0x000000003be33000 JavaThread "qtp1879034789-55" [_thread_blocked, id=8252, stack(0x000000003fbf0000,0x000000003fcf0000)]
      0x000000003be2e800 JavaThread "qtp1879034789-54" [_thread_blocked, id=28304, stack(0x000000003faf0000,0x000000003fbf0000)]
      0x000000003be31800 JavaThread "qtp1879034789-53" [_thread_blocked, id=12104, stack(0x000000003f9f0000,0x000000003faf0000)]
      0x000000003be30800 JavaThread "qtp1879034789-52" [_thread_blocked, id=23828, stack(0x000000003f8f0000,0x000000003f9f0000)]
      0x000000003be2f000 JavaThread "qtp1879034789-51" [_thread_blocked, id=26132, stack(0x000000003f7f0000,0x000000003f8f0000)]
      0x000000003be30000 JavaThread "qtp1879034789-50" [_thread_blocked, id=3632, stack(0x000000003f6f0000,0x000000003f7f0000)]
      0x000000003be2a800 JavaThread "qtp1879034789-49" [_thread_blocked, id=17164, stack(0x000000003f5f0000,0x000000003f6f0000)]
      0x000000003be2d000 JavaThread "qtp1879034789-48" [_thread_blocked, id=21492, stack(0x000000003f4f0000,0x000000003f5f0000)]
      0x000000003be2d800 JavaThread "qtp1879034789-47" [_thread_blocked, id=20032, stack(0x000000003f3f0000,0x000000003f4f0000)]
      0x000000003be2c000 JavaThread "qtp1879034789-46" [_thread_blocked, id=5980, stack(0x000000003f2f0000,0x000000003f3f0000)]
      0x000000003be2b800 JavaThread "qtp1879034789-45" [_thread_blocked, id=9924, stack(0x000000003f1f0000,0x000000003f2f0000)]
      0x000000003be2a000 JavaThread "qtp1879034789-44" [_thread_blocked, id=15000, stack(0x000000003f0f0000,0x000000003f1f0000)]
      0x000000003be29000 JavaThread "qtp1879034789-43" [_thread_blocked, id=25408, stack(0x000000003eff0000,0x000000003f0f0000)]
      0x000000003be25000 JavaThread "qtp1879034789-42" [_thread_blocked, id=29588, stack(0x000000003eef0000,0x000000003eff0000)]
      0x000000003be28800 JavaThread "qtp1879034789-41" [_thread_blocked, id=6388, stack(0x000000003edf0000,0x000000003eef0000)]
      0x000000003be24800 JavaThread "qtp1879034789-40" [_thread_blocked, id=10996, stack(0x000000003ecf0000,0x000000003edf0000)]
      0x000000003be27800 JavaThread "qtp1879034789-39" [_thread_blocked, id=23580, stack(0x000000003ebf0000,0x000000003ecf0000)]
      0x000000003be23800 JavaThread "qtp1879034789-38" [_thread_blocked, id=9096, stack(0x000000003eaf0000,0x000000003ebf0000)]
      0x000000003be26000 JavaThread "qtp1879034789-37" [_thread_blocked, id=8156, stack(0x000000003e9f0000,0x000000003eaf0000)]
      0x000000003bd16000 JavaThread "qtp1879034789-36" [_thread_blocked, id=25508, stack(0x000000003d720000,0x000000003d820000)]
      0x000000003bd15000 JavaThread "qtp1879034789-35" [_thread_blocked, id=14536, stack(0x000000003d620000,0x000000003d720000)]
      0x000000003bd14000 JavaThread "qtp1879034789-34" [_thread_blocked, id=2024, stack(0x000000003d520000,0x000000003d620000)]
      0x000000003bd8b000 JavaThread "qtp1879034789-33" [_thread_blocked, id=10432, stack(0x000000003d420000,0x000000003d520000)]
      0x000000003bd8a800 JavaThread "qtp1879034789-32" [_thread_blocked, id=11444, stack(0x000000003d320000,0x000000003d420000)]
      0x000000003bd89800 JavaThread "qtp1879034789-31" [_thread_blocked, id=8552, stack(0x000000003d220000,0x000000003d320000)]
      0x000000003bd89000 JavaThread "qtp1879034789-30" [_thread_blocked, id=12412, stack(0x000000003d120000,0x000000003d220000)]
      0x0000000036af0000 JavaThread "FileWatchdog" daemon [_thread_blocked, id=11548, stack(0x000000003cd20000,0x000000003ce20000)]
      0x000000003bc84000 JavaThread "FileWatchdog" daemon [_thread_blocked, id=15212, stack(0x000000003cc20000,0x000000003cd20000)]
      0x000000003669a800 JavaThread "Service Thread" daemon [_thread_blocked, id=17892, stack(0x000000003ba50000,0x000000003bb50000)]
      0x0000000036601000 JavaThread "C1 CompilerThread17" daemon [_thread_blocked, id=25788, stack(0x000000003b950000,0x000000003ba50000)]
      0x00000000365ff800 JavaThread "C1 CompilerThread16" daemon [_thread_blocked, id=3312, stack(0x000000003b850000,0x000000003b950000)]
      0x00000000365ff000 JavaThread "C1 CompilerThread15" daemon [_thread_blocked, id=20440, stack(0x000000003b750000,0x000000003b850000)]
      0x00000000365fb000 JavaThread "C1 CompilerThread14" daemon [_thread_blocked, id=11200, stack(0x000000003b650000,0x000000003b750000)]
      0x00000000365fb800 JavaThread "C1 CompilerThread13" daemon [_thread_blocked, id=22536, stack(0x000000003b550000,0x000000003b650000)]
      0x00000000365fe000 JavaThread "C1 CompilerThread12" daemon [_thread_blocked, id=24380, stack(0x000000003b450000,0x000000003b550000)]
      0x00000000365fd000 JavaThread "C2 CompilerThread11" daemon [_thread_blocked, id=15276, stack(0x000000003b350000,0x000000003b450000)]
      0x00000000365fc800 JavaThread "C2 CompilerThread10" daemon [_thread_blocked, id=23172, stack(0x000000003b250000,0x000000003b350000)]
      0x00000000365fa000 JavaThread "C2 CompilerThread9" daemon [_thread_blocked, id=23640, stack(0x000000003b150000,0x000000003b250000)]
      0x00000000365f1800 JavaThread "C2 CompilerThread8" daemon [_thread_blocked, id=28040, stack(0x000000003b050000,0x000000003b150000)]
      0x00000000365cd800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=26432, stack(0x000000003af50000,0x000000003b050000)]
      0x000000003441e000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=6968, stack(0x000000003ae50000,0x000000003af50000)]
      0x00000000309da000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9324, stack(0x000000003ad50000,0x000000003ae50000)]
      0x0000000034419800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=3476, stack(0x000000003ac50000,0x000000003ad50000)]
      0x0000000034418800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=22068, stack(0x000000003ab50000,0x000000003ac50000)]
      0x0000000034418000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=20540, stack(0x000000003aa50000,0x000000003ab50000)]
      0x0000000034417000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=17360, stack(0x000000003a950000,0x000000003aa50000)]
    =>0x0000000034407000 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=29192, stack(0x000000003a850000,0x000000003a950000)]
      0x000000003657c000 JavaThread "Attach Listener" daemon [_thread_blocked, id=3168, stack(0x000000003a750000,0x000000003a850000)]
      0x000000003657b800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5192, stack(0x000000003a650000,0x000000003a750000)]
      0x00000000343e7800 JavaThread "Finalizer" daemon [_thread_blocked, id=12236, stack(0x000000003a450000,0x000000003a550000)]
      0x00000000343e1800 JavaThread "Reference Handler" daemon [_thread_blocked, id=5200, stack(0x000000003a350000,0x000000003a450000)]
      0x0000000002f22800 JavaThread "main" [_thread_in_vm, id=19580, stack(0x0000000000850000,0x0000000000950000)]

    Other Threads:
      0x00000000309c0800 VMThread [stack: 0x000000003a250000,0x000000003a350000] [id=27916]
      0x00000000366d0800 WatcherThread [stack: 0x000000003bb50000,0x000000003bc50000] [id=24348]

    VM state:not at safepoint (normal execution)

    VM Mutex/Monitor currently owned by a thread: None

    Heap:
     PSYoungGen      total 16091136K, used 9929630K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 69% used [0x0000000dd5580000,0x00000010336678d8,0x0000001140d00000)
      from space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 46550528K, used 306079K [0x0000000080000000, 0x0000000b99380000, 0x0000000dd5580000)
      object space 46550528K, 0% used [0x0000000080000000,0x0000000092ae7f70,0x0000000b99380000)
     Metaspace       used 115068K, capacity 117258K, committed 117504K, reserved 118784K

    Card table byte_map: [0x00000000123e0000,0x000000001c3f0000] byte_map_base: 0x0000000011fe0000

    Marking Bits: (ParMarkBitMap*) 0x000000005a13a6c0
     Begin Bits: [0x0000001480000000, 0x00000014d0000000)
     End Bits:   [0x00000014d0000000, 0x0000001520000000)

    Polling page: 0x00000000009b0000

    CodeCache: size=245760Kb used=46370Kb max_used=46370Kb free=199389Kb
     bounds [0x0000000003020000, 0x0000000005e20000, 0x0000000012020000]
     total_blobs=10504 nmethods=9603 adapters=805
     compilation: enabled

    Compilation events (10 events):
    Event: 100.674 Thread 0x00000000365fe000 nmethod 12297 0x000000000517e510 code [0x000000000517e6a0, 0x000000000517eb38]
    Event: 100.674 Thread 0x00000000365fb800 nmethod 12298 0x0000000003afa4d0 code [0x0000000003afa660, 0x0000000003afaa98]
    Event: 100.677 Thread 0x00000000365ff000 12299       3       org.eclipse.jetty.xml.XmlParser$Node$1::hasNext (99 bytes)
    Event: 100.677 Thread 0x0000000036601000 12300       3       org.eclipse.jetty.xml.XmlParser$Node::get (65 bytes)
    Event: 100.677 Thread 0x00000000365ff800 12302  s    3       org.eclipse.jetty.xml.XmlParser$Node::toString (221 bytes)
    Event: 100.677 Thread 0x00000000365fe000 12301  s    3       org.eclipse.jetty.xml.XmlParser$Node::toString (19 bytes)
    Event: 100.678 Thread 0x0000000034407000 12303       4       sun.nio.fs.WindowsPath::hashCode (52 bytes)
    Event: 100.680 Thread 0x00000000365fe000 nmethod 12301 0x0000000003435650 code [0x0000000003435800, 0x0000000003435e78]
    Event: 100.681 Thread 0x0000000036601000 nmethod 12300 0x0000000003c007d0 code [0x0000000003c009a0, 0x0000000003c01328]
    Event: 100.681 Thread 0x00000000365ff000 nmethod 12299 0x00000000059e5c50 code [0x00000000059e5e40, 0x00000000059e6868]

    GC Heap History (10 events):
    Event: 26.651 GC heap before
    {Heap before GC invocations=5 (full 2):
     PSYoungGen      total 12233216K, used 27783K [0x0000000dd5580000, 0x0000001239700000, 0x0000001480000000)
      eden space 10485760K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001055580000)
      from space 1747456K, 1% used [0x0000001055580000,0x00000010570a1f60,0x00000010c0000000)
      to   space 1747456K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001239700000)
     ParOldGen       total 27962368K, used 625741K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 2% used [0x0000000080000000,0x00000000a63137f8,0x000000072ab00000)
     Metaspace       used 34654K, capacity 35328K, committed 35416K, reserved 36864K
    Event: 27.271 GC heap after
    Heap after GC invocations=5 (full 2):
     PSYoungGen      total 12233216K, used 0K [0x0000000dd5580000, 0x0000001239700000, 0x0000001480000000)
      eden space 10485760K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001055580000)
      from space 1747456K, 0% used [0x0000001055580000,0x0000001055580000,0x00000010c0000000)
      to   space 1747456K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001239700000)
     ParOldGen       total 27962368K, used 347576K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x000000009536e3f8,0x000000072ab00000)
     Metaspace       used 34654K, capacity 35328K, committed 35416K, reserved 36864K
    }
    Event: 38.958 GC heap before
    {Heap before GC invocations=6 (full 2):
     PSYoungGen      total 12233216K, used 1525617K [0x0000000dd5580000, 0x0000001239700000, 0x0000001480000000)
      eden space 10485760K, 14% used [0x0000000dd5580000,0x0000000e3275c4d8,0x0000001055580000)
      from space 1747456K, 0% used [0x0000001055580000,0x0000001055580000,0x00000010c0000000)
      to   space 1747456K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001239700000)
     ParOldGen       total 27962368K, used 347576K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x000000009536e3f8,0x000000072ab00000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    Event: 39.130 GC heap after
    Heap after GC invocations=6 (full 2):
     PSYoungGen      total 15512576K, used 122189K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1168896K, 10% used [0x00000011cec80000,0x00000011d63d36a8,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 27962368K, used 347649K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x0000000095380410,0x000000072ab00000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    }
    Event: 39.130 GC heap before
    {Heap before GC invocations=7 (full 3):
     PSYoungGen      total 15512576K, used 122189K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1168896K, 10% used [0x00000011cec80000,0x00000011d63d36a8,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 27962368K, used 347649K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x0000000095380410,0x000000072ab00000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    Event: 39.734 GC heap after
    Heap after GC invocations=7 (full 3):
     PSYoungGen      total 15512576K, used 0K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1168896K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 34104832K, used 365482K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964eaa38,0x00000008a1980000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    }
    Event: 67.747 GC heap before
    {Heap before GC invocations=8 (full 3):
     PSYoungGen      total 15512576K, used 7498360K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 52% used [0x0000000dd5580000,0x0000000f9f01e008,0x0000001140d00000)
      from space 1168896K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 34104832K, used 365482K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964eaa38,0x00000008a1980000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    Event: 67.974 GC heap after
    Heap after GC invocations=8 (full 3):
     PSYoungGen      total 16091136K, used 142921K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1747456K, 8% used [0x0000001140d00000,0x0000001149892538,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 34104832K, used 365522K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964f4a50,0x00000008a1980000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    }
    Event: 67.974 GC heap before
    {Heap before GC invocations=9 (full 4):
     PSYoungGen      total 16091136K, used 142921K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1747456K, 8% used [0x0000001140d00000,0x0000001149892538,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 34104832K, used 365522K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964f4a50,0x00000008a1980000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    Event: 68.533 GC heap after
    Heap after GC invocations=9 (full 4):
     PSYoungGen      total 16091136K, used 0K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 46550528K, used 306079K [0x0000000080000000, 0x0000000b99380000, 0x0000000dd5580000)
      object space 46550528K, 0% used [0x0000000080000000,0x0000000092ae7f70,0x0000000b99380000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    }

    Deoptimization events (10 events):
    Event: 84.458 Thread 0x000000003e0c5800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000395e364 method=org.eclipse.jetty.annotations.AnnotationParser.normalize(Ljava/lang/String;)Ljava/lang/String; @ 21
    Event: 84.481 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x00000000039e70d0 method=java.util.zip.ZipEntry.setExtra0([BZ)V @ 192
    Event: 84.490 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000005d34e24 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 84.490 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000003943ef4 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 84.501 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000005cd53fc method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 84.507 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000003c90f80 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 85.333 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000005d96908 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 86.767 Thread 0x000000003e0b8000 Uncommon trap: reason=range_check action=make_not_entrant pc=0x00000000056be4e8 method=java.util.zip.ZipUtils.get16([BI)I @ 10
    Event: 100.670 Thread 0x000000157773a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000565432c method=org.eclipse.jetty.util.log.JettyAwareLogger.log(Lorg/slf4j/Marker;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V @ 2
    Event: 100.683 Thread 0x0000000002f22800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000003b2bdfc method=java.io.FilePermission.init(I)V @ 136

    Internal exceptions (10 events):
    Event: 82.340 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f155c5410) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.342 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f155d2f48) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.390 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15863428) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.413 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f159658d8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.472 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15d2b2b0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.484 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15d7d400) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.505 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15ea3c60) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 83.849 Thread 0x000000003be2d800 Implicit null exception at 0x0000000003b7f899 to 0x0000000003b82a75
    Event: 100.669 Thread 0x000000157773a000 Exception <a 'java/lang/OutOfMemoryError'> (0x00000010334b3d18) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 735]
    Event: 100.674 Thread 0x000000157773a000 Exception <a 'java/lang/OutOfMemoryError': unable to create new native thread> (0x00000010334cf740) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jvm.cpp, line 3020]

    Events (10 events):
    Event: 100.673 loading class org/eclipse/jetty/jsp/JettyJspServlet done
    Event: 100.675 loading class java/lang/Throwable$WrappedPrintStream
    Event: 100.675 loading class java/lang/Throwable$WrappedPrintStream done
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet done
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet done
    Event: 100.683 Thread 0x0000000002f22800 Uncommon trap: trap_request=0xffffff65 fr.pc=0x0000000003b2bdfc
    Event: 100.683 Thread 0x0000000002f22800 DEOPT PACKING pc=0x0000000003b2bdfc sp=0x000000000094c3d0
    Event: 100.683 Thread 0x0000000002f22800 DEOPT UNPACKING pc=0x000000000306582a sp=0x000000000094c360 mode 2


    Dynamic libraries:
    0x00007ff71b180000 - 0x00007ff71b1d4000  D:\CA\DevTest\bin\Portal.exe
    0x00007ffa95de0000 - 0x00007ffa95f8a000  C:\Windows\SYSTEM32\ntdll.dll
    0x00007ffa943b0000 - 0x00007ffa944ea000  C:\Windows\system32\KERNEL32.DLL
    0x00007ffa93550000 - 0x00007ffa9365f000  C:\Windows\system32\KERNELBASE.dll
    0x00007ffa937d0000 - 0x00007ffa93941000  C:\Windows\system32\USER32.dll
    0x00007ffa95d30000 - 0x00007ffa95dd5000  C:\Windows\system32\ADVAPI32.dll
    0x00007ffa90e00000 - 0x00007ffa9105a000  C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.9600.17031_none_6242a4b3ecbb55a1\COMCTL32.dll
    0x00007ffa93d20000 - 0x00007ffa93e65000  C:\Windows\system32\GDI32.dll
    0x00007ffa94120000 - 0x00007ffa941c7000  C:\Windows\system32\msvcrt.dll
    0x00007ffa94000000 - 0x00007ffa94057000  C:\Windows\SYSTEM32\sechost.dll
    0x00007ffa94730000 - 0x00007ffa94866000  C:\Windows\system32\RPCRT4.dll
    0x00007ffa93790000 - 0x00007ffa937c4000  C:\Windows\system32\IMM32.DLL
    0x00007ffa93950000 - 0x00007ffa93a89000  C:\Windows\system32\MSCTF.dll
    0x0000000059920000 - 0x000000005a1ba000  d:\ca\devtest\jre\bin\server\jvm.dll
    0x00007ffa886a0000 - 0x00007ffa886a9000  C:\Windows\SYSTEM32\WSOCK32.dll
    0x00007ffa865a0000 - 0x00007ffa865bf000  C:\Windows\SYSTEM32\WINMM.dll
    0x00007ffa86ca0000 - 0x00007ffa86caa000  C:\Windows\SYSTEM32\VERSION.dll
    0x00007ffa93aa0000 - 0x00007ffa93aa7000  C:\Windows\system32\PSAPI.DLL
    0x000000005a260000 - 0x000000005a332000  C:\Windows\SYSTEM32\MSVCR100.dll
    0x00007ffa94060000 - 0x00007ffa940b8000  C:\Windows\system32\WS2_32.dll
    0x00007ffa86570000 - 0x00007ffa8659a000  C:\Windows\SYSTEM32\WINMMBASE.dll
    0x00007ffa93a90000 - 0x00007ffa93a99000  C:\Windows\system32\NSI.dll
    0x00007ffa93450000 - 0x00007ffa9349a000  C:\Windows\SYSTEM32\cfgmgr32.dll
    0x00007ffa92040000 - 0x00007ffa92066000  C:\Windows\SYSTEM32\DEVOBJ.dll
    0x0000000059910000 - 0x000000005991f000  d:\ca\devtest\jre\bin\verify.dll
    0x00000000598e0000 - 0x0000000059909000  d:\ca\devtest\jre\bin\java.dll
    0x00000000598c0000 - 0x00000000598d6000  d:\ca\devtest\jre\bin\zip.dll
    0x00007ffa94910000 - 0x00007ffa95d27000  C:\Windows\system32\SHELL32.dll
    0x00007ffa944f0000 - 0x00007ffa946c6000  C:\Windows\SYSTEM32\combase.dll
    0x00007ffa940c0000 - 0x00007ffa94111000  C:\Windows\system32\SHLWAPI.dll
    0x00007ffa91810000 - 0x00007ffa918af000  C:\Windows\SYSTEM32\SHCORE.dll
    0x00007ffa93150000 - 0x00007ffa93164000  C:\Windows\SYSTEM32\profapi.dll
    0x0000000059890000 - 0x000000005989d000  D:\CA\DevTest\jre\bin\management.dll
    0x00000000598a0000 - 0x00000000598ba000  D:\CA\DevTest\jre\bin\net.dll
    0x00007ffa92a80000 - 0x00007ffa92ad8000  C:\Windows\system32\mswsock.dll
    0x0000000059870000 - 0x0000000059881000  D:\CA\DevTest\jre\bin\nio.dll
    0x00007ffa92ae0000 - 0x00007ffa92afe000  C:\Windows\SYSTEM32\CRYPTSP.dll
    0x00007ffa926f0000 - 0x00007ffa92725000  C:\Windows\system32\rsaenh.dll
    0x00007ffa92d10000 - 0x00007ffa92d36000  C:\Windows\SYSTEM32\bcrypt.dll
    0x00007ffa927e0000 - 0x00007ffa927fe000  C:\Windows\SYSTEM32\USERENV.dll
    0x00007ffa93000000 - 0x00007ffa93060000  C:\Windows\system32\bcryptprimitives.dll
    0x00007ffa93140000 - 0x00007ffa9314a000  C:\Windows\SYSTEM32\CRYPTBASE.dll
    0x00007ffa8fc80000 - 0x00007ffa8fca9000  C:\Windows\SYSTEM32\IPHLPAPI.DLL
    0x00007ffa8fc70000 - 0x00007ffa8fc7a000  C:\Windows\SYSTEM32\WINNSI.DLL
    0x00007ffa8f5f0000 - 0x00007ffa8f604000  C:\Windows\SYSTEM32\dhcpcsvc6.DLL
    0x00007ffa8f5d0000 - 0x00007ffa8f5e9000  C:\Windows\SYSTEM32\dhcpcsvc.DLL
    0x00007ffa8ded0000 - 0x00007ffa8dee4000  C:\Windows\system32\napinsp.dll
    0x00007ffa90680000 - 0x00007ffa90699000  C:\Windows\system32\NLAapi.dll
    0x00007ffa92890000 - 0x00007ffa92933000  C:\Windows\SYSTEM32\DNSAPI.dll
    0x00007ffa8dec0000 - 0x00007ffa8decc000  C:\Windows\System32\winrnr.dll
    0x00007ffa8f610000 - 0x00007ffa8f678000  C:\Windows\System32\fwpuclnt.dll
    0x00007ffa8de40000 - 0x00007ffa8de49000  C:\Windows\System32\rasadhlp.dll
    0x0000000059840000 - 0x0000000059864000  D:\CA\DevTest\jre\bin\sunec.dll

    VM Arguments:
    jvm_args: -Dexe4j.semaphoreName=Local\d:_ca_devtest_bin_portal.exe -Dexe4j.isInstall4j=true -Dexe4j.moduleName=D:\CA\DevTest\bin\Portal.exe -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Djava.library.path=D:\CA\DevTest\bin\..\bin;D:\CA\DevTest\bin\..\lib\native;D:\TimesTent11.2.2\bin;D:\TimesTent11.2.2\ttoracle_home\instantclient_11_2;D:\app\devtest\product\12.1.0\dbhome_1\bin;D:\app\devt\product\12.1.0\dbhome_1\bin;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\CA\DevTest\lib\core;D:\CA\DevTest\bin;d:\ca\devtest\jre\bin -Dexe4j.consoleCodepage=cp936 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\CA\DevTest\bin\/../tmp -Djava.endorsed.dirs=D:\CA\DevTest\bin\/../lib/endorsed -DLISA_HOME=D:\CA\DevTest\bin\/../ -DLISA_LOG=portal.log -Xms24m -Xmx1024m -Dfile.encoding=UTF-8 -Dorg.springframework.boot.logging.LoggingSystem=com.ca.devtest.springframework.boot.logging.NonConfiguringWebAppLoggingSystem -Djava.net.preferIPv4Stack=true ${LISA_MORE_VM_PROPS} -Dlisa.version.number=10.0.0 -Dinstall4j.launcherId=12902 -Dinstall4j.swt=false -Xms40960m -Xmx81920m
    java_command: D:\CA\DevTest\bin\Portal.exe
    java_class_path (initial): D:\CA\DevTest\.install4j\i4jruntime.jar;D:\CA\DevTest\bin\..\lib\phoenix\phoenix-runner.jar;D:\CA\DevTest\bin\..\lib\shared\owner-1.0.6.jar;D:\CA\DevTest\bin\..\lib\shared\jetty-all-9.3.11.v20160721-uber.jar;D:\CA\DevTest\bin\..\lib\shared\javax.servlet-api-*.jar;D:\CA\DevTest\bin\..\lib\shared\slf4j-api-1.7.7.jar;D:\CA\DevTest\bin\..\lib\shared\slf4j-log4j12-1.7.7.jar;D:\CA\DevTest\bin\..\lib\shared\log4j-1.2.17.jar;D:\CA\DevTest\bin\..\lib\shared\commons-logging-1.2.jar;D:\CA\DevTest\bin\..\lib\shared\commons-codec-1.9.jar;D:\CA\DevTest\bin\..\lib\shared\bcprov-jdk15on-1.54.jar;D:\CA\DevTest\bin\..\lib\shared\asm-5.0.1.jar;
    Launcher Type: generic

    Environment Variables:
    PATH=D:\CA\DevTest\bin\..\bin;D:\CA\DevTest\bin\..\lib\native;D:\TimesTent11.2.2\bin;D:\TimesTent11.2.2\ttoracle_home\instantclient_11_2;D:\app\devtest\product\12.1.0\dbhome_1\bin;D:\app\devt\product\12.1.0\dbhome_1\bin;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPo

    #
    # There is insufficient memory for the Java Runtime Environment to continue.
    # Native memory allocation (malloc) failed to allocate 34816 bytes for Chunk::new
    # Possible reasons:
    #   The system is out of physical RAM or swap space
    #   In 32 bit mode, the process size limit was hit
    # Possible solutions:
    #   Reduce memory load on the system
    #   Increase physical memory or swap space
    #   Check if swap backing store is full
    #   Use 64 bit Java on a 64 bit OS
    #   Decrease Java heap size (-Xmx/-Xms)
    #   Decrease number of Java threads
    #   Decrease Java thread stack sizes (-Xss)
    #   Set larger code cache with -XX:ReservedCodeCacheSize=
    # This output file may be truncated or incomplete.
    #
    #  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208
    #
    # JRE version: Java(TM) SE Runtime Environment (8.0_102-b14) (build 1.8.0_102-b14)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.102-b14 mixed mode windows-amd64 )
    # Failed to write core dump.
    #

    ---------------  T H R E A D  ---------------

    Current thread (0x0000000034407000):  JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=29192, stack(0x000000003a850000,0x000000003a950000)]

    Stack: [0x000000003a850000,0x000000003a950000]
    [error occurred during error reporting (printing stack bounds), id 0xc0000005]

    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)


    Current CompileTask:
    C2: 100700 12303       4       sun.nio.fs.WindowsPath::hashCode (52 bytes)


    ---------------  P R O C E S S  ---------------

    Java Threads: ( => current thread )
      0x0000001577749000 JavaThread "elasticsearch[Dr. Otto Octavius][management][T#2]" daemon [_thread_blocked, id=26476, stack(0x00000015b6da0000,0x00000015b6ea0000)]
      0x0000001577748800 JavaThread "ServoPollScheduler-9" daemon [_thread_blocked, id=12804, stack(0x00000015b6ca0000,0x00000015b6da0000)]
      0x0000001577744800 JavaThread "ServoPollScheduler-8" daemon [_thread_blocked, id=11196, stack(0x00000015b6ba0000,0x00000015b6ca0000)]
      0x0000001577747800 JavaThread "ServoPollScheduler-7" daemon [_thread_blocked, id=9088, stack(0x00000015b69a0000,0x00000015b6aa0000)]
      0x0000001577746000 JavaThread "ServoPollScheduler-6" daemon [_thread_blocked, id=20284, stack(0x00000015b70a0000,0x00000015b71a0000)]
      0x0000001577745800 JavaThread "qtp1879034789-198" [_thread_blocked, id=4788, stack(0x00000015b64f0000,0x00000015b65f0000)]
      0x000000157773c800 JavaThread "org.eclipse.jetty.server.session.HashSessionManager@3847fe81Timer" daemon [_thread_blocked, id=25596, stack(0x00000015b67a0000,0x00000015b68a0000)]
      0x000000157773f800 JavaThread "threadPoolExecutor-6" daemon [_thread_blocked, id=26556, stack(0x00000015b66a0000,0x00000015b67a0000)]
      0x000000157773e800 JavaThread "threadPoolExecutor-5" daemon [_thread_blocked, id=10588, stack(0x00000015b61f0000,0x00000015b62f0000)]
      0x000000157773e000 JavaThread "threadPoolExecutor-4" daemon [_thread_blocked, id=28880, stack(0x00000015b60f0000,0x00000015b61f0000)]
      0x0000001577738800 JavaThread "ServoPollScheduler-5" daemon [_thread_blocked, id=15944, stack(0x00000015b5ff0000,0x00000015b60f0000)]
      0x000000157773b800 JavaThread "threadPoolExecutor-3" daemon [_thread_blocked, id=28872, stack(0x00000015b5d80000,0x00000015b5e80000)]
      0x0000001577738000 JavaThread "threadPoolExecutor-2" daemon [_thread_blocked, id=20040, stack(0x00000015b5c80000,0x00000015b5d80000)]
      0x0000001577739800 JavaThread "threadPoolExecutor-1" daemon [_thread_blocked, id=9428, stack(0x00000015b5980000,0x00000015b5a80000)]
      0x000000157773b000 JavaThread "ServoPollScheduler-4" daemon [_thread_blocked, id=15984, stack(0x00000015b5880000,0x00000015b5980000)]
      0x0000001577737000 JavaThread "elasticsearch[Dr. Otto Octavius][listener][T#3]" daemon [_thread_blocked, id=14400, stack(0x00000015b5780000,0x00000015b5880000)]
      0x0000001577733800 JavaThread "elasticsearch[Dr. Otto Octavius][listener][T#2]" daemon [_thread_blocked, id=21196, stack(0x00000015b5480000,0x00000015b5580000)]
      0x0000001577735000 JavaThread "elasticsearch[Dr. Otto Octavius][listener][T#1]" daemon [_thread_blocked, id=22876, stack(0x00000015b5380000,0x00000015b5480000)]
      0x0000001577734000 JavaThread "elasticsearch[Dr. Otto Octavius][warmer][T#1]" daemon [_thread_blocked, id=27088, stack(0x00000015b5030000,0x00000015b5130000)]
      0x0000001577735800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_boss][T#1]{New I/O server boss #65}" daemon [_thread_in_native, id=28628, stack(0x00000015b4d30000,0x00000015b4e30000)]
      0x000000157772f800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#64]{New I/O worker #64}" daemon [_thread_in_native, id=22748, stack(0x00000015b4c30000,0x00000015b4d30000)]
      0x0000001577730800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#63]{New I/O worker #63}" daemon [_thread_in_native, id=24424, stack(0x00000015b4b30000,0x00000015b4c30000)]
      0x0000001577732800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#62]{New I/O worker #62}" daemon [_thread_in_native, id=27944, stack(0x00000015b4a30000,0x00000015b4b30000)]
      0x000000157772f000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#61]{New I/O worker #61}" daemon [_thread_in_native, id=22348, stack(0x00000015b4930000,0x00000015b4a30000)]
      0x0000001577732000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#60]{New I/O worker #60}" daemon [_thread_in_native, id=24588, stack(0x00000015b4830000,0x00000015b4930000)]
      0x0000001577731000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#59]{New I/O worker #59}" daemon [_thread_in_native, id=24500, stack(0x00000015b4730000,0x00000015b4830000)]
      0x000000157772e000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#58]{New I/O worker #58}" daemon [_thread_in_native, id=13032, stack(0x00000015b4630000,0x00000015b4730000)]
      0x000000157772a800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#57]{New I/O worker #57}" daemon [_thread_in_native, id=22020, stack(0x00000015b4530000,0x00000015b4630000)]
      0x000000157772c800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#56]{New I/O worker #56}" daemon [_thread_in_native, id=7188, stack(0x00000015b4430000,0x00000015b4530000)]
      0x000000157772c000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#55]{New I/O worker #55}" daemon [_thread_in_native, id=16000, stack(0x00000015b4330000,0x00000015b4430000)]
      0x000000157772b000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#54]{New I/O worker #54}" daemon [_thread_in_native, id=9936, stack(0x00000015b4230000,0x00000015b4330000)]
      0x000000157772d800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#53]{New I/O worker #53}" daemon [_thread_in_native, id=15308, stack(0x00000015b4130000,0x00000015b4230000)]
      0x0000001577858800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#52]{New I/O worker #52}" daemon [_thread_in_native, id=22120, stack(0x00000015b4030000,0x00000015b4130000)]
      0x0000001577858000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#51]{New I/O worker #51}" daemon [_thread_in_native, id=12560, stack(0x00000015b3f30000,0x00000015b4030000)]
      0x0000001577859800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#50]{New I/O worker #50}" daemon [_thread_in_native, id=14828, stack(0x00000015b3e30000,0x00000015b3f30000)]
      0x0000001577855800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#49]{New I/O worker #49}" daemon [_thread_in_native, id=29516, stack(0x00000015b3d30000,0x00000015b3e30000)]
      0x0000001577855000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#48]{New I/O worker #48}" daemon [_thread_in_native, id=27228, stack(0x00000015b3c30000,0x00000015b3d30000)]
      0x0000001577857000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#47]{New I/O worker #47}" daemon [_thread_in_native, id=25140, stack(0x00000015b3b30000,0x00000015b3c30000)]
      0x0000001577856800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#46]{New I/O worker #46}" daemon [_thread_in_native, id=27876, stack(0x00000015b3a30000,0x00000015b3b30000)]
      0x0000001577852000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#45]{New I/O worker #45}" daemon [_thread_in_native, id=20964, stack(0x00000015b3930000,0x00000015b3a30000)]
      0x0000001577854000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#44]{New I/O worker #44}" daemon [_thread_in_native, id=22292, stack(0x00000015b3830000,0x00000015b3930000)]
      0x0000001577850800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#43]{New I/O worker #43}" daemon [_thread_in_native, id=28524, stack(0x00000015b3730000,0x00000015b3830000)]
      0x0000001577852800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#42]{New I/O worker #42}" daemon [_thread_in_native, id=12516, stack(0x00000015b3630000,0x00000015b3730000)]
      0x0000001577853800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#41]{New I/O worker #41}" daemon [_thread_in_native, id=29688, stack(0x00000015b3530000,0x00000015b3630000)]
      0x0000001577851000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#40]{New I/O worker #40}" daemon [_thread_in_native, id=27164, stack(0x00000015b3430000,0x00000015b3530000)]
      0x000000157784f800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#39]{New I/O worker #39}" daemon [_thread_in_native, id=428, stack(0x00000015b3330000,0x00000015b3430000)]
      0x000000157784c000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#38]{New I/O worker #38}" daemon [_thread_in_native, id=26536, stack(0x00000015b3230000,0x00000015b3330000)]
      0x000000157784f000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#37]{New I/O worker #37}" daemon [_thread_in_native, id=15284, stack(0x00000015b3130000,0x00000015b3230000)]
      0x000000157784e000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#36]{New I/O worker #36}" daemon [_thread_in_native, id=19896, stack(0x00000015b3030000,0x00000015b3130000)]
      0x000000157784c800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#35]{New I/O worker #35}" daemon [_thread_in_native, id=22708, stack(0x00000015b2f30000,0x00000015b3030000)]
      0x000000157784b000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#34]{New I/O worker #34}" daemon [_thread_in_native, id=948, stack(0x00000015b2e30000,0x00000015b2f30000)]
      0x000000157784a800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#33]{New I/O worker #33}" daemon [_thread_in_native, id=29432, stack(0x00000015b2d30000,0x00000015b2e30000)]
      0x000000157784d800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#32]{New I/O worker #32}" daemon [_thread_in_native, id=17848, stack(0x00000015b2c30000,0x00000015b2d30000)]
      0x000000003e322000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#31]{New I/O worker #31}" daemon [_thread_in_native, id=18408, stack(0x00000015b2b30000,0x00000015b2c30000)]
      0x000000003e325000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#30]{New I/O worker #30}" daemon [_thread_in_native, id=27776, stack(0x00000015b2a30000,0x00000015b2b30000)]
      0x000000003e323800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#29]{New I/O worker #29}" daemon [_thread_in_native, id=16720, stack(0x00000015b2930000,0x00000015b2a30000)]
      0x000000003e324800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#28]{New I/O worker #28}" daemon [_thread_in_native, id=17288, stack(0x00000015b2830000,0x00000015b2930000)]
      0x000000003e323000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#27]{New I/O worker #27}" daemon [_thread_in_native, id=6392, stack(0x00000015b2730000,0x00000015b2830000)]
      0x000000003e31f000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#26]{New I/O worker #26}" daemon [_thread_in_native, id=24584, stack(0x00000015b2630000,0x00000015b2730000)]
      0x000000003e31e800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#25]{New I/O worker #25}" daemon [_thread_in_native, id=14360, stack(0x00000015b2530000,0x00000015b2630000)]
      0x000000003e320800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#24]{New I/O worker #24}" daemon [_thread_in_native, id=13636, stack(0x00000015b2430000,0x00000015b2530000)]
      0x000000003e320000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#23]{New I/O worker #23}" daemon [_thread_in_native, id=17912, stack(0x00000015b2330000,0x00000015b2430000)]
      0x000000003e321800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#22]{New I/O worker #22}" daemon [_thread_in_native, id=5280, stack(0x00000015b2230000,0x00000015b2330000)]
      0x000000156d301800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#21]{New I/O worker #21}" daemon [_thread_in_native, id=28016, stack(0x00000015b2130000,0x00000015b2230000)]
      0x000000156d302800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#20]{New I/O worker #20}" daemon [_thread_in_native, id=27652, stack(0x00000015b2030000,0x00000015b2130000)]
      0x000000156d301000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#19]{New I/O worker #19}" daemon [_thread_in_native, id=21556, stack(0x00000015b1f30000,0x00000015b2030000)]
      0x000000156d2fd000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#18]{New I/O worker #18}" daemon [_thread_in_native, id=20412, stack(0x00000015b1e30000,0x00000015b1f30000)]
      0x000000156d300000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#17]{New I/O worker #17}" daemon [_thread_in_native, id=20364, stack(0x00000015b1d30000,0x00000015b1e30000)]
      0x000000156d2ff800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#16]{New I/O worker #16}" daemon [_thread_in_native, id=20672, stack(0x00000015b1c30000,0x00000015b1d30000)]
      0x000000156d2fc800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#15]{New I/O worker #15}" daemon [_thread_in_native, id=5932, stack(0x00000015b1b30000,0x00000015b1c30000)]
      0x000000156d2fe800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#14]{New I/O worker #14}" daemon [_thread_in_native, id=8384, stack(0x00000015b1a30000,0x00000015b1b30000)]
      0x000000156d2fb800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#13]{New I/O worker #13}" daemon [_thread_in_native, id=29220, stack(0x00000015b1930000,0x00000015b1a30000)]
      0x000000156d2fe000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#12]{New I/O worker #12}" daemon [_thread_in_native, id=26952, stack(0x00000015b1830000,0x00000015b1930000)]
      0x000000156d2fa000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#11]{New I/O worker #11}" daemon [_thread_in_native, id=26948, stack(0x00000015b1730000,0x00000015b1830000)]
      0x000000156d2fb000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#10]{New I/O worker #10}" daemon [_thread_in_native, id=1228, stack(0x00000015b1630000,0x00000015b1730000)]
      0x000000156d2f8800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#9]{New I/O worker #9}" daemon [_thread_in_native, id=7164, stack(0x00000015b1530000,0x00000015b1630000)]
      0x000000156d2f8000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#8]{New I/O worker #8}" daemon [_thread_in_native, id=2356, stack(0x00000015b1430000,0x00000015b1530000)]
      0x000000156d2f9800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#7]{New I/O worker #7}" daemon [_thread_in_native, id=21944, stack(0x00000015b1330000,0x00000015b1430000)]
      0x000000156d2f4000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#6]{New I/O worker #6}" daemon [_thread_in_native, id=28732, stack(0x00000015b1230000,0x00000015b1330000)]
      0x000000156d2f7000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#5]{New I/O worker #5}" daemon [_thread_in_native, id=11148, stack(0x00000015b1130000,0x00000015b1230000)]
      0x000000156d2f3800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#4]{New I/O worker #4}" daemon [_thread_in_native, id=22080, stack(0x00000015b1030000,0x00000015b1130000)]
      0x000000156d2f5000 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#3]{New I/O worker #3}" daemon [_thread_in_native, id=28120, stack(0x00000015b0f30000,0x00000015b1030000)]
      0x000000156d2f6800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#2]{New I/O worker #2}" daemon [_thread_in_native, id=13080, stack(0x00000015b0e30000,0x00000015b0f30000)]
      0x000000156d2f5800 JavaThread "elasticsearch[Dr. Otto Octavius][http_server_worker][T#1]{New I/O worker #1}" daemon [_thread_in_native, id=7908, stack(0x00000015b0d30000,0x00000015b0e30000)]
      0x000000158f52b000 JavaThread "elasticsearch[Dr. Otto Octavius][management][T#1]" daemon [_thread_blocked, id=5920, stack(0x00000015b0c30000,0x00000015b0d30000)]
      0x00000015a676c000 JavaThread "elasticsearch[Dr. Otto Octavius][riverClusterService#updateTask][T#1]" daemon [_thread_blocked, id=23628, stack(0x00000015b0b30000,0x00000015b0c30000)]
      0x00000015a676b000 JavaThread "elasticsearch[Dr. Otto Octavius][generic][T#1]" daemon [_thread_blocked, id=13320, stack(0x00000015b0830000,0x00000015b0930000)]
      0x00000015a676a800 JavaThread "elasticsearch[Dr. Otto Octavius][clusterService#updateTask][T#1]" daemon [_thread_blocked, id=21312, stack(0x00000015b0730000,0x00000015b0830000)]
      0x0000001593a1f800 JavaThread "elasticsearch[Dr. Otto Octavius][[ttl_expire]]" daemon [_thread_blocked, id=22972, stack(0x00000015b0630000,0x00000015b0730000)]
      0x0000001593a1e800 JavaThread "elasticsearch[Dr. Otto Octavius][master_mapping_updater]" [_thread_blocked, id=10860, stack(0x00000015b0530000,0x00000015b0630000)]
      0x0000001593a1b800 JavaThread "elasticsearch[Dr. Otto Octavius][scheduler][T#1]" daemon [_thread_blocked, id=12792, stack(0x00000015b0430000,0x00000015b0530000)]
      0x0000001593a1d000 JavaThread "ServoPollScheduler-3" daemon [_thread_blocked, id=28020, stack(0x00000015af280000,0x00000015af380000)]
      0x0000001593a1e000 JavaThread "elasticsearch[Dr. Otto Octavius][[timer]]" daemon [_thread_blocked, id=4556, stack(0x000000003d020000,0x000000003d120000)]
      0x0000001593a1c800 JavaThread "Actor Thread 1" daemon [_thread_blocked, id=7096, stack(0x0000000075110000,0x0000000075210000)]
      0x0000001593a17000 JavaThread "ServoPollScheduler-2" daemon [_thread_blocked, id=20556, stack(0x0000000071070000,0x0000000071170000)]
      0x0000001593a15800 JavaThread "ServoPollScheduler-1" daemon [_thread_blocked, id=24300, stack(0x00000015a8c20000,0x00000015a8d20000)]
      0x0000001593a12000 JavaThread "ServoPollScheduler-0" daemon [_thread_blocked, id=29672, stack(0x000000007fea0000,0x000000007ffa0000)]
      0x0000001593a15000 JavaThread "SimpleHostRoutingFilter.CONNECTION_MANAGER_TIMER" daemon [_thread_blocked, id=29236, stack(0x00000000582f0000,0x00000000583f0000)]
      0x000000158f526800 JavaThread "__DEFAULT__" daemon [_thread_blocked, id=26280, stack(0x000000006bfe0000,0x000000006c0e0000)]
      0x000000158f528000 JavaThread "Tomcat JDBC Pool Cleaner[724125922:1506335175053]" daemon [_thread_blocked, id=4456, stack(0x0000000054290000,0x0000000054390000)]
      0x00000015a6760000 JavaThread "FileWatchdog" daemon [_thread_blocked, id=24328, stack(0x00000000423f0000,0x00000000424f0000)]
      0x000000003e13f000 JavaThread "qtp1879034789-94" [_thread_blocked, id=12484, stack(0x00000000424f0000,0x00000000425f0000)]
      0x000000003e13b000 JavaThread "qtp1879034789-92" [_thread_blocked, id=568, stack(0x00000000422f0000,0x00000000423f0000)]
      0x000000003e13e000 JavaThread "qtp1879034789-91" [_thread_blocked, id=17244, stack(0x00000000421f0000,0x00000000422f0000)]
      0x000000003e13a800 JavaThread "qtp1879034789-90" [_thread_blocked, id=25836, stack(0x00000000420f0000,0x00000000421f0000)]
      0x000000003e13d800 JavaThread "qtp1879034789-89" [_thread_blocked, id=12752, stack(0x0000000041ff0000,0x00000000420f0000)]
      0x000000003e139800 JavaThread "qtp1879034789-88" [_thread_blocked, id=29152, stack(0x0000000041ef0000,0x0000000041ff0000)]
      0x000000003e13c800 JavaThread "qtp1879034789-87" [_thread_blocked, id=6348, stack(0x0000000041df0000,0x0000000041ef0000)]
      0x000000003e139000 JavaThread "qtp1879034789-86" [_thread_blocked, id=17000, stack(0x0000000041cf0000,0x0000000041df0000)]
      0x000000003e13c000 JavaThread "qtp1879034789-85" [_thread_blocked, id=21816, stack(0x0000000041bf0000,0x0000000041cf0000)]
      0x000000003e138000 JavaThread "qtp1879034789-84" [_thread_blocked, id=24008, stack(0x00000000418f0000,0x00000000419f0000)]
      0x000000003e135000 JavaThread "qtp1879034789-83" [_thread_blocked, id=14128, stack(0x00000000417f0000,0x00000000418f0000)]
      0x000000003e134800 JavaThread "qtp1879034789-82" [_thread_blocked, id=27428, stack(0x00000000416f0000,0x00000000417f0000)]
      0x000000003e137800 JavaThread "qtp1879034789-81" [_thread_blocked, id=20996, stack(0x00000000415f0000,0x00000000416f0000)]
      0x000000003e136000 JavaThread "qtp1879034789-80" [_thread_blocked, id=28476, stack(0x00000000414f0000,0x00000000415f0000)]
      0x000000003e133800 JavaThread "qtp1879034789-79" [_thread_blocked, id=16964, stack(0x00000000413f0000,0x00000000414f0000)]
      0x000000003e136800 JavaThread "qtp1879034789-78" [_thread_blocked, id=3672, stack(0x00000000412f0000,0x00000000413f0000)]
      0x000000003e0c7000 JavaThread "qtp1879034789-77" [_thread_blocked, id=24628, stack(0x00000000411f0000,0x00000000412f0000)]
      0x000000003e0c6800 JavaThread "qtp1879034789-76" [_thread_blocked, id=11804, stack(0x00000000410f0000,0x00000000411f0000)]
      0x000000003e0c5800 JavaThread "qtp1879034789-75" [_thread_blocked, id=28996, stack(0x0000000040ff0000,0x00000000410f0000)]
      0x000000003e0c5000 JavaThread "qtp1879034789-74" [_thread_blocked, id=7240, stack(0x0000000040ef0000,0x0000000040ff0000)]
      0x000000003e0c3800 JavaThread "qtp1879034789-73" [_thread_blocked, id=23144, stack(0x0000000040df0000,0x0000000040ef0000)]
      0x000000003e0c2800 JavaThread "qtp1879034789-72" [_thread_blocked, id=13248, stack(0x0000000040cf0000,0x0000000040df0000)]
      0x000000003e0c4000 JavaThread "qtp1879034789-71" [_thread_blocked, id=11736, stack(0x0000000040bf0000,0x0000000040cf0000)]
      0x000000003e0bf000 JavaThread "qtp1879034789-70" [_thread_blocked, id=15844, stack(0x0000000040af0000,0x0000000040bf0000)]
      0x000000003e0c2000 JavaThread "qtp1879034789-69" [_thread_blocked, id=18096, stack(0x00000000409f0000,0x0000000040af0000)]
      0x000000003e0bf800 JavaThread "qtp1879034789-68" [_thread_blocked, id=18524, stack(0x00000000408f0000,0x00000000409f0000)]
      0x000000003e0be000 JavaThread "qtp1879034789-67" [_thread_blocked, id=21560, stack(0x00000000407f0000,0x00000000408f0000)]
      0x000000003e0c1000 JavaThread "qtp1879034789-66" [_thread_blocked, id=3608, stack(0x00000000406f0000,0x00000000407f0000)]
      0x000000003e0c0800 JavaThread "qtp1879034789-65" [_thread_blocked, id=7048, stack(0x00000000405f0000,0x00000000406f0000)]
      0x000000003e0bd800 JavaThread "qtp1879034789-64" [_thread_blocked, id=24096, stack(0x00000000404f0000,0x00000000405f0000)]
      0x000000003e0b9800 JavaThread "qtp1879034789-63" [_thread_blocked, id=25260, stack(0x00000000403f0000,0x00000000404f0000)]
      0x000000003e0bc800 JavaThread "qtp1879034789-62" [_thread_blocked, id=27012, stack(0x00000000402f0000,0x00000000403f0000)]
      0x000000003e0ba800 JavaThread "qtp1879034789-61" [_thread_blocked, id=18200, stack(0x00000000401f0000,0x00000000402f0000)]
      0x000000003e0bb000 JavaThread "qtp1879034789-60" [_thread_blocked, id=26020, stack(0x00000000400f0000,0x00000000401f0000)]
      0x000000003e0bc000 JavaThread "qtp1879034789-59" [_thread_blocked, id=23656, stack(0x000000003fff0000,0x00000000400f0000)]
      0x000000003e0b9000 JavaThread "qtp1879034789-58" [_thread_blocked, id=19780, stack(0x000000003fef0000,0x000000003fff0000)]
      0x000000003e0b8000 JavaThread "qtp1879034789-57" [_thread_blocked, id=4460, stack(0x000000003fdf0000,0x000000003fef0000)]
      0x000000003be32000 JavaThread "qtp1879034789-56" [_thread_blocked, id=10736, stack(0x000000003fcf0000,0x000000003fdf0000)]
      0x000000003be33000 JavaThread "qtp1879034789-55" [_thread_blocked, id=8252, stack(0x000000003fbf0000,0x000000003fcf0000)]
      0x000000003be2e800 JavaThread "qtp1879034789-54" [_thread_blocked, id=28304, stack(0x000000003faf0000,0x000000003fbf0000)]
      0x000000003be31800 JavaThread "qtp1879034789-53" [_thread_blocked, id=12104, stack(0x000000003f9f0000,0x000000003faf0000)]
      0x000000003be30800 JavaThread "qtp1879034789-52" [_thread_blocked, id=23828, stack(0x000000003f8f0000,0x000000003f9f0000)]
      0x000000003be2f000 JavaThread "qtp1879034789-51" [_thread_blocked, id=26132, stack(0x000000003f7f0000,0x000000003f8f0000)]
      0x000000003be30000 JavaThread "qtp1879034789-50" [_thread_blocked, id=3632, stack(0x000000003f6f0000,0x000000003f7f0000)]
      0x000000003be2a800 JavaThread "qtp1879034789-49" [_thread_blocked, id=17164, stack(0x000000003f5f0000,0x000000003f6f0000)]
      0x000000003be2d000 JavaThread "qtp1879034789-48" [_thread_blocked, id=21492, stack(0x000000003f4f0000,0x000000003f5f0000)]
      0x000000003be2d800 JavaThread "qtp1879034789-47" [_thread_blocked, id=20032, stack(0x000000003f3f0000,0x000000003f4f0000)]
      0x000000003be2c000 JavaThread "qtp1879034789-46" [_thread_blocked, id=5980, stack(0x000000003f2f0000,0x000000003f3f0000)]
      0x000000003be2b800 JavaThread "qtp1879034789-45" [_thread_blocked, id=9924, stack(0x000000003f1f0000,0x000000003f2f0000)]
      0x000000003be2a000 JavaThread "qtp1879034789-44" [_thread_blocked, id=15000, stack(0x000000003f0f0000,0x000000003f1f0000)]
      0x000000003be29000 JavaThread "qtp1879034789-43" [_thread_blocked, id=25408, stack(0x000000003eff0000,0x000000003f0f0000)]
      0x000000003be25000 JavaThread "qtp1879034789-42" [_thread_blocked, id=29588, stack(0x000000003eef0000,0x000000003eff0000)]
      0x000000003be28800 JavaThread "qtp1879034789-41" [_thread_blocked, id=6388, stack(0x000000003edf0000,0x000000003eef0000)]
      0x000000003be24800 JavaThread "qtp1879034789-40" [_thread_blocked, id=10996, stack(0x000000003ecf0000,0x000000003edf0000)]
      0x000000003be27800 JavaThread "qtp1879034789-39" [_thread_blocked, id=23580, stack(0x000000003ebf0000,0x000000003ecf0000)]
      0x000000003be23800 JavaThread "qtp1879034789-38" [_thread_blocked, id=9096, stack(0x000000003eaf0000,0x000000003ebf0000)]
      0x000000003be26000 JavaThread "qtp1879034789-37" [_thread_blocked, id=8156, stack(0x000000003e9f0000,0x000000003eaf0000)]
      0x000000003bd16000 JavaThread "qtp1879034789-36" [_thread_blocked, id=25508, stack(0x000000003d720000,0x000000003d820000)]
      0x000000003bd15000 JavaThread "qtp1879034789-35" [_thread_blocked, id=14536, stack(0x000000003d620000,0x000000003d720000)]
      0x000000003bd14000 JavaThread "qtp1879034789-34" [_thread_blocked, id=2024, stack(0x000000003d520000,0x000000003d620000)]
      0x000000003bd8b000 JavaThread "qtp1879034789-33" [_thread_blocked, id=10432, stack(0x000000003d420000,0x000000003d520000)]
      0x000000003bd8a800 JavaThread "qtp1879034789-32" [_thread_blocked, id=11444, stack(0x000000003d320000,0x000000003d420000)]
      0x000000003bd89800 JavaThread "qtp1879034789-31" [_thread_blocked, id=8552, stack(0x000000003d220000,0x000000003d320000)]
      0x000000003bd89000 JavaThread "qtp1879034789-30" [_thread_blocked, id=12412, stack(0x000000003d120000,0x000000003d220000)]
      0x0000000036af0000 JavaThread "FileWatchdog" daemon [_thread_blocked, id=11548, stack(0x000000003cd20000,0x000000003ce20000)]
      0x000000003bc84000 JavaThread "FileWatchdog" daemon [_thread_blocked, id=15212, stack(0x000000003cc20000,0x000000003cd20000)]
      0x000000003669a800 JavaThread "Service Thread" daemon [_thread_blocked, id=17892, stack(0x000000003ba50000,0x000000003bb50000)]
      0x0000000036601000 JavaThread "C1 CompilerThread17" daemon [_thread_blocked, id=25788, stack(0x000000003b950000,0x000000003ba50000)]
      0x00000000365ff800 JavaThread "C1 CompilerThread16" daemon [_thread_blocked, id=3312, stack(0x000000003b850000,0x000000003b950000)]
      0x00000000365ff000 JavaThread "C1 CompilerThread15" daemon [_thread_blocked, id=20440, stack(0x000000003b750000,0x000000003b850000)]
      0x00000000365fb000 JavaThread "C1 CompilerThread14" daemon [_thread_blocked, id=11200, stack(0x000000003b650000,0x000000003b750000)]
      0x00000000365fb800 JavaThread "C1 CompilerThread13" daemon [_thread_blocked, id=22536, stack(0x000000003b550000,0x000000003b650000)]
      0x00000000365fe000 JavaThread "C1 CompilerThread12" daemon [_thread_blocked, id=24380, stack(0x000000003b450000,0x000000003b550000)]
      0x00000000365fd000 JavaThread "C2 CompilerThread11" daemon [_thread_blocked, id=15276, stack(0x000000003b350000,0x000000003b450000)]
      0x00000000365fc800 JavaThread "C2 CompilerThread10" daemon [_thread_blocked, id=23172, stack(0x000000003b250000,0x000000003b350000)]
      0x00000000365fa000 JavaThread "C2 CompilerThread9" daemon [_thread_blocked, id=23640, stack(0x000000003b150000,0x000000003b250000)]
      0x00000000365f1800 JavaThread "C2 CompilerThread8" daemon [_thread_blocked, id=28040, stack(0x000000003b050000,0x000000003b150000)]
      0x00000000365cd800 JavaThread "C2 CompilerThread7" daemon [_thread_blocked, id=26432, stack(0x000000003af50000,0x000000003b050000)]
      0x000000003441e000 JavaThread "C2 CompilerThread6" daemon [_thread_blocked, id=6968, stack(0x000000003ae50000,0x000000003af50000)]
      0x00000000309da000 JavaThread "C2 CompilerThread5" daemon [_thread_blocked, id=9324, stack(0x000000003ad50000,0x000000003ae50000)]
      0x0000000034419800 JavaThread "C2 CompilerThread4" daemon [_thread_blocked, id=3476, stack(0x000000003ac50000,0x000000003ad50000)]
      0x0000000034418800 JavaThread "C2 CompilerThread3" daemon [_thread_blocked, id=22068, stack(0x000000003ab50000,0x000000003ac50000)]
      0x0000000034418000 JavaThread "C2 CompilerThread2" daemon [_thread_blocked, id=20540, stack(0x000000003aa50000,0x000000003ab50000)]
      0x0000000034417000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=17360, stack(0x000000003a950000,0x000000003aa50000)]
    =>0x0000000034407000 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=29192, stack(0x000000003a850000,0x000000003a950000)]
      0x000000003657c000 JavaThread "Attach Listener" daemon [_thread_blocked, id=3168, stack(0x000000003a750000,0x000000003a850000)]
      0x000000003657b800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5192, stack(0x000000003a650000,0x000000003a750000)]
      0x00000000343e7800 JavaThread "Finalizer" daemon [_thread_blocked, id=12236, stack(0x000000003a450000,0x000000003a550000)]
      0x00000000343e1800 JavaThread "Reference Handler" daemon [_thread_blocked, id=5200, stack(0x000000003a350000,0x000000003a450000)]
      0x0000000002f22800 JavaThread "main" [_thread_in_vm, id=19580, stack(0x0000000000850000,0x0000000000950000)]

    Other Threads:
      0x00000000309c0800 VMThread [stack: 0x000000003a250000,0x000000003a350000] [id=27916]
      0x00000000366d0800 WatcherThread [stack: 0x000000003bb50000,0x000000003bc50000] [id=24348]

    VM state:not at safepoint (normal execution)

    VM Mutex/Monitor currently owned by a thread: None

    Heap:
     PSYoungGen      total 16091136K, used 9929630K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 69% used [0x0000000dd5580000,0x00000010336678d8,0x0000001140d00000)
      from space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 46550528K, used 306079K [0x0000000080000000, 0x0000000b99380000, 0x0000000dd5580000)
      object space 46550528K, 0% used [0x0000000080000000,0x0000000092ae7f70,0x0000000b99380000)
     Metaspace       used 115068K, capacity 117258K, committed 117504K, reserved 118784K

    Card table byte_map: [0x00000000123e0000,0x000000001c3f0000] byte_map_base: 0x0000000011fe0000

    Marking Bits: (ParMarkBitMap*) 0x000000005a13a6c0
     Begin Bits: [0x0000001480000000, 0x00000014d0000000)
     End Bits:   [0x00000014d0000000, 0x0000001520000000)

    Polling page: 0x00000000009b0000

    CodeCache: size=245760Kb used=46370Kb max_used=46370Kb free=199389Kb
     bounds [0x0000000003020000, 0x0000000005e20000, 0x0000000012020000]
     total_blobs=10504 nmethods=9603 adapters=805
     compilation: enabled

    Compilation events (10 events):
    Event: 100.674 Thread 0x00000000365fe000 nmethod 12297 0x000000000517e510 code [0x000000000517e6a0, 0x000000000517eb38]
    Event: 100.674 Thread 0x00000000365fb800 nmethod 12298 0x0000000003afa4d0 code [0x0000000003afa660, 0x0000000003afaa98]
    Event: 100.677 Thread 0x00000000365ff000 12299       3       org.eclipse.jetty.xml.XmlParser$Node$1::hasNext (99 bytes)
    Event: 100.677 Thread 0x0000000036601000 12300       3       org.eclipse.jetty.xml.XmlParser$Node::get (65 bytes)
    Event: 100.677 Thread 0x00000000365ff800 12302  s    3       org.eclipse.jetty.xml.XmlParser$Node::toString (221 bytes)
    Event: 100.677 Thread 0x00000000365fe000 12301  s    3       org.eclipse.jetty.xml.XmlParser$Node::toString (19 bytes)
    Event: 100.678 Thread 0x0000000034407000 12303       4       sun.nio.fs.WindowsPath::hashCode (52 bytes)
    Event: 100.680 Thread 0x00000000365fe000 nmethod 12301 0x0000000003435650 code [0x0000000003435800, 0x0000000003435e78]
    Event: 100.681 Thread 0x0000000036601000 nmethod 12300 0x0000000003c007d0 code [0x0000000003c009a0, 0x0000000003c01328]
    Event: 100.681 Thread 0x00000000365ff000 nmethod 12299 0x00000000059e5c50 code [0x00000000059e5e40, 0x00000000059e6868]

    GC Heap History (10 events):
    Event: 26.651 GC heap before
    {Heap before GC invocations=5 (full 2):
     PSYoungGen      total 12233216K, used 27783K [0x0000000dd5580000, 0x0000001239700000, 0x0000001480000000)
      eden space 10485760K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001055580000)
      from space 1747456K, 1% used [0x0000001055580000,0x00000010570a1f60,0x00000010c0000000)
      to   space 1747456K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001239700000)
     ParOldGen       total 27962368K, used 625741K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 2% used [0x0000000080000000,0x00000000a63137f8,0x000000072ab00000)
     Metaspace       used 34654K, capacity 35328K, committed 35416K, reserved 36864K
    Event: 27.271 GC heap after
    Heap after GC invocations=5 (full 2):
     PSYoungGen      total 12233216K, used 0K [0x0000000dd5580000, 0x0000001239700000, 0x0000001480000000)
      eden space 10485760K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001055580000)
      from space 1747456K, 0% used [0x0000001055580000,0x0000001055580000,0x00000010c0000000)
      to   space 1747456K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001239700000)
     ParOldGen       total 27962368K, used 347576K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x000000009536e3f8,0x000000072ab00000)
     Metaspace       used 34654K, capacity 35328K, committed 35416K, reserved 36864K
    }
    Event: 38.958 GC heap before
    {Heap before GC invocations=6 (full 2):
     PSYoungGen      total 12233216K, used 1525617K [0x0000000dd5580000, 0x0000001239700000, 0x0000001480000000)
      eden space 10485760K, 14% used [0x0000000dd5580000,0x0000000e3275c4d8,0x0000001055580000)
      from space 1747456K, 0% used [0x0000001055580000,0x0000001055580000,0x00000010c0000000)
      to   space 1747456K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001239700000)
     ParOldGen       total 27962368K, used 347576K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x000000009536e3f8,0x000000072ab00000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    Event: 39.130 GC heap after
    Heap after GC invocations=6 (full 2):
     PSYoungGen      total 15512576K, used 122189K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1168896K, 10% used [0x00000011cec80000,0x00000011d63d36a8,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 27962368K, used 347649K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x0000000095380410,0x000000072ab00000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    }
    Event: 39.130 GC heap before
    {Heap before GC invocations=7 (full 3):
     PSYoungGen      total 15512576K, used 122189K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1168896K, 10% used [0x00000011cec80000,0x00000011d63d36a8,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 27962368K, used 347649K [0x0000000080000000, 0x000000072ab00000, 0x0000000dd5580000)
      object space 27962368K, 1% used [0x0000000080000000,0x0000000095380410,0x000000072ab00000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    Event: 39.734 GC heap after
    Heap after GC invocations=7 (full 3):
     PSYoungGen      total 15512576K, used 0K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1168896K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 34104832K, used 365482K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964eaa38,0x00000008a1980000)
     Metaspace       used 58127K, capacity 59028K, committed 59028K, reserved 59392K
    }
    Event: 67.747 GC heap before
    {Heap before GC invocations=8 (full 3):
     PSYoungGen      total 15512576K, used 7498360K [0x0000000dd5580000, 0x0000001216200000, 0x0000001480000000)
      eden space 14343680K, 52% used [0x0000000dd5580000,0x0000000f9f01e008,0x0000001140d00000)
      from space 1168896K, 0% used [0x00000011cec80000,0x00000011cec80000,0x0000001216200000)
      to   space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
     ParOldGen       total 34104832K, used 365482K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964eaa38,0x00000008a1980000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    Event: 67.974 GC heap after
    Heap after GC invocations=8 (full 3):
     PSYoungGen      total 16091136K, used 142921K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1747456K, 8% used [0x0000001140d00000,0x0000001149892538,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 34104832K, used 365522K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964f4a50,0x00000008a1980000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    }
    Event: 67.974 GC heap before
    {Heap before GC invocations=9 (full 4):
     PSYoungGen      total 16091136K, used 142921K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1747456K, 8% used [0x0000001140d00000,0x0000001149892538,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 34104832K, used 365522K [0x0000000080000000, 0x00000008a1980000, 0x0000000dd5580000)
      object space 34104832K, 1% used [0x0000000080000000,0x00000000964f4a50,0x00000008a1980000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    Event: 68.533 GC heap after
    Heap after GC invocations=9 (full 4):
     PSYoungGen      total 16091136K, used 0K [0x0000000dd5580000, 0x00000012b0600000, 0x0000001480000000)
      eden space 14343680K, 0% used [0x0000000dd5580000,0x0000000dd5580000,0x0000001140d00000)
      from space 1747456K, 0% used [0x0000001140d00000,0x0000001140d00000,0x00000011ab780000)
      to   space 582656K, 0% used [0x000000128cd00000,0x000000128cd00000,0x00000012b0600000)
     ParOldGen       total 46550528K, used 306079K [0x0000000080000000, 0x0000000b99380000, 0x0000000dd5580000)
      object space 46550528K, 0% used [0x0000000080000000,0x0000000092ae7f70,0x0000000b99380000)
     Metaspace       used 96631K, capacity 98358K, committed 98380K, reserved 100352K
    }

    Deoptimization events (10 events):
    Event: 84.458 Thread 0x000000003e0c5800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000395e364 method=org.eclipse.jetty.annotations.AnnotationParser.normalize(Ljava/lang/String;)Ljava/lang/String; @ 21
    Event: 84.481 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x00000000039e70d0 method=java.util.zip.ZipEntry.setExtra0([BZ)V @ 192
    Event: 84.490 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000005d34e24 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 84.490 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000003943ef4 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 84.501 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000005cd53fc method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 84.507 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000003c90f80 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 85.333 Thread 0x000000003be30000 Uncommon trap: reason=null_assert action=make_not_entrant pc=0x0000000005d96908 method=java.util.zip.ZipEntry.<init>(Ljava/util/zip/ZipEntry;)V @ 70
    Event: 86.767 Thread 0x000000003e0b8000 Uncommon trap: reason=range_check action=make_not_entrant pc=0x00000000056be4e8 method=java.util.zip.ZipUtils.get16([BI)I @ 10
    Event: 100.670 Thread 0x000000157773a000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000565432c method=org.eclipse.jetty.util.log.JettyAwareLogger.log(Lorg/slf4j/Marker;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V @ 2
    Event: 100.683 Thread 0x0000000002f22800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000003b2bdfc method=java.io.FilePermission.init(I)V @ 136

    Internal exceptions (10 events):
    Event: 82.340 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f155c5410) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.342 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f155d2f48) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.390 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15863428) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.413 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f159658d8) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.472 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15d2b2b0) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.484 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15d7d400) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 82.505 Thread 0x000000003e0b9800 Exception <a 'sun/nio/fs/WindowsException'> (0x0000000f15ea3c60) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 709]
    Event: 83.849 Thread 0x000000003be2d800 Implicit null exception at 0x0000000003b7f899 to 0x0000000003b82a75
    Event: 100.669 Thread 0x000000157773a000 Exception <a 'java/lang/OutOfMemoryError'> (0x00000010334b3d18) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jni.cpp, line 735]
    Event: 100.674 Thread 0x000000157773a000 Exception <a 'java/lang/OutOfMemoryError': unable to create new native thread> (0x00000010334cf740) thrown at [C:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u102\7268\hotspot\src\share\vm\prims\jvm.cpp, line 3020]

    Events (10 events):
    Event: 100.673 loading class org/eclipse/jetty/jsp/JettyJspServlet done
    Event: 100.675 loading class java/lang/Throwable$WrappedPrintStream
    Event: 100.675 loading class java/lang/Throwable$WrappedPrintStream done
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet done
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet
    Event: 100.676 loading class org/eclipse/jetty/jsp/JettyJspServlet done
    Event: 100.683 Thread 0x0000000002f22800 Uncommon trap: trap_request=0xffffff65 fr.pc=0x0000000003b2bdfc
    Event: 100.683 Thread 0x0000000002f22800 DEOPT PACKING pc=0x0000000003b2bdfc sp=0x000000000094c3d0
    Event: 100.683 Thread 0x0000000002f22800 DEOPT UNPACKING pc=0x000000000306582a sp=0x000000000094c360 mode 2


    Dynamic libraries:
    0x00007ff71b180000 - 0x00007ff71b1d4000  D:\CA\DevTest\bin\Portal.exe
    0x00007ffa95de0000 - 0x00007ffa95f8a000  C:\Windows\SYSTEM32\ntdll.dll
    0x00007ffa943b0000 - 0x00007ffa944ea000  C:\Windows\system32\KERNEL32.DLL
    0x00007ffa93550000 - 0x00007ffa9365f000  C:\Windows\system32\KERNELBASE.dll
    0x00007ffa937d0000 - 0x00007ffa93941000  C:\Windows\system32\USER32.dll
    0x00007ffa95d30000 - 0x00007ffa95dd5000  C:\Windows\system32\ADVAPI32.dll
    0x00007ffa90e00000 - 0x00007ffa9105a000  C:\Windows\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.9600.17031_none_6242a4b3ecbb55a1\COMCTL32.dll
    0x00007ffa93d20000 - 0x00007ffa93e65000  C:\Windows\system32\GDI32.dll
    0x00007ffa94120000 - 0x00007ffa941c7000  C:\Windows\system32\msvcrt.dll
    0x00007ffa94000000 - 0x00007ffa94057000  C:\Windows\SYSTEM32\sechost.dll
    0x00007ffa94730000 - 0x00007ffa94866000  C:\Windows\system32\RPCRT4.dll
    0x00007ffa93790000 - 0x00007ffa937c4000  C:\Windows\system32\IMM32.DLL
    0x00007ffa93950000 - 0x00007ffa93a89000  C:\Windows\system32\MSCTF.dll
    0x0000000059920000 - 0x000000005a1ba000  d:\ca\devtest\jre\bin\server\jvm.dll
    0x00007ffa886a0000 - 0x00007ffa886a9000  C:\Windows\SYSTEM32\WSOCK32.dll
    0x00007ffa865a0000 - 0x00007ffa865bf000  C:\Windows\SYSTEM32\WINMM.dll
    0x00007ffa86ca0000 - 0x00007ffa86caa000  C:\Windows\SYSTEM32\VERSION.dll
    0x00007ffa93aa0000 - 0x00007ffa93aa7000  C:\Windows\system32\PSAPI.DLL
    0x000000005a260000 - 0x000000005a332000  C:\Windows\SYSTEM32\MSVCR100.dll
    0x00007ffa94060000 - 0x00007ffa940b8000  C:\Windows\system32\WS2_32.dll
    0x00007ffa86570000 - 0x00007ffa8659a000  C:\Windows\SYSTEM32\WINMMBASE.dll
    0x00007ffa93a90000 - 0x00007ffa93a99000  C:\Windows\system32\NSI.dll
    0x00007ffa93450000 - 0x00007ffa9349a000  C:\Windows\SYSTEM32\cfgmgr32.dll
    0x00007ffa92040000 - 0x00007ffa92066000  C:\Windows\SYSTEM32\DEVOBJ.dll
    0x0000000059910000 - 0x000000005991f000  d:\ca\devtest\jre\bin\verify.dll
    0x00000000598e0000 - 0x0000000059909000  d:\ca\devtest\jre\bin\java.dll
    0x00000000598c0000 - 0x00000000598d6000  d:\ca\devtest\jre\bin\zip.dll
    0x00007ffa94910000 - 0x00007ffa95d27000  C:\Windows\system32\SHELL32.dll
    0x00007ffa944f0000 - 0x00007ffa946c6000  C:\Windows\SYSTEM32\combase.dll
    0x00007ffa940c0000 - 0x00007ffa94111000  C:\Windows\system32\SHLWAPI.dll
    0x00007ffa91810000 - 0x00007ffa918af000  C:\Windows\SYSTEM32\SHCORE.dll
    0x00007ffa93150000 - 0x00007ffa93164000  C:\Windows\SYSTEM32\profapi.dll
    0x0000000059890000 - 0x000000005989d000  D:\CA\DevTest\jre\bin\management.dll
    0x00000000598a0000 - 0x00000000598ba000  D:\CA\DevTest\jre\bin\net.dll
    0x00007ffa92a80000 - 0x00007ffa92ad8000  C:\Windows\system32\mswsock.dll
    0x0000000059870000 - 0x0000000059881000  D:\CA\DevTest\jre\bin\nio.dll
    0x00007ffa92ae0000 - 0x00007ffa92afe000  C:\Windows\SYSTEM32\CRYPTSP.dll
    0x00007ffa926f0000 - 0x00007ffa92725000  C:\Windows\system32\rsaenh.dll
    0x00007ffa92d10000 - 0x00007ffa92d36000  C:\Windows\SYSTEM32\bcrypt.dll
    0x00007ffa927e0000 - 0x00007ffa927fe000  C:\Windows\SYSTEM32\USERENV.dll
    0x00007ffa93000000 - 0x00007ffa93060000  C:\Windows\system32\bcryptprimitives.dll
    0x00007ffa93140000 - 0x00007ffa9314a000  C:\Windows\SYSTEM32\CRYPTBASE.dll
    0x00007ffa8fc80000 - 0x00007ffa8fca9000  C:\Windows\SYSTEM32\IPHLPAPI.DLL
    0x00007ffa8fc70000 - 0x00007ffa8fc7a000  C:\Windows\SYSTEM32\WINNSI.DLL
    0x00007ffa8f5f0000 - 0x00007ffa8f604000  C:\Windows\SYSTEM32\dhcpcsvc6.DLL
    0x00007ffa8f5d0000 - 0x00007ffa8f5e9000  C:\Windows\SYSTEM32\dhcpcsvc.DLL
    0x00007ffa8ded0000 - 0x00007ffa8dee4000  C:\Windows\system32\napinsp.dll
    0x00007ffa90680000 - 0x00007ffa90699000  C:\Windows\system32\NLAapi.dll
    0x00007ffa92890000 - 0x00007ffa92933000  C:\Windows\SYSTEM32\DNSAPI.dll
    0x00007ffa8dec0000 - 0x00007ffa8decc000  C:\Windows\System32\winrnr.dll
    0x00007ffa8f610000 - 0x00007ffa8f678000  C:\Windows\System32\fwpuclnt.dll
    0x00007ffa8de40000 - 0x00007ffa8de49000  C:\Windows\System32\rasadhlp.dll
    0x0000000059840000 - 0x0000000059864000  D:\CA\DevTest\jre\bin\sunec.dll

    VM Arguments:
    jvm_args: -Dexe4j.semaphoreName=Local\d:_ca_devtest_bin_portal.exe -Dexe4j.isInstall4j=true -Dexe4j.moduleName=D:\CA\DevTest\bin\Portal.exe -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Djava.library.path=D:\CA\DevTest\bin\..\bin;D:\CA\DevTest\bin\..\lib\native;D:\TimesTent11.2.2\bin;D:\TimesTent11.2.2\ttoracle_home\instantclient_11_2;D:\app\devtest\product\12.1.0\dbhome_1\bin;D:\app\devt\product\12.1.0\dbhome_1\bin;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\CA\DevTest\lib\core;D:\CA\DevTest\bin;d:\ca\devtest\jre\bin -Dexe4j.consoleCodepage=cp936 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\CA\DevTest\bin\/../tmp -Djava.endorsed.dirs=D:\CA\DevTest\bin\/../lib/endorsed -DLISA_HOME=D:\CA\DevTest\bin\/../ -DLISA_LOG=portal.log -Xms24m -Xmx1024m -Dfile.encoding=UTF-8 -Dorg.springframework.boot.logging.LoggingSystem=com.ca.devtest.springframework.boot.logging.NonConfiguringWebAppLoggingSystem -Djava.net.preferIPv4Stack=true ${LISA_MORE_VM_PROPS} -Dlisa.version.number=10.0.0 -Dinstall4j.launcherId=12902 -Dinstall4j.swt=false -Xms40960m -Xmx81920m
    java_command: D:\CA\DevTest\bin\Portal.exe
    java_class_path (initial): D:\CA\DevTest\.install4j\i4jruntime.jar;D:\CA\DevTest\bin\..\lib\phoenix\phoenix-runner.jar;D:\CA\DevTest\bin\..\lib\shared\owner-1.0.6.jar;D:\CA\DevTest\bin\..\lib\shared\jetty-all-9.3.11.v20160721-uber.jar;D:\CA\DevTest\bin\..\lib\shared\javax.servlet-api-*.jar;D:\CA\DevTest\bin\..\lib\shared\slf4j-api-1.7.7.jar;D:\CA\DevTest\bin\..\lib\shared\slf4j-log4j12-1.7.7.jar;D:\CA\DevTest\bin\..\lib\shared\log4j-1.2.17.jar;D:\CA\DevTest\bin\..\lib\shared\commons-logging-1.2.jar;D:\CA\DevTest\bin\..\lib\shared\commons-codec-1.9.jar;D:\CA\DevTest\bin\..\lib\shared\bcprov-jdk15on-1.54.jar;D:\CA\DevTest\bin\..\lib\shared\asm-5.0.1.jar;
    Launcher Type: generic

    Environment Variables:
    PATH=D:\CA\DevTest\bin\..\bin;D:\CA\DevTest\bin\..\lib\native;D:\TimesTent11.2.2\bin;D:\TimesTent11.2.2\ttoracle_home\instantclient_11_2;D:\app\devtest\product\12.1.0\dbhome_1\bin;D:\app\devt\product\12.1.0\dbhome_1\bin;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\CA\DevTest\lib\core;D:\CA\DevTest\bin;d:\ca\devtest\jre\bin
    USERNAME=Administrator
    OS=Windows_NT
    PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 62 Stepping 7, GenuineIntel

     

    ---------------  S Y S T E M  ---------------

    OS: Windows Server 2012 R2 , 64 bit Build 9600 (6.3.9600.17056)

    CPU:total 64 (8 cores per cpu, 2 threads per core) family 6 model 62 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, ht, tsc, tscinvbit

    Memory: 4k page, physical 536761968k(132535260k free), swap 578705008k(4972k free)

    vm_info: Java HotSpot(TM) 64-Bit Server VM (25.102-b14) for windows-amd64 JRE (1.8.0_102-b14), built on Jun 22 2016 13:15:21 by "java_re" with MS VC++ 10.0 (VS2010)

    time: Mon Sep 25 18:27:07 2017
    elapsed time: 100 seconds (0d 0h 1m 40s)

    werShell\v1.0\;D:\CA\DevTest\lib\core;D:\CA\DevTest\bin;d:\ca\devtest\jre\bin
    USERNAME=Administrator
    OS=Windows_NT
    PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 62 Stepping 7, GenuineIntel

     

    ---------------  S Y S T E M  ---------------

    OS: Windows Server 2012 R2 , 64 bit Build 9600 (6.3.9600.17056)

    CPU:total 64 (8 cores per cpu, 2 threads per core) family 6 model 62 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, ht, tsc, tscinvbit

    Memory: 4k page, physical 536761968k(132535260k free), swap 578705008k(4972k free)

    vm_info: Java HotSpot(TM) 64-Bit Server VM (25.102-b14) for windows-amd64 JRE (1.8.0_102-b14), built on Jun 22 2016 13:15:21 by "java_re" with MS VC++ 10.0 (VS2010)

    time: Mon Sep 25 18:27:07 2017
    elapsed time: 100 seconds (0d 0h 1m 40s)



  • 2.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Broadcom Employee
    Posted Sep 26, 2017 04:45 PM

    Can I get a bit more information from you?

     

    Looks like you are on DevTest 10.0.0.  

     

    What were you doing when you got this thread dump?

     

    What DevTest components are running on this Windows 2012 machine?



  • 3.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Posted Sep 26, 2017 08:21 PM

    all of them,include databases



  • 4.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Broadcom Employee
    Posted Sep 27, 2017 09:33 AM

    What external database are you using?

     

    What is the memory on the machine?

     

    Are you using .vmoptions entries to start the components?  If so,  please list them for the Registry, Portal, Coordinator, Simulator and VSE.



  • 5.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Broadcom Employee
    Posted Sep 27, 2017 11:08 AM

    From the error  it looks like you are running out of Physical RAM.  Can you please verify?  



  • 6.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208
    Best Answer

    Posted Sep 27, 2017 12:37 PM

    Hello Jade,

     

    It looks like you were facing issues with DevTest Portal.

    jvm_args: -Dexe4j.semaphoreName=Local\d:_ca_devtest_bin_portal.exe

     

    Is it correct that the Portal is configured with the limits below?

    -Xms40960m

    -Xmx81920m

     

    What is the heap size configuration for DevTest services? Please, take a look at the vmoptions files for each component and verify.

     

    The recommendation is to start with low maximum heap value and then increase as needed.

    You can start with 2G, monitor it and increase the heap size if needed.

     

    Hope it helps.

    Heloisa



  • 7.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Broadcom Employee
    Posted Sep 27, 2017 03:42 PM

    Hi Heloisa,

     

    Did you intend to set the values to:

    -Xms4096m

    -Xmx8192m

    without the extra zero?

     

    --Mike



  • 8.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Posted Sep 27, 2017 03:49 PM

    Hi Mike,

     

    I wanted to verify what is set for now. From the information that was provided, it looks like the portal is set to -Xms40960m -Xmx81920m.

     

    I am looking at the piece of information below:

    VM Arguments:
    jvm_args: -Dexe4j.semaphoreName=Local\d:_ca_devtest_bin_portal.exe -Dexe4j.isInstall4j=true -Dexe4j.moduleName=D:\CA\DevTest\bin\Portal.exe -Dexe4j.tempDir= -Dexe4j.unextractedPosition=0 -Djava.library.path=D:\CA\DevTest\bin\..\bin;D:\CA\DevTest\bin\..\lib\native;D:\TimesTent11.2.2\bin;D:\TimesTent11.2.2\ttoracle_home\instantclient_11_2;D:\app\devtest\product\12.1.0\dbhome_1\bin;D:\app\devt\product\12.1.0\dbhome_1\bin;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;D:\CA\DevTest\lib\core;D:\CA\DevTest\bin;d:\ca\devtest\jre\bin -Dexe4j.consoleCodepage=cp936 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\CA\DevTest\bin\/../tmp -Djava.endorsed.dirs=D:\CA\DevTest\bin\/../lib/endorsed -DLISA_HOME=D:\CA\DevTest\bin\/../ -DLISA_LOG=portal.log -Xms24m -Xmx1024m -Dfile.encoding=UTF-8 -Dorg.springframework.boot.logging.LoggingSystem=com.ca.devtest.springframework.boot.logging.NonConfiguringWebAppLoggingSystem -Djava.net.preferIPv4Stack=true ${LISA_MORE_VM_PROPS} -Dlisa.version.number=10.0.0 -Dinstall4j.launcherId=12902 -Dinstall4j.swt=false -Xms40960m -Xmx81920m
    java_command: D:\CA\DevTest\bin\Portal.exe

     

    Let me know if I answered your questions.

     

    Heloisa



  • 9.  Re:  Out of Memory Error (allocation.cpp:390), pid=8780, tid=0x0000000000007208

    Posted Sep 29, 2017 10:46 AM

    ths,Heloisa 

    Your answer to help me solve it, is memory allocation is unreasonable.