Hello Samantha,
I have seen the same error/ stack trace when running long queries and seems to be related to the load
Could you verify the below?
1) The exception itself is about broken TCP connection, it could be caused in case the client refuses to get more data from server in case of large data portion or the network error.
Check if the problem occurs when running specific queries : Introcsope webservices calls, workstation,CLW queries or when using webview
2)I have found the same stack trace in Defect DE142508, but without a heapdump it not possible to confirm that the root cause is the same:
+++++++++++
“We took a heap dump and examined the object graphs for the database connections on the MoM, which pointed to a Commons DBCP pool used by AppMap. The pool instantiated internally by the EM, and is configured with an initial size of 10 connections, and a “maxIdle” size of 40 connections. What this means is that even though the connections are not being used, the pool will hold them open. What it looks like is a series of events, starting with the db aggregation job being configured on the wrong collector combined with the extra connections being held open in an IDE, which led to a burst of activity on Feb 18. This caused the connection pool in question to grow, and at that point the extra connections are never released, causing the problems to continue.
So there are two issues – 1) reduce the <property name="maxIdle” setting value so that the pool can return to normal after a burst, 2) add the pool to the sustainability metrics so that we can accurately track what is happening.
Specifically, in em\com.wily.introscope.appmap.em\src\main\resources\spring\appmap-server-context.xml:
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="${hibernate.connection.driver_class}" />
<property name="url" value="${hibernate.connection.url}" />
<property name="username" value="${hibernate.connection.username}" />
<property name="password" value="${hibernate.connection.password}" />
<property name="initialSize" value="10" />
<property name="maxActive" value="250" />
<property name="maxIdle" value="40" />
<property name="poolPreparedStatements" value="true" />
<property name="maxOpenPreparedStatements" value="250" />
</bean>
The quick workaround would be to reconfigure the setting in the xml file by unpacking the jar, editing the file, and repackaging the jar in the MoM installation, and then restart the MoM.]
+++++++++++
The above file is located inside EM_HOME\product\enterprisemanager\plugins\ com.wily.introscope.appmap.em_10.5.1.jar\spring\appmap-server-context.xml
You can use 7zip or similar utility to update the file, try to reduce maxIdle to 20
If the problem persists, open a support case and attach EM logs + a series of threadumps from MOM + apm db logs.
I hope this helps,
Regards,
Sergio