DX Application Performance Management

Hi,

We are amidst migration of CA APM from 10.3 to 10.7 to make use of APMSQL as a datasource for external dashboarding purposes (Grafana in our case). While we see mentions of Grafana in the APMSQL cookbook, we would however like to know further on the feasibility and how to actually get this done. We are using both CA APM and Grafana extensively and it would be awesome to see a combination of the two. Note that we use Postgres as the APM database.

Thanks,

Abhishek Rao

Converted to a discussion

Abhishek:

We are going to ship an official Grafana to APMSQL implementation with 10.7 SP1.  In the meantime, we are looking for early adopters to install and play with the integration.  I have draft documentation available and several customers have been successful with it, but the installation discussion is not yet complete.  If you would like to try it, or have a customer that would, we would love to get your feedback and incorporate it into the final release.

Grafana expects a JSON REST service.  In short, you can deploy this as a WAR in your own Tomcat, or use it as a built-in service in the APMSQL server instance.  You need APMSQL server running regardless.  Postgres is not involved; only APMSQL (and SmartStor).

May I have the draft you create. I already using APMSQL and want to try Grafana integration. FYI we are using 10.5.2 SP2.  is it ok to integrate APMSQL with Grafana on 10.5.2?

Send me a direct email and I will set you up with the WAR and instructions.

david.b.martin@ca.com

Hi David,

Could you send me instructions for 10.5.2, please ? I try to send you a mail directly but it doesn t work, my message is not distributed because your address is uknown. 

Regards

Philippe

Hi Philippe Merlin,

The official Grafana support with APMSQL is in 10.7 and you can also connect a 10.7 APMSQL server to a 10.5.x EM which Lutz Mader did successfully in this thread: APMSQL Grafana with APM 10.5.1 

More details about Grafana in 10.7 docops here:  Using Grafana with CA APMSQL - CA Application Performance Management - 10.7 - CA Technologies Documentation 

Also for query syntax see this related Community thread: No data points (grafana+apmsql) and KB: CA APM Grafana APMSQL queries - CA Knowledge 

Hope that helps

Regards,

Lynn

That's awesome. I'm glad that we can help you in this regard. Our current plan is to use APMSQL for Performance Testing environments to get valuable inputs/reports from the APM data. We would like it if you could share the draft documentation so we can get started on it right away . Also, do you have any timelines for the official release?

Thanks,

Abhishek

You don't have to wait for anything to use APMSQL; it is an official product feature today!  Grafana support will come in a service pack later this year.

On a separate, but related note, I've heard some reports of frustrations with APMSQL query performance.  So far, it seems to be around the "distinct" keyword, which is truly a stinker.  If you struggle to get the performance you expect from a query, I hope you'll come to me so we can figure out alternative queries.  There are better performing replacements for "distinct".

Hi

getting this error message in APMSQL server  while executing a apmsql from grafana. any suggestion much appreciated

17262 [NIO1] WARN org.apache.cxf.phase.PhaseInterceptorChain - Interceptor for {http://wilyapm:8444/apm/appmap/private/token/temporaryToken}WebClient has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
        at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
        at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1100)
        at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:894)
        at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
        at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:331)
        at org.apache.cxf.jaxrs.client.WebClient.post(WebClient.java:340)
        at com.ca.apm.server.teiid.APMSqlSecurityHelper.isValid(APMSqlSecurityHelper.java:182)
        at com.ca.apm.server.teiid.APMSqlSecurityHelper.authenticate(APMSqlSecurityHelper.java:128)
        at com.ca.apm.server.teiid.APMSqlSecurityHelper.authenticate(APMSqlSecurityHelper.java:1)
        at org.teiid.services.SessionServiceImpl.createSession(SessionServiceImpl.java:206)
        at org.teiid.transport.LogonImpl.logon(LogonImpl.java:142)
        at org.teiid.transport.LogonImpl.logon(LogonImpl.java:127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.teiid.transport.ServerWorkItem.run(ServerWorkItem.java:87)
        at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276)
        at org.teiid.transport.SocketClientInstance.processMessagePacket(SocketClientInstance.java:236)
        at org.teiid.transport.SocketClientInstance.receivedMessage(SocketClientInstance.java:222)
        at org.teiid.transport.SSLAwareChannelHandler.messageReceived(SSLAwareChannelHandler.java:212)
        at org.teiid.transport.SSLAwareChannelHandler.channelRead(SSLAwareChannelHandler.java:218)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: SocketException invoking http://wilyapm:8444/apm/appmap/private/token/temporaryToken: Unexpected end of file from server
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1365)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1349)
        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
        at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:652)
        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
        ... 46 more
Caused by: java.net.SocketException: Unexpected end of file from server
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:789)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:275)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1563)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1533)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1336)
        ... 49 more
7/23/18 12:28:32.784 PM BST [ERROR] [NIO1] [SECURITY] TEIID40017 Unexpected exception for session null
javax.ws.rs.ProcessingException: java.net.SocketException: SocketException invoking http://wilyapm:8444/apm/appmap/private/token/temporaryToken: Unexpected end of file from server
        at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:557)
        at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:539)
        at org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1151)
        at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1101)
        at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:894)
        at org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:865)
        at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:331)
        at org.apache.cxf.jaxrs.client.WebClient.post(WebClient.java:340)
        at com.ca.apm.server.teiid.APMSqlSecurityHelper.isValid(APMSqlSecurityHelper.java:182)
        at com.ca.apm.server.teiid.APMSqlSecurityHelper.authenticate(APMSqlSecurityHelper.java:128)
        at com.ca.apm.server.teiid.APMSqlSecurityHelper.authenticate(APMSqlSecurityHelper.java:1)
        at org.teiid.services.SessionServiceImpl.createSession(SessionServiceImpl.java:206)
        at org.teiid.transport.LogonImpl.logon(LogonImpl.java:142)
        at org.teiid.transport.LogonImpl.logon(LogonImpl.java:127)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.teiid.transport.ServerWorkItem.run(ServerWorkItem.java:87)
        at org.teiid.dqp.internal.process.DQPWorkContext.runInContext(DQPWorkContext.java:276)
        at org.teiid.transport.SocketClientInstance.processMessagePacket(SocketClientInstance.java:236)
        at org.teiid.transport.SocketClientInstance.receivedMessage(SocketClientInstance.java:222)
        at org.teiid.transport.SSLAwareChannelHandler.messageReceived(SSLAwareChannelHandler.java:212)
        at org.teiid.transport.SSLAwareChannelHandler.channelRead(SSLAwareChannelHandler.java:218)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1304)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:921)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:135)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: SocketException invoking http://wilyapm:8444/apm/appmap/private/token/temporaryToken: Unexpected end of file from server
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1365)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1349)
        at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
        at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:652)
        at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
        at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:624)
        at org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1100)
        ... 43 more
Caused by: java.net.SocketException: Unexpected end of file from server
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:792)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
        at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:789)
        at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
        at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
        at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:275)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1563)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1533)
        at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1336)
        ... 49 more

this is my property setting

introscope.webview.enterprisemanager.rest.base=https://wilyapm:8444/apm/appmap

Grafana console error

introscope.webview.enterprisemanager.rest.base=https://wilyapm:8444/apm/appmap

Please open a case to resolve. Seems an issue with SSL communications

Thanks Hallett

Opened a case with CA

Thanks

Ranga