DX Application Performance Management

New to Agile Ops, I just installed the starter 4 node setup as documented, trying to see what this adds to our APM deployments. I have everything installed and running. I'm trying to connect one of my APM Clusters to it (assuming Elasticsearch), and getting the following errors. I created the tenant "UPS" during install it said cohort "UPS-USERSTORE", so these were the values I used:

On the Collector, I'm getting this error after configuring it:

12/10/18 03:00:41.651 PM EST [ERROR] [Jarvis-Tenant-Checker] [Manager.TraceStore] Error creating tenant for APM product in Jarvis.I/O error on GET request for "http://<server#2>:8080/onboarding/tenants(product_id='ao',tenant_id='UPS-USERSTORE')": Connection reset; nested exception is java.net.SocketException: Connection reset
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://<server#2>:8080/onboarding/tenants(product_id='ao',tenant_id='UPS-USERSTORE')": Connection reset; nested exception is java.net.SocketException: Connection reset

Collector updates:

###############################################################################
# AO Platform Integration Configuration
###############################################################################
cohortId=UPS-USERSTORE
com.ca.apm.ttstore=jarvis
ca.apm.ttstore.jarvis.es.url=http://<server#2>:9200
ca.apm.ttstore.jarvis.ingestion.url=http://<server#2>:8080/ingestion
ca.apm.ttstore.jarvis.onboarding.url=http://<server#2>:8080/onboarding

If I hit port 9200 on Server#2, I get this response:

name "MASTER-DATA-<Server#2>"
cluster_name "CA_AOPlatform-TEST"
cluster_uuid "Qx_aEFbOQciUwlUBSJigA"
version 
number "5.6.5"
build_hash "6a37571"
build_date "2017-12-04T07:50:10.466Z"
build_snapshot false
lucene_version "6.6.1"
tagline "You Know, for Search"

Also when I try the documented command for updating the doc-type, it fails:

>curl -v -H "Content-Type: application/json" -X POST "http://<Server#2>:8080/onboarding/doc_type" -d "@doc-type-traces.json"
* About to connect() to <Server#2> port 8080 (#0)
*   Trying x.x.x.x...
* Connected to <Server#2> (x.x.x.x) port 8080 (#0)
> POST /onboarding/doc_type HTTP/1.1
> User-Agent: curl/7.29.0
> Host: <Server#2>:8080
> Accept: */*
> Content-Type: application/json
> Content-Length: 1887
> Expect: 100-continue
>
* Done waiting for 100-continue
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Hello,

Can you please try to run this command from your browser:

http://jarvis_host_name:8080/onboarding/tenants(product_id='ao')

You should see all tenants onboarded successfully.

Ex:

If you see the tenant you created show up on this list, then JARVIS has onboarded your (Ex: UPS tenant)

Then the only other thing to check for is firewall between your APM EM and the JARVIS machine.

Make sure firewall is turned off on both APM and JARVIS boxes

Check if the port 8080 is responsive on the JARVIS machine

If you dont see the tenant (Ex: UPS) on the list when you run the command, then that means the tenant was not successfully created in JARVIS

There are scripts to manually onboard the tenant in JARVIS

Please reach out to me via email at harish1@broadcom.com and i can assist via webex.

Cheers,

Harish