AnsweredAssumed Answered

ASM Checkpoint connection error: 503

Question asked by Arnab Mukhopadhyay on Feb 6, 2018
Latest reply on Feb 8, 2018 by Arnab Mukhopadhyay

Like • Show 1 Like1
Comment • 8

Hi,

We are seeing an issue where the On Premise Station is not relaying status of our Internal Website to asm.ca.com. The issue started happening last week. We have verified with our Security Engineers and they don't see anything blocked. I am able to ping asm.ca.com from opms and also ping the internal websites. I see the below error message.

(-98) No checkpoint available for check type https/IPvANY
(-95) Checkpoint connection error: 503

The Opms station shows as Active when I login to asm.ca.com. I also restarted the opms station and that did not resolve the issue. Any help will be greatly appreciated.

Thanks,

Arnab

Lynn_Williams

Feb 6, 2018 5:41 AM

Correct Answer

Hi Arnab,

This scenario may require a support case to analyse the opms logs in detail.

In the meantime have you tried command "monit restart all" to see if it will hep to resolve the problem?

If not resolved after that what is the output from the command "monit summary" ?

Troubleshooting - CA App Synthetic Monitor - CA Technologies Documentation

Thanks

Lynn

See the reply in context

No one else had this question

Outcomes

8 Replies

Lynn_Williams Feb 6, 2018 5:41 AM
Unmark Correct
Correct Answer
Hi Arnab,
This scenario may require a support case to analyse the opms logs in detail.
In the meantime have you tried command "monit restart all" to see if it will hep to resolve the problem?
If not resolved after that what is the output from the command "monit summary" ?
Troubleshooting - CA App Synthetic Monitor - CA Technologies Documentation

Thanks

Lynn
2 people found this helpful
Like • Show 2 Likes2
Actions
- Arnab Mukhopadhyay @ Lynn_Williams on Feb 7, 2018 2:55 AM
  Mark Correct
  Correct Answer
  Hi Lynn,
  
  Thank you for your reply.
  
  Unfortunately the restart did not resolve the issue. I will provide you the output of the monit summary.
  
  Here is the bottom of the log file.
  
  2018-02-06 09:56:28,113 client          INFO     [8967faac-e276-4989-a9e4-e630f6d1e763] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (3.026381) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30
  2018-02-06 09:56:30,316 client          INFO     [46b390a9-a375-47b5-a716-da46dc441aaa] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.537756) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30
  2018-02-06 09:56:40,381 optunnel        INFO     OnPremise PoP tunnel started with config: /etc/asm/optunnel.yaml
  2018-02-06 09:56:40,410 client          INFO     opPoP client is initiating
  2018-02-06 09:56:40,410 client          INFO     Using pid file: /opt/asm/var/run/optunnel-client.pid
  2018-02-06 09:56:40,410 client          ERROR    PID file exists /opt/asm/var/run/optunnel-client.pid
  2018-02-06 09:56:40,410 client          ERROR    Found stale PID file in /opt/asm/var/run/optunnel-client.pid, removing...
  2018-02-06 09:56:40,410 client          INFO     Created PID file in /opt/asm/var/run/optunnel-client.pid
  2018-02-06 09:56:43,749 client          INFO     Trying to connect at wss://opp.asm.ca.com:443/4a8317d4-21ab-11e6-aff1-782bcb56f074
  2018-02-06 09:56:44,310 client          INFO     WebSocket client connected to server (with cid=4a8317d4-21ab-11e6-aff1-782bcb56f074)
  2018-02-06 09:56:44,323 client          INFO     opPoP client is up & ready
  2018-02-06 10:00:55,830 client          INFO     [99464b02-788d-4385-b4e1-f2ac7c52578a] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=rbm&direct=1&age=30 (3.01754) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=rbm&direct=1&age=30
  2018-02-06 10:01:28,444 client          INFO     [839bfcce-0cdc-4aa7-83a3-7e4b6c838f4a] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.431061) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30
  2018-02-06 10:01:29,945 client          INFO     [03522980-a546-43d7-a425-c30e33a09b8c] cURL OK: (Status: 503) http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30 (1.515884) - Effective URL http://dotstollopms101.tolls.dot.state.fl.us/api/monitor/check?agent=cbot&direct=1&age=30
  Like • Show 0 Likes0
  Actions
  - Lynn_Williams @ Arnab Mukhopadhyay on Feb 7, 2018 3:09 AM
    Mark Correct
    Correct Answer
    Hi Arnab,
    As well as the "monit summary" output please also provide the output of "cat /proc/loadavg" and advise the OPMS version being used.
    Thanks
    Lynn
    1 person found this helpful
    Like • Show 1 Like1
    Actions
    - Arnab Mukhopadhyay @ Lynn_Williams on Feb 7, 2018 2:27 PM
      Mark Correct
      Correct Answer
      Hi Lynn,
      
      Below is the information from the opms station.
      
      # cat /opt/asm/opms/version.txt
      8.4.0.2
      
      # cat /proc/loadavg
      0.00 0.03 0.05 1/199 12235
      
      # monit summary
      The Monit daemon 5.9 uptime: 4d 18h 47m
      
      Process 'Xvfb'                      Running
      Process 'tunnel-client'             Running
      System 'dotstollopms101'            Running
      Process 'api'                       Not monitored
      Process 'httpbroker'                Not monitored
      Process 'assetsmanager'             Not monitored
      Process 'resultbroker'              Not monitored
      Process 'fpm'                       Not monitored
      Process 'redis-server'              Not monitored
      Process 'rbtm1'                     Not monitored
      Process 'rbtm1-Xvfb'                Running
      Process 'rbtm2'                     Not monitored
      Process 'rbtm2-Xvfb'                Running
      Process 'php-fpm'                   Running
      Process 'nginx'                     Running
      File 'jmeter-jail-proc'             Accessible
      Filesystem 'rootfs_warning'         Accessible
      Filesystem 'rootfs_critical'        Accessible
      Filesystem 'devshmfs_warning'       Accessible
      Filesystem 'devshmfs_critical'      Accessible
      Process 'fcgiwrap'                  Running
      Process 'bm_proxy'                  Running
      
      Like • Show 0 Likes0
      Actions
      - Lynn_Williams @ Arnab Mukhopadhyay on Feb 7, 2018 10:34 PM
        Mark Correct
        Correct Answer
        Hi Arnab,
        Thanks for the output.
        The fact that "monit summary" shows that several process are not running is a cause for concern.
        To troubleshoot in more detail please login and create a support case on support.ca.com (https://comm.support.ca.com/csupport/CaseManagement/cases/new)
        Please upload the log files from these directories to the case and one of our ASM SMEs will review the case logs and contact you.
        /var/log/optunnel
        /var/log/nginx
        /var/log/redis
        /var/log/puppet
        /var/log/smartpop
        Also /var/log/syslog*
        
        Thanks
        
        Lynn
        Like • Show 0 Likes0
        Actions
Arnab Mukhopadhyay Feb 8, 2018 3:03 AM
Mark Correct
Correct Answer
Thanks Lynn, I opened up a case. I see the below error in the syslog file

Does OPMS rely on puppet certificate?

Feb 7 21:35:01 XXXXXCRON[4727]: (root) CMD (/usr/bin/diff /etc/resolv.conf /opt/asm/jmeter/jail/etc/resolv.conf || /bin/cp /etc/resolv.conf /opt/asm/jmeter/jail/etc/resolv.conf >/dev/null)
Feb 7 21:35:43 XXXXX puppet-agent[886]: Could not request certificate: getaddrinfo: Name or service not known
Like • Show 1 Like1
Actions
- Lynn_Williams @ Arnab Mukhopadhyay on Feb 8, 2018 9:15 PM
  Mark Correct
  Correct Answer
  Hi Arnab,
  I see you are working with Mat via case 00957989 and the main problem seems to be with the "redis-server" process not running which would prevent any monitors from running. Looks like Mat will be engaging Engineering to troubleshoot further.
  Regards,
  Lynn
  1 person found this helpful
  Like • Show 1 Like1
  Actions
  - Arnab Mukhopadhyay @ Lynn_Williams on Feb 8, 2018 9:36 PM
    Mark Correct
    Correct Answer
    Thanks Lynn, I see the below message in the log file. I will wait for Mat to respond.
    
    [20238] 08 Feb 13:03:44.445 # Server started, Redis version 2.8.17
    [20238] 08 Feb 13:03:44.445 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
    [20238] 08 Feb 13:03:44.962 # Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>
    Like • Show 1 Like1
    Actions

ASM Checkpoint connection error: 503

Outcomes

Related Content

Recommended Content