DX NetOps Manager

  • Private Community
Back to Library

Tech Tip: Debugging trap processing in a Spectrum Distributed SpectroSERVER environment with Trap Director enabled 

Jul 10, 2015 11:38 AM

Tech Tip: Debugging trap processing in a Spectrum Distributed SpectroSERVER environment with Trap Director enabled

 

CA Spectrum Tech Tip by: Roger Nason, Support Delivery Manager

 

How to debug trap processing in a Distributed SpectroSERVER environment with Trap Director enabled

 

Description: The following tech tip describes how to monitor a trap from receipt on the Trap Director system to its processing on the destination SpectroSERVER.

 

Notes:

  • It is assumed that a packet capture has already been performed and shows the traps being received at the SpectroSERVER’s defined trap port (default is port 162).
  • All commands are run from the $SPECROOT/vnmsh directory.
  • All debug output is written to the $SPECROOT/SS/VNM.OUT file on the individual SpectroSERVERs

 

Solution:

Trap Director SS

On the SpectroSERVER that is receiving the trap (trap director enabled) enable all alert debugging for traps coming from a specific IP address; send the following CLI commands, substituting the VNM model's handle:

  1. Turn on debug
    1. update action=0x00010291 mh=<model handle of VNM>
      1. Run this to capture trap stats prior to processing
      2. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding (most of these counters can also be seen as model attributes on the VNM model)
    2. update action=0x10245 mh=<VNM mh> index=0,attr=1,type=0x13,val=aaa.bbb.ccc.ddd
        1. aaa.bbb.ccc.ddd = the actual IP address of the device sending the trap
        2. This needs to be running when the trap is received
        3. This will enable alert manager debug for all traps received from the device at the IP address specified in the command

 

Destination SS

On the final destination SpectroSERVER (SS where trap recipient is modeled) send the following CLI commands:

  1. Turn on debug
  2. update action=0x00010291 mh=<model handle of VNM>
    1. Run this to capture trap stats prior to processing
    2. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding (most of these counters can also be seen as model attributes on the VNM model)
  3. update action=0x10245 mh=<VNM mh> index=0,attr=1,type=0x13,val=aaa.bbb.ccc.ddd
    1. This needs to be running when the trap is received
    2. This will enable alert manager debug for all traps received from the device at the IP address specified in the command

 

 

** Now send the trap from a device on a remote landscape that you know is modeled in spectrum**

 

 

Trap Director SS

Once the trap has been sent make sure the debug is disabled to prevent the VNM.OUT file from filling up. In addition we also want to capture the contents of the mux cache and dump the Alert Manager statistics again.

 

  1. To turn off the debug by sending action 0x10246 to the VNM model.
    1. update action=0x10246 mh=<VNM mh>
  2. To capture the contents of the mux cache
    1. update action=0x0001011c mh=<VNM mh>
      1. This should be run after the trap has been received
  3. Dump the Alert Manager Statistics
    1. update action=0x00010291 mh=<model handle of VNM>
      1. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding (most of these counters can also be seen as model attributes on the VNM model)

 

Destination SS

Turn off debugging and dump the Alert Manager statistics again

  1. To turn off the debug by sending action 0x10246 to the VNM model.
    1. update action=0x10246 mh=<VNM mh>
  2. Dump the Alert Manager Statistics
    1. update action=0x00010291 mh=<model handle of VNM>
      1. This will dump the alert manager statistics, such as how many traps received/processed, plus some more counters covering remote trap forwarding

 

 

 

 

Sample Debug output from Trap Director SS:

 

The following output is intended to facilitate debugging, please forward

to CA technical support. The files listed are not part of the customer

installation of SPECTRUM.

**************************************************************************

Output from Alert Manager Stat dump

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 2

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 1

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:07:03 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Output from Alert Manager debug

Jun 18 13:07:26 ALERTMGR TRACE at CsAlertMgr.cc(3201): Alert Manager Debugging Enabled

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertMgMT.cc(1066): Alert manager received a trap:

  source address: 123.4.5.6

  alert code: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertMgMT.cc(1263): queueing trap to be forwarded to remote server

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(829): searching remote cache...

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(847): ...done

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(897): CsAlertFwdMgr::forward_trap_to_remote_servers() - forwarding the trap to 1 remote models.

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(922): forwarding trap to remote model: 0x700069

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(947): trap forwarding done :

  forward list count: 1

  ignore list count : 0

Jun 18 13:09:07 ALERTMGR TRACE at CsAlertFwdMgr.cc(1149): forwarding processing complete

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertMgMT.cc(1066): Alert manager received a trap:

  source address: 123.4.5.6

  alert code: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertMgMT.cc(1263): queueing trap to be forwarded to remote server

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(829): searching remote cache...

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(847): ...done

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(897): CsAlertFwdMgr::forward_trap_to_remote_servers() - forwarding the trap to 1 remote models.

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(922): forwarding trap to remote model: 0x700069

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(947): trap forwarding done :

  forward list count: 1

  ignore list count : 0

Jun 18 13:09:09 ALERTMGR TRACE at CsAlertFwdMgr.cc(1149): forwarding processing complete

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertMgMT.cc(1066): Alert manager received a trap:

  source address: 123.4.5.6

  alert code: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertMgMT.cc(1263): queueing trap to be forwarded to remote server

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(829): searching remote cache...

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(847): ...done

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(897): CsAlertFwdMgr::forward_trap_to_remote_servers() - forwarding the trap to 1 remote models.

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(922): forwarding trap to remote model: 0x700069

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(947): trap forwarding done :

  forward list count: 1

  ignore list count : 0

Jun 18 13:09:10 ALERTMGR TRACE at CsAlertFwdMgr.cc(1149): forwarding processing complete

Jun 18 13:09:40 ALERTMGR TRACE at CsAlertMgr.cc(3325): Alert Manager Debugging Disabled

Output from second Alert Manager stat dump

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 2

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 4

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:09:45 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Output from dump of Mux cache

Jun 18 13:10:39 : The following landscapes are participating in trap forwarding:

              Landscape 0x100000

              Landscape 0x400000

              Landscape 0x700000

Jun 18 13:10:39 : The remote trap forwarding cache contains 1 entries.

Cache entry(1):

              model handle = 0x700069

              address = 123.4.5.6

              secure domain =

              traps received = 4

              trap storm size = 20

              trap storm length = 5

              trap storm squelch = 0

              map using IP header = FALSE

              registers for alerts = TRUE

              ignore traps = FALSE

              trapStormHistory is 5 long

              Trap count per bucket is:

              bucket1=0

              bucket2=1

              bucket3=0

              bucket4=1

              bucket5=1

              historyUpdateTime = 1434647350

              lastVisitedTime = 1434647350

              storm started = 0

              cache insert time = Thursday, June 18, 2015 1:03:47 PM

              remote landscape unresponsive = FALSE

 

 

Sample Debug output from Destination SS

SPC-SHD-29021:

**************************************************************************

The following output is intended to facilitate debugging, please forward

to CA technical support. The files listed are not part of the customer

installation of SPECTRUM.

**************************************************************************

Output from Alert Manager stat dump

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 1

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 1

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:03:19 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Output from Alert Manager debug

Jun 18 13:03:40 ALERTMGR TRACE at CsAlertMgr.cc(3201): Alert Manager Debugging Enabled

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(2089): alert manager is processing trap: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(3400): alert is handled by default alert handler

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(3605): did not find any alert entries which would handle the alert, generating unknown alert event

Jun 18 13:03:58 ALERTMGR TRACE at CsAlertMgr1.cc(2163): failed to handle alert

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(2089): alert manager is processing trap: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(3400): alert is handled by default alert handler

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(3605): did not find any alert entries which would handle the alert, generating unknown alert event

Jun 18 13:03:59 ALERTMGR TRACE at CsAlertMgr1.cc(2163): failed to handle alert

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(2089): alert manager is processing trap: 1.3.6.1.4.1.9.9.1.6.5

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(3400): alert is handled by default alert handler

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(3605): did not find any alert entries which would handle the alert, generating unknown alert event

Jun 18 13:04:00 ALERTMGR TRACE at CsAlertMgr1.cc(2163): failed to handle alert

Jun 18 13:04:48 ALERTMGR TRACE at CsAlertMgr.cc(3325): Alert Manager Debugging Disabled

Output from second Alert Manager stat dump

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3689): Alert manager statistics:

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3695):  traps received                                          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3697):  traps handled which were received locally              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3699):  traps handled which were forwarded from a remote server : 4

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3701):  traps processed locally                                : 4

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3703):  traps discarded locally due to trap storm              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3705):  traps ignored                                          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3709):  alerts which were tried to be forwarded (local as well) : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3711):  alerts which were solely tried to be forwarded          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3713):  distributed find attempts                              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3715):  alerts forwarded successfully                          : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3717):  alerts forwarding failures                              : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3719):  alerts discarded remotely due to trap storm            : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3721):  alerts ignored in forward processing                    : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3726):  alert queue length                                      : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3728):  remote trap forwarding queue length                    : 0

Jun 18 13:04:56 ALERTMGR TRACE at CsAlertMgr.cc(3732): Event processing queue statistics:

Number of work queue nodes: 0

Number of model event queues: 0

Statistics
0 Favorited
19 Views
0 Files
0 Shares
0 Downloads

Related Entries and Links

No Related Resource entered.