Event Processing

Document created by MichielHelder Champion on Jan 6, 2012Last modified by SamCreek on Dec 17, 2016
Version 8Show Document
  • View in full screen mode

Back to:

CA Spectrum

IM Community WIKI Front Page


Events under Control

Events are stored in files like Eventffff0001 and referenced by EventDisp files.

Everytime you edit such files (via Spectrum tools or manually) you have to update Spectrum to re-read these files. Doing so you have a chance to get "notified" about errors and/or warnings regarding your EventDisp files by adding this to the .vnmrc file and restart SS:

event_disp_error_file=/tmp/event_parse.out

Of course you can change the output filename to whatever you want (the example above is Unix style).


SS will write to that file every time you update event configuration via SpectroSERVER Control and when it detects an error or warning during event processing.

 

How is the Event generated and get alarmed?

Spectrum events are created for any interesting or potentially interesting occurrence that the SpectroSERVER is aware of. Events that operators need to be made aware of trigger the creation of Alarms.

A Spectrum event is a small data structure. It contains a type code, a timestamp and from 0 to many event variables. It is always associated with exactly one model. Most are logged to the Archive Manager's data base (DDMDB) for long term storage and can be retrieved and displayed by applications acting as clients of Archive Manager.

Most events represent occurrences or state changes on devices in the managed network. Spectrum monitors the network by exchanging SNMP packets with devices. The exchange is both through periodic polls of the devices initiated by the SpectroSERVER (SNMP GET <> SNMP RESPONSE) and unsolicited alerts sent from the devices for conditions that they have been configured to notify Spectrum of (SNMP TRAP).


AlertMap and EventDisp Files

Two types of files determine how events are handled by the SpectroSERVER. Alert Map files (all called AlertMap) define what event type is created when a given trap type is received for a modelled device. Event Disposition files (all called EventDisp) define the behaviour that results when an event of a given type is created.

These files are distributed under $SPECROOT/CsVendor  in directories that represent vendors. The entries from all AlertMap end EventDisp files directly under CsVendor/<vendor> directories are compiled into a list and apply equally to all models. They are said to be global. There are also subdirectories under the vendor directories named for Spectrum model types. Any AlertMap or EventDisp file in a CsVendor/<vendor>/<model type> directory will only be applied to models of that type.

When a trap arrives at the SpectroSERVER Spectrum first determine whether there are alert mappings and event dispositions that apply. The sequence of events that determine the proper event and disposition when a trap arrives is:

  • A trap arrives at the SpectroSERVER.
    Spectrum checks whether there is a model in the SpectroSERVER data base for the agent IP address indicated in the trap (if Alert Forwarding is configured on the VNM it will check other SpectroSERVERs in the DSS and forward the trap if appropriate).
    • If no device model is found an event is generated on the VNM model with all of the trap details and the sequence stops.
    • If a device model is found Spectrum checks for an AlertMap entry specific to the model's type and then for a global entry.
    • If no entry is found an "Unmapped trap" event is generated on the target device model with all of the trap details and the sequence stops.
    • If an entry is found the mapped event is generated on the target device model.

 

For the generated event type Spectrum checks for an EventDisp entry specific to the model's type and then for a global entry. Spectrum executes the steps defined in the EventDisp entry

 

To customize how the SpectroSERVER behaves in response to traps and events it is necessary to edit the appropriate AlertMap or EventDisp files. It may be convenient to create a new vendor directory under $SPECROOT/CsVendor for a user's files.

 

Here is an example AlertMap Entry:

Alert Code                      Event Code  OID Mappings

1.3.6.1.4.1.9.9.13.3.6.5        0x180000    1.3.6.1.4.1.9.9.13.1.5.1.2(1,2) \

                                            1.3.6.1.4.1.9.9.13.1.5.1.3(3,0)

     

 

There are in fact two mappings that are defined here. First the trap with enterprise oid 1.3.6.1.4.1.9.9.13.3, generic type 6 and specific type 5 is mapped to event type 0x180000. Second the varbinds carried in the trap are mapped to the event variables indicated within the parenthesis. The value of the varbind with oid 1.3.6.1.4.1.9.9.13.1.5.1.2 is copied to event variable 1, and that of 1.3.6.1.4.1.9.9.13.1.5.1.3 to event variable 3. The event variable indicated following the comma holds the index portion of the oid if it is desired. The event variables are referenced by small integer values and do not need to be mapped in order, although it is more readable if you do so. A zero in one of these positions indicates that the value or the index is not to be saved. Also the trap varbinds are referenced by their oid and do not need to be mapped in the order they are defined in the trap. The AlertMap does not need to account for every varbind in the trap, only those of interest.

Here is an example EventDisp Entry:

<eventcode> E <eventseverity> A <alarmseverity>,<alarmcause>,U,N,T

0x180000    E 50              A 2,              0x180000,    N

     

The capitol letters have predefined meaning and if present must be in the relative position depicted. This entry states that when an event of type 0x180000 occurs, it will be logged (E) with a severity of 50. Also it will generate an alarm (A) of severity Major (2) and of alarm type 0x180000. The alarm is not user clearable (N).

The E can be present or absent. If absent the event is not logged to the Archive Manager and will not be observed in any client view unless the client is displaying the event history when the event occurs. The event severity is not currently used but is logged with the event. If the E is absent the severity should also be absent.

The A indicates that when this event occurs an alarm should be generated. Following the A are the alarm severity, alarm cause (type) and three qualifiers that can appear in any combination. The alarm type does not need to have the same value as the type of the event that generates it, however making them equal when possible enhances readability. The possible qualifiers are:

  • U - A unique alarm should be generated for each occurrence of this event (otherwise no new alarm is generated for subsequent occurrences, instead the existing alarm is noted)
  • N - The alarm is not user clearable
  • T - the alarm is not persistent (it will not be maintained through a SpectroSERVER restart)

In place of the A there can be a C. The C would be followed by an alarm cause value and indicates that when the event occurs it should clear an alarm of the type indicated if one exists on the model.

 

In place of the U one or more event variables can be listed as event discriminators:

<eventcode> E <eventseverity> A <alarmseverity>,<alarmcause>,<discriminators>,N,T

When discriminators are specified then they are considered together with the alarm cause in determining the effective type of the alarm. Consider the following:

0x180000 E 50 A 2,0x180000, 2,3,N

0x180001 E 50 C  0x180000, 2,3

     

This states that event 0x180000 should generate an alarm of type 0x180000 and that the values for event variables 2 and 3 should be saved with the alarm. If event 0x180000 occurs again and there is already an alarm present, but the new values of event variables 2 and 3 are not both equal to those stored with the alarm, then a new alarm will be generated, even though the unique indicator is not specified. The new alarm is considered to be of a different type since the values for the discriminators are different. Similarly if event 0x180001 occurs, it will only clear an alarm on this model of type 0x180000 if that alarm was generated with the same values of event variables 2 and 3 that were stored when the alarm was created. The specification of the discriminators with the clear event is required.


Event Rules

An event can also be disposed to invoke one of the event rules EventPair, EventRateWindow, EventRateCounter, EventSequence, EventCombo or EventCondition. For details refer to the Event Configuration Files Guide. The format is as follows:

  • <FirstEvent> R Aprisma.EventPair, <SecondEvent>, <GeneratedEvent>, <time>
  • <TriggerEvent> R Aprisma.EventRateWindow, <NumberofOccurrences>, <time>,<GeneratedEvent>
  • <TriggerEvent> R Aprisma.EventRateCounter, <NumberofOccurrences>, <time>,<GeneratedEvent>
  • <FirstEvent> R Aprisma.EventSequence, <GeneratedEvent>, <time>, <SecondEvent>, <ThirdEvent>, ...<NthEvent>
  • <TriggerEvent> R Aprisma.EventCombo, <GeneratedEvent>, <time>, <EventA>, <EventB>, ...<EventN>
  • <TriggerEvent> R Aprisma.EventCondition, "<conditional expression 1>", <event to generate when 1 is TRUE>, ..."<conditional expression n>", <event to generate when n is TRUE>, "default", <default event>

All of the rules except the EventCondition rule define a combination of events on a model that must occur within a time interval and when this is observed a new resulting event is generated.

The EventCondition rule allows for event variables and model attributes to be compared against constant values, regular expressions, or each other, and resulting events to be generated based on the results of the comparisons.

All AlertMap and EventDisp files are read when the SpectroSERVER is started. A reload of the files can be triggered from the command line interface by sending action code 0x100a2 to the VNM model. Alternatively as of Spectrum release 8.0 there is a button to reload events in the SpectroSERVER Control view under the Information Tab for the VNM model in a OneClick Client.

A very useful mechanism for debugging edits to EventDisp files is available by specifying a file name for the event_disp_error_file entry in the .vnmrc file. Once specified any errors encountered while parsing or loading EventDisp entries will be recorded in this file.

There is a legacy application, the Event Configuration Editor (ECE), available to assist in editing EventDisp files. It does not address AlertMap files. This will be ported to OneClick in a future release.


Event Format and Probable Cause Files

There are two other types of files that determine how events and alarms are presented by client applications (now primarily the OneClick server). Event Format files (EventXXXXXXXX) define the text presented for a given event type. They support a mechanism to display the values of the event variables. Probable Cause files (ProbXXXXXXXX) define the text presented for a given alarm type. They do not have a mechanism for any variable content. Unlike the AlertMap and EventDisp files these are all uniquely named with XXXXXXXX being the hex representation of the event code or alarm cause value respectively. These are all under $SPECROOT/SG-Support, either on the OneClick server machine or for the legacy event and alarm clients on each SpectroSERVER.

The Event Format files specify the text presented for a given event type and support an "escape sequence" of the form {<type> <variable id>} to insert event variable values in the text at the logical location. There are also several general variables that can be accessed such as date, model name, model type and event type. For example:

{d "%w- %d %m-, %Y - %T"} Model {m} of type {t} has reported that a PSA microcode reload has occurred for entity {S 1}: {S 2}

This entity table index is {I 4}

The current bundle id is {I 3}

The previous bundle id was {I 5}

Newly enabled features: {T PsaMicrocodeFeatures 6}

Newly disabled features: {T PsaMicrocodeFeatures 7}

(event [{e}])

     

The Probable Cause files specify the text presented for a given alarm type. They do not support variable content but the adhere to a standard format. The first line of the file, usually all capitalized, is used as the "Alarm Title". For Example:

PSA MICROCODE RELOAD

SYMPTOMS:

A trap has been received indicating an entity has experienced a microcode reload

PROBABLE CAUSES:

Planned configuration change

RECOMMENDED ACTIONS:

Verify required features were not disabled

     

 

Event Format and Probable Cause files are not used by the SpectroSERVER and customizations do not require any action there. Instead the client application (legacy Event or Alarm views or the OneClick server) will reread them upon a restart. As of Spectrum release 8.1 there is a button to reload these files to the OneClick server on the Administration web page under the link titled "EvFormat/PCause".

 

Net-SNMP trap utility

When developing new trap mappings or event dispositions or when debugging unexpected behavior in response to traps, it is helpful to be able to generate the traps in question on demand. The trap utility included with the Net-SNMP utilities allows you to send a trap and specify the agent address that it will carry.
Here is an excerpt from the usage for this utility:

./snmptrap -h

USAGE: snmptrap [OPTIONS] AGENT TRAP-PARAMETERS

OPTIONS:

-h, --help display this help message

-v 1|2c|3 specifies SNMP version to use

-c COMMUNITY set the community string

-v 1 TRAP-PARAMETERS:

enterprise-oid agent trap-type specific-type uptime [OID TYPE VALUE]...

or

-v 2 TRAP-PARAMETERS:

uptime trapoid [OID TYPE VALUE] ...

    

The argument "agent" is used twice with opposite meaning. The upper case "AGENT" in the first usage line refers to the management entity, in our case the SpectroSERVER. The lower case "agent" under v 1 TRAP-PARAMETERS refers to the device agent. This is where you specify the source of the trap.


This information comes from CA and is covered in the official documentation and on TechDocument ID: TEC444071 on CA Support


How to generate a trap

The command under any Unix is:

snmptrap -v 1 -c public 10.0.1.160 .1.3.6.1.4.1.28116.20 10.0.1.250 6 5247 13:33:00 1.3.6.1.4.1.28116.20.1 s "Test Trap"

    

Explanation: -v Version of SNMP (here SNMP version 1) -c Community - Agent address - OID (.1.3.6.1.4.1.28116.20 ist a Sub-Range, the 1.3.6.1.4.1 is iso.org.dod.internet.private.enterprise as defined in RFC, and 28116 is the OID from 4-tune) - Sender IP Address (where does it come from) - 6 means Generic Trap (0=coldStart, 1=warmStart, 2=linkDown, 3=linkUp, 4=authentication Failure, 5=egpNeighborLoss) - Trap ID (This Identificationnummer may be chosen without limits) - time (actual time HH:MM:SS) - OID for Databinding (you may take whatever you like. I take a continuous number..) - Type (a=IP Address, c=counter, d=decimal string, i=integer, o=Object ID, s=string, t=Time ticks, u=Unsigned integer, x=Hexadecimal string) - Value (depending on Type)

 

How to make sure something is connected?

If you are afraid of getting "Bad Link Detected" alarms you could put this example rule in place:

0x00010d11 E 50 R Aprisma.EventCondition, \

     "{ I 0 } == { attr 0x12a79 }", "0xffffffff -:-", \

     "default", "0x0001040a -:-"

0xffffffff

0x0001040a E 50 A 3, 0x0001040a

 

Eventrule to generate an alarm (fffffff1) if an event (ffffffff) is not generated in 10 Minutes

 

Something like the following should work:

0xffffffff R Aprisma.Eventpair, 0xffffffff, 0xfffffff1, 600

 

Of course fffffff1 needs to be defined as an alarm.

 

Thanks to John O'Mahony

 


All of the below comes from Rob Edberg's Wiki (http://spectrum.knowsitall.info/wiki/) Thank you Rob!!
This little config uses SEB's OID to create a generic snmptrap send/receive solution which can be used by almost anything.

 

Net-Snmp trap and Inform integration

Net-SNMP is a opensource snmp package that includes both a agent that Spectrum supports and a set of tools. The below command syntax uses the net-snmp snmptrap command. Net-SNMP support a wide range of operating systems and is standard in Solaris, Linux.

 

Command syntax Trap

This will create a Critical alarm in Spectrum:

c:\usr\bin\snmptrap -v 1 -c snmpcommunity spectroserver 1.3.6.1.4.1.10473.100.6 "" 6 0 "" \

1.3.6.1.4.1.10473.100.6.1.0 s "testdevice" \

1.3.6.1.4.1.10473.100.6.2.0 i 4 \

1.3.6.1.4.1.10473.100.6.3.0 s "Something is fishy in Denmark" \

1.3.6.1.4.1.10473.100.6.4.0 s "VMS_ALERT_TEST" \

1.3.6.1.4.1.10473.100.6.5.0 s "AlarmID: 12345" \

1.3.6.1.4.1.10473.100.6.6.0 s “http://systemsmanagement.sebank.se”

    

This will clear the same alarm:

c:\usr\bin\snmptrap -v 1 -c snmpcommunity spectroserver 1.3.6.1.4.1.10473.100.6 "" 6 0 "" \

1.3.6.1.4.1.10473.100.6.1.0 s "testdevice" \

1.3.6.1.4.1.10473.100.6.2.0 i 0 \

1.3.6.1.4.1.10473.100.6.3.0 s "Something is fishy in Denmark" \

1.3.6.1.4.1.10473.100.6.4.0 s "VMS_ALERT_TEST" \

1.3.6.1.4.1.10473.100.6.5.0 s "AlarmID: 12345" \

1.3.6.1.4.1.10473.100.6.6.0 s “http://systemsmanagement.sebank.se”

    

Net-Snmp INFORM integration

The big problem with TRAPs is that they're unacknowledged . SNMPv2 PDUs fixed this by introducing the notion of an INFORM. which is nothing more than an acknowledged TRAP. IE, when the remote application receives the INFORM it sends back an "I got it" The text and information is from: Net-SNMP

With this command you get a good return code when Spectrum received the Message. An returm message is sent on port 162 udp.

./snmptrap -v 2c -Ci -c public SpectroServer "" 1.3.6.1.4.1.10473.100.6.6.0 \

1.3.6.1.4.1.10473.100.6.1.0 s "testdevice" \

1.3.6.1.4.1.10473.100.6.2.0 i 0 \

1.3.6.1.4.1.10473.100.6.3.0 s "Something is fishy in Denmark" \

1.3.6.1.4.1.10473.100.6.4.0 s "VMS_ALERT_TEST" \

1.3.6.1.4.1.10473.100.6.5.0 s "AlarmID: 12345" \

1.3.6.1.4.1.10473.100.6.6.0 s “http://systemsmanagement.sebank.se”

    


Explanation


100.6.1.0

This varbind is only used if the complete Southbound GW logic is installed, which isn't the case here. This snmptrap will only create events and alarms on the host which send the trap.

 

100.6.2.0

This the severity:

4 = Red alert
3 = Orange alert
2 = Yellow alert
1 = Event message only
0 = Clear alarm

Regarding Clear alarm: It will clear the alarm were varbind 6.4.0 and 6.5.0 matches - only.


100.6.3.0

Event message varbind. Put anything you want here.


100.6.4.0

This is a unique alarm identifier, which can be supplied through this varbind or if none is supplied - be autocreated using oneclick xml customizations below. This is a solution align with having external "Operation manual per alert".


100.6.5.0

A unique ID. If for example you forward alert from a third party monitoring system, then add the specific alarmID from here from that system and when the alarms is cleared in the third party system send a new trap with severity 0 and the same alarmID, and the alarm in Spectrum will autoclose.


100.6.6.0

If you put a web url in this varbind, oneclick alarmview will autocreate a clickable button which will open this url, if clicked.

 

Spectrum Event Config


<spec_home>/SS/CsVendor/SEB_DATA/AlertMap

##################################################################

# Localhost Custom Net-SNMP Traps (by Roberth Edberg, 2006.11.23)

##################################################################

1.3.6.1.4.1.10473.100.6.6.0 0x00f1002f 1.3.6.1.4.1.10473.100.6.1.0(16,0)\

                                        1.3.6.1.4.1.10473.100.6.2.0(101,0)\

                                        1.3.6.1.4.1.10473.100.6.3.0(102,0)\

                                        1.3.6.1.4.1.10473.100.6.4.0(142,0)\

                                        1.3.6.1.4.1.10473.100.6.5.0(143,0)\

                                        1.3.6.1.4.1.10473.100.6.6.0(0x12a63,0)

  


<spec_home>/SS/CsVendor/SEB_DATA/EventDisp

##################################################################

# Localhost Custom Net-SNMP Traps (by Roberth Edberg, 2006.11.23)

# Related and will not work witout logic below

##################################################################

0x00f1002f R Aprisma.EventCondition, \

"({v 101} == {I 4})", "0x00f1001c 16:16,101:101,102:102,142:142,143:143,76387:76387", \

"({v 101} == {I 3})", "0x00f1001d 16:16,101:101,102:102,142:142,143:143,76387:76387", \

"({v 101} == {I 2})", "0x00f1001e 16:16,101:101,102:102,142:142,143:143,76387:76387", \

"({v 101} == {I 1})", "0x00f1001f 16:16,101:101,102:102,142:142,143:143", \

"({v 101} == {I 0})", "0x00f10020 16:16,101:101,102:102,142:142,143:143"

 

0x00f1001c E 90 A 3,0x00f10019,142,143,76387 # Critical alarm

0x00f1001d E 70 A 2,0x00f10019,142,143,76387 # Major alarm

0x00f1001e E 50 A 1,0x00f10019,142,143,76387 # Minor alarm

0x00f1001f E 30 # Log event only

0x00f10020 E 30 C 0x00f10019,142,143 # Clear alarm

  


<spec_home>/SG-Support/CsPCause/Prob00f10019

THIRD PARTY EVENT

 

SYMPTOMS:

 

This alarm was received through an event gateway

 

PROBABLE CAUSES:

 

1) Check event information

2) Check separate Focal instructions if key was parsed with the event

 

RECOMMENDED ACTIONS:

 

1) Check event information

2) Check separate Focal instructions if key was parsed with the event

  


<spec_home>/SG-Support/CsEvFormat/Event00f1002f & Event00f1001f

{d "%w- %d %m-, %Y - %T"} - Host = '{S 16}', Action = '{S 101}', Message = '{S 102}', Fokalkey = '{S 142}', Identifier = '{S 143}' - [event: {e}]

  

 

<spec_home>/SG-Support/CsEvFormat/Event00f1001c, Event00f1001d, Event00f1001e & Event00f10020

{d "%w- %d %m-, %Y - %T"} {S 102} - [event: {e}]

  

 

Oneclick Alarmview Customization


<spechome>/custom/alarm/config/alarm-table-config.xml

Create two new columns in alarmview that shows the new varbinds.

<!--

  Begin additions by SEB, Roberth Edberg

-->

 

  <column>

    <name>Focalkey (SEB)</name>

    <content>

      <attribute>0x8e</attribute>

      <expression>

        ( value() == null || value().toString().length() == 0 ) ?

            (Long.toHexString(attrLong(0x11f50)) + "_SPECTRUM_EVT") : value()

      </expression>

    </content>

    <default-width>115</default-width>

  </column>

 

 

  <column>

    <name>UniqueID (SEB)</name>

    <content>

      <attribute>0x8f</attribute>

    </content>

    <default-width>115</default-width>

  </column>

 

<!--

  End additions by SEB, Roberth Edberg

-->

  


<spechome>/custom/console/config/attr-config.xml

Register the two varbinds.

<config>

  <attribute>

    <id>0x8e</id>

    <value-type>CSC_OCTET_STRING</value-type>

  </attribute>

  <attribute>

    <id>0x8f</id>

    <value-type>CSC_OCTET_STRING</value-type>

  </attribute>

</config>

  


<spechome>/custom/console/config/custom-menu-config.xml

You are also able to add a button in the top of Oneclick is clicked will open a url using the new varbinds as arguments. Example:

<menu name="com.aprisma.spectrum.app.swing.window.menu.Tools">

  <item name="Show Focal instruction">

    <toolbar-image>

      /images/SEB_DATA/aiwa.png

    </toolbar-image>

    <action>

      <context>com.aprisma.spectrum.app.alarm.client.group.AlarmContext</context>

      <launch-browser>

        <url>http://corpintra.sebank.se/ITService/alarmdb.nsf/controlcenter/{1}?OpenDocument&hostname={0}&</url>

        <param>

          <attribute>0x1006e</attribute>

        </param>

        <param>

          <attribute>0x8e</attribute>

          <expression>

            ( value() == null || value().toString().length() == 0 ) ?

              (Long.toHexString(attrLong(0x11f50)) + "_SPECTRUM_EVT") : value()

          </expression>

        </param>

      </launch-browser>

    </action>

  </item>

</menu>

  

 

The icon button image should exist here: <spechome>/custom/images/SEB_DATA/aiwa.png

 

Alarm Notifier Script

These new varbind are usable also in Notifier scripts.

The below has to be included in Alarm Notifier config (.alarmrc file).

USE_NEW_INTERFACE=true EXTRA_ATTRS_AS_ENVVARS=0X8E,0X8F

 

AND within the (Set|Update|Clear) script the varbind's will be used as $SANM_0X8E, $SANM_0X8F

AND if the information contained is nonreadable, use this little function:

if [ "$SANM_0X8F" = "N/A" ] ; then

         UNIQEID=""

else

         UNIQEID=`<spechome>/vnmsh/sample_scripts/octet_to_ascii.pl $SANM_0X8F | cut -c10-`

fi

 
4 people found this helpful

Attachments

    Outcomes