DX NetOps Manager

Back to Library

Alarming

Recommend

Jan 06, 2012 07:03 AM

Anon Anon

Back to:

CA Spectrum

IM Community WIKI Front Page

How to show Alarmtitle in Email Header or SMS

You will have to edit your SetScript, which is found in the $SPECROOT/Notifier directory. In there add a line like this:

ALARMTITLE=`echo "$PCAUSE" | /usr/bin/head -1`

After that you will have to change your line with $mail -s and add $ALARMTITLE to it.

Make sure you have set in the .alarmrc the switch GET_PROBABLE_CAUSES=true!!

It is so easy but has been asked several time and it is hard to find if you don't know where to look at.

Information about Bad Link Alarms

PollPortStatus (0x1280a) - Controls status polling of a port whose connectivity is not modeled. Valid values are TRUE or FALSE.

ok_to_poll (0x11dd8) - Used to make the pipe associated with this port "live". The status of the port will be polled. Valid values are TRUE or FALSE.

AlarmOnLinkDownTrap (0x11fc2) - Determines how SPECTRUM handles a Link Down trap on this particular port (0 for ignore the trap, 1 for process the trap).

AssertLinkDownAlarm (0x12957) - Determines if SPECTRUM should generate a yellow alarm on the device model when a link down trap is received for this port. Valid values are TRUE or FALSE.

GeneratePortStatusAlarms (0x12a54) - Used to tell SPECTRUM that a port status alarm should never be generated on this port. Valid values are TRUE or FALSE.

Info provided by Stephen Warne

How to check Host via special port?

Here is a small shell script to monitor a remote system via telnet on a defined port. You may change this script to you needs and use it as usual free of charge ;-)

#!/bin/sh host=127.0.0.1 port=3389 stunde=`date +%H` minute=`date +%M` sekunde=`date +%S` zeit=$stunde:$minute:$sekunde  ( echo open $host $port sleep 2 echo quit )| telnet > result  if [ "$result" = "Connection closed by foreign host." ]   then   snmptrap -v 1 -c public 10.0.1.160 .1.3.6.1.4.1.28116.20 10.0.1.250 6 5247 $zeit 1.3.6.1.4.1.28116.20.1 s "host could not get connected" fi

Please change on the line snmptrap the ip after public, which should be your SpectroServer address. The OID string followed is the one of 4-tune which was used here as an example, the number 6 means generic trap (aka Enterprise specific trap) followed by a unique trap id which may be more or less any number. The s after the OID means textstring which is quoted just after this.

All you have to do is map this trap in an AlertMap file and create the appropriate Event and Alarm for this..

How to send Notification for specific Devices only to somebody?

Use SANM which is in your $SPECTRUM/SANM (start the PolicyAdmin). Open a policy or create a new one. Define the filter as required. In the Notification field put in the e-mail addresses you wish (separated by comma). Save it and select the policy you just created or changed, on the left side you select AlarmNotifier and click on Associate (in the middle).

# Guide if you are running AlarmNotifier on Windows with mail enabled
#
####################################################################
SENDMAIL=True #True or False
VARFORMAIL=Both #RepairPerson, NotificationData, or Both

Change to the $SPECTRUM/Notifier/ and edit the SetScript. At the end of the first part which is documented with # you will find sendmail=True/False make sure it is set to True! In the next line you will find VARFORMAIL. Make sure it is set to both.

How to check for the presence of a file?

This little script should do it. Please change it to your needs.

#!/bin/bash while true do   if test -e /directory/checkfile;   then   snmptrap -v 1 -c public 10.0.1.160 .1.3.6.1.4.1.28116.20 10.0.1.250 6 5247 13:33:00 1.3.6.1.4.1.28116.20.1 s "file exists";   fi   sleep 5 done

This will check every 5 seconds if the file checkfile exists, if it does it will generate a trap. All you have to do in Spectrum is creating the event- and alarmdefinition with ECE and do not forget to add a line to the AlertsMap..

How to start Alarm Notification Services automatically?

If you would like to get your Alarm Notifier started when your SpectroServer is started follow these simple steps:

Change into the $SPECROOT/lib/SDPM/partslist directory on your Spectro-Server
Copy one of the .idb files and name the copy AlarmNotifier.idb
Change the content of the file so it looks like this:

# Processd Install Ticket for Notification Service.
PARTNAME;AlarmNotifier;
APPNAME;Alarm Notification Service;
WORKPATH;$SPECROOT/Notifier;
LOGNAMEPATH;$WORKPATH/NOTIFIER.OUT;
ADMINPRIVS;y;
AUTORESTART;y;
AUTOBOOTSTART;y;
#STATEBASED;N;
NUMPROCS;1; // one per host
RETRYTIMEOUT;30; // 30 seconds
#TICKETUSER;$USER;
RETRYMAX;5; // 5 retries allowed
STARTPRIORITY;30;
SERVERPROCESS;Y;
#ENV;<var>=<value>;
ARGV;$WORKPATH/AlarmNotifier<CSEXE>; //

Note the line AUTOBOOTSTART;y; this will make Notifier auto-start the next time your processd is (re)started...

How to get informed about Alarms automatically cleared?

We all know this situation. We come to the office, have a look into our Spectrum and all is green. What a nice day! But half an hour later the responsible guy for backup is telling us, there have been problems during backup. I think it is network related. How can we find out if there was some errors during night or our absence?

Create a Global Collection
1. Attribute FlashGreen=TRUE
2. Acknowledged=NO
3. Condition=NORMAL
Set the update interval to 24 hours

Now you will see in your newly created collection those devices which had an alarm and recovered already so having the status green (normal). To reset the attribute FlashGreen, select all devices in that collection change the setting via Utilities, Attribute Editor back to NO.

How to reduce Bad Link Detected alarms

This will prevent Spectrum to create BAD-LINK detected alarms when the user turn-off their workstations based on the link type. This steps will produce alarms only when the Bad Link has an impact on the network. Please make sure all physical connections between devices are properly modeled!

in policy-rules.xml:

<policy-rule name="Default Router Link Fault Management" >
<port-policy-search-criteria-name name="Router Link Ports" />
<policy-settings-name name="Live Pipes" />
</policy-rule>

<policy-rule name="Default Switch Link Fault Management" >
<port-policy-search-criteria-name name="Switch Link Ports" />
<policy-settings-name name="Passive Port Monitoring" />
</policy-rule>

<policy-rule name="Default Shared Access Link Fault Management">
<port-policy-search-criteria-name name="Shared Access Link Ports" />
<policy-settings-name name="Disable Port Monitoring" />
</policy-rule>

<policy-rule name="Default End Station Link Fault Management" >
<port-policy-search-criteria-name name="End Station Link Ports" />
<policy-settings-name name="Disable Port Monitoring" />
</policy-rule>

<policy-rule name="Default Wide Area Link Fault Management" >
<port-policy-search-criteria-name name="Wide Area Link Ports" />
<policy-settings-name name="Live Pipes" />
</policy-rule>

<policy-rule name="Default Unknown Link Fault Management" >
<port-policy-search-criteria-name name="Unknown Link Ports" />
<policy-settings-name name="Live Pipes" />
</policy-rule>

<policy-rule name="Default Unlinked Port Fault Management" >
<port-policy-search-criteria-name name="Unlinked Ports"/> -->
<policy-settings-name name="Disable Port Monitoring" />
</policy-rule>

In policy-settings.xml (some exist in policy-settings-spectrum.xml, but have been moved):

<policy-settings name="Disable Port Monitoring">
<set-attr-value id="0x1280a" value="FALSE" />
#PollPortStatus
<set-attr-value id="0x11dd8" value="FALSE" />
#ok_to_poll
<set-attr-value id="0x11fc2" value="0" />
#AlarmOnLinkDownTrap
<set-attr-value id="0x12957" value="FALSE" />
#AssertLinkDownAlarm
<set-attr-value id="0x12a54" value="FALSE" />
#GeneratePortStatusAlarms
</policy-settings>

<policy-settings name="Passive Port Monitoring">
<set-attr-value id="0x1280a" value="FALSE" />
#PollPortStatus
<set-attr-value id="0x11dd8" value="FALSE" />
#ok_to_poll
<set-attr-value id="0x11fc2" value="1" />
#AlarmOnLinkDownTrap
<set-attr-value id="0x12957" value="TRUE" />
#AssertLinkDownAlarm
<set-attr-value id="0x12a54" value="TRUE" />
#GeneratePortStatusAlarms
</policy-settings>

<policy-settings name="Live Pipes">
<set-attr-value id="0x1280a" value="FALSE" />
#PollPortStatus
<set-attr-value id="0x11dd8" value="TRUE" />
#ok_to_poll
<set-attr-value id="0x11fc2" value="1" />
#AlarmOnLinkDownTrap
<set-attr-value id="0x12957" value="TRUE" />
#AssertLinkDownAlarm
<set-attr-value id="0x12a54" value="TRUE" />
#GeneratePortStatusAlarms
</policy-settings>

In policy-search-criteria.xml (some exist in policy-search-criteria-spectrum.xml, but have been moved):

<policy-search-criteria name="Router Link Ports">
<attr-match match-type="equal-to" id="0x12a79" value="1"/>
</policy-search-criteria>

<policy-search-criteria name="Switch Link Ports">
<attr-match match-type="equal-to" id="0x12a79" value="2"/>
</policy-search-criteria>

<policy-search-criteria name="Shared Access Link Ports">
<attr-match match-type="equal-to" id="0x12a79" value="3"/>
</policy-search-criteria>

<policy-search-criteria name="End Station Link Ports">
<attr-match match-type="equal-to" id="0x12a79" value="4"/>
</policy-search-criteria>

<policy-search-criteria name="Wide Area Link Ports">
<attr-match match-type="equal-to" id="0x12a79" value="5"/>
</policy-search-criteria>

<policy-search-criteria name="Unknown Link Ports">
<attr-match match-type="equal-to" id="0x12a79" value="7"/>
</policy-search-criteria>

<policy-search-criteria name="Unlinked Ports">
<attr-match match-type="equal-to" id="0x12a79" value="0"/>
</policy-search-criteria>

Thanks to James Cooleman who brought this solotion to our attention

How to have multiple alarm queues (for each type or group 1)

1. cd into your $SPECROOT/Notifier directory

2. create a new directory like "EmailNotification"

3. copy the .alarmrc, AlarmNotifier.exe, SetScript, UpdateScript, ClearScript into the newly created directory

4. rename AlarmNotifier.exe to something meaningful like MailNotifier.exe

5. open .alarmrc with your favorite editor and change the paths for SetScript etc. Change the name of the Notifier (APPLICATION field) to MailNotifier

6. either start your MailNotifier with ./MailNotifier -r $SPECROOT/Notifier/EmailNotification/.alarmrc or create an IDB file to start automatically

6.1 To create a IDB file.

cd $SPECROOT/lib/SDPM/partslist. Copy the existing NOTIFIER.idb to a new name. edit the file and change "AlarmNotifier" to your new "MailNotifier" name. If you create an IDB file you will need to restart processd to activate it.

7.1 Pre 9.x start your SANM which is in $SPECROOT/SANM/PolicyManager.exe create a new application "MailNotifier"

7.2 Post 9.x In the locater tab navigate to SANM. Double click on "All Applications". Your new notifier should appear.

8. create new policies or choose existing ones and press associate or schedule association.
If you schedule association, keep in mind you can not unassociate but you can schedule to associate on Monday morning WorkdayPolicy with MailNotifier and on Friday 5pm WeekendPolicy with MailNotifier.

Repeat these steps as you need different queues.

Missing red alarms for Pingables

In order to avoid generating massive Pingable alarms, the contact lost alarm is turned off on Pingables.

You can set Value_When_Red from 0 to 7 to generate the alarm.

How to send Syslog messages to Spectrum

I have put a small example-script in Perl to do this. It is straight forward and not complicated at all.

#!/usr/bin/perl  $offset = (-s $LOG);  while(1) { sleep(1);  open(LOGFILE, "</var/log/everything/current"); seek(LOGFILE, $offset, 0);  while($newstuff = <LOGFILE>) { if($newstuff =~ m/Error/i) { snmptrap -v 1 -c public 10.0.1.160 .1.3.6.1.4.1.28116.20 10.0.1.250 6 5247 13:33:00 1.3.6.1.4.1.28116.20.1 s "Error in Syslog detected"; } }  $offset = tell(LOGFILE); close(LOGFILE); }

All you have to do is add an entry in the AlertMap file and map a Event to this Trap. If you would like to receive an alarm put the Alarm-flag on in the ECE.

Redundant AlarmNotifier

If you have a Primary/Secondary SpectroServer Solution, there is no out-of-the-box way the setup an Fault Tolerant AlarmNotifier. With some work, this is no problem. All you have to do is to add a new attribute to the .alarmrc and edit the Set-/Update- and ClearScripts.

Every alarm has an attribute with the assiciated precedense. On a primary SpectroServer this value is normmally "10", on the backup SS it is "20". So you have to add this alarmattribute into the .alarmrc of the AlarmNotifier, so that this attribute will forwarded to the scripts:

EXTRA_ATTRS_AS_ENVVARS=0x12c0a

No you can use the attribute in you scripts. On the Primary and on the Secondary SS the same AlarmNotifier is running. But the scripts are not the same.
Add the following IF-clause to the scripts you use (normally SetScript):

On the primary SS:

if [ "$SANM_0X12C0A" = "10" ]
then
...Your commands...
fi

On the secondary SS:

if [ "$SANM_0X12C0A" = "20" ]
then
...You commands...
fi

Both AlarmNotifiers will get all alarms. But with this IF-clause the scripts check, from which SpectroServer the alarm is comming. If the primary SS send an alarm, only the AlarmNotifier on the primary SS will handle the alarm.

Notifier mail options on windows platform

To get the Notifier to work correctly and sending email, you will need to configure it once. This is done executing the mail command from within a command window. Here you see the needed parameters. This need to be done only once!

SimpleMail Version 1.1
usage: mail [-fhmsu] recipient,recipient...
-f "Full Name"
-h SMTP_Host (only needed once)
-m Return_Hostname (only needed once)
-s "Subject"
-u Return_Username (only needed once)

The three keys are all REG_SZ types in HKEY_CURRENT_USER\Software\SMail

How to create HTML PCause Files

Tired of maintaining Probable Cause files for customized alarms? Need more flexiblity in your Probable Cause files? Why not use web pages instead?

Beginning in Spectrum 7.3, The CsEvFormat files on the OneClick server can be written as an HTML page. This is extremely useful if you want to have a central repository for customized Probable Cause files, and the best part is, they won’t be touched by an upgrade, since they can be located on a different web server of your choice.

Here’s how to make this work.

First, we defined four parameters to be used in the default.asp page that is launched when the link is clicked. You can configure as many parameters as you wish, and call them what you want, but in our case, we found that four is more than enough for most situations.

The parameter names we chose are:

<Device Type> - Defines the folder where each different 3^rd-party app’s ProbCause files are located

<Alarm Type> - Can be filled with any variable passed by the Alarm

<Alarm Trigger> - Can be filled with any variable passed by the alarm

<Device Name> - Can be filled with any variable passed by the alarm

The default.asp page defines the parameter names and strings them together. If any of the parameter fields are empty, the next highest parameter applies. This allows you to build a structure of either generic or very detailed Probable Cause files.

In the Eventffffnnnn file, we then create a link to launch a web page containing instructions specific to the alarm. Based on variables passed along with traps, you can be very specific with your set of instructions.

To iIlustrate, I will use one of our custom traps from Oracle Grid Control as an example. For this application, we only need three of the parameters, so we use the first three, <DeviceType>, <Alarm Type> and <Alarm Trigger>. We populate these parameters with values received as variables with the trap.

Note that the variable names used in the Event Format file are meaningful only to the people who watch the events in Spectrum, and need not match the parameter names of the .asp page. What we really want are the varbind variables sent in the trap.

Here is the EvFormatFile for our Grid Control trap:

{d "%w- %d %m-, %Y - %T"}-
<html>
An Oracle Grid Control Alert has been received for '{S 1}'.
Host: {S 3}
Target_Name: {S 1}
Target_Type: {S 2}
Rule_Name: {S 10}
Event_Type: {S 8}
Event_Details: {S 9}
Agent_Event_Time: {S 7}
<br>
<a href="http://<your webserver>/Instructions/Default.asp?DeviceType=OracleGridControl&AlarmType={S 2}&AlarmTrigger={S 10}"><font color="#FFFF80" style="background-color:#556879">Click here for Instructions</font></a>
<br>
(event [{e}])
</html>

Note that we hard-code the <Device Type> parameter, then use varbind {S 2} for the <Alarm Type>, and {S10} for the <Alarm Trigger>. This builds a specific URL for instructions to the operator. For Grid Control, we have six different Alarm Types, and several Alarm triggers per type. Trying to do this with conventional Spectrum Pcause files would result in a file several pages long that an operator would have to sift through to find the right set of instructions. In this case, a specific set of instructions is availabe depending on the specific combination of varibles in the trap.

Here’s what the Event Details look like in OneClick for a trap:

The specific link that is built for “Click here for Instructions” would look like this:

http://<yourwebserver>/Instructions/Default.asp?DeviceType=OracleGridControl&AlarmType=Database%20Instance&AlarmTrigger=mon_rule_archiver

Next, our set of instructions for an Oracle Grid Control Database Instance triggered by an archiver rule has to be placed in the correct location on our website, and has to have a specific name:

[Home Web Folder]

[Instructions] – (contains the default.asp page)

[OracleGridControl] – (folder for all Grid Control Pcause files)

Database Instance.htm

Database Instance-mon_rule_archiver.htm

Database Instance-mon_rule_corruption

Database Instance-mon_rule_db

Database Instance-mon_rule_tablespace

Listener-mon_rule_lsnr.htm

Listener-mon_rule_blackout.htm

Etc.

Note also that we create a more generic “Database Instance.htm” file, in the off-chance that we receive a Database Instance trap that has a trigger that has not been defined. This ensures that unexpected traps do not result in a “Page not found” error.

Within our default.asp page, the parameters are separated by a dash (-), so that must be included as part of the filename.

Attached is a copy of our default.asp page, with all company-specific data removed. We chose a minimalist approach to our web-based Pcause files, but since they are in HTML, you can be as creative as you wish.

How to add ifAlias in AlarmMail?

You can pass any Spectrum attributes to SetScript/ClearScript/UpdateScript. Complete information is detailed in Spectrum_AlarmNotifier_User_ENU.pdf.

To summarize:

1. Add the desired attribute (ifAlias) in .alarmrc file:

EXTRA_ATTRS_AS_ARGS=0x23000e,0x11f7e

Note that 0x23000e means Device Type and it's included by default.

2. Add the following line after the default DTYPE in Set/Clear Script.

DTYPE="$1" IFALIAS_HEX="$2"

Note that $1, $2 refers to attributes declared in EXTRA_ATTRS_AS_ARGS

3. IfAlias is coded as Octed String. You'll need to convert it to ASCII before printing it:

if ([ "$IFALIAS_HEX" != "N/A" ] && [ "$IFALIAS_HEX" != "" ])
then
for line in $(echo "$IFALIAS_HEX"| gawk -F"." '{ for (i=1; i <= NF;
i++) { print $i } }' )
do
IFALIAS=$IFALIAS$line
done
else
IFALIAS="N/A"
fi

4. Restart Notifier

Statistics

0 Favorited

28 Views

0 Files

0 Shares

0 Downloads