We need to forward alarms generated in Spectrum to Splunk.
By any chance anyone has tried this. If yes, please help.
The method for forwarding alarms with which I'm familiar consists of SANM and AlarmNotifier.
You can read about it here:
You need to decide what kind of alarms you want to forward, and apply this filter as your SANM policy filter.
Afterwards you need to configure the AlarmNotifier to forward/consolidate these alarms however you see fit.
For example, the following thread shows how alarms can be sent to a mailbox:
Here is a guide for AlarmNotifier:
It's actually quite flexible, especially if you have third party tools that can parse and collect logs (eg. Logstash). You may want to replicate the pre-installed AlarmNotifier files and create a new alarmrc specific for your needs.
How the alarms are handled is up to the SetScript which you'll need to modify.
I don't think Alarm notifier can forward alarm to other third party tool unless you use some kind of scripting along with trap exploder option in the set script.
Are you going to be at CA World? If so, look me up, we can chat about what you need.
Here are some ideas, most of these are going to require some scripting or programming.
The easiest for for you would be to simply setup an alarmNotifier that will output the alarms to a file. Then install a Splunk client on your Spectrum system where you are running the AlarmNotifer and have it scrape the output file. You can use the standard SetScript, UpdateScript, ClearScript. I would suggest using something like logrotate to rotate the output file so that it doesn't fill up your system. Some would consider this a bit clunky because you end up with that output file that could get huge. One benefit is that you then have the Splunk client on your system to scrape other log files, like VNM.OUT, ARCHIVEMGR.OUT. We have put it on our simply for scraping Spectrum log files.
Another route would be to use AlarmNotifier, but update the SetScript/UpdateScript/ClearScript to call the Splunk api to insert the data into Splunk.
You could also utilize the REST interface to write an Alarm Subscription that takes the alarms as they come in and then use the Splunk api to insert them into Splunk. This is what we do for our ELK reporting stack and for other tools that need bulk, real time, alarm data.
Some would say you can also use SRM's MySQL databases to pull in alarm data, but I think that is more complicated and cumbersome than you probably want to deal with.
Another method that we have used for sending off alarms to other systems is Service Operations Insight (SOI). SOI also gives you the benefit of additional features and focuses alarms around services. Again, tho this will require programming to build the adapter.
As you can see, one of the fun thinks with Spectrum, is that there is almost an endless number of options.
Thanks William. I will try using the splunk API.
I did found a way to send the alarm details into Splunk via Setscript in Spectrum.
Did Modify the Setscript to foward the alarm details into Splunk using curl and sending data to HTTP token
However, the issue what i am facing is, if the alarm message has special character like double quotes or @, curl fails to send the message into Splunk. Curl throws a message invalid data
By any chance do you know can we remove these special characters. And on you ELK reporting stack how this is done ?
Retrieving data ...