Service Operations Insight

So upon returning from my vacation, I noticed after I got back that certain things happened that were missing. I had SOI running for 11 days and didnt' have any issues so the connectors were up and running the entire time

But I noticed that w/in one of our Domain Manger a site was down and I needed to dig into SOI to figure out when the alert happened, when it hit the console and then who closed it? This is not modeled within a service so i am completely relying on the Alert Queues that I have setup which categorizes everything.

I can find this info in the Domain Manager easily but I must know if SOI can provide this information as is?

I tried the following, selected the Alert Queue that the alert would reside in, Looked in the Information Tab and looked in the Cleared Alert History section. The initial 24 result was maxed out at 5000 already so 1st question is how do I increase the # of alerts returned from this query? I looked in View Preferences but the section for Cleared Alerts does not have this listed at something I can modify.

I was able to find the alert via using the time frame from which I occurred w/in the Domain Manager but there was no information as to who cleared it. There is no r-click ability to examine this alert further. All available columns are displayed yet it's still missing the information I need.

Even if I find the alert in the cleared Alert section, I then tried opening the "Auditor" tool, searched the Action Type of "CLEARED" and there is 0 cross reference between the Alert Queue | Information Tab: Closed Alerts section and the result from the Auditor tool to tell me the information I need.

This looks like a serious feature flaw. Unless I'm missing something can someone shed light on this simple query that an MSP would use on a daily basis to determine the life cycle of an alert?

Thank you, Dan

I opened an idea time ago about this.

ALERT CLEARED BY WHO?

have you voted it?

HI,

SOI 3.2 has Auditor feature which you can use to look at the alerts cleared/ackknowledged by specific user.
Thanks

Shaheen

Hi Shaheen, please elaborate on this as there is 0 correlation between the results and the alerts.

The results of the "CLEARED" action type are useless.

They don't contain the message details, nor the summary just time opened, closed and Alarm ID which ##########-####### doesn't help me.

Hi Dan,

If you go to the Alert Queues and select any queue, on the right hand side bring up the information tab, expand the "Cleared Alert History". You can filter by date and time to see the cleared alerts. You can use the "Cleared On" time from the "Cleared Alert History"and search in the Auditor to see who cleared the alerts during that time.

Thanks

Shaheen

Hi Shaheen, Yes that is the long & daunting way of figuring out who cleared an alarm but that really is too much work to do for a simple query such as this.

If development can add the two columns from the results of the Cleared Audit results:  Component & User Name then this Use Case would be satisfied by looking in that one section of the Information tab.

Yes I did vote it up..

I am using a 'workaround' for this issue. I use a simple 'echo' *.cmd file to write the required parms to a normal .txt file (For now anyway.)

I have a policy with Attributes 'Is Cleared Equal to "Yes". If that criteria is satisfied, I have an Action to 'Execute Command'.

In the Command Action, define the variables that you will pass according to your requirements. (Ex. Alert ID, Description, Login User, Cleared date etc.....)

in the *.cmd file, I declare the variables that will be received from SOI and echo these to a simple *.txt\*.log file.

Hi Madelaine, What did you specify on the Command to run text box? I tried the following:

echo $[Alert ID] $[Alert Severity] $[Cleared Date] $[Detail] $[Login User] $[Detail] >> c:\SOI_Alert_Cleared_Log.txt

It never produced anything? I was getting could not find echo, excho.exe. I cannot find echo in c:\windows nor windows\system32.

Thanks, Dan

Have a look at this process described in CA SOI Event and Alert Management Best Practices Guide on p103. That should help you. What I did is (1) Create a *.bat or *.cmd file on the SOI Manager to declare your variables that will be passed from SOI. (Ex. set 'Alert ID'=%1) After that is done, do an 'echo AlertID:%1 > c.......' (2) Create an Action for 'Execute Command' and complete the variables that you want to write to a file (basically what you have done, but with the full pathname of the *.cmd \ *.bat file that should be executed.

I would suggest reading the referenced piece I mentioned above, 'then construct the routine.'

There is way to see what has come in other than Alert Queues. Have you eve explored the tools -> Event Policies area? When you go in there their is a Source button on the Event Search tab. Select the connector for the domain manager in question. Then use the button for Time Range and go back to the day you last looked at it where is was good. Then hit the Search button. You can use the filter box to narrow things down and if you know what your after you can limit it even more by using the Event Patterns a few lines above.  I would start out at a higher level until you know what you are looking for then you can progress into the alarms to get honed into the time it went down. Once get familiar if you want to use the Event Patterns just click the ? in the bubble to the right for syntax. That may at least help you find the alarm in question. I gotta ask...you had more than 5000 alarms from  in 11 days? Wow...

As far as increasing the max number of alarms displayed I am not sure. I know that you can increase the number of CI's when doing searches using the locator and that is on the Preferences Tab under Locator.

To increase the number of Events returned when running an Event Search from the gui, follow the process as described in CA SOI Event and Alert Management Best Practises Guid p165. Due to the massive number of events from Nimsoft, I would recommend testing this in your Test environment first.

Just attaching this screen shot to explain exactly what change this portion of the product needs since its lacking this vital info.

And to be clear, the assign'd column isn't necessary the person who closed the alert. I tested that I put "Dan" in the assigned field then closed the alert and the user name logged into SOI was entered in the "User Name" which wasn't Dan.

But anyway to increase the # of items returned in the closed Alerts section since over a period of 24 hours we have way more than the default value of 5000 you can do the following:

CLEARED ALERT HISTORY FOR ALERT QUEUES

   This patch introduces a query limit for cleared alert history in the information tab for alert queues. The default limit is set to 5000. The sequence of entries fetched from the database is not guaranteed. The limit may be updated to a value #### using the following SQL query which can be run on the SOI database while services are running:

    update AdminConfiguration set ConfValue=150000 where ConfType='AlertHistorySetting' and ConfKey='MaxCount'

But be warned that if the result is very high then you will get performance hit to the SOI console. So if you change this then don't go over 200,000 results.

Use the new “Audit” feature and you can see this info.

Hi Scott, how's it going? Are you talking about SOIv3.3? The screen shot above is from v3.2's auditor tool and the results don't provide what you need to figure this out in an easy manner.

The top use cases are:

- alerts per service in a range of time

- search who close an alert

- search alerts per text, per type

We need a single pane of glass to accomplish this (not using alert queues where info is fragmented)

So I agree we need a big enhancement on this

Easy to do with SOI 3.2 “Auditing” feature.

BUMP....

Use the new SOI 3.2+ [Auditor] to do this ☺ You’ll have to create your own step by step procedure.

MAYBE, you can also modify the OneClick “Cleared Alerts” table view and add in the attribute for who closed the alert. Never done it, but in theory it may be possible as long as the attribute data exists.

Thanks

Scott

Hi Scott, please stop repeating this. This is useless. We are running SOI v3.3 and there is no help with the auditor tool and the "Cleared Alerts History" section. There is 0 linkage between the two except for the different time stamps in 1 side its 12HR in the Auditor its 24HR format. 

In the Cleared alerts section you have the Alert and the Alert Summary/message.

In the Auditor you have Action=Cleared and what you get is a time stamp and the AlarmID #. Not at all helpful.

Serious lacking on the SOI side to track an alert once it's been closed/cleared from an alert queue since you can search on say CI name, Alert Message Summary/details. The results on Action=Cleared is Missing alert history, who cleared it etc...