Service Operations Insight

  • 1.  How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 06, 2015 10:51 PM

    How many alerts per minute can a SNMP connector handle? and a generic connector?

    If a clients infra is generating thousands of alerts per minute, will SOI be able to handle it?



  • 2.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 01:00 AM

    Hi, SOI not being Event Management application you should limit the alerts send to SOI via connector (Alert should be filtered at connector / MDR level if possible) and should send only mission critical alerts to SOI which needs to monitor. SOI can not function if you send huge alerts and you may experience SOI outage / performance problems. Thanks, Ashay



  • 3.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 03:07 AM

    Hi Pramod,

     

    following are some figures (performance tests on a single CA SOI 3 Instance, 64-bit JVM):

    • Event store captures  incoming 1.000 events/second from connectors
    • Manager processes  50 events/sec  (modeled services) and 25 event/sec (event queues)


    As Ashay mentions, SOI is not an Event Management system, but a Service Management system.

    There are many processes running in background to evaluate the impact of Alerts towards Services, performing Escalations, assigning Alerts to queueus, keeping control of Access rights, etc.

    Thus, the limiting factor of processing Alerts is not the Connector, but the complexity of the entire system, e.g. the SOI Manager.

    The messages coming in will all be processed, but it might take a while until this is finished in case of a peek (Message Storm).  This should never be the "normal" situation.

    SOI is designed to handle "actionable" Alerts, e.g. messages that require an action (by an Operator or autmatically) such as opening a ticket, sending a mail.

    No Operating Center can handle thousands of such Alerts on a regular basis.

    You have to limit the information that comes into SOI (as Ashay says, best on Domain Manager level) to only forward to SOI what is really "actionable".

     

    MichaelBoehm



  • 4.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 06:10 AM

    Thanks for the reply Ashay & Michael.

     

    If we need to supress the alerts at connector level, as both of you have suggested, how can it be done? Is there some inbuilt capability available in the connectors? If not, can this capability be built into connectors?

    Secondly, is it a good idea to employ a tool like Spectrum to suppress the unwanted alerts/events?



  • 5.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 06:29 AM

    In general, all Domain Managers should be properly configured to only report about problems that are worthwhile to mention.

    I recommend to look at the following video to get a good understanding on Monitoring Governance: https://www.youtube.com/watch?v=1IqojteO6m0

     

    Depending on the scenario there are different ways to achieve the limitation of Alarms to a managable amount.

    If the above does not create the expected result, Filters or other Event Rules can be created to suppress Alarms in the SOI process flow.

    The basic question is: What kind of "thousands of alerts per minute" do you talk about in your question..

    Is this a one-time Message Storm or a permanent situation?

    What is the scenario?

    When you explain that a little more in detail, we can think about possible solutions.

    Only then we can also evaluate if the introduction of Spectrum would bring a benefit.

     

    If you like we can also discuss your case internally: Michael.Boehm@ca.com

     

    MichaelBoehm



  • 6.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 07:25 AM

    Thanks again Michael.

    The scenario is that a large Telco has multiple domains which are NMS'ed by respective monitoring solutions - the IT landscape has 75000 devices which constitute their business services (these services are to be monitored by SoI).

    The primary domains are - MES NMS (Dataminer); GVS NMS (REGI); IPSOC NMS (Monolith); RFNOC NMS (Wirelessone); ASD NMS (CA UIM) etc.

    The respective NMS monitor the alerts about respective (relevant) infra.

    Now they want to employ SOI so that they can relate an alert to a potential service issue.

     

    We have been told by client that roughly 50000 alerts get generated per minute - which triggered this thread.

    Is a tiered deployment another possible solution (read it in some other thread, not sure of the technicalities)?



  • 7.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 08:09 AM

    I think the problem is, that the difference between Event Management and Service Management is not clear.

     

    There may be 50.000 Events/minute (even that is difficult to believe on a permanent basis).

    But there will definitely not be 50.000 (actionable) Alerts/minute, which are showing different problems all the time - or the entire environment (75.000 devices) is completely in trouble.

    You would need several thousand people to work on problems permanently, and I dont think that anybody would like to offer Services which such an environment .

     

    The underlying NMSs have to be analyzed to figure out what requires attention and is "actionable".

    Most of above messages will be ignored all the time, because nobody can follow 50.000 Messages/minute on a console.

    The proper first step would be to ask which Events are the important ones they really deal with and forward them to SOI.

     

    This can all be done in a CA Services project but cannot be resolved here in the Community by some generic answers.

    Also the setup of a Tiered Architecture to pre-sort and load-balance all the messages from different NMSs can only be discussed following a thorough analysis of the environment.

     

    MichaelBoehm



  • 8.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 11:46 PM

    Michael & Daniel - thanks a lot for the replies.

    I think I know how to take this forward with the customer. I will seek your guidance & help again, whenever needed.

    Thanks indeed.



  • 9.  Re: How many alerts per minute can a SNMP connector handle? and a generic connector?

    Posted Oct 07, 2015 08:49 AM

    ^+1 to what Mike mentioned. I would not recommend throwing SOI into the picture in that environment UNTIL you have all your alarms filtered, cleaned up and tweak the underlying domain managers. You need a more manageable state before throwing SOI on top of a pile a mess. That environment needs to take a few months and clean up, fine tune, cut out the noise their NMS/UIM environment before standing up SOI.