I am trying to determine how to prevent duplicate alarms that would be generated should my primary hub fail.
My scenario is such that I have a primary and secondary hub (non-clustered) running monitoring probes such as URL_response, Logmon, Net_Connect, etc. I have the HA probe installed/configured on the secondary hub to detect a failover event on the primary hub. The probe configurations are synced between the primary monitoring probes and the secondary monitoring probes. The monitoring probes on the secondary hub are not enabled until the HA probes detects a failover event on the primary hub.
My issue is how would I prevent currently active alerts (count incrementing) from effectively re-generating in the event I have to failover to the secondary hub? So for example I have URL "site A down" alert alerting on the primary monitoring hub. The primary monitoring hub crashes. As soon as the secondary monitoring hub enables the URL_response probe I will get a "new" URL site A down alert only from a different hub.
The only solution I can think of is some sort of trigger in the NAS that becomes active once a failover event has occurred and build and creating a script that would handle the duplicate alarms, however I suspect there may be a simpler way of dealing with scenario that I am missing.
I am wondering how others handle this type of situation?