DX Unified Infrastructure Management

Hi 

Currently we have integration with ServiceNow through sdgtw and what we observed that there are duplicate tickets are getting created for same nimid.

Currently in CDM probe disk Interval is set to 5 mins and collecting 3 samples and average of 3 samples is above the thresholds then it should generate the alarm. But we are observing that its not working as required as few alarms getting in seconds.

Example -

2018-12-07 02:46:14 Message: Disk free on /orafiles/ is 10% , which is below critical threshold (10%) AlarmID: BO55079028-26297 
2018-12-07 02:46:25 Message: Disk free on /orafiles/ is 10% , which is below critical threshold (10%) AlarmID: BO55079028-26297 
2018-12-07 02:46:35 Message: Disk free on /orafiles/ is 10% , which is below critical threshold (10%) AlarmID: BO55079028-26297 
2018-12-07 02:46:47 Message: Disk free on /orafiles/ is 10% , which is below critical threshold (10%) AlarmID: BO55079028-26297 

Similar issue, I can see with different subsystem id alarms and duplicate tickets for the same too.

UIM 8.51

nas 9.0HF4

sdgtw 2.0

I have tried to change the AO profile overdue timings and verify but no luck too.

Regards - Ripple

What version of CDM probe?

Do you have the cdm probe set to send alarm on each sample?

this might explain what you are seeing.

Usually when using the sdgtw you would have an AO profile that is doing an assign for the username that triggers the ticket creation.

you can set the ao for on overdue 1 minute and this might help as well.

This could be complicated to track down exactly where the problem is so you might want to open a support case.

Hi

What version of CDM probe?  - CDM 6.30

Do you have the cdm probe set to send alarm on each sample? No

Already chasing with support but til now no luck.

Regards - Ripple

- do you have AO profiles on these messages? (because each AO rule can duplicate the alarm)

- do you have multiple NAS probes ? (because routing alarms between several nas probes can create a ping-pong effect)

We have AO profile rules but we are filtering it with the message or probe based only. So no two rules are same.

Perhaps you can try, with a query like:

select * from NAS_TRANSACTION_LOG where message like '%orafiles%' order by time desc

to determine why/how/where the duplicate is coming from?

Can you send me the full output with all columns (in csv) to luc.christiaens10@telenet.be (or attach it here)

Sorry I am not able to attach the same not sure any other issue.

Do you have integrations, like Spectrum, that can manipulate alarms?

Do you have another instance of the nas running?

Else i would suspect an AO rule. (can you send me/attach the AO rule that assigns alarms to Database)

Do you have integrations, like Spectrum, that can manipulate alarms?   No

Do you have another instance of the nas running?   No

I can see that within friction of seconds second alarm is getting generated for same conditions and as first alarm did not got acknowledgement from the servicenow, as per rule it passed second one also and due to this we got two tickets.

But not able to understand why we can see the same alarm in friction of seconds? as polling interval is every 5 mins.

Please attach (or mail) your AO rule (copy it in txt format from nas.cfg)

<auto_operator>
   <setup>
      interval = 5min
      active = yes
      ignore_import = no
   </setup>
   <definitions>
      <EMAIL critical (repost)>
         active = no
         action = repost EMAIL
         overdue = 5m
         level = critical
         order = 1
      </EMAIL critical (repost)>
      <Automatic cleanup of low-severity messages (3 days)>
         active = yes
         action = close
         overdue = 3d
         level = information
         visible = 0
         order = 2
         break = no
      </Automatic cleanup of low-severity messages (3 days)>
      <Test_duplicPQRation_Dev>
         active = yes
         action = assign ABC
         overdue = on_arrival
         level = information,warning,minor,major,critical
         source = 10.10.96.129|10.11.17.132
         visible = 0
         order = 3
         break = no
      </Test_duplicPQRation_Dev>
      <Test_duplicPQRation_Dev2>
         active = yes
         action = assign ABC
         overdue = on_arrival
         level = warning,minor
         source = 10.11.63.135
         counter = eq 1
         visible = 0
         order = 3
         break = no
      </Test_duplicPQRation_Dev2>
      <ABC_FS_Email>
         active = yes
         action = EMAIL
         overdue = 2m
         level = warning,minor,major,critical
         subsystems = Disk
         counter = eq 1
         probe = cdm
         visible = 0
         order = 4
         origin = ABC-UIM
         break = no
      </ABC_FS_Email>
      <AutoAssigXYZUserSNOW>
         active = yes
         action = assign Database-Team/database
         overdue = on_arrival
         level = minor,major,critical
         counter = eq 1
         probe = db2|mysql|oracle
         visible = 0
         order = 5
         origin = ABC-UIM|XYZ-UIM|PQR-UIM
         break = no
      </AutoAssigXYZUserSNOW>
      <AutoAssigXYZFilesystems>
         active = yes
         action = assign Database-Team/database
         overdue = on_arrival
         message = /.*mysql|treasury|oracle|orafiles.*/
         level = minor,major,critical
         subsystems = Disk
         source = 10.11.64.225|10.10.52.225|10.10.96.129
         counter = eq 1
         probe = cdm
         visible = 0
         invert = 2
         order = 6
         origin = ABC-UIM|XYZ-UIM|PQR-UIM
         break = no
      </AutoAssigXYZFilesystems>
      <Email alarms for DBA group>
         active = yes
         action = EMAIL DBAGroup
         overdue = 1m
         level = information,warning
         counter = eq 1
         probe = db2|mysql|oracle
         visible = 0
         category = Email
         order = 7
         origin = ABC-UIM|XYZ-UIM|PQR-UIM
         break = no
      </Email alarms for DBA group>
      <AutoAssignWindowsUserSNOW>
         active = yes
         action = assign Windows-Team/windows_team
         overdue = on_arrival
         level = minor,major,critical
         counter = eq 1
         visible = 0
         order = 8
         origin = ABC-UIM|XYZ-UIM|PQR-UIM
         break = no
         user_tag2 = WINDOWS
      </AutoAssignWindowsUserSNOW>
      <AutoAssignUNIXUserSNOW>
         active = yes
         action = assign Unix-Team/unix
         overdue = on_arrival
         message = /.*mysql|treasury|oracle|orafiles.*/
         level = minor,major,critical
         counter = eq 1
         probe = cdm|processes
         visible = 0
         invert = 16
         order = 9
         origin = ABC-UIM|XYZ-UIM|PQR-UIM
         break = no
         user_tag2 = UNIX
      </AutoAssignUNIXUserSNOW>
      <Email alarms for Windows group for XYZ Client>
         active = yes
         action = EMAIL WindowsGroup
         overdue = on_arrival
         level = warning,minor,major,critical
         counter = eq 1
         probe = cdm
         visible = 0
         order = 10
         origin = XYZ-UIM
         break = no
      </Email alarms for Windows group for XYZ Client>
      <Email alarms for Unix group [cdm]>
         active = no
         action = EMAIL UnixGroup
         overdue = on_arrival
         level = warning
         probe = cdm
         visible = 0
         category = Email
         order = 11
         break = no
         user_tag2 = *UNIX*
      </Email alarms for Unix group [cdm]>
      <Test for ticket creation>
         active = yes
         action = assign ABC
         overdue = 3m
         message = */orafiles/QOS*
         level = information
         robot = QOS
         visible = 0
         order = 13
         break = no
      </Test for ticket creation>
   </definitions>
   <triggers>
      <example.network>
         active = no
         level = major,critical
         subsystems = Network
         category = example
      </example.network>
      <example.system>
         active = no
         level = major,critical
         subsystems = Disk,CPU,Memory,Filesystem,Process,NT-Services,Security,Application,System
         category = example
      </example.system>
      <example.SLM>
         active = no
         sid = 1.3|1.3.*
         category = example
      </example.SLM>
   </triggers>
</auto_operator>

- Is it possible to attach or mail the output of the previous sql query because I'm missing some important columns, like source, host.. (so use the proposed * for the select statement)

- Is it your goal that an alarm can match multiple AO rules or that he has to stop after the first match?

This is with the same limited sql statement as your previous post; please use something like: (where the * is important to have all fields)

select * from NAS_TRANSACTION_LOG where message like '%orafiles%' order by time desc

 Is it your goal that an alarm can match multiple AO rules or that he has to stop after the first match?

Right after first match should stop.

But what we noticed that in few seconds only second alarm is getting generated for same condition and as workflow is not completed its sending again.

The problem with AO rules can be:

- by default an alarm trie sto match every rule, except if you code: break=yes

- if you have an AO action like assign, UIM will REgenerate the message, and depending your rules..match again.

(you can perhaps add in your assign rule a NOT clause for assigned to database?)

Currently under AO profile - Message counter is set to 1 then in that case second alarm should not be forwarded to sdgtw but in our case still we see rule is bypassed.

Hi, I think the fact that your AO profiles are configured as "on_arrival" negates the counter = 1 criteria. When an alarm comes into the nas (on_arrival), the counter is always 1. The on_arrival mode precedes the alarm suppression process.

Also, I can see in the AO profiles that they are not mutually exclusive. Just a quick scan shows that there are at least two that will execute an assignment for the same alarm - one assigns to the database group and the other to the Unix group. Since the nas is single threaded, the profiles are evaluated sequentially and one will fire, putting an alarm_assign message into the sdgtw queue and then the other will fire putting a subsequent alarm_assign message in the queue for the same alarm.

AO profiles are configured as "on_arrival" negates the counter = 1 criteria --- yeah but as we were getting duplicate alarms, I put this to see if I can restrict the second count of same alarm.

there are at least two that will execute an assignment for the same alarm - one assigns to the database group and the other to the Unix group. ---- Yeah as per requirement we are sending DB related alarms to database team and the OS related alarms to unix team and it was further filtered based on the User Tag, Message etc.. so that it will not execute the next profile.