DX Unified Infrastructure Management

Expand all | Collapse all

logmon messages and clearing alarms

Anon Anon

Anon AnonSep 09, 2013 10:55 PM

  • 1.  logmon messages and clearing alarms

    Posted Sep 05, 2013 08:21 PM

    Good afternoon,

     

    I will start by saying I am no logmon pro, but I have spent the last few hours testing a new profile and I am stumped. I have a log file that I want to search (update mode) for "concurrent clients=X".

     

    My watcher is as follows

    Standard

    Match Expression - /concurrent.clients=([0-9]?[0-9])/

    Message to send on match - concurrent clients=$connections (just for testing)

     

    Varibles

    connections $1

     

    Advanced

    Send clear alarm *checked*

     

    QoS

    QoS on variables - connections *checked*

     

    Alarm

    Message subsystem id - 3.1.1

    Suppression Key - $PROFILE.$WATCHER

    Pattern match threshold - 25

     

    Problem 1: Anytime it runs and finds a match, it sends an alarm, But it sends the entire line of text from the log as the message instead of the message I set it to send.

     

    Problem 2: It seems to send an alarm ANY time it finds a match, not if the variable is 26 as dictated in the Alarm tab.

     

    Problem 3: The alarm that comes across is the default subsystem id, not the one I specified and created in the NAS probe.

     

    Any help is greatly appreciated.

     

    Gordon



  • 2.  Re: logmon messages and clearing alarms

    Posted Sep 05, 2013 08:36 PM

    Hi,

     

    Could you post the profile from your config? I think it might be better for trying to figure out what's wrong with it.

     

    -jon



  • 3.  Re: logmon messages and clearing alarms

    Posted Sep 05, 2013 08:42 PM
      |   view attached

    Thank you John,

     

    I am attaching a file as the profile was cut off.

    Attachment(s)

    txt
    _logmon.cfg.txt   3 KB 1 version


  • 4.  Re: logmon messages and clearing alarms
    Best Answer

    Posted Sep 05, 2013 09:42 PM

    Hi,

     

    I think #2 is because you don't seem to set the threshold in the variable. You should specify the threshold there instead of the alarm tab in this case.

     

    as for #1, If that's the whole line that is logged in the logfile, it's awfully similar to the message you specified. I wonder if it's just a case of the variable not working properly? If I remember correctly logmon suggests you type variables this way: ${variable}, but I can't remember now if both work and can't get to Nimbus.

     

    #3 I have no clue without testing :smileysad:

     

    -jon



  • 5.  Re: logmon messages and clearing alarms

    Posted Sep 05, 2013 10:19 PM

    Hi Gordon,

     

    Could you attach a screenshot of one of the alarms? 

     

    Thanks



  • 6.  Re: logmon messages and clearing alarms

    Posted Sep 06, 2013 04:32 PM

    Jon,

     

    That was it, the threshol in the actual variable! Now the alarm comes in with the desired text AND it clears when it matches a lower number. Perfect!

     

    Now here is another hitch.. the file naming.. the file name is 20130719153712-serverCaptureLog. It breaks down as YYYYmmddhhmmss-serverCaptureLog. It appears to create a new log everyday, but sometimes a couple times a day. I am stuck on the naming for finding the latest file. I have %Y%m%d which matches todays year month and day, but I cannot figure out the hour minute second portion, especially if there is more than one say created at 20130906083412 and 20130906134532. Any ideas?

     

    Gordon



  • 7.  Re: logmon messages and clearing alarms

    Posted Sep 06, 2013 06:15 PM

    aww man I wrote a long reply and accidentally pressed back button.. here goes somewhat shorter version:

     

    You could use time primives like %M:%S to catch MINUTE:smileyfrustrated:ECOND, but it's problematic as the profile would need to run the same second as the file was created. You could get around that by using wildcard (undocumented feature), like FILE%H%M*. However, if I remember correctly, it doesn't work properly. It might be that it was unable to keep tabs on the last checksum of the file and scans all content of the files on every interval.

     

    What I usually do with more complicated cases, is that I write a script that handles the logfiles and have logmon run that script instead. It seems much easier to me oftentimes.

     

    -jon



  • 8.  Re: logmon messages and clearing alarms

    Posted Sep 06, 2013 11:55 PM

    Jon,

     

    That is awful, have done it a time or two myself.

     

    I did not think it was going to be easy, my lack of scripting only compounds the problem. Would you be willing to share how exactly you use the scripts to handle the log files and then how lomon handles the script? I am attempting to learn more as using scripts seems to be a very powerful way to get things done.

     

    Thank you for your help so far,

     

    Gordon



  • 9.  Re: logmon messages and clearing alarms

    Posted Sep 08, 2013 12:05 PM

    Hi,

     

    I don't have anything that pertains to your situation, I've been able to handle them mostly with the primitives. Commonly it really just comes down to how the logging system works. I could post some example next week about using script and it's output with logmon, but the ones that I come to mind are from different kind of scenarios. For example custom URL checking and dicom echo.

     

    -jon



  • 10.  Re: logmon messages and clearing alarms

    Posted Sep 09, 2013 04:17 PM

    Here is how I handled a similar situation recently:

    1. Created a logmon profile to monitor that file with it's current file name. 
    2. Wrote a batch script which outputs the name of the newest file in that directory matching the naming scheme.
    3. Created a logmon profile to run that script every 5 minutes and send the filename as an alarm.
    4. Wrote a Lua script which runs on arrival whenever that alarm comes in: 
    • Reach out to controller of robot running the logmon profile. Check the filename specified in the profile created in step 1.
    • Compare that file name to file name sent in the alarm from step 3. 
    • If they match, don't do anything. Just close the alarm.
    • If they don't match, send a command to update the filename in the logmon config, restart the probe, then close the alarm.

     

    It is kind of complicated and involves a lot of moving parts, but it was the only way I could figure out how to do this. It guarantees the filename in the profile is updated within 5 minutes of a new file being created.



  • 11.  Re: logmon messages and clearing alarms

    Posted Sep 09, 2013 06:31 PM

    Jon,

     

    Thanks again, any help I can get right now is more than appreciated.

     

    Craig,

     

    Would you be able to provide that batch and Lua text? It does sound complicated, but it also sounds like exactly what I am going to have to do since there is really no commonality to the log creation or time. I would really like to get this onto my system and see how it handles.

     

    Gordon



  • 12.  Re: logmon messages and clearing alarms

    Posted Sep 09, 2013 07:55 PM

    I think Craig's solution sounds good, I've used the same technique in other situations. I'll try to dig around tomorrow and see if there's something I can post as well.

     

    -jon



  • 13.  Re: logmon messages and clearing alarms

    Posted Sep 09, 2013 10:55 PM

    Thank you both.

     

    Gordon



  • 14.  Re: logmon messages and clearing alarms

    Posted Sep 10, 2013 12:55 AM

    I don't have the actual scripts right now, but here are the main tools I used:

     

    • I used the following link to help write the batch script: http://superuser.com/questions/123347/windows-batch-scripting-newest-file-matching-a-pattern
    • I used the "probe_config_get" probe callback to find the current filename from the logmon profile. I set the name variable to "logmon" and the var variable to something like "/profiles/<profile name>/scanfile"
    • I compared the two values and used "probe_config_set" to change the filename if needed.  Probe_config_set uses the following variables: name=logmon, section=<your profile name>, key=scanfile, value=<your file path>

     

    Hopefully this helps. 

     

     



  • 15.  Re: logmon messages and clearing alarms

    Posted Sep 10, 2013 08:11 AM

    I have a similar powershell script that gets the latest filename. It's a part of bigger a bigger script that I can't post, but it's not all that relevant either:

     

    $file = (Get-ChildItem -filter *.log | sort -property LastWriteTime | select -last 1).name
    write-host "log: $($file)"

     You can run powershell in logmon by selecting mode to "command" and putting something like this in the "Command":

     

    powershell.exe c:\machine\scripts\errfilechecker.ps1

     If you created a watcher to capture "log: logfile". You could get the filenime in the alarm. I'm not sure but I think the powershell interpreteter puts out some other text too everytime it runs, so you I put the "log: " in there just to be on the safe side. You could also remove that and just match your logname, which would reduce steps you need to take in logmon.

     

    I think I've once posted my scripts that do what craig mentioned, I'll see if I can find them here so I don't have to repost. Otherwise I'll try to sanitize them later and post.

     

    -jon



  • 16.  Re: logmon messages and clearing alarms

    Posted Sep 10, 2013 08:16 AM

    Here, this thread has some script examples: http://forum.nimsoft.com/t5/General/Maintenance-Mode-at-the-endpoint/m-p/21103#M7727

     

    If you combine that with Craig's pointers I think it might help you.



  • 17.  Re: logmon messages and clearing alarms

    Posted Sep 10, 2013 04:11 PM

    Thank you so much guys! I am starting my journey and will let you know how it turns out.

     

    Gordon



  • 18.  Re: logmon messages and clearing alarms

    Posted Sep 10, 2013 07:27 PM

    Well, I got this so far..

     

    args = pds.create()

    pds.putString (args,"name","logmon")

     

    cfg,rc = nimbus.request ("controller","probe_config_get",args)


    if rc == 0 then

     

       log = cfg["/profiles/AMCC Cloud"]

      

       if log ~= nilthen

          printf(log.scanfile)

       end

    end

     

    This returns C:\Gordon\20130719153712-serverCaptureLog.txt

     

    I tried to run the powershel executable in the logmon command but I kept getting a message it failed because running scripts was disabled on the server.

     

    Gordon



  • 19.  Re: logmon messages and clearing alarms

    Posted Sep 10, 2013 09:59 PM

    Hi,

     

    That's a default in windows. You need to do set-executionpolicy 1 (remotesigned, preferrable) or 0 (unrestricted, not recommended).

     

    -jon



  • 20.  Re: logmon messages and clearing alarms

    Posted Sep 11, 2013 03:22 PM

    Jon,

     

    That did the trick, had to look up the 64bit version of it though :smileyhappy:. Chose RemoteSigned, thanks for the help... so here is where I am.

     

    I have a logmon profile the runs command "powershell C\:Gordon\newestfile.ps1.

     

    newestfile.ps1

    cd c:\Gordon
    $file = (Get-ChildItem -filter *.txt | sort -property LastWriteTime | select -last 1).name
    write-host "log: $($file)"

     

    Then my watcher in logmon - /^log\:.(\d{14}-serverCaptureLog\.txt)/ catpures the filename as var1 and sends it as the message.

     

    So now I have a message (alarm) in nimsoft with the newest log file created as the message.

     

    I also set up the script to retrieve the file name from the current logmon setup.

     

    Now I have to get the file name from the alarm, do the comparrisson with the name I retrieved and replace if needed. I will really need you scripting guidance here when you have some time. I will keep tinkering and as said many times before, thank you so much for the help.

     

    Gordon



  • 21.  Re: logmon messages and clearing alarms

    Posted Sep 11, 2013 05:37 PM

    Hi Gordon,

     

    For the comparison, you should set up an auto-operator which runs on arrival for all alarms coming from your logmon/powershell profile. I recommend configuring that profile with a unique Subsystem ID in logmon so it will be easier to set up your auto-operator matching criteria. Set the Auto-Operator to run a script.

     

    For the script, you can do something like the following to pull the information from the incoming alarm:

     

    alarm1 = alarm.get()

    alarmMessage = alarm1.message

     

    Then, you can paste in the code you have already written to pull the filename from the existing logmon profile into another variable. 

     

    Compare the two variables. If they match, close the alarm with:

    action.close(alarm1.nimid)

     

    If they don't match, you can write some code to update the logmon profile. This should be very similar to the code you have already written to check the current filename in the logmon profile. However, you would be using probe_config_set instead of probe_config_get. SET requires a couple more variables which you will need to add to your PDS. 

     

    Let me know if you have any questions on any of this.

     

    Thanks



  • 22.  Re: logmon messages and clearing alarms

    Posted Sep 12, 2013 04:53 PM

    Craig,

     

    Thank you for your reply, I do need help indeed. I am very new to scripting and am kinda Frankensteining from from examples my coworker created. Some of it (5%) makes sense, most does not.

     

    Based on the replies so far (yours included) I have the following script. The printf is so I can see it is working, I dont think I need it in the final script, but again, do not know.

     

    alarm1 =

    alarm.get("BZ76128661-38512") <- My test alarm

    alarmMessage = alarm1.message

      

    printf(alarm1.message)

     

    args =

    pds.create()

    pds.putString

    (args,"name","logmon")

     

    cfg,rc =

    nimbus.request ("MedQuistNOC/NOC_HUB_01/mq1dstnoc01/controller","probe_config_get",args)

    -- Extract the named section '/cpu'

    if

    rc == 0 then

       log = cfg["/profiles/AMCC Cloud"]

      

    if log ~= nilthen

         

    printf(log.scanfile)

      

    end

    end

     

    This outputs the following:

     

    ----------- Executing script at 9/12/2013 9:34:59 AM ----------

    20130000000003-serverCaptureLog.txt

    C:\Gordon\20130719153712-serverCaptureLog.txt

     

    At this point, I assume I need to trim off the C:\Gordon\ from the second return, compare the two, then return the C:\Gordon\ to the new file name on update.

     

    I have no idea how to do this. I am working with Google to read up on LUA and kinda test out code as I think of it. Any and all help is greatly appreciated. You guys have been great so far, I cannot thank you enough!

     

    Gordon



  • 23.  Re: logmon messages and clearing alarms

    Posted Sep 12, 2013 05:05 PM

    Hey Gordon,

     

    You can use the "string.len" and "right" functions to trim that string. 

     

    You should be able to do something like:

     

    longFile = log.scanfile

    trimmedFile = right(longFile,string.len(longFile)-10))

     

    That should trim the first 10 characters from the string. Then you can compare them with an if statement like:

     

    if alarm1.message ~= trimmedFile then

        -- insert your code to update the file using probe_config_set

    end

     

    Craig



  • 24.  Re: logmon messages and clearing alarms

    Posted Sep 16, 2013 09:04 PM

    Guys,

     

    I did it! Thank you for the help. A coworker helped me polish it a bit, but I got it running. Here is the breakdown and script.

     

    1. Powershell runs the script to identify the newest file and send in an alarm.

    2. AO profile runs script on arrival. Script grabs file name from the alarm, then retrieves current file name from logmon profile. Compares the two, if they match, the alarm is cleared. If they do not match, the filename from the alarm is formatted with the proper directory, uploaded to the probe config, probe is restarted  and the alarm is cleared.

     

    -- Get message from the alarm and set as variable alarm1

    alarm1 = alarm.get()

    alarmMessage = alarm1.message

    --   printf(alarm1.message)

     

    -- Set probe to logmon

    args = pds.create()

    pds.putString(args,"name","logmon")

     

    -- Get the contents of /profiles/AMCC Cloud/scanfile

    cfg,rc = nimbus.request ("domain/hub/robot/controller","probe_config_get",args)

      

    if rc == 0 then

          log = cfg["/profiles/AMCC Cloud"]

         if log ~= nilthen

             logfile1 = printf(right (log.scanfile, 35))

         end

    end

     

    -- trims directory from file name

    logfile1 = (right (log.scanfile, 35))

     

    -- Adds directory to file name from alarm message.

    logfile2 = "X:\\directory\\"..alarm1.message

     

    -- Compares files from alarm message and current probe config for differences and closes alarm if none

    --      print(alarm1.message.." " ..logfile1)

    if alarm1.message == logfile1 then

       printf("File "..alarm1.message.." matches "..logfile1)

       action.close (nimid)

      else

      printf("File "..alarm1.message.." does not match "..logfile1)

     

    -- preparing new file name for upload

    profile = "PROFILE NAME"

    options = pds.create()

    pds.putString (options,"/profiles/"..profile.."/scanfile",logfile2)

     

    -- Uplodaing new config to probe

    args = pds.create()

    pds.putString (args, "name", "logmon")

    pds.putPDS (args, "as_pds",options)

    resp,rc = nimbus.request ("domain/hub/robot/controller","probe_config_set",args)

     

    -- restarting probe

    mypds = pds.create ()

    output,return_status = nimbus.request ("domain/hub/robot/logmon", "_restart" )

    if

    return_status == NIME_OKthen

      printf ("Restart successful")

    else

      printf ("Restart failed")

    end

     

    -- Closing the alarm

    action.close (nimid)

    end



  • 25.  Re: logmon messages and clearing alarms

    Posted Sep 16, 2013 10:11 PM

    good job! I see you took advantage of as_pds as well :smileyhappy:

     

    -jon



  • 26.  Re: logmon messages and clearing alarms

    Posted Sep 17, 2013 03:10 PM

    Yes I did, though I do not fully understand it :smileyembarrassed: I still have so much to learn about scripting.