DX Unified Infrastructure Management

Expand all | Collapse all

Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

  • 1.  Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Broadcom Employee
    Posted Sep 12, 2017 08:10 AM

    Good day everyone, I thought this tip might help some of you.  

    Using the logmon probe and a shell script, you can alarm on server certificate expiration if it is coming up soon.

    This example uses:

    UIM 8.5.1

    logmon probe version 3.90 on a Linux robot.

     

    1- Shell script to check server certificate expiration

    First, create a shell script to check the server certificates.  Here is a SAMPLE script I am offering.  You will need to customize it to suit your needs. Pay attention to the variables: "servers_to_check"  and "warn_exp_days".

    Modify these as appropriate for your needs.

     

    #!/bin/sh

    # Sample script to check for cert expirations offered by CA Support
    # with no guarantees or maintenance.

    DEBUG=true
    # Number of days to warn about soon-to-expire certs
    warn_exp_days=90
    servers_to_check='google.com:443
    yahoo.com:443
    ca.com:443'

    for CERT in $servers_to_check
    do
    $DEBUG && echo "Checking cert: [$CERT]"

    output=$(echo | openssl s_client -connect ${CERT} 2>/dev/null |\
    sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' |\
    openssl x509 -noout -subject -dates 2>/dev/null)

    if [ "$?" -ne 0 ]; then
    $DEBUG && echo "Error connecting to host for cert [$CERT]"
    logger -p local6.warn "Error connecting to host for cert [$CERT]"
    continue
    fi

    start_dt=$(echo $output | sed 's/.*notBefore=\(.*\).*not.*/\1/g')
    end_dt=$(echo $output | sed 's/.*notAfter=\(.*\)$/\1/g')

    start_epoch=$(date +%s -d "$start_dt")
    end_epoch=$(date +%s -d "$end_dt")

    epochNow=$(date +%s)

    if [ "$start_epoch" -gt "$epochNow" ]; then
    $DEBUG && echo "Certificate for [$CERT] is not yet valid"
    logger -p local6.warn "Certificate for $CERT is not yet valid"
    fi

    secs_to_exp=$(($end_epoch - $epochNow))
    days_to_exp=$(($secs_to_exp / 86400))

    $DEBUG && echo "Days to expiry: ($days_to_exp)"

    warn_secs=$((86400 * $warn_exp_days))

    if [ "$secs_to_exp" -lt "$warn_secs" ]; then
    $DEBUG && echo "Cert [$CERT] is soon to expire ($secs_to_exp seconds)"
    logger -p local6.warn "cert [$CERT] is soon to expire ($secs_to_exp seconds)"
    fi
    done

    Save the script and make it executable.  chmod 755 test.sh

    Note the path to the script.  In this example:  /opt/nimsoft/sample/scripts/test.sh

     

    2- Deploy the logmon probe to the same Linux robot.

    Using the IM console, create a new profile for the logmon probe.
    The "mode" for the profile should be "command".
    The command should be the shell script above.

    Check off "generate alarm" on the "General" tab.

    Set it to run every 24 or 48 hours.

     

    3- Navigate to the Watcher Rules tab. 

    use a Matcher Rule like the following:
    Match Expression:
    Cert \[[a-z|.|]*:[0-9]*\] is soon to expire \([0-9]* seconds\)

    Message to Send on Match:
    Cert Expiration Soon: ${scriptMessage}

    4- Create a variable for the script message.

    On the Variables tab, create a new variable called: "scriptMessage" 

    with "text block" checked.

    Save and restart the probe.
    You should now see alarms for any server certificate that is due to expire in 90 days.
    I would advise running the script manually a few times to make sure it works in your environment.



  • 2.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Oct 12, 2017 08:37 AM

    This looks great! Thanks for sharing

     

    Looking forward to trying it!



  • 3.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Jan 30, 2018 10:05 AM

    Thanks a lot for sharing such a helpful piece of information. I am looking forward for doing this configuration for the servers in our infrastructure. can u plz help me to understand if we can put the host names or IP addresses for the servers (windows/Unix)where u have used yahoo.com or google.com. I am kinda new to Nimsoft as well. Please help



  • 4.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Jan 30, 2018 10:21 AM

    how can we configure the log string from those servers for server certification alerts



  • 5.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Broadcom Employee
    Posted Jan 30, 2018 04:46 PM

    Hi Vidyullata

    You may want to first pursue using the "url_response" (URL check in UMP console) as a solution for checking certificate expiration.

    If using the UMP to setup the monitoring, here is an example:

     

    Alternatively, you can use the IM console to configure the "url_response" probe.

    It has an option to check the certificate expiration.

     

    You should then get alarms if the certificate is due to expire within the "90 days" of expiration

    For example:

               URL Certificate due to expire in '88' days for profile 'Check CA Web Site'

    Hope that helps



  • 6.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Jan 31, 2018 12:12 AM

    Thanks Shawn for the quick response to my query. I understand the process u explained above and at the initial thread is for HTTP url certificate. I need to monitor the certificate expiry of  windows and Unix servers in our infrastructure. Kindly put some light on that configuration. 

    Thanks in advance

    Vidyu



  • 7.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Broadcom Employee
    Posted Jan 31, 2018 04:44 PM

    Hi Vidyu,

     

    Unfortunately, I am not aware of anything in UIM that can monitor / test certificate expiration outside of https certificates.

    Can you tell us a little more about what your goal is, like what types of certificates these are?  We may think of some creative options.

     

    Thanks,

    Shawn



  • 8.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Feb 06, 2018 07:10 AM

    Hi Shawn,

     

    I've visited this issue a number of times and always struggled to check Server Certificate expiry dates.  From your initial post above it looks to cover this off, but reading the script example it points to google.com:443.

     

    Could (for example) the cert for google.com:443 be monitored using URL response? As shown in your later comments?

     

    As Vidyu has mentioned, I also I'm struggling to define what to put in the "Server" field.  I think the two different approaches need defining on what they are monitoring and if/how they differ.

     

    Personally I think the difference is between a URL cert and a Server Cert, but both solutions above suggest they do the same thing.

     

    From the script example and your description it looks like the Script approach will check Server Certs and URL response will check URL certs.  But again, I'm confused as to what to put in the servers_to_check field of the script.

     

    I hope that makes sense?!?!

     

    Cheers,

    Sam



  • 9.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Feb 06, 2018 07:12 AM

    There is also this post, but it'd never really worked for me.  This sounds great for Windows systems:

     

    Probe to check and store the days until a certificate expires 



  • 10.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Broadcom Employee
    Posted Feb 06, 2018 02:53 PM

    Thanks for the feedback Sam.  The script example above is something that could be built on.  My intent was to show that this is something that could be done and that similar things could be accomplished with the logmon probe.

    It is not a complete solution by itself.  But the script and the url_response - check expiration are doing the same thing.

    The url_response method would be the supported and "out of box" method and easier to setup.

    In the script example, the servers to check is a list of servers and their respective ports.  It is an array separated by spaces and the next section loops through each item in the list.

    I was not aware of James' post Probe to check and store the days until a certificate expires 

    Openssl is available on Linux too, so it should work there also with a little work.



  • 11.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Feb 08, 2018 06:47 AM

    Hi Shawn,

     

    That's great, thanks for clarifying and thank you for sharing the script!

     

    Cheers,

    Sam



  • 12.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Feb 08, 2018 06:51 AM

    For windows users you can also use powershell to check Cert expiry dates.  Here's a link to a script which looks like it could be useful:

     

    Checking SSL certificate values with Powershell | Nathan's Thoughts and Notes 



  • 13.  Re: Tuesday Tip: Monitoring Server Certificate Expiration with the Logmon Probe

    Posted Feb 08, 2018 09:26 AM

    Further to this...

     

    Whilst I'm investigating the Powershell route, there was an urgency to have our expiry dates monitored.  To provide this assurance I have implemented a rudimentary way of monitoring these dates.  We hold all our expiry dates in a spreadsheet and I currently have logmon reading the "date" column.  This date is then ran through a LUA script to ensure that the date is no more, or no less than 10 days away from today.

     

    This way, whenever the spreadsheet is updated and a date in the date column is 10 days away from today - exactly we get alerted.

     

    Here's the script:

     

    a = alarm.get()

    message = "Certificate Expiry Alert - Issue with data"
    SUPPKEY = "Certificate_Expiry"
    SUBSYS = "1.1"
    SOURCE = "192.168.***.*"

    -- Settings
    local allowed_age = 20160 -- In minutes

    -- Input line (for testing only)
    --local inputstr = "DOMAIN\\USERNAME,Web Server (WebServer),13/01/2017 09:13,13/01/2019,COMPANY_NAME,HOSTNAME_FQDN,SITE"

    -- Separate line into 7 variables by token ","
    local path, server, time, date, company_name, hostname, site = string.match(a.message, "([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+),([^,]+)")

    -- Note, some additional checks could be here (eg. regex to match DD/MM/YYYY format)
    --if date == nil then
    --nimbus.alarm (1, message , SUPPKEY , SUBSYS , SOURCE)
    --print("Error reading line: "..a.message)
    --end

    -- Get current time minus 9000 minutes (in format DD/MM/YYYY)
    local target_date = os.date("%d/%m/%Y", os.time() + allowed_age * 60)

    -- Printing what we got (for testing)
    print("Target date: "..target_date..", Input date: "..date)

    -- Testing the match
    if target_date == date then
    nimbus.alarm (2, a.message , SUPPKEY , SUBSYS , SOURCE)
    -- print("Dates are matched!")
    else
    print("Dates are not matched!")
    end

     

    This would need amending to suit your particular needs, the above script takes the entire line from the spreadsheet and obviously vary for others with different needs.

     

    There's also room to add extra error checking such as a date been input as an invalid format. 

     

    You could also get it to alert every day, once the threshold has been breached.  In our case we only want one alert - but if you think there's a potential for this to be missed you could easily edit the script to realert on a daily basis until the Cert is renewed.

     

    Hope that helps someone out there!