DX Application Performance Management

  • 1.  Email Shell Script

    Posted Jul 17, 2017 10:53 AM

    With the rich email feature within 10.0 not able to use a list within the subject, we are having to fall back to modifying our email_script to construct a dynamic subject line that will pull the first row of the alert list and include it into the subject.

     

    What is different from running the email_script.sh from command line versus calling it from an APM action?

     

    --------------- details ---------------

    We are running into a problem and we are hopeful that someone in the community has already hit this issue.

     

    We can run an email_script.sh from command line with no issue, and we receive an email with a well formed subject. We run the same email_script.sh called from a APM action and the subject line is blank.

     

    We are reading a subject line template and replacing tokens then piping that to a subject file.  Reading in the file for the first line, which should be the fully formed subject line.  This is basically the same thing we are doing with the body.

     

    The difference is the way we call the mail command with the values, subject, body.

     

       # Define the text body file with the system PID ($$) so calls won't cross over.
       OUTPUT="/wilyMom/alerts/alert.$$.output"
       SUBJECT="/wilyMom/alerts/subject.$$.output"

     

       # Since there are more than 9 parameters, shifting the values to get them all
       userdefined=$1
       shift
       texttoshell=$1
       shift
       emname=$1
       shift
       emhost=$1
       shift
       emip=$1
       shift
       alertname=$1
       shift
       alertstatus=$1
       shift
       alerttime=$1
       shift
       agentnames=$1
       shift
       metricnames=$1
       shift
       previousalertstate=$1
       shift
       hasalertstatechanged=$1
       shift
       thresholdvalue=$1

    # Parse of userdefined parameter
    userdefined=`echo "$userdefined"|sed s/\"//g`
    email=`echo $userdefined |cut -d '+' -f1`
    onCall=`echo $userdefined |cut -d '+' -f2`
    instSet=`echo $userdefined |cut -d '+' -f3`
    lmf=`echo $userdefined |cut -d '+' -f4`
    action=`echo $userdefined |cut -d '+' -f5`
    if [ -z "$action" ]; then
    action="Danger"
    fi

     

    # Build subject line
    if [ -f /wilyMom/alerts/${instSet}Caution.subject ]; then
    # Grab the offending value from the "text" variable.
    offendingvalue=$(echo "$text" | tail -n1 | awk '{print $NF}')
    # Use customer-defined subject line, if provided.
    cat /wilyMom/alerts/${instSet}Caution.subject | sed -e s/::agentserver::/$agentserver/g -e s/::alertname::/$alertname/g -e s/::offendingvalue::/$offendingvalue/g -e s/::alertstat::/$alertstat/g -e s/::environment::/$environment/g >> $SUBJECT
    else
    # Use default subject line
    cat /wilyMom/alerts/defaultCaution.subject | sed -e s/::alertname::/$alertname/g -e s/::alertstat::/$alertstat/g -e s/::environment::/$environment/gi >> $SUBJECT

    fi

     

    if [ -f /wilyMom/alerts/${instSet}Caution.instructions ]; then
    # Check for user-defined instructions
    cat /wilyMom/alerts/${instSet}Caution.instructions |sed -e s/::environment::/$environment/g -e s/::server::/$emhost/g -e 's/EE//g' -e '/GG.*GG/d' >> $OUTPUT
    else
    # Default instructions are standard caution (SiteScope instructions)
    cat /wilyMom/alerts/defaultCaution.instructions |sed -e s/::environment::/$environment/g -e s/::server::/$emhost/g -e 's/EE//g' -e '/GG.*GG/d' >> $OUTPUT
    fi

     

    We have tried several combinations

     

    # Read one-line SUBJECT file into $subject variable
    read -r subject <"$SUBJECT"

    mail -r APM_ALERT -s "$subject" $email < $OUTPUT

     

    When ran from command line, the subject file has the right content but when we run this from action, the file and email subject is empty.

     

    This is the user-defined parameter in the action.  The + is used for the field delimiter since , can be within the email addresses.

     

    <emailaddresses>+MONITORING_ADMIN+defalt+f+caution

     

    The third field, "defalt" is misspelled since it should never be located and the default template used.

     

    Thank you,

     

    Billy



  • 2.  Re: Email Shell Script

    Posted Jul 28, 2017 08:24 AM

    We punted.  We couldn't get the subject token method to work so we had to fall back to concat the subject line directly instead of having a subject template file with replacement tokens.  This limits the different subjects that we can offer since each will require code changes to the email script shell file.  

     

    Our theory is that APM action uses the standard in/out/error and will not allow the use of the standard pipes in order to more|cat|less a text file back into memory.  The body-file method works because we are piping the file directly into the mail command, which accepts a file for the body but does not accept a file for the subject.

     

    A bit more detail on why we want to do this.  Within our alert we have the "Notify by individual metric" which causes the "pass text to shell" to be a single metric.  We tail the last line and parse it out to get the alert name, host, metric and metric value to inject into the subject line.   This is so the receiver of the email does not need to open the email to understand what the problem is, the subject line has the server, metric and metric value and they can start triage before they even log into the email client to read the body of the email or open the APM interface/GUI.

     

    Billy



  • 3.  Re: Email Shell Script

    Broadcom Employee
    Posted Jul 28, 2017 09:27 AM

    Thanks Billy for sharing your experiences in detail and why you want to do things a certain way. Please keep sharing, asking, and answering questions!