ESP Workload Automation

We currently have a few processes that looks something similar to below, they run every couple of minutes. At times this process will stop for different reasons (Agent Notified, Queued for Submission, Suberror, etc.). With Suberror we get a notification as it fails but when in agent notified or queued for submission, or possibly just executing for too long we have no notification as these processes are critical. Trying to find the best way to receive alerts when these processes stop for the different reasons. I have found I can put MAXRUNTIME=1 on the NTJOB and then the following code in the JSDELAY job and that seems notify when NTJOB has been running over a minute.  

ESPCMDNM AJ CDD0810FWL1_24X7 RESET LATEEND('RESET') +
APPL(%ESPAPPL..%ESPAPGEN)

Looking for best practice to capture the other scenarios, like agent notified or queued for submission. Any ideas would be appreciated.

NT_JOB CDD0810FWL1_24X7 

RUN TODAY
EXITCODE 1-9999 SUCCESS

MAXDELAY=1
AGENT CLT_A145_LNG01
USER connectd
CMDNAME D:\DATATRANS\SCRIPTS\FILEWATCHER.BAT
RESOURCE ADD(1,DRES)
RELEASE ADD(JSDELAY_CDD0810FWL1_24X7)
ENDJOB

JOB JSDELAY_CDD0810FWL1_24X7 TASK SELFCOMPLETING
RUN TODAY
ESPCMDNM AJ RETRIG_CDD0810FWL1_24X7 RESET DELAYSUB('REALNOW +
PLUS 2 MINUTES') APPL(%ESPAPPL..%ESPAPGEN)

ESPCMDNM AJ CDD0810FWL1_24X7 RESET LATEEND('RESET') +
APPL(%ESPAPPL..%ESPAPGEN)
RELEASE ADD(RETRIG_CDD0810FWL1_24X7)
ENDJOB

JOB RETRIG_CDD0810FWL1_24X7 TASK SELFCOMPLETING
RUN TODAY
ESPCMDNM RERUNM ROOTJOBS(CDD0810FWL1_24X7+) +

APPL(%ESPAPPL..%ESPAPGEN)
ENDJOB

Thank you.

Sharon

Hi Sharon,

Using MAXRUNTIME and NOTIFY OVERDUE can make it much simpler. Following is an example User's Guide:

Example: Specifying Application-level Minimum and Maximum Run Times with Global Notification
In the following example, application PAYROLL has two jobs, PAY1 and PAY2. The minimum run time for each job is set to 25% of its current average runtime; the maximum run time for each job is set to 175% of its current average run time. If either job runs over the maximum time or ends before having run for the specified minimum time, a notification message is sent to user CYBER02.
APPL PAYROLL
JCLLIB 'CYBER.JCL.CNTL'
IF %ESP_APPL_PROC THEN DO
MINRUNTIME = AVGRUNTIME * 25 / 100
MAXRUNTIME = AVGRUNTIME * 175 / 100
ENDDO

NOTIFY OVERDUE USER(CYBER02)
NOTIFY PREMEND USER(CYBER02)
JOB PAY1
RUN NOW
ENDJOB
JOB PAY2
RUN NOW
ENDJOB

----

Of course, you can set MAXRUNTIME=x within each job definition.

Hope this helps,

Lucy

Hi Lucy,

Yes we can get MAXRUNTIME to work but sometimes the job has not started as it will go into "Agent Notified", will MAXRUNTIME pick this up as well, what about if it stays in "Queued for Submission"?

Thank you.

Sharon

Hi Sharon,

MAXRUNTIME is counted from the job starts running. So if the jobs stay in "Agent Notified" or "Queued for Submission" for long time, then it won't function as expected.

Then like what you are doing, DUEOUT EXEC time and LATEEND (on AJ command) should be set/used to help mark the job as overdue.

Hope other users can provide more helpful input,

Lucy

Hi Lucy,

Will DUEOUT EXEC catch those jobs in "Agent Notified" or "Queued for Submission"? Just want to make sure I understand the last response.

Thank you.

Sharon

I have verified that DUEOUT EXEC will detect jobs in "Agent Notified".

The DUEOUT along with NOTIFY OVERDUE ALERT(<Alert ID>) should provide what you need.

DUEOUT EXEC does work but the issue I am having is how to code the DUEOUT EXEC and how to reset it since this is a cyclic process. I have tried DUEOUT EXEC REALNOW PLUS x MINUTES (also tried DUEOUT EXEC NOW PLUX x MINUTES) and then tried to RESET it but that doesn't seem to work.The job I am trying to monitor gets reset to run every 2 minutes by code like below.

ESPCMDNM AJ B35209C_RETRIG RESET DELAYSUB('REALNOW +
PLUS 2 MINUTES') APPL(%ESPAPPL..%ESPAPGEN)

I've tried to use this to reset the DUEOUT EXEC but that doesn't work.

ESPCMDNM AJ B35209C RESET LATEEND('RESET') +
APPL(%ESPAPPL..%ESPAPGEN)

Any thoughts on how to reset DUEOUT EXEC on a cyclic process like this?

Thanks,

Sharon

Would adding something like this as a successor to CDD0810FWL1_24X7 work?

Commented out SEND and SAY for testing.
                                                                                                 
JOB LCSF TASK SELFCOMPLETING 
/* SEND '>>> LCSF: %LCSF_CMD.' USER(*) */ 
  LCSF_CMD = "LCSF (AGENT EQ 'CLT_A145_LNG01')"
  RUN ANYDAY
  DELAYSUB criteria 
  REXXON /* PROC */
    Y = TRAPOUT("LCSF.")
      ESP "%LCSF_CMD" 
    Y = TRAPOUT("OFF")
/*  SAY "LCSF. LINES: "LCSF.0 */ 
    DO I = 1 TO LCSF.0 
      PARSE VAR LCSF.I . "SystemStatus("SystemStatus")" . 
      SAY "SystemStatus: "SystemStatus"."  
/*    Add logic to do what you need for a given SystemStatus */
    END   
  REXXOFF 
ENDJOB 

Thanks for the idea, I'll see if I can make something work with this.

Question, when running this it comes back with SystemStatus: "Agent Notified", how do I drop those quotes so I only get SystemStatus: Agent Notified?

Here is the code I have.

REXXON PROC
Y = TRAPOUT("LCSF.")
ESP "%LCSF_CMD"
Y = TRAPOUT("OFF")
DO I = LCSF.0 TO LCSF.0
PARSE VAR LCSF.I . "SystemStatus("SystemStatus")" .
SAY "SystemStatus: "SystemStatus""
"SystemStat='"SystemStatus"'"
END
REXXOFF
IF '%SystemStat' EQ '"Agent Notified"' THEN
DO......

Thank you for all your assistance on this.

Sharon

Add REXX "Strip" Command to remove both "B" leading and trailing quotes.

REXXON PROC
Y = TRAPOUT("LCSF.")
ESP "%LCSF_CMD"
Y = TRAPOUT("OFF")
DO I = LCSF.0 TO LCSF.0
PARSE VAR LCSF.I . "SystemStatus("SystemStatus")" .

SystemStatus = Strip(SystemStatus,B,"'")  
SAY "SystemStatus: "SystemStatus""
"SystemStat='"SystemStatus"'"
END
REXXOFF
IF '%SystemStat' EQ '"Agent Notified"' THEN
DO......

Thanks, I added this and it worked. Appreciate all your help with this.

SystemStatus = Strip(SystemStatus,B,'"')