ESP Workload Automation

Expand all | Collapse all

Running Jobs Only After 7 Files Are Present

  • 1.  Running Jobs Only After 7 Files Are Present

    Posted Jan 06, 2017 01:46 PM

    Does anyone have a suggestion on the best way to handle the following:

    Execute all the jobs within an appl when there are 7 files ready to process.

    The files could come at any time - all at once or spread throughout the day.

    I was thinking of creating a resource to 7 file trigger jobs and when the 7th file arrives, set the resource count to 7 and allow the other jobs in the appl to run.  The event would be pre-loaded.  If the file names have a constant in them it would be easy to setup a file trigger but what if the file name could be anything?  Am I thinking about this the right way?  Any suggestions are most welcome!



  • 2.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 06, 2017 03:40 PM

    here is a take on it. Let me know if its not what you are looking for and ill get back to the drawing board ! 

    Its quite similar to your idea i think but its just a different perspective. 

     

     

    you could setup a file monitor which would look for anything that gets created in your folder.

    using the CONTINUOUS(* function you could kick an event which would check a Global table to check if the number is 7, if its not you increment the current number by 1. if its 7 then you release/trigger the job you need to run then reset the counter to 0 

     

    Let me know if this is clear or if i miss understood the requirement ! 

     

    Thanks 

    Alex 



  • 3.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 06, 2017 03:56 PM

    Do you know the format of the file names?

    Are they all in the same directory?

     

    Are the filename the same every day or do they have a date/time/sequence number?

     

    Are the files coming from one source or multiple?

    If the 7 file names are unique and you could create a FILE_TRIGGER Workload Object for each file.  This would be easier to troubleshoot when one of more files did not arrive.  Each FILE_TRIGGER Workload Object would RELEASE the job that processes the 7 files.  When a FILE_TRIGGER Workload Object detects a file the hold count would be decremented and there would not be a need for the resource count. 

     

    You would then schedule the event daily and put a DUEOUT of the job that processes the files to trigger an alert mechanism when all 7 files have not arrived 



  • 4.  Re: Running Jobs Only After 7 Files Are Present

    Broadcom Employee
    Posted Jan 06, 2017 04:01 PM

    If you can define 7 file_triggers, then each of them can release APPLSTART WOB. This is a special ESP wob, it will release all jobs in the APPL except its predecessors.

     

    Otherwise I think use global variable to store the number of files (as Alex mentioned) is a good idea. Other than use CONTINUOUS FILE_TRIGGER, you can also use WOBTRIG in the event definition, and in the INVOKED ESP Proc, add something like below:

    IF ESP_APPL_GEN THEN

      VGET COUNT TABLE(***)

      IF %COUNT < 7 THEN EXIT

      IF %COUNT = 7 THEN VSET COUNT '0' TABLE(***)

    END

     

    It will count 7 times before continue to run all jobs in the APPL.

     

    Hope this helps,

     

    Lucy



  • 5.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 09, 2017 09:30 AM

    Thanks for your responses!  I am asking a few more questions of the programmer and will respond when I know a little more.  Again, thanks!



  • 6.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 09, 2017 03:06 PM

    I have a little more information now. 

    Each file will have a unique name each day/time it is run (most likely no duplicate names over time). 

    The same file folder will hold all the files. 

    All files will have the same qualifier on them - .lst

    There will always be 7 files.  If there are not 7 files, nothing should be processed.

    The programmer indicated all 7 files need to be processed at the same time. 

    All files should arrive within an hour of the first file.

     

    I was trying to find information on the CONTINUOUS parm but really didn't find anything that went along with file or wob triggers.  Is there anything in the documentation that will help me to better understand this parm?

     

    How does one know if they are currently using a global variable anywhere?  I have a feeling we do not currently use these.  This would not be something one could set inside the appl like a gentime?  I also looked for a %COUNT but didn't find this in the command reference.  I'd like to know how the field within this is defined - length of field, can decimals be added, etc..

     

    If we setup a global variable, what are the implications of using it?  Pros or cons.

     

    I am also a little unsure of setting up 7 file triggers having each release an APPLSTART WOB.  I am in the process of reading about this now.  So this would be like having a appl that releases an appl?  If you were keeping to a standard naming convention, how might one do this?  We currently require all appl names, internal appl names, and event names to reference the same name.  Would I be able to do that with this?

     

    I apologize for my lack of knowledge on this - but I do want to learn and understand it.  Thanks!

     

     

     

     

     

     

     

     

     



  • 7.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 10, 2017 08:25 AM
      |   view attached

    If you are going to count files and they all do not come in for a given day then you will need to reset the counter every day.  If not they you could start processing at the wrong time.  For example if you received 6 files on Monday and the 7th file never was created you would not want to start processing on Tuesday after the first file is created.  Would you?

     

    In the CA Workload Automation ESP Edition Command Reference Guide Command Reference Guide Release 11.4

     

    There are examples for:

    • APPLSTART Statement
    • FILENAME Statement

     

    I have attached an example of a PROC with a FILE_TRIGGER Workload Object for each file (1-7) that should provide a starting point for a solution. 

    • The PROC would be scheduled to run daily. 
    • If all 7 files are received it will then run the script that process the files. 
    • If one of more files is not created it will COMPLETE the application.

     

    Let me know if you have any questions.

    Attachment(s)

    zip
    FILETRIG.TXT.zip   1 KB 1 version


  • 8.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 10, 2017 10:05 AM

    Thanks for the code.  Here is what I have.  My question now is how does ESP know to only process 1 of the file triggers at a time if all the files it is looking for are named the  *.LST?  Again, thanks for your help - it is much appreciated!

     

    APPL TESTME                                                   
                                                                  
    FILE_TRIGGER FILE_TRIG1                                       
     AGENT AGENT05                                                
     FILENAME +                                                   
      '\\cinfs01p\Billingtest\*.LST'   CREATE                        
       RELEASE ADD(APPLSTART_PROCESS_fILES)                       
       RUN WORKDAYS                                               
    ENDJOB                                                        
                                                                  
    FILE_TRIGGER FILE_TRIG2                                       
     AGENT AGENT05                                                
     FILENAME +                                                   
      '\\cinfs01p\Billingtest\*.LST'    CREATE                        
       RELEASE ADD(APPLSTART_PROCESS_fILES)                       
       RUN WORKDAYS                                               
    ENDJOB       

                                                     
    FILE_TRIGGER FILE_TRIG3                                            
     AGENT AGENT05                                                     
     FILENAME +                                                        
      '\\cinfs01p\Billingtest\*.LST'   CREATE                             
       RELEASE ADD(APPLSTART_PROCESS_fILES)                            
       RUN WORKDAYS                                                    
    ENDJOB     

    (4 additional file triggers - all the same except the FT name.

     

    APPLSTART APPLSTART_PROCESS_FILES              
      RELEASE ADD(BILLPRINT_PROCESS)               
    ENDJOB                                          
                                                     
    NT_JOB BILLPRINT_PROCESS                       
      AGENT AGENT05                                
      CMDNAME C:\ESP\Apps\BILLPRINT.bat            
      RUN WORKDAYS                                 
    ENDJOB                                         

     

                                                            
                                                                                                                                      

     



  • 9.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 10, 2017 10:33 AM

    You stated "Each file will have a unique name each day/time it is run)

    You will need to replace the * in the filename with something that makes each file unique.
    In my example each file had a unique number (1-7), but it can be anything that differentiates the 7 different files.

     

    In my example the last 2 jobs ensure that the PROC runs to completion everyday whether or not all 7 files are created.

     

    Can you provide an example of the 7 file names that would be created for a given day?



  • 10.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 10, 2017 12:39 PM

    I am suffering from Programmer Aggravation today.  Had the programmer answered all my questions in the beginning, I would not have needed to post anything here.  The first 3 levels of the job name will be distinct, hence the file triggers will be distinct.  I have learned something though about the APPLSTART which I will use with this.  Thanks for everyone who has added a suggestion - I appreciate your help!



  • 11.  Re: Running Jobs Only After 7 Files Are Present

    Broadcom Employee
    Posted Jan 25, 2017 08:35 AM

    Hi Barb,

     

    Have you got the answer you need? If yes, will you mark this as "Answered"?

     

    Thank you,

     

    Lucy



  • 12.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 11, 2017 03:25 AM

    Hello

     

    And what if you run a simple command job to get the number of *.lst files in the directory then trigger a successor job.

    To get the number of *.lst file in a given directory on Windows, you can run this command:

     

    dir *.lst /b/a-d | find /v /c "::"

     

    For example:

    C:\testdir>dir *.lst /b/a-d | find /v /c "::"
    2

     

     

    And my test directory contains 2 *.lst files

     

    C:\testdir>dir
    Volume in drive C has no label.
    Volume Serial Number is 76E8-C15B

    Directory of C:\testdir

    01/11/2017 09:15 AM <DIR> .
    01/11/2017 09:15 AM <DIR> ..
    01/11/2017 09:15 AM 4 a.lst
    01/11/2017 09:15 AM 4 b.lst
    2 File(s) 8 bytes
    2 Dir(s) 42,444,754,944 bytes free

     

    On Unix/Linux

     

    ls *.lst |wc -l

     

    Regards

    Jean Paul



  • 13.  Re: Running Jobs Only After 7 Files Are Present

    Posted Jan 30, 2017 02:39 AM

    Hi Barb

    Did any of these solutions help?

    If so, don't forget to mark correct the answer

    Regards

    Jean Paul