Automic Workload Automation

  • 1.  Validating the content of object scripting tabs

    Posted Jun 26, 2017 08:29 AM

    We recently identified a gap in our user authorization model, and we are developing a solution. First, a little background info...

     

    We have four separate staging environments:

    Staging environmentDescriptionUser authorizations
    EXPExperimental: testing new versions, new features, and anything that requires interrupting the system
    • Full access to all objects
    DEVDevelopment: design, creation, and initial testing of workflows and related scheduling objects
    • Read access to all objects
    • Write access only to project-specific objects
    • Execute access only to project-specific objects
    ITEIntegrated Testing Environment: testing & validation of workflows using real-world data
    • Read access to all objects
    • Execute access only to project-specific objects
    • No write access
    PRODProduction
    • Read access to all objects
    • No write
    • No execute access

    In DEV and ITE, we ensure that users can execute only objects that belong to their particular project, and we control this using a naming convention. All objects begin with a three-character project prefix. Users belong to one or more user group associated with these project prefixes. Each user group grants rights to work with objects of that project prefix.

     

    In PROD, normal users are not allowed to execute objects. Instead, objects must be scheduled using a JSCH object, and these schedule objects are executed automatically as an administrative user when they are deployed to production. This administrative user has broad authorizations in the production system, because it must be able to execute objects belonging to any project prefix. In production, almost all tasks run as this administrative user. Up until now, we have not checked the scripting tabs of objects being deployed to PROD to ensure that users are not doing things the should not. We recently discovered that some users have been using such statements to execute objects outside of their project. These statements — e.g., ACTIVATE_UC_OBJECT, would fail in DEV or ITE because they would be run as ordinary users that lack broad access rights; but in PROD, because the calling tasks are started automatically as an administrative user, these statements essentially run unchecked.

     

    We would ideally prefer to solve this problem by executing the schedules are project-specific users — technical (non-personal) users that have limited execution authority for a specific project prefix, but no broader access rights. Unfortunately, until the Automation Engine provides a mechanism for user impersonation, this approach will not be feasible.

     

    So until this User impersonation enhancement is delivered, we need to make sure users are not inserting ACTIVATE_UC_OBJECT statements into their objects that run objects to which they would normally not have access. This means during a deployment to production, we must check, line-by-line, the content of the scripting tabs of all relevant object types, look for ACTIVATE_UC_OBJECT statements, and validate that the name of the executable objects referenced in such statements begin with the same project prefix as the calling object.

     

    I already had a script that extracted the scripting lines of z/OS jobs, parse_zOS_jobs_in_transport_case.sh, so I took the code from this script and merged it into the more general transport case parsing script, parse_transport_case.sh. Version 0.7 of this script, posted earlier today, is now capable of printing all scripting tab content of relevant objects — objects with one or more scripting tab. These are: CALL, EVNT, JOBF, JOBG, JOBI, JOBP, JOBS, JSCH, and SCRI.

     

    I wrote a separate script, validate_obj_script.sh, that parses the scripting tab content and performs the required validation of ACTIVATE_UC_OBJECT statements. I will post this script once it is complete. I realized earlier today that I still have some work to do before it’s ready, because the scope of potentially problematic scripting statements is much broader than just ACTIVATE_UC_OBJECT statements.

     

    AE scripting statements that could cause harm include:

    Scripting statementDescriptionScope
    ACTIVATE_UC_OBJECTExecute an arbitrary object in the system.Task
    CANCEL_UC_OBJECTTerminate an active task.Task
    CREATE_OBJECTCreate a calendar, login, or static VARA object.Object
    MODIFY_OBJECTModify a calendar, login, or static VARA object.Object
    MODIFY_SYSTEMPerform various system administration tasks.System
    MODIFY_TASKModify a task within an active workflow.Task
    MODIFY_UC_OBJECTModifiy an attribute of an active task.Task
    MOVE_OBJECTMove an object to a different folder.Object
    REMOVE_OBJECTRemove an AE object from the system.Object
    :REPLACE_STRUCTUREReplace the structure of a workflow with the structure of another workflow.Task
    RERUN_UC_OBJECTContinue a workflow that has been rolled back.Task
    RESTART_UC_OBJECTRestart a task.Task
    ROLLBACK_UC_OBJECTRoll back a task.Task
    SET_SYNCPerform an action defined in a SYNC object.Sync
    :SET_UC_SETTINGChange system settings during system operation.System
    :SHUTDOWNShut down the Automation Engine completely.System
    :TERMINATEStop an agent or Automation Engine server process.System
    TOGGLE_OBJECT_STATUSStop or start the automatic processing of a task.Task
    TOGGLE_SYSTEM_STATUSStop or start the automatic processing of a client.System


  • 2.  Validating the content of object scripting tabs

    Posted Jun 26, 2017 10:49 AM
    For the elements that affect the system, I think we’re safe. The administrative user that executes objects does not have system administration privileges.

    For the elements that affect objects, we will simply prohibit their use, because they essentially circumvent the standard deployment process.

    For SET_SYNC statements, we can verify that only the project’s own SYNC objects are referenced.

    After reviewing the syntax of the script elements that affect individual tasks, I have come to the following conclusion:
    There is no way to allow users to use most of these statements without also granting them the access necessary to accidentally or intentionally interfere with tasks belonging to other projects.
    This is because with two exceptions (ACTIVATE_UC_OBJECT and RESTART_UC_OBJECT), all of these scripting statements take a run ID, and not an object name, as their primary argument. There is therefore no way to ensure that these statements are used only to manipulate objects belonging to the same project prefix as the calling task. To ensure prefix segregation, we will have to prohibit the use of scripting statements that alter active tasks based on a run ID.


  • 3.  Validating the content of object scripting tabs

    Posted Jun 27, 2017 10:05 AM

    I categorized these scripting statements based on what they change and how they are specified. I also listed how frequently they are used in our production environment.

    Category of scripting statementScript elementsPrevalence
    Start/restart tasks based on object nameACTIVATE_UC_OBJECT, RESTART_UC_OBJECTHigh
    Change tasks based on run IDCANCEL_UC_OBJECT, MODIFY_TASK, MODIFY_UC_OBJECT,  RERUN_UC_OBJECT, ROLLBACK_UC_OBJECT, TOGGLE_OBJECT_STATUSMedium
    Perform SYNC actionsSET_SYNCLow
    Change tasks based on object name:REPLACE_STRUCTUREVery low
    Change regular objectsCREATE_OBJECT, MODIFY_OBJECT, MOVE_OBJECT, REMOVE_OBJECTVery low
    Change systemMODIFY_SYSTEM, :SET_UC_SETTING, :SHUTDOWN, :TERMINATE, TOGGLE_SYSTEM_STATUSVery low


  • 4.  Validating the content of object scripting tabs

    Posted Jun 27, 2017 03:37 PM
    I just realized that the problem with privilege escalation is not limited to the content of scripting tabs. A user could also accomplish similar things via task pre- or post-conditions.

    Does anyone have an SQL query that will list all workflow tasks whose pre- or post-conditions include an execute object action?


  • 5.  Validating the content of object scripting tabs

    Posted Jun 28, 2017 11:37 AM

    I added the new validate_obj_script.sh script. Based on how the various script elements work, and how frequently they are used, I decided to handle them as follows:

     

     

    Category of scripting statementScript elementsPrevalenceHandling
    Start/restart tasks based on object nameACTIVATE_UC_OBJECT, RESTART_UC_OBJECTHighAllowed only if referenced executable object belongs to the same business unit
    Change tasks based on run ID — group 1CANCEL_UC_OBJECT, MODIFY_TASKMediumAllowed only if referenced executable object belongs to the same business unit
    Change tasks based on run ID — group 2MODIFY_UC_OBJECT,  RERUN_UC_OBJECT, ROLLBACK_UC_OBJECT, TOGGLE_OBJECT_STATUSLowProhibited
    Perform SYNC actionsSET_SYNCLowAllowed only if referenced SYNC object belongs to the same business unit
    Change tasks based on object name:REPLACE_STRUCTUREVery lowAllowed only if loaded workflow belongs to the same business unit .
    Change regular objectsCREATE_OBJECT, MODIFY_OBJECT, MOVE_OBJECT, REMOVE_OBJECTVery lowProhibited
    Change systemMODIFY_SYSTEM, :SET_UC_SETTING, :SHUTDOWN, :TERMINATE, TOGGLE_SYSTEM_STATUSVery lowProhibited


  • 6.  Validating the content of object scripting tabs

    Posted Jun 29, 2017 05:48 AM

    I think I figured it out. This SQL query will list the details of all workflow tasks that have a pre/post-condition that execute an object.

    with task_conditions as
    (select * from OH
    inner join JPP on JPP_OH_Idnr = OH_Idnr
    inner join JPOV on JPOV_OH_Idnr = OH_Idnr
    inner join JPPO on JPPO_OH_Idnr = OH_Idnr)

    select OH_Name,OH_Title,JPP_Alias,JPP_Object,JPP_LNR,JPP_Row,JPP_Col,
    case JPOV_Location when 1 then 'Pre-conditions'
    when 2 then 'Post-conditions' end as Location,
    cast(JPOV_Value as VARCHAR2(200)) as Obj_to_Exec
    from task_conditions
    where 1=1
    and JPPO_JPP_Lnr = JPP_Lnr
    and JPOV_JPP_Lnr = JPP_Lnr
    and JPOV_JPPO_Lnr = JPPO_Lnr
    and JPPO_Location = JPOV_Location
    and JPPO_CarName = 'EXECUTE OBJECT'
    and JPPO_Type = 'A'
    and JPOV_Type = 'V'
    and JPOV_Value is not null
    and JPOV_Value not like '##%'
    and JPOV_VName = 'XC_P01'


    Extracting this information from a transport case unload file would be a bit involved because of the multiple levels of abstraction between the workflow, task, condition, and action.

     

    Update: After a bit more testing, I got a better understanding of the relationships and cleaned up the query. It works reliably now.



  • 7.  Validating the content of object scripting tabs

    Posted Jun 29, 2017 11:51 AM
    Because we are in the process of switching to an XML-based approach to batch deployment, I also identified the corresponding structures in the JOBP portion of the AE object XML schema. Even when the XML schema was partly documented, the schema for task pre- and post-conditions was not  documented.

    Below are a couple of Xpath queries based on my best guesses of the structure.

    1. Identify workflows with tasks with EXECUTE OBJECT actions defined in task pre- or post-conditions:
    /uc-export/JOBP/JOBP/JobpStruct/task/preconditions/PreCon/conditions/action[@id="EXECUTE OBJECT"]/params/param[@name="XC_P01"]/../../../../../../../../../@name
    2. List the names of the executable objects referenced by such actions:
    /uc-export/JOBP/JOBP/JobpStruct/task/preconditions/PreCon/conditions/action[@id="EXECUTE OBJECT"]/params/param[@name="XC_P01"]/@value



  • 8.  Validating the content of object scripting tabs

    Posted Jun 30, 2017 07:28 AM
      |   view attached

    Here is a bit of XSLT that will generate an HTML table listing all EXECUTE OBJECT actions defined in task pre- or post-conditions of workflows in an XML export file.

    <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
         <!-- Idendtity template -->
         <xsl:template match="/">
              <html>
                   <body>
                        <h2>Workflows with <i>EXECUTE OBJECT</i> actions defined in pre- or post-conditions.</h2>
                        <table border="1">
                             <tr bgcolor="#66ccff">
                                  <th>Workflow</th>
                                  <th>Running #</th>
                                  <th>Row</th>
                                  <th>Column</th>
                                  <th>Object</th>
                                  <th>Pre/Post</th>
                                  <th>Object to execute</th>
                             </tr>
                             <xsl:for-each select="/uc-export/JOBP/JOBP/JobpStruct/task//conditions//action[@id='EXECUTE OBJECT']/params/param[@name='XC_P01']">
                                  <tr>
                                       <td>
                                            <xsl:value-of select="ancestor::JOBP/../@name"/>
                                       </td>
                                       <td>
                                            <xsl:value-of select="ancestor::task/@Lnr"/>
                                       </td>
                                       <td>
                                            <xsl:value-of select="ancestor::task/@Row"/>
                                       </td>
                                       <td>
                                            <xsl:value-of select="ancestor::task/@Col"/>
                                       </td>
                                       <td>
                                            <xsl:value-of select="ancestor::task/@Object"/>
                                       </td>
                                       <td>
                                            <xsl:value-of select="name(ancestor::conditions/..)"/>
                                       </td>
                                       <td>
                                            <xsl:value-of select="@value"/>
                                       </td>
                                  </tr>
                             </xsl:for-each>
                        </table>
                   </body>
              </html>
         </xsl:template>
    </xsl:stylesheet>

     

    Here is the output, when run against the attached XML export of a JOBP object.

    Workflows with EXECUTE OBJECT actions defined in pre- or post-conditions.

    WorkflowRunning #RowColumnObjectPre/PostObject to execute
    UC0.MAL.TEST2.JOBP212UC0.MAL.TEST.SCRIPreConPRECOND_NEST1_EXECUTEABLE1
    UC0.MAL.TEST2.JOBP212UC0.MAL.TEST.SCRIPreConPRECOND_NEST2_EXECUTEABLE1
    UC0.MAL.TEST2.JOBP212UC0.MAL.TEST.SCRIPreConPRECOND_NEST2_EXECUTEABLE2
    UC0.MAL.TEST2.JOBP212UC0.MAL.TEST.SCRIPreConPRECOND_NEST1_EXECUTEABLE2
    UC0.MAL.TEST2.JOBP212UC0.MAL.TEST.SCRIPreConPRECOND_FINAL_EXECUTEABLE1
    UC0.MAL.TEST2.JOBP212UC0.MAL.TEST.SCRIPostConEXECUTE_OBJECT.POSTCOND

    Enjoy!