Automic Workload Automation

Expand all | Collapse all

Setting file mode (permissions) of a file after a file transfer

Michael A. Lowry

Michael A. LowrySep 20, 2013 05:55 AM

  • 1.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 12, 2013 11:08 AM
    I’m participating in a migration of some file transfer jobs from Stonebranch UDM to UC4. UDM allows specifying the mode (file permissions) of the target file. As far as I can tell, UC4 file transfer jobs do not provide this capability, so I came up with a work-around. The user stores the desired file mode in a variable in the JOBF object. He then adds a bit of code to the process and post-process tabs. These UC4 scripting commands read the agent name, login object name, and target file name from the predefined variables associated with the JOBF instance. It also reads the file mode specified by the user. It then kicks off a separate job that uses these pieces of information to set the file mode of the target file. This work-around works pretty well, and I’d be glad to share it here if anyone is interested. Right now it works only on UNIX, but it could conceivably be expanded to work on Windows too.


  • 2.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 12, 2013 11:29 AM
    Why not just use the JOBF functionality to do that and put 644 or 777 or whatever is required in the attributes line after the target file name I have also used "touch" to do this via the command line as well as chmod Stephen


  • 3.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 13, 2013 05:34 AM
    Support has created a 'Best Practice' article providing a possible solution for Unix/Linux files. KBE0010607 An excerpt from the article: Best practice is, changing owner group and / or permissions after the Filetransfer, if necessary. The easiest way is, to use the post-process tab of the file transfer object. In the "Post-Process" tab a so called "Prep_Process UNIXCMD" job is submitted, which runs a "chgrp" and "chgmod" command. -Jenn


  • 4.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 16, 2013 04:10 AM
    Why not just use the JOBF functionality to do that and put 644 or 777 or whatever is required in the attributes line after the target file name
    As far as I know, the file attributes field does not work this way. See the section of of the documentation entitled “UNIX Agent - FileTransfer Support.”


  • 5.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 16, 2013 04:59 AM
    Support has created a 'Best Practice' article providing a possible solution for Unix/Linux files. KBE0010607 An excerpt from the article: Best practice is, changing owner group and / or permissions after the Filetransfer, if necessary. The easiest way is, to use the post-process tab of the file transfer object. In the "Post-Process" tab a so called "Prep_Process UNIXCMD" job is submitted, which runs a "chgrp" and "chgmod" command.
    Thanks for the link to the KB article. The article omits many of the implementation details, but I believe I was able to get the general idea. Using PREP_PROCESS with the EventType EVENT.UNIXCMD is an interesting approach. I suppose the advantage of doing it this way is that no separate job needs to be created. However, there is essentially a separate job being run behind the scenes anyway. Moreover, the PREP_PROCESS approach is better suited to situations where additional processing of command output is required. When it comes to setting the permissions on a file, this is not the case. A disadvantage of this approach is that it also requires setting redundant information. Specifically, the destination agent, destination file, and destination login. These pieces of information are stored already in the FileTransfer tab. Duplicating the data in another location requires additional work and increases the likelihood of errors due to inconsistencies between the data in the two locations. I believe my solution is more flexible and easier to implement. I’ll post the details here. First, the two include files: FT.SET_FILE_MODE_PROCESS & FT.SET_FILE_MODE_POSTPROCESS: FT.SET_FILE_MODE_PROCESS ! Include this INC file in the process tab of your JOBF using this line: ! :INC FT.SET_FILE_MODE_PROCESS :IF &FT_DST_FILE_MODE# <> "" : PRINT "File mode has been specified. Setting variables..." : PRINT "Runtime warnings such as this one can be ignored:" : PRINT "U0020239 Runtime warning in object ... Could not find parent task of type JOBP or JSCH." : PSET &FT_DST_AGENT# = &$DST_AGENT# : PSET &FT_DST_LOGIN# = &$DST_LOGIN# : PSET &FT_DST_FILE# = &$DST_FILE# : PRINT "Destination host name : &FT_DST_AGENT#" : PRINT "Destination login     : &FT_DST_LOGIN#" : PRINT "Destination file name : &FT_DST_FILE#" : PRINT "Destination file mode : &FT_DST_FILE_MODE#" :ENDIF ! The job that sets the file mode is activated from the post-process tab FT.SET_FILE_MODE_POSTPROCESS ! Include this INC file in the post-process tab of your JOBF using this line: ! :INC FT.SET_FILE_MODE_POSTPROCESS :IF &FT_DST_FILE_MODE# <> "" : PRINT "Starting job to set file mode" :  SET &ACTOBJ# = ACTIVATE_UC_OBJECT(FT.SET_FILE_MODE,,,,,PASS_VALUES,UC0) :  IF &ACTOBJ#  = "0" :   PRINT "Error activating the job FT.SET_FILE_MODE" :   SET &RC# = 1 :  ENDIF : WHILE SYS_STATE_ACTIVE('FT.SET_FILE_MODE')= 'Y' :  WAIT 1 : ENDWHILE :  SET &STATUS# = GET_UC_OBJECT_STATUS(JOBS,&ACTOBJ#) :  IF &STATUS# <> 1900 :   PRINT "Job FT.SET_FILE_MODE &ACTOBJ# Finished with error code &STATUS#." :   SET &RC# = &STATUS# :  ENDIF :ENDIF Next, the short UNIX job that sets the file mode, FT.SET_FILE_MODE: PreProcess Tab: !Print values of variables passed to job by original FT job. :PRINT "Destination host name : &FT_DST_AGENT#" :PRINT "Destination login     : &FT_DST_LOGIN#" :PRINT "Destination file name : &FT_DST_FILE#" :PRINT "Destination file mode : &FT_DST_FILE_MODE#" !Set HOST and LOGIN based on details of original file transfer job. :PUT_ATT HOST = "&FT_DST_AGENT#" :PUT_ATT LOGIN = "&FT_DST_LOGIN#" Process Tab: echo "*** FT.SET_FILE_MODE ***" echo "Set file mode based on value of &&FT_DST_FILE_MODE# object variable." echo "Destination host name : &$AGENT#" echo "Destination login     : &$LOGIN#" echo "Destination file name : &FT_DST_FILE#" echo "Destination file mode : &FT_DST_FILE_MODE#" echo "File before chmod:" ls -lha &FT_DST_FILE# echo "Running \"chmod &FT_DST_FILE_MODE# &FT_DST_FILE#\"" chmod &FT_DST_FILE_MODE# &FT_DST_FILE# echo "File after chmod:" ls -lha &FT_DST_FILE# With these objects defined, make a new file transfer job (JOBF), and fill in these details: FileTransfer Tab: Source and destination host name, file name and login object name. Variables & Prompts Tab: Create a variable named &FT_DST_FILE_MODE#, and set the value to the desired octal file mode, e.g., 644. Process Tab: :INC FT.SET_FILE_MODE_PROCESS PostProcess Tab: :INC FT.SET_FILE_MODE_POSTPROCESS Note: The separate instructions in the process tab are necessary because the predefined variables &$AGENT# and &$LOGIN# contain different values when these values are read in the post-process tab. This is probably a bug. The advantage of my solution is that you need to define just one variable in the Variables & Prompts tab of the file transfer job. Limitations: • The solution is specific to UNIX; it has not been designed to work on other OSes. • It supports only ordinary octal or symbolic file modes as one might pass as the first argument to chmod. It does not support ACLs. • It does not perform validation of the specified file mode. • It has limited error handling, and currently does not even check whether the file transfer was successful before running the chmod job. • It has not been tested with wildcards (*) in the file name; using this solution with file transfers that use wildcards may has unintended results, such as setting the mode of other files in the target location.


  • 6.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 16, 2013 05:47 AM
      |   view attached
    I have attached an XML export of the objects described above.

    Attachment(s)

    xml
    FT.SET_FILE_MODE.xml   12 KB 1 version


  • 7.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 16, 2013 12:34 PM
      |   view attached
    I have updated the solution to support setting file modes on Windows too, using the icacls command. The queue of the job that sets the file mode is also now set dynamically, based on the &$QUEUE# attribute of the original JOBF. The next step would be to add support for file transfers that use wildcards. Anyone want to take a stab at this?

    Attachment(s)

    xml
    FT.SET_FILE_MODE_2.xml   22 KB 1 version


  • 8.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 18, 2013 04:18 AM
    Untested but here you go. Basically encapsulating your code in a PREP_PROCESS which grabs filenames
    :SET &HND# = :PREP_PROCESS_REPORT(,,"REP","*U0011133*","COL=DELIMITER","DELIMITER=*'*") :PROCESS &HND# : PSET &FT_DST_FILE# = GET_PROCESS_LINE(&HND#,8) : PRINT "Starting job to set file mode" : SET &ACTOBJ# = ACTIVATE_UC_OBJECT(FT.SET_FILE_MODE,,,,,PASS_VALUES,UC0) : IF &ACTOBJ# = "0" :   PRINT "Error activating the job FT.SET_FILE_MODE" :   SET &RC# = 1 : ENDIF : WHILE SYS_STATE_ACTIVE('FT.SET_FILE_MODE')= 'Y' :   WAIT 1 : ENDWHILE : SET &STATUS# = GET_UC_OBJECT_STATUS(JOBS,&ACTOBJ#) : IF &STATUS# <> 1900 :   PRINT "Job FT.SET_FILE_MODE &ACTOBJ# Finished with error code &STATUS#." :   SET &RC# = &STATUS# : ENDIF :ENDPROCESS :CLOSE_PROCESS &HND#
    P.S. you could use a WAIT as the 2nd parameter of the ACTIVATE_UC_OBJECT instead of looping a while statement


  • 9.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 18, 2013 08:20 AM
    Untested but here you go. Basically encapsulating your code in a PREP_PROCESS which grabs filenames
    ...

    P.S. you could use a WAIT as the 2nd parameter of the ACTIVATE_UC_OBJECT instead of looping a while statement
    Good idea — thanks. I had forgotten about the WAIT option. I believe your change would work with wildcards.

    Also, ideally, I would prefer to avoid launching a separate job to change the mode of each file. I thought about making a string array to hold the list of files, and then passing this variable to the job that sets the mode. However, an array must be created with a specific size, and I don’t know ahead of time how many files will match the pattern. Nor does there seem to be an attribute containing the number of files that were transferred. KB article KB0011630 claims to provide a solution to this, but it is wrong. The RECORDS option of GET_STATISTIC_DETAIL does not work as explained in the article. It always returns 0 for binary transfers, and the number of lines for text transfers.

    I suppose the PostProcess could look in the report and count the lines corresponding to successful file transfers, and then do another PROCESS loop to fill the array with the file names. Or I could use CREATE_OBJECT to make a temporary VARA to store the file list. With VARAs, you do not need to know the size ahead of time.

    In any event, I cannot make progress on this until I fix an error I am currently seeing whenever I try to use wildcards:
    U0063016 FT '3225582': The file '/home/uc4a/testfile*.txt' does not exist.
    The file definitely exists. If you know what might be causing this problem, I would be glad to learn of your ideas.


  • 10.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 18, 2013 08:28 AM


    In any event, I cannot make progress on this until I fix an error I am currently seeing whenever I try to use wildcards:
    U0063016 FT '3225582': The file '/home/uc4a/testfile*.txt' does not exist.
    The file definitely exists. If you know what might be causing this problem, I would be glad to learn of your ideas.
    I found a work-around for the wildcard problem above. It started working once I changed to a directory that is world-readable. It seems like the agent trying to list the files matching the pattern using a user other than the one defined in the login object. The user defined in the login object obviously has read access because it works in the non-world-readable directory as long as I don’t use wildcards. Enumeration of file lists during wildcard expansion should obviously be performed by the user defined in the login object. This seems like a bug.


  • 11.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 18, 2013 09:34 AM
      |   view attached
    I have updated my solution to use a temporary VARA to store the file list in cases where a wildcard is used. I also added better error checking. The job that sets the file mode now will not launch if the file transfer was unsuccessful. The example objects are attached. It seems to work pretty well. I still believe it would be preferable to use a string array. What do you think?

    Attachment(s)

    xml
    FT.SET_FILE_MODE_3.xml   25 KB 1 version


  • 12.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 18, 2013 10:29 PM
    Perhaps what you can do is pass over the RUNID of the FT job and have the filemod job PREP_PROCESS_REPORT the FT job's report using the RUNID as an identifier. This way you can do a loop within your Process tab for multiple chmod/icalcs lines That FT wildcard issue does look like the bug based on the way you are describing it. Usually i get that error if i put a wildcard without marking the wildcard checkbox in the FT job but i guess this is not what happened in your case.


  • 13.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 19, 2013 03:23 AM
    Perhaps what you can do is pass over the RUNID of the FT job and have the filemod job PREP_PROCESS_REPORT the FT job's report using the RUNID as an identifier. This way you can do a loop within your Process tab for multiple chmod/icalcs lines.
    Another good idea. This approach would eliminate the need for the VARA. It would also allow a more straightforward structure to the code. Right now I'm relying on many levels of nested IF statements to exit in case of an error, but normally I would use STOP for these cases. STOP is not allowed in the post process. (If you can suggest a better approach to error handling in post process, I would be glad to read about it.) In any event, it occurred to me that exporting the list of successfully transferred files to a temporary VARA might have more general utility.
    That FT wildcard issue does look like the bug based on the way you are describing it. Usually i get that error if i put a wildcard without marking the wildcard checkbox in the FT job but i guess this is not what happened in your case.
    Yes, the wildcard checkbox was definitely checked. I’ll perform some more tests today and report the bug if that’s what it turns out to be.


  • 14.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 20, 2013 03:52 AM
    I’ll perform some more tests today and report the bug if that’s what it turns out to be.
    Changing the group of the directory containing the files appears to have fixed the problem. The group of the directory was a group to which the user did not belong. I’m not sure why this would prevent enumerating a list of matching files though. Even when the group was set to that other user, I could list files from the command line using ls. Odd. If I have time, I’ll investigate this more later.


  • 15.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 20, 2013 04:11 AM
    Perhaps what you can do is pass over the RUNID of the FT job and have the filemod job PREP_PROCESS_REPORT the FT job's report using the RUNID as an identifier. This way you can do a loop within your Process tab for multiple chmod/icalcs lines.
    I updated my solution based on your suggestion. If wildcards were used in the FT job, the SET_FILE_MODE job now sets the mode of each file that was successfully transferred. It does this by examining the job log of the FT job. In cases where wildcards were not used, the job just uses the destination file name from the FT job. This solution is much simpler and smaller than the previous version. It’s also much easier to understand and debug. The objects are attached in an XML file. Attachment removed due to typo/bug. Corrected file attached in later comment. Writing the list of successfully transferred files to a temporary VARA  might still be useful for more complicated operations; but for something as simple as setting the file mode, I felt that it added unnecessary complexity. For your information, it’s not necessary to pass the original FT job’s RUNID as a parameter. The SET_FILE_MODE job can look up this piece of information using GET_PARENT_NR. Thanks again for your feedback and suggestions.


  • 16.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 20, 2013 04:19 AM
    Here are some ideas for potential future enhancements:
    • Add support for UNIX ACLs via setfacl command
    • Add support for z/OS via RACF commands
    • Add support for changing the owner and group of the file(s)


  • 17.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 20, 2013 05:55 AM
    --


  • 18.  Setting file mode (permissions) of a file after a file transfer

    Posted Sep 20, 2013 07:25 AM
      |   view attached
    Sorry about that. I had misplaced an :ENDSWITCH statement. I also placed the portion of the SET_FILE_MODE jobs that actually does the work inside functions to reduce code redundancy. The corrected & updated version is attached.

    Attachment(s)

    xml
    UC4.FT.SET_FILE_MODE.xml   18 KB 1 version


  • 19.  Setting file mode (permissions) of a file after a file transfer

    Posted Oct 09, 2013 08:28 AM
      |   view attached
    I have updated the solution with the following improvements:
    • It now usesIbrahim Ghouth’s GET_SCRIPT_VAR method for getting script variable values when the variable may or may not be set. This is an improvement over the method I had been using, based on GET_PUBLISHED_VALUE.
    • I also fixed a small bug related to the way variable names are echoed. It turns out that although the &&VAR_NAME# trick for echoing variable names (and not their values) to the log works in SCRIs and the Activation phase of OS jobs, it does not work in the OS-targeted statements of OS jobs. This probably has to do with the way UC4 variables are resolved during Generation. Anyway, I came up with OS-specific work-arounds.
    Note: there is a bug (PRB0306261) that causes some job attributes to contain incorrect information when read in the Post Process of a JOBF. Once this bug has been fixed, the SET_FILE_MODE.JOBF_PROCESS JOBI object will no longer be necessary, and it will be possible to incorporate its instructions into SET_FILE_MODE.JOBF_POSTPROCESS.

    Attachment(s)