Best practices for file transfer polling?

Discussion created by Bill_Corcoran_1199 on Jul 21, 2014
Latest reply on Jul 25, 2014 by Jean-FrancoisAnctil606475

So I was recently tasked with fixing up a flow for a file transfer process.  The first step in the flow is a free-form FTP job that downloads files from an FTP server not running a UC4 agent.  Then it checks if a file exists on the intermediary FTP server (via UC4 condition) which is running an agent with FTP RA before ultimately transferring it offsite.  If there are no files, it cancels the flow.  After making some corrections, testing and approving the flow to be migrated to production, I learn they want to check for files to transfer every 5 minutes.

The way it is setup right now, the whole flow is on a schedule to run every 5 minutes.  Most of the time there are no files, so it just cancels itself.  But that's 288 requests a day, with multiple cancelled jobs per requested flow.  And that's just for one file transfer process.  I feel like this pollutes our history, and seems like there must be a more elegant way to do this.  Below are a couple of my ideas, but surely this is a common scenario, so I was wondering what the general consensus on the best way to do this was.

- A "monitoring" job that runs continuously, regularly checks for files on the remote FTP server somehow - either putting an agent on the source FTP server or integrating FTP commands into UC4 logic somehow?

- Offloading the initial part of the flow to a script called regularly from cron and if files are retrieved requesting the flow from the host OS?

It just feels kind of kludgy and like I must be missing something obvious but maybe it has to be that way because of requirements/limitations.  Solutions/recommendations?