marco_tizzoni_automic

Download files from a SFTP in parallel

Discussion created by marco_tizzoni_automic on Jul 5, 2017
Latest reply on Aug 29, 2017 by marco_tizzoni_automic
Hello,
we have to download a couple of thousands files from a remote SFTP server. We would like to parallelize the download on multiple servers to speed up the process.
I thought a possible implementation could be:
  1. execute a ls command using the RA_FTP agent 
  2. parse the ls output to get the list of files in a variable 
  3. for each file trigger a file download
I could not get past the first point because the ls output is mixed with SSH debug information and parsing is hard and too easy too break. 
Any hints on how to overcome the issue or other possible implementations?

Best,
Marco

Outcomes