MarcoTizzoni604411

Run file processing in parallel

Discussion created by MarcoTizzoni604411 on Jul 11, 2017
Latest reply on Jul 12, 2017 by joel_wiesmann_automic
Hi everybody,
I have a shared folder where a workflow downloads a number of files (thousands). I have to process those files and I want to do it in parallel on a number of different systems so we can scale out easily in case the load increases.

My idea was to list all files to process in a variable, read the variable in a for loop and spawn a new instance of the processing workflow for each file letting AWA to deal with the complexity of parallelism (how many instances per server, where to run the next instance, putting on hole executions if no free slots are available, ...) , however I find it rather difficult because the for-each workflow does not support parallel execution and, LTM, a custom implementation is quite complex.

What would be the best approach to implement this?

Any comments appreciated.

Thanks in advance,
Marco

Outcomes