Sameer_Chandragiri

Submitting multiple remote publish jobs to run concurrently (Data Finder)

Discussion created by Sameer_Chandragiri Employee on Mar 18, 2015

At times, owing to the requirement of running multiple publish jobs with varying durations, it becomes necessary and extremely helpful to be able to run these publish jobs concurrently via a Data Finder Remote Engine.

 

The pointers below show how Data Finder and the Remote Engine enable one to do so:

 

1.       A given Data Finder remote engine instance can listen to jobs of multiple threads in PARALLEL. By default it listens on THREAD1. But it can be configured to listen on comma separated thread list too such as THREAD1,THREAD2,THREAD3 in PARALLEL.

remote_engine_parallel_threads.jpg

 

2.       For a given remote engine instance the job processing of all jobs in a given thread is serial. So if

          a. THREAD 1 – job queue =  1001,1003,1005,1006

          b. THREAD 2 – job queue = 1002,1004

          Then job 1001 and 1002 will be executed in parallel. If they both finish around the same time – then job 1003 and 1004 will be executed in parallel, Then jobs 1005,1006 are executed in serial.

 

3.       Remote engine also understands a ‘trick’ thread name = ‘ANY’. So if Remote Engine is configured for three threads – THREAD1,THREAD2,THREAD3, then if following jobs are submitted from Data Finder with thread name ‘ANY’:

          a. ANY – 1001,1002,1003,1004,1005,1006

         Then 1001,1002,1003 will automatically be allocated to available free threads i.e. THREAD1, THREAD2 and THREAD3 and will be processed concurrently. If they all finish around the same time, then 1004,1005,1006 will be

          processed in parallel.

          Note: This particular feature is available from Remote Engine (DMBatch.exe) version 1.82.0.0 onwards.

Outcomes