Test Data Manager

Hi,

We are trying to create synthetic data in CA TDM where we have created 10 rows with specific layout.

Now, we want 2 files with 1 million record each. When we are repeating 10 rows for 100000 using Batch option; unable to get any result and it shows error.

Step followed:

1. Registered layout.

2. Created 10 rows with Ca functions.

3. Publish to file

4. Row = 100000

5. Publish to CSV.

6. Batch.

7. CMD and XML file generated, excuted CMD file.

8. Got error of failure.

Successfull run fro 1000 rows and generated 10000 rows but unable to get result fro more rows.

Please confirm, if we have any upper limit to repeat any row in CA TDM or any upper limit to generate sythetic data using CA TDM in single File.

Please suggest what process can be followed.

Thanks,

Richa Joshi

Hi,

    Can you confirm which version of CA Test Data Manager you are using and also if you see the same problem if you use the option to publish immediately or using the TDM Portal publish engine?

Regards

Keith

hi Keith,

I m using 4.4.0.23 version of CA TDM.

Not sure about other details you askedabout TDM Portal publish engine.

Thanks,

Richa Joshi

Hi,

      I have run a few tests and have been able to generate over 1 million records into a file with no issues using 4.4, On the publish screen do you enable the option "Enterprise Mode" , this switch will use the publish engine in the TDM portal instead of the Data Maker Publish engine. Are these the same settings as you used when you created the publish job?

Regards

Keith

hi,

I submitted in same way but unable to find job in submiited request selection.

seems some with my id.

Any suggestion for the same.

Thanks,

Richa Joshi

Hi,

    If you submit the publish job from within the TDM portal you can review the job by going to the screen in the screenshot above and selecting "Generators" from the drop down. if you create the publish job in datamaker the publish jobs do not appear on this TDM Portal screen instead open datamaker and select the option "View and Authorise jobs" from the Tools menu. this will show the status of the datmaker publish jobs.

Regards

Keith

Hi,

Thanks. Any estimate time how long does it take to create 1 Million rows?

Thanks Richa

Hi,

   It will depend on how complex the data generation is and also the performance / resources of the publish engine, on a small test environment with a fairly simple rule generation publish job i was able to create a CSV with one million rows in about 15 minutes. 

Regards

Keith

Hi Keith,

I have submitted job via generator in TDM portal itself.

Even after 1 h 30 min, It's still in running status. PFB

Thanks Richa

Richa,

The publish time depends on a lot of factors like your hardware specs, complexity of rules, number of columns etc so as long as there is no error, you should continue to wait and the file would be generated.

Also we may need to take a closer look at the rules you have used to see if they can be optimized.

Thanks,

Anil

Hi Richa,

Do you see any increase in rows in the files?  i.e. they are growing - right? 

Are you using a Trial License?

Thanks!
Les.

Hello,

Can you show us the error message, so We can get a better understanding of which type of error is occurring during the publish?

Great!  Now, what is inside of that file?

Cheers!
Les

Thats screen shot is of the error file itself.

check the name.

Richa,

Can you publish the same job directly through TDM Portal directly and see what happens? 

Thanks,

Anil

Hi,  

    The job output from the portal publish engine will by default be in this location on the portal server  "C:\ProgramData\CA\CA Test Data Manager Portal\Jobs"  can you cehck if there is a job folder in this location? 

Regards

Keith  

Hi Keith,

Job is still in running status under CA TDm Portal however I am unable to find path"C:\ProgramData\CA\CA Test Data Manager Portal\Jobs" in my machine.

Thanks,

Richa Joshi

Hi,

     The programdata folder is normally hidden with the default windows settings, Can you use the %programdata% variable i.e   %programdata%\CA\CA Test Data Manager Portal\

    This should take you to the root folder for the portal data, from this folder you can access jobs / logs etc

Regards

Keith

Hi Richa,

It might be useful for you to open a Support Case and get on a WebEx with us.  That way we can do some trouble shooting and get to the root of the issue.  During the WebEx we willlook at your logs, make sure you are running them in Debug mode, and get this resolved for you.

Cheers!

Les.

Xavier-Batllori-i-Ros is working on the support case. 

Hi Richa, been looking for performance issues related to this case and found a suggestion to run large data generation jobs in Portal instead of Datamaker and also set the following C:\Program Files\CA\CA Test Data Manager Portal\conf\application.properties parameters and restart TDM portal and test.

tdmweb.publish.batchCommit=true
tdmweb.publish.iterationsBeforeCommit=20000

batchCommit=true means that commits to the database are made in batches rather for each row

application.properties file and restart TDM portal and test.

tdmweb.publish.batchCommit=true
tdmweb.publish.iterationsBeforeCommit=20000

batchCommit=true means that commits to the database are made in batches rather for each row
iterationsBeforeCommit=20000 means that commit to the database is done by batches of 20000 rows.

However, being the output a CSV file we don't think it will affect the performance in this case.