Test Data Manager

  • Private Community

  • 1.  InPlace masking Using Fast data masker and a lookup table

    Posted Jun 14, 2018 01:39 AM

    Hi , 

     

    I am trying to perform in-place masking using Fast data masker. The data base that i am trying to mask is in SQL and the lookup table is in oracle. When i run the masking script , it is trying to load all the data in the lookup table first which is taking a lot of time.For masking 5000 rows that too only column it is taking 1 hour.

     

    Regards

    Velentina



  • 2.  Re: InPlace masking Using Fast data masker and a lookup table

    Posted Jun 14, 2018 02:48 AM

    Hi,

     

    What are you trying to achieve? It is a known limitation with FDM that if you want to maintain consistency with a 100% guarantee and hence are using XREF method, it will be time consuming because of the additional table update and lookup involved.

     

    However, if you 100% is not important then you can use other functions that can server the purpose. Once we know what is it that you are actually trying to achieve, we can guide you further.

     

    Regards,

    Aditi



  • 3.  Re: InPlace masking Using Fast data masker and a lookup table

    Posted Jun 14, 2018 03:04 AM

    Hi Kevin,

     

    Thanks for the information. We are trying achieve consistent masking across databases with one single lookup table as a source of truth 

    for masking and yes we need 100% guarantee. Our data are in IMS, DB2 , SQL, ORACLE.

    Our main concern is that , in the start of the masking uisng FDM , the time to load the lookup table is consuming a lot of time.

    Our lookup table has 2 million records and it was taking 1 hour aprox to load the data and start the masking.

    Is there a way to minimise this time?

     

    Regards

    Velentina



  • 4.  Re: InPlace masking Using Fast data masker and a lookup table
    Best Answer

    Posted Jun 14, 2018 06:43 AM

    Hi,

     

    Please ensure the following activities are performed before masking.

     

    Disable Triggers 

    Disable Index 

    Rebuild Index 

    Disable Constraints 

     

    You may also need to give FDM more memory. The following link will be helpful. 

     

    https://comm.support.ca.com/kb/fdm-outofmemory-error/kb000005241 

     

    You can also set the Max Memory (MB) in the FDM GUI.( you can find it in the bottom of the page)The number you enter in that depends on how many rows you have. It should always be in line or more. 

     

    For huge amounts of data, you should split the masking using where clauses into 3 or 4 and re-try. Pick a row that has unique values in them and put the where clauses using that. 

     

    Fire multiple jobs one after the other. I do not think you can run that in parallel. 

     

    Regards,

    Aditi