Symantec IGA

Expand all | Collapse all
  • 1.  Bulk Loader

    Posted Jun 29, 2018 07:56 AM

    Hey

    can we make bulk loader task synchronous,

    like after completion of create user task for 1st entry in csv
    next entry in csv , create user task must execute.

     

     

    because we are using custom userid generation where in this case it is failing

     

    ex:

    1)test,case

    2)test,case

     

    expected outpt:

    userid=:tcase

    userid: tcase1

     

    output:

    userid=tcase

    for next user the task is failing saying entry already exists

    here the task is generating asynchronus

    can we make it synchrounous for this usecase?

     

     

    i have user UI as policy type for userid generation



  • 2.  Re: Bulk Loader

    Posted Jul 02, 2018 07:57 AM

    IM will attempt to process all of the tasks fed into the bulk loader concurrently. You would need to try having a something like a PX Policy call out to an external service/application to serialize and generate the IDs as IM will not serialize. I do not have any example.



  • 3.  Re: Bulk Loader

    Posted Jul 02, 2018 08:05 AM

    PX Policy call out to an external service/application to serialize and generate the IDs

    i didnt get your point can u please tell me how to do
    or is there any other solution for this because this is the main usecase for us,

    how actually others do for this usecase?



  • 4.  Re: Bulk Loader

    Posted Jul 02, 2018 08:08 AM

    As I already mentioned, I do not have any example. Maybe others in the community can assist further.



  • 5.  Re: Bulk Loader

    Broadcom Employee
    Posted Jul 02, 2018 05:35 PM

    Your problem is that both create requests are getting picked up at the same time by IDM. I guess your PX checks whether the proposed uid <first-initial><last-name> already exists or not. If it doesn't exist, you use it. If it does exist, you add a counter to the end.

    So when both requests are picked up by IDM, the proposed uid <first-initial><last-name> does not yet exist. So your PX uses that uid for both. Now the actual attempt to create each user is not simultaneous. One happens first and is successful. The second now fails because the proposed unique id now does exist.

     

    A simple solution is to use the -b 1 option when submitting the bulk load. This submits each line as a separate request. So assuming IDM is not completely overloaded, the first request should have gone through before the second one even starts.

     

    Another option is to have a step in PX also checks whether the proposed uid exists in a runtime DB. If not, use it and reserve it by writing it to the runtime DB. If it does exist, increment by one and continue. On the off-chance that two PX's attempt to write to DB at the same time (and the key is unique in the DB) then you can deal with this failure and you increment the proposed uid by one and continue.



  • 6.  Re: Bulk Loader

    Posted Jul 03, 2018 02:45 AM

    Thanks pearse.

    1) as you mentioned that -b 1 option,where we can specify this

    as i was doing from userconsole(under systems tab)

    2) runtime db:i didnt get your point (you are saying to create a custom table and store in it through px)like as i was new to this 

    can you please describe me,how can we achieve this,through runtime db.



  • 7.  Re: Bulk Loader

    Broadcom Employee
    Posted Jul 05, 2018 05:08 AM

    Sorry for delayed reply. When using the -b 1 flag, I was referring to the bulk load client.

    I would recommend using this instead of the bulk loader in the UI. Once you get it up, it's much quicker for submitting your CSV files.

     

    I think you've already answered your second question. I saw your subsequent post where you were testing an SQL Insert. That was exactly what I meant. I.e. create a new DB and a new datasource in JBoss/Wildfly (use an existing datasource as a template). Then you can store any useful data and run whatever SQL queries/procedures that you want.

     

    Pearse



  • 8.  Re: Bulk Loader

    Posted Jul 05, 2018 05:59 AM

    can we do using BLTH task like creating one static list variable in class and adding user id to list.

    can we check our java blth code in standalone whether the code which we have did is working or not?



  • 9.  Re: Bulk Loader

    Posted Jul 05, 2018 09:47 AM

    hi pearse

    the execution is behaving same like earlier one

    i created a runtime db i was checking whether userid is there or not, again its failing for 2nd record? 



  • 10.  Re: Bulk Loader

    Broadcom Employee
    Posted Jul 05, 2018 09:58 AM

    Is the field in the DB where you store the uid a unique field? If so, then the second attempt to insert the uid should fail. Then at that stage you need to build policy logic to handle that failure - e.g. increment the uid by adding a digit to the end of it and repeating the policy.



  • 11.  Re: Bulk Loader

    Posted Jul 06, 2018 04:00 AM

    Hi   PEARSE Kennedy

     

     

    For the failed one i created a px on task failed and wrote some logic to generate unique one ,from that again iam calling the first policy for userid insert

    its failed,when i did this 

    the following is the error i was facing

     

     

    Task Failed:CreateBulkUser - Failed to execute action 'callagain'(calling the first policy at this action). Error category 'System Failure' with response 'Fail Event'. 


    POLICYXPRESS ERROR MESSAGE: Failed finding policy set User Values for Unique UserID in the list for the current step
    Failed performing action type action.name.change.process.flow

     

    as you said in the previous discussion

    you need to build policy logic to handle that failure,how can we achieve this case



  • 12.  Re: Bulk Loader
    Best Answer

    Broadcom Employee
    Posted Jul 06, 2018 05:10 AM

    Here's an outline example of what I was thinking of. Note, I haven't actually tested this, so you may need to modify

    It assumes all Policies are of type "UI" and fire on "submission", e.g. for requests from the bulk load client. 

     

    P1 - Set variables
    Data:
    First_Name (user attribute)
    Last_Name (user attribute)
    Counter (constant = 0)
    Action rule
    Set variable: Insert_Success_var=false
    Set variable: First_Name_var={First_Name}
    Set variable: Last_Name_var={Last_Name}
    Set variable: Counter_var={Counter}

     

    P2 - Check if UID candidate is unique
    Data:
    First_Name (variable = First_Name_var)
    Last_Name (variable = Last_Name_var)
    Counter (variable = Counter_var)
    Counter_Plus_One (math - increment Counter by 1)
    Uid_Candidate (constant = {First_Name}{Last_Name}{Counter})
    Is_Uid_Candidate_Unique (attribute - uniqueness checker)
    Is_Uid_Candidate_In_Runtime_DB (SQL query)
    Action Rule
    Condition: Uid candididate is unique and not in runtime DB
    Action: SQL - run procedure to insert into DB
        Set Variable: Insert_Success_var=true (this must come after the SQL query)
        Set Variable Uid_var = {Uid_Candidate}
    Condition: Uid candidate not unique and/or in runtime DB
    Action: Set variable: Counter_var={Counter_Plus_One}
        Redo policy

     


    P3 - handle SQL Insert failures
    Data
    Counter (variable = Counter_var)
    Counter_Plus_One (math - increment Counter by 1)
    Insert_Success (variable = Insert_Success_var)
    Entry Rule
    Insert_Success = false
    Action Rule
    Action: Set variable: Counter_var={Counter_Plus_One}
        Goto previous policy

     


    P4 - Set user attributes
    Data
    Uid (variable = Uid_var)
    Action rule
    Set User Attribute - %USER_ID% = {Uid}

     

    Now the trick is in Policy 2 to make sure that the entire task doesn't fail if you fail to write the UID Candidate into your runtime DB. To do this, in Policy 2, go to the "advanced" tab. Change the "validation" and "environment" errors to "fail policy" (I think the default is "fail task"). This means that the policy will fail but the task does not, so you will proceed to the next policy

     

    So if the UID Candidate you choose in P2 is unique, you then write it to the the DB. If this succeeds, the next action is to update the "Insert_Success_var" variable to "true". If it fails (e.g. because another request has just written the same value to the DB), then the policy fails before it gets the chance to update the "Insert_Success_var" - i.e. it is still set to "false" from P1

     

    So when P3 runs, it checks the "Insert_Success_var" value. If it is "true", P3 does nothing (entry rule condition not met). If it is false, P3 increments the counter and returns to previous policy to try again with a new UID Candidate.

     

    Make sure to write the log for each action in each policy to troubleshoot.

     

    Pearse



  • 13.  Re: Bulk Loader

    Posted Jul 06, 2018 07:37 AM

    Thanks PEARSE Kennedy.

     



  • 14.  Re: Bulk Loader

    Posted Jan 08, 2019 10:00 PM

    Same problem we were also facing and resolved by externalising code. I mean before CSV submitted to IDM uniqueness checked and UID generated. 

     

    We have maintained uniqueness by checking uniqueness against CORP store, and maintaining an array list at runtime. 

     

    Thanks! Alok