Critical snapshot filtering fix for large scale environments.

Discussion created by Chris_Thomas Employee on Sep 12, 2011
Latest reply on Jun 3, 2013 by Chris_Thomas

Within your version of IM prior to SP8, it’s simply impossible to leverage a simple user filter. When your environment starts to scale arbitrarily large in the range of tens of thousands of Global Users, with even more endpoint accounts, the snapshot process starts to become very cumbersome. Additionally, even if you’re successful with capturing a large snapshot, which encompasses all of your population, the data volume may simply be too much for the OOTB report to consume. If after implementing additional indexing on the database, you’re then able to produce a report, it will likely have thousands of pages and still isn’t “human consumable”. Reports should be concise and specific to fit the criteria of the audience, which they’re intended for. These reports should be easily shared to the customer’s IT management team and not only visible by the IM System Manager who created them. No single consumer or manager would specifically request one massive report, in fact conversely they would probably request a report that has a specific subset of the information they require. It’s important to narrow down your business requirements and craft the snapshot process to match those requirements and produce more meaningful reports for your users’. Without the fixes from SP7, it’s nearly impossible to effectively implement this for any account based reports, which can result in unacceptable snapshot collection times, failed snapshots, failed reports etc. It’s important that any snapshot process be implemented during off-peak periods, preferably on an underutilized cluster node. If the snapshot takes days to complete, this will affect end-user performance and degrade your system. If possible snapshots should be staggered and set on the recurrence scheduler to run during non-business hours.


User accounts filter was not working in r12.5 prior to SP8…

Here's the background on the problem:
When attempting to leverage filtering, which worked within r12, we're unable to filter the user accounts snapshot. When user accounts snapshot is filtered based on user accounts object for %USER_ID% equals to "a*", it doesn't limit collected data simply to the corporate / global user id's who "start with a" and their accounts'.

This SP8 fix will improve the performance of both the snapshot export and also the report generation on whatever reports the accounts are used.

This will help the user to customize their snapshot export to minimize the data to export thus makes the performance better and when they use this in the reports also, the report data will be to the point of what the user wanted to see rather than exporting 1 million users to view the set of users.

This issue was introduced when the snapshot export module was refactored. In r12, the user accounts export took care of exporting the global users and their associated accounts. In R12.5 it was split into two XML elements one for just exporting the global users and the accounts relationships, and the second one to export the accounts residing in the endpoints.

The problem was the filter that was designed to export the global users was not added to the endpoint accounts export section.

A new filter set is added for exporting the accounts which should be working closely with the global user filter defined in the "useraccounts" export element. For ex. for the user accounts, the global user filter will be defined like this:

<export object="useraccounts">

<where attr="%USER_ID%" satisfy="ANY">
<value op="EQUALS">corp*</value>

To obtain the accounts for the user(s) "corp*" the "endpoint" export element also should define the same as follows:

<export object="Endpoint">
<where attr="%USER_ID%" satisfy="ANY">
<!--op attribute in value expression has following allowed values -->
<!-- 1.EQUALS (Default) -->
<value op="EQUALS">corp*</value>

[color=#f60707]After applying SP8 or higher, please refer to the sample "UserAccountsReportSnapshot.xml" attached to this thread.[color]

Once the account information is defined, the account information will be filtered.

[u]Note: This functionality can be used in the following OOTB snapshot exports:
[*]Account Details
[*]Endpoint Accounts
[*]Non-Standard Accounts
[*]Non-Standard Accounts Trend
[*]Orphan Accounts
[*]User Accounts
[*]User Entitlements
[list]Account Details

Implement SP8 and then please complete the following steps to take advantage of the fix.
1) Setup corp = prov or corp != prov IME and acquire, explore & correlate at least one endpoint.
2) Make sure one or more accounts are exception and orphan accounts. Refer the provisioning administration guide for how to setup the accounts for exception and the orphan.
3) Use the sample "UserAccountsReportSnapshot.xml" attached in the SRF. Note: Do not just copy and test the sample. Make sure the filters are appropriate for your setup.
4) Replace the UserAccountsReportSnapshot.xml with the sample in <iam-ear>\config\com\netegrity\config\imrexport\sample folder.
5) In the IM user console "Reports->Snapshot Tasks->Capture Snapshot Data and select the "User Accounts Report Snapshot". Make sure you have setup the snapshot database connection prior to this. Refer IM administration guide's Reporting section.
6) Run the task. Wait for this to complete.
7) Associate the snapshot to the "User Accounts Report" task. Refer IM administration Guide "Reporting" section if you need to know how to.
8) From "Reports->Reporting Tasks->Request a report->User Accounts Report, execute the task.
9) Using "Reports->Reporting Tasks->View My Reports" view the report you ran.
10) The report should show only the accounts you exported.

Additionally, please review this thread on more strategies to cut down on your snapshot size.

Please post any questions or concerns.
Thank you.

Chris Thomas
CA Technologies
Principal Support Engineer
Identity Manager Reporting Expert
Tel: +1-631-342-4360