IDM could be configured to connect to only 1 corporate user store, but in the course of processing, you may see LDAP referrals to other LDAP servers which may not be desired and could be causing performance lags.
Proof of an LDAP referral can be seen in the IDM log as below:
18:59:32,357 DEBUG [ims.llsdk.directory.jndi] extraProp:[java.naming.referral]=[follow]
or via any Wireshark network trace.
Active Directory in particular is well known for returning referrals with search results, often pointing to the subschema entry and site configuration data if a non-existent site or similar is defined in AD.
To override LDAP refferrals from happening:
Put an extra property in your directory XML as follows
(section should appear right after Managed Objects declarations):
<PropertyDict name="LDAP_CONNECTION_SETTINGS">
<Property name=" java.naming.referral">ignore</Property>
</PropertyDict>