CA SSO : Policy Server trying to SEARCH over a CLOSE_WAIT Connection.

Idea created by Hubert Dennis Employee on Apr 8, 2016
    Under review
    Score9
    • ranebshekhar
    • Kaladhar.Brahmanapally
    • malsi07
    • SamatBoA
    • Hubert Dennis
    • brian.w.jones
    • AnandKaturi
    • michael.protulipac
    • rminnj

    In continuation of the following Tech note : CA SSO : Policy Server VS 3rd party components closing Idle Connection. & If needed I could share the Support Case number internally (same questions have been posted on the support case).

     

     

    We opted OPTION-2 i.e. LDAP Closing the Connection before Firewall and LDAP sending a FIN to Policy Server.

     

     

    What we identified was the following.

     

     

    1) LDAP notifies Policy Server it is closing the connection. LDAP sent a packet (actually a formal LDAP response) to the client (Policy Server) indicating it will be terminating the connection.

     

     

    2) Policy Server then ACK's this message. LDAP sends FIN, and Policy Server returns a ACK... We would have expected Policy Server to send a FIN and ACK. The concern here is that that the socket remains in a CLOSE_WAIT on the client (Policy Server) side - likely a direct result of client not sending a FIN/ACK (the policy server is still hanging onto the connection and hasn't given the OS the ok to tear it down).

     

     

    3) When Policy Server initiates the next SEARCH request (e.g. for a IsAuth call) it tries to reuse the same CLOSE_WAIT connection first. Policy Server see's that connection is in CLOSE_WAIT, hence issues a CLOSE PENDING on its side (believe thats what tears down the connection from Policy Server Side). Then tries a rebind using a new connection.

     

     

    [SmDsLdapConnMgr.cpp:1190][LogMessage:ERROR:[sm-Ldap-02230] Error# '81' during search: 'error: Can't contact LDAP server' Search Query = '(cn=AAAAAA)']

    [SmDsLdapConnMgr.cpp:1201][CSmDsLdapConn::SearchExts][LDAP search of (cn=AAAAAA) took 0 seconds and 8237 microseconds]

    [SmDsLdapFunctionImpl.cpp:3155][CSmDsLdapProvider::SearchExts][Ldap Search failed, ErrorMsg is Can't contact LDAP server]

     

     

    The Enhancement Request is to look at the design and see if there are opportunities for improving this.

     

     

    As we know Policy Server maintains 3 connections DIR, USR and PING. We assume that the above action of a Firewall OR LDAP sending a RST, puts the DIR and/or USR in CLOSE_WAIT. However PING connection is actively returning success. Could we do something from the Policy Server side (within DIR and/or USR connection, similar to introducing a PING within DIR and/or USR) that allows Policy Server to send a FIN/ACK and close down the connection immediately, rather than Policy Server send only an ACK and keep the connection on CLOSE WAIT. This would eliminate the additional call over a CLOSE_WAIT connection during a actual SEARCH request and Policy Server starts off with a fresh connection.

     

     

     

     

     

     

    Log Snippets :

     

     

    [SmDsDir.cpp:66][CSmDsDir::CSmDsDir][Start of call InitDir.][About to initialize directory, Oid='0e-000231d3-2718-16f4-83ec-693a0a0a909d', Name='LDAP-DIR'][

    [SmDsLdapProvider.cpp:1424][CSmDsLdapProvider::InitDir][Using LDAP server bank #1][ldapserver1.ca.com][1389]

    [SmDsLdapFunctionImpl.cpp:1952][ImproveLDAPConnection][Enter ImproveLDAPConnection]

    [ImproveLDAPConnection][Exit ImproveLDAPConnection]

    [SmDsDir.cpp:81][CSmDsDir::CSmDsDir][Return from call InitDir.]

    [SmDsObj.cpp:94][CSmDsObj::IsValid][Start of call IsValid.]

    [SmDsObj.cpp:96][CSmDsObj::IsValid][Return from call IsValid.][true]

    [SmDsDir.cpp:1080][CSmDsDir::GetDirectoryVersionInfo][Enter function CSmDsDir::GetDirectoryVersionInfo]

    [SmDsDir.cpp:1082][CSmDsDir::GetDirectoryVersionInfo][Leave function CSmDsDir::GetDirectoryVersionInfo][18][00:00:00.000023]

    [SmObjCache.cpp:824][CSmObjCache::Fetch][Retrieve an object from the object cache.]

    [SmObjStore.cpp:3363][IsADEnhanced][Global Preferences:]

    [SmDsDir.cpp:194][CSmDsDir::GetConnectionObject][Start of call GetDirConnectionObject.][Get dir connection object.]

    [SmDsLdapFunctionImpl.cpp:1952][ImproveLDAPConnection][Enter ImproveLDAPConnection]

    [ImproveLDAPConnection][Exit ImproveLDAPConnection]

    [SmDsLdapFunctionImpl.cpp:1740][GetConHandle][Enter GetConHandle]

    [SmDsLdapFunctionImpl.cpp:1741][GetConHandle][host=ldapserver1.ca.com, port=1389, secure=0, automatic=1, search=1]

    [GetConHandle][Exit GetConHandle]

    [SmDsDir.cpp:196][CSmDsDir::GetConnectionObject][Return from call GetDirConnectionObject.][Ok]

    [SmDsDir.cpp:202][CSmDsDir::GetRawHandle][Start of call GetDirRawHandle.][Get dir raw handle]

    [SmDsDir.cpp:204][CSmDsDir::GetRawHandle][Return from call GetDirRawHandle.][Ok]

    [SmAuthUser.cpp:5117][CSmAuthUser::Authenticate][Enter function CSmAuthUser::Authenticate]

    [SmAuthHtml.cpp:279][SmAuthenticate][Enter function SmAuthenticate]

    [SmAuthHtml.cpp:284][SmAuthenticate][Leave function SmAuthenticate][6][00:00:00.000181]

    [SmAuthUser.cpp:1695][CSmAuthUser::SavePasswordState][Enter function CSmAuthUser::SavePasswordState]

    [SmAuthUser.cpp:1697][CSmAuthUser::SavePasswordState][Leave function CSmAuthUser::SavePasswordState][false][00:00:00.000038]

    [SmAuthUser.cpp:5385][CSmAuthUser::Authenticate][Leave function CSmAuthUser::Authenticate][6][00:00:00.002154]

    [SmDsDir.cpp:272][CSmDsDir::IsValidUsername][Start of call IsValidUsername.][User ='AAAAAA']

    [SmDsDir.cpp:274][CSmDsDir::IsValidUsername][Return from call IsValidUsername.][true]

    [SmDsDir.cpp:425][CSmDsDir::Search][Start of call Search.][Advanced search, Root='dc=ca,dc=com',Filter='(cn=AAAAAA)']

    [SmDsAliases.cpp:328][CSmDsAliases::GetSmDsAliases][Enter function CSmDsAliases::GetSmDsAliases]

    [SmDsAliases.cpp:377][CSmDsAliases::GetSmDsAliases][Leave function CSmDsAliases::GetSmDsAliases][true][00:00:00.000757]

    [SmDsAliases.cpp:428][CSmDsAliases::GetAttributeMapping][Enter function CSmDsAliases::GetAttributeMapping]

    [SmDsAliases.cpp:435][CSmDsAliases::GetAttributeMapping][Leave function CSmDsAliases::GetAttributeMapping][false][00:00:00.000169]

    [SmDsLdapFunctionImpl.cpp:1952][ImproveLDAPConnection][Enter ImproveLDAPConnection]

    [ImproveLDAPConnection][Exit ImproveLDAPConnection]

    [SmDsLdapProvider.cpp:1783][CSmDsLdapProvider::SearchImpl][search filter is : (cn=AAAAAA)]

    [SmDsLdapFunctionImpl.cpp:3127][SearchExts][Enter SearchExts]

    [SmDsLdapConnMgr.cpp:1190][LogMessage:ERROR:[sm-Ldap-02230] Error# '81' during search: 'error: Can't contact LDAP server' Search Query = '(cn=AAAAAA)']

    [SmDsLdapConnMgr.cpp:1201][CSmDsLdapConn::SearchExts][LDAP search of (cn=AAAAAA) took 0 seconds and 8237 microseconds]

    [SmDsLdapFunctionImpl.cpp:3155][CSmDsLdapProvider::SearchExts][Ldap Search failed, ErrorMsg is Can't contact LDAP server]

    [SmDsLdapFunctionImpl.cpp:2013][RebindServer][Enter RebindServer]

    [SmDsLdapFunctionImpl.cpp:2044][RebindServer][server OK]

    [SmDsLdapConnMgr.cpp:501][CSmDsLdapConnMgr::AddDeadHandleList][Marked dir connection (seq: 3) ldapserver1.ca.com:1389 as Close Pending]

    [SmDsLdapConnMgr.cpp:501][CSmDsLdapConnMgr::AddDeadHandleList][Marked dir connection (seq: 1) ldapserver1.ca.com:1389 as Close Pending]

    [SmDsLdapConnMgr.cpp:501][CSmDsLdapConnMgr::AddDeadHandleList][Marked user connection (seq: 2) ldapserver1.ca.com:1389 as Close Pending]

    [SmDsLdapFunctionImpl.cpp:3002][LdapBind][Enter LdapBind]

    [SmDsLdapFunctionImpl.cpp:3053][LdapBind][err=ErrCode: 0]

    [LdapBind][Exit LdapBind]

    [SmDsLdapConnMgr.cpp:895][IsAvailable][Successful V3 Bind server][ldapserver1.ca.com][1389]

    [SmDsLdapConnMgr.cpp:628][PingServer][LDAP Server Ping Successful][ldapserver1.ca.com][1389]

    [SmDsLdapFunctionImpl.cpp:2110][CSmDsLdapProvider::RebindServer][Reconnect to server 'ldapserver1.ca.com:1389' as it's previous connections are closed and it is available for connecting now]

    [SmDsLdapFunctionImpl.cpp:2132][RebindServer][GetData(pDs)->m_szServer=ldapserver1.ca.com:1389, nRebindTimestamp=1459965750, szBestServer=, nBestServerTimestamp=0][

    [SmDsLdapFunctionImpl.cpp:2169][RebindServer][ to rebind to same server as current connection is closed by server]

    [SmDsLdapFunctionImpl.cpp:2170][RebindServer][szBestServer=ldapserver1.ca.com:1389]

    [SmDsLdapFunctionImpl.cpp:2203][CSmDsLdapProvider::RebindServer][Rebind attempt on 'dir' connection to best LDAP server 'ldapserver1.ca.com:1389']

    [SmDsLdapFunctionImpl.cpp:2408][BindServer][Enter BindServer]

    [SmDsLdapFunctionImpl.cpp:2409][BindServer][szServer=ldapserver1.ca.com:1389, szBindDN=cn=adminr1252,ou=admin,dc=ca,dc=com, nSearchResults=0, nSearchTimeout=90]

    [SmDsLdapFunctionImpl.cpp:2410][BindServer][bRequireCredentials=1, bSSL=0, bAutomatic=0]

    [SmDsLdapFunctionImpl.cpp:2564][BindServer][(Bind) For this handle LDAP automatic referrals are disabled.]

    [SmDsLdapFunctionImpl.cpp:3002][LdapBind][Enter LdapBind]

    [SmDsLdapFunctionImpl.cpp:3053][LdapBind][err=ErrCode: 0]

    [LdapBind][Exit LdapBind]

    [SmDsLdapFunctionImpl.cpp:2733][BindServer][szBindDN << nSearchResults << nSearchTimeout (cn=adminr1252,ou=admin,dc=ca,dc=com, 0, 90)]

    [BindServer][Exit BindServer]

    [SmDsLdapFunctionImpl.cpp:2256][RebindServer][szBestServer=ldapserver1.ca.com:1389]

    [SmDsLdapFunctionImpl.cpp:2294][RebindServer][pDsLdap->m_nCurr+1=1, szBestServer=ldapserver1.ca.com:1389]

    [SmDsLdapFunctionImpl.cpp:2408][BindServer][Enter BindServer]

    [SmDsLdapFunctionImpl.cpp:2409][BindServer][szServer=ldapserver1.ca.com:1389, szBindDN=cn=adminr1252,ou=admin,dc=ca,dc=com, nSearchResults=0, nSearchTimeout=90]

    [SmDsLdapFunctionImpl.cpp:2410][BindServer][bRequireCredentials=1, bSSL=0, bAutomatic=0]

    [SmDsLdapFunctionImpl.cpp:2564][BindServer][(Bind) For this handle LDAP automatic referrals are disabled.]

    [SmDsLdapFunctionImpl.cpp:3002][LdapBind][Enter LdapBind]

    [SmDsLdapFunctionImpl.cpp:3053][LdapBind][err=ErrCode: 0]

    [LdapBind][Exit LdapBind]

    [SmDsLdapFunctionImpl.cpp:2733][BindServer][szBindDN << nSearchResults << nSearchTimeout (cn=adminr1252,ou=admin,dc=ca,dc=com, 0, 90)]

    [BindServer][Exit BindServer]

    [SmDsLdapFunctionImpl.cpp:2322][RebindServer][szBestServer=ldapserver1.ca.com:1389]

    [RebindServer][Exit RebindServer]

    [SmDsLdapConnMgr.cpp:1201][CSmDsLdapConn::SearchExts][LDAP search of (cn=AAAAAA) took 0 seconds and 2981 microseconds]

    [SearchExts][Exit SearchExts]

    [SmDsLdapProvider.cpp:2311][CSmDsLdapProvider::Search][Ldap Search callout succeeds.][(Search) Base: 'dc=ca,dc=com', Filter: '(cn=AAAAAA)'. Status: 1 entries]

     

     

     

    Regards

     

    Hubert