Kelly Wong

Tech Tip - CA Privileged Access Manager: Cluster constantly going out-of-sync

Blog Post created by Kelly Wong Employee on Nov 9, 2017

CA Privileged Access Manager Tech Tip by Kelly Wong, Principal Support Engineer for 9th November 2017

Symptoms

Cluster constantly going out-of-sync. The errors in the log highlight database replication issues.

 

php_error.log:

[ 14:08:10 10/15/17 ] [ error ] [Request-59e06ce426937]: My database (***.xx.x.xx) is out-of-sync with member ***.xx.x.xx database: Array
(
[status] => out-sync

...

[ 14:15:01 10/15/17 ] [ info ] [Request-59e36d65d3f29]:  Logging sql exec = DELETE FROM saml_idp_replay_requests WHERE expire < NOW()

Issued From /sbin/cleanSAMLReplayCache.php:28 [ /var/www/htdocs/uag/services/main/common/Database.php : 64 ]

[ 14:15:01 10/15/17 ] [ debug ] [Request-59e36d65d3f29]:  Query will be replicated to cluster:  DELETE FROM saml_idp_replay_requests WHERE expire < NOW() [ /var/www/htdocs/uag/services/main/common/QueryReplicator.php : 271 ]

[ 14:15:01 10/15/17 ] [ debug ] [Request-59e36d65d3f29]:  Replication Query:  Sending Query To Member ***.xx.x.xx [ /var/www/htdocs/uag/services/main/common/QueryReplicator.php : 206 ]

[ 14:15:01 10/15/17 ] [ debug ] [Request-59e36d65d3f29]:  Post Request Response for https://***.xx.x.xx/ajax_cmd.php?cmd=ACTACT&shared_key=d41e98d1eafa6d6011d3a70f1a5b92f0&cmdtype=EXECSQL:  Array

(

    [0] => ____J___F__º_C=¼ã0eÇW_m—»F_&8d©¶œ__{Í_¨F[

Ä”__wwÐÖµ"ÜÔ@vJù_k‚>0k_žîz[ÆDkˆ_5_____Û
__×__Ô__Ñ0‚_Í0‚_µ _____    _éTæç_¼Ð¢0

_    *†H†÷

_____0Ÿ1
0    __U____US1_0___U___

New Jersey1_0___U_ _
Jersey City1_0___U_

_

Xceedium, Inc1_0___U_
_ Support1_0___U___
xceedium.com1$0"_     *†H†÷

_    ___rgallant@xceedium.com0__

150223191836Z_

200222191836Z0Ÿ1
0    __U____US1_0___U___

New Jersey1_0___U_ _
Jersey City1_0___U_

_

Xceedium, Inc1_0___U_
_ Support1_0___U___
xceedium.com1$0"_     *†H†÷

_    ___rgallant@xceedium.com0‚_"0

_    *†H†÷

______‚___0‚_

_‚___ ê ºr_2H™ ´_¦‚«õÞ%ÁÅ+QÓR …0Ц²ÉIÛ„©P_Ó]–,·ì«Â‚éÀF_„_ã_`e^‚,_£mÞT0

›£   Îg€'¨-У'_D_ÿ'Ž¥;šyñ²{ù_¯ËP±ms·†¼_Vš2üT_ë/  …ð½

l;eIðPˆµÚç~5X_ZgJ]__@_ƒŸ2#&OõmxӁ
š_«Ù96ÉÜÊh/ìmpÄ&„ŪåIÓ>p‡j‚½

ˆZÔØî¡ÁK·†Û±zºVl÷ÿ¡_¢YvÙզ.T_#¹)"E*Pìs]<ˆ9’óÚØg¢éÀ>»“X__¹GŽm·0bç ë_____£‚__0‚__0___U______,í«_]·‰H˜"™ƒ



¼©{0Ô__U_#_Ì0É€_,í«_]·‰H˜"™ƒ



¼©{¡¥¤¢0Ÿ1
0    __U____US1_0___U___

New Jersey1_0___U_ _
Jersey City1_0___U_

_

Xceedium, Inc1_0___U_
_ Support1_0___U___
xceedium.com1$0"_     *†H†÷

_    ___rgallant@xceedium.com‚  _éTæç_¼Ð¢0
__U____0___ÿ0

_    *†H†÷

______‚___{µ†_Ü_´·__k,Ž__k΢5(œ_ôƹ¹&F,_é~Õ†¹“ÔT’F׍݈—oh½_á»ëLò©JŽÈ܁3oC`–TR_ñQHÛ›Bz³-Ù_"_^Ámï_"FØj…å³ÓB-__”üúâZI&ÓUÍi_¨Ú%Šò)Fdá!_ó_B5÷¸çÿ¯òdåÑo
¼èWSqÛ¢®¨%_¶‡ÿ•½_:™ØÎ_[víÒ‹Œ_x'Ÿ-ÝUÄuÂôµKyçkmPf_43‚_˜¾Æ¦À†‹_Ÿ:L|1ª¦_a©ù‚VN·h<PF“‹ÄÙ\ÿ8TôCërF“î___¦št|gÍ_»ÉvV

GØ_&N________________F

)

 [ /var/www/htdocs/uag/services/main/common/ReplicationDataSender.php : 110 ]

[ 14:15:01 10/15/17 ] [ error ] [Request-59e36d65d3f29]:  Failed to replicate encoded query (c3H1cQ1xVXAL8vdVKE7MzYnPTCmIL0otyEmsBFKFpanFJcUK4R6uQa4KqRUFmUWpCjYKfv7hGpoA) to member ***.xx.x.xx.  Response was:  ____J___F__º_C=¼ã0eÇW_m—»F_&8d©¶œ__{Í_¨F[

Ä”__wwÐÖµ"ÜÔ@vJù_k‚>0k_žîz[ÆDkˆ_5_____Û
__×__Ô__Ñ0‚_Í0‚_µ _____    _éTæç_¼Ð¢0

_    *†H†÷

_____0Ÿ1
0    __U____US1_0___U___

New Jersey1_0___U_ _
Jersey City1_0___U_

_

Xceedium, Inc1_0___U_
_ Support1_0___U___
xceedium.com1$0"_     *†H†÷

_    ___rgallant@xceedium.com0__

150223191836Z_

200222191836Z0Ÿ1
0    __U____US1_0___U___

New Jersey1_0___U_ _
Jersey City1_0___U_

_

Xceedium, Inc1_0___U_
_ Support1_0___U___
xceedium.com1$0"_     *†H†÷

_    ___rgallant@xceedium.com0‚_"0

_    *†H†÷

______‚___0‚_

_‚___ ê ºr_2H™ ´_¦‚«õÞ%ÁÅ+QÓR …0Ц²ÉIÛ„©P_Ó]–,·ì«Â‚éÀF_„_ã_`e^‚,_£mÞT0

›£   Îg€'¨-У'_D_ÿ'Ž¥;šyñ²{ù_¯ËP±ms·†¼_Vš2üT_ë/  …ð½

l;eIðPˆµÚç~5X_ZgJ]__@_ƒŸ2#&OõmxӁ
š_«Ù96ÉÜÊh/ìmpÄ&„ŪåIÓ>p‡j‚½

ˆZÔØî¡ÁK·†Û±zºVl÷ÿ¡_¢YvÙզ.T_#¹)"E*Pìs]<ˆ9’óÚØg¢éÀ>»“X__¹GŽm·0bç ë_____£‚__0‚__0___U______,í«_]·‰H˜"™ƒ



¼©{0Ô__U_#_Ì0É€_,í«_]·‰H˜"™ƒ



¼©{¡¥¤¢0Ÿ1
0    __U____US1_0___U___

New Jersey1_0___U_ _
Jersey City1_0___U_

_

Xceedium, Inc1_0___U_
_ Support1_0___U___
xceedium.com1$0"_     *†H†÷

_    ___rgallant@xceedium.com‚  _éTæç_¼Ð¢0
__U____0___ÿ0

_    *†H†÷

______‚___{µ†_Ü_´·__k,Ž__k΢5(œ_ôƹ¹&F,_é~Õ†¹“ÔT’F׍݈—oh½_á»ëLò©JŽÈ܁3oC`–TR_ñQHÛ›Bz³-Ù_"_^Ámï_"FØj…å³ÓB-__”üúâZI&ÓUÍi_¨Ú%Šò)Fdá!_ó_B5÷¸çÿ¯òdåÑo
¼èWSqÛ¢®¨%_¶‡ÿ•½_:™ØÎ_[víÒ‹Œ_x'Ÿ-ÝUÄuÂôµKyçkmPf_43‚_˜¾Æ¦À†‹_Ÿ:L|1ª¦_a©ù‚VN·h<PF“‹ÄÙ\ÿ8TôCërF“î___¦št|gÍ_»ÉvV

GØ_&N________________F [ /var/www/htdocs/uag/services/main/common/ReplicationDataSender.php : 132 ]

 

Resolution

To resolve this database replication issue, we need to check the ensure the symbolic link for libcurl.so.4 is correctly referencing libcurl.so.4.1.0.

 

SSH to the PAM server (all the cluster nodes) and perform the following steps:

  1. Check the output of the command:
    > ls -l /usr/lib/libcurl.so.4
    output will be as follows:
    lrwxrwxrwx 1 root root 16 2017-10-02 21:39 /usr/lib/libcurl.so.4 -> libcurl.so.4.1.1
    (notice 4.1.1 at the end of this output)
  2. If the command output matched above, run the following command
    > ln -sf /usr/lib/libcurl.so.4.1.0 /usr/lib/libcurl.so.4
  3. Check the output of the following command:
    > ls -l /usr/lib/libcurl.so.4
    It should match :
    lrwxrwxrwx 1 root root 25 2017-10-17 15:44 /usr/lib/libcurl.so.4 -> /usr/lib/libcurl.so.4.1.0
    (notice 4.1.0 at the end of the output)
  4. Restart Apache
  5. Restart cluster after all the nodes are updated -- not mandatory, but recommended
  6. Verify that the cluster comes up in-sync

Outcomes