Symantec Access Management

Expand all | Collapse all

CA Directory : Issue in enabling replication over SSL

  • 1.  CA Directory : Issue in enabling replication over SSL

    Posted Oct 31, 2014 11:46 AM

    Hello All

     

    Any clues why replication is failing over SSL. I have 3 DataDSAs i.e.

     

     

    kstore

    kstore2

    kstore3

     

     

    I added an entry in “kstore” and it tries to replicate to other two. Whilst doing so reports these messages in the WARNING log file.

     

    [0] 20141031.100732.635 WARN : max-local-ops has no effect

    [0] 20141031.100732.639 WARN : Loading cache

    [0] 20141031.100732.653 WARN : Datastore was created at: 20141027193625Z

    [0] 20141031.100732.653 WARN : Datastore was created for: kstore

    [0] 20141031.100732.680 WARN : Cache loaded, 9296 entries

    [0] 20141031.100732.800 WARN : Memory used by cache: 5094970 + 7975679

    [0] 20141031.100732.800 WARN : Found new MW DSA: kstore2

    [0] 20141031.100732.800 WARN : Found new MW DSA: kstore3

    [5] 20141031.101404.461 WARN : Verify error 20: unable to get local issuer certificate

    [5] 20141031.101404.461 WARN : SSL Error

    [5] 20141031.101404.461 WARN : 5:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:979:

     

     

    [5] 20141031.101404.461 WARN : ssld_ssl_request failed

    [5] 20141031.101404.461 WARN : Remote DSA 'kstore2' aborted

    [5] 20141031.101404.461 WARN : Marking DSA 'kstore2' as down

    [8] 20141031.101404.466 WARN : Verify error 20: unable to get local issuer certificate

    [8] 20141031.101404.466 WARN : SSL Error

    [8] 20141031.101404.466 WARN : 8:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:979:

     

     

    [8] 20141031.101404.466 WARN : ssld_ssl_request failed

    [8] 20141031.101404.466 WARN : Remote DSA 'kstore3' aborted

    [8] 20141031.101404.466 WARN : Marking DSA 'kstore3' as down

     

     

    In kstore2 I see the following

     

    [7] 20141031.113949.755 WARN : TLS/SSL handshake failed for call from 138.42.177.173:46823

    [4] 20141031.114050.852 WARN : TLS/SSL handshake failed for call from 138.42.177.173:46825

    [1] 20141031.114151.951 WARN : TLS/SSL handshake failed for call from 138.42.177.173:48584

    [9] 20141031.114252.047 WARN : TLS/SSL handshake failed for call from 138.42.177.173:48586

    [3] 20141031.114353.149 WARN : TLS/SSL handshake failed for call from 138.42.177.173:48588

    [5] 20141031.114454.247 WARN : TLS/SSL handshake failed for call from 138.42.177.173:48590

    [4] 20141031.114555.340 WARN : TLS/SSL handshake failed for call from 138.42.177.173:48592

    [1] 20141031.114656.439 WARN : TLS/SSL handshake failed for call from 138.42.177.173:49630

    [9] 20141031.114757.537 WARN : TLS/SSL handshake failed for call from 138.42.177.173:49632

    [2] 20141031.114858.637 WARN : TLS/SSL handshake failed for call from 138.42.177.173:49634

     

     

     

    NOTE : I did get a feedback from a peer suggesting I should have done it the other way round i.e. first get replication working and then enable SSL. However since I invested time and energy to get all the 3 DataDSAs on enabling SSL, I would like to continue ahead on the same path and see if it could be resolved. Also I would like to state that I am using my own openssl CA to sign the server certificates. SSL connections to individual dataDSA using Jxexplorer is working fine. SSL Connections from Policy Server / SmConsole is also working fine.

     

     

    I probably think I am missing a step for interworking of remote dataDSA over SSL, however if someone could suggest if these ideas are correct OR the steps forward, it would be helpful.

     

     

    A. Currently I have in ssld folder in each server, the respective certificates only.Do I need to add the other server/dataDSA server certificates too. If so where should I add it? in trusted.pem OR elsewhere?

     

    B. Anything else?

     

     

    Regards

     

    Hubert



  • 2.  Re: CA Directory : Issue in enabling replication over SSL
    Best Answer

    Posted Nov 02, 2014 06:31 PM

    Hi Hubert,

     

    This looks like the CA Directory unable to get the certificate that it expects. Maybe server certificate placed in wrong path? Not an expert in CA Directory SSL. Probably need to move this thread to CA Directory communities to get more insight from CA Directory expert.

     

    The specified item was not found.

     

    Regards,

    Kar Meng



  • 3.  Re: CA Directory : Issue in enabling replication over SSL

    Broadcom Employee
    Posted Nov 02, 2014 11:43 PM

    Hi Hubert,

     

    I assume kstore1 is the primary DSA and you’ve done the following
    1. On kstore1/kstore2/kstore3, each .dxi file has source all the 3 knowledge files for (kstore1/kstore2/kstore3)
    2. On kstore1, you’ve run dxcertgen generate certificates for ALL 3 DSA’s

     

    Then, you only need to duplicate ALL the .pem files from the following 2 folders to the corresponding folders on kstore2 and kstore3
    * %dxhome%\config\ssld
    * %dxhome%\config\ssld\personalities


    This is the way to align all certificates across all your CA directory machines.

     

    If you didn't run dxcertgen before , you need to do the following

    1. Verify that all of the DSA knowledge files are available in the primary DSA (kstore1)
       domain installation.

    2. Execute the dxcertgen command to generate personality files for the configured DSAs in %DXHOME%\config\knowledge, as shown in the following example:
       dxcertgen -d 365 -i "dc=keystore" certs
       -d
         Specifies the number of days that certificates are valid, one year in this example.
       -i
         Specifies the issuer of the certificate, dc=keystore in this example.
       certs
         Subcommand of dxcertgen, which means certificates

    3. You can run the following command for reporting after the personality files are generated:
       dxcertgen report

     

    Hope this is helpful.

     

    Best Regards,

     

    Yong



  • 4.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Nov 03, 2014 10:55 AM

    Thank You All

     

    The way I did build the setup is as follows

     

    A. Setup KStore1/DataDSA.

    B. Enable SSL for KStore1 using dxcertgen command on KStore1.

    C. Setup KStore2/DataDSA.

    D. Enable SSL for KStore2 using dxcertgen command on KStore2.

    E. Setup KStore3/DataDSA.

    F. Enable SSL for KStore3 using dxcertgen command on KStore3.

     

    Therefore this stage...

    1. All 2 DataDSA's have only their respective certificates in the personalities folder.

    2. The RootCA is the same for all 3 certificates. Hence in trusted.pem there is only one RootCA certificate

    3. All DataDSA's have the same structure i.e. dc=ca,dc=com.

    4. Only DSA name is unique, i.e. KStore1, KStore2, KStore3.

     

    Post this stage....

    G. Make changes to knowledge file for Multi-Write.

    H. Copy knowledge files for all the 3 DataDSA on all Servers.

    I. Make a Group file and then call (source) this in the server's .dxi file.

    J. Start the servers. Check logs to see if all knowledge files are read.

     

     

    This is basically where I am at in a nutshell.

     

     

    Now based on the suggestion from Yong, it feels that the personalities folder needs to have *.pem files of all Servers. Plus the trusted.pem needs to include RootCA entries of all 3 Servers. My current stage has only respective *.pem files in respective servers and same with trusted.pem.

     

    Am going to manually copy the *.key and *.cert from other servers across all servers. Then run these below commands on all server to have the other servers certs added into personalities and trusted.pem.

     

    Add the certs in KStore1

    >> dxcertgen -D kstore2 -n kstore2.cer certmerge

    >> dxcertgen -n kstore2.cer importca

    >> dxcertgen -D kstore3 -n kstore3.cer certmerge

    >> dxcertgen -n kstore3.cer importca

    Similar do the same on KStore2 and KStore3.

     

     

    Hopefully, it should resolve (fingers crossed). Otherwise it looks like I have to dismantle and start from scratch by doing replication first and then enabling SSL.

     

     

    Regards

     

    Hubert



  • 5.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Nov 03, 2014 04:04 PM

    Thank You All,

     

    Finally managed to get it working....

     

    • The documentation is a bit confusing in terms of how certificates should be generated across DataDSAs (no RouterDSA in this mix).
    • The normal recommendation is to generate at one place and copy the certs.

     

     

    Here's what I did to get it to work with only DataDSA's (no Router DSA). Again I stayed on the same approach i.e. enabled SSL on each DataDSA first, then did replication over SSL.

     

     

    SSL Enablement

     

    A. Server-1, Setup KStore1/DataDSA. [dxnewdsa kstore1 7771 "dc=ca,dc=com"].

    B. Enable SSL for KStore1 using dxcertgen command on KStore1.

    • dxcertgen -D kstore1 -Z SHA256 -k 2048 certreq
    • dxcertgen -D kstore1 -n kstore1.cer certmerge
    • dxcertgen -n ca.cer importca

    NOTE :

    The 1st bullet >> generates a key and CSR.

    The 2nd bullet >> imports the signed certificate and the key into a pem file under personalities folder.

    The 3rd bullet >> imports the Root CA who signed the CSR inorder to generate the signed certificate into trusted.pem.

     

    C. Server-2, Setup KStore2/DataDSA. [dxnewdsa kstore2 7771 "dc=ca,dc=com"].

    D. Enable SSL for KStore2 using dxcertgen command on KStore2.

    • dxcertgen -D kstore2 -Z SHA256 -k 2048 certreq
    • dxcertgen -D kstore2 -n kstore2.cer certmerge
    • dxcertgen -n ca.cer importca

    NOTE :

    The 1st bullet >> generates a key and CSR.

    The 2nd bullet >> imports the signed certificate and the key into a pem file under personalities folder.

    The 3rd bullet >> imports the Root CA who signed the CSR inorder to generate the signed certificate into trusted.pem.

     

     

    E. Server-3, Setup KStore3/DataDSA. [dxnewdsa kstore3 7771 "dc=ca,dc=com"].

    F. Enable SSL for KStore3 using dxcertgen command on KStore3.

    • dxcertgen -D kstore3 -Z SHA256 -k 2048 certreq
    • dxcertgen -D kstore3 -n kstore3.cer certmerge
    • dxcertgen -n ca.cer importca

    NOTE :

    The 1st bullet >> generates a key and CSR.

    The 2nd bullet >> imports the signed certificate and the key into a pem file under personalities folder.

    The 3rd bullet >> imports the Root CA who signed the CSR inorder to generate the signed certificate into trusted.pem.

     

     

    G. Make modification to the knowledge file to allow changes for SSL enablement. All changes to default knowledge file is shown in italics.

    set dsa "kstore1" =

    {

        prefix              = <dc com><dc ca>

        dsa-name        = <dc com><dc ca><cn "kstore1">

        dsa-password  = "secret"

        address            = tcp "host1" port 7771, tcp "host1" port 7443

        disp-psap          = DISP

        snmp-port          = 7771

        console-port      = 7772

        auth-levels  = anonymous, clear-password, ssl-auth

        trust-flags  = allow-check-password

        link-flags    = ssl-encryption, ssl-encryption-remote

    };

     

    H. Repeat Step-G for KStore2 and KStore3.

     

    I. To force SSL encryption on anonymous bindings, include the following parameter in the settings configuration file of the DSA (DXHOME\config\settings) under the # security controls section. When this is set to “true”, if a user tries to create an anonymous binding without SSL, the DSA disallows it and returns an "Inappropriate authentication" error.

    set force-encrypt-anon = true;

     

    J. To force SSL encryption on authenticated bindings, include the following parameter in the settings configuration file of the DSA (DXHOME\config\settings) under the # security controls section. Note: The ‘set force-encrypt-auth’ setting does not prevent the credentials from being sent unencrypted over the network. However it refuses any unencrypted binding request.

    set force-encrypt-auth = true;

     

     

    I. Stop and Start services.

     

    Now you could use a LDAP Tool e.g. Jxplorer to connect to each of these servers / instances and test SSL Connection.

     

     

     

     

     

    Replication

     

    A. Go to KStore1 Server. Add the below flags for replication in the knowledge files of the DataDSA KStore1.

     

    set dsa "kstore1" =

    {

        prefix        = <dc com><dc ca>

        dsa-name      = <dc com><dc ca><cn "kstore1">

        dsa-password  = "secret"

        address      = tcp "host1" port 7771, tcp "host1" port 7443

        disp-psap    = DISP

        snmp-port    = 7771

        console-port  = 7772

        auth-levels  = anonymous, clear-password, ssl-auth

        dsa-flags    = multi-write

        trust-flags  = allow-check-password, trust-conveyed-originator, trust-dsa-triggered-operations

        link-flags    = ssl-encryption, ssl-encryption-remote

    };

     

    B. Repeat the same Step-A in knowledge files by logging into servers KStore2 and KStore3.

     

    C. Copy the knowledge files from KStore2 and KStore3 into the knowledge folder in KStore1. Do the same for KStore2 and KStore3, such that all 3 server/DataDSA now have local copies of all 3 knowledge files in the knowledge folder within the respective server.

     

    D. Go to KStore1 server. Create a Group file "repDSAGrp.dxg" in KStore1 under knowledge folder and source all the knowledge files.

    -bash-4.1$ more ../knowledge/repDSAGrp.dxg

    source "kstore1.dxc";

    source "kstore2.dxc";

    source "kstore3.dxc";

     

    E. Copy the Group file into the knowledge folder in KStore2 and KStore3.

     

    F. Go to KStore1 server. Modify the initialization file kstore1.dxi under DX_HOME/config/servers, to include the Group file. Note I had to disable the knowledge file otherwise an error about not being unique is thrown up.

    # knowledge

    clear dsas;

    #source "../knowledge/kstore1.dxc";

    source "../knowledge/repDSAGrp.dxg";

     

    G. Go to KStore2 and KStore3. Repeat the same Step-F.

     

    H. Stop and Start all DataDSA's.

     

     

    Now you could use a LDAP Tool e.g. Jxplorer to connect to each of these servers / instances using SSL Connection. Connect to KStore1 and create a test object e.g. create a user and submit. Now login using JXplorer to KStore2 and KStore3 on SSL, you should see the object replicated.

     

     

    Error Logging

     

    If there are any errors, they'd be logged in the WARNING log file under DXHOME/logs.

    The name of the log file would be <DSANAME>_warn_<timestamp>.log e.g. kstore_warn_20141103.log

     

     

     

    I hope I have recorded all the steps that I performed to get this working.

     

     

    Regards

     

    Hubert



  • 6.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 09:20 AM

    Hello Hubert,

    I am facing an issue which has similar message.

    I have few Data DSAs running with replication  enabled.

    However all but one Data DSA's certificates are getting expired.

    I just renewed the cert in one of the DSAs and it went fine and showed the message about cert getting updated.

    however in the warning and trace logs, I can see "TLS/SSL handshake failed for call from another Data DSA" and also "unable to synchronize with peer" .

     though the instance is up and running. 



  • 7.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 10:14 AM

    One possibility, Was the new cert issued by the same Root CA? Compare the earlier Root CA even if it was the same issuer and if you see a difference in the Root CA have it imported as well across all DSA.

     

    Can you also run DEBUG / QUERY in console mode with "set trace" flags enabled? that'd help more output lines.



  • 8.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 10:50 AM

    Hello Hubert

    I also thought the same as probable root cause. It is indeed the case. Other Data DSAs have certs issued by different CA.  

      But in this case we will end up renewing those certs also which are not expiring if we want all of them to be issued by same CA.

    Can we just copy the new root CA in the existing trusted.pem files, would that fix our problem?

    Or, Is there any possibility of other issue as well ?

    Which command you want me to run for query/debug ?

     

    Thanks

    Satyendra



  • 9.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 11:08 AM

    We may need to run "dxcertgen -n Newca.cer importca" in all the DSA's. So that all DSA's now trust the new RootCA.

     

    Is this PROD or NONPROD? Just being careful. Backup the personalities folder in all DSA before you do this change.



  • 10.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 11:56 AM

    Thanks Hubert.

    We do take back up of dsaname.pem and trusted.pem certificates. its Prod.

    I think adding new root CA in trusted.pem would do the same thing as this command because that the path mentioned for root CA in configuration. What are your thoughts.



  • 11.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 12:26 PM

    That is correct == "I think adding new root CA in trusted.pem would do the same thing as this command".



  • 12.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 12:34 PM

    Thanks a lot Hubert.

    in the same issue, I am getting below error in the warning log:

    WARN : ssld_ssl_request failed [2] 20170811.122004.465 WARN : Verify error 19: self signed certificate in certificate chain [2] 20170811.122004.465 WARN : SSL Error

    WARN : 2:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:s3_clnt.c:1264:

    [2] 20170811.122004.465 WARN : ssld_ssl_request failed

    Though my root CA or server cert is not self signed, Could it be related?

     

    Thanks

    Satyendra



  • 13.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 01:13 PM

    Could be. Since the Certificate was updated but the trusted.pem was not updated with the new RootCA.

     

    <SNIP> from an earlier case.

    The Customer may have not imported the CAcert, trusted.pem, and/or the intermediary certificate into the trusted.pem in CA Directory, using the dxcertgen importca  command.

    <SNIP>



  • 14.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 01:22 PM

    New root CA is added in trusted.pem .



  • 15.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 11, 2017 02:32 PM

    Did it work thereafter? You may also have to restart the DSA instances.

     

    Regards,

    Hubert



  • 16.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 14, 2017 09:33 AM

    NO, It did not. I restarted the instance, even then it didn't work.



  • 17.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 15, 2017 01:20 PM

    Did the logs message have any difference before OR is the same lines in the log? It is always best to open a new thread for a new issue.



  • 18.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 16, 2017 12:10 PM

    NO, its still the same message.

    ssld_ssl_request failed
    [WARN : TLS/SSL handshake failed for call from 'ipaddres'
    WARN : MW-DISP not in sync for 'dsaname'



  • 19.  Re: CA Directory : Issue in enabling replication over SSL

    Posted Aug 16, 2017 12:24 PM

    This is a different error message from the earlier one. We want to avoid discussing multiple issues on the same thread.

     

    I have opened a new thread for you. We will discuss further on that thread. Please review the new thread.

     

    https://communities.ca.com/message/242000231-error-after-updating-cert-in-one-dsa