Symantec Access Management

I have following setup -

- 3 Router DSA
- 2 Data DSA connecting to each Router DSA, so a total of 6 Data DSAs.
- All Data DSAs are added to the Knowledge group of each other for replication.
- DSA Flags is set to 'multi-write' and 'Multiwrite with DISP recovery' is enabled on all DSAs.

- Configurations are done using 'CA Directory Manager 14.0'

If a change is published on DSA1 (Added a system object), how this change will replicate to remaining DSAs. Will more than 1 Data DSA try to replicate their data to another one at the same.
If so, which DSA change will be replicated and how the conflict will be managed?

Thanks,

Pankaj

Have a read here at this link:

Set Up Replication - CA Directory - 14.0 - CA Technologies Documentation 

• Multiwrite replication with DISP recovery (Multiwrite-DISP)
  Multiwrite-DISP replication is a replication scheme that uses multiwrite replication for real-time updates and DISP for recovery.
  Multiwrite and DISP have complementary strengths: multiwrite replays updates to another DSA, and is fast and secure for small differences. However, if multiwrite has more updates to be replayed to another DSA than it can fit in its queue, it stops using the queue and drops all update information for that DSA. This means that multiwrite replication cannot cope with a DSA being down indefinitely. By contrast, DISP compares the current states of two DSAs and can handle a DSA being down indefinitely.
  Multiwrite-DISP is the recommended method of replication with CA Directory because it combines the efficiency of multiwrite when DSAs are online, with the robustness of DISP to allow DSAs to recover after being offline.

thanks,

Bill Patton

Pankaj Pankaj.Negi

If we have simply configured a vanilla MW configuration, the router which receives the UPDATE Request would send it to one of the Data DSA based on the Order of DSA defined in configuration file. The Data DSA that receives the UPDATE Request, becomes the Master for that UPDATE Request and is then tasked with updating the peers (slaves). I don't think the peers (slaves) update each other.

We can further our vanilla MW configuration by setting "write-precedence". Once we set a write-precedence, all UPDATE Request would go in the order defined by "write-precedence". This command is effective when set at the Router DSA level. If the set write-precedence command is not present, updates follow the normal precedence. The normal precedence is defined by the order of the DSAs in the configuration file, or the set precedence command.

When multiple DSAs (Routers and their respective DataDSAs) are configured like in our case here. The Routers should give precedence to the local data DSAs. There are configuration items that dictate which DSAs a router will communicate with, using "set precedence" and "set write-precedence". The precedence lists should include local DSAs first and additional key note, precedence only needs to be configured on router DSAs.

We can further extend this by creating MW-Groups.

We don't have to configure everything, but based on our needs have to design these aspects.

Reference Links : 

Multiwrite Replication - CA Directory - 14.0 - CA Technologies Documentation  

set write-precedence Command - CA Directory - 14.0 - CA Technologies Documentation  

set precedence Command - CA Directory - 14.0 - CA Technologies Documentation 

Large Deployment - CA Directory - 14.0 - CA Technologies Documentation  

Sample Topology and Disaster Recovery - CA Directory - 14.0 - CA Technologies Documentation 

Regards

Hubert

Thanks, Hubert HubertDennis

For replication to work in my setup, I did the following -

1. Start only 1 Data DSA and import Policy store data.

2. Took an online backup of the same. It gives us the .zdb file.

3. Remove .tx and .db file on other DSA and copy the .zdb file from DSA1 and make it .db file.

4. Start the DSA.

When I start the Data DSA2, I found following in the warn logs -

ERROR1- 

[0] 20181108.121953.266 WARN : Datastore was created at: 20180911124559Z
[0] 20181108.121953.266 WARN : Datastore was created for: DataDSA1

It should write DataDSA2 while startup. Could it be due to the fact that I copied .zdb file of Data DSA1?

But, that's one of the suggested ways to add a new DSA to MW group.

ERROR2-

[0] 20181108.121953.705 WARN : Cannot find certificate file DataDSA2.pem
[0] 20181108.121953.705 WARN : Creating client SSL_CTX without certificate/key
[2] 20181108.121953.797 WARN : HTTP send failure HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Users"
Date: Thu, 08 Nov 2018 12:19:53 GMT
Connection: keep-alive
Content-Length: 12

I did not enable SSL on Data DSA. So I should not get this in the logs. Is there any specific setting to fix this or is it something expected by design?

Thanks,

Pankaj

Pankaj Pankaj.Negi

ERROR-1

It is unclear if the correct steps were followed based on the four steps that you listed above. There are errors in the four steps and also it seems other critical steps have been missed as well.

One clear error I can spot is in Step-3 we not only change file extension ".zdb" to ".db"; but also must rename filename to match DSA name. Hence we have to change DataDSA1.zdb to DataDSA2.db once ".zdb" file is  copied over from DataDSA1 to DataDSA2. 

Refer the link below for the correct steps to be followed.

Reference Links : 

Add a DSA to a Multiwrite-DISP System - CA Directory - 14.0 - CA Technologies Documentation 

Backing Up Data - CA Directory - 14.0 - CA Technologies Documentation  for command "dxserver onlinebackup {dsaname}".

dump dxgrid-db Command -- Take a Consistent Snapshot Copy of a Datastore - CA Directory - 14.0 - CA Technologies Documen…  for command "dump dxgrid-db".

ERROR-2

What is the frequency of this error ? Is any client trying to connect using SSL ? I think may be more detailed logging may shed some light on who initiated this request and how (SSL / non SSL).

Regards

Hubert

Hi HubertDennis

I did follow the correct steps as per the documentation but missed mentioning a few steps in the last update

1. Stop all DSA except one (say DataDSA1).

2. Import Policy store data using XPSImport.

3. Take a backup of DataDSA1 using 'ONLINE BACKUP' feature of CA Directory Manager. It generates a file       DataDSA1.zdb

4. Copy this to the Data folder of DataDSA2. Delete .tx and .db file of DataDSA2.

5. Rename DataDSA1.zdb to DataDSA2.db

6. Start DataDSA2.

I observed the following error in the warn log of DataDSA2 -

[0] 20181108.121953.266 WARN : Datastore was created at: 20180911124559Z
[0] 20181108.121953.266 WARN : Datastore was created for: DataDSA1

ERROR2- 

This error is seen once per minute in the warn log without any other message.

I found 1 more thread on this error - http warning in DSA warn log  

The only component which is talking to DSA is Policy server and I did not enable SSL connection from there. I could see this error in the logs when I did not even point it to the Policy server.

Thanks,

Pankaj

Pankaj Pankaj.Negi

ERROR-1

I think this is as per design. The reason I say that is in general, these lines should be also printed in DataDSA1 warn log. The key question is does it work functionally. If we shutdown DataDSA1 and restart Policy Server, does Policy Server bulk fetch complete successfully. Further any admin UI operation, do they function successfully.

[0] 20181108.121953.266 WARN : Datastore was created at: 20180911124559Z
[0] 20181108.121953.266 WARN : Datastore was created for: DataDSA1

ERROR-2

It could be DataDSA1 trying to poll its peers as well since it is a MW peer. Would let know if I find anything.

Thanks, Hubert HubertDennis

ERROR1 looks like a possible Warn message setting which is kind of hardcoded in the backup.

I am more interested to know the cause for Error2 as this may be impacting some feature. Will wait to see if you find something around it.

I will update here in case I find anything relating to this

Regards,

Pankaj

Pankaj Pankaj.Negi

Yes though ERROR-1 show itself as a WARN message, I kind of like it, as an insight into from which peer / remote DataDSA was this local DataDSA restored. Would keep you posted if I find anything on ERROR-2.

Regards

Hubert

Hi Hubert HubertDennis

I guess these errors are due to Directory Management monitoring. Error frequency is 1/minute and the default interval at which the node server receives the request for charts or alerts information is also 1 minute.

When I log in to Directory Manager, I can't see anything in Health -> Charts. This is a bit strange because I could see the DSA operations graph from Charts initially.

Health - CA Directory - 14.0 - CA Technologies Documentation 

The documentation says that If you are using a Directory version which is older than 14.0, do the following before the DSAs can send monitoring information to the Management UI:

1. Generate Personality Certificates for the DSA.

2. Restart the DSA.

I am using CA Directory 14.0.

I did not find any content on how to generate DSA personality certificates. Is it necessary to do this?

Regards,

Pankaj

Pankaj Pankaj.Negi

One of the steps of adding a HOST (dxagent) to an ENVIRONMENT ( * using Directory Management UI ), needs certificates. However these certificates are generated by the install wizard and then used as input, whilst adding a HOST. From experience, if there is an issue with the certificates, then adding to HOST to ENVIRONMENT fails at Directory Management UI layer itself. However I'm assuming that is not the case here.

I mentioned earlier, did we enable detailed logging via DXConsole and investigate the request incoming to the DSA. That logging on console is far more detailed and in real time. We can change the tracing level temporarily using the DSA console. When the DSA is next restarted, the tracing returns to the level set in the configuration file. Open the DataDSA Console by SSH into the linux machine running the DSA instance [telnet localhost port]. Console Port is DSA running port + 1 e.g. if DSA port is 8881, console port will be 8882. 

Reference Link : 

Environments - CA Directory - 14.0 - CA Technologies Documentation 

set trace Command -- Define Trace Levels - CA Directory - 14.0 - CA Technologies Documentation 

trace Command - CA Directory - 14.0 - CA Technologies Documentation 

If we don't see insights even after tracing (I'd be surprised if we don't find any clue, tracing has always helped me), I'd say lets open a Support Case and get this clarified.

Regards

Hubert

Hi Hubert HubertDennis

I didn't find any error in the trace log as well. I will raise a case with CA support for clarification.

Event in trace log -

{ "dxserver-monitor": { "host-name": "hostname", "dsa-name": "DataSA2", "time": "20181111.132600.854Z", "message-id": 51, "stats" : { "anonymous-binds": 7, "simple-binds": 0, "strong-bi
nds": 0, "bind-security-errors": 0, "total-operations": 14, "compare-entry-operations": 0, "add-entry-operations": 0, "remove-entry-operations": 0, "modify-entry-operations": 7, "rename-entry-operations": 0, "list-operations": 0, "search-operations": 0, "one-level-searches": 0, "whole-subtree-searches": 0, "security-errors": 0, "operation-errors": 0
: [0] rsbuff_alloc: 0x7f1c680603c8 size 33000 used 1026
: [0] ssl_send 4864
: [0] lockCtx 4864
: [0] .. state 4 0
: [0] send_to_openssl 4864 (state 4)
: [0] ..calling OpenSSL for ssl=0
: [0] ssld_ssl_request(0, 2, 1026)
: [0] SSL_write(1026) = 1026
: [0] case *: BIO_read() --> 1055
: [0] ssld_ssl_response(4864, 1, 1055)
: [0] findCtx 4864
: [0] rsbuff_alloc: 0x7f1c58140738 size 33000 used 1055
: [0] comms_send 4864 58140738
: [0] => ctx 2504ef48 busy 1
: [0] Sending fd=19 buff=0x7f1c58140738 size=1055
: [0] rsbuff_free: 0x7f1c58140738 size 33000
: [0] ..called OpenSSL for ssl=0
: [0] unlockCtx 4864
: [0] .. state 4 0
: [0] rsbuff_free: 0x7f1c680603c8 size 33000
: [0] xm_freeTree(9595): 0x7f1c78063688
: [0] unbusy_netcon: 1 0
: [0] comms_sched: select returned 1 events
: [5] comms_worker in
: [5] comms_read 19
: [5] rsbuff_alloc: 0x7f1c58121c58 size 33000 used 33000
: [5] ssl_more_data 4864
: [5] lockCtx 4864
: [5] .. state 4 0
: [5] send_to_openssl 4864 (state 4)
: [5] ..calling OpenSSL for ssl=0
: [5] ssld_ssl_request(0, 1, 190)
: [5] ssld_ssl_read
: [5] Write 190 bytes to 25135fa0: returned 190
: [5] Read 5 bytes from 25135fa0: returned 5
: [5] Read 185 bytes from 25135fa0: returned 185
: [5] case 1: SSL_read() --> 161
: [5] ssld_ssl_response(4864, 2, 161)
: [5] findCtx 4864
: [5] ssl_data_in 4864
: [5] rsbuff_alloc: 0x7f1c58148868 size 33000 used 33000
: [5] http_more_data 4864 0x55672512d0a9
: [5] http_more_data: header len 149
: [5] http_more_data: Content-Length: 12
: [5] http_more_data: body len 12
: [5] http_event
? [5] 20181111.132600.898 WARN : HTTP send failure HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Users"
Date: Sun, 11 Nov 2018 13:26:00 GMT
Connection: keep-alive
Content-Length: 12
 

Regards,

Pankaj

Hi Hubert HubertDennis

Did you find anything around ERROR 2 -

[2] 20181108.121953.797 WARN : HTTP send failure HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="Users"
Date: Thu, 08 Nov 2018 12:19:53 GMT
Connection: keep-alive
Content-Length: 12

I raised a case with CA support 3 weeks back but still, there is no answer to it.

Thanks,

Pankaj

Hi Pankaj,

As you support case 01240225 is now on track (with engineering involved as well), do you mind closing out this thread?

Thanks,

Hitesh Patel

Broadcom Support

Hi Hitesh Hitesh_Patel

Yes this thread can be closed now

Thanks,

Pankaj

Experts,

I am trying to enable MW Replication among ODSEE and CA Dir 14.0. I am half way configuring it, below is the URL that i am following.

Phased Migration - CA Directory - 14.0 - CA Technologies Documentation 

I am kinda struck at the "Validate Replication" section within the above document. 

odsee instance, ca dir instance both on same box -- running as different instances.

1) enabled replication as mentioned in the above url.

2) Created an entry in CA Dir 14.0 and expecting it to reflect in Odsee. -- entry not seen yet in odsee

3) trying to run DXconsole and 'get dsp'. Not sure, how i can trigger it. -- inputs appreciated !