DX NetOps

After successful installation of primary and secondary spectroServers not able to see the status of secondary server in oneclick console. For this we need to do any other configuration in primary and secondary.

And also not able to see secondary spectro server in  Connection Status icon

Followed below steps all are done. 

1. Install Primary 
2. Install Secondary with same precedance as parimary.Edit the .hosts file on both servers for access. Edit the .locrc file on secondary and make sure that the main location server is the name of the primary server.While installing secondary server make sure you give the primary server name in the MLS field.Once done with installation edit the .vnmrc file on the secondary and add secondary_polling=yes .Save the file.This way we are configuring the FT in hot standby.
3. To check if the FT is successful run ./MapUpdate -v on primary and secondary server and check if both the servers are reflected in both the output.
bash -l ->cd SS-Tools -> ./MapUpdate -v
3. Now discover models on Primary. 
4. Make sure Archmgr is stopped on the secondary 
5. Backup your Primary SSDB
6. Load the same on secondary using this command
i) Copy the SSDB into SS directory on Secondary server.
ii) Go to bash -l -> cd SS -> ../SS-Tools /SSdbload -a 20 -il <Primary SSDB name>
7. Once done start the seocndary server.

8.To verify the setup, use the MapUpdate command with the view argument to display the current landscape map.

Hello,

In item 2, did you add the OC hostname in the $SPECROOT/.hostrc file of the secondary SS?

Can you share the output of "MapUpdate.exe -v"?

Thanks,

Silvio

It is Linux environment 

[ge@****** SS-Tools]$ ./MapUpdate -v
Connecting to Location Server running at: IXXXXXB port 0xdaff

Landscape Service Type Name Hostname Port Precedence Notes
---------- ------------------ ------------------------------ ------ ---------- ------------------
0x800000 Events iXXXXXb 0xbafe 10 1M model mask
0x800000 Landscape Default iXXXXXb 0xbeef 10 1M model mask
0x800000 Landscape iXXXXXb 0xbeef 10 1M model mask

In .Hostrc file we added primary and secondary host name only (no ip's)

rebooted servers started both.

It was supposed to have this:

0x800000 Events iXXXXXb 0xbafe 10 1M model mask
0x800000 Landscape Default iXXXXXb 0xbeef 10 1M model mask
0x800000 Landscape iXXXXXb 0xbeef 10 1M model mask

0x800000 Events iXXXXXy 0xbafe 20 1M model mask
0x800000 Landscape Default iXXXXXy 0xbeef 20 1M model mask
0x800000 Landscape iXXXXXy 0xbeef 20 1M model mask

The secondary SS is not visible.

What is the problem for this? Why i am not getting  these , what i need to do for this.

Please help me on this.

0x800000 Events iXXXXXy 0xbafe 20 1M model mask
0x800000 Landscape Default iXXXXXy 0xbeef 20 1M model mask
0x800000 Landscape iXXXXXy 0xbeef 20 1M model mask

iXXXXXb this is secondary server output.

0x800000 Events iXXXXXb 0xbafe 10 1M model mask
0x800000 Landscape Default iXXXXXb 0xbeef 10 1M model mask
0x800000 Landscape iXXXXXb 0xbeef 10 1M model mask

this is primary server out put 

[ge@IXXXXXP SS-Tools]$ ./MapUpdate -v
Connecting to Location Server running at: IXXXXXP port 0xdaff

Landscape Service Type Name Hostname Port Precedence Notes
---------- ------------------ ------------------------------ ------ ---------- ------------------
0x800000 Events iXXXXXp 0xbafe 10 1M model mask
0x800000 Landscape Default iXXXXXp 0xbeef 10 1M model mask
0x800000 Landscape iXXXXXXp 0xbeef 10 1M model mask

****************************************

>>>>while loding SSDB got this out put.

Reading new catalog from  /usr/Spectrum/SS/SS-DB-Backup/db_20171211_1505.SSdb ...

        Number of model types read : 6,281

        Number of relations read : 250

    Validating new catalog...

    Writing new catalog to the database.

       Please be patient...

    Loading new models into the database.

        Number of models loaded : 249

    The database now indicates these landscapes loaded on these servers:

        Landscape   Hostname          Port    Precedence

        ----------  ----------------  ------  ----------

        0x800000    iXXXXXp      0xbeef  10

        0x800000    iXXXXXb     0x0     20

SSdbload completed successfully.

Maybe you have not followed the correct instruction to load the SSdb on the secondary SS.

Did you load the SSdb from the $SPECROOT/SS directory?

[spectrum@Spectrum]# cd SS
[spectrum@SS] ../SS-Tools/SSdbload -il a 20 db_20171211_1505.SSdb

As Silvio noted make sure you ran the SSdbload command  from the SS directory and not the SS-Tools directory.  Other items to check:

User – make sure the user you are running the SS is the same on both SpectroSERVERs

$SPECROOT/.hostrc – make sure it has just a + symbol or machine names

$SPECROOT/LS/.locrc – on the backup SS make sure the MAIN_LOCATION_HOST_NAME=  entry is set to the name of the primary SS.  If it isn’t, set it to that and kill the LocServer process (processd will automatically restart it).

Cheers

Jay

iXXXXXp = Primary

iXXXXXb = Secondary.

I got below out put is is correct ?

For secondary:

[geetika@IXXXXXB SS-Tools]$ ./MapUpdate -v
Connecting to Location Server running at: ICB port 0xdaff

Landscape Service Type Name Hostname Port Precedence Notes
---------- ------------------ ------------------------------ ------ ---------- ------------------
0x800000 Events iXXXXXb 0xbafe 20 1M model mask
0x800000 Landscape Default iXXXXb 0xbeef 20 1M model mask
0x800000 Landscape iXXXXXb 0xbeef 20 1M model mask
0x800000 Landscape iXXXXXp 0xbeef 10 1M model mask

For Primary:

[geetika@IXXXXXP SS-Tools]$ ./MapUpdate -v
Connecting to Location Server running at: IXXXXXP port 0xdaff

Landscape Service Type Name Hostname Port Precedence Notes
---------- ------------------ ------------------------------ ------ ---------- ------------------
0x800000 Landscape Default iXXXXXp 0xbeef 10 1M model mask
0x800000 Landscape iXXXXXp 0xbeef 10 1M model mask

But still not able to see the status of secondary server in oneclick console

 And also not able to see secondary spectro server in  Connection Status icon

In first query having screen shots.

It is expected to see six entries in the MapUpdate -v output, three for the primary and three for the secondary SpectroSERVER.

Make sure you added the OC hostname and the secondary SpectroSERVER hostname in the $SPECROOT/.hostrc file.

Example:

On the primary SpectroSERVER, in the SPECROOT/.hostrc file:

iXXXXXp  --> (primary SS)

iXXXXXb  --> (secondary SS)

iXXXXXc  --> (OneClick web server)

On the secondary SpectroSERVER, in the SPECROOT/.hostrc file:

iXXXXXp  --> (primary SS)

iXXXXXb  --> (secondary SS)

iXXXXXc  --> (OneClick web server)

Then stop the SpectroSERVER application and stop the Spectrum Process Daemon. Start the Spectrum Process Daemon and start the SpectroSERVER application.

Thanks,

Silvio

The landscape map should be the same on both fault tolerant server, so this is not correct.  Setting up a fault tolerant environment and adding a secondary SpectroSERVER, always works when all the steps have been followed as documented.  However, it is easy to run one of those steps, in a manner slightly different than required and documented.

e.g. As Jason and Silvio mentioned, please retry paying particular to the following:

Make sure you ran the SSdbload command  from the SS directory and not the SS-Tools directory. 

User – make sure the user you are running the SS is the same on both SpectroSERVERs

$SPECROOT/.hostrc – make sure it has just a + symbol or machine names

$SPECROOT/LS/.locrc – on the backup SS make sure the MAIN_LOCATION_HOST_NAME=  entry is set to the name of the primary SS.  If it isn’t, set it to that and kill the LocServer process (processd will automatically restart it).

If that does not work, please open an issue with support and we can resolve this more quickly, as it is difficult to read the map, with the hostnames removed like this. 

regards

Shane

What is the reason i am not getting 6 entrees in primary and secondary. Any other process to follow.

In both SPECROOT/.hostrc files we already have   primary SS,secondary SS host names.

If i want to re lode SSDB data to secondary what prerequisites i need to follow.  

Why the OC hostname was not added into the hostrc file?

Have you cycled the SpectroSERVER and Process Daemon service?

Is there any firewall enabled? Check the ports as per the following docs:

SpectroSERVER and OneClick Web Server Communication Across Firewalls - CA Spectrum - 10.2 to 10.2.2 - CA Technologies Do… 

Primary and Secondary SpectroSERVER Communication Across Firewalls - CA Spectrum - 10.2 to 10.2.2 - CA Technologies Docu… 

Why the OC hostname was not added into the hostrc file?

We are using only 2   servers 

one is ***116 as primary and OneClick web server 

two is ***119 as secondary so we added both servers host names in .hostrc file.

Have you cycled the SpectroSERVER and Process Daemon service?

Means need to restart SpectroSERVER right, How we ca do this Process Daemon service?

Is there any firewall enabled? Check the ports as per the following docs

For this we stopped  iptables in primary and secondary. So we will not get port or firewall issues.

iptables: Firewall is not running.

To stop/start the Spectrum Process Daemon, navigate to the $SPECROOT/lib/SDPM/ directory, then run:

./processd.pl stop

./processd.pl start

Run telnet to make sure you can connect to the required ports.

Did This step :stop the SpectroSERVER application and stop the Spectrum Process Daemon. Start the Spectrum Process Daemon and start the SpectroSERVER application

But same out put in Mapupdate -v 

And checked all ports all are ok.  only two ports are not open

Primary to secondary 51966 port is not open

[root@IXXXXP SDPM]# telnet XXXXX119 51966
Trying XXXX119...
telnet: connect to address XXXX119: Connection refused
telnet: Unable to connect to remote host: Connection refused

secondary to primary 47870 is not open.
[root@IXXXXB SDPM]# telnet XXXXX116 47870
Trying XXXXX116...
telnet: connect to address XXXX116: Connection refused
telnet: Unable to connect to remote host: Connection refused

Port 51966 (0xcafe) is used by rcpd (Remote Copy Process Daemon) when the SSdb database synchronization is in progress. So, no worries about this port if OLB is not in progress.

Check on the primary SS is it is listening on port 47870 (0xbafe). Run telnet locally (on the primary SS) to see if you connect on port 47870.

Also run the netstat -ano and check if the connection is established on the port used by Spectrum. 

47870 port is not listening in primary server 

[geetika@IXXXXP SS-Tools]$ telnet XXXX116 47870
Trying XXXX116...
telnet: connect to address XXXX116: Connection refused
telnet: Unable to connect to remote host: Connection refused

For this 47870 port reason i am not getting six entrees in primary and secondary?

If i want to re lode SSDB data to secondary what prerequisites i need to follow.?

But still not able to see the status of secondary server in oneclick console.

What is the main reason for this  what i need to do?

What is the output of netstat? Is the machine listening on por 47870?

netstat -an | grep -i listening

netstat -an | grep 47870

Archive Manager uses the port 47870. 

Are you sure the Archive Manager is up and running?

Check the $SPECROOT/SS/DDM/ARCHMGR.OUT file.

If the primary SSdb already has the secondary SS entry, just perform the regular SSdb load. Can be either in SCP (Spectrum Control Panel), or CLI.

../SS-Tools/SSdbload -il db_xxxxx

Does the OLB (OnLineBackup) synchronize the SSdb from primary to secondary SS?

47870 port is now listening on both primary and secondary we started arch manager in primary.

[geetika@IXXXXP SS-Tools]$netstat -an | grep 47870 is 

tcp 0 0 0.0.0.0:47870 0.0.0.0:* LISTEN

Does the OLB (OnLineBackup) synchronize the SSdb from primary to secondary SS? Where we can do this pls tell me .

There are two ways to manually trigger the OLB:

a) In the SCP (Spectrum Control Panel)

b) On the VNM model - OneClick Console

Make sure "Primary backup successful. Database synchronization in progress" message was displayed in the Status field.

During the Database synchronization, a copy of SSdb is transferred from the primary to the secondary, then the secondary SS is stopped, the SSdb database is loaded and then the SS is restarted.

Got this error what we need to do?

Check the Events and/or Alarms for this failure.

What is the Alarms Cause Code? (in the Alarms TAB, add the Cause Code column)

What are the OnLineBackup events? (In the Locater TAB, expand the Application Models folder, double-click on By Name and type: OnLineBackup. You should see the OnLineBackup application model in the Results TAB. Select it and then click on Events TAB in Component Detail pane)

Alarms Cause Code is 0x10521 

We don't have any OnLineBackup events.

Below is the alarm symptoms probable cause details 

Please help me on this what we need to do next.

Based on the 0x10521 Cause Code, this problem may be caused by the exhaustion of some system resource.  For example, the machine on which the SpectroSERVER is running may be low on real memory and/or configured swap space.

Has this ever worked before?

How did you make the SSdb backup to copy to the secondary SS?

Did you check the following?

a) Free disk space in  /usr, the minimum required disk space is 20 MB based on your settings.

b) How much memory do you have on the primary SS?

How did you make the SSdb backup to copy to the secondary SS?

We took SSDB primary backup from SCP (Spectrum Control Panel). before that SCP was working fine when we took SSDB backup now it's not taking backup.

b) How much memory do you have on the primary SS?

[geetika@IXXXXP usr]$ free -m
total used free shared buffers cached
Mem: 5853 5043 809 1 58 1907
-/+ buffers/cache: 3077 2775
Swap: 14499 39 14460

a) Free disk space in  /usr

[geetika@IXXXXP ~]$ df -h /usr/
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 95G 28G 62G 31% /

To further investigate this issue we need to gather additional log files which will expose sensitive data. I would recommend to open a case/ticket at CA support (http://support.ca.com)

Thank you very much for your help Silvio Okamoto.

We will check with  CA support.

Hi,

Any document or link for this, how to open a ticket at CA support.

How to raise a support case in the new Support Online portal 

You can also call CA Support to open a support case:

Contact CA Support - CA Technologies 

Hi Please help me on this.

Not able to stop spectro server. When we tried to stop using below command it is running only processes is not stopping. what is the problem ?

[geetika@IXXXXXP bin]$ ./stopSS.pl
Archive Manager is not running
Stopping SpectroSERVER

Hi

Could you go to %SPECROOT%/SS directory and share the output of:

tail VNM.OUT