DX NetOps

While we are trying to start archmgr getting below error?

What is the problem what we need to do for this ?

[geetika@IXXXXXP DDM]$ ./ArchMgr
Dec 20 15:58:50 : ArchMgr started as user 'geetika'
Dec 20 15:58:50 : ArchMgr validating database.
Error opening the ArchMgr database.

A lockfile exists indicating that another process is using the database.

Database already locked by: geetika,
by process: ArchMgr,
by process ID: 15785,
on network node: IXXXXXP,
which started at: 12/20/2017_15:54:45.

You must shut the other process down before running this program.
If the indicated process is no longer running, then you can
remove the .DDMDB.LOCK file and restart this process.
Dec 20 15:58:50 : ArchMgr error opening database.

Hi,

Looks Archive Manager is running, what does the file ARCHMRG.OUT (location %SPECROOT%/SS/DDM) say?

sorry, a typo in my previous update, what does ARCHMGR.OUT say (location %SPECROOT%/SS/DDM)?

Not able to stop spectro server. When we tried to stop using below command it is running only processes is not stopping. what is the problem ?

[geetika@IXXXXXP bin]$ ./stopSS.pl
Archive Manager is not running
Stopping SpectroSERVER

Hi

Could you go to %SPECROOT%/SS directory and share the output of:

tail VNM.OUT

vi VNM.OUT

Dec 12 19:41:53 : /usr/Spectrum/SS/SpectroSERVER starting
**************************************************************************

CA Technologies
One CA Plaza
Islandia, NY 11749 USA
Copyright (c) 2017 CA Technologies
All Rights Reserved

This software is protected by Copyright law and international treaties.

Unauthorized reproduction or distribution of this software may result in
civil and criminal penalties.

This software incorporates the following issued U.S. patents
and foreign patents, other patents pending:
5,261,044; 5,295,244; 5,436,909; 5,504,921;
5,559,955; 5,590,120; 5,649,103; 5,666,481;
5,696,486; 5,706,436; 5,734,642; 5,748,781;
5,751,933; 5,751,965; 5,768,501; 5,777,549;
5,793,362; 5,812,750; 5,822,305; 5,832,503;
5,872,928; 5,907,696; 6,000,045; 6,014,697;
6,049,828; 6,057,757; 6,064,304; 6,115,362;
6,131,112; 6,205,563; 6,233,623; 6,243,747;
6,255,943; 6,374,293; 6,373,383; 6,430,712;
682272 AU; 681972 AU; 69132279.1 DE; 0737920 FR;
0737920 GB; 685335 AU; 69132280.5 DE; 0737921 FR;
0737921 GB; 711060 AU; 69608107.5 DE; 0838056 FR;
0838056 GB; 674572 AU; 720061 AU; 700957 AU;
699509 AU; 69611542.5 DE; 0865698 FR; 0865698 GB;
702607 AU; 715918 AU; 720079 AU.

Subject to CA Spectrum license agreements enclosed with this software.

This notice shall be marked on any reproduction of this computer software,
in whole or in part.

**************************************************************************
Version 10.2.0.000

Dec 12 19:41:54 : Validating and loading SPECTRUM Intelligence Modules.
Dec 12 19:41:57 : All SPECTRUM Intelligence Modules successfully loaded.

Dec 12 19:41:57 : Please wait. SpectroSERVER
is loading landscape 0x800000 at precedence 10...

Number of models loaded : 0 ^M Number of models loaded : 254

Dec 12 19:42:03 : /usr/Spectrum/SS/SpectroSERVER
is now ready on port 0xbeef...

Dec 13 15:20:26 : SpectroSERVER has received shut down signal - scheduling shut down
Dec 13 15:20:26 : /usr/Spectrum/SS/SpectroSERVER is shutting down...
Dec 13 15:20:27 : Closing all client connections...
Dec 13 15:20:27 : Stopping CORBA communication...
"VNM.OUT" [readonly] 70L, 2833C

Looks you would need to load a good known database, have a look at the below:

https://support.ca.com/us/knowledge-base-articles.tec1348589.html

Why so? Wouldn't it be enough to move the lockfile and do an ArchMGR repair? 

[EDIT]

Overread that the SpectroSERVER is not stopping.  

If i kill spectro server pid it will work?

Need to load old SSDB ? and  how?

Hi Sudheerreddy

If the SpectroSERVER will not shut down and you have to stop it with a Kill -9 command, then likely you will have to load a previous database. You can do this with Spectrum Control Panel, or by using SSdbload command. See info here: 

Load a Database - CA Spectrum - 10.1 to 10.1.2 - CA Technologies Documentation 

Before killing the SpectroSERVER process, Collect all VNM.OUT and VNM.OUT.bak files and save off to a safe location for review. You can also try to collect some data on the process in its hung state. You would do this by running a "pstack" command on the process PID: 

pstack <PID> >> pstack.out

Run this command a few times. I.e.

pstack <PID> >> pstack1.out

pstack <PID> >> pstack2.out

etc

More info on pstack: https://linux.die.net/man/1/pstack 

You can run the pstack command on the SpectroSERVER process, ArchMgr process, any process which may be in hung state (e.g. oneclickservice process, etc.)  - collect the pstack.out files and open a case with Support for review. 

(For Windows you can collect data on the hung process with MS procmon ( Process Monitor - Windows Sysinternals | Microsoft Docs )

You can also check OS Application Logs for any clues or messages to the hung process state:

Linux OS - check /var/logs directory

Or check process status in 

/proc/[pid]/task/[thread ids]/status
How to check if a process is in hang state (Linux) - Stack Overflow 

Windows OS - Event Viewer

HTH

~Jay V

Hi All,

We killed spectro server pid and tried to start spectro server. But we are getting below logs with 12,13 th dates it's not getting today dates.

And please see  last 13 th dec date logs it's showing like /usr/Spectrum/SS/SpectroSERVER has successfully shut down. how to start this

even we are not able to login oneclick console.

please help me on this.

Dec 12 19:41:53 : /usr/Spectrum/SS/SpectroSERVER starting
**************************************************************************

CA Technologies
One CA Plaza
Islandia, NY 11749 USA
Copyright (c) 2017 CA Technologies
All Rights Reserved

This software is protected by Copyright law and international treaties.

Unauthorized reproduction or distribution of this software may result in
civil and criminal penalties.

This software incorporates the following issued U.S. patents
and foreign patents, other patents pending:
5,261,044; 5,295,244; 5,436,909; 5,504,921;
5,559,955; 5,590,120; 5,649,103; 5,666,481;
5,696,486; 5,706,436; 5,734,642; 5,748,781;
5,751,933; 5,751,965; 5,768,501; 5,777,549;
5,793,362; 5,812,750; 5,822,305; 5,832,503;
5,872,928; 5,907,696; 6,000,045; 6,014,697;
6,049,828; 6,057,757; 6,064,304; 6,115,362;
6,131,112; 6,205,563; 6,233,623; 6,243,747;
6,255,943; 6,374,293; 6,373,383; 6,430,712;
682272 AU; 681972 AU; 69132279.1 DE; 0737920 FR;
0737920 GB; 685335 AU; 69132280.5 DE; 0737921 FR;
0737921 GB; 711060 AU; 69608107.5 DE; 0838056 FR;
0838056 GB; 674572 AU; 720061 AU; 700957 AU;
699509 AU; 69611542.5 DE; 0865698 FR; 0865698 GB;
702607 AU; 715918 AU; 720079 AU.

Subject to CA Spectrum license agreements enclosed with this software.

This notice shall be marked on any reproduction of this computer software,
in whole or in part.

**************************************************************************
Version 10.2.0.000

Dec 12 19:41:54 : Validating and loading SPECTRUM Intelligence Modules.
Dec 12 19:41:57 : All SPECTRUM Intelligence Modules successfully loaded.

Dec 12 19:41:57 : Please wait. SpectroSERVER
is loading landscape 0x800000 at precedence 10...

Number of models loaded : 0 Number of models loaded : 254

Dec 12 19:42:03 : /usr/Spectrum/SS/SpectroSERVER
is now ready on port 0xbeef...

Dec 13 15:20:26 : SpectroSERVER has received shut down signal - scheduling shut down
Dec 13 15:20:26 : /usr/Spectrum/SS/SpectroSERVER is shutting down...
Dec 13 15:20:27 : Closing all client connections...
Dec 13 15:20:27 : Stopping CORBA communication...
Dec 13 15:20:29 : Stopping /usr/Spectrum/SS/SpectroSERVER activity...

----- NOTE --------------------------------------------------------------
CA Technologies recommends that the SpectroSERVER be
allowed to complete the shutdown process.
Database corruption may result if the SpectroSERVER is prematurely stopped.
------------------------------------------------------------- NOTE ------

Dec 13 15:20:29 : waiting for model activates to complete...
Dec 13 15:20:29 : waiting for model destroys to complete...
Dec 13 15:20:29 : Closing /usr/Spectrum/SS/SpectroSERVER database...
Dec 13 15:20:29 : /usr/Spectrum/SS/SpectroSERVER has successfully shut down.

What are the permissions for files in the <SPECROOT>/SS directory? 

What user did you start the SpectroSERVER with? You have to start/stop the server with your Spectrum Admin user. Do not start the server with ROOT. That will cause permissions problems. 

~Jay V

Hi,

SS and SpectroSERVER both having same users.

drwxrwxrwx 7 geetika geetika 4096 Dec 21 18:27 SS-

rwxrwxrwx 1 geetika geetika 12841 Oct 13 17:22 SpectroSERVER

and same errors we are getting wile we are staring SpectroSERVER

Dec 13 15:20:26 : /usr/Spectrum/SS/SpectroSERVER is shutting down...
Dec 13 15:20:27 : Closing all client connections...
Dec 13 15:20:27 : Stopping CORBA communication...
Dec 13 15:20:29 : Stopping /usr/Spectrum/SS/SpectroSERVER activity...

----- NOTE --------------------------------------------------------------
CA Technologies recommends that the SpectroSERVER be
allowed to complete the shutdown process.
Database corruption may result if the SpectroSERVER is prematurely stopped.
------------------------------------------------------------- NOTE ------

Dec 13 15:20:29 : waiting for model activates to complete...
Dec 13 15:20:29 : waiting for model destroys to complete...
Dec 13 15:20:29 : Closing /usr/Spectrum/SS/SpectroSERVER database...
Dec 13 15:20:29 : /usr/Spectrum/SS/SpectroSERVER has successfully shut down.

Hello Sudheer,

Can you try to initialize the Spectroserver using legacy database. Please make sure, not to use Root user. Once the Spectroserver is initialized, load the last good back. 

----- NOTE --------------------------------------------------------------
CA Technologies recommends that the SpectroSERVER be
allowed to complete the shutdown process.
Database corruption may result if the SpectroSERVER is prematurely stopped.

I think the SSDB is crashed as you have killed the Spectroserver process to stop spectrum, and that is the reason, Spectrum in not getting started. 

Please try to run the following command from the "$SPECROOT/SS" directory to initialize the database to the legacy:

../SS-Tools/SSdbload -il legacy.SSdb

After this try to load the last good know SSDB backup.

HTH

Amit

How  to load the last good know SSDB backup. any command what process i need to follow.

./SS-Tools/SSdbload -il legacy.SSdb   done this 

The specified save file: legacy.SSdb ,
was created with the following options:
-c (catalog)

***Warning***

You have chosen the '-i' option. This option is used to initialize
the database. This means ALL data in the database will be deleted. If
you choose to continue with this option, you must be prepared to load
your modeling catalog (model types and relations), before starting the
SpectroSERVER.

Do you really want to initialize the database? (Y or N): Y

Initializing database...

Reading new catalog from legacy.SSdb ...
Number of model types read : 6,278
Number of relations read : 250

Validating new catalog...

Writing new catalog to the database.
Please be patient...

SSdbload completed successfully.

1. Open a command line (Solaris/Linux) or 'bash -login' shell (Windows) as the Spectrum Owner account.
2. cd $SPECROOT/SS directory.
3. Run ../SS-Tools/SSdbload -il <your_backup_filename>
    The -il options indicate we are initializing the files and loading new files from the legacy database.
 
    If your backup file is in another directory, such as $SPECROOT/SS-DB-Backup, you will need to specify the path to the file.
    For example: ../SS-Tools/SSdbload -il ../SS-DB-Backup/<your_backup_filename>
 
4. Once the load completes restart Spectrum.

Ok thank you.

But we are having multiple .SSDB files which one we need to lode and all are same names dates are different.

or we need lode .gz files after compression?

Try loading the db_20171211_1501.SSdb

Why like this ?

[geetika@IXXXXP SS]$ ../SS-Tools/SSdbload -il ../SS-DB-Backup/db_20171211_1501.SSdb
SPC-DBS-23007: This save file cannot be loaded by version 10.2.0.000 of Spectrum.
It was saved as version loaded landscape address {0} .

Looks database is corrupted - would you mind opening a ticket with us please?

what ever i used command is correct? pls check

[geetika@IXXXXP SS]$ ../SS-Tools/SSdbload -il ../SS-DB-Backup/db_20171211_1501.SSdb

Looks good

Can u see below command 

[geetika@IXXXXXP SS]$ ../SS-Tools/SSdbload -il ../usr/Spectrum/SS/SS-DB-Backup/db_20171211_1501.SSdb
SPC-DBS-20081: Error: Unable to open specified save file: ../usr/Spectrum/SS-DB-Backup/db_20171211_1501.SSdb

Looking into the error you have mention:

SPC-DBS-23007

I doubt if you are loading the DB of older version. Did you try a models only load ? This will load only the models from your db into Spectrum. But you need to initialize the current db before doing it In this case all the changes done to the current catalog from MTE will be lost.

Please try to load the database using below command with Spectrum install owner:

cd $SPECROOT/SS directory.
Run ../SS-Tools/SSdbload -il -m ../SS-DB-Backup/db_20171211_1501.SSdb

Hi,

Getting below options means i need to give "models" in the place of -m in the command ? what i need to do?

is below command is correct?

[geetika@IXXXXP SS]$ ../SS-Tools/SSdbload -il -m ../SS-DB-Backup/db_20171211_1501.SSdb
SSdbload: The -load flag may not be used with -catalog or -models.

Usage: SSdbload [-quiet] [-initialize] [-developer <DEV_INFO_FILE>]
[-load] [-models] [-catalog] [-replace <PRECEDENCE>]
[-add <PRECEDENCE>] [-port <PORT_NO>] [-showmap]
[-version] [-extension] [-new_primary <HOST_NAME>]
[-UpgradeFrom <ENCODING_NAME>] [-TestEncoding]
[<SAVE_FILE>]

Options/Arguments:

-quiet disable interactive mode
-initialize initialize the database
-developer <DEV_INFO_FILE> load specified developer info
file
-load load all objects from SAVE_FILE
-models load models from SAVE_FILE
-catalog load modeling catalog
-replace <PRECEDENCE> replace landscape map entry
-add <PRECEDENCE> add landscape map entry
-port <PORT_NO> port number for landscape map
entry
-showmap print landscape map info
-version print database version
-extension disable file extension enforcing
-new_primary <HOST_NAME> set local landscape model name
-UpgradeFrom <ENCODING_NAME> Name of the Encoding used to
store non-UTF8 data in
SpectroServer DB
-TestEncoding Check if there are attribute
values that are not in UTF-8
encoding
<SAVE_FILE> file from which to load

I will suggest you to inatilaize the Spectrum by Spectrum Control Panel itself, and then load the database with model only option. Below will be the command:

[geetika@IXXXXP SS]$ ../SS-Tools/SSdbload -m ../SS-DB-Backup/db_20171211_1501.SSdb

HTH

Amit

Getting below error

[geetika@IXXXXX SS]$ ../SS-Tools/SSdbload -m ../SS-DB-Backup/db_20171211_1501.SSdb

SPC-DBS-20053:
Error: The file to be loaded: ../SS-DB-Backup/db_20171211_1501.SSdb ,
does not contain any models.

Please verify that the load options are consistent with the types of data in
the save file. The database has not been changed.

Can you confirm, if you have initialize the SS from SCP, and after that you are loading the SSDB from command line. Make sure, you are not using root user. 

Also I will suggest you to try loading another SSDB with the above given command.

I am not using root user. tried another 3 SSDB's but getting same errors.

Can you confirm, if you have initialize the SS from SCP?

this is done from command line.

./SS-Tools/SSdbload -il legacy.SSdb   done this 

The specified save file: legacy.SSdb ,
was created with the following options:
-c (catalog)

***Warning***

You have chosen the '-i' option. This option is used to initialize
the database. This means ALL data in the database will be deleted. If
you choose to continue with this option, you must be prepared to load
your modeling catalog (model types and relations), before starting the
SpectroSERVER.

Do you really want to initialize the database? (Y or N): Y

Initializing database...

Reading new catalog from legacy.SSdb ...
Number of model types read : 6,278
Number of relations read : 250

Validating new catalog...

Writing new catalog to the database.
Please be patient...

SSdbload completed successfully.

I see the last line says that SSdbload completed successfully, after this you have load good know backup. Can you share the screen shot of the error you are facing.

For few files it is notable to open and for few files  does not contain any models.