DX NetOps

I've seen it done before, but i can't find it in the documentation anywhere. I don't want to restart my server, nor any of my existing stuff. I just want to startup the alarm notifier as specified in the alarmnotifier.idb parts list.

I thought i'd call processd.exe, but i can't remember the parameters and i don't trust that if i just run it without any parameters it would run safely (i.e. without messing up the already running process).

You can use the /bin/launchinstdbapp utility:

meaja05-F164217%/c/win32app/Spectrum/bin

./launchinstdbapp.exe

Usage:

  launchinstdbapp

       i.e.: launchinstdbapp hostname SS y VNM.OUT

Obviously change it for AlarmNotifier…

Cheers

Jay

# Here's a very specific example to stop and restart a notifier that is defined in your parts list:  (without having to restart spectrum or processd or anything else)  (assumes *nix environment)  (maybe someone else can help you on Windows!)

# Here is a notifier process that's been running since I last rebooted my spectroserver:

ps -ef |grep dba

spectrum   17520  4060  0 Jan12 ?        00:00:01 /usr/local/CA/spectrum/Notifier/rail_custom/pg_dba/AlarmNotifier_pg_dba

# I want to kill this one to stop these alerts from going out for a while - say during a planned maintenance event or whatever.

kill 17520

# Then, when I want to start up my notifier once the activity is over, this is the command to run to start it back up:

$SPECROOT/bin/launchinstdbapp vrhmonssprd01 AlarmNotifier_pg_dba n AlarmNotifier_pg_dba.OUT

# Then I go check the AlarmNotifier_pg_dba.OUT file to see that it is running again...

# I think that's what you were looking for...  It took me a while to figure this out back in the day, too!

HTH,

Chris

One additional point for you to keep in mind if you go to script something like this...  if you have several notifiers you want to stop and restart, make sure you start them back up one at a time and give them a few seconds between start ups...  I have tried starting up a bunch all at once and sometimes it works.  But, I have seen conditions where a few of the notifiers were not able to connect to <whatever they connect to, I don't recall exactly>.  When you see a message in the log file that it couldn't connect to something, that means your notifier is running (you'll see it in your process tree), but it's not actually doing anything!  You'll want to check for this condition.  I haven't had this issue since I began starting multiple notifiers individually.

Chris

Hi Stuart,

You can simply do a processd restart via:

lib/SDPM/processd.pl restart

(in a bash -login as the root (unix) or Administrative user (win) not the Spectrum owner user).

All the .idb files get re-read. Processd will now start up any processes with valid .idb files, and that have  AUTOBOOTSTART set to Y, that are currently not running and should be.  Any currently running ones (e.g. your SS, ARCHMGR etc) will be entirely unaffected. Just an easy alternative to the (let's face it!) somewhat arcane launchinstdbapp and cmdC commands if you are working manually at the command line trying to get a new Notifier working properly!

You can kill the Notifier process or task whenever you want to stop it, then  do the above when you want to start it up again.

Cheers

Dan.

...so for this - you need the ALARMNOTIFIER.idb  - or whatever you have called yours - configured with AUTOBOOTSTART;Y and AUTORESTART;N

Cheers

Dan.

In a standard 10.0 install:

[root@o2usspr01 partslist]# pwd

/opt/spectrum/lib/SDPM/partslist

[root@o2usspr01 partslist]# cat ALARMNOTIFIER.idb

# Processd Install Ticket for AlarmNotifier

PARTNAME;ALARMNOTIFIER;

APPNAME;AlarmNotifier;

WORKPATH;$SPECROOT/Notifier;

LOGNAMEPATH;$WORKPATH/NOTIFIER.OUT;

#ADMINPRIVS;N;

AUTORESTART;y;

AUTOBOOTSTART;n;

#STATEBASED;N;

NUMPROCS;1; // one per host

RETRYTIMEOUT;600; // 10 minutes

TICKETUSER;specadmin;

RETRYMAX;3;    // 3 retries allowed

STARTPRIORITY;30;

SERVERPROCESS;Y;

#ENV;<var>=<value>;

ARGV;$SPECROOT/Notifier/AlarmNotifier<CSEXE>;

[root@o2usspr01 partslist]#

As Dan mentioned, you need to set:

AUTOBOOTSTART;y;

AUTORESTART;n;

AUTOBOOTSTART set to 'y' tells the process daemon to start the process when processd starts - and to kill it when it shuts down.

Setting AUTORESTART to 'n' is just so it doesn't try restart it if the process fails.

Note the STARTPRIORITY. This is used when starting up / shutting down the process. The lower numbers start up before the higher numbers. So e.g. the value of 30 here means that Alarm Notifier will start AFTER the SpectroSERVER process (set to 20).

When setting up the IDB files, you need to make sure the start-up priorities are set correctly. If you set these up incorrectly you can get db corruption. E.g. when you try to shutdown spectrum while alarm notifier is running.

When processd shuts down it also does the shutdown procedure in reverse - shutting down the processes with the higher STARTPRIORITY before shutting down the lower ones.

This is incorrect, Processd --restart will not just start any .idb files, and that have  AUTOBOOTSTART set to Y.

You have to use the launchinstdbapp utility for that.

Regards,

Dencel

Hi Dencel,

When I do:

processd.pl restart

in a bash -login prompt it has always worked for me  (- for causing the start up of any AUTOBOOTSTART processes that are not currently running and ought to be).  Providing processd is already running and I'm logged in as a windows administrator, (like 'root' on Unix) rather than the Spectrum install owner.

I agree with you that if one tries to do the same thing using the processd.exe directly by means of processd.exe --restart

in the way you mention (instead of by running the .pl)  - it does not work and gives an error.  For reasons I never tried to understand.

I agree one needs to use that (strange!) launchinstdbapp interface for starting processes that are not set to AUTOBOOTSTART.

Cheers

Dan

Dear all, 

it truly appears the current CA Spectrum documentation is not fully covering the details around manual "processd" managed services. So - adding here some more details. 

The processd service can handle sub-services - and is controlled by *.idb files under ./lib/SDPM/partlist. 

Once the processd service (i.e. via Windows-Services / or via shell - per ./lib/SDPM/processd.pl start) is started it scans all files named *.idb under ./lib/SDPM/partlist - and will start those having "AUTOBOOTSTART" set to "y". 

But - due to administrative needs you may have to stop and start the socalled "tickets" manually. 

The tools to manage and advise "processd" service to do so - are: 

   ./bin/launchinstdbapp   - for ticket/service start

   ./bin/cmdC   - for managing already running tickets

By default the following files/tickets will exist per default:

   [spectro@labspec02 partslist]$ ls
   ALARMNOTIFIER.idb       MYSQL.idb          RCPD.idb             SS.idb
   ARCHMGR.idb                  NAMINGSERVICE.idb       REXCON.idb TELNETD.idb
   LOCSERV.idb             NCMSERV.idb          SSDBLOAD.idb          TOMCAT.idb
   MTE.idb                PVIEW.idb                SSDBSAVE.idb                VNMSHD.idb

This is for an all_in_one install - and using OC-server / or SpectroSERVER_only install you will 

find less. But - in any case - you may add your additional "processd" managed service ticket files.

Potentially - you can configure them to cover AUTOBOOTSTART;y;  - but you can keep them also

on "n"/no - and starting them manually. 

Use ./bin/launchinstdbapp  for this - do so per Spectrum install owner from a shell.

Usage:

         launchinstdbapp <hostname> <install partname> <autorestart> <logname>

Sample:

       ./bin/launchinstdbapp localhost  AN22  Y  /usr/spectrum/AlarmNotifier22.OUT

Note - the "logname"/logfile have to be a full specified path/file-name - to ensure the proper directory.

The "hostname" set to "localhost" should work in case name resolution is fine for this "local host". 

To "manage" the running "tickets" - use the ./bin/cmdC command - good view is per: 

     ./bin/cmdC  localhost  6   - then i.e. filtered for "part_name"

Sample:

[spectro@labspec02 bin]$ ./cmdC localhost 6 | grep -i part_n
ticket part_name = NAMINGSERVICE
ticket part_name = TELNETD
ticket part_name = MYSQL
ticket part_name = LOCSERV
ticket part_name = NCMSERV
ticket part_name = TOMCAT
ticket part_name = SS
ticket part_name = REXCON
ticket part_name = ARCHMGR

Now - to "stop" one "ticket" - just run:  ./bin/cmdC localhost 2  AN22  (see above sample).

hth, Joerg 

Best explanation i've seen so far. Now I might actually be able to setup multiple alarmnotifier tickets.  One question:

ticket = part = sub-process? 

You used those three terms in different parts of your explanation and it seems that you're referring to the same thing.

Hi Stuart - right / but better here few more words about our internal jargon.

The "ticket" refers to the "./lib/SDPM/partlist -> "ticket_file_name".idb

The "part_name" is the name how the processd-service handles this started/spawned task. (it is not truly a sub-service - nor a thread). 

in a Linux runtime - you see "parent process-id" which is then the "processd" - and the process-ID of the started "ticket"/sub-process/service. As the initial "tickets" for Location-Server, Namingserver and mysql or ncmservice are "sub-services" we had started to talk about "sub-services". Clearly these are standalone tasks doing an IP-related communication. 

So: 

- processd configuration is done via "ticket" files (*.idb)

- processd internal management and output will report "managed" "tickets" having 

  parameter per ticket. 

See output here:

[spectro@labspec02 bin]$ ./cmdC localhost 6 | more
DATE OF THIS REQUEST

ticket function id = 266
ticket cb function id = 256
ticket exchange id = 3

...

ticket part_name = NAMINGSERVICE
ticket application_name = Visibroker Naming Service
ticket work_pathname = /home/spec9/bin/VBNS
ticket log_name = /home/spec9/bin/VBNS/NAMINGSERVICE.OUT

So - you see the "ticket" stands for "this managed instance".   Sorry for the confusion. - Joerg -

P.S: As all is managed by "processd" it is required to have many "parameter" being unique - 

so the processd can proper handle them. Also the "output"/logname have to be unique (due to open_output file options / it will not allow to share one file across multiple tickets).

Right, either way, ticket, part, sub-service, sub-process, etc. are all names for the same thing or names of pointers that point to the same thing.  For example, if i were to startup a second AlarmNotifier "instance", I'd have a new part/process/ticket/sub-service/sub-process.

I think the slight nuance in the names doesn't really matter to most people. When a nuance is slight, it makes better design to simplify the structure/convention so that the nuance doesn't have to be understood to be able to consume the design.  The nuance may be why the documentation does such a poor job of explaining it (and is why I've waited over 18 months before even approaching the subject of setting up a second, third, fourth AN instance).

Thanks for the clarification. CA, if you're reading this, please add documentation like this to your product. 

I have followed a rule of thumb that I create a new alarm notifier for each separate team that needs alerting.  That way I can turn on/off alerting for just that team specifically without disturbing the notifications for other teams simply by stopping their notifier.  Plus, this helps me simplify the event rules and other things that look into the great big bucket of everything and need to send alerts to folks. 

I don't know if this helps you or not, but I tried doing all of my notifications in a single notifier in the beginning.  I found that to be more convoluted than it was worth, so I started separating things out by teams or organizations.  Much more simple.  And, once you found the trick to stopping/starting them separately with their "ticket" files, life is good.

That mirrors my experience exactly, except that i've been procrastinating separating out the notifier processes because the concepts of separate AN instances was too unclear for me to feel confident starting that process. I have much less of an excuse to put it off now.

Hi Joerg

I have been unable to start a new processd sub-service using launchinstdbapp as you describe. In the past I have used Dan White's suggestion to use processd.pl --restart and this has worked for me.  I have just tried again now to add a new sub-service using launchinstdbapp and get the following.

spectrum]$ ./bin/launchinstdbapp localhost MYNOTIFIER y /opt/spectrum/Notifier/MYNOTIFIER.OUT
Problem with extracting / launching the MYNOTIFIER install ticket:

No such file or directory:

spectrum]$

For info the MYNOTIFIER.idb file exists and looks good to me. 

Are you saying that this should work for a new, properly configured .idb file? Or is there some step I am missing?

Regards, John

John,

What is the contents of you MYNOTIFIER.idb file?

Joe

Hi Joe

Here it is...

PARTNAME;MYNOTIFIER;
APPNAME;MYNotifier;
WORKPATH;$SPECROOT/Notifier;
LOGNAMEPATH;$WORKPATH/MYNOTIFIER.OUT;
#ADMINPRIVS;N;
AUTORESTART;Y;
AUTOBOOTSTART;Y;
#STATEBASED;N;
NUMPROCS;1; // one per host
RETRYTIMEOUT;600; // 10 minutes
TICKETUSER;specadmin;
RETRYMAX;3; // 3 retries allowed
STARTPRIORITY;30;
SERVERPROCESS;Y;
#ENV;<var>=<value>;
ARGV;$SPECROOT/Notifier/AlarmNotifier<CSEXE> -r .myalarmrc;

John

Hi John

Maybe try ./bin/launchinstdbapp localhost MYNOTIFIER y MYNOTIFIER.OUT

Regards, 

Jay V

Hi Jay

I had alrerady tried that and get the same thing.

John

Instead of using localhost, try using the hostname of the system

Joe

Hi

I get the same error.

For info I have tried with the full path to the OUT file and also just using MYNOTIFIER.OUT.

John

I found the root cause of the error. 

I placed processd in debug mode. When I run launchinstdbapp, I see the following in the processd_log file:

ticket part_name = MYNOTIFIER
ticket application_name =
ticket work_pathname =
ticket log_name = /app/spectrum/Notifier/MYNOTIFIER.OUT

For some reason, it is not picking up the ticket application_name or work_pathname from the idb file.

Joe

And do you have a solution... 

Or do I revert to running "processd.pl --restart" to get it going?

John

No solution at this time. Looks like it may be a bug. Use the processd.pl -restart 

Joe

Hi,

I thought parts had to be registered with processd first, before launchinstdbapp could be used to start it manually?

i.e. that processd would have had to be restarted,  to get it to read in this new idb and register it.