CA Spectrum Tuesday Tip by Scott Orzechowski, Principal Support Engineer, December 5, 2011
Description:
Some Spectrum client/server or server/server communication uses UDP, which is connectionless. That means that packets are sent without confirmation that they got there. One example is the Corba connection between a SpectroSERVER and a OneClick server.
Other ports are connection based, meaning that each end waits for confirmation from the other port that the data was received. If it doesn’t receive confirmation from the other port, the sending port will wait before sending more data.
For the TCP protocol, the procedure is called a handshake. Ports on TCP sockets can back up and eventually close. The operating system will gradually shrink the size of the windows over which it will send TCP for each port when data queues up. The sending process will ultimately initiate the procedure that closes the socket, thereby ending communication between the client and server.
Sockets back up when one or both of the ports is slow to acknowledge the packets from the other side. Usually, this is due to either poor connectivity between the client and server (like a slow network path) or an unresponsive process (a busy SpectroSERVER process is a common cause).
Check the send queue (Send-Q) and receive queue (Recv-Q) (not shown in the Windows netstat command) to see whether messages are backing up. On Solaris, run “netstat –an” and redirect it to a file. You can then view the file in an editor to make it easier to find Spectrum ports (listed in the Communication Across Firewalls section of Chapter 3: Communication Across Firewalls of the Distributed SpectroSERVER Environment, in the Distributed SpectroSERVER Administrator Guide). Note that the netstat output for each socket spans two lines, so if you grep for a port, you will only see part of the data. Here’s a sample of the columns:
TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -----------
*.48879 *.* 0 0 49152 0 LISTEN
*.* *.* 0 0 49152 0 IDLE
On Linux, run “netstat –anp” This will show the process name or process ID for the port. On Windows, run “netstat –anb”
When messages on a port get back up, the operating system can shrink the send window for that port, and you could experience poor performance. Check the send window (Swind) and receive window (Rwind) columns. These columns will only be visible on Solaris and Linux.
Check the state to see whether the socket is ESTABLISHED, or whether one side has initiated the handshake that closes the socket (CLOSE-WAIT, TIME-WAIT, FIN-WAIT, etc.). A port that is in any state other than ESTABLISHED or LISTENING is in some phase of tearing down the socket. Sometimes improperly stopped client processes will result in an orphaned port (only one end of the socket remains). Sometimes you will see many iterations of ports in CLOSED_WAIT or FIN_WAIT state. These sometimes correspond with defunct AlarmNotifer client processes that were stopped ungracefully. A reboot will clean these up.
For more information and the definitions of the TCP port states, do a man on netstat on Solaris or Linux. There are plenty of resources on the web that explain the TCP handshake more thoroughly. One such example (which CA is not endorsing) is http://en.wikipedia.org/wiki/Transmission_Control_Protocol.