Chandru_V

Did you know? lsof (list of open files) command on Linux

Blog Post created by Chandru_V Employee on Nov 24, 2015

You may be familiar the /usr/sbin/lsof command on Linux systems. I’ve known this one since a few years. However, until recently I’ve used it to just do what the command’s name says list open files (lsof).

 

As you may already know, UNIX and UNIX-like systems treat every object as a file. Devices, directories, pipers, sockets, inodes, all of them.

 

In this sense, the lsof command can be used to get far more diagnostic information from our customers.

 

I am listing a few purposes I find this command serves, as below:

 

Simply typing lsof will provide a list of all open files belonging to all active processes.

 

# lsof | head -10

COMMAND     PID      USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME

init          1      root  cwd       DIR              253,0       4096          2 /

init          1      root  rtd       DIR              253,0       4096          2 /

init          1      root  txt       REG              253,0     150352    1314508 /sbin/init

init          1      root  mem       REG              253,0      65928    2883613 /lib64/libnss_files-2.12.so

init          1      root  mem       REG              253,0    1926800    2884051 /lib64/libc-2.12.so

init          1      root  mem       REG              253,0      93320    2884065 /lib64/libgcc_s-4.4.7-20120601.so.1

init          1      root  mem       REG              253,0      47064    2884054 /lib64/librt-2.12.so

init          1      root  mem       REG              253,0     145896    2884053 /lib64/libpthread-2.12.so

init          1      root  mem       REG              253,0     268232    2884055 /lib64/libdbus-1.so.3.4.0

 

(I’ve taken just the top 10 lines [head -10] for brevity)

 

By default One file per line is displayed. Most of the columns are self-explanatory. We will see the details about couple of columns (FD and TYPE).

 

FD – Represents the file descriptor. Some of the values of FDs are,

·    cwd – Current Working Directory

·    txt – Text file

·    mem – Memory mapped file

·    mmap – Memory mapped device

·    NUMBER – Represent the actual file descriptor. The character after the number i.e ’1u’, represents the mode in which the file is opened. r for read, w for write, u for read and write.

 

TYPE – Specifies the type of the file. Some of the values of TYPEs are,

·    REG – Regular File

·    DIR – Directory

·    FIFO – First In First Out

·    CHR – Character special file

 

For a complete list of FD & TYPE, refer man lsof.

 

Now, let’s move on to some cool examples…

 

1.   List processes which opened a specific file

You can list only the processes which opened a specific file, by providing the filename as arguments.

 

Example:

 

# lsof /var/log/messages

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME

rsyslogd 1643 root    1w   REG  253,0     3909 3017037 /var/log/messages

 

2.    List opened files under a directory

You can list the processes which opened files under a specified directory (recursively) using ‘+D’ option. If you don’t want to do recursive, then ‘+d’ option is your friend.

 

Example:

 

# lsof +d /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/

COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME

cybAgent. 7443 root  cwd    DIR  253,0     4096 1321830 /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT

cybAgent. 7443 root  txt    REG  253,0   689912 1322102 /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/cybAgent.bin

cybAgent. 7443 root  mem    REG  253,0    12105 1322098 /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/libfilefilter.so

cybAgent. 7443 root    1w   REG  253,0        0 1322129 /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/nohup.stdout

cybAgent. 7443 root    2w   REG  253,0        0 1322130 /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/nohup.stderr

cybAgent. 7443 root    3uW  REG  253,0        5 1318191 /opt/CA/WorkloadAutomationAE/SystemAgent/WA_AGENT/.lock

 

Sometime when we apply a fix/patch, it fails with an error that the file or directory is in use. We need to find out what are all the processes using that directory and kill those processes.

By using lsof we can find those processes.

 

3.    List opened files based on process names (that start with)

·    You can list the files opened by process names starting with a string, using ‘-c’ option.

·    -c followed by the process name will list the files opened by the process starting with that processes name.

·    You can use multiple -c switch on a single command line (works as OR operator)

 

Example:

The following command lists all files currently held/open by CA’s license process (lic98fds)

 

# lsof -c lic98

COMMAND    PID    USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME

lic98fds 10987 autosys  cwd    DIR              253,0     4096  1321469 /opt/CA/WorkloadAutomationAE/autouser.PRD

lic98fds 10987 autosys  rtd    DIR              253,0     4096        2 /

lic98fds 10987 autosys  txt    REG              253,0   156682  1190235 /opt/CA/SharedComponents/ca_lic/lic98fds

lic98fds 10987 autosys  mem    REG              253,0   142536  3149427 /lib/ld-2.12.so

lic98fds 10987 autosys  mem    REG              253,0  1910572  3149428 /lib/libc-2.12.so

lic98fds 10987 autosys    0r   CHR                1,3      0t0     3698 /dev/null

lic98fds 10987 autosys    1w   CHR                1,3      0t0     3698 /dev/null

lic98fds 10987 autosys    2u  unix 0xffff8800328d8080      0t0 31133642 /tmp/CALIC98-FD-SERVER

 

4.    List files opened by a specific user

 

Example:

Listing files opened by CA (eTrust) Directory user dsa (only bottom 10, for brevity)

 

# lsof -u dsa | tail -10

dxserver 5607  dsa   10u  IPv4              18698       0t0     TCP testserver.ca.com:snare (LISTEN)

dxserver 5607  dsa   11u  unix 0xffff880139d41380       0t0   18700 socket

dxserver 5607  dsa   12u  IPv4              19581       0t0     TCP testserver.ca.com:snare->testserver.ca.com:36746 (ESTABLISHED)

dxserver 5607  dsa   13r   CHR                1,9       0t0    3703 /dev/urandom

dxserver 5607  dsa   14u  IPv4              19586       0t0     TCP testserver.ca.com:snare->testserver.ca.com:36747 (ESTABLISHED)

dxserver 5607  dsa   15u  IPv4           31293438       0t0     TCP testserver.ca.com:snare->testserver.ca.com:26178 (ESTABLISHED)

dxserver 5607  dsa   16u  IPv4           31293520       0t0     TCP testserver.ca.com:snare->testserver.ca.com:26181 (ESTABLISHED)

dxserver 5607  dsa   17u  IPv4           31296281       0t0     TCP testserver.ca.com:snare->testserver.ca.com:26348 (ESTABLISHED)

dxserver 5607  dsa   18u  IPv4           15394232       0t0     TCP testserver.ca.com:snare->testserver.ca.com:9398 (ESTABLISHED)

dxserver 5607  dsa   19u  IPv4          123487647       0t0     TCP testserver.ca.com:snare->testserver.ca.com:55893 (ESTABLISHED)

 

And this one is for file open by all users, except dsa user

 

# lsof -u ^dsa

 

            (Output skipped, too long)

 

            The –t option can be used to get just the process ID (PID) list:

            Example:

          

# lsof -t -u autosys

7585

7786

10987

 

Tip: What to know what processes are currently owned by a specific user? Just type ps –u <userid>

 

Example:

 

# ps -u autosys

  PID TTY          TIME CMD

7585 ?        07:35:49 as_server

7786 ?        03:34:01 event_demon

10987 ?        00:03:50 lic98fds

 

The –r option can be used to run the lsof in repeat mode. Meaning, it loops through the same command at interval specified. It can be interrupted with a CTRL+C key combination.

          

Example:

            This command lists all process IDs owned by autosys indefinitely at 5 seconds intervals, until interrupted (notice the ^C at the bottom of the output, that is CTRL+C)

          

# lsof -t -u autosys -r5

7585

7786

10987

=======

7585

7786

10987

=======

7585

7786

10987

=======

^C

 

5.   List all open files by a specific process

You can list all the files opened by a specific process using ‘-p’ option.

This is extremely useful to find out what files specific process is holding on to.

 

Example:

Let’s first get the process ID for the CA (Dylan) Socket Adapter process csampmuxf

 

# ps -ef|grep csampmux | grep -v grep

root     12693     1  0 Apr07 ?        02:12:41 /opt/CA/SharedComponents/Csam/SockAdapter/bin/csampmuxf

 

Next, let’s ask lsof what files are open for process ID 12693.

 

# lsof -p 12693 | tail -15

csampmuxf 12693 root    2r   CHR                1,9      0t0      3703 /dev/urandom

csampmuxf 12693 root    3u  IPv4           59309649      0t0       TCP *:cacsambroker (LISTEN)

csampmuxf 12693 root    4u  unix 0xffff880040dd4980      0t0  59309651 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root    5u  unix 0xffff88013bc62980      0t0 100823963 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root    6u  unix 0xffff8800b96a86c0      0t0 100823965 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root    7u  unix 0xffff88013b46e3c0      0t0 100825952 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root    8u  unix 0xffff8801014d03c0      0t0 100825987 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root    9u  unix 0xffff880139808080      0t0 120822917 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root   11u  unix 0xffff8800385cac80      0t0 120822921 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root   12u  unix 0xffff8800b956b680      0t0 120824424 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root   13u  unix 0xffff88010307a9c0      0t0 120822923 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root   14u  unix 0xffff8801106099c0      0t0  59314819 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root   15u  unix 0xffff8801106090c0      0t0 120824429 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root   16u  unix 0xffff8801014d09c0      0t0 100826163 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

csampmuxf 12693 root   18u  unix 0xffff8800b2ae03c0      0t0 126316250 /opt/CA/SharedComponents/Csam/SockAdapter/.csam_broker

 

Want more? There is much more. It just gets better….

 

6.   List all network connections

·    You can list all the network connections opened by using ‘-i’ option.

·    You can also use ‘-i4′ or ‘-i6′ to list only ‘IPV4′ or ‘IPV6‘ respectively.

 

Examples:

 

a.    List all network connection opened by the Oracle TNS listener

 

# lsof -i -a -c tnslsnr

COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

tnslsnr 31190 oracle   11u  IPv6  74297      0t0  TCP *:ncube-lm (LISTEN)

tnslsnr 31190 oracle   13u  IPv6  74623      0t0  TCP testserver.ca.com:ncube-lm->testserver.ca.com:44678 (ESTABLISHED)

 

b.    List all network connections opened by PID

 

# lsof -i -a -p 5607

COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

dxserver 5607  dsa    8u  IPv6     18630      0t0  UDP *:snare

dxserver 5607  dsa    9u  IPv4     18697      0t0  TCP testserver.ca.com:snare (LISTEN)

dxserver 5607  dsa   10u  IPv4     18698      0t0  TCP testserver.ca.com:snare (LISTEN)

dxserver 5607  dsa   12u  IPv4     19581      0t0  TCP testserver.ca.com:snare->testserver.ca.com:36746 (ESTABLISHED)

dxserver 5607  dsa   14u  IPv4     19586      0t0  TCP testserver.ca.com:snare->testserver.ca.com:36747 (ESTABLISHED)

dxserver 5607  dsa   15u  IPv4  31293438      0t0  TCP testserver.ca.com:snare->testserver.ca.com:26178 (ESTABLISHED)

dxserver 5607  dsa   16u  IPv4  31293520      0t0  TCP testserver.ca.com:snare->testserver.ca.com:26181 (ESTABLISHED)

dxserver 5607  dsa   17u  IPv4  31296281      0t0  TCP testserver.ca.com:snare->testserver.ca.com:26348 (ESTABLISHED)

dxserver 5607  dsa   18u  IPv4  15394232      0t0  TCP testserver.ca.com:snare->testserver.ca.com:9398 (ESTABLISHED)

dxserver 5607  dsa   19u  IPv4 123487647      0t0  TCP testserver.ca.com:snare->testserver.ca.com:55893 (ESTABLISHED)

 

c.    List all open network connections on a specific port.

 

# lsof -i :22

COMMAND   PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME

sshd     2657 root    3u  IPv4 138591901      0t0  TCP testserver.ca.com:ssh->vench06-e7440.ca.com:65094 (ESTABLISHED)

sshd     5137 root    3u  IPv4     17643      0t0  TCP *:ssh (LISTEN)

sshd     5137 root    4u  IPv6     17647      0t0  TCP *:ssh (LISTEN)

sshd    32403 root    3u  IPv4 119929825      0t0  TCP testserver.ca.com:ssh->testserver.ca.com:52897 (ESTABLISHED)

 

d.    List all TCP network connections for a specific process name

 

# lsof -i tcp -a -c csampmuxf

COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME

csampmuxf 12693 root    3u  IPv4 59309649      0t0  TCP *:cacsambroker (LISTEN)

 

e.    List all UDP connections on the server.

 

# lsof -i udp

COMMAND     PID    USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME

portreser  1636    root    5u  IPv4    12140      0t0  UDP *:ipp

rpcbind    1686     rpc    6u  IPv4    12372      0t0  UDP *:sunrpc

rpcbind    1686     rpc    7u  IPv4    12374      0t0  UDP *:1013

rpcbind    1686     rpc    9u  IPv6    12377      0t0  UDP *:sunrpc

rpcbind    1686     rpc   10u  IPv6    12379      0t0  UDP *:1013

rpc.statd  1794 rpcuser    5u  IPv4    12709      0t0  UDP *:olsr

rpc.statd  1794 rpcuser    8u  IPv4    12713      0t0  UDP *:61893

rpc.statd  1794 rpcuser   10u  IPv6    12721      0t0  UDP *:35001

dxserver   5607     dsa    8u  IPv6    18630      0t0  UDP *:snare

gdm-binar  5857    root    7u  IPv4    19496      0t0  UDP *:xdmcp

dhclient  29299    root    6u  IPv4    67386      0t0  UDP *:bootpc

oracle    30759  oracle   17u  IPv6    71848      0t0  UDP [::1]:12328

oracle    30779  oracle   31u  IPv6 33964308      0t0  UDP *:18963

oracle    30787  oracle   24u  IPv6    72962      0t0  UDP *:40853

oracle    30791  oracle   16u  IPv6    72021      0t0  UDP [::1]:40753

oracle    30793  oracle   16u  IPv6    72040      0t0  UDP [::1]:13978

 

I hope you find this one a tad helpful in your troubleshooting process.

 

Happy exploring!

 

Cheers,

Chandru

Outcomes