When trying to start vi as root using /etc/init.d/vdc start you notice a message in stdout:
touch: Unable to create file “/var/$filename” No space left on device
"Note: $filename could be any file"
There are two ways in which a filesystem can run out of space: it can consume all the space for adding new data (i.e., to existing files or to new files), or it can use up all the inodes. Both situations can cause the system to stop, because exhaustion of the inodes will prohibit the creation of additional files even if sufficient HDD space exists. It is particularly easy to run out of inodes if a filesystem contains a very large number of very small files.
Whenever you start VI, lock files needs to be written to the disk. If the script can't write VI won't start.
Check below commands that could be used to troubleshoot inodes usage in Linux environment.
df –i -> Checking the percentage of used inodes in a partition.
[root@dcimtrain ~]# df -hi /var
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/vg_centos64-lv_root
3.2M 3.2M 0M 100% /
The IUse% column shows the percentage of used inode. In this example, it shows 100%.
Although df -h shows plenty of available disk space available in the Vi server the application is unable to start.
find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -nr|head -10 -> This will dump a list of top 10 directory on the filesystem prefixed with the number of files (and subdirectories) in that directory. To make things simple, you can assume -> more files = more inodes used
[root@dcimtrain ~]# find / -xdev -printf '%h\n' | sort | uniq -c | sort -k 1 -nr|head -10
122822 /var/spool/postfix/maildrop
8993 /opt/VDC/ibuilder/logs
6454 /usr/local/pgsql/data/base/16384
6103 /usr/local/pgsql/data/base/4190477
5918 /usr/local/pgsql/data/base/1768489
3276 /usr/share/man/man3
3181 /usr/share/foomatic/db/source/printer
2180 /usr/lib64
2131 /usr/bin
1815 /usr/share/man/man1
In this example we can see /var/spool/postfix/maildrop contains many files. And it is the cause of our inode problem.
/var is a OS file system, so we have to be careful before erasing any content here. Especially if you are dealing with production environment.
/var/spool/postfix/maildrop is a file system that holds postfix mail queue. postfix is a sendmail alternative and it is the default mail server in centOs 6++.
The number of files in this directory can grow in some situations, for example, if a cronjob fails.
For this specific case, there is a cronjob in vdc user crontab that is failing to execute. When this happens the OS tries to send an e-mail to the user using postfix. As the receiver address is non-existent or was never set, the server won't be able to deliver this messages.
In order to clean the mail queue, we can use the command.
Postsuper –d ALL
This will delete the mail queue for all users. And will free up some inodes temporarily.
We still need to look into why this job is failing but i'll leave this for another document.
Summarizing:
Make sure your problem is not related to space.
Use df -hi to get an idea on the amount of free inodes you have for VI and OS file system.
Use the find command to find the top 10 directories with more files.
Look for 0KB files that could be erased. Be careful, make sure you know what you needs to be removed.
References:
Inode definition by The Linux Information Project (LINFO)
The Postfix Home Page