I received an alert notifying me that the /opt on one of our Soalris 10 boxes was 100% full:
# df -h /opt
Filesystem size used avail capacity Mounted on
/dev/dsk/c0t0d0s5 32G 8192E 0K 27983662414130% /opt
I was surprised that the numbers on “used” and “capacity” were surprisingly huge. I checked out exactly how much space had been used on /opt:
# du -sh /opt
1.9G /opt
Also there were still plenty of inode capacity:
# df -o i /opt
Filesystem iused ifree %iused Mounted on
/dev/dsk/c0t0d0s5 17543 3989369 0% /opt
lsof output showed that httpd processes were still hanging on to some lingering disk space after the corresponding files were removed:
# /usr/local/bin/lsof -a +L1 /opt
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
httpd 18129 root 24w VREG 32,13 0 0 9701 /opt (/dev/dsk/c0t0d0s5)
httpd 18129 root 30w VREG 32,13 0 0 10099 /opt (/dev/dsk/c0t0d0s5)
httpd 18133 mobimg 24w VREG 32,13 0 0 9701 /opt (/dev/dsk/c0t0d0s5)
httpd 18134 mobimg 24w VREG 32,13 0 0 9701 /opt (/dev/dsk/c0t0d0s5)
...
But the “SIZE/OFF” all showed up as 0. I had some ideas: either restart the httpd processes or fsck /opt. But since this is a production box, I couldn’t try things out without coordinating a downtime. I spent some time on Sunsolve trying to see if this was a known problem or bug but didn’t find anything useful. I might not hit the right keyword (and it turned out that I didn’t).
So as the last resort, I sent a message to Sunmanagers:
http://www.sunmanagers.org/pipermail/sunmanagers/2006-December/042648.html
I got a couple of replies and most of them suggested the inconsistency between df(1M) and du(1M) might be caused by some running processes which were still hanging on to the lingering disk space after the corresponding file(s) was/were removed. A few sulutions were suggested:
1) restart the https processes
2) fsck /opt
3) simply reboot
While one reply pointed right to the heart of the problem: this is a bug of Solaris 10 and the bug id is 1-26-102285-1. The Sun Alert notification is here:
http://sunsolve.sun.com/search/document.do?assetkey=1-26-102285-1
Sunmanagers is always the right place to go if you are desperately seeking help :)
