I have been administering Linux servers for a while now so coming across a new problem is both exciting and stressful, especially when that problem is on a high demand production server. I recently came across one of these scenarios, and was surprised how difficult it was to solve.
I had a single directory on a server that contained millions and millions of tiny 32Kb text files (which were actually non-sharded PHP session files that didn’t get caught in garbage collection, and went unnoticed for months on a busy server). The interim fix to that problem was easy:
mv /var/php/sessions /var/php/sessions.evil && mkdir /var/php/sessions
The real problem was that I was left with this sessions.evil directory that I couldn’t delete, ignore in snapshot backups, or even list of the contents of because there were just too many files and going anywhere near it would use too much disk I/O and cause the load on the server to spike.
After using / writing various Bash, Python, PHP, etc, scripts that caused too much load on the server I happened across this genius solution by Zhenyu Lee (and a comment by Paul Reiber)… to use rsync instead of rm, find, xargs, etc:
On a CentOS box and using my example directory of /var/php/sessions.evil, which was owned by root:apache do this:
cd /var/php
mkdir empty_dir
chown root:apache empty_dir
rsync -vvvv -a –delete empty_dir/ sessions.evil/
Depending on how many files are in your sessions.evil directory this could take a while (2 days in my case), but the 5 minute load average on the server stayed between a manageable 2.0 and 3.0. My twist to Zhenyu’s solution was to add some verbosity (-vvvv) in there so I could tell that rsync was actually doing something.
Paul’s next comments are important though, so once rsync is eventually done mirroring your sessions.evil directory with your empty_dir pay attention:
rmdir sessions.evil
mkdir sessions.evil
rmdir sessions.evil empty_dir
The first rmdir sessions.evil will cause a bit of a load average spike for a few minutes, but once it’s gone… whew.
Well there you go, happy rsyncing, and a huge thanks to Zhenyu Lee for posting that unique and brilliant solution.