Cleaning Up Is Hard To Do

Linux TuxI have been administering Linux servers for a while now so coming across a new problem is both exciting and stressful, especially when that problem is on a high demand production server. I recently came across one of these scenarios, and was surprised how difficult it was to solve.

I had a single directory on a server that contained millions and millions of tiny 32Kb text files (which were actually non-sharded PHP session files that didn’t get caught in garbage collection, and went unnoticed for months on a busy server). The interim fix to that problem was easy:

mv /var/php/sessions /var/php/sessions.evil && mkdir /var/php/sessions

The real problem was that I was left with this sessions.evil directory that I couldn’t delete, ignore in snapshot backups, or even list of the contents of because there were just too many files and going anywhere near it would use too much disk I/O and cause the load on the server to spike.

After using / writing various Bash, Python, PHP, etc, scripts that caused too much load on the server I happened across this genius solution by Zhenyu Lee (and a comment by Paul Reiber)… to use rsync instead of rm, find, xargs, etc:

On a CentOS box and using my example directory of /var/php/sessions.evil, which was owned by root:apache do this:

cd /var/php
mkdir empty_dir
chown root:apache empty_dir
rsync -vvvv -a –delete empty_dir/ sessions.evil/

Depending on how many files are in your sessions.evil directory this could take a while (2 days in my case), but the 5 minute load average on the server stayed between a manageable 2.0 and 3.0. My twist to Zhenyu’s solution was to add some verbosity (-vvvv) in there so I could tell that rsync was actually doing something.

Paul’s next comments are important though, so once rsync is eventually done mirroring your sessions.evil directory with your empty_dir pay attention:

rmdir sessions.evil
mkdir sessions.evil
rmdir sessions.evil empty_dir

The first rmdir sessions.evil will cause a bit of a load average spike for a few minutes, but once it’s gone… whew.

Well there you go, happy rsyncing, and a huge thanks to Zhenyu Lee for posting that unique and brilliant solution.

Cisco WRVS4400N Is Upsetting

A little over a year ago I decided it was time to replace my aging Linksys WRT54G wireless router in my home office with something new and fancy. I happened to be visiting a local computer shop one Saturday afternoon and came across the Cisco WRVS4400N Wireless-N Gigabit Security Router – VPN v2.0, which on the box sounded amazing. It was a little on the expensive side comparatively, but hey… it was Cisco, and I could setup VPN profiles on the device and easily connect remotely to my local network… so the extra money would be worth it.

Boy was I wrong. This device has never functioned even close to my expectations, despite running firmware version My biggest complaint is that I am finding the wireless connectivity terribly unreliable with frequent disconnects. My second largest issue is that the supported QuickVPN client is a sad sad Windows hunker, and the only way I can get the VPN connection working reliably on my Mac laptop is use to VPNTracker, which is beautiful but significantly more money that I am willing to pay for something that should work using Mac OS X’s built in L2TP over IPSEC or Cisco IPSEC connection. AFAIK the only other option for Mac users is to use IPSecuritas, but unfortunately it wasn’t working with Mac OS 10.7 at the time.

In summary (as I sit here connected to my Airport Express) I am far from impressed with the WRVS4400N, and this reaffirms that I should only purchase equipment after reading reviews not the other way around (as you may be now).

Good luck, and move on to something else. If you find something else awesome and worth it, please feel free to post it.

Find All WordPress Installations

I was trying to figure out how many WordPress installations lived on one of my hosting servers recently, and I also wanted to see how good the site owners are at keeping them up-to-date, which means I also wanted to find out the WordPress version number of each installation.

I couldn’t find any quick way of doing this so I turned the linux find command and grep to give me a hand. Here is the command (which must be run as a privileged user):

find /var/www/vhosts -type d -name "wp-includes" -print -exec grep "wp_version" {}/version.php \;

Just replace /var/www/vhosts with the root directory of where all of your domains are stored. Hope this helps someone. Cheers.

Updated: I switched the above command to use “-type d” vs “-d” as per James’ comment below. Thanks James.

Where The Heck Is Java Web Start?

So you have a from-the-browser Java application that requires Java Web Start to load on your beautiful Mac, and that Java application is named something crazy like:
viewer.jnlp([email protected]@idrac-BR2XXXX1,+PowerEdge+R710,[email protected])

What do you do? It doesn’t open automatically, and when you double click the filename it just asks you what application you want to use to open the file.

Well I will tell you what you do!

  1. On the “Launch Application” dialog that opens when you double click the filename, click “Choose…” beside “Choose an Application”.
  2. Navigate to Macintosh HD > System > Library > CoreServices.
    /System/Library/CoreServices/Java Web
  3. Choose “Java Web Start” and click Open.

There you go. A post like this would have saved me 15 minutes anyways, hope it helps someone else.

Sony Blu-Ray Players Don’t Play NetFlix in Canada

The bad newsI was somewhat frustrated this evening to find out that despite NetFlix being available in Canada now, Sony devices such my new Sony BDP-S370 blu-ray player will still not allow me to connect to NetFlix. So if you were holding out to find that perfect device to connect to the newly available NetFlix Canada service with, Sony devices are definitely not an option for you at this point. I’ve attached a Sony eSupport chat transcript for you to wallow over. Boooo, back to the store you go blu-ray.

Bitnami Redmine Stack

I ran across Bitnami today, and they have some really great software stacks with flexible install options (i.e. Native / VM / Cloud), including PowerPC support so I can put some of these older G5 Powermac’s into use again.

I downloaded the VM of their Bitnami Redmine Stack and it was so quick to get up and running, but here’s a tip to save others some frustration perhaps:

Despite the README.txt stating the default Redmine username and password is the username and password you entered during setup, it is actually (as of 0.9.4) Username: user | Password: bitnami

Because Everyone Has To Rant Once In A While

As a web developer in a graphic designers body I spend a lot of time making what I consider to be very usable and “pretty” interfaces, so I truly resent when Neophyte Bob arbitrarily and publicly posts something like “this interface is too techie” when what he actually mean is “this interface is not completely idiot proof”.

Come on Neophyte Bob it’s a blinkin’ select box, and by the way… stop using IE6.

Recent Spam Influx

If your inbox is at all like mine, you have all-of-a-sudden started to receive a heck of a lot more spam than you are used to in the past month or so. Have no fear, you are not alone. The Internet has recently experienced a large influx of spam e-mail thanks to Russian hackers who control a network of hacked PC’s some 70,000 computers strong.

I don’t normally post these messages; however, this affects so many people I want to make sure you are all aware of it. Please read the following article by eWeek for more information:,1895,2060235,00.asp

A final note: Windows users… does your computer have the “SpamThru” trojan?

E-Mail Scams Scram

Being a long time “netizen” I probably deal with more spam, scam and virus e-mails than anyone else I know. Thanks to SpamAssassin and Thunderbirds’ fantastic bayesian filtering techniques I luckily only see about 5-10 of these messages per day in my inbox, the rest (at least 100 – 200 messages per day) are being filtered through to my Junk folder and deleted.

Spam is one thing, but some of these damn scam e-mails are something much more diabolical. Now I’m not talking about the ridiculous Nigerian 411 scams that only work because of human greed… I’m talking about messages like:

Dear Matt [which just happens to be the first part of my e-mail address]
Yada Yada… this is notice to inform you that the credit card information that we have on file for your account will be expiring in the coming months, please log into our website and update your billing details. For your personal security, please type “” into your web-browser’s location window or click the following link [evil_link][/evil_link].

Real Name
Valid Company Inc.

Now being that long time experienced “netizen” I can spot these things (usually really quite easily because I use text-only e-mail)… and I know not to click on links in e-mails, period… but how am I supposed to explain that to unsuspecting friends, my father, etc? They look at me like I’m a nutzo paranoid crazy man if I tell them they can’t click links in e-mails at all, never ever, no matter what or who it’s from.

Here’s something that made me laugh today (again because of all my net-experience)… I received a virus e-mail that actually got me concerned (for a few seconds anyways). I actually called WorldPay before I did anything, just to confirm my suspicions and that this was infact a hoax message. It is honestly the first time I’ve ever given any significant number of seconds thought to one of these messages; therefore, it’s a valid share and besides Google has nothing on this yet:


My name is Dave and I am from the Support of WorldPay.

We have received the payment order (ID 0220712,Receipt Date 09/07/2006) from you and we need to make a verification of the details you have filled in, as we have received a notice from your card service stating that there was a chargeback made by the owner of the card with which you have made the payment and that your level of authorization has been altered during your last transaction.

This is a very serious matter. We have deducted the amount of the chargeback, GBP 149.89, from your account and added our standard fee of GBP 24.00 as well (you can see your payment details in the attachment).

We have failed to contact you using the telephone number you have provided earlier, meeting no response.

As a precaution, we have limited access to your account in order to protect against future unauthorized transactions.Please understand that this is a security measure intended to help protect you and your personal information.

Please contact your credit card company to resolve this matter.

Best Regards,
Dave Gollick
[email protected]

Yes there was some pour grammar in the message and true there was no “Hello [firstname] [lastname]”, but I deal with card processing quite a bit and I actually know WorldPay and their services, plus there were no links off-message… It was enough to get me thinking anyways. Of course the major tip off was the attachment. Why the heck would WorldPay attach a .zip file of the unauthorized transaction? Either way, that quick call to WorldPay confirmed that indeed it was a hoax. Question solved.

I’m actually not sure where I was going with this blog entry now… I was just annoyed and thought it would be fun to share… so yeah, I’m leaving now.

Ugly Coding Standards Are Standard?

I just read through the manual on the much anticipated Zend Framework and for the most part, I really like the implementation. It is a collection of a number of classes I already use from around the net, which is fine by me. True, PHP is in dire need of a good framework that everyone can standardize on. Right? Yeah, well, I think so, but I’m not really sure… Isn’t that what Pear is? (as a friend reminded me this morning).

My major problem is their stupid coding standards; I honestly hardly agree with a single one of them… it actually makes me angry. Here are a few that I read through until I got so mad I just closed the damn manual website and did something else (wrote this angry blog post).

B.2.2. Indentation
Use an indent of 4 spaces, with no tabs.

I despise spaces in code… why hit the space bar 4 times when you could hit tab 1 time? So what, you have different editors with different default tab stops… who cares, it’s a preference. I refuse to use spaces.

B.2.3. Maximum Line Length
The target line length is 80 characters, i.e. developers should aim keep code as close to the 80-column boundary as is practical. However, longer lines are acceptable. The maximum length of any line of PHP code is 120 characters.

No way, not in a million years. I like long lines… sure, I may have to scroll horizontally to see some code, but it is a hell of a lot simpler to read if you’re looking at the whole file or a large chunk of code.

B.3.3. Filenames
Files that are containers for single classes are derived from the class names (see above) in the format “ZClassName.php”.

Whatever… this is silly. Have they ever used an FTP client before? Some clients by default change filenames to lowercase when the upload files… like they should be. As far as I’m concerned filenames on the net should be a-z 0-9 – _ . period. If everything is in lowercase you never have to worry about case-sensitive operating systems.

B.3.5. Variables
Variable names may only contain alphanumeric characters. Underscores are not permitted. Numbers are permitted in variable names but are discouraged.

Whatever… again, I use $variable_name and refuse to use $variableName… If you have the words “some string” does it make more sense to say someString or does it look more accurate to say some_string. My opinion is the later. camelCaps is ugly in PHP code. I agree with their definition of Constants, but it leads to the question of… if underscores are permitted in constants, why not in variable names? If the answer is to clearly separate the two, then why shouldn’t variable names be all lowercase and constants be all upper case.

B.4.2.4. String Concatenation
Strings may be concatenated using the “.” operator. A space must always be added before and after the “.” operator to improve readability.

$whatever = “that is “.$tupid;

I just give up… I refuse to conform to that less-efficient “standard”. My own personal “standard” is almost the exact opposite of this Zend outline and in my opinion it is much nicer to read through and work from. I’m actually a Graphic Designer turned developer… my whole education and philosophy revolve around making things look good and stay functional… I personally think I’m qualified enough to say those standards stink.