Lost some boxcars

Pushed a new site live today and I got a call this evening from my VP, she was getting a 500 error on the site. There is no way this site is getting to much traffic and it works just fine when I pull it up from home. I start log hunting and make my way to the Rails log:

/usr/lib/ruby/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/bin/mongrel_rails:16:in `load’
/usr/bin/mongrel_rails:16
/!\ FAILSAFE /!\  Wed Oct 08 19:33:05 -0500 2008
Status: 500 Internal Server Error
IP spoofing attack?!
HTTP_CLIENT_IP=”1.2.3.4″
HTTP_X_FORWARDED_FOR=”1.2.3.4, 4.5.6.7, 10.168.1.81″

Awesome, so it’s Rails that’s tossing an error, at least we know what is up now. I send out an email to the code gurus and get a quick response back pointing out this site (because I’m in a company of freaks that likes to stay up late, read work email and figure out problems… it’s good kind of freak). The answer is that RoR is pissy in later versions and if your HTTP_CLIENT_IP header differs from your HTTP_X_FORWARDED_FOR header it going to put the breaks on for you and throw a 500 error. Unfortunately were working in an environment with a loadbalancer as well a Apache/mod_proxy in from of mongrel and this will happen to us a lot. The solution is to add

RequestHeader unset Client-IP

to you VirtualHost config and make sure you have mod_headers enabled. At this point it should clear your HTTP_CLIENT_IP and stop the error.

Now I can go back to eating my chili.