Rails 1.1.6, backports, and full disclosure
Posted by David August 10, 2006 @ 05:34 PM
The cat is out of the bag, so here’s the full disclosure edition of the current security vulnerability. With Rails 1.1.0 through 1.1.5 (minus the short-lived 1.1.3), you can trigger the evaluation of Ruby code through the URL because of a bug in the routing code of Rails. This means that you can essentially take down a Rails process by starting something like /script/profiler, as the code will run for a long time and that process will be hung while it happens. Other URLs can even cause data loss.
We’ve backported a fix to all the affected versions for those of you that can’t update. You’ll have to apply the diff for your version:
- Patch for Rails 1.1.0
- Patch for Rails 1.1.1
- Patch for Rails 1.1.2
- Patch for Rails 1.1.4
- Patch for Rails 1.1.5: Upgrade to Rails 1.1.6.
These patches (and 1.1.6) will break applications using the 3rd party engines idea. So if you can’t upgrade because of dependencies to those, you can also add the following URL blocking while engines are being updated. Here’s how to do it with mod_rewrite under Apache:
RewriteRule ^(app|components|config|db|doc|lib|log|public|script|test|tmp|vendor)/ - [F]
Here’s how to do it under lighttpd:
url.rewrite-once = ( "^/(app|components|config|db|doc|lib|log|public|script|test|tmp|vendor)/" => "index.html" )
Unfortunately, the 1.1.5 update from yesterday only partly closed the hole (getting rid of the worst data loss trigger). After learning more about the extent of the problem, we’ve now put together a 1.1.6 release that completely closes all elements of the hole (using the same technique as the backports above).
So if you upgraded to 1.1.5 yesterday, you need to upgrade again. The approach stays the same, but since the Rubyforge gem server can be very slow at distributing gem updates, you should grab this fix straight from the Rails server:
sudo gem install rails --source http://gems.rubyonrails.org --include-dependencies
If you’re running of trunk (also known as edge) using revision 4394 or later, you’re not affected by all this in any form.
We’ll follow up with more information as it becomes available. Needless to say, this is all the Rails core team is working on right now and we’ve recruited a whole band of testers to help us play this out. We’ll make sure to evaluate all the feedback that’s been coming in and develop some scar tissue a policy for dealing with security issues in the future. Thanks for your continued understanding.
We’ve also started #rails-security on Freenet for people with IRC available to get and share more information.
UPDATE: If you’re floating on gems (don’t have vendor/rails), then make sure you update RAILS_GEM_VERSION in your config/environment.rb. Otherwise you’ll still be bound to that earlier version of Rails even as you install the new gems.
UPDATE 2: Rails 1.1.6 is now available on the official gem server, so you no longer need to add the—source http://gems.rubyonrails.org parameter.
Good choice going ahead and providing patches for the previous versions. Just wanted to say again that we do appreciate the considerable amount of work and effort you all are putting into fixing this issue.
Should I just wait ‘till tomorrow to upgrade to 1.1.7 or should I take the prescription and upgrade today?
Do Rails apps need to add a 404 route to block /breakpoint_client in production or does the latest release protect against this behavior?
In the spirit of “full disclosure” what were the additional holes that warranted a 1.1.6?
You are the ones, guys.
I want to give you a recommendation: please, try to maintain backwards compatibility since next version. PHP is always under the discussion for this, and so are the commercial VB6, VB.NET, and the less-commented Python (P3000 wont be compatible with P2.4) and Perl (6.0 comes with things that aren’t compatible with 5.x)
It’s the only way on having a good product, and normal people doesn’t have to use “svn, svn, svn” to know if the next release will break the applications that are working right now.
Aside from the fact that I just finished upgrading all my sites and have to do it again, I want to commend you guys on the speed in which you patched this up. I notified someone about the additional vulnerabilies and literally saw a patch within the next 15 minutes. Thank you again for staying on top of things!
Eric, some Rails internals, like /breakpoint_client, was still being exposed with 1.1.5. Not as critical, but still critical.
Casey, you should upgrade today. If we should learn more about the problem that requires a 1.1.7, you should then upgrade to 1.1.7. We’re providing patches that fixes the issues we know about as soon as we know about them.
Steve, 1.1.6 and the backported patches all protect fully against /breakpoint_client and any other internal or app-related lookups.
Routing is:
QED: routing is “weak sauce”
Is there a way we can just turn it off?
Security through obscurity is no security at all. The crackers had this wide open in half an hour. We couldn’t tell whether our publicly facing systems had even been compromised, because we didn’t know what the problem was. We had to stop work, diff the code, and manually audit everything. Now we find out a day later that the “fix” didn’t fix things. This would have been caught in minutes rather than days if the vulnerability had been fully disclosed from the beginning.
There doesn’t seem to be a tag for 1.1.6 yet—is that coming?
Thanks!
I get the following errors when trying to get Rails 1.1.6:
bash-2.05b# gem update rails Upgrading installed gems… Updating Gem source index for: http://gems.rubyforge.org Attempting remote upgrade of rails Attempting remote installation of ‘rails’ Install required dependency actionpack? [Yn] Y ERROR: While executing gem … (OpenURI::HTTPError) 404 Not Found bash-2.05b# gem update rails Upgrading installed gems… Attempting remote upgrade of rails Attempting remote installation of ‘rails’ Install required dependency actionpack? [Yn] Y ERROR: While executing gem … (OpenURI::HTTPError) 404 Not Found
@DHH
Do y’all have some functional tests for routing that prove this works?
:)
Rex, I believe you’re seeing an error due to rubygem’s slow propegation.
Use the command DHH listed:
sudo gem install rails—source http://gems.rubyonrails.org—include-dependencies
Yeah, none of the mirrors have the gem files yet it seems.
A thought on future policy: The combination of limited disclosure and (originally) asking users to upgrade from 1.0 to 1.1 had a lot of nasty consequences.
There were pockets of Rails developers (on IRC and presumably in the real world) who had to spend their evening in emergency mode, trying to forward-port applications from 1.0 to 1.1.5—which ironically made safe sites vulnerable at an enormous cost in stress.
Some groups of us were lucky to (partially) figure out the bug, and realize that 1.0 was unaffected before the core team’s second announcement.
Let me be honest: This can’t happen again, or some of us won’t be able to use Rails.
In any case, many thanks to everybody who’s helped fix this problem.
Scratch that, I can get to the gems via the gem list command, but the actual gem files haven’t been uploaded to rubyforge yet.
Grab the beta gems or freeze to the 1.1.6 tag—
sudo gem install rails --source http://gems.rubyonrails.org --include-dependencies --no-rdocI don’t way to gem install, I just want the gem file. Trying to update our linux distribution and I need the md5sum of the gem.
DHH et al., First, I hope you get some much deserved rest today. Thanks for the hard work and the disclosure. Let me layout what I believe to be the source of frustration for many posters over the last day. With a security breach there are 3 discreet tasks. 1) plug the hole, 2) assess the damage if there has actually been a breach, 3) take steps to correct the damage. As a framework developer, all you can do is work on Step 1. And assuming thigs are smoothed out, your job is done and in qucik fashion to boot. For the people using your framework, steps 2 and 3 are equally if not more important to be handled in a timely fashion. Think compromised bank accounts or credit cards. The sooner clients know about this, the sooner they can protect themselves, and the sooner they will get over their anger. By issuing a dire warning and then not revealing the problem, developers had no way to judge whether they should shut down their app, do nothing, put in place other security measures. And they had no means to judge whether the fix actually worked or not. I do sympathize that hiding the attack vector to reduce detection by lowerlevel crackers, while you and the team fervishly worked on a solution might have been the most logical approach from the framework standpoint, it was a tough position for some members of the community. And while you’ve primarily created a framework, you’ve also created a community. Let me also say that I also regret that many who disagreed with your decision expressed that disagreement in an immature fashion. How people state a point can diminish the validity of that point, and I hope that is not the case here. Some posters on both sides should really take some time and think about what they say before they hit send. This isn’t a black and white issue and treating as such reflects poorly on the posters and the community. Thanks for your effort (in this case and in general), handling the situation with aplomb, and taking proactive measures for future security issues.
DHH et al., First, I hope you get some much deserved rest today. Thanks for the hard work and the disclosure. Let me layout what I believe to be the source of frustration for many posters over the last day. With a security breach there are 3 discreet tasks. 1) plug the hole, 2) assess the damage if there has actually been a breach, 3) take steps to correct the damage. As a framework developer, all you can do is work on Step 1. And assuming thigs are smoothed out, your job is done and in qucik fashion to boot. For the people using your framework, steps 2 and 3 are equally if not more important to be handled in a timely fashion. Think compromised bank accounts or credit cards. The sooner clients know about this, the sooner they can protect themselves, and the sooner they will get over their anger. By issuing a dire warning and then not revealing the problem, developers had no way to judge whether they should shut down their app, do nothing, put in place other security measures. And they had no means to judge whether the fix actually worked or not. I do sympathize that hiding the attack vector to reduce detection by lowerlevel crackers, while you and the team fervishly worked on a solution might have been the most logical approach from the framework standpoint, it was a tough position for some members of the community. And while you’ve primarily created a framework, you’ve also created a community. Let me also say that I also regret that many who disagreed with your decision expressed that disagreement in an immature fashion. How people state a point can diminish the validity of that point, and I hope that is not the case here. Some posters on both sides should really take some time and think about what they say before they hit send. This isn’t a black and white issue and treating as such reflects poorly on the posters and the community. Thanks for your effort (in this case and in general), handling the situation with aplomb, and taking proactive measures for future security issues.
Huh, I’m using Rails 1.1.3 – so I’m not affected? Any other reason why I should update? (And why was 1.1.3 short lived?)
Uh, I’m sorry… what does “floating on gems” mean?
rails_newbie: 1.1.3 is unaffected. It had a bug with controllers nested in modules, so 1.1.4 was released quickly. You’re fine aiming for a move to 1.2 if you’re in development now.
‘Floating on gems’ means you’re running with the default latest gem version, rather than specifying a specific version in config/environment.rb
Thanks fixing this so quickly guys.
We’re serving all the gems from rubyforge.org itself now until this gem propagates out. 4 Mb/sec of traffic right now, whew!
David and the core team,
Like everyone else above I very much applaud your change in attitude over the “security through obscurity” criticism. It really shows you’ve listened to the community and responded properly, and for that you should be commended.
http://digg.com/programming/Rails_1_1_5_Mandatory_security_patch Digg the story and get the word out
Well, this has been fun. I added in routes to manually deal with the breakpoint_client thing on my blog, just to be sure, and also to make it so that any exploratory scripts are quickly sent to a benign page instead of going through all of the routes. Now that I know about the data loss issue, I’m suddenly feeling very glad that I have a cron job that does a complete dump of my database every hour.
If this is this serious, for new Railer, could you give some details on “You’ll have to apply the diff for your version”? What this means and how. Remmber r4r is for mainstream not just bunch of gurus. Thanks!
I love rails, but I think maybe this whole issue could have been handled more elegantly than it has been so far.
”/(app|components|config|db|doc|lib|log|public|script|test|tmp|vendor)”
Wow. Hole big enough for a Mac truck? I’m stunned.
But thanks again for fixing (no upgrade problems here)...
Now THIS I am happy to see. Excellent work guys.
I appreciate the point fixes.
I have to second that. The fixes came very quickly, but they were incomplete and completely disabling engines as a known bug caused us no end of trouble. I spent last night patching, testing, patching again and then tweaking to fix the breakage… and then came in and did it all over again today. I’m kind of unclear how such a huge vulnerability was introduced in the first place unless it was simply one of those moments where no-one is paying attention, but I’m glad it was at least caught and fixed. Now if only we’d had enough disclosure to make an informed decision…
If this has been asked already I apologize, but…. Was this a simple mistake, or was it a maliciously submitted patch that unknowingly got accepted / applied?
Whenever I try to upgrade our production servers from 1.1.0 to 1.1.4, 1.1.5, or 1.1.6, we suddenly get routing errors. (Recognition failed for ”/Foo”)
This is on a (nearly) fresh RedHat Enterprise Linux box with a clean install of rails, and this critical failure happens regardless of running WEBrick or Apache. It also affects a newly created rails project with a single controller generated from script/generate.
I guess I’ll have to patch with the diff until I can get things working.
Thanks to the core team for providing all necessary fixes in a timely fashion.
What doesn’t kill you make you stronger! ;)
Many more incidents of nondisclosure of security bugs will effectively kill off Rails in the commercial software world.
I have to echo the comments above. Because of the incomplete nature of your disclosure, I couldn’t test this problem for myself. I was running on an older system that wasn’t affected by this bug, yet by following your advice I upgraded to a version that was affected. Now I’m on a version (1.1.6) that hopefully isn’t affected, but please do a gutcheck. Other people have already published this bug, and how to do it I’m sure. So the only people who are still in the dark, are those of use who come to the official source of information. NOT those who are out looking for ways to hack into a Ruby site, I’m quite sure they have all of the information we still lack.
I’m still a huge fan of Rails, but please, please, next time, fully disclose the nature of the problem and how to reproduce it. This way we can fully assess our own vulnerability, and potentially contribute fixes for the problem.
I think I figured it out.
It appears that if you give the controller names as camelcase instead of underscored, it doesn’t do the necessary conversion of the file path.
The problem seems to be with the an argument passed to the ‘attempt_load’ method in routing.rb.
More specifically, if I go to “localhost:3000/FooBar” it looks for “script/../config/../app/controllers/FooBar_controller.rb”, instead of “ficript/../config/../app/controllers/foo_bar_controller.rb”.
The fix appears to be the addition of a line in ‘traverse_to_controller’ that converts the path_suffix to underscores.
I haven’t worked out all the details and tested for bugs yet, but it seems to solve my problems.
(I plan on submitting this bug officially once I’ve completed the fix)
I’ve read through a lot of the comments on this page. While it would have been deeply unwise to advertise one of these ‘killer urls’, a targeted patch would have been much better…. and would have been very helpful to the people running older versions as they would have avoided a full upgrade.
What I think you should do now is create an internal security board composed of trusted team members and trusted outsiders. (i.e. a rails mass hosting provider or a large company installation.) This would allow discussion of a unique issue like this within a larger forum before any public disclosure. This board would also be tasked with creating a published policy to be followed when new vulnerabilities are discovered.
Really guys, how about getting your act together. The hole was not previously disclosed, so you had time to be thorough in your fix.
And now what? .. a day later, we got another freakin patch after you so urgently advised us to upgrade to 1.1.5? Right after spending the time to upgrade a few hundred servers. Perfect.
I understand this is an open source project, but if you want to continue to see the adoption grow, especially by commercial users, do us a favor and be more thorough before you rush out to tell the world that disaster is moments away.
It’s bad enough to deal with this type of thing from Microsoft. Now I have to spend time and resources for this update as well.
Get your act together, I’m sure you guys have professional development or IT jobs as well, and one massive deployment to fix a problem that should have been solved in yesterday’s massive deployment should upset you as well.
I’m not concerned that there was a security problem in Rails, that happens. What bothers me is the haste in which the first “fix” was rushed.
Commercial: yes, because we see so many commercial vendors who disclose all their exploits immediately, and have the kind of patch turn-around that rails has had in the past few weeks.
Commercial Rails User,
Microsoft, to take one example, consistently fails to patch/announce numerous major security holes in its products a timely manner (instead frequently pushing them off to monthly updates, or relying on third-party patches.) By your security standard, Windows would scarcely qualify to run the toaster in a school cafeteria, let alone “commercial software.” Yet Windows doesn’t seem to be having too much trouble making headway in the enterprise world.
Having merely been cautious about when to disclose a security hole the day it was discovered doesn’t disqualify Rails for anything.
That being said, the history of Linux-
and FOSS generally-shows us that openness is those platforms’ source of stability and security. I do hope the core team will consider a more forthcoming approach in future.I’m puzzled, what’s a commercial Rails user? I don’t recall there being a sticker price on the box. We care equally for all users since everyone is paying the same price: zilch.
Okay, that’s not true. We actually care more about users who don’t feel the need to hide behind aliases like “Commercial Rails user”.
If you have something to say, please stand by your words by signing it with your real name. Anything else is a surefire way to get anything you have to say discounted as trolling.
Can you explain more about “floating on gems” and what’s needed?
I don’t have a “vendor/rails” and I also don’t have an environment.rb RAILS_GEM_VERSION
Any guidance would be much appreciated—this is for a production server so the more detailed the info the better. Thanks!
Thanks for cranking so hard on your freaking birthday.
I’ve read these and other posts over the last few days and it’s sad that there are so many selfish a$$holes out there.
From the many who greatly appreciate what you do for us, great job guys.
Joel: That means you are “floating on gems”. The term is used because your app is at the whim of whoever can update the gems on your system – this is particularily dangerous if you are on a shared host, and your app depends upon a certain version of rails. If its a dedicated box where you would be the one updating rubygems, its not as big a deal.
I would like to throw in my $.02. I am the IT Director at a small tribal college (read we don’t have much money). We depend on many open source software apps and frameworks including Rails. We also have to use Windows in various flavors.
So, I see this security issue come out with Rails. (No disclosure. You can argue either side of that to death. I certainly do not have a answer). But the problem was sorted out quickly with a little hesitation in getting a patch out, but it was patched and the problem understood. Now compare that to my Windows updates I had to do today. We patched 6 servers, that is not all of them, just the ones we decided to do. (Keep in mind we paid a lot of money for these licenses, see DHH’s comment’s, we paid NOTHING for Rails). There were numerous updates, some for shit like Windows Media Player or IE (this was on Windows SERVERS). Many of these failed, so me and my sys admin now have to spend a whole shitload of time upgrading, BLAH, BLAH, BLAH. The Rails patch was nothing, a blip in my day compared to that mess. I know some Microsoft fanboy will smoke me for not having the latest geewiz patch role-out software installed or god forbid automatic updates running, but these are publicly exposed production servers and I would rather not have them rebooting on their own accord. It would be nice if they just ran the stuff I wanted and nothing more, like Rails, in a way that a simple minded fool like me can understand. But then again I guess I should hire some MCSA consultant to come and fix all of my problems, tomorrow, the next day, the next day, etc. No, I think I will stick with Linux and Mac OS X.
It is OK that Microsoft and other commercial software companies hide their issues and role-out semi-mysterious patches that us simpletons are told by Homeland Security (see GW I listened) to install. But god forbid some FREE software has an issue and has some hiccups in handling the issue. Go back to .NET, please.
Thanks guys!
Thanks for the update and for RoR! Is there a quick way to test if we have updated correctly and are no longer vulnerable to this security hole?
I don’t understand any of this griping. I think the Rails team’s response was flawless. I can’t think of even a single example of a commercial software package’s security flaw being handled as well.
DHH et al., thanks a million times over for all your hard work. Rails is an inspired piece of software.
If you’ve got 100 servers that needed to be updated, maybe you should TEST THE UPDATE before you deploy it to every server.
You’re obviously a webhoster, and shared hosting clients expect you to make sure you to make sure things are in order before you rush it into production.
You’d think webhosters would have learned with the whole Rails update with Typo fiasco.
I’m not able to apply the patches or upgrade because of dependencies on the Engines plugin, so I’m using the Apache mod_rewrite fix. It doesn’t seem to protect against the /breakpoint_client – is that expected or am I just doing something wrong? Thanks.
Quit your bickering, all of you. As far as I can tell, the Ruby team did what any good development group would do if they had a potentially crippling problem in their fundamental code: Rolled out a fix as fast as they could and warning their users.
Compare that to Microsoft, where it takes months to get ANY kind of disclosure, and it takes a massive attack for them to admit they f*cked up their code, and to have them tell us it’s gonna be at least 2 weeks before they could possibly fix it – thanks guys, you’ve just told the entire cracker community how to break any Windoze box in the world, and no fixes for 2 weeks? You’d think they WANTED people using their softwares to get fried.
I too think this issue was handled extremely well by DHH and whichever other Rails core/people involved.
Thank you all, for your continued commitment to Rails and its users.
David, interesting turn of events to this security problem today. I hope it’s been a valuable learning experience to you. What would you change in the handling of it next time if anything at all?
John, I added this to my .htaccess
RewriteRule ^(app|components|config|db|doc|lib|log|public|script|test|tmp|vendor|)/ – [F] RewriteRule ^(breakpoint_client|cgi) – [F]
Missed anything important?
thank DHH,
quick and effective bug fixes, makes rails developers like me happy.
We are considering Ruby on Rails as a platform with some internal test servers setup for it but to be honest I’m pushing against it.
You’re too new and too cocky and you got found out this week.
Then seeing ridiculously pouty responses like the one from DHH at 11 Aug 00:40 (45)
“I’m puzzled, what’s a commercial Rails user? I don’t recall there being a sticker price on the box. We care equally for all users since everyone is paying the same price: zilch.”
further makes me think that to use Ruby on Rails in a commercial environment is taking a really big risk.
Maybe when the product and the developer mature things might be better.
Maybe this is a stupid question, but if you have an app that doesn’t use “RAILS_GEM_VERSION” and you upgrade Rails on the server, does the app need to be restarted to benefit from the fix?
I recently inherited a rails application and know next to nothing about how it’s configured. How do I determine what version of Rails we are running so that I can apply the proper patches?
Thanks.
I just wish people would stop complaining about Engines breaking. They’re not a part of Rails, therefore the core team is not required to make them work, support them, or even acknowledge their existence.
Perhaps the problem lies with engines doing something they shouldn’t be doing, or “developers” looking for packages of code to do their work for them without understanding what the code really does or how it works.
Anyone know the status of the rails engines patch or if it’s in the works?
I suggest to create a test which will generate random url requests. Urls from empty string to very long strings.
Other test should generate semi-random urls. Simply parse RoR code (even etags might be good for beginning), build dictionary of keywords and then use these keywords to generate urls. Combine keywords, add some random parameters, etc.
Run above tests on separate machine and don’t stop them. I guess that after few days something will be discovered. Of course this is additional test to do besides conventional methods like code reviews.
I used this technique in few projects with good results. Hardware people who test new CPU designs use that kind of random checks very extensively.
Anyway I believe this security lesson will make RoR just more stronger. And don’t worry, this isn’t a first or last security bug in software.
DDH and core team members, thank you for the rapid (and I am sure sleepless) work fixing this issue.
Updating to 1.1.5 and 1.1.6 on 6 applications was painless thanks to your amazing design philosophy guys.
How unfortunate. People complaining about having to update each and every one of their precious servers to a new version of Rails. If you’re the one doing the updates, then it’s your job and you chose it. Maybe it’s time for a career switch. If your application is a spaghetti of code built on countless 3rd party plugins, that was your choice; not the Rails core team so it isn’t fair to complain about backwards compatibility when you’ve cornered yourself. Rails is free. Patches and updates surface extremely quickly. Relax and say thank you.
Andrew, yes, its probably better that you go with vendor-backed software if you think that people who go under cover names like “Commercial Rails User” deserves special treatment.
Since everyone is paying the same price, the only way to raise your status in the community is through valuable contributions to the commons. Sitting behind a cover name full of indignation and entitlement does not count as a valuable contribution, quite the contrary.
If you choose to label this state of affairs “immaturity”, you might not want to hold your breath for a change. You’ll just end up blue in your face.
ROR team. Ignore the hate and continue to succeed.
John & Chuck:
Engines is up to speed on 1.1.6, just update against:
http://svn.rails-engines.org/engines/branche s/rb_1.1
Given that this is probably the first ‘serious’ security patch I can ever remember with Rails, I can’t see what all the fuss is about. I hear BEA, Microsoft et al are looking for ‘Commerical Users’ – good luck and remember to feel the pain.
DHH wrote: “I’m puzzled, what’s a commercial Rails user?”
Someone using Rails to run a business. Like, you know, 37signals.
I’m not at all saying that “commercial Rails users” deserve preferential treatment just because they’re using your free product to make money. But they do have different priorities than some random guy running a personal blog. Several “commercial Rails users” have left you quite constructive and detailed feedback in the past few days, trying to explain the environment in which they use Rails. That environment includes things like managerial justification for upgrades, formal test procedures, and so on—things which were hindered by your initial unwillingness to disclose information.
One “commercial Rails user” in this thread described his own experience of upgrading his Rails 1.0 application to Rails 1.1.5—an upgrade which you explicitly and mistakenly said was both mandatory and urgent. Without any information about the vulnerability (which didn’t actually affect him), he followed your advice and ended up less secure than if he had done nothing at all.
Calling such people names or telling them to stop using Rails does not seem to be a terribly valuable contribution to the commons.
“its probably better that you go with vendor-backed software if you think that people who go under cover names like “Commercial Rails User” deserves special treatment.”
It’s also very telling that you choose to focus on this one aspect while also very conveniently ignoring his actual post.
How about something like:
RewriteRule ^.*(cgi|breakpoint_client|blankslate|dependencies) – [F]
to catch some of the rest?
Justin
Greg – great idea. Continuous fuzzing would be a welcome adjunct to continuous integration testing.
In fact, a precious crew of volunteer testers assembled in #rails-security and thoroughly fuzzed the space of exploitable URLs on every affected release (1.1.0, 1.1.1, 1.1.2, 1.1.4, 1.1.5) on every common deployment platform (apache1+fcgi, apache2+fcgi, lighttpd+fcgi, webrick, mongrel) to conclusively demonstrate closure of the hole.
Thank you all!
OK, since David has asked people to go on the record with their real names, here’s my story.
Rails is a marvellous framework, and it offers huge productivity advantages. I’ve deployed 7 Rails applications since last fall, and helped convince several developers and managers to start investing in it. Right now, I support one production application on Rails 1.1, three on Rails 1.0, and one soon to be deployed on Edge Rails.
On Wednesday, I was working on another project (on deadline) when the first security alert went up, including the following notice:
UPDATE: This problem affects 0.13, 0.14, 1.0, and 1.1.x. So here’s a happy opportunity to upgrade if you still haven’t.
In my experience, migrating a Rails application from 1.0 to 1.1 takes anywhere from a minute to several hours, depending on the amount of code changes and testing required. And in a production environment, many applications will only be upgraded when (a) major changes are being made or (b) the vendor specifically deprecates certain versions of their framework, as David did with Rails 0.13 and 0.14.
Unfortunately, when a vendor makes an after-hours announcement that says “This is a MANDATORY upgrade… The security issue is severe and you do not want to be caught unpatched,” (without any further details), and calls it a “happy opportunity to upgrade if you haven’t,” well, it means a bunch of hard work after hours.
And if the result of that hard work is taking a secure 1.0-based application and turning it into an insecure 1.1.5-based application, well, that doesn’t make anyone happy. Fortunately, plenty of people determined that 1.0 was unaffected before the second announcement went out.
Now, it’s perfectly true that none of us paid money for Rails. And the core team has every right to handle security fixes any way they want (or even leave holes unpatched), and tell people who complain to use another framework.
But I genuinely want to see Rails succeed. And it seems like a new security policy might make it easier for some folks to use Rails.
I too see the events of the past week as a pivotal point for Rails. While it may still too young of a project for some, it is maturing fast.
I wrote a post on my site on my impressions of Rails the past few days. I’m not a Rails user, so hopefully my perspective on the events will seem fair to most here. The URL for the post is: http://cmsreport.com/node/287 .
Looking forward to digging more into Ruby on Rails when I get the chance. Cheers!
In case anyone is still having engine plguin problems, ruby-engines has a confirmed fix found here: http://dev.rails-engines.org/tickets/196. This worked for us using rails 1.1.6 in production and development mode for both WEBrick and fastcgi.
Upgradeing Rails: Securiety through Obscenity My hero DDT has shown what a treu Hacxker he is.
No messin’ arown – he uses his martian arts to TELL peoples when to upgrade. They is so afrid that they do!
He says
“UPGRADE NOW BITCH! OR ILL RIP YOUR HEART OUT LIKE CHUCK NORUS!”
I has upgarded already.. but lets me tell you. I was sitting as my conputer sweating and stuff until it was finshed. I is afraid of being hackxer and pwdned as mccuh as anyone, but I is more efraid to fight DDT (and I is GOOD at martian arts and Rubby, let’s me tell you)
My friend Clegg hasn’ not left his room in too days after he read the Rubby Mailer List and he is hasing some, er, brown trowser poblems.
The Javacs must be cowering in theys Enterprises with feer. I am glad I is on DDT’s side.
I bet most people writing these harsh or unfair comments are just jealous pythonistas, either forced to use Rails or not even using it. Seriously, if you don’t think RoR’s advantages are big enough for you that you can’t deal with a securiy update, you probably should try something else. On the other hand, fair criticism and controlling tone of your comment is very much welcome.
Ryan: By your security standard, Windows would scarcely qualify to run the toaster in a school cafeteria, let alone “commercial software.”
Well, I’ve known a number of professional Unix sysadmins who claim exactly that. :-)
More seriously, there’s a large market for “better than Microsoft” security policies out there, which is why so many firewalls run OpenBSD or hardened Linux.
Open source software can frequently meet those security needs, provided that you avoid a few packages with chronic security issues. I’d be very sad to see Rails wind up on that list of packages to avoid; it’s too good a framework.
Hi guys,
first I wanna thank you for delivering a patch. May be I have a litte OT Question here in sence of Upgrading and future secure developement:
We run actually both Apache 1.3 and Apach 2.0x with Rails, sliding on fcgi. As we all know, there is no further developement for the fcgi module.
Can everybody tell or has an Idea, whats a real alternative using Rails on a high scalable http server? What is the prefered way to do it, by the developers or even by 37 Signals
Thnx a lot
Thnx
Hahahaha—I see Rails Blob has made an appearance!
http://railsblob.blogspot.com/2006/08/rubby-links-on-ww-w.html
What it comes down to, DHH, is: Do you want Rails to be the standard web platform? Or do you want it to be a hobbyist platform for guys running their personal blogs? You mock and ridicule anyone who dares question your judgement (and not just here.) People love Rails. They use it to replace J2EE. But the money that drives all this doesn’t spring out of nowhere. Commercial users come to you with legitimate and respectfully phrased concerns about how they can possibly use Rails out in the real world with this kind of security policy, and you basically tell them to shove it, since it was free.
You did a lot of work and gave it away for free, yes, and we all appreciate it. But this does not entitle you to sit in an ivory tower for the rest of your life and cast disparagement on the ignorant peons who call you on your bad decisions (notwithstanding the legions of DHH cheerleaders who have no customers to answer to, just use Rails for their personal blogs and side projects, and don’t really care about their systems being vulnerable for a few days because “hey, it’s better than Microsoft’s security policy”.)
And the handling of this security patch was a bad decision. 37signals, which supports Rails, is pretty unusual as a commercial case, and the money it makes is ONLY due to OUR use of Rails out here in the real world, which drives traffic to it in droves. It sprang up pretty fast out of nowhere, and it will dry up just as fast (or fork away from your control) if this attitude towards your users continues.
Maybe what we need is an independent rails-full-disclosure mailing list, so that those of us who care about security in our apps can keep each other posted on what the situation actually is. It is evident that the Rails core team is not interested in any of the perfectly reasonable criticisms that have been levied; the only acknowledgement it’s got at all has been dismissive mockery.
Thanks for all the hard work, RoR team. Microsoft could learn a thing or two (or three…) from you.
Rails core guys – thank you for a premier product, and thanks for making it free!
Yes, it would have been better to fix it once. Yes it would be nice if those making their money from Rails usage could get the info they want when they wanted it. No, not everyone needs full disclosure of a URL-based attack, especially before a fix is available (any Windows admins remember CodeRed?). Please, folks, there is a balance here.
I humbly suggest to post this kind of announcement on an “updates” mailing list as well as a security-focused list like BugTraq, rather than a widely read blog. Keep full disclosure for a day, a week, two weeks – enough time for those with the most to lose to take action. This way anyone needing this info has the power to get it, while keeping mass hysteria an a minimum.
With this, and “freezing”, users are empowered to control their updates and keep their app secure. After all, the Rails core team has enough to do, and they are giving it away!
Thanks for an awesome, free product!
Cheers,
Kevin
Tom Barrick: It is evident that the Rails core team is not interested in any of the perfectly reasonable criticisms that have been levied…
Even though this update caused me some headaches, I don’t think that’s a fair accusation to make towards the core team.
From what I’ve heard on IRC, the core team recognizes that (a) they need to be clear about which versions of Rails will be supported with security updates, and (b) they need to manage the update process more cleanly.
I’m waiting to see the new security policy. Once that’s out, we’ll be able to judge what kinds of security and deployment needs the core team wishes to support. And we can choose our tools appropriately.
Eric Kidd: Fair enough, perhaps that was spoken too soon. I was not aware that a new security policy was in the works. (Nothing about a new policy was announced on the blog, for example.) I will wait and see what it says, and hope that lessons have been learned.
With the depcrecation of FastCGI, what are shared Rails web hosts supposed to use?
planetmcd++
I asked this question on the 1.1.5 notification post, and didn’t receive an answer, so here it is again.
Is it required to run rake rails:update after updating?
Ahh, Barrick, what a simple world we all live in. If you’re satisfied with how we deal with this incident, you’re a cheerleader, don’t have real customers, and must just be using Rails for a blog. On the other hand, if you think things were done badly, you’re an important person of stature with real customers and real needs.
I don’t tell people with legitimate concerns to “shove it” when their suggestions are not hidden by fake names or worded with the kind of indignation and entitlement you here exhibit. We’ve taken many of the suggestions to heart already.
Within 24 hours of the first announcement, we released the full disclosure, a complete patch, backports for all affected versions, work-arounds for lighttpd and Apache, and announced a new security mailing list for future incidents.
On the back of your rambling, wild accusations, and laughable sense of control or understanding of the 37signals bussines, I have absolutely no issue with telling you to shove it. Please do go find another standard web platform. I’m sure they’re dying to get your valuable input.
Adam, no that’s not necessary. Only the vendor core files were updated.
tgmeth, FCGI has not been deprecated for Rails use. Nor will it be for the foreseeable future. If you’re a shared hoster, it’s still worth looking into personal Mongrels and a shared proxy setup, though.
Kevin, we actually did release a bugtraq notice early this morning. We’ll make sure to have it happen simultaneously next time.
“Within 24 hours of the first announcement, we released the full disclosure”
but in the post
“The cat is out of the bag, so here’s the full disclosure edition”
way to take credit after the fact
DHH and Core Team:
Immediately after all future patch releases, I expect you to log in to all my servers, update them, and ensure everything works properly.
Thank you login/pw: root/12345
“Within 24 hours of the first announcement, we released the full disclosure, a complete patch, backports for all affected versions, work-arounds for lighttpd and Apache, and announced a new security mailing list for future incidents.”
DHH, IMHO it would have been better to delay the first announcement until you are totally clear about which versions are affected and you can provide patches for all supported branches. While you can never be sure that you have complete patches, you can make sure you have patches for all supported branches. A precondition is a clear statement which branches are supported, of corse.
At the time you brought the first announcement, there was no public knowledge about the problem and no zero-day-exploit was known. There was no pressure to do the announcement at that time and releasing an untested patch. You could have be doing proper investigations in 24 hours or in a week (at your liking) without any stress (stress provokes errors). You would not give away any information to the bad guys.
With the first announcement you have put yourself under heavy pressure to do the investigations you haven’t done before as fast as possible because everybody out there was screaming and complaining.
Having said that, I want to thank you for the great framework! I am pretty sure the team and the project both have matured very much during this week.
Thank you very much, guys!
It’s intriguing. I don’t see how DHH can have the attitude he does and expect Rails to ever be used by serious professionals other than his own team. When it’s not your baby, you do take issues of how things are handled and how the people handling it respond to your concerns seriously.
Telling people to shove it because they are irked (rightfully so, based on the bizarre way in which this whole thing was handled) is stupid. People who’ve done a lot of work to install a faulty patch only to patch again later are GOING to be irritated. To not understand and accept this is to ignore human nature, to admit you deal better with computers than people. And that is not going to make Rails go beyond niche to mainstream in production environments. Again, it’s your baby, you don’t notice that it misbehaves, but everyone else does.
If you can’t understand that people look at this kind of issue when deciding what kind of platform to use for deploying web apps, you are insane. Telling people to shove it and find a different platform guarantees that true professionals who base decisions not on how cool you think you and your platform are, but on whether you can be objective about whether or not they have rightful concerns over this major inconvenience and the stupidly worded original notice of the issue (and on the merits of the platform itself) will indeed go elsewhere. As will their co-workers, their friends, and others who they influence.
If this is not a toy, then accept that people who do this for a living will have concerns. And they will be frustrated with how this was handled. Accept it, fix the problems, and do not resort to name calling and telling people to shove it. That’s an approach that works for toy platforms, not serious ones.
I think we all learned few things during this process, rails is still maturing but the core team is doing a phenomenal job, the best they can with this project. I really appreciate their effort. As wih other open-source projects the vulnerability got fixed swiftly but it took 2 iterations in a period of 2 days, so what? It is a lot better than what other industry leading commercial middleware companies are doing. I am sure lots of “commercial user” would know the times when they needed a particular fix (security or not) but had to wait not days but months for those to get delivered. My rule of thumb, for mission critical applications is to run an earlier release and to wait a little bit after a patch is released. For other type of apps I like to test the latest and as soon as possible. Again thank you core team and thanks for the new security mailing list.
Thank you core team guys. The rails framework is just awesome – I try to imagine myself writing the same product I did in the j2ee framework… very unpleasant view… I have two question to the world: 1) Is adding the mentioned rewite rule to apache config file, enough? Or must I upgrade rails anyway? (I am using 1.1.2). 2) I am a bit green in svn – how exactly do I apply the 1.1.2 patch?
Thanks all
nadav, the rewrite rule is a temporary measure. You should definitely upgrade to 1.1.6. I’d recommend doing that directly. If you can’t do that, for whatever reason, then you can do “curl http://www.rubyonrails.org/files/aug_10_security/rel_1-1-2.diff | patch -p0” while standing vendor/rails.
This post has thoroughly convinced me to not use Rails in any real project. The attitude kills it. Really.
Could you please include Ishikawa’s patch to ticket 3030 (IMHO also quite security-relevant) in the next security releases so that it needn’t be re-applied manually? See: http://dev.rubyonrails.org/ticket/3030
Re: the 3030 ticket. Well, I get a Trak error. Can’t even get the patch. :(
This comment thread really is quite painful to read.
A humble “thanks for the suggestions, we understand that there have been a few bumps and we’re taking things to heart so that next time out this process will go much more smoothly” would have done wonders.
It’s all that was necessary, and appears to be all that the great majority of commenters were looking for.
The bumping of heads here is bewildering.
Thanks for the patch.
I understand why people who’ve invested a lot of time and money in deploying Rails commercially are upset with this patch and the way it was handled.
On the other hand, these folks have deployed a 1.x version of a software platform. That in itself is a risk, and any truly ‘enterprise-y’ Rails user should have evaluated and accepted that risk before deciding to go with Rails.
Furthermore, the Rails core folks have always isolated themselves from the gibbering hordes to an extent, probably to maintain their personal sanity. I’m sure this has been a learning experience for all of them. This was a publicity challenge as much as it was a technical one.
While I’m not going to applaud all aspects of the way this patch was handled, I’m extremely glad that the core team swung into action as quickly as they did. They did their best, and they fixed the problem in the end.
As for the bluster in these comments, it’s just frustration, people. Don’t pass judgement until you see how problems like this are handled in the future.
This is fun!
Hi people I suggest to look at the bottom line – there was a problem, there was a bug – this sort of things happen everywhere all the times, in the software world. The problem was fixed very quickly, the first fix was not good enough so a day after came the final fix. The whole thing took something like what – about 24 hours ? If this isn’t fast then neither is FastCGI … (joking). Some people think DHH was not kind or polite enough. Ok so what? Did he come up with a great framework? yes. Did he open sourced it for all of us to use ? yes. Was the security bug fixed quickly ? yes. Those are the important things here, and I thank all the people involved in maintaining Rails.
Nadav
I am not sure why my messages are not getting to the mailing list (at least I can’t see them) and I can’t find an answer anywhere. I have tested on a unpatched version of 1.1.4 (works) upgraded to 1.1.6 as well as a clean instance where 1.1.6 is the first version of rails I installed and have had all require statements made in the environment.rb file blowup. Has anyone else ran into this? if so what is the fix? it boils down to dependencies.rb line 100 and const_missing when referencing plugins (sentry, guff, etc.)
Good work guys!
Such anger, such hatered, dam that rap music. Everyone needs to calm down before they go flaming away. The open source community nearly always does a better job at critical bug fixes then Microsoft has ever done. While have 2 point point one patch in 2 days is not fun. At least they discovered the problem and fixed it right away. People make mistakes, but they admitted their error and fixed it right away. I for one am very happy with rails and so is my boss. It’s allowed us to get changes made quickly and realiably.
Thank you for creating such a wonderful system. Without I’d still be stuck with PHP. Bleh.
I keep getting errors when trying to upgrade rails:
Fetching external item into ‘vendor/rails’ svn: REPORT request failed on ’/svn/rails/!svn/vcc/default’ svn: REPORT of ’/svn/rails/!svn/vcc/default’: 400 Bad Request (http://dev.rubyonrails.org)
I have set svn:externals to http://dev.rubyonrails.org/svn/rails/tags/rel_1-1-6
I first thought it was because the trac was down, but the problem don’t go away. Any ideas?
What a painless patch! & thank god that Ruby persuaded me to adopt TDD—I know everything works fine, too! Cheers again, Doug.
Hej David, synes lige at du skulle have en enkelt besked paa dansk ind imellem alt det udenbyske :)
du kan jo oversaette til Core Team, hvis du synes at det er noedvendigt.
RoR er brilliant – og selv om der har vaeret masser af huller i vejen, saa vil jeg ikke tilbage paa motorvejen at koere (hvis jeg paa nogen maade kan undgaa det)
synd at saa faa forstaar den saerlige danske ironi, som jeg synes at fornemme i visse af dine indlaeg ;)
glem dem – og fokuser paa det gode arbejde!
de bedste oensker Walther
ps. jeg har en ide til hvordan core team kunne advare rails applikationer – men skriv mig, hvis du orker at hoere den :) walt at alco dot dk
Hallo—I suddenly cannot run migrations to my production database (which is SQL Server, sadly). This has happened since I updated Rails earlier today (due to the security problems). The attempt to run migrate with trace enabled looks like this:
W:\holly> rake db:migrate RAILS_ENV=production—trace (in W:/holly)This is for an enterprise application that is currently running, so is concerning me quite alot. It appears to be an error in ActiveRecord. I know you’re all very busy at the moment, but if a response to this was possible, I’d be very grateful! Thanks, Doug.
Hi!
Great work! I’m new to rails, but I feel that this is the first critical security bug found in rails.
I think that you should do something like this next time:
+ When you get a security problem: calm down. You can’t think if you don’t calm down. + Then, you must study it (including test cases and stuff). + test ALL of the maintained versions for the bug. + Try to create workarounds if the fix is too long. + Write the fix.
While you are working on the fix, you can post the security advisory with the workarounds (if any). The time to post the security advisory is when you are sure on the time-window for the fix… and off course, if the time-window is not too long.
In my experience, in Open Source projects, it never takes more than three days to fix a critical security bug (once reported, off course).
People who use this for critical aplications, will decide if they should shutdown their servers or keep using it while the fix is out. The thing is: if you use rails for critical aplications, you would be looking at the security page oftem, or you could create a “security report list”, where you would receive all the security-related messages.
Off course, you can add/remove steps, you can change the order. This is just a suggestion.
Thanks for creating such a good project!, and keep up the good work!
Well guys, I would say that my two cents is this:
1. I’m appreciative that you guys worked so hard and addressed the issue so quickly.
2. I understand the frustration of the people using rails in enterprise environments. When they dub themselves ‘commercial’ they are more so referring to using rails in a large scale server environment where redeployment can be a huge process. Not all companies are as agile as 37S or other small businesses.
Any who these are all people who have adopted your creation and are a part of your community. It’s clear you’ve made a mistake in the way you handled this fix though you had the best of intentions. All that I would ask is that you learn from this experience and handle the situation a little more traditional or professionally in the future.
I’m a big fan of your framework, let’s not hurt our community support by reacting to tough situations improperly and further lashing back at those very people who are supporting your community.
Think about it this way. 90% of customers don’t complain they just leave. Whoever that guy was.. he’s doing a favor by complaining to you.. now the cat is out of the bag again.. you know that you’ve turned off some people in the community and scared some folks away from rails. Not becuase of the security flaw but becuase of they way it was handled. Let’s not let it happen again!
Given that bugs exist and that it is likely more security holes will be found in future, is it wise to recommend Rails applications are frozen before deployment?
I could well imagine Rails-aware hosts emailing their clients claiming that they were now running the latest Rails 1.1.6 and the clients sitting back with a warm fuzzy feeling – while all their applications continue to run on problematic earlier versions lurking in vendor/rails. The idea of people knowing if they’re “floating on Gems’ only really makes sense for those who wrote/deployed an application, rather than someone who just took an existing app off the shelf and installed it.
One good way to tackle this is to design an API then stick with it, at least until the next major version number. As long as the API will always be backwards compatible for applications written on earlier versions of Rails (bug fixes that change behaviour accepted), there should be no need to freeze. One would always “float on Gems”, all applications installed on a particular server benefiting simultaneously from a central Rails upgrade.
IMHO this kind of change control is something that OSS is often quite bad at – “just fix your apps and recompile to use library version <foo>” – but as we see with Rails 1.1.[0-5] -> 1.1.6, it can lead to security problems.
DHH: Grow up.
I think it is good to freeze your gems. I know first hand that updating the gems can break your app. YOU want to be in control of what version your app is running on not REACTING to changes in the sys environment. If you’re smart enough to freeze your gems hopefully you are not silly enough not to realize the flip side to doing so.