Friday, December 22, 2006

Mongrel 1.0

I had a chance to talk with Zed about the mongrel 1.0 prereleases he's doing, and the big 1.0 release they'll lead up to. As always, Zed has some great things to say:


Getting to 1.0 has been a hard row to hoe, and Win32 has been a big part of that. How much work does Ruby need to just work on Win32?

Zed: Luis has done quite a lot of work building a nice mongrel_service for win32 that manages all the processes and does a better job of keeping things tidy. That really was the biggest issue with the win32 support for Mongrel 1.0. Part of the problem is that, well, win32 is just weird and doesn't have any real notion of a "process". Do all the hackery you want it still isn't quite the same. In order to run a Mongrel server and keep it going after you log out, you need to make a service, and that's a whole huge domain not covered by Ruby very well.

Dan Berger has made a significant dent in the problem, but it's an uphill battle since he doesn't get very much support from the ruby-core folks.

I think if Ruby is going to run on win32 very well it'll take an effort similar to ActiveState's Python and Perl. A completely cohesive install that has *everything* someone needs for win32, and then all the other goodies people want to have. Make it all built with the same compiler (instead of this three different compilers crap) and push it as the standard win32 install. Banish anyone who isn't using it as their Ruby platform on win32 and we can finally get some stability.

Just doing that would go a long way to help win32 people out.

What five things are the biggest draws for Mongrel 1.0?

Zed:
5) Your boss will have no problem with you using it now (even though it's not different from 0.3.20).
4) All the Java heads in your office who spend hours clicking around in their "app containers" can't snicker at your lousy web server's 0.3.x status anymore.
3) It works better on win32, even though we had to use FreeBASIC to make it work better.
2) Mongrel 1.0 has that fresh Enterprisey smell and will soon get TWO version numbers so it can keep up with the traditional "baffle them with versions" marketing strategy so common in the BigBiz world.

And the best reason you should use Mongrel 1.0:
1) This was recently overheard in the #django channel:

"16:25 < daibatzu> Also, is there a python based server I can use to run my Django apps, rails has mongrel"
Even the django guys want a Mongrel. How hot is that?

On a more serious note, 1.0 doesn't add anything but a few tweaks and expanded static file handling (MIME types, etc.). For most people it's just a minor upgrade that helps them feel better about running an open source project.

At RubyConf, you talked about using RFuzz to find problems in Mongrel, how did you do that? How much did it help?

Zed: RFuzz helps a ton since there's many problems that I can't verify unless I have a full running Ruby on Rails stack and can hit it with a test case. In the RFuzz source I've got a full external test suite that runs Mongrel with Rails and then tests out a bunch of problems people have reported. This makes fixing the problem easier, and also makes sure the problem doesn't show up again.

It also has a bunch of thrashing I've done to make sure the parser works well, sockets getting closed violently work well, and tons of other problems related to stability and nasty clients. It's not quite as extensive as I'd like, but it's really helping keep my work load down.

About the only thing that sucks is I've got to run this before I do a release and the automated build is now taking about 4 minutes.

I know the JRuby guys are trying to get mongrel working over on the JVM. How much have you worked with them?

Zed: A bit here and there. Mostly they're waiting on the 1.0 release to get out and then I'm gonna try helping them get a Mongrel running that works under JRuby. One major breakthrough was Adrian (the author of Ragel) added a Java generator for Ragel. Using that I should be able to port the HTTP parser over to Java for them.

What's next for Mongrel? (What other Zed projects should we be keeping an eye on?)

Zed: I have a process of putting a release of Mongrel out and then letting it simmer. I judge when it's time to put out a new release based on the number of bugs I've received and how severe they are. Security problems get an immediate release, but other stuff I plan for and do meticulously. So, for the immediate future there's probably not too much more to be done to Mongrel. I have received requests to do a better process monitoring system to replace the nasty spinner/spawner/reaper/mongrel_cluster stuff, but that's about it for now.

After thinking about what should be for the "Mongrel 2.0" stream I've decided that Mongrel should really be a vehicle for cleaning up the way all of these frameworks are deployed. Big on the hit list is removing their addiction to CGI semantics and cgi.rb by coming up with a Servlets/WSGI style standard.

No, I don't mean "complex" like Servlets, but rather a standard about how applications are packaged and deployed as well as a standard API that makes using cgi.rb useless. This standard would abstract away all the IO processing and deployment so that each framework just worried about how to process requests. Chief problem to be solved is how frameworks seem to love raping and reinventing things like MIME processing, error reporting, statistics, logging, starting and stopping the *server*, clustering, and even just putting the app into a package for the deployment server.

The problem is getting the various framework developers to play ball might not happen without severe beatings and bribery. I do think Mongrel's in a unique position though to use "gentle enforcement" to make this happen, it's just a matter of whether the various framework developers are interested.

5 comments:

Luis Lavena said...

H pat, nice and quick interview!

Zed forgot to mention that even we used FreeBASIC for mongrel_service, is the only with built-in process monitoring and respawning :-)

Kirk Haines said...

You need to use the right carrot.

Simply replacing cgi.rb isn't necessarily enough. IOWA doesn't even use CGI (it used to use it for the escape/unescape methods, but no more), but if I did, a major selling point for me would be to have a Six Million Dollar Man (faster, stronger, better)implementation of functionality that I need.

I do have questions, though:

-- What are you thinking about regarding the standardization of logging and error reporting?

-- How do your thoughts about standardizing things like application deployment and clustering play with things other than Rails and Rails-a-like's? My ideal clustering solution, for example, should be capable of handling session affinity demands for serverside sessions. It should also work just fine if I am NOT using Mongrel as part of my application deployment.

My new apps are most commonly deployed using what I call a hybrid deployment. It uses EventMachine to handle the network traffic, and Mongrel's HTTP parser to make sense of the incoming request that is delivered to IOWA because for my typical site or application it's faster and scales better when request concurrencies exceed 1. I'm not going to a fledgling standard if it forces me to lose the flexibility to choose the deployment mechanism that best suits the application.

Zed A. Shaw said...

Kirk,

It's pretty early to be getting into how it's actually going to be done. It's all very pie-in-the-sky right now since I want to get Mongrel 1.0 out and then take a breather and really think the problem through.

But, this statement is exactly the kind of stubborness I'm talking about:

"I'm not going to a fledgling standard if it forces me to lose the flexibility to choose the deployment mechanism that best suits the application."

However, your only real advantage in your custom EM+1/2Mongrel stuff is it's slightly faster for IOWA. You've done it your way, you're very proud of it, and I'd have to get a 2X performance to bring you along. I mean, c'mon you couldn't even use regular Mongrel without reinventing it as your own thing.

My complaint about this stubborness is that it makes it hard to improve the quality of Ruby web applications in general. Your stance that you won't budge unless I completely solve the problem first means that I can't find out what you'd want. I basically have to do the exact same thing I did with Mongrel: write it, document it, and then let people use it if they want.

But, most likely you'll take what I've written and simply roll your own again. That's fine, it's what open source is all about. But it's so sad because it's a ton of wasted effort for only a slightly marginal improvement.

And yes, your performance gains are marginal, not astronomically different from other fast frameworks.

Anyway, I'd wait until I start writing about it before you jump in to comment on the idea. Just think about the idea that things could be better if everyone quit trying to roll their own versions of this stuff.

Kirk Haines said...

"However, your only real advantage in your custom EM+1/2Mongrel stuff is it's slightly faster for IOWA. You've done it your way, you're very proud of it, and I'd have to get a 2X performance to bring you along. I mean, c'mon you couldn't even use regular Mongrel without reinventing it as your own thing."

Well, no. In cases where there are external latencies that make the requests slow, where Ruby threading can thereby be taken advantage of, Mongrel is a better choice for me than the hybrid EM + Mongrel HTTP parser, so I use it.

When that's not the case, though, I get significant benefits by using EM for the network traffic, especially when there are going be concurrencies greater than one.

That's all rather beside the point, though. The point is this:

If there's no benefit to using a tool, why use it?

If you define something that you would like to see as a standard, there has to be a benefit to conforming to that standard in order to convince people, like me, to adopt it.

Mongrel provided a significant benefit over webrick. People adopted it. Yay. I like it. It wasn't hard to write a handler to use it. I use it. Just not for everything.

Your comment that my stance makes it impossible for you to find out what I want is untrue. I can tell you very clearly what I want and what I need. Just ask.

And if what is built overlaps with what I need, I'll be happy to do the work on my end to make things work with it.


Kirk

Anonymous said...

Just found out that Ragel has now an experimental ruby code generator, Victor Hugo Borja is working on this, maybe this could be used to have Mongrel's HTTP parser compiled to java bytecodes by jruby and maybe have a common mongrel parser code to share with rubinius.