RubyConf 2006: Implementers Summit

To me, one of the highlights of RubyConf 2006 was the implementers summit we held. The idea was that we'd grab a quick dinner together, eat and chat, then sit down to some serious discussion of what it means to implement Ruby and how to get the various projects working together.

We had a number of people attend: Paul Brannan, Nathaniel Talbott (Test::Unit), Charles Nutter (JRuby), Evan Phoenix (rubinius), John Lam (RubyCLR), Nick Sieger, Devlin Daley, Kevin Tew (cardinal), matz (Ruby), Koichi (YARV), Eric Hodel (metaruby), Tim Bray (not pictured), Zed Shaw (RFuzz, not pictured), Ryan Davis (not pictured, metaruby), and I. There are five people in the photo who hung out to listen, Cease Larry and Doug Tolton came in from Utah, and I know another is Jack Wohr (from Dr. Dobbs) — if you can identify the others, please, let me know.

Since this was the first time we've tried something like this, a lot of the discussion was centered around "What kind of problems are you having?" and "How are you dealing with that?". This actually proved very useful in pointing out common ground for the more valuable (to me at least) part of the discussion. How the various projects could work together. (You can see an (ugly) pile of notes at wiki.rubygarden.org/Ruby/page/show/RubyImplemntersSummit2006Nov.)

There were a couple of specific tasks that came out of the meeting. First, there was consensus around the need for a 1.8.5+ spec that the various groups can work against. Charles Nutter has already put together a Wiki (and matz has already contributed to it), but it's down due to hardware problem at the moment.

The second task was to build a common testing suite. Ryan and Eric are contributing their BFTS code and tests, which Kevin and Evan are moving into the rubytests project at Rubyforge. The other projects will all add their own tests into this new repository. I'll also be adding Legion to it, and Zed will be helping build a fuzzing framework for the implementations to share.

We also agreed to hold more of these summits going forward. I'm hoping we'll see them as a semi-annual event.

Photographs curtesy of Tim Bray who, oddly enough, doesn't appear in either of them ;). Tim also mentioned the summit in this post

**Doug Tolton has identified himself (and corrected my spelling of Ceaser's name.**

Legion Improving

I've been doing a bit of tinkering with Legion and it's already showing improvements. I've also found my first victim, err, volunteer to help get more tests running. Best of all though, Legion has found a couple of issues that can be directly addressed (one of them already has).

Improvements first: I've added another library (Ruport) to the test suite, and added the Ruby 1.9 CVS tree to implementations. The new test results look like this:

ruby ran 730 tests with 2452 assertions in 17.178 seconds.
There were 0 failures and 23 errors.
The average time per assert was 0.007 seconds.

ruby-1.9 ran 188 tests with 819 assertions in 4.549 seconds.
There were 1 failures and 7 errors.
The average time per assert was 0.006 seconds.

ruby-yarv ran 596 tests with 1425 assertions in 12.055 seconds.
There were 3 failures and 146 errors.
The average time per assert was 0.008 seconds.

jruby ran 578 tests with 1987 assertions in 96.666 seconds.
There were 29 failures and 16 errors.
The average time per assert was 0.049 seconds.

Now I need to work on some tools to drill down from this. (I can do so by hand, but I'd like to automate it.) If you've got ideas about what kinds of reports you'd like to see, let me know.

I got an email from Ben Bleything offering to help. If anyone else is interested, I'd like to get the conversation going this coming week (18-24 Sep). Drop me an email, or leave me a comment.

While I was adding Ruport, Legion identified a (really minor) problem. Gregory Brown has already fixed it.

I've fixed this in trunk and stable, revision 211. Legion is already helping out! :)

w00t! It also looks like Enumerator still needs to be built into JRuby. I'm not sure if it was on the list yet, but I've reported it to the developers mailing list to make sure.

I still have a lot of cleanup to go to get this really useful, but it's getting there.

Tests, thy name is Legion

It's no secret that I think we in the Ruby community can learn a lot for the communities around other programming languages. The Perl community is huge, mature, and a good source of ideas. One of the ideas that I'm trying to steal from them right now is the Phalanx Project — a testing tool for new releases of Perl.

Instead of building a large suite of unit tests, Phalanx is taking the top 100 Perl modules from CPAN and using their test suites to exercise Perl. This has several benefits. First, they can see how new versions of Perl perform in 'real world' settings. Second they can watch changes in correctness (nd performance) over time. Third, the testing wizards in the Phalanx project can help module owners improve their own testing.

I'm working on a project I call Legion, with the intent of doing much the same thing. So far, I'm using four libraries from RubyForge with a total of 4062 assertions. I'm running the tests against the latest releases of Ruby, JRuby, and YARV. As I collect data from these tests, I can feed it back into each Ruby implementation project to help them build a better Ruby.

I also want to feed the data back into the projects who's test suites I'm using. If I can identify gaps in their test coverage, I'll certainly work at building tests to close those gaps. Perhaps we'll even find some bugs through more strenuous testing.

There are a couple of things Legion isn't doing yet. With some help, I think they'll be easy to implement:

• More platforms — currently, I'm only testing on x86 Linux. I'm planning on adding PPC OS X shortly. I'd love to add x86 OpenSolaris, Win32, and any other OSes people are willing to run Legion on (as soon as it's ready for public consumption).
• Better reporting — I'm still trying to figure out how much information to record, what to report on, and where to make those reports. I need feedback (especially from Ruby implementers) to make sure I'm providing a tool that's useful.
• More libraries — I'd like to add more libraries to the test suite. Right now, they need to be pure Ruby (i.e., no C), and need to have a non-Rake method of running all their tests. I may move to Rake to automate the testing, but then will need to write a Legion specific task for each library in the suite.
• More Ruby implementations — as soo as I have things a bit more stable, I'm going to add CVS/SVN versions of Ruby (both 1.8 and 1.9 trees), JRuby, and YARV. I'f like to add Cardinal, Metaruby, and other implementations as soon as they're ready.

If you're interested in helping with any of these, please let me know.

I've already collected some data (and posted it to the dev lists for YARV and JRuby). I'd like to share the highlights here. As I can coordinate with the library maintainers and the developers, I'll open up the reporting a bit more. For now though, here's a high level view of the "big three" Ruby implementations according to Legion:

Ruby  
  total time:     115.094538 seconds
  time/assertion:    .02819  seconds
    total asserts:  4082
    total failures:   29
    total errors:     27
  
YARV
  total time:      12.012201 seconds
  time/assertion:    .00842  seconds
    total asserts:  1425
    total failures:    3
    total errors:    146

JRuby:
  total time:      97.978000 seconds
  time/assertion:    .04996  seconds
    total asserts:  1961
    total failures:   29
    total errors:     17

I noticed three things on a simple look at the tests:

• JRuby seems more compliant with Ruby than YARV right now.
• Ruby is nearly 2x faster than JRuby right now.
• YARV is ~3x faster than Ruby.

I expect to see these change over time. For example, JRuby is just starting a push for optimization now that the developer feel like they're close to being functionally complete (see Charles Nutter's blog posts, Performance: Block Variables Breakdown, Performance: Inlining Strings, and Nibbling Away at Performance).