Tests, thy name is Legion

It's no secret that I think we in the Ruby community can learn a lot for the communities around other programming languages. The Perl community is huge, mature, and a good source of ideas. One of the ideas that I'm trying to steal from them right now is the Phalanx Project — a testing tool for new releases of Perl.

Instead of building a large suite of unit tests, Phalanx is taking the top 100 Perl modules from CPAN and using their test suites to exercise Perl. This has several benefits. First, they can see how new versions of Perl perform in 'real world' settings. Second they can watch changes in correctness (nd performance) over time. Third, the testing wizards in the Phalanx project can help module owners improve their own testing.

I'm working on a project I call Legion, with the intent of doing much the same thing. So far, I'm using four libraries from RubyForge with a total of 4062 assertions. I'm running the tests against the latest releases of Ruby, JRuby, and YARV. As I collect data from these tests, I can feed it back into each Ruby implementation project to help them build a better Ruby.

I also want to feed the data back into the projects who's test suites I'm using. If I can identify gaps in their test coverage, I'll certainly work at building tests to close those gaps. Perhaps we'll even find some bugs through more strenuous testing.

There are a couple of things Legion isn't doing yet. With some help, I think they'll be easy to implement:

• More platforms — currently, I'm only testing on x86 Linux. I'm planning on adding PPC OS X shortly. I'd love to add x86 OpenSolaris, Win32, and any other OSes people are willing to run Legion on (as soon as it's ready for public consumption).
• Better reporting — I'm still trying to figure out how much information to record, what to report on, and where to make those reports. I need feedback (especially from Ruby implementers) to make sure I'm providing a tool that's useful.
• More libraries — I'd like to add more libraries to the test suite. Right now, they need to be pure Ruby (i.e., no C), and need to have a non-Rake method of running all their tests. I may move to Rake to automate the testing, but then will need to write a Legion specific task for each library in the suite.
• More Ruby implementations — as soo as I have things a bit more stable, I'm going to add CVS/SVN versions of Ruby (both 1.8 and 1.9 trees), JRuby, and YARV. I'f like to add Cardinal, Metaruby, and other implementations as soon as they're ready.

If you're interested in helping with any of these, please let me know.

I've already collected some data (and posted it to the dev lists for YARV and JRuby). I'd like to share the highlights here. As I can coordinate with the library maintainers and the developers, I'll open up the reporting a bit more. For now though, here's a high level view of the "big three" Ruby implementations according to Legion:

Ruby  
  total time:     115.094538 seconds
  time/assertion:    .02819  seconds
    total asserts:  4082
    total failures:   29
    total errors:     27
  
YARV
  total time:      12.012201 seconds
  time/assertion:    .00842  seconds
    total asserts:  1425
    total failures:    3
    total errors:    146

JRuby:
  total time:      97.978000 seconds
  time/assertion:    .04996  seconds
    total asserts:  1961
    total failures:   29
    total errors:     17

I noticed three things on a simple look at the tests:

• JRuby seems more compliant with Ruby than YARV right now.
• Ruby is nearly 2x faster than JRuby right now.
• YARV is ~3x faster than Ruby.

I expect to see these change over time. For example, JRuby is just starting a push for optimization now that the developer feel like they're close to being functionally complete (see Charles Nutter's blog posts, Performance: Block Variables Breakdown, Performance: Inlining Strings, and Nibbling Away at Performance).