It's no secret that I think we in the Ruby community can learn a lot for the communities around other programming languages. The Perl community is huge, mature, and a good source of ideas. One of the ideas that I'm trying to steal from them right now is the Phalanx Project — a testing tool for new releases of Perl.
Instead of building a large suite of unit tests, Phalanx is taking the top 100 Perl modules from CPAN and using their test suites to exercise Perl. This has several benefits. First, they can see how new versions of Perl perform in 'real world' settings. Second they can watch changes in correctness (nd performance) over time. Third, the testing wizards in the Phalanx project can help module owners improve their own testing.
I'm working on a project I call Legion, with the intent of doing much the same thing. So far, I'm using four libraries from RubyForge with a total of 4062 assertions. I'm running the tests against the latest releases of Ruby, JRuby, and YARV. As I collect data from these tests, I can feed it back into each Ruby implementation project to help them build a better Ruby.
I also want to feed the data back into the projects who's test suites I'm using. If I can identify gaps in their test coverage, I'll certainly work at building tests to close those gaps. Perhaps we'll even find some bugs through more strenuous testing.
There are a couple of things Legion isn't doing yet. With some help, I think they'll be easy to implement:
- More platforms — currently, I'm only testing on x86 Linux. I'm planning on adding PPC OS X shortly. I'd love to add x86 OpenSolaris, Win32, and any other OSes people are willing to run Legion on (as soon as it's ready for public consumption).
- Better reporting — I'm still trying to figure out how much information to record, what to report on, and where to make those reports. I need feedback (especially from Ruby implementers) to make sure I'm providing a tool that's useful.
- More libraries — I'd like to add more libraries to the test suite. Right now, they need to be pure Ruby (i.e., no C), and need to have a non-Rake method of running all their tests. I may move to Rake to automate the testing, but then will need to write a Legion specific task for each library in the suite.
- More Ruby implementations — as soo as I have things a bit more stable, I'm going to add CVS/SVN versions of Ruby (both 1.8 and 1.9 trees), JRuby, and YARV. I'f like to add Cardinal, Metaruby, and other implementations as soon as they're ready.
I've already collected some data (and posted it to the dev lists for YARV and JRuby). I'd like to share the highlights here. As I can coordinate with the library maintainers and the developers, I'll open up the reporting a bit more. For now though, here's a high level view of the "big three" Ruby implementations according to Legion:
Ruby total time: 115.094538 seconds time/assertion: .02819 seconds total asserts: 4082 total failures: 29 total errors: 27 YARV total time: 12.012201 seconds time/assertion: .00842 seconds total asserts: 1425 total failures: 3 total errors: 146 JRuby: total time: 97.978000 seconds time/assertion: .04996 seconds total asserts: 1961 total failures: 29 total errors: 17
I noticed three things on a simple look at the tests:
- JRuby seems more compliant with Ruby than YARV right now.
- Ruby is nearly 2x faster than JRuby right now.
- YARV is ~3x faster than Ruby.