Will rubinius Be An Acceptable Lisp

Yesterday (Wednesday, January 10th, 2007), there was a short discussion on the #rubinius irc channel which prompted a few questions which I thought would be best asked and answered here. Before I get to them though, I thought I'd share some context:

olabini: pate said something about Lisp on rubinius? wanna elaborate Evan? I'm drooling at the thought...

evan: Sure, since i started the project, i've always wanted to write a parser for a lisp-dialect that i could feed to the existing compiler/assembler

olabini: Yeah, pate said so. it sounds very interesting. My first thought is it sounds nice to be able to mix and match ruby and lisp within the same runtime. The basic Ruby operations should make it very simple to create a basic Lisp dialect.

---

So, what would kinds of uses do you think would Lisp on rubinius really see?

MenTaLguY: Well, speculatively, it'd pretty much be a Lisp dialect with Ruby's smalltalk-esque object system bolted on, I guess. Maybe otherwise undefined functions would be tried as method calls on their first argument? Most likely people would be doing things like using lisp macros to glue together DSLs that would be grotty in straight Ruby. You'd probably also see Ruby methods implemented in Risp when there was a compelling reason to do so (e.g. because it was easier in Lisp or because the author had a Lisp fetish).

It's interesting to note that (unless I'm making this up) Ruby itself started out as "matzlisp", a Scheme dialect (as still evidenced by many things like Bignums, the numeric tower, callcc, immediate types packed into VALUE, symbols, and the general feel of the C API). So it'd be sort of a return to Ruby's roots.

Ola: For me it's a matter of taste. I love those parenthesis.

No, what's more interesting is combining Ruby with code that can do real macros. That's one thing I really miss in Ruby, and also something I have written about in my blog on occasion. The most prominent use case would be writing some of the code in pure, simple Lisp, have a few macros that transform these in Common Lisp-style, and be able to require that file and call the methods defined from regular Ruby-code. Once again it's one of those best things from both worlds stuff. Would it be possible for me to write Lisp and fall back on Ruby when I wanted too, integrated with Ruby, it would be like heaven. Almost. =)

Evan: I'm motivated by the ability to have a simple language that I can use to write tests to stress different parts of the VM. It should be noted that I've only really done lisp once and it was for class. I'm mainly drawn to it because the parser is dead simple and the grammar could use all the functionality of the VM.

Is this really putting the cart before the horse? How do you know rubinius will even be stable/performant enough to handle this?

MenTaLguY: No. It's always time for Lisp.

Realistically, I'm sure Ruby-on-Rubinius is going to take priority anyway.

Ola: Regarding performance, it's very easy to get Lisp performing well enough. And since Rubinius is built on a Smalltalk VM architecture, it's operations fit very well with Lisp. A Lisp on Rubinius would perform as well as Ruby on Rubinius, no doubt. (Remember that the first Smalltalks were implemented in Lisp, btw).

Regarding stable... Well, another language will use the machine in different ways, which would possibly help increase stability. And the more interesting use cases you can find for Rubinius, the more hype it will generate, and the more people will contribute. So, the question isn't really if it's stable enough. Stability can be a consequence if it instead.

MenTaLguY: If it's stable/performant enough for Ruby, it's stable/performant enough for Lisp. But let's say it's not -- them the demands of implementing Risp will make it so. Everyone wins.

Evan: Sure, It will run the same speed as ruby, it's going to get compiled down and run on the VM. It should be noted that when I said lisp, I actually said a lisp-dialect. I don't indend for this to confirm to ANY lisp standard out there. I only expect to have a lisp that lets me perform the same operations you can do in ruby.

Would a programmer really be able to mix and match as they went along?

Ola: They should be. The probable delimiter would be on file-basis, or possible a RubyInline-variation.

Evan: Since the lisp will compile to rubinius bytecode, sure!

MenTaLguY: Don't see why not. Mixing in the same file, though? I don't know. Maybe by playing Dylan-like syntax games. The other option would be string evals, but I have a rather low opinion of those in general.

What are some of the bigger technical issues you see involved in making this happen?

Evan: Well, writing a simple lisp parser first, and then continuing to stabilize the VM and compiler.

Ola: There are no real technical issues. Implementing a basic Lisp is very easy. Rather, the two big difficulties I see are not technical at all.

First of all, some of us need to have time enough to do it. Secondly, we should decide some ground rules for what Lisp constructs should map to which Ruby constructs. (For example, should all Lisp methods be added to Object, or can we add something like CL packages, that map to modules or classes? Should it be able to use XSTR syntax in the RL Strings? (XSTR is strings like "abc #{3+2}")) Lots of fun issues to think about.

Another thing that needs to be decided is how the macro support should look. Should it by hygienic or all-powerful? Or both?

MenTaLguY: Figuring out what to take from other Lisps. I don't expect Risp to be a Common Lisp -- the Common Lisp standard library is nice, and it's standard, but it's huge, maybe a little redundant and the naming conventions are so ... un-Ruby.

What follows are personal preferences; don't take this as any kind of statement about what Rubinius might actually do.

As far as syntax goes, I'd like to take some cues from MISP actually. (I've collected links to MISP postings here.)

In particular, I'd like to see this familiar-looking syntax from MISP:

{|x| ...}

being shorthand for:

(fn (x) (...))

[fn being MISP's name for lambda]

An important thing being that this would be shorthand (as in Arc), not new syntax (as in Dylan).

Other than MISP, probably stealing anything that isn't nailed down from Paul Graham's Arc writings. At least the bits that make sense.

One other big deal is going to be making arrays pleasant to use. Since people are going to want to use Risp with the Ruby libraries, arrays are going to have to be more natural to work with in Risp than most Lisp dialects make them.

I think that means at least that Arrays should be usable anywhere a list can (the cdr/tail/rest of an Array would be an external iterator into that array which duck-types as a Pair and #to_a's to an appropriate Array slice).

Ola, you've mentioned wanting to see a Java based rubinius. Does this mean we should be watching out for JLispR or something?

Ola: You can count on a JRuby-based Rubinius. It will happen. And making a Lispinius possible on JRuby would be a major reason for it.

MenTaLguY: JRisp?

And for the questions that's on everyone's mind — Might this be a way to get macros into Ruby?

MenTaLguY: Hygienic macros, please. The fact that they avoid name collisions is just icing — their real value is that they're easier to reason about and an IDE can do smarter things with them.

Ola: Probably not. Or not completely. It would allow macros to exist in the Ruby VM, but not actually expanding Ruby-code.

Evan: It depends. Lisp macros are easy because they take lisp code/data in and output lisp code/data. For ruby to have macros, it has to be able to take ruby code in and at least output something the compiler understands. A macro that accepts ruby code on it's input would have to incorporate that code into the output, which would mean either outputting ruby code or converting the ruby code into another interpretation to be incorporated and output.

That means you probably couldn't have a macro that takes ruby code in and just outputs lisp because it would be lisp with ruby code as strings stuck in the middle.

Something that is possible is to leverage the fact that currently, the compiler takes sexp's as input and turns that into bytecode. So perhaps you could do something like...

(macro debug (code) (
(if $DEBUG (
   ('puts (to_sexp code))
))
))

so that code is ruby code that is converted to a sexp at compile time and the integrated into the output from the macro, which the compiler then processes.

But it might not work either, I've just come up with this off the top of my head. :)