Thursday, January 22, 2009

Looking for serious Ruby Hacktitude?

I just noticed that zenspider and some of my other buddies up in Seattle.rb have hung their shingle out again. If you need some big Ruby guns, these guys are the deal — hire them now before someone else does.

Besides, if they're not working on your project, they're liable to extend Wilson to produce PPC output or something equally dangerous. For the sake of humanity, don't let them sit too long.

Wednesday, January 14, 2009

Ruby Best Practices: mini-Interview 2

As Gregory Brown (@seacreature) and I discussed the Ruby Best Practices Contest, we also talked about the current state of Ruby Best Practices (or get the Rough Cut). You can read the resulting interview below.

Before we get to that though, we should announce the tow winners of the contest: Jamis Buck (actually entered by Daniel Berger) and Eric Hodel. Eric and Jamis both get free Rough Cut Access to Ruby Best Practices — who knows, they might even find some of their own ideas in there.


How is the writing process itself going? How are you dealing with feedback from readers/reviewers?

Gregory The process we've put together really seems to be working. When I first pitched the book, I set the pre-condition that we'd need to have a broad panel of advisers from the get-go. If this book is going to have the title "Ruby Best Practices", I want it to be able to stand on more than my own name, for sure. So I asked a few of my friends, a few of my students, and a few folks I think of as truly masterful Rubyists to be part of an active review process. So far, that has been working really well.

Every chapter gets two levels of review, at a minimum. I start by posting to our internal reviewers a plain text file and a PDF, and collect their comments. After a about a week, I make revisions based on their suggestions and send things off to O'Reilly for a RoughCut update. Usually within a day or so, the changes hit Safari and those who pre-ordered the book can comment on them. That group has already caught a couple problems and made suggestions, so it's definitely useful to have more eyes on the content. I think it's really important to release changes as often as possible, to encourage further review. Of course, we use some great tech to help us with that.

Because the book's sources are in Asciidoc, there is no separate typesetting process. My plaintext files are the same source that the PDF is generated from. This means I can make whatever revisions I want whenever I want, and I don't need to be worried about getting bogged down by the formatting. We can also go back and make changes to old chapters later on without much worry.

So far, O'Reilly has basically let me run things exactly how I wanted to, and it has been working out great. Fast, tight iterations will help make the book grow organically based on the input from our internal reviewers as well as the wider reader base. That's what I'm hoping for, anyway.

I hear that you're converting testing examples from RSpec to Test::Unit is pretty huge. Why are you making the change?

Gregory Oh, for the "Driving Code Through Tests" chapter? I just sent that out to the internal reviewers, so we'll see what they think. :)

But actually, it was their suggestion. The initial RSpec based chapter was meant to reflect the change in the tides in the Ruby community, but it wasn't too hard to convince me that I should reconsider things. I used to think RSpec was easier to teach than Test::Unit until I wrote a chapter about it, and found myself having to show reference implementations of how things like foo.should be_bar work, whereas assert foo.bar? needs no extra explanation. I had to explain what a proxy object was for folks to understand how to build custom matchers, and what a lambda was for people to know how to test their exception raising. I'll be blunt, that sucks.

But to be fair, I use RSpec for a good bit of my work, and I think it's okay once you get the hang of it. But the ideas I wanted to express in RBP weren't really tech-specific. I was mostly talking about overarching strategies like keeping your tests atomic, writing some helpers here and there, that kind of stuff. So some of the 'features' of RSpec were actually getting in the way. Test::Unit (well, now minitest/unit I guess), is standard Ruby, is dirt simple, and was able to demonstrate the strategies I wanted to share with folks with less magic.

So, maybe it's not such a big change. RSpec, Test:Unit, Minispec, Test:Spec ... How much does the specific tool really matter versus being serious about using it?

Gregory The interesting thing is I didn't really need to change the chapter much, aside from translating the source. Some sections were dropped and others will be added in, but for the most part, it's the same chapter and it follows the same general blueprint. This really underscores the fact that what matters are the ideas behind testing, not the technology. I'm hoping the reader will benefit from the simplicity of Test::Unit for my examples, but then take the knowledge and apply it wherever they want, whether it be to RSpec, Shoulda, or whatever it is the cool kids will be using when the book hits the shelf.

I think the specific tool matters, but only when it comes to comfort, not concepts. It's arguable that certain frameworks encourage you to embrace a concept more than others, but I don't think that's as big a factor as people might think.

How important is mocking as a Best Practice? What about Stubbing? Where do they fit in relation to each other?

Gregory Well, I think that mocking and stubbing are both pretty important when you need them, and completely worthless when you don't. I've seen systems in which every interaction with every other class was fully mocked out with all the behaviors specified. These systems are absolutely nightmarish to work with, because the tests become so brittle that you need to re-write your giant, complicated mocks every time you refactor. Tests should aid in refactoring, not get in the way, whenever possible. I've also seen (*cough* written *cough*) systems that stub out so much and produce canned results for everything which gives you wonderful passing tests but, well... don't actually test anything.

But of course, responsible use of mock objects to minimize your dependencies on expensive external resources can really help make your tests run faster and more clearly show what's going on. Stubbing out a method here or there to prevent you from testing things you don't care about can definitely be helpful. In general, I prefer to use whatever technique is the right combination of cheap, fast, and easy. That combination changes wildly from project to project so anyone who tells you that you should never make use of mocking / stubbing is crazy, and so are the people who want you to do it all the time.

As far as where all these things fit together, I think that's more a question for folks with a stronger grasp of theory than me. The best I can do is point you at Martin Fowler, then tell you that I don't really care about their distinctions at the high level. I only care about using the right approach for the job. Libraries like flexmock that intentionally blur the lines between mocks and stubs are great in the hands of responsible users, if you ask me. :)

While Ruby Best Practices is going to end up Free, O'Reilly has even more open in the writing of a couple of other books that are cropping up. How does this go 'round compare with your completely open work on the Ruport Book?

Gregory Well, the Ruport Book taught me a lot, both good and bad, about writing a community oriented book. The idea of having an internal review team is something we had on the Ruport Book as well, and I carried it on to RBP. It seems like because the topic is more general, this has worked out much better on this book than it did for the Ruport Book, simply because most of my internal review team knows a hell of a lot more about Ruby than I do!

But the Ruport Book and RBP are two very different books, so it's hard for me to compare them.

I'm having difficulty expressing my feelings on this, but basically, the way this book is formed in my head doesn't feel much like a textbook. It's more like a quasi-fictional narrative "based on a true story" with notable mentions of various Ruby hackers and their projects. It's sort of like I've stitched together a series of blog posts so that it bleeds into this textbook undercurrent. The book is meant to be read from cover to cover, or at least in whole chapters. This makes it tricky to make wide open in the early phases.

But once the work is done, I want it to be able to bend under the pressures of peer review. I want people to be able to freely use and remix what I've done, however they see fit. And of course, that's already planned to happen and so it will, it will just take some time.

I believe it can be a virtue to keep something a little quiet until you know for sure where it is going. It's sort of like waiting to do a first release of an open source project until you clean up the code a bit and make it useful and approachable to others. That's exactly what I'm doing with RBP now, and hopefully, that will benefit those who might be interested in contributing to the book down the line.

Lots of people coming into Ruby from other languages bring Best Practices (and sometimes Pessimal Practices) with them. What languages/communities do you think offer the best opportunity for finding good practices for Ruby?

Gregory That's a good question. I don't really know. I think we end up getting more bad patterns than good from other languages. This isn't to say anything bad about other languages, but it is to say that best practices aren't always transferable. Look how many patterns from Java aren't even remotely relevant in Ruby. Or how many cool functional programming techniques fall down and die when you try to import them. I think maybe the fact that we only got a few responses to our blog contest here was an indication of that, to some extent.

However, I think many communities within Ruby are a hotbed for best practices. RBP draws almost all of its key points via the influences of notable open source Ruby projects. Rails utterly changed the way people write Ruby, in many ways for better, in other ways for worse. Merb has really made me re-think issues like performance. Smaller libraries also pack a punch. FasterCSV is the source of a lot of inspiration for me, and in turn, I think FasterCSV has been influenced to some extent by Ruport. After-all, it has its roots there, anyway.

So if you pay attention to the conversations between well trodden Ruby projects, both in words, and in code, you'll see a whole lot of great strategies emerge. My hope is that this book will give people a snapshot of some of them so that they can pick up new skills in the way many of us hackers are doing: One new patch at a time.

I see that you've set up a twitter account for RBP. How do you think that will impact the book?

Gregory It'll motivate me a little to keep posting updates, give me a way to keep people informed about book related news (such as this interview), and give readers (or potential readers) a way to connect with me and share their thoughts. I'm not a big twitter addict, but I'll at least let people know when to expect new content and whenever something I think might be interesting comes along. Beyond that, I guess we'll see how it goes.

Tuesday, January 13, 2009

Sequel Interview with Jeremy Evans

With the recent release of Sequel 2.9.0, I've finally taken the time to interview Jeremy Evans, the project's maintainer. Back in Jan 2008, I interview Sharon Rosner (the former maintainer). Sequel has come a long way since then, so it's about time I revisited the project.


You took over Sequel from Sharon Rosner almost a year ago. How and why did you end up holding the reins on this project?

Jeremy My first experience with Sequel was with Sequel 1.2, in February of 2008. I was looking for an additional ORM to support in my Scaffolding Extensions project, and had already added support for ActiveRecord and DataMapper. At the time, Sequel didn't really have support for model associations, and the recommendation was to just implement your own methods to get associated objects.

I worked on a simple associations implementation for Sequel that was only around 60 lines and handled the three main association types (many_to_one, one_to_many, many_to_many) with full reflection support. I posted on the Sequel mailing list with it and Sharon liked it. It became the basis for the association support Sequel introduced in 1.3, and I was given subversion commit rights.

In the middle of March 2008, Sharon emailed four of the developers with subversion commit rights and asked if we could assume leadership of Sequel, since he was planning to move on from programming and do something completely different. I was one of the two developers that responded, and initially I was only going to be responsible for sequel_model. I ended up taking over maintenance of sequel_core a couple months afterward, since the other developer didn't have enough time to maintain it.

With Merb being merged into Rails, do you see a future for Sequel in Rails 3.0?

Jeremy Yes. I'm guessing the merb_sequel plugin will be ported over to Rails 3.0.

It's fairly easy to use Sequel in Rails already, just by requiring it and setting up a database connection with Sequel.connect. I have 5 Rails projects that use Sequel in this manner.

It's good to have people using the projects they're working on. What kinds of changes have you made to Sequel based on the way you use it?

Jeremy The association code was designed to be easy to use by my Scaffolding Extensions project. The eager loading code was designed to be fairly similar in usage to ActiveRecord's, making it easy for me to convert projects from ActiveRecord to Sequel.

On my largest Rails project, I often had cases where associated objects needed to refer back to their parent objects:


 artist.albums.each{|albums| p album.artist}

That causes issues because ActiveRecord will do a query for each album to get the artist. I originally handled this in ActiveRecord by doing something like:


 Artist.has_many :albums, :include=>:artist

I never liked this way of doing things, so I changed Sequel's association implementation so that the parent association is cached (the ActiveRecord parental_control plugin does something similar, I hear).

I only add things to Sequel itself if they are generic and not tied to any specific implementation. The most recent example of this is support I added to Sequel for Giftsmas. Giftsmas runs on PostgreSQL and uses triggers to handle some constraints. I added {create,drop}_{language,function,trigger} methods to the PostgreSQL adapter, which are generic enough they can be used by any PostgreSQL user. I then created the sequel_postgresql_triggers project, which has specific implementations of some common column types (counter/sum cache columns, immutable columns, timestamp columns). Giftsmas uses sequel_postgresql_triggers in it's migration to set up the database.

I didn't add the code in sequel_postgresql_triggers to Sequel itself because they are specific implementations, and there are other (perhaps better) ways of doing the same thing.

What kind of tools are you using to help with code quality — test coverage, complexity, refactoring, etc.?

Jeremy I test coverage before every release, anything less than 100% gets fixed. 100% code coverage means nothing, but less than 100% code coverage means something.

I don't use any IDE refactoring support (I code in vim and SciTE depending on the situation). I don't use any ABC metrics to measure code complexity. There are parts of Sequel that could definitely do with refactoring, but I generally wait to refactor until I'm going do make changes to the code that refactoring will help. I don't refactor for it's own sake.

I use TDD when the problem space is known and I know what outputs I want for each input. I'd say I do TDD for new features about 50% of the time, and about 90% of the time for bug fixes.

In an interview with InfoQ, you talked about eight things you thought had gotten better with Sequel. What's improved since then?

Jeremy That was shortly after the 2.7.0 release, so the major improvements since then are:

  • Database stored procedure support in the MySQL and JDBC adapters.
  • Much better support for database schemas.
  • Much improved compound SQL statement support (i.e. UNION, EXCEPT, and INTERSECT).

For full details, please see the release notes:

What do you see happening in the next major Sequel release?

Jeremy I expect the 2.10.0 release of Sequel to include:

  • A Firebird adapter.
  • A DataObjects (the underlying database connection libary used by DataMapper) adapter.
  • Better handling of MySQL CREATE TABLE options, such as the ability to specify an engine.

I don't have any other major plans for 2.10.0, but most of Sequel's new features originate in the community (in ideas if not code), so I'm sure there will be other improvements.

Sequel 2.10.0 will probably be released in the first half of February 2009.

That's pretty aggressive. How do you keep your release cycle short?

Jeremy I put all but the simplest commits through the same test suites I put the releases. Releases are generally time based, not feature based. If a feature isn't ready and is going to hold up the release more than a week, it just goes in the next release. For example, the firebird adapter is already mostly complete and was going to be in 2.9.0, but the developer working on it asked for a little more time to polish it, so it didn't make 2.9.0 and will have to wait for 2.10.0.

How closely do you watch other ORMs (Ruby or non-Ruby)? Which ones seem most interesting to you? What do you learn from them?

Jeremy I browse the DataMapper mailing list and periodically chat with the DataMapper developers on IRC. I generally look at the "What's New in Rails" posts to see what is going on in ActiveRecord. However, I don't follow either very closely. I also spent a bit of time looking at Lone, a PostgreSQL-specific ruby ORM.

DataMapper is interesting in that their focus is very different from Sequel's, in that they are aiming to be a persistance framework for generic classes. I don't have any experience with their recent code, though, the last time I used DataMapper was in the 0.2.5/0.3 timeframe.

I'm sure ActiveRecord taught me a few things, as I used it for years, but that's probably at a more subconcious level. Lone's prepared statement support gave me some ideas for the prepared statement interface in Sequel. I don't think I spent enough time using DataMapper to learn anything specific.

Why do you think sequel is a compelling option for a Ruby ORM?

Jeremy

  • Sequel code is generally clear and consise, and it has a very rubyish syntax.
  • Sequel::Model's associations are the most powerful of any ruby ORM, as Sequel allows the user much more control over how the associations work.
  • Sequel is especially compelling whenever you are dealing with sets of objects instead of single objects.
  • Sequel is very easy to contribute to. You don't need 3 +1s. Submit a pull request, post on the mailing, or ping me on IRC and I generally review patches quickly and decide if they are a good fit or not.
  • Bugs in Sequel are generally fixed quickly. The Sequel bug tracker has no open bugs currently, and that is how it is most of the time.
  • Sequel's internals are clean and easy to extend and modify if you need to.
  • Sequel's connection pool doesn't require the user or framework clean up connections manually.

Any closing thoughts or advice for people looking at Sequel?

Jeremy The Sequel community is very friendly, so if you have any issues getting things setup or have questions about how things work after reading the documentation, stop by the Google Group) or IRC (#sequel on Freenode) and hopefully we can help you out.

Thursday, January 08, 2009

Editor Interview: Talking about Open Content with O'Reilly's Mike Loukides

With a recent string of interviews with authors working on (open) books for O'Reilly, I wanted to see what the folks inside O'Reilly had to say about this trend. Mike Loukides (@mikeloukides) was good enough to answer my questions in a short interview. There's some great stuff in here whether you're an aspiring author, interested in open content, or how thinking conversations (in the web 2.0 sense) impact book marketing. Read on to see what Mike had to say.


Mike, it looks like you guys are riding a string of Creative Commons books either out or in the pipeline (Real World Haskell, Relax with CouchDB, and Ruby Best Practices). Historically, you've done a number of other open/free books as well. I'd like to pick your brains a bit about your willingness to head down this road.

Noah Slater told me "Our editor told us a surprising rule of thumb, that releasing a good book under a free license makes it sell more copies, and releasing a bad book under a free license makes it sell less copies." Why do you think this is?

Mike First: I'm not trying to back down from what I told Noah, because that is what I tell authors. But there's no hard data, and very little soft data: it's just our sense of what happens. It would be next to impossible to do a controlled experiment.

And I do want to correct one thing. I don't think in terms of good books and bad books; I think in terms of successful and unsuccessful. Plenty of good books are unsuccessful. What really drives success is the community that forms around the technology and the book.

That said, the mechanism is fairly simple. If there's a strong, thriving community around the technology—let's say CouchDB, since you're talking to Noah—the free online edition of the book will increase buzz, and make more people aware of the print book. A lot of people will download the online book, and decide they want the print book.

On the other hand, if the community around the technology is small, isn't thriving for one reason or another, etc., the existence of a free version will soak up what limited demand exists.

So I think free licenses for books is an intensifier: if a book is going to succeed, a free license will make it more successful. If it's going to fail, a free license will make it fail worse.

In the same vein, what can prospective authors do to make sure the open book they want to write is a good book? What makes the proposal stand out and say "I'd be a winner under a free license!"?

Mike Although I have a fair amount of ego tied up in making sure the book itself is as good as possible, I think in the long run it's only partially about the book. It's really about the technology and the community. If a community is growing, and people are excited, a good book will be successful (free license or not).

That said, there is an awful lot that authors can do to make their book more successful. None of this is magic: blogs, trade show talks, tutorials, all of that. A free online version of the book gives you a few more tools to play with. The authors of our Haskell book have done an excellent job of motivating the Haskell community.

As far as the book itself: readers want practical books. Readers want books that help them to solve the problem. If Noah and the other CouchDB authors had wanted to write a couple hundred pages explaining the principles behind CouchDB, non-relational databases, REST, and so on, without a single line of working code, it would be a disaster. I do get lots of proposals like this. They're generally disguised as "books targeted at management".

That's not to say that I'd turn down more abstract books on topics like software engineering. But I also think the case for writing that kind of a book with a free license isn't as strong. Would Martin Fowler's Refactoring or Kent Beck's Test Driven Development have been more successful if there was a version with an open source license? Possibly, but I don't think it's as clear a case.

O'Reilly seems to embrace open, flowing communication. You've got active bloggers, you work closely with User Groups, and you seem to have jumped on Twitter in a big way. How does this willingness to have conversations with your customers change the way you bring books to market?

Mike That's probably a better question for someone in Marketing.

But yes, all of these things give us more tools to work with. It certainly helps to get people talking about a book early on, it certainly helps to get people motivated and excited so that they want to read the book. And with some books, like the Haskell book, we got huge amounts of technical feedback from the public. That was a real help in making it a great book.

Other than technology titles, what other genres do you think would benefit from more openess (license, development, and conversational)?

Mike That's a really interesting question. One thing I like to point out is that we almost lost Shakespeare's entire works because there was no such thing as copyright protection in the 17th century. Plays were trade secrets, and the few plays that were published were generally published in unauthorized editions: several years after the fact, a few actors sat around a table and tried to remember the lines.

At the same time, I think the DMCA was a ridiculous intellectual property land grab. Music has always thrived on artists ripping off bits and pieces from each other: that's really central to how musical creativity works. But in the current climate it's entirely too easy to write a song and end up being sued because it's similar to some song that was published 5 years ago and happened to be buried in your subconscious. (Part of the problem in music, I think, is that you're dealing with relatively short sequences taken from a relatively limited repetoire—24 notes in a typical singer's range, 88 notes on a piano keyboard.)

It's worth noting that Noel Paul Stookey, from Peter, Paul, and Mary, started something equivalent to the FSF back in the late 70s. So free licensing didn't start with RMS and the GPL. (I'm trying to look up info on what Paul did back then, but I can't find it now. If you dig into this and find anything, I'd appreciate a link. I don't think he got very far with it. But in 78, I don't think it was needed the way it is now.)

Sticking with books: I think anywhere you can build a book out of a conversation, you'll do well. We've got some interesting experiments coming out: 97 Things Every Software Architect Should Know (Feb.) was built on a wiki, with contributions from roughly 40 software architects. We had the book about halfway done, with contributions from a couple dozen architects, and then opened it to the public. The response was great.

So, if you start from a conversation, almost anything is possible. There's a sense in which all the design patterns books are really about conversations. So can I see freely licensed books in fields like software engineering? Definitely. Can I see it in science? It runs against the way scientific institutions currently work, but open, collaborative science—doing science in public, on a wiki, as it were—was discussed a lot at our last SciFOO camp, and you should have seen how excited people were. Science changes a lot when it becomes more open and less tied to traditional publishing institutions.

Sunday, January 04, 2009

New Year's Road Trip

In December, we heard that the Sahpeech chapter was going to hold a Fellowship with a Brotherhood ceremony in Gunnison on January 2-3. Since we had an Ordeal candidate and a Brotherhood candidate just waiting for the opportunity, we decided to take a road trip.

On Friday afternoon, we packed up our gear (and lots of warm clothes) and dinner to eat on the road, and we headed off. The youth talked about scouting, movies, and the things they'd done over Christmas break, then they settled in to a discussion of the OA and upcoming chapter events.

We weren't the only travellers at the fellowship. There were two young men from the Nez Pierce chapter, two young men and an adult from the Sioux chapter, and one from the Todebeda Cheda Toonle chapter down in St. George. It was pretty cool to see arrowmen and candidates from all over the lodge.

The candidates, elangomats, and other arrowmen present put in a lot of service. We cleaned an elementary school from top to bottom, mounted a bunch of video projectors on classroom ceilings, and shoveled a lot of snow to clear the elementary school and nearby High School. We even retired a flag while we were there.

Before wrapping this up I'd like to welcome Ian, our newest Ordeal member, and congratulate Mike, our newest Brotherhood member. Mike's also the Ordealmaster for our upcoming May Fellowship and he's looking for elangomats to serve there, if you're interested please send an email to lakotasecretary@gmail.com and I'll pass your name and contact information along to him.

Thanks to the Sahpeech arrowmen for all the work and planning you put into this.

Oops, somehow I managed to post this to the wrong blog ... oh well. If you're interested in my scouting and OA activities, have fun with it. If not ... just ignore it and I'll post something more pertinent to 'On Ruby' early this week.