Hacking Night — May 2, 2006

We had a small group last night (only three people), but we still had a productive hacking night. Kevin, Devlin, and I decided to work on Ruby. Since we've got access to the Coverity scan data, we thought it would be worthwhile to look at some of the defects to see if there were reasonable ways to fix them.

We looked at four problems, identified one of them as a likely false positive, and provided patches for the other three. All of the patches ended up being one or two lines of code changes, little things that are easy to lose in a big pile of code. All of our changes have been posted back to the ruby-core mailing list, and hopefuly will be applied soon.

They may have to wait a bit though, as matz posted a message saying that he was in the middle of a large set of changes and it might be a couple of weeks before he applies incoming patches. Hopefully ours are small and simple enough that they can be snuck in quickly, without disrupting matz' work.

---

postscript — Today, as I sat on the bus in to Salt Lake, someone sat down next to me and opened up a copy of Agile Web Development with Rails. We started talking about Ruby, and it turns out that he works with the guys who are driving the urug/slc.rb. Since he lives down close to me though, I invited him to come out to our urug/utahvalley.rb meetings/hacking nights. Hopefully we'll have a new member to help bolster the ranks.

Addicted to Coverity

I was IMing with a friend today, and asked him if he was paying attention to the Coverity scan. He told me he wasn't, and I jokingly commented that it had the potential to be as addictive as fantasy baseball.

I little bit later, I realized that I was speaking more truly than I'd thought. I often catch myself looking at different projects and their defect rates, comparing one against another. I'm registered to look at the Ruby results, and occasionally find myself wondering what the other projects look like.

It is possible to see some of the rolled up data both on a per project and on a sitewide basis. For example, although Coverity identifies eight different classes of defects, the three most common account for over 60% of the total tracked defects.

There are some things I'd like to see -- a project's velocity in defects corrected over time and the average lifetime of a defect after identification (by project, sitewide, or by class of problem) for example. Some of the data could probably be harvested from their website, but doing that would mean admitting that I'd crossed the line and was addicted to coverity scan data.

I guess I should be glad there isn't a fantasy coverity project league out there.

Coverity and Velocity

I'm glad to see some gathering momentum around the coverity scans on the ruby-core list. I've posted a table of the 29 remaining possible defects there, and two of the possible defects in their entirity. (I'll be posting a third, which I think is a false positive) as soon as I'v finished this post.) The developers seem interested, and I know that a couple have registered with coverity.

I've been amazed to see the way the Perl and Python camps have attacked their defect list though. Perl is down to one verified defect and one uninspected possible defect (for an error rate of 0.004). Python has done even better and has no remaining defects (an error rate of 0.000). Wow!

Even more impressive is that there are eight projects with a defect rate of 0.000 and four more with a defect rate under 0.010. I hope we'll see Ruby joining them soon. It'll take some work to identify the false positives and fix the real problems, but we've made a start and it looks like people are interested in finishing the job.

Maybe once we get to zero defects, we can start chasing another big item — getting Ruby to run well under valgrind.

Coverity, 1 down and 29 to go

When I first brought up the idea of Coverity's scans, some of my Ruby friends told me that it would be a waste of time. "There will be too many false positives", or similar reasons were given. When the invitation to sign up for the scan readers was posted to the ruby-core mailing list and didn't seem to draw any interest, I was worried that they were right.

I was wrong. I decided to sign up with coverity, and post one of the errors to the ruby-core list. Less than twelve hours later, I got an email from Hidetoshi NAGAI. It said: "Than you for your report. I've fixed it on CVS."

Then I went over to the Coverity's Report and found that they'd already updated it to show just 29 defects and a 4% drop in the defect rate. I'm posting another defect now.

Coverity Code Analysis and Ruby

Recently, Coverity added Ruby to it's list of scanned projects (overview data is available here). Because Perl and Python were already in Coverity's target list, this allows us to look at a code quality comparison between these three languages. Below, I've listed each langage with its Lines of Code (LoC), current number of defects, original number of defects, current rate of defects per KLoC, and the original rate of defects per KLoC. I've had to calculate the last figure since it's not explicitly available on Coverity's wesite.

Lang	LoC	cur defects	defect rate
Perl	485,001	67	0.138
Python	273,980	14	0.051
Ruby	258,908	30	0.116

Python looks very good here, probably because they've been aggressive about attacking the reported defects. In the grand scheme of things though, All three languages look pretty good.

Lang	LoC	orig defects	defect rate
Perl	485,001	89	0.185
Python	273980	96	0.350
Ruby	258,908	30	0.116

This is where Ruby shows a bit better than the 'competition'. Ruby has a pretty good (low) defect rate before we've even gotten any feedback from Coverity. Hopefully we can be at least as aggressive about attacking our defects as the Python community. In fact, there's an even better target — AMANDA:

Lang	LoC	cur defects	defect rate
AMANDA	0	88,414	0.000

(By the way, they started out with a rate of 1.227)