cc.rb -> Integrity

A few months ago, after upgrading cc.rb to the newest version (coinciding with our move to GitHub), CI started going really slow. The load on the single CPU machine would often go over 10. CCTray/Menu would say CI was down, when really it just took >1minute to render anything.

We weren’t sure what was up, so after some reading, thought maybe it had to do with the number of historical builds we were keeping. So I cronjob-duct taped that to kill builds and kick off new ones nightly, but that didn’t seem to fix it.

We decided to try the CI instance on EC2 (funny enough, on a wussier box, but we didn’t think about that at the time), and EC2 turned out not to be the solution either.

Fed up, I switched us to Integrity, which has some nice notifiers (IRC!), a better user interface (imo), and had the possibility of being faster. After putting 5 projects on it, it wasn’t. Load was already up to 5.

Turns out that RCov is pretty CPU-intensive, and for some of our projects we were running it for every commit. After turning that off, load hasn’t been a problem, but I liked Integrity enough that I decided to stick with it. Unfortunately, it was missing a few features I really liked, like automagic build log updating and build locking and CCTray/Menu support (out of the box).

CCTray/Menu

Nobody on CCTray has made this work yet, so that’s a little misleading, but the CCMenu users have had success. There’s a gem called integritray that’s supposed to provide support with just a config line, but that wasn’t intercepting the requests correctly, so I ended up just dumping the integritray code into Integrity’s app.rb.

Build Log Auto-Update

So part of this I did really well (I think), and part of this I did terri-bad (definitely). First up, the way the build gets called is with IO::popen, which is all fine and dandy, except the build has to finish before output gets written to a field on a DataMapper Build object. This doesn’t work for live updating.

You could probably do all of this by just editing the build command per project, but I wanted to get my hands dirty with Integrity, so I opened up lib/integrity/builder.rb and changed the run method to this:

That’s going to do a few things. First, it’s going to write the output to stdout (for future consumption) as well as a build log named by the id of the build, and it’s also going to maintain the exit status from Rake (instead of tee, which is always going to be 0). I’ve only tried this on Bash, so $PIPESTATUS might not work if you’re not using that shell. Sorry. Next, I just gave the Build model a default value for output, so the method looks like this:

I added a cronjob that deletes all of the old build logs nightly, because after they’ve run, they’re saved to the Integrity database.

Build Locking

One of the nice things about cc.rb was that it didn’t try to start a new build for a project if one was already running. This might not be a problem for lots of projects, but with Rails projects with fixtures (ugh) and migrations, it can cause db deadlocks. I decided to add a rake task (that we use for all our Rails projects now) for Integrity to use locking. Here’s what that looks like:

So now doing a rake integrity:run will guarantee only one of those tasks runs at once. Note that it doesn’t guarantee to run the calls in order, and it’s possible that a build will run on a codebase that’s been updated since it started (I think I might be able to avoid that by loading up the Rails env before looking at the lock), but at least it prevents db deadlocks.

I haven’t added any of this as a pull request or anything to Integrity, because it’s not that polished yet, but feel free to pull whatever you see if it helps out.