terrbear.org

cc.rb -> Integrity

A few months ago, after upgrading cc.rb to the newest version (coinciding with our move to GitHub), CI started going really slow. The load on the single CPU machine would often go over 10. CCTray/Menu would say CI was down, when really it just took >1minute to render anything.

We weren’t sure what was up, so after some reading, thought maybe it had to do with the number of historical builds we were keeping. So I cronjob-duct taped that to kill builds and kick off new ones nightly, but that didn’t seem to fix it.

We decided to try the CI instance on EC2 (funny enough, on a wussier box, but we didn’t think about that at the time), and EC2 turned out not to be the solution either.

Fed up, I switched us to Integrity, which has some nice notifiers (IRC!), a better user interface (imo), and had the possibility of being faster. After putting 5 projects on it, it wasn’t. Load was already up to 5.

Turns out that RCov is pretty CPU-intensive, and for some of our projects we were running it for every commit. After turning that off, load hasn’t been a problem, but I liked Integrity enough that I decided to stick with it. Unfortunately, it was missing a few features I really liked, like automagic build log updating and build locking and CCTray/Menu support (out of the box).

CCTray/Menu

Nobody on CCTray has made this work yet, so that’s a little misleading, but the CCMenu users have had success. There’s a gem called integritray that’s supposed to provide support with just a config line, but that wasn’t intercepting the requests correctly, so I ended up just dumping the integritray code into Integrity’s app.rb.

Build Log Auto-Update

So part of this I did really well (I think), and part of this I did terri-bad (definitely). First up, the way the build gets called is with IO::popen, which is all fine and dandy, except the build has to finish before output gets written to a field on a DataMapper Build object. This doesn’t work for live updating.

You could probably do all of this by just editing the build command per project, but I wanted to get my hands dirty with Integrity, so I opened up lib/integrity/builder.rb and changed the run method to this:

That’s going to do a few things. First, it’s going to write the output to stdout (for future consumption) as well as a build log named by the id of the build, and it’s also going to maintain the exit status from Rake (instead of tee, which is always going to be 0). I’ve only tried this on Bash, so $PIPESTATUS might not work if you’re not using that shell. Sorry. Next, I just gave the Build model a default value for output, so the method looks like this:

I added a cronjob that deletes all of the old build logs nightly, because after they’ve run, they’re saved to the Integrity database.

Build Locking

One of the nice things about cc.rb was that it didn’t try to start a new build for a project if one was already running. This might not be a problem for lots of projects, but with Rails projects with fixtures (ugh) and migrations, it can cause db deadlocks. I decided to add a rake task (that we use for all our Rails projects now) for Integrity to use locking. Here’s what that looks like:

So now doing a rake integrity:run will guarantee only one of those tasks runs at once. Note that it doesn’t guarantee to run the calls in order, and it’s possible that a build will run on a codebase that’s been updated since it started (I think I might be able to avoid that by loading up the Rails env before looking at the lock), but at least it prevents db deadlocks.

I haven’t added any of this as a pull request or anything to Integrity, because it’s not that polished yet, but feel free to pull whatever you see if it helps out.

Secure Passwords sans SSL

I use a lot of webapps that don’t use SSL. I don’t think this is necessarily bad – SSL costs money, is slower, introduces development and IT headaches – but I do worry that I’m sending my password in cleartext.

I set out tonight to offer a solution for that. I haven’t tried to work it into any existing plugin, but just the links and the code should suffice for someone who doesn’t want to let passwords go readable across the wire.

I decided that I would send an HMAC using the SHA-256 encrypted password as the key, and the username and a unix style timestamp concatenated as the data. To make this work I dropped password salts. I haven’t decided if it’s safer to send the salt to the user or just to drop it. Hints in this area would be appreciated.

Prerequisites for this are jQuery and the ridiculously-easy jQuery sha256 plugin.

To start things out, I created a Rails app (using 2.3.4, if you’re curious) with a User model that had a login and password field. I meant for the login to be, well, the login, and the password to be a SHA-256 hexdigest of the actual user’s password. If you adapt this code to work with RestfulAuthentication, which handles the password digests nicely, you’ll have to solve the salt problem mentioned above either by dropping it or sending it to the client.

Here’s how I set up my user:

Then, I needed a login form. Here’s the relevant HTML:

As you can see, as soon as a user submits the form, I hash the password and create the digest, and then blank the password field out. Obviously that last part is important, because otherwise you’re sending the user’s password as cleartext.

This code is more of a jumping off point than a landing place, but here’s what my (trivial) action to login looks like:

The meat and potatoes of the authentication goes in the User model, and the code looks painfully similar to its Javascript counterpart:

Essentially this goes, “ok, if you took a hash of your password locally and then used that to hash whatever timestamp you received and your username, and I get the same result on my end, it’s really likely that you’re the right user.”

Now, this is almost secure, but not quite. There’s still nothing preventing someone from re-posting the same data and logging in. This is just as bad as simply sending the digested password over to the server. I suggest (if you’re not using SSL), sending neither the password or the digest of the password itself over the wire.

The benefit of using the timestamp is that you can let the client send 1-time valid digests. So, I added a new model to my Rails app, LoginRequest, consisting of a timestamp attribute, and a User has many of them.

Here’s what the more secure version of authenticate looks like:

Now, even if a bad guy snoops everything you sent to the server, he (or she!) can’t reuse it, because that timestamp’s been used. Changing the timestamp should significantly change the hash, and without the secret information necessary for the encryption (the password hash), it’s not feasible to guess.

Also, brute forcing against a single timestamp isn’t an option, because even *failed* login attempts are registered as a LoginRequest. You get one try per timestamp to login.

If you wanted to, you could even put a check in the authenticate method that required that the timestamp be from the last day or some other arbitrary time limit.

So, if you aren’t storing important data, and for some reason can’t use OpenID, this seems to be a candidate for at least keeping passwords from going out in the open. I’m pretty sure there’s some hole in it that I haven’t seen, so if you see something, please put it in the comments so I might fix it. Thanks!

* Note: I used OpenSSL’s HMAC functionality over ruby-hmac because (1) the author of ruby-hmac suggests as much and (2) Nathaniel Bibler showed significantly better performance from the OpenSSL implementation here: http://blog.nathanielbibler.com/post/63031273/openssl-hmac-vs-ruby-hmac-benchmarks

Async Observer sans Rails

My favorite background job pairing is beanstalkd with async_observer. I think the code that comes out is clean, easily pushed out to a queue, and beanstalkd is stupid easy to get going.

Both at work and on a side project, I’ve run into a situation where I need to put something into beanstalkd to be processed by a Rails-backed worker, but don’t want to load up the Rails framework to do that. Easy example: receiving email, I’d like to queue the processing how Jason Seifer suggests, but don’t want to have my own beanstalkd tube dedicated to email processing.

When you do an async_send, it just gets serialized as a string out to beanstalkd, with something like Model.find(id).method()

So if you want to push stuff into your queue in a rake task without having to load up the Rails environment, just use this method:

At work, we have a job system, and cron runs every 2 minutes to find scheduled jobs that should run and put them into the queue. Initially this was coded as a Rails-backed rake task, which meant every 2 minutes you’d see a ruby process load up for 10-15 seconds and steal 95% of a CPU. As we had more applications on the same machine needing jobs loaded, this wasn’t effective. Making the task only use beanstalk-client (and mysql), the task now runs in <1 second and doesn’t even register on top.

You’re Right

A few weeks ago, there was a fun argument over a feature at work. It was split down the middle. Feature arguments are a lot of fun, because everyone gets to pretend they know what the ambiguous “User” wants, when nobody really does.

The lines were clearly drawn, I was on the opposite side of our PM, and he had to go to a meeting. While he was gone, a (really smart) fellow dev hopped into the conversation, asking what we were arguing about. I explained the issue (surely with dripping bias), and asked him what he thought.

He sided with the PM, and explained his (different line) of reasoning. It was cogent and persuasive. As he was closing his argument, I just cut him off. “No, you’re right.”

The PM came back in, my boss in tow, started up the discussion again, and I exclaimed that I’d switched sides. I regurgitated the (really smart) fellow dev’s reasoning, and that was that.

There were a lot of important things that happened there for me. One, I just outright said that someone was right, which until then I hadn’t done very much. When I was younger it’d get to the point that I’d draw up ridiculous straw men or insane special pleading fallacies to make my point, even when I knew I was wrong, and would “win” the argument.

When you “win” with shit reasoning, everybody loses, for two reasons. One, other smart people recognize your shit reasoning, and even if they can’t argue their way out of it, they lose respect for you. Two, and this counts especially if you’re making something that other people have to use, shit reasoning leads to poorly modeled and horribly thought out products.

There’s a lot of value in being honest and working hard to identify the best ideas. One of the consequences of this is the realization that your ideas aren’t always the best and aren’t always right.

The unpredictable effect of me conceding the argument with, “you’re right?” Feeling liberated. Suddenly I wasn’t stuck to a losing side, had seen the light, and was then arguing for something that made even more sense. Also, I think by reiterating the (really smart) dev’s reasoning, I’d helped bring everyone onto the same page, but that was only by luck – if the (really smart) dev had been in the original discussion, the discourse probably wouldn’t have lasted more than 2 minutes.

There are arguments that are fun to have, where neither side can be 100% right, and there are arguments that aren’t fun to have, because one side is 100% wrong. In the second type of argument, it’s important to be on the right side, even if you’re late to the game.

Most importantly? Make sure your teammates and coworkers know you’re aware of your own fallibility.

Bgtwt Open Source’d!

After (I think) removing the passwords and everything else sensitive in the code, I’ve open sourced the code behind bgtwt.com.

I’m not sure if it will get any patch submissions or not, but I figured with it being closed, it definitely wouldn’t. If you submit anything you’re proud of (you can find the code here: http://github.com/terrbear/bgtwt), and I like it, I’ll happily put it up on the site.

You’re also free to copy the code and do whatever you want with it. I’ll attach the proper license in a few days.

If you’re looking for a feature that would be awesome and want to start playing with Rails, I’d like to integrate acts_as_unvlogable into bgtwt.

Rails and OpenID: Start Your Engines

(Apologies for the ridiculously lame title, but I laughed while writing it, so it stays.)

If you care about your user’s security, and your time, it’s time you stop asking them for passwords. Storing user passwords and handling authentication offers a bunch of problems that have already been solved, and requires a lot of work to do it right.

Think about it. If you’re storing user’s passwords in your database, you’ve automatically got sensitive information on your server. Maybe it’s just a College Football Pick ‘Em site, which stores nothing too important … until you store a user’s password. Let’s face it, most users don’t use a new password for every site, so if someone breaks into your server, you’re responsible if that information is lost.

So, you’ve got to make sure your server’s security is up to date. Then, it’s still possible there’s a break in, or you have a malicious developer. So you have to encrypt the passwords. But not just encrypt them, because that would still be susceptible to dictionary attacks. You have to salt each password.

What about brute force login attempts on your sign in page? Have you handled that?

These are just some of the problems that you have to solve the minute you decide to store someone’s password.

A better solution? OpenID. Below I’ll explain how to use my newly written (really repackaged) OpenID Rails engine, and you can be on your way to making the world an easier place to sign in.

Read the rest of this entry »

Rails Rumble Recap

@meezy and @jcooksey came down over the weekend so we could all hang out and compete in Rails Rumble. It was a lot of fun, we had good food, got a lot of coding done, and even tried to impose some culture on Cooksey.

We made a college football pickem game, similar to the one I ran a few years ago, but with lots of improvements and social networking hooks. Check it out at pickem.navoty.com.

We managed the scope really well and dropped features when we realized we wouldn’t make it otherwise, and as a result got in our last minute fixes and polishing and tagged and released with 12 minutes to spare. The guys running the competition did a great job providing tools to help confirm what things needed to be done to be eligible when things closed.

Onto a postmortem:

Bad

• We thought the competition started at midnight Saturday, not 7pm Friday. By the time I realized it, Cooksey and Meezy were about to leave Dallas, and there wasn’t any way we were going to be able to compensate. Further, we went to go see Inglourious Basterds Friday night, which I think was worth the lost time.
• All of us had used git, but mostly in non-conflicting or single-user ways, such that we got to learn how to deal with conflicts over the weekend. Being more familiar with it would have saved us a little time.
• There’s a bug on the Twitter feed widget, where our code isn’t hooking up to memcached correctly. We introduced memcached with about 45 minutes left, because just from our viewing, we’d reached Twitter’s API limit. This will be fixed when we get out of competition mode.
• Rails has changed a lot, and Cooksey hadn’t done Rails coding in a few years, so lots of times he’d be like, “wtf is this voodoo?” I think that’s good in that Rails has been maturing rapidly, but if you put it down for a few months, the learning curve grows rapidly.
• Facebook sucked 3 of my hours on Sunday. I tried to use a different library, Facebooker, instead of RFacebook, and the documentation was terrible. Most of the writeups I’d seen were out of date and using the old Facebook API. I should have played with Facebook ahead of time to know what I was doing.
• Cooksey hasn’t seen Princess Bride or Big Lebowski.

Good

• Lots of the tools we needed, we’d had experience setting up, so it was easy. Beanstalkd, Monit, Passenger, Apache, Memcached (it’s running, even if we’re not using it right for the Twitter widget), Twitter OAuth, all went really smoothly and were done in no time.
• Most of the plumbing and real coding was done Saturday, so we had lots of tests written then. Most of the front end work was done Sunday, which was nice, because we weren’t introducing regressions in the codebase and continually banging on that and wasting time.
• Scope was managed really well, and we even realized there were parts that didn’t need to be completed until the season actually starts, so we could justify pretty easily pushing those features back a week or so.
• We used campfire for pasting, but that’s about it. I requested that anyone who wanted to compete be in the same room, because I didn’t want to waste time with IM or whatever. It worked really well, because jumping over to someone’s computer screen or just asking “what are you working on?” is that much faster.
• Pen and paper proved to be fantastic tools for tracking who was working on what and our progress. There’s a lot to be said about a tool that lets you draw where you want, without any rules. I think I’ve blogged about really good screwdrivers before, though, so I won’t repeat myself.
• I really like the app.

We’re going to introduce a few new features after the competition is over, move it to a permanent (EC2, we think) host, and then you’ll all have a solid pickem site for years to come. Unless you want to win, which is unpossible, since I’m playing.

Lamenting the Lack of Science in Parenting

I assume most people who read this know me, and know I write software. I really like thinking in terms of code, because it’s unambiguous, and there’s an answer to almost every problem.

Parenting? Not so much.

Most people don’t realize that things that they see babies do, well, they had to be taught to do that. Eat from a bottle? Have to teach them that. Suck on a pacifier? Have to teach them that. Stop sucking on a pacifier? Have to teach them that.

Sleep through the night? Yep, taught. Go to sleep without crying? Taught. Doesn’t hit you in the face (most of the time)? Taught.

The problems come up when it’s time to teach those things. Take, for example, sleeping. There are several different ways to teach a kid to go to sleep. You can abstain, and just feed them until they’re big enough that they sleep through the night (our niece was lucky enough to get this, I’m pretty sure she went to 18 months until the parents changed methods out of necessity). You can go in at increasing timed intervals – The Ferber Method (Nate) – to comfort the child. You can sit in the room and try to comfort from the corner. You can ignore them by turning off the monitor – the Extinguish Method (Evie, after a failed Ferber attempt).

I imagine a lot of kidless people will say that the right thing to do is to wake up with them until they’re ready to sleep through the night. Good for you, and let me know how that works out when you’re going on your 6th month (or 180th consecutive night) of bad sleep.

Of course, whatever choice you make, you worry. If you get up with the kid forever, are they going to have attachment issues? If you turn off the monitor, are they going to have separation issues? Will this affect their growth? Will they turn to drugs at 14 and run away, never to be seen again until they’re 22, wanting drug money and another fix, if you give in and give them ice cream before the last bite of chicken?

(Also, random aside, I never entertain plausible tragedies for my kids. I only think of outlandish ones, convince myself that they’ll happen, and then act accordingly. I won’t drive with my trunk open with kids in the car for this reason, because I know a ladder will fly off the top of a truck and slam the back seat and squish my kids in horrendous ways.)

Discipline approaches cause me similar anxiety. We’ve decided that we will put Nate in time out for anything that we consider to be dangerous or extremely antisocial. For everything else, we raise our voice and talk sternly. Touch a knife, go to time out. Hit, go to time out. Throw something that isn’t a ball, go to time out. But are we over-using timeout? Is spanking really as bad as I think it is? Is yelling effective? Am I damaging Nate’s ears? His ego? His appreciation of acceptable conversational volume?

This lack of science has some parents turning to garbage for advice, and you know what sucks? I have no evidence that they’re doing anything wrong. The number of studies that actually link different child rearing tactics with long term efficacy and child success? Tiny & apocryphal. Instead, for example, parents will do “parenting studies,” and it’s based on, say, the Bible, interpreted through the author’s perspective of how it should be applied. (tl;dr: get the belt).

And, what I think is worse, is that because there isn’t good science for lots of problems that all of us run into, parents will ignore the science when it’s available. Jenny McCarthy has got parents thinking that they know better than doctors, the FDA, the CDC, and the WHO. Those vaccines have toxins! They’re evil! Don’t give them to your kids, they’ll cause autism!

Even though they won’t, and study after study has shown no link between the two.

So, in short, I think I misplaced my kid instruction manual, and if any of you out there have a spare one because you’re on your 18th kid or whatever, we could really use one.

Twitter::Deploy

There are a lot of plugins and libraries that I consider must-haves for a Rails project. Capistrano is one of them.

At work, we use cap to build to all of our different environments. It’s a pretty big script, since some of our instances span several boxes, some don’t, they all require special restart orders, etc. We use the same script to build to our staging environments, and push out to different staging environments several times a day for QE.

This started to become a bit of a problem, because we couldn’t be sure when someone had deployed (we have timestamps, but who keeps up with the timestamps of when they check things in?). So, while taking one of my pretzel breaks, I added this to our cap script:

task :tweet_build_started, :roles => :db do
  users = {
    'theath' => "@terrbear",
    'mroeder' => "@donkeyhighway",
    'rdejuana' => "@rdejuana"
  }

  env_user = ENV['USER'] || ENV['USERNAME']
  user = users[env_user] || env_user

  username = "FILTERED"
  password = "FILTERED"
  message = "building #{application} from revision #{revision} by #{user}!"
  `curl -u #{username}:#{password} -d status="#{message}" http://twitter.com/statuses/update.json`
end

task :tweet_build, :roles => :db do
  users = {
    'theath' => "@terrbear",
    'mroeder' => "@donkeyhighway",
    'rdejuana' => "@rdejuana"
  }

  env_user = ENV['USER'] || ENV['USERNAME']
  user = users[env_user] || env_user

  username = "FILTERED"
  password = "FILTERED"
  message = "new build pushed to #{application} from revision #{revision} by #{user}!"
  `curl -u #{username}:#{password} -d status="#{message}" http://twitter.com/statuses/update.json`
end

This worked out pretty nicely, so if anyone builds, I see the tweet, and know that any issues I’ve marked resolved, I can drop to QE (part of our workflow).

So that was kind of cool. But then, talking with the QE guys, we realized that there’s lots of times where QE needs a fresh build to test things, and they have to get a hold of one of us for that to happen. I don’t want the QE guys knowing how to use our cap script, or even trying to, because that could be horrible (cap deploy production -> everybody fired).

So I wrote a bot that runs on our CI server that polls our builder account for direct messages. If it receives one (using Twitter security, where it can only receive a DM from someone it’s following), it checks the environment to see if it’s a staging one, and then builds to the appropriate environment. It even tweets its progress. Here’s the script:

# Sample loop to show process
twitter = Twitter::Client.new(:login => "FILTERED", :password => "FILTERED")

ACCEPTABLE_BUILD_ENVS = /staging1|staging2|staging3/

loop do
  begin
  messages = twitter.messages(:received)
  messages.each do |m|
    env = m.text[ACCEPTABLE_BUILD_ENVS]
    unless(env.nil? || env.empty?)
      `cd ~/builder && cap deploy env=#{env} twitter=#{m.sender.screen_name}`
    end
    twitter.message(:delete, m)
  end
  rescue Twitter::RESTError
    puts "rest problem, continuing as normal"
  rescue
    puts "problem: #{$!}"
  end
  sleep 90
end

I just run that with nohup, and it’s been going strong for several days now. QE doesn’t even have to ask us for a build, and they’re notified when their build is done automatically, instead of asking for a build, then waiting for us to get around to it, then waiting for us to tell them when it’s ready to go.

Piston + OSX Fail / Fix

We recently moved all of our external plugins to Piston, which at first seemed great. There were times where the external plugin devs (in house) were keeping code on their machines because they didn’t want it to get deployed, which seemed wrong.

So I got everything imported happily, and then yesterday I tried to update one of the plugins, only to see “svn: can’t get password.”

Turns out there’s a bug/glitch/whatever for svn on OSX when using svn in non-interactive mode, where it won’t ask OSX’s Keychain for the credentials. A bug has been reported to both Subversion and Apple, but in the mean time, I forked Piston and just run svn without the non-interactive flag.

I don’t think it’s worthy of a pull request, because it’d be nicer if Apple or Subversion would get things fixed, but for now, you can gem install terrbear-piston if you need to use Piston on OSX.