In which I explain why you should not set your system time manually.

I almost ruined my Friday by deploying to our production servers before the office hours ended.

The application is on Rails 3.0.x and we use Bundler and Capistrano - old fashion - to manage dependencies and deployment. Getting something fixed, I hit bundle exec cap production deploy wondering what great foods my wife would be preparing. Then BANG! I got alert telling me the api server (and of course the others) failed to start!

A quick check showed Bundler couldn’t locate any installed gems. Going to the current release path I found there was no .bundle folder generated. I guessed Bundler had installed gems into a wrong place so reading the horrible capistrano outputs must help:

executing "cd /var/www/example.com/releases/20111215140542 && bundle install --gemfile /var/www/example.com/releases/20111215140542/Gemfile --path /var/www/example.com/shared/bundle --deployment --quiet --without development test"

Do you notice the problem? Bundler had actually installed all gems to the correct directory which was shared/bundle(that’s the default behavior of bundler/capistrano). The problem was Bundler had ran install under the wrong directory! Today is Dec 2nd but that directory was like Dec 12th.

Since other symlink and assets complication tasks had ran against the correct release path probably there was something odd with Bundler. Let’s check https://github.com/carlhuda/bundler/blob/master/lib/bundler/deployment.rb:

current_release = context.fetch(:current_release)　# ln 44

Oh yeah if I recalled correctly many capistrano tasks use release_path to do jobs after deploy:finalize_update or deploy:update_code, but Bundler’s recipe uses current_release.

It appeared someone had got this issue too, and he’s kind enough to make a fix, unfortuately his pull request was rejected:

https://github.com/sunaku/bundler/commit/a4633eec154af0bca9aa4ebdeffbf5d2bde5b3e3

Maybe we should have a look at what is current_release from capistrano, https://github.com/capistrano/capistrano/blob/master/lib/capistrano/recipes/deploy.rb:

_cset(:releases)          { capture("ls -x #{releases_path}", :except => { :no_release => true }).split.sort }     #60
_cset(:current_release)   { releases.length > 0 ? File.join(releases_path, releases.last) : nil }

ls and sort, that gave us the newest directory by name, be it in the past or future.

I had to admit, this was really a user error, as the discussions implied here: https://github.com/capistrano/capistrano/issues/55.

My fault was I was playing with some local test a while ago, setting my Mac’s time to the future. At the same time I deployed our application, which lead to a release path with a timestamp from the parallel universe.

If you use Bundler and Capistrano, don’t set your system time manually, ever!