On the Path

The Power of Names

I'm working on an interesting project with a few other Sherpas (Arild and Charles) - building an inter-office workflow and legal reporting tool in Ruby on Rails.  Arild and Charles were just having a conversation about a named scope, and what they should call it (a named_scope is a construct in Rails that, basically, returns a collection of objects that match certain criteria).  The purpose of this named scope was to return the number of 'change records' the system had to deal with in a certain time period.

The actual business requirement says "5 minutes"... but being experienced at this kind of thing, we know that when requirements specify absolute values, those are usually the things that change.  Taking it into account now is cheap and easy.  Trying to account for it the week before launch, or when a legal report is overdue, expensive and painful.  So they decided to parameterize it, and they came up with something that looked like:

Record.changed_in_last_x_minutes(5)

They looked at that for a minute and asked me what I thought.  "It's not hateful..." I said, and they both laughed.

It was a diplomatic answer.  We have all spent time in languages and projects where that would have been perfectly acceptable; but in the past few years, developers have spent a lot of time concerned about aesthetics, readability, and maintainability... and this just didn't seem to live up to that expectation.  I was proud of them just for asking - most developers would have just coded it up and moved to the next problem.

"I'd like to have it read like a sentence, and that just doesn't do it for me", I said.

Arild said, "I agree... but the problem is trying to get the number value at the end of the sentence with the 'minutes' in there"...  and both of our eyes lit up.  I think I managed to verbalize it first, but we both had the same flash of insight...  we wanted to pass in a duration, not a number.

Thirty seconds later, and we had an API that looks like:

Record.changed_in_last(5.minutes)

and that api also works now for variations like changed_in_last(2.hours), changed_in_last(47.seconds), changed_in_last(5.days), changed_in_last(2.weeks), and just about every variation you can imagine.

A brief conversation, a small insight, and an additional 30 seconds of code, and we have a fluent interface that can easily implement many requirement variations.  We can also show to a client and ask "is this what you mean?".

It might seem like one small change, but these add up.  Imagine an entire system designed that way - where the code begins not to read so much as code, but more like a formal description with a lot of flexibility...  A system where the developers care what the names of things are and how they read with that attention to detail.

As soon as we had it done, I remembered this story by a friend of mine.  I'm happy to steal his title and write this story as an homage to the point he so elequently made.

Cool kids don't store plain-text

or How I simplified database.yml to avoid storing sensitive passwords in plain-text

Nobody likes putting important passwords in a plain text config file. So what are your options?

1) Use a one-time, throw-away password
I typically do this for most Rails projects that I work on (e.g. user: "acme", pass: "acme")

2) Prompt for the password
Unfortunately there are times when access is tied to a real password. On my current project, the DB password is the same as my email password (served via LDAP).

Prompting in a config file? Yes, this is available by default in Rails, and can be done easily in other Ruby projects. This is all you need to get a command-line password prompt when loading database.yml:

  # Within config/database.yml or your custom config file...
    <%
      # Cool kids don't store plaintext
      require 'highline/import'
      def prompt_for_password
        @passwd ||= ask("DB password: ") { |q| q.echo = false }
      end
    %>
  development:
    host: somehost.example.com
    username: arild.shirazi@example.com
    password: <%= prompt_for_password %>

If you are not using Rails, you will need to evaluate ERB within your config file whenever you load it:

  # If loading your own yaml outside of Rails
  config_erb = ERB.new(IO.read config_file)
  YAML::load config_erb.result

BONUS: When using this method, your checked-in config/database-example.yml could be setup to work for all users out-of-the box. Just add some ERB to determine the correct mysql socket to use (<% system(mysql_config --socket) %>), and create a rake dependency that copies database-example.yml to database.yml when needed and you're good to go.

NOTE: Feel free to roll your own prompts:

  stty -echo; pass = gets.chomp; stty echo

But using the highline library really improves readability. Also, using highline is a great way to promt from inside rake scritps.

Installing CentOS 5, ImageMagick, and RMagick, September 2009 Edition

For some reason, this process seems to get more convoluted with time, not less. Before you go through this pain, let me ask you, "are you sure you need to use ImageMagick and RMagick?" There are other options in today's Rails environment.

I wrote about this previously on my older blog. While just going through this pain again, I found several things:

• CentOS has been updated, but still relies on older-than-dirt gems for ImageMagick. Still, there is something to be said for relying on stock gems rather than messing with building and installing yourself.
• The version of RMagick I specifically call out in the old blog post isn't available anymore.
• The newer version of RMagick adds a new default html documentation option you need to turn off.

So sit back, relax, and read my previous blog post, now updated to work as of September, 2009:

I don‘t normally blog about obscure, specific technical topics, mainly because 99% of the time I find a blog entry someplace that has already covered it. This is an exception, as the amount of information out there on this topic is confusing and dated.

Installing ImageMagick and RMagick on CentOS5 seems to be a bit of a dark art, mainly because of the amount of stale information and differences between linux distributions. This entry will eventually be stale as well, so check the date and version numbers – your mileage may vary.

The ImageMagick/RMagick combination is arguably one of the coolest yet painful combinations of ruby/native code out there. There are a lot of attempts to replace, simplify or otherwise solve this pain, but today, rmagic is often a necessary tool in the rubyist‘s toolbox. So here‘s my attempt to lessen that pain.

I'll include details and explanations with each command line. I have you as a captive audience, and the details will give this post more relevance if the version numbers have moved on since this writing. There are only 3 commands, so bear with me.

One more caveat – I‘m taking the short path using the most recent CentOS5 packaged versions. If you need the latest version of ImageMagick, have fun compiling it all yourself. There are plenty of sites out there with directions. For me, one of the reasons I‘m using CentOS5 (with this client) is for the stability and ease of package management.

1. First we install ImageMagick as an rpm using yum. As of this install, the latest CentOS package is 6.2.8.0-4.el5_1.1. This is lagging considerably from the newest version, and that causes some of our issues. Use sudo on these commands as appropriate for your user management style.

   yum install ImageMagick

   This will ask to download and install all kinds of stuff (43 other packages on a stock CentOS server install). Let it install whatever it wants. If the version number of ImageMagick is up to the 6.3.x range by the time you read this, then this post probably isn‘t want you‘re looking for. (although explanations below will still help you). If you are on an x86_64 box, this will also install that rpm file for you.

2. Now install the ImageMagick-devel library. Yes, that could have been done in the same step above, I‘m just trying to be explicit for your benefit.

   yum install ImageMagick-devel

   That wants to install 16 packages on my stock CentOS5 server. This also does the right thing os x86_64.

   So no surprises so far. It is the ruby gem that causes us all the problems...

3. Install the RMagick gem. I‘m about to make this look easy, but it wasn‘t. The first problem is that the latest RMagick gem needs the 6.3.x series of ImageMagick, which isn‘t packaged for CentOS5 yet. So I had to determine the latest version of the RMagick gem that works on the 6.2.x series. As of this writing, that is 1.15.17, and you can find that out at the RMagick gem page.

   But installing that just might fail in a blaze of glory that only a software engineer with a minor in forensic investigation would enjoy. Near as I can tell, the install of the gem would succeed if it weren‘t for some failures building the documentation. These failures vary from machine to machine, as it seems to depend on what fonts you have installed (which might depend on what version of OpenOffice, etc). Solution? We dont need no stinkin' docs! Furthermore, sometime between the last time I did this and now, they have a new option to disable the html documentation. This has to be passed through the gem installer to their script (this the obscure little '--' sitting all by itself in the command.)

   gem install rmagick -v 1.15.17 --no-rdoc --no-ri -- --disable-htmldoc

   a minute later, I get the output:

   Successfully installed rmagick-1.15.17
1 gem installed

By the way, I‘m not writing this from memory. I just installed CentOS 5.3 x86_64 on a stock server at ServerBeach, and these commands are clipped right out of my console window.

Anecdotes on Passenger Memory Usage

We manage a rails site for a client that will go unnamed for now, and for the past year they have experienced about a 23% increase in traffic per month.  The front end is run on a single box; a dual core with 2 gige of ram.  Up until the switch to passenger, we were using a cluster of mongrels.  As the site grew, memory usage (and mongrel restarts with monit) were starting to cause some problems.

So we switched to passenger.  I bet you can tell when by looking at this chart.  This chart represents the last year of memory usage on that box, as sampled every 5 minutes.

Switching to Passenger is nothing new; in fact, if you're still hosting with mongrels, thats getting pretty old-school in the rails community.  I just haven't seen much real-world data with a year of measurements behind it, thought this graph was pretty, and thought I'd share.

Introducing Crondonkulous

Do you write rails apps? Have you ever wanted to automate rake tasks with cron? Easy peasy, right? Well, its not that hard for the most part, but there are some issues to think about:

• What happens if you try to re-deploy your rails app while your cronned tasks are running?
• What happens if you migrate your database while a long-running cron is processing?
• What happens if you get the cron syntax wrong? Ever have a script start every second when it should run once a day?
• Ever have a cron task start before the previous one had completed?
• Ever try to find your cronned task's log intermixed with your controller log entries?
• Ever want to change your cron entries, and think its ugly that everything else about your deploy is automated?

Wouldn't it be great to solve all these issues with one gem? Crondonkulous!

install the gem:

gem sources -a http://gems.github.com
gem install bokmann-crondonkulous


in your rails app, generate the files:
./script/generate crondonkulous


The generate will give you some further instructions - in a nutshell:

1. add require 'crondonkulous/recipes' to the top of your deploy.rb cap file
2. add config.gem "lockfile" to your environment.rb
3. uncomment whatever cron entries you want to run in the generated config/crontab.erb file
4. write your periodic tasks where it says 'your code here' in the lib/tasks/crondonkulous.rake file

Pretty easy, right?

What you get:

• a lib/tasks/crondonkulous.rake file that you add your code to. Stubbed out tasks include every_minute, every_5_minutes, every_15_minutes, every_hour, every_day, every_week, and every_month.
• a config/ctontab.erb file that gets rendered and put into the cron table. This file already includes the proper syntax for tasks that run every minute, every 5 minutes, every 15 minutes, hourly, daily, weekly, and monthly.
• conventions for naming/logging. Each periodic task gets its own log file, like log/every_minute.log
• protection via the LockFile gem. crondonkulous rake tasks won't run at the same time (unless you let them), and will wait/retry to run the number of times you specify.
• Capistrano integration to remove the old cron entries while deploying and insert new cron entries before application restart

Motivation

A while back I wrote the 'Automate Repeatable Tasks' recipe (#72) for the 'Advanced Rails Recipes' book. Those ideas, and more, are embodied in this gem. In particular, the management of entries into the cron tab is a lot easier. I have done this for various apps time and time again, and wanted to promote some common conventions.

Extras

Crondonkulous inserts the crontab entries into the user doing the deployment by default. When I have total control over the hosting environment, I prefer to create a user specifically for the application and install the cron entries there. you can add:

set :cron_user, "my_app_user"

to your deploy.rb if you want to do the same.

You can follow the project on github.

Have fun!