From TextMate to vim

TextMate was my trusty Mac code editor for many years. It seemed everyone who was anyone was using TextMate, especially if they were living in the Ruby on Rails world. Unfortunately, this great editor did have some shortcomings, and improvements were few and far between, with an uncertain future. As many developers began migrating to other, more powerful editors, I resisted for a while, but ultimately made the decision early this year to leave TextMate behind and move to MacVim. It’s also noteworthy that while I’d been a TextMate fan for five years or so, for the 19 years prior I coded almost exclusively in Emacs.

Moving to MacVim was a major decision for me, as I’d never been a vi user before, and the key thing I’d learned through 24 years of “near misses” with vi was that it was a different breed of editor than anything else out there. I knew that hardcore vi users swore by it, but I also knew that it had a steep learning curve, due the drastically different method of editing it employed. Over the years, I only accumulated enough vi knowledge to make the simplest edits to unix configuration files, and save my work. When I made an error, I’d quit without saving, and try again. But I’d also always had a strong preference for using the keyboard for as much as possible, foregoing the mouse (or more recently, the trackpad) whenever possible, and ultimately this was what drew me to finally tackle vi rather than move to Coda, Sublime Text, or other traditional editors.

Now that I’ve been using MacVim as my editor exclusively for nine months, I’ve been reflecting on the journey. Most importantly, I’m “sold” on the power of vi, and have no regrets about my switch. As expected, the learning curve was substantial, but I found many helpful pointers along the way, and I have a long way yet to go. Some of the critical guides and tools that helped me get started include:

Vim Text Objects: The Definitive Guide was one of the later guides I followed, but helped the most in providing me with the “a ha!” moment where all the puzzle pieces fell into place and finally stayed stuck in my brain. The Janus distribution of plugins and configurations turned vim into a powerful integrated development environment, and eased my transition from TextMate.

My current development environment is now built as follows:

  • brew install macvim
  • curl -Lo- http://bit.ly/janus-bootstrap | bash
  • Map jj to Escape (because the Escape key is so far away for such a common function)
  • Set color scheme to Tomorrow Night
  • Set font to Bitstream Vera Sans Mono 12px

It’s truly amazing what can be done with vi and MacVim, and I know I’ve only scratched the surface. Being able to make very precise and complex changes with only a few keystrokes, where previously the task would require various combinations of mouse selection and typing, has been eye-opening and time-saving.

Now, I need to go back through this blog post and delete all the occurrences of "jjjj" where my fingers forgot that I’m in WordPress, and not vi…

Preparing Mac OSX Lion for Rails Development

With Mac OSX Lion being released recently, I decided to do a clean re-install my development iMac from scratch, rather than doing an upgrade. This offered me the opportunity to fine tune and document my development environment setup. Without further ado, here’s how I set up my environment. I hope it helps any new Lion or Rails developers out as well.

Here’s an overview of my process:

  1. Install OSX Lion
  2. Install Apple developer tools
  3. Install RVM
  4. Install Ruby 1.9.2
  5. Install Homebrew
  6. Install MySQL

Note: this is not an introduction to Ruby, Rails, RVM, terminal, or other developer tools. I assume you already understand these, so I won’t be spelling out, for example, how to edit your .bash_profile from the terminal.

Install OSX Lion

The App Store wants you to download and install Lion on top of Snow Leopard.  In order to do a “clean install”, you’ll need to create a Lion installation DVD or USB drive. I used an 8GB USB stick for mine, but you can use any external media with at least 5 GB of space.  It’s pretty simple, and a very detailed tutorial is available at Information Week for creating the installer.

Reboot your Mac with the DVD or USB stick plugged in, and hold down the option key while booting to select the new installation media. Once the installer is running, choose Disk Utility from the Utilities menu and erase your main hard drive partition.

Note: a clean install means reformatting and completely erasing your hard drive, so be sure you 1) have a complete backup of all your applications and data, and 2) are ready to spend the time re-installing said applications and data when you’re done.

A clean install from scratch is not necessary, but I like using it to clean up years of accumulated junk and to know my operating system is in a reliable state.

Install Developer Tools

Now that you’re running a nice, clean, new version of Lion, you should immediately run “Software Update” from the Apple menu, to ensure you’re on the latest version.

Next, you’ll need to install Xcode, Apple’s software development environment—it’s a free download from the App Store. Once the download finishes, you will need to run the “Install Xcode” application found in your Applications folder.

The Xcode installer is a 3.2 GB application, so if if you have a small hard disk (for example, a solid state disk), you can safely move or remove the installer from your Applications folder to save space once the installation is complete.

If you’re not planning to do native iPhone app development, you can also delete the two iPhone folders in /Developer/Platforms/ to recover another 4.6 GB. You’ll still be able to create web applications for the iPhone, just not native apps. Or, to be extra clever, you can prevent installation of these files altogether: instead of running the “Install Xcode” application, right-click on it and select “Show Package Contents”. Navigate to Contents/Resources and run the Xcode.mpkg installer instead. This installer lets you select which components to install, and you can deselect the “iOS Platform” tools before they get installed.

Installing Ruby and RVM

Even though OSX comes with Ruby, you will almost certainly want to install a newer version and maybe multiple different versions, and that means using RVM. Go ahead and install it following the RVM installation instructions:

[cc lang=’bash’ line_numbers=’false’]
bash Install Homebrew and MySQL

I need to use a variety of third-party and open source development tools, and there are several ways to install these. I use Homebrew, which describes itself as “the easiest and most flexible way to install the UNIX tools Apple didn’t include with OS X”. Run the installer:

[cc lang=’bash’ line_numbers=’false’]
ruby -e “$(curl -fsSL https://raw.github.com/gist/323731)”
[/cc]

Now you can easily install Mysql and any other tools you may need to do some Rails development:

[cc lang=’bash’ line_numbers=’false’]
brew install mysql
[/cc]

After the MySQL installation completes, you have a few more steps to follow, that will be outlined in the brew output. These configure MySQL and set it to start automatically:

[cc lang=’bash’ line_numbers=’false’]
mysql_install_db –verbose –user=`whoami` –basedir=”$(brew –prefix mysql)” –datadir=/usr/local/var/mysql –tmpdir=/tmp
mkdir -p ~/Library/LaunchAgents
cp /usr/local/Cellar/mysql/5.5.14/com.mysql.mysqld.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/com.mysql.mysqld.plist
[/cc]

Brew installs programs into /usr/local/bin, which is after /usr/bin in the standard OSX command search path. This means if you were to brew install git to get a newer version of git, you would still end up running the standard OSX git unless you edit $PATH in your .bash_profile to move /usr/local/bin to the beginning of the list.

Trying It Out

Now, let’s see if things are working:

[cc lang=’bash’ line_numbers=’false’]
gem install rails –no-ri –no-rdoc
rails new foo -d mysql
bundle exec rake db:create
[/cc]

Looks like things are working! Now, on to choosing the all-important desktop background picture…

Radial Org Charts

As my department at work has grown over time, I’ve thought a few times about the tradition of maintaining organization charts in Visio. These charts typically display an increasingly awkward and messy collection of boxes and lines, showing which employees report to which manager.

Apart from being difficult to effectively automate the layout once a department reaches more than a few dozen people (most tools end up overlapping boxes or choosing awkward-looking layouts), these charts don’t convey much information other than direct reporting structure of employees.

While researching the Protovis javascript graphing library for some more traditional numeric data visualization, I decided to try applying the Sunburst graph type to organizational charts. Here is a small example of how one department looks. The names have all been replaced with fake names created with the Forgery Ruby gem.

Click to enlarge

While a little non-conventional, this style organization “chart” shows the same hierarchical reporting data as a traditional chart, and more. At a glance, I can now see the relative size of each team, and the depth (number of management layers) of each part of the organization. Best of all, since the chart is generated automatically from a flat file, there’s no more moving boxes around to make that one new employee fit into an over-crowded page, fixing stray line connectors, etc.

Plus it looks pretty cool.

Rails hosting

A few weeks ago I migrated the Labrador Retriever Rescue web site from a shared hosting service at Bluehost to a virtual private server at Linode. My main motivation was to gain an element of control, including managing my Apache configuration, but I was also hoping to gain some performance improvements by getting away from Bluehost’s overloaded servers and FastCGI based Rails implementation (which isn’t fast).

Well, the performance improvements have certainly panned out better than expected! Here’s Google’s view of the site, taken from their Webmaster Tools page:

After the migration to Linode, page load times went from an average of 1.5 seconds per page, to an average of 0.2 seconds per page! Of course, this doesn’t come without a price. Bluehost charges $7 per month for hosting, while a Linode server is $17 per month (on a two-year plan). Linode is more than twice as expensive, but still a very reasonable price considering the many benefits.

Resources:
Linode VPS
Google Webmaster Tools

Dropbox Dock Drawer for OSX

When Mac OSX Leopard was released in 2007, one of the clever new features was “stacks”. These were essentially a way to quickly access a folder of files by putting it in the dock. The stacks don’t look particularly nice though, simply showing the icons of all the items in the folder stacked on top of each other, and there was no way to tell what folder the stack represented.

Not long after, a clever icon artist realized that by managing the sort order of the stack, he could cause one file’s icon to always appear at the “top” of the stack. He created a set of “drawer” icons which users could drag into their stacks, which when forced to be the front-most icon, magically make the stack look much more interesting:

Dock Drawers: Before and After

You can download his full set of icons (grayscale and color versions) at http://www.geocities.jp/chy065/

If you sort your stack by “Name”, everything works right because these icon files all have a space at the beginning of their name, causing them to sort ahead of normal files.

If you sort your stack by “Date Added” or one of the other date formats, you’ll need to set the timestamp of the drawer icon to a date far in the future (the author of the icons did this three years ago, but unfortunately he only set the timestamp to October 2010, so it no longer works). To update the timestamp, open a terminal, choose which drawer to update (I’ll use “Downloads” in this example), and run:

[cc lang=’bash’ line_numbers=’false’]
touch -mt 202001010101.01 ” Download .app”
[/cc]

Be sure to include a single space before and after the word “Download” in the filename, as in the example above.

And then there was Dropbox…

In 2010 I moved almost all of my day-to-day files onto Dropbox so that I could access them from any computer, iPhone, iPad, and via the web. It’s worked out fantastically well, and I’m a big fan. If you haven’t tried it out, give it a try and I’ll get extra free storage space too!

Once all my files were on Dropbox, I realized it made sense for Dropbox to be one of the main stacks on my dock, but unfortunately there was no Dropbox drawer icon, so I decided to make one. After a bit of time working inside gimp, I have a dock drawer for Dropbox (center drawer icon below):

You can download the new Dropbox drawer icon here: Dropbox Dock Drawer.zip

Resources:
Dock Drawer Icons
Sign up for Dropbox
Dropbox Dock Drawer download

Rails 3 Application Template

For a long time I’ve been disappointed with the “out of the box” look and feel of a new Rails app: no useful stylesheets whatsoever, just plain unformatted text on a white background. I often like to throw together quick apps to prototype a concept, or even create a long-term app that just doesn’t need to be visually unique, but needs to look decent.

I’ve also found that every time I create a new Rails app I end up going into one of my more “established” applications and stealing whole bunches of code to bring my new app up to my standards.

Since Rails allows the creation of Application Templates, I figured it was about time I dove in and spent some time creating one of my own. This strikes me as one of those projects that will never be done but is now good enough to be very useful. My template is on GitHub at https://github.com/greendog99/greendog-rails-template along with some basic documentation to help get started.

After cloning the github repository locally (into /tmp in the example below), a new Rails app can be created by specifying the template file, like this:

rails new appname -d mysql -m /tmp/template/template.rb

A standard rails application will be created as usual, but the template will then make many changes. Some of the bigger ones are:

  • Install and set up the compass-html-boilerplate gem by Peter Gumeson (sporkd). This provides:
    • All templates, stylesheets, and views use haml and sass.
    • Paul Irish’s HTML5 Boilerplate is installed for layout and style reset.
    • Compass is installed for managing CSS mixins and frameworks.
  • Adds a simple 960px 12-column grid framework from http://www.1kbgrid.com/
  • Adds a basic stylesheet with header, navigation bar, body, footer, tables, forms, buttons, and flash messages, so web pages look good out of the box.
  • Adds a default Gemfile with my most commonly used gems, like friendly_id.
  • Creates a custom RVM gemset for the application, and installs all needed gems via bundler.
  • Sets up RSpec and factory_girl for testing, instead of the default test-unit.
  • Creates a Git repository and commits the entire project into version control.

There are several other smaller features that get added, which are listed in the GitHub documentation. The template also adds a couple sample views to help get started using the framework. Here’s what a brand new Rails application looks like after running the “rails new” command above:

Developing Qnotifier plugins

Qnotifier is a combination of a daemon that runs on your Unix system (Linux or OSX), and an iPhone/iPad app to provide a dashboard-like view of data collected from your system. It can report text-based information, time-based graphs, or alerts which can be delivered to you in a variety of ways—for example, when your CPU usage gets too high. Qnotifier is free right now while in beta, but no word yet on whether it will switch to a paid model in the future.

One of the things I like about Qnotifier is that the daemon is a Ruby gem, and supports user-developed plug-ins to collect and report new kinds of information. I just finished setting up a Squid cache to sit in front of a Rails site to improve performance, and decided to see what it would take to add Squid statistics to my Qnotifier dashboard. It turns out, getting the Squid statistics was the hard part—delivering them to Qnotifier was easy!

Getting Squid statistics

I couldn’t find much information on how to retrieve statistics. Every web page said to install and use the Squid cachemgr.cgi tool to get a GUI into viewing stats. This wasn’t what I wanted, but it got me there eventually. The CGI is compiled C code, and quick look through the source gave some hints. Using tcpdump to observe it in action gave away the answer: send the following request to the running Squid server, and get back a ton of information:
GET cache_object://localhost/info HTTP/1.0

Writing the Qnotifier plugin

Qnotifier was just released a week ago, fairly stealthily, and thus there’s no documentation at all yet. Even looking through the plugins shipped with the package, half of them are non-functional yet (apache, iostat, etc.).

Fortunately, it was pretty easy to figure out the general plugin architecture from reading the working plugins (Ruby, system, Passenger, etc.):

  • Create a new plugin file in /var/lib/qnotifier/plugins
  • Create a new Class, and inherit from Qnotifier::Plugin
  • Collect your metrics in whatever way is appropriate.
  • Call stat(key, message) to deliver text-based metrics to the “Stats” dashboard in the app.
  • Call report(key, val) to deliver one time-series data point for a graph in the “Charts” dashboard.
  • Call alert(key, message) to send a real-time alert, and reset_alert(key, message) to clear one.

Here’s the short Ruby plugin I wrote to retrieve the Squid statistics (using netcat) and push them to Qnotifier. This is an hour’s proof-of-concept work, and just gathers a couple metrics without error checking. If Qnotifier comes out of beta and is free (or cheap), I’ll clean it up some more since I see a lot of potential here.


module Qnotifier
class Squid

Confluence Ruby API

At work, we use the most excellent Confluence wiki system by Atlassian for all of our shared documentation. We also use it for weekly status reports, which are based on a template that needs some minor changes each week before being created.

Creating these weekly pages for people to fill in used to be a manual process—copying the template to a new wiki document, then editing it to make some pre-defined changes (such as the date)—until I learned about a Ruby gem for the Confluence API. Now a small Ruby program run weekly does the work for me:


#!/usr/bin/ruby

require 'rubygems'
require 'confluence4r'

# Some variables specific for my Confluence setup
url = 'https://your.confluence.wiki.com/'
user = 'jamesbond'
password = '007'
wikispace = 'statusreports'
template = 'Status for week ending YYYY-MM-DD'

# Find the date of the next Friday
friday = Date.today
friday += 1 while friday.wday != 5

# Log in to Confluence and retrieve the template page
server = Confluence::Wiki.new(url)
server.login(user, password)
page = server.getPage(space, template)

# Create a new page based on the template, changing the
# date placeholder with this Friday's date
newpage = Hash.new
newpage['space'] = space
newpage['title'] = template.gsub(/YYYY-MM-DD/, friday.to_s)
newpage['content'] = page['content'].gsub(/YYYY-MM-DD/, friday.to_s)
newpage['parentId'] = page['parentId']

# Save the new page
server.storePage(newpage)
server.logout()

Resources:
* Confluence by Atlassian
* confluence4r gem on github
* confluence4r documentation by Atlassian

Javascript date selector in Rails

Many projects I work on require a user to select dates while entering information into a form. Rails makes it trivial to create date selectors and save the results into models via the date_select form helper. Here’s what it looks like by default:

Functional, but hardly friendly to require the user to manipulate three different controls to select a date. Searching the web I found a few different pop-up-calendar JavaScript apps, and settled on using Brian McAllister’s Unobtrusive Date-Picker Widget via Brian landau’s Rails plugin.

I’ve used this successfully for the past year, but noticed this message on the plugin’s home page recently: The latest release of the Unobtrusive Date-Picker Widget JavaScript library makes this plugin unnecessary so I’ve decided to no longer maintain it. Using the date-picker without needing a plug-in sounded great to me! Unfortunately, I couldn’t find any references or instructions on how to use the new (version 5) date-picker with Rails, and there was clearly no built-in support within new new date-picker.

I set out to solve this problem and this is the solution I came up with. The previous plug-in version required changes to my Rails applications’ views in order to activate the date-picker. Reading through the documentation for the new version 5 date-picker JavaScript app, it quickly became obvious that I’d need to handle the date-picker activation completely in JavaScript. This has advantages (the date-picker is truly unobtrusive, since no Rails app changes are needed) and disadvantages (flexibility is lost since the Rails app can’t easily configure the date-picker).

The date-picker needs to bind to the existing date_select form elements in order to function. This example from the documentation demonstrates, with three form elements with DOM IDs of inp1, inp2, and inp3:

[cc lang=”javascript” line_numbers=”false”]
datePickerController.createDatePicker({
formElements:{
// “inp1” represents the day part
“inp1″:”d”,
// “inp2” represents the month part
“inp2″:”m”,
// “inp3” represents the year part
“inp3″:”Y”
}
});
[/cc]

Rails view helpers create form element IDs automatically. For example, a form for a “person” model with a “firstname” input would have an ID of person_firstname. Even though dates are stored as a single field in models, the date_select helper creates three form elements (year, month, and date), which are re-combined when saving to the model. Each of these elements’ IDs are generated as above, but with an identifier (_1i, _2i, or _3i) appended to differentiate the three fields. My first thought was to retrieve the computed IDs in my view code, and pass them to the JavaScript. Looking through the DateHelper source code yielded some promising hints such as the input_id_from_type method, but I eventually gave up as nothing seemed easily accessible from public-facing methods. Instead I decided to rely on Rails’ current convention, and search for the _1i identifier across all form select elements in the page.

[cc lang=”javascript” line_numbers=”false”]
$(document).ready(function() {
// Rails creates date_select ids as
// #{model}_#{attribute}_#{field}i
// where #{field} is a digit representing
// each element in the date/time.
$(“select[id$=’_1i’]”).each(function() {
var id1 = this.id; // e.g. person_birthday_1i
var id2 = this.id.replace(/1i$/, “2i”);
var id3 = this.id.replace(/1i$/, “3i”);
var eles = {};
eles[id1] = “Y”;
eles[id2] = “n”;
eles[id3] = “j”;
datePickerController.createDatePicker({formElements: eles});
});
});
[/cc]

The above snippet from application.js requires jQuery and does the following:

  • Finds all select elements that have an ID ending in _1i. These represent individual date_select instances on the page (in case a page has multiple dates to be selected).
  • For each one, computes the IDs of the three components of the date_select (_1i, _2i, and _3i)
  • Creates a JSON configuration array telling the date-picker which form element describes each part of the date, and in what format. See the Date Formats section of the version 5 date-picker documentation for more details.
  • Instantiates the date-picker

Here’s what the date selection form looks like with the above code in place:

Of course, the magic happens when the user clicks on the calendar icon which now appears to the right of the existing date selection elements:

To get all this working in your project, you’ll of course need to download the date-picker files from Brian McAllister’s site, move the files into your Rails app’s /public directory, and include them with something like the following:

[cc lang=”rails” line_numbers=”false”]



People:







[/cc]

Home automation via iPhone

When we had our house built a little over 10 years ago, we had a home automation system installed, which included a Lutron Homeworks lighting control system. This meant every light circuit in the house was wired to Lutron’s controller, which could be programmed with their custom Windows application.

The installation company distributed multi-button keypads throughout the house and programmed them to do interesting things, like turning on a dim path of lights from the bedroom to the kitchen, or the ever-popular “30 seconds from now, turn off all lights in the house” button we use when going out or going to bed. One requirement I had was that they leave me with a copy of the application so I could re-program the keypads myself.

I quickly grew dissatisfied with being limited to the keypads, so did some digging around and discovered the documentation for the RS-232 command set used to program the keypads. I hooked up the serial port of the Lutron controller to my Linux server in the basement and quickly whipped up some HTML+Javascript to show me the status of all the lights in the house and let me change the level of any of the lights. Since we’re a very laptop-enabled household, this meant no more getting up off the couch to adjust the lights! There’s a generally-accepted belief that programmers are lazy, since we often write programs to do things for us, rather than do the things ourselves.

Now that we’re also an iPhone-enabled household, I decided it was time to get with the times. After all, it’s been almost 10 years since my original web-based version. I looked into several UI frameworks for creating iPhone web apps, and decided to go with jQTouch. After one evening I was up and running with an iPhone web app that would let me control the lights The next evening I had the app updated to query the lighting system to display the status of the lights in the house. Touching one of the lights brings up the iPhone radial selector to change the brightness of that light. Touching done sends a command to the Linux server with the RS-232 connection to the Lutron controller.

With jQTouch, the HTML and CSS were very straightforward, and only a little bit of JavaScript glue was needed to bring in the interactivity.

Here’s a bit of the Haml for the main page of the web app:

[cc lang=”haml” line_numbers=”false”]
%body
#home
.toolbar
%h1 Lighting
%form
%h2 Main Floor
%ul.rounded.lights
%li
.title Kitchen
%select{:id => ‘z_01_04_01_08_03’}
%optgroup{:label => “Kitchen”}
%option{:value => ‘0’} 0%
%option{:value => ’20’} 20%
%option{:value => ’40’} 40%
%option{:value => ’60’} 60%
%option{:value => ’80’} 80%
%option{:value => ‘100’} 100%
%li
.title Breakfast
%select{:id => ‘z_01_04_01_08_04’}
%optgroup{:label => “Breakfast”}
%option{:value => ‘0’} 0%
[/cc]

And the HTML which is delivered to the iPhone after Haml conversion:

[cc lang=”html4strict” line_numbers=”false”]

Lighting

Main Floor

  • Kitchen

  • Breakfast

    jQTouch’s CSS turns the unordered list with rounded class into a pretty iPhone screen, and its JavaScript enables effects like moving between screens (not used in this example).

    The JavaScript (jQuery) glue I mentioned earlier consists of two parts. First, when the web application loads, it makes an AJAX query to the Linux server and gathers information about the lights, then sets the initial state of the select form fields:

    [cc lang=’javascript’ line_numbers=”false”]
    // Populate the data
    $(document).ready(function() {
    $.getJSON(‘/automate/get_lutron.php’, function(data) {
    $.each(data, function(key, val) {
    var v = Math.round(parseFloat(val)/20)*20;
    $(“#z_” + key).val(v);
    });
    });
    });
    [/cc]

    The Lutron system identifies each light with a 5-part numeric string, like “02_04_01_02_01”. In order to do as little data transformation as possible, I used the same string as the selector IDs in the form (with z_ prepended since XHTML IDs can’t start with a number). Since I’m using 20% brightness granularity, a little math needs to be done for lights that are set to other values. All the lights have manual analog dimmer switches, so a light might be set to any value from 0-100% the old fashioned way (you know, walking over to the switch and physically moving the slider).

    The second part of the JavaScript glue listens for changes to any of the selectors in the web app, and then sends the new value to the Linux server’s Lutron RS-232 interface:

    [cc lang=’javascript’ line_numbers=”false”]
    // Lutron SELECT elements get wired to AJAX
    $(document).ready(function() {
    $(‘select’).change(function() {
    id = this.id.substring(2).replace(/_/g, ‘.’);
    val = $(this).val();
    var command = ‘fadedim,’ + val + ‘,1,0,’ + id;
    $.get(‘/automate/send_lutron.cgi’, {cmd: command});
    });
    });
    [/cc]

    The end result works beautifully, and now it’s even easier to turn off that forgotten kitchen light after settling down with popcorn in the family room!