Actually better yet, let me show you:

                     

Our hue matching system clearly has some issues. The main problem is that it’s all-or-nothing; a pixel is either masked or not. There’s no gradation. It’s not a surprise that it confuses the two greens. The whole reason we used HSL instead of RGB was because we can find similar colors easily with the hue component! But why can’t greens be gradually more opaque the further they are from the key color?

Wouldn’t it be great if there was a way to do chroma key where colors close to the key color but not quite had a transparency value somewhere in between on and off? There is a way, it’s called alpha compositing.

Sorry, ImageMagick

I was researching other chroma key programs and ran across this plugin for Paint.NET. I tried it on some images that I knew were a problem and was amazed. It worked way better than my ImageMagick method, even for the pixels that were not quite the right shade of green.

As much as I love the .NET open source community though, I didn’t really want to try to automate Paint.NET actions on the fly inside of the Photobot 3000. I also figured that I might need to tweak things in the algorithm to get it just right, so I was leaning heavily towards trying to implement chroma key myself. The creator of the Paint.NET plugin conveniently included a link to the paper that they used to create the plugin. The paper turned out to be pretty digestible, so I decided to writing the algorithm myself in C#.

The Algorithm

The algorithm starts with looping over the pixels in the image in RGB color space. For each pixel, it checks to see if the green component (we’re going to use green but they use blue in the paper) is greater than the blue and red components. If not, it leaves the pixel how it is. If so, it decides an alpha value (how transparent to make the pixel) based on how much more green the pixel is than it is red or blue.

For pixels where the green color is only slightly dominant, the alpha stays at 255 (completely opaque.) Then at some configurable threshold, the alphas start counting down until a second threshold where the alpha becomes zero. Every pixel that has a more green than red or blue value higher than the second threshold gets its alpha value set to 0 (completely transparent.)

The Code

First, let’s create the alpha map:

private void initalphaMap()
{
    List<int> alphaMapArray = new List<int>(510);

    //int fullyTransparentEndIndex = 15;
    //int semiTransparentEndIndex = 75;

    int fullyTransparentEndIndex = 15;
    int semiTransparentEndIndex = 100;

    for (int i = 0; i < 510; i++)
    {
        if (i < fullyTransparentEndIndex)
            alphaMapArray.Add(255);
        else if (i >= fullyTransparentEndIndex
                 && i < semiTransparentEndIndex)
        {
            double lengthOfSemis = semiTransparentEndIndex
                                   - fullyTransparentEndIndex;

            double indexValue = (double)(i - fullyTransparentEndIndex);

            double multiplier = 1 - (indexValue / (double)lengthOfSemis);

            int alphaValue = (int)(multiplier * 255.0);

            alphaMapArray.Add(alphaValue);
        }
        else
            alphaMapArray.Add(0);
    }
    alphaMap = alphaMapArray.ToArray();
}

Next, we need to loop through the pixels of our image. There are ways of looping through the image data which are safe and managed by .NET, but on the advice of this post, I decided to throw caution to the wind and access the memory directly inside of an unsafe block. Here’s the code I’m using to loop through all of the bits:

private Bitmap operateOnEachPixelOfImage(Bitmap original
                                         , CopyPixelOperation operation)
{
    Rectangle rect = new Rectangle(Point.Empty, original.Size);

    Bitmap output = original.Clone(rect
         , System.Drawing.Imaging.PixelFormat.Format32bppArgb);

    var modifications = output.LockBits(rect
         , System.Drawing.Imaging.ImageLockMode.ReadWrite
         , System.Drawing.Imaging.PixelFormat.Format32bppArgb);

    unsafe
    {
        int pixelSize = 4;

        for (int y = 0; y < modifications.Height; y++)
        {

            byte* row = (byte*)modifications.Scan0
                        + (y * modifications.Stride);

            for (int x = 0; x < modifications.Width; x++)
            {
                byte[] pixel = {row[(x*pixelSize)]
                                , row[(x*pixelSize)+1]
                                , row[(x*pixelSize)+2]
                                , row[(x*pixelSize)+3]};

                byte[] newPixel = operation(pixel);

                row[(x * pixelSize)] = newPixel[0];
                row[(x * pixelSize)+1] = newPixel[1];
                row[(x * pixelSize)+2] = newPixel[2];
                row[(x * pixelSize)+3] = newPixel[3];

            }

        }

    }
    output.UnlockBits(modifications);
    return output;
}

You may have noticed that I’m not changing the pixels at all so far. I wanted to be able to reuse the bit-looping code with other operations for which I’d need to work with each individual pixel. I’m allowing the caller of this method to inject their own pixel manipulation operation into the loop through a delegate. This makes it trivial to create things like an alpha booster, or a method that makes a photo darker or lighter, and it helped in creating a hue-shifting method that I’m using for an enhanced version of the chroma key algorithm (which is a topic for another post.)

The actual method that creates our mask is implemented like this:

public Bitmap chromaKey(Bitmap original)
{

    return operateOnEachPixelOfImage(original,
            delegate(byte[] pixel)
            {
                int redValue = pixel[2];
                int greenValue = pixel[1];
                int blueValue = pixel[0];

                if (greenValue > redValue && greenValue > blueValue)
                    pixel[3] = (byte)alphaMap[(greenValue * 2)
                                - redValue - blueValue];
                else
                    pixel[3] = 255;

                return pixel;
            });
}

This method outputs a Bitmap that has a transparency where we want the background to be inserted. Once you have this, it’s simply a matter of overlaying the Laser image, which you can do using the built-in Graphics object.

Results

The results are hard to argue with. The new method still picks up some other greens but when it does, they’re only partially transparent. Here are the results, the old ImageMagick hue matching method is on the left, and the new algorithm is on the right.

                     

Not only does this method work better, it’s also much faster and more configurable. If we happen to change the color of the background, it will be really easy to change the code to work with the different color. Also, we can keep honing the alpha map to produce the desired results, and maybe add new features that allow us to perfect this process.

We were already using the amazing and versatile ImageMagick to add a custom logo for each event to the images, so it made sense to see if ImageMagick could do something like chroma key. I was hoping it’d be as simple as calling ImageMagick with the right arguments, but it turned out to be a bit of a rabbit hole.

Before I started looking into this, I had a skewed idea about how chroma key works. I thought that you just set your key color and software removes every pixel with that color, with some fudge factor so that if a pixel is close enough to be under a certain threshold, it will be removed as well. That mental model is only partially true, and it’s only one way of implementing chroma key. The point that my brain was glossing over is the definition of the word “close enough.” Since I’d only ever seen images with color information represented as either RGB or CMYK, I assumed that “close enough” meant nearby in RGB color space.

In RGB color space however, similar colors aren’t really next to each other. The lightest green is #00FF00, but the darkest green of the same shade is #000000. In fact, the darkest green is so dark that you can’t even tell that it’s green. It’s just black like every other black. The darkest red is the exact same color as the darkest green in RGB. Because of this, just replacing colors that are close in RGB is not enough.

                           

Prove it

To demonstrate this, we’re going to do some ImageMagick acrobatics. Basically what we have to do is create a mask image based on closeness to our key color in RGB space, then overlay our background image, but mask out the areas that are not close to our key color.

Step 1 is creating the mask image. If we start with a JPG or other compressed image, we need to convert it to a bitmap so that the compare command will work. Similar images in a compressed format may have differences that have nothing to do with the image itself, but are the result of compression artifacts.

The first part of creating the mask is changing every instance of a pixel with a color close to our key color to white and saving the results.

Next, we compare the original image to the mask and then re-save it as the completed mask.

convert IMG_7938.JPG IMG_7938.BMP
convert IMG_7938.BMP \
   -fuzz 20% -fill white -opaque \#467c46 IMG_7938_Mask.BMP
compare IMG_7938.BMP IMG_7938_Mask.BMP \
   -compose SRC IMG_7938_Mask.BMP

Now that we have the mask, we do some smoothing to try and remove some of the noise

convert IMG_7938_Mask.BMP \
   -morphology smooth square IMG_7938_Mask.BMP

Then we sandwich the background, the mask, and the original JPG together

composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \
   IMG_7938_Chromakeyed.JPG

and get the resulting images:

The Original Photo


The Mask


The Composited Result

Hue

If we look at that color map a little more closely, we will find the answer to our problem:

                           

Well OK color scientist, but how can we use this? Well, if we just change the colorspace we’re working with to something that has hue as a component, such as HSL or HSV, we can identify pixels of our image that have the same hue (or are close within a threshold) regardless of how luminous or saturated they are.

Luckily, we can do this pretty easily with ImageMagick. The ImageMagick documentation on chroma key gives us a hint as to how to proceed. We can use the -separate option on convert to separate the image into it’s distinct hues.

convert IMG_7938.JPG \
   -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG

If we combine that with what we already have, we get

convert IMG_7938.JPG \
   -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG

convert IMG_7938_Hues.JPG IMG_7938_Hues.BMP

convert IMG_7938_Hues.BMP \
   -fuzz 20% -fill white -opaque \#606060 IMG_7938_Mask.BMP
compare IMG_7938_Hues.BMP IMG_7938_Mask.BMP \
   -compose SRC IMG_7938_Mask.BMP
convert IMG_7938_Mask.BMP \
   -morphology smooth square IMG_7938_Mask.BMP

composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \
   IMG_7938_Chromakeyed.JPG

#606060 is the specific grey that our background’s hue turns out to be in the separated file:

The Original Photo


Hue Separated Image (notice that the background is #606060)


HSL Mask


HSL Chroma Key

I’ve been working on a project for which a long list of past tweets might not be the best for visibility, layout or usability. The Photobot 3000 is a portable photo booth without the booth. It’s a small box that you set up facing a wall with chairs and a green screen. Party guests sit in the chairs and hit the big red button on the front of the box. A series of lights counts down and the box takes their picture. All of the pictures are then projected onto a screen for everyone to see.

The projected image is a browser running a web page in full screen mode. For certain events, we also want to show tweets adjacent to the photos using a custom hashtag. We’ve been using Twitter’s widget and it’s been working great, but there are a few things that could make it better.

First, the text is very small, making it hard to read from far away. If we’re at a venue that doesn’t have a proper screen and are instead projecting the images on a non-homogenous wall, the tweets are even harder to read. If people can’t read the tweets, they’ll be less likely to use the hashtag themselves and the Twitter widget will just become one big waste of space.

Second, having the list of tweets displayed vertically to the right of the photos limits how large we can display the photos. We need to make sure that the Twitter widget adds to the experience and doesn’t detract from the important portion of the display: the images.

Third, there are some filters that we’d like to put on the tweets to be displayed that the default Twitter widget doesn’t support. For example, we’d like to only show tweets from the same day as the event.

So I decided to start playing around with getting data from Twitter and try to see if I could create my own Twitter widget that only shows one tweet at a time and position it right below each photo.

Before we continue, let me just say that I’ve been skittish about using ticker-type displays ever since the introduction of the second most hated HTML tag: <marquee> (second to <blink>). But I think I’ve made a reasonable case for a Twitter ticker widget as long as I keep it tasteful.
I promise it’ll be better than this!

Twitter’s really easy

It turns out that Twitter is really easy to work with if you’re just reading public tweets. They expose an interface to get search results in JSON format without having to log-in. If you add in the ease of working with JSON data which jQuery provides (which you should because jQuery is awesome,) the code to get Twitter search results becomes pretty trivial:

function getTweets()
{
	var hashTag = "%23flyers";
	var maxTweets = 99;

	var JSONQuery = "http://search.twitter.com/search.json?q="
		+ hashTag + "&callback=?&rpp=" + maxTweets;

 	$.getJSON(JSONQuery, getTweetCallback);

}
function getTweetCallback(ob)
{
	tweets = ob;
}

This code’s pretty straightforward. I’m asynchronously calling the Twitter search with an arbitrary hashtag (%23 is the octothorpe character.) When the request is complete, the getTweetCallback method is invoked. The rpp option stands for Results Per Page. The maximum is 100 which is fine for my uses, but if you need more than that, you’ll have to call the JSON search multiple times with the paging query options and aggregate the results.

Ok, but what do we do with it?

So we’ve got all of the tweets, now what do we do? the variable ob now contains a member called results which is an array of tweets. Each element has a bunch of data about each tweet, but the members that we’re going to use are profile_image_url, from_user, and text. So for example, to access the text of the latest tweet, you would do:

ob.results[0].text

It’s not enough to just show the latest tweet though. We want to cycle through the list, showing each tweet for a few seconds. For this, we’re going to use the Javscript’s setTimeout function. setTimeout sets a one-time timer and after the prescribed duration invokes the specified callback method.

timer = setTimeout(showNext, interval);

The final piece of the puzzle is making it look pretty. Another reason that jQeury is a joy to use is jQuery UI. jQuery UI adds a method called .toggle that shows or hides an HTML element with a few different options for visual effects. The effects look great and are really easy to invoke.

var toggleType = "blind"
$( "#TwitterTicker" ).toggle(toggleType, options, "slow");

Pulling it all together, we have the following Javascript:

function getTweets()
{
	refreshNum = 0;

	if(timer)
		clearTimeout(timer);

	var JSONQuery = "http://search.twitter.com/search.json?q="
		+ hashTag + "&callback=?&rpp=" + maxTweets;

 	$.getJSON(JSONQuery, getTweetCallback);

}
function getTweetCallback(ob)
{
	tweets = ob;
	showNext();
}
function showNext()
{
	if(tweets.results.length > refreshNum+1)
 	{
 		var toggleType = "blind";
 		var options = {}

 		//Calling toggle from jQuery UI to
 		//make the previous tweet invisible.
		$( "#TwitterTicker" ).toggle(toggleType, options, "slow",
		function(){

 			//Using the tweet data
 			var text =
			   "<img style='vertical-align:middle' src='"
			   + tweets.results[refreshNum].profile_image_url
			   + "'/> <a href='twitter.com'>"
			   + tweets.results[refreshNum].from_user
			   + "</a> "
			   + tweets.results[refreshNum].text;

			$( "#TwitterTicker" ).html(text);

 			if(refreshNum >= maxTweets)
 				getTweets();
 			else
 				refreshNum = refreshNum + 1;

 		});

		 //Calling toggle from jQuery UI to
 		 //make the new tweet visible
		 $( "#TwitterTicker" ).toggle(toggleType, options, "slow");

		 //Setting the timeout timer
		 timer = setTimeout(showNext, interval);
	 }
	 else
	 	getTweets();
}

The only slightly tricky part here is that I’m setting the text of the div element TwitterTicker in the callback for the toggle method that hides the div. This is so that the text of the tweet doesn’t change until the div is completely hidden. Here’s what it looks like:

One more thing

One more thing. setTimeout has this weird behavior on many browsers where if you switch tabs or the tab otherwise loses focus, when you return the browser tries to “catch up”. If you leave this code as-is, you may see a bunch of tweets flashing by quickly if you leave the tab and then come back. That’s pretty undesirable behavior here, so I added this code to the end of the Javascript file:

 $(document).blur(function(e) {
 	clearTimeout(timer);
 });
 $(document).focus(function(e) {
  	if(timer)
 	 	setTimeout(showNext, interval);
 });

This code disables the timer when the tab loses focus and restarts it when it gains focus again. The timer will start over at zero seconds when the focus comes back, but it’s better than playing catch up!

You can download the Javascript source here and an example html file here!

I knew the speaker system was meant for a computer, but didn’t think it made any difference. My TV has an audio-out jack, the speakers have an audio-in jack, what could go wrong? I got the speakers home and connected them to the TV. The sound was amazing. My ears ached retroactively from the abuse they had been subjected to from the built-in speakers. Plugging in the new speakers turned Snooki into a virtuoso, Walter White into Louis Armstrong, and the Bridalplasty contestants into sirens.

There was a problem though. When you hook up external speakers to my TV, the TV assumes that the speakers will now control the volume and locks you out of its volume control.

That would be fine if the speakers could recognize the volume up and volume down buttons on the TV remote, but this particular set of speakers is geared toward computers and has no IR input whatsoever. In place of an IR reciever it has a big standalone knob, which as an input device is pretty awesome, but you can’t use it remotely.

So for months, I would set the volume to a middle level and only change it in dire situations. It’s only a foot away from the sofa, but it might as well be inside of Yucca Mountain.

This was all just a minor inconvenience, but I would think about the problem a lot. I wasn’t really trying to solve the issue, I was just using it as a brain teaser. I would try to come up with the most Goldbergian contraption I could. The funniest one was something that intercepts the volume signals from the normal TV remote and then turns a gear that turns the big knob in a direction that corresponds to which volume button you pressed. The conversion of mechanical signal (button press) to IR signal to mechanical signal (gear turning) to electrical signal (from the knob to the speakers) to audio signal (change in volume) seemed hilariously complicated.

It would have stayed just a funny idea if I hadn’t started playing around with an Arduino. Arduinos are tiny programmable microcontrollers that work really well with any sort of electronic component. They use the Processing language which is super simple and fun to code in. Making this crazy contraption was the perfect project to get me started with the Arduino. I ran down to the local Radio Shack and bought the two main components of the system: an IR reciever module and a Servo (servo being a fancy word for turny thing) and got started.

Getting the IR module to respond to my remote

Getting the IR module to work was not as hard as I thought it would be. I basically used an amazing blog post as the outline for my code. I added some Serial.print statements into the pulse_to_bits method so that I could see the codes that the receiver was getting from my particular remote. I ran the code with the Arduino IDE’s built-in Serial Monitor and was able to find the codes for both the Volume Up button and the Volume Down button on my Samsung remote.

The real bit sequence is actually much longer, but the length I collected seems to be enough to be basically unique. At least in my living room.

Getting the servo to turn the right amount in the right direction

Getting the servo turning took a little bit more work. Arduino has a Servo library called SoftwareServo and there’s another Arduino Servo library on Google Code, but neither of them worked with the servos I had lying around. So not knowing anything about servos, I did some extensive research and more than a little trial and error to come up with the following code.

void rotateServo(int servoPin, boolean clockwise, int duration)
{
  int pulse = clockwise?1200:1800;
  unsigned long endTime = millis() + duration;

  int i = 1;
  while(endTime > millis())
  {
    digitalWrite(servoPin,HIGH);
    delayMicroseconds(pulse);
    digitalWrite(servoPin,LOW);
    delay(20);
    i++;
  }
}

If you send the servo I was working with a stream of ones pulsed at 1200 milliseconds, it will turn clockwise. If you send the same digits but with an 1800ms pulse, it will turn counterclockwise. I sincerely hope someone else looking for this information finds this page, it will save them a ton of time.

Tying it all together

Once I had the IR receiver and the servo figured out, I made this circuit.

Here is a video of it in action.

And this is the processing code that I uploaded to the Arduino.

#include <Servo.h>;
// 0.1 by pmalmsten
// 0.2 by farkinga
//This code is mostly verbatim from this blog post
//http://goo.gl/R3Zm8 with the primary modifications
//being the addition of a servo and the magic bit strings
//that represent the volume-up and volume-down buttons
//on my Samsung TV remote.

#define IR_BIT_LENGTH 32
#define BIT_1 1000          //Binary 1 threshold (Microseconds)
#define BIT_0 400           //Binary 0 threshold (Microseconds)
#define BIT_START 2000      //Start bit threshold (Microseconds)
#define DEBUG 1             //Serial connection must be started
                            //to debug

#define IR_PIN 0            //Sensor pin 1 wired through a
                            //220 ohm resistor

#define LED_PIN 9           //"Ready to Recieve" flag, not
                            //needed but nice
#define POWER_PIN 11       // the red LED that indicates
                           //if the power button is pressed.

#define SERVO_PIN 9

#define SERVO_INCREMENT 50

int runtime_debug = 0;
int output_key = 1;
int power_button = 0;

int servoPos = 0;

//IR Codes for my remote
int samsungVolumeUp[] = {1,1,1,0,0,0,0,0,1,1,1,0,0,0,0,0,1,1,1,0,0,0,0,0,0,0,0,1,1,1,1,1};
int samsungVolumeDown[] = {1,1,1,0,0,0,0,0,1,1,1,0,0,0,0,0,1,1,0,1,0,0,0,0,0,0,1,0,1,1,1,1};

Servo volumeKnob;

void setup() {
  pinMode(IR_PIN, INPUT);

  volumeKnob.attach(SERVO_PIN);
  volumeKnob.write(servoPos);
  delay(100);
  volumeKnob.detach();

  Serial.begin(9600);
}

void loop() {

  int bitStream[IR_BIT_LENGTH];

  get_ir_key(bitStream);		    //Fetch the key
  do_response(bitStream);

  delay(200);
}

void do_response(int key[])
{
  if(intArraysAreEqual(key, samsungVolumeUp))
  {
    Serial.println("Smasung Volume UP!");
    rotateServo(SERVO_PIN, true, SERVO_INCREMENT);

  }
  else if(intArraysAreEqual(key, samsungVolumeDown))
  {
    Serial.println("Samsung Volume DOWN!");
    rotateServo(SERVO_PIN, false, SERVO_INCREMENT);

  }
  else
    Serial.println("Unrecognized Key.");

}

void read_pulse(int data[], int num_bits)
{
  for (int i = 0; i < num_bits; i++)
  {
    data[i] = pulseIn(IR_PIN, HIGH);
  }
}

void pulse_to_bits(int pulse[], int bits[], int num_bits)
{

  if (DEBUG || runtime_debug) { Serial.println("-----"); }

  for(int i = 0; i < num_bits ; i++)
  {
    //if (DEBUG || runtime_debug) { Serial.println(pulse[i]); }

    if(pulse[i] > BIT_1) //is it a 1?
    {
      bits[i] = 1;
    }
    else if(pulse[i] > BIT_0) //is it a 0?
    {
      bits[i] = 0;
    } 

    else //data is invalid...
    {
      Serial.println("Error");
    }
    Serial.print(bits[i]);
  }
  Serial.println();

}

void get_ir_key(int bits[])
{
  int pulse[IR_BIT_LENGTH];

  do {} //Wait for a start bit
  while(pulseIn(IR_PIN, LOW) < BIT_START);

  read_pulse(pulse, IR_BIT_LENGTH);

  pulse_to_bits(pulse, bits, IR_BIT_LENGTH);

}
void printBits(int bits[])
{
  for(int i = 0; i < IR_BIT_LENGTH; i++)
    Serial.print(bits[i]);
  Serial.println();
}
boolean intArraysAreEqual(int Array1[], int Array2[])
{ 

  for(int i = 0; i < IR_BIT_LENGTH; i++)
  {
    if(Array1[i] != Array2[i])
      return false;
  }

  return true;
}
void rotateServo(int servoPin, boolean clockwise, int duration)
{
  int pulse = clockwise?1200:1800;
  unsigned long endTime = millis() + duration;

  int i = 1;
  while(endTime > millis())
  {
    digitalWrite(servoPin,HIGH);
    delayMicroseconds(pulse);
    digitalWrite(servoPin,LOW);
    delay(20);
    i++;
  }
}

Don't worry, broseph! I got this on lock.

And Joel Spolsky agrees. Our backup strategy needs to be verified or it’s worthless. If you’re not testing your backups, you’re begging the data gods to hit you with the double whammy: losing your primary data and then realizing your backups have never worked (or maybe you have that one backup from last spring when you were setting up the process.) Think of how terrible your future self will feel when he or she realizes that they were backing up the wrong database every day. How will the Family Guy fanfic community react to the content vacuum created by the loss of your blog?

They’ll move on. Some other Peter will start a blog and become more successful than you ever were. He’ll be lauded by the community and you’ll be shunned. You won’t be invited to Stewie-con next year, and people will think you’re crazy when you try to point out that the plot of the new guy’s opus, “Cirque de Family” is suspiciously similar to your masterpiece, “Brian’s Carni-fail.”

Neither of us want that to happen, so we have a choice. We can put all our trust into some broseph-spewing robot, or we can test our backups by faking an emergency and trying to restore to a new database. It’ll be like a fire drill, except we don’t have to stand outside our apartment building at 3 am in the middle of February (I know that’s a great opportunity to chat-up your cute neighbor, but you wouldn’t need the “perfect opportunity” if you just grew a pair and said hi in elevator.)

The Emergency

Just for narrative interest, lets invent an emergency. We want something that covers all aspects of the backup process. Since the main point of backing up to a local drive was so that Time Machine would pick it up, we should include a failure of the hard drive to which the backups are being written.

Picture this: you’re on an airplane returning from a vacation in Italy (fun time, bad drivers). A terrorist sitting behind you shoulder-surfs your MySQL password and overwrites your database with threats against American freedom using his CLEAR 4G card. Then he stands up and starts threatening the pilot with bombs that he smuggled in his shoes AND underwear. He’s standing right in front of you with his back turned, so you quietly shut your laptop and wait for just the right moment. At the crescendo of his anti-imperialist soliloquy, you whack him with your MacBook so hard the drive crashes, knocking him unconcious. Your database is corrupted, your hard drive is destroyed, but you’re a hero.

Weeks later, after the parade down Broadway in your honor and the loads of free hardware Apple has thrown at you, you decide it’s time to restore all of your data from your external Time Machine drive.

Get the MySql Backup File from Time Machine

Open a Finder window and navigate to the folder where your WordPress backups are stored. Click the Time Machine toolbar icon and click Enter Time Machine. Scroll through history to a time that has the most recent backup. Right-click the file and choose Restore.

I put it here somewhere...

Time Machine then prompts you for a location to put the restored file. I chose to create a new folder called Restore, just so I don’t get my restored backups and my actual backups mixed up.

Create a Restore Environment

We obviously don’t want this restore to overwrite your existing Family Guy fanfic blog (Fanficly Guy). After all, what if the backup is incomplete or corrupted? We’d end up destroying the data we’re trying to protect. We’ve got to create a separate restore environment.

Note: This section is going to cover creating a restore environment for WordPress on DreamHost. If you’re looking for how to restore from a MySqlDataDump in general, skip to Finally do the restore!

To recreate a similar environment, we’re going to make new Worpress and MySQL instances. That way we can restore the data to a new clean database. Plus, once we’re done restoring, we can check that it worked by hitting the new WordPress instance in our browser. If everything works correctly, we should see an exact clone of our original blog.

Just to keep the two environments separated, we’re also going to create a new subdomain. To add the new subdomain, go to Dreamhost’s web panel, and click Domains > Manage Domains > Add New Domain/Sub-Domain. Name it whatever you want; I named mine restore.pr.ogra.ms. Leave all of the other options at their defaults and click Fully host this domain.

Now to install a new WordPress instance, click Goodies > One-Click Installs > WordPress. Click Custom Installation so that it gives you the option of creating a new MySql database. Make sure that Automatically Create Database is highlighted in the Select Database dropdown and click Install it for me now!

They make it too easy.

Verify the MySql Connection

You should be redirected to a confirmation page that tells you the name of the new MySql database. In my case, it’s restore_pr_ogra_ms.

We need to make sure we can access the new database from our local machine, but what username and password do we use? If we click Goodies > MySql Databases, we can see the new database in the list. It also tells us the name of a new user that the installation created for us. We still need the password though!

Don’t worry, the web panel can do anything. Click on the user with access (in my case, restoreprograms). It will show the autogenerated password in the New Password section (after Currently:…). You should probably not change the autogenerated password because WordPress has connection information hard-coded in the wp-config.php file. If you change it here, you’ll have to change it there. We will be making changes to that file, but keeping the autogenerated password just makes troubleshooting easier.

While we’re in this part of the web panel, we need to add your IP address into the Allowable Hosts box. Otherwise, we will not be able to connect to this database from your local machine. Copy the IP address of your current computer (it’s displayed on the page under Your current computer is:) and paste it into the Allowable Hosts box. Save your settings.

Now we’re ready to test connectivity from our local machine. Open up a terminal window and run the MySqlDataDump command that we’ve been using to create these backups, but change the address and login credentials to point to your restore database. You could also connect via MySql and run some SELECT statements, but MySqlDataDump is a quick, simple and unambiguous way to check the connection.

/usr/local/mysql/bin/mysqldump
	-h restore.pr.ogra.ms
	--user=restoreprograms
	--password=AReallyStrongPassword restore_pr_ogra_ms

If you get a scrolling window of gibberish SQL code, congratulations! Your MySql restore environment is ready.

Verify the Restore WordPress Instance

Since we’re verifying everything, we might as well navigate to the new WordPress installation in a browser. Set up the Admin user and check to make sure the default blog entries show up. Logging in as Admin for the first time creates some database tables and sets up some configuration variables, so it’s important to do this step. Don’t worry about any of the settings too much, we’re going to replace all of the new tables with our restored data. Just make sure that you give the Admin user a password and login at least once. If you can see the new blog with the default posts when you navigate to your restore WordPress instance, you’re all set.

Finally do the restore!

This part is almost really simple, but there’s one trick. First, open a terminal window and navigate to the folder to which you restored your Time Machine backup. Now execute the following command:

/usr/local/mysql/bin/mysql
	-h mysql.restore.pr.ogra.ms
	--user=restoreprograms
	--password=AutoGeneratedPassword restore_pr_ogra_ms
	< 02022011-pr.ogra.ms.MySqlBackup.sql

make sure to substitute the credentials of your restore database (restore_pr_ogra_ms is the database within my restore MySql instance). Also substitute the name of your extracted backup file for 02022011-pr.ogra.ms.MySqlBackup.sql.

The backup file is a SQL script that CREATEs all of the WordPress tables and INSERTS all of your data (posts, comments, styles, etc). The above command just pipes the backup file to MySql, which knows to run the script against your restore database. So after this command is run, all of your backed-up data should be in the restore database.

Here’s the tricky part though: Every WordPress site has a unique prefix that it uses to name its tables. When we logged in to our new WordPress instance as Admin the first time, it generated all of the tables with this unique prefix. The same thing happened the first time you logged in to your real blog. When we ran the backup script, it recreated the tables from our real WordPress blog, prefix and all. This means that now we have two sets of tables in our restore database. That’s fine, we just need to tell the WordPress config file to use the prefix from our primary site instead of the new blank site and we should be golden.

Go back to the DreamHost web panel and click Goodies > MySql Databases. Find your restore database’s hostname (it should be of the form mysql.<new subdomain that we made for the restore>) and click phpMyAdmin.

phpYourAdmin

This brings up phpMyAdmin, a nice little web interface for MySql. We’re going to use this interface to figure out the prefix of our restored tables. To get a listing of all the tables in the WordPress database, click on your database name in the left sidebar (under the phpMyAdmin logo.)

Pre Fixe

When the list loads, we can see that there are two sets of tables with the same names but different prefixes. One set is the default tables that were installed when we logged in to our new WordPress instance as Admin, and the other is our restored blog data. Which is which though?

To figure that out, we are going to look at the configuration file that tells WordPress which prefix to look for. The, we’re going to change the file to point to the real tables. If you have SSH configured for passwordless login, run this command to login:

ssh TheBestHuman@pr.ogra.ms

Substituting your username for TheBestHuman and your primary domain name (not the new restore subdomain) for pr.ogra.ms. Now cd into the directory of your restore WordPress instance.

cd restore.pr.ogra.ms/blog

Open the wp-config.php file in your favorite editor. Find the line that says

$table_prefix  = 'wp_5qvylv_';

This tells us that ‘wp_5qvylv_’ is the prefix of the new default installation of WordPress. We want to point it to our restored tables, so simply change this value to the prefix of the other set of tables.

$table_prefix  = 'wp_wg0xrn_';

Finally, navigate to your restore instance (restore.pr.ogra.ms/blog) and BOOM, we have an exact copy of our blog!

This is Important, Just Do It

If you follow these instructions, it should take less than half an hour to verify a WordPress backup. If you weigh that half-hour against the hours and days you’ve spent crafting your online identity with an interesting and useful blog, it becomes laughable not to do a test restore at least once a month. If you never do a test restore, you have no idea whether or not your backups are working. Why spend time setting up and running a backup process every day if you’re not going to test it? As far as I’m concerned, an untested backup is the same as no backup.

Don’t just trust your robots; you never know when they’re going to turn evil and kill your data.

That may sound a little codependent, and unit testing in objective-c is a lot like an emotionally abusive relationship. There are a bunch of options out there, but OCUnit just feels comfortable and besides, it’s been with you since the beginning (because it came bundled with Xcode). OCUnit can’t stay terrible forever, it’s got to change right? right? What’s so bad about OCUnit anyway?

What’s Wrong with OCUnit?

Well for starters, all OCUnit tests happen at build time, not as separate run-time tasks. This small distinction makes it very difficult to debug unit tests. You can set things up so that when you choose Build and Debug in Xcode, it runs the tests once at build time and then a second time as your app starts running. On the second run-through, breakpoints will get hit, but the debugger doesn’t really work. It’s a mess that’s annoying to deal with, so you’re probably going to start falling back to NSLogging everything and just running the tests through Build.

Another problem is that there’s no separate GUI that shows all of your tests as green or red for pass or fail. If a test fails, the build fails. To find out which one failed, you have to drill down to the specific test in the build results window. It’s hard to do Red, Green, Refactor without red or green. There are third-party GUIs such as OCRunner that are doing their best to usher OCUnit into the mid-aughts by providing a nice GUI. I haven’t been able to get them working yet though (maybe a future post).

Finally, OCUnit requires specific naming conventions for test classes and methods. I admit that this is more of a style choice than a genuine shortcoming. In some other unit test packages such as NUnit for .NET, classes and methods are identified as test cases in metadata (NUnit uses attributes.) With OCUnit however, all test class names must end with “TestCase” and each test method must begin with “test” in order for them to be recognized by OCUnit. This may seem like just the choice of convention over configuration, but it’s really convention as configuration. Instead of making things easier and more intuitive, it hides information about the behavior of a class or method in its name.

What’s Right With OCUnit?

Ok, now you’re about to leave a comment “If you think OCUnit sucks so much, don’t use it!”, but that’s not true. I don’t think OCUnit is that bad. It gets the job done and could grow into the testing framework that we all want it to be. It does have some things going for it too.

When you’re choosing tools, a real concern is that development on a particular tool will stagnate and you’ll be left in the cold with outdated tools that you can’t migrate from. OCUnit is bundled with Xcode, so development is probably going to continue for the foreseeable future. Who knows, maybe Apple already has developers dedicated to improving it. Or writing a unit testing framework that’s compatible with it.

Also, despite my ranting, once you get OCUnit running and know its quirks, it’s pretty easy to use. Its shortcomings only really cost you time when setting it up and learning how to use it. Most of the problems I’ve mentioned above don’t have recurring costs.

Conveniently, you can minimize that initial cost of setting up and learning OCUnit by reading the rest of this post. It will not only help you survive the wasteland, but thrive with Unit Testing in the Objective-C world.

Add a new testing target

The first thing we need to do is add a new Unit Test Bundle target to an existing project that we want to unit test. Right click on the Targets section of the project file list and click Add>New Target.

Target Aqcuired.

Choose Unit Test Bundle and click Next.

Bundle Up!

Usually people name it HotPotatoTests where HotPotato is the name of their project. I don’t think the naming affects anything though, so feel free name it a cuss word if you like and click Finish.

I recommend naming it something safe for work if you're at work.

Add a new group for your test files

We’re probably going to end up writing at least one test class per functionality class, so the project’s going to start to get messy. Let’s just nip that clutter in the bud right now and create a group called Test into which we will put all of our test case files. Right click your project’s name in the Groups & Files pane, and go to Add>New Group. You should probably not name this a cuss word, you’re just going to confuse yourself.

Add a new Test Case file

Now it’s time to add a new testing class. Right click your newly created Test group and go to Add>New File… Choose Objective-C Test Case Class from the Cocoa Class section and hit Next.

This next part is VERY IMPORTANT. It is the first of two things that are absolutely vital to saving yourself frustration. Every test case class in OCUnit must end with the suffix TestCase. A nice convention is to name it HotPotatoTestCase where HotPotato is the name of the class under test. That way it’s stupid easy to find the tests for that class. Don’t forget to add TestCase to the end though. Promise me you won’t forget. I’m serious, pinky swear in the comments.

Once we have a test case class, the first thing we should do is write a failing test and run it to make sure it fails. Add the following code to the .m file of the test case class:

-(void) testThatTestingIsWorking
{
	STFail(@"YOU WILL ALWAYS BE A FAILURE");
}

This is the second VERY IMPORTANT part. It is again crucial to your sanity that you remember to do the following exactly: You must name all test methods so that they begin with the word test. Don’t forget! Oh please don’t forget.

Run your failing test

Now click that little unlabeled drop down in the upper left corner of the main Xcode window and switch the Active Target to your test target.

The next part is optional, but you probably want it. Expand the Targets section of your project. Now click and drag your executable target onto your test target. This will make your executable a dependency of your test target. Now, every time you attempt to build the test target, it will build the executable first. This ensures that the code under test always reflects the current code and not just whatever is lying around from the last time you built your executable.

Now comes the fun part, running the test! Just go to Build>Build and your test should run. The first time I did this, I was like “oh shit there are build errors, I’m doing this tutorial wrong.” Don’t worry, you’re not doing it wrong. Well you may be doing it wrong, but you should be expecting build errors here because a test failure counts as a build error. We need to dig a little to see exactly what the build error is though.

Finding the Failure

First bring up the Build Results window by doing command+shift+b. At this point, we could simply do a full expansion on the line where you see the errors by clicking the little button. But that gets you a big long log file that sucks to parse with your eyes. We can get a slightly more readable list of results by expanding Run custom shell script ‘Run Script’ > Run unit tests for architecture > Run test suite <octest path>. Here you will see a list of test case classes with an exclamation point if they’ve failed. You can further expand that to see the actual test methods that failed, and then drill down to the actual failure.

Ok, I guess it is red.

It’s annoying, but you’ll get used to it. Because you have to.

Test your actual code from your actual project

Most of the tutorials I’ve encountered stop here without telling you how to reference the code in your executable from your test case classes. It’s not intuitive but it’s just something you have to learn to do. I don’t know why they usually exclude this part, maybe the authors didn’t actually try to unit test anything significant. We live in the real world though, so we need to actually reference our actual code.

The first thing we’ll do is go to the Groups & Files pane and drag the code files (just the .m files) under test into the Compile Sources folder inside your test target. This is the magic that tells Xcode that you’re going to need access to the code from these files when you build the test target. So, if you’re testing a class called StringBasedTime, you grab StringBasedTime.m and drop them into the folder under Targets>HotPotatoTests>Compile Sources (where HotPotatoTests is the name of your test target.)

Cuss words everywhere!

Then in the test case file that you’re using to test this code (StringBasedTimeTest.m,) you simply add a directive to import the header file:

#import "StringBasedTime.h"

And that’s it, you can reference your executable class in your test code!

Example Unit Test

Here’s a simple unit test that I wrote in real life for a real reason. I’m using it to test the initializer of a class I’m building called StringBasedTime. StringBasedTime takes an argument in it’s initalizer that is a string representation of a time interval. The string representation is based on the output of QTStringFromTime, which is a method in QTKit, Cocoa’s QuickTime framework. It converts a QTime to an NSString in the format dd:hh:mm:ss.ff/ts. For example, @”0:00:00:00.00/00″ is the very beginning of a QuickTime movie. The test we’re looking at just makes sure that if someone initializes a new StringBasedTime with @”0:00:00:00.00/00″, each of the properties gets populated with zero. If one doesn’t, I want the failure output to tell the tester which property is non-zero. Here’s the code from the test case’s .m file:

#import "StringBasedTimeTestCase.h"
#import "StringBasedTime.h"

@implementation StringBasedTimeTestCase

-(void) testInitWithStringZeroTime
{
	NSString *ZeroTimeString = @"0:00:00:00.00/00";
	StringBasedTime *ZeroTime =
		[[StringBasedTime alloc] initWithString:ZeroTimeString];

	NSString* FailText = @"";

	if([ZeroTime days] != 0)
		FailText = [FailText stringByAppendingFormat:
					@"%d days when 0 expected."
					, [ZeroTime days]];

	if([ZeroTime hours] != 0)
		FailText = [FailText stringByAppendingFormat:
					@"%d hours when 0 expected."
					, [ZeroTime hours]];

	if([ZeroTime minutes] != 0)
		FailText = [FailText stringByAppendingFormat:
					@"%d minutes when 0 expected."
					, [ZeroTime minutes]];

	if([ZeroTime seconds] != 0)
		FailText = [FailText stringByAppendingFormat:
					@"%d seconds when 0 expected."
					, [ZeroTime seconds]];

	if([ZeroTime timescale] != 0)
		FailText = [FailText stringByAppendingFormat:
					@"%d timescale when 0 expected."
					, [ZeroTime timescale]];

	STAssertEquals(@"", FailText, FailText);

}
@end

Debugging

Remember, NSLog is your friend when debugging. I know printf debugging sucks, but sometimes it’s your best option. It might be possible to get the debugger working with OCUnit, but I haven’t been able to crack that code. If you figure it out, please feel free to leave a comment!

Further Reading

I figured out how to do all of this from a few good tutorials on OCUnit (and a few bad ones!):

• What is the best way to unit test Objective-C code?
• Chris Hanson’s series on Unit Testing in Xcode – Very comprehensive series of blog posts on OCUnit
• OCUnit: Integrated Unit Testing in Xcode
• Test Driving Your Code With OCUnit
• iOS Development Guide: Unit Testing Applications – This is where Apple lies and tells you that when a test fails, Xcode will highlight the line on which it failed.

MySQLDataDump

MySQL comes bundled with an awesome utility called MySQLDataDump. You throw it a database, user name and password, and it outputs a SQL script that creates all the tables and inserts all the data. First, you have to install MySQL locally though. Finding the right download for MySQL is a little like finding the installer for Java was back in the day. They try to hide it to make you accidentally pay money for it or something. I’ve gone ahead and sussed out the link for you though. After you install MySQL locally, it’s really easy to get a dump of your database from Terminal (This should all be on one line, but I didn’t want to make you have to scroll to the right forever so I broke it up a little):

/usr/local/mysql/bin/mysqldump
	-h WordPressDatabase.pr.ogra.ms
	--user=TheBestHuman
	--password=AReallyStrongPassword WordPressDatabaseName

Once the terminal window stops scrolling, we can talk about piping this output to a nice text file and zipping it up, but first a Dreamhost digression: In order to access a MySQL database remotely, you need to add the host to the Allowable Hosts list on their web panel. Don’t believe me? Check out the wiki. I’m not sure what I’m going to do when my IP address changes, but for now, I just went to Goodies>MySQL Databases on the web panel, then clicked on the user that I wanted to have remote access, then added my current IP address to the list of Allowable Hosts.

Put The Output of MySQLDataDump Into a File

I knew I would probably want to schedule this backup with a cron job later, so I wanted to put a date stamp in the filename that we pipe the output of MySQLDataDump to. Fortunately, Unix’s date does exactly what we need it to (again, this was put onto multiple lines for readability, the real command should be just a single line).

/usr/local/mysql/bin/mysqldump
	-h  WordPressDatabase.pr.ogra.ms
	--user=TheBestHuman
	--password=AReallyStrongPassword WordPressDatabaseName
>
	~/Documents/Projects/pr.ogra.ms/Wordpress Database Backup/
	$(date +%m%d%Y)-pr.ogra.ms.MySqlBackup.sql.zip

That’s nice… real nice. All this crap is going to start filling up the drive though, so let’s zip it up

/usr/local/mysql/bin/mysqldump
	-h WordPressDatabase.pr.ogra.ms
	--user=TheBestHuman
	--password=AReallyStrongPassword WordPressDatabaseName
	| gzip >
	~/Documents/Projects/pr.ogra.ms/Wordpress Database Backup/
	$(date +%m%d%Y)-pr.ogra.ms.MySqlBackup.sql.zip

Make sure you name the output file .sql.zip so that when you unzip, it has the .sql extension. Who knows, we might run this script in Windows some day.

Schedule it wih Crontab

I had never used crontab before this, though I knew it existed. It’s a little idiosyncratic, but easy to pick up. You can always use a third-party task scheduler like Lingon, but I find cron jobs to be way simpler to use.

In order to schedule a task, you need to add a line to crontab’s configuration text file. The line format is split into two sections, the first is five numbers separated by spaces indicating the minute, hour, day of the month, month, and day of the week that you want the job to run. You can substitute an asterisk (*) for any of these numbers to tell cron to execute the job every hour, minute, etc. The second section (after another space) is the actual command you want to execute. Here’s a nice little diagram that I stole from this site:

*     *     *   *    *        command to be executed
-     -     -   -    -
|     |     |   |    |
|     |     |   |    +----- day of week (0 - 6) (Sunday=0)
|     |     |   +------- month (1 - 12)
|     |     +--------- day of        month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)

How do you get to this config file though? just type

crontab -e

and you get a nice VI window with the file opened. If you’re confused or just don’t want to think right now, here is the line I inserted to tell crontab to execute the data dump every day at noon:

0 12 * * * /usr/local/mysql/bin/mysqldump
	-h   WordPressDatabase.pr.ogra.ms
	--user=TheBestHuman
	--password=AReallyStrongPassword
	WordpressDatabaseName
	| gzip >
	~/Documents/Projects/pr.ogra.ms/Wordpress Database Backup/
	$(date +%m%d%Y)-pr.ogra.ms.MySqlBackup.sql.zip

This should all be on one line though, here it is for copying:

0 12 * * * /usr/local/mysql/bin/mysqldump -h   WordPressDatabase.pr.ogra.ms --user=TheBestHuman --password=AReallyStrongPassword WordPressDatabaseName | gzip > ~/Documents/Projects/pr.ogra.ms/Wordpress Database Backup/ $(date +%m%d%Y)-pr.ogra.ms.MySqlBackup.sql.zip

Just paste that line into the crontab config file and your MySQL(YourSQL?) database is on the Time Machine Train!

"Your data's safe with me, Marty!"

I’ve always wondered why Mercurial’s executable was called hg. I thought that maybe it was a holdover from an ancestor project- the way Firefox’s executable was called mozilla for a while. It turns out that I was selling Mercurial’s developers short and that they do, in fact, have their shit together:

Chemistry Nerds!

It was named after the elemental symbol for mercury, Hg. The Mercurial Logo even looks like a glop of Mercury.

Liquid at Room Temperature

The symbol Hg is short for hydrargyrum which comes from the Latinized Greek prefix hyrda- meaning watery or runny and the word argyros meaning silver. So it means runny silver. I can imagine the Greeks making prank phone calls: “Is your silver runny? Then you better go catch it!” and getting people all mad and Latinized over it. Anyway, back to

3. Clone my existing local repositories to the server and then make that server the parent

Should be easy, right? It is. Ned Batchelder’s answer to this Stack Overflow question nails it. First, ssh into your Dreamhost account, then create a new repository on the server to house the repo that’s already on your local machine. Make sure that it’s in the repo parent directory (specified in hgweb.config) so that hgweb.cgi will pick it up.

cd ~/hg/repos

mkdir HotPotato

cd HotPotato

hg init

Now logout of your ssh session and use Terminal to navigate to the local directory of the repository you want to clone to the server. Use the hg push command to get the local repo up to your newly created server repo.

hg push ssh://TheBestHuman@pr.ogra.ms/hg/repos/HotPotato

Make sure you put the ssh:// prefix on the URI or you’ll get a repository not found error message. Now you should be able to go to hgweb.cgi and see your new (old) repo.

Your "New" Repo!

Your repo is available on the server, but we need to make it so that when you hg push from your local repo, it remembers what server to push to. This will make it so you don’t have to specify the server repo’s URI every time you want to push your changes up.

vi .hg/hgrc

with vi open, add the following lines to the hgrc file.

[paths]
default = ssh://TheBestHuman@pr.ogra.ms/hg/repos/HotPotato

Any changes you commit locally from now on will get pushed to the server if you run hg push!

But wait, if you ssh back in to the server and ls in the new repo’s directory, you won’t see any files!

Dub Tee Eff

What was all that console output if it wasn’t uploading the repo to the server? And how come when you navigate to hgweb.cgi, there’s a bunch of history about files that don’t seem to exist? Well, it has to do with some Mercurial fundamentals. The repo’s root directory (~/hg/repos/HotPotato) contains the working copy of the repo. Mercurial, however, only cares about committed code. Since we hg pushed the local repo to the server, Mercurial is aware of the changes, but has not applied those changes to the working copy on the server. The changes are all sitting in the .hg subfolder (~/hg/repos/HotPotato/.hg). If you never want to work on code and commit changes from the server, this is fine. It doesn’t really matter what the working copy on the server contains. But if you ever want to be able to edit code directly from an ssh session on the server, you’ll have to bring the working copy up to date.

hg update

You’ll also have to do this from any repo that you clone from the server copy.

And that’s it! Don’t worry, the workflow seems like a lot to remember but it will start to feel really natural after a while.

Terrorism is also distributed

Joel’s tutorial had me intrigued, Mercurious if you will. So when I finally got some time to work on personal projects, I started using it. And I’m hooked. I haven’t had much need for the D in DVCS yet though because I’m the only one committing any code. I’ve been using Mercurial as local revision control, but I’ve been using it for EVERYTHING; my resume redesign, Arduino code, even Movable Type theme modifications. I can’t get over how easy and quick it is to turn any normal project into a legitimate source controlled juggernaut. And there’s no guilt about polluting the codebase! After a while, I started thinking that maybe I should version control my OkCupid profile, or my Facebook friends. I know I’m addicted because sometimes when I’m at the bar, I think “I wish I could hg commit right now and revert if it turns out that I drank too much.”

I started to get nervous though. Any important data that’s not in the cloud makes me nervous. I had some time on my hands the other day, so I decided to try to install Mercurial on my DreamHost account and keep a main repository there. It would also give me a chance to commit code from different machines, or different partitions on a single machine without breaking the flow of work. I was thirsting for that all-powerful D.

Usually when I want to do something new with DreamHost, I pray that they have a one-click install in their web panel and if they don’t I give up. I’ve been writing software for most of my life and am embarrassed to say, installing things in Unix baffles me. I wasn’t so lucky on this one, there is no one-click install, but they did have the next best thing. One of the great things about DreamHost is that they have a large user base and their support wiki is packed with enough documentation to hang yourself. Sure enough, there’s a page on installing Mercurial. That’s not the end of the story though, there are really three things I want to do here:

1. Install Mercurial on my DreamHost account
2. Configure Mercurial to allow me to push and pull code from the server
3. Clone my existing local repositories to the server and then make that server the parent

1. Install Mercurial on my DreamHost account

Turtles are shell users too

In order to do this, we’re going to need a shell account and we’re going to need to log in to it. If you don’t already have a shell account, go to the DreamHost web panel and click Users > Manage Users. Now click Add User and set the Type of User Account to Shell.

Next, we need to SSH into the account. Our trusty DreamHost support wiki also has an entry for configuring SSH for passwordless login, which you can do to make it dead simple to login to your DreamHost shell:

ssh TheBestHuman@pr.ogra.ms

Now that we’re logged in, we can actually install Mercurial from the source code. This part is pretty much just following the instructions from this page of the support wiki. I’m going to repeat the steps here with some explanations so we’ll have everything in one place.

First, make a directory for the Mercurial source and download it. I wanted to make things easier for myself so I went with the same version that’s in the tutorial.

mkdir -p ~/srcs
cd ~/srcs
wget http://mercurial.selenic.com/release/mercurial-1.6.3.tar.gz

Mercurial’s latest stable release is 1.7.3, so we’re not that far off. Or are we? Who cares, I’m new to Mercurial, I won’t notice the difference. Next, unzip, make and make install it.

tar xvzf mercurial-1.6.3.tar.gz
cd mercurial-1.6.3
make all
make install-home

Now do some bash configuration stuff, changing the default path for Python and the Mercurial configuration file or something.

vi ~/.bash_profile

* add these lines at the bottom:
export PYTHONPATH=~/lib/python
export PATH=~/bin:$PATH
export HGRCPATH=TheBestHuman/.hgrc

Make sure to replace TheBestHuman with your DreamHost username because, well, there can be only one. After all this, you should be able to do

hg --version

and get 1.6.3 or whatever version you decided to go rogue and install.

So now we have Mercurial installed on the server. Badass. It’s only accessible if we’re shelled into our account though; we want to be able to push and pull code from Malaysia if we happen to be there and the McDonald’s has free wifi.

2. Configure Mercurial to allow us to push and pull code from the server

The heart of Mercurial’s code publishing capabilities is a Python script called HgWeb.cgi. It’s a nice little piece of code that not only serves the repos up to remote users, but also provides you with a slick read-only web interface to your revisions (basically a friendly web version of the hg log command). Sounds awesome, right? Let’s set it up!

First, make a subdirectory for your repos.

mkdir -p ~/hg/repos

Now create the configuration file for the publishing script.

vi ~/hg/hgweb.config

* ...and add the following lines:
[collections]
repos/ = repos/
[web]
style = gitweb

The default script is packaged with the Mercurial source, so copy it from there to your hg directory and add execute privileges to the copied file.

cp ~/srcs/mercurial-1.6.3/hgweb.cgi ~/hg
chmod +x ~/hg/hgweb.cgi

At this point, the wiki tells you to change the python version in hgweb.cgi, but I followed the directions and eventually (after a painful 20 minutes) had to switch this back to its original value. I’d recommend leaving the script as-is at first. If you’re getting a 500 error trying to access the hgweb.cgi page later on, you can go back and force the script to use python 2.4.

Next, change the following lines in hgweb.cgi

config = "/path/to/repo/or/config"
#import sys; sys.path.insert(0, "/path/to/python/lib")

to:

config = "/home/TheBestHuman/hg/hgweb.config"
import sys; sys.path.insert(0, "/home/TheBestHuman/lib/python")

Remember to replace TheBestHuman with your shell user name. Next, do some Apache crap:

vi ~/hg/.htaccess

* ...add the following lines (Comments optional) and then save the file 

# Taken from http://www.pmwiki.org/wiki/Cookbook/CleanUrls#samedir
# Used at http://ggap.sf.net/hg/
Options +ExecCGI
RewriteEngine On
#write base depending on where the base url lives
RewriteBase /hg
RewriteRule ^$ hgweb.cgi  [L]
# Send requests for files that exist to those files.
RewriteCond %{REQUEST_FILENAME} !-f
# Send requests for directories that exist to those directories.
RewriteCond %{REQUEST_FILENAME} !-d
# Send requests to hgweb.cgi, appending the rest of url.
RewriteRule (.*) hgweb.cgi/$1  [QSA,L]

After that, all that’s left to do is map the hg directory to somewhere on your domain. You can do this by creating a symbolic link to it from a sub directory of your domain.

cd ~/pr.ogra.ms
ln -s ~/hg

You might need to change some permissions.

 chmod 755 ~
 find ~/pr.ogra.ms -type d -print0 | xargs -0r chmod 755
 find ~/pr.ogra.ms -type f -print0 | xargs -0r chmod 644

At this point, you should be able to see an empty Mercurial page if you navigate to http://yourdomain/hg. Don’t worry, when you get some projects in there, it’ll look like this:

A preview of what's to come

So right now, we have an empty Mercurial installation with no repositories that’s absolutely useless. Mission Accomplished!

Not really, the next post will show you how to move an existing repo to your newly pimped Mercurial server and making it the root repo.

Continue on to Getting Mercurial Running on DreamHost Part II: Cloning an Existing Local Repo to the Server and Making it the Parent

]]> http://pr.ogra.ms/blog/?feed=rss2&p=17 1