Chroma Key Adventures Part 1: ImageMagick
In my last post, I mentioned the Photobot 3000, a portable photobooth-without-the-booth project I’ve been working on. After renting it out for a few events, we decided to investigate the possibility of doing in-software chroma key to add fun backgrounds to the photos before they get displayed on the projector.
We were already using the amazing and versatile ImageMagick to add a custom logo for each event to the images, so it made sense to see if ImageMagick could do something like chroma key. I was hoping it’d be as simple as calling ImageMagick with the right arguments, but it turned out to be a bit of a rabbit hole.
Before I started looking into this, I had a skewed idea about how chroma key works. I thought that you just set your key color and software removes every pixel with that color, with some fudge factor so that if a pixel is close enough to be under a certain threshold, it will be removed as well. That mental model is only partially true, and it’s only one way of implementing chroma key. The point that my brain was glossing over is the definition of the word “close enough.” Since I’d only ever seen images with color information represented as either RGB or CMYK, I assumed that “close enough” meant nearby in RGB color space.
In RGB color space however, similar colors aren’t really next to each other. The lightest green is #00FF00, but the darkest green of the same shade is #000000. In fact, the darkest green is so dark that you can’t even tell that it’s green. It’s just black like every other black. The darkest red is the exact same color as the darkest green in RGB. Because of this, just replacing colors that are close in RGB is not enough.
To demonstrate this, we’re going to do some ImageMagick acrobatics. Basically what we have to do is create a mask image based on closeness to our key color in RGB space, then overlay our background image, but mask out the areas that are not close to our key color.
Step 1 is creating the mask image. If we start with a JPG or other compressed image, we need to convert it to a bitmap so that the compare command will work. Similar images in a compressed format may have differences that have nothing to do with the image itself, but are the result of compression artifacts.
The first part of creating the mask is changing every instance of a pixel with a color close to our key color to white and saving the results.
Next, we compare the original image to the mask and then re-save it as the completed mask.
convert IMG_7938.JPG IMG_7938.BMP convert IMG_7938.BMP \ -fuzz 20% -fill white -opaque \#467c46 IMG_7938_Mask.BMP compare IMG_7938.BMP IMG_7938_Mask.BMP \ -compose SRC IMG_7938_Mask.BMP
Now that we have the mask, we do some smoothing to try and remove some of the noise
convert IMG_7938_Mask.BMP \ -morphology smooth square IMG_7938_Mask.BMP
Then we sandwich the background, the mask, and the original JPG together
composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \ IMG_7938_Chromakeyed.JPG
and get the resulting images:
The Original Photo
The Composited Result
This sucks. It can’t tell the difference between differently colored shadows because it’s in RGB. We need to account for the difference between things like the green-black in the folds of the background and the red-black in the folds of my shirt. This is where hue comes in.
If we look at that color map a little more closely, we will find the answer to our problem:
That’s right, the brightest green and the darkest green have the same exact hues! That’s because hue refers to distinct colors, “…without tint or shade (added white or black pigment, respectively).”
Well OK color scientist, but how can we use this? Well, if we just change the colorspace we’re working with to something that has hue as a component, such as HSL or HSV, we can identify pixels of our image that have the same hue (or are close within a threshold) regardless of how luminous or saturated they are.
Luckily, we can do this pretty easily with ImageMagick. The ImageMagick documentation on chroma key gives us a hint as to how to proceed. We can use the -separate option on convert to separate the image into it’s distinct hues.
convert IMG_7938.JPG \ -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG
If we combine that with what we already have, we get
convert IMG_7938.JPG \ -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG convert IMG_7938_Hues.JPG IMG_7938_Hues.BMP convert IMG_7938_Hues.BMP \ -fuzz 20% -fill white -opaque \#606060 IMG_7938_Mask.BMP compare IMG_7938_Hues.BMP IMG_7938_Mask.BMP \ -compose SRC IMG_7938_Mask.BMP convert IMG_7938_Mask.BMP \ -morphology smooth square IMG_7938_Mask.BMP composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \ IMG_7938_Chromakeyed.JPG
#606060 is the specific grey that our background’s hue turns out to be in the separated file:
The Original Photo
Hue Separated Image (notice that the background is #606060)
HSL Chroma Key
This is much better. There are still some problems though, which I will go over in my next post. I think this is the best result we can reasonably get short of writing our own chroma key functionality and operating on the pixels individually. That turns out to not be that scary though…