Chroma Key Adventures Part 2: This Is How We Do It
Now that I’ve convinced you all that hue matching is the way to go with chroma key, let me tell you what’s wrong with the approach I championed in Chroma Key Adventures Part 1.
Actually better yet, let me show you:
My green jacket is a very different green than the background, but it’s getting completely obliterated by ImageMagick.
Our hue matching system clearly has some issues. The main problem is that it’s all-or-nothing; a pixel is either masked or not. There’s no gradation. It’s not a surprise that it confuses the two greens. The whole reason we used HSL instead of RGB was because we can find similar colors easily with the hue component! But why can’t greens be gradually more opaque the further they are from the key color?
Wouldn’t it be great if there was a way to do chroma key where colors close to the key color but not quite had a transparency value somewhere in between on and off? There is a way, it’s called alpha compositing.
Sorry, ImageMagick
I was researching other chroma key programs and ran across this plugin for Paint.NET. I tried it on some images that I knew were a problem and was amazed. It worked way better than my ImageMagick method, even for the pixels that were not quite the right shade of green.
As much as I love the .NET open source community though, I didn’t really want to try to automate Paint.NET actions on the fly inside of the Photobot 3000. I also figured that I might need to tweak things in the algorithm to get it just right, so I was leaning heavily towards trying to implement chroma key myself. The creator of the Paint.NET plugin conveniently included a link to the paper that they used to create the plugin. The paper turned out to be pretty digestible, so I decided to writing the algorithm myself in C#.
The Algorithm
The algorithm starts with looping over the pixels in the image in RGB color space. For each pixel, it checks to see if the green component (we’re going to use green but they use blue in the paper) is greater than the blue and red components. If not, it leaves the pixel how it is. If so, it decides an alpha value (how transparent to make the pixel) based on how much more green the pixel is than it is red or blue.
For pixels where the green color is only slightly dominant, the alpha stays at 255 (completely opaque.) Then at some configurable threshold, the alphas start counting down until a second threshold where the alpha becomes zero. Every pixel that has a more green than red or blue value higher than the second threshold gets its alpha value set to 0 (completely transparent.)
And that’s basically it. The paper makes it seem hard, but it’s really simple. And really fast, especially if we create a lookup table for the values in that graph. The paper doesn’t provide any actual values for the two thresholds, so we will have to play with them to get just the right alpha map.
The Code
First, let’s create the alpha map:
private void initalphaMap()
{
List<int> alphaMapArray = new List<int>(510);
//int fullyTransparentEndIndex = 15;
//int semiTransparentEndIndex = 75;
int fullyTransparentEndIndex = 15;
int semiTransparentEndIndex = 100;
for (int i = 0; i < 510; i++)
{
if (i < fullyTransparentEndIndex)
alphaMapArray.Add(255);
else if (i >= fullyTransparentEndIndex
&& i < semiTransparentEndIndex)
{
double lengthOfSemis = semiTransparentEndIndex
- fullyTransparentEndIndex;
double indexValue = (double)(i - fullyTransparentEndIndex);
double multiplier = 1 - (indexValue / (double)lengthOfSemis);
int alphaValue = (int)(multiplier * 255.0);
alphaMapArray.Add(alphaValue);
}
else
alphaMapArray.Add(0);
}
alphaMap = alphaMapArray.ToArray();
}
Next, we need to loop through the pixels of our image. There are ways of looping through the image data which are safe and managed by .NET, but on the advice of this post, I decided to throw caution to the wind and access the memory directly inside of an unsafe block. Here’s the code I’m using to loop through all of the bits:
private Bitmap operateOnEachPixelOfImage(Bitmap original
, CopyPixelOperation operation)
{
Rectangle rect = new Rectangle(Point.Empty, original.Size);
Bitmap output = original.Clone(rect
, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
var modifications = output.LockBits(rect
, System.Drawing.Imaging.ImageLockMode.ReadWrite
, System.Drawing.Imaging.PixelFormat.Format32bppArgb);
unsafe
{
int pixelSize = 4;
for (int y = 0; y < modifications.Height; y++)
{
byte* row = (byte*)modifications.Scan0
+ (y * modifications.Stride);
for (int x = 0; x < modifications.Width; x++)
{
byte[] pixel = {row[(x*pixelSize)]
, row[(x*pixelSize)+1]
, row[(x*pixelSize)+2]
, row[(x*pixelSize)+3]};
byte[] newPixel = operation(pixel);
row[(x * pixelSize)] = newPixel[0];
row[(x * pixelSize)+1] = newPixel[1];
row[(x * pixelSize)+2] = newPixel[2];
row[(x * pixelSize)+3] = newPixel[3];
}
}
}
output.UnlockBits(modifications);
return output;
}
You may have noticed that I’m not changing the pixels at all so far. I wanted to be able to reuse the bit-looping code with other operations for which I’d need to work with each individual pixel. I’m allowing the caller of this method to inject their own pixel manipulation operation into the loop through a delegate. This makes it trivial to create things like an alpha booster, or a method that makes a photo darker or lighter, and it helped in creating a hue-shifting method that I’m using for an enhanced version of the chroma key algorithm (which is a topic for another post.)
The actual method that creates our mask is implemented like this:
public Bitmap chromaKey(Bitmap original)
{
return operateOnEachPixelOfImage(original,
delegate(byte[] pixel)
{
int redValue = pixel[2];
int greenValue = pixel[1];
int blueValue = pixel[0];
if (greenValue > redValue && greenValue > blueValue)
pixel[3] = (byte)alphaMap[(greenValue * 2)
- redValue - blueValue];
else
pixel[3] = 255;
return pixel;
});
}
This method outputs a Bitmap that has a transparency where we want the background to be inserted. Once you have this, it’s simply a matter of overlaying the Laser image, which you can do using the built-in Graphics object.
Results
The results are hard to argue with. The new method still picks up some other greens but when it does, they’re only partially transparent. Here are the results, the old ImageMagick hue matching method is on the left, and the new algorithm is on the right.
As you can see, it works much better. More of the background is getting through, so it’s brighter, and it’s not making the jacket completely transparent. If you look closely at my right arm in the new photo, you’ll see a blue laser faintly dissolving into my jacket. That’s good! You’ll never see that at first glance. It’s unlikely we’ll ever have a chroma key method that’s as good at differentiating colors as the human eye, but we can trick the human eye into seeing what we want it to see.
Not only does this method work better, it’s also much faster and more configurable. If we happen to change the color of the background, it will be really easy to change the code to work with the different color. Also, we can keep honing the alpha map to produce the desired results, and maybe add new features that allow us to perfect this process.
Chroma Key Adventures Part 1: ImageMagick
In my last post, I mentioned the Photobot 3000, a portable photobooth-without-the-booth project I’ve been working on. After renting it out for a few events, we decided to investigate the possibility of doing in-software chroma key to add fun backgrounds to the photos before they get displayed on the projector.
We were already using the amazing and versatile ImageMagick to add a custom logo for each event to the images, so it made sense to see if ImageMagick could do something like chroma key. I was hoping it’d be as simple as calling ImageMagick with the right arguments, but it turned out to be a bit of a rabbit hole.
Before I started looking into this, I had a skewed idea about how chroma key works. I thought that you just set your key color and software removes every pixel with that color, with some fudge factor so that if a pixel is close enough to be under a certain threshold, it will be removed as well. That mental model is only partially true, and it’s only one way of implementing chroma key. The point that my brain was glossing over is the definition of the word “close enough.” Since I’d only ever seen images with color information represented as either RGB or CMYK, I assumed that “close enough” meant nearby in RGB color space.
In RGB color space however, similar colors aren’t really next to each other. The lightest green is #00FF00, but the darkest green of the same shade is #000000. In fact, the darkest green is so dark that you can’t even tell that it’s green. It’s just black like every other black. The darkest red is the exact same color as the darkest green in RGB. Because of this, just replacing colors that are close in RGB is not enough.
Prove it
To demonstrate this, we’re going to do some ImageMagick acrobatics. Basically what we have to do is create a mask image based on closeness to our key color in RGB space, then overlay our background image, but mask out the areas that are not close to our key color.
Step 1 is creating the mask image. If we start with a JPG or other compressed image, we need to convert it to a bitmap so that the compare command will work. Similar images in a compressed format may have differences that have nothing to do with the image itself, but are the result of compression artifacts.
The first part of creating the mask is changing every instance of a pixel with a color close to our key color to white and saving the results.
Next, we compare the original image to the mask and then re-save it as the completed mask.
convert IMG_7938.JPG IMG_7938.BMP convert IMG_7938.BMP \ -fuzz 20% -fill white -opaque \#467c46 IMG_7938_Mask.BMP compare IMG_7938.BMP IMG_7938_Mask.BMP \ -compose SRC IMG_7938_Mask.BMP
Now that we have the mask, we do some smoothing to try and remove some of the noise
convert IMG_7938_Mask.BMP \ -morphology smooth square IMG_7938_Mask.BMP
Then we sandwich the background, the mask, and the original JPG together
composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \ IMG_7938_Chromakeyed.JPG
and get the resulting images:
The Original Photo
The Mask
The Composited Result
This sucks. It can’t tell the difference between differently colored shadows because it’s in RGB. We need to account for the difference between things like the green-black in the folds of the background and the red-black in the folds of my shirt. This is where hue comes in.
Hue
If we look at that color map a little more closely, we will find the answer to our problem:
That’s right, the brightest green and the darkest green have the same exact hues! That’s because hue refers to distinct colors, “…without tint or shade (added white or black pigment, respectively).”
Well OK color scientist, but how can we use this? Well, if we just change the colorspace we’re working with to something that has hue as a component, such as HSL or HSV, we can identify pixels of our image that have the same hue (or are close within a threshold) regardless of how luminous or saturated they are.
Luckily, we can do this pretty easily with ImageMagick. The ImageMagick documentation on chroma key gives us a hint as to how to proceed. We can use the -separate option on convert to separate the image into it’s distinct hues.
convert IMG_7938.JPG \ -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG
If we combine that with what we already have, we get
convert IMG_7938.JPG \ -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG convert IMG_7938_Hues.JPG IMG_7938_Hues.BMP convert IMG_7938_Hues.BMP \ -fuzz 20% -fill white -opaque \#606060 IMG_7938_Mask.BMP compare IMG_7938_Hues.BMP IMG_7938_Mask.BMP \ -compose SRC IMG_7938_Mask.BMP convert IMG_7938_Mask.BMP \ -morphology smooth square IMG_7938_Mask.BMP composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \ IMG_7938_Chromakeyed.JPG
#606060 is the specific grey that our background’s hue turns out to be in the separated file:
The Original Photo
Hue Separated Image (notice that the background is #606060)
HSL Mask
HSL Chroma Key
This is much better. There are still some problems though, which I will go over in my next post. I think this is the best result we can reasonably get short of writing our own chroma key functionality and operating on the pixels individually. That turns out to not be that scary though…
