Skip to content

Recent Articles

Chroma Key Adventures Part 2: This Is How We Do It

Now that I’ve convinced you all that hue matching is the way to go with chroma key, let me tell you what’s wrong with the approach I championed in Chroma Key Adventures Part 1.

Actually better yet, let me show you:

Green Jacket Original                     Green Jacket Chroma Key in HSL Space

My green jacket is a very different green than the background, but it’s getting completely obliterated by ImageMagick.

Our hue matching system clearly has some issues. The main problem is that it’s all-or-nothing; a pixel is either masked or not. There’s no gradation. It’s not a surprise that it confuses the two greens. The whole reason we used HSL instead of RGB was because we can find similar colors easily with the hue component! But why can’t greens be gradually more opaque the further they are from the key color?

Wouldn’t it be great if there was a way to do chroma key where colors close to the key color but not quite had a transparency value somewhere in between on and off? There is a way, it’s called alpha compositing.

Sorry, ImageMagick

I was researching other chroma key programs and ran across this plugin for Paint.NET. I tried it on some images that I knew were a problem and was amazed. It worked way better than my ImageMagick method, even for the pixels that were not quite the right shade of green.

As much as I love the .NET open source community though, I didn’t really want to try to automate Paint.NET actions on the fly inside of the Photobot 3000. I also figured that I might need to tweak things in the algorithm to get it just right, so I was leaning heavily towards trying to implement chroma key myself. The creator of the Paint.NET plugin conveniently included a link to the paper that they used to create the plugin. The paper turned out to be pretty digestible, so I decided to writing the algorithm myself in C#.

The Algorithm

The algorithm starts with looping over the pixels in the image in RGB color space. For each pixel, it checks to see if the green component (we’re going to use green but they use blue in the paper) is greater than the blue and red components. If not, it leaves the pixel how it is. If so, it decides an alpha value (how transparent to make the pixel) based on how much more green the pixel is than it is red or blue.

For pixels where the green color is only slightly dominant, the alpha stays at 255 (completely opaque.) Then at some configurable threshold, the alphas start counting down until a second threshold where the alpha becomes zero. Every pixel that has a more green than red or blue value higher than the second threshold gets its alpha value set to 0 (completely transparent.)


AlphaMap

And that’s basically it. The paper makes it seem hard, but it’s really simple. And really fast, especially if we create a lookup table for the values in that graph. The paper doesn’t provide any actual values for the two thresholds, so we will have to play with them to get just the right alpha map.

The Code

First, let’s create the alpha map:

private void initalphaMap()
{
    List<int> alphaMapArray = new List<int>(510);

    //int fullyTransparentEndIndex = 15;
    //int semiTransparentEndIndex = 75;

    int fullyTransparentEndIndex = 15;
    int semiTransparentEndIndex = 100;

    for (int i = 0; i < 510; i++)
    {
        if (i < fullyTransparentEndIndex)
            alphaMapArray.Add(255);
        else if (i >= fullyTransparentEndIndex 
                 && i < semiTransparentEndIndex)
        {
            double lengthOfSemis = semiTransparentEndIndex 
                                   - fullyTransparentEndIndex;

            double indexValue = (double)(i - fullyTransparentEndIndex);

            double multiplier = 1 - (indexValue / (double)lengthOfSemis);

            int alphaValue = (int)(multiplier * 255.0);

            alphaMapArray.Add(alphaValue);
        }
        else
            alphaMapArray.Add(0);
    }
    alphaMap = alphaMapArray.ToArray();
}

Next, we need to loop through the pixels of our image. There are ways of looping through the image data which are safe and managed by .NET, but on the advice of this post, I decided to throw caution to the wind and access the memory directly inside of an unsafe block. Here’s the code I’m using to loop through all of the bits:

private Bitmap operateOnEachPixelOfImage(Bitmap original
                                         , CopyPixelOperation operation)
{
    Rectangle rect = new Rectangle(Point.Empty, original.Size);

    Bitmap output = original.Clone(rect
         , System.Drawing.Imaging.PixelFormat.Format32bppArgb);

    var modifications = output.LockBits(rect
         , System.Drawing.Imaging.ImageLockMode.ReadWrite
         , System.Drawing.Imaging.PixelFormat.Format32bppArgb);

    unsafe
    {
        int pixelSize = 4;

        for (int y = 0; y < modifications.Height; y++)
        {

            byte* row = (byte*)modifications.Scan0 
                        + (y * modifications.Stride);

            for (int x = 0; x < modifications.Width; x++)
            {
                byte[] pixel = {row[(x*pixelSize)]
                                , row[(x*pixelSize)+1]
                                , row[(x*pixelSize)+2]
                                , row[(x*pixelSize)+3]};

                byte[] newPixel = operation(pixel);

                row[(x * pixelSize)] = newPixel[0];
                row[(x * pixelSize)+1] = newPixel[1];
                row[(x * pixelSize)+2] = newPixel[2];
                row[(x * pixelSize)+3] = newPixel[3];

            }

        }

    }
    output.UnlockBits(modifications);
    return output;
}

You may have noticed that I’m not changing the pixels at all so far. I wanted to be able to reuse the bit-looping code with other operations for which I’d need to work with each individual pixel. I’m allowing the caller of this method to inject their own pixel manipulation operation into the loop through a delegate. This makes it trivial to create things like an alpha booster, or a method that makes a photo darker or lighter, and it helped in creating a hue-shifting method that I’m using for an enhanced version of the chroma key algorithm (which is a topic for another post.)

The actual method that creates our mask is implemented like this:

public Bitmap chromaKey(Bitmap original)
{

    return operateOnEachPixelOfImage(original,
            delegate(byte[] pixel)
            {
                int redValue = pixel[2];
                int greenValue = pixel[1];
                int blueValue = pixel[0];

                if (greenValue > redValue && greenValue > blueValue)
                    pixel[3] = (byte)alphaMap[(greenValue * 2)
                                - redValue - blueValue];
                else
                    pixel[3] = 255;

                return pixel;
            });
}

This method outputs a Bitmap that has a transparency where we want the background to be inserted. Once you have this, it’s simply a matter of overlaying the Laser image, which you can do using the built-in Graphics object.

Results

The results are hard to argue with. The new method still picks up some other greens but when it does, they’re only partially transparent. Here are the results, the old ImageMagick hue matching method is on the left, and the new algorithm is on the right.


Green Jacket Chroma Key in HSL Space                      Green Jacket Chroma Key Alpha Blending

As you can see, it works much better. More of the background is getting through, so it’s brighter, and it’s not making the jacket completely transparent. If you look closely at my right arm in the new photo, you’ll see a blue laser faintly dissolving into my jacket. That’s good! You’ll never see that at first glance. It’s unlikely we’ll ever have a chroma key method that’s as good at differentiating colors as the human eye, but we can trick the human eye into seeing what we want it to see.

Not only does this method work better, it’s also much faster and more configurable. If we happen to change the color of the background, it will be really easy to change the code to work with the different color. Also, we can keep honing the alpha map to produce the desired results, and maybe add new features that allow us to perfect this process.

1 Comment

Chroma Key Adventures Part 1: ImageMagick

In my last post, I mentioned the Photobot 3000, a portable photobooth-without-the-booth project I’ve been working on. After renting it out for a few events, we decided to investigate the possibility of doing in-software chroma key to add fun backgrounds to the photos before they get displayed on the projector.

We were already using the amazing and versatile ImageMagick to add a custom logo for each event to the images, so it made sense to see if ImageMagick could do something like chroma key. I was hoping it’d be as simple as calling ImageMagick with the right arguments, but it turned out to be a bit of a rabbit hole.

Before I started looking into this, I had a skewed idea about how chroma key works. I thought that you just set your key color and software removes every pixel with that color, with some fudge factor so that if a pixel is close enough to be under a certain threshold, it will be removed as well. That mental model is only partially true, and it’s only one way of implementing chroma key. The point that my brain was glossing over is the definition of the word “close enough.” Since I’d only ever seen images with color information represented as either RGB or CMYK, I assumed that “close enough” meant nearby in RGB color space.

In RGB color space however, similar colors aren’t really next to each other. The lightest green is #00FF00, but the darkest green of the same shade is #000000. In fact, the darkest green is so dark that you can’t even tell that it’s green. It’s just black like every other black. The darkest red is the exact same color as the darkest green in RGB. Because of this, just replacing colors that are close in RGB is not enough.


Bright Green                           Dark Green

Prove it

To demonstrate this, we’re going to do some ImageMagick acrobatics. Basically what we have to do is create a mask image based on closeness to our key color in RGB space, then overlay our background image, but mask out the areas that are not close to our key color.

Step 1 is creating the mask image. If we start with a JPG or other compressed image, we need to convert it to a bitmap so that the compare command will work. Similar images in a compressed format may have differences that have nothing to do with the image itself, but are the result of compression artifacts.

The first part of creating the mask is changing every instance of a pixel with a color close to our key color to white and saving the results.

Next, we compare the original image to the mask and then re-save it as the completed mask.

convert IMG_7938.JPG IMG_7938.BMP
convert IMG_7938.BMP \
   -fuzz 20% -fill white -opaque \#467c46 IMG_7938_Mask.BMP
compare IMG_7938.BMP IMG_7938_Mask.BMP \
   -compose SRC IMG_7938_Mask.BMP

Now that we have the mask, we do some smoothing to try and remove some of the noise

convert IMG_7938_Mask.BMP \
   -morphology smooth square IMG_7938_Mask.BMP

Then we sandwich the background, the mask, and the original JPG together

composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \
   IMG_7938_Chromakeyed.JPG

and get the resulting images:


Original
The Original Photo

Masked in RGB Space
The Mask

Chromakey in RGB Space
The Composited Result


This sucks. It can’t tell the difference between differently colored shadows because it’s in RGB. We need to account for the difference between things like the green-black in the folds of the background and the red-black in the folds of my shirt. This is where hue comes in.

Hue

If we look at that color map a little more closely, we will find the answer to our problem:


Dark Green Hue                           Bright Green Hue

That’s right, the brightest green and the darkest green have the same exact hues! That’s because hue refers to distinct colors, “…without tint or shade (added white or black pigment, respectively).”

Well OK color scientist, but how can we use this? Well, if we just change the colorspace we’re working with to something that has hue as a component, such as HSL or HSV, we can identify pixels of our image that have the same hue (or are close within a threshold) regardless of how luminous or saturated they are.

Luckily, we can do this pretty easily with ImageMagick. The ImageMagick documentation on chroma key gives us a hint as to how to proceed. We can use the -separate option on convert to separate the image into it’s distinct hues.

convert IMG_7938.JPG \
   -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG

If we combine that with what we already have, we get

convert IMG_7938.JPG \
   -colorspace HSL -channel Hue -separate IMG_7938_Hues.JPG

convert IMG_7938_Hues.JPG IMG_7938_Hues.BMP

convert IMG_7938_Hues.BMP \
   -fuzz 20% -fill white -opaque \#606060 IMG_7938_Mask.BMP
compare IMG_7938_Hues.BMP IMG_7938_Mask.BMP \
   -compose SRC IMG_7938_Mask.BMP
convert IMG_7938_Mask.BMP \
   -morphology smooth square IMG_7938_Mask.BMP

composite IMG_7938.JPG Lasers.gif IMG_7938_Mask.BMP \
   IMG_7938_Chromakeyed.JPG

#606060 is the specific grey that our background’s hue turns out to be in the separated file:


Original
The Original Photo

Original Hue Separated
Hue Separated Image (notice that the background is #606060)

Hue Mask
HSL Mask

Chromakey in HSL Space
HSL Chroma Key


This is much better. There are still some problems though, which I will go over in my next post. I think this is the best result we can reasonably get short of writing our own chroma key functionality and operating on the pixels individually. That turns out to not be that scary though…

No Comments