Skip to content

October 11, 2011

1

Chroma Key Adventures Part 2: This Is How We Do It

Now that I’ve convinced you all that hue matching is the way to go with chroma key, let me tell you what’s wrong with the approach I championed in Chroma Key Adventures Part 1.

Actually better yet, let me show you:

Green Jacket Original                     Green Jacket Chroma Key in HSL Space

My green jacket is a very different green than the background, but it’s getting completely obliterated by ImageMagick.

Our hue matching system clearly has some issues. The main problem is that it’s all-or-nothing; a pixel is either masked or not. There’s no gradation. It’s not a surprise that it confuses the two greens. The whole reason we used HSL instead of RGB was because we can find similar colors easily with the hue component! But why can’t greens be gradually more opaque the further they are from the key color?

Wouldn’t it be great if there was a way to do chroma key where colors close to the key color but not quite had a transparency value somewhere in between on and off? There is a way, it’s called alpha compositing.

Sorry, ImageMagick

I was researching other chroma key programs and ran across this plugin for Paint.NET. I tried it on some images that I knew were a problem and was amazed. It worked way better than my ImageMagick method, even for the pixels that were not quite the right shade of green.

As much as I love the .NET open source community though, I didn’t really want to try to automate Paint.NET actions on the fly inside of the Photobot 3000. I also figured that I might need to tweak things in the algorithm to get it just right, so I was leaning heavily towards trying to implement chroma key myself. The creator of the Paint.NET plugin conveniently included a link to the paper that they used to create the plugin. The paper turned out to be pretty digestible, so I decided to writing the algorithm myself in C#.

The Algorithm

The algorithm starts with looping over the pixels in the image in RGB color space. For each pixel, it checks to see if the green component (we’re going to use green but they use blue in the paper) is greater than the blue and red components. If not, it leaves the pixel how it is. If so, it decides an alpha value (how transparent to make the pixel) based on how much more green the pixel is than it is red or blue.

For pixels where the green color is only slightly dominant, the alpha stays at 255 (completely opaque.) Then at some configurable threshold, the alphas start counting down until a second threshold where the alpha becomes zero. Every pixel that has a more green than red or blue value higher than the second threshold gets its alpha value set to 0 (completely transparent.)


AlphaMap

And that’s basically it. The paper makes it seem hard, but it’s really simple. And really fast, especially if we create a lookup table for the values in that graph. The paper doesn’t provide any actual values for the two thresholds, so we will have to play with them to get just the right alpha map.

The Code

First, let’s create the alpha map:

private void initalphaMap()
{
    List<int> alphaMapArray = new List<int>(510);

    //int fullyTransparentEndIndex = 15;
    //int semiTransparentEndIndex = 75;

    int fullyTransparentEndIndex = 15;
    int semiTransparentEndIndex = 100;

    for (int i = 0; i < 510; i++)
    {
        if (i < fullyTransparentEndIndex)
            alphaMapArray.Add(255);
        else if (i >= fullyTransparentEndIndex 
                 && i < semiTransparentEndIndex)
        {
            double lengthOfSemis = semiTransparentEndIndex 
                                   - fullyTransparentEndIndex;

            double indexValue = (double)(i - fullyTransparentEndIndex);

            double multiplier = 1 - (indexValue / (double)lengthOfSemis);

            int alphaValue = (int)(multiplier * 255.0);

            alphaMapArray.Add(alphaValue);
        }
        else
            alphaMapArray.Add(0);
    }
    alphaMap = alphaMapArray.ToArray();
}

Next, we need to loop through the pixels of our image. There are ways of looping through the image data which are safe and managed by .NET, but on the advice of this post, I decided to throw caution to the wind and access the memory directly inside of an unsafe block. Here’s the code I’m using to loop through all of the bits:

private Bitmap operateOnEachPixelOfImage(Bitmap original
                                         , CopyPixelOperation operation)
{
    Rectangle rect = new Rectangle(Point.Empty, original.Size);

    Bitmap output = original.Clone(rect
         , System.Drawing.Imaging.PixelFormat.Format32bppArgb);

    var modifications = output.LockBits(rect
         , System.Drawing.Imaging.ImageLockMode.ReadWrite
         , System.Drawing.Imaging.PixelFormat.Format32bppArgb);

    unsafe
    {
        int pixelSize = 4;

        for (int y = 0; y < modifications.Height; y++)
        {

            byte* row = (byte*)modifications.Scan0 
                        + (y * modifications.Stride);

            for (int x = 0; x < modifications.Width; x++)
            {
                byte[] pixel = {row[(x*pixelSize)]
                                , row[(x*pixelSize)+1]
                                , row[(x*pixelSize)+2]
                                , row[(x*pixelSize)+3]};

                byte[] newPixel = operation(pixel);

                row[(x * pixelSize)] = newPixel[0];
                row[(x * pixelSize)+1] = newPixel[1];
                row[(x * pixelSize)+2] = newPixel[2];
                row[(x * pixelSize)+3] = newPixel[3];

            }

        }

    }
    output.UnlockBits(modifications);
    return output;
}

You may have noticed that I’m not changing the pixels at all so far. I wanted to be able to reuse the bit-looping code with other operations for which I’d need to work with each individual pixel. I’m allowing the caller of this method to inject their own pixel manipulation operation into the loop through a delegate. This makes it trivial to create things like an alpha booster, or a method that makes a photo darker or lighter, and it helped in creating a hue-shifting method that I’m using for an enhanced version of the chroma key algorithm (which is a topic for another post.)

The actual method that creates our mask is implemented like this:

public Bitmap chromaKey(Bitmap original)
{

    return operateOnEachPixelOfImage(original,
            delegate(byte[] pixel)
            {
                int redValue = pixel[2];
                int greenValue = pixel[1];
                int blueValue = pixel[0];

                if (greenValue > redValue && greenValue > blueValue)
                    pixel[3] = (byte)alphaMap[(greenValue * 2)
                                - redValue - blueValue];
                else
                    pixel[3] = 255;

                return pixel;
            });
}

This method outputs a Bitmap that has a transparency where we want the background to be inserted. Once you have this, it’s simply a matter of overlaying the Laser image, which you can do using the built-in Graphics object.

Results

The results are hard to argue with. The new method still picks up some other greens but when it does, they’re only partially transparent. Here are the results, the old ImageMagick hue matching method is on the left, and the new algorithm is on the right.


Green Jacket Chroma Key in HSL Space                      Green Jacket Chroma Key Alpha Blending

As you can see, it works much better. More of the background is getting through, so it’s brighter, and it’s not making the jacket completely transparent. If you look closely at my right arm in the new photo, you’ll see a blue laser faintly dissolving into my jacket. That’s good! You’ll never see that at first glance. It’s unlikely we’ll ever have a chroma key method that’s as good at differentiating colors as the human eye, but we can trick the human eye into seeing what we want it to see.

Not only does this method work better, it’s also much faster and more configurable. If we happen to change the color of the background, it will be really easy to change the code to work with the different color. Also, we can keep honing the alpha map to produce the desired results, and maybe add new features that allow us to perfect this process.

1 Comment Post a comment
  1. Jeremy
    December 14, 2015

    Wish we could get these results from the software me and my kids use. Been trying to translate your code to VB.NET, but no luck. Would be a great edition to a stop-motion program I have in mind.

Leave a Reply

Your email address will not be published.