Change image background

 In order to change the image background, it is important to distinguish the boundary between the person and the background well.  In YouTube broadcasting and TV broadcasting, the chroma key technique is used to accurately distinguish the boundary between the person and the background.

Background removal using chroma key

A chroma key is a technique used in film, video and still photography to replace a portion of an image with a new image. This is most commonly used to replaced a coloured background with a different setting.  The background color used for chroma key is usually blue or green. Persons who appear in the background of Chromakey must not wear clothes of the same color as Chromakey. If possible, wear complementary colors to accurately distinguish the boundary.

<Chromakey example>

Change chroma key photo background in OpenCV

This task is very easy. For chroma key photos, the boundary line can be distinguished fairly accurately only with the OpenCV function.

Convert chroma key image to BGR -> HSV color space. Some of you may be wondering why it is changed to HSV color instead of RGB color. The biggest reason is the difference in brightness of the chroma key background. A part of the chroma key background may have a different brightness due to a difference in lighting. For HSV colors, the same Hue value works very well with different brightnesses.

HSV

The HSV color space or HSV model is a method of expressing colors and a method of arranging colors according to the method. Specify a specific color using the coordinates of Hue, Saturation, and Value.

<from https://en.wikipedia.org/wiki/HSL_and_HSV>

And I will use the cv2.inRange() function to create a mask image .  If you use green chromakey, test the  cv2.inRange() with hue range 80 ~ 160, and if you use blue chromakey, test with hue range 180 ~ 260.

<hue <->angle >

I'll use this image for testing.

<chromakey image sample>

If you want to know the exact HSV value of the chroma key background color, This can be seen by outputting the pixel value where the background is located in the image. 

In the example below, when the HSV value of (100,100) pixels with a green background was output, a value of [59 247 250] was obtained. That is, it is the value of H:59, S:247, V:250. This value is slightly different from the Hue value in the figure above.

The following is the code to extract the green background.  

import numpy as np
import cv2
import os, sys
from matplotlib import pyplot as plt, gridspec

image_name =  'C:\\lsh\\study\\image\\chromakey.jpg' 
img = cv2.imread(image_name, cv2.IMREAD_UNCHANGED)
print(img.shape)

hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
print(hsv.shape)
print(hsv[100, 100,:])
mask = cv2.inRange(hsv, (55, 200, 0), (65, 255, 255))
plt.figure(figsize=(15,15))
grid = gridspec.GridSpec(1,2)

# Setup axes
ax0 = plt.subplot(grid[0])
ax1 = plt.subplot(grid[1])
ax0.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
print(mask.shape)
ax1.imshow(cv2.cvtColor(mask, cv2.COLOR_GRAY2RGB))
plt.show()

You can see the following output. The parameters (55, 200, 0), (65, 255, 255) of the cv2.inRange function are adjusted again by referring to the output value [59 247 250].

<output>

Now let's change the background image using a mask. The cv2.copyTo function is defined as follows.

• copyTo(src, mask[, dst]) -> dst

That is, after changing the mask area in the src image to dst, it is saved in dst.

beach_name =  'C:\\lsh\\study\\image\\beach.jpg' 

beach = cv2.imread(beach_name, cv2.IMREAD_UNCHANGED)

cv2.copyTo(beach, mask, img)
plt.figure(figsize=(15,15))

ax0 = plt.subplot(grid[0])
ax1 = plt.subplot(grid[1])
ax0.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
ax1.imshow(cv2.cvtColor(beach, cv2.COLOR_BGR2RGB))
plt.show()

You can see the following output. 

Changing the background of an image using chroma key is very easy and the quality is excellent. However, it is not easy to remove the background from a normal photo without using chroma key. In general, artificial intelligence is used to remove the background from a person image.

Remove background using rembg

There are many artificial intelligence solutions that remove the background. An easily available package in Python is rembg. To use rembg, you need to install Pytorch and torchvision beforehand. Be sure to install these packages already for your operating system and whether you are using a GPU or not.

Requirements

• python 3.8 or newer
• torch and torchvision stable version (https://pytorch.org)

A description of rembg can be found at https://pypi.org/project/rembg/. The rembg is based on u2net. u2net is at https://github.com/xuebinqin/U-2-Net.

Now let's work with the following image.

<Leonardo-dicaprio announcing his 2016 Oscars>

Since this image does not use chroma key, it is impossible to remove the background using the cv2.inRange function used earlier. We will first remove the background using rembg and then apply the background image.

from rembg.bg import remove
import numpy as np
import io
from PIL import Image, ImageFile
import cv2
from matplotlib import pyplot as plt
import sys, time

ImageFile.LOAD_TRUNCATED_IMAGES = True

image_dir =  'C:\\lsh\\study\\image\\' 
input_path = image_dir + 'dicaprio.jpg'

f = np.fromfile(input_path)
result = remove(f)
pil_img = Image.open(io.BytesIO(result)).convert("RGBA")
img = cv2.imread(input_path, cv2.IMREAD_UNCHANGED)
plt.figure(figsize=(15,15))

ax0 = plt.subplot(grid[0])
ax1 = plt.subplot(grid[1])
ax0.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
ax1.imshow(pil_img)
plt.show()

You can see the following output. In general, the background has been erased cleanly.

Now let's make a mask just like in Chromakey. 

#convert to openCV format
plt_array = np.asarray(pil_img, dtype="uint8")
cv_array = cv2.cvtColor(plt_array, cv2.COLOR_RGBA2BGRA)
H, W, C = cv_array.shape
print(H,W,C)
mask = cv_array[:,:, 3]
mask = cv2.bitwise_not(mask)
print(mask[200,300])

plt.figure(figsize=(15,15))

ax0 = plt.subplot(grid[0])
ax1 = plt.subplot(grid[1])
ax0.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
ax1.imshow(cv2.cvtColor(mask, cv2.COLOR_GRAY2RGB))
plt.show()

You can see the following output.

Let's change the background image using a mask. 

beach = cv2.resize(beach,(W, H), cv2.INTER_LINEAR)

cv2.copyTo(beach, mask, img)
plt.figure(figsize=(15,15))

ax0 = plt.subplot(grid[0])
ax1 = plt.subplot(grid[1])
ax0.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
ax1.imshow(cv2.cvtColor(beach, cv2.COLOR_BGR2RGB))
plt.show()

However, the results are a bit odd.

The cause of the strange result is the mask value. Although it looks black to our eyes, the value of the black part of the mask image (200,300) is 1, not 0. The reason for this is that the mask created by rembg may not be exactly zero because it is created based on probability. Therefore, we work by forcing the value of the mask to be 0 or 255.

Later, we will see how to smoothly change the background while maintaining the mask value.

Add the following line to the code and change it to 0 if the value of the mask image is 30 or less. The 30 used here can be adjusted according to the characteristics of the mask.

• mask[np.where(mask < 30)] = 0

The changed code is as follows.

mask = cv2.bitwise_not(mask)
mask[np.where(mask < 30)] = 0
print(mask[200,300]) 

Let's run it again. The (200, 300) coordinate value of the mask image is now output as 0, and the picture we want is shown as follows.

Smoother blending with alpha value

The alpha value of the person and the background boundary changes abruptly from 0 to 255. In order to express this boundary smoothly, the pixel value of the area can be created by mixing the pixel value of the person image and the background image in a ratio equal to the alpha value using the alpha value. However, this method takes a lot of time because all Python operations must be performed in pixel units.

beach = cv2.resize(beach,(W, H), cv2.INTER_LINEAR)

plt_array = np.asarray(pil_img, dtype="uint8")
im = cv2.cvtColor(plt_array, cv2.COLOR_RGBA2BGRA)

canvas_image = np.zeros((H, W, 3), np.uint8)

for y in range(0, H):
    for x in range(0, W):
        alpha = im[y,x,3]
        b = int((float(im[y,x,0]) * alpha + float(beach[y,x,0])*(255 - alpha) ) / 255)
        g = int((float(im[y,x,1]) * alpha + float(beach[y,x,1])*(255 - alpha) ) / 255)
        r = int((float(im[y,x,2]) * alpha + float(beach[y,x,2])*(255 - alpha) ) / 255)
        if b > 255 or g > 255 or r > 255:
            print('y[%d] x[%d]'%(y, x),   im[y,x])
        canvas_image[y, x] = (b, g, r)
        
plt.figure(figsize=(15,15))

ax0 = plt.subplot(grid[0])
ax1 = plt.subplot(grid[1])
ax0.imshow(cv2.cvtColor(canvas_image, cv2.COLOR_BGR2RGB))
ax1.imshow(cv2.cvtColor(beach, cv2.COLOR_BGR2RGB))
plt.show()

This is the result image.

The interface is made a little softer than when using a mask. And, unlike the use of a mask, the red circle is overlapped with the background.

Wrapping up

There are two ways to change the background of an image. An image using chroma key can easily change the background, but it has a disadvantage in that it is necessary to establish a shooting environment from the image production stage. Many images, such as selfies using a cell phone, have a background. Rembg does a good job of removing this background in most cases.

We also learned about two ways to change the background. The most commonly used method is to use a mask. However, since this method cuts the interface, a sense of heterogeneity can be felt at the interface. Using the alpha value for pixel unit operation has the advantage of creating a smooth interface, but it also has the disadvantage of taking a lot of time.