Creating TetrisClock using OpenCV #1

Do you know TetrisClock?
TetrisClock is a WiFi clock made of falling tetris blocks. Runs on an ESP32 with an RGB LED Matrix.

img
<TetrisClock on the RGB LED Matrix by Brian Lough>


I'm a big fan of RGB LED matrix and I wrote many posts about RGB LED Matrix in my blog.
I like to display the screen using OpenCV on the Raspberry Pi to the RGB LED Matrix.
However, the TetrisClock shown in the figure above works on the Arduino family of ESP32 MCUs. I also posted a post implementing TetrisClock on ESP32 at https://iot-for-maker.blogspot.com/2020/04/led-9-rgb-led-matrix-drive-with-esp-32.html. But I wanted to implement this beautiful clock in Raspberry Pi, so I googled hard, but couldn't find any good examples. Eventually, I decided to analyze the code written in C language and implement it in Python and OpenCV.
In the picture above, it consists of four large numbers indicating hours, minutes and small letters indicating morning(AM) and afternoon(PM).


Original code analysis

Number Analysis

You can find out by looking at https://github.com/toblum/TetrisAnimation/blob/master/src/TetrisNumbers.h for Tetris, which makes large numbers first.

First, this header file declares the following structure.

// Type that describes how a brick is falling down
typedef struct
{
  int blocktype;  // Number of the block type
  int color; // Color of the brick
  int x_pos;      // x-position (starting from the left number staring point) where the brick should be placed
  int y_stop;     // y-position (1-16, where 16 is the last line of the matrix) where the brick should stop falling
  int num_rot;    // Number of 90-degree (clockwise) rotations a brick is turned from the standard position
} fall_instr_let;

The first member of this structure specifies the shape of the Tetris block. This value is from 0 to 6, and the figure corresponding to the number is indicated in the figure below.




Next member is color. The uint16 variable is used as a variable to store color. In general, this is strange compared to using 24-bit variables to store RGB colors or 32-bit variables including alpha channels. The reason for this can be found in the Adafruit-GFX-Library graphics library, which is widely used in Arduino MCUs.
The Adafruit-GFX-Library graphics library stores and converts 32-bit RGB values ​​to R (5 bits), G (6 bits), and B (5 bits).
Storing in this way makes it difficult to store accurate color values. But instead, the memory space occupied by the variable that stores the color is halved. This may be useful for MCUs with insufficient memory space, but it is not needed at all on systems using Linux operating systems such as the Raspberry Pi. So, just store it in a 32-bit variable.

The original maker(Tobias Blum) of TetrisClock uses simple colors like this.

uint16_t myRED = display.color565(255, 0, 0);
uint16_t myGREEN = display.color565(0, 255, 0);
uint16_t myBLUE = display.color565(48, 73, 255);
uint16_t myWHITE = display.color565(255, 255, 255);
uint16_t myYELLOW = display.color565(255, 255, 0);
uint16_t myCYAN = display.color565(0, 255, 255);
uint16_t myMAGENTA = display.color565(255, 0, 255);
uint16_t myORANGE = display.color565(255, 96, 0);
uint16_t myBLACK = display.color565(0, 0, 0);

uint16 myCOLORS[8] = {myRED, myGREEN, myBLUE, myWHITE, myYELLOW, myCYAN, myMAGENTA, myBLACK};


But Brian's code uses a slightly different format. However, the color values ​​of the two codes are exactly the same. Therefore, referencing the original author's code for porting Raspberry Pi is much easier to understand and more accurate.

void TetrisMatrixDraw::intialiseColors(){
    this->tetrisRED = 0xF800;
    this->tetrisGREEN = 0x07E0;
    this->tetrisBLUE = 0x325F;
    this->tetrisWHITE = 0xFFFF;
    this->tetrisYELLOW = 0xFFE0;
    this->tetrisCYAN = 0x07FF;
    this->tetrisMAGENTA = 0xF81F;
    this->tetrisORANGE = 0xFB00;
    this->tetrisBLACK = 0x0000;
    
    this->tetrisColors[0] = this->tetrisRED; 
    this->tetrisColors[1] = this->tetrisGREEN; 
    this->tetrisColors[2] = this->tetrisBLUE; 
    this->tetrisColors[3] = this->tetrisWHITE; 
    this->tetrisColors[4] = this->tetrisYELLOW; 
    this->tetrisColors[5] = this->tetrisCYAN; 
    this->tetrisColors[6] = this->tetrisMAGENTA;
    this->tetrisColors[7] = this->tetrisORANGE; 
    this->tetrisColors[8] = this->tetrisBLACK;
}

The next value is the X coordinate of the Tetris block. This value is between 0 and 5.
The next value is the Y coordinate of the Tetris block. This value is between 1 and 16. The origin of these values ​​can be found in the RGB LED Matrix. This is because the early TetrisClock developers used an 8x16 RGB LED matrix. After that, while using the RGB LED Matrix of size 32X64, the multiple of these values ​​is applied.

The last member is the number of rotations needed to complete the number shape.


Now let's see the numbers. This code snippet is a Tetris implementation of the number 0.

// *********************************************************************
// Number 0
// *********************************************************************
#define SIZE_NUM_0 13
fall_instr num_0[SIZE_NUM_0] = {
    {2, myCYAN, 4, 16, 0},
    {4, myORANGE, 2, 16, 1},
    {3, myYELLOW, 0, 16, 1},
    {6, myMAGENTA, 1, 16, 1},
    {5, myGREEN, 4, 14, 0},
    {6, myMAGENTA, 0, 13, 3},
    {5, myGREEN, 4, 12, 0},
    {5, myGREEN, 0, 11, 0},
    {6, myMAGENTA, 4, 10, 1},
    {6, myMAGENTA, 0, 9, 1},
    {5, myGREEN, 1, 8, 1},
    {2, myCYAN, 3, 8, 3}};

If you interpret it in order, the line{2, myCYAN, 4, 16, 0}, means that Cyan's 2nd shape has 4 X coordinates, 16 Y coordinates, and no rotation. If you interpret the second line, it is the orange (4, 2) position of figure 4 and requires 1 rotation. And so on..... If you analyze all the numbers in this way, you get the following shape. Since the black background is used, the black color of the shape was used (45,45,45) rather than (0,0,0).




Python Porting

Python Code

Let's use Testris to represent Tetris numbers in Python. Since the length of the code is quite large, please download the entire code from github and take a look. Here, we will only load the important parts.

import numpy as np
import cv2
import time

num_0 = (
    (2, 5, 4, 16, 0),
    (4, 7, 2, 16, 1),
    (3, 4, 0, 16, 1),
    (6, 6, 1, 16, 1),
    (5, 1, 4, 14, 0),
    (6, 6, 0, 13, 3),
    (5, 1, 4, 12, 0),
    (5, 1, 0, 11, 0),
    (6, 6, 4, 10, 1),
    (6, 6, 0, 9, 1),
    (5, 1, 1, 8, 1),
    (2, 5, 3, 8, 3))


num_1 = (
    (2, 5, 4, 16, 0),
    (3, 4, 4, 15, 1),
    (3, 4, 5, 13, 3),
    (2, 5, 4, 11, 2),
    (0, 0, 4, 8, 0))

'''skip
'''

num_9 = (
    (0, 0, 0, 16, 0),
    (3, 4, 2, 16, 0),
    (1, 2, 2, 15, 3),
    (1, 2, 4, 15, 2),
    (3, 4, 1, 12, 2),
    (3, 4, 5, 12, 3),
    (5, 1, 0, 12, 0),
    (1, 2, 2, 11, 3),
    (5, 1, 4, 9, 0),
    (6, 6, 0, 10, 1),
    (5, 1, 0, 8, 1),
    (6, 6, 2, 8, 2))

'''BGR'''
myRED = (0, 0, 255)
myGREEN = (0, 255, 0)
myBLUE = (255, 73, 48)
myWHITE = (255, 255, 255)
myYELLOW = (0, 255, 255)
myCYAN = (255, 255, 0)
myMAGENTA = (255, 0, 255)
myORANGE = (0, 96, 255)
myBLACK = (45, 45, 45)

myCOLORS = (myRED, myGREEN, myBLUE, myWHITE, myYELLOW, myCYAN, myMAGENTA, myBLACK)
mynums = (num_0, num_1, num_2, num_3, num_4, num_5, num_6, num_7, num_8, num_9)
scale = 20
x_shift = 2
y_shift = 1

'''
Don't make shape, just paint the pixel.
rotate:  %4 => 0 ~ 3 
'''
def draw_shape(canvas, x, y, color, shape, rotate, y_pos):
    tcanvas = canvas.copy()
    rot = rotate % 4
    ret = False
    if shape == 0:  #rantangle
        if rot == 0 or rot == 1 or rot == 2 or rot == 3:
            for i in range(x, x + 2*scale):
                for j in range(y + 0*scale, y + 2*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
    elif shape == 1: 
        if rot == 3:
            for i in range(x + 2*scale, x + 3*scale):
                for j in range(y + 1*scale, y + 2*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
            for i in range(x , x + 3*scale):
                for j in range(y + 2*scale, y + 3*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
        elif rot == 0:
            for i in range(x , x + 1*scale):
                for j in range(y, y + 3*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
            for i in range(x + 1*scale, x + 2*scale):
                for j in range(y + 2*scale, y + 3*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True


     ''' skip '''


    elif shape == 6:
        if rot == 0:
            for i in range(x + 1*scale, x + 2*scale):
                for j in range(y, y + 1*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
            for i in range(x, x + 3*scale):
                for j in range(y + 1*scale, y + 2*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
        elif rot == 1:
            for i in range(x, x + 1*scale):
                for j in range(y, y + 3*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
            for i in range(x + 1*scale, x + 2*scale):
                for j in range(y + 1*scale, y + 2*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
        elif rot == 2:
            for i in range(x, x + 3*scale):
                for j in range(y, y + 1*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
            for i in range(x + 1*scale, x + 2*scale):
                for j in range(y + 1*scale, y + 2*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
        elif rot == 3:
            for i in range(x, x + 1*scale):
                for j in range(y + 1*scale, y + 2*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True
            for i in range(x + 1*scale, x + 2*scale):
                for j in range(y, y + 3*scale):
                    tcanvas[j, i] = color
                    if j == (y_pos - 1):
                        ret = True

    return tcanvas, ret

def animate_number(canvas, n, x_shift,  y_shift):
    tcanvas = canvas.copy()
    num = mynums[n]
    for i in num:
        print(i)
        shape = i[0]
        color = myCOLORS[i[1]]
        x_pos = i[2] + x_shift
        y_pos = i[3] - y_shift
        rotation = i[4]
        y = 0
        rot = 0

        while True:
            mycanvas, ret = draw_shape(tcanvas, x_pos * scale, y * scale, color, shape, rot, y_pos * scale)
            # cv2.imshow("%d"%(n), mycanvas)
            cv2.imshow("Tetris", mycanvas)
            cv2.waitKey(50)
            y += 1
            if rot != rotation:
                rot += 1
            if ret == True:
                break    
        tcanvas = mycanvas.copy()
        cv2.waitKey(100)
    return mycanvas

    cv2.waitKey(0)

def make_canvas(h, w, color):
    # canvas = np.zeros([(h + 1) * scale,(w + 1) * scale,3], dtype=np.uint8)
    canvas = np.zeros([(h ) * scale,(w ) * scale,3], dtype=np.uint8)
    canvas.fill(color)
    return canvas

def test_shape():
    for shape in range(0, 7):
        for rotate in range(0, 4):
            print(shape)
            tcanvas, _ = draw_shape(canvas, 0, 0, myBLUE, shape, rotate, y_shift)
            cv2.imshow("tetris", tcanvas)
            cv2.waitKey(0)

def set_scale(s):
    global scale
    scale = s


if __name__ == '__main__':
    set_scale(20)
    canvas = make_canvas(16 * int(scale / 20 + 0.5), 32 * int(scale / 4 + 0.5), 0)
    # test_shape()
    tcanv = canvas
    shift = x_shift
    for i in range(0, 10):
        tcanv = animate_number(tcanv, i,  shift , y_shift)
        shift += 11
    cv2.waitKey(0)
    cv2.destroyAllWindows()
<tetris.py>

If you run the code, you can see this window.

python3 tetris.py




Wrapping up

By analyzing the Tetris Clock C code running on the existing ESP32 MCU, numbers from 0 to 9 were made to work with Python using OpenCV. In the above example, the Tetris effect was applied one number at a time, but in the next part, I will try to implement non-numeric characters in Tetris.
You can download the source code at https://github.com/raspberry-pi-maker/OpenCV





















댓글

댓글 쓰기

이 블로그의 인기 게시물

Image Processing #7 - OpenCV Text

이미지
You can download the source codes here( https://github.com/raspberry-pi-maker/OpenCV ) Basic text output using putText putText function Draws a text string. Python: cv2. putText ( img, text, org, fontFace, fontScale, color [ , thickness [ , lineType [ , bottomLeftOrigin ] ] ] ) → None Python: cv. PutText ( img, text, org, font, color ) → None Parameters: img – Image. text – Text string to be drawn. org – Bottom-left corner of the text string in the image. font – CvFont structure initialized using InitFont() . fontFace – Font type. One of FONT_HERSHEY_SIMPLEX , FONT_HERSHEY_PLAIN , FONT_HERSHEY_DUPLEX , FONT_HERSHEY_COMPLEX , FONT_HERSHEY_TRIPLEX , FONT_HERSHEY_COMPLEX_SMALL , FONT_HERSHEY_SCRIPT_SIMPLEX , or FONT_HERSHEY_SCRIPT_COMPLEX , where each of the font ID’s can be combined with FONT_ITALIC to get the slanted letters. fontScale – Font scale factor that is multiplied by the font-specific base size. color – Text color. thickness – Th
자세한 내용 보기

Playing YouTube videos using OpenCV

이미지
So far, we have dealt with many examples of playing video files such as mp4, avi, or frames captured using an input device such as a webcam. In this post, we will learn how to play YouTube videos using OpenCV. You can download the source codes here( https://github.com/raspberry-pi-maker/OpenCV ) prerequisite You only need to install the papy, youtube-dl packages. pip3 install papy pip3 install --upgrade youtube-dl Pafy Objects and Stream Objects Pafy objects relate to videos hosted on YouTube. They hold metadata such as title, viewcount, author and video ID. Stream objects relate to individual streams of a YouTube video. They hold stream-specific data such as resolution, bitrate and url. Each Pafy object contains multiple stream objects. If you use papy, you can get information about youtube videos as follows. import pafy url = 'https://www.youtube.com/watch?v=M-UKXCUI0rE' video = pafy . new(url) #The author of the video (str) print(video . author
자세한 내용 보기

OpenCV Installation - Rasbian Buster, Jessie, DietPi Buster

이미지
In this post, we will install OpenCV on Raspberry Pi . Raspberry Pie doesn't have a GPU, so you don't need to set up acceleration with CUDA. Therefore, it is easiest to install a package that has already been created. But old version of Rasbian(Jessie) doesn't have python package. So you have to build the source code to install the OpenCV. And I'm going to install OpenCV 4.1.1 to DietPi. DietPi is a lightweight version of Debian. Prepare Rasbian Buster headless sd card image Prepare the image using Etcher. I used 2019-09-26-raspbian-buster-lite.zip image. Do not boot the Raspberry Pi immediately after creating the image, but create a size 0 ssh file on the / boot partition. The reason for this file is to make it a headless image. This allows remote ssh operations without having to connect a monitor to the Raspberry Pi. However, to get an IP address using dhcp, the network cable must be connected. <Create ssh file in the boot partition> Then run the
자세한 내용 보기