Introduction

In this session, you will learn how to use machine vision to make new controls for a game. This worksheet will guide you through the different steps to make a basic hand gesture controller. But you are free to spice up things and explore other types of interactions.

Objectives

• To understand the concept of particle systems and implement it

• To integrate a gesture/object recognition system into a game as a controller

Preliminary steps

To complete this workshop, you will need a computer with a webcam. You will also need to know Python and have installed mediapipe as well as pygame.

We recommend you to setup a virtual environment for this:

python3 -m venv mp_env && source mp_env/bin/activate

You can then install mediapipe

pip install mediapipe
pip install pygame

• Part 1 Getting Familiar with OpenCV and MediaPipe
• 1.2 Recognizing and displaying the Hands
• 1.3 Using your Webcam with VideoCapture
• Part 2 Simple Hand Controller using Mediapipe
• Part 3 The Hand-Controlled Pong Game