This is a thesis done on hand gesture recognition to create a program that can control a device (PC) using hand gestures. This is based on the source code by Kazuhito Takahashi, the original readme for his project is found below. original repo.
The major changes to this project can be found in the Skelett branch. In the main branch the original alpha prototype can be found in app.py. This code has not been edited very much. In the beta branch Skelett, all changes and additions made by us have been marked as "By FJ" or something similar.
The alpha prototype works as a proof of concept and can control the device using static gestures. They can be found below. The beta prototype can be controlled by gestures that are done by moving the hand. This is done by storing 16 frames in a dequeue that is then trained using mainly the LSTM layer in the neural network.
- open hand - do nothing
- closed fist - left down click
- index finger - mouse control
- ok sign - marking and mouse control
- not ok sign - release the marking
- backhand - right click down
- rock n roll - scroll up
- C - 'Ctrl + C' to copy
- V/peace sign - 'Ctrl + V' to paste
- rotate - 'alt + tab' to switch tabs
- reverse pinch - zoom in
- ringing telephone - zoom out
- back - 'alt + back' to go back
- hand talking - mute
- wave - open the web (linkedin)
- basketball toss - volume up
- basketball bounce - volume down
- finger gun - 'esc' or turn off the program
Big thanks to Kazuhito Takahashi for providing this program under the apache license.
Estimate hand pose using MediaPipe (Python version).
This is a sample
program that recognizes hand signs and finger gestures with a simple MLP using the detected key points.
❗ ️This is English Translated version of the original repo. All Content is translated to english along with comments and notebooks ❗
This repository contains the following contents.
- Sample program
- Hand sign recognition model(TFLite)
- Finger gesture recognition model(TFLite)
- Learning data for hand sign recognition and notebook for learning
- Learning data for finger gesture recognition and notebook for learning
- mediapipe 0.8.1
- OpenCV 3.4.2 or Later
- Tensorflow 2.3.0 or Later
tf-nightly 2.5.0.dev or later (Only when creating a TFLite for an LSTM model) - scikit-learn 0.23.2 or Later (Only if you want to display the confusion matrix)
- matplotlib 3.3.2 or Later (Only if you want to display the confusion matrix)
Here's how to run the demo using your webcam.
python app.pyThe following options can be specified when running the demo.
- --device
Specifying the camera device number (Default：0) - --width
Width at the time of camera capture (Default：960) - --height
Height at the time of camera capture (Default：540) - --use_static_image_mode
Whether to use static_image_mode option for MediaPipe inference (Default：Unspecified) - --min_detection_confidence
Detection confidence threshold (Default：0.5) - --min_tracking_confidence
Tracking confidence threshold (Default：0.5)
│ app.py
│ keypoint_classification.ipynb
│ point_history_classification.ipynb
│
├─model
│ ├─keypoint_classifier
│ │ │ keypoint.csv
│ │ │ keypoint_classifier.hdf5
│ │ │ keypoint_classifier.py
│ │ │ keypoint_classifier.tflite
│ │ └─ keypoint_classifier_label.csv
│ │
│ └─point_history_classifier
│ │ point_history.csv
│ │ point_history_classifier.hdf5
│ │ point_history_classifier.py
│ │ point_history_classifier.tflite
│ └─ point_history_classifier_label.csv
│
└─utils
└─cvfpscalc.py
This is a sample program for inference.
In addition, learning data (key points) for hand sign recognition,
You can also collect training data (index finger coordinate history) for finger gesture recognition.
This is a model training script for hand sign recognition.
This is a model training script for finger gesture recognition.
This directory stores files related to hand sign recognition.
The following files are stored.
- Training data(keypoint.csv)
- Trained model(keypoint_classifier.tflite)
- Label data(keypoint_classifier_label.csv)
- Inference module(keypoint_classifier.py)
This directory stores files related to finger gesture recognition.
The following files are stored.
- Training data(point_history.csv)
- Trained model(point_history_classifier.tflite)
- Label data(point_history_classifier_label.csv)
- Inference module(point_history_classifier.py)
This is a module for FPS measurement.
Hand sign recognition and finger gesture recognition can add and change training data and retrain the model.
Press "k" to enter the mode to save key points（displayed as 「MODE:Logging Key Point」）
If you press "0" to "9", the key points will be added to "model/keypoint_classifier/keypoint.csv" as shown below.
1st column: Pressed number (used as class ID), 2nd and subsequent columns: Key point coordinates
The key point coordinates are the ones that have undergone the following preprocessing up to ④.
In the initial state, three types of learning data are included: open hand (class ID: 0), close hand (class ID: 1), and pointing (class ID: 2).
If necessary, add 3 or later, or delete the existing data of csv to prepare the training data.
Open "keypoint_classification.ipynb" in Jupyter Notebook and execute from top to bottom.
To change the number of training data classes, change the value of "NUM_CLASSES = 3"
and modify the label of "model/keypoint_classifier/keypoint_classifier_label.csv" as appropriate.
The image of the model prepared in "keypoint_classification.ipynb" is as follows.
Press "h" to enter the mode to save the history of fingertip coordinates (displayed as "MODE:Logging Point History").
If you press "0" to "9", the key points will be added to "model/point_history_classifier/point_history.csv" as shown below.
1st column: Pressed number (used as class ID), 2nd and subsequent columns: Coordinate history
The key point coordinates are the ones that have undergone the following preprocessing up to ④.
In the initial state, 4 types of learning data are included: stationary (class ID: 0), clockwise (class ID: 1), counterclockwise (class ID: 2), and moving (class ID: 4).
If necessary, add 5 or later, or delete the existing data of csv to prepare the training data.
Open "point_history_classification.ipynb" in Jupyter Notebook and execute from top to bottom.
To change the number of training data classes, change the value of "NUM_CLASSES = 4" and
modify the label of "model/point_history_classifier/point_history_classifier_label.csv" as appropriate.
The image of the model prepared in "point_history_classification.ipynb" is as follows.
The model using "LSTM" is as follows.
Please change "use_lstm = False" to "True" when using (tf-nightly required (as of 2020/12/16))
Kazuhito Takahashi(https://twitter.com/KzhtTkhs)
Nikita Kiselov(https://github.com/kinivi)
hand-gesture-recognition-using-mediapipe is under Apache v2 license.