Spatial Computing for macOS

A camera-based eye cursor and hand gesture demo built with MediaPipe, OpenCV, and PyAutoGUI.

Features

• Eye-controlled cursor with multi-point polynomial calibration
• Pinch to click, vertical pinch-hold to scroll, and horizontal pinch-hold to switch macOS desktops
• Double-pinch to pause or resume control
• Inverted vertical scroll and desktop swipe controls
• Guided calibration with progress, multi-sample capture, and an accuracy score
• Visual gesture feedback and a safe demo mode that never controls the computer

Install

Python 3.12 is recommended because it is supported by the pinned MediaPipe release. Newer MediaPipe releases may not include the legacy mp.solutions tracking API used by this project.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

For live control, grant the terminal or Python application access in:

• System Settings > Privacy & Security > Camera
• System Settings > Privacy & Security > Accessibility

Run

Start with safe demo mode:

python spatial-computing.py --demo

Enable live cursor and gesture control:

python spatial-computing.py

Press Q to quit, R or Command+Shift+R to recalibrate, and move the mouse to a screen corner to pause control with PyAutoGUI's emergency stop. The application remains open so control can be resumed after moving away.

Tests

The core calibration and gesture logic has no camera dependency:

python -m unittest discover -s tests -v

Personal ML Eye Tracker

The default tracker maps absolute iris positions to the screen and is sensitive to head movement. The optional personal gaze model instead uses cropped images of both eyes plus normalized face pose features.

Install the ML dependencies:

python -m pip install -r requirements-ml.txt

Collect at least three sessions under different lighting and head positions. One default session takes about four minutes and stores only local data:

python collect_gaze_data.py

For a useful personal model, collect 10-15 sessions across several days. Keep the webcam fixed relative to the screen. Training always reserves complete sessions for validation instead of mixing adjacent video frames:

python gaze_model.py --epochs 40

Training reads the screen dimensions from the sessions, prints median and 90th-percentile validation error, and exports the best TorchScript model to personal_gaze.pt.

Compare the exported model against the old iris-position baseline on the held out session:

python evaluate_gaze_model.py

Run the application with the trained model:

python spatial-computing.py --gaze-model personal_gaze.pt

The ML backend uses adaptive temporal smoothing and stops moving the cursor when prediction confidence is low. Without --gaze-model, the polynomial iris tracker remains available as a fallback.