HandFrame AI

A gesture-driven augmented-reality workspace that combines real-time hand tracking, temporal gesture recognition, interactive state management, procedural visual effects, and asynchronous AI image generation.

HandFrame AI transforms a standard webcam video feed into an interactive spatial computing environment. By tracking hand positions with MediaPipe, the system enables users to construct a virtual "floating frame" in physical space using two-hand framing gestures. Once framed, the sub-region can be manipulated in real time using intuitive physical gestures rather than traditional mouse and keyboard inputs.

The workspace operates across multiple interaction modes, allowing users to apply instant procedural OpenCV stylizations or trigger cloud-hosted generative AI diffusion models. Hand interaction is decoupled from application state through a multi-stage pipeline that interprets temporal hand dynamics (pinch lifecycles, directional swipes, open-palm pauses) while maintaining continuous 30+ FPS camera throughput.

By separating gesture recognition, intent resolution, and asynchronous inference into distinct architectural layers, HandFrame AI demonstrates how computer vision, spatial AR interfaces, and modern generative AI workflows can be unified into a responsive, cohesive product.

Author: M. Abdullah

Features

Real-Time Multi-Hand Tracking: Low-latency landmark extraction powered by MediaPipe Hands with adaptive 1-Euro temporal point smoothing to eliminate high-frequency jitter.
Temporal Gesture Recognition: Dedicated gesture engine evaluating multi-frame velocity, displacement, and duration rather than isolated single-frame snapshots.
L-Frame Workspace Activation: Dynamic two-hand framing gesture that anchors an interactive quadrilateral in 3D camera space.
Pinch State Lifecycle: Dual-mode pinch recognition distinguishing between quick release taps (effect selection) and sustained holds (AI generation triggers).
Multi-Frame Swipe Navigation: Horizontal hand motion tracking with velocity and linearity thresholds for seamless workspace mode transitions.
Open-Palm Pause & Resume: Five-finger extension detection to pause or resume spatial interactions without resetting workspace state.
Perspective-Aware AR Frame: Warps stylized content and AI outputs onto the user's hand-defined quadrilateral with holographic corner brackets and dynamic glow borders.
17 Procedural Visual Effects: Instant, real-time OpenCV filters (Cyberpunk, Anime, Watercolor, Pop Art, Thermal, Van Gogh, and more) with facial feature preservation masks.
Asynchronous AI Diffusion: Background worker thread dispatching image-to-image diffusion jobs to cloud models (fal.ai FLUX.2) without blocking the camera capture loop.
Deterministic Local Fallback: Automatic failover to local procedural filters if network drops, API credentials are missing, or requests time out.
Measurable Gesture Confidence: Mathematical confidence metrics normalized to $[0.0, 1.0]$ derived from hand geometry and velocity margins.
State Debouncing & Cooldowns: Enforced cooldown timers and history resets to prevent false or duplicate multi-frame triggers.
Spatial Telemetry HUD: Real-time on-screen display showing measured FPS, loop execution time in milliseconds, active mode, gesture meters, and action toasts.
Keyboard Fallback Controls: Full keyboard navigation support for accessibility, debugging, and testing.
Secure Environment Configuration: Centralized configuration using .env files with secret masking in logs and zero hardcoded credentials.

How It Works

HandFrame AI is architected around a strict four-layer pipeline that separates raw computer vision tracking from interaction logic and rendering:

Camera Stream (30 FPS)
  ↓
HandTracker (vision/hand_tracker.py)
  ↓ [MediaPipe landmark extraction & 1-Euro adaptive temporal smoothing]
GestureEngine (vision/gesture_engine.py)
  ↓ [Classifies poses & motions -> emits structured GestureEvents with confidence]
InteractionEngine (core/interaction_engine.py)
  ↓ [Decouples intent: maps GestureEvent + WorkspaceState -> InteractionActions]
Workspace (core/workspace.py)
  ↓ [Manages CAMERA, LIVE_EFFECTS, AI_ART, pause state, and AI lifecycle]
┌─────────────────────────────────┬─────────────────────────────────┐
│ Live Effects & AR Rendering     │ Asynchronous AI Inference       │
│ (effects/ & ui/ar_frame.py)     │ (inference/ & fal_backend.py)   │
└─────────────────────────────────┴─────────────────────────────────┘

HandTracker (Layer 1): Ingests BGR video frames, runs MediaPipe hand detection, normalizes landmark coordinates, and applies 1-Euro temporal smoothing filters to reduce point jitter while maintaining responsiveness.
GestureEngine (Layer 2): Evaluates geometric ratios, finger extensions, pinch durations, and multi-frame motion velocities to identify discrete gestures. Emits structured GestureEvent objects with normalized confidence scores.
InteractionEngine (Layer 3): Acts as the intent resolution layer. It consumes GestureEvent objects alongside the current Workspace state to produce decoupled InteractionAction commands (e.g., NEXT_WORKSPACE_MODE, START_AI_GENERATION).
Workspace & Rendering (Layer 4 & 5): The Workspace state machine updates its internal operational mode and dispatches tasks to either the real-time procedural EffectManager or the asynchronous AsyncInferenceEngine. The resulting image is perspective-warped onto the floating AR quad via FloatingPlane.

Gesture System

The gesture recognition engine evaluates continuous temporal information across recent frames rather than making isolated per-frame decisions.

Gesture	Physical Motion / Pose	Resolved Action	Description
`L_FRAME`	Both hands facing camera forming frame anchors	`ACTIVATE_WORKSPACE`	Pinned floating AR plane between fingertip anchors
`SWIPE_RIGHT`	Rapid horizontal hand sweep to right ($\ge 200\text{ px/s}$)	`NEXT_WORKSPACE_MODE`	Advances mode: `CAMERA` $\to$ `LIVE_EFFECTS` $\to$ `AI_ART`
`SWIPE_LEFT`	Rapid horizontal hand sweep to left ($\ge 200\text{ px/s}$)	`PREVIOUS_WORKSPACE_MODE`	Reverses mode switching: `AI_ART` $\to$ `LIVE_EFFECTS` $\to$ `CAMERA`
`SHORT_PINCH`	Right hand thumb + index tap ($< 0.55\text{ s}$)	`NEXT_EFFECT`	Advances to the next visual style in the 17-effect catalog
`LONG_PINCH`	Right hand thumb + index held ($\ge 0.55\text{ s}$)	`START_AI_GENERATION`	Captures frame and starts non-blocking background diffusion
`OPEN_PALM`	5 extended fingers facing camera	`TOGGLE_PAUSE`	Freezes/resumes interaction without destroying state

Temporal Tracking & Heuristics

Pinch Lifecycle: Tracks pinch contact across three explicit states: START $\to$ HOLD (if sustained $\ge 0.55\text{s}$, firing exactly one AI generation request) $\to$ RELEASE (if released $< 0.55\text{s}$, cycling visual styles).
Swipe Velocity & Linearity: A sliding time window ($\Delta t \in [0.08\text{s}, 0.35\text{s}]$) calculates horizontal displacement ($\Delta x \ge 60\text{px}$) and velocity ($v_x \ge 200\text{px/s}$) while enforcing horizontal dominance ($|\Delta x| > 1.35|\Delta y|$) and a $0.45\text{s}$ cooldown to prevent accidental multi-triggers.
Adaptive Smoothing: 1-Euro adaptive temporal filters dynamically tune their cutoff frequencies based on movement speed, providing steady hand anchoring when stationary and zero perceptible lag during rapid movement.

Workspace Modes

HandFrame AI supports three distinct operational modes:

1. `CAMERA` Mode

A clean passthrough camera stream with active holographic framing quad lines and corner brackets. Used for aligning the floating AR frame over a physical subject before applying stylization.

2. `LIVE_EFFECTS` Mode

An instant AR filter canvas that warps procedural OpenCV effects directly onto the framed quad in real time at full camera frame rates (30+ FPS). Users can quickly cycle through 17 curated style presets.

3. `AI_ART` Mode

An asynchronous generative AI canvas. Sustaining a long pinch captures the framed quadrilateral and dispatches an image-to-image diffusion request to cloud models. The camera feed and UI continue running without interruption; once complete, the AI-generated artwork freezes warped into the AR plane.

AI Architecture

The AI subsystem is decoupled through an abstract backend interface, isolating cloud networking and image encoding from the core application loop:

Application Coordinator (app.py)
      ↓
InferenceBackend (inference/base.py)
      ↓
FalCloudBackend (inference/fal_backend.py)
      ↓ (Cloud API: fal-ai/flux-2/klein/4b/edit)
Cloud Output

Asynchronous Execution & Failover

Non-Blocking Background Worker: Requests are submitted to a thread-backed queue managed by AsyncInferenceEngine. The main video capture loop polls for results non-blockingly, keeping the webcam stream fluid.
Automatic Local Fallback: If FAL_KEY is not configured, network connectivity drops, or the cloud API returns an error, the worker automatically routes the frame through the local procedural EffectManager filter pipeline. The application tags the result as used_fallback=True and continues execution without crashing.
Duplicate Request Protection: If a generation job is already in progress, subsequent long-pinch gestures are ignored to prevent request flooding and unnecessary cloud API consumption.

Tech Stack

Technology	Version / Requirement	Role in Project
Python	`3.11.9`	Primary runtime environment
OpenCV (`opencv-python`)	`~4.9.0`	Video capture, perspective warping, image processing, HUD rendering
MediaPipe	`~0.10.14`	Real-time multi-hand tracking and 21-landmark 3D coordinate extraction
NumPy	`~1.26.0`	Vectorized geometric math, matrix transformations, array operations
Pillow (`PIL`)	`~10.3.0`	Image serialization and encoding for AI backend transport
fal-client	`~0.5.0`	Client SDK for cloud FLUX.2 image-to-image diffusion models
python-dotenv	`~1.0.0`	Environment variable management and credential loading
pytest	`~8.0.0`	Automated testing framework for unit and regression suites

Project Structure

HandFrame-AI/
├── app.py                      # Main entry point coordinating the 4-layer pipeline
├── core/                       # Core workspace state, interaction engine, and config
│   ├── config.py               # Centralized configuration dataclass (.env + CLI)
│   ├── fps.py                  # Exponentially smoothed FPS meter
│   ├── interaction_engine.py   # Intent resolution engine mapping gestures to actions
│   ├── logging_config.py       # Standardized logging setup
│   ├── state.py                # Legacy state container (backward compatibility)
│   └── workspace.py            # Workspace manager & mode state machine
├── vision/                     # Computer vision & gesture recognition
│   ├── gesture_engine.py       # Multi-frame gesture engine & confidence calculation
│   ├── hand_tracker.py         # MediaPipe multi-hand tracking & landmark extraction
│   ├── perspective.py          # Perspective transforms, quad validation & floating plane
│   └── smoothing.py            # 1-Euro adaptive temporal point filter
├── inference/                  # Asynchronous AI image generation
│   ├── base.py                 # Abstract InferenceBackend interface
│   ├── engine.py               # Threaded AsyncInferenceEngine with local fallback
│   └── fal_backend.py          # fal.ai cloud FLUX.2 inference backend implementation
├── effects/                    # Stylization catalog & procedural filters
│   ├── filters.py              # Fast procedural OpenCV fallback filters & face masks
│   ├── manager.py              # EffectManager for enumeration and filter application
│   └── styles.py               # 17 artistic style presets & prompts
├── ui/                         # Spatial AR rendering & heads-up display
│   ├── ar_frame.py             # Spatial holographic corner brackets, glow borders, badges
│   └── overlay.py              # Workspace HUD, telemetry bars & loading spinners
├── tests/                      # Automated unit & integration test suite (55 tests)
│   ├── test_config.py          # Configuration & secret masking tests
│   ├── test_effects.py         # 17 fallback filter execution tests
│   ├── test_effects_manager.py # EffectManager style queries & cycling tests
│   ├── test_gesture_engine.py  # L-Frame, pinch, palm, fist, swipe & confidence tests
│   ├── test_hand_geometry.py   # Quad ordering & pinch geometry tests
│   ├── test_inference_backend.py # Backend abstraction, mock execution & fallback tests
│   ├── test_interaction_engine.py # Action resolution per workspace mode tests
│   ├── test_perspective.py     # Perspective warping & quad validation tests
│   ├── test_phase3_features.py # Spatial AR frame, HUD telemetry & duplicate protection tests
│   ├── test_smoothing.py       # 1-Euro filter jitter reduction tests
│   ├── test_validation_pass.py # End-to-end multi-frame interactive scenario tests
│   └── test_workspace.py       # Workspace mode & state transition tests
├── docs/                       # Architecture diagrams, state machines & ADRs
│   └── architecture.md
├── .env.example                # Environment configuration template
├── .gitignore                  # Git ignore rules protecting credentials
├── requirements.txt            # Python dependency manifest
├── LICENSE                     # MIT License
└── README.md

Installation & Setup

1. Prerequisites

Python: 3.11.9 is required for dependency compatibility with MediaPipe and OpenCV wheels.
Webcam: Any standard integrated or external USB camera.

2. Clone the Repository

git clone https://github.com/muhammad-abdullah-nova-dev/HandFrame-AI.git
cd HandFrame-AI

3. Create and Activate Virtual Environment

Windows (PowerShell):

py -3.11 -m venv .venv311
.\.venv311\Scripts\Activate.ps1

macOS / Linux:

python3.11 -m venv .venv311
source .venv311/bin/activate

4. Install Dependencies

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Environment Variables

Copy the .env.example file to create a local .env:

cp .env.example .env

Configure your local environment variables in .env:

# Optional: fal.ai API key for FLUX.2 cloud diffusion
# Obtain a key at https://fal.ai/dashboard/keys
FAL_KEY=your_api_key_here

# Camera & Capture
CAMERA_INDEX=0
FRAME_WIDTH=1280
FRAME_HEIGHT=720
FPS_TARGET=30

# Gesture Thresholds
HOLD_THRESHOLD=0.55
SWIPE_MIN_DIST=70.0
SWIPE_MIN_VELOCITY=260.0
SWIPE_COOLDOWN=0.45

# Logging
LOG_LEVEL=INFO

Security Note: The .env file is excluded from Git tracking in .gitignore. Never commit real API keys to version control. If FAL_KEY is not provided, HandFrame AI automatically runs in local offline mode using procedural OpenCV filters.

Running the Application

Ensure your Python 3.11 virtual environment is active, then run:

python app.py

Command-Line Arguments

Optional command-line overrides are supported:

# Select camera device index 1 with 1280x720 resolution
python app.py --camera 1 --width 1280 --height 720

# Disable selfie mirror flip
python app.py --no-mirror

# Set debug logging
python app.py --log-level DEBUG

Controls

Primary Gesture Interface

Gesture	Action	Visual Feedback
L-Frame (Both Hands)	Position and activate floating AR frame	Holographic corner brackets appear
Swipe Right	Switch to next workspace mode	HUD toast: `MODE: [NEXT_MODE]`
Swipe Left	Switch to previous workspace mode	HUD toast: `MODE: [PREV_MODE]`
Quick Pinch (< 0.55s)	Cycle to next visual style	HUD toast: `STYLE: [STYLE_NAME]`
Hold Pinch (≥ 0.55s)	Trigger AI image generation	Dual-arc rotating loading overlay
Open Palm	Toggle pause / resume interaction	Frame turns lavender with `[PAUSED]` badge

Keyboard Fallback Controls

Keyboard shortcuts exist for development, accessibility, and demonstration fallbacks:

Key	Action
`]`	Next visual style
`[`	Previous visual style
`M` or `Tab`	Next workspace mode (`CAMERA` $\to$ `LIVE_EFFECTS` $\to$ `AI_ART`)
`P`	Toggle workspace pause state
`R`	Reset workspace to default live state
`Q`	Quit application and release camera resources

Performance

The following measurements reflect observed development metrics on standard hardware (Intel i7 / Apple Silicon, 1280x720 webcam capture) rather than universal benchmarks:

Webcam Capture & Processing Loop: $\sim 30\text{--}35\text{ FPS}$ ($28\text{--}32\text{ ms}$ total frame latency).
1-Euro Temporal Point Smoothing: $< 0.15\text{ ms}$ overhead per frame for 2 hands (42 landmarks) and 4 quad corners.
Procedural OpenCV Filters: $\sim 1.2\text{--}4.5\text{ ms}$ per frame depending on filter complexity (color mapping vs. bilateral filtering).
Cloud Diffusion Latency: $\sim 1.8\text{--}3.2\text{ s}$ network round-trip for fal.ai FLUX.2 inference (runs asynchronously without freezing the 30 FPS camera feed).

Testing

The repository contains an automated test suite with 55 unit and integration tests covering geometry, gesture state machines, interaction resolution, asynchronous threading, and failover paths.

Run the test suite with pytest:

.\.venv311\Scripts\python.exe -m pytest tests/ -v

============================= test session starts =============================
platform win32 -- Python 3.11.9, pytest-9.1.1, pluggy-1.6.0 -- .venv311\Scripts\python.exe
collected 55 items

tests/test_config.py ......................... PASSED [ 5%]
tests/test_effects.py ........................ PASSED [ 9%]
tests/test_effects_manager.py ................ PASSED [ 14%]
tests/test_gesture_engine.py ................. PASSED [ 29%]
tests/test_hand_geometry.py .................. PASSED [ 34%]
tests/test_inference_backend.py .............. PASSED [ 38%]
tests/test_interaction_engine.py .............. PASSED [ 49%]
tests/test_perspective.py .................... PASSED [ 58%]
tests/test_phase3_features.py ................ PASSED [ 70%]
tests/test_smoothing.py ...................... PASSED [ 78%]
tests/test_validation_pass.py ................ PASSED [ 89%]
tests/test_workspace.py ...................... PASSED [100%]

============================= 55 passed in 4.38s ==============================

Test Suite Coverage

Configuration & Secrets: Verifies CLI flag parsing, default fallback values, and string masking of sensitive API keys.
Filter Pipelines: Validates that all 17 procedural fallback filters execute without exceptions and produce valid BGR arrays.
Gesture Engine: Tests L-frame detection, short vs. long pinch lifecycles, open-palm stability, fist detection, and swipe velocity calculations.
Interaction Engine: Tests intent mapping per workspace mode and verifies gesture blocking during pause states.
Inference Backends: Tests mock cloud backend execution, worker thread queue polling, and automatic fallback failover upon exceptions.
Perspective Geometry: Validates non-degenerate quad heuristics, inverse warping, and floating plane alpha-compositing.
Smoothing Filters: Tests 1-Euro point filter initialization, jitter reduction, and state reset behavior.
End-to-End Validation Pass: Tests 6 continuous multi-frame integration scenarios (L-frame activation, multi-step swipes, pinch releases, 5-second continuous holds, pause toggling, and backend failure resilience).

Security

Zero Hardcoded Secrets: No API keys or tokens are stored in the source code.
Environment Isolation: Secrets are loaded strictly through environment variables and .env files.
Git Protection: .env, .env.*, virtual environments, caches, and log files are gitignored.
Safe Representation: AppConfig.masked_repr() masks API keys in logs and debug representations (SET (masked)).

Architecture Decisions

Why separate `GestureEngine` and `InteractionEngine`?

GestureEngine answers "What physical gesture occurred?" by analyzing raw landmarks, measuring duration, and computing geometric confidence. InteractionEngine answers "What should the application do about it?" based on the active workspace state. This separation allows gesture thresholds or tracking backends to be modified without altering application business logic.

Why asynchronous inference?

Cloud diffusion models require $1.5\text{s} - 3.5\text{s}$ to execute. Running inference synchronously on the main thread would freeze video capture, causing severe frame drops. A daemon worker thread processes inference in the background while the main loop continues rendering the camera and HUD at 30+ FPS.

Why backend abstraction (`InferenceBackend`)?

By routing generation requests through the InferenceBackend protocol, workspace logic remains decoupled from specific cloud APIs. New backends (such as local ONNX, TensorRT, or alternate cloud APIs) can be added without modifying the main application coordinator.

Why local procedural fallback?

To ensure the application is reliable and demo-ready regardless of network conditions or API account status. When a cloud call fails or no API key is present, the worker immediately applies local OpenCV filters with zero downtime.

Why temporal gesture recognition over single-frame detection?

Single-frame heuristics are vulnerable to landmark jitter, brief tracking dropouts, and false triggers. HandFrame AI tracks multi-frame motion histories, evaluates duration thresholds, requires multi-frame pose stability, and enforces cooldowns to guarantee intentional user interaction.

Documentation

For an in-depth technical analysis of the runtime pipeline, mathematical heuristics, state machine diagrams, and Architecture Decision Records (ADRs), see:

docs/architecture.md

Known Limitations

Hand Overlap: When both hands cross over each other in front of the camera, MediaPipe hand chirality ("Left" vs. "Right") can temporarily invert.
Low-Light Environments: Dim lighting reduces camera contrast, which can lower MediaPipe landmark detection confidence and affect finger extension ratios.
Webcam Auto-Exposure: Webcams with aggressive auto-exposure adjustments may cause momentary brightness fluctuations across procedural color filters.
Cloud Latency: Generative AI diffusion turnaround depends on network latency and cloud provider queue times.

Roadmap

Two-Hand Multi-Touch Scaling: Pinch-to-zoom and rotational transformations on the floating AR quadrilateral plane.
Local ONNX / TensorRT AI Backends: Lightweight local diffusion model integrations (e.g., SD-Turbo) for zero-latency offline generation.
Spatial UI Buttons: Interactive floating virtual buttons rendered in camera space for direct fingertip tapping.
Session Video Recording: Gesture-triggered MP4 recording and animated GIF exporting of stylized creations.
Multi-Person Hand Association: Face-anchored hand tracking to isolate interactions to the primary user in crowded camera views.

Author

M. Abdullah

HandFrame AI is a computer-vision and AI interaction project focused on exploring gesture-driven augmented-reality interfaces and spatial computing workflows.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Name		Name	Last commit message	Last commit date
core		core
docs		docs
effects		effects
inference		inference
tests		tests
ui		ui
vision		vision
.env.example		.env.example
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
app.py		app.py
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation