An AI-Powered Assistive Tool for Visual Impairment
SecondSight is an iOS application that leverages state-of-the-art artificial intelligence to enhance safety and spatial awareness for individuals with visual impairment. Using real-time hazard detection and scene description, SecondSight aims to prevent accidents and improve independence for users with complete blindness or low vision.
demo.mov
SecondSight addresses critical safety gaps in existing assistive technologies for the visually impaired. While traditional tools like canes and guide dogs are invaluable, they can sometimes miss small hazardous objects at close proximity. SecondSight complements these methods by providing:
- Automated Hazard Detection: Continuous real-time detection of small obstacles (rocks, bottles, potholes, poles, stairs) within 2-3 strides distance
- Scene Description: On-demand AI-generated descriptions of the surrounding environment
- Accessible Feedback: Dual-channel alerts via haptic feedback and speech descriptions
- Enhance accident prevention and promote safety
- Increase awareness of surroundings for greater peace of mind
- Alleviate economic burden on the welfare system
- Enhance assistive tool accessibility across social, economic, and geographic boundaries
- Promote independence and enhance social wellbeing
When the camera is pointed toward the ground (covering approximately 2-3 strides distance), the app continuously detects 5 common hazardous objects:
- Rocks
- Bottles
- Potholes
- Poles
- Stairs
Detection Response:
- Haptic feedback on iPhone (and Apple Watch if paired)
- Speech alerts describing the detected hazard
- Visual bounding boxes with labels (for sighted caregivers)
- Response time: < 300ms for real-time accident prevention
Users can request detailed environmental descriptions at any time:
- One-finger tap anywhere on the screen to trigger scene description
- AI-generated descriptions using Apple's FastVLM (on-device)
- Provides context about hazards and general surroundings
- Fully offline - no internet connection required
Designed for accessibility with minimal interaction. Inclusive design cater for physical challenges such as tremor:
- Zero interaction to start: Detection begins automatically on launch
- One-finger tap: Request scene description
- Swipe down: Pause detection
- Swipe up: Resume detection
- Long press: Camera display ON/OFF
Hands-free usability - the Apple watch can worn on wirst or clipped to the white cane for detection with single touch control using a thumb:
- Full standalone companion app for wearable-only operation
- Receive haptic feedback and notifications directly on the wrist
- Allows for more discreet and convenient alerts
- Fully integrated with iPhone via Bluetooth pairing
- Automatic camera angle detection using gyroscope
- < 30° from ground: Activates hazard detection mode
- > 30° from ground: Switches to scene description mode
- Seamless transitions without manual intervention
SecondSight uses a component-based architecture with three key components:
-
User Interface (iOS App)
- Swift/SwiftUI implementation
- Single-screen design for simplicity
- High-contrast, large fonts for accessibility
- Gesture-based controls
-
Model Inferencing
- On-device: YOLOv11n CoreML for hazard detection
- On-device: Apple FastVLM for scene description
- Optimized for mobile device resource constraints
- No internet connection required for core functionality
-
Machine Learning Operations (MLOps)
- Automated pipelines using ClearML
- Two pipelines: Hazard Detection and Scene Description
- Continuous Integration/Continuous Deployment (CI/CD)
- Model training on Google Cloud remote agents
- FastAPI model deployment and serving
The system integrates:
- Video Processing: Real-time camera feed handling with gyroscope-based angle detection
- Hazard Detection: YOLOv11n inference on-device (camera angle < 30°)
- Scene Description: Apple FastVLM inference on-device (camera angle > 30°)
- User Feedback: Haptic, speech, and visual alerts
- Apple Watch Integration: Full companion app for standalone operation
- Device: iPhone 14 Pro or later (iPhone 15 Pro Max recommended for testing)
- Processor: A16 Bionic chip or later
- RAM: Minimum 6GB
- OS: iOS 17 or later
- Optional: Apple Watch for companion app functionality
-
Development:
- Xcode IDE
- Swift programming language
- Apple Developer Account
- iTunes Connect Account
-
ML Pipeline:
- Python 3.x
- ClearML for MLOps
- PyTorch
- FastAPI for remote inference
- CoreML for on-device deployment
- Camera access (required)
- Motion & orientation access (required for gyroscope-based mode switching)
- Ensure your device meets the hardware requirements
- Obtain the SecondSight app from your Apple Developer portal
- Install the app on your iPhone 14 Pro or later
- Grant camera permissions when prompted
- (Optional) Install the companion app on Apple Watch
-
Clone the repository:
git clone https://github.com/your-repo/SecondSight.git
-
Open the project in Xcode
-
Configure your Apple Developer Account and signing certificates
-
Download the CoreML model and add it to the project
-
Build and deploy to your device:
- Select your target device
- Build and run (⌘R)
-
For Apple Watch companion:
- Select the Watch target
- Build and deploy to paired Apple Watch
Note: Follow Apple's official development guidelines for detailed account setup, app signing, and deployment procedures.
- Launch the app: A splash screen will appear for 3 seconds showing the SecondSight logo
- Detection starts automatically: The app immediately begins hazard detection
- Point camera downward: Aim the camera toward the ground at an angle that covers 2-3 strides ahead
- Automatic Mode Switching:
- Point camera down (< 30°): Hazard detection mode activates
- Point camera up (> 30°): Scene description mode activates
- Hazard Detected: You'll receive haptic feedback and a voice alert (e.g., "Rock ahead")
- Request Scene Description: Two-finger tap anywhere on screen (works offline)
- Pause Detection: Swipe down on screen
- Resume Detection: Swipe up on screen
- Apple Watch: Receive all alerts and notifications on your wrist
- Use in outdoor daytime conditions with good lighting
- Point camera at ground level covering 2-3 strides distance
- Ensure phone is held steadily for optimal detection
- Use as a complementary tool with existing assistive devices (cane, guide dog, etc.)
- Do NOT use in low-light conditions or extremely hazardous environments
SecondSight has been optimized and tested to meet the following performance targets:
| Metric | Target | Achieved | Notes |
|---|---|---|---|
| Hazard Detection Recall | ≥ 75% | 75.1% | Macro recall on test set (IoU 0.5) |
| Hazard Detection Precision (mAP50) | ≥ 80% | 81.7% | Improved from 73.3% after tuning |
| Scene Description Quality (CIDEr) | ≥ 0.50 | 0.47 | Baseline; stretch goal of 0.50 |
| End-to-End Latency | ≤ 300ms | ~35ms on-device | Critical for accident prevention |
| Real-Time Throughput | ≥ 20 FPS | 21 FPS | On iPhone 14 Pro (100 layers, 6.3 GFLOPs) |
| Battery Efficiency | ≤ 10% per hour | TBD | Profiling pending |
| Memory Usage | ≤ 200MB | Within limit | During continuous operation |
- YOLOv11n (Hazard Detection): Custom-trained on specialized dataset for downward-angle camera captures at 2-3 stride distances
- ViT-GPT2 Student (Scene Description): Knowledge-distilled from LLaVA 1.5-7B for lightweight inference (earlier version)
- Framework: YOLOv11n converted to CoreML
- Deployment: On-device (embedded in iOS app)
- Classes: 5 hazard objects (rocks, bottles, potholes, poles, stairs)
- Input: Continuous video stream from camera
- Output: Bounding boxes with labels and confidence scores
- Performance: 21 FPS on iPhone 14 Pro
- Framework: Apple FastVLM (Vision Language Model)
- Deployment: On-device inference (fully offline)
- Input: Single still image from camera
- Output: Natural language description of scene
- Performance: Slight delay on first load, faster on subsequent requests
- Advantage: No internet connection required, maintains user privacy
- Dataset upload and analysis
- Data preprocessing
- Model training (YOLOv11n)
- Hyperparameter optimization
- Model evaluation
- Model publishing to registry
- Deployment to GitHub (CI/CD trigger)
- PyTorch → CoreML conversion
- Integration into iOS app
- Dataset preparation with caption generation
- Knowledge distillation training
- Model fine-tuning
- Evaluation (BLEU/CIDEr metrics)
- Model publishing
- Deployment to FastAPI server
- Model loading at application startup
- Frontend: Swift, SwiftUI
- ML Frameworks: PyTorch, CoreML, Vision, Apple FastVLM
- MLOps: ClearML, GitHub Actions
- Cloud: Google Cloud (for training agents)
- Sensors: CoreMotion (gyroscope for angle detection)
- Haptics: CoreHaptics framework
- Speech: AVFoundation (Text-to-Speech)
- Watch: WatchKit, WatchConnectivity
- Device Compatibility: iOS devices only (iPhone 14 Pro+); Android not supported
- Lighting Conditions: Best performance in outdoor daylight; not optimized for low-light or night conditions
- Weather: May not perform well in rain, fog, or other adverse weather conditions
- Hazard Classes: Limited to 5 predefined hazard types
- Battery Consumption: Continuous camera and detection may drain battery quickly
- Detection Distance: Effective range limited to 2-3 strides ahead
- Not TGA/FDA approved as a medical device
- No emergency call functionality
- No navigation or path guidance
- Should NOT replace certified assistive tools or guide dogs
- Requires visually-abled person for initial setup and installation
- Dataset availability constraints may limit accuracy improvements
- Training time requires significant compute resources
- False negative rate target: < 5% for critical hazards
- False positive rate: < 10% of total detections
- Offline Scene Description: Apple FastVLM for on-device operation (no internet required)
- Apple Watch Standalone: Full companion app integrated for wearable-only operation
- Gyroscope Integration: Automatic camera angle-based mode switching (< 30° = detection, > 30° = scene description)
- Cloud-based Detection: Move detection to cloud to reduce device resource consumption
- Microservices Architecture: Better resource allocation for model inferencing
- Android Support: Expand to Android devices for wider accessibility
- Voice Control: Hands-free interaction with voice commands
- Auto-Model Updates: Cloud-based model updates without app reinstall
- Battery Optimization: Improved power management for extended usage
- Low-Light Enhancement: Night vision mode with grayscale optimization
- Expanded Hazard Classes: More object types based on user feedback
- External Integration: Support for Braille displays and other assistive hardware
- Visual Question Answering (VQA): Interactive queries about the environment
- Navigation Assistance: Path planning and obstacle avoidance guidance
- Emergency Services: One-tap call for help functionality
According to the World Health Organization (WHO, 2023):
- 2.2 billion people worldwide have vision impairment or blindness
- In Australia: 453,000-840,000 people are blind or visually impaired
- By 2030: Expected to exceed 1.04 million in Australia alone
Economic Impact:
- Global productivity loss: ~$411 billion USD annually
- Australia: $27.6 billion AUD annually ($46,950 per person with vision loss aged 40+)
Safety Challenges:
- Higher rates of accidents requiring hospitalization (stairs, door collisions, burns, medication errors)
- Increased risk of serious life events (explosions, assaults, life-threatening injuries)
- Greater likelihood of falls, fractures, and early nursing home entry
Current Support Options:
- Canes: May miss small hazardous objects
- Guide Dogs: Expensive, require extensive training, limited availability
- Personal Assistants/Carers: Not available 24/7, require scheduling
- Existing Apps (Seeing AI, Envision AI, Be My Eyes): No specialized hazard detection for close-proximity obstacles
SecondSight's Unique Value:
- Hazard-specific detection for small objects at 2-3 stride distance
- 24/7 availability without need for human assistance
- Affordable and accessible (uses existing smartphone hardware)
- Complementary tool that works alongside traditional methods
- Improve safety and reduce accident-related medical costs
- Enhance independence and reduce caregiver burden
- Build self-confidence for environmental interaction
- Improve psychological wellbeing and reduce anxiety
- Promote equality by reducing discrimination based on disability
- Energy Efficiency: Lightweight models (YOLOv11n, distilled VLM) minimize GPU requirements
- Hardware Reuse: Leverages existing smartphones (no new hardware production)
- Resource Optimization: Distributed pipeline with CPU/GPU task separation
- Future Integration: Designed for third-party hardware compatibility to avoid closed ecosystems
- Custom-trained YOLOv11n CoreML hazard detection (5 classes)
- Dual-channel alerts (haptic + speech)
- On-demand scene description via two-finger tap (Apple FastVLM - fully offline)
- Automatic mode switching via gyroscope (< 30° detection, > 30° description)
- iPhone 14 Pro / 15 Pro (A16+) support running iOS 17+
- Custom ML pipelines (YOLO + distilled VLM)
- Full Apple Watch companion app with standalone operation
- Navigation / path guidance
- Emergency call-for-help
- External hardware/software integration
- Model performance recall > 80%
- Android / other edge devices
- Replacing certified assistive tools
- Visual Question Answering (VQA)
- TGA approval as medical device
This project was developed as part of academic research in AI-powered assistive technologies. Special thanks to:
- Vision 2020 Australia for background research and statistics
- WHO for global vision impairment data
- The visual impairment community for inspiring this work
- Ahmetovic, D., et al. (2020). ReCog: Supporting Blind People in Recognizing Personal Objects. CHI 2020.
- Brunes, A., & Heir, T. (2021). Serious Life Events in People with Visual Impairment. Int. J. Environ. Res. Public Health, 18(21).
- Cleveland Clinic (2025). Low Vision. Cleveland Clinic Health Library.
- Lundälv, J., & Thodelius, C. (2021). Risk of Injury Events in Patients With Visual Impairments. J. Visual Impairment & Blindness, 115(5).
- Stangl, A. J., et al. (2018). BrowseWithMe. ASSETS '18.
- Vision 2020 Australia (2022). 2022-23 Pre-Budget Submission.
- WHO (2023). Blindness and vision impairment: Facts.