by Nipun, Alesia, Aditya
ConnectAble is a personal Augmentative and Alternative Communication (AAC) system that helps non-verbal users communicate through a symbol-based board. The system learns how a specific user communicates over time and predicts what they want to say next using their phrase history, location, and time-of-day context.
User presses symbol buttons → builds a sentence → speaks it aloud
↓
phrase logged to SQLite
↓
nightly training (2 AM)
┌────────────────┴────────────────┐
Bigram map ChromaDB embeddings
(vocab_store.json) (chroma_db/)
When the user starts typing, GET /suggestions runs a 3-layer cascade:
Layer 1 — Bigram next-word lookup (instant, local dict)
Layer 2 — ChromaDB semantic similarity (vector search over past phrases)
Layer 3 — Ollama LLM fallback (phi3, used when < 5 phrases in DB)
The frontend is a React SPA with a grid of 66 AAC symbol buttons across 5 categories (food, feelings, actions, places, people). As the user builds a sentence, the top bar shows AI-powered phrase suggestions that update in real time. Tapping a suggestion appends it and logs it as accepted for future learning.
An agent tab lets users send natural language messages that are classified into intents (make a call, order food, set a reminder, or general chat) and dispatched to tool handlers.
├── backend/ FastAPI prediction API + agent (Python)
└── frontend/ Symbol board UI (React + TypeScript + Vite)
- Python 3.11+
- Node.js 18+
- Ollama (
brew install ollamaon macOS)
cd backend
# Install dependencies
pip install -r requirements.txt
# Generate the database encryption key (run once per machine, never commit)
python3 -c "import secrets; print(secrets.token_hex(32))" > aac.key && chmod 0600 aac.key
# Seed starter phrases and build the prediction models (run once)
python -m data.seed_phrases
python nightly_train.py
# Start Ollama in a separate terminal (needed for LLM fallback and agent)
ollama serve
ollama pull phi3
# Start the API on port 8000
uvicorn main:app --reload --port 8000API docs available at http://localhost:8000/docs.
cd frontend
# Install dependencies
npm install
# Start dev server on port 8080
npm run devOpen http://localhost:8080. The UI works standalone with fallback data if the backend is not running.
backend/user_config.json— set user locations, default location, and TTS mode ("offline"or"elevenlabs")backend/aac.key— AES-256 encryption key for SQLite (generated above, gitignored)backend/.env— addELEVENLABS_API_KEY=...if using ElevenLabs TTS
| Method | Path | Description |
|---|---|---|
GET |
/health |
Health check + phrase count |
POST |
/log_phrase |
Record a phrase the user spoke |
GET |
/suggestions |
3-layer word/phrase predictions |
POST |
/llm_suggest |
Direct Ollama LLM suggestion (bypasses vector store) |
POST |
/speak |
Text-to-speech (offline pyttsx3 or ElevenLabs streaming) |
GET |
/analytics/heatmap |
Top 50 words by frequency across logged phrases |
GET |
/analytics/summary |
Total phrases, acceptance rate, top phrases, top locations |
POST |
/autocomplete/accepted |
Log that a suggestion was accepted |
POST |
/autocomplete/dismissed |
Log that a suggestion was dismissed |
POST |
/agent |
Intent classification + tool dispatch |
GET |
/reminders |
List all reminders |
DELETE |
/reminders/{id} |
Delete a reminder by ID |
The LLM is the fallback, not the primary path — and that inversion is the point.
The obvious design for a predictive AAC board is "send context to an LLM, show
what it returns." We built the opposite: a bigram lookup runs first, ChromaDB
semantic search second, and the LLM only fires when the user has fewer than 5
logged phrases (VECTOR_MIN_PHRASES in backend/routers/suggestions.py).
Two constraints drove that:
- Latency budget. An AAC user is mid-sentence. A local phi3 call costs hundreds of milliseconds to seconds; a bigram lookup is a dict hit and a ChromaDB query over a few thousand phrases is single-digit milliseconds. A suggestion that arrives after the user has already tapped the next symbol is worse than no suggestion.
- The user's own phrasing beats a general model's. A model that has read the internet will suggest fluent, generic English. Someone's actual communication history is idiosyncratic, repetitive, and theirs. Ranking their own past phrases first produces suggestions they'll actually accept.
So the LLM solves exactly one problem — cold start, when there is no history to rank — and gets out of the way as soon as there is.
What this cost us: three prediction paths to maintain instead of one, and a
source field threaded through the response so we can tell which layer answered.
Debugging a bad suggestion means first working out which layer produced it. We
accepted that because the alternative was a system that felt sluggish exactly
when it mattered.
Everything runs locally. SQLite (AES-256 via SQLCipher), ChromaDB, and Ollama all run on-device, and offline TTS is the default. Communication history for a non-verbal user is about as sensitive as personal data gets — shipping it to a cloud API was not a tradeoff we were willing to make. ElevenLabs TTS is opt-in and off by default.
Nightly retraining instead of online learning. The bigram map and vector
store rebuild at 2 AM (backend/nightly_train.py) rather than updating per
phrase. Rebuilding is idempotent and easy to reason about; incremental index
updates would have been faster to reflect new phrases but far easier to corrupt.
For a system where a corrupted index means someone can't communicate, the boring
option won.
| Layer | Stack |
|---|---|
| Frontend | React 18, TypeScript, Vite, Tailwind CSS, shadcn/ui, Recharts |
| Backend | FastAPI, SQLite (AES-256 via SQLCipher), ChromaDB, sentence-transformers (all-MiniLM-L6-v2) |
| LLM | Ollama + phi3 (local, offline) |
| TTS | pyttsx3 / macOS say (offline) or ElevenLabs (streaming audio) |
Nipun Saini, Alesia, Aditya — built at AISC Hackathon 2026 (1st Place).