DADS – Stutter Detection App

DADS (Detecting and Analyzing Disfluencies in Speech) is an AI-powered application that detects stuttering in speech recordings. Users can record or upload audio, analyze it for five stutter classes, and visualize the results — via either a PyQt5 desktop app or a FastAPI web app.

---

Problem Statement

Stuttering is a speech disorder affecting millions of people worldwide. Identifying and categorizing stuttering events is important for diagnosis and for tracking the effectiveness of speech therapy — yet manual annotation is slow, subjective, and requires trained clinicians.

DADS addresses this by automatically detecting disfluencies in speech audio. The app analyzes an audio clip and reports whether each of the five most common stutter types is present, along with a confidence score, so speech-language pathologists, researchers, and individuals can get objective, repeatable feedback.

---

Approach

1. Audio preprocessing — Recordings are resampled to 16 kHz mono and split into 3-second chunks. Each chunk is either zero-padded or truncated to exactly 48,000 samples.
2. Feature extraction — A mel spectrogram is computed per chunk (n_fft=1024, hop_length=512), converted to dB scale, and z-score normalized.
3. Multi-model classification — One binary CNN is trained per stutter class. Each model independently predicts the probability that its class is present in a chunk. Per-model feature parameters (e.g. n_mels) are parsed from the model filename.
4. Aggregation — Results across chunks are aggregated per class using the maximum probability as the confidence score. A class is flagged as detected when its max probability exceeds a detection threshold (0.4 in the web app, 0.45 in the desktop app).
5. Visualization — A spectrogram (or waveform) of the recording is rendered, with playback controls (play/pause, ±5s seek, scrubber) and a results panel showing each class's confidence, detected/not-detected status, and detected-chunk count.

---

Models

Five binary CNN classifiers (one per stutter class), each trained independently:

Class	Model file	n_fft	hop_length	n_mels	epochs	Test accuracy
Prolongation	prolongation_model_1024_512_128_40.pth	1024	512	128	40	0.76
Block	block_model_1024_512_128_40.pth	1024	512	128	40	0.68
Sound Repetition	soundrep_model_1024_512_256_40.pth	1024	512	256	40	0.82
Word Repetition	wordrep_model_1024_512_64_40.pth	1024	512	64	40	0.81
Interjection	interjection_model_1024_512_128_40.pth	1024	512	128	40	0.71

Architecture (identical to the training notebook):

• 3 convolutional blocks: Conv2d(1→32) + BatchNorm + ReLU + MaxPool, Conv2d(32→64) + BatchNorm + ReLU + MaxPool, Conv2d(64→128) + BatchNorm + ReLU + AdaptiveAvgPool((1,1))
• Fully connected head: Flatten → Linear(128,64) → ReLU → Dropout(0.3) → Linear(64,1)
• Loss: BCEWithLogitsLoss · Optimizer: Adam (lr=1e-3) · Batch size: 32
• Classify as detected when sigmoid(output) > threshold

The 5 production weights live in Model/models/copy/ and are committed to the repo so both the desktop and web apps (including the Docker image) can load them without retraining.

---

Dataset

The models are trained on SEP-28k plus FluencyBank, two public stuttering-annotation datasets:

• SEP-28k — ~28,000 3-second clips from 385 publicly available stuttering YouTube episodes, annotated with 9 speech-disorder labels.
• FluencyBank — ~4,100 clips from 33 TalkBank interview recordings with the same annotation scheme.

In total ~32,000 clips across the two datasets are used. Labels used for training are the five stutter classes above (Prolongation, Block, SoundRep, WordRep, Interjection).

Clips are extracted from episode audio as 3-second WAV files (16 kHz mono) using the annotation timestamps. The dataset is only needed to re-train the models — it is not required to run the app.

Setup tooling:

• setup_dataset.sh / setup_dataset.py — download episode audio and slice clips from SEP-28k_labels.csv / fluencybank_labels.csv timestamps (requires ffmpeg).
• SEP-28k can also be obtained from Kaggle.

---

Tech Stack

Machine Learning

• PyTorch (torch, torchaudio, torchvision)
• Librosa (audio loading + mel spectrograms)
• NumPy, pandas, scikit-learn, matplotlib

Desktop App

• PyQt5 (PyQt5.QtMultimedia for recording/playback, PyMuPDF for the reading-passage viewer)

Web App

• FastAPI + uvicorn (backend, SSE streaming analysis)
• Jinja2 templating, vanilla JS/CSS frontend
• wavesurfer.js (waveform player) · pdf.js (passage viewer)
• Docker (python:3.12-slim + libsndfile1 + ffmpeg), deployable to Railway/Render via Dockerfile / render.yaml

Tooling

• Python 3.12, ruff (lint + format), GitHub Actions CI

---

Project Structure

DADS/
│-- App/                  # PyQt5 desktop application
│-- backend/              # FastAPI web backend (SSE analysis, spectrogram, UI)
│-- shared/               # Shared inference code (web backend detector)
│-- Model/                # Training notebook & saved models (Model/models/copy)
│-- dataset/              # SEP-28k + FluencyBank clips and labels
│-- setup_dataset.sh|py   # Dataset download/clip-extraction tooling
│-- Dockerfile            # Container image for the web app
│-- render.yaml           # Render deployment config
└-- requirements.txt      # Desktop app dependencies

---

How to Use the App

Option A — Web App (upload/record in the browser)

Run locally:

uvicorn backend.main:app --host 0.0.0.0 --port 8000

Then open http://localhost:8000.

Or deploy with Docker:

docker build -t dads .
docker run -p 8000:8000 dads

Usage:

1. Click Start Recording to capture a clip from your microphone, or upload a WAV/MP3/WebM/OGG/FLAC file.
2. Optionally open the Reading Passage PDF (with zoom/page controls) to read while recording.
3. Click Analyze — progress streams via SSE as each 3s chunk is scored.
4. View the spectrogram/waveform, listen with the player (play/pause, ±5s, scrubber), and read the per-class confidence + detection results.
5. Download Report to export the results.

Option B — Desktop App (PyQt5)

Step 1: Clone the Repository

git clone <repo-link>

Step 2: Create a Virtual Environment

• macOS / Linux:

python3 -m venv .venv

• Windows:

python -m venv .venv

Step 3: Activate the Environment and Install Dependencies

• macOS / Linux:

source .venv/bin/activate

• Windows:

.venv\Scripts\activate

pip install -r requirements.txt

Step 4: Run the App

• macOS / Linux:

python3 App/run_app.py

• Windows:

python App/run_app.py

Usage:

1. On the main screen, record a clip (Start/Stop, saved to Recordings/) or upload a WAV file (16-bit mono). Choose a reading passage from the PDF list (or upload your own) to read while recording.
2. Click Go to Analysis — detection runs in the background with a live progress indicator.
3. Toggle between Spectrogram / Waveform, play back the audio (play/pause, ±5s seek, scrubber), and read the per-class results (confidence %, detected ✓, chunk count).
4. Export Report to save a timestamped summary sorted by confidence to reports/.

---

Commit Guidelines

Follow this convention:

[TYPE] Commit message

Common types:

• [FIX] – Bug fixes
• [ADD] – New features
• [DOCS] – Documentation changes
• [MNT] – Code refactoring & Maintenance
• [TEST] – Tests additions or fixes