██████╗██╗██╗ ██╗██╗ ██╗ █████╗ ████████╗ ██████╗██╗ ██╗
██╔════╝██║██║ ██║██║ ██║██╔══██╗╚══██╔══╝██╔════╝██║ ██║
██║ ██║╚██╗ ██╔╝██║ █╗ ██║███████║ ██║ ██║ ███████║
██║ ██║ ╚████╔╝ ██║███╗██║██╔══██║ ██║ ██║ ██╔══██║
╚██████╗██║ ╚██╔╝ ╚███╔███╔╝██║ ██║ ██║ ╚██████╗██║ ██║
╚═════╝╚═╝ ╚═╝ ╚══╝╚══╝ ╚═╝ ╚═╝ ╚═╝ ╚═════╝╚═╝ ╚═╝
◈ CIVIC INTELLIGENCE. ANOMALY DETECTION. REAL-TIME TRUTH. ◈
Transform fragmented public records into normalized, anomaly-aware, queryable intelligence.
📡 Live Status · 🛣️ Roadmap · 📐 Architecture · 🔒 Security · 🐛 Issues
CIVWATCH is a full-stack civic intelligence platform purpose-built to surface patterns in public records that institutions don't want surfaced. It ingests agendas, minutes, contracts, budgets, and vote records — then normalizes, scores, and visualizes anomalies in real time.
Built for residents, journalists, investigators, and civic analysts who need machine-grade clarity on government data — not PDFs.
"Make civic data as actionable as a security feed."
┌─────────────────────────────────────────────────────────────────┐
│ CIVWATCH STACK │
├──────────────────┬──────────────────┬───────────────────────────┤
│ React/Vite │ Node/Express │ Python FastAPI │
│ Dashboard :4000 │ API Server :3000 │ ML Engine :5000 │
│ AnomalyUI │ /alerts │ DBSCAN /detect │
│ Charts (WIP) │ /analytics │ z-score /score/anomaly │
│ Auth Views │ /anomalies │ TextBlob → DistilBERT │
│ │ /ingest │ /analyze/sentiment │
├──────────────────┴──────────┬────────┴───────────────────────────┤
│ PostgreSQL │ Redis (cache layer — in progress) │
│ Data Store │ 30s TTL on alert routes │
├─────────────────────────────┴────────────────────────────────────┤
│ Docker Compose · CI/CD · 6 Security Workflows · Electron Shell │
└──────────────────────────────────────────────────────────────────┘
Port map: Frontend :4000 → Backend API :3000 → ML Engine :5000 → PostgreSQL :5432 → Redis :6379
Full design → docs/architecture.md
Last verified: May 18, 2026 · Full per-component matrix → STATUS.md
| Component | State | Notes |
|---|---|---|
| 🟢 Backend API | LIVE |
/status /health /alerts /analytics |
| 🟢 JWT Auth Middleware | LIVE |
requireAuth on all protected routes |
| 🟢 ML Engine (FastAPI :5000) | LIVE |
Health check passing |
🟢 DBSCAN /detect |
LIVE |
StandardScaler applied, outliers → -1 |
| 🟢 Anomaly Scoring v2 | LIVE |
z-score + threshold flags |
🟢 Sentiment /analyze/sentiment |
LIVE |
TextBlob MVP |
🟢 Batch Scoring /analyze/batch |
LIVE |
|
| 🟢 PostgreSQL | LIVE |
Pool, migrations, routes querying |
| 🟢 Docker Compose | LIVE |
All healthchecks resolved |
| 🟢 CI/CD Pipeline | LIVE |
Jest + pytest + 6 security workflows |
| 🟢 Electron Shell | LIVE |
IPC bridge complete |
🟡 /api/anomalies |
PARTIAL |
Returns empty array — ML bind pending |
🟡 /api/ingest |
PARTIAL |
Accepts POST — no storage/forwarding yet |
🟡 dataAnalyzer.ts |
BOTTLENECK |
Type errors blocking vector output (#3) |
| 🟡 Frontend AnomalyDashboard | PARTIAL |
Nav wired — no live data binding |
| 🟡 Redis Cache | WIRED |
Client declared — routes not hooked (#5) |
| 🔴 DBSCAN Scheduler | GAP |
No trigger from anomaly_scores table |
🔴 ML /predict Endpoint |
NOT STARTED |
Pydantic schema + /api/anomalies bind (#9) |
| 🔴 WebSocket / Real-Time | NOT STARTED |
socket.io + pg LISTEN/NOTIFY (#12) |
| 🔴 Rate Limiting | NOT STARTED |
(#7) |
🟢 Verified · 🟡 Partial / In Progress · 🔴 Not Started / Blocked
Requirements: Docker + Docker Compose · Node.js 20+ · Python 3.10+
# 1. Clone
git clone https://github.com/POWDER-RANGER/CIVWATCH.git && cd CIVWATCH
# 2. Configure environment
cp .env.example .env
# → Set: DB credentials · Redis URL · JWT secret
# 3. Spin everything up
docker-compose up
# or: npm run docker:up
# 4. Verify
curl http://localhost:3000/api/status # {"status":"ok"}
curl http://localhost:5000/health # {"status":"ok"}
open http://localhost:4000 # React dashboardnpm run setup # Install all deps across workspaces
npm run dev # All services concurrently
# Or individually:
npm run dev:backend # Node.js API → :3000
npm run dev:frontend # React/Vite UI → :4000
npm run dev:ml # FastAPI ML engine → :5000See SETUP.md for full environment configuration, secrets management, and database initialization.
The anomaly detection engine runs independently on :5000 and is the most production-ready component in the stack. It processes raw civic data vectors and returns cluster labels — outliers flagged as -1.
# Submit civic data points for DBSCAN anomaly scoring
curl -X POST http://localhost:5000/detect \
-H "Content-Type: application/json" \
-d '{"data": [[1.2, 0.5, 200000], [1.0, 0.4, 195000], [8.9, 7.1, 4500000]]}'
# → {"labels": [0, 0, -1]} ← last point flagged as anomaly
# z-score + threshold anomaly scoring
curl -X POST http://localhost:5000/score/anomaly \
-H "Content-Type: application/json" \
-d '{"values": [200000, 195000, 4500000]}'
# Sentiment analysis on civic text
curl -X POST http://localhost:5000/analyze/sentiment \
-H "Content-Type: application/json" \
-d '{"text": "The council approved the $4.5M no-bid contract unanimously."}'| Step | Status | Detail |
|---|---|---|
Data ingestion (POST /detect) |
✅ | Live |
| Normalization | ✅ | StandardScaler applied |
| Clustering | ✅ | DBSCAN via scikit-learn |
| Outlier flagging | ✅ | Label -1 = anomaly |
| z-score scoring | ✅ | POST /score/anomaly |
| Batch scoring | ✅ | POST /analyze/batch |
| Sentiment analysis | ✅ | TextBlob MVP live |
/predict endpoint |
🔴 | Pydantic schema needed (#9) |
| DistilBERT NLP swap | 🔴 | Replaces TextBlob (#8) |
| Model persistence | 🔴 | Planned |
CIVWATCH/
├── 📁 backend/ Node.js/Express API (:3000)
│ ├── /api/status ✅ LIVE
│ ├── /api/health ✅ LIVE
│ ├── /api/alerts ✅ LIVE
│ ├── /api/analytics ✅ LIVE
│ ├── /api/anomalies ⚠️ PARTIAL — empty array, ML bind needed
│ └── /api/ingest ⚠️ PARTIAL — no storage/forwarding
├── 📁 frontend/ React/Vite dashboard (:4000)
│ └── AnomalyDashboard ⚠️ PARTIAL — nav wired, no data binding
├── 📁 ml/ Python FastAPI ML engine (:5000)
│ ├── POST /detect ✅ DBSCAN live
│ ├── POST /score/anomaly ✅ z-score + flags
│ └── POST /analyze/* ✅ sentiment + batch
├── 📁 src/analytics/ dataAnalyzer.ts ⚠️ type errors (#3)
├── 📁 civwatch-desktop/ Electron wrapper — Phase 1 complete
├── 📁 tests/ Jest + pytest (~15% coverage)
├── 📁 docs/ Architecture · API spec · Testing strategy
├── 📁 .github/ 6 security scanning workflows
├── 🐳 docker-compose.yml
└── ⚙️ .env.example
| ✅ | Backend health / status / alerts / analytics APIs |
| ✅ | PostgreSQL pool + real migrations |
| ✅ | ML FastAPI + DBSCAN + StandardScaler |
| ✅ | JWT auth middleware |
| ✅ | Docker Compose — all healthchecks resolved |
| ✅ | CI/CD + 6 security scanning workflows |
| ✅ | Electron shell + IPC bridge |
| Priority | Task | Issue |
|---|---|---|
| 🔴 CRITICAL | Fix dataAnalyzer.ts type errors — verify real vector output |
#3 |
| 🔴 CRITICAL | Wire DBSCAN scheduler/trigger from anomaly_scores table |
— |
| 🔴 CRITICAL | ML /predict — Pydantic schema + bind return to /api/anomalies |
#9 |
| 🟡 HIGH | Complete POST /api/ingest — store, queue, forward to ML |
— |
| 🟡 HIGH | React anomaly charts + visualization components | #10 #11 |
| 🟡 HIGH | WebSocket real-time layer (socket.io + pg LISTEN/NOTIFY) | #12 |
| 🟡 MED | Wire Redis into alert routes (30s TTL) | #5 |
| 🟡 MED | Swap TextBlob → DistilBERT NLP | #8 |
| 🟡 MED | Composite indexes on anomaly_scores + alerts tables |
— |
| 🟡 MED | Integration tests — backend-to-ML + frontend-to-backend | #15 |
| 🟡 | OWASP A01–A05 security audit (#17) |
| 🟡 | Rate limiting + request throttling (#7) |
| 🟡 | PostgreSQL query tuning + load testing |
| 🟡 | 80%+ test coverage gate in CI (#16) |
| 🟡 | Packaged releases (.exe · .dmg · .AppImage) |
| 🟡 | Ops runbook + documentation review (#18) |
Current coverage: ~15% · Phase 2 target: 50%+ · Production gate: 80%+
npm test # TypeScript/Node suite
pytest tests/ -v # Python ML service
npm run test --workspaces && pytest tests/ -v # Full suiteSee docs/testing.md for strategy, coverage targets, and CI integration details.
| Task | Issue |
|---|---|
Fix TypeScript type errors in dataAnalyzer.ts |
#3 |
| Wire Redis into alert routes | #5 |
| Rate limiting middleware | #7 |
| Swap TextBlob → DistilBERT | #8 |
ML /predict endpoint |
#9 |
| React anomaly visualization components | #10 |
| WebSocket real-time layer | #12 |
| Dark mode theme support | #13 |
| Integration tests (backend ↔ ML) | #15 |
See CONTRIBUTING.md · Keep PRs small and focused. Every PR needs a short why in the description. Pick a good first issue to get started.
| File | Purpose |
|---|---|
| STATUS.md | Full per-component implementation matrix |
| IMPLEMENTATION_ROADMAP.md | Phased PR plan (PR0 → PR19) |
| NEXT_PHASE.md | This-week tasks + debugging guide |
| docs/architecture.md | System design, port map, data flow |
| CHANGELOG.md | Version history |
| SETUP.md | Detailed local environment setup |
| SECURITY.md | Security practices + threat model |
Do not open public GitHub issues for security vulnerabilities. Report exclusively via GitHub Security Advisories.
Full policy → SECURITY.md · RESPONSIBLE_DISCLOSURE.md
MIT — see LICENSE.
Built by Curtis Farrar
Independent Systems Engineer · AI Security Architect · Civic Monitoring · Iowa, USA
GitHub ·
Portfolio
This README reflects verified current state — not aspirations. Full truth table → STATUS.md