MeshCore Analytics

A real-time analytics platform for MeshCore networks. It ingests MQTT packets from mctomqtt-compatible observers, decodes them with @michaelhart/meshcore-decoder, stores them in TimescaleDB, and serves interactive dashboards plus public-facing site pages with live mapping, link intelligence, coverage modelling, packet analytics, and worker/system health.

---

Features

• Real-time node map with animated packet arcs and live WebSocket updates
• Progressive UK-wide HopReach RF coverage with Standard and Precision tiers
• Link intelligence overlay with directional observations and path-loss viability
• Public repeater-topology explorer with hub ranking, graph components, likely bridge repeaters, isolated nodes, and multibyte-evidence filtering
• Map modes, shareable viewport/filter URLs, selected-node popup, and bounded activity replay
• Regional health scoring and predicted-versus-observed RF validation
• Local saved searches/watchlists for nodes, observers, regions, packet types, and incidents
• Privacy-filtered CSV/GeoJSON exports with a versioned OpenAPI contract
• Beta path prediction model with concurrent worker pool and hourly path-learning prior rebuilds
• Multibyte path-hash support (1-byte, 2-byte, 3-byte) throughout the live ingest and pathing stack
• Decoded live packet feed (Advert, GroupText, DM, ACK, Path, Trace)
• Stats pages and chart endpoints for packet rates, radios, hops, and activity
• Public Health page with worker status/history + server resource metrics
• UK site Feed page for public MQTT observer traffic visibility
• Repeater owner portal with MQTT username/password login and encrypted cookie session
• Owner dashboard with repeater summary, direct sender map, live packets, advert trend, heard-by list, link health, and alerts
• Multi-network ingestion (meshcore/* and ukmesh/*) with per-site filtering
• Isolated test-feed support via meshcore-test/* and test.ukmesh.com
• Multi-observer deduplication by packet hash
• MQTT connection monitor with Mosquitto log parsing and reconnect tracking

---

Roadmap

Phase 1 - Core platform (complete)

• MQTT ingestion via mctomqtt with multi-observer support
• Packet decoding with @michaelhart/meshcore-decoder
• TimescaleDB storage and live WebSocket fan-out
• React dashboard: node map, animated packet arcs, decoded live feed
• Packet deduplication by hash across observers

Phase 2 - RF coverage and link intelligence (complete)

• HopReach v0.1.32 canonical terrain propagation with reference-parity tests
• Progressive Standard/Precision rasters rendered natively in MapLibre
• Link worker: observed relay-path processing into node-to-node link intelligence
• Directional link counts and path-loss viability modelling
• Versioned GB, Northern Ireland, Isle of Man, Jersey, and Guernsey boundary

Phase 3 - Path learning and predictions (beta)

• Hourly path-learning prior rebuild worker
• Beta path overlays and confidence scoring
• Historical calibration using observed packet behavior
• Multibyte path-hash aware path resolution
• Concurrent resolve worker pool for high-throughput path matching

Phase 4 - Public website and operations (complete)

• Separate public-facing website pages (install, MQTT, packets, stats)
• Public Health page with worker/system status and history
• Click-to-explain worker cards
• UK Feed page for live public observer traffic

Phase 5 - Repeater owner portal (complete)

• MQTT username/password owner login with encrypted cookie session
• Dedicated owner auth database for username → repeater ownership mapping
• Owner-facing dashboard: repeater summary, packet history, advert counts, direct sender map, heard-by list, link health, and alerts
• Planned repeater registration/claim workflow improvements
• Owner alerts for offline duration, low battery voltage, and excessive predicted path loss, with durable delivery history and test delivery

Phase 6 - Network intelligence expansion (complete)

• Bounded topology graph with hubs, graph components, isolated repeaters, and likely bridge nodes
• Regional health scoring based on traffic freshness and observer redundancy
• Timeline replay, map modes, shareable views, node details, and planning comparisons
• Path explanations with confidence, evidence, alternatives, and limitations
• Saved searches/watchlists plus versioned CSV and GeoJSON exports

Phase 7 - Predicted vs observed RF model validation (complete)

• Compare terrain-predicted links against real observed relay behavior
• Highlight high-confidence mismatches for network tuning
• Separate operator overrides and weak evidence from likely model mismatches

Phase 8 - Reliability and operations (complete)

• CI for backend, frontend, browser journeys, Python workers, and Compose configuration
• Independent synthetic HTTP/WebSocket monitoring with failure and recovery webhooks
• Liveness/readiness split, public status page, DB maintenance telemetry, and bounded load tooling
• Restartable, audited production-network label migration with snapshot-based rollback guidance

---

Current State

• Split worker architecture for resilience:
  • hopreach (canonical progressive coverage compute)
  • link-worker (link/path-loss processing)
  • path-learning-worker (hourly model rebuild)
  • health-worker (health snapshots)
  • link-backfill-worker (one-shot historical backfill)
• Path resolver runs a concurrent worker pool (resolveWorker, resolvePool, resolveCache) to handle high packet volumes without blocking the main ingest loop.
• Nginx frontend proxies use Docker DNS resolver-based upstreams to avoid stale backend IP issues after container recreates.
• Owner authentication uses MQTT credentials plus a separate owner-auth mapping database rather than public-key login.
• Live coverage is served by HopReach's canonical terrain model; calibrated variants remain disabled pending UK evidence validation.
• Public/test feeds are isolated at the topic level, with meshcore-test/* excluded from the public sites.
• MQTT connection state is tracked via Mosquitto log parsing — connect/disconnect events are available in the health feed.

---

Quick Start

# 1. Clone and enter the project
git clone https://github.com/gadgethd/ukmesh.git
cd ukmesh

# 2. Copy and configure environment
cp .env.example .env
# Edit .env. Required values are POSTGRES_PASSWORD, JWT_SECRET,
# MQTT_PASSWORD, REDIS_PASSWORD, OPERATOR_SITE_TOKEN,
# ANUBIS_ED25519_PRIVATE_KEY_HEX, GRAFANA_ADMIN_PASSWORD, and the full
# 40-character HEALTHCHECK_SOURCE_REF.

# 3. Create Mosquitto's backend credential and least-privilege ACL
set -a
. ./.env
set +a
scripts/bootstrap-mosquitto.sh

# 4. Start everything
docker compose up -d

# 5. Confirm readiness and inspect logs
curl --fail http://127.0.0.1:3000/readyz
docker compose logs -f backend

The calculator starts with the stack, publishes Standard tiles first, and continues Precision only after Standard is live and the disk gate passes:

docker compose up -d backend hopreach app-ukmesh link-worker
docker compose logs -f hopreach

Local endpoints:

• Backend API/WS: http://localhost:3000
• App (ukmesh): http://localhost:3003
• Website (ukmesh): http://localhost:3004
• Dev/test site: http://localhost:3006
• Liveness/readiness: http://localhost:3000/healthz and http://localhost:3000/readyz
• API discovery/OpenAPI: http://localhost:3000/api/v1 and http://localhost:3000/api/v1/openapi.yaml

To expose it publicly, configure a Cloudflare Tunnel (see below) or reverse proxy of your choice.

---

Environment Variables

Copy .env.example to .env and fill in your values. All variables used by the app:

Variable	Default	Description
POSTGRES_DB	meshcore	TimescaleDB database name
POSTGRES_USER	meshcore	TimescaleDB user
POSTGRES_PASSWORD	(required)	TimescaleDB password
POSTGRES_MAX_WORKER_PROCESSES	24	PostgreSQL worker slots; keep above the Timescale background-worker setting plus launcher/scheduler headroom
MQTT_BROKER_URL	ws://mosquitto:9001	Mosquitto WebSocket URL (internal)
MQTT_USERNAME	backend	MQTT client username
MQTT_PASSWORD	(required)	MQTT client password
REDIS_PASSWORD	(required)	Password for the bundled Redis service; passed separately from the URL so reserved characters are safe
REDIS_URL	redis://redis:6379	Redis URL for WebSocket pub/sub
JWT_SECRET	(required)	Secret for JWT verification
ALLOWED_ORIGINS	http://localhost:3001,http://localhost:3002	Comma-separated browser origins allowed for CORS and WebSocket
API_RATE_LIMIT_MAX	120	Per-client public API requests/minute; raise only in an isolated load-test project
VITE_APP_HOSTNAME	(blank — always shows dashboard)	If set, only this hostname serves the analytics dashboard; all others serve the public website layout
MESHCORE_CHANNEL_SECRETS	(blank)	Comma-separated channel secrets for decrypting GroupText packets. Format: name:hex or bare hex. The default MeshCore public channel key is always included.
OWNER_DATABASE_URL	(optional)	Separate Postgres database URL for owner portal username → repeater mappings
OWNER_COOKIE_SECRET	(optional but recommended)	Secret used to encrypt/sign the owner session cookie
OWNER_MQTT_USERNAME_MAP	(empty)	Operator-managed owner grants in the format `user=nodeId1
OWNER_AUTHORIZATION_MODE	shadow	shadow preserves read-only legacy ACL compatibility; enforce accepts verified database/config grants only
OWNER_ACL_MODE	shadow	shadow renders and validates without changing Mosquitto; apply atomically installs and verifies the canonical ACL
OWNER_ACL_UNMANAGED_USERS	backend,test,test2	Exact broker accounts intentionally preserved outside owner grant management
OWNER_ACL_ALLOW_EMPTY_USERS	(empty)	Explicitly reviewed owner accounts allowed to render with no publish grants
RF_COVERAGE_ENABLED	true	Compile the native HopReach layer into the production app image
HOPREACH_IMAGE	meshcore-analytics-hopreach:local	Immutable HopReach calculator image override
HOPREACH_CPUS	4.0	Calculator CPU limit
HOPREACH_CPU_WORKERS	4	Go CPU worker limit (GOMAXPROCS)
HOPREACH_MEMORY_LIMIT	8g	Calculator memory and swap limit
PUBLIC_FEATURE_INFERRED_NODES_ENABLED	true	Runtime kill switch for privacy-reviewed inferred map nodes
PUBLIC_FEATURE_PACKET_ARCS_ENABLED	true	Runtime kill switch for privacy-filtered live packet arcs
PUBLIC_FEATURE_HEATMAP_ENABLED	false	Runtime kill switch for the packet heatmap
PUBLIC_FEATURE_CONFIG_TTL_SECONDS	30	Client refresh interval for same-origin public feature configuration (bounded to 5–300 seconds)
CLOUDFLARE_TUNNEL_TOKEN	(optional)	Cloudflare Zero Trust tunnel token
PORT	3000	Internal app port

The inferred-node and packet-arc layers are enabled after privacy/correctness validation; the heatmap remains disabled by default. Each has an independent runtime kill switch. To operate an immediate kill switch, change the relevant value in .env, then recreate the backend without building an image:

docker compose up -d --no-build --force-recreate backend

Browsers load /api/runtime-config before the map starts and refresh it within PUBLIC_FEATURE_CONFIG_TTL_SECONDS. A failed, malformed, or timed-out request disables all three layers.

---

Mosquitto Setup

Mosquitto is configured for WebSocket-only access with password authentication. Create the backend credential and least-privilege read ACL before first startup:

# Reads MQTT_USERNAME/MQTT_PASSWORD from the environment.
set -a
. ./.env
set +a
scripts/bootstrap-mosquitto.sh
docker compose up -d

The bootstrap is idempotent, sets credential files to mode 0640, refuses symlinks or incomplete existing state, and never overwrites live credentials.

Do not create observer passwords manually. Provision every observer with ~/bin/newuser; it installs and verifies the publish ACL before enabling the credentials.

The host helper at ~/bin/newuser requires one or more full 64-character node public keys. It validates the keys, writes and verifies exact per-key publish ACLs, and only then creates the MQTT password. This ordering prevents an account from authenticating successfully while all of its publishes are silently denied. In MeshCore-HA, keep {PUBLIC_KEY} in the topic template; the helper requires the actual key only to provision the server-side ACL.

---

Cloudflare Tunnel (optional)

To expose the app and MQTT broker publicly without opening firewall ports:

1. Go to Cloudflare Zero Trust → Networks → Tunnels
2. Create a tunnel and copy the token
3. Add to .env: CLOUDFLARE_TUNNEL_TOKEN=<token>
4. Start with the tunnel profile: docker compose --profile tunnel up -d
5. Configure public hostnames in the Cloudflare dashboard (example):
   • app.example.com → http://anubis-app-ukmesh:8923
   • www.example.com → http://anubis-website-ukmesh:8923
   • mqtt.example.com → http://mosquitto:9001
   • healthcheck.example.com → http://anubis-mesh-health-check:8923

For UKMesh health checks, point healthcheck.ukmesh.com at http://anubis-mesh-health-check:8923 in the same tunnel. The container uses the internal Mosquitto WebSocket listener and persists observer/result state in the mesh_health_check_data Docker volume. A network-isolated one-shot Compose initializer normalizes that volume's ownership before the capability-free, non-root application starts. By default it uses HEALTHCHECK_TEST_CHANNEL_NAME=ukmeshtest and reuses the existing test:... entry from MESHCORE_CHANNEL_SECRETS via HEALTHCHECK_TEST_CHANNEL_SECRET_SOURCE_NAME=test.

---

MQTT Topic Structure

The backend subscribes to meshcore/#, ukmesh/#, and meshcore-test/#. MeshCore observers publish mctomqtt-compatible JSON envelopes to topics of the form:

meshcore/<IATA>/<observer-public-key>/packets   # received/transmitted packets
meshcore/<IATA>/<observer-public-key>/status    # node status advertisement
ukmesh/<IATA>/<observer-public-key>/packets
ukmesh/<IATA>/<observer-public-key>/status
meshcore-test/<IATA>/<observer-public-key>/packets
meshcore-test/<IATA>/<observer-public-key>/status

Payloads are JSON envelopes containing a raw hex field (the MeshCore packet) plus metadata such as RSSI, SNR, direction, and hash. The ingest path supports 1-byte, 2-byte, and 3-byte path hashes carried inside the raw packet.

---

Architecture

MeshCore Devices
     │ LoRa RF
     ▼
 mctomqtt-compatible observer
     │ MQTT over WebSocket/TLS
     ▼
 Mosquitto ─────────────────────────────── (optional Cloudflare Tunnel)
     │ subscribe meshcore/# + ukmesh/#
     ▼
 Backend (Node.js/TypeScript)
     │
     ├─ meshcore-decoder → TimescaleDB (packets, nodes, observed links, priors, health snapshots)
     │
     ├─ Path resolver worker pool (concurrent resolve workers + LRU cache)
     │
     ├─ Redis pub/sub
     │
     ├─ WebSocket → frontend live updates
     └─ REST API /api/*

 App/Web Frontends (Nginx + React)
     └─ app-ukmesh / website-ukmesh / website-dev (interactive dashboard + public site + owner portal)

 RF and Link Workers
     ├─ HopReach (persistent DEM cache, resumable Standard/Precision rasters)
     ├─ backend private compatibility API (positioned repeaters + observed evidence)
     └─ link-worker → node_links from observed paths

 Backend Workers (Node.js)
     ├─ path-learning-worker (hourly prior rebuild)
     ├─ health-worker (minute snapshots)
     └─ link-backfill-worker (one-shot historical backfill)

 Owner Auth
     └─ separate Postgres DB for MQTT username → repeater ownership mapping

---

Services

Service	Image	Purpose
timescaledb	Digest-pinned TimescaleDB/PostgreSQL image from Compose	Time-series and relational data storage
mosquitto	Digest-pinned Eclipse Mosquitto image from Compose	MQTT broker (WebSocket only)
mosquitto-reloader	Locally built, least-privilege helper	Authenticated broker-local ACL reload
redis	Digest-pinned Redis image from Compose	WebSocket fan-out pub/sub and bounded job queues
backend	Built from Dockerfile.backend	MQTT ingest, decoding, API, WebSocket
path-learning-worker	Built from Dockerfile.backend	Hourly path-learning model rebuilds
health-worker	Built from Dockerfile.backend	Periodic health snapshot capture
link-backfill-worker	Built from Dockerfile.backend	One-shot historical link backfill
synthetic-monitor	Built from Dockerfile.backend	Independent HTTP/WebSocket journey checks and alert delivery
link-worker	Built from viewshed-worker/Dockerfile	Link/path-loss processing from observed paths
hopreach	Built from third_party/hopreach/Dockerfile	Canonical terrain RF calculation and progressive raster publication
app-ukmesh	Built from Dockerfile.app	Interactive dashboard frontend
website-ukmesh	Built from Dockerfile.website	Public website frontend
mesh-health-check	Built from the configured gadgethd/meshcore-health-check ref	MeshCore observer coverage health-check app
website-dev	Built from Dockerfile.website	Isolated test/status site for meshcore-test/* traffic
cloudflared	cloudflare/cloudflared	Optional Cloudflare Tunnel (use --profile tunnel)
alloy, loki	Digest-pinned observability images	Bounded journal/container log collection and storage
prometheus, alertmanager, alert-receiver	Digest-pinned observability images plus local receiver	Metrics, rules, grouped alert delivery, and delivery history
grafana	Digest-pinned Grafana image	Provisioned read-only operational dashboards
postgres-exporter, redis-exporter, node-exporter, blackbox-exporter	Digest-pinned exporters	Database, queue, host, and endpoint telemetry

---

Data Retention

• Compression and destructive retention ship disabled. Raw packets and status samples remain intact until a table-specific, backup- and restore-gated lifecycle rollout is approved.
• The proposed raw retention window is 180 days, preserving the longest 120-day learner dependency. Privacy-safe hourly/daily aggregates and current node/link/model state remain longer lived. Legacy node_coverage is retained for one release as inactive rollback data only.
• Operational row-table cleanup is bounded and runs only when both DATA_LIFECYCLE_RETENTION_ENABLED=true and the exact table appears in DATA_LIFECYCLE_RETENTION_TARGETS.
• See docs/db-lifecycle.md for the exact dry-run inventory, compression, retention, owner export, and restore requirements.

---

Acknowledgements

This project is built on the following open source libraries and tools:

Frontend

Package	License
React	MIT
Vite	MIT
TypeScript	Apache 2.0
MapLibre GL JS	BSD 3-Clause
deck.gl	MIT
react-router-dom	MIT
Recharts	MIT
polygon-clipping	MIT

Backend

Package	License
Express	MIT
MQTT.js	MIT
ws	MIT
ioredis	MIT
node-postgres	MIT
cors	MIT
express-rate-limit	MIT
@michaelhart/meshcore-decoder	MIT

RF and link workers

Package	License
HopReach v0.1.32-ukmesh.3	AGPL-3.0 plus Commons Clause
NumPy	BSD 3-Clause
SciPy	BSD 3-Clause
Shapely	BSD 3-Clause
GDAL	MIT/X
psycopg2	LGPL v3
redis-py	MIT
Requests	Apache 2.0

Infrastructure

Tool	License
TimescaleDB	Apache 2.0 (Community)
Redis	BSD 3-Clause
Eclipse Mosquitto	EPL 2.0 / EDL 1.0
Docker	Apache 2.0

Data

Source	License
SRTM Elevation Data	Public Domain (NASA)
Natural Earth	Public Domain

---

License

UK Mesh-authored code is licensed under MIT — see LICENSE. Vendored HopReach and derived RF integration files retain AGPL-3.0 plus the Commons Clause; see third_party/hopreach/LICENSE and rf-coverage/SOURCE-OFFER.md.

Note on dependencies: Eclipse Mosquitto (EPL 2.0) is used as a dependency but not modified. Other runtime dependencies use MIT, BSD, or Apache 2.0 licenses.