Relay

A self-hostable webhook gateway that never loses an event. Verify, normalize, persist, retry, fan out — with published p99s.

Status: v0.1.0 — implemented and tested. 29 tests green: 14 integration tests, 7 property tests (256-case proptest suites), 5 unit tests (claim atomicity, failed-write rescue, SSRF egress allow/block), and 3 kill-mid-flight chaos tests run against the release binary. See ROADMAP and docs/gap-analysis.md for the feature comparison against Svix / Hookdeck / Convoy.

Why

Every team integrating Stripe, GitHub, or Shopify rebuilds the same fragile plumbing: signature checks, deduplication, retries with backoff, dead-letter handling, replay. Hosted platforms (Hookdeck, Svix) solve it but add a vendor and a bill. Ad-hoc handler endpoints solve it until the process dies between "received" and "processed," and an invoice.paid silently vanishes.

Relay is the missing middle: a single Rust binary you run next to your app. It makes exactly one promise, precisely:

If Relay answered 200 OK, the event is durable and will be delivered or visibly dead-lettered — even across kill -9.

Everything else serves that sentence.

Features (v1)

• Signature verification — Stripe, GitHub, and Standard Webhooks schemes; raw-byte HMAC-SHA256, constant-time comparison (subtle), timestamp tolerance checks
• Persist-then-ack ingest — the event hits an fsync-backed WAL before the 200 goes out; p99 ingest target ≤ 10 ms
• Durable queue — embedded SQLite (WAL); PostgreSQL backend is the first post-v0.1 milestone
• Inbound dedup — provider event IDs (evt_…, X-GitHub-Delivery, webhook-id) enforced unique per path at the storage layer
• Fan-out with event-type filters — one inbound path feeds many listeners; each listener subscribes to exact or prefix.* event types
• At-least-once delivery engine — jittered exponential schedule presets (~3 days), per-listener circuit breakers, dead-letter queue, Retry-After honored, 410 Gone auto-disables the endpoint
• Signed fan-out — outbound deliveries signed per the Standard Webhooks spec so your consumers get the same rigor (stable webhook-id across retries)
• Destination authentication — static extra headers per listener (e.g. Authorization: Bearer …)
• Replay anything — single event or a whole listener's failed batch, from API or console
• Console — event log, delivery timelines, replay links; server-rendered HTML, no node_modules, stays a single binary
• Operations — retention TTL purge, ops notification webhook on dead-letter/disable/breaker trips, SSRF egress guard, secret rotation window
• Observability — structured tracing spans per lifecycle, Prometheus /metrics with p50/p95/p99 histograms

Architecture

 Stripe/GitHub ──▶ ┌───────────────────────────────┐
                   │ Receiver (axum)               │
                   │ 1. verify HMAC signature      │
                   │ 2. dedup (provider event id)  │
                   │ 3. persist FIRST (WAL commit) │
                   │ 4. return 200 immediately     │
                   └────────────┬──────────────────┘
                                │
                   ┌────────────▼──────────────────┐
                   │ Durable queue                 │
                   │  SQLite WAL │ PG SKIP LOCKED  │
                   └────────────┬──────────────────┘
                                │ claim due work
                   ┌────────────▼──────────────────┐
                   │ Delivery workers              │
                   │ - per-endpoint circuit breaker│
                   │ - exponential backoff+jitter  │
                   │ - dead-letter after N tries   │
                   │ - sign per Standard Webhooks  │
                   └────────────┬──────────────────┘
                                │
                   ┌────────────▼──────────────────┐
                   │ Your service                  │
                   └───────────────────────────────┘

 Console/API: event log · delivery timelines · replay · /metrics

1. Provider POSTs to /in/{source}/{listener}. Raw bytes are never deserialized before verification.
2. Signature verified in constant time; bad signature → 400, never persisted.
3. Event inserted inside one transaction (dedup via unique index). Commit returns → 200 OK.
4. Workers claim due deliveries, POST to your endpoint with a 30 s timeout, capture status/latency/body snippet.
5. Failures reschedule with full-jitter backoff; repeated failures trip the endpoint's circuit breaker; exhausted attempts land in the dead-letter state.
6. Replay moves dead (or any) events back to pending. Nothing is ever deleted to hide a failure.

Design decisions (short version)

Full arguments live in docs/; this table is the honest summary.

Decision	Choice	Tradeoff accepted	Doc
Durability	Persist-then-ack (WAL commit before 200)	~1 ms added to p99 ingest; buys the core promise	persist-then-ack.md
Delivery guarantee	At-least-once + idempotency keys	Duplicates are possible; consumers dedup on webhook-id	delivery-semantics.md
Exactly-once	Rejected — impossible across trust boundaries you don't control	—	delivery-semantics.md
Storage	SQLite default, Postgres first-class alternative	Single-writer ingest ceiling on SQLite	design.md
Ordering	Best-effort per endpoint	Strict ordering would stall retries behind one slow event	delivery-semantics.md
Crash proof	Kill -9 mid-flight in CI, reconcile ledger afterward	Chaos suite requires Linux runners	testing.md

Honest limitations (v1)

• Single node. No HA story. Run it like you run single-instance Redis: reliable until the host dies; replay recovers what was in flight.
• SQLite only today. The Postgres backend (SKIP LOCKED claims) is the first post-v0.1 milestone; the schema and queue logic were designed for it.
• SQLite mode is process-crash durable. Survives kill -9 (WAL + synchronous=NORMAL). Full power-loss durability needs synchronous=FULL — supported via env, documented, off by default for latency. The distinction is tested separately.
• Chaos methodology note: the kill-mid-flight suite aborts the real binary at deterministic crash points (RELAY_CRASH_AFTER) — no destructors or cleanup run, matching SIGKILL semantics — rather than signaling from outside the process.
• No transformation/filtering yet. JMESPath filtering is the first stretch goal.
• Endpoint secrets are stored unencrypted in the local DB. Appropriate for a self-hosted sidecar, not for multi-tenant SaaS (yet).
• Ordering is best-effort, never guaranteed. Retries and concurrency reorder; consumers dedup instead.
• Console is functional, not fancy — server-rendered HTML tables and timelines. A UI polish pass (Hookdeck/Vercel-grade) is a roadmap item.

Benchmarks

Measured on a desktop-class Windows box (i9-14900KF / 64 GB / SSDs, Defender active), v0.1.0 release build — full disclosure, methodology, failure breakdowns, and analysis of every miss in BENCHMARKS.md. Reproduce with cargo run --release --bin relay-bench.

Metric	Target (pre-declared)	Measured v0.1.0
Sustained ingest	≥ 5,000 evt/s	2,049 evt/s @ c64 · 1,972 @ c16 ❌
Ingest latency p50	—	6.1 ms @ c16 · 30.1 ms @ c64
p99 ingest latency	≤ 10 ms	43.6 ms @ c16 · 66.2 ms @ c64 ❌
End-to-end delivered/s (healthy dest.)	≤ 250 ms p99	4,266 delivered/s, full drain ❌/—
Time to accept traffic after kill -9	< 1 s	≈ 1 s observed in chaos runs ✅
Idle RSS	< 50 MB	12.9 MB ✅

Honest read: durability-first SQLite gives sub-millisecond medians and a 4×-under-budget footprint, but sustained throughput plateaus at ~2k evt/s on the single-writer embedded stack — misses analyzed in BENCHMARKS.md; the Postgres backend (next milestone) removes that ceiling. Linux/NVMe rerun to follow on CI hardware.

Quickstart

# from a checkout
git clone https://github.com/sambai-dev/relay && cd relay
cp .env.example .env   # reference for every RELAY_* var (Relay reads env vars, not .env files)

# required BEFORE `serve` — it refuses to start without a token
export RELAY_ADMIN_TOKEN="$(openssl rand -hex 16)"

# keeps the DB in this working dir; the default /data/relay.db needs root
export RELAY_DB_URL="sqlite://relay.db"

cargo run -- serve                     # creates ./relay.db on first start

# second terminal — same two exports, then register a listener
export RELAY_ADMIN_TOKEN=... RELAY_DB_URL="sqlite://relay.db"
cargo run -- add-listener \
  --source stripe --path demo --url https://your-service.test/hooks \
  --event-types "invoice.*,customer.*"

add-listener prints the listener record with generated secrets:

{ "id": "01J…", "ingest_url_hint": "/in/{source}/{path}", "path": "demo", "secret": "whsec_…", "signing_secret": "whsec_…", "source": "stripe" }

Or install from crates.io instead of a checkout (crate relay-gateway, binary named relay): cargo install relay-gateway.

Point Stripe at it:

stripe listen --forward-to localhost:8080/in/stripe/demo

Every delivery your service receives is signed with webhook-id / webhook-timestamp / webhook-signature per the Standard Webhooks spec — verify with any of its existing libraries.

Admin API + console at http://localhost:8080/console?token=$RELAY_ADMIN_TOKEN; Prometheus metrics at /metrics (bearer token). Docker image ships with the v0.1 release tag.

Security note: the console authenticates via ?token= in the URL so it works from a bare browser. The tradeoff: that token can leak through browser history, server access logs, and Referer headers. This embedding is intended for local / trusted-network use only — do not expose the console on the public internet; use the Bearer-token admin API for programmatic access from untrusted networks instead.

Repository layout

relay/
├── src/
│   ├── main.rs        # CLI: serve | migrate | add-listener | generate-secret
│   ├── lib.rs         # module wiring for the library crate
│   ├── config.rs      # env config + retry presets
│   ├── db.rs          # SQLite pool, WAL setup, schema, startup recovery
│   ├── models.rs      # Listener / Event / Delivery records
│   ├── verify.rs      # stripe | github | standard verifiers + outbound signing
│   ├── ingest.rs      # POST /in/{source}/{path}: verify → dedup → persist → ack
│   ├── deliver.rs     # worker pool: claim (BEGIN IMMEDIATE) → sign → send → schedule
│   ├── backoff.rs     # full-jitter delays, breaker cooldowns
│   ├── breaker.rs     # per-listener closed/open/half-open state machine
│   ├── admin.rs       # /api/v1 REST: listeners, events, replay, metrics
│   ├── console.rs     # server-rendered HTML log/timeline
│   ├── server.rs      # router assembly + graceful serve
│   ├── state.rs       # shared App state: config, pool, HTTP client, breakers, metrics
│   ├── obs.rs         # tracing/OTLP init + shutdown (RELAY_OTLP_ENDPOINT)
│   ├── metrics.rs     # hand-rolled Prometheus text format, HDR-style buckets
│   ├── purge.rs       # retention TTL task
│   ├── notify.rs      # ops notification webhook
│   └── crash.rs       # RELAY_CRASH_AFTER hooks for the chaos suite
├── src/bin/
│   └── relay-bench.rs # load/benchmark harness (`cargo run --release --bin relay-bench`)
├── tests/
│   ├── common/mod.rs  # app spawn helpers, mock destination, signature builders
│   ├── integration.rs # end-to-end flows incl. fan-out, dedup, DLQ→replay, 410
│   ├── property.rs    # proptest: verifier bit-flips, delay bounds, filters
│   └── chaos.rs       # T2/T4/T5 kill-mid-flight proofs vs the real binary
├── docs/              # design · persist-then-ack · delivery-semantics ·
│                      # gap-analysis · testing · research · deploy (.md)
├── .github/workflows/ # ci.yml · release.yml · docker.yml · benchmarks.yml
├── .env.example       # reference for every RELAY_* env var
├── BENCHMARKS.md      # benchmark methodology + full v0.1 results
├── Dockerfile         # release image (also published to GHCR)
├── fly.toml           # fly.io deploy config
├── Cargo.toml / Cargo.lock / LICENSE / ROADMAP.md
└── README.md

Splitting into a workspace (relay-core IO-free vs relay-server) happens only when benchmarks demand an IO-free core — not before.

Docs

Doc	Contents
docs/design.md	Components, data model, delivery engine, API surface, observability, security
docs/persist-then-ack.md	The correctness essay: why the 200 means what it means
docs/delivery-semantics.md	At-least-once, why exactly-once is a lie, idempotency contract
docs/gap-analysis.md	Feature comparison vs Svix / Hookdeck / Convoy — adopted vs deliberately deferred
docs/testing.md	Chaos suite design, property tests, benchmark methodology, CI
docs/research.md	Signature schemes, prior art, backoff literature, crate survey
ROADMAP.md	Milestones with definitions of done

Launch plan

• Day 0: CI gates on (fmt --check, clippy -D warnings, tests) + release workflow building Linux/macOS binaries — even while the binary does nothing.
• Week 5: crates.io publish (name checked day 0), GHCR Docker image, public demo, then:
  • Show HN: "Show HN: Relay – an open-source webhook gateway that survives kill -9"
  • r/rust thread same week; prepared answers for the durability and "why not Svix/Hookdeck" questions
  • First-week issue SLA: responses within 24 h

License

MIT — same license as opencode.