An API gateway in Go that treats failure as the normal case. One binary in front of your services gives you distributed rate limiting with standard RateLimit headers, per-upstream circuit breakers, hedged retries, active health checks, JWT verification at the edge, and Prometheus metrics — configured from one YAML file that hot-reloads without dropping connections.
Built with the standard library's net/http at the core: the proxy loop, breaker state machine and limiter algorithms are implemented in this repo, not imported — the point is to show how they work.
- Rate limiting — token bucket per instance, or a Redis-backed sliding-window counter shared across replicas (atomic Lua, no boundary-burst flaw). Emits IETF
RateLimit/RateLimit-Policyheader fields (draft-ietf-httpapi-ratelimit-headers), not the legacyX-RateLimit-*. - Circuit breaking — per-upstream closed → open → half-open with capped concurrent probes; open breakers fail fast without consuming a retry attempt.
- Retries & hedging — exponential backoff with full jitter; idempotent requests can launch a parallel hedged attempt to a second upstream when the first is slow. Non-idempotent requests are retried only when the connection never happened.
- Health checking — active probes flip upstreams in and out of rotation; round-robin distributes fairly across whatever is left.
- Edge auth — JWT validation (HS256 via env secret, RS256 via cached JWKS) before traffic reaches upstreams; subject forwarded as
X-Auth-Subjectand usable as the rate-limit key. - Operations — hot config reload (file watch + SIGHUP) with rejected-if-invalid semantics, graceful shutdown, JSON logs with request IDs, Prometheus metrics, live route/breaker status endpoint.
flowchart LR
C[Client] --> MW[middleware<br/>request-id · logs · metrics]
MW --> RT{route match<br/>longest prefix}
RT --> A[JWT verify]
A --> RL[rate limiter<br/>token bucket / Redis sliding window]
RL --> F[forwarder<br/>retries · hedging]
F --> B1((breaker A)) --> U1[upstream A]
F --> B2((breaker B)) --> U2[upstream B]
HC[health checker] -.probes.-> U1 & U2
F -.selection.-> P[pool<br/>health-aware round robin]
ADM[admin :9090<br/>/metrics /routes /healthz]
Key decisions:
- Engines are immutable. A config reload builds a whole new engine and swaps it atomically; a broken config is rejected and the old engine keeps serving. No partially-applied state, ever.
- The limiter fails open. If Redis dies, requests pass and the event is logged — a degraded limiter should not become the outage itself.
- Exhausted retries relay the upstream's 5xx instead of masking it with a synthetic 502: the client sees what actually happened.
- Bodies are buffered up to 1 MiB to make retries possible; larger bodies stream through once and are never retried.
git clone https://github.com/aminyx/threshold
cd threshold
go run ./cmd/threshold -config configs/gateway.yamlOr the full demo topology (gateway + 2 echo upstreams + Redis + Prometheus):
docker compose -f deploy/docker-compose.yml up --buildcurl -i http://localhost:8080/echo/hello # proxied, RateLimit headers
curl http://localhost:9090/routes # live upstream health + breaker statesPrometheus UI: http://localhost:9091 — try rate(threshold_requests_total[1m]).
Kill an upstream and watch the gateway route around it:
docker stop deploy-upstream-a-1Everything lives in one YAML file — see the annotated configs/gateway.yaml. Per route: prefix, upstreams, timeout, rate limit (rate/burst/window/key), retry policy (max_attempts, base_delay, hedge_delay), breaker thresholds and health checks. Edit the file while the gateway runs — it reloads automatically (or send SIGHUP).
| Variable | Required | Description |
|---|---|---|
THRESHOLD_JWT_SECRET |
only with HS256 auth | Shared secret for HS256 verification. The config names the variable; the secret never appears in config files. |
go test -race ./...The suite covers the breaker state machine, both limiter algorithms (the Redis one against miniredis, including the window-boundary case), health-aware round robin, JWT verification (including alg=none and JWKS kid rotation), and end-to-end proxy behavior: failover retries, POST-not-retried semantics, open-breaker fast-fail, 429 headers, and hedging against a slow upstream.
Numbers from the included zero-dependency probe (go run ./loadtest/probe), Windows 11 dev machine, gateway and probe sharing the host — read them as relative overhead, not server capacity:
| Path | RPS | p50 | p95 | p99 | errors |
|---|---|---|---|---|---|
| through gateway (100 conc, 20s) | 6 244 | 13.6 ms | 36.4 ms | 55.3 ms | 0 |
| direct to upstream (same load) | 25 305 | 2.9 ms | 11.5 ms | 21 ms | 0 |
A k6 scenario with latency thresholds is included for the compose demo.
- HTTP/1.1 reverse proxying only — no WebSocket/upgrade passthrough or gRPC streaming yet.
- The sliding-window estimate weights the previous window linearly; it is an approximation (industry standard, but an approximation).
- Rate-limit keying by IP uses the socket peer address; running behind another proxy requires the
header:key mode instead of trustingX-Forwarded-For. - Single-process data plane: horizontal scale comes from running replicas behind a TCP balancer with Redis-backed limits.
- OpenTelemetry traces through the proxy hop (OTLP export)
- Response caching with singleflight collapse for hot idempotent routes
- Weighted upstreams and least-connections balancing
- WebSocket upgrade passthrough