diff --git a/adr/003-mojo-default.md b/adr/003-mojo-default.md index 58e1a36..75aacad 100644 --- a/adr/003-mojo-default.md +++ b/adr/003-mojo-default.md @@ -161,13 +161,11 @@ Each follow-up is a one-page issue. None blocks this ADR. 3. ~~**Rate limiting.**~~ **Shipped (per-process):** `RateLimit[Inner]` token bucket tuned by `RATE_LIMIT_RPS` / `RATE_LIMIT_BURST` (0 = disabled). Approximately global across workers via a shared atomic cell. Distributed rate limiting still needs Redis and stays open. -4. **TLS termination.** **Two upstream artefacts filed; local stand-in kept as reference.** (a) Reactor-native TLS serving in flare is a deferred upstream follow-up — ``STATE_TLS_HANDSHAKE`` is gated on parametric-trait method specialisation in a future Mojo nightly, and the server-side FFI has no ``SSL_read``/``SSL_write`` data path because the reactor integration was never wired. **Design doc filed at [ehsanmok/flare#7](https://github.com/ehsanmok/flare/issues/7)** describing the work for a ``serve_tls`` entry point (essentially: bind the existing ``flare_ssl_read``/``flare_ssl_write`` C exports on the server side and hand the wrapped stream to the unified reactor). (b) An in-binary stand-in via ``opengateway.mojo_bridge.tls_proxy`` is verified by ``tests/test_tls_proxy.py`` — three tests, real TLS round-trip, cleartext rejection, multi-chunk forwarding — but does not compose with the Mojo runtime's daemon-thread + ``GILReleased`` interaction: the proxy threads bind and the TLS handshake completes inside the binary, yet bytes do not flow from the TLS socket to the upstream socket when the reactor also calls into the Python bridge. **Diagnostic filed at [modular/modular#6865](https://github.com/modular/modular/issues/6865)** with a minimal reproducer (``feat/tls-thread-repro`` branch, ``opengateway/mojo/repro_tls_thread.mojo`` + ``opengateway/mojo_bridge/_repro_tls.py``). Production terminates TLS at the edge LB (ALB / NLB / nginx / Caddy), which is the standard gateway deployment shape anyway. - - The validated pure-Python proxy (``opengateway/mojo_bridge/tls_proxy.py`` + ``tests/test_tls_proxy.py``) is kept in-tree as a reference implementation and a building block for when flare#7 lands; the wiring in ``main.mojo`` was reverted. +4. **TLS termination.** **Shipped via [PR #36](https://github.com/echohello-dev/opengateway/pull/36) on the flare#7 commit (`edf3f22`).** Bumped `pixi.toml` from `v0.9.0` to `edf3f22`; rebuilt `libflare_tls.so` against the new flare source (one-line `-Wl,-u,_flare_ssl_read_ex` linker flag because clang treats unreferenced extern C as dead-code; gcc includes them by default). `main.mojo` binds via `HttpServer.bind_tls(addr, cert, key, alpn=["http/1.1"])` + `serve_tls(stack)` when `TLS_CERT_FILE`/`TLS_KEY_FILE` are set. e2e: `openssl s_client` handshake completes, ALPN negotiates `http/1.1`, non-streaming + streaming `/v1/chat/completions` round-trip works over HTTPS, structured logger emits JSON per line. The flare reactor-side TLS reactor is real and working. Self-signed cert hostname gotcha documented in README ("Self-signed cert gotcha") — curl needs `--resolve localhost:port:127.0.0.1` when the cert is `CN=localhost`. Production terminates TLS at the edge LB; the in-binary path is for the edge-binary deployment story (Fly.io machines, bare metal, Lambda-ish). The stdlib-ssl proxy (`opengateway/mojo_bridge/tls_proxy.py` + `tests/test_tls_proxy.py` + `opengateway/mojo/repro_tls_thread.mojo`) stays in-tree as a vendored fallback for the v0.9.0 pin and as a reproduction of the early daemon-thread diagnostic; the wrapper usage in `main.mojo` was reverted. The flare issue I filed earlier (#8) was closed because the only "stall" I observed was curl's cert hostname verification failing silently under `-s`, not a reactor bug. 5. ~~**DB-backed virtual keys.**~~ **Shipped via the Python bridge** (no Mojo Postgres driver required): `opengateway/mojo_bridge/db.py` defines a `VirtualKeyStore` protocol with an asyncpg-backed implementation, and `authenticate_authorization` consults it after the root-key short-circuit with a 60 s in-process TTL cache. One connection per lookup — the bridge's one-shot `asyncio.run` model means a pool cannot outlive its event loop; at gateway latencies the connect cost is noise. `database_url` is now unset-by-default; when unset the store seam returns `None` and auth is root-key-only. Schema is created by `PostgresVirtualKeyStore.ensure_schema` (`virtual_keys` table with models / budget / tpm / rpm columns). **Spend recording shipped with it:** the bridge increments `budget_used` by `usage.total_tokens` after each unary completion, and the streaming pump parses the terminal usage chunk (the bridge injects `stream_options: {"include_usage": true}` so upstreams always send one). Budgets are token-denominated; a per-model dollar pricing table is a deliberate product decision left open. -6. **HTTP/3 enablement.** **Blocked on two prerequisites:** (a) server-side TLS (follow-up 4, which is itself blocked on flare's reactor TLS wiring), since QUIC is always encrypted; and (b) the rustls QUIC cdylib (`libflare_rustls_quic.so`), which flare's activation script builds via cargo — not present in the default toolchain image. When both land, `HttpServer.bind_with_http3` + `QuicServerConfig` is the wiring point, gated behind an env flag. +6. **HTTP/3 enablement.** **Partially landed, then rolled back.** The rustls QUIC cdylib now builds in the pixi env and the Docker image (#35), and the flare pin bump (#36) pulls in `HttpServer.bind_with_http3` + `QuicServerConfig`. **The reactor-side driver is not integrated at the pinned commit** — `serve()` does not call `tick_http3_once` / `pump_http3_handler_once`, which the source itself labels "Test-only entry point". Binding via `bind_with_http3` accepts the UDP socket but never processes datagrams, so a working h3 path requires a custom reactor loop (select on the UDP fd, feed into `recv_datagram`, dispatch via `pump_http3_handler_once`, drain egress via `send_to`) which is real upstream flare work, not an opengateway concern. As a safety measure, `main.mojo` raises a clear `Error` if `HTTP3_ENABLED=1` is set rather than binding a UDP port that silently accepts nothing. Once flare integrates h3 into `serve()` (or another PR adds the loop in opengateway and upstreams it), the guard can be replaced with the wiring. 7. **CI parallelism for the Mojo build.** **Shipped:** the mojo CI job caches `~/.cache/modular` (the `.mojo_cache` compile cache) keyed on the Mojo sources + `pixi.lock`, turning the ~6 min cold compile into an incremental rebuild. The pixi version is pinned to v0.70.2 to match flare v0.9.0's build backend. diff --git a/opengateway/mojo/main.mojo b/opengateway/mojo/main.mojo index 4fe0786..bf91392 100644 --- a/opengateway/mojo/main.mojo +++ b/opengateway/mojo/main.mojo @@ -319,7 +319,32 @@ def serve( - RequestId: inject / propagate ``X-Request-Id``; close to the router so the id propagates through every header + log line. - Router: the application dispatcher. + + ``HTTP3_ENABLED=1`` was planned to bind a UDP/QUIC listener in + addition to the TCP/TLS one, but the flare reactor-side h3 driver + (the one ``tick_http3_once`` / ``pump_http3_handler_once`` / + ``emit_http3_response`` test helpers expose) is not yet integrated + into ``serve()`` at the pinned commit, so binding without driving + the socket would silently accept no packets. We surface a clear + error instead. See ADR-003 follow-up 6. """ + # Guard for ADR-003 #6: HTTP3_ENABLED was wired to bind_with_http3 + # in an earlier iteration, but at the pinned flare commit the + # reactor's main ``serve()`` does not drive the h3 UDP loop + # (``tick_http3_once`` / ``pump_http3_handler_once`` are test-only + # entry points). The HTTP/3 listener exists on ``bind`` but never + # processes datagrams. Refuse loudly rather than bind a port that + # silently accepts nothing -- a flapping h3 path is worse than a + # documented "pending upstream integration". + if getenv("HTTP3_ENABLED").byte_length() > 0: + raise Error( + "opengateway: HTTP3_ENABLED is set but flare's reactor-" + "side h3 driver is not yet integrated into serve() at the " + "pinned commit. ADR-003 follow-up 6. Disable HTTP3_ENABLED " + "(TLS over TCP still works) or upgrade opengateway's flare " + "pin once the reactor integration lands upstream." + ) + var router = Router() router.get("/health", health) router.post("/v1/chat/completions", chat_completions)