Skip to content

feat: add spinloop gateway, a fleet's OpenAI-compatible front door - #170

Draft
outofcoffee wants to merge 4 commits into
mainfrom
gateway
Draft

feat: add spinloop gateway, a fleet's OpenAI-compatible front door#170
outofcoffee wants to merge 4 commits into
mainfrom
gateway

Conversation

@outofcoffee

Copy link
Copy Markdown
Collaborator

Serves a fleet under one OpenAI-compatible endpoint: an agent's Spinloop names the gateway's address in its FLEET, and the agent needs only the gateway's token.

Summary

  • spinloop gateway command and internal/gateway: one OpenAI-compatible endpoint in front of a fleet.yaml. /v1/models lists what the fleet is running; completion requests are routed with the fleet's own selector; a node is woken from its own Spinloop source when nothing is serving, and the request is held until the engine answers; the caller's authorisation is swapped for the engine key the node's fleet entry names
  • A new top-level wake: on|off setting in the fleet file decides whether routing may start an engine; off refuses without starting anything, naming the node and the spinloop fleet start command that would
  • A Spinloop's FLEET may now name an endpoint (a URL) as well as a fleet file: the launch is pointed at it, with the OpenAI-compatible prefix added where needed, and authenticates with the endpoint's token through the launch's existing key chain. spinloop fleet route answers an endpoint without querying or starting a node
  • A node wake now carries the Spinloop's BASEURL into the pushed deploy config, translated to the engine's bind exactly as a local serve does. Without it a node with no preset woke onto the engine's own default bind — loopback for llama.cpp — which nothing but the node itself could reach
  • examples/gateway-docker/: two daemon nodes and two gateways over them (one waking, one wake: off) in containers. It is the gateway's integration test, run per PR by a new CI job
  • Docs: docs/commands/gateway.md, the endpoint form of FLEET in docs/spinloop-file.md, the wake setting in docs/commands/fleet.md

Implementation details

  • The gateway is the fleet client wearing a server: selection, waking, readiness and key handling all come from internal/fleet; the handler reuses a fan-out reading taken within the last two seconds rather than fanning out per request
  • The wake config's bind travels in the deploy config's serve args, as a final override layer — the preset's host/port were already dropped for a node target. The Spinloop's bind therefore wins over the preset's exactly as it does for a local serve, and the cloud path is untouched: the cloud still assigns its own port
  • A running engine bound to loopback, without an engine: override taking responsibility for reachability, is never selected — and when it is the only candidate, the failure says so
  • Concurrent requests for the same model wake at most one engine: a request that loses the start to the daemon's 409 takes the node the other one started

@outofcoffee outofcoffee added documentation Improvements or additions to documentation enhancement New feature or request go Pull requests that update go code labels Sep 6, 2026
A new `spinloop gateway` command serves a fleet.yaml behind one
OpenAI-compatible endpoint: it answers /v1/models with what the fleet
is running, routes completion requests with the fleet's own selector,
wakes a node from its own Spinloop source when nothing is serving,
swaps the caller's authorisation for the engine key its fleet entry
names, and honours a new top-level `wake: on|off` setting in the
fleet file.

A Spinloop's FLEET may now name an endpoint (a URL) as well as a fleet
file: the launch is pointed at it with the OpenAI-compatible prefix
added where needed, authenticating with the endpoint's token through
the launch's existing key chain, and `spinloop fleet route` answers
it without querying or starting a node.

A node wake now carries the Spinloop's own BASEURL into the pushed
deploy config, translated to the engine's bind exactly as a local
serve translates it. Without it a node with no preset woke onto the
engine's default bind - loopback for llama.cpp - which the gateway
then correctly refused to route to.

examples/gateway-docker/ brings up two daemon nodes and two gateways
(one waking, one with wake: off) in containers and is the gateway
integration test, run per PR by a new CI job.
The status branch sources the serving facts (runner, model id, served name) from the environment's deploy config — the same source the stats reply reads — not the daemon, whose model is the on-disk weights path, not a name a router can match against. The fleet and remote views now name a model the same.
A running remote environment reported its state and model but no engine
address: the control plane's published base url was dropped in the status
mapping, so the gateway could list a remote node's model and then refuse to
route a request to it. The status now carries the engine's host, port and
path from that address, and routing resolves a reported engine host in place
of the fleet file's (which names a remote node by environment alone).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request go Pull requests that update go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant