Split openai worker into per-engine backends (vllm/sglang/llama) - #88
Merged
Conversation
…e) — CON-1612 Make the openai worker consume its per-worker config from env, each defaulting to the previous hardcoded value, so base-image can bake per-backend values: - MODEL_LOG (was hardcoded /var/log/portal/vllm.log) — the original latent bug: the worker tailed a hardcoded path and ignored MODEL_LOG. Now os.environ.get(MODEL_LOG,...). - MODEL_HEALTH_ENDPOINT (the established env var; the framework's Backend.healthcheck_url default reads it and the vLLM serverless templates set it) — default /health. A path resolves against MODEL_SERVER_URL; a full URL is used as-is (worker venv is fresh, so aiohttp is current). - MODEL_LOAD_LOG_MSG / MODEL_ERROR_LOG_MSGS / MODEL_INFO_LOG_MSGS — newline-delimited env (each line stripped), vLLM defaults; lets an image supply per-backend log grammar (fixes on_error detection being dead for non-vLLM backends under the shared worker). - READINESS + READINESS_TIMEOUT + HEALTHCHECK_PROBE_TIMEOUT (vast-cli WS2) — feature- detected via inspect.signature(WorkerConfig): only passed when the installed vastai supports them, so a new worker on an OLD framework degrades to log mode, not TypeError. Timeout floats safe-parse AND require positive-finite (0/neg/inf/nan -> default, since they feed asyncio timeouts). Default READINESS=logs (current behaviour). Backwards-compat / merge note: absent-env reproduces today's behaviour, and READINESS stays 'logs' until an image/template opts in. NOT a pure no-op merge, though: reusing MODEL_HEALTH_ENDPOINT *activates* an env the vLLM templates already set (previously the worker hardcoded /health and ignored it). Audited: all deployed openai serverless templates (010/011/012) set http://127.0.0.1:18000/health == the prior hardcoded value, so it is a no-op for the CURRENT fleet — but the health URL gates readiness even in log mode, so any future openai template MUST set MODEL_HEALTH_ENDPOINT to :18000/health (keep this audit as a merge gate). Scoped to openai only.
…enchmark A single template must serve both on-demand and serverless. The on-demand recommended templates set only the engine-specific model var (VLLM_MODEL / SGLANG_MODEL / LLAMA_MODEL), never MODEL_NAME, so the serverless benchmark had no model id and raised. Resolve it with precedence MODEL_NAME -> VLLM_MODEL -> SGLANG_MODEL -> LLAMA_MODEL: an explicit MODEL_NAME still wins (templates can override), else the served-model id is recovered from the engine var. vLLM/SGLang serve under exactly that value (no --served-model-name rewrite) so it matches /v1/models; llama.cpp ignores the request model field. Done in the worker, not a boot script, because the engine var is present in the worker env on BOTH serverless launch paths (onstart-curl and base supervisor), whereas a boot-computed value never reaches the onstart-curl worker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Health-gated readiness should backstop a genuinely-wedged worker, not gate a slow-but- valid cold start. 300s trips on large-model cold starts (weight download + load + graph capture) on slower hosts, forcing operators to tune it just to get serving working. 1800s (30 min) covers realistic big-model starts so it works out of the box; lower it only to get faster failure detection. Still env-overridable via READINESS_TIMEOUT. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…framework Matches the vast-cli default bump: SGLang /health can do real work (~20s internally), so a 10s per-probe timeout risked a false regression error on a stall once loaded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
robballantyne
changed the base branch from
feat/openai-env-driven-config
to
main
August 4, 2026 10:11
robballantyne
force-pushed
the
feat/split-openai-per-engine
branch
2 times, most recently
from
August 4, 2026 10:49
49e6400 to
29b4140
Compare
Contributor
Author
|
Staging test templates, pinned to this branch (on-demand · serverless, differ only by
|
…-1612
pyworker proxies an OpenAI-compatible API, and vLLM, SGLang and llama.cpp all
expose that same surface, so the worker logic is identical across them. Extract
it into workers/openai/core.py (EngineDefaults + run()) and add thin
workers/{vllm,sglang,llama} adapters that pass only per-engine default log
grammar. This gives each engine its own BACKEND without triplicating the ~145
lines of shared logic.
BACKEND=openai becomes a backwards-compatible ALIAS for vllm: workers/openai/
worker.py runs the vLLM worker directly (import for side effect), so there is one
definition of the vLLM defaults and no second copy to drift. Existing templates
that declare openai keep working, provided they run a pyworker new enough to
contain this split.
On startup the worker prints which engine is serving ("Using worker backend:
vllm"), sourced from the engine identity, so BACKEND=openai reports the real
engine (vllm) and notes the alias rather than printing "openai".
Every default remains env-overridable: the image is the only layer version-locked
to the engine build, so engine/version-specific config (log path, health
endpoint, load/error/info log grammar) is image-supplied and worker-read.
The worker runs in log mode. Health-gated readiness depends on a framework change
that has not landed, so it is left out rather than feature-detected — keeping the
worker minimal instead of carrying speculative forward-compat.
The demo client (workers/openai/client.py) is shared, not copied per engine.
robballantyne
force-pushed
the
feat/split-openai-per-engine
branch
from
August 4, 2026 12:11
29b4140 to
e528b9d
Compare
robballantyne
marked this pull request as ready for review
August 4, 2026 12:16
Contributor
Author
|
@LucasArmandVast ready when you have a chance — the split we talked about. Test templates are in the comment above. |
LucasArmandVast
approved these changes
Aug 21, 2026
robballantyne
added a commit
to vast-ai/base-image
that referenced
this pull request
Aug 24, 2026
vast-ai/pyworker#88 merged, so workers/ now holds vllm, sglang and llama, with workers/openai/worker.py reduced to `import workers.vllm.worker` as a backwards-compatible alias. This image was baking the alias because the real name did not exist yet; it does now. Functionally identical either way — openai imports the same module and gets the same EngineDefaults — so this is about naming rather than behaviour. `vllm` is the honest name and it matches what sglang and llama.cpp already bake, instead of leaving one image on the legacy alias for no reason. The one cost: this image now REQUIRES a pyworker containing the split, which is always true because pyworker.sh fetches main at every boot, and only matters if that split were reverted. BACKEND IS ALSO REMOVED FROM THE QA CELL'S extra_env, which is the more interesting half. The image baking BACKEND is the entire claim of the serverless-enablement work — an image that self-configures rather than requiring every template to know the engine's wiring — and passing it from the caller overrode the bake with docker -e, testing the template and leaving the thing under test unexercised. Now the cell proves the bake. It also fails loudly if the bake is ever missing: start_server.sh resolves workers/$BACKEND/worker.py, an empty BACKEND matches nothing, no worker starts, and vllm.d/20-serverless-pyworker reds on "nothing is listening on :3000". Note for the next run: MODEL_LOAD_LOG_MSG, which this image has baked since the serverless branch was written, was INERT until #88 — the old worker carried it as a Python constant. core.py now reads it through os.environ, so the image's value is load-bearing for the first time. The same is true of MODEL_LOG, which the QA cell passes and the old worker ignored. Evidence: lint baseline CLEAN; linter and template_manager suites pass.
robballantyne
added a commit
to vast-ai/base-image
that referenced
this pull request
Aug 24, 2026
…ith the gates to prove it (#225) Bakes BACKEND, MODEL_LOAD_LOG_MSG and EXPOSE 3000 on the four engine images so each self-configures for serverless — and, because two of them had NO QA GATE AT ALL, builds the gates that prove it. All three engines are now verified on live GPUs. The bake is the claim: an image that configures its own serverless wiring rather than requiring every template to know the engine's internals. It became real when vast-ai/pyworker#88 landed the per-engine split (workers/{vllm,sglang,llama} over a shared openai/core.py, with openai kept as a backwards-compatible alias). EXPOSE 3000 is load-bearing rather than tidy: the platform injects VAST_TCP_PORT_<n> only for MAPPED ports and the SDK looks that variable up unguarded, so with 3000 unmapped the worker dies with KeyError before it binds or benchmarks. TWO IMAGES HAD NO GATE. build-sglang.yml and build-llama-cpp.yml had no qa job at all — merge-manifests depended only on build, so an image that COMPILED was promoted whether or not it served. Neither had a QA template, which is why. Both now have the gate vLLM has. PROVEN ON HARDWARE. Each engine resolved BACKEND from its own bake, ran the matching per-engine worker, and wrote a benchmark score during that run: vLLM 405.87, SGLang 3256.59, llama.cpp 222.23. Each serverless cell differs from its standard cell by SERVERLESS=true and nothing else — MODEL_NAME and MODEL_LOG were removed deliberately so the cell tests the image's resolution rather than our values. SIX DEFECTS FOUND BY RUNNING IT, four of them mine: - base/13's pinned PATH could not find supervisorctl, latent on every external image whose upstream base does not ship it in a system directory; vLLM passed only by accident of its base. Gated now by L075. - Cells sharing a template file shared ONE throwaway template and raced to delete it: four cells produced two ids, one was deleted at 14:12:51, and the cell still launching on it failed every attempt after that instant. - The serverless cell needed three extra env vars, so it was not testing the claim. - create.py <file.yml> raised TypeError unconditionally — run() splatted set_filters into a signature that did not accept it. - cap:embeddings reported a 501 as a failed capability, when that is the engine answering discovery correctly. - A --name-suffix built from an unset variable would silently restore the shared name. THE ENGINES DO NOT IMPLEMENT THE SAME CONTRACT, measured on first contact and recorded in docs/invariants.md: vLLM refuses an unknown model, a malformed body and an over-long max_tokens with 4xx; SGLang misses the first; llama.cpp misses all three. Those are declared deviations (ADR 0031 decision 6a) — reported every run, non-blocking, and automatically a violation again the day they stop reproducing, so a declaration cannot outlive the defect. llama.cpp's malformed-body bound is explicitly recorded as WEAK and accepted knowingly rather than argued harmless. Evidence: all cells green across vLLM (4), SGLang (4) and llama.cpp (2); 499 + 419 tests; lint baseline CLEAN throughout; L072/L073 observed FIRING on each new template before completion, so the rules demonstrably see the new images. Known gaps, stated: base/28 still skips under serverless; the contract checker is three copies differing only in one ENGINE block with no automated drift detection; llama.cpp gets one QA cell because its matrix is over base images and there is exactly one; and the promoted multi-arch manifest is never checked against the digest QA actually tested.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splits the single
openaiworker into per-engine backends, and folds in the env-driven config work from #87 so this is one self-contained change offmain. Supersedes #87.The logic doesn't change. pyworker is a proxy for an OpenAI-compatible API, and vLLM, SGLang and llama.cpp all expose that same surface, so rather than triplicate the worker I've pulled the shared implementation into
workers/openai/core.pyand given each engine a thin adapter that only supplies its own default log grammar.workers/openai/core.py— all the shared logic (EngineDefaults+run()).workers/{vllm,sglang,llama}/worker.py— thin adapters, one perBACKEND.workers/openai/worker.py— now an alias forvllm(it just runs the vllm worker), soBACKEND=openaikeeps working with a single definition of the defaults, no copy to drift.Every default is env-overridable. The image is the only layer pinned to a specific engine build, so the engine- and version-specific values (log path, health endpoint, the load/error/info log lines) are supplied by the image and read here.
The worker runs in log mode, same as today. I've deliberately left health-gated readiness out — it depends on a framework change that hasn't landed, and I'd rather not carry speculative forward-compat in the worker.