feat(server): OpenAI-compatible logprobs for chat and legacy completions - #224
feat(server): OpenAI-compatible logprobs for chat and legacy completions#224Artemowka22 wants to merge 2 commits into
Conversation
Adds the engine half of OpenAI logprobs support. SamplingParams gains logprobs/top_logprobs; the sampler computes log_softmax over the PRE-temperature logits (raw model distribution, so temperature/top-k/top-p do not change reported values), gathers the chosen token and batch-max top-k, and ships CPU copies covered by the existing copy_done_event. The scheduler attaches per-request values (cut to each request's own top_logprobs) to DetokenizeMsg; the detokenizer builds a neutral entry (token text via single-id decode + UTF-8 bytes, so clients can reassemble partial-UTF-8 pieces) onto UserReply.logprobs. Zero cost when off: no row asked -> no mask tensor, no log_softmax, ForwardOutput carries None. Message fields default to None, so old and new peers interoperate. Stop-string trimming can hide final visible text; entries still cover every sampled token.
API half of logprobs support. Chat: logprobs + top_logprobs (0..20) yield
choice.logprobs.content entries {token, logprob, bytes, top_logprobs[]},
streaming and non-streaming. Completions: the legacy integer field (0..5) yields
{tokens, token_logprobs, top_logprobs, text_offset} with absolute offsets across
the stream; echo+logprobs stays rejected (prompt logprobs need prefill logits
and are out of scope here).
The protocol-neutral event layer carries entries on ContentDelta (a list -- parser
buffering can release several tokens' text in one delta); entries always
accumulate on GenResult for the non-streaming path, and reasoning/tool-call
buffering carries pending entries onto the next content delta. Formatting lives
in server/logprobs.py; wire compatibility follows the OpenAI shapes.
|
Fresh compatibility check from current |
|
RTX 5090 / torch 2.11 cu130 microbenchmark on Qwen3.8 vocab size 248,320: the PR implementation shape ( |
|
One correctness issue before merge: streaming semantic parsing currently misattributes hidden-token logprobs to later visible content. In |
2 participants
Summary
Implements
logprobsfor sampled tokens across the whole pipeline; today/v1/completionsrejects the field ("logprobs is not supported") and/v1/chat/completionssilently swallows it (extra="allow").logprobs: true+top_logprobs: 0..20→ each choice carrieslogprobs.content[]entries{token, logprob, bytes, top_logprobs[]}, streaming and non-streaming.logprobs: 0..5→{tokens, token_logprobs, top_logprobs, text_offset}.echotogether withlogprobsstays rejected (prompt logprobs need prefill logits and are out of scope here).Design
log_softmaxover the pre-temperature logits, so temperature/top-k/top-p do not change what is reported (matches vLLM's default) and greedy eval harnesses get the true model distribution.log_softmax+topk+ a small D2H copy that rides the existingcopy_done_event.DetokenizeMsg.chosen_logprob/top_ids/top_logprobs,UserReply.logprobs), all defaulting to None — wire-compatible with older peers.tokenizer.decode([id])); thebytesfield carries UTF-8 so clients can reassemble partial-UTF-8 pieces, same trade-off OpenAI documents.Why
Any evaluation gate worth trusting (teacher-forced agreement/KL against a reference checkpoint, perplexity tracking of quantized variants) needs token logprobs from the OpenAI endpoint; with the radix prefix cache, per-position 1-token continuation calls make teacher-forced scoring practical without echo support.
Test plan
tests/engine/test_sample_logprobs.py) and the entry builder (tests/tokenizer/test_logprobs_entry.py)tests/server/test_logprobs_api.py) — no GPU, no weights, no network