feat(server): add per-request table log for chat completions - #8
Open
cdryzun wants to merge 1 commit into
Open
Conversation
Print one aligned table row per /v1/chat/completions call to stdout, covering sequence, time, model, mode, status, TTFB, output tokens, tokens-per-second and total duration, so operators can spot slow or throttled requests at a glance. - measure real time-to-first-token from request entry via content-aware SSE parsing; count streamed output tokens by UTF-8 runes and use the upstream usage field when present - emit a table row on error paths (bad request, upstream parse error, all-accounts-unavailable) with the real HTTP status code - keep the SSE stream passthrough unchanged for the client - silence /healthz and avoid duplicate logging of chat completions - add unit tests for SSE token counting, TTFB baseline and model parsing - Dockerfile: point GOPROXY at goproxy.cn to fix module downloads
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a table-formatted request log for
/v1/chat/completions. Each call prints one row to stdout with sequence number, time, model, mode (stream/sync), status, time-to-first-token, output token count, tokens/sec and total duration. This makes slow or throttled requests visible without digging through raw nginx access logs.Why
The service currently logs only its startup line and any upstream transport errors. There is no per-request visibility into latency or token usage, which makes it hard to tell when a specific model or account is underperforming. The table gives an operator a quick per-request picture.
What changed
internal/server/logging.go(new): achatStatsReaderthat wraps the upstream SSE stream. It counts streamed output tokens (by UTF-8 runes) and records the time to the first content frame. AlogChatRowhelper prints the aligned table row.internal/server/handler.go: wires the logger into the stream and non-stream paths. Error paths (bad request, upstream parse failure, all accounts unavailable) now also emit a row with the real HTTP status instead of being silent./healthzis excluded, and chat completions no longer duplicate the plain one-line log.internal/server/logging_test.go(new): unit tests for SSE token counting (includingreasoning_content), usage-field precedence, TTFB baseline, and model parsing.Dockerfile: setsGOPROXYtogoproxy.cnso the container build can fetch modules behind the GFW. Unrelated to logging, included separately for completeness.Notes
completion_tokensfrom the upstream usage field. The two are not directly comparable.