fix(kosong): conservative max_tokens fallback for custom Anthropic-compatible endpoints#2066
fix(kosong): conservative max_tokens fallback for custom Anthropic-compatible endpoints#2066Yorha9e wants to merge 2 commits into
Conversation
…mpatible endpoints Custom Anthropic-compatible endpoints (e.g., Volcano Ark, DeepSeek) often enforce a lower output token limit than the 128000 FALLBACK_MAX_TOKENS designed for official Claude models. This causes immediate 400 rejections for any non-Claude model on these endpoints. Two-layer fix: 1. Conservative fallback: resolveDefaultMaxTokens now accepts an optional baseUrl parameter. When the endpoint is custom (non-Anthropic) and the model is not a recognized Claude, the fallback drops from 128000 to 32768 (the most common ceiling among compatible providers). 2. Error-driven recovery: when a 400 response mentions a max_tokens limit, parseMaxTokensLimit extracts the provider's declared ceiling, and the v2 request loop clamps the completion budget and retries once. The discovered limit is cached per model so subsequent requests start pre-clamped. Closes MoonshotAI#2062
🦋 Changeset detectedLatest commit: ab120cd The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 026cf88706
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| // volcano / ark: "expected a value <= 32768" | ||
| /max_tokens[\s\S]{0,200}expected a value\s*<=\s*(\d+)/i, | ||
| // generic: "max_tokens must be at most N" / "max_tokens cannot exceed N" | ||
| /max_tokens[\s\S]{0,200}(?:must be at most|cannot exceed|maximum value[^\d]*)\s*(\d+)/i, |
There was a problem hiding this comment.
When an Anthropic-compatible provider returns the already-recognized wording max_tokens must be less than or equal to 4096 (there are existing normalizeAPIStatusError fixtures for this shape in both packages), none of these new patterns match, so parseMaxTokensLimit() returns null and the v2 retry path never clamps or resends the request. Please include the less than or equal to / bare <= form here, and mirror the fix in the duplicate legacy parser, so this recovery covers that common max_tokens-limit response.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in ab120cd — parseMaxTokensLimit now recognizes the less than or equal to wording and the bare <= form, mirrored in both packages/agent-core-v2/src/kosong/contract/errors.ts and the legacy packages/kosong/src/errors.ts, with test cases added in both packages.
1 participant
Related Issue
Resolve #2062
Problem
See linked issue.
What changed
Two-layer fix in both v1 (
packages/kosong) and v2 (packages/agent-core-v2):1. Conservative fallback for custom endpoints
resolveDefaultMaxTokensnow accepts an optionalbaseUrlparameter. When the endpoint is custom (non-api.anthropic.com) and the model isn't a recognized Claude, the fallback drops from 128000 to 32768 — the most common output ceiling among Anthropic-compatible providers. ExplicitdefaultMaxTokens(frommax_output_sizein config) and known Claude ceilings are unaffected.2. Error-driven recovery
When a 400 response mentions a
max_tokenslimit (e.g.,"expected a value <= 32768"), the newparseMaxTokensLimitfunction extracts the provider's declared ceiling. In the v2 request loop, the completion budget is clamped and the request retried once. The discovered limit is cached per model, so subsequent requests start pre-clamped — each model pays for at most one rejected request.Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.