perf: skip redundant deep copy in CostPolicy.apply_token_limit - #126
Open
shrdgn wants to merge 1 commit into
Open
perf: skip redundant deep copy in CostPolicy.apply_token_limit#126shrdgn wants to merge 1 commit into
shrdgn wants to merge 1 commit into
Conversation
Every panel-member call and every judge synthesis call deep-copied the request body twice: once by the caller to isolate it, then again inside apply_token_limit. This scales with panel size and debate rounds and adds needless CPU/memory work for large multi-turn or multimodal request bodies. Add an already_isolated flag so callers that already own an exclusive copy can skip the second one; default behavior (and the existing "does not mutate" contract) is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Every panel-member call (
openfusion/panel.py:_call_member) and every judge synthesis call (openfusion/synthesize.py:synthesize) deep-copied the request body twice: once by the caller to isolate it from the shared original, then a second time insideCostPolicy.apply_token_limit(openfusion/cost.py).This is pure wasted CPU/memory that scales with panel size and debate rounds, and is invisible in normal tests since correctness isn't affected — only latency/memory for large multi-turn or multimodal (e.g. base64 image) request bodies.
Fix: add an
already_isolated: bool = Falsekeyword-only flag toapply_token_limit. WhenTrue, the caller is asserting it already owns an exclusive copy ofbody, so the method mutates and returns it in place instead of deep-copying again. Default behavior — and the existing "does not mutate the original" contract relied on byserver.py's two call sites — is unchanged.How it was tested
ruff check .passespytest -qpasses (476 passed, no live network)tests/test_cost.py: identity check that no extra copy is made, in-place mutation, and thatreject_over_limitstill raises correctly underalready_isolated=True)bench/run.pynumber (if applicable) — n/a, this is a copy-elimination perf fix, not a quality/cost tradeoff; behavior is bit-for-bit identical, verified by the existing test suiteNotes for reviewers
Only the two hot-path callers that provably already own an isolated copy at the call site were switched to
already_isolated=True:panel.py:_call_member— deep-copiesrequest_bodyintobodyand already mutates it (.pop("model"),.pop("stream")) before callingapply_token_limit.synthesize.py:synthesize— deep-copiesrequest_bodyintojudge_bodyand already mutates it (setsmessages, popsmodel, strips tool fields) before callingapply_token_limit.server.py's two callers (policy.apply_token_limit(body, ...)) were deliberately left on the default (deep-copying) path since they call it directly on the request body without a prior isolating copy of their own.Generated by Claude Code