Skip to content

perf: skip redundant deep copy in CostPolicy.apply_token_limit - #126

Open
shrdgn wants to merge 1 commit into
mainfrom
claude/eliminate-redundant-deepcopy
Open

perf: skip redundant deep copy in CostPolicy.apply_token_limit#126
shrdgn wants to merge 1 commit into
mainfrom
claude/eliminate-redundant-deepcopy

Conversation

@shrdgn

@shrdgn shrdgn commented Jul 31, 2026

Copy link
Copy Markdown
Owner

What & why

Every panel-member call (openfusion/panel.py:_call_member) and every judge synthesis call (openfusion/synthesize.py:synthesize) deep-copied the request body twice: once by the caller to isolate it from the shared original, then a second time inside CostPolicy.apply_token_limit (openfusion/cost.py).

This is pure wasted CPU/memory that scales with panel size and debate rounds, and is invisible in normal tests since correctness isn't affected — only latency/memory for large multi-turn or multimodal (e.g. base64 image) request bodies.

Fix: add an already_isolated: bool = False keyword-only flag to apply_token_limit. When True, the caller is asserting it already owns an exclusive copy of body, so the method mutates and returns it in place instead of deep-copying again. Default behavior — and the existing "does not mutate the original" contract relied on by server.py's two call sites — is unchanged.

How it was tested

  • ruff check . passes
  • pytest -q passes (476 passed, no live network)
  • New behavior has a test (tests/test_cost.py: identity check that no extra copy is made, in-place mutation, and that reject_over_limit still raises correctly under already_isolated=True)
  • Docs updated if config / request surface / defaults changed — n/a, no public request/config surface changed
  • No secrets, prompts, or response bodies added to logs or metrics
  • Quality/cost claims backed by a reproducible bench/run.py number (if applicable) — n/a, this is a copy-elimination perf fix, not a quality/cost tradeoff; behavior is bit-for-bit identical, verified by the existing test suite

Notes for reviewers

Only the two hot-path callers that provably already own an isolated copy at the call site were switched to already_isolated=True:

  • panel.py:_call_member — deep-copies request_body into body and already mutates it (.pop("model"), .pop("stream")) before calling apply_token_limit.
  • synthesize.py:synthesize — deep-copies request_body into judge_body and already mutates it (sets messages, pops model, strips tool fields) before calling apply_token_limit.

server.py's two callers (policy.apply_token_limit(body, ...)) were deliberately left on the default (deep-copying) path since they call it directly on the request body without a prior isolating copy of their own.


Generated by Claude Code

Every panel-member call and every judge synthesis call deep-copied the
request body twice: once by the caller to isolate it, then again inside
apply_token_limit. This scales with panel size and debate rounds and adds
needless CPU/memory work for large multi-turn or multimodal request
bodies. Add an already_isolated flag so callers that already own an
exclusive copy can skip the second one; default behavior (and the
existing "does not mutate" contract) is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants