fix(gateway): deduplicate within a payload without requiring history (OX-M1) - #41
Merged
Conversation
Audit OX-M1, DECISIONS §67. runSessionDedupStage treated within-payload repetition as a side condition of cross-turn matching. The dedup branch was gated on previousBlockHashes.has(hash), and the stage returned early unless that set was non-empty -- so on turn 1 three identical blocks in one payload all survived. The README's savings table said "the same block repeated within one payload -> saves" with no qualifier. It was true from turn 2. The gate was doing no safety work there. `recoverable: true` means an intact copy survives in the SAME outbound payload, which rule 3 guarantees by preserving the first occurrence -- a claim verifiable from the payload alone. previousBlockHashes answers a different question (is this content old?) and was deciding something it does not bear on. Two gates relaxed by the same reasoning: shouldAttemptDedup now also fires on within-payload repetition (the occurrence map moved above the early return), and the per-item branch fires on seenInEarlierTurn || repeatsInThisPayload. Rule 3 and the isRecoverable computation are unchanged, which is what keeps sole-copy elision out of the new path -- the first occurrence is always preserved, so anything elided under repeatsInThisPayload is recoverable by construction. DECISIONS §16/§41 untouched. A SOLE copy elided across turns is still elided with recoverable: false, still scored in full by DriftTracker, and still fails the gate. The ordinary conversational shape still saves 0 bytes and still falls back -- the number the README leads with has not moved, and its pin is unchanged. Measured over real sockets, block repeated three times: turn 1 8,459 sent / 8,459 forwarded -> saves turn 2 saves -> saves cross-turn sole 0 bytes, falls back -> unchanged turn 1, distinct no saving -> no saving The last row is the control that keeps the first honest: turn 1 did not become "always saves". This stage never runs on the CLI or MCP paths, so the corpus cannot see the change; the instrument is the Gateway integration suite. Verified: typecheck, lint and build clean, 89 files / 819 tests. The turn-1 case was confirmed failing against the unfixed tree first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Audit OX-M1. DECISIONS §67. Third Lane B item.
The defect
runSessionDedupStagetreated within-payload repetition as a side condition of cross-turnmatching. The dedup branch was gated on
previousBlockHashes.has(item.contentHash), and the stagereturned early unless that set was non-empty — so on turn 1, three identical blocks in one payload
all survived.
The README's savings table said "the same block repeated within one payload → saves", with no
qualifier. It was true from turn two.
Why the gate was wrong, not merely conservative
recoverable: truemeans an intact copy survives elsewhere in the same outbound payload. Rule 3guarantees that by preserving the first occurrence and eliding the ones after it. That claim is
verifiable from the payload alone — the model has seen the content, in this request, so the marker
resolves.
Nothing in it refers to previous turns.
previousBlockHasheswas answering a different question —is this content old? — and its answer was deciding something it does not bear on.
DECISIONS §16 and §41 are untouched. They concern a sole copy elided across turns, where the
consumer is a stateless provider API with no rehydration mechanism, so the marker is deletion rather
than reference. That case is still elided with
recoverable: false, still scored in full byDriftTracker, and still fails the gate.Measured, over real sockets
Block repeated three times in one payload:
The last row is the control that keeps the first honest — turn 1 did not become "always saves". The
third is §41 still holding, and its pin in
gateway-dedup-reality.test.tsis unchanged.The change
Two gates, relaxed by the same reasoning:
shouldAttemptDedupis nowpreviousBlockHashes.size > 0 || hasRepeatedContent. The occurrencemap moved above the early return, since within-payload repetition is by itself a reason to run.
seenInEarlierTurn || repeatsInThisPayload.Rule 3 and the
isRecoverable = survivingHashes.has(hash)computation are unchanged, which iswhat keeps sole-copy elision out of the new path: the first occurrence is always preserved, so
anything elided under
repeatsInThisPayloadis recoverable by construction.What this does not establish
copy per payload, seen before — still saves nothing and still falls back. That is the number the
README leads with and it has not moved.
plan.stageIdsand don't include it. The instrument is the Gateway integration suite.code does. Whether real agent traffic repeats a block inside a single payload often enough to
matter was not measured, and the honest framing stays the README's: use Gateway mode for
interception, validation and metrics, not for compression.
Verification
npm run typecheck,npm run lint,npm run buildandnpx vitest runall pass: 89 files /819 tests. The turn-1 case was confirmed failing against the unfixed tree first, with both
pre-existing pins passing throughout.
🤖 Generated with Claude Code