Add docs-agent arm: agent-maintained notes baseline for swechat-plan - #161
Draft
AbdulWasih05 wants to merge 3 commits into
Draft
Add docs-agent arm: agent-maintained notes baseline for swechat-plan#161AbdulWasih05 wants to merge 3 commits into
AbdulWasih05 wants to merge 3 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extends the swechat-plan benchmark (#139) with a
docs-agentarm: an agent maintains its ownnotes.mdacross the same replayed prior sessions with no Greplica involvement, and retrieves by reading those notes at task time. The existingdocsarm (graph export) is kept for contrast, so the four arms form a 2x2 over memory acquisition and retrieval format (see the newevals/swechat-plan/README.md).Design choices
build-memory.tsgains a--memory graph|docs-agentmode flag instead of a sibling script, so the session-replay scaffolding (transcript filtering, edit reconstruction, working-tree staging) stays byte-identical across arms and the acquisition comparison is not confounded.run.tsadds the runner additively: seeds copy toagent-notes/notes.md, tool guards fall through to the existing non-greplica branch, and audit fields mirror the docs arm (agent_notes_*, plus a cross-arm access violation check). Judge,scoreJudgeChecks, and cost estimation are untouched.notes-seeds/with a manifest (order, final snapshot, per-step size stats), so acquisition can be inspected over time. Build transcripts are scanned post-hoc for greplica/codegraph usage; hits are recorded asforbidden_tool_warnings(zero across all committed seeds).Disclosed limitations
docs-agentvsdocsis confounded (multi-file export vs single notes file); primary contrasts aredocs-agentvsgreplicaand vsbaseline. Details in the README.greplica graph context; the docs-agent analog is re-reading its own notes. Disclosed as a scale limitation rather than patched.Seeds in this PR
swechat-rudel-project-trends-groupingandswechat-gemini-voyager-sync-auth-bug(each 1 bootstrap + 3 session updates, gpt-5.5, roughly 1 to 1.4 million tokens per case). A third case (iptvnator) will follow.1ead0508replays to a patch identical to the held-out fix commit83a8c08e(same checkpoint3e715f8184f4, created ~1.4 s before the held-out session), so all memory arms, not just this one, absorb the held-out answer for that case. Unique in the corpus by a scan of all 10 case.json files. Will report separately with full evidence; seeds are committed as faithful acquisition artifacts.Verification
tscclean;--fixture-onlysmokes pass for both memory modes and the new runner;--memory graphregression smoke unchanged. A full end-to-enddocs-agentplanning run is pending (needs a Linux/WSL environment for the POSIX tool guards).What follows
This is the first of a planned sequence: per-query retrieval-quality measurement (scoring each arm's returned context against relevance judgments, reusing
evals/lib/search-retrieval-scoring.ts), a staleness probe, and aggregation across cases with repeats and judge-agreement for variance. Several choices here exist to support those: per-claim path citations in the notes prompt (so notes can be scored on file routing), per-step snapshots, and byte-identical scaffolding across arms.Note: #160 explores the same roadmap item independently (parallel exploration). The two designs conflict mechanically (both add a fourth runner and a notes-seeds convention); flagging for the maintainer rather than resolving unilaterally.