Skip to content

Add docs-agent arm: agent-maintained notes baseline for swechat-plan - #161

Draft
AbdulWasih05 wants to merge 3 commits into
Autoloops:mainfrom
AbdulWasih05:docs-agent-arm
Draft

Add docs-agent arm: agent-maintained notes baseline for swechat-plan#161
AbdulWasih05 wants to merge 3 commits into
Autoloops:mainfrom
AbdulWasih05:docs-agent-arm

Conversation

@AbdulWasih05

Copy link
Copy Markdown

Extends the swechat-plan benchmark (#139) with a docs-agent arm: an agent maintains its own notes.md across the same replayed prior sessions with no Greplica involvement, and retrieves by reading those notes at task time. The existing docs arm (graph export) is kept for contrast, so the four arms form a 2x2 over memory acquisition and retrieval format (see the new evals/swechat-plan/README.md).

Design choices

  • build-memory.ts gains a --memory graph|docs-agent mode flag instead of a sibling script, so the session-replay scaffolding (transcript filtering, edit reconstruction, working-tree staging) stays byte-identical across arms and the acquisition comparison is not confounded.
  • run.ts adds the runner additively: seeds copy to agent-notes/notes.md, tool guards fall through to the existing non-greplica branch, and audit fields mirror the docs arm (agent_notes_*, plus a cross-arm access violation check). Judge, scoreJudgeChecks, and cost estimation are untouched.
  • Notes snapshots are stored per step in notes-seeds/ with a manifest (order, final snapshot, per-step size stats), so acquisition can be inspected over time. Build transcripts are scanned post-hoc for greplica/codegraph usage; hits are recorded as forbidden_tool_warnings (zero across all committed seeds).

Disclosed limitations

  • docs-agent vs docs is confounded (multi-file export vs single notes file); primary contrasts are docs-agent vs greplica and vs baseline. Details in the README.
  • Build-time asymmetry: graph-mode update agents may reuse memory via greplica graph context; the docs-agent analog is re-reading its own notes. Disclosed as a scale limitation rather than patched.

Seeds in this PR

  • swechat-rudel-project-trends-grouping and swechat-gemini-voyager-sync-auth-bug (each 1 bootstrap + 3 session updates, gpt-5.5, roughly 1 to 1.4 million tokens per case). A third case (iptvnator) will follow.
  • Caveat found while validating the gemini-voyager seeds: that case's prior session 1ead0508 replays to a patch identical to the held-out fix commit 83a8c08e (same checkpoint 3e715f8184f4, created ~1.4 s before the held-out session), so all memory arms, not just this one, absorb the held-out answer for that case. Unique in the corpus by a scan of all 10 case.json files. Will report separately with full evidence; seeds are committed as faithful acquisition artifacts.

Verification

  • tsc clean; --fixture-only smokes pass for both memory modes and the new runner; --memory graph regression smoke unchanged. A full end-to-end docs-agent planning run is pending (needs a Linux/WSL environment for the POSIX tool guards).

What follows

This is the first of a planned sequence: per-query retrieval-quality measurement (scoring each arm's returned context against relevance judgments, reusing evals/lib/search-retrieval-scoring.ts), a staleness probe, and aggregation across cases with repeats and judge-agreement for variance. Several choices here exist to support those: per-claim path citations in the notes prompt (so notes can be scored on file routing), per-step snapshots, and byte-identical scaffolding across arms.

Note: #160 explores the same roadmap item independently (parallel exploration). The two designs conflict mechanically (both add a fourth runner and a notes-seeds convention); flagging for the maintainer rather than resolving unilaterally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant