feat(eval): agent-maintained notes arm for the SWE-chat planning benchmark

[chiragbiradar]

Follows up on the Discord thread about the docs-folder comparison for the roadmap benchmark item — builds directly on the SWE-chat planning harness from #139.

What this adds

A fourth benchmark arm answering "how does Greplica compare with a docs folder maintained by an agent": alongside baseline, greplica, and the graph-export docs arm, --runner notes has the planning agent navigate from a docs/agent-notes.md that an agent maintained across the same prior sessions used to build the Greplica memory seeds.

That gives the benchmark both docs baselines:

• docs — Greplica's knowledge exported to markdown (structured memory, flat presentation)
• notes — what developers actually do today: an agent keeping its own notes file session by session, no Greplica involved

How it works

evals/swechat-plan/build-notes.ts (mirror of build-memory.ts): replays the case's prior sessions in chronological order using the same transcript-to-markdown conversion and reconstructed session edits, but instead of writing Greplica proposals the agent maintains a single agent-notes.md (bootstrap pass on the base snapshot first, then one update per session). The final notes are stored as a checked-in notes-seeds/agent-notes.md plus a manifest with per-step token/size stats, parallel to memory-seeds/.

One subtlety: the replay machinery does git reset --hard && git clean -fd between sessions, which would delete an untracked notes file — so notes are committed into the replay snapshots after each step.

run.ts: --runner notes copies the case's notes seed to docs/agent-notes.md in the target repo, prompts the agent to use it as its memory map (mirroring the docs-arm prompt shape), keeps greplica/codegraph blocked exactly like baseline, and extends the transcript audit: first_notes_memory_command / notes_memory_first_navigation_used for the notes arm, and a forbidden_notes_memory_access violation if any other runner touches the file. Notes size stats (chars / estimated tokens) are recorded next to the docs-arm stats for storage-cost comparison.

build-memory.ts: only change is exporting the transcript/edit replay helpers so build-notes reuses them instead of duplicating ~100 lines.

Checks run locally

• npm run typecheck — clean
• npm run build — clean
• --runner notes --fixture-only reaches fixture prep; --runner bogus rejected with the updated usage line
• git diff --check — clean

Honest caveat: I couldn't run the arms end-to-end here — that needs the SWE-chat transcript root (.context/swechat-data) and an OpenAI key, and additionally the harness's tar -C <absolute path> step doesn't work on Windows (GNU tar treats C:\... as a remote host — pre-existing, hits baseline the same way; happy to fix separately). Typecheck, build, and the argument/setup paths are verified.

To produce numbers

1. npm run eval:swechat-plan-build-notes -- --case <case-id> once per case (needs transcript root + key) to generate the checked-in notes-seeds/
2. npm run eval:swechat-plan -- --case <case-id> --runner notes

I can generate and check in the notes seeds for the four #139 cases if you can share the transcript root setup — or if it's easier, run build-notes on your side and I'll take it from there. Also happy to adjust the notes-agent prompts if you want the notes arm handicapped/equipped differently.