I built a code-graph context system to make coding agents cheaper on a 354-file Go codebase, wrote the pass/fail rule before collecting any data, measured it across 10 tasks, and shut it down when it missed.
This repo is the record: the methodology, the numbers, the two measurement bugs that invalidated the first pass, and what I'd try next.
Verdict: no-go. Two of three criteria failed, the third was negative. Full data in RESULTS.md, rule and protocol in METHODOLOGY.md.
A coding agent in an unfamiliar repository spends most of its budget re-deriving what the repository is. It greps, opens six files, reads most of them for nothing, and pays for every token.
The proposed fix: give it a pre-built map — a code graph of symbols and call edges, plus a one-or-two-sentence purpose description for every file, generated once and refreshed on commit. Query the map instead of reading the territory.
Prediction: fewer files opened, fewer tool calls, lower cost, no loss of correctness.
The failure mode here is obvious — build it, feel that it helps, ship it. So the decision rule went into the methodology document before a single measured task ran, explicitly so it "can't be loosened after seeing results."
| Criterion | Outcome | |
|---|---|---|
| A | Median files read −30%, tool calls −25% | Failed — better in 3 of 9 |
| B | Savings exceed bootstrap + ongoing cost within 60 days | Negative — cost $0.62 more |
| C | No task slower or worse than baseline | Failed — faster in 1 of 9 |
Ten tasks across comprehension, blast-radius analysis, bug fixing, feature scaffolding, and refactor safety. Each ran twice in fresh sessions with no history carryover — once with the system available, once with it blocked.
Two choices did the work of keeping it honest:
- The baseline was the real setup. Ad hoc grep/read plus the tooling already in use — not a stripped-down agent with nothing. The question is "better than what I do today," not "better than nothing."
- Two bug-fix tasks were re-derived from real commits. Revert the fix, re-pose it as a bug report, score the agent's diff against what actually shipped. Correctness becomes a diff match, not a judgment call.
Runs were non-interactive with cost and token counts read from the API response rather than estimated.
Initial results showed no advantage. Both reasons were measurement bugs, and both would silently corrupt anyone else's version of this experiment.
The hidden subagent. The baseline looked suspiciously efficient on one task — 3 tool calls against the treatment's 25. It had delegated the real work to a subagent whose transcript lives in a separate file. Those 38 tool calls appear in neither the parent transcript nor the turn count, but the cost does roll into the total. Fairly counted, that task flipped from an apparent loss to 41% fewer tool calls and 23% cheaper.
Silent permission denials. The treatment was barely using the system under test — one of five sessions called it at all, and gave up after two tries. Not reluctance: the workspace was untrusted and the query scripts were missing from the permission allowlist, so every invocation was denied without an error the agent could act on. In a transcript, "tried and was denied" and "chose not to" look identical.
Both were found by pushing on results that looked odd instead of accepting them.
Wall-clock time was the clearest signal, and the cause isn't mysterious: every query spawned a Python process that shelled out to a CLI that connected to the graph database — a fresh subprocess per call. Fewer, smarter tool calls still lost, because each cost more latency than the grep it replaced.
The one task requiring genuine multi-hop traversal of the call graph won decisively, for exactly the structural reason predicted: grep cannot follow an edge. The value is real and it is narrow.
-
A context system is worth exactly its invocation rate. Two failure modes look identical from outside — the agent doesn't know which tool fits the question, or the agent is being silently denied. They need completely different fixes. Check permission denials before concluding anything about tool choice.
-
The always-loaded instruction file is a bigger lever than any per-task saving. Mid-pilot I rewrote the repository's agent instruction file — loaded unconditionally into every session — from 641 lines to 180, and found a stale code pattern that had propagated into four other files and was actively teaching agents an API that does not exist. That one edit likely outweighs everything the graph would have saved.
-
Syntax parsing is not type checking. The graph's interface-implementation detection missed all four implementations of a central interface — it could not resolve embedded interfaces or method sets split across files. Replacing it with a small tool built on the language's real type checker found 76 implementation edges across 39 interfaces, and it exits with an error rather than emitting a partial result that looks complete.
Not building this now — recording it as the honest next hypothesis:
- Hold the query scripts against a persistent connection instead of spawning a subprocess per call. That should remove most of the latency that decided criterion C.
- Narrow the remit to the question shape it demonstrably wins on — multi-hop traversal — rather than offering it as a general replacement for search.
- Re-run the same ten tasks against that version. The harness was built to be reusable, which is the other reason to write the rule down first.
| Semantic layer bootstrap (one-time, hash-guarded) | $3.45 |
| Pre-build estimate for the same work | $14.64 |
| Files summarized | 354 |
| Engineering time, build phases | ~2.3 h |
| Measured task runs | 20 |
The bootstrap came in at a quarter of the estimate, almost certainly because the real run executed as a tight sequential loop and kept hitting the prompt cache the sampled estimate kept missing. That's labelled a hypothesis rather than a result — the cost ledger at the time didn't record cache hit rates. It does now, so the next run can confirm or refute it.
Total cost of establishing that a plausible, well-liked idea did not work: about four dollars and an afternoon.
MIT