home-assistant-class python emission volume: 38M edges at the 8-file cap (last open lever of the dcbench 60s tail)

[nikolay-e]

Residue of #116: 62 of 372 dcbench instances still exceed the 60s cap

The sentry (28->2) and gitpod (34->1) hang classes are fixed, but three repo classes remain, and each traces to a NAMED uncapped/slow builder (all complete at ~35-40s builder time, so these are 60-120s instances, not infinite hangs):

class	instances	culprit (DIFFCTX_TRACE_BUILDERS)	trace
kubernetes	21	GoEdgeBuilder: 14.3M edges / 39.8s on 078f462a955b	go caps landed (#116) but the per-import package match is a linear scan for (pkg, frag_ids) in &pkg_to_frags — O(imports x packages), plus emission volume
home-assistant	20	PythonEdgeBuilder: 38.0M edges / 7.8s on 21e51b83d3ae — capped channels still fan wide on a 10k-module monorepo; AND structural[testing]: 37.3s for 2,562 edges — compute-bound quadratic scan, not emission
polars	17	RustEdgeBuilder: 35.4M edges / 34.6s on 3b6875ae9879 — rust_type/rust_fn channels have NO max_files_per_name cap (the same disease python/js/go had)

Repro: git -C test-repos/<repo> worktree add --detach wt <commit> then DIFFCTX_TRACE_BUILDERS=1 diffctx . --diff '<commit>^..<commit>' -q -f json --budget 8000 --timeout 120.

Fix sketch, in expected-value order: (1) rust.rs gets the standard MAX_FILES_PER_NAME = 8 def-file cap (mechanical, same patch as python/js/go); (2) go.rs import loop gets a pkg_to_frags.get(&imp_pkg) lookup instead of the full-map scan; (3) profile testing.rs on the home-assistant instance (37s for 2.5k edges is a scan bug, likely same-stem or name-pair quadratic); (4) re-measure the 62. Post-emission single-threaded dedup (~50s of a 52s sentry run, noted at #116 close) is the second-order lever after these.

Full list of the 62 instance ids: results/final/v6/dcbench_new/*.jsonl rows with status != produced.

[nikolay-e]

Partial fix landed in 7156776 — three of four mechanisms addressed, measured on the trace instances:

instance	before	after	mechanism
kubernetes 078f462	92.7s	58.2s (under the 60s cap)	go per-import package lookup + last-component index for the dir-path scan (was O(imports x dirs) with 2 allocs/probe)
home-assistant 21e51b8	testing.rs 37.3s	2.1s	test loop iterates unique test FILES, not fragments (everything in it was file-level); total still 72s — python 38M-edge emission remains
polars 3b6875a	35.4M edges	18.1M edges, wall ~flat 49s	rust type/fn/use/mod channels now carry the standard <=8-file caps; remaining cost is in the emission loops themselves

Corpus: bit-identical (2902 pass, same 196 baseline) — the caps changed no oracle outcome.

Remaining for this issue: (a) polars-class — rust builder emission cost with capped-but-wide channels; (b) home-assistant-class — python identifiers channel emits 38M edges even at the 8-file cap on a 10k-module monorepo; (c) the second-order lever named at #116 close: post-emission single-threaded dedup over tens of millions of surviving emissions. Re-measure the full 62-instance list after (a)/(b).

[nikolay-e]

QA pass 2026-08-10, two additions from the diff review of the same epoch:

• edges/semantic/c_family.rs:179-181 deep-clones every C-family fragment (identifiers included) just to satisfy file_representatives(&[Fragment]), and the push_file_key/by_stem dedup uses Vec::contains — O(bucket²) over fragments in exactly the envoy-shape buckets (256 config.h, 520 config stems) this issue's classes hit. Both are inside the builder-cost budget this issue tracks; also filed in the cleanup cluster dead wiring after the last epoch: find_ignored_paths, duplicated BM25 top-k knob #218.
• The remaining home-assistant python 38M-edge emission item stands; nothing in this pass touched it.

[nikolay-e]

Progress in 1b3a8d2 (1.14.0), the two QA-added c_family items plus the second-order lever:

• c_family deep-clone for file_representatives removed (full-slice call, bit-equivalent) and both Vec::contains O(bucket^2) dedups (push_file_key basename/stem buckets, by_stem pairing) replaced with order-preserving seen-sets — exactly the envoy-shape buckets (256 config.h / 520 config stems) this issue's classes hit.
• Post-emission dedup sort parallelized (graph.rs dedup_compact_edges: sort_by -> rayon par_sort_by, stable so bit-identical) — the single-threaded pass named at near-dense edge emission on huge same-directory trees: 199M raw edges / 37GB peak on one mui instance #116 close as the second-order lever.
• The known flooding loops (config-key scan/emission, c_family, python identifier emission) now poll the per-run deadline inside the loop (compute deadline is process-global: concurrent MCP requests overwrite each other's ceiling #210), so a 60-120s instance aborts at the ceiling instead of running to completion.

Gates: bitcheck 24/24, corpus 2902 baseline untouched.

Remaining as before: (a) polars-class rust emission cost with capped-but-wide channels, (b) home-assistant python 38M-edge emission volume at the 8-file cap (fragment-fanout and dual fwd/rev insertion — reducing it changes edges, so it needs the corpus gate with measurement), (c) re-measure the 62-instance list after (a)/(b).

[nikolay-e]

Item (c) done — all 62 instances re-measured on 1.15.0 (f0fe9fb build, M4 Pro): first a 4-way parallel pass over the full list, then a serial confirmation pass for everything ≥60s (serial numbers reported below; the handful of serial rows that overlapped a concurrent corpus build are upper bounds).

Headline: zero hangs. Every instance either completes or aborts cleanly at the in-loop 120s deadline (#210) — the original list was 62 uniform hang rows at the 60s cap.

class	n	<60s	60–120s (completed)	hit 120s deadline
kubernetes	21	14	6	1 (ec84379)
home-assistant	20	8	12	0
polars	17	16	1 (62.5s max)	0
gitpod / sentry / nextcloud	3	0	3 (84–100s)	0
gitlab-foss	1	0	0	1 (80b2c3c)
total	62	38	22	2

Reading per remaining sub-item:

• (a) polars-class rust emission — effectively closed: 16 of 17 under the cap, worst case 62.5s.
• (b) home-assistant python 38M-edge emission — now the whole story: the 60–120s tail is 12 of its 20 instances plus kubernetes' 6. Reducing that volume changes edges (Q-class) and stays the one open lever.
• The two deadline-bound instances (kubernetes__ec84379, gitlab-foss__80b2c3c) degrade gracefully instead of hanging.

Narrowing this issue to sub-item (b); everything else it tracked is measured done.