Week 7 close-out: gauntlet G3+G4, perf bench, ADR-009 - #21
Merged
Conversation
…gauntlet G3) Function-level transforms (callback_to_async_await, class_to_dataclass, promise_chains_to_async, promise_constructor_to_async, commonjs_to_esm, manual_typecheck_to_hints) fire on the *declaration* line, so the default ±1 window only shows the signature + docstring and hides the actual code being refactored. Per-transform overrides give each pattern the right amount of after-context without bloating output for single-line transforms. Single-line transforms (format_to_fstring, var_to_const_let, etc.) keep the tight default ±1 window — verified by a regression test that asserts unrelated body lines are NOT pulled into the excerpt for a format_to_fstring finding. Two new tests in format-analysis.test.ts (8 → cover both 'extends context for function-level transforms' AND 'keeps tight context for single-line transforms — no over-pull').
…nd (gauntlet G4)
If a user clones Refactron and runs 'refactron run --apply' on the
repo itself, the test gate fails by design — the meta-tests in
tests/unit/transform/transforms/ exercise the transforms on the
fixtures, and refactoring those fixtures breaks the meta-tests' input.
This is exactly the safety guarantee firing as intended (no files
written when the test gate fails) but it confuses first-time users
on Refactron's own checkout.
Add a 'Known Limitations' section before Contributing with the
explanation and the .refactronrc.json { exclude: ['fixtures/**'] }
workaround.
… week 7 day 49 bench/gen-fixture.ts produces a directory tree of mixed Python + TypeScript files at a target LOC count, with every Refactron transform pattern represented so analyze finds work to do. 50/50 split between Python and TypeScript; ~100 files per subdirectory to avoid pathological single-dir sizes. Generated trees gitignored — they're large (a 100k-LOC tree is ~6 MB, a 500k-LOC tree is ~30 MB) and trivially regenerable. README documents the bench protocol + the Week 7 perf targets. Bench results captured in dev-docs/decisions/09-week-7-architecture.md: - 10k LOC analyze: 1.31s (target 6s, 4.6x headroom) - 100k LOC analyze: 11.48s (target 60s, 5.2x headroom) - 500k LOC: deferred to local-only; targets met without profiling needed.
|
Warning Rate limit exceeded
You’ve run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR closes out Week 7 by improving analyze output readability for function-level findings (G3), documenting the “self-test paradox” when running Refactron on its own repo (G4), and adding a local perf benchmark fixture generator + accompanying documentation and ADR.
Changes:
- Extend
formatAnalysisReportexcerpts for specific function/class-level transforms while keeping single-line transforms on a tight context window, with new unit coverage. - Add perf bench infrastructure (
bench/gen-fixture.ts,bench/README.md) and record the Week 7 architecture/bench results in ADR-009. - Document the Refactron self-run limitation in README and ignore generated bench trees via
.gitignore.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
src/cli/format-analysis.ts |
Adds per-transform extended “after context” when rendering source excerpts for findings. |
tests/unit/cli/format-analysis.test.ts |
Adds regression tests for extended context (function-level) vs non-extended context (single-line). |
README.md |
Documents the self-repo “test gate” behavior and an exclude workaround. |
bench/gen-fixture.ts |
New script to generate synthetic Python/TS trees at a target LOC for benchmarking analyze. |
bench/README.md |
Documents the bench protocol and target thresholds. |
dev-docs/decisions/09-week-7-architecture.md |
New ADR capturing Week 7 architecture decisions and benchmark snapshots. |
.gitignore |
Ignores generated bench/*-loc/ fixture directories. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+96
to
+103
| const EXTENDED_AFTER_BY_TRANSFORM: Partial<Record<string, number>> = { | ||
| callback_to_async_await: 6, | ||
| class_to_dataclass: 5, | ||
| promise_chains_to_async: 5, | ||
| promise_constructor_to_async: 6, | ||
| commonjs_to_esm: 3, | ||
| manual_typecheck_to_hints: 4, | ||
| }; |
Comment on lines
+68
to
+69
|
|
||
| async function generate(outDir: string, targetLoc: number): Promise<void> { |
…vidence file The original single-run bench results were misleading — they implied exact numbers (1.31s / 11.48s) that single-iteration timing can't support. Replaced with a reproducible N=5-iterations-with-warmup script and saved the raw evidence to bench/results-2026-05-14.txt. Methodology (bench/run-bench.sh): - 1 warm-up run (discarded — primes Node module cache, tree-sitter wasm load, OS file cache) - 5 measured runs via /usr/bin/time -p - Median + min + max reported - Hardware + Node version captured at top of results file Real results (2026-05-14, Apple M2, Node 24): - 10k LOC (448 files): median 1.31s range 1.16-1.64s - 100k LOC (4465 files): median 20.58s range 14.99-38.65s - 500k LOC: not run (~30 MB fixture; runnable locally) Honest correction: the prior single-run 11.48s for 100k was below the bench's measured MIN (14.99s) — that run probably had a fully-warm FS cache from the immediately-preceding fixture generation. The N=5 median (20.58s) is the correct number to claim. 100k LOC shows 2.6x run-to-run variance. Even the worst run (38.65s) beats the 60s target. Variance source likely OS file-cache churn under 8 GB RAM + GC pauses. Investigation deferred unless it becomes a ship-blocker.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wraps Week 7 with the gauntlet-#1 deferred fixes (G3 + G4), Day 49 perf bench infrastructure + results, and ADR-009 recording the architecture. Gauntlets #2-#5 + bug-fix-day stay as a separate user-driven workstream (needs real beta users to schedule with).
G3 — per-transform extended after-context (
format-analysis.ts)Function-level transforms (
callback_to_async_await,class_to_dataclass,promise_chains_to_async,promise_constructor_to_async,commonjs_to_esm,manual_typecheck_to_hints) fire on the declaration line — the default ±1 window only shows the signature + docstring and hides the actual code being refactored. Per-transform overrides give each pattern the right amount of after-context.Single-line transforms (
format_to_fstring,var_to_const_let, etc.) keep the tight ±1 window. Regression test asserts unrelated body lines are NOT pulled into the excerpt for aformat_to_fstringfinding.G4 — README self-test paradox note
First-time users cloning Refactron and running
refactron run --applyon the repo itself see the test gate fail — that's the safety guarantee firing as intended, but it's confusing without context. Added a 'Known Limitations' section before Contributing with the explanation + the.refactronrc.json { exclude: ['fixtures/**'] }workaround.Day 49 — perf bench infrastructure + results
bench/gen-fixture.tsproduces synthetic Python + TS trees at a target LOC count with every transform pattern represented. Generated trees gitignored.bench/README.mddocuments the protocol.Bench results (M-series macOS, Node 22):
analyzewall-clockBoth measured sizes pass with healthy headroom on first attempt — no profiling needed. 500k bench runs locally (≈25 MB fixture); deferred from CI.
ADR-009
dev-docs/decisions/09-week-7-architecture.mdrecords the three formatter modules, engine-side hooks (onGateComplete,onShadowRoot), gauntlet-#1 fixes, cross-platform consistency work, and the bench results. Also catalogs the deferred-to-v2.1 items.Out of scope (deferred to dedicated workstream)
--keep-shadowflagTest plan
tests/unit/cli/format-analysis.test.tsnpm run typecheck— cleannpm run lint— clean (max-warnings 0)npm run format:check— cleannpm run build— cleanbench/gen-fixture.ts+ analyze on 10k + 100k trees, results captured in ADR-009npm test— same pre-existing Python-subprocess parallel-load flakes seen on PRs Week 6 — Documentation Engine (LLM, scoped) #15, fix(cli): scope run by positional path, stop --apply from eating its argument #17, Week 7 (part 1) — CLI output redesign for analyze, run --dry-run, run --apply #19 (callback-to-async, format-to-fstring, etc.). All pass in isolation. Not regressions from this PR.