Phase PR CI to 7 minutes, then a coverage-preserving 5-minute median

[leehack]

Problem

Full-gate pull-request CI is coverage-bound and slow.

Across 30 recent successful PR runs, one per distinct head branch from 2026-08-16 through 2026-08-21:

• median: 514 seconds (8m34s)
• p90: 605 seconds (10m05s)
• Linux coverage was critical in 28/30 runs
• coverage median/p90: 500/580 seconds
• coverage median decomposition: about 29 seconds setup, 281 seconds VM tests, and 190 seconds single-worker coverage formatting

The pinned coverage formatter supports --workers/-j but defaults to one worker. Formatter parallelism is the safest first improvement. Reaching a five-minute full-gate median additionally requires balanced VM coverage and Chrome sharding. A five-minute p90 is not currently a realistic commitment on standard GitHub-hosted runners because later-job queue variability can dominate.

Evidence:

• median PR run
• p90 PR run
• coverage queue outlier
• Playwright installation tail
• runner scheduling tail
• CI workflow
• coverage worker option

Phase 1: low-risk critical-path reduction

1. Add durable per-step timing evidence.
2. Benchmark coverage formatting with two workers, then four if stable.
3. Require semantically identical sorted LCOV records, the unchanged 70% threshold, and no material memory instability.
4. Stabilize version-matched Playwright/Chromium provisioning without changing browser semantics.
5. Remove LiteRT-LM's serial OS cap only after confirming no hidden shared-resource constraint.
6. Cache the native prompt-reuse parity model.
7. Add strict path-aware gating beginning with provably docs-only changes; full code PRs retain every mandatory lane.

Acceptance:

• median <=420 seconds
• p90 <=510 seconds
• coverage-job median <=390 seconds
• coverage-formatting median <=120 seconds

Phase 2: coverage-preserving five-minute median

1. Split Linux VM tests across three balanced -j1 raw-coverage shards; use a fourth only if needed.
2. Assert that every prior VM test is assigned exactly once.
3. Merge raw coverage once, then run the pinned formatter, unchanged threshold, and Codecov upload.
4. Split Chrome tests into two complete, disjoint shards with an equivalent inventory assertion.
5. Keep Web Chat, native OS, companions, real-model parity, docs, analysis, and the final required aggregate check on the same PR completion DAG.

Acceptance:

• median <=300 seconds
• p90 <=420 seconds
• exact VM and Chrome test inventory parity
• coverage remains >=70%
• Codecov total does not regress by more than 0.1 percentage point
• runner minutes remain within 20% of the approximately 34m55s baseline unless explicitly accepted

A <=300-second p90 remains conditional. Promote it only after 30 successful distinct-head full-gate runs demonstrate queue p90 below 15 seconds and execution p90 below 280 seconds.

Mandatory retained validation

Do not remove, weaken, defer, or move outside the measured PR completion path:

• format, analysis, platform-boundary, release-doc, Pana, and publish dry-run checks
• both companion packages
• full Linux VM inventory with merged coverage and the 70% threshold
• root Chrome tests
• Web Chat VM and Chrome tests, deployable artifact validation, mock smoke, and real worker/WASM/GGUF smoke
• macOS and Windows native tests
• native prompt-reuse real-model parity
• LiteRT Ubuntu and Windows real-model/library smokes on relevant changes
• existing release approval boundaries

Non-goals

• Removing or quarantining tests to claim improvement
• Moving required work into a later workflow
• Raising global Dart test concurrency without an isolation audit
• Using stale docs output or a mismatched system browser
• Reporting docs-only latency as the full-gate result
• Using a persistent self-hosted runner for untrusted public PR code

Measurement guardrails

Evaluate each phase over at least 30 successful full-gate PR runs from distinct heads. Report failed/cancelled runs separately and separate queue from execution time. Record before/after median and p90, exact test discovery and pass counts, sorted LCOV semantics, threshold outcome, rerun rate, and runner-minute cost. Roll back any change that loses tests, weakens coverage, or trades latency for instability.