diff --git a/artifacts/design/sidebar-after-768.png b/artifacts/design/sidebar-after-768.png deleted file mode 100644 index 40c01f56..00000000 Binary files a/artifacts/design/sidebar-after-768.png and /dev/null differ diff --git a/artifacts/design/structural-context-evaluation.html b/artifacts/design/structural-context-evaluation.html deleted file mode 100644 index ade3647c..00000000 --- a/artifacts/design/structural-context-evaluation.html +++ /dev/null @@ -1,123 +0,0 @@ - - - - - - - - -Synthetic structural-context agent evaluation - - - -
-
CodeVetter · structural context evaluationsynthetic-structural-context-contract
-
-

Does the graph help the agent ship better?

-

The same tasks and agent identity, paired with and without CodeVetter structural context. Executable hidden checks decide success; activity metrics only explain the path.

-
- unqualified -

Synthetic contract fixture only. No real structural-context value claim is authorized.

-
-
- -
-

Paired outcome

2 complete A/B pairs across 2 tasks.

Success means every required hidden check passed with no recorded regression.

-
-
without structural context50%1/2 successful runs
-
VS
-
with structural context100%2/2 successful runs
-
-
+50% success delta1 treatment wins0 control wins1 ties0 regression delta
-

unqualified · synthetic receipts · Synthetic contract fixture only. No real structural-context value claim is authorized.

-
- -
-

What changed task by task

Open a decision trace to see which graph question oriented the treatment arm and which source paths it returned.

-
-
-
trial 1

Preserve authenticated route access

- tie pass -
-
-
without structural contextsuccess2/2 checks
- -
with structural contextsuccess2/2 checks
-
-
-

Changed checks

No check outcome changed between arms.

-
Agent decision trace
  1. graph_querycallers:requireAuthenticatedUser

    Confirmed the guard is shared by both protected route branches.

    src/auth.ts · src/routes.ts
-
-
-
-
trial 1

Invalidate stale checkout cache entries

- treatment win -
-
-
without structural contextcheck failure1/2 checks
- -
with structural contextsuccess2/2 checks
-
-
-

Changed checks

  • Stale checkout cache is clearedcontrol: failtreatment: pass
-
Agent decision trace
  1. graph_querysymbol:invalidateCheckoutCache

    Located the cache invalidator and its checkout caller.

    src/cache.ts · src/checkout.ts
  2. graph_pathcheckout route to cache invalidator

    Confirmed the missing invalidation edge on the write path.

    src/routes.ts · src/checkout.ts · src/cache.ts
-
-
-
- -
-

Can this result be trusted?

Qualification is policy-driven. Favorable synthetic numbers still fail the real-evidence gate.

-
  • FAILreal evidenceSynthetic fixtures cannot establish product value.
  • PASScomplete pairs2/2 complete A/B pairs
  • PASSdistinct tasks2/2 distinct tasks
  • PASSaa pairs2/2 complete A/A pairs
  • PASSaa noise0% discordance; maximum 0%
  • PASSpair integrity0 invalid or contaminated pairs
  • PASSsuccess improvement+50%; minimum +20%
  • PASSregression control0 regressions; maximum 0
- -
- -
-

Activity diagnostics

Secondary signals describe cost and search behavior. They never substitute for task success, and missing values remain missing.

-
Means use only pairs where both arms captured the metric. A negative delta means less activity, not necessarily a better result.
MetricControlTreatmentDeltaCoverage
Verification selections1102/2
Files inspected2.52-0.52/2
Files modified1102/2
Tool calls22.5+0.52/2
Input tokens19501700-2502/2
Output tokens520450-702/2
Elapsed time (ms)6450055500-90002/2
Cost (USD)0.0180.016-0.0022/2
-
- -
-

Boundaries

What this artifact does not prove.

-

Declared limitations

  • Synthetic receipts prove the scorer contract only.
  • No coding agent or hidden acceptance runner was invoked.
  • Activity diagnostic fields are illustrative and not product measurements.

Invalid or contaminated pairs

None detected.

-
Authorized claim

Synthetic contract fixture only. No real structural-context value claim is authorized.

-
- - -
- - - \ No newline at end of file diff --git a/artifacts/design/trex-after-1440.png b/artifacts/design/trex-after-1440.png deleted file mode 100644 index acbdf8cb..00000000 Binary files a/artifacts/design/trex-after-1440.png and /dev/null differ diff --git a/artifacts/design/trex-after-390.png b/artifacts/design/trex-after-390.png deleted file mode 100644 index 9bf63da6..00000000 Binary files a/artifacts/design/trex-after-390.png and /dev/null differ diff --git a/artifacts/design/trex-after-768.png b/artifacts/design/trex-after-768.png deleted file mode 100644 index 62836436..00000000 Binary files a/artifacts/design/trex-after-768.png and /dev/null differ diff --git a/artifacts/design/trex-before-1440.png b/artifacts/design/trex-before-1440.png deleted file mode 100644 index 7fc7c4ba..00000000 Binary files a/artifacts/design/trex-before-1440.png and /dev/null differ diff --git a/artifacts/design/trex-result-1440.png b/artifacts/design/trex-result-1440.png deleted file mode 100644 index 3a842f26..00000000 Binary files a/artifacts/design/trex-result-1440.png and /dev/null differ diff --git a/artifacts/design/usage-telemetry-after-1440.png b/artifacts/design/usage-telemetry-after-1440.png deleted file mode 100644 index dddec2ae..00000000 Binary files a/artifacts/design/usage-telemetry-after-1440.png and /dev/null differ diff --git a/artifacts/design/usage-telemetry-after-390.png b/artifacts/design/usage-telemetry-after-390.png deleted file mode 100644 index 77a40854..00000000 Binary files a/artifacts/design/usage-telemetry-after-390.png and /dev/null differ diff --git a/artifacts/design/usage-telemetry-after-768.png b/artifacts/design/usage-telemetry-after-768.png deleted file mode 100644 index abcfa19a..00000000 Binary files a/artifacts/design/usage-telemetry-after-768.png and /dev/null differ diff --git a/artifacts/design/usage-telemetry-before-1440.png b/artifacts/design/usage-telemetry-before-1440.png deleted file mode 100644 index 4a6aee59..00000000 Binary files a/artifacts/design/usage-telemetry-before-1440.png and /dev/null differ diff --git a/artifacts/performance/calorie-exercise-guidance-2026-08-10.md b/artifacts/performance/calorie-exercise-guidance-2026-08-10.md deleted file mode 100644 index f4ecc36c..00000000 --- a/artifacts/performance/calorie-exercise-guidance-2026-08-10.md +++ /dev/null @@ -1,132 +0,0 @@ -# Calorie exercise-guidance performance trial — 2026-08-10 - -This records a local proof run, not a customer-impact case study. The benchmark -used no production database, hosted endpoint, deployment, or paid model. The -publication ledger should label any later review URL as a synthetic stress -result unless representative product-scale evidence is added. - -## Scope - -- Project: `/Users/sarthak/Desktop/fleet/calorie` -- Starting revision: `4db2075170a27c5147ad88fe3622dd69760d90ec` -- Runtime: Node `v24.19.0` on Darwin arm64 -- Exact flow: `exercise guidance performance scales across representative food histories` -- Benchmark: `src/lib/recommendations.performance.test.ts` -- Candidate: `calculateGymGuidance` in `src/lib/recommendations.ts` -- Inputs: 1,000, 10,000, and 35,000 food-history entries; 100 in-process - iterations per input -- Correctness oracle: the selected exercise window must belong to the most - recent eligible meal, independent of input order; future and expired meals do - not qualify - -The benchmark is intentionally a scale/stress workload. A 35,000-entry local -history is useful for distinguishing algorithmic behavior, but this artifact -does not establish that typical users have that history size or that the -operation is currently a visible UX bottleneck. - -## Why CodeVetter selected it - -`runtime:plan-flow-campaign` discovered the timed Vitest flow without a manual -target hint. The independent source-profile passes inside a five-sample run -repeated `calculateGymGuidance` as the repository-owned CPU candidate. The -captured supported-scale cost after the change was approximately `0.17 ms/op` -at 35,000 entries. - -The planner classified the candidate as actionable but used neutral frequency -and user-impact weights because Calorie has no project-owned priority manifest. -Therefore production frequency and customer impact remain unverified. - -## Change - -The baseline implementation cloned the history, filtered it, fully sorted the -eligible entries, mapped every entry into a window object, and then selected the -first still-open window. The candidate performs one pass and retains only the -most recent eligible open window. - -The change preserves the baseline's stable-tie behavior: when two entries have -the same timestamp, the first input entry remains selected. - -## Paired result - -CodeVetter alternated baseline and candidate executions for ten measurement -pairs after one warmup per side. Both checkouts used the same exact workload -digest: - -`ebf9f02b4b14632ea6f245be39693a196b5276ea42bc6cedd46000819e80c0ec` - -| History entries | Baseline median | Candidate median | Change | -| ---: | ---: | ---: | ---: | -| 1,000 | 0.034 ms/op | 0.016 ms/op | -52.941% | -| 10,000 | 0.253 ms/op | 0.046 ms/op | -81.818% | -| 35,000 | 1.203 ms/op | 0.162 ms/op | -86.534% | - -The endpoint scale exponent moved from `1.003` to `0.651`. CodeVetter returned: - -- verdict: `confirmed` -- mechanically confirmed: `true` -- materially useful under the recorded benchmark policy: `true` -- shipping recommended at the ten-sample floor: `true` -- verifier limitations: none - -Interpretation: the algorithmic improvement is strongly supported. The absolute -benefit is only about `1.041 ms/op` at the largest stress input, so this is not -yet evidence of customer-visible latency. A future UI must show absolute and -relative movement together. - -## Correctness and repository checks - -- Focused Vitest run: 2 files, 17 tests passed -- Full Calorie test run before the final iteration-count increase: 28 files, - 120 tests passed -- TypeScript: `pnpm typecheck` passed -- Biome on the three touched files passed -- `git diff --check` passed - -## CodeVetter improvements caused by this trial - -The first paired Vitest attempt compared runner startup time and ignored the -benchmark's console `ms/op` series. That produced an inconclusive result even -though both sides emitted comparable domain metrics. CodeVetter now retains the -median console metrics from repeated paired Vitest measurements, matching the -existing Node-test and Node-script behavior. A regression test covers the -failure mode. - -The later campaign run also exposed a source-coordinate mismatch: the raw V8 -profile reported generated line 178 while bounded source inspection uniquely -anchored `calculateGymGuidance` to original TypeScript line 254. The campaign -planner now emits the source-anchored line and preserves the raw profiler line -as `reported_line`. This prevents an agent or UI from silently presenting the -generated coordinate as the original source location. - -Relevant CodeVetter validation after both fixes: 34 focused runtime-performance -tests passed, including paired execution, Vitest metric retention, source-line -alignment, Node profiling, and Go benchmark coverage. - -## Other candidate screening in this pass - -| Project | Evidence found | Decision | -| --- | --- | --- | -| India Standards | Existing benchmark depends on a local DuckDB/data path and requires explicit arguments. | Not executed in the default local campaign; the workload needs a declared fixture boundary first. | -| Karte | Generic tests but no direct timing evidence; important paths are database/network-shaped. | No synthetic optimization forced. | -| SWE Interview Prep | Generic deterministic tests over small fixed catalogs, with no representative timed flow. | Kept as a correctness surface; no scale claim manufactured. | -| ChatGPT Memory Insights | Local export normalization and semantic-analysis code were inspected. The dominant semantic flow depends on model inference, while the only current performance test checks timer bookkeeping. | No candidate changed. A representative staged analysis fixture is needed before optimizing source. | - -These negative decisions are part of the proof: discovery breadth is not the -same as evidence quality, and a campaign should be allowed to return “needs a -better workload.” - -## UI implications earned by this evidence - -Per application, the eventual surface should show: - -1. flow inventory: tested, excluded, and missing-workload flows; -2. exact revision, workload identity, samples, and correctness gate; -3. observed absolute and relative measurements; -4. source-anchored candidate plus raw profiler coordinate when they differ; -5. confirmed, rejected, guardrail, or needs-better-workload status; -6. inferred product impact separately from measured runtime impact; -7. the concrete tool improvement or product patch produced by the trial. - -This trial argues against building a generic profiler dashboard first. The -useful UI object is an evidence-backed application-flow review with explicit -coverage and missing-evidence states. diff --git a/artifacts/performance/old-local-portfolio-manifest.json b/artifacts/performance/old-local-portfolio-manifest.json deleted file mode 100644 index 36770752..00000000 --- a/artifacts/performance/old-local-portfolio-manifest.json +++ /dev/null @@ -1,17 +0,0 @@ -{ - "schema_version": "runtime-qualification-portfolio-manifest/v1", - "repositories": [ - { "id": "aliveville", "path": "../../../../fleet-inactive-projects/aliveville" }, - { "id": "companion-robot", "path": "../../../../fleet-inactive-projects/companion-robot" }, - { "id": "elves-hq", "path": "../../../../fleet-inactive-projects/elves-hq" }, - { "id": "everythingrated", "path": "../../../../fleet-inactive-projects/everythingrated" }, - { "id": "forecast-lab", "path": "../../../../fleet-inactive-projects/forecast-lab" }, - { "id": "materia", "path": "../../../../fleet-inactive-projects/materia" }, - { "id": "open-historia", "path": "../../../../fleet-inactive-projects/open-historia" }, - { "id": "protein-index", "path": "../../../../fleet-inactive-projects/protein-index" }, - { "id": "saas-ideas", "path": "../../../../fleet-inactive-projects/saas-ideas" }, - { "id": "today-little-log", "path": "../../../../fleet-inactive-projects/today-little-log" }, - { "id": "truehire", "path": "../../../../fleet-inactive-projects/truehire" }, - { "id": "web-playables", "path": "../../../../fleet-inactive-projects/web-playables" } - ] -} diff --git a/artifacts/performance/old-local-projects-results-2026-08-09.md b/artifacts/performance/old-local-projects-results-2026-08-09.md deleted file mode 100644 index e7bb1d21..00000000 --- a/artifacts/performance/old-local-projects-results-2026-08-09.md +++ /dev/null @@ -1,125 +0,0 @@ -# Old local project performance pass — 2026-08-09 - -## Scope - -This pass inspected every locally checked-out Fleet project whose canonical -lifecycle is `past`. It deliberately used local execution only: no production -database, external API, Cloud Player, deployment, or cloud benchmark was run. -The historical `mashup` project was excluded because it has been incorporated -into Foundry and has no independent checkout. - -The qualification manifest is -`artifacts/performance/old-local-portfolio-manifest.json`. - -## Portfolio result - -Before adding a benchmark, CodeVetter classified the 12 checked-out projects as: - -- `ready`: 0 -- `needs_selection`: 8 -- `no_representative_workload`: 4 - -After the Web Playables benchmark was added, CodeVetter independently selected -that exact test and classified the repository as `ready`: - -- adapter: `vitest` -- target: `games/idle-startup/test/performance.test.ts` -- test: `scales deterministic company ticks` -- qualification score: 70 - -This is a useful product result: qualification can rediscover a good workload, -but it cannot manufacture representative workloads from generic correctness -tests. The portfolio initially had zero workloads with direct timing evidence. - -## Verified optimization - -### Web Playables — idle startup tick - -The benchmark advances deterministic game state for 1,000, 10,000, and 50,000 -ticks. The production tick function calculated gross revenue and burn, then -called `netPerSec()` for milestone checks, redundantly calculating both again. -The change reuses `gross - burn` within the same tick. - -Observed CodeVetter comparison: - -| Scale | Baseline | Optimized | Change | -| ---: | ---: | ---: | ---: | -| 1,000 ticks | 3.613 ms | 2.807 ms | -22.31% | -| 10,000 ticks | 21.666 ms | 12.338 ms | -43.05% | -| 50,000 ticks | 104.190 ms | 57.964 ms | -44.37% | - -The enclosing Vitest wall time improved from 831 ms to 655 ms (-21.18%). -CodeVetter returned `confirmed`, `mechanically_confirmed: true`, and -`materially_useful: true`. It correctly withheld `shipping_recommended` -because the baseline had only three samples and independent Vitest profiles did -not all contain application source frames. - -Validation after the change: - -- 34 tests passed (27 simulation, 6 prestige, 1 performance) -- TypeScript `tsc --noEmit` passed -- Biome passed for the changed production and benchmark files -- `git diff --check` passed - -The checkout was clean before the experiment. It now contains only the scoped -simulation change, the new benchmark, and the local `.codevetter` evidence. -The lockfile-existing dependencies were installed locally with scripts disabled; -69 packages came from cache and four were downloaded. No dependency was added. - -## Project-by-project disposition - -| Project | Qualification / inspection result | Action | -| --- | --- | --- | -| Aliveville | Generic Vitest tests suggested simulation work, but no timed workload existed and the Web3D dependency tree was not installed. | Inspected the world tick and catch-up paths; no evidence-backed change made. Best next workload is a deterministic multi-NPC catch-up benchmark. | -| Companion Robot | No representative code/test workload; effectively a project shell. | No optimization to make. | -| Elves HQ | Build/dev workspace with no benchmark or test workload. | No evidence-backed optimization to make. | -| EverythingRated | Generic tests only; checkout already had 109 dirty entries. | Preserved existing work and made no speculative edit. | -| Forecast Lab | No representative code/test workload; effectively a project shell. | No optimization to make. | -| Materia | Generic content tests only; checkout already had 18 dirty entries. | Preserved existing work and made no speculative edit. | -| Open Historia | Correctness tests, no timed workload. Storage restore already uses keyed lookup rather than a quadratic scan. | No material local hotspot found; no edit made. | -| Protein Index | Highest-signal work depends on database/network behavior. | Excluded from this backend-independent pass. | -| SaaS Ideas | No representative workload; checkout already had three dirty entries. | Preserved existing work and made no speculative edit. | -| Today Little Log | Browser/auth-oriented tests and lifecycle is deleted. | Excluded from local CPU optimization. | -| TrueHire | Generic tests only; likely runtime cost is external repository fetching and inputs are small. | A synthetic scoring microbenchmark would not be representative, so no edit was made. | -| Web Playables | Newly qualified deterministic tick benchmark. | Verified 44.37% improvement at 50,000 ticks. | - -## CodeVetter gaps exposed - -1. **Representative workload discovery is the limiting step.** File and test - names are not enough; direct timing evidence was absent in all 12 projects. -2. **Vitest source profiling is incomplete.** V8 CPU profiling captured the - Vitest/Vite runner process, not useful application frames from the fork that - executed the test. The measurement and before/after verdict worked, but the - tool did not locate the redundant calculation by itself. -3. **Dirty-snapshot qualification is coarse.** The repository is marked dirty, - but untracked benchmark/evidence identity needs to remain explicit in every - receipt. -4. **The shipping gate is usefully conservative.** It did not turn a three-sample - baseline into a shipping claim even after a large observed improvement. - -## One Billion Row Challenge status - -The current CodeVetter artifact is a bounded Node parser experiment, not an -official 1BRC implementation: - -- largest measured input: 800,000 rows -- latest parser-only time: 34.325 ms (the retained supervised run was 32.465 ms) -- latest throughput: approximately 23.31 million rows/second -- naive linear parser-only projection to one billion rows: 42.91 seconds -- official 8-core winner: 1.535 seconds -- optimistic gap: 27.95 times slower, or 3.58% of the winner's throughput - -That gap is intentionally labelled optimistic. The current metric excludes file -I/O and startup, uses an in-memory string and a small generated station set, and -the file entry point reads the entire input with `readFile(..., "utf8")`. It is -therefore not yet capable of a credible official 12 GB run. Cross-machine -wall-clock results are not directly comparable either. - -The next honest milestone is not a smaller microbenchmark number. It is a -streaming or chunked parser that can process the official file with bounded -memory, followed by an end-to-end same-machine benchmark against a baseline. - -## Change policy - -No commit, push, deployment, production configuration, or cloud resource was -created by this pass. diff --git a/artifacts/reviews/pr157-codevetter-review.png b/artifacts/reviews/pr157-codevetter-review.png deleted file mode 100644 index 876be9ea..00000000 Binary files a/artifacts/reviews/pr157-codevetter-review.png and /dev/null differ diff --git a/artifacts/reviews/pr173-codevetter-review.png b/artifacts/reviews/pr173-codevetter-review.png deleted file mode 100644 index 31321657..00000000 Binary files a/artifacts/reviews/pr173-codevetter-review.png and /dev/null differ