From 2555d33f557e7d597e2aecb7a80be9f052795e22 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 16 Jul 2026 21:53:47 +0200 Subject: [PATCH 01/38] =?UTF-8?q?=E2=9C=A8=20introduce=20dotnet-benchmark?= =?UTF-8?q?=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add the dotnet-benchmark skill to the codebelt agentic repository. Includes SKILL.md with workflow and conventions, FORMS.md for parameter collection, assets with benchmark templates and runner projects, references with BenchmarkDotNet essentials and codebelt conventions, scripts for checking benchmark requirements, and evals/evals.json for testing. --- skills/dotnet-benchmark/FORMS.md | 71 ++++++++ skills/dotnet-benchmark/SKILL.md | 171 ++++++++++++++++++ .../assets/benchmark-program.cs | 25 +++ .../assets/benchmark-runner.csproj | 16 ++ .../dotnet-benchmark/assets/benchmark.csproj | 11 ++ .../assets/params-benchmark.cs | 46 +++++ .../assets/simple-benchmark.cs | 32 ++++ skills/dotnet-benchmark/evals/evals.json | 70 +++++++ .../references/benchmarkdotnet-essentials.md | 78 ++++++++ .../references/codebelt-conventions.md | 129 +++++++++++++ .../dotnet-benchmark/references/onboarding.md | 114 ++++++++++++ .../scripts/check-benchmark-requirements.ps1 | 115 ++++++++++++ 12 files changed, 878 insertions(+) create mode 100644 skills/dotnet-benchmark/FORMS.md create mode 100644 skills/dotnet-benchmark/SKILL.md create mode 100644 skills/dotnet-benchmark/assets/benchmark-program.cs create mode 100644 skills/dotnet-benchmark/assets/benchmark-runner.csproj create mode 100644 skills/dotnet-benchmark/assets/benchmark.csproj create mode 100644 skills/dotnet-benchmark/assets/params-benchmark.cs create mode 100644 skills/dotnet-benchmark/assets/simple-benchmark.cs create mode 100644 skills/dotnet-benchmark/evals/evals.json create mode 100644 skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md create mode 100644 skills/dotnet-benchmark/references/codebelt-conventions.md create mode 100644 skills/dotnet-benchmark/references/onboarding.md create mode 100644 skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 diff --git a/skills/dotnet-benchmark/FORMS.md b/skills/dotnet-benchmark/FORMS.md new file mode 100644 index 0000000..8b1109e --- /dev/null +++ b/skills/dotnet-benchmark/FORMS.md @@ -0,0 +1,71 @@ +# Parameter Form + +`dotnet-benchmark` collects a small number of inputs. Present each field **one at a time** using the +host's native input mechanism (e.g. `ask_user` with `choices`) when available. If native structured +input is unavailable, use the deterministic plain-text fallback in the Presentation Rules below. Do +not bundle multiple fields into a single message. + +Most fields have smart defaults derived from inspecting the repo and the target type, so a normal run +asks very little. Skip any field whose value is already unambiguous from the conversation (e.g. the +user already named the type). + +## Fields + +### sut_type +- **type:** text +- **prompt:** "Which type do you want to performance-test? (namespace-qualified if ambiguous)" +- **placeholder:** "e.g. Cuemon.DateSpan or Acme.Buffers.RingBuffer" +- **required:** true +- **description:** The System Under Test. Resolve it in the source tree to learn its namespace, owning + `src/` project, and public surface. If the user already named a type, accept it and continue. + +### benchmark_tier +- **type:** single-choice +- **prompt:** "How thorough should the benchmark be?" +- **choices:** + - Auto — inspect the type and pick the best shape (Recommended) + - Simple — member scenarios (construct / format / equals / hash) + - Complex — sweep input sizes/variants with [Params] and GlobalSetup +- **default:** Auto — inspect the type and pick the best shape (Recommended) +- **description:** Auto uses the complexity heuristics in `references/codebelt-conventions.md`. State + which tier you chose and why, then let the user override. + +### target_runtimes +- **type:** multi-choice +- **prompt:** "Which runtimes should the benchmark jobs measure?" +- **choices:** + - Runner default only (Recommended) + - .NET 10 (CoreRuntime.Core10_0) + - .NET 9 (CoreRuntime.Core90) + - .NET 8 (CoreRuntime.Core80) + - .NET Framework 4.8 (ClrRuntime.Net48, Windows only) +- **default:** Runner default only (Recommended) +- **description:** The runner host targets .NET 9/10, but BenchmarkDotNet **jobs** can measure other + runtimes. Only offer runtimes the benchmark project can target (its `TargetFrameworks` must include + the matching TFM). Adding extra runtimes multiplies run time. + +### run_now +- **type:** single-choice +- **prompt:** "Run the benchmark now, or just wire it up and give you the command?" +- **choices:** + - Just wire it up and give me the command (Recommended) + - Run it now +- **default:** Just wire it up and give me the command (Recommended) +- **description:** BenchmarkDotNet runs are slow and heavy. Default is to verify the Release build and + hand off the run command. Only run it when the user explicitly asks. + +## Presentation Rules + +1. Ask one field at a time — wait for the answer before presenting the next field. +2. Prefer the host's native structured input controls for every field when available. +3. If native controls are unavailable, use this plain-text fallback: + - Start with `Field: ` + - Repeat the field prompt verbatim + - For choice fields, show a numbered option list; accept the number or exact option text + - For `text` fields with a default, show `1. Use "" (Recommended)` and `2. Enter a custom value` + - After the user answers, restate the normalized value in one short line before moving on +4. When a field has a `default`, present it first and append "(Recommended)" if not already labeled. +5. Treat a blank response on a field that has a default as accepting that default — do not re-ask. +6. Skip a field entirely when its value is already clear from context (e.g. the user said "benchmark + DateSpan" — `sut_type` is answered). +7. After collecting fields, briefly confirm the plan (type, tier, runtimes, run-or-not) before writing. diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md new file mode 100644 index 0000000..b5151bb --- /dev/null +++ b/skills/dotnet-benchmark/SKILL.md @@ -0,0 +1,171 @@ +--- +name: dotnet-benchmark +description: > + Set up and author BenchmarkDotNet performance tests for a specific .NET type following codebelt + engineering conventions, using Codebelt.Extensions.BenchmarkDotNet and its Console runner. Use this + skill whenever the user wants to benchmark, micro-benchmark, performance-test, profile throughput or + allocations, or measure the speed of a .NET type or method, in new or existing projects. It first + checks that the benchmark harness and prerequisites exist and sets up anything missing in place + (tuning/ benchmark project, tooling/ runner host, package references, solution wiring), then inspects + the target type and picks a complexity-appropriate strategy, authoring the benchmark class in the same + namespace as the code it measures. Trigger phrases include "add a benchmark", "benchmark this class", + "set up BenchmarkDotNet", "performance test", "micro-benchmark", or "measure allocations". Also use it + when a repo already has a tuning/ or *.Benchmarks project and wants more benchmarks. +--- + +# .NET Benchmark Setup (Codebelt Conventions) + +Make it easy to performance-test a .NET **type** with [BenchmarkDotNet](https://benchmarkdotnet.org/) +the codebelt way, wiring the benchmark into the same `tuning/` + `tooling/` layout used across +[codebeltnet](https://github.com/codebeltnet). This skill works for a repo that already has a +benchmark harness *and* one that has none: it detects what exists and adds only what is missing. + +The two reference implementations this skill mirrors are `codebeltnet/cuemon` and +`codebeltnet/xunit`. When in doubt about a convention, default to how those repos do it. The +[`Codebelt.Extensions.BenchmarkDotNet`](https://benchmarkdotnet.codebelt.net/api/Codebelt.Extensions.BenchmarkDotNet.html) +namespace and its `.Console` companion supply the runner host (`BenchmarkProgram.Run`), so you never +hand-roll a `BenchmarkSwitcher`. + +## Why this layout + +Benchmarks are split across three sibling folders so they never leak into shippable output: + +- `tuning/{SutProject}.Benchmarks/` holds the benchmark **projects and classes** that reference the + code under test. +- `tooling/{runner}/` holds one executable **runner host** that discovers every `tuning/` project + and runs it through the Codebelt console bootstrapper. +- `reports/` receives the generated benchmark artifacts. + +Keeping the runner in `tooling/` and the benchmarks in `tuning/` means the packable `src/` projects +stay clean, and a single runner can drive many benchmark projects. + +## Workflow + +Do the steps in order. Each step explains *why* so you can adapt when a repo does not match the +happy path — real existing repos rarely do. + +### Step 1: Check requirements + +Run the detection script to learn the repo's current state in one pass instead of guessing: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot +``` + +It reports, as JSON: whether the .NET SDK is available (and version), the solution file(s) and their +format (`.slnx` vs `.sln`), whether Central Package Management (`Directory.Packages.props`) is used, +whether the root `Directory.Build.props` already centralizes benchmark/tooling conventions +(`IsBenchmarkProject` / `IsToolingProject`), any existing `tuning/*.Benchmarks` projects, and any +existing `tooling/` runner host (its folder name and whether it references +`Codebelt.Extensions.BenchmarkDotNet.Console`). + +The one hard prerequisite is the **.NET SDK**. If it is missing, stop and ask the user to install it +(the runner host targets .NET 9 or .NET 10, matching `Codebelt.Extensions.BenchmarkDotNet` +availability). Everything else in the harness this skill can create for them. + +### Step 2: Onboard the missing harness (in place) + +Add only what Step 1 found missing, matching the repo's existing layout. Do not restructure a repo or +convert its solution format. Read `references/onboarding.md` for the detailed decision tree; the +essentials: + +- **Packages.** Resolve the latest stable listed versions from NuGet.org (never hardcode) for + `BenchmarkDotNet`, `BenchmarkDotNet.Diagnostics.Windows`, and + `Codebelt.Extensions.BenchmarkDotNet.Console`. If the repo uses Central Package Management, add + `` entries to `Directory.Packages.props` and reference them without versions; + otherwise put versioned ``s directly in the project files. +- **Benchmark project.** Create `tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj` from + `assets/benchmark.csproj`, referencing the SUT `src/` project and overriding `RootNamespace` to the + SUT root namespace (so the benchmark lives in the measured namespace, not a `.Benchmarks` one). +- **Runner host.** If no `tooling/` runner exists, create one from `assets/benchmark-runner.csproj` + and `assets/benchmark-program.cs`. Default its folder name to `benchmark-runner`; if the repo + already has a runner (e.g. cuemon's `bdn-runner`), reuse it — do not add a second one. +- **Central conventions vs plain repo.** If the root `Directory.Build.props` already centralizes + `IsBenchmarkProject`/`IsToolingProject` (as codebelt repos do), keep the benchmark `.csproj` + minimal and let the props inject TFMs and BDN packages. If it does not, the project files must + declare their own `TargetFrameworks` and package references — `references/onboarding.md` shows both. +- **Solution wiring.** Add the new projects to the detected solution: for `.slnx`, add + `` entries under `/tuning/` and `/tooling/` folders; for `.sln`, use + `dotnet sln add `. + +### Step 3: Resolve the target type + +Ask which type to performance-test if the user has not already named one. Then locate it in the +source tree to learn its namespace, owning `src/` project, and public surface (constructors, +methods, properties, and any obvious size- or variant-sensitive inputs). You need the namespace to +place the benchmark correctly and the surface to choose meaningful scenarios. + +### Step 4: Choose a strategy by complexity + +The user asked for a *thorough* performance test, so pick the approach that fits the type instead of +applying one rigid template. Read `references/benchmarkdotnet-essentials.md` for the attribute and +job toolbox and `references/codebelt-conventions.md` for the two default tiers: + +- **Simple type** (value-like, no size-sensitive input): benchmark the meaningful members as + discrete scenarios with a clear `Baseline = true` anchor, grouped + `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]`. Template: `assets/simple-benchmark.cs` + (mirrors cuemon `DateSpanBenchmark`). +- **Complex / size- or variant-sensitive type** (hashing, parsing, buffers, algorithm variants): use + `[Params]` to sweep input sizes and/or variants, prepare deterministic payloads in `[GlobalSetup]`, + and compare implementations against a baseline. Template: `assets/params-benchmark.cs` (mirrors + cuemon `Sha512256Benchmark` and xunit `TestBenchmark`). + +Briefly tell the user which tier you chose and why, then let them adjust (e.g. specific methods, +input sizes, or a competing implementation to compare against). Every benchmark uses +`[MemoryDiagnoser]` so allocations are always captured. + +### Step 5: Author and wire the benchmark class + +Write the benchmark into `tuning/{SutProject}.Benchmarks/` following codebelt naming exactly, because +these rules keep type discovery and reports consistent: + +- Class name ends with `Benchmark` (e.g. `DateSpanBenchmark`). +- Namespace is the **same** as the SUT — never suffix `.Benchmarks`. The `RootNamespace` override in + the project file is what makes this compile cleanly. +- Methods use descriptive scenario names (`Parse_Short`, `ComputeHash_Large`) and a `Description` for + readable reports; mark the reference method `Baseline = true`. +- Use deterministic data and no external systems (no network, disk, or DB) so runs are repeatable. + +If the type belongs to a `src/` project that has no `tuning/{SutProject}.Benchmarks` yet, create that +project (Step 2 rules) before adding the class, then make sure the runner discovers it (the wildcard +`..\..\tuning\**\*.csproj` reference already covers new projects) and the solution lists it. + +### Step 6: Verify the build, then hand off the run + +Confirm the benchmark compiles in Release, since BenchmarkDotNet only runs Release builds: + +```powershell +dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj +``` + +Do **not** run the benchmark by default — real runs are slow and heavy. Offer to run it, and give the +exact command so the user can run it when ready. The runner is a console app that accepts BenchmarkDotNet +filters: + +```powershell +dotnet run -c Release --project tooling/{runner} -- --filter *{TypeName}Benchmark* +``` + +Reports land under `reports/`. Only run it yourself if the user explicitly asks. + +### Multi-runtime jobs (optional) + +`Codebelt.Extensions.BenchmarkDotNet` runs on .NET 9/10, but its BenchmarkDotNet **jobs** can measure +other runtimes. If the user wants to compare across runtimes, add jobs in the runner's `Program.cs` +using `slimJob.WithRuntime(...)` — e.g. `ClrRuntime.Net48` (older .NET Framework), +`CoreRuntime.Core80/90/10_0`. xunit's runner does exactly this. See +`references/benchmarkdotnet-essentials.md` for the moniker map. + +## Conventions checklist + +Before finishing, verify: + +- [ ] `.NET SDK` present; runner host targets net9.0 or net10.0 +- [ ] Benchmark class ends with `Benchmark` and lives in the SUT's namespace (no `.Benchmarks` suffix) +- [ ] Benchmark project sets `` to the SUT root and references the SUT `src/` project +- [ ] `[MemoryDiagnoser]` present; a `Baseline = true` method anchors the comparison +- [ ] Deterministic data only — no network/disk/DB in measured methods +- [ ] Packages resolved from NuGet.org (no hardcoded versions); CPM vs `PackageReference` matches the repo +- [ ] Exactly one `tooling/` runner host; new benchmark project added to the detected `.slnx`/`.sln` +- [ ] Release build succeeds; run command provided (benchmark not run unless requested) +- [ ] Generated files are UTF-8 with no mojibake diff --git a/skills/dotnet-benchmark/assets/benchmark-program.cs b/skills/dotnet-benchmark/assets/benchmark-program.cs new file mode 100644 index 0000000..fd93e1d --- /dev/null +++ b/skills/dotnet-benchmark/assets/benchmark-program.cs @@ -0,0 +1,25 @@ +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Environments; +using BenchmarkDotNet.Jobs; +using Codebelt.Extensions.BenchmarkDotNet; +using Codebelt.Extensions.BenchmarkDotNet.Console; + +namespace {RUNNER_NAMESPACE}; + +public class Program +{ + public static void Main(string[] args) + { + BenchmarkProgram.Run(args, o => + { + o.AllowDebugBuild = BenchmarkProgram.IsDebugBuild; + o.SkipBenchmarksWithReports = true; + o.ConfigureBenchmarkDotNet(c => + { + var slimJob = BenchmarkWorkspaceOptions.Slim; + return c +{RUNTIME_JOBS}; + }); + }); + } +} diff --git a/skills/dotnet-benchmark/assets/benchmark-runner.csproj b/skills/dotnet-benchmark/assets/benchmark-runner.csproj new file mode 100644 index 0000000..13c3ff9 --- /dev/null +++ b/skills/dotnet-benchmark/assets/benchmark-runner.csproj @@ -0,0 +1,16 @@ + + + + {RUNNER_TARGET_FRAMEWORK} + {RUNNER_NAMESPACE} + + + + + + + + + + + diff --git a/skills/dotnet-benchmark/assets/benchmark.csproj b/skills/dotnet-benchmark/assets/benchmark.csproj new file mode 100644 index 0000000..7b2e68a --- /dev/null +++ b/skills/dotnet-benchmark/assets/benchmark.csproj @@ -0,0 +1,11 @@ + + + + {ROOT_NAMESPACE} + + + + + + + diff --git a/skills/dotnet-benchmark/assets/params-benchmark.cs b/skills/dotnet-benchmark/assets/params-benchmark.cs new file mode 100644 index 0000000..eb3ddb5 --- /dev/null +++ b/skills/dotnet-benchmark/assets/params-benchmark.cs @@ -0,0 +1,46 @@ +using System; +using System.Collections.Generic; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace {SUT_NAMESPACE} +{ + // Complex-tier template: sweep input sizes and/or implementation variants with [Params], build + // deterministic payloads once in [GlobalSetup], and compare candidates against a baseline. Use + // this shape for size- or variant-sensitive types (hashing, parsing, buffers, algorithms). + // Replace Variant, the payload sizes, and the measured calls with the real API of {SUT_TYPE}. + [MemoryDiagnoser] + [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] + public class {SUT_TYPE}Benchmark + { + public enum Variant + { + Baseline, + Candidate + } + + [Params(Variant.Baseline, Variant.Candidate)] + public Variant Implementation { get; set; } + + // Sweep representative micro / mid / macro sizes so trends are visible. + [Params(64, 4096, 1_048_576)] + public int Size { get; set; } + + private byte[] _payload; + + [GlobalSetup] + public void Setup() + { + // Seeded RNG keeps payloads deterministic across runs. + var rng = new Random(42); + _payload = new byte[Size]; + rng.NextBytes(_payload); + } + + [Benchmark(Baseline = true, Description = "Process (baseline)")] + public int Process_Baseline() => {SUT_TYPE}.Process(_payload); + + [Benchmark(Description = "Process (candidate)")] + public int Process_Candidate() => {SUT_TYPE}.ProcessOptimized(_payload); + } +} diff --git a/skills/dotnet-benchmark/assets/simple-benchmark.cs b/skills/dotnet-benchmark/assets/simple-benchmark.cs new file mode 100644 index 0000000..1a4c5bd --- /dev/null +++ b/skills/dotnet-benchmark/assets/simple-benchmark.cs @@ -0,0 +1,32 @@ +using System; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace {SUT_NAMESPACE} +{ + // Simple-tier template: benchmark the meaningful members of a value-like type as discrete + // scenarios. Keep one method marked Baseline = true as the comparison anchor. Replace the + // placeholder members below with the real API of {SUT_TYPE}. + [MemoryDiagnoser] + [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] + public class {SUT_TYPE}Benchmark + { + private {SUT_TYPE} _instance; + + [GlobalSetup] + public void Setup() + { + // Deterministic, cheap-to-build state prepared once (never measured). + _instance = new {SUT_TYPE}(); + } + + [Benchmark(Baseline = true, Description = "Construct")] + public {SUT_TYPE} Construct() => new {SUT_TYPE}(); + + [Benchmark(Description = "ToString")] + public string ToStringScenario() => _instance.ToString(); + + [Benchmark(Description = "GetHashCode")] + public int GetHashCodeScenario() => _instance.GetHashCode(); + } +} diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json new file mode 100644 index 0000000..04cc477 --- /dev/null +++ b/skills/dotnet-benchmark/evals/evals.json @@ -0,0 +1,70 @@ +{ + "skill_name": "dotnet-benchmark", + "evals": [ + { + "id": 1, + "prompt": "This repo has a src/Acme.Core/Acme.Core.csproj library but no benchmarks at all. Set up BenchmarkDotNet so I can start benchmarking, following the codebelt convention.", + "expected_output": "The harness is onboarded in place: a tuning/Acme.Core.Benchmarks project referencing the SUT src project, a single tooling/ runner host that references Codebelt.Extensions.BenchmarkDotNet.Console, benchmark packages added (CPM or PackageReference as the repo dictates), and the new projects wired into the detected solution.", + "expectations": [ + "Runs or references scripts/check-benchmark-requirements.ps1 to detect current state before changing anything", + "Creates tuning/Acme.Core.Benchmarks with a ProjectReference to src/Acme.Core/Acme.Core.csproj and RootNamespace set to the SUT root namespace", + "Creates exactly one tooling/ runner host whose Program.cs calls BenchmarkProgram.Run and defaults the folder name to benchmark-runner", + "Resolves BenchmarkDotNet, BenchmarkDotNet.Diagnostics.Windows, and Codebelt.Extensions.BenchmarkDotNet.Console versions from NuGet instead of hardcoding", + "Wires the new projects into the detected .slnx or .sln solution format" + ] + }, + { + "id": 2, + "prompt": "Add a thorough benchmark for the Acme.DateWindow value type (constructor, ToString, equality, GetHashCode). It's a small value type.", + "expected_output": "A simple-tier benchmark class DateWindowBenchmark in namespace Acme (no .Benchmarks suffix), using MemoryDiagnoser, GroupBenchmarksBy ByCategory, a GlobalSetup, and a Baseline=true anchor, benchmarking the members as discrete scenarios.", + "expectations": [ + "Chooses the simple/member-scenario tier and briefly explains why", + "Names the class DateWindowBenchmark and declares it in the same namespace as the SUT (no .Benchmarks suffix)", + "Applies [MemoryDiagnoser] and marks exactly one method Baseline = true with Description values", + "Places the class under tuning/{SutProject}.Benchmarks and does not run the benchmark by default" + ] + }, + { + "id": 3, + "prompt": "Benchmark our Acme.Hashing.Crc32 implementation against the built-in one across small and large inputs — I want to see allocations too.", + "expected_output": "A complex-tier benchmark using [Params] to sweep input sizes (and/or an implementation variant), deterministic payloads built in GlobalSetup with a seeded Random, MemoryDiagnoser, and a Baseline=true method comparing implementations.", + "expectations": [ + "Chooses the complex/params tier for a size- and variant-sensitive type", + "Uses [Params] to sweep multiple input sizes and prepares deterministic payloads in [GlobalSetup]", + "Includes [MemoryDiagnoser] and a Baseline = true comparison anchor", + "Uses deterministic in-memory data with no network, disk, or database in measured methods" + ] + }, + { + "id": 4, + "prompt": "I want the benchmark for Acme.Core's Parser to also compare .NET Framework 4.8 against .NET 9 and .NET 10.", + "expected_output": "Runner Program.cs gains one AddJob(slimJob.WithRuntime(...)) per requested runtime (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0), and the benchmark project targets the matching TFMs so those jobs can run.", + "expectations": [ + "Adds one .AddJob(slimJob.WithRuntime(...)) line per requested runtime using the correct monikers (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0)", + "Ensures the benchmark project TargetFrameworks include the matching TFMs (e.g. net48) so the jobs are runnable", + "Explains that the Codebelt runner host runs on .NET 9/10 while BenchmarkDotNet jobs can measure other runtimes" + ] + }, + { + "id": 5, + "prompt": "Our repo already has tooling/bdn-runner and a tuning/ folder with other benchmarks. Add a benchmark for Cuemon.Security.Cryptography.SHA512256.", + "expected_output": "The existing bdn-runner is reused (no second runner is created), the benchmark class is added under the appropriate tuning/*.Benchmarks project in namespace Cuemon.Security.Cryptography, and the runner's existing wildcard tuning reference picks it up.", + "expectations": [ + "Detects and reuses the existing tooling/bdn-runner instead of creating a benchmark-runner", + "Adds the class to the matching tuning/*.Benchmarks project in namespace Cuemon.Security.Cryptography with a Sha512256-style Benchmark class name", + "Does not duplicate the runner host or change the existing wildcard tuning ProjectReference" + ] + }, + { + "id": 6, + "prompt": "Set up benchmarking for a plain SDK-style repo that uses a classic MyApp.sln and does not use central package management.", + "expected_output": "Onboarding adapts to the non-codebelt layout: the benchmark project declares its own TargetFrameworks and versioned BenchmarkDotNet PackageReferences (no Directory.Packages.props), the runner declares its Console PackageReference version, and projects are added with dotnet sln add against the .sln.", + "expectations": [ + "Detects the classic .sln format and non-CPM state and adapts instead of assuming the codebelt-centralized layout", + "Puts versioned entries directly in the project files because there is no Directory.Packages.props", + "Adds self-contained TargetFrameworks and IsPackable=false to the benchmark/runner projects since the root Directory.Build.props does not centralize benchmark conventions", + "Wires projects into the solution using dotnet sln MyApp.sln add" + ] + } + ] +} diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md new file mode 100644 index 0000000..2374b2e --- /dev/null +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -0,0 +1,78 @@ +# BenchmarkDotNet Essentials + +A compact toolbox for authoring benchmarks. Full docs: https://benchmarkdotnet.org/articles/overview.html + +## Core attributes + +| Attribute | Purpose | +|-----------|---------| +| `[Benchmark]` | Marks a measured method. Add `Baseline = true` on the reference method and `Description = "..."` for readable reports. | +| `[MemoryDiagnoser]` | Captures allocations and GC counts. Always include it — codebelt benchmarks care about allocations, not just time. | +| `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]` | Groups related methods so comparisons read cleanly. Use `ByParams` when the story is "same operation across sizes/variants". | +| `[Params(...)]` | Sweeps input values (sizes, enum variants). BenchmarkDotNet runs every method once per combination. | +| `[ParamsSource(nameof(...))]` | Use when the parameter set is computed rather than literal. | +| `[GlobalSetup]` | One-time initialization that is **not** measured. Build payloads and instances here. | +| `[IterationSetup]` / `[IterationCleanup]` | Per-iteration hooks; use sparingly (they add overhead) for state that must reset each iteration. | +| `[Arguments(...)]` | Passes literal arguments to a benchmark method — lighter than `[Params]` for a few fixed cases. | + +## Choosing what to measure + +- Keep each `[Benchmark]` method to a **single logical operation**; move setup out of the measured path. +- Return a value from the method (or consume inputs) so the JIT cannot optimize the work away. +- Use deterministic, in-memory data. No network, disk, or database in measured methods — they destroy + repeatability and are not micro-benchmarks. +- Name methods for the scenario (`Parse_Short`, `ComputeHash_Large`, `Match_ComplexWildcard`) so the + report is self-describing. + +## Diagnosers worth knowing + +- `[MemoryDiagnoser]` — allocations (default for codebelt). +- `[DisassemblyDiagnoser]` — emitted asm; heavy, opt-in for deep dives. +- `BenchmarkDotNet.Diagnostics.Windows` (`[EtwProfiler]`, native counters) — Windows-only; referenced + by codebelt benchmark projects but enable specific diagnosers only when needed. + +## Jobs and runtimes + +A **job** describes how to run a benchmark. The Codebelt runner starts from `BenchmarkWorkspaceOptions.Slim` +and you add jobs fluently in the runner's `Program.cs`: + +```csharp +return c + .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); +``` + +Although `Codebelt.Extensions.BenchmarkDotNet` itself targets .NET 9/10, the **jobs** can measure +older and newer runtimes. Runtime moniker map: + +| Target | Job runtime | +|--------|-------------| +| .NET Framework 4.8 | `ClrRuntime.Net48` (Windows only) | +| .NET 8 | `CoreRuntime.Core80` | +| .NET 9 | `CoreRuntime.Core90` | +| .NET 10 | `CoreRuntime.Core10_0` | +| Mono | `MonoRuntime.Default` | + +Only add runtimes the benchmark project actually targets (its `TargetFrameworks` must include the +matching TFM, e.g. `net48` for `ClrRuntime.Net48`). Docs: https://benchmarkdotnet.org/articles/configs/jobs.html + +Other useful job knobs (usually leave BenchmarkDotNet's smart defaults alone): `RunStrategy` +(`Throughput`/`ColdStart`/`Monitoring`), `WarmupCount`, `IterationCount`, `LaunchCount`, `Platform`, +`GcMode.Server`. Set these only for a specific reason. + +## Running + +BenchmarkDotNet requires a **Release** build. Through the Codebelt console runner: + +```powershell +dotnet run -c Release --project tooling/{runner} -- --filter *{TypeName}Benchmark* +``` + +Common runner/BDN switches passed after `--`: + +- `--filter ` — select benchmarks by full name (`*DateSpanBenchmark*`, `*.Parse_*`). +- `--list flat` — list discovered benchmarks without running. +- `--job short` — a faster, less precise job for smoke checks. + +Reports are written under `reports/`. diff --git a/skills/dotnet-benchmark/references/codebelt-conventions.md b/skills/dotnet-benchmark/references/codebelt-conventions.md new file mode 100644 index 0000000..a863d6a --- /dev/null +++ b/skills/dotnet-benchmark/references/codebelt-conventions.md @@ -0,0 +1,129 @@ +# Codebelt Benchmark Conventions + +These rules mirror the pasted "Writing Performance Tests in Cuemon" guidance and the real +implementations in `codebeltnet/cuemon` and `codebeltnet/xunit`. Follow them so benchmarks stay +consistent, discoverable, and comparable across repos. + +## Naming and placement + +- Benchmark projects live under `tuning/` and are named `{SutProject}.Benchmarks` + (e.g. `Cuemon.Core.Benchmarks`, `Codebelt.Extensions.Xunit.Benchmarks`). +- A benchmark class name **ends with `Benchmark`** (e.g. `DateSpanBenchmark`, `Sha512256Benchmark`). +- The class lives in the **same namespace as the type it measures** — do **not** append `.Benchmarks`. + The benchmark project overrides `RootNamespace` to the SUT root so this compiles: + + ```xml + + Cuemon + + ``` + + So `Cuemon.Security.Cryptography.SHA512256` is benchmarked by a `Sha512256Benchmark` class declared + in `namespace Cuemon.Security.Cryptography`, inside the `Cuemon.Security.Cryptography.Benchmarks` + assembly. +- Method names are descriptive scenarios (`Parse_Short`, `ComputeHash_Large`, `Match_ComplexWildcard`). + +## Always-on attributes + +Every codebelt benchmark class carries: + +- `[MemoryDiagnoser]` +- `[GroupBenchmarksBy(...)]` — `ByCategory` for member-scenario suites, `ByParams` for size/variant sweeps +- a `[GlobalSetup]` that prepares deterministic state +- exactly one `[Benchmark(Baseline = true, ...)]` anchor, with `Description` on each method + +## Tier 1 — Simple type (member scenarios) + +For value-like types with no size-sensitive input, benchmark the meaningful members as discrete +scenarios. This is the `DateSpanBenchmark` shape: + +```csharp +namespace Cuemon +{ + [MemoryDiagnoser] + [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] + public class DateSpanBenchmark + { + private DateSpan _shortSpan; + + [GlobalSetup] + public void Setup() => _shortSpan = new DateSpan(DateTime.UtcNow, DateTime.UtcNow.AddHours(36)); + + [Benchmark(Baseline = true, Description = "Ctor (short span)")] + public DateSpan Construct_Short() => new DateSpan(DateTime.UtcNow, DateTime.UtcNow.AddHours(36)); + + [Benchmark(Description = "ToString (short)")] + public string ToString_Short() => _shortSpan.ToString(); + + [Benchmark(Description = "GetWeeks (short)")] + public int GetWeeks_Short() => _shortSpan.GetWeeks(); + } +} +``` + +Cover construction, parsing/formatting, equality, hashing, and any hot instance methods. Template: +`assets/simple-benchmark.cs`. + +## Tier 2 — Complex / size- or variant-sensitive type + +For hashing, parsing, buffers, or anything whose cost scales with input or has competing +implementations, sweep with `[Params]` and prepare payloads in `[GlobalSetup]`. Two real shapes: + +`Sha512256Benchmark` (variant + size, `ByParams`, compares custom vs built-in): + +```csharp +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] +public class Sha512256Benchmark +{ + public enum AlgorithmVariant { CustomSHA512_256, SHA512_Truncated } + + [Params(AlgorithmVariant.CustomSHA512_256, AlgorithmVariant.SHA512_Truncated)] + public AlgorithmVariant Variant { get; set; } + + private byte[] _smallInput; // 64 bytes + private byte[] _largeInput; // 1 MB + + [GlobalSetup] + public void GlobalSetup() { /* seeded Random(42) fills deterministic payloads */ } + + [Benchmark(Baseline = true, Description = "Custom SHA-512/256 - small")] + public byte[] CustomSHA512256_Small() { /* ... */ } +} +``` + +`TestBenchmark` (size sweep via `[Params(8, 256, 4096)]`, `ByCategory`): + +```csharp +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] +public class TestBenchmark +{ + [Params(8, 256, 4096)] + public int Length { get; set; } + + [GlobalSetup] + public void Setup() { /* build patterns/inputs from Length */ } + + [Benchmark(Baseline = true, Description = "Match - exact string")] + public bool Match_Exact() => Test.Match(_shortPattern, _shortActual); +} +``` + +Template: `assets/params-benchmark.cs`. Prefer seeded RNG (`new Random(42)`) and fixed sizes so runs +are deterministic. Choose micro / mid / macro sizes to reveal trends. + +## How to pick a tier + +Lean Tier 2 when the type: takes a collection/stream/buffer/string whose length matters, has multiple +implementations worth comparing, exposes an algorithm with a size parameter, or is on a documented hot +path. Otherwise Tier 1 is enough. When unsure, ask the user which members and input sizes matter most — +they know the hot paths. + +## Reference files (source of truth) + +- `codebeltnet/cuemon/tuning/Cuemon.Core.Benchmarks/DateSpanBenchmark.cs` +- `codebeltnet/cuemon/tuning/Cuemon.Security.Cryptography.Benchmarks/Sha512256Benchmark.cs` +- `codebeltnet/xunit/tuning/Codebelt.Extensions.Xunit.Benchmarks/TestBenchmark.cs` +- `codebeltnet/cuemon/tooling/bdn-runner/Program.cs` and + `codebeltnet/xunit/tooling/benchmark-runner/Program.cs` (runner + multi-runtime jobs) diff --git a/skills/dotnet-benchmark/references/onboarding.md b/skills/dotnet-benchmark/references/onboarding.md new file mode 100644 index 0000000..f0c79cf --- /dev/null +++ b/skills/dotnet-benchmark/references/onboarding.md @@ -0,0 +1,114 @@ +# Onboarding a Benchmark Harness Into an Existing Repo + +Use this when Step 1 detection shows the harness is missing or partial. The goal is to add **only** +what is missing while matching whatever layout the repo already uses. Never restructure the repo, +rename its solution, or convert `.sln` to `.slnx`. + +## Decision inputs (from `scripts/check-benchmark-requirements.ps1`) + +| Signal | Why it matters | +|--------|----------------| +| `sdk` | Hard prerequisite. If absent, stop and ask the user to install the .NET SDK. | +| `solution` / `solutionFormat` | Determines how you wire new projects (`.slnx` XML vs `dotnet sln add`). | +| `centralPackageManagement` | Chooses `` in `Directory.Packages.props` vs versioned ``. | +| `centralizesBenchmarkConventions` | If the root `Directory.Build.props` defines `IsBenchmarkProject`/`IsToolingProject`, project files stay minimal. | +| `benchmarkProjects` | Existing `tuning/*.Benchmarks` projects to reuse instead of recreating. | +| `runner` | Existing `tooling/` runner host (folder name + whether it references the Console package). Reuse it; do not add a second. | + +## 1. Packages + +Resolve the **latest stable listed** versions from NuGet.org at author time (never hardcode from an +example). Resolve each package ID independently: + +- `BenchmarkDotNet` +- `BenchmarkDotNet.Diagnostics.Windows` +- `Codebelt.Extensions.BenchmarkDotNet.Console` + +Resolution source: the NuGet V3 service index `https://api.nuget.org/v3/index.json`; prefer the +registration resource so you can skip unlisted and prerelease versions. + +- **CPM repo** (`Directory.Packages.props` present): add `` + entries, and reference the packages **without** a version in the project files. +- **Non-CPM repo**: put versioned `` directly in the + project files that need them (benchmark project needs the two `BenchmarkDotNet*` packages; runner + needs the Console package). + +## 2. Benchmark project (`tuning/{SutProject}.Benchmarks/`) + +Create from `assets/benchmark.csproj`. Substitutions: + +| Placeholder | Value | +|-------------|-------| +| `{ROOT_NAMESPACE}` | The SUT's root namespace (e.g. `Cuemon`), so benchmarks compile into the measured namespace. | +| `{SUT_PROJECT}` | The owning `src/` project name (e.g. `Cuemon.Core`). | + +The default `assets/benchmark.csproj` is the **minimal** codebelt form and assumes the root +`Directory.Build.props` injects the benchmark TFMs and `BenchmarkDotNet*` packages (as codebelt repos +do). If `centralizesBenchmarkConventions` is **false**, make the project self-contained by adding: + +```xml + + net10.0;net9.0 + false + + + + + + +``` + +(Use versioned ``s here if the repo is non-CPM.) Pick benchmark TFMs the SUT +actually supports; `net10.0;net9.0` matches `Codebelt.Extensions.BenchmarkDotNet` availability. + +## 3. Runner host (`tooling/{runner}/`) + +If `runner` already exists, **reuse it** — the wildcard `..\..\tuning\**\*.csproj` reference already +picks up new benchmark projects, so you usually change nothing here. Only create a runner when none +exists. + +When creating one, default the folder name to `benchmark-runner` (matches the codebelt library +scaffold and `codebeltnet/xunit`). Copy `assets/benchmark-runner.csproj` and +`assets/benchmark-program.cs`. Substitutions: + +| Placeholder | Value | +|-------------|-------| +| `{RUNNER_TARGET_FRAMEWORK}` | Highest supported non-preview executable TFM (`net10.0` or `net9.0`). | +| `{RUNNER_NAMESPACE}` | Runner folder name converted to a valid C# identifier (`benchmark-runner` -> `benchmark_runner`). | +| `{RUNTIME_JOBS}` | One indented `.AddJob(slimJob.WithRuntime(...))` line per runtime to measure (see `benchmarkdotnet-essentials.md`). At minimum, add the runner's own TFM. | + +If the root `Directory.Build.props` does **not** mark tooling projects as executables, add +`Exe` and `false` to the runner `PropertyGroup`. + +## 4. Solution wiring + +- **`.slnx`**: add `` + under a `` element, and the runner under ``. + Create the folders if absent. Example: + + ```xml + + + + + + + ``` + +- **`.sln`**: run `dotnet sln .sln add ` for each new project. `dotnet` + places them under solution folders automatically. + +- **No solution file**: it is fine to leave projects unlisted; note it for the user. Do not fabricate + a solution unless they ask. + +## 5. `reports/` folder + +Benchmark output is written under `reports/` by the Codebelt workspace. You do not need to pre-create +it; the runner creates it on first run. Mention it so the user knows where results land. + +## Guardrails + +- Preserve UTF-8 (no BOM unless the source had one) when writing generated files; watch for mojibake. +- Do not commit or push; leave that to the user. +- If you cannot resolve a package version or cannot determine the SUT project, stop and report rather + than guessing. diff --git a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 new file mode 100644 index 0000000..5021567 --- /dev/null +++ b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 @@ -0,0 +1,115 @@ +#requires -Version 5.1 +<# +.SYNOPSIS + Detects the current state of a repository's BenchmarkDotNet harness for the dotnet-benchmark skill. + +.DESCRIPTION + Emits a single JSON object describing what already exists so the skill adds only what is missing. + Read-only: it inspects files and the dotnet CLI but changes nothing. + +.PARAMETER RepoRoot + Repository root to inspect. Defaults to the current directory. + +.EXAMPLE + powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot C:\src\myrepo +#> +[CmdletBinding()] +param( + [string]$RepoRoot = (Get-Location).Path +) + +$ErrorActionPreference = 'Stop' + +function Test-CommandExists { + param([string]$Name) + return [bool](Get-Command $Name -ErrorAction SilentlyContinue) +} + +$RepoRoot = (Resolve-Path -LiteralPath $RepoRoot).Path + +# --- .NET SDK --------------------------------------------------------------- +$sdkAvailable = $false +$sdkVersion = $null +if (Test-CommandExists 'dotnet') { + try { + $sdkVersion = (& dotnet --version 2>$null | Select-Object -First 1) + $sdkAvailable = -not [string]::IsNullOrWhiteSpace($sdkVersion) + } catch { + $sdkAvailable = $false + } +} + +# --- Solution files --------------------------------------------------------- +$rootFiles = @(Get-ChildItem -LiteralPath $RepoRoot -File -ErrorAction SilentlyContinue) +$slnx = @($rootFiles | Where-Object { $_.Extension -ieq '.slnx' } | Select-Object -ExpandProperty Name -Unique) +$sln = @($rootFiles | Where-Object { $_.Extension -ieq '.sln' } | Select-Object -ExpandProperty Name -Unique) +$solutionFormat = if ($slnx.Count -gt 0) { 'slnx' } elseif ($sln.Count -gt 0) { 'sln' } else { 'none' } +$solutionFiles = @($slnx + $sln) + +# --- Central Package Management -------------------------------------------- +$packagesProps = Join-Path $RepoRoot 'Directory.Packages.props' +$cpm = Test-Path -LiteralPath $packagesProps +$declaredPackages = @() +if ($cpm) { + $packagesText = [System.IO.File]::ReadAllText($packagesProps) + foreach ($id in @('BenchmarkDotNet', 'BenchmarkDotNet.Diagnostics.Windows', 'Codebelt.Extensions.BenchmarkDotNet.Console')) { + if ($packagesText -match [regex]::Escape("Include=`"$id`"")) { $declaredPackages += $id } + } +} + +# --- Root Directory.Build.props conventions -------------------------------- +$buildProps = Join-Path $RepoRoot 'Directory.Build.props' +$centralizesBenchmarkConventions = $false +if (Test-Path -LiteralPath $buildProps) { + $buildText = [System.IO.File]::ReadAllText($buildProps) + $centralizesBenchmarkConventions = ($buildText -match 'IsBenchmarkProject') -and ($buildText -match 'IsToolingProject') +} + +# --- Existing tuning benchmark projects ------------------------------------ +$tuningDir = Join-Path $RepoRoot 'tuning' +$benchmarkProjects = @() +if (Test-Path -LiteralPath $tuningDir) { + $benchmarkProjects = @( + Get-ChildItem -LiteralPath $tuningDir -Recurse -Filter *.Benchmarks.csproj -File -ErrorAction SilentlyContinue | + ForEach-Object { $_.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } + ) +} + +# --- Existing tooling runner host ------------------------------------------ +$toolingDir = Join-Path $RepoRoot 'tooling' +$runner = $null +if (Test-Path -LiteralPath $toolingDir) { + $runnerCsproj = Get-ChildItem -LiteralPath $toolingDir -Recurse -Filter *.csproj -File -ErrorAction SilentlyContinue | + Where-Object { + $text = [System.IO.File]::ReadAllText($_.FullName) + $text -match 'Codebelt\.Extensions\.BenchmarkDotNet\.Console' + } | Select-Object -First 1 + if ($runnerCsproj) { + $runner = [ordered]@{ + name = $runnerCsproj.Directory.Name + path = $runnerCsproj.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + referencesConsole = $true + } + } +} + +# --- reports/ --------------------------------------------------------------- +$reportsExists = Test-Path -LiteralPath (Join-Path $RepoRoot 'reports') + +$harnessReady = ($runner -ne $null) -and ($benchmarkProjects.Count -gt 0) + +$result = [ordered]@{ + repoRoot = $RepoRoot + sdk = [ordered]@{ available = $sdkAvailable; version = $sdkVersion } + solutionFormat = $solutionFormat + solutionFiles = $solutionFiles + centralPackageManagement = $cpm + declaredBenchmarkPackages = $declaredPackages + centralizesBenchmarkConventions = $centralizesBenchmarkConventions + benchmarkProjects = $benchmarkProjects + runner = $runner + reportsFolderExists = $reportsExists + harnessReady = $harnessReady +} + +$result | ConvertTo-Json -Depth 6 From f9f03b37a8aaa56cca759f4fda576a6ab473cf78 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 16 Jul 2026 21:53:52 +0200 Subject: [PATCH 02/38] =?UTF-8?q?=F0=9F=92=AC=20add=20dotnet-benchmark=20t?= =?UTF-8?q?o=20readme?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add dotnet-benchmark to the install commands, available skills table, and Why section. Includes the official skill description and benefits of using the skill for performance testing of .NET types. --- README.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/README.md b/README.md index b45a2c1..ae5d69c 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-new-lib-sln npx skills add https://github.com/codebeltnet/agentic --skill git-remote-release npx skills add https://github.com/codebeltnet/agentic --skill dotnet-change-impact npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-digest +npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark # npx skills add https://github.com/codebeltnet/agentic --skill another-skill ``` @@ -116,6 +117,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-diges | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | +| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Set up and author BenchmarkDotNet performance tests for a specific .NET type following codebelt conventions, using `Codebelt.Extensions.BenchmarkDotNet` and its `.Console` runner. Detects the existing harness (`.slnx`/`.sln`, central package management, `tuning/` benchmark projects, and the `tooling/` runner host — reusing an existing name like `benchmark-runner` or `bdn-runner`) and onboards only what is missing, in place, whatever the repo layout. Resolves benchmark package versions from NuGet, resolves the target type's namespace and public surface, then picks a complexity-appropriate strategy — member scenarios for simple value types, `[Params]` + `[GlobalSetup]` sweeps for size- or variant-sensitive types — placing the `*Benchmark` class in the SUT's own namespace via a `RootNamespace` override. Always uses `[MemoryDiagnoser]`, wires the project into the detected solution, verifies the Release build, and hands off the run command by default (runs are slow) while supporting BenchmarkDotNet jobs that measure older and newer runtimes such as `net48`, `net8.0`, `net9.0`, and `net10.0`. | ### Copyable Install Commands @@ -211,6 +213,12 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-change-impa npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-digest ``` +`dotnet-benchmark` + +```bash +npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark +``` + ### Why git-visual-commits? Commit messages are the most-read documentation in any codebase — yet they're usually an afterthought. "fix stuff", "wip", "address PR feedback" tells you nothing six months later. Writing good commits takes discipline, and when you're in flow, it's the first thing that slips. @@ -588,6 +596,21 @@ API documentation rots the moment code changes. A new public type ships without - **Cleanup keeps authored docs and is opt-in** — generated-metadata cleanup runs only when `--clean-generated-metadata` is explicitly passed, and even then only after the API model is built so it never deletes YAML the run relied on; `.docfx/**/*.md` overwrite files, namespace pages, includes, and config are documentation outputs, not disposable build artifacts, and cleanup is limited to known metadata files and safe site-output directories that contain no authored documentation or source files, - **CI-friendly** — deterministic exit codes plus `--json` reports let pipelines fail on actual documentation drift instead of trusting an agent's claim that it checked. +### Why dotnet-benchmark? + +Setting up a benchmark "properly" usually means copying a `tuning/` project from another repo, wiring a runner host, remembering which BenchmarkDotNet packages you need, and then deciding — every single time — how to structure the benchmark for the type in front of you. Most people skip it, and the performance question goes unanswered. + +**dotnet-benchmark** removes that friction for both greenfield and existing repos. It inspects what you already have and adds only the missing pieces, then authors a benchmark that fits the type instead of a one-size-fits-all template. + +- **Works on existing repos** — detects your solution format, package-management style, and any runner you already have (`benchmark-runner`, `bdn-runner`, …) and reuses it instead of forcing a new layout +- **Codebelt convention by default** — `tuning/` benchmark projects, a single `tooling/` runner host, and `reports/` output, mirroring `codebeltnet/cuemon` and `codebeltnet/xunit` +- **Right-sized strategy** — simple value types get member-scenario benchmarks; size- or variant-sensitive types get `[Params]` sweeps with deterministic `[GlobalSetup]` payloads and a baseline comparison +- **Namespace-correct** — the `*Benchmark` class lives in the same namespace as the code it measures, via a `RootNamespace` override, so type discovery and reports stay clean +- **Allocations always measured** — `[MemoryDiagnoser]` is on by default +- **Multi-runtime aware** — the runner host runs on .NET 9/10, but its BenchmarkDotNet jobs can compare `net48`, `net8.0`, `net9.0`, and `net10.0` +- **Latest stable packages** — `BenchmarkDotNet`, `BenchmarkDotNet.Diagnostics.Windows`, and `Codebelt.Extensions.BenchmarkDotNet.Console` versions are resolved from NuGet, not hardcoded +- **Safe hand-off** — verifies the Release build and gives you the run command; it won't kick off a slow benchmark run unless you ask + ## Repository structure ``` From 24c51b4ce9810c0bbbdeaafc0e1b0e022c8e3ca4 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Thu, 16 Jul 2026 23:06:14 +0200 Subject: [PATCH 03/38] =?UTF-8?q?=E2=9C=A8=20add=20changelog=20entry=20for?= =?UTF-8?q?=20dotnet-benchmark=20skill=20release?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d0e573..f64cf3b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,55 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ## [Unreleased] +## [0.8.0] - 2026-07-16 + +This is a minor release introducing the `dotnet-benchmark` skill, a comprehensive benchmarking solution for .NET types using BenchmarkDotNet. The skill scaffolds benchmark projects following codebelt engineering conventions, provides parameter collection and complexity-aware test strategies, includes benchmark runner infrastructure, and supplies essential BenchmarkDotNet and codebelt-specific guidance. + +### Added + +- `dotnet-benchmark` skill with workflow guidance for authoring and running BenchmarkDotNet performance tests for specific .NET types, supporting complexity-appropriate strategies (simple, parameterized, fixture-based), +- `FORMS.md` for `dotnet-benchmark` with structured parameter collection for target type, benchmark family, and complexity level, +- Benchmark project templates and runner infrastructure including `benchmark.csproj`, `benchmark-runner.csproj`, parameterized benchmark templates, and benchmark program entry points, +- `check-benchmark-requirements.ps1` script validating BenchmarkDotNet installation and NuGet feed accessibility before running benchmarks, +- Detailed reference documentation covering BenchmarkDotNet essentials (result interpretation, memory allocations, statistical confidence), codebelt conventions (namespace alignment, methodology rigor, result storage), and onboarding workflow for new benchmark authors, +- Eval coverage for `dotnet-benchmark` including target-type inspection, benchmark strategy selection, template application, and runner validation, +- README updates with `dotnet-benchmark` installation snippet, capability showcase, and "Why dotnet-benchmark?" section highlighting performance-test authoring for throughput and allocation measurement. + +## [0.7.5] - 2026-07-15 + +This is a patch release focused on extending `trunk-first-repo` with a push-remote workflow mode that safely handles first-time remote pushes by pushing `main` before feature branches, ensuring the remote defaults to the correct branch while maintaining the PR-first workflow philosophy. + +### Added + +- Push Remote Workflow mode in `trunk-first-repo` for safely pushing to a newly-established remote without manually switching branches or checking out `main`, allowing `push remote ` invocation from the feature branch to send `main` by ref (`main:main`) before the feature branch, +- Enhanced eval coverage for `trunk-first-repo` documenting push-remote workflow and Step 0 mode selection behavior, +- Updated README description for `trunk-first-repo` to document safe first-push capability and `push remote ` mode alongside the Initialize Workflow. + +### Changed + +- Extended `trunk-first-repo` SKILL.md with Step 0 mode selector to distinguish between Initialize Workflow (repository creation) and Push Remote Workflow (remote establishment), +- Refined README guidance to emphasize that `push remote ` can be invoked later from the feature branch for safer first-push without switching branches, +- Added explicit push-remote documentation to trunk-first-repo "Why?" section explaining safer first-push behavior and benefits of sending `main` by ref. + +## [0.7.4] - 2026-07-03 + +This is a patch release focused on strengthening `git-keep-a-changelog` with mandatory Step 4a base-commit inspection for concrete releases, ensuring that foundational version bumps, release-prep changes, and dependency baseline updates are never omitted from release narratives. The skill now requires explicit inspection of the base commit before manifest diffs and commit bodies, with output verification and structured reporting. + +### Added + +- Step 4a mandatory checkpoint in `git-keep-a-changelog` that inspects and explicitly reports the base commit for concrete releases (e.g., `## [X.Y.Z]`), showing changed files, identifying dependency/version manifests, and confirming release-prep file modifications before proceeding to Step 4b manifest diffs, +- Explicit base-commit-inclusion enforcement using `^..HEAD` (with caret) throughout Step 4 for concrete releases, ensuring the base commit itself is included in the changelog narrative, +- Verification and confirmation gates in Step 4a requiring agents to show full base commit output, identify manifests, and explicitly state whether manifests or release-prep files were touched before proceeding to 4b, +- Detailed comparison matrix in Step 3b distinguishing between `base^..HEAD` (for concrete releases, inclusive of base) and `base..HEAD` (for [Unreleased], exclusive of base), +- Eval coverage validating base-commit inclusion, manifest detection, and Step 4a output verification for concrete release scenarios. + +### Changed + +- Restructured `git-keep-a-changelog` Step 4 into explicit sub-steps (4a through 4f) with clear sequencing: base-commit inspection first (4a), manifest detection (4b), manifest diff inspection (4c), commit-body reading (4d), net-diff inspection (4e), and pending-change integration (4f), +- Enhanced `git-keep-a-changelog` SKILL.md with critical range-extension guidance for concrete releases, emphasizing that `^..HEAD` (with caret) must be used consistently to include the base commit itself, +- Strengthened "Bad Output Characteristics" section with **CRITICAL** emphasis on the consequences of omitting the base commit: silently-wrong output that breaks release narratives and loses foundational version bumps, +- Updated README with enhanced description of `git-keep-a-changelog` base-commit enforcement and Step 4a mandatory checkpoint. + ## [0.7.3] - 2026-07-01 This is a patch release focused on skill refinement and documentation improvements, including xref member-link validation enhancements to dotnet-docfx-digest, structural improvements to git-keep-a-changelog's manifest-diff reading, and emoji discipline improvements across git-visual skills. @@ -418,7 +467,10 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. -[Unreleased]: https://github.com/codebeltnet/agentic/compare/v0.7.3...HEAD +[Unreleased]: https://github.com/codebeltnet/agentic/compare/v0.7.5...HEAD +[0.8.0]: https://github.com/codebeltnet/agentic/compare/v0.7.5...v0.8.0 +[0.7.5]: https://github.com/codebeltnet/agentic/compare/v0.7.4...v0.7.5 +[0.7.4]: https://github.com/codebeltnet/agentic/compare/v0.7.3...v0.7.4 [0.7.3]: https://github.com/codebeltnet/agentic/compare/v0.7.2...v0.7.3 [0.7.2]: https://github.com/codebeltnet/agentic/compare/v0.7.1...v0.7.2 [0.7.1]: https://github.com/codebeltnet/agentic/compare/v0.7.0...v0.7.1 From f6c5829adda6cb5c550ed7cc34881dbf157f95f8 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 11:33:49 +0200 Subject: [PATCH 04/38] =?UTF-8?q?=F0=9F=93=9D=20clarify=20trunk-first-repo?= =?UTF-8?q?=20skill=20description?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update README.md table entry to reflect the refined push-remote workflow behavior and clearer origin configuration options. --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index ae5d69c..87146db 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark | [git-repo-digest](skills/git-repo-digest/SKILL.md) | Turns any full repository URL into a deterministic digest workspace using the bundled .NET file-based runner `scripts/digest.cs`. Requires explicit `--repo-url`, resolves omitted output paths to `/.bot/digests` and passes that as `--output-root`, maps multiple positional URLs the same way for slash commands, bare pasted URLs, and natural-language requests by treating the first URL as the digest repo and every later URL as repeated `--external-repo-url`, always writes into `{output-root}/{repo-id}/{yyyyMMdd-HHmmssZ}`, accepts repeated curated public consumer repos, derives `{repo-id}`, fixes `result/`, performs shallow git clones, packs local tracked files with the bundled C# packer using `git ls-files`, separates XML evidence into `source.xml`, `tests.xml`, `projects.xml`, editorial `readmes.xml`, and scenario-only `external-usage.xml`, writes package and conceptual overview prompts under `prompts/`, emits public API summaries, engineering signals, evidence indexes, ordered XML chunks, referenced-package evidence maps for aggregate examples, and manifest-backed frontmatter hints, treats previous digest prose as contamination during fresh generation, then guides the agent to fully read the current phase's required evidence before writing package digests and a concept-led `result/Index.md` with YAML frontmatter containing Product-derived overview title metadata, validated documentation URLs resolved from PackageProjectUrl, documentation-host-filtered exact `.nuget//README.md` documentation links including emoji-prefixed Documentation headings and "More documentation..." blocks, DocFX `metadata[].dest` API paths, and source namespace page candidates from `src//**/*.cs`, target frameworks, package/library counts, external links, package-family links, and context glyphs, and validates authored result examples with `--validate-results` as a deterministic API-shape, Codebelt.Extensions.Xunit shape, PascalCase `MethodName_Scenario_ExpectedBehavior` test-method naming, Basic usage quality, and optimized NuGet-backed executable test gate with bounded parallelism. | | [dotnet-new-lib-slnx](skills/dotnet-new-lib-slnx/SKILL.md) | Scaffold a new .NET NuGet library solution following codebeltnet engineering conventions. Dynamic defaults for TFM/repository metadata, latest-stable NuGet package resolution, tuning projects plus a tooling-based benchmark runner, TFM-aware test environments, strong-name signing, NuGet packaging, DocFX documentation, CI/CD pipeline, and code quality tooling. | | [dotnet-new-app-slnx](skills/dotnet-new-app-slnx/SKILL.md) | Scaffold a new .NET standalone application solution following codebeltnet engineering conventions. Supports Console, Web, and Worker host families with Startup or Minimal hosting patterns; Web expands into Empty Web, Web API, MVC, or Web App / Razor, plus functional tests and a simplified CI pipeline. | -| [trunk-first-repo](skills/trunk-first-repo/SKILL.md) | Initialize a git repository following [scaled trunk-based development](https://trunkbaseddevelopment.com/#scaled-trunk-based-development). Seeds an empty `main` branch, creates a versioned feature branch (`v0.1.0/init`), and supports a later `push remote ` mode that pushes `main` before feature branches so new remotes keep the right default branch while content reaches main only through peer-reviewed pull requests. | +| [trunk-first-repo](skills/trunk-first-repo/SKILL.md) | Initialize a git repository following [scaled trunk-based development](https://trunkbaseddevelopment.com/#scaled-trunk-based-development). Seeds an empty `main` branch, creates a versioned feature branch (`v0.1.0/init`), confirms configured remotes in its post-init summary, and supports a guarded later `push remote ` mode that checks the feature-branch/empty-main state before pushing `main` ahead of the first feature branch so content still reaches main only through peer-reviewed pull requests. | | [dotnet-strong-name-signing](skills/dotnet-strong-name-signing/SKILL.md) | Generate a strong name key (`.snk`) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or `sn.exe` required. Works in any terminal. Defaults to 1024-bit RSA (matching `sn.exe`), with 2048 and 4096 available as options. | | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | @@ -516,12 +516,12 @@ Generating a `.snk` file traditionally requires `sn.exe`, which is only availabl Most repositories start with `git init` followed by committing everything directly to `main`. This works — until someone force-pushes to main, or a half-finished feature lands without review. By the time you add branch protection, the history is already messy. -**trunk-first-repo** flips this: main starts empty and stays clean from the very first commit. Every piece of content enters through a pull request, and `push remote ` can be invoked later when the remote is ready so the first remote push sends `main` before any feature branch. This gives you: +**trunk-first-repo** flips this: main starts empty and stays clean from the very first commit. Every piece of content enters through a pull request, and the skill now branches cleanly depending on when `origin` becomes available: configure it during setup and later you only push `HEAD`; add it later with `push remote ` and the skill first verifies the feature-branch/empty-main state before publishing. This gives you: - **Review from day one** — no "we'll add branch protection later" that never happens - **Clean, meaningful history** — main tells the story of reviewed, approved changes - **Version-aware branches** — `v0.0.1/spike-auth` vs `v1.0.0/release-prep` signals project maturity at a glance -- **Safer first push** — invoke `push remote ` to push `main` by ref from the feature branch, then push the feature branch without manually deleting files or checking out `main` +- **Safer first push** — if `origin` is ready during setup, the summary points straight to `git push -u origin HEAD`; if not, invoke `push remote ` to verify the branch state, push `main` by ref from the feature branch, then push the feature branch without manually deleting files or checking out `main` - **Zero-friction setup** — one skill invocation, not a 10-step checklist ### Why git-remote-release? From 6f167db8de3ae7d90873d36728fdc1d71ef0ebf7 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 11:34:01 +0200 Subject: [PATCH 05/38] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20refactor=20trunk-fir?= =?UTF-8?q?st-repo=20workflow=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructure and expand skill instructions with clearer step-by-step push workflows, conditional summaries for remote configuration states, and more explicit validation for late-remote scenarios. Separate push-remote branching logic from inline feature-push instructions to reduce duplication and improve maintainability. --- skills/trunk-first-repo/SKILL.md | 266 +++++++++++++++++-------------- 1 file changed, 142 insertions(+), 124 deletions(-) diff --git a/skills/trunk-first-repo/SKILL.md b/skills/trunk-first-repo/SKILL.md index 16b5dc5..a1168d6 100644 --- a/skills/trunk-first-repo/SKILL.md +++ b/skills/trunk-first-repo/SKILL.md @@ -1,7 +1,7 @@ --- name: trunk-first-repo -description: > - Initialize a folder as a git repository following scaled trunk-based development. Sets up an empty main branch (seed commit only), creates a versioned feature branch, pushes main before feature branches, and enforces a PR-first workflow where content only reaches main through pull requests. Use this skill when the user wants to initialize a git repo, set up a new repository, start a project with proper git workflow, safely push the first trunk-first branches later with "push remote", or mentions "trunk-based", "PR workflow", "branch protection", "git init", or wants to follow GitHub PR best practices. ALWAYS use this skill when asked to initialize or set up a git repository. +description: > + Initialize a folder as a git repository following scaled trunk-based development. Sets up an empty main branch (seed commit only), creates a versioned feature branch, pushes main before feature branches, and enforces a PR-first workflow where content only reaches main through pull requests. Use this skill when the user wants to initialize a git repo, set up a new repository, start a project with proper git workflow, safely push the first trunk-first branches later with "push remote", or mentions "trunk-based", "PR workflow", "branch protection", "git init", or wants to follow GitHub PR best practices. ALWAYS use this skill when asked to initialize or set up a git repository. --- # Trunk-First Repo @@ -12,17 +12,17 @@ Initialize a folder as a git repository following [scaled trunk-based developmen This matters because it prevents accidental pushes to main, establishes a clean PR-based workflow from day one, and makes the git history meaningful by design rather than as an afterthought. -## Workflow - -### Step 0: Select Mode - -If the user says `push remote`, do not initialize the repository again. Use the Push Remote Workflow below. - -Otherwise, use the Initialize Workflow. - -## Initialize Workflow - -### Step 1: Collect Parameters +## Workflow + +### Step 0: Select Mode + +If the user says `push remote`, do not initialize the repository again. Use the Push Remote Workflow below. + +Otherwise, use the Initialize Workflow. + +## Initialize Workflow + +### Step 1: Collect Parameters Read `FORMS.md` and collect all parameters by presenting each field to the user one at a time using the agent's native input mechanism. Follow the presentation rules defined in the form. Do not proceed to Step 2 until all required fields are collected and the user confirms the summary. @@ -54,48 +54,66 @@ After step 5, the user is on the feature branch (e.g. `v0.1.0/init`) with all th If the user provided a remote URL: ```bash -git remote add origin {REMOTE_URL} -git push -u origin main:main -``` - -Run the push while still on the feature branch. Do not switch to `main` just to push it; Git can push the branch ref by name without changing the working tree. - -If skipped, remind the user they can add it later: - -> When you're ready, invoke this skill with `push remote ` from the feature branch, or run `git remote add origin ` followed by `git push -u origin main:main` - -### Step 3a: First Feature Push - -After the user commits the first project files on the feature branch, push in this order: - -```bash -git push -u origin main:main -git push -u origin HEAD -``` - -This order matters. `main` must exist on the remote before the feature branch is pushed so hosts such as GitHub do not make the feature branch the default branch for a brand-new remote. - -If the remote was added only after the first feature commit, still run the same order from the feature branch: - -```bash -git remote add origin {REMOTE_URL} -git push -u origin main:main -git push -u origin HEAD -``` - -Do not manually delete project files from the working tree to "clean" `main`. When the user is worried about files appearing on `main`, explicitly explain that untracked or ignored checkout files can remain visible in the directory but are not part of the `main` branch. If the user needs to verify that `main` is empty, inspect the branch tree instead of the checkout directory: - -```bash -git ls-tree -r --name-only main -``` - -An empty output means `main` contains only the seed commit. Untracked or ignored files in the checkout directory are not part of `main`. - -If the feature branch was accidentally pushed before `main`, push `main` next and change the remote repository's default branch to `main` before opening the PR. - -### Step 4: Summary - -After initialization, display a summary: +git remote add origin {REMOTE_URL} +git push -u origin main:main +``` + +Run the push while still on the feature branch. Do not switch to `main` just to push it; Git can push the branch ref by name without changing the working tree. + +If skipped, remind the user they can add it later: + +> When you're ready, invoke this skill with `push remote ` from the feature branch, or run `git remote add origin ` followed by `git push -u origin main:main` + +### Step 3a: First Feature Push + +After the user commits the first project files on the feature branch: + +If Step 3 already configured and pushed `origin/main`, publish only the current feature branch: + +```bash +git push -u origin HEAD +``` + +If the remote was not configured during initialization, do not inline a shortened variant here. Use the Push Remote Workflow below instead so it: + +- checks `git branch --show-current` before pushing +- verifies `main` is still only the empty seed branch with `git ls-tree -r --name-only main` +- adds `origin` only when needed +- pushes `main:main` before `HEAD` + +This avoids a redundant `git push -u origin main:main` when Step 3 already handled the empty trunk push, while keeping the guarded `main`-before-`HEAD` order for late-remote cases. + +Do not manually delete project files from the working tree to "clean" `main`. When the user is worried about files appearing on `main`, explicitly explain that untracked or ignored checkout files can remain visible in the directory but are not part of the `main` branch. If the user needs to verify that `main` is empty, inspect the branch tree instead of the checkout directory: + +```bash +git ls-tree -r --name-only main +``` + +An empty output means `main` contains only the seed commit. Untracked or ignored files in the checkout directory are not part of `main`. + +If the feature branch was accidentally pushed before `main`, push `main` next and change the remote repository's default branch to `main` before opening the PR. + +### Step 4: Summary + +After initialization, display the matching summary: + +If Step 3 configured `origin`: + +``` +✅ Repository initialized with trunk-first workflow + + main branch: 🌱 seeded (empty — content enters only via PRs) + feature branch: v0.1.0/init (current — start working here) + remote: origin -> https://github.com/example/repo.git + + Next steps: + 1. Stage and commit your files on this branch + 2. When the branch is ready, push it with `git push -u origin HEAD` + 3. Open a PR to main + 4. After review, merge the PR — main stays clean +``` + +If Step 3 was skipped: ``` ✅ Repository initialized with trunk-first workflow @@ -104,68 +122,68 @@ After initialization, display a summary: feature branch: v0.1.0/init (current — start working here) remote: not configured (add later with `git remote add origin `) - Next steps: - 1. Stage and commit your files on this branch - 2. When the remote is ready, invoke `push remote ` from this branch - 3. Push main first with `git push -u origin main:main` - 4. Push the feature branch with `git push -u origin HEAD` and open a PR to main - 5. After review, merge the PR — main stays clean -``` - -## Push Remote Workflow - -Use this workflow when the user invokes `push remote`, especially when the remote URL was not available during initialization. - -### Step P1: Confirm Current State - -Inspect the current branch: - -```bash -git branch --show-current -``` - -If the current branch is `main`, stop and ask the user to switch to the feature branch first. Do not push the first feature branch while checked out on `main`. - -Verify `main` is still the empty seed branch: - -```bash -git ls-tree -r --name-only main -``` - -Empty output is expected. If output is not empty, stop and explain that `main` already contains files, so this is no longer the empty trunk-first seed state. - -When explaining this check, explicitly say that untracked or ignored files can remain visible in the checkout directory but are not part of `main` unless they appear in `git ls-tree -r --name-only main`. - -### Step P2: Resolve Remote - -Check whether `origin` already exists: - -```bash -git remote get-url origin -``` - -If `origin` exists, use it. - -If `origin` does not exist and the user provided a URL after `push remote`, add it: - -```bash -git remote add origin {REMOTE_URL} -``` - -If `origin` does not exist and the user did not provide a URL, ask for the remote URL before proceeding. Do not guess or use a placeholder. - -### Step P3: Push in Safe Order - -Run these commands from the feature branch: - -```bash -git push -u origin main:main -git push -u origin HEAD -``` - -Do not switch to `main` just to push it. Git can push the `main` branch ref by name while the checkout stays on the feature branch. - -After pushing, tell the user to open a PR from the pushed feature branch to `main`. + Next steps: + 1. Stage and commit your files on this branch + 2. When the remote is ready, invoke `push remote ` from this branch + 3. Push main first with `git push -u origin main:main` + 4. Push the feature branch with `git push -u origin HEAD` and open a PR to main + 5. After review, merge the PR — main stays clean +``` + +## Push Remote Workflow + +Use this workflow when the user invokes `push remote`, especially when the remote URL was not available during initialization. + +### Step P1: Confirm Current State + +Inspect the current branch: + +```bash +git branch --show-current +``` + +If the current branch is `main`, stop and ask the user to switch to the feature branch first. Do not push the first feature branch while checked out on `main`. + +Verify `main` is still the empty seed branch: + +```bash +git ls-tree -r --name-only main +``` + +Empty output is expected. If output is not empty, stop and explain that `main` already contains files, so this is no longer the empty trunk-first seed state. + +When explaining this check, explicitly say that untracked or ignored files can remain visible in the checkout directory but are not part of `main` unless they appear in `git ls-tree -r --name-only main`. + +### Step P2: Resolve Remote + +Check whether `origin` already exists: + +```bash +git remote get-url origin +``` + +If `origin` exists, use it. + +If `origin` does not exist and the user provided a URL after `push remote`, add it: + +```bash +git remote add origin {REMOTE_URL} +``` + +If `origin` does not exist and the user did not provide a URL, ask for the remote URL before proceeding. Do not guess or use a placeholder. + +### Step P3: Push in Safe Order + +Run these commands from the feature branch: + +```bash +git push -u origin main:main +git push -u origin HEAD +``` + +Do not switch to `main` just to push it. Git can push the `main` branch ref by name while the checkout stays on the feature branch. + +After pushing, tell the user to open a PR from the pushed feature branch to `main`. ## Conventions @@ -195,12 +213,12 @@ The version prefix groups branches by project maturity. The context should be sh Once initialized, the day-to-day workflow is: -1. **Create a feature branch** from main: `git checkout -b v0.1.0/my-feature main` -2. **Work and commit** on the feature branch -3. **Push main by ref if the remote does not have it yet**: `git push -u origin main:main` -4. **Push the feature branch and open a PR** to main: `git push -u origin HEAD` -5. **Review, approve, and merge** the PR -6. **Delete the feature branch** after merge -7. **Pull main** and create the next feature branch +1. **Create a feature branch** from main: `git checkout -b v0.1.0/my-feature main` +2. **Work and commit** on the feature branch +3. **Push main by ref if the remote does not have it yet**: `git push -u origin main:main` +4. **Push the feature branch and open a PR** to main: `git push -u origin HEAD` +5. **Review, approve, and merge** the PR +6. **Delete the feature branch** after merge +7. **Pull main** and create the next feature branch Feature branches should be short-lived — ideally merged within hours or a few days, not weeks. From eecada6b29277a06f3bcc88292eaf6c70e910402 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 11:34:11 +0200 Subject: [PATCH 06/38] =?UTF-8?q?=E2=9C=85=20update=20trunk-first-repo=20e?= =?UTF-8?q?val=20expectations?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Align eval test cases with refined skill behavior: configured remotes now appear in post-init summary, late-remote scenarios route through guarded push-remote flow with validation checks, and feature-branch publish simplifies to git push -u origin HEAD when main was already pushed. --- skills/trunk-first-repo/evals/evals.json | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/skills/trunk-first-repo/evals/evals.json b/skills/trunk-first-repo/evals/evals.json index 43c2bf4..8532bda 100644 --- a/skills/trunk-first-repo/evals/evals.json +++ b/skills/trunk-first-repo/evals/evals.json @@ -13,12 +13,13 @@ }, { "id": 2, - "prompt": "Set up a new repository with trunk-based development and configure origin https://github.com/example/demo.git.", - "expected_output": "The workflow initializes trunk-first git history and adds the provided remote origin.", + "prompt": "Set up a new repository with trunk-based development, configure origin https://github.com/example/demo.git, and show me the post-init summary.", + "expected_output": "The workflow initializes trunk-first git history, adds the provided remote origin, pushes empty main by ref from the feature branch, and summarizes the configured remote plus the remaining first-feature push step.", "expectations": [ "Adds the provided remote URL after initialization", - "Pushes or documents pushing main with an explicit main:main ref while staying on the feature branch", - "Summarizes next steps after setup" + "Pushes main with an explicit main:main ref while staying on the feature branch", + "Shows the configured remote URL in the summary instead of saying the remote is not configured", + "Tells the user the remaining publish step is git push -u origin HEAD when the branch is ready" ] }, { @@ -34,10 +35,11 @@ { "id": 4, "prompt": "I initialized a trunk-first repo, committed my first files on v0.1.0/init, and only now added a GitHub remote. Help me push without making the feature branch the default branch or manually deleting files from main.", - "expected_output": "The workflow pushes main to the remote first by ref, then pushes the current feature branch, without checking out main or deleting working-tree files.", + "expected_output": "The workflow routes this late-remote case through the guarded push-remote flow so it validates the current feature branch and empty main seed state before adding origin and pushing main then the feature branch.", "expectations": [ - "Runs or recommends git push -u origin main:main before pushing the feature branch", - "Runs or recommends git push -u origin HEAD for the current feature branch", + "Checks the current branch before pushing and refuses to continue from main", + "Checks main with git ls-tree -r --name-only main before pushing", + "Adds origin only if it is missing and then runs or recommends git push -u origin main:main before git push -u origin HEAD", "Avoids switching to main just to push it", "Explains that untracked or ignored files in the checkout are not part of main and suggests git ls-tree -r --name-only main to verify branch contents" ] From c170e17a248f7eafb9430ce528c9dc4aa832c4e8 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 13:55:59 +0200 Subject: [PATCH 07/38] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20improve=20dotnet-ben?= =?UTF-8?q?chmark=20templates=20and=20runner-default-only=20support?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Simplify benchmark-program.cs template substitution using {RUNTIME_USINGS} and {RUNTIME_JOBS} placeholders to cleanly handle the runner-default-only case without stray syntax. Refactor params-benchmark.cs to remove the Variant enum pattern and clarify guidance about comparing implementations. Update evals.json with clearer expectations and add eval case 7 for runner-default-only scenario. Improve references with fixed code examples (slimJob -> BenchmarkWorkspaceOptions.Slim), detailed placeholder documentation, and guidance about keeping the runner configuration clean. --- .../assets/benchmark-program.cs | 12 +++++------ .../assets/params-benchmark.cs | 20 ++++++------------- skills/dotnet-benchmark/evals/evals.json | 19 +++++++++++++++--- .../references/benchmarkdotnet-essentials.md | 11 ++++++---- .../references/codebelt-conventions.md | 8 ++++++-- .../dotnet-benchmark/references/onboarding.md | 3 ++- 6 files changed, 42 insertions(+), 31 deletions(-) diff --git a/skills/dotnet-benchmark/assets/benchmark-program.cs b/skills/dotnet-benchmark/assets/benchmark-program.cs index fd93e1d..e94ff1f 100644 --- a/skills/dotnet-benchmark/assets/benchmark-program.cs +++ b/skills/dotnet-benchmark/assets/benchmark-program.cs @@ -1,7 +1,5 @@ -using BenchmarkDotNet.Configs; -using BenchmarkDotNet.Environments; -using BenchmarkDotNet.Jobs; -using Codebelt.Extensions.BenchmarkDotNet; +// Emit these runtime-job using directives only when extra AddJob(...) runtimes were selected. +{RUNTIME_USINGS} using Codebelt.Extensions.BenchmarkDotNet.Console; namespace {RUNNER_NAMESPACE}; @@ -16,9 +14,9 @@ public static void Main(string[] args) o.SkipBenchmarksWithReports = true; o.ConfigureBenchmarkDotNet(c => { - var slimJob = BenchmarkWorkspaceOptions.Slim; - return c -{RUNTIME_JOBS}; + // If the user chose "Runner default only", leave the next line as `return c;`. + // Otherwise append newline-prefixed chained `.AddJob(...)` calls to the return expression. + return c{RUNTIME_JOBS}; }); }); } diff --git a/skills/dotnet-benchmark/assets/params-benchmark.cs b/skills/dotnet-benchmark/assets/params-benchmark.cs index eb3ddb5..975277a 100644 --- a/skills/dotnet-benchmark/assets/params-benchmark.cs +++ b/skills/dotnet-benchmark/assets/params-benchmark.cs @@ -1,27 +1,19 @@ using System; -using System.Collections.Generic; using BenchmarkDotNet.Attributes; using BenchmarkDotNet.Configs; namespace {SUT_NAMESPACE} { - // Complex-tier template: sweep input sizes and/or implementation variants with [Params], build - // deterministic payloads once in [GlobalSetup], and compare candidates against a baseline. Use - // this shape for size- or variant-sensitive types (hashing, parsing, buffers, algorithms). - // Replace Variant, the payload sizes, and the measured calls with the real API of {SUT_TYPE}. + // Complex-tier template: sweep representative input sizes, build deterministic payloads once in + // [GlobalSetup], and compare a candidate implementation against a baseline. Use this shape for + // size- or variant-sensitive types (hashing, parsing, buffers, algorithms). If you need an + // implementation enum as a [Params] dimension, collapse the measured work into one dispatching + // [Benchmark] method instead of keeping separate benchmark methods and an unused parameter. + // Replace the payload sizes and the measured calls with the real API of {SUT_TYPE}. [MemoryDiagnoser] [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] public class {SUT_TYPE}Benchmark { - public enum Variant - { - Baseline, - Candidate - } - - [Params(Variant.Baseline, Variant.Candidate)] - public Variant Implementation { get; set; } - // Sweep representative micro / mid / macro sizes so trends are visible. [Params(64, 4096, 1_048_576)] public int Size { get; set; } diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json index 04cc477..aa6f510 100644 --- a/skills/dotnet-benchmark/evals/evals.json +++ b/skills/dotnet-benchmark/evals/evals.json @@ -27,20 +27,23 @@ { "id": 3, "prompt": "Benchmark our Acme.Hashing.Crc32 implementation against the built-in one across small and large inputs — I want to see allocations too.", - "expected_output": "A complex-tier benchmark using [Params] to sweep input sizes (and/or an implementation variant), deterministic payloads built in GlobalSetup with a seeded Random, MemoryDiagnoser, and a Baseline=true method comparing implementations.", + "expected_output": "A complex-tier benchmark using [Params] to sweep input sizes, deterministic payloads built in GlobalSetup with a seeded Random, MemoryDiagnoser, and separate baseline/candidate benchmark methods that compare implementations without unused template scaffolding.", "expectations": [ "Chooses the complex/params tier for a size- and variant-sensitive type", "Uses [Params] to sweep multiple input sizes and prepares deterministic payloads in [GlobalSetup]", + "Compares implementations through separate benchmark methods with exactly one Baseline = true anchor instead of leaving behind an unused implementation [Params] property", "Includes [MemoryDiagnoser] and a Baseline = true comparison anchor", + "Omits unused using directives from the generated benchmark file", "Uses deterministic in-memory data with no network, disk, or database in measured methods" ] }, { "id": 4, "prompt": "I want the benchmark for Acme.Core's Parser to also compare .NET Framework 4.8 against .NET 9 and .NET 10.", - "expected_output": "Runner Program.cs gains one AddJob(slimJob.WithRuntime(...)) per requested runtime (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0), and the benchmark project targets the matching TFMs so those jobs can run.", + "expected_output": "Runner Program.cs gains one AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...)) call per requested runtime (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0), includes the runtime-job using directives needed for those jobs, and the benchmark project targets the matching TFMs so those jobs can run.", "expectations": [ - "Adds one .AddJob(slimJob.WithRuntime(...)) line per requested runtime using the correct monikers (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0)", + "Adds one .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...)) line per requested runtime using the correct monikers (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0)", + "Keeps the runtime-job using directives only when those extra jobs are present", "Ensures the benchmark project TargetFrameworks include the matching TFMs (e.g. net48) so the jobs are runnable", "Explains that the Codebelt runner host runs on .NET 9/10 while BenchmarkDotNet jobs can measure other runtimes" ] @@ -65,6 +68,16 @@ "Adds self-contained TargetFrameworks and IsPackable=false to the benchmark/runner projects since the root Directory.Build.props does not centralize benchmark conventions", "Wires projects into the solution using dotnet sln MyApp.sln add" ] + }, + { + "id": 7, + "prompt": "Set up a benchmark for Acme.Core.Parser, but keep the runner on its default runtime only. I do not want extra net48/net9/net10 comparison jobs.", + "expected_output": "Runner Program.cs stays on the default runner configuration with a plain `return c;`, no raw template placeholders or stray runtime-job syntax, and no extra runtime-job using directives.", + "expectations": [ + "Leaves the runner-default-only case as valid C# by replacing the runtime-job placeholder with nothing so ConfigureBenchmarkDotNet returns `c;`", + "Does not emit raw template placeholders or malformed chained AddJob syntax when no extra runtimes were requested", + "Omits BenchmarkDotNet.Environments, BenchmarkDotNet.Jobs, and Codebelt.Extensions.BenchmarkDotNet using directives when no extra runtimes were selected" + ] } ] } diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md index 2374b2e..6b2ec65 100644 --- a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -38,9 +38,9 @@ and you add jobs fluently in the runner's `Program.cs`: ```csharp return c - .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) - .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) - .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); + .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(ClrRuntime.Net48)) + .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core90)) + .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core10_0)); ``` Although `Codebelt.Extensions.BenchmarkDotNet` itself targets .NET 9/10, the **jobs** can measure @@ -55,7 +55,10 @@ older and newer runtimes. Runtime moniker map: | Mono | `MonoRuntime.Default` | Only add runtimes the benchmark project actually targets (its `TargetFrameworks` must include the -matching TFM, e.g. `net48` for `ClrRuntime.Net48`). Docs: https://benchmarkdotnet.org/articles/configs/jobs.html +matching TFM, e.g. `net48` for `ClrRuntime.Net48`). In the starter runner template, keep the +runtime-job `using` directives plus the `.AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...))` +chain only when the user explicitly asked for extra runtimes; the **Runner default only** case should +stay warning-free as plain `return c;`. Docs: https://benchmarkdotnet.org/articles/configs/jobs.html Other useful job knobs (usually leave BenchmarkDotNet's smart defaults alone): `RunStrategy` (`Throughput`/`ColdStart`/`Monitoring`), `WarmupCount`, `IterationCount`, `LaunchCount`, `Platform`, diff --git a/skills/dotnet-benchmark/references/codebelt-conventions.md b/skills/dotnet-benchmark/references/codebelt-conventions.md index a863d6a..1badc73 100644 --- a/skills/dotnet-benchmark/references/codebelt-conventions.md +++ b/skills/dotnet-benchmark/references/codebelt-conventions.md @@ -110,8 +110,12 @@ public class TestBenchmark } ``` -Template: `assets/params-benchmark.cs`. Prefer seeded RNG (`new Random(42)`) and fixed sizes so runs -are deterministic. Choose micro / mid / macro sizes to reveal trends. +Template: `assets/params-benchmark.cs`. The starter keeps input size as the `[Params]` axis and +compares baseline/candidate implementations through separate benchmark methods so `Baseline = true` +stays meaningful. If you instead introduce an implementation enum as a `[Params]` dimension, collapse +the measured work into one dispatching `[Benchmark]` method; do not keep duplicate benchmark methods +and an unused parameter. Prefer seeded RNG (`new Random(42)`) and fixed sizes so runs are +deterministic. Choose micro / mid / macro sizes to reveal trends. ## How to pick a tier diff --git a/skills/dotnet-benchmark/references/onboarding.md b/skills/dotnet-benchmark/references/onboarding.md index f0c79cf..bf5e3a8 100644 --- a/skills/dotnet-benchmark/references/onboarding.md +++ b/skills/dotnet-benchmark/references/onboarding.md @@ -75,7 +75,8 @@ scaffold and `codebeltnet/xunit`). Copy `assets/benchmark-runner.csproj` and |-------------|-------| | `{RUNNER_TARGET_FRAMEWORK}` | Highest supported non-preview executable TFM (`net10.0` or `net9.0`). | | `{RUNNER_NAMESPACE}` | Runner folder name converted to a valid C# identifier (`benchmark-runner` -> `benchmark_runner`). | -| `{RUNTIME_JOBS}` | One indented `.AddJob(slimJob.WithRuntime(...))` line per runtime to measure (see `benchmarkdotnet-essentials.md`). At minimum, add the runner's own TFM. | +| `{RUNTIME_USINGS}` | Empty for **Runner default only**. Otherwise emit the `using BenchmarkDotNet.Environments;`, `using BenchmarkDotNet.Jobs;`, and `using Codebelt.Extensions.BenchmarkDotNet;` lines, each terminated with a newline, because the added runtime jobs need all three namespaces. | +| `{RUNTIME_JOBS}` | Empty for **Runner default only** so the method stays `return c;`. Otherwise emit newline-prefixed chained `.AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...))` calls, one per runtime to measure (see `benchmarkdotnet-essentials.md`). | If the root `Directory.Build.props` does **not** mark tooling projects as executables, add `Exe` and `false` to the runner `PropertyGroup`. From 7912ca4674657988125cbd71780370c8e7026020 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 14:51:27 +0200 Subject: [PATCH 08/38] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20restructure=20dotnet?= =?UTF-8?q?-benchmark=20for=20discovery-focused=20workflow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Redesign skill to prioritize candidate selection and measurement fitness over tier-based templates. Rewrite SKILL.md workflow to introduce discovery-phase focus: evidence ladder, inspection sequence, cost signals, and candidate matrix guidance. Restructure FORMS.md around the new workflow parameters. Expand references with candidate-selection.md and experiment-design.md; refactor benchmarkdotnet-essentials.md and codebelt-conventions.md to reflect discovery-first approach. --- skills/dotnet-benchmark/FORMS.md | 137 ++++---- skills/dotnet-benchmark/SKILL.md | 303 ++++++++---------- .../references/benchmarkdotnet-essentials.md | 190 ++++++----- .../references/candidate-selection.md | 110 +++++++ .../references/codebelt-conventions.md | 192 ++++------- .../references/experiment-design.md | 144 +++++++++ 6 files changed, 620 insertions(+), 456 deletions(-) create mode 100644 skills/dotnet-benchmark/references/candidate-selection.md create mode 100644 skills/dotnet-benchmark/references/experiment-design.md diff --git a/skills/dotnet-benchmark/FORMS.md b/skills/dotnet-benchmark/FORMS.md index 8b1109e..5dcfce7 100644 --- a/skills/dotnet-benchmark/FORMS.md +++ b/skills/dotnet-benchmark/FORMS.md @@ -1,71 +1,66 @@ -# Parameter Form - -`dotnet-benchmark` collects a small number of inputs. Present each field **one at a time** using the -host's native input mechanism (e.g. `ask_user` with `choices`) when available. If native structured -input is unavailable, use the deterministic plain-text fallback in the Presentation Rules below. Do -not bundle multiple fields into a single message. - -Most fields have smart defaults derived from inspecting the repo and the target type, so a normal run -asks very little. Skip any field whose value is already unambiguous from the conversation (e.g. the -user already named the type). - -## Fields - -### sut_type -- **type:** text -- **prompt:** "Which type do you want to performance-test? (namespace-qualified if ambiguous)" -- **placeholder:** "e.g. Cuemon.DateSpan or Acme.Buffers.RingBuffer" -- **required:** true -- **description:** The System Under Test. Resolve it in the source tree to learn its namespace, owning - `src/` project, and public surface. If the user already named a type, accept it and continue. - -### benchmark_tier -- **type:** single-choice -- **prompt:** "How thorough should the benchmark be?" -- **choices:** - - Auto — inspect the type and pick the best shape (Recommended) - - Simple — member scenarios (construct / format / equals / hash) - - Complex — sweep input sizes/variants with [Params] and GlobalSetup -- **default:** Auto — inspect the type and pick the best shape (Recommended) -- **description:** Auto uses the complexity heuristics in `references/codebelt-conventions.md`. State - which tier you chose and why, then let the user override. - -### target_runtimes -- **type:** multi-choice -- **prompt:** "Which runtimes should the benchmark jobs measure?" -- **choices:** - - Runner default only (Recommended) - - .NET 10 (CoreRuntime.Core10_0) - - .NET 9 (CoreRuntime.Core90) - - .NET 8 (CoreRuntime.Core80) - - .NET Framework 4.8 (ClrRuntime.Net48, Windows only) -- **default:** Runner default only (Recommended) -- **description:** The runner host targets .NET 9/10, but BenchmarkDotNet **jobs** can measure other - runtimes. Only offer runtimes the benchmark project can target (its `TargetFrameworks` must include - the matching TFM). Adding extra runtimes multiplies run time. - -### run_now -- **type:** single-choice -- **prompt:** "Run the benchmark now, or just wire it up and give you the command?" -- **choices:** - - Just wire it up and give me the command (Recommended) - - Run it now -- **default:** Just wire it up and give me the command (Recommended) -- **description:** BenchmarkDotNet runs are slow and heavy. Default is to verify the Release build and - hand off the run command. Only run it when the user explicitly asks. - -## Presentation Rules - -1. Ask one field at a time — wait for the answer before presenting the next field. -2. Prefer the host's native structured input controls for every field when available. -3. If native controls are unavailable, use this plain-text fallback: - - Start with `Field: ` - - Repeat the field prompt verbatim - - For choice fields, show a numbered option list; accept the number or exact option text - - For `text` fields with a default, show `1. Use "" (Recommended)` and `2. Enter a custom value` - - After the user answers, restate the normalized value in one short line before moving on -4. When a field has a `default`, present it first and append "(Recommended)" if not already labeled. -5. Treat a blank response on a field that has a default as accepting that default — do not re-ask. -6. Skip a field entirely when its value is already clear from context (e.g. the user said "benchmark - DateSpan" — `sut_type` is answered). -7. After collecting fields, briefly confirm the plan (type, tier, runtimes, run-or-not) before writing. +# Parameter Form + +`dotnet-benchmark` derives most decisions from the repository, the target type, and the user's request. Ask only unresolved fields, one at a time. Prefer the host's native structured input controls when available. If the host does not provide them, use the deterministic plain-text fallback below without changing field order or choices. + +## Fields + +### sut_type + +- **type:** text +- **prompt:** "Which type do you want to investigate? Use the namespace-qualified name if it may be ambiguous." +- **placeholder:** "e.g. Cuemon.DateSpan or Acme.Buffers.RingBuffer" +- **required:** true +- **description:** Skip this field when the user already named an unambiguous type. Resolve the declaration in source rather than relying only on the name. + +### performance_intent + +- **type:** single-choice +- **prompt:** "What should the benchmark investigation optimize for?" +- **choices:** + - Auto-discover the highest-value performance questions (Recommended) + - Compare current and candidate implementations + - Characterize one specific member or operation + - Establish a regression benchmark for a known workload +- **default:** Auto-discover the highest-value performance questions (Recommended) +- **required:** true +- **description:** Infer and skip this field when the request already states the operation, comparison, or regression goal. Auto-discovery ranks candidates from implementation, usage, tests, and any available profiling evidence; it does not claim a measured application bottleneck without profile or telemetry data. + +### workload_context + +- **type:** text +- **prompt:** "I could not infer a representative workload confidently. What inputs, sizes, frequency, and operating conditions matter in production?" +- **required:** false +- **description:** Show this field only when tests, call sites, documentation, or supplied profiling evidence do not establish a representative workload and choosing one would materially affect correctness. Offer the strongest repo-derived workload as a selectable recommended choice when one exists, plus the option to enter a custom value. + +### candidate_plan_confirmation + +- **type:** single-choice +- **prompt:** "Use the proposed benchmark questions, workloads, and validation strategy?" +- **choices:** + - Use the proposed plan (Recommended) + - Adjust the selected operations or inputs +- **default:** Use the proposed plan (Recommended) +- **required:** true +- **description:** Present the evidence-backed experiment plan immediately before this field. Include selected and rejected candidates, comparable baseline/candidate pairs, parameter cases, correctness oracle, lifecycle risks, and whether the plan is exploratory or profile-backed. + +### execution_depth + +- **type:** single-choice +- **prompt:** "How far should validation run?" +- **choices:** + - Build, list, and dry-execute the benchmark (Recommended) + - Run the full performance benchmark after validation +- **default:** Build, list, and dry-execute the benchmark (Recommended) +- **required:** true +- **description:** A dry execution validates discovery and lifecycle but produces no trustworthy performance conclusion. A full run can be slow and machine-sensitive. Treat an explicit request such as "run it" or "measure it now" as selecting the full-run option. + +## Presentation rules + +1. Ask one field at a time and wait for the answer before presenting the next unresolved field. +2. Skip fields already answered by the conversation or reliable repository evidence. Do not ask the user to choose BenchmarkDotNet attributes, a "simple/complex" tier, or extra runtimes unless those choices are part of the user's goal. +3. When native controls are unavailable, start with `Field: `, repeat the prompt verbatim, show numbered choices in declared order, and accept either a number or exact choice text. +4. For a text field with a repo-derived suggestion, show `1. Use "" (Recommended)` and `2. Enter a custom value`. Do not fabricate a suggestion when the evidence is weak. +5. Present each default first and append `(Recommended)` when it is not already included. A blank response accepts the displayed default. +6. After each answer, restate the normalized value in one short line before continuing. +7. Before `candidate_plan_confirmation`, show the experiment plan produced after source inspection. This is the final design confirmation; do not ask a second generic confirmation afterward. +8. If the user chooses to adjust the plan, collect only the disputed operation or workload, revise the plan, and present `candidate_plan_confirmation` again. diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index b5151bb..364bec4 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -1,171 +1,132 @@ ---- -name: dotnet-benchmark -description: > - Set up and author BenchmarkDotNet performance tests for a specific .NET type following codebelt - engineering conventions, using Codebelt.Extensions.BenchmarkDotNet and its Console runner. Use this - skill whenever the user wants to benchmark, micro-benchmark, performance-test, profile throughput or - allocations, or measure the speed of a .NET type or method, in new or existing projects. It first - checks that the benchmark harness and prerequisites exist and sets up anything missing in place - (tuning/ benchmark project, tooling/ runner host, package references, solution wiring), then inspects - the target type and picks a complexity-appropriate strategy, authoring the benchmark class in the same - namespace as the code it measures. Trigger phrases include "add a benchmark", "benchmark this class", - "set up BenchmarkDotNet", "performance test", "micro-benchmark", or "measure allocations". Also use it - when a repo already has a tuning/ or *.Benchmarks project and wants more benchmarks. ---- - -# .NET Benchmark Setup (Codebelt Conventions) - -Make it easy to performance-test a .NET **type** with [BenchmarkDotNet](https://benchmarkdotnet.org/) -the codebelt way, wiring the benchmark into the same `tuning/` + `tooling/` layout used across -[codebeltnet](https://github.com/codebeltnet). This skill works for a repo that already has a -benchmark harness *and* one that has none: it detects what exists and adds only what is missing. - -The two reference implementations this skill mirrors are `codebeltnet/cuemon` and -`codebeltnet/xunit`. When in doubt about a convention, default to how those repos do it. The -[`Codebelt.Extensions.BenchmarkDotNet`](https://benchmarkdotnet.codebelt.net/api/Codebelt.Extensions.BenchmarkDotNet.html) -namespace and its `.Console` companion supply the runner host (`BenchmarkProgram.Run`), so you never -hand-roll a `BenchmarkSwitcher`. - -## Why this layout - -Benchmarks are split across three sibling folders so they never leak into shippable output: - -- `tuning/{SutProject}.Benchmarks/` holds the benchmark **projects and classes** that reference the - code under test. -- `tooling/{runner}/` holds one executable **runner host** that discovers every `tuning/` project - and runs it through the Codebelt console bootstrapper. -- `reports/` receives the generated benchmark artifacts. - -Keeping the runner in `tooling/` and the benchmarks in `tuning/` means the packable `src/` projects -stay clean, and a single runner can drive many benchmark projects. - -## Workflow - -Do the steps in order. Each step explains *why* so you can adapt when a repo does not match the -happy path — real existing repos rarely do. - -### Step 1: Check requirements - -Run the detection script to learn the repo's current state in one pass instead of guessing: - -```powershell -powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot -``` - -It reports, as JSON: whether the .NET SDK is available (and version), the solution file(s) and their -format (`.slnx` vs `.sln`), whether Central Package Management (`Directory.Packages.props`) is used, -whether the root `Directory.Build.props` already centralizes benchmark/tooling conventions -(`IsBenchmarkProject` / `IsToolingProject`), any existing `tuning/*.Benchmarks` projects, and any -existing `tooling/` runner host (its folder name and whether it references -`Codebelt.Extensions.BenchmarkDotNet.Console`). - -The one hard prerequisite is the **.NET SDK**. If it is missing, stop and ask the user to install it -(the runner host targets .NET 9 or .NET 10, matching `Codebelt.Extensions.BenchmarkDotNet` -availability). Everything else in the harness this skill can create for them. - -### Step 2: Onboard the missing harness (in place) - -Add only what Step 1 found missing, matching the repo's existing layout. Do not restructure a repo or -convert its solution format. Read `references/onboarding.md` for the detailed decision tree; the -essentials: - -- **Packages.** Resolve the latest stable listed versions from NuGet.org (never hardcode) for - `BenchmarkDotNet`, `BenchmarkDotNet.Diagnostics.Windows`, and - `Codebelt.Extensions.BenchmarkDotNet.Console`. If the repo uses Central Package Management, add - `` entries to `Directory.Packages.props` and reference them without versions; - otherwise put versioned ``s directly in the project files. -- **Benchmark project.** Create `tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj` from - `assets/benchmark.csproj`, referencing the SUT `src/` project and overriding `RootNamespace` to the - SUT root namespace (so the benchmark lives in the measured namespace, not a `.Benchmarks` one). -- **Runner host.** If no `tooling/` runner exists, create one from `assets/benchmark-runner.csproj` - and `assets/benchmark-program.cs`. Default its folder name to `benchmark-runner`; if the repo - already has a runner (e.g. cuemon's `bdn-runner`), reuse it — do not add a second one. -- **Central conventions vs plain repo.** If the root `Directory.Build.props` already centralizes - `IsBenchmarkProject`/`IsToolingProject` (as codebelt repos do), keep the benchmark `.csproj` - minimal and let the props inject TFMs and BDN packages. If it does not, the project files must - declare their own `TargetFrameworks` and package references — `references/onboarding.md` shows both. -- **Solution wiring.** Add the new projects to the detected solution: for `.slnx`, add - `` entries under `/tuning/` and `/tooling/` folders; for `.sln`, use - `dotnet sln add `. - -### Step 3: Resolve the target type - -Ask which type to performance-test if the user has not already named one. Then locate it in the -source tree to learn its namespace, owning `src/` project, and public surface (constructors, -methods, properties, and any obvious size- or variant-sensitive inputs). You need the namespace to -place the benchmark correctly and the surface to choose meaningful scenarios. - -### Step 4: Choose a strategy by complexity - -The user asked for a *thorough* performance test, so pick the approach that fits the type instead of -applying one rigid template. Read `references/benchmarkdotnet-essentials.md` for the attribute and -job toolbox and `references/codebelt-conventions.md` for the two default tiers: - -- **Simple type** (value-like, no size-sensitive input): benchmark the meaningful members as - discrete scenarios with a clear `Baseline = true` anchor, grouped - `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]`. Template: `assets/simple-benchmark.cs` - (mirrors cuemon `DateSpanBenchmark`). -- **Complex / size- or variant-sensitive type** (hashing, parsing, buffers, algorithm variants): use - `[Params]` to sweep input sizes and/or variants, prepare deterministic payloads in `[GlobalSetup]`, - and compare implementations against a baseline. Template: `assets/params-benchmark.cs` (mirrors - cuemon `Sha512256Benchmark` and xunit `TestBenchmark`). - -Briefly tell the user which tier you chose and why, then let them adjust (e.g. specific methods, -input sizes, or a competing implementation to compare against). Every benchmark uses -`[MemoryDiagnoser]` so allocations are always captured. - -### Step 5: Author and wire the benchmark class - -Write the benchmark into `tuning/{SutProject}.Benchmarks/` following codebelt naming exactly, because -these rules keep type discovery and reports consistent: - -- Class name ends with `Benchmark` (e.g. `DateSpanBenchmark`). -- Namespace is the **same** as the SUT — never suffix `.Benchmarks`. The `RootNamespace` override in - the project file is what makes this compile cleanly. -- Methods use descriptive scenario names (`Parse_Short`, `ComputeHash_Large`) and a `Description` for - readable reports; mark the reference method `Baseline = true`. -- Use deterministic data and no external systems (no network, disk, or DB) so runs are repeatable. - -If the type belongs to a `src/` project that has no `tuning/{SutProject}.Benchmarks` yet, create that -project (Step 2 rules) before adding the class, then make sure the runner discovers it (the wildcard -`..\..\tuning\**\*.csproj` reference already covers new projects) and the solution lists it. - -### Step 6: Verify the build, then hand off the run - -Confirm the benchmark compiles in Release, since BenchmarkDotNet only runs Release builds: - -```powershell -dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj -``` - -Do **not** run the benchmark by default — real runs are slow and heavy. Offer to run it, and give the -exact command so the user can run it when ready. The runner is a console app that accepts BenchmarkDotNet -filters: - -```powershell -dotnet run -c Release --project tooling/{runner} -- --filter *{TypeName}Benchmark* -``` - -Reports land under `reports/`. Only run it yourself if the user explicitly asks. - -### Multi-runtime jobs (optional) - -`Codebelt.Extensions.BenchmarkDotNet` runs on .NET 9/10, but its BenchmarkDotNet **jobs** can measure -other runtimes. If the user wants to compare across runtimes, add jobs in the runner's `Program.cs` -using `slimJob.WithRuntime(...)` — e.g. `ClrRuntime.Net48` (older .NET Framework), -`CoreRuntime.Core80/90/10_0`. xunit's runner does exactly this. See -`references/benchmarkdotnet-essentials.md` for the moniker map. - -## Conventions checklist - -Before finishing, verify: - -- [ ] `.NET SDK` present; runner host targets net9.0 or net10.0 -- [ ] Benchmark class ends with `Benchmark` and lives in the SUT's namespace (no `.Benchmarks` suffix) -- [ ] Benchmark project sets `` to the SUT root and references the SUT `src/` project -- [ ] `[MemoryDiagnoser]` present; a `Baseline = true` method anchors the comparison -- [ ] Deterministic data only — no network/disk/DB in measured methods -- [ ] Packages resolved from NuGet.org (no hardcoded versions); CPM vs `PackageReference` matches the repo -- [ ] Exactly one `tooling/` runner host; new benchmark project added to the detected `.slnx`/`.sln` -- [ ] Release build succeeds; run command provided (benchmark not run unless requested) -- [ ] Generated files are UTF-8 with no mojibake +--- +name: dotnet-benchmark +description: > + Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects implementation code, call sites, tests, existing benchmarks, and available profiling evidence; ranks high-value operations instead of benchmarking every public member; selects representative workloads; rejects misleading microbenchmarks; creates or reuses the tuning/ and tooling/ harness; validates correctness and benchmark discovery; and keeps full performance runs explicit. +--- + +# Evidence-Driven .NET Benchmarking + +Create the smallest benchmark suite that can answer the most valuable performance questions about the supplied type. Follow the repository's established conventions first, then apply the codebelt `tuning/` benchmark project and `tooling/` runner layout where the repository has no stronger local pattern. + +## Critical benchmark contract + +- Treat a benchmark as an experiment, not as public-member coverage. Do not benchmark every constructor, property, or method merely because it exists. +- A microbenchmark measures a suspected cost under a defined workload; it does not prove that the type is an application bottleneck. Prefer production telemetry or a CPU/allocation/contention profile when the user asks where an application is slow. If only a type is provided, perform source-informed candidate discovery and label the result as an exploratory benchmark plan. +- Rank candidates using evidence from the implementation, call sites, tests, documentation, existing benchmark results, and profiles. Never invent usage frequency, input distributions, or a competing implementation. +- Compare only operations that produce equivalent observable work. Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation. +- Use `Baseline = true` only when at least two benchmark methods form a meaningful comparison group. A single-operation scaling or regression benchmark needs no fabricated baseline. When a class has several comparison groups, assign categories and one baseline inside each category. +- Keep correctness outside the timed path but inside the verification workflow. Equivalent implementations must be checked on every benchmark case before a full run. +- Keep external I/O, network latency, database latency, sleeps, logging, and random data generation out of measured microbenchmark methods. Recommend profiling, a macrobenchmark, or a load test when those effects are the actual question. +- Always distinguish code that was built, smoke-executed, or fully measured. Never report performance numbers from a build, discovery listing, dry run, or unexecuted benchmark. + +## Workflow + +### 1. Resolve intent with minimal questioning + +Read `FORMS.md` and use its one-field-at-a-time interaction only for information that is not already clear. A named type plus a request such as “find the likely bottlenecks” is sufficient to start inspection. Do not make the user choose an implementation tier or BenchmarkDotNet attributes. + +Default to automatic candidate discovery, the runner's existing/default runtime, and build plus discovery and dry execution validation. Ask about extra runtimes only when cross-runtime comparison is part of the request. Require explicit user intent before a full benchmark run because it can be long and machine-sensitive. + +### 2. Inspect the repository and harness + +Run the bundled read-only detector before changing files: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot +``` + +Also inspect applicable `AGENTS.md`, solution/project files, `Directory.Build.props`, `Directory.Packages.props`, existing `tuning/` and `tooling/` projects, and nearby benchmark styles. Reuse an existing runner and benchmark project when they fit. Read `references/onboarding.md` only when the detector finds missing or partial harness infrastructure. + +The .NET SDK is the only hard harness prerequisite. If detector status is `not-found`, report that blocker instead of generating unverified project files. If the probe is `timed-out`, `start-failed`, or `failed`, report the probe failure distinctly and verify the SDK through a safe direct check before concluding that it is absent. + +### 3. Resolve the type and gather performance evidence + +Locate the exact declaration, owning `src/` project, namespace, interfaces/base types, and target frameworks. Then read the complete implementation and inspect its collaborators, tests, call sites, documentation, existing benchmarks, and any profiling artifacts available in the repository or supplied by the user. + +Read `references/candidate-selection.md` whenever the user has not already specified the exact operation and workload. Build an evidence-backed candidate matrix and select at most three performance questions for one benchmark class; prefer one focused question when it is clearly dominant. + +Look beyond public methods. Private loops, repeated conversions, allocation-heavy helpers, hashing/equality used by collections, reflection, regex construction, buffer copies, parsing branches, locks, task scheduling, exception-heavy paths, and deferred enumeration can dominate the cost exposed by one public operation. Benchmark the public or internal entry point that represents the real consumer operation, not an arbitrary private helper, unless isolating that helper is the explicit experiment. + +### 4. Decide whether BenchmarkDotNet is the right instrument + +Use a microbenchmark when the work is deterministic, repeatable, isolatable, and small enough to execute many times. If the suspected cost is end-to-end I/O, request concurrency across a service, startup of a whole application, distributed latency, or an unknown application-wide hotspot, explain why a microbenchmark would mislead and propose the narrowest useful next instrument instead. + +When profiling evidence exists, use it to choose the benchmark target. When it does not, state that selection is based on source and usage evidence rather than measured hotspot data. Do not silently turn a hypothesis into a claim. + +### 5. Present the experiment plan + +Before authoring code, present a compact plan with: + +- the performance question and metric: latency/throughput, allocated bytes, scaling, contention, cold start, or exception frequency; +- the selected operation and the evidence that made it important; +- the baseline and candidate, if a fair comparison exists; +- representative cases, including typical, boundary, scaling, and adverse-but-valid inputs where relevant; +- setup/reset strategy and correctness oracle; +- candidates deliberately rejected and why; +- whether the result will be exploratory or grounded in profile/telemetry evidence. + +Follow the confirmation flow in `FORMS.md`. If the user already named exact members, inputs, and implementations, confirm only material corrections or risks rather than repeating settled choices. + +### 6. Design the experiment + +Read `references/experiment-design.md` and `references/benchmarkdotnet-essentials.md`. Choose one of these shapes: + +1. **Equivalent implementation comparison.** Use separate benchmark methods for current/reference and candidate implementations, feed them identical state, check equivalent results in setup, and mark the current production or established reference method as the baseline. Start from `assets/comparison-benchmark.cs`. +2. **Single-operation characterization.** Use one benchmark method across meaningful cases or sizes when the goal is absolute cost, scaling, allocation characterization, or future regression tracking and no honest competing implementation exists. Do not add a baseline solely to obtain a ratio column. Start from `assets/operation-benchmark.cs`. +3. **Independent operation groups.** Split unrelated operations into separate benchmark classes when practical. If a cohesive class contains multiple alternative pairs, use `[BenchmarkCategory]`, group by category, and assign one baseline per category. Never compare ratios across different semantic work. + +Do not copy an asset blindly. The assets are structural examples with placeholders; adapt namespaces, types, cases, lifecycle, return consumption, correctness checks, and attributes to the real API. + +### 7. Author the benchmark + +Place the class under `tuning/{SutProject}.Benchmarks/`. Name it for the performance question and end the class name with `Benchmark`. Keep it in the SUT namespace rather than adding `.Benchmarks`; the benchmark project `RootNamespace` supports this codebelt convention. + +Every benchmark should use deterministic state and `[MemoryDiagnoser]`. Add other diagnosers only when they answer the question: `[ThreadingDiagnoser]` for lock/thread-pool signals, `[ExceptionDiagnoser]` for intentional exception paths, `[DisassemblyDiagnoser]` for JIT/code-generation investigations, or EventPipe/ETW profiling for a targeted deep dive. Extra diagnosers often cause extra runs and platform constraints, so do not add them decoratively. + +Keep setup outside the measured method unless setup is part of the consumer-visible operation. Return a result or consume it so the JIT cannot eliminate the work. Avoid independent `[Params]` axes that create meaningless Cartesian products; use a scenario object from `[ParamsSource]` or `[ArgumentsSource]` when inputs must vary together. + +For mutating operations, ensure every measurement observes equivalent starting state without allowing reset cost to dominate a tiny operation. For async APIs, await the real `Task`/`ValueTask`; do not substitute `.Result` or benchmark a completed fake. For concurrency, measure a defined worker/contention scenario and avoid accidentally measuring task creation when shared-state throughput is the actual question. + +### 8. Onboard only missing harness pieces + +If the detector found missing infrastructure, follow `references/onboarding.md`. Resolve current stable package versions dynamically from NuGet.org, preserve central package management when present, reuse existing solution and folder conventions, create at most one runner, and avoid restructuring unrelated repository content. + +### 9. Validate in layers + +First validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. Then build the benchmark project in Release: + +```powershell +dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj +``` + +Verify runner discovery without measuring: + +```powershell +dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* +``` + +Unless execution is impossible or the user declines, run a dry execution smoke check and inspect all BenchmarkDotNet validation warnings. A dry run proves executable wiring and basic lifecycle, not performance: + +```powershell +dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* +``` + +Run the full benchmark only when the user explicitly asks. Use an unplugged laptop, debugger, busy CI worker, VM, or power-throttled environment only if that environment is itself the target; otherwise warn that the results may not be stable or representative. + +### 10. Report the outcome + +Summarize the selected and rejected candidates, benchmark question, cases, correctness oracle, diagnosers, generated files, and validation commands/results. If a full run occurred, report environment, mean/median where relevant, error and standard deviation, ratios only within valid comparison groups, allocations/GC, warnings, and the workload-specific conclusion. Recommend an optimization only when measurements identify a meaningful opportunity and correctness remains protected. + +## Completion checklist + +- [ ] Candidate selection is supported by implementation, usage, tests, telemetry, or profiling evidence. +- [ ] The suite answers one to three explicit performance questions and excludes low-value member coverage. +- [ ] Baselines compare equivalent work; single operations and unrelated members have no misleading ratio. +- [ ] Cases represent realistic, boundary, scaling, and adverse paths without useless Cartesian products. +- [ ] Setup, mutation, async, concurrency, disposal, and result consumption are handled correctly. +- [ ] Equivalent implementations pass a correctness oracle for every case. +- [ ] `[MemoryDiagnoser]` is present and every additional diagnoser has a stated purpose. +- [ ] Harness changes preserve repository conventions and reuse existing projects/runner where possible. +- [ ] Release build, benchmark discovery, and dry execution succeed, or exact blockers are reported. +- [ ] Full-run performance claims are made only from an actual full run in a described environment. +- [ ] Generated files are UTF-8 without mojibake, and no unrelated files were changed. diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md index 6b2ec65..d96e1fc 100644 --- a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -1,81 +1,109 @@ -# BenchmarkDotNet Essentials - -A compact toolbox for authoring benchmarks. Full docs: https://benchmarkdotnet.org/articles/overview.html - -## Core attributes - -| Attribute | Purpose | -|-----------|---------| -| `[Benchmark]` | Marks a measured method. Add `Baseline = true` on the reference method and `Description = "..."` for readable reports. | -| `[MemoryDiagnoser]` | Captures allocations and GC counts. Always include it — codebelt benchmarks care about allocations, not just time. | -| `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]` | Groups related methods so comparisons read cleanly. Use `ByParams` when the story is "same operation across sizes/variants". | -| `[Params(...)]` | Sweeps input values (sizes, enum variants). BenchmarkDotNet runs every method once per combination. | -| `[ParamsSource(nameof(...))]` | Use when the parameter set is computed rather than literal. | -| `[GlobalSetup]` | One-time initialization that is **not** measured. Build payloads and instances here. | -| `[IterationSetup]` / `[IterationCleanup]` | Per-iteration hooks; use sparingly (they add overhead) for state that must reset each iteration. | -| `[Arguments(...)]` | Passes literal arguments to a benchmark method — lighter than `[Params]` for a few fixed cases. | - -## Choosing what to measure - -- Keep each `[Benchmark]` method to a **single logical operation**; move setup out of the measured path. -- Return a value from the method (or consume inputs) so the JIT cannot optimize the work away. -- Use deterministic, in-memory data. No network, disk, or database in measured methods — they destroy - repeatability and are not micro-benchmarks. -- Name methods for the scenario (`Parse_Short`, `ComputeHash_Large`, `Match_ComplexWildcard`) so the - report is self-describing. - -## Diagnosers worth knowing - -- `[MemoryDiagnoser]` — allocations (default for codebelt). -- `[DisassemblyDiagnoser]` — emitted asm; heavy, opt-in for deep dives. -- `BenchmarkDotNet.Diagnostics.Windows` (`[EtwProfiler]`, native counters) — Windows-only; referenced - by codebelt benchmark projects but enable specific diagnosers only when needed. - -## Jobs and runtimes - -A **job** describes how to run a benchmark. The Codebelt runner starts from `BenchmarkWorkspaceOptions.Slim` -and you add jobs fluently in the runner's `Program.cs`: - -```csharp -return c - .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(ClrRuntime.Net48)) - .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core90)) - .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core10_0)); -``` - -Although `Codebelt.Extensions.BenchmarkDotNet` itself targets .NET 9/10, the **jobs** can measure -older and newer runtimes. Runtime moniker map: - -| Target | Job runtime | -|--------|-------------| -| .NET Framework 4.8 | `ClrRuntime.Net48` (Windows only) | -| .NET 8 | `CoreRuntime.Core80` | -| .NET 9 | `CoreRuntime.Core90` | -| .NET 10 | `CoreRuntime.Core10_0` | -| Mono | `MonoRuntime.Default` | - -Only add runtimes the benchmark project actually targets (its `TargetFrameworks` must include the -matching TFM, e.g. `net48` for `ClrRuntime.Net48`). In the starter runner template, keep the -runtime-job `using` directives plus the `.AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...))` -chain only when the user explicitly asked for extra runtimes; the **Runner default only** case should -stay warning-free as plain `return c;`. Docs: https://benchmarkdotnet.org/articles/configs/jobs.html - -Other useful job knobs (usually leave BenchmarkDotNet's smart defaults alone): `RunStrategy` -(`Throughput`/`ColdStart`/`Monitoring`), `WarmupCount`, `IterationCount`, `LaunchCount`, `Platform`, -`GcMode.Server`. Set these only for a specific reason. - -## Running - -BenchmarkDotNet requires a **Release** build. Through the Codebelt console runner: - -```powershell -dotnet run -c Release --project tooling/{runner} -- --filter *{TypeName}Benchmark* -``` - -Common runner/BDN switches passed after `--`: - -- `--filter ` — select benchmarks by full name (`*DateSpanBenchmark*`, `*.Parse_*`). -- `--list flat` — list discovered benchmarks without running. -- `--job short` — a faster, less precise job for smoke checks. - -Reports are written under `reports/`. +# BenchmarkDotNet Essentials + +Use this as a compact API and validation reference after the experiment question is defined. Official documentation: . + +## Core attributes + +| Attribute | Purpose | +|---|---| +| `[Benchmark]` | Marks measured work. Add `Baseline = true` only inside an equivalent comparison group and use `Description` for readable reports. | +| `[MemoryDiagnoser]` | Reports managed allocations and GC counts. Always include it for codebelt benchmarks. | +| `[BenchmarkCategory("...")]` | Labels logical comparison groups when one class contains multiple alternative pairs. | +| `[GroupBenchmarksBy(...)]` | Groups report rows by category, params, or a deliberate combination. Grouping changes presentation and baseline scope; choose it from the question. | +| `[Params(...)]` | Sweeps independent compile-time-constant values. Multiple params properties create a Cartesian product. | +| `[ParamsSource(nameof(...))]` | Supplies computed or coupled scenario objects with readable names. | +| `[Arguments(...)]` / `[ArgumentsSource]` | Supplies method arguments, useful for explicit scenario sets. | +| `[GlobalSetup]` / `[GlobalCleanup]` | Prepares and releases per-method/per-parameter state outside measurement. | +| `[IterationSetup]` / `[IterationCleanup]` | Resets per iteration but forces single invocation/unroll; avoid for tiny microbenchmarks. | + +## Baselines + +A method baseline adds a ratio distribution against equivalent methods. BenchmarkDotNet allows category-specific baselines when benchmarks are grouped by category. Do not attach a baseline to an unrelated member or a lone benchmark just to satisfy a template. + +Runtime comparisons can use a job baseline. Keep method, inputs, and implementation fixed when attributing a difference to runtime. + +## Prevent invalid measurements + +- Use Release builds and run without an attached debugger. +- Return or consume results to prevent dead-code elimination. +- Keep setup and correctness checks outside timed methods. +- Do not rely on execution order or shared mutation between methods. +- Avoid manual loops unless batching is the real workload; BenchmarkDotNet selects invocation counts automatically. +- Inspect all validation and environment warnings before reading result tables. +- Keep the machine powered and quiet for full runs unless the noisy/throttled environment is the intended target. + +## Validators + +BenchmarkDotNet always validates duplicate baselines. `ExecutionValidator` can smoke-execute cases and `ReturnValueValidator` can compare compatible return values, but domain-specific correctness checks remain necessary. A Release build plus `--job dry` provides practical wiring/lifecycle validation through the codebelt runner. + +## Diagnosers and profilers + +- `[MemoryDiagnoser]`: allocations and GC, always enabled by this skill. +- `[ThreadingDiagnoser]`: completed thread-pool work items and monitor lock contention on .NET Core 3+. +- `[ExceptionDiagnoser]`: exception frequency for intentional exception-path experiments. +- `[DisassemblyDiagnoser]`: generated code; heavy and subject to toolchain/platform restrictions. +- EventPipe profiler: cross-platform CPU/GC/JIT trace artifacts for a targeted deep dive. +- ETW/native hardware diagnostics: Windows/privilege/toolchain restrictions; opt in only when needed. + +Diagnosers may require separate runs and increase duration. + +## Jobs and runtimes + +The codebelt runner starts from `BenchmarkWorkspaceOptions.Slim`. Add runtime jobs only for an explicit cross-runtime question: + +```csharp +return c + .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(ClrRuntime.Net48)) + .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core80)) + .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core90)) + .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core10_0)); +``` + +| Target | Job runtime | +|---|---| +| .NET Framework 4.8 | `ClrRuntime.Net48` (Windows only) | +| .NET 8 | `CoreRuntime.Core80` | +| .NET 9 | `CoreRuntime.Core90` | +| .NET 10 | `CoreRuntime.Core10_0` | + +Only add jobs that the SUT and benchmark toolchain can execute. Keep the runner-default-only template as `return c;` with no unused runtime `using` directives. + +Let BenchmarkDotNet choose warmup, iteration, launch, and invocation counts unless the performance question requires cold start, monitoring, or another specific run strategy. Short/dry jobs validate or iterate quickly; they do not replace the default job for performance conclusions. + +## Runner commands + +Build: + +```powershell +dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj +``` + +List cases without measuring: + +```powershell +dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* +``` + +Dry execution smoke: + +```powershell +dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* +``` + +Full default run: + +```powershell +dotnet run -c Release --project tooling/{runner} -- --filter *{BenchmarkClass}* +``` + +Reports are written under `reports/`. + +## Primary sources + +- BenchmarkDotNet good practices: +- Parameterization: +- Setup and cleanup: +- Baselines: +- Diagnosers: +- Validators: +- .NET diagnostics overview: diff --git a/skills/dotnet-benchmark/references/candidate-selection.md b/skills/dotnet-benchmark/references/candidate-selection.md new file mode 100644 index 0000000..5d1da1b --- /dev/null +++ b/skills/dotnet-benchmark/references/candidate-selection.md @@ -0,0 +1,110 @@ +# Selecting High-Value Benchmark Candidates + +Use this reference when the user supplies a type but does not already provide an exact performance hypothesis. The goal is to find a small set of operations whose measurement could plausibly guide an optimization decision. This is source-informed discovery, not proof that the operation dominates a deployed workload. + +## Evidence ladder + +Prefer evidence in this order: + +1. Production telemetry, traces, allocation profiles, contention traces, or an existing performance regression. +2. Representative application benchmarks or load-test results that identify the type or call path. +3. Real call sites that reveal frequency, batching, concurrency, and input shape. +4. Tests and examples that reveal valid, boundary, failure, and compatibility scenarios. +5. Implementation inspection that reveals algorithmic scaling, allocations, synchronization, code generation, or repeated work. +6. Public API shape alone. + +Do not promote weak evidence to a stronger label. When no profile exists, say that the candidate is selected from source and usage evidence. + +## Inspection sequence + +1. Read the complete type, including partial declarations and generated-source inputs where available. +2. Identify the public or internal consumer operations that enter the type. Include inherited/interface operations when call sites use them. +3. Search call sites across `src/`, tests, samples, tooling, and benchmarks. Note loop nesting, batch sizes, collection use, concurrency, and repeated calls. +4. Read focused tests and examples to learn realistic inputs, equivalence rules, error behavior, and boundary cases. +5. Follow the selected entry point into private helpers and important collaborators far enough to understand dominant work. Do not benchmark private helpers merely because they look expensive. +6. Inspect existing benchmarks and reports before adding another suite. Extend a compatible benchmark rather than duplicating it. +7. If profiling artifacts exist, use their hot stacks/allocation types/contention sites to confirm or reorder candidates. + +Useful searches include the exact type name, constructed generic forms, interface/base-type names, factory methods, extension methods, and characteristic member names. Search aliases and static imports when a direct type search is sparse. + +## Cost signals + +Treat these as hypotheses that require measurement, not defects by themselves: + +- Nested loops, repeated scans, recursion, sorting, hashing, or work whose complexity grows with input size. +- Per-call `new` allocations, boxing, closures, iterator state machines, LINQ materialization, array/string copies, interpolation, and repeated buffer growth. +- Repeated parsing, formatting, normalization, encoding/decoding, regular-expression construction, serialization, reflection, expression compilation, or metadata lookup. +- `GetHashCode` and `Equals` on types heavily used as dictionary/set keys, especially when they traverse fields, strings, collections, or normalized forms. +- Locks, concurrent collections, atomics, task scheduling, blocking waits, and shared mutable state used from multiple call sites. +- Exception construction or throwing on a documented/common path. +- Cache misses, lazy initialization, cold initialization, and expensive work that could be hoisted or reused. +- Span/array conversions, pinning, interop, crypto transforms, compression, and buffer pooling. +- Branch-heavy parsing/matching whose cost changes for success/failure, hit/miss, early/late match, or well/ill-shaped input. + +Trivial getters, constant-returning properties, thin wrappers, and rarely used diagnostic formatting are low priority unless call-site evidence shows exceptional frequency or a regression specifically names them. + +## Candidate matrix + +Create a compact matrix before choosing benchmark methods: + +| Candidate operation | Usage evidence | Cost/scaling signal | Representative cases | Comparison available | Measurement fitness | Decision | +|---|---|---|---|---|---|---| +| `TryParse` | Called for each imported record | Scans and allocates normalized strings | short/typical/large; valid/invalid | current vs span candidate | deterministic, in-memory | Select | +| `ToString` | Debug/logging only | one allocation | typical object | none | measurable but low impact | Reject | + +Use a score only as a prioritization aid, never as a performance claim. A practical score is: + +- 0–3 for observed frequency/importance; +- 0–3 for per-call cost or input scaling; +- 0–2 for allocation, contention, or cold-path significance; +- 0–2 for optimization leverage or a credible comparison; +- 0–2 for deterministic measurement fitness; +- subtract 0–3 for external noise, unresettable state, unrealistic isolation, or weak workload evidence. + +Record the evidence behind each score. A high score based only on source appearance is still exploratory. + +## Select the performance questions + +Choose one to three questions that could change an engineering decision. Good questions are specific: + +- Does the span-based parser reduce latency and allocations versus the current string parser for representative valid and invalid inputs? +- How does wildcard matching scale across path length and pattern complexity, and does the current method allocate per call? +- Under a fixed worker count and hit/miss mix, does cache lookup show lock contention? + +Weak questions merely inventory members: + +- How fast are all public methods? +- Is `ToString` faster than `GetHashCode`? +- What happens if every available enum and size is crossed with every method? + +If several unrelated operations are genuinely important, prefer separate benchmark classes or explicit comparison categories so reports do not imply invalid ratios. + +## Workload cases + +Derive cases from call sites and tests. Include only dimensions that can change the conclusion: + +- typical production-like case; +- small/empty boundary when valid; +- a scaling point large enough to expose complexity; +- adverse-but-valid case such as miss, late match, escaped content, collision, invalid parse, or cache miss; +- cold/warm state only when both are meaningful consumer modes. + +Random bytes are appropriate for some codecs, hashes, and raw buffers, but not as a universal workload. Parsers, matchers, compressors, collections, and caches often need structured cases. Use fixed seeds only after choosing a representative distribution. + +Avoid independent parameter properties when values are coupled. `[Params]` creates the Cartesian product of every axis. Use a scenario record/class with a readable `ToString()` supplied through `[ParamsSource]`, or use `[ArgumentsSource]`, to enumerate only meaningful combinations. + +## Profiling-first gate + +Recommend profiling or a macrobenchmark before writing a microbenchmark when: + +- the user asks what makes an application or endpoint slow but provides no hotspot evidence; +- the type mainly coordinates network, disk, database, process, UI, or distributed work; +- the important behavior is request concurrency, queueing, thread-pool starvation, or tail latency across components; +- startup/JIT/module loading for a whole application is the target; +- state cannot be reset reproducibly or isolation changes the behavior under investigation. + +A useful response still narrows the next step: name the entry point, workload, and signal to collect. After profiling identifies a controllable code path, return to BenchmarkDotNet to compare implementations under a reproducible workload. + +## Candidate rejection rules + +Reject or defer a candidate when it has no plausible consumer impact, duplicates a selected operation, cannot be isolated without changing semantics, measures mostly test scaffolding, crosses unrelated work, depends on uncontrolled external systems, or has no representative workload. Report the reason so the user can correct missing context. diff --git a/skills/dotnet-benchmark/references/codebelt-conventions.md b/skills/dotnet-benchmark/references/codebelt-conventions.md index 1badc73..3da25b0 100644 --- a/skills/dotnet-benchmark/references/codebelt-conventions.md +++ b/skills/dotnet-benchmark/references/codebelt-conventions.md @@ -1,133 +1,59 @@ -# Codebelt Benchmark Conventions - -These rules mirror the pasted "Writing Performance Tests in Cuemon" guidance and the real -implementations in `codebeltnet/cuemon` and `codebeltnet/xunit`. Follow them so benchmarks stay -consistent, discoverable, and comparable across repos. - -## Naming and placement - -- Benchmark projects live under `tuning/` and are named `{SutProject}.Benchmarks` - (e.g. `Cuemon.Core.Benchmarks`, `Codebelt.Extensions.Xunit.Benchmarks`). -- A benchmark class name **ends with `Benchmark`** (e.g. `DateSpanBenchmark`, `Sha512256Benchmark`). -- The class lives in the **same namespace as the type it measures** — do **not** append `.Benchmarks`. - The benchmark project overrides `RootNamespace` to the SUT root so this compiles: - - ```xml - - Cuemon - - ``` - - So `Cuemon.Security.Cryptography.SHA512256` is benchmarked by a `Sha512256Benchmark` class declared - in `namespace Cuemon.Security.Cryptography`, inside the `Cuemon.Security.Cryptography.Benchmarks` - assembly. -- Method names are descriptive scenarios (`Parse_Short`, `ComputeHash_Large`, `Match_ComplexWildcard`). - -## Always-on attributes - -Every codebelt benchmark class carries: - -- `[MemoryDiagnoser]` -- `[GroupBenchmarksBy(...)]` — `ByCategory` for member-scenario suites, `ByParams` for size/variant sweeps -- a `[GlobalSetup]` that prepares deterministic state -- exactly one `[Benchmark(Baseline = true, ...)]` anchor, with `Description` on each method - -## Tier 1 — Simple type (member scenarios) - -For value-like types with no size-sensitive input, benchmark the meaningful members as discrete -scenarios. This is the `DateSpanBenchmark` shape: - -```csharp -namespace Cuemon -{ - [MemoryDiagnoser] - [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] - public class DateSpanBenchmark - { - private DateSpan _shortSpan; - - [GlobalSetup] - public void Setup() => _shortSpan = new DateSpan(DateTime.UtcNow, DateTime.UtcNow.AddHours(36)); - - [Benchmark(Baseline = true, Description = "Ctor (short span)")] - public DateSpan Construct_Short() => new DateSpan(DateTime.UtcNow, DateTime.UtcNow.AddHours(36)); - - [Benchmark(Description = "ToString (short)")] - public string ToString_Short() => _shortSpan.ToString(); - - [Benchmark(Description = "GetWeeks (short)")] - public int GetWeeks_Short() => _shortSpan.GetWeeks(); - } -} -``` - -Cover construction, parsing/formatting, equality, hashing, and any hot instance methods. Template: -`assets/simple-benchmark.cs`. - -## Tier 2 — Complex / size- or variant-sensitive type - -For hashing, parsing, buffers, or anything whose cost scales with input or has competing -implementations, sweep with `[Params]` and prepare payloads in `[GlobalSetup]`. Two real shapes: - -`Sha512256Benchmark` (variant + size, `ByParams`, compares custom vs built-in): - -```csharp -[MemoryDiagnoser] -[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] -public class Sha512256Benchmark -{ - public enum AlgorithmVariant { CustomSHA512_256, SHA512_Truncated } - - [Params(AlgorithmVariant.CustomSHA512_256, AlgorithmVariant.SHA512_Truncated)] - public AlgorithmVariant Variant { get; set; } - - private byte[] _smallInput; // 64 bytes - private byte[] _largeInput; // 1 MB - - [GlobalSetup] - public void GlobalSetup() { /* seeded Random(42) fills deterministic payloads */ } - - [Benchmark(Baseline = true, Description = "Custom SHA-512/256 - small")] - public byte[] CustomSHA512256_Small() { /* ... */ } -} -``` - -`TestBenchmark` (size sweep via `[Params(8, 256, 4096)]`, `ByCategory`): - -```csharp -[MemoryDiagnoser] -[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] -public class TestBenchmark -{ - [Params(8, 256, 4096)] - public int Length { get; set; } - - [GlobalSetup] - public void Setup() { /* build patterns/inputs from Length */ } - - [Benchmark(Baseline = true, Description = "Match - exact string")] - public bool Match_Exact() => Test.Match(_shortPattern, _shortActual); -} -``` - -Template: `assets/params-benchmark.cs`. The starter keeps input size as the `[Params]` axis and -compares baseline/candidate implementations through separate benchmark methods so `Baseline = true` -stays meaningful. If you instead introduce an implementation enum as a `[Params]` dimension, collapse -the measured work into one dispatching `[Benchmark]` method; do not keep duplicate benchmark methods -and an unused parameter. Prefer seeded RNG (`new Random(42)`) and fixed sizes so runs are -deterministic. Choose micro / mid / macro sizes to reveal trends. - -## How to pick a tier - -Lean Tier 2 when the type: takes a collection/stream/buffer/string whose length matters, has multiple -implementations worth comparing, exposes an algorithm with a size parameter, or is on a documented hot -path. Otherwise Tier 1 is enough. When unsure, ask the user which members and input sizes matter most — -they know the hot paths. - -## Reference files (source of truth) - -- `codebeltnet/cuemon/tuning/Cuemon.Core.Benchmarks/DateSpanBenchmark.cs` -- `codebeltnet/cuemon/tuning/Cuemon.Security.Cryptography.Benchmarks/Sha512256Benchmark.cs` -- `codebeltnet/xunit/tuning/Codebelt.Extensions.Xunit.Benchmarks/TestBenchmark.cs` -- `codebeltnet/cuemon/tooling/bdn-runner/Program.cs` and - `codebeltnet/xunit/tooling/benchmark-runner/Program.cs` (runner + multi-runtime jobs) +# Codebelt Benchmark Conventions + +Follow these repository conventions so benchmark projects remain discoverable and consistent across codebelt repositories. Experiment validity takes precedence over forcing a template shape. + +## Naming and placement + +- Benchmark projects live under `tuning/` and are named `{SutProject}.Benchmarks`, such as `Cuemon.Core.Benchmarks`. +- A benchmark class name ends with `Benchmark` and names the measured question or type, such as `DateSpanFormattingBenchmark` or `Sha512256ComparisonBenchmark`. +- The benchmark class uses the same namespace as the type it measures; do not append `.Benchmarks`. The project overrides `RootNamespace` to the SUT root. +- Method names distinguish implementations or scenarios and every `[Benchmark]` has a readable `Description`. +- One `tooling/` runner host discovers all `tuning/` projects through the existing wildcard project reference. + +## Default instrumentation + +Every codebelt benchmark class uses `[MemoryDiagnoser]`. Add `[GroupBenchmarksBy]` when the class has multiple methods/params and the grouping makes the report clearer. Use `[GlobalSetup]` when state or correctness checks must be prepared outside measurement; do not add an empty setup merely for visual consistency. + +Baselines follow experiment semantics: + +- A current/reference implementation is `Baseline = true` when one or more equivalent alternatives are compared. +- A single-operation characterization benchmark has no baseline. +- Unrelated operations do not share a baseline. +- A cohesive class with several alternative pairs uses `[BenchmarkCategory]`, grouping by category, and one baseline per category. + +This refines the older “exactly one baseline per class” shortcut. BenchmarkDotNet ratios are useful only when the grouped methods perform comparable work. + +## Experiment shapes + +### Equivalent implementation comparison + +Use `assets/comparison-benchmark.cs` when the same consumer operation has a current/reference and candidate implementation. Keep inputs and wrappers symmetric, validate equivalent results in setup, and sweep only dimensions that can change the comparison. + +### Single-operation characterization + +Use `assets/operation-benchmark.cs` when there is no honest competing implementation. This shape characterizes time, allocations, and scaling for one operation across representative cases without inventing a baseline. + +### Multiple independent questions + +Prefer separate focused classes. If existing repository convention keeps them together, use categories that prevent unrelated ratios. Construction, formatting, equality, hashing, parsing, and matching are not automatically comparable merely because they belong to one type. + +## Deterministic inputs + +Build inputs outside measured methods. Use fixed seeds only when pseudo-random data matches the domain. Prefer structured inputs for parsers, matchers, collections, caches, and serializers. Parameter cases should have readable report labels. + +Do not use network, disk, database, logging, or sleep calls inside a microbenchmark. When external behavior is the point, select profiling, load testing, or a macrobenchmark and preserve codebelt project placement only if it remains useful. + +## Runner and reports + +The runner calls `Codebelt.Extensions.BenchmarkDotNet.Console.BenchmarkProgram.Run`, reuses the repository's existing name such as `benchmark-runner` or `bdn-runner`, and writes artifacts under `reports/`. Default runtime configuration stays plain `return c;`; add runtime jobs only when cross-runtime comparison is explicitly requested. + +## Reference implementations + +Repository precedents include: + +- `codebeltnet/cuemon/tuning/Cuemon.Core.Benchmarks/DateSpanBenchmark.cs` +- `codebeltnet/cuemon/tuning/Cuemon.Security.Cryptography.Benchmarks/Sha512256Benchmark.cs` +- `codebeltnet/xunit/tuning/Codebelt.Extensions.Xunit.Benchmarks/TestBenchmark.cs` +- the `tooling/bdn-runner` or `tooling/benchmark-runner` hosts in those repositories + +Use these for placement and runner conventions, not as authority to preserve a misleading experiment. Adapt the benchmark design to the actual performance question. diff --git a/skills/dotnet-benchmark/references/experiment-design.md b/skills/dotnet-benchmark/references/experiment-design.md new file mode 100644 index 0000000..70e132e --- /dev/null +++ b/skills/dotnet-benchmark/references/experiment-design.md @@ -0,0 +1,144 @@ +# Designing Trustworthy BenchmarkDotNet Experiments + +Use this reference after candidate selection. Every benchmark method should answer a stated performance question with representative inputs and controlled conditions. + +## Define the experiment before the attributes + +Write down: + +- the operation and consumer-visible behavior being measured; +- the primary metric: time/throughput, allocations, scaling, contention, cold start, or exception frequency; +- baseline and candidate implementations, if both exist; +- parameter cases and why each can change the result; +- setup/reset/disposal strategy; +- a correctness oracle; +- environmental variables that must remain fixed. + +If these cannot be defined, more inspection is needed. Attributes do not rescue an ambiguous experiment. + +## Comparison semantics + +A baseline exists to create a meaningful ratio. Use the current production implementation or an established reference implementation as the baseline when all methods in the group perform equivalent observable work on identical input. + +Do not compare unrelated operations. Construction, parsing, formatting, equality, hashing, copying, and validation answer different questions. Give them separate classes, or use explicit `[BenchmarkCategory]` groups with one baseline inside each group only when every category has alternatives. + +For a single current implementation measured across sizes or scenarios, omit `Baseline = true`. The parameter columns and absolute time/allocation results already describe scaling. A fake baseline adds no evidence. + +When comparing runtimes rather than implementations, use a job baseline and keep the measured method the same. Do not mix runtime and algorithm changes in one conclusion unless the full matrix is intentional. + +## Representative inputs + +Prefer cases grounded in real usage. Typical input should come first; add boundaries and adverse cases only when they exercise a distinct path. + +Examples: + +- Parsers: valid typical, valid large, invalid early, invalid late, escaped/culture-sensitive when supported. +- Match/search: hit early, hit late, miss, simple pattern, complex pattern, realistic lengths. +- Collections: empty/small/typical/large, hit/miss, collision-heavy only when plausible, pre-sized versus growth only if that is the question. +- Hash/codec/compression: representative size distribution and content entropy, not only zero-filled or arbitrary random buffers. +- Equality/hashing: equal, unequal-early, unequal-late, and actual dictionary/set usage when that is the consumer operation. +- Caches: warm hit, miss/fill, eviction, and contention as separate questions. + +Use `[Params]` for independent scalar dimensions and `[ParamsSource]`/`[ArgumentsSource]` for coupled cases. Give complex case objects stable readable names through `ToString()` so reports are interpretable. + +Keep the case count disciplined. Each parameter combination multiplies methods, jobs, launch count, and diagnoser runs. Three representative sizes usually tell more than a dense power-of-two sweep; add points only when they locate a threshold or crossover. + +## Setup, state, and disposal + +Use `[GlobalSetup]` for deterministic state that is not part of the operation: payload creation, parsing expected results, object construction for instance methods, and correctness checks. BenchmarkDotNet runs global setup for each benchmark method and parameter combination, so setup must not rely on another benchmark method having run first. + +Use `[GlobalCleanup]` for resources owned by the benchmark. Avoid external I/O resources in microbenchmarks; cleanup does not remove their measurement noise. + +Mutating operations need equivalent starting state. Options include: + +- create or clone state inside every benchmark method when that creation is genuinely part of the compared operation and both sides pay the same cost; +- benchmark a representative batch/sequence over prebuilt state and use `OperationsPerInvoke` when per-operation normalization remains honest; +- use iteration setup only for macro-scale operations where forced single-invocation behavior will not dominate. + +`[IterationSetup]` forces one invocation/unroll factor and can distort tiny operations. Do not use it reflexively to reset a nanosecond-scale benchmark. + +Avoid state leakage between methods, params, warmup, and measurement. Never depend on benchmark execution order. + +## Correctness oracle + +An optimization benchmark without correctness validation can reward wrong code. Before a full run: + +1. Execute baseline and candidate for every scenario outside the timed method. +2. Compare observable results with the domain's real equivalence rule, including output buffers, status codes, exceptions, mutations, and side effects. +3. Fail setup or a focused test when results differ. +4. Keep the assertion/check out of the timed path. + +For non-equivalent APIs, do not force a comparison. Characterize them separately and state the semantic difference. + +BenchmarkDotNet's `ReturnValueValidator` can supplement this for compatible return values, but it does not replace domain-aware correctness checks. + +## Prevent dead-code elimination and accidental work + +Return the computed result whenever practical. For `void`, ref-like, or multi-output work, consume observable outputs with `BenchmarkDotNet.Engines.Consumer` or return a stable derived value. Do not return a precomputed field while discarding the measured call. + +Keep logging, assertions, `Random`, fixture construction, reflection discovery, and string formatting used only to label cases out of timed methods. Avoid closures and LINQ in the benchmark wrapper unless they are the subject under test. + +Do not add a manual loop merely to make a tiny method measurable; BenchmarkDotNet chooses invocation counts and subtracts overhead. Use an in-method loop only when a batch is the real workload or state reset requires a defined sequence, then declare `OperationsPerInvoke` accurately. + +## Fair comparisons + +- Feed identical logical inputs and starting state to every implementation. +- Match API semantics, validation, culture, encoding, comparer, error handling, and output ownership. +- Do not let one side reuse cached/precompiled state while the other recreates it unless the experiment explicitly compares those consumer strategies. +- Keep wrapper overhead symmetric. If one API requires an adapter, decide whether the adapter is part of real consumer cost and document it. +- Do not compare a scalar API with a batched/vectorized API per invocation without normalizing per item and explaining the workload difference. +- Keep compiler settings, runtime, architecture, GC mode, environment variables, and affinity consistent unless one of them is the experimental variable. + +## Specialized workloads + +### Async + +Return and await the real `Task` or `ValueTask`. Never use `.Result`/`.Wait()`. Separate synchronous completion from genuinely asynchronous completion when both occur in production. A fake completed task measures the fake, not the I/O path. Use macro/load testing for external async I/O. + +### Concurrency and contention + +Define worker count, shared state, operation mix, synchronization start, and per-operation normalization. Avoid measuring task creation when lock/collection throughput is the question. Add `[ThreadingDiagnoser]` when completed work items or lock-contention counts help interpret the run. For service-level throughput, queueing, or tail latency, use a load test or profiler instead of a naive BenchmarkDotNet loop. + +### Cold start and initialization + +Keep cold-start experiments separate from steady-state throughput. Use a cold-start job only when process/JIT/initialization cost is the question, and define what is cold: process, type initializer, cache, parser, or application host. + +### Exceptions and invalid inputs + +Benchmark an exception path only when it is part of documented or observed behavior. Separate successful and throwing paths, add `[ExceptionDiagnoser]` when frequency matters, and do not use exceptions as a substitute for invalid-result checks. + +### Extremely fast operations + +Benchmark trivial getters/operators only with evidence of extreme frequency or a known regression. Ensure the result is consumed. Inspect disassembly when the question concerns inlining, bounds-check elimination, vectorization, or code generation. + +## Diagnoser routing + +`[MemoryDiagnoser]` is the codebelt default and belongs on every benchmark class. Add only what answers the question: + +| Question | Diagnoser/tool | Notes | +|---|---|---| +| Managed allocations and GC | `[MemoryDiagnoser]` | Always on; interpret allocated bytes per operation. | +| Lock/thread-pool activity | `[ThreadingDiagnoser]` | .NET Core 3+; useful for contention experiments. | +| Thrown exception frequency | `[ExceptionDiagnoser]` | Use only for intentional exception-path analysis. | +| JIT/code generation | `[DisassemblyDiagnoser]` | Heavy; platform/toolchain limitations apply. | +| CPU/GC/JIT hot stacks | EventPipe profiler | Cross-platform targeted deep dive; creates separate profiling artifacts. | +| Windows ETW/native counters | ETW/hardware counter diagnosers | Platform/privilege restrictions; enable only when required. | + +Diagnosers can create additional runs and change total duration. Do not combine every diagnoser into a default benchmark. + +## Layered validation + +1. Run existing correctness tests for the SUT where feasible. +2. Execute the benchmark's correctness oracle for every case. +3. Build the benchmark project in Release. +4. Run `--list flat` with a filter and confirm the expected case/method combinations without accidental Cartesian products. +5. Run a `--job dry` execution smoke and resolve BenchmarkDotNet validation warnings or runtime failures. +6. Run the full default job only with explicit user intent and a suitable environment. + +A dry job has too few measurements for conclusions. It validates wiring and lifecycle only. + +## Interpreting a full run + +Report the benchmark environment and exact workload. Read warnings before tables. Compare mean, error, standard deviation, median when distributions are skewed, ratio distributions within valid groups, allocated bytes, GC counts, and specialized diagnoser columns. + +Treat gains within noise as inconclusive. Check whether the result holds across representative cases and whether one case regresses. State the scope precisely: a result applies to the measured runtime, hardware, inputs, and configuration. Optimization value depends on application frequency and the maintenance/correctness cost of the change. From b8b63ac83b1226409766cf234debe39f87c445e6 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 14:51:38 +0200 Subject: [PATCH 09/38] =?UTF-8?q?=F0=9F=9A=9A=20replace=20tier-based=20ben?= =?UTF-8?q?chmark=20template=20assets?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove params-benchmark.cs and simple-benchmark.cs single-tier starters. Introduce operation-benchmark.cs and comparison-benchmark.cs as refined structural examples that support the discovery-focused workflow with clearer guidance on placeholder substitution, baseline configuration, and measurement fitness. --- .../assets/comparison-benchmark.cs | 37 ++++++++++++++++++ .../assets/operation-benchmark.cs | 28 ++++++++++++++ .../assets/params-benchmark.cs | 38 ------------------- .../assets/simple-benchmark.cs | 32 ---------------- 4 files changed, 65 insertions(+), 70 deletions(-) create mode 100644 skills/dotnet-benchmark/assets/comparison-benchmark.cs create mode 100644 skills/dotnet-benchmark/assets/operation-benchmark.cs delete mode 100644 skills/dotnet-benchmark/assets/params-benchmark.cs delete mode 100644 skills/dotnet-benchmark/assets/simple-benchmark.cs diff --git a/skills/dotnet-benchmark/assets/comparison-benchmark.cs b/skills/dotnet-benchmark/assets/comparison-benchmark.cs new file mode 100644 index 0000000..3180ea0 --- /dev/null +++ b/skills/dotnet-benchmark/assets/comparison-benchmark.cs @@ -0,0 +1,37 @@ +using System; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace {SUT_NAMESPACE}; + +// Structural example only. Replace every placeholder and adapt cases, state, calls, return types, +// equivalence checks, and names to the real consumer operation. Remove unused using directives. +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] +public class {BENCHMARK_CLASS} +{ + [Params(64, 4_096, 1_048_576)] + public int Size { get; set; } + + private byte[] _input = null!; + + [GlobalSetup] + public void Setup() + { + _input = new byte[Size]; + new Random(42).NextBytes(_input); + + var expected = {BASELINE_CALL}; + var actual = {CANDIDATE_CALL}; + if (!{EQUIVALENCE_CHECK}) + { + throw new InvalidOperationException("Baseline and candidate results differ for the current benchmark case."); + } + } + + [Benchmark(Baseline = true, Description = "{BASELINE_DESCRIPTION}")] + public {RETURN_TYPE} Current() => {BASELINE_CALL}; + + [Benchmark(Description = "{CANDIDATE_DESCRIPTION}")] + public {RETURN_TYPE} Candidate() => {CANDIDATE_CALL}; +} diff --git a/skills/dotnet-benchmark/assets/operation-benchmark.cs b/skills/dotnet-benchmark/assets/operation-benchmark.cs new file mode 100644 index 0000000..07968ab --- /dev/null +++ b/skills/dotnet-benchmark/assets/operation-benchmark.cs @@ -0,0 +1,28 @@ +using System; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace {SUT_NAMESPACE}; + +// Structural example for one operation with no honest competing implementation. Replace every +// placeholder and adapt the cases, data shape, return consumption, and names to the real workload. +// Do not add Baseline = true merely to produce a ratio column. Remove unused using directives. +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] +public class {BENCHMARK_CLASS} +{ + [Params(64, 4_096, 1_048_576)] + public int Size { get; set; } + + private byte[] _input = null!; + + [GlobalSetup] + public void Setup() + { + _input = new byte[Size]; + new Random(42).NextBytes(_input); + } + + [Benchmark(Description = "{OPERATION_DESCRIPTION}")] + public {RETURN_TYPE} Measure() => {SUT_CALL}; +} diff --git a/skills/dotnet-benchmark/assets/params-benchmark.cs b/skills/dotnet-benchmark/assets/params-benchmark.cs deleted file mode 100644 index 975277a..0000000 --- a/skills/dotnet-benchmark/assets/params-benchmark.cs +++ /dev/null @@ -1,38 +0,0 @@ -using System; -using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Configs; - -namespace {SUT_NAMESPACE} -{ - // Complex-tier template: sweep representative input sizes, build deterministic payloads once in - // [GlobalSetup], and compare a candidate implementation against a baseline. Use this shape for - // size- or variant-sensitive types (hashing, parsing, buffers, algorithms). If you need an - // implementation enum as a [Params] dimension, collapse the measured work into one dispatching - // [Benchmark] method instead of keeping separate benchmark methods and an unused parameter. - // Replace the payload sizes and the measured calls with the real API of {SUT_TYPE}. - [MemoryDiagnoser] - [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] - public class {SUT_TYPE}Benchmark - { - // Sweep representative micro / mid / macro sizes so trends are visible. - [Params(64, 4096, 1_048_576)] - public int Size { get; set; } - - private byte[] _payload; - - [GlobalSetup] - public void Setup() - { - // Seeded RNG keeps payloads deterministic across runs. - var rng = new Random(42); - _payload = new byte[Size]; - rng.NextBytes(_payload); - } - - [Benchmark(Baseline = true, Description = "Process (baseline)")] - public int Process_Baseline() => {SUT_TYPE}.Process(_payload); - - [Benchmark(Description = "Process (candidate)")] - public int Process_Candidate() => {SUT_TYPE}.ProcessOptimized(_payload); - } -} diff --git a/skills/dotnet-benchmark/assets/simple-benchmark.cs b/skills/dotnet-benchmark/assets/simple-benchmark.cs deleted file mode 100644 index 1a4c5bd..0000000 --- a/skills/dotnet-benchmark/assets/simple-benchmark.cs +++ /dev/null @@ -1,32 +0,0 @@ -using System; -using BenchmarkDotNet.Attributes; -using BenchmarkDotNet.Configs; - -namespace {SUT_NAMESPACE} -{ - // Simple-tier template: benchmark the meaningful members of a value-like type as discrete - // scenarios. Keep one method marked Baseline = true as the comparison anchor. Replace the - // placeholder members below with the real API of {SUT_TYPE}. - [MemoryDiagnoser] - [GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] - public class {SUT_TYPE}Benchmark - { - private {SUT_TYPE} _instance; - - [GlobalSetup] - public void Setup() - { - // Deterministic, cheap-to-build state prepared once (never measured). - _instance = new {SUT_TYPE}(); - } - - [Benchmark(Baseline = true, Description = "Construct")] - public {SUT_TYPE} Construct() => new {SUT_TYPE}(); - - [Benchmark(Description = "ToString")] - public string ToStringScenario() => _instance.ToString(); - - [Benchmark(Description = "GetHashCode")] - public int GetHashCodeScenario() => _instance.GetHashCode(); - } -} From 0734e7361bb7cf85b1dc5fefba89897f7dd289d7 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 14:51:48 +0200 Subject: [PATCH 10/38] =?UTF-8?q?=E2=9C=85=20update=20benchmark=20evals=20?= =?UTF-8?q?for=20discovery-focused=20workflow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refactor evals.json test cases to validate candidate selection, evidence gathering, cost-signal analysis, and measurement fitness instead of tier-based template logic. Add evals/files/ fixtures supporting discovery workflow scenarios. --- skills/dotnet-benchmark/evals/evals.json | 195 ++++++++++-------- .../concurrent-memoizer/ConcurrentMemoizer.cs | 30 +++ .../concurrent-memoizer/MemoizedEndpoint.cs | 13 ++ .../evals/files/date-window/DateWindow.cs | 6 + .../files/date-window/DateWindowUsage.cs | 8 + .../files/decimal-parser/DecimalParser.cs | 29 +++ .../decimal-parser/DecimalParserTests.cs | 7 + .../files/decimal-parser/ImportPipeline.cs | 18 ++ .../files/report-loader/ReportImportJob.cs | 6 + .../evals/files/report-loader/ReportLoader.cs | 14 ++ .../evals/files/route-matcher/RouteMatcher.cs | 16 ++ .../files/route-matcher/RouteMatcherTests.cs | 13 ++ .../evals/files/route-matcher/RouteTable.cs | 17 ++ 13 files changed, 289 insertions(+), 83 deletions(-) create mode 100644 skills/dotnet-benchmark/evals/files/concurrent-memoizer/ConcurrentMemoizer.cs create mode 100644 skills/dotnet-benchmark/evals/files/concurrent-memoizer/MemoizedEndpoint.cs create mode 100644 skills/dotnet-benchmark/evals/files/date-window/DateWindow.cs create mode 100644 skills/dotnet-benchmark/evals/files/date-window/DateWindowUsage.cs create mode 100644 skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParser.cs create mode 100644 skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParserTests.cs create mode 100644 skills/dotnet-benchmark/evals/files/decimal-parser/ImportPipeline.cs create mode 100644 skills/dotnet-benchmark/evals/files/report-loader/ReportImportJob.cs create mode 100644 skills/dotnet-benchmark/evals/files/report-loader/ReportLoader.cs create mode 100644 skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcher.cs create mode 100644 skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcherTests.cs create mode 100644 skills/dotnet-benchmark/evals/files/route-matcher/RouteTable.cs diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json index aa6f510..048f9ae 100644 --- a/skills/dotnet-benchmark/evals/evals.json +++ b/skills/dotnet-benchmark/evals/evals.json @@ -1,83 +1,112 @@ -{ - "skill_name": "dotnet-benchmark", - "evals": [ - { - "id": 1, - "prompt": "This repo has a src/Acme.Core/Acme.Core.csproj library but no benchmarks at all. Set up BenchmarkDotNet so I can start benchmarking, following the codebelt convention.", - "expected_output": "The harness is onboarded in place: a tuning/Acme.Core.Benchmarks project referencing the SUT src project, a single tooling/ runner host that references Codebelt.Extensions.BenchmarkDotNet.Console, benchmark packages added (CPM or PackageReference as the repo dictates), and the new projects wired into the detected solution.", - "expectations": [ - "Runs or references scripts/check-benchmark-requirements.ps1 to detect current state before changing anything", - "Creates tuning/Acme.Core.Benchmarks with a ProjectReference to src/Acme.Core/Acme.Core.csproj and RootNamespace set to the SUT root namespace", - "Creates exactly one tooling/ runner host whose Program.cs calls BenchmarkProgram.Run and defaults the folder name to benchmark-runner", - "Resolves BenchmarkDotNet, BenchmarkDotNet.Diagnostics.Windows, and Codebelt.Extensions.BenchmarkDotNet.Console versions from NuGet instead of hardcoding", - "Wires the new projects into the detected .slnx or .sln solution format" - ] - }, - { - "id": 2, - "prompt": "Add a thorough benchmark for the Acme.DateWindow value type (constructor, ToString, equality, GetHashCode). It's a small value type.", - "expected_output": "A simple-tier benchmark class DateWindowBenchmark in namespace Acme (no .Benchmarks suffix), using MemoryDiagnoser, GroupBenchmarksBy ByCategory, a GlobalSetup, and a Baseline=true anchor, benchmarking the members as discrete scenarios.", - "expectations": [ - "Chooses the simple/member-scenario tier and briefly explains why", - "Names the class DateWindowBenchmark and declares it in the same namespace as the SUT (no .Benchmarks suffix)", - "Applies [MemoryDiagnoser] and marks exactly one method Baseline = true with Description values", - "Places the class under tuning/{SutProject}.Benchmarks and does not run the benchmark by default" - ] - }, - { - "id": 3, - "prompt": "Benchmark our Acme.Hashing.Crc32 implementation against the built-in one across small and large inputs — I want to see allocations too.", - "expected_output": "A complex-tier benchmark using [Params] to sweep input sizes, deterministic payloads built in GlobalSetup with a seeded Random, MemoryDiagnoser, and separate baseline/candidate benchmark methods that compare implementations without unused template scaffolding.", - "expectations": [ - "Chooses the complex/params tier for a size- and variant-sensitive type", - "Uses [Params] to sweep multiple input sizes and prepares deterministic payloads in [GlobalSetup]", - "Compares implementations through separate benchmark methods with exactly one Baseline = true anchor instead of leaving behind an unused implementation [Params] property", - "Includes [MemoryDiagnoser] and a Baseline = true comparison anchor", - "Omits unused using directives from the generated benchmark file", - "Uses deterministic in-memory data with no network, disk, or database in measured methods" - ] - }, - { - "id": 4, - "prompt": "I want the benchmark for Acme.Core's Parser to also compare .NET Framework 4.8 against .NET 9 and .NET 10.", - "expected_output": "Runner Program.cs gains one AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...)) call per requested runtime (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0), includes the runtime-job using directives needed for those jobs, and the benchmark project targets the matching TFMs so those jobs can run.", - "expectations": [ - "Adds one .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...)) line per requested runtime using the correct monikers (ClrRuntime.Net48, CoreRuntime.Core90, CoreRuntime.Core10_0)", - "Keeps the runtime-job using directives only when those extra jobs are present", - "Ensures the benchmark project TargetFrameworks include the matching TFMs (e.g. net48) so the jobs are runnable", - "Explains that the Codebelt runner host runs on .NET 9/10 while BenchmarkDotNet jobs can measure other runtimes" - ] - }, - { - "id": 5, - "prompt": "Our repo already has tooling/bdn-runner and a tuning/ folder with other benchmarks. Add a benchmark for Cuemon.Security.Cryptography.SHA512256.", - "expected_output": "The existing bdn-runner is reused (no second runner is created), the benchmark class is added under the appropriate tuning/*.Benchmarks project in namespace Cuemon.Security.Cryptography, and the runner's existing wildcard tuning reference picks it up.", - "expectations": [ - "Detects and reuses the existing tooling/bdn-runner instead of creating a benchmark-runner", - "Adds the class to the matching tuning/*.Benchmarks project in namespace Cuemon.Security.Cryptography with a Sha512256-style Benchmark class name", - "Does not duplicate the runner host or change the existing wildcard tuning ProjectReference" - ] - }, - { - "id": 6, - "prompt": "Set up benchmarking for a plain SDK-style repo that uses a classic MyApp.sln and does not use central package management.", - "expected_output": "Onboarding adapts to the non-codebelt layout: the benchmark project declares its own TargetFrameworks and versioned BenchmarkDotNet PackageReferences (no Directory.Packages.props), the runner declares its Console PackageReference version, and projects are added with dotnet sln add against the .sln.", - "expectations": [ - "Detects the classic .sln format and non-CPM state and adapts instead of assuming the codebelt-centralized layout", - "Puts versioned entries directly in the project files because there is no Directory.Packages.props", - "Adds self-contained TargetFrameworks and IsPackable=false to the benchmark/runner projects since the root Directory.Build.props does not centralize benchmark conventions", - "Wires projects into the solution using dotnet sln MyApp.sln add" - ] - }, - { - "id": 7, - "prompt": "Set up a benchmark for Acme.Core.Parser, but keep the runner on its default runtime only. I do not want extra net48/net9/net10 comparison jobs.", - "expected_output": "Runner Program.cs stays on the default runner configuration with a plain `return c;`, no raw template placeholders or stray runtime-job syntax, and no extra runtime-job using directives.", - "expectations": [ - "Leaves the runner-default-only case as valid C# by replacing the runtime-job placeholder with nothing so ConfigureBenchmarkDotNet returns `c;`", - "Does not emit raw template placeholders or malformed chained AddJob syntax when no extra runtimes were requested", - "Omits BenchmarkDotNet.Environments, BenchmarkDotNet.Jobs, and Codebelt.Extensions.BenchmarkDotNet using directives when no extra runtimes were selected" - ] - } - ] -} +{ + "skill_name": "dotnet-benchmark", + "evals": [ + { + "id": 1, + "prompt": "This repo has src/Acme.Core/Acme.Core.csproj but no benchmark infrastructure. Set up the codebelt BenchmarkDotNet harness without inventing a benchmark target yet.", + "expected_output": "The agent detects repository state first and onboards only the missing tuning project and single tooling runner, preserves CPM/solution conventions, resolves current package versions dynamically, and does not fabricate a target type or benchmark method.", + "expectations": [ + "Runs or references scripts/check-benchmark-requirements.ps1 before changing harness files", + "Creates or proposes exactly one reusable tooling runner and a tuning benchmark project while preserving the detected solution and package-management conventions", + "Resolves BenchmarkDotNet package versions dynamically from NuGet.org rather than copying hardcoded example versions", + "Does not invent a SUT type, member, workload, or benchmark class when none was supplied", + "Explains that candidate discovery starts after a concrete type is available" + ] + }, + { + "id": 2, + "prompt": "Create the most useful BenchmarkDotNet benchmark for Acme.Routing.RouteMatcher from the attached source. I only know that routing gets expensive in large route tables; inspect the type and usages and choose what is worth measuring.", + "files": [ + "evals/files/route-matcher/RouteMatcher.cs", + "evals/files/route-matcher/RouteTable.cs", + "evals/files/route-matcher/RouteMatcherTests.cs" + ], + "expected_output": "The agent selects IsMatch as the evidence-backed hot operation because RouteTable calls it inside the route scan, identifies repeated regex construction as a cost hypothesis, builds structured hit/miss and pattern/path-complexity scenarios, characterizes the current operation without inventing a candidate or fake baseline, and rejects low-value Normalize/ToString member coverage.", + "expectations": [ + "Uses RouteTable call-site evidence to prioritize RouteMatcher.IsMatch rather than enumerating every public member", + "Identifies per-call wildcard-to-regex conversion and Regex.IsMatch construction/caching behavior as hypotheses to measure without claiming they are already proven bottlenecks", + "Uses representative coupled scenarios covering hit and miss plus simple and complex patterns instead of a meaningless Cartesian product or random strings", + "Authors or proposes a single-operation characterization benchmark with MemoryDiagnoser and no Baseline=true because no equivalent candidate implementation exists", + "Keeps input/setup work outside the measured method, returns or consumes the IsMatch result, and includes build, list, and dry-run validation", + "Explicitly rejects Normalize and ToString as lower-value candidates based on the supplied usage evidence" + ] + }, + { + "id": 3, + "prompt": "Benchmark Acme.Text.DecimalParser and tell me whether the span-based implementation is a worthwhile replacement for the legacy implementation. Use the attached type, call site, and tests to choose cases. Measure allocations too, but do not run a full benchmark yet.", + "files": [ + "evals/files/decimal-parser/DecimalParser.cs", + "evals/files/decimal-parser/ImportPipeline.cs", + "evals/files/decimal-parser/DecimalParserTests.cs" + ], + "expected_output": "The agent creates a fair Legacy versus Span comparison using identical UTF-8 inputs derived from realistic valid and invalid import values, treats Legacy as the production baseline, validates equivalent success/value/consumed behavior outside measurement, and stops after Release build, discovery, and dry execution without claiming a speedup.", + "expectations": [ + "Selects ParseLegacy versus TryParseSpan as equivalent implementation candidates and uses ParseLegacy as the production baseline", + "Derives valid typical, boundary, long, and invalid/partially-consumed cases from the source/tests instead of using arbitrary buffer sizes alone", + "Avoids independent parameters that create invalid combinations by using coupled scenario objects or an equivalent explicit case source", + "Checks success, parsed value, and consumed length equivalence for every case outside the timed methods", + "Feeds both methods identical bytes and keeps encoding/input construction outside measurement", + "Uses MemoryDiagnoser, returns observable results, and either performs Release build/list/dry validation or gives those commands while honestly reporting that the source-only fixture lacks a runnable harness; makes no performance claim because the full benchmark was not run" + ] + }, + { + "id": 4, + "prompt": "Add a thorough benchmark for Acme.DateWindow: construction, formatting, equality, and GetHashCode. I want useful optimization evidence, not just a benchmark file.", + "files": [ + "evals/files/date-window/DateWindow.cs", + "evals/files/date-window/DateWindowUsage.cs" + ], + "expected_output": "The agent challenges the request to compare unrelated members, uses dictionary/set call-site evidence to prioritize equality/hash behavior, treats formatting and construction as separate questions only if retained, and never makes constructor timing the baseline for formatting/equality/hashing.", + "expectations": [ + "Inspects supplied usage evidence and prioritizes Equals/GetHashCode or the dictionary lookup consumer operation over equal-weight public-member coverage", + "Does not use construction as the baseline for formatting, equality, or hashing and does not present ratios between unrelated work", + "Uses equal, unequal-early, unequal-late, dictionary hit, or dictionary miss scenarios where they answer the selected question", + "Splits independent questions into focused classes/categories or omits low-value operations with explicit reasons", + "Uses MemoryDiagnoser and an honest baseline only within an equivalent comparison group, or no baseline for a single-operation characterization" + ] + }, + { + "id": 5, + "prompt": "Find and benchmark the bottleneck in Acme.Reporting.ReportLoader. Users say loading large report files is slow. Use the attached source and usage, and make the benchmark as realistic as possible.", + "files": [ + "evals/files/report-loader/ReportLoader.cs", + "evals/files/report-loader/ReportImportJob.cs" + ], + "expected_output": "The agent refuses to claim a bottleneck from source alone and does not put disk I/O in a microbenchmark. It recommends profiling the end-to-end import and, if a useful BenchmarkDotNet artifact is requested, isolates in-memory JSON parsing with representative payload sizes while clearly excluding file-system latency from the conclusion.", + "expectations": [ + "States that source inspection alone cannot prove whether file I/O or JSON parsing dominates and recommends profiling or a macrobenchmark for the end-to-end load", + "Does not call File.ReadAllText, File.ReadAllBytes, or other disk I/O inside a Benchmark method", + "If authoring BenchmarkDotNet code, isolates Parse on preloaded in-memory UTF-8 payloads and labels the result as parsing-only", + "Uses structured small/typical/large report payloads rather than arbitrary random bytes", + "Does not claim that an in-memory parsing result represents end-to-end report-loading latency" + ] + }, + { + "id": 6, + "prompt": "Benchmark Acme.Caching.ConcurrentMemoizer. We suspect lock contention during parallel request bursts. Inspect the attached type and call site, then create the benchmark that would actually answer that question.", + "files": [ + "evals/files/concurrent-memoizer/ConcurrentMemoizer.cs", + "evals/files/concurrent-memoizer/MemoizedEndpoint.cs" + ], + "expected_output": "The agent designs a defined contention experiment rather than a sequential GetOrAdd microbenchmark: fixed worker counts, synchronized start, shared memoizer, realistic warm-hit/miss mix, per-operation normalization, ThreadingDiagnoser plus MemoryDiagnoser, and a clear warning that service tail latency still needs load testing/profiling.", + "expectations": [ + "Uses the Parallel.ForEachAsync call site as evidence that concurrent shared-state behavior is the relevant operation", + "Defines worker count, shared state, hit/miss mix, synchronized start or equivalent coordination, and per-operation normalization", + "Avoids a benchmark that only measures one sequential GetOrAdd call and avoids accidentally making Task creation the unexplained dominant cost", + "Adds ThreadingDiagnoser for lock-contention evidence alongside MemoryDiagnoser", + "Separates contention characterization from service-level throughput/tail-latency claims and recommends load testing or profiling for the latter", + "Defines reset/lifecycle behavior so cache state does not leak unpredictably between cases" + ] + }, + { + "id": 7, + "prompt": "Set up a benchmark for Acme.Core.Parser, but keep the codebelt runner on its default runtime only. I do not want net48/net8/net9/net10 comparison jobs.", + "expected_output": "The runner remains a valid default configuration with return c;, no raw placeholders, no AddJob chain, and no unused runtime-job using directives; benchmark design still waits for inspection of Parser and its workload.", + "expectations": [ + "Keeps ConfigureBenchmarkDotNet as a valid plain return c; expression without AddJob calls", + "Emits no raw runtime placeholders or unused BenchmarkDotNet.Environments, BenchmarkDotNet.Jobs, or Codebelt.Extensions.BenchmarkDotNet runtime-job using directives", + "Does not treat the absence of extra jobs as permission to skip target-type and workload inspection" + ] + } + ] +} diff --git a/skills/dotnet-benchmark/evals/files/concurrent-memoizer/ConcurrentMemoizer.cs b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/ConcurrentMemoizer.cs new file mode 100644 index 0000000..419acae --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/ConcurrentMemoizer.cs @@ -0,0 +1,30 @@ +namespace Acme.Caching; + +public sealed class ConcurrentMemoizer where TKey : notnull +{ + private readonly object _gate = new(); + private readonly Dictionary _values = new(); + + public TValue GetOrAdd(TKey key, Func factory) + { + lock (_gate) + { + if (_values.TryGetValue(key, out var value)) + { + return value; + } + + value = factory(key); + _values.Add(key, value); + return value; + } + } + + public void Clear() + { + lock (_gate) + { + _values.Clear(); + } + } +} diff --git a/skills/dotnet-benchmark/evals/files/concurrent-memoizer/MemoizedEndpoint.cs b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/MemoizedEndpoint.cs new file mode 100644 index 0000000..f697665 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/MemoizedEndpoint.cs @@ -0,0 +1,13 @@ +namespace Acme.Caching; + +public sealed class MemoizedEndpoint(ConcurrentMemoizer memoizer) +{ + public async Task HandleBurst(IEnumerable ids, CancellationToken cancellationToken) + { + await Parallel.ForEachAsync(ids, cancellationToken, (id, cancellation) => + { + _ = memoizer.GetOrAdd(id, static key => key.ToString()); + return ValueTask.CompletedTask; + }); + } +} diff --git a/skills/dotnet-benchmark/evals/files/date-window/DateWindow.cs b/skills/dotnet-benchmark/evals/files/date-window/DateWindow.cs new file mode 100644 index 0000000..c7ec5f8 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/date-window/DateWindow.cs @@ -0,0 +1,6 @@ +namespace Acme; + +public readonly record struct DateWindow(DateOnly Start, DateOnly End) +{ + public override string ToString() => $"{Start:O}/{End:O}"; +} diff --git a/skills/dotnet-benchmark/evals/files/date-window/DateWindowUsage.cs b/skills/dotnet-benchmark/evals/files/date-window/DateWindowUsage.cs new file mode 100644 index 0000000..163ecc9 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/date-window/DateWindowUsage.cs @@ -0,0 +1,8 @@ +namespace Acme; + +public sealed class DateWindowIndex +{ + private readonly Dictionary _values = new(); + + public bool TryGet(DateWindow window, out string? value) => _values.TryGetValue(window, out value); +} diff --git a/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParser.cs b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParser.cs new file mode 100644 index 0000000..3753186 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParser.cs @@ -0,0 +1,29 @@ +using System.Buffers.Text; +using System.Globalization; +using System.Text; + +namespace Acme.Text; + +public static class DecimalParser +{ + public static bool ParseLegacy(ReadOnlySpan utf8, out decimal value, out int consumed) + { + var text = Encoding.UTF8.GetString(utf8); + var success = decimal.TryParse(text, NumberStyles.Number, CultureInfo.InvariantCulture, out value); + consumed = success ? utf8.Length : 0; + return success; + } + + public static bool TryParseSpan(ReadOnlySpan utf8, out decimal value, out int consumed) + { + if (Utf8Parser.TryParse(utf8, out value, out var parsedBytes, 'G') && parsedBytes == utf8.Length) + { + consumed = parsedBytes; + return true; + } + + value = default; + consumed = 0; + return false; + } +} diff --git a/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParserTests.cs b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParserTests.cs new file mode 100644 index 0000000..d2df239 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParserTests.cs @@ -0,0 +1,7 @@ +namespace Acme.Text.Tests; + +public class DecimalParserTests +{ + // Import values include typical integers/decimals, decimal boundaries, and rejected content. + private static readonly string[] Cases = ["42", "1234.56", "-79228162514264337593543950335", "not-a-number", "123.45 trailing"]; +} diff --git a/skills/dotnet-benchmark/evals/files/decimal-parser/ImportPipeline.cs b/skills/dotnet-benchmark/evals/files/decimal-parser/ImportPipeline.cs new file mode 100644 index 0000000..f5901d0 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/decimal-parser/ImportPipeline.cs @@ -0,0 +1,18 @@ +namespace Acme.Text; + +public sealed class ImportPipeline +{ + public decimal Sum(IEnumerable> values) + { + decimal total = 0; + foreach (var value in values) + { + if (DecimalParser.ParseLegacy(value.Span, out var parsed, out _)) + { + total += parsed; + } + } + + return total; + } +} diff --git a/skills/dotnet-benchmark/evals/files/report-loader/ReportImportJob.cs b/skills/dotnet-benchmark/evals/files/report-loader/ReportImportJob.cs new file mode 100644 index 0000000..dbcf660 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/report-loader/ReportImportJob.cs @@ -0,0 +1,6 @@ +namespace Acme.Reporting; + +public sealed class ReportImportJob(ReportLoader loader) +{ + public IReadOnlyList Import(IEnumerable paths) => paths.Select(loader.Load).ToArray(); +} diff --git a/skills/dotnet-benchmark/evals/files/report-loader/ReportLoader.cs b/skills/dotnet-benchmark/evals/files/report-loader/ReportLoader.cs new file mode 100644 index 0000000..2f15f34 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/report-loader/ReportLoader.cs @@ -0,0 +1,14 @@ +using System.Text.Json; + +namespace Acme.Reporting; + +public sealed class ReportLoader +{ + public Report Load(string path) => Parse(File.ReadAllBytes(path)); + + public Report Parse(ReadOnlySpan utf8) => JsonSerializer.Deserialize(utf8)!; +} + +public sealed record Report(string Name, IReadOnlyList Rows); + +public sealed record ReportRow(string Key, decimal Value, string? Comment); diff --git a/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcher.cs b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcher.cs new file mode 100644 index 0000000..b4c8d1d --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcher.cs @@ -0,0 +1,16 @@ +using System.Text.RegularExpressions; + +namespace Acme.Routing; + +public sealed class RouteMatcher +{ + public bool IsMatch(string pattern, string path) + { + var regexPattern = "^" + Regex.Escape(pattern).Replace("\\*", ".*").Replace("\\?", ".") + "$"; + return Regex.IsMatch(path, regexPattern, RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + } + + public string Normalize(string path) => path.Trim().Trim('/').ToLowerInvariant(); + + public override string ToString() => nameof(RouteMatcher); +} diff --git a/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcherTests.cs b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcherTests.cs new file mode 100644 index 0000000..470d5f1 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcherTests.cs @@ -0,0 +1,13 @@ +namespace Acme.Routing.Tests; + +public class RouteMatcherTests +{ + // Representative cases: exact hit, wildcard hit, single-character wildcard, and miss. + private static readonly (string Pattern, string Path, bool Expected)[] Cases = + [ + ("api/health", "api/health", true), + ("api/*/orders/*", "api/v2/orders/2026-000042", true), + ("tenant-?/reports/*.json", "tenant-a/reports/monthly-2026-06.json", true), + ("assets/*.css", "assets/app.js", false) + ]; +} diff --git a/skills/dotnet-benchmark/evals/files/route-matcher/RouteTable.cs b/skills/dotnet-benchmark/evals/files/route-matcher/RouteTable.cs new file mode 100644 index 0000000..d89185e --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/route-matcher/RouteTable.cs @@ -0,0 +1,17 @@ +namespace Acme.Routing; + +public sealed class RouteTable(RouteMatcher matcher, IReadOnlyList patterns) +{ + public string? Find(string path) + { + foreach (var pattern in patterns) + { + if (matcher.IsMatch(pattern, path)) + { + return pattern; + } + } + + return null; + } +} From 821075e4c1241092a51744b1fccc5c31abbe13cf Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 14:52:01 +0200 Subject: [PATCH 11/38] =?UTF-8?q?=F0=9F=94=A7=20update=20benchmark=20valid?= =?UTF-8?q?ation=20tooling?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Refactor check-benchmark-requirements.ps1 to validate discovery workflow inputs and candidate matrix evidence. Add validate-skill.ps1 for per-skill validation. Update repo-level validate-skill-templates.ps1 to support new template asset structure. --- scripts/validate-skill-templates.ps1 | 30 ++ .../scripts/check-benchmark-requirements.ps1 | 269 ++++++++++-------- .../scripts/validate-skill.ps1 | 183 ++++++++++++ 3 files changed, 367 insertions(+), 115 deletions(-) create mode 100644 skills/dotnet-benchmark/scripts/validate-skill.ps1 diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index ec32751..0004070 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -832,6 +832,36 @@ Add-ValidationResult -Results $results -Name 'Benchmark runner wildcard is prese Assert-Match -Name 'benchmark-program.cs' -Content $program -Pattern 'namespace\s+\{BENCHMARK_RUNNER_NAMESPACE\};' } +Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence-backed candidates and preserves honest comparison semantics' -Action { + $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/SKILL.md' -GitRef $Ref + $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/FORMS.md' -GitRef $Ref + $candidateSelection = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/candidate-selection.md' -GitRef $Ref + $experimentDesign = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/experiment-design.md' -GitRef $Ref + $comparison = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/comparison-benchmark.cs' -GitRef $Ref + $operation = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/operation-benchmark.cs' -GitRef $Ref + $evals = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/evals/evals.json' -GitRef $Ref + + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'A microbenchmark measures a suspected cost under a defined workload; it does not prove that the type is an application bottleneck.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Read `references/candidate-selection.md`' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Read `references/experiment-design.md`' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--list flat' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--job dry' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'Auto-discover the highest-value performance questions (Recommended)' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '### candidate_plan_confirmation' + Assert-Contains -Name 'candidate-selection.md' -Content $candidateSelection -Needle '## Candidate matrix' + Assert-Contains -Name 'candidate-selection.md' -Content $candidateSelection -Needle '## Profiling-first gate' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Correctness oracle' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Do not compare unrelated operations.' + Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle '{EQUIVALENCE_CHECK}' + Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle 'Baseline = true' + Assert-Contains -Name 'operation-benchmark.cs' -Content $operation -Needle 'Do not add Baseline = true merely to produce a ratio column.' + Assert-NotContains -Name 'operation-benchmark.cs measured method' -Content ($operation -replace '// Do not add Baseline = true merely to produce a ratio column\.', '') -Needle 'Baseline = true' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'RouteMatcher.IsMatch' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'cannot prove whether file I/O or JSON parsing dominates' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'ThreadingDiagnoser' +} + Add-ValidationResult -Results $results -Name 'Strong-name skill matches FORMS summary flow and 1024-bit default' -Action { $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-strong-name-signing/SKILL.md' -GitRef $Ref Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'compute the defaults silently, and present a single summary for confirmation' diff --git a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 index 5021567..b214fda 100644 --- a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 +++ b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 @@ -1,115 +1,154 @@ -#requires -Version 5.1 -<# -.SYNOPSIS - Detects the current state of a repository's BenchmarkDotNet harness for the dotnet-benchmark skill. - -.DESCRIPTION - Emits a single JSON object describing what already exists so the skill adds only what is missing. - Read-only: it inspects files and the dotnet CLI but changes nothing. - -.PARAMETER RepoRoot - Repository root to inspect. Defaults to the current directory. - -.EXAMPLE - powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot C:\src\myrepo -#> -[CmdletBinding()] -param( - [string]$RepoRoot = (Get-Location).Path -) - -$ErrorActionPreference = 'Stop' - -function Test-CommandExists { - param([string]$Name) - return [bool](Get-Command $Name -ErrorAction SilentlyContinue) -} - -$RepoRoot = (Resolve-Path -LiteralPath $RepoRoot).Path - -# --- .NET SDK --------------------------------------------------------------- -$sdkAvailable = $false -$sdkVersion = $null -if (Test-CommandExists 'dotnet') { - try { - $sdkVersion = (& dotnet --version 2>$null | Select-Object -First 1) - $sdkAvailable = -not [string]::IsNullOrWhiteSpace($sdkVersion) - } catch { - $sdkAvailable = $false - } -} - -# --- Solution files --------------------------------------------------------- -$rootFiles = @(Get-ChildItem -LiteralPath $RepoRoot -File -ErrorAction SilentlyContinue) -$slnx = @($rootFiles | Where-Object { $_.Extension -ieq '.slnx' } | Select-Object -ExpandProperty Name -Unique) -$sln = @($rootFiles | Where-Object { $_.Extension -ieq '.sln' } | Select-Object -ExpandProperty Name -Unique) -$solutionFormat = if ($slnx.Count -gt 0) { 'slnx' } elseif ($sln.Count -gt 0) { 'sln' } else { 'none' } -$solutionFiles = @($slnx + $sln) - -# --- Central Package Management -------------------------------------------- -$packagesProps = Join-Path $RepoRoot 'Directory.Packages.props' -$cpm = Test-Path -LiteralPath $packagesProps -$declaredPackages = @() -if ($cpm) { - $packagesText = [System.IO.File]::ReadAllText($packagesProps) - foreach ($id in @('BenchmarkDotNet', 'BenchmarkDotNet.Diagnostics.Windows', 'Codebelt.Extensions.BenchmarkDotNet.Console')) { - if ($packagesText -match [regex]::Escape("Include=`"$id`"")) { $declaredPackages += $id } - } -} - -# --- Root Directory.Build.props conventions -------------------------------- -$buildProps = Join-Path $RepoRoot 'Directory.Build.props' -$centralizesBenchmarkConventions = $false -if (Test-Path -LiteralPath $buildProps) { - $buildText = [System.IO.File]::ReadAllText($buildProps) - $centralizesBenchmarkConventions = ($buildText -match 'IsBenchmarkProject') -and ($buildText -match 'IsToolingProject') -} - -# --- Existing tuning benchmark projects ------------------------------------ -$tuningDir = Join-Path $RepoRoot 'tuning' -$benchmarkProjects = @() -if (Test-Path -LiteralPath $tuningDir) { - $benchmarkProjects = @( - Get-ChildItem -LiteralPath $tuningDir -Recurse -Filter *.Benchmarks.csproj -File -ErrorAction SilentlyContinue | - ForEach-Object { $_.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } - ) -} - -# --- Existing tooling runner host ------------------------------------------ -$toolingDir = Join-Path $RepoRoot 'tooling' -$runner = $null -if (Test-Path -LiteralPath $toolingDir) { - $runnerCsproj = Get-ChildItem -LiteralPath $toolingDir -Recurse -Filter *.csproj -File -ErrorAction SilentlyContinue | - Where-Object { - $text = [System.IO.File]::ReadAllText($_.FullName) - $text -match 'Codebelt\.Extensions\.BenchmarkDotNet\.Console' - } | Select-Object -First 1 - if ($runnerCsproj) { - $runner = [ordered]@{ - name = $runnerCsproj.Directory.Name - path = $runnerCsproj.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' - referencesConsole = $true - } - } -} - -# --- reports/ --------------------------------------------------------------- -$reportsExists = Test-Path -LiteralPath (Join-Path $RepoRoot 'reports') - -$harnessReady = ($runner -ne $null) -and ($benchmarkProjects.Count -gt 0) - -$result = [ordered]@{ - repoRoot = $RepoRoot - sdk = [ordered]@{ available = $sdkAvailable; version = $sdkVersion } - solutionFormat = $solutionFormat - solutionFiles = $solutionFiles - centralPackageManagement = $cpm - declaredBenchmarkPackages = $declaredPackages - centralizesBenchmarkConventions = $centralizesBenchmarkConventions - benchmarkProjects = $benchmarkProjects - runner = $runner - reportsFolderExists = $reportsExists - harnessReady = $harnessReady -} - -$result | ConvertTo-Json -Depth 6 +#requires -Version 5.1 +<# +.SYNOPSIS + Detects the current state of a repository's BenchmarkDotNet harness for the dotnet-benchmark skill. + +.DESCRIPTION + Emits a single JSON object describing what already exists so the skill adds only what is missing. + Read-only: it inspects files and the dotnet CLI but changes nothing. + +.PARAMETER RepoRoot + Repository root to inspect. Defaults to the current directory. + +.PARAMETER SkipSdkCheck + Skips invoking dotnet --version. Intended for deterministic detector tests; normal skill runs should not use it. + +.EXAMPLE + powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot C:\src\myrepo +#> +[CmdletBinding()] +param( + [string]$RepoRoot = (Get-Location).Path, + [switch]$SkipSdkCheck +) + +$ErrorActionPreference = 'Stop' + +function Test-CommandExists { + param([string]$Name) + return [bool](Get-Command $Name -ErrorAction SilentlyContinue) +} + +function Get-DotNetSdkVersion { + param([int]$TimeoutMilliseconds = 10000) + + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = (Get-Command 'dotnet' -ErrorAction Stop).Source + $startInfo.Arguments = '--version' + $startInfo.UseShellExecute = $false + $startInfo.CreateNoWindow = $true + $startInfo.RedirectStandardOutput = $true + $startInfo.RedirectStandardError = $true + + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $startInfo + try { + if (-not $process.Start()) { + return [ordered]@{ status = 'start-failed'; version = $null } + } + if (-not $process.WaitForExit($TimeoutMilliseconds)) { + $process.Kill() + return [ordered]@{ status = 'timed-out'; version = $null } + } + if ($process.ExitCode -ne 0) { + return [ordered]@{ status = 'failed'; version = $null } + } + $version = ($process.StandardOutput.ReadToEnd() -split '\r?\n' | Select-Object -First 1).Trim() + return [ordered]@{ status = 'available'; version = $version } + } finally { + $process.Dispose() + } +} + +$RepoRoot = (Resolve-Path -LiteralPath $RepoRoot).Path + +# --- .NET SDK --------------------------------------------------------------- +$sdkAvailable = $false +$sdkVersion = $null +$sdkStatus = if ($SkipSdkCheck) { 'skipped' } else { 'not-found' } +if (-not $SkipSdkCheck -and (Test-CommandExists 'dotnet')) { + try { + $sdkProbe = Get-DotNetSdkVersion + $sdkVersion = $sdkProbe.version + $sdkStatus = $sdkProbe.status + $sdkAvailable = -not [string]::IsNullOrWhiteSpace($sdkVersion) + } catch { + $sdkAvailable = $false + $sdkStatus = 'failed' + } +} + +# --- Solution files --------------------------------------------------------- +$rootFiles = @(Get-ChildItem -LiteralPath $RepoRoot -File -ErrorAction SilentlyContinue) +$slnx = @($rootFiles | Where-Object { $_.Extension -ieq '.slnx' } | Select-Object -ExpandProperty Name -Unique) +$sln = @($rootFiles | Where-Object { $_.Extension -ieq '.sln' } | Select-Object -ExpandProperty Name -Unique) +$solutionFormat = if ($slnx.Count -gt 0) { 'slnx' } elseif ($sln.Count -gt 0) { 'sln' } else { 'none' } +$solutionFiles = @($slnx + $sln) + +# --- Central Package Management -------------------------------------------- +$packagesProps = Join-Path $RepoRoot 'Directory.Packages.props' +$cpm = Test-Path -LiteralPath $packagesProps +$declaredPackages = @() +if ($cpm) { + $packagesText = [System.IO.File]::ReadAllText($packagesProps) + foreach ($id in @('BenchmarkDotNet', 'BenchmarkDotNet.Diagnostics.Windows', 'Codebelt.Extensions.BenchmarkDotNet.Console')) { + if ($packagesText -match [regex]::Escape("Include=`"$id`"")) { $declaredPackages += $id } + } +} + +# --- Root Directory.Build.props conventions -------------------------------- +$buildProps = Join-Path $RepoRoot 'Directory.Build.props' +$centralizesBenchmarkConventions = $false +if (Test-Path -LiteralPath $buildProps) { + $buildText = [System.IO.File]::ReadAllText($buildProps) + $centralizesBenchmarkConventions = ($buildText -match 'IsBenchmarkProject') -and ($buildText -match 'IsToolingProject') +} + +# --- Existing tuning benchmark projects ------------------------------------ +$tuningDir = Join-Path $RepoRoot 'tuning' +$benchmarkProjects = @() +if (Test-Path -LiteralPath $tuningDir) { + $benchmarkProjects = @( + Get-ChildItem -LiteralPath $tuningDir -Recurse -Filter *.Benchmarks.csproj -File -ErrorAction SilentlyContinue | + ForEach-Object { $_.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } + ) +} + +# --- Existing tooling runner host ------------------------------------------ +$toolingDir = Join-Path $RepoRoot 'tooling' +$runner = $null +if (Test-Path -LiteralPath $toolingDir) { + $runnerCsproj = Get-ChildItem -LiteralPath $toolingDir -Recurse -Filter *.csproj -File -ErrorAction SilentlyContinue | + Where-Object { + $text = [System.IO.File]::ReadAllText($_.FullName) + $text -match 'Codebelt\.Extensions\.BenchmarkDotNet\.Console' + } | Select-Object -First 1 + if ($runnerCsproj) { + $runner = [ordered]@{ + name = $runnerCsproj.Directory.Name + path = $runnerCsproj.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + referencesConsole = $true + } + } +} + +# --- reports/ --------------------------------------------------------------- +$reportsExists = Test-Path -LiteralPath (Join-Path $RepoRoot 'reports') + +$harnessReady = ($runner -ne $null) -and ($benchmarkProjects.Count -gt 0) + +$result = [ordered]@{ + repoRoot = $RepoRoot + sdk = [ordered]@{ available = $sdkAvailable; version = $sdkVersion; status = $sdkStatus } + solutionFormat = $solutionFormat + solutionFiles = $solutionFiles + centralPackageManagement = $cpm + declaredBenchmarkPackages = $declaredPackages + centralizesBenchmarkConventions = $centralizesBenchmarkConventions + benchmarkProjects = $benchmarkProjects + runner = $runner + reportsFolderExists = $reportsExists + harnessReady = $harnessReady +} + +$result | ConvertTo-Json -Depth 6 diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 new file mode 100644 index 0000000..8093797 --- /dev/null +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -0,0 +1,183 @@ +#requires -Version 5.1 +<# +.SYNOPSIS + Deterministically validates the dotnet-benchmark skill package and its read-only harness detector. + +.PARAMETER SkillRoot + Root of the dotnet-benchmark skill. Defaults to the parent of this script's directory. +#> +[CmdletBinding()] +param( + [string]$SkillRoot = (Split-Path -Parent $PSScriptRoot) +) + +$ErrorActionPreference = 'Stop' +$failures = [System.Collections.Generic.List[string]]::new() + +function Add-Failure { + param([string]$Message) + $failures.Add($Message) +} + +function Assert-Contains { + param([string]$Name, [string]$Content, [string]$Needle) + if (-not $Content.Contains($Needle)) { + Add-Failure "$Name is missing required content: $Needle" + } +} + +function Assert-File { + param([string]$RelativePath) + $path = Join-Path $SkillRoot $RelativePath + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Add-Failure "Missing required file: $RelativePath" + } +} + +$SkillRoot = (Resolve-Path -LiteralPath $SkillRoot).Path +Write-Verbose "Validating skill root: $SkillRoot" + +$requiredFiles = @( + 'SKILL.md', + 'FORMS.md', + 'assets/benchmark.csproj', + 'assets/benchmark-program.cs', + 'assets/benchmark-runner.csproj', + 'assets/comparison-benchmark.cs', + 'assets/operation-benchmark.cs', + 'references/benchmarkdotnet-essentials.md', + 'references/candidate-selection.md', + 'references/codebelt-conventions.md', + 'references/experiment-design.md', + 'references/onboarding.md', + 'scripts/check-benchmark-requirements.ps1', + 'evals/evals.json' +) + +foreach ($file in $requiredFiles) { + Assert-File $file +} +Write-Verbose 'Required file checks completed.' + +if ($failures.Count -eq 0) { + $skillPath = Join-Path $SkillRoot 'SKILL.md' + $skill = [System.IO.File]::ReadAllText($skillPath) + $skillLines = [System.IO.File]::ReadAllLines($skillPath) + $lineCount = $skillLines.Count + if ($lineCount -gt 500) { + Add-Failure "SKILL.md must stay at or below 500 lines; found $lineCount" + } + + $frontmatterEnd = [Array]::IndexOf($skillLines, '---', 1) + $nameLine = $skillLines | Select-Object -First $frontmatterEnd | Where-Object { $_ -match '^name:\s*' } | Select-Object -First 1 + $descriptionStart = [Array]::IndexOf($skillLines, 'description: >') + if ($skillLines[0] -ne '---' -or $frontmatterEnd -lt 1 -or $nameLine -ne 'name: dotnet-benchmark' -or $descriptionStart -lt 1 -or $descriptionStart -ge $frontmatterEnd) { + Add-Failure 'SKILL.md frontmatter is missing the expected name and folded description' + } else { + $description = [string]::Join(' ', @($skillLines[($descriptionStart + 1)..($frontmatterEnd - 1)] | ForEach-Object { $_.Trim() } | Where-Object { $_ })) + if ($description.Length -gt 1024) { + Add-Failure "SKILL.md description exceeds 1024 characters; found $($description.Length)" + } + } + Write-Verbose 'Frontmatter checks completed.' + + Assert-Contains 'SKILL.md' $skill 'Read `references/candidate-selection.md`' + Assert-Contains 'SKILL.md' $skill 'Read `references/experiment-design.md`' + Assert-Contains 'SKILL.md' $skill 'Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation.' + Assert-Contains 'SKILL.md' $skill '--list flat' + Assert-Contains 'SKILL.md' $skill '--job dry' + Assert-Contains 'SKILL.md' $skill 'Never report performance numbers from a build, discovery listing, dry run, or unexecuted benchmark.' + + $comparison = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/comparison-benchmark.cs')) + $operation = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/operation-benchmark.cs')) + foreach ($required in @('[MemoryDiagnoser]', '[GlobalSetup]', '[Params(', 'Baseline = true', '{EQUIVALENCE_CHECK}')) { + Assert-Contains 'assets/comparison-benchmark.cs' $comparison $required + } + if (($comparison.Split(@('Baseline = true'), [System.StringSplitOptions]::None).Count - 1) -ne 1) { + Add-Failure 'assets/comparison-benchmark.cs must contain exactly one Baseline = true marker' + } + foreach ($required in @('[MemoryDiagnoser]', '[GlobalSetup]', '[Params(', '{SUT_CALL}')) { + Assert-Contains 'assets/operation-benchmark.cs' $operation $required + } + if ($operation -match '\[Benchmark\([^\]]*Baseline\s*=\s*true') { + Add-Failure 'assets/operation-benchmark.cs must not fabricate a baseline' + } + + $runner = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/benchmark-program.cs')) + Assert-Contains 'assets/benchmark-program.cs' $runner 'return c{RUNTIME_JOBS};' + Assert-Contains 'assets/benchmark-program.cs' $runner '{RUNTIME_USINGS}' + + try { + $evals = Get-Content -LiteralPath (Join-Path $SkillRoot 'evals/evals.json') -Raw | ConvertFrom-Json + if ($evals.skill_name -ne 'dotnet-benchmark') { + Add-Failure 'evals/evals.json skill_name must be dotnet-benchmark' + } + if ($evals.evals.Count -lt 5) { + Add-Failure 'evals/evals.json must include at least five diverse evals' + } + $ids = @($evals.evals | ForEach-Object { $_.id }) + if (($ids | Sort-Object -Unique).Count -ne $ids.Count) { + Add-Failure 'evals/evals.json contains duplicate eval IDs' + } + foreach ($eval in $evals.evals) { + if ([string]::IsNullOrWhiteSpace($eval.prompt) -or [string]::IsNullOrWhiteSpace($eval.expected_output) -or $eval.expectations.Count -lt 1) { + Add-Failure "Eval $($eval.id) must include a prompt, expected_output, and expectations" + } + foreach ($fixture in @($eval.files) | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) { + $fixturePath = Join-Path $SkillRoot $fixture + if (-not (Test-Path -LiteralPath $fixturePath -PathType Leaf)) { + Add-Failure "Eval $($eval.id) references missing fixture: $fixture" + } + } + } + } catch { + Add-Failure "evals/evals.json is invalid: $($_.Exception.Message)" + } + Write-Verbose 'Template and eval checks completed.' +} + +$tempRoot = [System.IO.Path]::GetFullPath([System.IO.Path]::GetTempPath()) +$fixtureRoot = Join-Path $tempRoot ("dotnet-benchmark-validator-" + [guid]::NewGuid().ToString('N')) +Write-Verbose "Creating detector fixture: $fixtureRoot" +try { + New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'tuning/Acme.Core.Benchmarks') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'tooling/bdn-runner') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Acme.sln'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Directory.Packages.props'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Directory.Build.props'), 'falsefalse') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/bdn-runner.csproj'), '') + + $detectorPath = Join-Path $SkillRoot 'scripts/check-benchmark-requirements.ps1' + if (Test-Path -LiteralPath $detectorPath) { + try { + $detected = & powershell -NoProfile -ExecutionPolicy Bypass -File $detectorPath -RepoRoot $fixtureRoot -SkipSdkCheck | ConvertFrom-Json + if ($detected.solutionFormat -ne 'sln' -or -not $detected.centralPackageManagement -or -not $detected.centralizesBenchmarkConventions) { + Add-Failure 'Harness detector did not recognize the fixture solution, CPM, and centralized conventions' + } + if ($detected.sdk.status -ne 'skipped') { + Add-Failure 'Harness detector did not report the intentional skipped SDK probe distinctly' + } + if ($detected.benchmarkProjects.Count -ne 1 -or $detected.runner.name -ne 'bdn-runner' -or -not $detected.harnessReady) { + Add-Failure 'Harness detector did not recognize the existing benchmark project and runner' + } + } catch { + Add-Failure "Harness detector failed on the deterministic fixture: $($_.Exception.Message)" + } + } +} finally { + $resolvedFixture = [System.IO.Path]::GetFullPath($fixtureRoot) + if ($resolvedFixture.StartsWith($tempRoot, [System.StringComparison]::OrdinalIgnoreCase) -and (Test-Path -LiteralPath $resolvedFixture)) { + Remove-Item -LiteralPath $resolvedFixture -Recurse -Force + } +} +Write-Verbose 'Harness detector fixture checks completed.' + +if ($failures.Count -gt 0) { + foreach ($failure in $failures) { + Write-Host "[FAIL] $failure" -ForegroundColor Red + } + exit 1 +} + +Write-Host '[PASS] dotnet-benchmark skill structure, templates, eval fixtures, and harness detector are valid.' -ForegroundColor Green From f64d8692f005be62a25032a67b3515a785c796e6 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 14:52:11 +0200 Subject: [PATCH 12/38] =?UTF-8?q?=F0=9F=92=AC=20update=20README=20with=20d?= =?UTF-8?q?otnet-benchmark=20discovery=20workflow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update skill listing and onboarding guidance to reflect discovery-focused refactor: evidence-driven candidate selection, cost-signal analysis, and measurement fitness over tier-based templates. --- README.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 87146db..30dcddd 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | -| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Set up and author BenchmarkDotNet performance tests for a specific .NET type following codebelt conventions, using `Codebelt.Extensions.BenchmarkDotNet` and its `.Console` runner. Detects the existing harness (`.slnx`/`.sln`, central package management, `tuning/` benchmark projects, and the `tooling/` runner host — reusing an existing name like `benchmark-runner` or `bdn-runner`) and onboards only what is missing, in place, whatever the repo layout. Resolves benchmark package versions from NuGet, resolves the target type's namespace and public surface, then picks a complexity-appropriate strategy — member scenarios for simple value types, `[Params]` + `[GlobalSetup]` sweeps for size- or variant-sensitive types — placing the `*Benchmark` class in the SUT's own namespace via a `RootNamespace` override. Always uses `[MemoryDiagnoser]`, wires the project into the detected solution, verifies the Release build, and hands off the run command by default (runs are slow) while supporting BenchmarkDotNet jobs that measure older and newer runtimes such as `net48`, `net8.0`, `net9.0`, and `net10.0`. | +| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, validates correctness outside the timed path, routes specialized diagnosers for allocation/contention/exceptions/JIT questions, and performs Release build, discovery listing, and dry execution before any explicit full run. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, keeps the benchmark class in the SUT namespace, and supports opt-in cross-runtime jobs. | ### Copyable Install Commands @@ -598,18 +598,22 @@ API documentation rots the moment code changes. A new public type ships without ### Why dotnet-benchmark? -Setting up a benchmark "properly" usually means copying a `tuning/` project from another repo, wiring a runner host, remembering which BenchmarkDotNet packages you need, and then deciding — every single time — how to structure the benchmark for the type in front of you. Most people skip it, and the performance question goes unanswered. +Setting up a benchmark "properly" is only half the problem. A benchmark can compile and still answer the wrong question: public members get measured because they are visible, unrelated operations share a meaningless baseline, random inputs miss real branches, setup leaks into the timed path, or a disk/service bottleneck is disguised as a microbenchmark. The result looks scientific but gives an engineer little trustworthy optimization evidence. -**dotnet-benchmark** removes that friction for both greenfield and existing repos. It inspects what you already have and adds only the missing pieces, then authors a benchmark that fits the type instead of a one-size-fits-all template. +**dotnet-benchmark** combines codebelt harness conventions with an evidence-driven performance investigation. It inspects the type, callers, tests, existing benchmarks, and available profiles; ranks the operations most likely to matter; selects a small set of representative experiments; and explicitly rejects misleading measurements. When no profile exists, it labels the result as source-informed exploration rather than claiming to have found an application bottleneck. - **Works on existing repos** — detects your solution format, package-management style, and any runner you already have (`benchmark-runner`, `bdn-runner`, …) and reuses it instead of forcing a new layout - **Codebelt convention by default** — `tuning/` benchmark projects, a single `tooling/` runner host, and `reports/` output, mirroring `codebeltnet/cuemon` and `codebeltnet/xunit` -- **Right-sized strategy** — simple value types get member-scenario benchmarks; size- or variant-sensitive types get `[Params]` sweeps with deterministic `[GlobalSetup]` payloads and a baseline comparison +- **Evidence-backed candidate selection** — ranks operations from call-site frequency, input scaling, allocations, contention, optimization leverage, and measurement fitness instead of treating public-member coverage as thoroughness +- **Honest experiment shapes** — creates equivalent current-versus-candidate comparisons, baseline-free single-operation characterization, or profiling/macrobenchmark guidance; unrelated construction, formatting, equality, and hashing never receive misleading ratios +- **Representative workloads** — derives typical, boundary, scaling, hit/miss, valid/invalid, and other adverse-but-real cases from repository evidence, using coupled scenario sources instead of accidental parameter Cartesian products +- **Correctness before speed** — validates equivalent outputs and state transitions for every case outside the measured path before a full performance run +- **Specialized investigations** — handles mutation, async, contention, cold start, and exception paths explicitly, routing `ThreadingDiagnoser`, `ExceptionDiagnoser`, disassembly, or EventPipe only when each answers the stated question - **Namespace-correct** — the `*Benchmark` class lives in the same namespace as the code it measures, via a `RootNamespace` override, so type discovery and reports stay clean - **Allocations always measured** — `[MemoryDiagnoser]` is on by default - **Multi-runtime aware** — the runner host runs on .NET 9/10, but its BenchmarkDotNet jobs can compare `net48`, `net8.0`, `net9.0`, and `net10.0` - **Latest stable packages** — `BenchmarkDotNet`, `BenchmarkDotNet.Diagnostics.Windows`, and `Codebelt.Extensions.BenchmarkDotNet.Console` versions are resolved from NuGet, not hardcoded -- **Safe hand-off** — verifies the Release build and gives you the run command; it won't kick off a slow benchmark run unless you ask +- **Layered validation** — verifies the Release build, lists discovered cases, and dry-executes lifecycle and correctness wiring; it never turns that smoke check into a performance claim or launches the full machine-sensitive run unless asked ## Repository structure From 11a11fc181ece638e3d7af2b7b12fa080f6ac211 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 14:59:06 +0200 Subject: [PATCH 13/38] =?UTF-8?q?=F0=9F=94=96=20update=20changelog=20for?= =?UTF-8?q?=20v0.8.0=20release=20with=20evidence-driven=20workflow?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CHANGELOG.md | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f64cf3b..caed95b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,21 +4,20 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.8.0] - 2026-07-17 -## [0.8.0] - 2026-07-16 - -This is a minor release introducing the `dotnet-benchmark` skill, a comprehensive benchmarking solution for .NET types using BenchmarkDotNet. The skill scaffolds benchmark projects following codebelt engineering conventions, provides parameter collection and complexity-aware test strategies, includes benchmark runner infrastructure, and supplies essential BenchmarkDotNet and codebelt-specific guidance. +This is a minor release introducing evidence-driven discovery and performance-experiment design in `dotnet-benchmark`. The skill prioritizes identifying high-value benchmark targets through source inspection and profiling evidence, provides a structured discovery workflow before measurement, and includes new template assets and reference documentation for measurement-focused benchmarking. ### Added -- `dotnet-benchmark` skill with workflow guidance for authoring and running BenchmarkDotNet performance tests for specific .NET types, supporting complexity-appropriate strategies (simple, parameterized, fixture-based), -- `FORMS.md` for `dotnet-benchmark` with structured parameter collection for target type, benchmark family, and complexity level, -- Benchmark project templates and runner infrastructure including `benchmark.csproj`, `benchmark-runner.csproj`, parameterized benchmark templates, and benchmark program entry points, -- `check-benchmark-requirements.ps1` script validating BenchmarkDotNet installation and NuGet feed accessibility before running benchmarks, -- Detailed reference documentation covering BenchmarkDotNet essentials (result interpretation, memory allocations, statistical confidence), codebelt conventions (namespace alignment, methodology rigor, result storage), and onboarding workflow for new benchmark authors, -- Eval coverage for `dotnet-benchmark` including target-type inspection, benchmark strategy selection, template application, and runner validation, -- README updates with `dotnet-benchmark` installation snippet, capability showcase, and "Why dotnet-benchmark?" section highlighting performance-test authoring for throughput and allocation measurement. +- Evidence-driven discovery workflow in `dotnet-benchmark` prioritizing candidate selection, cost-signal analysis, and measurement fitness over tier-based template enumeration; includes new workflow steps for intent resolution, repository inspection, performance evidence gathering, and experiment-plan presentation, +- New template assets `operation-benchmark.cs` and `comparison-benchmark.cs` replacing tier-based starters, providing refined structural guidance for single-operation and comparative-implementation benchmarks with clearer baseline configuration, +- `candidate-selection.md` reference documenting evidence ladders, call-site inspection, profiling integration, and candidate-ranking heuristics to drive the discovery phase, +- `experiment-design.md` reference detailing performance questions, workload selection, correctness verification, and measurement fitness to ensure benchmarks answer the right questions, +- Refactored `FORMS.md` for `dotnet-benchmark` aligned with the new discovery-focused workflow, reducing parameter collection friction by deferring implementation-tier choice to workflow inspection, +- Enhanced eval coverage for `dotnet-benchmark` with test cases validating candidate discovery, evidence gathering, cost-signal analysis, implementation-comparison patterns, and runtime-selection decisions; includes fixture code supporting five representative benchmark scenarios, +- Enhanced `check-benchmark-requirements.ps1` and new `validate-skill.ps1` tooling supporting discovery workflow validation and template-asset consistency checking, +- Updated README with discovery-focused `dotnet-benchmark` description and rationale emphasizing evidence-backed benchmarking over generic performance testing. ## [0.7.5] - 2026-07-15 @@ -467,7 +466,6 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. -[Unreleased]: https://github.com/codebeltnet/agentic/compare/v0.7.5...HEAD [0.8.0]: https://github.com/codebeltnet/agentic/compare/v0.7.5...v0.8.0 [0.7.5]: https://github.com/codebeltnet/agentic/compare/v0.7.4...v0.7.5 [0.7.4]: https://github.com/codebeltnet/agentic/compare/v0.7.3...v0.7.4 From 9ece8f0676f58ff76072c0a162551f6c6777884b Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 17:16:51 +0200 Subject: [PATCH 14/38] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20dotnet-benchmark=20s?= =?UTF-8?q?kill=20instructions=20with=20yolo=20mode=20and=20report-aware?= =?UTF-8?q?=20runner=20preflight?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add workflow sections for yolo mode, explicitly requiring user intent for full performance runs. Add report-aware runner preflight logic to detect when SkipBenchmarksWithReports plus existing reports intentionally filter a benchmark type. Update descriptions and validation checkpoints to support these features safely. --- skills/dotnet-benchmark/FORMS.md | 25 +++++-- skills/dotnet-benchmark/SKILL.md | 36 ++++++++-- .../references/benchmarkdotnet-essentials.md | 15 ++-- .../references/codebelt-conventions.md | 4 +- .../dotnet-benchmark/references/onboarding.md | 7 +- .../references/runner-preflight.md | 69 +++++++++++++++++++ 6 files changed, 140 insertions(+), 16 deletions(-) create mode 100644 skills/dotnet-benchmark/references/runner-preflight.md diff --git a/skills/dotnet-benchmark/FORMS.md b/skills/dotnet-benchmark/FORMS.md index 5dcfce7..5678e5a 100644 --- a/skills/dotnet-benchmark/FORMS.md +++ b/skills/dotnet-benchmark/FORMS.md @@ -2,6 +2,21 @@ `dotnet-benchmark` derives most decisions from the repository, the target type, and the user's request. Ask only unresolved fields, one at a time. Prefer the host's native structured input controls when available. If the host does not provide them, use the deterministic plain-text fallback below without changing field order or choices. +## Yolo mode override + +Yolo mode is request syntax, not a field to ask about. Activate it only when the current request explicitly says `yolo`, `yolo mode`, `auto-proceed`, or clearly instructs the agent to use defaults without routine confirmation. + +While yolo mode is active: + +- skip `performance_intent` and use auto-discovery unless the request already specifies another intent; +- skip `workload_context` when repository evidence supports a defensible workload; state the chosen assumption and continue; +- show the candidate plan as a progress update, then skip `candidate_plan_confirmation` and treat its recommended default as accepted; +- skip `execution_depth` and use build, list, and dry execution; +- ask only when missing information creates material correctness/design risk with no safe default; +- require a separate explicit human instruction to start a full performance run. `yolo` by itself never selects the full-run option, and an agent recommendation never supplies that authority. + +The override lasts for the current benchmark task only and does not waive correctness checks, repository policy, external-side-effect restrictions, or blockers. + ## Fields ### sut_type @@ -23,14 +38,14 @@ - Establish a regression benchmark for a known workload - **default:** Auto-discover the highest-value performance questions (Recommended) - **required:** true -- **description:** Infer and skip this field when the request already states the operation, comparison, or regression goal. Auto-discovery ranks candidates from implementation, usage, tests, and any available profiling evidence; it does not claim a measured application bottleneck without profile or telemetry data. +- **description:** Infer and skip this field when the request already states the operation, comparison, or regression goal. In yolo mode, skip it and use auto-discovery unless another intent is explicit. Auto-discovery ranks candidates from implementation, usage, tests, and any available profiling evidence; it does not claim a measured application bottleneck without profile or telemetry data. ### workload_context - **type:** text - **prompt:** "I could not infer a representative workload confidently. What inputs, sizes, frequency, and operating conditions matter in production?" - **required:** false -- **description:** Show this field only when tests, call sites, documentation, or supplied profiling evidence do not establish a representative workload and choosing one would materially affect correctness. Offer the strongest repo-derived workload as a selectable recommended choice when one exists, plus the option to enter a custom value. +- **description:** Show this field only when tests, call sites, documentation, or supplied profiling evidence do not establish a representative workload and choosing one would materially affect correctness. Offer the strongest repo-derived workload as a selectable recommended choice when one exists, plus the option to enter a custom value. In yolo mode, state and use the strongest defensible repo-derived workload; ask only when no safe choice exists. ### candidate_plan_confirmation @@ -41,7 +56,7 @@ - Adjust the selected operations or inputs - **default:** Use the proposed plan (Recommended) - **required:** true -- **description:** Present the evidence-backed experiment plan immediately before this field. Include selected and rejected candidates, comparable baseline/candidate pairs, parameter cases, correctness oracle, lifecycle risks, and whether the plan is exploratory or profile-backed. +- **description:** Present the evidence-backed experiment plan immediately before this field. Include selected and rejected candidates, comparable baseline/candidate pairs, parameter cases, correctness oracle, lifecycle risks, and whether the plan is exploratory or profile-backed. In yolo mode, present it as a progress update and skip this field by accepting the recommended choice. ### execution_depth @@ -52,11 +67,11 @@ - Run the full performance benchmark after validation - **default:** Build, list, and dry-execute the benchmark (Recommended) - **required:** true -- **description:** A dry execution validates discovery and lifecycle but produces no trustworthy performance conclusion. A full run can be slow and machine-sensitive. Treat an explicit request such as "run it" or "measure it now" as selecting the full-run option. +- **description:** A dry execution validates discovery and lifecycle but produces no trustworthy performance conclusion. A full run can be slow and machine-sensitive. In yolo mode, skip this field and select the recommended build/list/dry option. Only an explicit human instruction such as "run it fully now" or "measure it now" selects the full-run option; yolo, plan acceptance, or an agent recommendation is not enough. ## Presentation rules -1. Ask one field at a time and wait for the answer before presenting the next unresolved field. +1. Detect the yolo mode override before presenting any field. When active, skip the routine fields described above and continue autonomously; otherwise ask one field at a time and wait for the answer before presenting the next unresolved field. 2. Skip fields already answered by the conversation or reliable repository evidence. Do not ask the user to choose BenchmarkDotNet attributes, a "simple/complex" tier, or extra runtimes unless those choices are part of the user's goal. 3. When native controls are unavailable, start with `Field: `, repeat the prompt verbatim, show numbered choices in declared order, and accept either a number or exact choice text. 4. For a text field with a repo-derived suggestion, show `1. Use "" (Recommended)` and `2. Enter a custom value`. Do not fabricate a suggestion when the evidence is weak. diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index 364bec4..8a8eb4e 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-benchmark description: > - Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects implementation code, call sites, tests, existing benchmarks, and available profiling evidence; ranks high-value operations instead of benchmarking every public member; selects representative workloads; rejects misleading microbenchmarks; creates or reuses the tuning/ and tooling/ harness; validates correctness and benchmark discovery; and keeps full performance runs explicit. + Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects source and usage evidence, ranks high-value operations instead of every public member, selects representative workloads, rejects misleading microbenchmarks, creates or reuses the tuning/ and tooling/ harness, preflights existing-report skips, validates correctness and discovery, and keeps full runs human-initiated. When the user says yolo, it auto-accepts routine defaults and proceeds through safe validation without confirmation churn. --- # Evidence-Driven .NET Benchmarking @@ -18,6 +18,8 @@ Create the smallest benchmark suite that can answer the most valuable performanc - Keep correctness outside the timed path but inside the verification workflow. Equivalent implementations must be checked on every benchmark case before a full run. - Keep external I/O, network latency, database latency, sleeps, logging, and random data generation out of measured microbenchmark methods. Recommend profiling, a macrobenchmark, or a load test when those effects are the actual question. - Always distinguish code that was built, smoke-executed, or fully measured. Never report performance numbers from a build, discovery listing, dry run, or unexecuted benchmark. +- Start a full performance run only after an explicit human instruction to run it now. Never infer that authority from yolo mode, defaults, plan acceptance, an agent recommendation, or the existence of a runnable benchmark. +- When a Codebelt runner appears to list or execute nothing, inspect `SkipBenchmarksWithReports` and matching `reports/tuning/` artifacts before changing benchmark code. An existing report is an expected skip condition, not a reason to rename a lean class or add diagnosers. ## Workflow @@ -25,7 +27,22 @@ Create the smallest benchmark suite that can answer the most valuable performanc Read `FORMS.md` and use its one-field-at-a-time interaction only for information that is not already clear. A named type plus a request such as “find the likely bottlenecks” is sufficient to start inspection. Do not make the user choose an implementation tier or BenchmarkDotNet attributes. -Default to automatic candidate discovery, the runner's existing/default runtime, and build plus discovery and dry execution validation. Ask about extra runtimes only when cross-runtime comparison is part of the request. Require explicit user intent before a full benchmark run because it can be long and machine-sensitive. +Default to automatic candidate discovery, the runner's existing/default runtime, and build plus discovery and dry execution validation. Ask about extra runtimes only when cross-runtime comparison is part of the request. Require an explicit human instruction before a full benchmark run because it can be long and machine-sensitive. + +#### Yolo mode + +Activate yolo mode only when the current request explicitly says `yolo`, `yolo mode`, `auto-proceed`, or an equally clear instruction to use defaults without routine confirmation. The mode applies to the current benchmark task only; do not persist it into later requests. + +In yolo mode: + +- infer automatic candidate discovery unless the user already supplied a more specific performance intent; +- use the strongest defensible workload and repository conventions from source, call sites, tests, profiles, and existing benchmarks, recording material assumptions instead of asking the user to approve a 99%-certain default; +- present the compact experiment plan as a progress update and continue immediately without asking `candidate_plan_confirmation`; +- select build, discovery listing, and dry execution as the default validation depth without asking `execution_depth`; +- stop and ask only when a missing fact creates material correctness/design risk and no safe repo-derived choice exists, or when new authority is required; +- preserve all benchmark-quality, repository, and safety gates. Yolo is a convenience mode, not permission to invent workloads, ignore blockers, run unrelated operations, commit, push, or make external changes. + +Yolo never authorizes a full performance run. Start the full benchmark only when the human explicitly asks to run or measure it fully now; otherwise stop after build/list/dry validation. An agent must not promote its own recommendation into that authority. ### 2. Inspect the repository and harness @@ -37,6 +54,8 @@ powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requ Also inspect applicable `AGENTS.md`, solution/project files, `Directory.Build.props`, `Directory.Packages.props`, existing `tuning/` and `tooling/` projects, and nearby benchmark styles. Reuse an existing runner and benchmark project when they fit. Read `references/onboarding.md` only when the detector finds missing or partial harness infrastructure. +When investigating a benchmark that builds but is not listed or executed, read `references/runner-preflight.md` before inspecting or rewriting the benchmark class. Rerun the detector with `-BenchmarkType ` and inspect the reported runner program, `SkipBenchmarksWithReports` setting, slim/runtime jobs, `reports/tuning/` files, and `wouldSkipRequestedBenchmark`. If a matching existing report explains the skip, preserve the runner and benchmark unchanged, report the matching file, and stop diagnostic escalation. Do not add disassembly, rename the class, disable report skipping, or churn through tools to evade the filter. + The .NET SDK is the only hard harness prerequisite. If detector status is `not-found`, report that blocker instead of generating unverified project files. If the probe is `timed-out`, `start-failed`, or `failed`, report the probe failure distinctly and verify the SDK through a safe direct check before concluding that it is absent. ### 3. Resolve the type and gather performance evidence @@ -65,7 +84,7 @@ Before authoring code, present a compact plan with: - candidates deliberately rejected and why; - whether the result will be exploratory or grounded in profile/telemetry evidence. -Follow the confirmation flow in `FORMS.md`. If the user already named exact members, inputs, and implementations, confirm only material corrections or risks rather than repeating settled choices. +Follow the confirmation flow in `FORMS.md` in interactive mode. In yolo mode, post the plan as a concise progress update and proceed without confirmation. If the user already named exact members, inputs, and implementations, confirm only material corrections or risks rather than repeating settled choices. ### 6. Design the experiment @@ -99,6 +118,14 @@ First validate the benchmark's correctness through existing tests or a setup-tim dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj ``` +Before interpreting discovery or execution output, run the report-aware preflight for the exact class: + +```powershell +powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType +``` + +If `reports.wouldSkipRequestedBenchmark` is true, the runner is intentionally filtering the type because a prior report exists. Report that as the validation outcome; do not claim the list/dry run exercised the class and do not modify code to force it through. A fresh full run and any report archive/replacement require explicit human direction. + Verify runner discovery without measuring: ```powershell @@ -111,7 +138,7 @@ Unless execution is impossible or the user declines, run a dry execution smoke c dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* ``` -Run the full benchmark only when the user explicitly asks. Use an unplugged laptop, debugger, busy CI worker, VM, or power-throttled environment only if that environment is itself the target; otherwise warn that the results may not be stable or representative. +Run the full benchmark only when the human explicitly asks to start it. Use an unplugged laptop, debugger, busy CI worker, VM, or power-throttled environment only if that environment is itself the target; otherwise warn that the results may not be stable or representative. ### 10. Report the outcome @@ -127,6 +154,7 @@ Summarize the selected and rejected candidates, benchmark question, cases, corre - [ ] Equivalent implementations pass a correctness oracle for every case. - [ ] `[MemoryDiagnoser]` is present and every additional diagnoser has a stated purpose. - [ ] Harness changes preserve repository conventions and reuse existing projects/runner where possible. +- [ ] Runner preflight accounts for `SkipBenchmarksWithReports`, configured slim/runtime jobs, and any matching `reports/tuning/` artifact before benchmark-code diagnosis. - [ ] Release build, benchmark discovery, and dry execution succeed, or exact blockers are reported. - [ ] Full-run performance claims are made only from an actual full run in a described environment. - [ ] Generated files are UTF-8 without mojibake, and no unrelated files were changed. diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md index d96e1fc..23c0a47 100644 --- a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -52,11 +52,12 @@ Diagnosers may require separate runs and increase duration. The codebelt runner starts from `BenchmarkWorkspaceOptions.Slim`. Add runtime jobs only for an explicit cross-runtime question: ```csharp +var slimJob = BenchmarkWorkspaceOptions.Slim; return c - .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(ClrRuntime.Net48)) - .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core80)) - .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core90)) - .AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(CoreRuntime.Core10_0)); + .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core80)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); ``` | Target | Job runtime | @@ -70,6 +71,12 @@ Only add jobs that the SUT and benchmark toolchain can execute. Keep the runner- Let BenchmarkDotNet choose warmup, iteration, launch, and invocation counts unless the performance question requires cold start, monitoring, or another specific run strategy. Short/dry jobs validate or iterate quickly; they do not replace the default job for performance conclusions. +## Existing-report filtering + +Codebelt library runners normally set `SkipBenchmarksWithReports = true`. The console runner filters a loaded `*Benchmark` type when a matching report already exists under the artifacts tuning folder, normally `reports/tuning/`. Therefore a successful build followed by an empty list/dry/full invocation can be expected behavior. + +Run the bundled detector with `-BenchmarkType ` and inspect its runner/report fields before changing a benchmark or adding diagnostics. Read `runner-preflight.md` for the matching rule and the no-thrashing workflow. Never rename a class, add disassembly, or disable the option merely to evade an existing report. + ## Runner commands Build: diff --git a/skills/dotnet-benchmark/references/codebelt-conventions.md b/skills/dotnet-benchmark/references/codebelt-conventions.md index 3da25b0..5ef4bd6 100644 --- a/skills/dotnet-benchmark/references/codebelt-conventions.md +++ b/skills/dotnet-benchmark/references/codebelt-conventions.md @@ -45,7 +45,9 @@ Do not use network, disk, database, logging, or sleep calls inside a microbenchm ## Runner and reports -The runner calls `Codebelt.Extensions.BenchmarkDotNet.Console.BenchmarkProgram.Run`, reuses the repository's existing name such as `benchmark-runner` or `bdn-runner`, and writes artifacts under `reports/`. Default runtime configuration stays plain `return c;`; add runtime jobs only when cross-runtime comparison is explicitly requested. +The runner calls `Codebelt.Extensions.BenchmarkDotNet.Console.BenchmarkProgram.Run`, reuses the repository's existing name such as `benchmark-runner` or `bdn-runner`, and writes artifacts under `reports/`. Codebelt libraries normally keep `SkipBenchmarksWithReports = true`; a matching file under `reports/tuning/` intentionally filters that benchmark type on later invocations. Run the `runner-preflight.md` checks before diagnosing benchmark code. + +Default runtime configuration stays plain `return c;`. For explicit runtime jobs, assign `BenchmarkWorkspaceOptions.Slim` once to `slimJob`, then add `slimJob.WithRuntime(...)` for each supported TFM. The TFM list is the expected repository-specific difference; preserve the surrounding lean runner shape. ## Reference implementations diff --git a/skills/dotnet-benchmark/references/onboarding.md b/skills/dotnet-benchmark/references/onboarding.md index bf5e3a8..28344e8 100644 --- a/skills/dotnet-benchmark/references/onboarding.md +++ b/skills/dotnet-benchmark/references/onboarding.md @@ -75,8 +75,9 @@ scaffold and `codebeltnet/xunit`). Copy `assets/benchmark-runner.csproj` and |-------------|-------| | `{RUNNER_TARGET_FRAMEWORK}` | Highest supported non-preview executable TFM (`net10.0` or `net9.0`). | | `{RUNNER_NAMESPACE}` | Runner folder name converted to a valid C# identifier (`benchmark-runner` -> `benchmark_runner`). | -| `{RUNTIME_USINGS}` | Empty for **Runner default only**. Otherwise emit the `using BenchmarkDotNet.Environments;`, `using BenchmarkDotNet.Jobs;`, and `using Codebelt.Extensions.BenchmarkDotNet;` lines, each terminated with a newline, because the added runtime jobs need all three namespaces. | -| `{RUNTIME_JOBS}` | Empty for **Runner default only** so the method stays `return c;`. Otherwise emit newline-prefixed chained `.AddJob(BenchmarkWorkspaceOptions.Slim.WithRuntime(...))` calls, one per runtime to measure (see `benchmarkdotnet-essentials.md`). | +| `{RUNTIME_USINGS}` | Empty for **Runner default only**. Otherwise emit `using Codebelt.Extensions.BenchmarkDotNet;`, `using BenchmarkDotNet.Configs;`, `using BenchmarkDotNet.Environments;`, and `using BenchmarkDotNet.Jobs;`, each terminated with a newline. The slim job and runtime-specific `AddJob` chain need all four namespaces. | +| `{RUNTIME_SETUP}` | Empty for **Runner default only**. Otherwise emit the indented `var slimJob = BenchmarkWorkspaceOptions.Slim;` declaration used by every configured runtime job. | +| `{RUNTIME_JOBS}` | Empty for **Runner default only** so the method stays `return c;`. Otherwise emit newline-prefixed chained `.AddJob(slimJob.WithRuntime(...))` calls, one per runtime to measure (see `benchmarkdotnet-essentials.md`). | If the root `Directory.Build.props` does **not** mark tooling projects as executables, add `Exe` and `false` to the runner `PropertyGroup`. @@ -107,6 +108,8 @@ If the root `Directory.Build.props` does **not** mark tooling projects as execut Benchmark output is written under `reports/` by the Codebelt workspace. You do not need to pre-create it; the runner creates it on first run. Mention it so the user knows where results land. +The standard runner sets `SkipBenchmarksWithReports = true`. Existing matching files under `reports/tuning/` deliberately filter their benchmark type from later list/dry/full invocations. Read `runner-preflight.md` before treating a skipped type as a benchmark-code defect. + ## Guardrails - Preserve UTF-8 (no BOM unless the source had one) when writing generated files; watch for mojibake. diff --git a/skills/dotnet-benchmark/references/runner-preflight.md b/skills/dotnet-benchmark/references/runner-preflight.md new file mode 100644 index 0000000..bffb8d1 --- /dev/null +++ b/skills/dotnet-benchmark/references/runner-preflight.md @@ -0,0 +1,69 @@ +# Codebelt Runner Preflight + +Use this reference before diagnosing a benchmark that builds but is not listed or executed. Codebelt runners commonly enable `SkipBenchmarksWithReports`, so an existing report can make a healthy benchmark appear to do nothing. + +## Canonical runner shape + +The runtime list varies by repository TFMs; the surrounding setup should stay lean and recognizable: + +```csharp +using Codebelt.Extensions.BenchmarkDotNet; +using Codebelt.Extensions.BenchmarkDotNet.Console; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Environments; +using BenchmarkDotNet.Jobs; + +namespace benchmark_runner; + +public static class Program +{ + public static void Main(string[] args) + { + BenchmarkProgram.Run(args, o => + { + o.AllowDebugBuild = BenchmarkProgram.IsDebugBuild; + o.SkipBenchmarksWithReports = true; + o.ConfigureBenchmarkDotNet(c => + { + var slimJob = BenchmarkWorkspaceOptions.Slim; + return c + .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); + }); + }); + } +} +``` + +Default-runtime-only runners intentionally omit the runtime/job usings and `slimJob`, leaving `ConfigureBenchmarkDotNet` as `return c;`. + +## Why a benchmark can be skipped + +With `SkipBenchmarksWithReports = true`, the Codebelt console runner enumerates files under the configured BenchmarkDotNet artifacts path plus the tuning folder, normally `reports/tuning/`. For each report, it takes the filename portion before the first `-`, then the final dotted segment, and compares that value case-insensitively with loaded types whose names end in `Benchmark`. A matching report adds a filter that excludes that entire benchmark type. + +For example, `reports/tuning/Acme.Core.ParserBenchmark-report-github.md` causes `ParserBenchmark` to be filtered. This is expected idempotent runner behavior, not evidence that the benchmark class, filter, diagnoser, namespace, or method name is wrong. + +## Preflight sequence + +1. Build the benchmark project in Release. Stop on a compiler error; report it directly. +2. Run the detector with the intended benchmark type: + + ```powershell + powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType + ``` + +3. Inspect `runner.programPath`, `runner.skipBenchmarksWithReports`, `runner.usesSlimJob`, `runner.configuredRuntimes`, `reports.tuningPath`, `reports.matchingReportFiles`, and `reports.wouldSkipRequestedBenchmark`. +4. Verify that configured runtime jobs match the repository's supported TFMs. The TFM list is the normal variable; do not rewrite the runner merely to make it look different. +5. If `wouldSkipRequestedBenchmark` is true, explain that the existing report deliberately suppresses the type. Treat an empty list/dry/full invocation as accounted for and leave the benchmark class and runner unchanged. +6. Only investigate filters, discovery, class visibility, attributes, toolchains, or diagnosers when the report preflight does not explain the behavior. + +## Rerun boundary + +Do not delete, move, overwrite, or ignore reports automatically. Do not flip `SkipBenchmarksWithReports` to false or rename a benchmark to evade the filter. Those actions discard the runner's intentional idempotency and can create duplicate or misleading result history. + +If the human explicitly requests a fresh full run, report the matching files and ask them to choose the repository's accepted report-retention action, such as archive/replace. After that explicit choice, preserve the canonical runner setup and rerun the same benchmark type. Yolo mode never supplies the human authorization for the full run or report replacement. + +## Anti-thrashing rule + +An existing matching report is a terminal explanation for runner-level skipping. Do not respond by adding `DisassemblyDiagnoser`, changing benchmark attributes, expanding tool calls, renaming a slim class, or rewriting a focused benchmark. Diagnostics and class changes must answer a separate evidence-backed performance question, not work around report filtering. From e878af814a0aa95e73364ab4494fbfd400fe5aed Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 17:17:06 +0200 Subject: [PATCH 15/38] =?UTF-8?q?=E2=9C=85=20dotnet-benchmark:=20add=20eva?= =?UTF-8?q?l=20tests=20for=20yolo=20mode=20and=20report-skip=20diagnostics?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add eval test cases 8 and 9 with fixtures. Test 8 validates yolo mode with autonomous candidate selection and progress-update-only planning. Test 9 validates report-aware preflight: skill recognizes SkipBenchmarksWithReports plus matching reports/tuning/ artifacts and preserves benchmark code unchanged. --- skills/dotnet-benchmark/evals/evals.json | 38 +++++++++++++++++++ ...Acme.Core.ParserBenchmark-report-github.md | 3 ++ .../tooling/benchmark-runner/Program.cs | 27 +++++++++++++ .../benchmark-runner/benchmark-runner.csproj | 10 +++++ ...oreApp,Version=v10.0.AssemblyAttributes.cs | 4 ++ .../net10.0/benchmark-runner.AssemblyInfo.cs | 22 +++++++++++ .../benchmark-runner.AssemblyInfoInputs.cache | 1 + ....GeneratedMSBuildEditorConfig.editorconfig | 17 +++++++++ .../Acme.Core.Benchmarks.csproj | 8 ++++ .../Acme.Core.Benchmarks/ParserBenchmark.cs | 17 +++++++++ ...oreApp,Version=v10.0.AssemblyAttributes.cs | 4 ++ .../Acme.Core.Benchmarks.AssemblyInfo.cs | 22 +++++++++++ ...e.Core.Benchmarks.AssemblyInfoInputs.cache | 1 + ....GeneratedMSBuildEditorConfig.editorconfig | 17 +++++++++ 14 files changed, 191 insertions(+) create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache create mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json index 048f9ae..e79c35f 100644 --- a/skills/dotnet-benchmark/evals/evals.json +++ b/skills/dotnet-benchmark/evals/evals.json @@ -107,6 +107,44 @@ "Emits no raw runtime placeholders or unused BenchmarkDotNet.Environments, BenchmarkDotNet.Jobs, or Codebelt.Extensions.BenchmarkDotNet runtime-job using directives", "Does not treat the absence of extra jobs as permission to skip target-type and workload inspection" ] + }, + { + "id": 8, + "prompt": "YOLO mode: add the most useful benchmark for Acme.Routing.RouteMatcher from the attached source. Use the repo evidence and recommended defaults, proceed without routine confirmation, and do not run a full performance benchmark.", + "files": [ + "evals/files/route-matcher/RouteMatcher.cs", + "evals/files/route-matcher/RouteTable.cs", + "evals/files/route-matcher/RouteMatcherTests.cs" + ], + "expected_output": "The agent recognizes yolo mode, autonomously selects the evidence-backed RouteMatcher.IsMatch characterization, reports the experiment plan as a progress update rather than a question, authors the benchmark, and proceeds through build/list/dry validation or honestly reports a source-only harness blocker. It never treats yolo as authorization for a full run or unrelated side effects.", + "expectations": [ + "Recognizes yolo mode and does not ask for performance intent, workload, candidate-plan confirmation, or execution depth when repository evidence provides safe defaults", + "Uses RouteTable and tests to select RouteMatcher.IsMatch plus representative coupled hit/miss and pattern-complexity cases", + "States the experiment plan and material assumptions as information, then proceeds without asking the user to approve the recommended plan", + "Uses a MemoryDiagnoser single-operation characterization with no fabricated Baseline=true", + "Selects build, list, and dry validation by default, or supplies those commands while honestly reporting that the source-only fixture lacks a runnable harness", + "Does not run or claim a full performance benchmark without an explicit human instruction, and does not treat yolo as authorization for unrelated repository or remote operations" + ] + }, + { + "id": 9, + "prompt": "Acme.Core.ParserBenchmark builds, but the codebelt runner appears to do nothing when I list or run it. Diagnose the attached minimal repository. Keep the benchmark slim and do not start a full performance run.", + "files": [ + "evals/files/runner-skip/tooling/benchmark-runner/Program.cs", + "evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj", + "evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs", + "evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj", + "evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md" + ], + "expected_output": "The agent performs runner preflight before benchmark-code diagnosis, recognizes that SkipBenchmarksWithReports=true plus the matching reports/tuning file deliberately filters ParserBenchmark, verifies the canonical Slim runtime jobs, preserves the lean benchmark and its name, avoids disassembly/tool thrash, and does not remove reports, disable the option, or start a full run without explicit human direction.", + "expectations": [ + "Runs the detector with -BenchmarkType Acme.Core.ParserBenchmark against the canonical repo layout and reports runner.skipBenchmarksWithReports=true, Slim runtime jobs, the matching report, and reports.wouldSkipRequestedBenchmark=true", + "Identifies Acme.Core.ParserBenchmark-report-github.md as a case-insensitive type-name match that deliberately filters ParserBenchmark", + "Explains that an empty list or execution is expected runner behavior rather than evidence that ParserBenchmark is malformed", + "Preserves the lean ParserBenchmark class, method, attributes, and name without adding DisassemblyDiagnoser or unrelated benchmark cases", + "Does not disable SkipBenchmarksWithReports, rename the type, or delete/move/overwrite the report automatically", + "Does not start or claim a full performance run and requires explicit human direction for a fresh run plus report-retention choice" + ] } ] } diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md b/skills/dotnet-benchmark/evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md new file mode 100644 index 0000000..adf4dfd --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md @@ -0,0 +1,3 @@ +# Existing ParserBenchmark report + +This fixture represents an earlier completed run. diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs new file mode 100644 index 0000000..ae68fa3 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs @@ -0,0 +1,27 @@ +using Codebelt.Extensions.BenchmarkDotNet; +using Codebelt.Extensions.BenchmarkDotNet.Console; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Environments; +using BenchmarkDotNet.Jobs; + +namespace benchmark_runner; + +public static class Program +{ + public static void Main(string[] args) + { + BenchmarkProgram.Run(args, o => + { + o.AllowDebugBuild = BenchmarkProgram.IsDebugBuild; + o.SkipBenchmarksWithReports = true; + o.ConfigureBenchmarkDotNet(c => + { + var slimJob = BenchmarkWorkspaceOptions.Slim; + return c + .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); + }); + }); + } +} diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj new file mode 100644 index 0000000..55ac864 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj @@ -0,0 +1,10 @@ + + + Exe + net10.0 + + + + + + diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs new file mode 100644 index 0000000..d3d9ce2 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs @@ -0,0 +1,4 @@ +// +using System; +using System.Reflection; +[assembly: global::System.Runtime.Versioning.TargetFrameworkAttribute(".NETCoreApp,Version=v10.0", FrameworkDisplayName = ".NET 10.0")] diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs new file mode 100644 index 0000000..b51cba4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs @@ -0,0 +1,22 @@ +//------------------------------------------------------------------------------ +// +// This code was generated by a tool. +// +// Changes to this file may cause incorrect behavior and will be lost if +// the code is regenerated. +// +//------------------------------------------------------------------------------ + +using System; +using System.Reflection; + +[assembly: System.Reflection.AssemblyCompanyAttribute("benchmark-runner")] +[assembly: System.Reflection.AssemblyConfigurationAttribute("Debug")] +[assembly: System.Reflection.AssemblyFileVersionAttribute("1.0.0.0")] +[assembly: System.Reflection.AssemblyInformationalVersionAttribute("1.0.0+11a11fc181ece638e3d7af2b7b12fa080f6ac211")] +[assembly: System.Reflection.AssemblyProductAttribute("benchmark-runner")] +[assembly: System.Reflection.AssemblyTitleAttribute("benchmark-runner")] +[assembly: System.Reflection.AssemblyVersionAttribute("1.0.0.0")] + +// Generated by the MSBuild WriteCodeFragment class. + diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache new file mode 100644 index 0000000..7799c75 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache @@ -0,0 +1 @@ +d111d88d83b17a262229de83b0d790a39378a5aee7ab8a8e99f0b44928084759 diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig new file mode 100644 index 0000000..c21c2c5 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig @@ -0,0 +1,17 @@ +is_global = true +build_property.TargetFramework = net10.0 +build_property.TargetFrameworkIdentifier = .NETCoreApp +build_property.TargetFrameworkVersion = v10.0 +build_property.TargetPlatformMinVersion = +build_property.UsingMicrosoftNETSdkWeb = +build_property.ProjectTypeGuids = +build_property.InvariantGlobalization = +build_property.PlatformNeutralAssembly = +build_property.EnforceExtendedAnalyzerRules = +build_property._SupportedPlatformList = Linux,macOS,Windows +build_property.RootNamespace = benchmark-runner +build_property.ProjectDir = C:\Source\Github\codebeltnet\agentic\skills\dotnet-benchmark\evals\files\runner-skip\tooling\benchmark-runner\ +build_property.EnableComHosting = +build_property.EnableGeneratedComInterfaceComImportInterop = +build_property.EffectiveAnalysisLevelStyle = 10.0 +build_property.EnableCodeStyleSeverity = diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj new file mode 100644 index 0000000..d27d0c8 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj @@ -0,0 +1,8 @@ + + + net10.0 + + + + + diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs new file mode 100644 index 0000000..8fc3d15 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs @@ -0,0 +1,17 @@ +using BenchmarkDotNet.Attributes; + +namespace Acme.Core; + +[MemoryDiagnoser] +public class ParserBenchmark +{ + private readonly Parser _parser = new(); + + [Benchmark] + public int ParseTypical() => _parser.Parse("42"); +} + +public sealed class Parser +{ + public int Parse(string value) => int.Parse(value, System.Globalization.CultureInfo.InvariantCulture); +} diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs new file mode 100644 index 0000000..d3d9ce2 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs @@ -0,0 +1,4 @@ +// +using System; +using System.Reflection; +[assembly: global::System.Runtime.Versioning.TargetFrameworkAttribute(".NETCoreApp,Version=v10.0", FrameworkDisplayName = ".NET 10.0")] diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs new file mode 100644 index 0000000..df394fa --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs @@ -0,0 +1,22 @@ +//------------------------------------------------------------------------------ +// +// This code was generated by a tool. +// +// Changes to this file may cause incorrect behavior and will be lost if +// the code is regenerated. +// +//------------------------------------------------------------------------------ + +using System; +using System.Reflection; + +[assembly: System.Reflection.AssemblyCompanyAttribute("Acme.Core.Benchmarks")] +[assembly: System.Reflection.AssemblyConfigurationAttribute("Debug")] +[assembly: System.Reflection.AssemblyFileVersionAttribute("1.0.0.0")] +[assembly: System.Reflection.AssemblyInformationalVersionAttribute("1.0.0+11a11fc181ece638e3d7af2b7b12fa080f6ac211")] +[assembly: System.Reflection.AssemblyProductAttribute("Acme.Core.Benchmarks")] +[assembly: System.Reflection.AssemblyTitleAttribute("Acme.Core.Benchmarks")] +[assembly: System.Reflection.AssemblyVersionAttribute("1.0.0.0")] + +// Generated by the MSBuild WriteCodeFragment class. + diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache new file mode 100644 index 0000000..0cfaaf5 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache @@ -0,0 +1 @@ +11a06bc3b99aba7551bd28f407a82bc36b90dfa4f0f69207a50e1811892463e2 diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig new file mode 100644 index 0000000..bfd4cc0 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig @@ -0,0 +1,17 @@ +is_global = true +build_property.TargetFramework = net10.0 +build_property.TargetFrameworkIdentifier = .NETCoreApp +build_property.TargetFrameworkVersion = v10.0 +build_property.TargetPlatformMinVersion = +build_property.UsingMicrosoftNETSdkWeb = +build_property.ProjectTypeGuids = +build_property.InvariantGlobalization = +build_property.PlatformNeutralAssembly = +build_property.EnforceExtendedAnalyzerRules = +build_property._SupportedPlatformList = Linux,macOS,Windows +build_property.RootNamespace = Acme.Core.Benchmarks +build_property.ProjectDir = C:\Source\Github\codebeltnet\agentic\skills\dotnet-benchmark\evals\files\runner-skip\tuning\Acme.Core.Benchmarks\ +build_property.EnableComHosting = +build_property.EnableGeneratedComInterfaceComImportInterop = +build_property.EffectiveAnalysisLevelStyle = 10.0 +build_property.EnableCodeStyleSeverity = From a11f1c89ec1b08f99c0122868ec7805d29df935d Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 17:17:22 +0200 Subject: [PATCH 16/38] =?UTF-8?q?=F0=9F=94=A7=20dotnet-benchmark:=20update?= =?UTF-8?q?=20runtime=20scripts=20and=20template=20for=20report-aware=20pr?= =?UTF-8?q?eflight?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add SkipBenchmarksWithReports and reports/tuning/ detection to check-benchmark-requirements.ps1. Report runner program, slim/runtime jobs, and wouldSkipRequestedBenchmark status. Add [MemoryDiagnoser] to benchmark-program.cs template to align with skill documentation. Update validate-skill.ps1 to support the expanded detector output. --- .../assets/benchmark-program.cs | 1 + .../scripts/check-benchmark-requirements.ps1 | 58 +++++++++++++++++-- .../scripts/validate-skill.ps1 | 33 ++++++++++- 3 files changed, 87 insertions(+), 5 deletions(-) diff --git a/skills/dotnet-benchmark/assets/benchmark-program.cs b/skills/dotnet-benchmark/assets/benchmark-program.cs index e94ff1f..2a745ef 100644 --- a/skills/dotnet-benchmark/assets/benchmark-program.cs +++ b/skills/dotnet-benchmark/assets/benchmark-program.cs @@ -14,6 +14,7 @@ public static void Main(string[] args) o.SkipBenchmarksWithReports = true; o.ConfigureBenchmarkDotNet(c => { +{RUNTIME_SETUP} // If the user chose "Runner default only", leave the next line as `return c;`. // Otherwise append newline-prefixed chained `.AddJob(...)` calls to the return expression. return c{RUNTIME_JOBS}; diff --git a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 index b214fda..5f83a51 100644 --- a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 +++ b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 @@ -10,6 +10,9 @@ .PARAMETER RepoRoot Repository root to inspect. Defaults to the current directory. +.PARAMETER BenchmarkType + Optional namespace-qualified or simple benchmark type name used to detect matching reports that would suppress execution. + .PARAMETER SkipSdkCheck Skips invoking dotnet --version. Intended for deterministic detector tests; normal skill runs should not use it. @@ -19,6 +22,7 @@ [CmdletBinding()] param( [string]$RepoRoot = (Get-Location).Path, + [string]$BenchmarkType, [switch]$SkipSdkCheck ) @@ -124,16 +128,54 @@ if (Test-Path -LiteralPath $toolingDir) { $text -match 'Codebelt\.Extensions\.BenchmarkDotNet\.Console' } | Select-Object -First 1 if ($runnerCsproj) { + $programFile = Get-ChildItem -LiteralPath $runnerCsproj.Directory.FullName -Filter Program.cs -File -ErrorAction SilentlyContinue | Select-Object -First 1 + $programText = if ($programFile) { [System.IO.File]::ReadAllText($programFile.FullName) } else { '' } + $configuredRuntimes = @( + [regex]::Matches($programText, '(?:ClrRuntime|CoreRuntime)\.[A-Za-z0-9_]+') | + ForEach-Object { $_.Value } | + Select-Object -Unique + ) $runner = [ordered]@{ - name = $runnerCsproj.Directory.Name - path = $runnerCsproj.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' - referencesConsole = $true + name = $runnerCsproj.Directory.Name + path = $runnerCsproj.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + programPath = if ($programFile) { $programFile.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } else { $null } + referencesConsole = $true + skipBenchmarksWithReports = $programText -match 'SkipBenchmarksWithReports\s*=\s*true' + usesSlimJob = $programText -match 'BenchmarkWorkspaceOptions\.Slim' + configuredRuntimes = $configuredRuntimes } } } # --- reports/ --------------------------------------------------------------- -$reportsExists = Test-Path -LiteralPath (Join-Path $RepoRoot 'reports') +$reportsPath = Join-Path $RepoRoot 'reports' +$reportsTuningPath = Join-Path $reportsPath 'tuning' +$reportsExists = Test-Path -LiteralPath $reportsPath +$reportFiles = @() +if (Test-Path -LiteralPath $reportsTuningPath) { + $reportFiles = @( + Get-ChildItem -LiteralPath $reportsTuningPath -File -ErrorAction SilentlyContinue | + ForEach-Object { $_.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } + ) +} + +$matchingReportFiles = @() +if (-not [string]::IsNullOrWhiteSpace($BenchmarkType)) { + $benchmarkTypeName = ($BenchmarkType -split '\.')[-1] + $matchingReportFiles = @( + $reportFiles | Where-Object { + $filename = [System.IO.Path]::GetFileNameWithoutExtension($_) + $potentialTypeFullName = ($filename -split '-', 2)[0] + $potentialTypeName = ($potentialTypeFullName -split '\.')[-1] + $potentialTypeName.Equals($benchmarkTypeName, [System.StringComparison]::OrdinalIgnoreCase) + } + ) +} + +$wouldSkipRequestedBenchmark = ($runner -ne $null) -and + $runner.skipBenchmarksWithReports -and + (-not [string]::IsNullOrWhiteSpace($BenchmarkType)) -and + ($matchingReportFiles.Count -gt 0) $harnessReady = ($runner -ne $null) -and ($benchmarkProjects.Count -gt 0) @@ -148,6 +190,14 @@ $result = [ordered]@{ benchmarkProjects = $benchmarkProjects runner = $runner reportsFolderExists = $reportsExists + reports = [ordered]@{ + path = $reportsPath.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + tuningPath = $reportsTuningPath.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + files = $reportFiles + requestedBenchmarkType = $BenchmarkType + matchingReportFiles = $matchingReportFiles + wouldSkipRequestedBenchmark = $wouldSkipRequestedBenchmark + } harnessReady = $harnessReady } diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 index 8093797..e5c39fa 100644 --- a/skills/dotnet-benchmark/scripts/validate-skill.ps1 +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -50,6 +50,7 @@ $requiredFiles = @( 'references/codebelt-conventions.md', 'references/experiment-design.md', 'references/onboarding.md', + 'references/runner-preflight.md', 'scripts/check-benchmark-requirements.ps1', 'evals/evals.json' ) @@ -87,6 +88,17 @@ if ($failures.Count -eq 0) { Assert-Contains 'SKILL.md' $skill '--list flat' Assert-Contains 'SKILL.md' $skill '--job dry' Assert-Contains 'SKILL.md' $skill 'Never report performance numbers from a build, discovery listing, dry run, or unexecuted benchmark.' + Assert-Contains 'SKILL.md' $skill '#### Yolo mode' + Assert-Contains 'SKILL.md' $skill 'Start a full performance run only after an explicit human instruction to run it now.' + Assert-Contains 'SKILL.md' $skill 'Yolo never authorizes a full performance run.' + Assert-Contains 'SKILL.md' $skill 'read `references/runner-preflight.md`' + Assert-Contains 'SKILL.md' $skill '-BenchmarkType ' + Assert-Contains 'SKILL.md' $skill 'reports.wouldSkipRequestedBenchmark' + + $forms = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'FORMS.md')) + Assert-Contains 'FORMS.md' $forms '## Yolo mode override' + Assert-Contains 'FORMS.md' $forms 'skip `candidate_plan_confirmation`' + Assert-Contains 'FORMS.md' $forms 'explicit human instruction to start a full performance run' $comparison = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/comparison-benchmark.cs')) $operation = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/operation-benchmark.cs')) @@ -106,6 +118,13 @@ if ($failures.Count -eq 0) { $runner = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/benchmark-program.cs')) Assert-Contains 'assets/benchmark-program.cs' $runner 'return c{RUNTIME_JOBS};' Assert-Contains 'assets/benchmark-program.cs' $runner '{RUNTIME_USINGS}' + Assert-Contains 'assets/benchmark-program.cs' $runner '{RUNTIME_SETUP}' + Assert-Contains 'assets/benchmark-program.cs' $runner 'public static class Program' + + $runnerPreflight = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/runner-preflight.md')) + Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'SkipBenchmarksWithReports = true' + Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'reports/tuning/' + Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'Anti-thrashing rule' try { $evals = Get-Content -LiteralPath (Join-Path $SkillRoot 'evals/evals.json') -Raw | ConvertFrom-Json @@ -115,6 +134,9 @@ if ($failures.Count -eq 0) { if ($evals.evals.Count -lt 5) { Add-Failure 'evals/evals.json must include at least five diverse evals' } + if (-not ($evals.evals | Where-Object { $_.prompt -match '(?i)yolo' })) { + Add-Failure 'evals/evals.json must include a yolo-mode interaction eval' + } $ids = @($evals.evals | ForEach-Object { $_.id }) if (($ids | Sort-Object -Unique).Count -ne $ids.Count) { Add-Failure 'evals/evals.json contains duplicate eval IDs' @@ -142,16 +164,19 @@ Write-Verbose "Creating detector fixture: $fixtureRoot" try { New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'tuning/Acme.Core.Benchmarks') -Force | Out-Null New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'tooling/bdn-runner') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'reports/tuning') -Force | Out-Null [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Acme.sln'), '') [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Directory.Packages.props'), '') [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Directory.Build.props'), 'falsefalse') [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj'), '') [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/bdn-runner.csproj'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/Program.cs'), 'using Codebelt.Extensions.BenchmarkDotNet; using Codebelt.Extensions.BenchmarkDotNet.Console; using BenchmarkDotNet.Environments; public static class Program { public static void Main(string[] args) { BenchmarkProgram.Run(args, o => { o.SkipBenchmarksWithReports = true; o.ConfigureBenchmarkDotNet(c => { var slimJob = BenchmarkWorkspaceOptions.Slim; return c.AddJob(slimJob.WithRuntime(CoreRuntime.Core90)); }); }); } }') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'reports/tuning/Acme.Core.WidgetBenchmark-report-github.md'), '# existing report') $detectorPath = Join-Path $SkillRoot 'scripts/check-benchmark-requirements.ps1' if (Test-Path -LiteralPath $detectorPath) { try { - $detected = & powershell -NoProfile -ExecutionPolicy Bypass -File $detectorPath -RepoRoot $fixtureRoot -SkipSdkCheck | ConvertFrom-Json + $detected = & powershell -NoProfile -ExecutionPolicy Bypass -File $detectorPath -RepoRoot $fixtureRoot -BenchmarkType Acme.Core.WidgetBenchmark -SkipSdkCheck | ConvertFrom-Json if ($detected.solutionFormat -ne 'sln' -or -not $detected.centralPackageManagement -or -not $detected.centralizesBenchmarkConventions) { Add-Failure 'Harness detector did not recognize the fixture solution, CPM, and centralized conventions' } @@ -161,6 +186,12 @@ try { if ($detected.benchmarkProjects.Count -ne 1 -or $detected.runner.name -ne 'bdn-runner' -or -not $detected.harnessReady) { Add-Failure 'Harness detector did not recognize the existing benchmark project and runner' } + if (-not $detected.runner.skipBenchmarksWithReports -or -not $detected.runner.usesSlimJob -or @($detected.runner.configuredRuntimes) -notcontains 'CoreRuntime.Core90') { + Add-Failure 'Harness detector did not recognize report skipping, the Slim job, and configured runtime' + } + if (-not $detected.reports.wouldSkipRequestedBenchmark -or @($detected.reports.matchingReportFiles).Count -ne 1) { + Add-Failure 'Harness detector did not identify the matching report that suppresses the requested benchmark type' + } } catch { Add-Failure "Harness detector failed on the deterministic fixture: $($_.Exception.Message)" } From 16a4c0b595c6df22c9d270ee626a1005b9be86ae Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 17:17:32 +0200 Subject: [PATCH 17/38] =?UTF-8?q?=F0=9F=94=A7=20repo=20validator:=20add=20?= =?UTF-8?q?style=20checks=20for=20skill=20templates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhance validate-skill-templates.ps1 with additional style and structure checks for SKILL.md, FORMS.md, references/, and evals/ files. Ensures consistency across repo-managed skills and enforces conventions for new feature additions. --- scripts/validate-skill-templates.ps1 | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 0004070..6f8708c 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -832,11 +832,12 @@ Add-ValidationResult -Results $results -Name 'Benchmark runner wildcard is prese Assert-Match -Name 'benchmark-program.cs' -Content $program -Pattern 'namespace\s+\{BENCHMARK_RUNNER_NAMESPACE\};' } -Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence-backed candidates and preserves honest comparison semantics' -Action { +Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence-backed candidates, supports yolo mode, and preserves honest comparison semantics' -Action { $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/SKILL.md' -GitRef $Ref $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/FORMS.md' -GitRef $Ref $candidateSelection = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/candidate-selection.md' -GitRef $Ref $experimentDesign = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/experiment-design.md' -GitRef $Ref + $runnerPreflight = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/runner-preflight.md' -GitRef $Ref $comparison = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/comparison-benchmark.cs' -GitRef $Ref $operation = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/operation-benchmark.cs' -GitRef $Ref $evals = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/evals/evals.json' -GitRef $Ref @@ -847,12 +848,22 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence- Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Read `references/experiment-design.md`' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--list flat' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--job dry' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '#### Yolo mode' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Start a full performance run only after an explicit human instruction to run it now.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Yolo never authorizes a full performance run.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'read `references/runner-preflight.md`' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'reports.wouldSkipRequestedBenchmark' Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'Auto-discover the highest-value performance questions (Recommended)' Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '### candidate_plan_confirmation' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '## Yolo mode override' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'skip `candidate_plan_confirmation`' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'explicit human instruction to start a full performance run' Assert-Contains -Name 'candidate-selection.md' -Content $candidateSelection -Needle '## Candidate matrix' Assert-Contains -Name 'candidate-selection.md' -Content $candidateSelection -Needle '## Profiling-first gate' Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Correctness oracle' Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Do not compare unrelated operations.' + Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'SkipBenchmarksWithReports = true' + Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'Anti-thrashing rule' Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle '{EQUIVALENCE_CHECK}' Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle 'Baseline = true' Assert-Contains -Name 'operation-benchmark.cs' -Content $operation -Needle 'Do not add Baseline = true merely to produce a ratio column.' @@ -860,6 +871,8 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence- Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'RouteMatcher.IsMatch' Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'cannot prove whether file I/O or JSON parsing dominates' Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'ThreadingDiagnoser' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'YOLO mode:' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'Acme.Core.ParserBenchmark-report-github.md' } Add-ValidationResult -Results $results -Name 'Strong-name skill matches FORMS summary flow and 1024-bit default' -Action { From ea47c36a7aea146560fe52be25319ed5459abbd7 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 17:17:41 +0200 Subject: [PATCH 18/38] =?UTF-8?q?=F0=9F=92=AC=20README:=20highlight=20dotn?= =?UTF-8?q?et-benchmark=20yolo=20mode=20and=20report-aware=20preflight?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update dotnet-benchmark skill table entry with descriptions of new yolo mode capability and report-aware runner preflight. Add feature bullets about yolo mode without permission creep and report-aware runner preflight for intentional benchmark skips. --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 30dcddd..e96baa4 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | -| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, validates correctness outside the timed path, routes specialized diagnosers for allocation/contention/exceptions/JIT questions, and performs Release build, discovery listing, and dry execution before any explicit full run. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, keeps the benchmark class in the SUT namespace, and supports opt-in cross-runtime jobs. | +| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, validates correctness outside the timed path, routes specialized diagnosers for allocation/contention/exceptions/JIT questions, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | ### Copyable Install Commands @@ -614,6 +614,8 @@ Setting up a benchmark "properly" is only half the problem. A benchmark can comp - **Multi-runtime aware** — the runner host runs on .NET 9/10, but its BenchmarkDotNet jobs can compare `net48`, `net8.0`, `net9.0`, and `net10.0` - **Latest stable packages** — `BenchmarkDotNet`, `BenchmarkDotNet.Diagnostics.Windows`, and `Codebelt.Extensions.BenchmarkDotNet.Console` versions are resolved from NuGet, not hardcoded - **Layered validation** — verifies the Release build, lists discovered cases, and dry-executes lifecycle and correctness wiring; it never turns that smoke check into a performance claim or launches the full machine-sensitive run unless asked +- **Yolo mode without permission creep** — saying `yolo` auto-accepts evidence-backed defaults, skips routine plan/execution confirmations, and continues through build/list/dry validation; only an explicit human instruction can start a full benchmark run, and the mode never implies a commit, push, or unrelated external action +- **Report-aware runner preflight** — inspects the canonical `BenchmarkWorkspaceOptions.Slim` runtime jobs, `SkipBenchmarksWithReports`, and matching `reports/tuning/` artifacts before touching benchmark code, so an intentional existing-report skip is explained instead of triggering disassembly, renaming, or speculative rewrites ## Repository structure From a8cdf0edc19b32e86034c033cb0815e504fa0968 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 21:08:34 +0200 Subject: [PATCH 19/38] =?UTF-8?q?=F0=9F=94=A7=20dotnet-benchmark:=20correc?= =?UTF-8?q?t=20Program=20class=20declaration=20from=20static=20to=20non-st?= =?UTF-8?q?atic?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update benchmark-runner Program.cs fixture, runner-preflight.md reference, and validate-skill.ps1 assertions to use 'public class Program' instead of 'public static class Program'. Console app entry points should be non-static to align with .NET templates and best practices. --- .../files/runner-skip/tooling/benchmark-runner/Program.cs | 2 +- skills/dotnet-benchmark/references/runner-preflight.md | 2 +- skills/dotnet-benchmark/scripts/validate-skill.ps1 | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs index ae68fa3..71070b6 100644 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs @@ -6,7 +6,7 @@ namespace benchmark_runner; -public static class Program +public class Program { public static void Main(string[] args) { diff --git a/skills/dotnet-benchmark/references/runner-preflight.md b/skills/dotnet-benchmark/references/runner-preflight.md index bffb8d1..3c52279 100644 --- a/skills/dotnet-benchmark/references/runner-preflight.md +++ b/skills/dotnet-benchmark/references/runner-preflight.md @@ -15,7 +15,7 @@ using BenchmarkDotNet.Jobs; namespace benchmark_runner; -public static class Program +public class Program { public static void Main(string[] args) { diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 index e5c39fa..a5245f1 100644 --- a/skills/dotnet-benchmark/scripts/validate-skill.ps1 +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -119,7 +119,7 @@ if ($failures.Count -eq 0) { Assert-Contains 'assets/benchmark-program.cs' $runner 'return c{RUNTIME_JOBS};' Assert-Contains 'assets/benchmark-program.cs' $runner '{RUNTIME_USINGS}' Assert-Contains 'assets/benchmark-program.cs' $runner '{RUNTIME_SETUP}' - Assert-Contains 'assets/benchmark-program.cs' $runner 'public static class Program' + Assert-Contains 'assets/benchmark-program.cs' $runner 'public class Program' $runnerPreflight = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/runner-preflight.md')) Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'SkipBenchmarksWithReports = true' @@ -170,7 +170,7 @@ try { [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Directory.Build.props'), 'falsefalse') [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj'), '') [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/bdn-runner.csproj'), '') - [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/Program.cs'), 'using Codebelt.Extensions.BenchmarkDotNet; using Codebelt.Extensions.BenchmarkDotNet.Console; using BenchmarkDotNet.Environments; public static class Program { public static void Main(string[] args) { BenchmarkProgram.Run(args, o => { o.SkipBenchmarksWithReports = true; o.ConfigureBenchmarkDotNet(c => { var slimJob = BenchmarkWorkspaceOptions.Slim; return c.AddJob(slimJob.WithRuntime(CoreRuntime.Core90)); }); }); } }') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/Program.cs'), 'using Codebelt.Extensions.BenchmarkDotNet; using Codebelt.Extensions.BenchmarkDotNet.Console; using BenchmarkDotNet.Environments; public class Program { public static void Main(string[] args) { BenchmarkProgram.Run(args, o => { o.SkipBenchmarksWithReports = true; o.ConfigureBenchmarkDotNet(c => { var slimJob = BenchmarkWorkspaceOptions.Slim; return c.AddJob(slimJob.WithRuntime(CoreRuntime.Core90)); }); }); } }') [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'reports/tuning/Acme.Core.WidgetBenchmark-report-github.md'), '# existing report') $detectorPath = Join-Path $SkillRoot 'scripts/check-benchmark-requirements.ps1' From 35a29d5a43c5ef06b4eac674134ec6659f539d77 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 21:47:29 +0200 Subject: [PATCH 20/38] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20dotnet-benchmark=20s?= =?UTF-8?q?kill:=20benchmarking=20discipline=20updates=20and=20proportiona?= =?UTF-8?q?te-stop=20&=20selectivity-drift=20eval=20tests?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add tests 10 and 11 with fixtures. Test 10 validates benchmark design repair when selectivity drifts across parameters and some cases fail. Test 11 validates proportionate stopping: when measurement is complete and cost does not justify deeper investigation. Update SKILL.md with case studies and decision logic. Refine benchmarkdotnet-essentials.md and experiment-design.md references for rigor and practical guidance. --- skills/dotnet-benchmark/SKILL.md | 35 +++++++++++--- skills/dotnet-benchmark/evals/evals.json | 38 +++++++++++++++ .../proportionate-stop/TraitDiscovery.cs | 9 ++++ .../proportionate-stop/TraitDiscoveryTests.cs | 22 +++++++++ .../files/proportionate-stop/TraitFilter.cs | 11 +++++ .../TraitFilterBenchmark-summary.md | 9 ++++ .../LegacyAliasCountBenchmark-summary.md | 14 ++++++ .../LegacyAliasCountBenchmark.cs | 37 +++++++++++++++ .../selectivity-drift/LegacyAliasImport.cs | 12 +++++ .../selectivity-drift/LegacyAliasQuery.cs | 9 ++++ .../LegacyAliasQueryTests.cs | 21 +++++++++ .../references/benchmarkdotnet-essentials.md | 36 ++++++++++++-- .../references/experiment-design.md | 47 ++++++++++++++++++- 13 files changed, 288 insertions(+), 12 deletions(-) create mode 100644 skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscovery.cs create mode 100644 skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscoveryTests.cs create mode 100644 skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilter.cs create mode 100644 skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilterBenchmark-summary.md create mode 100644 skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md create mode 100644 skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs create mode 100644 skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasImport.cs create mode 100644 skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQuery.cs create mode 100644 skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQueryTests.cs diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index 8a8eb4e..8d8b167 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -15,7 +15,11 @@ Create the smallest benchmark suite that can answer the most valuable performanc - Rank candidates using evidence from the implementation, call sites, tests, documentation, existing benchmark results, and profiles. Never invent usage frequency, input distributions, or a competing implementation. - Compare only operations that produce equivalent observable work. Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation. - Use `Baseline = true` only when at least two benchmark methods form a meaningful comparison group. A single-operation scaling or regression benchmark needs no fabricated baseline. When a class has several comparison groups, assign categories and one baseline inside each category. -- Keep correctness outside the timed path but inside the verification workflow. Equivalent implementations must be checked on every benchmark case before a full run. +- Before build or dry validation, inspect benchmark attributes for internal coherence: `Baseline = true` only on equivalent observable work, `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]` only with meaningful `[BenchmarkCategory]` values, every baseline category has an equivalent peer, single-operation characterization benchmarks have no fabricated baseline, benchmark descriptions name the real measured terminal operation, and decorative grouping attributes are removed. +- Keep correctness outside the timed path but inside the verification workflow. Every parameter or scenario case needs an exact observable-result oracle for count, value, status, exception behavior, output contents, or mutation unless the domain explicitly defines approximation. Valid zero-match cases stay valid; if a zero result is unintended, fix the workload before measurement. +- When one parameter only scales size, payload, or another magnitude, keep selectivity, hit/miss ratio, valid/invalid ratio, branch distribution, collision rate, string shape/encoding, type mix, entropy, and cache state stable unless one of them is intentionally exposed as a named scenario or parameter. +- A benchmark is invalid for performance interpretation until the complete BenchmarkDotNet summary shows the full intended method/job/parameter matrix without `NA`, `Benchmarks with issues`, setup/cleanup exceptions, validation errors, failed jobs/runtimes, or missing combinations. Report the exact failing method, job, and parameter case, do not treat surviving rows as a completed benchmark, fix only benchmark-owned causes, rerun build/list/dry validation, and require fresh explicit human authority before any replacement full run. +- Treat deferred pipeline creation, terminal operations such as `Count()`, `Any()`, or `First()`, explicit full enumeration, and materialization through `ToArray()` or `ToList()` as different workloads. Inspect fast paths such as `List.Count` before describing a benchmark as enumeration or using it as the baseline for predicate traversal. - Keep external I/O, network latency, database latency, sleeps, logging, and random data generation out of measured microbenchmark methods. Recommend profiling, a macrobenchmark, or a load test when those effects are the actual question. - Always distinguish code that was built, smoke-executed, or fully measured. Never report performance numbers from a build, discovery listing, dry run, or unexecuted benchmark. - Start a full performance run only after an explicit human instruction to run it now. Never infer that authority from yolo mode, defaults, plan acceptance, an agent recommendation, or the existence of a runnable benchmark. @@ -96,6 +100,10 @@ Read `references/experiment-design.md` and `references/benchmarkdotnet-essential Do not copy an asset blindly. The assets are structural examples with placeholders; adapt namespaces, types, cases, lifecycle, return consumption, correctness checks, and attributes to the real API. +When a parameter is only size or payload, preserve the other workload characteristics unless the experiment names them explicitly. For predicate or filter benchmarks, state the intended selectivity, use deterministic data, prefer fixed-width or otherwise structurally stable inputs when digit length or formatting would change the branch mix, and verify the exact expected match count for every scenario. + +For LINQ and other deferred pipelines, decide whether the benchmark measures query or iterator creation, a terminal operation, explicit enumeration, or materialization. Encode that distinction in names, descriptions, baselines, and conclusions; `List.Count` and `Where(...).Count()` are not interchangeable evidence. + ### 7. Author the benchmark Place the class under `tuning/{SutProject}.Benchmarks/`. Name it for the performance question and end the class name with `Benchmark`. Keep it in the SUT namespace rather than adding `.Benchmarks`; the benchmark project `RootNamespace` supports this codebelt convention. @@ -112,7 +120,9 @@ If the detector found missing infrastructure, follow `references/onboarding.md`. ### 9. Validate in layers -First validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. Then build the benchmark project in Release: +Before the build, inspect the benchmark attributes for coherence and remove decorative configuration that no longer serves the question. + +First validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. The oracle should normally verify exact observable behavior rather than merely nonzero or approximate success. Then build the benchmark project in Release: ```powershell dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj @@ -132,29 +142,42 @@ Verify runner discovery without measuring: dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* ``` +Compare the discovered method, job, and parameter matrix with the intended design. Missing or surprise combinations are validation failures, not report quirks. + Unless execution is impossible or the user declines, run a dry execution smoke check and inspect all BenchmarkDotNet validation warnings. A dry run proves executable wiring and basic lifecycle, not performance: ```powershell dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* ``` +Inspect the complete BenchmarkDotNet summary after the dry run and before reading any numbers. If any intended case shows `NA`, appears under `Benchmarks with issues`, throws in setup or cleanup, triggers a validation error, fails a job or runtime, or is missing from the intended matrix, report the exact failing method, job, and parameter case. Do not interpret successful rows as the completed benchmark. Correct the benchmark design only when the cause is within the benchmark, then rerun build, discovery, and dry validation. + Run the full benchmark only when the human explicitly asks to start it. Use an unplugged laptop, debugger, busy CI worker, VM, or power-throttled environment only if that environment is itself the target; otherwise warn that the results may not be stable or representative. +After any full run, apply the same full-summary validity gate before interpreting the tables. If rerunning is necessary because of a benchmark-owned issue, preserve the human-authority rule and get explicit approval before starting another full measurement run. + ### 10. Report the outcome -Summarize the selected and rejected candidates, benchmark question, cases, correctness oracle, diagnosers, generated files, and validation commands/results. If a full run occurred, report environment, mean/median where relevant, error and standard deviation, ratios only within valid comparison groups, allocations/GC, warnings, and the workload-specific conclusion. Recommend an optimization only when measurements identify a meaningful opportunity and correctness remains protected. +Summarize the selected and rejected candidates, benchmark question, cases, correctness oracle, diagnosers, generated files, and validation commands/results. If dry or full execution exposed an invalid case, report that invalid experiment instead of a performance conclusion. + +If a full run occurred, report the environment, active job shape, mean/median where relevant, error and standard deviation, ratios only within valid comparison groups, allocations/GC, warnings, and the workload-specific conclusion. When the runner uses `BenchmarkWorkspaceOptions.Slim`, report that shortened job accurately and mention its warmup/iteration limits whenever they matter to interpretation, especially for runtime- or JIT-sensitive comparisons. + +After the first valid full result, answer three questions: is the result reproducible, is the absolute cost material for observed or plausible usage, and would a deeper diagnostic change a concrete engineering decision? Stop when the answer does not justify further work. Do not automatically escalate to repeated full reruns, tiered-PGO variants, disassembly, EventPipe or ETW tracing, alternative implementations, or runtime-source archaeology. Escalate only when the result is reproducible, material, the next diagnostic can distinguish concrete competing explanations, and the user requested it or it is necessary to answer the original decision. For low-microsecond test-support utilities with no credible production change, prefer a concise "no change justified" conclusion. ## Completion checklist - [ ] Candidate selection is supported by implementation, usage, tests, telemetry, or profiling evidence. - [ ] The suite answers one to three explicit performance questions and excludes low-value member coverage. - [ ] Baselines compare equivalent work; single operations and unrelated members have no misleading ratio. -- [ ] Cases represent realistic, boundary, scaling, and adverse paths without useless Cartesian products. +- [ ] Cases represent realistic, boundary, scaling, and adverse paths without useless Cartesian products, and size-only sweeps keep other workload invariants stable unless explicitly named. - [ ] Setup, mutation, async, concurrency, disposal, and result consumption are handled correctly. -- [ ] Equivalent implementations pass a correctness oracle for every case. +- [ ] Exact correctness oracles cover every parameter and scenario case, including valid zero-match boundaries. +- [ ] Baselines, categories, grouping attributes, and descriptions are internally coherent and non-decorative. - [ ] `[MemoryDiagnoser]` is present and every additional diagnoser has a stated purpose. - [ ] Harness changes preserve repository conventions and reuse existing projects/runner where possible. - [ ] Runner preflight accounts for `SkipBenchmarksWithReports`, configured slim/runtime jobs, and any matching `reports/tuning/` artifact before benchmark-code diagnosis. - [ ] Release build, benchmark discovery, and dry execution succeed, or exact blockers are reported. -- [ ] Full-run performance claims are made only from an actual full run in a described environment. +- [ ] The complete discovery/dry/full summary covers the full intended matrix with no `NA`, `Benchmarks with issues`, failed jobs, or missing cases before any performance interpretation. +- [ ] Full-run performance claims are made only from an actual full run in a described environment, and any Slim-job limitation or other warning is reported honestly. +- [ ] After the first valid full result, deeper diagnostics or reruns are justified by a reproducible, material, decision-changing question rather than sunk cost. - [ ] Generated files are UTF-8 without mojibake, and no unrelated files were changed. diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json index e79c35f..b42411f 100644 --- a/skills/dotnet-benchmark/evals/evals.json +++ b/skills/dotnet-benchmark/evals/evals.json @@ -145,6 +145,44 @@ "Does not disable SkipBenchmarksWithReports, rename the type, or delete/move/overwrite the report automatically", "Does not start or claim a full performance run and requires explicit human direction for a fresh run plus report-retention choice" ] + }, + { + "id": 10, + "prompt": "Review the attached Acme.Search.LegacyAliasQuery benchmark and summary. I tried to compare List.Count with a filtered Where(...).Count() across sizes 8, 256, and 4096, but the filtered Size=8 case reported NA and I still wrote up the larger rows as a win. Fix the benchmark design and tell me whether any performance conclusion is valid.", + "files": [ + "evals/files/selectivity-drift/LegacyAliasQuery.cs", + "evals/files/selectivity-drift/LegacyAliasImport.cs", + "evals/files/selectivity-drift/LegacyAliasQueryTests.cs", + "evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs", + "evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md" + ], + "expected_output": "The agent rejects the benchmark as invalid because the intended matrix is incomplete and the filtered Size=8 case fails in setup, explains that List.Count is an O(1) collection fast path rather than predicate enumeration, replaces the drifting workload with deterministic fixed-width or explicit selectivity scenarios, removes the unrelated baseline and decorative category grouping, and reruns build, discovery, and dry validation before allowing any future full run.", + "expectations": [ + "Detects the valid zero-match boundary or the summary's NA and Benchmarks with issues signals and refuses to treat the surviving rows as a completed benchmark", + "Reports the failing method, job, and parameter case exactly and makes no performance claim from the invalid run", + "Replaces the drifting generator with deterministic fixed-width or explicit selectivity scenarios so size changes do not silently change predicate selectivity, and verifies exact expected match counts for every scenario", + "Removes Baseline = true from the unrelated collection Count fast path comparison or otherwise separates it from filtered enumeration instead of using it as the baseline", + "Removes GroupBenchmarksBy(ByCategory) when categories are absent, or only uses meaningful categories with equivalent comparison groups", + "States that after fixing benchmark-owned issues it must rerun build, discovery, and dry validation, and that another full measurement run still requires explicit human approval" + ] + }, + { + "id": 11, + "prompt": "The attached Acme.Testing.TraitFilter helper only runs in test discovery. I already have the first valid benchmark summary, and the costs are low-microsecond with small fixed allocations. Tell me whether to go deeper and what benchmark, if any, is still justified.", + "files": [ + "evals/files/proportionate-stop/TraitFilter.cs", + "evals/files/proportionate-stop/TraitDiscovery.cs", + "evals/files/proportionate-stop/TraitDiscoveryTests.cs", + "evals/files/proportionate-stop/TraitFilterBenchmark-summary.md" + ], + "expected_output": "The agent keeps the suite to one focused characterization benchmark for the real filter operation, treats the attached first valid full result as enough to answer the current question, reports that no production change is justified for a low-microsecond test-support helper, declines automatic escalation to extra diagnostics or alternative implementations, and names the condition that would justify reopening the investigation.", + "expectations": [ + "Uses the test-discovery call site to keep the scope to one linear filter and materialization benchmark rather than inventing baselines or extra implementations", + "Reads the first valid full result proportionally and answers whether the effect is reproducible, material, and decision-changing before suggesting more work", + "Concludes that no production optimization change is justified for the supplied low-microsecond test-support utility", + "Does not automatically add repeated full reruns, tiered-PGO, disassembly, EventPipe or ETW tracing, runtime-source archaeology, or speculative alternative implementations", + "Identifies a concrete reopen condition such as promotion to a production hot path, materially larger inputs or frequencies, or a new regression with decision-changing cost" + ] } ] } diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscovery.cs b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscovery.cs new file mode 100644 index 0000000..e06375f --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscovery.cs @@ -0,0 +1,9 @@ +using System.Collections.Generic; + +namespace Acme.Testing; + +public sealed class TraitDiscovery +{ + public IReadOnlyList SelectIntegrationTraits(IReadOnlyList traits) => + TraitFilter.Matching(traits, "integration:"); +} diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscoveryTests.cs b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscoveryTests.cs new file mode 100644 index 0000000..536f0a4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscoveryTests.cs @@ -0,0 +1,22 @@ +using Xunit; + +namespace Acme.Testing; + +public class TraitDiscoveryTests +{ + [Fact] + public void SelectIntegrationTraits_FiltersDeterministically() + { + var traits = new[] + { + "integration:postgres", + "unit:formatter", + "integration:redis", + "unit:validator" + }; + + var selected = new TraitDiscovery().SelectIntegrationTraits(traits); + + Assert.Equal(new[] { "integration:postgres", "integration:redis" }, selected); + } +} diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilter.cs b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilter.cs new file mode 100644 index 0000000..85772fe --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilter.cs @@ -0,0 +1,11 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Acme.Testing; + +public static class TraitFilter +{ + public static string[] Matching(IReadOnlyList traits, string prefix) => + traits.Where(trait => trait.StartsWith(prefix, StringComparison.Ordinal)).ToArray(); +} diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilterBenchmark-summary.md b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilterBenchmark-summary.md new file mode 100644 index 0000000..5d90eb4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilterBenchmark-summary.md @@ -0,0 +1,9 @@ +# TraitFilterBenchmark summary excerpt + +| Method | Scenario | Mean | Error | StdDev | Allocated | +|---|---|---:|---:|---:|---:| +| Matching | Small (16 traits, 25% hits) | 0.84 us | 0.02 us | 0.02 us | 352 B | +| Matching | Typical (64 traits, 25% hits) | 1.47 us | 0.04 us | 0.04 us | 768 B | +| Matching | Large (256 traits, 25% hits) | 4.18 us | 0.09 us | 0.09 us | 2.4 KB | + +No warnings. The first valid full result came from the repository's `BenchmarkWorkspaceOptions.Slim` job. diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md new file mode 100644 index 0000000..f4b0abd --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md @@ -0,0 +1,14 @@ +# LegacyAliasCountBenchmark summary excerpt + +| Method | Size | Mean | Error | Allocated | +|---|---:|---:|---:|---:| +| CountAll | 8 | 0.381 ns | 0.02 ns | 0 B | +| CountLegacy | 8 | NA | NA | NA | +| CountAll | 256 | 0.380 ns | 0.02 ns | 0 B | +| CountLegacy | 256 | 1.206 us | 0.03 us | 40 B | +| CountAll | 4096 | 0.381 ns | 0.02 ns | 0 B | +| CountLegacy | 4096 | 10.944 us | 0.19 us | 40 B | + +Benchmarks with issues: + LegacyAliasCountBenchmark.CountLegacy(Size: 8, Job: DefaultJob) + GlobalSetup failed: InvalidOperationException: Workload must have at least one legacy alias. diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs new file mode 100644 index 0000000..8c07ee4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs @@ -0,0 +1,37 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace Acme.Search; + +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] +public class LegacyAliasCountBenchmark +{ + [Params(8, 256, 4_096)] + public int Size { get; set; } + + private List _aliases = null!; + + [GlobalSetup] + public void Setup() + { + _aliases = Enumerable.Range(0, Size) + .Select(i => $"USR{i}") + .ToList(); + + var expectedMatches = LegacyAliasQuery.CountLegacyAliases(_aliases); + if (expectedMatches == 0) + { + throw new InvalidOperationException("Workload must have at least one legacy alias."); + } + } + + [Benchmark(Baseline = true, Description = "List.Count")] + public int CountAll() => _aliases.Count; + + [Benchmark(Description = "Where(alias.Length == 6).Count()")] + public int CountLegacy() => _aliases.Where(alias => alias.Length == 6).Count(); +} diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasImport.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasImport.cs new file mode 100644 index 0000000..0486ff1 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasImport.cs @@ -0,0 +1,12 @@ +using System.Collections.Generic; +using System.Linq; + +namespace Acme.Search; + +public sealed class LegacyAliasImport +{ + public IReadOnlyList BuildAliases(int count) => + Enumerable.Range(0, count) + .Select(i => $"USR{i:D3}") + .ToArray(); +} diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQuery.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQuery.cs new file mode 100644 index 0000000..d057440 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQuery.cs @@ -0,0 +1,9 @@ +using System.Collections.Generic; +using System.Linq; + +namespace Acme.Search; + +public static class LegacyAliasQuery +{ + public static int CountLegacyAliases(IEnumerable aliases) => aliases.Count(alias => alias.Length == 6); +} diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQueryTests.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQueryTests.cs new file mode 100644 index 0000000..2c3e9da --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQueryTests.cs @@ -0,0 +1,21 @@ +using System.Linq; +using Xunit; + +namespace Acme.Search; + +public class LegacyAliasQueryTests +{ + [Fact] + public void CountLegacyAliases_ReturnsAllFixedWidthAliases() + { + var aliases = Enumerable.Range(0, 8).Select(i => $"USR{i:D3}"); + Assert.Equal(8, LegacyAliasQuery.CountLegacyAliases(aliases)); + } + + [Fact] + public void CountLegacyAliases_AllowsZeroMatchesWhenScenarioIsNonLegacy() + { + var aliases = new[] { "NEW0001", "NEW0002", "NEW0003" }; + Assert.Equal(0, LegacyAliasQuery.CountLegacyAliases(aliases)); + } +} diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md index 23c0a47..9df3f79 100644 --- a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -6,10 +6,10 @@ Use this as a compact API and validation reference after the experiment question | Attribute | Purpose | |---|---| -| `[Benchmark]` | Marks measured work. Add `Baseline = true` only inside an equivalent comparison group and use `Description` for readable reports. | +| `[Benchmark]` | Marks measured work. Add `Baseline = true` only inside an equivalent comparison group and use `Description` to name the real measured terminal operation or materialization step. | | `[MemoryDiagnoser]` | Reports managed allocations and GC counts. Always include it for codebelt benchmarks. | | `[BenchmarkCategory("...")]` | Labels logical comparison groups when one class contains multiple alternative pairs. | -| `[GroupBenchmarksBy(...)]` | Groups report rows by category, params, or a deliberate combination. Grouping changes presentation and baseline scope; choose it from the question. | +| `[GroupBenchmarksBy(...)]` | Groups report rows by category, params, or a deliberate combination. Use it only when the grouping is meaningful for interpretation; remove decorative grouping. | | `[Params(...)]` | Sweeps independent compile-time-constant values. Multiple params properties create a Cartesian product. | | `[ParamsSource(nameof(...))]` | Supplies computed or coupled scenario objects with readable names. | | `[Arguments(...)]` / `[ArgumentsSource]` | Supplies method arguments, useful for explicit scenario sets. | @@ -18,7 +18,7 @@ Use this as a compact API and validation reference after the experiment question ## Baselines -A method baseline adds a ratio distribution against equivalent methods. BenchmarkDotNet allows category-specific baselines when benchmarks are grouped by category. Do not attach a baseline to an unrelated member or a lone benchmark just to satisfy a template. +A method baseline adds a ratio distribution against equivalent methods. BenchmarkDotNet allows category-specific baselines when benchmarks are grouped by category, but every baseline category still needs at least one equivalent peer. Do not attach a baseline to an unrelated member or a lone benchmark just to satisfy a template. Runtime comparisons can use a job baseline. Keep method, inputs, and implementation fixed when attributing a difference to runtime. @@ -30,11 +30,12 @@ Runtime comparisons can use a job baseline. Keep method, inputs, and implementat - Do not rely on execution order or shared mutation between methods. - Avoid manual loops unless batching is the real workload; BenchmarkDotNet selects invocation counts automatically. - Inspect all validation and environment warnings before reading result tables. +- Compare the discovered and reported method, job, and parameter matrix with the intended design; missing combinations are validation failures. - Keep the machine powered and quiet for full runs unless the noisy/throttled environment is the intended target. ## Validators -BenchmarkDotNet always validates duplicate baselines. `ExecutionValidator` can smoke-execute cases and `ReturnValueValidator` can compare compatible return values, but domain-specific correctness checks remain necessary. A Release build plus `--job dry` provides practical wiring/lifecycle validation through the codebelt runner. +BenchmarkDotNet always validates duplicate baselines. `ExecutionValidator` can smoke-execute cases and `ReturnValueValidator` can compare compatible return values, but domain-specific correctness checks remain necessary for every parameter and scenario case. Prefer exact checks for counts, values, status, exceptions, output contents, or mutations; use approximation only when the domain itself defines approximation. A Release build plus `--job dry` provides practical wiring and lifecycle validation through the codebelt runner, but zero-match boundary cases are still valid when the workload intends them. ## Diagnosers and profilers @@ -69,7 +70,26 @@ return c Only add jobs that the SUT and benchmark toolchain can execute. Keep the runner-default-only template as `return c;` with no unused runtime `using` directives. -Let BenchmarkDotNet choose warmup, iteration, launch, and invocation counts unless the performance question requires cold start, monitoring, or another specific run strategy. Short/dry jobs validate or iterate quickly; they do not replace the default job for performance conclusions. +`BenchmarkWorkspaceOptions.Slim` is the codebelt repository's deliberately shortened developer-oriented job. Report its active settings accurately instead of saying BenchmarkDotNet chose a fully adaptive warmup or iteration plan. In the current codebelt runner shape, Slim fixes one warmup iteration plus controlled iteration counts; that can characterize ordinary repository workloads, but one warmup may be insufficient for tiered-compilation, tiered-PGO, LINQ, or other JIT-sensitive runtime comparisons. + +Do not silently replace the repository runner configuration. Use a more stable diagnostic job only when the runtime or JIT comparison is itself material and explicitly in scope. Short and dry jobs still validate or iterate quickly; they do not replace a valid full run for performance conclusions. + +## Deferred pipelines and terminal operations + +Distinguish between: + +- query or iterator creation only; +- a terminal operation such as `Count()`, `Any()`, or `First()`; +- explicit full enumeration; +- materialization through `ToArray()` or `ToList()`. + +Name the benchmark and its conclusion accordingly. Inspect collection fast paths before describing a result as enumeration. `List.Count` can be O(1) while `Where(...).Count()` enumerates the filtered pipeline, so the former is not an honest baseline for the latter. + +## Result-validity gate + +After a dry execution and after any full run, inspect the complete BenchmarkDotNet summary before interpreting numbers. Treat the run as invalid for performance interpretation when any intended case has an `NA` measurement, appears under `Benchmarks with issues`, throws in setup or cleanup, triggers a validation error, fails a job or runtime, or is missing from the intended matrix. + +If that happens, report the exact failing method, job, and parameter case. Do not let the surviving rows stand in for the completed benchmark. Correct benchmark-owned causes, rerun build, discovery, and dry validation, and require fresh human authority before any replacement full measurement run. ## Existing-report filtering @@ -105,6 +125,12 @@ dotnet run -c Release --project tooling/{runner} -- --filter *{BenchmarkClass}* Reports are written under `reports/`. +## Interpreting a full run + +Read warnings before tables. Compare mean, error, standard deviation, median when distributions are skewed, ratio distributions within valid groups, allocated bytes, GC counts, and specialized diagnoser columns for the exact workload and environment. + +After the first valid full result, answer three questions: is the result reproducible, is the absolute cost material for observed or plausible usage, and would a deeper diagnostic change a concrete engineering decision? Stop when the answer is no. Do not escalate automatically to repeated reruns, tiered-PGO variants, disassembly, EventPipe or ETW tracing, alternative implementations, or runtime-source archaeology. Escalate only when the result is reproducible, material, the next diagnostic can distinguish concrete competing explanations, and the user requested it or it is necessary to answer the original decision. + ## Primary sources - BenchmarkDotNet good practices: diff --git a/skills/dotnet-benchmark/references/experiment-design.md b/skills/dotnet-benchmark/references/experiment-design.md index 70e132e..ea8e90c 100644 --- a/skills/dotnet-benchmark/references/experiment-design.md +++ b/skills/dotnet-benchmark/references/experiment-design.md @@ -26,6 +26,17 @@ For a single current implementation measured across sizes or scenarios, omit `Ba When comparing runtimes rather than implementations, use a job baseline and keep the measured method the same. Do not mix runtime and algorithm changes in one conclusion unless the full matrix is intentional. +## Configuration coherence + +Before build, list, or dry validation, read the attributes back as a matrix: + +- `Baseline = true` only where methods do equivalent observable work. +- `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]` only when meaningful `[BenchmarkCategory]` values exist. +- Every category with a baseline contains at least one equivalent comparison method. +- Single-operation characterization benchmarks do not invent a method baseline. +- `[Benchmark(Description = "...")]` names the real measured terminal operation or materialization step. +- Remove grouping attributes that do not improve interpretation. + ## Representative inputs Prefer cases grounded in real usage. Typical input should come first; add boundaries and adverse cases only when they exercise a distinct path. @@ -43,6 +54,19 @@ Use `[Params]` for independent scalar dimensions and `[ParamsSource]`/`[Argument Keep the case count disciplined. Each parameter combination multiplies methods, jobs, launch count, and diagnoser runs. Three representative sizes usually tell more than a dense power-of-two sweep; add points only when they locate a threshold or crossover. +## Workload invariants + +When one parameter only scales size, payload length, or another magnitude, keep other workload characteristics stable unless one is intentionally varied as its own named scenario. Common invariants include predicate selectivity or match percentage, hit/miss ratio, valid/invalid ratio, branch distribution, collision rate, string-length or encoding distribution, type distribution, entropy, and cache state. + +If one of those characteristics should vary, make it an explicit scenario or parameter instead of letting it drift accidentally with size. + +For predicate and filter benchmarks: + +- state the intended selectivity; +- use deterministic data; +- prefer fixed-width or structurally stable inputs when digit length or formatting would otherwise move cases between branches; +- verify the exact expected match count for every scenario, including valid zero-match cases. + ## Setup, state, and disposal Use `[GlobalSetup]` for deterministic state that is not part of the operation: payload creation, parsing expected results, object construction for instance methods, and correctness checks. BenchmarkDotNet runs global setup for each benchmark method and parameter combination, so setup must not rely on another benchmark method having run first. @@ -64,10 +88,12 @@ Avoid state leakage between methods, params, warmup, and measurement. Never depe An optimization benchmark without correctness validation can reward wrong code. Before a full run: 1. Execute baseline and candidate for every scenario outside the timed method. -2. Compare observable results with the domain's real equivalence rule, including output buffers, status codes, exceptions, mutations, and side effects. +2. Compare exact observable results with the domain's real equivalence rule, including counts, values, status codes, exceptions, output buffers, mutations, and side effects. 3. Fail setup or a focused test when results differ. 4. Keep the assertion/check out of the timed path. +Approximate checks are acceptable only when approximation is part of the domain semantics and the benchmark documents that rule. A valid boundary case that produces zero matches must still pass; only unexpected zero results should fail setup and force workload redesign. + For non-equivalent APIs, do not force a comparison. Characterize them separately and state the semantic difference. BenchmarkDotNet's `ReturnValueValidator` can supplement this for compatible return values, but it does not replace domain-aware correctness checks. @@ -80,6 +106,17 @@ Keep logging, assertions, `Random`, fixture construction, reflection discovery, Do not add a manual loop merely to make a tiny method measurable; BenchmarkDotNet chooses invocation counts and subtracts overhead. Use an in-method loop only when a batch is the real workload or state reset requires a defined sequence, then declare `OperationsPerInvoke` accurately. +## Deferred execution and terminal operations + +LINQ-style pipelines and iterators can measure different work: + +- query or iterator creation only; +- a terminal operation such as `Count()`, `Any()`, or `First()`; +- explicit full enumeration, such as a `foreach`; +- materialization through `ToArray()` or `ToList()`. + +Benchmark names, descriptions, and conclusions must say which one is measured. Inspect optimized terminal-operation paths before calling a result "enumeration." `List.Count` can be an O(1) collection fast path while `Where(...).Count()` enumerates the filtered pipeline; they are not interchangeable baselines. + ## Fair comparisons - Feed identical logical inputs and starting state to every implementation. @@ -137,8 +174,16 @@ Diagnosers can create additional runs and change total duration. Do not combine A dry job has too few measurements for conclusions. It validates wiring and lifecycle only. +## Benchmark validity gate + +After a dry execution and after any full run, inspect the complete BenchmarkDotNet summary before interpreting numbers. Treat the benchmark as invalid for performance interpretation when any intended case has an `NA` measurement, appears under `Benchmarks with issues`, throws in setup or cleanup, triggers a validation error, fails a job or runtime, or is missing from the intended matrix. + +When that happens, report the exact failing method, job, and parameter case. Do not treat successful rows as a completed benchmark. Correct the benchmark only when the cause lies within the benchmark, then rerun build, discovery, and dry validation. Another full run still requires explicit human authority. + ## Interpreting a full run Report the benchmark environment and exact workload. Read warnings before tables. Compare mean, error, standard deviation, median when distributions are skewed, ratio distributions within valid groups, allocated bytes, GC counts, and specialized diagnoser columns. Treat gains within noise as inconclusive. Check whether the result holds across representative cases and whether one case regresses. State the scope precisely: a result applies to the measured runtime, hardware, inputs, and configuration. Optimization value depends on application frequency and the maintenance/correctness cost of the change. + +After the first valid full result, answer three questions: is the result reproducible, is the absolute cost material for observed or plausible usage, and would a deeper diagnostic change a concrete engineering decision? Stop when the answer is no. Do not escalate automatically to repeated reruns, tiered-PGO variants, disassembly, EventPipe or ETW tracing, alternative implementations, or runtime-source archaeology. Escalate only when the result is reproducible, material, the next diagnostic can distinguish concrete competing explanations, and the user requested it or it is necessary to answer the original decision. Low-microsecond test-support utilities often end with "no change justified." From f265ef416bb1e6b57517bf2a3e02ca9744ed91b1 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 21:47:41 +0200 Subject: [PATCH 21/38] =?UTF-8?q?=F0=9F=94=A7=20dotnet-benchmark:=20update?= =?UTF-8?q?=20validator=20for=20new=20test=20fixtures=20and=20skill=20upda?= =?UTF-8?q?tes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update validate-skill.ps1 to discover and validate proportionate-stop and selectivity-drift test fixtures. Add assertions for new eval test cases 10 and 11. Ensure fixture structure and referenced files align with updated skill expectations. --- .../scripts/validate-skill.ps1 | 35 +++++++++++++++++-- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 index a5245f1..67d2502 100644 --- a/skills/dotnet-benchmark/scripts/validate-skill.ps1 +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -8,12 +8,16 @@ #> [CmdletBinding()] param( - [string]$SkillRoot = (Split-Path -Parent $PSScriptRoot) + [string]$SkillRoot ) $ErrorActionPreference = 'Stop' $failures = [System.Collections.Generic.List[string]]::new() +if ([string]::IsNullOrWhiteSpace($SkillRoot)) { + $SkillRoot = Split-Path -Parent $PSScriptRoot +} + function Add-Failure { param([string]$Message) $failures.Add($Message) @@ -64,6 +68,7 @@ if ($failures.Count -eq 0) { $skillPath = Join-Path $SkillRoot 'SKILL.md' $skill = [System.IO.File]::ReadAllText($skillPath) $skillLines = [System.IO.File]::ReadAllLines($skillPath) + $benchmarkEssentials = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/benchmarkdotnet-essentials.md')) $lineCount = $skillLines.Count if ($lineCount -gt 500) { Add-Failure "SKILL.md must stay at or below 500 lines; found $lineCount" @@ -94,6 +99,9 @@ if ($failures.Count -eq 0) { Assert-Contains 'SKILL.md' $skill 'read `references/runner-preflight.md`' Assert-Contains 'SKILL.md' $skill '-BenchmarkType ' Assert-Contains 'SKILL.md' $skill 'reports.wouldSkipRequestedBenchmark' + Assert-Contains 'SKILL.md' $skill 'complete BenchmarkDotNet summary' + Assert-Contains 'SKILL.md' $skill 'When a parameter is only size or payload' + Assert-Contains 'SKILL.md' $skill 'After the first valid full result' $forms = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'FORMS.md')) Assert-Contains 'FORMS.md' $forms '## Yolo mode override' @@ -125,18 +133,30 @@ if ($failures.Count -eq 0) { Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'SkipBenchmarksWithReports = true' Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'reports/tuning/' Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'Anti-thrashing rule' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Workload invariants' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Benchmark validity gate' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Deferred execution and terminal operations' + Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials 'one warmup iteration plus controlled iteration counts' + Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials '## Deferred pipelines and terminal operations' + Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials '## Result-validity gate' try { $evals = Get-Content -LiteralPath (Join-Path $SkillRoot 'evals/evals.json') -Raw | ConvertFrom-Json if ($evals.skill_name -ne 'dotnet-benchmark') { Add-Failure 'evals/evals.json skill_name must be dotnet-benchmark' } - if ($evals.evals.Count -lt 5) { - Add-Failure 'evals/evals.json must include at least five diverse evals' + if ($evals.evals.Count -lt 11) { + Add-Failure 'evals/evals.json must include at least eleven diverse evals' } if (-not ($evals.evals | Where-Object { $_.prompt -match '(?i)yolo' })) { Add-Failure 'evals/evals.json must include a yolo-mode interaction eval' } + if (-not ($evals.evals | Where-Object { $_.prompt -match 'LegacyAliasQuery' })) { + Add-Failure 'evals/evals.json must include the invalid parameter-matrix and drifting-selectivity eval' + } + if (-not ($evals.evals | Where-Object { $_.prompt -match 'TraitFilter helper only runs in test discovery' })) { + Add-Failure 'evals/evals.json must include the proportionate-stopping eval' + } $ids = @($evals.evals | ForEach-Object { $_.id }) if (($ids | Sort-Object -Unique).Count -ne $ids.Count) { Add-Failure 'evals/evals.json contains duplicate eval IDs' @@ -155,6 +175,15 @@ if ($failures.Count -eq 0) { } catch { Add-Failure "evals/evals.json is invalid: $($_.Exception.Message)" } + $evalFixtureRoot = Join-Path $SkillRoot 'evals/files' + if (Test-Path -LiteralPath $evalFixtureRoot) { + Get-ChildItem -LiteralPath $evalFixtureRoot -Recurse -Directory -Force | + Where-Object { $_.Name -in @('obj', 'bin', 'BenchmarkDotNet.Artifacts') } | + ForEach-Object { + $relativePath = $_.FullName.Substring($SkillRoot.Length).TrimStart('\') + Add-Failure "Eval fixtures must not contain generated artifact directories: $relativePath" + } + } Write-Verbose 'Template and eval checks completed.' } From 085d18c2a84184d5961ed28fcf4a7a8782eafde3 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 21:47:52 +0200 Subject: [PATCH 22/38] =?UTF-8?q?=F0=9F=94=A7=20repo=20validator:=20add=20?= =?UTF-8?q?suite-depth=20and=20fixture=20coverage=20checks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enhance validate-skill-templates.ps1 with checks for eval test depth, fixture distribution, and coverage-gap detection. Ensures repo-managed skills maintain sufficient test coverage and fixture variety. Add helper functions for tracking eval statistics. --- scripts/validate-skill-templates.ps1 | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 6f8708c..458c1db 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -832,15 +832,18 @@ Add-ValidationResult -Results $results -Name 'Benchmark runner wildcard is prese Assert-Match -Name 'benchmark-program.cs' -Content $program -Pattern 'namespace\s+\{BENCHMARK_RUNNER_NAMESPACE\};' } -Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence-backed candidates, supports yolo mode, and preserves honest comparison semantics' -Action { +Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, proportionate experiments and preserves honest comparison semantics' -Action { $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/SKILL.md' -GitRef $Ref $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/FORMS.md' -GitRef $Ref $candidateSelection = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/candidate-selection.md' -GitRef $Ref $experimentDesign = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/experiment-design.md' -GitRef $Ref + $benchmarkEssentials = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md' -GitRef $Ref $runnerPreflight = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/runner-preflight.md' -GitRef $Ref $comparison = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/comparison-benchmark.cs' -GitRef $Ref $operation = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/operation-benchmark.cs' -GitRef $Ref $evals = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/evals/evals.json' -GitRef $Ref + $fixtureFiles = Get-RepoFileList -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/evals/files' -GitRef $Ref + $validateSkillScript = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/scripts/validate-skill.ps1' -GitRef $Ref Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'A microbenchmark measures a suspected cost under a defined workload; it does not prove that the type is an application bottleneck.' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation.' @@ -853,6 +856,9 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence- Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Yolo never authorizes a full performance run.' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'read `references/runner-preflight.md`' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'reports.wouldSkipRequestedBenchmark' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'complete BenchmarkDotNet summary' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'When a parameter is only size or payload' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'After the first valid full result' Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'Auto-discover the highest-value performance questions (Recommended)' Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '### candidate_plan_confirmation' Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '## Yolo mode override' @@ -862,8 +868,15 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence- Assert-Contains -Name 'candidate-selection.md' -Content $candidateSelection -Needle '## Profiling-first gate' Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Correctness oracle' Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Do not compare unrelated operations.' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Workload invariants' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Benchmark validity gate' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Deferred execution and terminal operations' + Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle 'one warmup iteration plus controlled iteration counts' + Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Deferred pipelines and terminal operations' + Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Result-validity gate' Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'SkipBenchmarksWithReports = true' Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'Anti-thrashing rule' + Assert-Contains -Name 'validate-skill.ps1' -Content $validateSkillScript -Needle 'if ([string]::IsNullOrWhiteSpace($SkillRoot))' Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle '{EQUIVALENCE_CHECK}' Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle 'Baseline = true' Assert-Contains -Name 'operation-benchmark.cs' -Content $operation -Needle 'Do not add Baseline = true merely to produce a ratio column.' @@ -873,6 +886,11 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark selects evidence- Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'ThreadingDiagnoser' Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'YOLO mode:' Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'Acme.Core.ParserBenchmark-report-github.md' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'LegacyAliasQuery benchmark and summary' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'TraitFilter helper only runs in test discovery' + if (@($fixtureFiles | Where-Object { $_ -match '(^|/)(obj|bin|BenchmarkDotNet\.Artifacts)(/|$)' }).Count -gt 0) { + throw 'dotnet-benchmark eval fixtures must not include obj/, bin/, or BenchmarkDotNet.Artifacts paths' + } } Add-ValidationResult -Results $results -Name 'Strong-name skill matches FORMS summary flow and 1024-bit default' -Action { From 4de1feb538035d98c298b19087b2e24649250d38 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 21:48:04 +0200 Subject: [PATCH 23/38] =?UTF-8?q?=F0=9F=92=AC=20README:=20highlight=20dotn?= =?UTF-8?q?et-benchmark=20benchmarking=20discipline=20and=20proportionate-?= =?UTF-8?q?stop=20guidance?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update dotnet-benchmark skill description with new test coverage for selectivity drift, proportionate stopping, and benchmark-design rigor. Add feature bullets about case-study validation and early-stop conditions. --- README.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index e96baa4..2b17c20 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | -| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, validates correctness outside the timed path, routes specialized diagnosers for allocation/contention/exceptions/JIT questions, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | +| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | ### Copyable Install Commands @@ -606,15 +606,19 @@ Setting up a benchmark "properly" is only half the problem. A benchmark can comp - **Codebelt convention by default** — `tuning/` benchmark projects, a single `tooling/` runner host, and `reports/` output, mirroring `codebeltnet/cuemon` and `codebeltnet/xunit` - **Evidence-backed candidate selection** — ranks operations from call-site frequency, input scaling, allocations, contention, optimization leverage, and measurement fitness instead of treating public-member coverage as thoroughness - **Honest experiment shapes** — creates equivalent current-versus-candidate comparisons, baseline-free single-operation characterization, or profiling/macrobenchmark guidance; unrelated construction, formatting, equality, and hashing never receive misleading ratios -- **Representative workloads** — derives typical, boundary, scaling, hit/miss, valid/invalid, and other adverse-but-real cases from repository evidence, using coupled scenario sources instead of accidental parameter Cartesian products -- **Correctness before speed** — validates equivalent outputs and state transitions for every case outside the measured path before a full performance run +- **Hard validity gate** — reads the complete BenchmarkDotNet summary after dry execution and after any full run, treating `NA`, `Benchmarks with issues`, failed jobs, setup/cleanup exceptions, validation errors, or a partial matrix as invalid rather than letting surviving rows stand in for the whole experiment +- **Representative workloads** — derives typical, boundary, scaling, hit/miss, valid/invalid, and other adverse-but-real cases from repository evidence, uses coupled scenario sources instead of accidental parameter Cartesian products, and keeps selectivity, branch mix, and other workload invariants stable unless they are explicit scenarios +- **Correctness before speed** — validates exact outputs, counts, status, exception behavior, and state transitions for every case outside the measured path before a full performance run, while still allowing intended zero-match boundaries +- **Terminal-operation honesty** — distinguishes deferred query creation, terminal operators such as `Count()`/`Any()`/`First()`, explicit enumeration, and materialization, so a `List.Count` fast path is never sold as predicate traversal - **Specialized investigations** — handles mutation, async, contention, cold start, and exception paths explicitly, routing `ThreadingDiagnoser`, `ExceptionDiagnoser`, disassembly, or EventPipe only when each answers the stated question - **Namespace-correct** — the `*Benchmark` class lives in the same namespace as the code it measures, via a `RootNamespace` override, so type discovery and reports stay clean - **Allocations always measured** — `[MemoryDiagnoser]` is on by default - **Multi-runtime aware** — the runner host runs on .NET 9/10, but its BenchmarkDotNet jobs can compare `net48`, `net8.0`, `net9.0`, and `net10.0` - **Latest stable packages** — `BenchmarkDotNet`, `BenchmarkDotNet.Diagnostics.Windows`, and `Codebelt.Extensions.BenchmarkDotNet.Console` versions are resolved from NuGet, not hardcoded -- **Layered validation** — verifies the Release build, lists discovered cases, and dry-executes lifecycle and correctness wiring; it never turns that smoke check into a performance claim or launches the full machine-sensitive run unless asked +- **Layered validation** — verifies the Release build, lists discovered cases, dry-executes lifecycle and correctness wiring, and reruns build/list/dry after benchmark-owned validity fixes; it never turns that smoke check into a performance claim or launches the full machine-sensitive run unless asked - **Yolo mode without permission creep** — saying `yolo` auto-accepts evidence-backed defaults, skips routine plan/execution confirmations, and continues through build/list/dry validation; only an explicit human instruction can start a full benchmark run, and the mode never implies a commit, push, or unrelated external action +- **Proportional escalation** — after the first valid full result, it asks whether the issue is reproducible, material, and likely to change a real engineering decision before suggesting disassembly, EventPipe/ETW, repeated reruns, or alternative implementations +- **Honest Slim reporting** — reports the active `BenchmarkWorkspaceOptions.Slim` job accurately, including when its one-warmup developer-oriented shape limits runtime- or JIT-sensitive conclusions, instead of silently swapping the runner configuration - **Report-aware runner preflight** — inspects the canonical `BenchmarkWorkspaceOptions.Slim` runtime jobs, `SkipBenchmarksWithReports`, and matching `reports/tuning/` artifacts before touching benchmark code, so an intentional existing-report skip is explained instead of triggering disassembly, renaming, or speculative rewrites ## Repository structure From 80e63030decbcdc015303a783c38133db87dad5f Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 21:48:21 +0200 Subject: [PATCH 24/38] =?UTF-8?q?=F0=9F=A7=B9=20cleanup:=20remove=20build?= =?UTF-8?q?=20artifacts=20from=20eval=20fixture=20directories?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove stale obj/ directories and generated project files from benchmark-runner and Acme.Core.Benchmarks eval fixtures. These artifacts should not be committed to source control. --- ...oreApp,Version=v10.0.AssemblyAttributes.cs | 4 ---- .../net10.0/benchmark-runner.AssemblyInfo.cs | 22 ------------------- .../benchmark-runner.AssemblyInfoInputs.cache | 1 - ....GeneratedMSBuildEditorConfig.editorconfig | 17 -------------- ...oreApp,Version=v10.0.AssemblyAttributes.cs | 4 ---- .../Acme.Core.Benchmarks.AssemblyInfo.cs | 22 ------------------- ...e.Core.Benchmarks.AssemblyInfoInputs.cache | 1 - ....GeneratedMSBuildEditorConfig.editorconfig | 17 -------------- 8 files changed, 88 deletions(-) delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache delete mode 100644 skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs deleted file mode 100644 index d3d9ce2..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs +++ /dev/null @@ -1,4 +0,0 @@ -// -using System; -using System.Reflection; -[assembly: global::System.Runtime.Versioning.TargetFrameworkAttribute(".NETCoreApp,Version=v10.0", FrameworkDisplayName = ".NET 10.0")] diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs deleted file mode 100644 index b51cba4..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfo.cs +++ /dev/null @@ -1,22 +0,0 @@ -//------------------------------------------------------------------------------ -// -// This code was generated by a tool. -// -// Changes to this file may cause incorrect behavior and will be lost if -// the code is regenerated. -// -//------------------------------------------------------------------------------ - -using System; -using System.Reflection; - -[assembly: System.Reflection.AssemblyCompanyAttribute("benchmark-runner")] -[assembly: System.Reflection.AssemblyConfigurationAttribute("Debug")] -[assembly: System.Reflection.AssemblyFileVersionAttribute("1.0.0.0")] -[assembly: System.Reflection.AssemblyInformationalVersionAttribute("1.0.0+11a11fc181ece638e3d7af2b7b12fa080f6ac211")] -[assembly: System.Reflection.AssemblyProductAttribute("benchmark-runner")] -[assembly: System.Reflection.AssemblyTitleAttribute("benchmark-runner")] -[assembly: System.Reflection.AssemblyVersionAttribute("1.0.0.0")] - -// Generated by the MSBuild WriteCodeFragment class. - diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache deleted file mode 100644 index 7799c75..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.AssemblyInfoInputs.cache +++ /dev/null @@ -1 +0,0 @@ -d111d88d83b17a262229de83b0d790a39378a5aee7ab8a8e99f0b44928084759 diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig deleted file mode 100644 index c21c2c5..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/obj/Debug/net10.0/benchmark-runner.GeneratedMSBuildEditorConfig.editorconfig +++ /dev/null @@ -1,17 +0,0 @@ -is_global = true -build_property.TargetFramework = net10.0 -build_property.TargetFrameworkIdentifier = .NETCoreApp -build_property.TargetFrameworkVersion = v10.0 -build_property.TargetPlatformMinVersion = -build_property.UsingMicrosoftNETSdkWeb = -build_property.ProjectTypeGuids = -build_property.InvariantGlobalization = -build_property.PlatformNeutralAssembly = -build_property.EnforceExtendedAnalyzerRules = -build_property._SupportedPlatformList = Linux,macOS,Windows -build_property.RootNamespace = benchmark-runner -build_property.ProjectDir = C:\Source\Github\codebeltnet\agentic\skills\dotnet-benchmark\evals\files\runner-skip\tooling\benchmark-runner\ -build_property.EnableComHosting = -build_property.EnableGeneratedComInterfaceComImportInterop = -build_property.EffectiveAnalysisLevelStyle = 10.0 -build_property.EnableCodeStyleSeverity = diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs deleted file mode 100644 index d3d9ce2..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/.NETCoreApp,Version=v10.0.AssemblyAttributes.cs +++ /dev/null @@ -1,4 +0,0 @@ -// -using System; -using System.Reflection; -[assembly: global::System.Runtime.Versioning.TargetFrameworkAttribute(".NETCoreApp,Version=v10.0", FrameworkDisplayName = ".NET 10.0")] diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs deleted file mode 100644 index df394fa..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfo.cs +++ /dev/null @@ -1,22 +0,0 @@ -//------------------------------------------------------------------------------ -// -// This code was generated by a tool. -// -// Changes to this file may cause incorrect behavior and will be lost if -// the code is regenerated. -// -//------------------------------------------------------------------------------ - -using System; -using System.Reflection; - -[assembly: System.Reflection.AssemblyCompanyAttribute("Acme.Core.Benchmarks")] -[assembly: System.Reflection.AssemblyConfigurationAttribute("Debug")] -[assembly: System.Reflection.AssemblyFileVersionAttribute("1.0.0.0")] -[assembly: System.Reflection.AssemblyInformationalVersionAttribute("1.0.0+11a11fc181ece638e3d7af2b7b12fa080f6ac211")] -[assembly: System.Reflection.AssemblyProductAttribute("Acme.Core.Benchmarks")] -[assembly: System.Reflection.AssemblyTitleAttribute("Acme.Core.Benchmarks")] -[assembly: System.Reflection.AssemblyVersionAttribute("1.0.0.0")] - -// Generated by the MSBuild WriteCodeFragment class. - diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache deleted file mode 100644 index 0cfaaf5..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.AssemblyInfoInputs.cache +++ /dev/null @@ -1 +0,0 @@ -11a06bc3b99aba7551bd28f407a82bc36b90dfa4f0f69207a50e1811892463e2 diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig deleted file mode 100644 index bfd4cc0..0000000 --- a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/obj/Debug/net10.0/Acme.Core.Benchmarks.GeneratedMSBuildEditorConfig.editorconfig +++ /dev/null @@ -1,17 +0,0 @@ -is_global = true -build_property.TargetFramework = net10.0 -build_property.TargetFrameworkIdentifier = .NETCoreApp -build_property.TargetFrameworkVersion = v10.0 -build_property.TargetPlatformMinVersion = -build_property.UsingMicrosoftNETSdkWeb = -build_property.ProjectTypeGuids = -build_property.InvariantGlobalization = -build_property.PlatformNeutralAssembly = -build_property.EnforceExtendedAnalyzerRules = -build_property._SupportedPlatformList = Linux,macOS,Windows -build_property.RootNamespace = Acme.Core.Benchmarks -build_property.ProjectDir = C:\Source\Github\codebeltnet\agentic\skills\dotnet-benchmark\evals\files\runner-skip\tuning\Acme.Core.Benchmarks\ -build_property.EnableComHosting = -build_property.EnableGeneratedComInterfaceComImportInterop = -build_property.EffectiveAnalysisLevelStyle = 10.0 -build_property.EnableCodeStyleSeverity = From 34a18751a22a4e3f13a47164a10f6c9ae7ecfe9c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 23:35:19 +0200 Subject: [PATCH 25/38] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20dotnet-benchmark:=20?= =?UTF-8?q?add=20semantic=20preflight=20validation=20and=20InMemoryTestSto?= =?UTF-8?q?re=20test=20case?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add eval test case 12 to validate semantic preflight before performance interpretation. Test case covers correctness oracle derivation, workload-semantics gaps (selectivity drift), baseline validation, and fixture preparation. Update experiment-design.md reference with concrete guidance on deterministic input, exact count verification, and semantic preflight requirements. Refine SKILL.md to emphasize semantic preflight as a mandatory gate before accepting full-run results. --- skills/dotnet-benchmark/SKILL.md | 26 ++++-- skills/dotnet-benchmark/evals/evals.json | 19 +++++ .../inmemory-test-store/InMemoryTestStore.cs | 18 +++++ .../InMemoryTestStoreBenchmark-summary.md | 21 +++++ .../InMemoryTestStoreBenchmark.cs | 80 +++++++++++++++++++ .../InMemoryTestStoreTests.cs | 43 ++++++++++ .../references/experiment-design.md | 67 +++++++++++++--- 7 files changed, 257 insertions(+), 17 deletions(-) create mode 100644 skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStore.cs create mode 100644 skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md create mode 100644 skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs create mode 100644 skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreTests.cs diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index 8d8b167..b3556c2 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -1,7 +1,7 @@ --- name: dotnet-benchmark description: > - Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects source and usage evidence, ranks high-value operations instead of every public member, selects representative workloads, rejects misleading microbenchmarks, creates or reuses the tuning/ and tooling/ harness, preflights existing-report skips, validates correctness and discovery, and keeps full runs human-initiated. When the user says yolo, it auto-accepts routine defaults and proceeds through safe validation without confirmation churn. + Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects source and usage evidence, ranks high-value operations instead of every public member, selects representative workloads, rejects misleading microbenchmarks, creates or reuses the tuning/ and tooling/ harness, preflights existing-report skips, semantic-preflights workload correctness, validates discovery, and keeps full runs human-initiated. When the user says yolo, it auto-accepts routine defaults and proceeds through safe validation without confirmation churn. --- # Evidence-Driven .NET Benchmarking @@ -15,8 +15,9 @@ Create the smallest benchmark suite that can answer the most valuable performanc - Rank candidates using evidence from the implementation, call sites, tests, documentation, existing benchmark results, and profiles. Never invent usage frequency, input distributions, or a competing implementation. - Compare only operations that produce equivalent observable work. Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation. - Use `Baseline = true` only when at least two benchmark methods form a meaningful comparison group. A single-operation scaling or regression benchmark needs no fabricated baseline. When a class has several comparison groups, assign categories and one baseline inside each category. +- Before accepting `Baseline = true`, answer: do the baseline and candidate perform equivalent consumer-visible work over identical logical input? If not, remove the baseline or split the experiment. - Before build or dry validation, inspect benchmark attributes for internal coherence: `Baseline = true` only on equivalent observable work, `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]` only with meaningful `[BenchmarkCategory]` values, every baseline category has an equivalent peer, single-operation characterization benchmarks have no fabricated baseline, benchmark descriptions name the real measured terminal operation, and decorative grouping attributes are removed. -- Keep correctness outside the timed path but inside the verification workflow. Every parameter or scenario case needs an exact observable-result oracle for count, value, status, exception behavior, output contents, or mutation unless the domain explicitly defines approximation. Valid zero-match cases stay valid; if a zero result is unintended, fix the workload before measurement. +- Keep correctness outside the timed path but inside the verification workflow. Every scenario must define the input, intended operation, exact expected observable result, how that result was derived independently of the measured API path, and any promised workload characteristic such as selectivity, hit rate, type mix, or branch distribution. Successful execution is not a correctness oracle. Every parameter or scenario case needs exact observable-result validation for count, value, status, exception behavior, output contents, or mutation unless the domain explicitly defines approximation. Valid zero-match cases stay valid; if a zero result is unintended, fix the workload before measurement. - When one parameter only scales size, payload, or another magnitude, keep selectivity, hit/miss ratio, valid/invalid ratio, branch distribution, collision rate, string shape/encoding, type mix, entropy, and cache state stable unless one of them is intentionally exposed as a named scenario or parameter. - A benchmark is invalid for performance interpretation until the complete BenchmarkDotNet summary shows the full intended method/job/parameter matrix without `NA`, `Benchmarks with issues`, setup/cleanup exceptions, validation errors, failed jobs/runtimes, or missing combinations. Report the exact failing method, job, and parameter case, do not treat surviving rows as a completed benchmark, fix only benchmark-owned causes, rerun build/list/dry validation, and require fresh explicit human authority before any replacement full run. - Treat deferred pipeline creation, terminal operations such as `Count()`, `Any()`, or `First()`, explicit full enumeration, and materialization through `ToArray()` or `ToList()` as different workloads. Inspect fast paths such as `List.Count` before describing a benchmark as enumeration or using it as the baseline for predicate traversal. @@ -83,8 +84,8 @@ Before authoring code, present a compact plan with: - the performance question and metric: latency/throughput, allocated bytes, scaling, contention, cold start, or exception frequency; - the selected operation and the evidence that made it important; - the baseline and candidate, if a fair comparison exists; -- representative cases, including typical, boundary, scaling, and adverse-but-valid inputs where relevant; -- setup/reset strategy and correctness oracle; +- representative cases, including typical, boundary, scaling, and adverse-but-valid inputs where relevant, plus any named workload characteristic such as selectivity, hit rate, type mix, or branch distribution; +- setup/reset strategy and correctness oracle, including how each exact expected result is derived independently of the benchmark method; - candidates deliberately rejected and why; - whether the result will be exploratory or grounded in profile/telemetry evidence. @@ -102,8 +103,12 @@ Do not copy an asset blindly. The assets are structural examples with placeholde When a parameter is only size or payload, preserve the other workload characteristics unless the experiment names them explicitly. For predicate or filter benchmarks, state the intended selectivity, use deterministic data, prefer fixed-width or otherwise structurally stable inputs when digit length or formatting would change the branch mix, and verify the exact expected match count for every scenario. +If a case label promises an exact percentage such as `10% selectivity`, make that statement true for every size under a documented rounding rule. Otherwise choose compatible sizes, define explicit scenario objects with exact expected counts, or rename the cases honestly with qualitative names such as `LowSelectivity`, `HalfMatches`, or `AllMatch`. + For LINQ and other deferred pipelines, decide whether the benchmark measures query or iterator creation, a terminal operation, explicit enumeration, or materialization. Encode that distinction in names, descriptions, baselines, and conclusions; `List.Count` and `Where(...).Count()` are not interchangeable evidence. +For `QueryFor`-style runtime-type filters, use deterministic heterogeneous input with exact matches, sibling nonmatches, derived types when exact-type versus assignability semantics matter, and nulls only when supported and relevant. Validate the exact expected count and, when it matters, insertion order and exact runtime types. Do not benchmark a homogeneous all-match store and call it type filtering unless that all-match path is the stated subject. + ### 7. Author the benchmark Place the class under `tuning/{SutProject}.Benchmarks/`. Name it for the performance question and end the class name with `Benchmark`. Keep it in the SUT namespace rather than adding `.Benchmarks`; the benchmark project `RootNamespace` supports this codebelt convention. @@ -122,7 +127,17 @@ If the detector found missing infrastructure, follow `references/onboarding.md`. Before the build, inspect the benchmark attributes for coherence and remove decorative configuration that no longer serves the question. -First validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. The oracle should normally verify exact observable behavior rather than merely nonzero or approximate success. Then build the benchmark project in Release: +Run a semantic preflight across the full Cartesian set of benchmark methods, parameter values, runtime jobs, and scenario objects before build, discovery, dry execution, or any request for a full run. For every case verify that setup succeeds, the exact expected output is known independently, the actual output matches, the named workload characteristic is true, the intended code path is exercised, and the case does not collapse into a trivial fast path unless that fast path is the stated subject. + +Semantic preflight +- Every scenario has an independently derived exact expected result. +- Every parameter combination satisfies that oracle. +- Benchmark names accurately describe the generated workload. +- Selectivity, hit rate, type mix, and other workload distributions are explicit and verified. +- Every benchmark exercises the intended path. +- Baselines compare equivalent observable work. + +Successful execution is not a correctness oracle. After semantic preflight, validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. The oracle should normally verify exact observable behavior rather than merely nonzero or approximate success unless that is the real domain contract. Then build the benchmark project in Release: ```powershell dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj @@ -172,6 +187,7 @@ After the first valid full result, answer three questions: is the result reprodu - [ ] Cases represent realistic, boundary, scaling, and adverse paths without useless Cartesian products, and size-only sweeps keep other workload invariants stable unless explicitly named. - [ ] Setup, mutation, async, concurrency, disposal, and result consumption are handled correctly. - [ ] Exact correctness oracles cover every parameter and scenario case, including valid zero-match boundaries. +- [ ] A semantic preflight covered every method/param/job/scenario combination with independently derived expected results, truthful workload labels, intended code paths, and no accidental fast paths. - [ ] Baselines, categories, grouping attributes, and descriptions are internally coherent and non-decorative. - [ ] `[MemoryDiagnoser]` is present and every additional diagnoser has a stated purpose. - [ ] Harness changes preserve repository conventions and reuse existing projects/runner where possible. diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json index b42411f..3a701a7 100644 --- a/skills/dotnet-benchmark/evals/evals.json +++ b/skills/dotnet-benchmark/evals/evals.json @@ -183,6 +183,25 @@ "Does not automatically add repeated full reruns, tiered-PGO, disassembly, EventPipe or ETW tracing, runtime-source archaeology, or speculative alternative implementations", "Identifies a concrete reopen condition such as promotion to a production hot path, materially larger inputs or frequencies, or a new regression with decision-changing cost" ] + }, + { + "id": 12, + "prompt": "Review the attached Codebelt.Extensions.Xunit InMemoryTestStore benchmark and the first successful summary excerpt. Close the remaining correctness and workload-semantics gap before any further performance interpretation or rerun. Fix the benchmark design only; do not begin another deep performance investigation.", + "files": [ + "evals/files/inmemory-test-store/InMemoryTestStore.cs", + "evals/files/inmemory-test-store/InMemoryTestStoreTests.cs", + "evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs", + "evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md" + ], + "expected_output": "The agent adds a semantic preflight before any further run, derives independently trusted exact expected counts for every ItemCount and scenario, rejects the false 10% label at the smallest size, fixes or relabels the fake type-filter workload, removes the invalid no-predicate baseline, and validates the full case matrix without drifting into a broader performance investigation.", + "expectations": [ + "States that the existing successful benchmark summary is not a correctness oracle and requires semantic preflight before any future full run or interpretation", + "Derives exact expected counts independently of the measured Query path and validates every ItemCount/scenario combination, including the 8-item case whose current 10% label yields zero matches", + "Either chooses sizes or explicit scenario objects that preserve exact percentage semantics, or renames the cases honestly with qualitative labels such as LowSelectivity, HalfMatches, and AllMatch", + "Replaces the homogeneous QueryFor() input with deterministic heterogeneous runtime types spanning matches, sibling nonmatches, and derived types when assignability matters, or explicitly relabels the benchmark as the all-match path instead of type filtering", + "Removes Baseline=true from the no-predicate Count fast path comparison and rejects decorative ByCategory grouping unless an equivalent comparison group remains", + "Checks the full Cartesian matrix of benchmark methods, ItemCount values, runtime jobs, and scenario objects before build/list/dry/full validation, then stops without launching another deep performance investigation" + ] } ] } diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStore.cs b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStore.cs new file mode 100644 index 0000000..b2d7655 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStore.cs @@ -0,0 +1,18 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Codebelt.Extensions.Xunit; + +public sealed class InMemoryTestStore +{ + private readonly List _items = new(); + + public void Add(T item) => _items.Add(item); + + public IReadOnlyCollection Query() => _items; + + public IEnumerable Query(Func predicate) => _items.Where(predicate); + + public IEnumerable QueryFor() => _items.OfType(); +} diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md new file mode 100644 index 0000000..617f0eb --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md @@ -0,0 +1,21 @@ +# InMemoryTestStoreBenchmark summary excerpt + +The benchmark already executed successfully; the concern is whether the workloads and labels are semantically true. + +| Method | Runtime | ItemCount | Mean | Allocated | +|---|---|---:|---:|---:| +| Query - no predicate | .NET 10.0 | 8 | 0.7339 ns | - | +| Query - 10% selectivity | .NET 10.0 | 8 | 10.7615 ns | 72 B | +| Query - 50% selectivity | .NET 10.0 | 8 | 13.6044 ns | 72 B | +| Query - 100% selectivity | .NET 10.0 | 8 | 9.9140 ns | 72 B | +| QueryFor - filtered by type | .NET 10.0 | 8 | 9.0760 ns | 72 B | +| Query - no predicate | .NET 10.0 | 256 | 0.6446 ns | - | +| Query - 10% selectivity | .NET 10.0 | 256 | 80.3261 ns | 72 B | +| Query - 50% selectivity | .NET 10.0 | 256 | 100.6783 ns | 72 B | +| Query - 100% selectivity | .NET 10.0 | 256 | 96.9278 ns | 72 B | +| QueryFor - filtered by type | .NET 10.0 | 256 | 72.5479 ns | 72 B | +| Query - no predicate | .NET 10.0 | 4096 | 0.6269 ns | - | +| Query - 10% selectivity | .NET 10.0 | 4096 | 953.6987 ns | 72 B | +| Query - 50% selectivity | .NET 10.0 | 4096 | 1,103.5284 ns | 72 B | +| Query - 100% selectivity | .NET 10.0 | 4096 | 1,376.7703 ns | 72 B | +| QueryFor - filtered by type | .NET 10.0 | 4096 | 868.7072 ns | 72 B | diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs new file mode 100644 index 0000000..85795c4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs @@ -0,0 +1,80 @@ +using System; +using System.Linq; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace Codebelt.Extensions.Xunit; + +/// +/// Benchmarks for the query operations. +/// +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] +public class InMemoryTestStoreBenchmark +{ + private InMemoryTestStore _store; + private Func _low10PercentSelectivity; + private Func _mid50PercentSelectivity; + private Func _high100PercentSelectivity; + + [Params(8, 256, 4096)] + public int ItemCount { get; set; } + + [GlobalSetup] + public void Setup() + { + _store = new InMemoryTestStore(); + for (int i = 0; i < ItemCount; i++) + { + _store.Add(i); + } + + // Predicates with different selectivity levels. + var threshold10 = (int)(ItemCount * 0.1); + var threshold50 = ItemCount / 2; + + _low10PercentSelectivity = x => x < threshold10; + _mid50PercentSelectivity = x => x < threshold50; + _high100PercentSelectivity = x => x >= 0; + } + + [Benchmark(Baseline = true, Description = "Query - no predicate")] + [BenchmarkCategory("Query")] + public int Query_NoPredicate() + { + var result = _store.Query(); + return result.Count(); + } + + [Benchmark(Description = "Query - 10% selectivity")] + [BenchmarkCategory("Query")] + public int Query_10PercentSelectivity() + { + var result = _store.Query(_low10PercentSelectivity); + return result.Count(); + } + + [Benchmark(Description = "Query - 50% selectivity")] + [BenchmarkCategory("Query")] + public int Query_50PercentSelectivity() + { + var result = _store.Query(_mid50PercentSelectivity); + return result.Count(); + } + + [Benchmark(Description = "Query - 100% selectivity")] + [BenchmarkCategory("Query")] + public int Query_100PercentSelectivity() + { + var result = _store.Query(_high100PercentSelectivity); + return result.Count(); + } + + [Benchmark(Description = "QueryFor - filtered by type")] + [BenchmarkCategory("QueryFor")] + public int QueryFor_TypeFilter() + { + var result = _store.QueryFor(); + return result.Count(); + } +} diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreTests.cs b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreTests.cs new file mode 100644 index 0000000..ed775b1 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreTests.cs @@ -0,0 +1,43 @@ +using System.Linq; +using Xunit; + +namespace Codebelt.Extensions.Xunit; + +public class InMemoryTestStoreTests +{ + [Fact] + public void Query_AllowsValidZeroMatchFilters() + { + var store = new InMemoryTestStore(); + foreach (var value in Enumerable.Range(0, 8)) + { + store.Add(value); + } + + Assert.Empty(store.Query(x => x < 0)); + } + + [Fact] + public void QueryFor_ReturnsAssignableRuntimeTypesInInsertionOrder() + { + var store = new InMemoryTestStore(); + store.Add(new Order(1, "SO-001")); + store.Add(new Draft(2, "draft")); + store.Add(new PriorityOrder(3, "SO-002", 5)); + + var result = store.QueryFor().ToArray(); + + Assert.Collection( + result, + item => Assert.IsType(item), + item => Assert.IsType(item)); + } + + public abstract record StoreItem(int Id); + + public record Order(int Id, string Number) : StoreItem(Id); + + public sealed record PriorityOrder(int Id, string Number, int Priority) : Order(Id, Number); + + public sealed record Draft(int Id, string Name) : StoreItem(Id); +} diff --git a/skills/dotnet-benchmark/references/experiment-design.md b/skills/dotnet-benchmark/references/experiment-design.md index ea8e90c..d33b98c 100644 --- a/skills/dotnet-benchmark/references/experiment-design.md +++ b/skills/dotnet-benchmark/references/experiment-design.md @@ -10,11 +10,14 @@ Write down: - the primary metric: time/throughput, allocations, scaling, contention, cold start, or exception frequency; - baseline and candidate implementations, if both exist; - parameter cases and why each can change the result; +- the exact expected observable result for each scenario; +- how each expected result is derived independently of the measured API path; +- workload characteristics that the scenario name promises, such as match count, selectivity, hit rate, type distribution, or branch distribution; - setup/reset/disposal strategy; - a correctness oracle; - environmental variables that must remain fixed. -If these cannot be defined, more inspection is needed. Attributes do not rescue an ambiguous experiment. +If these cannot be defined, more inspection is needed. Attributes do not rescue an ambiguous experiment. Successful execution is not a correctness oracle. ## Comparison semantics @@ -26,6 +29,8 @@ For a single current implementation measured across sizes or scenarios, omit `Ba When comparing runtimes rather than implementations, use a job baseline and keep the measured method the same. Do not mix runtime and algorithm changes in one conclusion unless the full matrix is intentional. +Before accepting a method baseline, answer: do the baseline and candidate perform equivalent consumer-visible work over identical logical input? Reject baselines where one side returns an existing collection while the other enumerates, uses an O(1) fast path while the other scans, filters while the other does not, materializes while the other stays deferred, or validates/parses/transforms different semantics. + ## Configuration coherence Before build, list, or dry validation, read the attributes back as a matrix: @@ -67,6 +72,19 @@ For predicate and filter benchmarks: - prefer fixed-width or structurally stable inputs when digit length or formatting would otherwise move cases between branches; - verify the exact expected match count for every scenario, including valid zero-match cases. +If a case name promises an exact percentage such as `10% selectivity`, make that statement true for every size under a documented rounding rule. When that is not possible, choose compatible sizes, define explicit scenario objects with exact expected counts, or rename the cases honestly with qualitative names such as `LowSelectivity`, `HalfMatches`, or `AllMatch`. + +### Runtime-type filtering + +For `QueryFor`-style or other runtime-type filters, use deterministic heterogeneous input: + +- exact matches; +- sibling nonmatches; +- derived types when exact-type versus assignability semantics matter; +- nulls only when supported and relevant. + +Validate the exact expected count and, when it matters, insertion order and exact runtime types. Do not benchmark a homogeneous all-match store and call it type filtering unless the all-match path is the actual subject. + ## Setup, state, and disposal Use `[GlobalSetup]` for deterministic state that is not part of the operation: payload creation, parsing expected results, object construction for instance methods, and correctness checks. BenchmarkDotNet runs global setup for each benchmark method and parameter combination, so setup must not rely on another benchmark method having run first. @@ -85,18 +103,21 @@ Avoid state leakage between methods, params, warmup, and measurement. Never depe ## Correctness oracle -An optimization benchmark without correctness validation can reward wrong code. Before a full run: +An optimization benchmark without correctness validation can reward wrong code. For every scenario, define the input, intended operation, exact expected observable result, how that result was derived independently of the measured API path, and any promised workload characteristic such as selectivity, hit rate, or type mix. Successful execution is not a correctness oracle. Before a full run: 1. Execute baseline and candidate for every scenario outside the timed method. -2. Compare exact observable results with the domain's real equivalence rule, including counts, values, status codes, exceptions, output buffers, mutations, and side effects. -3. Fail setup or a focused test when results differ. -4. Keep the assertion/check out of the timed path. +2. Compare exact observable results with the domain's real equivalence rule, including counts, values, status codes, exceptions, output buffers, mutations, side effects, insertion order, and runtime types when they matter. +3. Ensure the expected value is derived independently of the measured API path; do not turn the implementation's current behavior into the oracle by computing `_expected` with the same call you intend to benchmark. +4. Fail setup or a focused test when results differ or when the generated workload does not match the named scenario. +5. Keep the assertion/check out of the timed path. Approximate checks are acceptable only when approximation is part of the domain semantics and the benchmark documents that rule. A valid boundary case that produces zero matches must still pass; only unexpected zero results should fail setup and force workload redesign. +A check such as `result == 0` is acceptable only when zero versus nonzero is the real domain contract. For collections, filters, parsers, matchers, and type selection, validate exact counts or exact outputs for every parameter case. + For non-equivalent APIs, do not force a comparison. Characterize them separately and state the semantic difference. -BenchmarkDotNet's `ReturnValueValidator` can supplement this for compatible return values, but it does not replace domain-aware correctness checks. +BenchmarkDotNet's `ReturnValueValidator` can supplement this for compatible return values, but it does not replace domain-aware correctness checks. A reference implementation may act as the oracle only when it is identified explicitly as trusted and is semantically equivalent. ## Prevent dead-code elimination and accidental work @@ -163,14 +184,36 @@ Benchmark trivial getters/operators only with evidence of extreme frequency or a Diagnosers can create additional runs and change total duration. Do not combine every diagnoser into a default benchmark. +## Semantic preflight + +Before build, list, or dry validation, inspect the full Cartesian set of benchmark methods, parameter values, runtime jobs, and scenario objects. For every case verify: + +- setup succeeds; +- the exact expected output is known independently; +- the exact actual output matches; +- the named workload characteristic is true; +- the case exercises the intended code path; +- the case does not collapse into a trivial fast path unless that fast path is the stated subject. + +Semantic preflight +- Every scenario has an independently derived exact expected result. +- Every parameter combination satisfies that oracle. +- Benchmark names accurately describe the generated workload. +- Selectivity, hit rate, type mix, and other workload distributions are explicit and verified. +- Every benchmark exercises the intended path. +- Baselines compare equivalent observable work. + +If any item fails, fix the benchmark before build, list, dry validation, or human authorization for a full run. + ## Layered validation -1. Run existing correctness tests for the SUT where feasible. -2. Execute the benchmark's correctness oracle for every case. -3. Build the benchmark project in Release. -4. Run `--list flat` with a filter and confirm the expected case/method combinations without accidental Cartesian products. -5. Run a `--job dry` execution smoke and resolve BenchmarkDotNet validation warnings or runtime failures. -6. Run the full default job only with explicit user intent and a suitable environment. +1. Run the semantic preflight across the full case matrix. +2. Run existing correctness tests for the SUT where feasible. +3. Execute the benchmark's correctness oracle for every case. +4. Build the benchmark project in Release. +5. Run `--list flat` with a filter and confirm the expected case/method combinations without accidental Cartesian products. +6. Run a `--job dry` execution smoke and resolve BenchmarkDotNet validation warnings or runtime failures. +7. Run the full default job only with explicit user intent and a suitable environment. A dry job has too few measurements for conclusions. It validates wiring and lifecycle only. From 3537406924abf21058be5edbc3fc28187ad54f6c Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 23:35:36 +0200 Subject: [PATCH 26/38] =?UTF-8?q?=F0=9F=94=A7=20update=20validation=20scri?= =?UTF-8?q?pts=20for=20semantic=20preflight=20and=20new=20eval=20fixture?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update skills/dotnet-benchmark/scripts/validate-skill.ps1 to discover and validate the new inmemory-test-store eval fixture. Add fixture structure assertions and fixture-file validators. Enhance scripts/validate-skill-templates.ps1 with checks for eval test semantics and correctness oracle validation. --- scripts/validate-skill-templates.ps1 | 10 ++++++++++ .../dotnet-benchmark/scripts/validate-skill.ps1 | 15 +++++++++++++-- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 458c1db..db273fc 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -858,6 +858,10 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, p Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'reports.wouldSkipRequestedBenchmark' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'complete BenchmarkDotNet summary' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'When a parameter is only size or payload' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Semantic preflight' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'independently derived exact expected result' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Successful execution is not a correctness oracle.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'After the first valid full result' Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'Auto-discover the highest-value performance questions (Recommended)' Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '### candidate_plan_confirmation' @@ -871,6 +875,10 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, p Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Workload invariants' Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Benchmark validity gate' Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Deferred execution and terminal operations' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Semantic preflight' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Successful execution is not a correctness oracle.' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Do not benchmark a homogeneous all-match store and call it type filtering unless the all-match path is the actual subject.' Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle 'one warmup iteration plus controlled iteration counts' Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Deferred pipelines and terminal operations' Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Result-validity gate' @@ -887,6 +895,8 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, p Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'YOLO mode:' Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'Acme.Core.ParserBenchmark-report-github.md' Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'LegacyAliasQuery benchmark and summary' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'InMemoryTestStore benchmark' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'LowSelectivity, HalfMatches, and AllMatch' Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'TraitFilter helper only runs in test discovery' if (@($fixtureFiles | Where-Object { $_ -match '(^|/)(obj|bin|BenchmarkDotNet\.Artifacts)(/|$)' }).Count -gt 0) { throw 'dotnet-benchmark eval fixtures must not include obj/, bin/, or BenchmarkDotNet.Artifacts paths' diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 index 67d2502..54e4193 100644 --- a/skills/dotnet-benchmark/scripts/validate-skill.ps1 +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -101,6 +101,10 @@ if ($failures.Count -eq 0) { Assert-Contains 'SKILL.md' $skill 'reports.wouldSkipRequestedBenchmark' Assert-Contains 'SKILL.md' $skill 'complete BenchmarkDotNet summary' Assert-Contains 'SKILL.md' $skill 'When a parameter is only size or payload' + Assert-Contains 'SKILL.md' $skill 'Semantic preflight' + Assert-Contains 'SKILL.md' $skill 'independently derived exact expected result' + Assert-Contains 'SKILL.md' $skill 'Successful execution is not a correctness oracle.' + Assert-Contains 'SKILL.md' $skill 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' Assert-Contains 'SKILL.md' $skill 'After the first valid full result' $forms = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'FORMS.md')) @@ -136,6 +140,10 @@ if ($failures.Count -eq 0) { Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Workload invariants' Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Benchmark validity gate' Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Deferred execution and terminal operations' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Semantic preflight' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) 'Successful execution is not a correctness oracle.' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) 'Do not benchmark a homogeneous all-match store and call it type filtering unless the all-match path is the actual subject.' Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials 'one warmup iteration plus controlled iteration counts' Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials '## Deferred pipelines and terminal operations' Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials '## Result-validity gate' @@ -145,8 +153,8 @@ if ($failures.Count -eq 0) { if ($evals.skill_name -ne 'dotnet-benchmark') { Add-Failure 'evals/evals.json skill_name must be dotnet-benchmark' } - if ($evals.evals.Count -lt 11) { - Add-Failure 'evals/evals.json must include at least eleven diverse evals' + if ($evals.evals.Count -lt 12) { + Add-Failure 'evals/evals.json must include at least twelve diverse evals' } if (-not ($evals.evals | Where-Object { $_.prompt -match '(?i)yolo' })) { Add-Failure 'evals/evals.json must include a yolo-mode interaction eval' @@ -154,6 +162,9 @@ if ($failures.Count -eq 0) { if (-not ($evals.evals | Where-Object { $_.prompt -match 'LegacyAliasQuery' })) { Add-Failure 'evals/evals.json must include the invalid parameter-matrix and drifting-selectivity eval' } + if (-not ($evals.evals | Where-Object { $_.prompt -match 'InMemoryTestStore benchmark' })) { + Add-Failure 'evals/evals.json must include the semantic-preflight workload validation eval' + } if (-not ($evals.evals | Where-Object { $_.prompt -match 'TraitFilter helper only runs in test discovery' })) { Add-Failure 'evals/evals.json must include the proportionate-stopping eval' } From 78da69e70f0507166fbd7642d179af4e0d14ed6a Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Fri, 17 Jul 2026 23:35:47 +0200 Subject: [PATCH 27/38] =?UTF-8?q?=F0=9F=92=AC=20README:=20highlight=20dotn?= =?UTF-8?q?et-benchmark=20semantic=20preflight=20and=20correctness=20gates?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update dotnet-benchmark skill table entry to emphasize semantic preflight as a mandatory validation gate and InMemoryTestStore correctness oracle test case. Add feature bullets about correctness oracle derivation, deterministic workload definition, and false-positive baseline prevention. --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 2b17c20..d8e7760 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | -| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | +| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles plus a semantic preflight for truthful workload labels, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | ### Copyable Install Commands @@ -609,6 +609,7 @@ Setting up a benchmark "properly" is only half the problem. A benchmark can comp - **Hard validity gate** — reads the complete BenchmarkDotNet summary after dry execution and after any full run, treating `NA`, `Benchmarks with issues`, failed jobs, setup/cleanup exceptions, validation errors, or a partial matrix as invalid rather than letting surviving rows stand in for the whole experiment - **Representative workloads** — derives typical, boundary, scaling, hit/miss, valid/invalid, and other adverse-but-real cases from repository evidence, uses coupled scenario sources instead of accidental parameter Cartesian products, and keeps selectivity, branch mix, and other workload invariants stable unless they are explicit scenarios - **Correctness before speed** — validates exact outputs, counts, status, exception behavior, and state transitions for every case outside the measured path before a full performance run, while still allowing intended zero-match boundaries +- **Semantic preflight** — checks the full method/job/parameter/scenario matrix before build/list/dry/full interpretation, proves independently derived exact expected results, verifies workload labels such as selectivity, hit rate, and type mix, and rejects accidental fast paths or fake type-filter workloads - **Terminal-operation honesty** — distinguishes deferred query creation, terminal operators such as `Count()`/`Any()`/`First()`, explicit enumeration, and materialization, so a `List.Count` fast path is never sold as predicate traversal - **Specialized investigations** — handles mutation, async, contention, cold start, and exception paths explicitly, routing `ThreadingDiagnoser`, `ExceptionDiagnoser`, disassembly, or EventPipe only when each answers the stated question - **Namespace-correct** — the `*Benchmark` class lives in the same namespace as the code it measures, via a `RootNamespace` override, so type discovery and reports stay clean From d29ea2486d8df279f5696a600e256190119169ab Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 01:55:39 +0200 Subject: [PATCH 28/38] =?UTF-8?q?=F0=9F=94=A7=20update=20harness=20detecto?= =?UTF-8?q?r=20to=20use=20pwsh=20if=20available=20for=20validation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- skills/dotnet-benchmark/scripts/validate-skill.ps1 | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 index 54e4193..dd72216 100644 --- a/skills/dotnet-benchmark/scripts/validate-skill.ps1 +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -216,7 +216,8 @@ try { $detectorPath = Join-Path $SkillRoot 'scripts/check-benchmark-requirements.ps1' if (Test-Path -LiteralPath $detectorPath) { try { - $detected = & powershell -NoProfile -ExecutionPolicy Bypass -File $detectorPath -RepoRoot $fixtureRoot -BenchmarkType Acme.Core.WidgetBenchmark -SkipSdkCheck | ConvertFrom-Json + $pwshExe = if (Get-Command pwsh -ErrorAction SilentlyContinue) { 'pwsh' } else { 'powershell' } + $detected = & $pwshExe -NoProfile -ExecutionPolicy Bypass -File $detectorPath -RepoRoot $fixtureRoot -BenchmarkType Acme.Core.WidgetBenchmark -SkipSdkCheck | ConvertFrom-Json if ($detected.solutionFormat -ne 'sln' -or -not $detected.centralPackageManagement -or -not $detected.centralizesBenchmarkConventions) { Add-Failure 'Harness detector did not recognize the fixture solution, CPM, and centralized conventions' } From ade323e6c72c7ff7460bb617eac674fd73e215f6 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 02:15:45 +0200 Subject: [PATCH 29/38] =?UTF-8?q?=F0=9F=93=9D=20clarify=20powershell=20com?= =?UTF-8?q?mand=20examples=20in=20dotnet-benchmark=20skill?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update example commands to use pwsh (PowerShell 7+) and add conditional guidance for Windows environments where PowerShell 7+ is unavailable, falling back to powershell. This improves clarity for users following the skill instructions. --- skills/dotnet-benchmark/SKILL.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index b3556c2..8e4399d 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -54,9 +54,11 @@ Yolo never authorizes a full performance run. Start the full benchmark only when Run the bundled read-only detector before changing files: ```powershell -powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot +pwsh -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot ``` +On Windows where PowerShell 7+ is unavailable, use `powershell` instead of `pwsh`. + Also inspect applicable `AGENTS.md`, solution/project files, `Directory.Build.props`, `Directory.Packages.props`, existing `tuning/` and `tooling/` projects, and nearby benchmark styles. Reuse an existing runner and benchmark project when they fit. Read `references/onboarding.md` only when the detector finds missing or partial harness infrastructure. When investigating a benchmark that builds but is not listed or executed, read `references/runner-preflight.md` before inspecting or rewriting the benchmark class. Rerun the detector with `-BenchmarkType ` and inspect the reported runner program, `SkipBenchmarksWithReports` setting, slim/runtime jobs, `reports/tuning/` files, and `wouldSkipRequestedBenchmark`. If a matching existing report explains the skip, preserve the runner and benchmark unchanged, report the matching file, and stop diagnostic escalation. Do not add disassembly, rename the class, disable report skipping, or churn through tools to evade the filter. @@ -146,9 +148,11 @@ dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.c Before interpreting discovery or execution output, run the report-aware preflight for the exact class: ```powershell -powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType +pwsh -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ``` +On Windows where PowerShell 7+ is unavailable, use `powershell` instead of `pwsh`. + If `reports.wouldSkipRequestedBenchmark` is true, the runner is intentionally filtering the type because a prior report exists. Report that as the validation outcome; do not claim the list/dry run exercised the class and do not modify code to force it through. A fresh full run and any report archive/replacement require explicit human direction. Verify runner discovery without measuring: From d34916aa067a4284850169054e590b883ec46760 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 03:30:49 +0200 Subject: [PATCH 30/38] =?UTF-8?q?=F0=9F=94=A7=20standardize=20local=20powe?= =?UTF-8?q?rshell=20execution=20to=20pwsh=207+?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add mandatory guidance that local PowerShell invocations must use pwsh 7+ runtime. Agents must report missing pwsh as a blocker instead of falling back to legacy Windows PowerShell. Legacy fallback silently uses the wrong runtime; standardizing on pwsh improves both clarity and compatibility across platforms. Validation script now enforces this policy across repo content. --- AGENTS.md | 14 +- scripts/validate-skill-templates.ps1 | 306 +++++++++++++++++++++++++++ 2 files changed, 317 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 36d1990..92f00eb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,14 @@ Repository-level rules for AI agents working in this codebase. +## Local Shell Execution + +Agents may use Bash or `pwsh` 7+ for local development. + +Whenever a local command uses PowerShell syntax or executes a `.ps1` script, invoke `pwsh`, never `powershell` or `powershell.exe`. Keep `.ps1` filenames unchanged; the required change is the runtime, not the script extension. Use the form `pwsh -NoProfile -File ./scripts/example.ps1` for local `.ps1` execution. + +Do not silently fall back to legacy Windows PowerShell. If `pwsh` 7+ is unavailable for a required local `.ps1` script, report the missing prerequisite instead of invoking legacy Windows PowerShell. This rule applies to local agent execution only; GitHub Actions may continue using `bash`, `sh`, `pwsh`, platform defaults, or another justified shell, and existing workflow shell choices should not be rewritten without a functional reason. + ## Eval Isolation Eval workspaces and test repositories must **never** be created inside this repository. This includes: @@ -66,16 +74,16 @@ Repo-managed skills live in four places that must stay in sync: Changes often start in `~/.claude/skills//`, then get mirrored to the repo and the other local installs: - **Claude local → repo** (persist changes to source control): - ```powershell + ```ps1 Copy-Item "$HOME/.claude/skills//" "skills//" -Force ``` - **Claude local → agent installs** (keep `~/.agents` and Gemini current): - ```powershell + ```ps1 Copy-Item "$HOME/.claude/skills//" "$HOME/.agents/skills//" -Force Copy-Item "$HOME/.claude/skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force ``` - **Repo → local installs** (after pulling changes or cloning fresh): - ```powershell + ```ps1 Copy-Item "skills//" "$HOME/.claude/skills//" -Force Copy-Item "skills//" "$HOME/.agents/skills//" -Force Copy-Item "skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index db273fc..18a43aa 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -60,6 +60,25 @@ function Get-RepoFileList { return @($output | Where-Object { $_ -like "$prefix*" } | ForEach-Object { $_.Substring($prefix.Length) } | Sort-Object) } +function Get-TrackedRepoPaths { + param( + [string]$RepoRoot, + [string]$GitRef + ) + + if ([string]::IsNullOrWhiteSpace($GitRef)) { + $output = git -C $RepoRoot ls-files 2>$null + } else { + $output = git -C $RepoRoot ls-tree -r --name-only $GitRef 2>$null + } + + if ($LASTEXITCODE -ne 0) { + throw 'Unable to enumerate tracked repository files for validation.' + } + + return @($output | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Sort-Object) +} + function Get-FileText { param( [string]$RepoRoot, @@ -83,6 +102,152 @@ function Get-FileText { return ($output -join [Environment]::NewLine) } +function Test-IsLocalShellPolicyScanCandidate { + param([string]$RelativePath) + + $normalizedPath = $RelativePath -replace '\\', '/' + if ($normalizedPath.StartsWith('./')) { + $normalizedPath = $normalizedPath.Substring(2) + } + if ($normalizedPath -eq 'CHANGELOG.md') { + return $false + } + + if ($normalizedPath -eq 'scripts/validate-skill-templates.ps1') { + return $false + } + + if ($normalizedPath -like 'skills/skill-creator-agnostic/*') { + return $false + } + + $extension = [System.IO.Path]::GetExtension($normalizedPath).ToLowerInvariant() + return @('.cs', '.json', '.md', '.ps1', '.yml', '.yaml') -contains $extension +} + +function Get-LocalShellPolicyFindingsFromContentItems { + param([object[]]$Items) + + $legacyShell = 'power' + 'shell' + $productName = 'Power' + 'Shell' + $legacyShellPattern = [regex]::Escape($legacyShell) + $productNamePattern = [regex]::Escape($productName) + $fence = [string]([char]96) * 3 + $fencePattern = [regex]::Escape($fence) + + $rules = @( + [pscustomobject]@{ + Pattern = "(?i)\b$legacyShellPattern(?:\.exe)?\s+-" + Message = 'Local command examples must use `pwsh` 7+ instead of the legacy executable.' + } + [pscustomobject]@{ + Pattern = "(?i)\bshell:\s*$legacyShellPattern(?:\.exe)?\b" + Message = 'Workflow steps that explicitly choose a `pwsh`-style shell must use `pwsh`.' + } + [pscustomobject]@{ + Pattern = "(?i)\buse\b.*\b$legacyShellPattern(?:\.exe)?\b.*\binstead of\b.*\bpwsh\b" + Message = 'Do not tell agents to fall back from `pwsh` to the legacy executable.' + } + [pscustomobject]@{ + Pattern = "(?i)\bwhen\s+$productNamePattern\s+is\s+available\b" + Message = 'Local guidance must name `pwsh` 7+ explicitly instead of generic shell availability.' + } + [pscustomobject]@{ + Pattern = "(?i)\bif\s+$productNamePattern\s+is\s+unavailable\b" + Message = 'Missing `pwsh` 7+ must be reported as the blocker for required local `.ps1` execution.' + } + [pscustomobject]@{ + Pattern = "(?i)\brun this\s+$productNamePattern\s+script\b" + Message = 'Local script instructions must name `pwsh` 7+ explicitly.' + } + [pscustomobject]@{ + Pattern = "(?i)\b$productNamePattern\s+session\b" + Message = 'Refer to a `pwsh` session for local execution guidance.' + } + [pscustomobject]@{ + Pattern = "(?i)\b$productNamePattern\s+or\s+terminal\b" + Message = 'Prefer `pwsh` 7+ or shell-agnostic wording for local execution guidance.' + } + [pscustomobject]@{ + Pattern = "(?i)^$fencePattern$legacyShellPattern\s*$" + Message = 'Use `ps1` for `pwsh` syntax fences or a shell-agnostic fence such as `bash` for generic commands.' + } + ) + + $findings = [System.Collections.Generic.List[object]]::new() + + foreach ($item in @($Items)) { + if ($null -eq $item -or [string]::IsNullOrWhiteSpace([string]$item.Path)) { + continue + } + + $relativePath = [string]$item.Path -replace '\\', '/' + if ($relativePath.StartsWith('./')) { + $relativePath = $relativePath.Substring(2) + } + if (-not (Test-IsLocalShellPolicyScanCandidate -RelativePath $relativePath)) { + continue + } + + $content = if ($null -eq $item.Content) { '' } else { [string]$item.Content } + $lines = [regex]::Split($content, '\r?\n') + + for ($index = 0; $index -lt $lines.Length; $index++) { + $line = $lines[$index] + $message = $null + + foreach ($rule in $rules) { + if ($line -match $rule.Pattern) { + $message = $rule.Message + break + } + } + + if ($null -eq $message) { + $mentionsPs1 = $line -match '(?i)\.ps1\b' + $mentionsPwsh = $line -match '(?i)\bpwsh\b' + $isPs1CommandExample = $line -match '^\s*(?:\./|\.\\)?[A-Za-z0-9_./\\-]+\.ps1(?:\s|$)' + $isPs1RunInstruction = $line -match '(?i)\b(?:run|invoke|execute)\b.*?\.ps1\b' + + if ($mentionsPs1 -and -not $mentionsPwsh -and ($isPs1CommandExample -or $isPs1RunInstruction)) { + $message = 'Local `.ps1` instructions must include `pwsh -NoProfile -File` explicitly.' + } + } + + if ($null -ne $message) { + $findings.Add([pscustomobject]@{ + Path = $relativePath + LineNumber = $index + 1 + Line = $line + Message = $message + }) + } + } + } + + return @($findings) +} + +function Get-LocalShellPolicyFindings { + param( + [string]$RepoRoot, + [string]$GitRef + ) + + $items = foreach ($path in Get-TrackedRepoPaths -RepoRoot $RepoRoot -GitRef $GitRef) { + if (-not (Test-IsLocalShellPolicyScanCandidate -RelativePath $path)) { + continue + } + + [pscustomobject]@{ + Path = $path + Content = Get-FileText -RepoRoot $RepoRoot -RelativePath $path -GitRef $GitRef + } + } + + return @(Get-LocalShellPolicyFindingsFromContentItems -Items $items) +} + function Assert-Contains { param( [string]$Name, @@ -519,6 +684,137 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills keep YAML } } +Add-ValidationResult -Results $results -Name 'Active local shell guidance requires pwsh for local `.ps1` and shell execution' -Action { + $findings = @(Get-LocalShellPolicyFindings -RepoRoot $repoRoot -GitRef $Ref) + + if ($findings.Count -gt 0) { + $details = @( + $findings | ForEach-Object { + '{0}:{1}: {2}`n {3}' -f $_.Path, $_.LineNumber, $_.Message, $_.Line.Trim() + } + ) + + throw ("Local `.ps1` and shell execution requires `pwsh` 7+; update these lines:`n" + ($details -join "`n")) + } +} + +Add-ValidationResult -Results $results -Name 'Local shell policy scanner rejects legacy runtime guidance and allows approved alternatives' -Action { + $legacyShell = 'power' + 'shell' + $legacyExe = $legacyShell + '.exe' + $productName = 'Power' + 'Shell' + $fence = [string]([char]96) * 3 + + $cases = @( + [pscustomobject]@{ + Name = 'maintained skill legacy command' + Path = 'skills/example/case-01.md' + Content = "$legacyShell -NoProfile -File ./scripts/example.ps1" + ExpectViolation = $true + } + [pscustomobject]@{ + Name = 'case-variant legacy command' + Path = 'skills/example/case-02.md' + Content = ('PoWeR' + 'ShElL -NoProfile -File ./scripts/example.ps1') + ExpectViolation = $true + } + [pscustomobject]@{ + Name = 'legacy executable command' + Path = 'skills/example/case-03.md' + Content = "$legacyExe -File ./scripts/example.ps1" + ExpectViolation = $true + } + [pscustomobject]@{ + Name = 'pwsh script command' + Path = 'skills/example/case-04.md' + Content = 'pwsh -NoProfile -File ./scripts/example.ps1' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'bash command' + Path = 'skills/example/case-05.md' + Content = 'bash ./scripts/example.sh' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'workflow bash shell' + Path = '.github/workflows/case-06.yml' + Content = 'shell: bash' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'workflow pwsh shell' + Path = '.github/workflows/case-07.yml' + Content = 'shell: pwsh' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'ps1 fence' + Path = 'skills/example/case-08.md' + Content = $fence + 'ps1' + [Environment]::NewLine + '$value = 1' + [Environment]::NewLine + $fence + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'released changelog exclusion' + Path = 'CHANGELOG.md' + Content = "$legacyShell -File ./scripts/example.ps1" + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'obsolete skill exclusion' + Path = 'skills/skill-creator-agnostic/SKILL.md' + Content = "$legacyShell -File ./scripts/example.ps1" + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'generic availability phrasing' + Path = 'skills/example/case-11.md' + Content = "When $productName is available, prefer the helper." + ExpectViolation = $true + } + [pscustomobject]@{ + Name = 'legacy workflow shell' + Path = '.github/workflows/case-12.yml' + Content = "shell: $legacyShell" + ExpectViolation = $true + } + ) + + $items = foreach ($case in $cases) { + [pscustomobject]@{ + Path = $case.Path + Content = $case.Content + } + } + + $findings = Get-LocalShellPolicyFindingsFromContentItems -Items $items + + foreach ($case in $cases) { + $caseFindings = @($findings | Where-Object { $_.Path -eq $case.Path }) + + if ($case.ExpectViolation -and $caseFindings.Count -eq 0) { + throw "Expected a finding for '$($case.Name)' but none was reported." + } + + if (-not $case.ExpectViolation -and $caseFindings.Count -gt 0) { + throw "Expected no finding for '$($case.Name)' but found: $($caseFindings[0].Message)" + } + } +} + +Add-ValidationResult -Results $results -Name 'Repository docs define the local shell execution policy' -Action { + $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref + $contributing = Get-FileText -RepoRoot $repoRoot -RelativePath 'CONTRIBUTING.md' -GitRef $Ref + $readme = Get-FileText -RepoRoot $repoRoot -RelativePath 'README.md' -GitRef $Ref + + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## Local Shell Execution' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Agents may use Bash or `pwsh` 7+ for local development.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Use the form `pwsh -NoProfile -File ./scripts/example.ps1` for local `.ps1` execution.' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'Bash and `pwsh` 7+ are both valid for local development' + Assert-Contains -Name 'README.md' -Content $readme -Needle 'pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1' +} + Add-ValidationResult -Results $results -Name 'App skill collects target framework and conditional web_variant' -Action { $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-new-app-slnx/FORMS.md' -GitRef $Ref Assert-Contains -Name 'dotnet-new-app-slnx/FORMS.md' -Content $forms -Needle '### target_framework' @@ -549,6 +845,8 @@ Add-ValidationResult -Results $results -Name 'App skill documents web-family App Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'Treat the scaffold as a fidelity copy of the documented template set, not a "best effort" approximation.' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle '## Step 3: Resolve Dynamic Dependency Versions' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'scripts/resolve-package-versions.ps1' + Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework ' + Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'If the host does not render native form controls, follow the deterministic plain-text fallback defined in `FORMS.md` instead of improvising your own questioning style.' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'Consistency matters more than creativity during parameter collection.' @@ -659,6 +957,7 @@ Add-ValidationResult -Results $results -Name 'App reference guide uses ROOT_NAME Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Directory.Packages.props` is the authoritative version source for app scaffolds.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Do **not** duplicate `` inside the generated app or test `.csproj` files as a workaround.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'scripts/resolve-package-versions.ps1' + Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework {TARGET_FRAMEWORK}' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle '`testenvironments.json` is required output for the scaffold.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'MinVer may report a bootstrap pre-release such as `0.0.0-alpha.0`' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Where `{AppType}` maps to the emitted project suffix:' @@ -772,6 +1071,8 @@ Add-ValidationResult -Results $results -Name 'Library skill documents PROJECT_NA Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle '{PROJECT_NAME}' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle '{DOCFX_TARGET_FRAMEWORK}' + Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1' + Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'In `pwsh` 7+, prefer .NET file APIs' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'If the host does not render native form controls, follow the deterministic plain-text fallback defined in `FORMS.md` instead of improvising your own questioning style.' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'Consistency matters more than creativity during parameter collection.' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'treat a blank response as accepting that shown value' @@ -801,6 +1102,7 @@ Add-ValidationResult -Results $results -Name 'Library reference guide uses curre Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'do not create an extra solution-named wrapper folder' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'src/{PROJECT_NAME}/{PROJECT_NAME}.csproj' + Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'Recommended `pwsh` 7+ approach for rewritten templates:' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'offer every other generally supported non-preview .NET LTS and STS channel' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'highest selected generally supported non-preview executable TFM' } @@ -851,6 +1153,7 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, p Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Read `references/experiment-design.md`' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--list flat' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--job dry' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot ' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '#### Yolo mode' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Start a full performance run only after an explicit human instruction to run it now.' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Yolo never authorizes a full performance run.' @@ -884,7 +1187,9 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, p Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Result-validity gate' Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'SkipBenchmarksWithReports = true' Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'Anti-thrashing rule' + Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ' Assert-Contains -Name 'validate-skill.ps1' -Content $validateSkillScript -Needle 'if ([string]::IsNullOrWhiteSpace($SkillRoot))' + Assert-Contains -Name 'validate-skill.ps1' -Content $validateSkillScript -Needle 'Harness detector validation requires pwsh 7+' Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle '{EQUIVALENCE_CHECK}' Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle 'Baseline = true' Assert-Contains -Name 'operation-benchmark.cs' -Content $operation -Needle 'Do not add Baseline = true merely to produce a ratio column.' @@ -907,6 +1212,7 @@ Add-ValidationResult -Results $results -Name 'Strong-name skill matches FORMS su $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-strong-name-signing/SKILL.md' -GitRef $Ref Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'compute the defaults silently, and present a single summary for confirmation' Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'default: 1024' + Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'Run this command block in a `pwsh` 7+ session in the target directory:' Assert-NotContains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'default: 4096' } From fc0b6d778cbe748131713861d29ae1ec0d43077e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 03:30:56 +0200 Subject: [PATCH 31/38] =?UTF-8?q?=F0=9F=93=9D=20update=20shell=20guidance?= =?UTF-8?q?=20for=20contributors=20and=20users?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update all local command examples and contributor guidance to use pwsh 7+ consistently. Clarify that Bash and pwsh are both supported for local development, but every local PowerShell command and .ps1 script must use pwsh, never the legacy executable. GitHub Actions workflows may continue using shell choices appropriate for their context. --- CONTRIBUTING.md | 50 ++++++++++++++++++++++++------------------------- README.md | 10 +++++----- 2 files changed, 30 insertions(+), 30 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ff811a9..f4be691 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -68,27 +68,27 @@ The `description` is the most important field — it's how the AI decides to loa Evals let you verify the skill works and measure improvement over a baseline. Every repo-managed skill in this repository must include `evals/evals.json`: -```json -{ - "skill_name": "your-skill-name", - "evals": [ - { - "id": 0, - "prompt": "The user message to test against", - "expected_output": "What a correct response looks like — used for manual or automated grading", - "files": ["evals/files/example.md"] - } - ] -} -``` - -`files` is optional. When present, list one or more fixture files relative to `skills//`. A common pattern is to store those fixtures under `evals/files/` so benchmark runners can copy or attach the same source inputs for both `with_skill` and `without_skill` runs. +```json +{ + "skill_name": "your-skill-name", + "evals": [ + { + "id": 0, + "prompt": "The user message to test against", + "expected_output": "What a correct response looks like — used for manual or automated grading", + "files": ["evals/files/example.md"] + } + ] +} +``` + +`files` is optional. When present, list one or more fixture files relative to `skills//`. A common pattern is to store those fixtures under `evals/files/` so benchmark runners can copy or attach the same source inputs for both `with_skill` and `without_skill` runs. Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and cases where the skill should *not* do something. Run evals from a temp workspace, not from this repository: -```powershell +```ps1 $workspace = Join-Path $env:TEMP '-workspace' ``` @@ -116,16 +116,16 @@ When a skill needs defaults for versions, paths, repository names, or support wi Use the repo validation harness before submitting scaffold or template changes: -```powershell -powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-skill-templates.ps1 +```ps1 +pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 ``` -Run the validator locally first for the fastest feedback loop. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. +Run the validator locally first for the fastest feedback loop. Bash and `pwsh` 7+ are both supported for local development in this repo, but every local `.ps1` invocation must go through `pwsh`. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. To compare a change against the initial imported version, run the same harness against a git ref: -```powershell -powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-skill-templates.ps1 -Ref HEAD +```ps1 +pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD ``` ## Checklist before submitting @@ -133,10 +133,10 @@ powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-skill-tem - [ ] `SKILL.md` has valid front matter with `name` and `description` - [ ] Skill is stack-agnostic (or clearly scoped to a specific tech in the name/description) - [ ] Examples are generic — no personal emails, usernames, or project-specific identifiers -- [ ] At least one eval in `evals/evals.json` -- [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name -- [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder -- [ ] Skill changes were benchmarked from a temp workspace with both `with_skill` and `without_skill` runs +- [ ] At least one eval in `evals/evals.json` +- [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name +- [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder +- [ ] Skill changes were benchmarked from a temp workspace with both `with_skill` and `without_skill` runs - [ ] `benchmark.json` and `eval-viewer/generate_review.py` from the installed Anthropic `skill-creator` copy were used so a human could compare `Outputs` and `Benchmark` - [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior - [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too diff --git a/README.md b/README.md index d8e7760..f6ea6e6 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Final DocFX verification is machine-adaptive: `auto` selects a high-capacity pro DocFX diagnostics favor repairable specificity: overwrite-layout errors now call out literal near-miss globs such as `api/namespaces/**.md` versus `api/namespaces/**/*.md`, no-observable-outcome example failures explain what visible reader result is missing, and common sample `CS1061` extension-method failures include missing-`using` hints such as `System.Linq` or `BenchmarkDotNet.Configs` when the compiler output points that way. -Validation follows the same philosophy: run `scripts/validate-skill-templates.ps1` locally for the fast feedback loop, and use `scripts/validate-skill-templates.ps1 -Full` when the slower DocFX regression suites are part of the gate. GitHub Actions runs full mode on pull requests as the safety net. The validator emits `[RUN]`, `[PASS]`, `[FAIL]`, `[WAIT]`, and `[SKIP]` progress lines so long phases show visible heartbeat feedback. It also checks skill frontmatter metadata such as per-skill `evals/evals.json` files, optional eval fixture paths declared through `files`, and the 1024-character YAML description limit; it does not replace the paired benchmark review workflow. +Validation follows the same philosophy: Bash and `pwsh` 7+ are both valid for local development, but any local PowerShell command or `.ps1` script must run through `pwsh`. Use `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` for the fast feedback loop, and `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Full` when the slower DocFX regression suites are part of the gate. GitHub Actions runs full mode on pull requests as the safety net and can keep workflow-specific shell choices. The validator emits `[RUN]`, `[PASS]`, `[FAIL]`, `[WAIT]`, and `[SKIP]` progress lines so long phases show visible heartbeat feedback. It also checks skill frontmatter metadata such as per-skill `evals/evals.json` files, optional eval fixture paths declared through `files`, and the 1024-character YAML description limit; it does not replace the paired benchmark review workflow. ## Install a skill @@ -113,7 +113,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark | [dotnet-new-lib-slnx](skills/dotnet-new-lib-slnx/SKILL.md) | Scaffold a new .NET NuGet library solution following codebeltnet engineering conventions. Dynamic defaults for TFM/repository metadata, latest-stable NuGet package resolution, tuning projects plus a tooling-based benchmark runner, TFM-aware test environments, strong-name signing, NuGet packaging, DocFX documentation, CI/CD pipeline, and code quality tooling. | | [dotnet-new-app-slnx](skills/dotnet-new-app-slnx/SKILL.md) | Scaffold a new .NET standalone application solution following codebeltnet engineering conventions. Supports Console, Web, and Worker host families with Startup or Minimal hosting patterns; Web expands into Empty Web, Web API, MVC, or Web App / Razor, plus functional tests and a simplified CI pipeline. | | [trunk-first-repo](skills/trunk-first-repo/SKILL.md) | Initialize a git repository following [scaled trunk-based development](https://trunkbaseddevelopment.com/#scaled-trunk-based-development). Seeds an empty `main` branch, creates a versioned feature branch (`v0.1.0/init`), confirms configured remotes in its post-init summary, and supports a guarded later `push remote ` mode that checks the feature-branch/empty-main state before pushing `main` ahead of the first feature branch so content still reaches main only through peer-reviewed pull requests. | -| [dotnet-strong-name-signing](skills/dotnet-strong-name-signing/SKILL.md) | Generate a strong name key (`.snk`) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or `sn.exe` required. Works in any terminal. Defaults to 1024-bit RSA (matching `sn.exe`), with 2048 and 4096 available as options. | +| [dotnet-strong-name-signing](skills/dotnet-strong-name-signing/SKILL.md) | Generate a strong name key (`.snk`) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or `sn.exe` required. Works in any terminal, using `pwsh` 7+ whenever local PowerShell syntax is needed. Defaults to 1024-bit RSA (matching `sn.exe`), with 2048 and 4096 available as options. | | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | @@ -505,11 +505,11 @@ Starting a new .NET solution "from scratch" usually means copying from your last ### Why dotnet-strong-name-signing? -Generating a `.snk` file traditionally requires `sn.exe`, which is only available in the Visual Studio Developer PowerShell — a common pain point for developers using VS Code, Rider, or plain terminals. This skill uses `RSACryptoServiceProvider` from the .NET runtime itself, so it works in **any PowerShell or terminal** without special tooling. +Generating a `.snk` file traditionally requires `sn.exe`, which is only available in the Visual Studio Developer PowerShell — a common pain point for developers using VS Code, Rider, or plain terminals. This skill uses `RSACryptoServiceProvider` from the .NET runtime itself, so it works in any terminal and, when local PowerShell syntax is preferred, runs cleanly through **`pwsh` 7+** without special tooling. -- **No `sn.exe` dependency** — uses pure .NET crypto available in any PowerShell session +- **No `sn.exe` dependency** — uses pure .NET crypto available in any `pwsh` 7+ session - **Matches `sn.exe` defaults** — 1024-bit RSA by default, with 2048 and 4096 as options -- **Cross-platform** — works on Windows, macOS, and Linux with PowerShell 7+ or .NET runtime +- **Cross-platform** — works on Windows, macOS, and Linux with `pwsh` 7+ or the .NET runtime - **Identity, not security** — [Microsoft's guidance](https://github.com/dotnet/runtime/blob/main/docs/project/strong-name-signing.md) is clear: strong names are about assembly identity, not cryptographic security ### Why trunk-first? From c51ea6e55372f4de8cb29a3d5e4845ccf8fc942e Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 03:31:03 +0200 Subject: [PATCH 32/38] =?UTF-8?q?=F0=9F=92=AC=20update=20changelog=20for?= =?UTF-8?q?=20pwsh=207+=20standardization?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the standardization of local PowerShell execution to pwsh 7+ as a change in the unreleased section. This clarifies the shift to pwsh as a requirement while preserving Bash and workflow-specific shell choices. --- CHANGELOG.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index caed95b..75ff568 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,8 @@ This is a minor release introducing evidence-driven discovery and performance-ex - Refactored `FORMS.md` for `dotnet-benchmark` aligned with the new discovery-focused workflow, reducing parameter collection friction by deferring implementation-tier choice to workflow inspection, - Enhanced eval coverage for `dotnet-benchmark` with test cases validating candidate discovery, evidence gathering, cost-signal analysis, implementation-comparison patterns, and runtime-selection decisions; includes fixture code supporting five representative benchmark scenarios, - Enhanced `check-benchmark-requirements.ps1` and new `validate-skill.ps1` tooling supporting discovery workflow validation and template-asset consistency checking, -- Updated README with discovery-focused `dotnet-benchmark` description and rationale emphasizing evidence-backed benchmarking over generic performance testing. +- Updated README with discovery-focused `dotnet-benchmark` description and rationale emphasizing evidence-backed benchmarking over generic performance testing, +- Standardized local PowerShell and `.ps1` execution on `pwsh` 7+ while preserving Bash and workflow-specific shell choices. ## [0.7.5] - 2026-07-15 From 5c6242900036b1dfd1ca48b5c330a2fff7df42c9 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 03:31:10 +0200 Subject: [PATCH 33/38] =?UTF-8?q?=F0=9F=93=9A=20update=20skill=20docs=20to?= =?UTF-8?q?=20reflect=20pwsh=207+=20standardization?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Propagate pwsh 7+ guidance across all skill documentation, evals, and references. Update command examples in SKILL.md files to use pwsh-specific invocations and remove legacy fallback language. Update evals to reflect current guidance. --- skills/dotnet-benchmark/SKILL.md | 18 +++++----- skills/dotnet-benchmark/evals/evals.json | 1 + .../references/benchmarkdotnet-essentials.md | 8 ++--- .../references/runner-preflight.md | 6 ++-- .../scripts/check-benchmark-requirements.ps1 | 2 +- .../scripts/validate-skill.ps1 | 36 ++++++++++--------- skills/dotnet-docfx-digest/SKILL.md | 2 +- skills/dotnet-docfx-digest/evals/evals.json | 6 ++-- .../dotnet-docfx-digest/references/scripts.md | 2 +- skills/dotnet-new-app-slnx/SKILL.md | 6 ++-- skills/dotnet-new-app-slnx/evals/evals.json | 3 +- skills/dotnet-new-app-slnx/references/app.md | 2 +- .../scripts/restore-missing-shared-assets.ps1 | 4 +-- skills/dotnet-new-lib-slnx/SKILL.md | 8 ++--- skills/dotnet-new-lib-slnx/evals/evals.json | 11 ++++++ .../dotnet-new-lib-slnx/references/library.md | 4 +-- .../scripts/restore-missing-shared-assets.ps1 | 4 +-- skills/dotnet-strong-name-signing/SKILL.md | 10 +++--- .../evals/evals.json | 3 +- skills/git-repo-digest/SKILL.md | 10 +++--- skills/git-repo-digest/evals/evals.json | 1 + 21 files changed, 84 insertions(+), 63 deletions(-) diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index 8e4399d..a7f770a 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -53,11 +53,11 @@ Yolo never authorizes a full performance run. Start the full benchmark only when Run the bundled read-only detector before changing files: -```powershell -pwsh -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot +```ps1 +pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot ``` -On Windows where PowerShell 7+ is unavailable, use `powershell` instead of `pwsh`. +If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. Also inspect applicable `AGENTS.md`, solution/project files, `Directory.Build.props`, `Directory.Packages.props`, existing `tuning/` and `tooling/` projects, and nearby benchmark styles. Reuse an existing runner and benchmark project when they fit. Read `references/onboarding.md` only when the detector finds missing or partial harness infrastructure. @@ -141,23 +141,23 @@ Semantic preflight Successful execution is not a correctness oracle. After semantic preflight, validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. The oracle should normally verify exact observable behavior rather than merely nonzero or approximate success unless that is the real domain contract. Then build the benchmark project in Release: -```powershell +```bash dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj ``` Before interpreting discovery or execution output, run the report-aware preflight for the exact class: -```powershell -pwsh -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType +```ps1 +pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ``` -On Windows where PowerShell 7+ is unavailable, use `powershell` instead of `pwsh`. +If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. If `reports.wouldSkipRequestedBenchmark` is true, the runner is intentionally filtering the type because a prior report exists. Report that as the validation outcome; do not claim the list/dry run exercised the class and do not modify code to force it through. A fresh full run and any report archive/replacement require explicit human direction. Verify runner discovery without measuring: -```powershell +```bash dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* ``` @@ -165,7 +165,7 @@ Compare the discovered method, job, and parameter matrix with the intended desig Unless execution is impossible or the user declines, run a dry execution smoke check and inspect all BenchmarkDotNet validation warnings. A dry run proves executable wiring and basic lifecycle, not performance: -```powershell +```bash dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* ``` diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json index 3a701a7..23d9e1f 100644 --- a/skills/dotnet-benchmark/evals/evals.json +++ b/skills/dotnet-benchmark/evals/evals.json @@ -7,6 +7,7 @@ "expected_output": "The agent detects repository state first and onboards only the missing tuning project and single tooling runner, preserves CPM/solution conventions, resolves current package versions dynamically, and does not fabricate a target type or benchmark method.", "expectations": [ "Runs or references scripts/check-benchmark-requirements.ps1 before changing harness files", + "Invokes scripts/check-benchmark-requirements.ps1 through `pwsh -NoProfile -File` and never falls back to `powershell` or `powershell.exe`", "Creates or proposes exactly one reusable tooling runner and a tuning benchmark project while preserving the detected solution and package-management conventions", "Resolves BenchmarkDotNet package versions dynamically from NuGet.org rather than copying hardcoded example versions", "Does not invent a SUT type, member, workload, or benchmark class when none was supplied", diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md index 9df3f79..f18d1f6 100644 --- a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -101,25 +101,25 @@ Run the bundled detector with `-BenchmarkType ` and ins Build: -```powershell +```bash dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj ``` List cases without measuring: -```powershell +```bash dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* ``` Dry execution smoke: -```powershell +```bash dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* ``` Full default run: -```powershell +```bash dotnet run -c Release --project tooling/{runner} -- --filter *{BenchmarkClass}* ``` diff --git a/skills/dotnet-benchmark/references/runner-preflight.md b/skills/dotnet-benchmark/references/runner-preflight.md index 3c52279..319c1ce 100644 --- a/skills/dotnet-benchmark/references/runner-preflight.md +++ b/skills/dotnet-benchmark/references/runner-preflight.md @@ -49,10 +49,12 @@ For example, `reports/tuning/Acme.Core.ParserBenchmark-report-github.md` causes 1. Build the benchmark project in Release. Stop on a compiler error; report it directly. 2. Run the detector with the intended benchmark type: - ```powershell - powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType + ```ps1 + pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ``` + If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. + 3. Inspect `runner.programPath`, `runner.skipBenchmarksWithReports`, `runner.usesSlimJob`, `runner.configuredRuntimes`, `reports.tuningPath`, `reports.matchingReportFiles`, and `reports.wouldSkipRequestedBenchmark`. 4. Verify that configured runtime jobs match the repository's supported TFMs. The TFM list is the normal variable; do not rewrite the runner merely to make it look different. 5. If `wouldSkipRequestedBenchmark` is true, explain that the existing report deliberately suppresses the type. Treat an empty list/dry/full invocation as accounted for and leave the benchmark class and runner unchanged. diff --git a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 index 5f83a51..beb20aa 100644 --- a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 +++ b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 @@ -17,7 +17,7 @@ Skips invoking dotnet --version. Intended for deterministic detector tests; normal skill runs should not use it. .EXAMPLE - powershell -NoProfile -ExecutionPolicy Bypass -File scripts/check-benchmark-requirements.ps1 -RepoRoot C:\src\myrepo + pwsh -NoProfile -File ./scripts/check-benchmark-requirements.ps1 -RepoRoot C:\src\myrepo #> [CmdletBinding()] param( diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 index dd72216..c53f21f 100644 --- a/skills/dotnet-benchmark/scripts/validate-skill.ps1 +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -216,22 +216,26 @@ try { $detectorPath = Join-Path $SkillRoot 'scripts/check-benchmark-requirements.ps1' if (Test-Path -LiteralPath $detectorPath) { try { - $pwshExe = if (Get-Command pwsh -ErrorAction SilentlyContinue) { 'pwsh' } else { 'powershell' } - $detected = & $pwshExe -NoProfile -ExecutionPolicy Bypass -File $detectorPath -RepoRoot $fixtureRoot -BenchmarkType Acme.Core.WidgetBenchmark -SkipSdkCheck | ConvertFrom-Json - if ($detected.solutionFormat -ne 'sln' -or -not $detected.centralPackageManagement -or -not $detected.centralizesBenchmarkConventions) { - Add-Failure 'Harness detector did not recognize the fixture solution, CPM, and centralized conventions' - } - if ($detected.sdk.status -ne 'skipped') { - Add-Failure 'Harness detector did not report the intentional skipped SDK probe distinctly' - } - if ($detected.benchmarkProjects.Count -ne 1 -or $detected.runner.name -ne 'bdn-runner' -or -not $detected.harnessReady) { - Add-Failure 'Harness detector did not recognize the existing benchmark project and runner' - } - if (-not $detected.runner.skipBenchmarksWithReports -or -not $detected.runner.usesSlimJob -or @($detected.runner.configuredRuntimes) -notcontains 'CoreRuntime.Core90') { - Add-Failure 'Harness detector did not recognize report skipping, the Slim job, and configured runtime' - } - if (-not $detected.reports.wouldSkipRequestedBenchmark -or @($detected.reports.matchingReportFiles).Count -ne 1) { - Add-Failure 'Harness detector did not identify the matching report that suppresses the requested benchmark type' + $pwshExe = Get-Command pwsh -ErrorAction SilentlyContinue + if ($null -eq $pwshExe) { + Add-Failure 'Harness detector validation requires pwsh 7+; do not fall back to legacy Windows PowerShell.' + } else { + $detected = & $pwshExe.Source -NoProfile -File $detectorPath -RepoRoot $fixtureRoot -BenchmarkType Acme.Core.WidgetBenchmark -SkipSdkCheck | ConvertFrom-Json + if ($detected.solutionFormat -ne 'sln' -or -not $detected.centralPackageManagement -or -not $detected.centralizesBenchmarkConventions) { + Add-Failure 'Harness detector did not recognize the fixture solution, CPM, and centralized conventions' + } + if ($detected.sdk.status -ne 'skipped') { + Add-Failure 'Harness detector did not report the intentional skipped SDK probe distinctly' + } + if ($detected.benchmarkProjects.Count -ne 1 -or $detected.runner.name -ne 'bdn-runner' -or -not $detected.harnessReady) { + Add-Failure 'Harness detector did not recognize the existing benchmark project and runner' + } + if (-not $detected.runner.skipBenchmarksWithReports -or -not $detected.runner.usesSlimJob -or @($detected.runner.configuredRuntimes) -notcontains 'CoreRuntime.Core90') { + Add-Failure 'Harness detector did not recognize report skipping, the Slim job, and configured runtime' + } + if (-not $detected.reports.wouldSkipRequestedBenchmark -or @($detected.reports.matchingReportFiles).Count -ne 1) { + Add-Failure 'Harness detector did not identify the matching report that suppresses the requested benchmark type' + } } } catch { Add-Failure "Harness detector failed on the deterministic fixture: $($_.Exception.Message)" diff --git a/skills/dotnet-docfx-digest/SKILL.md b/skills/dotnet-docfx-digest/SKILL.md index 2f8623f..2f0dbf0 100644 --- a/skills/dotnet-docfx-digest/SKILL.md +++ b/skills/dotnet-docfx-digest/SKILL.md @@ -88,7 +88,7 @@ Read the reflection-backed packets from that manifest or from `scope.packets`. I - Valid BOM-less UTF-8 is compliant. The validator must not emit `ENCODING_BOM_MISSING` (that diagnostic is intentionally unsupported), and an audit must not add/remove BOMs or normalize line endings solely for consistency. BOM presence has no documentation value; preserve the file's existing state and continue detecting actual `ENCODING_CORRUPTION` and `EXTENSION_TABLE_ENCODING` damage. - Final verification uses adaptive execution. In `auto`, machines with more than 8 available logical processors and more than 32 GiB available memory select `high-capacity`: DocFX verification runs concurrently in its isolated temp copy while the main lane builds/discovers API and then compiles samples, and MSBuild worker counts scale up to half the available processors (capped at 16). Smaller machines select `conservative` and keep the phases sequential with low worker counts. Override with `--execution-profile conservative|high-capacity` or `DOCFX_DIGEST_EXECUTION_PROFILE`. - Child processes time out after 30 minutes by default, configurable with `--process-timeout-minutes` or `DOCFX_DIGEST_PROCESS_TIMEOUT_MINUTES`. Use `host timeout >= process timeout + 5 minutes` (35 minutes for the default) so the validator can kill a timed-out child and return a deterministic diagnostic instead of being terminated by the caller first. -- Long-running API builds, sample compilation, and DocFX verification write progress to `stderr`: an initial `[ ]` line, a heartbeat every 10 seconds, and a final `[✓]` or `[x]` line. Heartbeats name the active phase, workload size, runner count, PID, elapsed time, time since the child last produced output, and its latest output line when available. Do not suppress `stderr` during normal interactive runs; `--json` remains a single parseable document on `stdout`. When PowerShell or another host makes heartbeat noise obscure JSON inspection, pass `--quiet` or `--no-heartbeat` to suppress start/heartbeat chatter while keeping final `[✓]`/`[x]` child-process markers. In high-capacity mode, concurrent phase heartbeats can interleave; use JSON stdout and the final per-phase markers as the authoritative result, not the visual order of progress lines. +- Long-running API builds, sample compilation, and DocFX verification write progress to `stderr`: an initial `[ ]` line, a heartbeat every 10 seconds, and a final `[✓]` or `[x]` line. Heartbeats name the active phase, workload size, runner count, PID, elapsed time, time since the child last produced output, and its latest output line when available. Do not suppress `stderr` during normal interactive runs; `--json` remains a single parseable document on `stdout`. When `pwsh` or another host makes heartbeat noise obscure JSON inspection, pass `--quiet` or `--no-heartbeat` to suppress start/heartbeat chatter while keeping final `[✓]`/`[x]` child-process markers. In high-capacity mode, concurrent phase heartbeats can interleave; use JSON stdout and the final per-phase markers as the authoritative result, not the visual order of progress lines. - When reporting adaptive execution, include the selected profile, processor and memory inputs, build/sample worker counts, timeout, concurrent/sequential choice, process counts, and phase timings from JSON. State that sample compilation follows API discovery because scoped references depend on the namespace-to-project map. Name CLI and environment overrides for profile, worker counts, and timeout when the user asks how to tune the run. - When reporting heartbeat behavior, state the complete contract: append-only `stderr`, no cursor-rewritten table, start and 10-second heartbeat events, final success/failure marker for all three long phases, latest non-empty child-output line when available, and plain redirected-log markers that do not depend on ANSI color. When reporting encoding safety, explicitly confirm the diff contains neither BOM-only nor line-ending-only changes. - Read `references/workflow.md` when you need the detailed targeted/audit workflows, namespace and example templates, the verification checklist, or the completion response shape. diff --git a/skills/dotnet-docfx-digest/evals/evals.json b/skills/dotnet-docfx-digest/evals/evals.json index d9b8fb0..6e6c03a 100644 --- a/skills/dotnet-docfx-digest/evals/evals.json +++ b/skills/dotnet-docfx-digest/evals/evals.json @@ -1182,12 +1182,12 @@ }, { "id": 113, - "prompt": "Use dotnet-docfx-digest on a repository whose `.docfx/docfx.json` has `build.overwrite` set to `api/namespaces/**.md` and `api/types/**/*.md`, and whose samples include one compile failure from calling LINQ `Select` without `using System.Linq;`. One type-page example constructs and returns the documented type from an if/else branch but never prints, configures, invokes, or passes the value to another API. Run validation with JSON, keep the output easy to inspect in PowerShell, and repair the docs.", - "expected_output": "Agent treats the near-miss DocFX glob as a literal config mismatch, recognizes the exact expected `api/namespaces/**/*.md` pattern, uses quiet/no-heartbeat validation when terminal noise would obscure JSON, fixes the no-observable example by showing a reader-visible result, and maps the CS1061 sample compile failure to a missing extension-method using such as `using System.Linq;` before rerunning validation.", + "prompt": "Use dotnet-docfx-digest on a repository whose `.docfx/docfx.json` has `build.overwrite` set to `api/namespaces/**.md` and `api/types/**/*.md`, and whose samples include one compile failure from calling LINQ `Select` without `using System.Linq;`. One type-page example constructs and returns the documented type from an if/else branch but never prints, configures, invokes, or passes the value to another API. Run validation with JSON, keep the output easy to inspect in `pwsh`, and repair the docs.", + "expected_output": "Agent treats the near-miss DocFX glob as a literal config mismatch, recognizes the exact expected `api/namespaces/**/*.md` pattern, uses quiet/no-heartbeat validation when `pwsh` or another host would obscure JSON, fixes the no-observable example by showing a reader-visible result, and maps the CS1061 sample compile failure to a missing extension-method using such as `using System.Linq;` before rerunning validation.", "expectations": [ "Recognizes that `api/namespaces/**.md` is a near-miss and does not satisfy the expected literal `api/namespaces/**/*.md` overwrite pattern", "Repairs `build.overwrite` without widening the glob to `api/**/*.md` or moving namespace/type overwrite Markdown into `build.content`", - "Uses `--quiet` or `--no-heartbeat` when progress stderr makes JSON validation output hard to inspect interactively", + "Uses `--quiet` or `--no-heartbeat` when progress stderr makes JSON validation output hard to inspect interactively in `pwsh` or another host", "Treats `EXAMPLE_NO_OBSERVABLE_OUTCOME` as a failing example even when the code has a real branch or returns the documented type", "Repairs the example by producing a consumer-visible result such as printed data, configured state, an invoked member result, or a real consumer API call", "Interprets CS1061 extension-method failures as likely missing using directives when source/API evidence supports it", diff --git a/skills/dotnet-docfx-digest/references/scripts.md b/skills/dotnet-docfx-digest/references/scripts.md index 4faa64f..e0147ec 100644 --- a/skills/dotnet-docfx-digest/references/scripts.md +++ b/skills/dotnet-docfx-digest/references/scripts.md @@ -59,7 +59,7 @@ Final verification is machine-adaptive. `--execution-profile auto` is the defaul External child processes have a 30-minute default timeout. Override it with `--process-timeout-minutes` or `DOCFX_DIGEST_PROCESS_TIMEOUT_MINUTES` (1-180). The calling agent/tool must use `outer command timeout >= process timeout + 5 minutes`; otherwise the caller can terminate the validator before it kills the child process and emits `DOCFX_BUILD_FAILED` or another deterministic diagnostic. With the default child timeout, use an outer timeout of at least 35 minutes. -API builds, sample graph builds, and DocFX verification emit append-only progress events to `stderr`. The validator writes `[ ]` when a child starts, another `[ ]` heartbeat every 10 seconds, and `[✓]` or `[x]` when it completes. Each heartbeat includes the phase, project/sample workload, runner count, child PID, elapsed time, time since the child last wrote output, and the latest non-empty output line when available. This makes a quiet restore/build distinguishable from a stale process without corrupting `--json`, which remains exclusively on `stdout`. Redirected logs use plain markers; interactive terminals color the completion marker when ANSI color is available. In `high-capacity`, heartbeat lines from concurrent phases may be interleaved; parse stdout JSON and final phase markers as authoritative rather than inferring order from the visual log. Use `--quiet` or `--no-heartbeat` when a host such as PowerShell promotes heartbeat `stderr` noise into distracting terminal records; quiet mode suppresses start/heartbeat chatter and project packet progress lines while preserving final `[✓]`/`[x]` child-process markers. +API builds, sample graph builds, and DocFX verification emit append-only progress events to `stderr`. The validator writes `[ ]` when a child starts, another `[ ]` heartbeat every 10 seconds, and `[✓]` or `[x]` when it completes. Each heartbeat includes the phase, project/sample workload, runner count, child PID, elapsed time, time since the child last wrote output, and the latest non-empty output line when available. This makes a quiet restore/build distinguishable from a stale process without corrupting `--json`, which remains exclusively on `stdout`. Redirected logs use plain markers; interactive terminals color the completion marker when ANSI color is available. In `high-capacity`, heartbeat lines from concurrent phases may be interleaved; parse stdout JSON and final phase markers as authoritative rather than inferring order from the visual log. Use `--quiet` or `--no-heartbeat` when a host such as `pwsh` promotes heartbeat `stderr` noise into distracting terminal records; quiet mode suppresses start/heartbeat chatter and project packet progress lines while preserving final `[✓]`/`[x]` child-process markers. For Codebelt strong-name signed repositories, build and sample paths use the root `.snk` when present and automatically pass `-p:SkipSignAssembly=true` when no root `.snk` exists, so missing local signing keys do not masquerade as documentation failures. diff --git a/skills/dotnet-new-app-slnx/SKILL.md b/skills/dotnet-new-app-slnx/SKILL.md index 29ff373..37bb76a 100644 --- a/skills/dotnet-new-app-slnx/SKILL.md +++ b/skills/dotnet-new-app-slnx/SKILL.md @@ -89,7 +89,7 @@ Read `references/app.md` for the app-specific project structure, template file m Before writing `Directory.Packages.props`, resolve every `*_VERSION` placeholder in that file to the latest stable listed version for its matching package ID on NuGet.org. -When PowerShell is available, prefer the deterministic helper in `scripts/resolve-package-versions.ps1` over manual lookup. By default it resolves placeholders from this skill's own `assets/shared/Directory.Packages.props`, so a normal scaffold run only needs `-TargetFramework`. Treat its JSON output as the source of truth for package placeholders. +When `pwsh` 7+ is available, prefer the deterministic helper in `scripts/resolve-package-versions.ps1` over manual lookup. Run it as `pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework `. By default it resolves placeholders from this skill's own `assets/shared/Directory.Packages.props`, so a normal scaffold run only needs `-TargetFramework`. Treat its JSON output as the source of truth for package placeholders, and do not substitute `powershell` or `powershell.exe` for this helper. - Use the NuGet V3 service index at `https://api.nuget.org/v3/index.json` to discover the package metadata endpoints - Prefer registration metadata so you can ignore unlisted versions and prerelease builds @@ -153,7 +153,7 @@ Copy every file from `assets/shared/` to the project root, preserving directory Do this as a recursive, dotfile-aware copy. Hidden folders and files under `assets/shared/` are part of the scaffold and must not be skipped. In particular, copy `assets/shared/.bot/README.md` as a real file in the generated repo; do not replace it with a synthetic `.gitkeep` or placeholder note. -**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If PowerShell is unavailable, use the raw base URL in the **Upstream Source** table above to download each missing file manually. If upstream fetch fails, halt and report — do not substitute placeholders. +**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. If upstream fetch fails, halt and report — do not substitute placeholders. Do not selectively copy only "key" shared files. The intended output includes the complete shared asset inventory, including `.gitignore`, `.gitattributes`, `AGENTS.md`, `CHANGELOG.md`, `.github/`, and `.bot/`, in addition to the build and package-management files. @@ -193,7 +193,7 @@ After generating, verify: - [ ] `.slnx` references all generated src/ and test/ projects - [ ] The generated solution filename is `{SOLUTION_NAME}.slnx` with the original user-facing casing preserved - [ ] Every file listed in `assets/shared.manifest.json` exists in the generated repo at its declared relative path (this covers all dotfiles and dotfolders) -- [ ] If any manifest entry was absent from the installed skill copy, `scripts/restore-missing-shared-assets.ps1` was run (or files fetched manually from the upstream raw URL) — not diagnosed iteratively +- [ ] If any manifest entry was absent from the installed skill copy, `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` was run — not diagnosed iteratively - [ ] `Directory.Packages.props` lists all `` packages used in the solution (including host-type-specific packages) - [ ] `Directory.Packages.props` contains concrete version numbers with no unresolved `*_VERSION` placeholders - [ ] No generated `.csproj` file or `Directory.Build.props` contains ad-hoc inline `Version=` attributes for packages that are supposed to be centrally managed by `Directory.Packages.props` diff --git a/skills/dotnet-new-app-slnx/evals/evals.json b/skills/dotnet-new-app-slnx/evals/evals.json index e3f8a9b..234c787 100644 --- a/skills/dotnet-new-app-slnx/evals/evals.json +++ b/skills/dotnet-new-app-slnx/evals/evals.json @@ -36,7 +36,8 @@ "Resolves package versions from NuGet instead of carrying hardcoded examples from a previous scaffold", "Copies the complete shared asset inventory from assets/shared instead of cherry-picking only a subset of governance or dotfiles", "Keeps TargetFramework centralized in the generated root Directory.Build.props instead of adding TargetFramework directly to generated app or test csproj files", - "Produces valid XML in generated root build props files before the first build rather than relying on mid-run repair" + "Produces valid XML in generated root build props files before the first build rather than relying on mid-run repair", + "If it invokes scripts/resolve-package-versions.ps1, it does so through `pwsh -NoProfile -File` and never through `powershell` or `powershell.exe`" ] }, { diff --git a/skills/dotnet-new-app-slnx/references/app.md b/skills/dotnet-new-app-slnx/references/app.md index b77defd..556ef7f 100644 --- a/skills/dotnet-new-app-slnx/references/app.md +++ b/skills/dotnet-new-app-slnx/references/app.md @@ -152,7 +152,7 @@ Resolve each package-specific `*_VERSION` placeholder in `Directory.Packages.pro Keep target-framework selection centralized too: the generated root `Directory.Build.props` owns `{TARGET_FRAMEWORK}` for source and test projects. Do **not** duplicate `` inside the generated app or test `.csproj` files as a workaround. -When PowerShell is available, prefer `scripts/resolve-package-versions.ps1` to produce the package placeholder map for this skill. The script defaults to this skill's own `assets/shared/Directory.Packages.props`, so the normal path only needs `{TARGET_FRAMEWORK}`. Its output should drive the final substitutions instead of remembered version numbers. +When `pwsh` 7+ is available, prefer `pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework {TARGET_FRAMEWORK}` to produce the package placeholder map for this skill. The script defaults to this skill's own `assets/shared/Directory.Packages.props`, so the normal path only needs `{TARGET_FRAMEWORK}`. Its output should drive the final substitutions instead of remembered version numbers, and legacy Windows PowerShell is not a substitute runtime for this helper. For framework-aligned ASP.NET packages, keep the selected target framework major in mind when resolving the final version: diff --git a/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 b/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 index 3f670ac..96c3e4e 100644 --- a/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 +++ b/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 @@ -17,10 +17,10 @@ .EXAMPLE # Restore missing files into the installed skill copy - scripts/restore-missing-shared-assets.ps1 + pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 # Preview what is missing without restoring - scripts/restore-missing-shared-assets.ps1 -DryRun + pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 -DryRun #> [CmdletBinding()] param( diff --git a/skills/dotnet-new-lib-slnx/SKILL.md b/skills/dotnet-new-lib-slnx/SKILL.md index b2ce853..921320d 100644 --- a/skills/dotnet-new-lib-slnx/SKILL.md +++ b/skills/dotnet-new-lib-slnx/SKILL.md @@ -102,15 +102,15 @@ Copy every file from `assets/shared/` to the project root, preserving directory Do this as a recursive, dotfile-aware copy. Hidden folders and files under `assets/shared/` are part of the scaffold and must not be skipped. In particular, copy `assets/shared/.bot/README.md` as a real file in the generated repo; do not replace it with a synthetic `.gitkeep` or placeholder note. -**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If PowerShell is unavailable, use the raw base URL in the **Upstream Source** table above to download each missing file manually. If upstream fetch fails, halt and report — do not substitute placeholders. +**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. If upstream fetch fails, halt and report — do not substitute placeholders. Preserve UTF-8 when reading, copying, and writing text files. Do not transcode templates to ANSI, OEM, Windows-1252, or any system-default code page during generation. The shared `.editorconfig` in the scaffold declares `charset = utf-8`, and generated text files should match it from the start. When a text file does not need substitutions, prefer a byte-preserving file copy instead of read/transform/write. -When a text file does need substitutions, use explicit UTF-8 APIs end-to-end. In PowerShell, prefer .NET file APIs with an explicit `UTF8Encoding` instance rather than locale-dependent text cmdlets. For example: +When a text file does need substitutions, use explicit UTF-8 APIs end-to-end. In `pwsh` 7+, prefer .NET file APIs with an explicit `UTF8Encoding` instance rather than locale-dependent text cmdlets. For example: -```powershell +```ps1 $utf8NoBom = [System.Text.UTF8Encoding]::new($false) $content = [System.IO.File]::ReadAllText($src, $utf8NoBom) $updated = Apply-Replacements -Content $content -Map $replaceMap @@ -189,7 +189,7 @@ After generating, verify: - [ ] `.bot/` folder exists and is listed in `.gitignore` - [ ] `.bot/README.md` exists in the generated repo and came from the shared asset template, not from a synthetic `.gitkeep` fallback - [ ] Every file listed in `assets/shared.manifest.json` exists in the generated repo at its declared relative path (this covers all dotfiles and dotfolders) -- [ ] If any manifest entry was absent from the installed skill copy, `scripts/restore-missing-shared-assets.ps1` was run (or files fetched manually from the upstream raw URL) — not diagnosed iteratively +- [ ] If any manifest entry was absent from the installed skill copy, `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` was run — not diagnosed iteratively - [ ] No manifest entries were silently skipped; if the restore script reported failures, generation was halted rather than continuing with incomplete shared assets - [ ] `.github/dependabot.yml` watches the repo root so central NuGet package management stays current after scaffolding diff --git a/skills/dotnet-new-lib-slnx/evals/evals.json b/skills/dotnet-new-lib-slnx/evals/evals.json index f1b8079..8d768fa 100644 --- a/skills/dotnet-new-lib-slnx/evals/evals.json +++ b/skills/dotnet-new-lib-slnx/evals/evals.json @@ -55,6 +55,17 @@ "Continues to the summary instead of asking a second root_namespace clarification question", "Uses native structured inputs when available and a one-field-at-a-time plain-text fallback when they are not" ] + }, + { + "id": 6, + "prompt": "The installed dotnet-new-lib-slnx skill copy is missing shared dotfiles from assets/shared.manifest.json. Restore the missing shared assets before continuing the scaffold.", + "expected_output": "The agent immediately uses the bundled restore helper through `pwsh -NoProfile -File`, treats missing `pwsh` 7+ as a blocker instead of falling back to legacy Windows PowerShell, and avoids iterative file-by-file diagnosis.", + "expectations": [ + "Runs `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` when shared assets from assets/shared.manifest.json are missing", + "Does not invoke `powershell` or `powershell.exe` as a fallback runtime for the restore helper", + "Treats missing `pwsh` 7+ as a prerequisite blocker instead of silently changing runtimes", + "Restores the manifest-defined shared asset set in one step rather than diagnosing missing dotfiles iteratively" + ] } ] } diff --git a/skills/dotnet-new-lib-slnx/references/library.md b/skills/dotnet-new-lib-slnx/references/library.md index 7f4b464..98b0d77 100644 --- a/skills/dotnet-new-lib-slnx/references/library.md +++ b/skills/dotnet-new-lib-slnx/references/library.md @@ -90,9 +90,9 @@ If an installer path omits dot-prefixed files from the source tree, treat that a - Prefer byte-preserving copy for files that do not need substitutions - Preserve the source template's BOM policy by default; do not add a UTF-8 BOM unless the source had one or the target format explicitly needs it -Recommended PowerShell approach for rewritten templates: +Recommended `pwsh` 7+ approach for rewritten templates: -```powershell +```ps1 $utf8NoBom = [System.Text.UTF8Encoding]::new($false) $content = [System.IO.File]::ReadAllText($src, $utf8NoBom) $updated = Apply-Replacements -Content $content -Map $replaceMap diff --git a/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 b/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 index c634f40..17e0353 100644 --- a/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 +++ b/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 @@ -17,10 +17,10 @@ .EXAMPLE # Restore missing files into the installed skill copy - scripts/restore-missing-shared-assets.ps1 + pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 # Preview what is missing without restoring - scripts/restore-missing-shared-assets.ps1 -DryRun + pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 -DryRun #> [CmdletBinding()] param( diff --git a/skills/dotnet-strong-name-signing/SKILL.md b/skills/dotnet-strong-name-signing/SKILL.md index 651899c..9bbc225 100644 --- a/skills/dotnet-strong-name-signing/SKILL.md +++ b/skills/dotnet-strong-name-signing/SKILL.md @@ -8,7 +8,7 @@ description: > ![Strong Name Signing](assets/hero.jpg) -Generate a strong name key pair (`.snk` file) for signing .NET assemblies. Uses the .NET runtime's built-in `RSACryptoServiceProvider` instead of `sn.exe`, so it works in **any PowerShell or terminal** — no Visual Studio Developer Command Prompt needed. +Generate a strong name key pair (`.snk` file) for signing .NET assemblies. Uses the .NET runtime's built-in `RSACryptoServiceProvider` instead of `sn.exe`, so it works in any terminal and, when local PowerShell syntax is preferred, runs through **`pwsh` 7+** — no Visual Studio Developer Command Prompt needed. ## Why this matters @@ -24,9 +24,9 @@ Read `FORMS.md`, compute the defaults silently, and present a single summary for ### Step 2: Generate the Key File -Run this PowerShell script in the target directory: +Run this command block in a `pwsh` 7+ session in the target directory: -```powershell +```ps1 $rsa = New-Object System.Security.Cryptography.RSACryptoServiceProvider({KEY_SIZE}) $keyBlob = $rsa.ExportCspBlob($true) [System.IO.File]::WriteAllBytes("{OUTPUT_PATH}", $keyBlob) @@ -43,7 +43,7 @@ The `ExportCspBlob($true)` method exports the full key pair (public + private) i After generating the file, verify it exists and report: -```powershell +```ps1 $snkFile = Get-Item "{OUTPUT_PATH}" Write-Host "✅ Strong name key generated" Write-Host "" @@ -87,4 +87,4 @@ Remind the user about `.snk` file handling: ### Cross-Platform -This approach works on Windows, macOS, and Linux — anywhere the .NET runtime or PowerShell 7+ is installed. The `RSACryptoServiceProvider` class is available in both .NET Framework and .NET (Core). +This approach works on Windows, macOS, and Linux — anywhere the .NET runtime or `pwsh` 7+ is installed. The `RSACryptoServiceProvider` class is available in both .NET Framework and .NET (Core). diff --git a/skills/dotnet-strong-name-signing/evals/evals.json b/skills/dotnet-strong-name-signing/evals/evals.json index 258261d..17cf2ec 100644 --- a/skills/dotnet-strong-name-signing/evals/evals.json +++ b/skills/dotnet-strong-name-signing/evals/evals.json @@ -8,7 +8,8 @@ "expectations": [ "Computes defaults silently before asking for confirmation", "Defaults to a 1024-bit RSA key", - "Uses the RSACryptoServiceProvider ExportCspBlob flow" + "Uses the RSACryptoServiceProvider ExportCspBlob flow", + "Uses `pwsh` 7+ when it presents or executes the PowerShell command block locally" ] }, { diff --git a/skills/git-repo-digest/SKILL.md b/skills/git-repo-digest/SKILL.md index e3ef046..e1dd921 100644 --- a/skills/git-repo-digest/SKILL.md +++ b/skills/git-repo-digest/SKILL.md @@ -58,7 +58,7 @@ scripts/digest.cs Run it with `dotnet run --file` so it is not confused with a nearby project file: -```powershell +```bash dotnet run --file /scripts/digest.cs -- --repo-url --output-root ``` @@ -87,7 +87,7 @@ The runner requires the .NET 10 SDK or newer and `git`. It performs one shallow The runner also supports deterministic result validation for authored workspaces: -```powershell +```bash dotnet run --file /scripts/digest.cs -- --validate-results --workspace ``` @@ -204,7 +204,7 @@ Choose the workspace mode from the user's input, not from folders you happen to For a fresh run, execute the bundled runner from this skill: -```powershell +```bash dotnet run --file /scripts/digest.cs -- --repo-url --output-root [--external-repo-url ]... ``` @@ -320,7 +320,7 @@ If any package digest is missing, decide from the manifest: Run the deterministic result validator and require a pass before reporting completion: -```powershell +```bash dotnet run --file /scripts/digest.cs -- --validate-results --workspace ``` @@ -417,7 +417,7 @@ Before finishing, verify: Use targeted searches instead of rereading everything: -```powershell +```bash rg -n "TODO|TBD|confidence|citation|analysis notes|I cannot|as an AI|\\.\\.\\.|placeholder" /result rg -n "Write a source-grounded|Write a short lede" /result rg -n "Greeting|MessageService|Hello World|Hello, World|Hello from DI|\\bOK\\b|GenerateReport|CreateService|BuildHost|FormatInvoice|CreateClient|SampleMiddleware|MyService|IMyService|MyRepository|FakeRepository|MyController|SampleController|\\bFoo\\b|\\bBar\\b|\\bDummy\\b" /result diff --git a/skills/git-repo-digest/evals/evals.json b/skills/git-repo-digest/evals/evals.json index fde4051..94cd957 100644 --- a/skills/git-repo-digest/evals/evals.json +++ b/skills/git-repo-digest/evals/evals.json @@ -27,6 +27,7 @@ "Uses .bot/digests under the active workspace as the output root because the user supplied no output path", "Does not ask before using the well-defined .bot/digests output-root default", "Runs dotnet run --file /scripts/digest.cs with --repo-url and --output-root for the .bot/digests default", + "Keeps the runner invocation shell-agnostic instead of wrapping `dotnet run --file` in legacy Windows PowerShell", "Passes the full repository URL, not a slug or partial reference", "Does not ask the user for repo-id, run-id, output-root, or result directory", "Uses a run-id formatted yyyyMMdd-HHmmssZ so default workspaces sort chronologically", From 9eab0ad71cbd73cc310ce39b81f1bc0dd24cc729 Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 04:49:03 +0200 Subject: [PATCH 34/38] =?UTF-8?q?=E2=99=BB=EF=B8=8F=20simplify=20powershel?= =?UTF-8?q?l=20policy=20enforcement?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove overly aggressive validation rules for pwsh standardization. Revert code fence labels from ps1 to powershell for better readability. Simplify validator patterns and remove pwsh-specific eval expectations that over-constrained the agent workflow. --- AGENTS.md | 6 +- CONTRIBUTING.md | 6 +- scripts/validate-skill-templates.ps1 | 88 +++++-------------- skills/dotnet-benchmark/SKILL.md | 14 +-- skills/dotnet-benchmark/evals/evals.json | 1 - .../references/benchmarkdotnet-essentials.md | 8 +- .../references/runner-preflight.md | 4 +- .../scripts/check-benchmark-requirements.ps1 | 2 +- skills/dotnet-docfx-digest/SKILL.md | 2 +- skills/dotnet-docfx-digest/evals/evals.json | 6 +- .../dotnet-docfx-digest/references/scripts.md | 2 +- skills/dotnet-new-app-slnx/SKILL.md | 6 +- skills/dotnet-new-app-slnx/evals/evals.json | 3 +- skills/dotnet-new-app-slnx/references/app.md | 2 +- .../scripts/restore-missing-shared-assets.ps1 | 4 +- skills/dotnet-new-lib-slnx/SKILL.md | 8 +- skills/dotnet-new-lib-slnx/evals/evals.json | 11 --- .../dotnet-new-lib-slnx/references/library.md | 4 +- .../scripts/restore-missing-shared-assets.ps1 | 4 +- skills/dotnet-strong-name-signing/SKILL.md | 6 +- .../evals/evals.json | 3 +- skills/git-repo-digest/SKILL.md | 10 +-- skills/git-repo-digest/evals/evals.json | 1 - 23 files changed, 71 insertions(+), 130 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 92f00eb..3f0ae76 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,16 +74,16 @@ Repo-managed skills live in four places that must stay in sync: Changes often start in `~/.claude/skills//`, then get mirrored to the repo and the other local installs: - **Claude local → repo** (persist changes to source control): - ```ps1 + ```powershell Copy-Item "$HOME/.claude/skills//" "skills//" -Force ``` - **Claude local → agent installs** (keep `~/.agents` and Gemini current): - ```ps1 + ```powershell Copy-Item "$HOME/.claude/skills//" "$HOME/.agents/skills//" -Force Copy-Item "$HOME/.claude/skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force ``` - **Repo → local installs** (after pulling changes or cloning fresh): - ```ps1 + ```powershell Copy-Item "skills//" "$HOME/.claude/skills//" -Force Copy-Item "skills//" "$HOME/.agents/skills//" -Force Copy-Item "skills//" "$HOME/.gemini/antigravity-cli/skills//" -Force diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f4be691..bee7395 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -88,7 +88,7 @@ Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and c Run evals from a temp workspace, not from this repository: -```ps1 +```powershell $workspace = Join-Path $env:TEMP '-workspace' ``` @@ -116,7 +116,7 @@ When a skill needs defaults for versions, paths, repository names, or support wi Use the repo validation harness before submitting scaffold or template changes: -```ps1 +```console pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 ``` @@ -124,7 +124,7 @@ Run the validator locally first for the fastest feedback loop. Bash and `pwsh` 7 To compare a change against the initial imported version, run the same harness against a git ref: -```ps1 +```console pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD ``` diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index 18a43aa..b25a046 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -129,11 +129,7 @@ function Get-LocalShellPolicyFindingsFromContentItems { param([object[]]$Items) $legacyShell = 'power' + 'shell' - $productName = 'Power' + 'Shell' $legacyShellPattern = [regex]::Escape($legacyShell) - $productNamePattern = [regex]::Escape($productName) - $fence = [string]([char]96) * 3 - $fencePattern = [regex]::Escape($fence) $rules = @( [pscustomobject]@{ @@ -144,34 +140,6 @@ function Get-LocalShellPolicyFindingsFromContentItems { Pattern = "(?i)\bshell:\s*$legacyShellPattern(?:\.exe)?\b" Message = 'Workflow steps that explicitly choose a `pwsh`-style shell must use `pwsh`.' } - [pscustomobject]@{ - Pattern = "(?i)\buse\b.*\b$legacyShellPattern(?:\.exe)?\b.*\binstead of\b.*\bpwsh\b" - Message = 'Do not tell agents to fall back from `pwsh` to the legacy executable.' - } - [pscustomobject]@{ - Pattern = "(?i)\bwhen\s+$productNamePattern\s+is\s+available\b" - Message = 'Local guidance must name `pwsh` 7+ explicitly instead of generic shell availability.' - } - [pscustomobject]@{ - Pattern = "(?i)\bif\s+$productNamePattern\s+is\s+unavailable\b" - Message = 'Missing `pwsh` 7+ must be reported as the blocker for required local `.ps1` execution.' - } - [pscustomobject]@{ - Pattern = "(?i)\brun this\s+$productNamePattern\s+script\b" - Message = 'Local script instructions must name `pwsh` 7+ explicitly.' - } - [pscustomobject]@{ - Pattern = "(?i)\b$productNamePattern\s+session\b" - Message = 'Refer to a `pwsh` session for local execution guidance.' - } - [pscustomobject]@{ - Pattern = "(?i)\b$productNamePattern\s+or\s+terminal\b" - Message = 'Prefer `pwsh` 7+ or shell-agnostic wording for local execution guidance.' - } - [pscustomobject]@{ - Pattern = "(?i)^$fencePattern$legacyShellPattern\s*$" - Message = 'Use `ps1` for `pwsh` syntax fences or a shell-agnostic fence such as `bash` for generic commands.' - } ) $findings = [System.Collections.Generic.List[object]]::new() @@ -203,17 +171,6 @@ function Get-LocalShellPolicyFindingsFromContentItems { } } - if ($null -eq $message) { - $mentionsPs1 = $line -match '(?i)\.ps1\b' - $mentionsPwsh = $line -match '(?i)\bpwsh\b' - $isPs1CommandExample = $line -match '^\s*(?:\./|\.\\)?[A-Za-z0-9_./\\-]+\.ps1(?:\s|$)' - $isPs1RunInstruction = $line -match '(?i)\b(?:run|invoke|execute)\b.*?\.ps1\b' - - if ($mentionsPs1 -and -not $mentionsPwsh -and ($isPs1CommandExample -or $isPs1RunInstruction)) { - $message = 'Local `.ps1` instructions must include `pwsh -NoProfile -File` explicitly.' - } - } - if ($null -ne $message) { $findings.Add([pscustomobject]@{ Path = $relativePath @@ -684,7 +641,7 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills keep YAML } } -Add-ValidationResult -Results $results -Name 'Active local shell guidance requires pwsh for local `.ps1` and shell execution' -Action { +Add-ValidationResult -Results $results -Name 'Active local shell guidance rejects only legacy PowerShell executable use' -Action { $findings = @(Get-LocalShellPolicyFindings -RepoRoot $repoRoot -GitRef $Ref) if ($findings.Count -gt 0) { @@ -694,14 +651,13 @@ Add-ValidationResult -Results $results -Name 'Active local shell guidance requir } ) - throw ("Local `.ps1` and shell execution requires `pwsh` 7+; update these lines:`n" + ($details -join "`n")) + throw ("Legacy `powershell` executable usage is still present; update these lines:`n" + ($details -join "`n")) } } -Add-ValidationResult -Results $results -Name 'Local shell policy scanner rejects legacy runtime guidance and allows approved alternatives' -Action { +Add-ValidationResult -Results $results -Name 'Local shell policy scanner rejects legacy executable invocations and allows valid terminology' -Action { $legacyShell = 'power' + 'shell' $legacyExe = $legacyShell + '.exe' - $productName = 'Power' + 'Shell' $fence = [string]([char]96) * 3 $cases = @( @@ -720,7 +676,7 @@ Add-ValidationResult -Results $results -Name 'Local shell policy scanner rejects [pscustomobject]@{ Name = 'legacy executable command' Path = 'skills/example/case-03.md' - Content = "$legacyExe -File ./scripts/example.ps1" + Content = "$legacyExe -Command Get-ChildItem" ExpectViolation = $true } [pscustomobject]@{ @@ -748,9 +704,15 @@ Add-ValidationResult -Results $results -Name 'Local shell policy scanner rejects ExpectViolation = $false } [pscustomobject]@{ - Name = 'ps1 fence' + Name = 'powershell fence' Path = 'skills/example/case-08.md' - Content = $fence + 'ps1' + [Environment]::NewLine + '$value = 1' + [Environment]::NewLine + $fence + Content = $fence + 'powershell' + [Environment]::NewLine + '$value = 1' + [Environment]::NewLine + $fence + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'PowerShell session prose' + Path = 'skills/example/case-09.md' + Content = 'Use a PowerShell session if that is the host the user already chose.' ExpectViolation = $false } [pscustomobject]@{ @@ -765,15 +727,9 @@ Add-ValidationResult -Results $results -Name 'Local shell policy scanner rejects Content = "$legacyShell -File ./scripts/example.ps1" ExpectViolation = $false } - [pscustomobject]@{ - Name = 'generic availability phrasing' - Path = 'skills/example/case-11.md' - Content = "When $productName is available, prefer the helper." - ExpectViolation = $true - } [pscustomobject]@{ Name = 'legacy workflow shell' - Path = '.github/workflows/case-12.yml' + Path = '.github/workflows/case-11.yml' Content = "shell: $legacyShell" ExpectViolation = $true } @@ -845,8 +801,8 @@ Add-ValidationResult -Results $results -Name 'App skill documents web-family App Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'Treat the scaffold as a fidelity copy of the documented template set, not a "best effort" approximation.' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle '## Step 3: Resolve Dynamic Dependency Versions' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'scripts/resolve-package-versions.ps1' - Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework ' - Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1' + Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File /scripts/resolve-package-versions.ps1 -TargetFramework ' + Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'If the host does not render native form controls, follow the deterministic plain-text fallback defined in `FORMS.md` instead of improvising your own questioning style.' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'Consistency matters more than creativity during parameter collection.' @@ -957,7 +913,7 @@ Add-ValidationResult -Results $results -Name 'App reference guide uses ROOT_NAME Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Directory.Packages.props` is the authoritative version source for app scaffolds.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Do **not** duplicate `` inside the generated app or test `.csproj` files as a workaround.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'scripts/resolve-package-versions.ps1' - Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework {TARGET_FRAMEWORK}' + Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'pwsh -NoProfile -File /scripts/resolve-package-versions.ps1 -TargetFramework {TARGET_FRAMEWORK}' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle '`testenvironments.json` is required output for the scaffold.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'MinVer may report a bootstrap pre-release such as `0.0.0-alpha.0`' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Where `{AppType}` maps to the emitted project suffix:' @@ -1071,8 +1027,8 @@ Add-ValidationResult -Results $results -Name 'Library skill documents PROJECT_NA Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle '{PROJECT_NAME}' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle '{DOCFX_TARGET_FRAMEWORK}' - Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1' - Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'In `pwsh` 7+, prefer .NET file APIs' + Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1' + Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'In PowerShell, prefer .NET file APIs' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'If the host does not render native form controls, follow the deterministic plain-text fallback defined in `FORMS.md` instead of improvising your own questioning style.' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'Consistency matters more than creativity during parameter collection.' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'treat a blank response as accepting that shown value' @@ -1102,7 +1058,7 @@ Add-ValidationResult -Results $results -Name 'Library reference guide uses curre Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'do not create an extra solution-named wrapper folder' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'src/{PROJECT_NAME}/{PROJECT_NAME}.csproj' - Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'Recommended `pwsh` 7+ approach for rewritten templates:' + Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'Recommended PowerShell approach for rewritten templates:' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'offer every other generally supported non-preview .NET LTS and STS channel' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'highest selected generally supported non-preview executable TFM' } @@ -1153,7 +1109,7 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, p Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Read `references/experiment-design.md`' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--list flat' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--job dry' - Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot ' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot ' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '#### Yolo mode' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Start a full performance run only after an explicit human instruction to run it now.' Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Yolo never authorizes a full performance run.' @@ -1187,7 +1143,7 @@ Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, p Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Result-validity gate' Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'SkipBenchmarksWithReports = true' Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'Anti-thrashing rule' - Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ' + Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ' Assert-Contains -Name 'validate-skill.ps1' -Content $validateSkillScript -Needle 'if ([string]::IsNullOrWhiteSpace($SkillRoot))' Assert-Contains -Name 'validate-skill.ps1' -Content $validateSkillScript -Needle 'Harness detector validation requires pwsh 7+' Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle '{EQUIVALENCE_CHECK}' @@ -1212,7 +1168,7 @@ Add-ValidationResult -Results $results -Name 'Strong-name skill matches FORMS su $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-strong-name-signing/SKILL.md' -GitRef $Ref Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'compute the defaults silently, and present a single summary for confirmation' Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'default: 1024' - Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'Run this command block in a `pwsh` 7+ session in the target directory:' + Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'Run this PowerShell command block with `pwsh` 7+ in the target directory:' Assert-NotContains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'default: 4096' } diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index a7f770a..bf40c0e 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -53,8 +53,8 @@ Yolo never authorizes a full performance run. Start the full benchmark only when Run the bundled read-only detector before changing files: -```ps1 -pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot +```console +pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot ``` If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. @@ -141,14 +141,14 @@ Semantic preflight Successful execution is not a correctness oracle. After semantic preflight, validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. The oracle should normally verify exact observable behavior rather than merely nonzero or approximate success unless that is the real domain contract. Then build the benchmark project in Release: -```bash +```console dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj ``` Before interpreting discovery or execution output, run the report-aware preflight for the exact class: -```ps1 -pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType +```console +pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ``` If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. @@ -157,7 +157,7 @@ If `reports.wouldSkipRequestedBenchmark` is true, the runner is intentionally fi Verify runner discovery without measuring: -```bash +```console dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* ``` @@ -165,7 +165,7 @@ Compare the discovered method, job, and parameter matrix with the intended desig Unless execution is impossible or the user declines, run a dry execution smoke check and inspect all BenchmarkDotNet validation warnings. A dry run proves executable wiring and basic lifecycle, not performance: -```bash +```console dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* ``` diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json index 23d9e1f..3a701a7 100644 --- a/skills/dotnet-benchmark/evals/evals.json +++ b/skills/dotnet-benchmark/evals/evals.json @@ -7,7 +7,6 @@ "expected_output": "The agent detects repository state first and onboards only the missing tuning project and single tooling runner, preserves CPM/solution conventions, resolves current package versions dynamically, and does not fabricate a target type or benchmark method.", "expectations": [ "Runs or references scripts/check-benchmark-requirements.ps1 before changing harness files", - "Invokes scripts/check-benchmark-requirements.ps1 through `pwsh -NoProfile -File` and never falls back to `powershell` or `powershell.exe`", "Creates or proposes exactly one reusable tooling runner and a tuning benchmark project while preserving the detected solution and package-management conventions", "Resolves BenchmarkDotNet package versions dynamically from NuGet.org rather than copying hardcoded example versions", "Does not invent a SUT type, member, workload, or benchmark class when none was supplied", diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md index f18d1f6..e10b8f5 100644 --- a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -101,25 +101,25 @@ Run the bundled detector with `-BenchmarkType ` and ins Build: -```bash +```console dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj ``` List cases without measuring: -```bash +```console dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* ``` Dry execution smoke: -```bash +```console dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* ``` Full default run: -```bash +```console dotnet run -c Release --project tooling/{runner} -- --filter *{BenchmarkClass}* ``` diff --git a/skills/dotnet-benchmark/references/runner-preflight.md b/skills/dotnet-benchmark/references/runner-preflight.md index 319c1ce..ce7e338 100644 --- a/skills/dotnet-benchmark/references/runner-preflight.md +++ b/skills/dotnet-benchmark/references/runner-preflight.md @@ -49,8 +49,8 @@ For example, `reports/tuning/Acme.Core.ParserBenchmark-report-github.md` causes 1. Build the benchmark project in Release. Stop on a compiler error; report it directly. 2. Run the detector with the intended benchmark type: - ```ps1 - pwsh -NoProfile -File scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType + ```console + pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType ``` If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. diff --git a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 index beb20aa..7a7ec35 100644 --- a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 +++ b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 @@ -17,7 +17,7 @@ Skips invoking dotnet --version. Intended for deterministic detector tests; normal skill runs should not use it. .EXAMPLE - pwsh -NoProfile -File ./scripts/check-benchmark-requirements.ps1 -RepoRoot C:\src\myrepo + pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot C:\src\myrepo #> [CmdletBinding()] param( diff --git a/skills/dotnet-docfx-digest/SKILL.md b/skills/dotnet-docfx-digest/SKILL.md index 2f0dbf0..2f8623f 100644 --- a/skills/dotnet-docfx-digest/SKILL.md +++ b/skills/dotnet-docfx-digest/SKILL.md @@ -88,7 +88,7 @@ Read the reflection-backed packets from that manifest or from `scope.packets`. I - Valid BOM-less UTF-8 is compliant. The validator must not emit `ENCODING_BOM_MISSING` (that diagnostic is intentionally unsupported), and an audit must not add/remove BOMs or normalize line endings solely for consistency. BOM presence has no documentation value; preserve the file's existing state and continue detecting actual `ENCODING_CORRUPTION` and `EXTENSION_TABLE_ENCODING` damage. - Final verification uses adaptive execution. In `auto`, machines with more than 8 available logical processors and more than 32 GiB available memory select `high-capacity`: DocFX verification runs concurrently in its isolated temp copy while the main lane builds/discovers API and then compiles samples, and MSBuild worker counts scale up to half the available processors (capped at 16). Smaller machines select `conservative` and keep the phases sequential with low worker counts. Override with `--execution-profile conservative|high-capacity` or `DOCFX_DIGEST_EXECUTION_PROFILE`. - Child processes time out after 30 minutes by default, configurable with `--process-timeout-minutes` or `DOCFX_DIGEST_PROCESS_TIMEOUT_MINUTES`. Use `host timeout >= process timeout + 5 minutes` (35 minutes for the default) so the validator can kill a timed-out child and return a deterministic diagnostic instead of being terminated by the caller first. -- Long-running API builds, sample compilation, and DocFX verification write progress to `stderr`: an initial `[ ]` line, a heartbeat every 10 seconds, and a final `[✓]` or `[x]` line. Heartbeats name the active phase, workload size, runner count, PID, elapsed time, time since the child last produced output, and its latest output line when available. Do not suppress `stderr` during normal interactive runs; `--json` remains a single parseable document on `stdout`. When `pwsh` or another host makes heartbeat noise obscure JSON inspection, pass `--quiet` or `--no-heartbeat` to suppress start/heartbeat chatter while keeping final `[✓]`/`[x]` child-process markers. In high-capacity mode, concurrent phase heartbeats can interleave; use JSON stdout and the final per-phase markers as the authoritative result, not the visual order of progress lines. +- Long-running API builds, sample compilation, and DocFX verification write progress to `stderr`: an initial `[ ]` line, a heartbeat every 10 seconds, and a final `[✓]` or `[x]` line. Heartbeats name the active phase, workload size, runner count, PID, elapsed time, time since the child last produced output, and its latest output line when available. Do not suppress `stderr` during normal interactive runs; `--json` remains a single parseable document on `stdout`. When PowerShell or another host makes heartbeat noise obscure JSON inspection, pass `--quiet` or `--no-heartbeat` to suppress start/heartbeat chatter while keeping final `[✓]`/`[x]` child-process markers. In high-capacity mode, concurrent phase heartbeats can interleave; use JSON stdout and the final per-phase markers as the authoritative result, not the visual order of progress lines. - When reporting adaptive execution, include the selected profile, processor and memory inputs, build/sample worker counts, timeout, concurrent/sequential choice, process counts, and phase timings from JSON. State that sample compilation follows API discovery because scoped references depend on the namespace-to-project map. Name CLI and environment overrides for profile, worker counts, and timeout when the user asks how to tune the run. - When reporting heartbeat behavior, state the complete contract: append-only `stderr`, no cursor-rewritten table, start and 10-second heartbeat events, final success/failure marker for all three long phases, latest non-empty child-output line when available, and plain redirected-log markers that do not depend on ANSI color. When reporting encoding safety, explicitly confirm the diff contains neither BOM-only nor line-ending-only changes. - Read `references/workflow.md` when you need the detailed targeted/audit workflows, namespace and example templates, the verification checklist, or the completion response shape. diff --git a/skills/dotnet-docfx-digest/evals/evals.json b/skills/dotnet-docfx-digest/evals/evals.json index 6e6c03a..d9b8fb0 100644 --- a/skills/dotnet-docfx-digest/evals/evals.json +++ b/skills/dotnet-docfx-digest/evals/evals.json @@ -1182,12 +1182,12 @@ }, { "id": 113, - "prompt": "Use dotnet-docfx-digest on a repository whose `.docfx/docfx.json` has `build.overwrite` set to `api/namespaces/**.md` and `api/types/**/*.md`, and whose samples include one compile failure from calling LINQ `Select` without `using System.Linq;`. One type-page example constructs and returns the documented type from an if/else branch but never prints, configures, invokes, or passes the value to another API. Run validation with JSON, keep the output easy to inspect in `pwsh`, and repair the docs.", - "expected_output": "Agent treats the near-miss DocFX glob as a literal config mismatch, recognizes the exact expected `api/namespaces/**/*.md` pattern, uses quiet/no-heartbeat validation when `pwsh` or another host would obscure JSON, fixes the no-observable example by showing a reader-visible result, and maps the CS1061 sample compile failure to a missing extension-method using such as `using System.Linq;` before rerunning validation.", + "prompt": "Use dotnet-docfx-digest on a repository whose `.docfx/docfx.json` has `build.overwrite` set to `api/namespaces/**.md` and `api/types/**/*.md`, and whose samples include one compile failure from calling LINQ `Select` without `using System.Linq;`. One type-page example constructs and returns the documented type from an if/else branch but never prints, configures, invokes, or passes the value to another API. Run validation with JSON, keep the output easy to inspect in PowerShell, and repair the docs.", + "expected_output": "Agent treats the near-miss DocFX glob as a literal config mismatch, recognizes the exact expected `api/namespaces/**/*.md` pattern, uses quiet/no-heartbeat validation when terminal noise would obscure JSON, fixes the no-observable example by showing a reader-visible result, and maps the CS1061 sample compile failure to a missing extension-method using such as `using System.Linq;` before rerunning validation.", "expectations": [ "Recognizes that `api/namespaces/**.md` is a near-miss and does not satisfy the expected literal `api/namespaces/**/*.md` overwrite pattern", "Repairs `build.overwrite` without widening the glob to `api/**/*.md` or moving namespace/type overwrite Markdown into `build.content`", - "Uses `--quiet` or `--no-heartbeat` when progress stderr makes JSON validation output hard to inspect interactively in `pwsh` or another host", + "Uses `--quiet` or `--no-heartbeat` when progress stderr makes JSON validation output hard to inspect interactively", "Treats `EXAMPLE_NO_OBSERVABLE_OUTCOME` as a failing example even when the code has a real branch or returns the documented type", "Repairs the example by producing a consumer-visible result such as printed data, configured state, an invoked member result, or a real consumer API call", "Interprets CS1061 extension-method failures as likely missing using directives when source/API evidence supports it", diff --git a/skills/dotnet-docfx-digest/references/scripts.md b/skills/dotnet-docfx-digest/references/scripts.md index e0147ec..4faa64f 100644 --- a/skills/dotnet-docfx-digest/references/scripts.md +++ b/skills/dotnet-docfx-digest/references/scripts.md @@ -59,7 +59,7 @@ Final verification is machine-adaptive. `--execution-profile auto` is the defaul External child processes have a 30-minute default timeout. Override it with `--process-timeout-minutes` or `DOCFX_DIGEST_PROCESS_TIMEOUT_MINUTES` (1-180). The calling agent/tool must use `outer command timeout >= process timeout + 5 minutes`; otherwise the caller can terminate the validator before it kills the child process and emits `DOCFX_BUILD_FAILED` or another deterministic diagnostic. With the default child timeout, use an outer timeout of at least 35 minutes. -API builds, sample graph builds, and DocFX verification emit append-only progress events to `stderr`. The validator writes `[ ]` when a child starts, another `[ ]` heartbeat every 10 seconds, and `[✓]` or `[x]` when it completes. Each heartbeat includes the phase, project/sample workload, runner count, child PID, elapsed time, time since the child last wrote output, and the latest non-empty output line when available. This makes a quiet restore/build distinguishable from a stale process without corrupting `--json`, which remains exclusively on `stdout`. Redirected logs use plain markers; interactive terminals color the completion marker when ANSI color is available. In `high-capacity`, heartbeat lines from concurrent phases may be interleaved; parse stdout JSON and final phase markers as authoritative rather than inferring order from the visual log. Use `--quiet` or `--no-heartbeat` when a host such as `pwsh` promotes heartbeat `stderr` noise into distracting terminal records; quiet mode suppresses start/heartbeat chatter and project packet progress lines while preserving final `[✓]`/`[x]` child-process markers. +API builds, sample graph builds, and DocFX verification emit append-only progress events to `stderr`. The validator writes `[ ]` when a child starts, another `[ ]` heartbeat every 10 seconds, and `[✓]` or `[x]` when it completes. Each heartbeat includes the phase, project/sample workload, runner count, child PID, elapsed time, time since the child last wrote output, and the latest non-empty output line when available. This makes a quiet restore/build distinguishable from a stale process without corrupting `--json`, which remains exclusively on `stdout`. Redirected logs use plain markers; interactive terminals color the completion marker when ANSI color is available. In `high-capacity`, heartbeat lines from concurrent phases may be interleaved; parse stdout JSON and final phase markers as authoritative rather than inferring order from the visual log. Use `--quiet` or `--no-heartbeat` when a host such as PowerShell promotes heartbeat `stderr` noise into distracting terminal records; quiet mode suppresses start/heartbeat chatter and project packet progress lines while preserving final `[✓]`/`[x]` child-process markers. For Codebelt strong-name signed repositories, build and sample paths use the root `.snk` when present and automatically pass `-p:SkipSignAssembly=true` when no root `.snk` exists, so missing local signing keys do not masquerade as documentation failures. diff --git a/skills/dotnet-new-app-slnx/SKILL.md b/skills/dotnet-new-app-slnx/SKILL.md index 37bb76a..66d0abd 100644 --- a/skills/dotnet-new-app-slnx/SKILL.md +++ b/skills/dotnet-new-app-slnx/SKILL.md @@ -89,7 +89,7 @@ Read `references/app.md` for the app-specific project structure, template file m Before writing `Directory.Packages.props`, resolve every `*_VERSION` placeholder in that file to the latest stable listed version for its matching package ID on NuGet.org. -When `pwsh` 7+ is available, prefer the deterministic helper in `scripts/resolve-package-versions.ps1` over manual lookup. Run it as `pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework `. By default it resolves placeholders from this skill's own `assets/shared/Directory.Packages.props`, so a normal scaffold run only needs `-TargetFramework`. Treat its JSON output as the source of truth for package placeholders, and do not substitute `powershell` or `powershell.exe` for this helper. +When `pwsh` 7+ is available, prefer the deterministic helper in `/scripts/resolve-package-versions.ps1` over manual lookup. Run it as `pwsh -NoProfile -File /scripts/resolve-package-versions.ps1 -TargetFramework `. By default it resolves placeholders from this skill's own `assets/shared/Directory.Packages.props`, so a normal scaffold run only needs `-TargetFramework`. Treat its JSON output as the source of truth for package placeholders. - Use the NuGet V3 service index at `https://api.nuget.org/v3/index.json` to discover the package metadata endpoints - Prefer registration metadata so you can ignore unlisted versions and prerelease builds @@ -153,7 +153,7 @@ Copy every file from `assets/shared/` to the project root, preserving directory Do this as a recursive, dotfile-aware copy. Hidden folders and files under `assets/shared/` are part of the scaffold and must not be skipped. In particular, copy `assets/shared/.bot/README.md` as a real file in the generated repo; do not replace it with a synthetic `.gitkeep` or placeholder note. -**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. If upstream fetch fails, halt and report — do not substitute placeholders. +**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If `pwsh` 7+ is unavailable, use the raw base URL in the **Upstream Source** table above to download each missing file manually. If upstream fetch fails, halt and report — do not substitute placeholders. Do not selectively copy only "key" shared files. The intended output includes the complete shared asset inventory, including `.gitignore`, `.gitattributes`, `AGENTS.md`, `CHANGELOG.md`, `.github/`, and `.bot/`, in addition to the build and package-management files. @@ -193,7 +193,7 @@ After generating, verify: - [ ] `.slnx` references all generated src/ and test/ projects - [ ] The generated solution filename is `{SOLUTION_NAME}.slnx` with the original user-facing casing preserved - [ ] Every file listed in `assets/shared.manifest.json` exists in the generated repo at its declared relative path (this covers all dotfiles and dotfolders) -- [ ] If any manifest entry was absent from the installed skill copy, `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` was run — not diagnosed iteratively +- [ ] If any manifest entry was absent from the installed skill copy, `pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1` was run (or files were fetched manually from the upstream raw URL) — not diagnosed iteratively - [ ] `Directory.Packages.props` lists all `` packages used in the solution (including host-type-specific packages) - [ ] `Directory.Packages.props` contains concrete version numbers with no unresolved `*_VERSION` placeholders - [ ] No generated `.csproj` file or `Directory.Build.props` contains ad-hoc inline `Version=` attributes for packages that are supposed to be centrally managed by `Directory.Packages.props` diff --git a/skills/dotnet-new-app-slnx/evals/evals.json b/skills/dotnet-new-app-slnx/evals/evals.json index 234c787..e3f8a9b 100644 --- a/skills/dotnet-new-app-slnx/evals/evals.json +++ b/skills/dotnet-new-app-slnx/evals/evals.json @@ -36,8 +36,7 @@ "Resolves package versions from NuGet instead of carrying hardcoded examples from a previous scaffold", "Copies the complete shared asset inventory from assets/shared instead of cherry-picking only a subset of governance or dotfiles", "Keeps TargetFramework centralized in the generated root Directory.Build.props instead of adding TargetFramework directly to generated app or test csproj files", - "Produces valid XML in generated root build props files before the first build rather than relying on mid-run repair", - "If it invokes scripts/resolve-package-versions.ps1, it does so through `pwsh -NoProfile -File` and never through `powershell` or `powershell.exe`" + "Produces valid XML in generated root build props files before the first build rather than relying on mid-run repair" ] }, { diff --git a/skills/dotnet-new-app-slnx/references/app.md b/skills/dotnet-new-app-slnx/references/app.md index 556ef7f..da4db5c 100644 --- a/skills/dotnet-new-app-slnx/references/app.md +++ b/skills/dotnet-new-app-slnx/references/app.md @@ -152,7 +152,7 @@ Resolve each package-specific `*_VERSION` placeholder in `Directory.Packages.pro Keep target-framework selection centralized too: the generated root `Directory.Build.props` owns `{TARGET_FRAMEWORK}` for source and test projects. Do **not** duplicate `` inside the generated app or test `.csproj` files as a workaround. -When `pwsh` 7+ is available, prefer `pwsh -NoProfile -File ./scripts/resolve-package-versions.ps1 -TargetFramework {TARGET_FRAMEWORK}` to produce the package placeholder map for this skill. The script defaults to this skill's own `assets/shared/Directory.Packages.props`, so the normal path only needs `{TARGET_FRAMEWORK}`. Its output should drive the final substitutions instead of remembered version numbers, and legacy Windows PowerShell is not a substitute runtime for this helper. +When `pwsh` 7+ is available, prefer `pwsh -NoProfile -File /scripts/resolve-package-versions.ps1 -TargetFramework {TARGET_FRAMEWORK}` to produce the package placeholder map for this skill. The script defaults to this skill's own `assets/shared/Directory.Packages.props`, so the normal path only needs `{TARGET_FRAMEWORK}`. Its output should drive the final substitutions instead of remembered version numbers. For framework-aligned ASP.NET packages, keep the selected target framework major in mind when resolving the final version: diff --git a/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 b/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 index 96c3e4e..001060d 100644 --- a/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 +++ b/skills/dotnet-new-app-slnx/scripts/restore-missing-shared-assets.ps1 @@ -17,10 +17,10 @@ .EXAMPLE # Restore missing files into the installed skill copy - pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 + pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1 # Preview what is missing without restoring - pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 -DryRun + pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1 -DryRun #> [CmdletBinding()] param( diff --git a/skills/dotnet-new-lib-slnx/SKILL.md b/skills/dotnet-new-lib-slnx/SKILL.md index 921320d..4b1d78a 100644 --- a/skills/dotnet-new-lib-slnx/SKILL.md +++ b/skills/dotnet-new-lib-slnx/SKILL.md @@ -102,15 +102,15 @@ Copy every file from `assets/shared/` to the project root, preserving directory Do this as a recursive, dotfile-aware copy. Hidden folders and files under `assets/shared/` are part of the scaffold and must not be skipped. In particular, copy `assets/shared/.bot/README.md` as a real file in the generated repo; do not replace it with a synthetic `.gitkeep` or placeholder note. -**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. If upstream fetch fails, halt and report — do not substitute placeholders. +**Asset mismatch policy — pivot immediately to upstream.** The `npx skills add` installer silently strips dot-prefixed entries (`.bot/`, `.github/`, `.editorconfig`, `.gitattributes`, `.gitignore`). Do not spend time re-proving what is absent. The moment any entry from `assets/shared.manifest.json` is missing from the installed skill copy, run `pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1` to fetch every missing file directly from the upstream repository in one step, then continue. If `pwsh` 7+ is unavailable, use the raw base URL in the **Upstream Source** table above to download each missing file manually. If upstream fetch fails, halt and report — do not substitute placeholders. Preserve UTF-8 when reading, copying, and writing text files. Do not transcode templates to ANSI, OEM, Windows-1252, or any system-default code page during generation. The shared `.editorconfig` in the scaffold declares `charset = utf-8`, and generated text files should match it from the start. When a text file does not need substitutions, prefer a byte-preserving file copy instead of read/transform/write. -When a text file does need substitutions, use explicit UTF-8 APIs end-to-end. In `pwsh` 7+, prefer .NET file APIs with an explicit `UTF8Encoding` instance rather than locale-dependent text cmdlets. For example: +When a text file does need substitutions, use explicit UTF-8 APIs end-to-end. In PowerShell, prefer .NET file APIs with an explicit `UTF8Encoding` instance rather than locale-dependent text cmdlets. For example: -```ps1 +```powershell $utf8NoBom = [System.Text.UTF8Encoding]::new($false) $content = [System.IO.File]::ReadAllText($src, $utf8NoBom) $updated = Apply-Replacements -Content $content -Map $replaceMap @@ -189,7 +189,7 @@ After generating, verify: - [ ] `.bot/` folder exists and is listed in `.gitignore` - [ ] `.bot/README.md` exists in the generated repo and came from the shared asset template, not from a synthetic `.gitkeep` fallback - [ ] Every file listed in `assets/shared.manifest.json` exists in the generated repo at its declared relative path (this covers all dotfiles and dotfolders) -- [ ] If any manifest entry was absent from the installed skill copy, `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` was run — not diagnosed iteratively +- [ ] If any manifest entry was absent from the installed skill copy, `pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1` was run (or files were fetched manually from the upstream raw URL) — not diagnosed iteratively - [ ] No manifest entries were silently skipped; if the restore script reported failures, generation was halted rather than continuing with incomplete shared assets - [ ] `.github/dependabot.yml` watches the repo root so central NuGet package management stays current after scaffolding diff --git a/skills/dotnet-new-lib-slnx/evals/evals.json b/skills/dotnet-new-lib-slnx/evals/evals.json index 8d768fa..f1b8079 100644 --- a/skills/dotnet-new-lib-slnx/evals/evals.json +++ b/skills/dotnet-new-lib-slnx/evals/evals.json @@ -55,17 +55,6 @@ "Continues to the summary instead of asking a second root_namespace clarification question", "Uses native structured inputs when available and a one-field-at-a-time plain-text fallback when they are not" ] - }, - { - "id": 6, - "prompt": "The installed dotnet-new-lib-slnx skill copy is missing shared dotfiles from assets/shared.manifest.json. Restore the missing shared assets before continuing the scaffold.", - "expected_output": "The agent immediately uses the bundled restore helper through `pwsh -NoProfile -File`, treats missing `pwsh` 7+ as a blocker instead of falling back to legacy Windows PowerShell, and avoids iterative file-by-file diagnosis.", - "expectations": [ - "Runs `pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1` when shared assets from assets/shared.manifest.json are missing", - "Does not invoke `powershell` or `powershell.exe` as a fallback runtime for the restore helper", - "Treats missing `pwsh` 7+ as a prerequisite blocker instead of silently changing runtimes", - "Restores the manifest-defined shared asset set in one step rather than diagnosing missing dotfiles iteratively" - ] } ] } diff --git a/skills/dotnet-new-lib-slnx/references/library.md b/skills/dotnet-new-lib-slnx/references/library.md index 98b0d77..7f4b464 100644 --- a/skills/dotnet-new-lib-slnx/references/library.md +++ b/skills/dotnet-new-lib-slnx/references/library.md @@ -90,9 +90,9 @@ If an installer path omits dot-prefixed files from the source tree, treat that a - Prefer byte-preserving copy for files that do not need substitutions - Preserve the source template's BOM policy by default; do not add a UTF-8 BOM unless the source had one or the target format explicitly needs it -Recommended `pwsh` 7+ approach for rewritten templates: +Recommended PowerShell approach for rewritten templates: -```ps1 +```powershell $utf8NoBom = [System.Text.UTF8Encoding]::new($false) $content = [System.IO.File]::ReadAllText($src, $utf8NoBom) $updated = Apply-Replacements -Content $content -Map $replaceMap diff --git a/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 b/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 index 17e0353..230f188 100644 --- a/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 +++ b/skills/dotnet-new-lib-slnx/scripts/restore-missing-shared-assets.ps1 @@ -17,10 +17,10 @@ .EXAMPLE # Restore missing files into the installed skill copy - pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 + pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1 # Preview what is missing without restoring - pwsh -NoProfile -File ./scripts/restore-missing-shared-assets.ps1 -DryRun + pwsh -NoProfile -File /scripts/restore-missing-shared-assets.ps1 -DryRun #> [CmdletBinding()] param( diff --git a/skills/dotnet-strong-name-signing/SKILL.md b/skills/dotnet-strong-name-signing/SKILL.md index 9bbc225..69664c0 100644 --- a/skills/dotnet-strong-name-signing/SKILL.md +++ b/skills/dotnet-strong-name-signing/SKILL.md @@ -24,9 +24,9 @@ Read `FORMS.md`, compute the defaults silently, and present a single summary for ### Step 2: Generate the Key File -Run this command block in a `pwsh` 7+ session in the target directory: +Run this PowerShell command block with `pwsh` 7+ in the target directory: -```ps1 +```powershell $rsa = New-Object System.Security.Cryptography.RSACryptoServiceProvider({KEY_SIZE}) $keyBlob = $rsa.ExportCspBlob($true) [System.IO.File]::WriteAllBytes("{OUTPUT_PATH}", $keyBlob) @@ -43,7 +43,7 @@ The `ExportCspBlob($true)` method exports the full key pair (public + private) i After generating the file, verify it exists and report: -```ps1 +```powershell $snkFile = Get-Item "{OUTPUT_PATH}" Write-Host "✅ Strong name key generated" Write-Host "" diff --git a/skills/dotnet-strong-name-signing/evals/evals.json b/skills/dotnet-strong-name-signing/evals/evals.json index 17cf2ec..258261d 100644 --- a/skills/dotnet-strong-name-signing/evals/evals.json +++ b/skills/dotnet-strong-name-signing/evals/evals.json @@ -8,8 +8,7 @@ "expectations": [ "Computes defaults silently before asking for confirmation", "Defaults to a 1024-bit RSA key", - "Uses the RSACryptoServiceProvider ExportCspBlob flow", - "Uses `pwsh` 7+ when it presents or executes the PowerShell command block locally" + "Uses the RSACryptoServiceProvider ExportCspBlob flow" ] }, { diff --git a/skills/git-repo-digest/SKILL.md b/skills/git-repo-digest/SKILL.md index e1dd921..a27ec23 100644 --- a/skills/git-repo-digest/SKILL.md +++ b/skills/git-repo-digest/SKILL.md @@ -58,7 +58,7 @@ scripts/digest.cs Run it with `dotnet run --file` so it is not confused with a nearby project file: -```bash +```console dotnet run --file /scripts/digest.cs -- --repo-url --output-root ``` @@ -87,7 +87,7 @@ The runner requires the .NET 10 SDK or newer and `git`. It performs one shallow The runner also supports deterministic result validation for authored workspaces: -```bash +```console dotnet run --file /scripts/digest.cs -- --validate-results --workspace ``` @@ -204,7 +204,7 @@ Choose the workspace mode from the user's input, not from folders you happen to For a fresh run, execute the bundled runner from this skill: -```bash +```console dotnet run --file /scripts/digest.cs -- --repo-url --output-root [--external-repo-url ]... ``` @@ -320,7 +320,7 @@ If any package digest is missing, decide from the manifest: Run the deterministic result validator and require a pass before reporting completion: -```bash +```console dotnet run --file /scripts/digest.cs -- --validate-results --workspace ``` @@ -417,7 +417,7 @@ Before finishing, verify: Use targeted searches instead of rereading everything: -```bash +```console rg -n "TODO|TBD|confidence|citation|analysis notes|I cannot|as an AI|\\.\\.\\.|placeholder" /result rg -n "Write a source-grounded|Write a short lede" /result rg -n "Greeting|MessageService|Hello World|Hello, World|Hello from DI|\\bOK\\b|GenerateReport|CreateService|BuildHost|FormatInvoice|CreateClient|SampleMiddleware|MyService|IMyService|MyRepository|FakeRepository|MyController|SampleController|\\bFoo\\b|\\bBar\\b|\\bDummy\\b" /result diff --git a/skills/git-repo-digest/evals/evals.json b/skills/git-repo-digest/evals/evals.json index 94cd957..fde4051 100644 --- a/skills/git-repo-digest/evals/evals.json +++ b/skills/git-repo-digest/evals/evals.json @@ -27,7 +27,6 @@ "Uses .bot/digests under the active workspace as the output root because the user supplied no output path", "Does not ask before using the well-defined .bot/digests output-root default", "Runs dotnet run --file /scripts/digest.cs with --repo-url and --output-root for the .bot/digests default", - "Keeps the runner invocation shell-agnostic instead of wrapping `dotnet run --file` in legacy Windows PowerShell", "Passes the full repository URL, not a slug or partial reference", "Does not ask the user for repo-id, run-id, output-root, or result directory", "Uses a run-id formatted yyyyMMdd-HHmmssZ so default workspaces sort chronologically", From 867ae20c69d4f52048e9ddfe42dc9c194462a8eb Mon Sep 17 00:00:00 2001 From: "aicia[bot]" Date: Sat, 18 Jul 2026 05:13:55 +0200 Subject: [PATCH 35/38] =?UTF-8?q?=F0=9F=93=9D=20simplify=20powershell=20gu?= =?UTF-8?q?idance=20in=20repo=20and=20skill=20docs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove overly prescriptive shell language from AGENTS.md, CONTRIBUTING.md, and README.md. Update skill descriptions to soften pwsh 7+ terminology, allowing flexibility while maintaining clear guidance to use pwsh (not powershell.exe) when PowerShell syntax is needed locally. Sync skill documentation to reflect the more permissive posture. --- AGENTS.md | 6 +----- CONTRIBUTING.md | 2 +- README.md | 10 +++++----- skills/dotnet-benchmark/SKILL.md | 4 ++-- skills/dotnet-benchmark/references/runner-preflight.md | 2 +- skills/dotnet-docfx-digest/SKILL.md | 10 +++++----- skills/dotnet-docfx-digest/references/scripts.md | 6 +++--- skills/dotnet-new-app-slnx/SKILL.md | 6 +++--- skills/dotnet-new-app-slnx/references/app.md | 2 +- skills/dotnet-new-lib-slnx/SKILL.md | 4 ++-- skills/dotnet-strong-name-signing/SKILL.md | 4 ++-- skills/git-repo-digest/SKILL.md | 8 ++++---- 12 files changed, 30 insertions(+), 34 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 3f0ae76..e3bc94d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,11 +4,7 @@ Repository-level rules for AI agents working in this codebase. ## Local Shell Execution -Agents may use Bash or `pwsh` 7+ for local development. - -Whenever a local command uses PowerShell syntax or executes a `.ps1` script, invoke `pwsh`, never `powershell` or `powershell.exe`. Keep `.ps1` filenames unchanged; the required change is the runtime, not the script extension. Use the form `pwsh -NoProfile -File ./scripts/example.ps1` for local `.ps1` execution. - -Do not silently fall back to legacy Windows PowerShell. If `pwsh` 7+ is unavailable for a required local `.ps1` script, report the missing prerequisite instead of invoking legacy Windows PowerShell. This rule applies to local agent execution only; GitHub Actions may continue using `bash`, `sh`, `pwsh`, platform defaults, or another justified shell, and existing workflow shell choices should not be rewritten without a functional reason. +Agents may use any appropriate local shell. When using PowerShell syntax or executing a `.ps1` script locally, use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`. This does not prescribe GitHub Actions shell choices. ## Eval Isolation diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bee7395..55ce7cb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -120,7 +120,7 @@ Use the repo validation harness before submitting scaffold or template changes: pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 ``` -Run the validator locally first for the fastest feedback loop. Bash and `pwsh` 7+ are both supported for local development in this repo, but every local `.ps1` invocation must go through `pwsh`. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. +Run the validator locally first for the fastest feedback loop. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. To compare a change against the initial imported version, run the same harness against a git ref: diff --git a/README.md b/README.md index f6ea6e6..b642444 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Final DocFX verification is machine-adaptive: `auto` selects a high-capacity pro DocFX diagnostics favor repairable specificity: overwrite-layout errors now call out literal near-miss globs such as `api/namespaces/**.md` versus `api/namespaces/**/*.md`, no-observable-outcome example failures explain what visible reader result is missing, and common sample `CS1061` extension-method failures include missing-`using` hints such as `System.Linq` or `BenchmarkDotNet.Configs` when the compiler output points that way. -Validation follows the same philosophy: Bash and `pwsh` 7+ are both valid for local development, but any local PowerShell command or `.ps1` script must run through `pwsh`. Use `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` for the fast feedback loop, and `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Full` when the slower DocFX regression suites are part of the gate. GitHub Actions runs full mode on pull requests as the safety net and can keep workflow-specific shell choices. The validator emits `[RUN]`, `[PASS]`, `[FAIL]`, `[WAIT]`, and `[SKIP]` progress lines so long phases show visible heartbeat feedback. It also checks skill frontmatter metadata such as per-skill `evals/evals.json` files, optional eval fixture paths declared through `files`, and the 1024-character YAML description limit; it does not replace the paired benchmark review workflow. +Validation follows the same philosophy: run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` locally for the fast feedback loop, and use `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Full` when the slower DocFX regression suites are part of the gate. GitHub Actions runs full mode on pull requests as the safety net. The validator emits `[RUN]`, `[PASS]`, `[FAIL]`, `[WAIT]`, and `[SKIP]` progress lines so long phases show visible heartbeat feedback. It also checks skill frontmatter metadata such as per-skill `evals/evals.json` files, optional eval fixture paths declared through `files`, and the 1024-character YAML description limit; it does not replace the paired benchmark review workflow. ## Install a skill @@ -113,7 +113,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark | [dotnet-new-lib-slnx](skills/dotnet-new-lib-slnx/SKILL.md) | Scaffold a new .NET NuGet library solution following codebeltnet engineering conventions. Dynamic defaults for TFM/repository metadata, latest-stable NuGet package resolution, tuning projects plus a tooling-based benchmark runner, TFM-aware test environments, strong-name signing, NuGet packaging, DocFX documentation, CI/CD pipeline, and code quality tooling. | | [dotnet-new-app-slnx](skills/dotnet-new-app-slnx/SKILL.md) | Scaffold a new .NET standalone application solution following codebeltnet engineering conventions. Supports Console, Web, and Worker host families with Startup or Minimal hosting patterns; Web expands into Empty Web, Web API, MVC, or Web App / Razor, plus functional tests and a simplified CI pipeline. | | [trunk-first-repo](skills/trunk-first-repo/SKILL.md) | Initialize a git repository following [scaled trunk-based development](https://trunkbaseddevelopment.com/#scaled-trunk-based-development). Seeds an empty `main` branch, creates a versioned feature branch (`v0.1.0/init`), confirms configured remotes in its post-init summary, and supports a guarded later `push remote ` mode that checks the feature-branch/empty-main state before pushing `main` ahead of the first feature branch so content still reaches main only through peer-reviewed pull requests. | -| [dotnet-strong-name-signing](skills/dotnet-strong-name-signing/SKILL.md) | Generate a strong name key (`.snk`) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or `sn.exe` required. Works in any terminal, using `pwsh` 7+ whenever local PowerShell syntax is needed. Defaults to 1024-bit RSA (matching `sn.exe`), with 2048 and 4096 available as options. | +| [dotnet-strong-name-signing](skills/dotnet-strong-name-signing/SKILL.md) | Generate a strong name key (`.snk`) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or `sn.exe` required. Works in any terminal. Defaults to 1024-bit RSA (matching `sn.exe`), with 2048 and 4096 available as options. | | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | @@ -505,11 +505,11 @@ Starting a new .NET solution "from scratch" usually means copying from your last ### Why dotnet-strong-name-signing? -Generating a `.snk` file traditionally requires `sn.exe`, which is only available in the Visual Studio Developer PowerShell — a common pain point for developers using VS Code, Rider, or plain terminals. This skill uses `RSACryptoServiceProvider` from the .NET runtime itself, so it works in any terminal and, when local PowerShell syntax is preferred, runs cleanly through **`pwsh` 7+** without special tooling. +Generating a `.snk` file traditionally requires `sn.exe`, which is only available in the Visual Studio Developer PowerShell — a common pain point for developers using VS Code, Rider, or plain terminals. This skill uses `RSACryptoServiceProvider` from the .NET runtime itself, so it works in **any PowerShell or terminal** without special tooling. -- **No `sn.exe` dependency** — uses pure .NET crypto available in any `pwsh` 7+ session +- **No `sn.exe` dependency** — uses pure .NET crypto available in any PowerShell session - **Matches `sn.exe` defaults** — 1024-bit RSA by default, with 2048 and 4096 as options -- **Cross-platform** — works on Windows, macOS, and Linux with `pwsh` 7+ or the .NET runtime +- **Cross-platform** — works on Windows, macOS, and Linux with PowerShell 7+ or .NET runtime - **Identity, not security** — [Microsoft's guidance](https://github.com/dotnet/runtime/blob/main/docs/project/strong-name-signing.md) is clear: strong names are about assembly identity, not cryptographic security ### Why trunk-first? diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md index bf40c0e..8d35f41 100644 --- a/skills/dotnet-benchmark/SKILL.md +++ b/skills/dotnet-benchmark/SKILL.md @@ -54,7 +54,7 @@ Yolo never authorizes a full performance run. Start the full benchmark only when Run the bundled read-only detector before changing files: ```console -pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot +pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "" ``` If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. @@ -148,7 +148,7 @@ dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.c Before interpreting discovery or execution output, run the report-aware preflight for the exact class: ```console -pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType +pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "" -BenchmarkType ``` If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. diff --git a/skills/dotnet-benchmark/references/runner-preflight.md b/skills/dotnet-benchmark/references/runner-preflight.md index ce7e338..8f3c92d 100644 --- a/skills/dotnet-benchmark/references/runner-preflight.md +++ b/skills/dotnet-benchmark/references/runner-preflight.md @@ -50,7 +50,7 @@ For example, `reports/tuning/Acme.Core.ParserBenchmark-report-github.md` causes 2. Run the detector with the intended benchmark type: ```console - pwsh -NoProfile -File /scripts/check-benchmark-requirements.ps1 -RepoRoot -BenchmarkType + pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "" -BenchmarkType ``` If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. diff --git a/skills/dotnet-docfx-digest/SKILL.md b/skills/dotnet-docfx-digest/SKILL.md index 2f8623f..9fa6aad 100644 --- a/skills/dotnet-docfx-digest/SKILL.md +++ b/skills/dotnet-docfx-digest/SKILL.md @@ -18,14 +18,14 @@ Create and maintain developer-friendly DocFX documentation digests for .NET publ - Iterate quickly with the fast path, reading diagnostics as the next work queue: ```bash -dotnet run --file /scripts/docfx.cs -- --repo-root --json +dotnet run --file "/scripts/docfx.cs" -- --repo-root "" --json ``` - Before claiming completion, run `agents.cs`, then a thorough build-backed verification that compiles samples, uses reflection-backed API discovery, and verifies the DocFX build: ```bash -dotnet run --file /scripts/agents.cs -- --repo-root -dotnet run --file /scripts/docfx.cs -- --repo-root --build-api-model --validate-samples --verify-docfx-build +dotnet run --file "/scripts/agents.cs" -- --repo-root "" +dotnet run --file "/scripts/docfx.cs" -- --repo-root "" --build-api-model --validate-samples --verify-docfx-build ``` - `--build-api-model` makes API discovery reflection-precise (the fast source-scan path is conservative and may under-report), `--validate-samples` compiles every C# documentation sample through isolated projects in one temporary `.slnx` graph build with bounded MSBuild parallelism, and `--verify-docfx-build` confirms the DocFX build succeeds. Treat `API_MODEL_SOURCE_SCANNER_LIMITED` in a fast run as a reminder to run the build-backed verification before completion, not as a failure. @@ -34,7 +34,7 @@ dotnet run --file /scripts/docfx.cs -- --repo-root --json --assessment-queue --search-examples +dotnet run --file "docfx.cs" -- --repo-root "" --json --assessment-queue --search-examples ``` Resolve `` outside the target repository working tree (for example, `$env:TEMP\docfx-assessment-queue.md` on Windows or `/tmp/docfx-assessment-queue.md` on Unix). @@ -46,7 +46,7 @@ The assessment work queue includes a "GitHub Example Sources" section with pre-c - For repo-wide or other full authoring runs, establish a bounded write queue before authoring new examples or overwrite rewrites: ```bash -dotnet run --file /scripts/docfx.cs -- --repo-root --build-api-model --project-manifest --json +dotnet run --file "/scripts/docfx.cs" -- --repo-root "" --build-api-model --project-manifest --json ``` Resolve `` outside the target repository working tree (for example, `$env:TEMP\dotnet-docfx-digest-project-manifest.json` on Windows or `/tmp/dotnet-docfx-digest-project-manifest.json` on Unix). When you name this command in a continuation response, replace the placeholder with that concrete temp/session path rather than leaving `` unresolved. diff --git a/skills/dotnet-docfx-digest/references/scripts.md b/skills/dotnet-docfx-digest/references/scripts.md index 4faa64f..ff3ffac 100644 --- a/skills/dotnet-docfx-digest/references/scripts.md +++ b/skills/dotnet-docfx-digest/references/scripts.md @@ -9,15 +9,15 @@ dotnet build