diff --git a/.gitignore b/.gitignore index 0035e34..06296cf 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,20 @@ +# Build results +[Dd]ebug/ +[Dd]ebugPublic/ +[Rr]elease/ +[Rr]eleases/ +x64/ +x86/ +[Ww][Ii][Nn]32/ +[Aa][Rr][Mm]/ +[Aa][Rr][Mm]64/ +bld/ +[Bb]in/ +[Oo]bj/ +[Oo]ut/ +[Ll]og/ +[Ll]ogs/ + # OS .DS_Store Thumbs.db diff --git a/AGENTS.md b/AGENTS.md index 36d1990..e3bc94d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,6 +2,10 @@ Repository-level rules for AI agents working in this codebase. +## Local Shell Execution + +Agents may use any appropriate local shell. When using PowerShell syntax or executing a `.ps1` script locally, use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`. This does not prescribe GitHub Actions shell choices. + ## Eval Isolation Eval workspaces and test repositories must **never** be created inside this repository. This includes: diff --git a/CHANGELOG.md b/CHANGELOG.md index 2d0e573..560e836 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,62 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [0.8.0] - 2026-07-18 + +This is a minor release introducing the `dotnet-benchmark` skill for performance testing of .NET types with evidence-driven discovery and measurement discipline. The release emphasizes candidate selection through profiling evidence, semantic correctness validation before performance interpretation, and proportionate-stopping decision logic. Additionally, the release standardizes local PowerShell execution to `pwsh` 7+ and strengthens validation discipline across repo-managed skills. + +### Added + +- `dotnet-benchmark` skill with evidence-driven workflow for identifying high-value benchmark targets, designed to avoid low-signal performance testing and over-measurement; includes step-by-step discovery phases from intent resolution through experiment planning, +- Discovery-focused FORMS.md parameter collection for `dotnet-benchmark` reducing implementation-tier choice friction by deferring tier selection to workflow inspection, +- New template assets `operation-benchmark.cs` and `comparison-benchmark.cs` providing refined structural guidance for single-operation and comparative-implementation benchmarks, +- `candidate-selection.md` reference documenting evidence ladders, call-site inspection, profiling integration, and candidate-ranking heuristics to drive the discovery phase, +- `experiment-design.md` reference detailing performance questions, workload selection, semantic preflight and correctness oracle validation, measurement fitness assessment, and early-stop conditions, +- Mandatory semantic preflight validation gate in `dotnet-benchmark` SKILL.md requiring deterministic correctness oracle derivation before accepting full-run results, preventing false-positive baseline misinterpretation, +- Proportionate-stopping decision logic in `dotnet-benchmark` recognizing when measurement is complete and cost does not justify deeper investigation; includes case studies and selectivity-drift repair guidance, +- Yolo mode support in `dotnet-benchmark` for autonomous candidate selection and progress-update-only planning when user intent is explicit, +- Report-aware runner preflight in `dotnet-benchmark` recognizing when SkipBenchmarksWithReports plus matching reports/tuning/ artifacts intentionally filter a benchmark type, preserving benchmark code unchanged, +- Comprehensive eval coverage for `dotnet-benchmark` with 12 test cases covering discovery workflow, candidate selection, evidence gathering, cost-signal analysis, implementation-comparison patterns, semantic preflight validation, selectivity-drift repair, proportionate stopping, yolo mode, and report-aware preflight; includes fixture code supporting five representative benchmark scenarios, +- Enhanced `check-benchmark-requirements.ps1` and new `validate-skill.ps1` tooling supporting discovery workflow validation and template-asset consistency checking. + +### Changed + +- Standardized local PowerShell execution to `pwsh` 7+ while preserving Bash and workflow-specific shell choices; updated all local command examples and contributor guidance accordingly. + +## [0.7.5] - 2026-07-15 + +This is a patch release focused on extending `trunk-first-repo` with a push-remote workflow mode that safely handles first-time remote pushes by pushing `main` before feature branches, ensuring the remote defaults to the correct branch while maintaining the PR-first workflow philosophy. + +### Added + +- Push Remote Workflow mode in `trunk-first-repo` for safely pushing to a newly-established remote without manually switching branches or checking out `main`, allowing `push remote ` invocation from the feature branch to send `main` by ref (`main:main`) before the feature branch, +- Enhanced eval coverage for `trunk-first-repo` documenting push-remote workflow and Step 0 mode selection behavior, +- Updated README description for `trunk-first-repo` to document safe first-push capability and `push remote ` mode alongside the Initialize Workflow. + +### Changed + +- Extended `trunk-first-repo` SKILL.md with Step 0 mode selector to distinguish between Initialize Workflow (repository creation) and Push Remote Workflow (remote establishment), +- Refined README guidance to emphasize that `push remote ` can be invoked later from the feature branch for safer first-push without switching branches, +- Added explicit push-remote documentation to trunk-first-repo "Why?" section explaining safer first-push behavior and benefits of sending `main` by ref. + +## [0.7.4] - 2026-07-03 + +This is a patch release focused on strengthening `git-keep-a-changelog` with mandatory Step 4a base-commit inspection for concrete releases, ensuring that foundational version bumps, release-prep changes, and dependency baseline updates are never omitted from release narratives. The skill now requires explicit inspection of the base commit before manifest diffs and commit bodies, with output verification and structured reporting. + +### Added + +- Step 4a mandatory checkpoint in `git-keep-a-changelog` that inspects and explicitly reports the base commit for concrete releases (e.g., `## [X.Y.Z]`), showing changed files, identifying dependency/version manifests, and confirming release-prep file modifications before proceeding to Step 4b manifest diffs, +- Explicit base-commit-inclusion enforcement using `^..HEAD` (with caret) throughout Step 4 for concrete releases, ensuring the base commit itself is included in the changelog narrative, +- Verification and confirmation gates in Step 4a requiring agents to show full base commit output, identify manifests, and explicitly state whether manifests or release-prep files were touched before proceeding to 4b, +- Detailed comparison matrix in Step 3b distinguishing between `base^..HEAD` (for concrete releases, inclusive of base) and `base..HEAD` (for [Unreleased], exclusive of base), +- Eval coverage validating base-commit inclusion, manifest detection, and Step 4a output verification for concrete release scenarios. + +### Changed + +- Restructured `git-keep-a-changelog` Step 4 into explicit sub-steps (4a through 4f) with clear sequencing: base-commit inspection first (4a), manifest detection (4b), manifest diff inspection (4c), commit-body reading (4d), net-diff inspection (4e), and pending-change integration (4f), +- Enhanced `git-keep-a-changelog` SKILL.md with critical range-extension guidance for concrete releases, emphasizing that `^..HEAD` (with caret) must be used consistently to include the base commit itself, +- Strengthened "Bad Output Characteristics" section with **CRITICAL** emphasis on the consequences of omitting the base commit: silently-wrong output that breaks release narratives and loses foundational version bumps, +- Updated README with enhanced description of `git-keep-a-changelog` base-commit enforcement and Step 4a mandatory checkpoint. ## [0.7.3] - 2026-07-01 @@ -418,7 +473,9 @@ This is a minor release that introduces two complementary git workflow skills, e - Improved scaffold fidelity with hidden `.bot` asset preservation, explicit UTF-8 and BOM handling, and checks aimed at preventing mojibake or incomplete generated output. -[Unreleased]: https://github.com/codebeltnet/agentic/compare/v0.7.3...HEAD +[0.8.0]: https://github.com/codebeltnet/agentic/compare/v0.7.5...v0.8.0 +[0.7.5]: https://github.com/codebeltnet/agentic/compare/v0.7.4...v0.7.5 +[0.7.4]: https://github.com/codebeltnet/agentic/compare/v0.7.3...v0.7.4 [0.7.3]: https://github.com/codebeltnet/agentic/compare/v0.7.2...v0.7.3 [0.7.2]: https://github.com/codebeltnet/agentic/compare/v0.7.1...v0.7.2 [0.7.1]: https://github.com/codebeltnet/agentic/compare/v0.7.0...v0.7.1 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ff811a9..55ce7cb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -68,21 +68,21 @@ The `description` is the most important field — it's how the AI decides to loa Evals let you verify the skill works and measure improvement over a baseline. Every repo-managed skill in this repository must include `evals/evals.json`: -```json -{ - "skill_name": "your-skill-name", - "evals": [ - { - "id": 0, - "prompt": "The user message to test against", - "expected_output": "What a correct response looks like — used for manual or automated grading", - "files": ["evals/files/example.md"] - } - ] -} -``` - -`files` is optional. When present, list one or more fixture files relative to `skills//`. A common pattern is to store those fixtures under `evals/files/` so benchmark runners can copy or attach the same source inputs for both `with_skill` and `without_skill` runs. +```json +{ + "skill_name": "your-skill-name", + "evals": [ + { + "id": 0, + "prompt": "The user message to test against", + "expected_output": "What a correct response looks like — used for manual or automated grading", + "files": ["evals/files/example.md"] + } + ] +} +``` + +`files` is optional. When present, list one or more fixture files relative to `skills//`. A common pattern is to store those fixtures under `evals/files/` so benchmark runners can copy or attach the same source inputs for both `with_skill` and `without_skill` runs. Aim for 3–5 evals that cover distinct scenarios: happy path, edge cases, and cases where the skill should *not* do something. @@ -116,16 +116,16 @@ When a skill needs defaults for versions, paths, repository names, or support wi Use the repo validation harness before submitting scaffold or template changes: -```powershell -powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-skill-templates.ps1 +```console +pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 ``` -Run the validator locally first for the fastest feedback loop. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. +Run the validator locally first for the fastest feedback loop. GitHub Actions also runs the same script on pull requests, but CI is the backstop, not the primary authoring loop. To compare a change against the initial imported version, run the same harness against a git ref: -```powershell -powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-skill-templates.ps1 -Ref HEAD +```console +pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD ``` ## Checklist before submitting @@ -133,10 +133,10 @@ powershell -NoProfile -ExecutionPolicy Bypass -File .\scripts\validate-skill-tem - [ ] `SKILL.md` has valid front matter with `name` and `description` - [ ] Skill is stack-agnostic (or clearly scoped to a specific tech in the name/description) - [ ] Examples are generic — no personal emails, usernames, or project-specific identifiers -- [ ] At least one eval in `evals/evals.json` -- [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name -- [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder -- [ ] Skill changes were benchmarked from a temp workspace with both `with_skill` and `without_skill` runs +- [ ] At least one eval in `evals/evals.json` +- [ ] The skill's `evals/evals.json` exists and its `skill_name` matches the folder/frontmatter name +- [ ] Any optional `files` entries in `evals/evals.json` point to real fixture files under the same skill folder +- [ ] Skill changes were benchmarked from a temp workspace with both `with_skill` and `without_skill` runs - [ ] `benchmark.json` and `eval-viewer/generate_review.py` from the installed Anthropic `skill-creator` copy were used so a human could compare `Outputs` and `Benchmark` - [ ] `scripts/validate-skill-templates.ps1` passes for the current working tree when changing scaffold or template behavior - [ ] If CI is enabled for the branch, the GitHub Actions validation job passes too diff --git a/README.md b/README.md index b45a2c1..b642444 100644 --- a/README.md +++ b/README.md @@ -36,7 +36,7 @@ Final DocFX verification is machine-adaptive: `auto` selects a high-capacity pro DocFX diagnostics favor repairable specificity: overwrite-layout errors now call out literal near-miss globs such as `api/namespaces/**.md` versus `api/namespaces/**/*.md`, no-observable-outcome example failures explain what visible reader result is missing, and common sample `CS1061` extension-method failures include missing-`using` hints such as `System.Linq` or `BenchmarkDotNet.Configs` when the compiler output points that way. -Validation follows the same philosophy: run `scripts/validate-skill-templates.ps1` locally for the fast feedback loop, and use `scripts/validate-skill-templates.ps1 -Full` when the slower DocFX regression suites are part of the gate. GitHub Actions runs full mode on pull requests as the safety net. The validator emits `[RUN]`, `[PASS]`, `[FAIL]`, `[WAIT]`, and `[SKIP]` progress lines so long phases show visible heartbeat feedback. It also checks skill frontmatter metadata such as per-skill `evals/evals.json` files, optional eval fixture paths declared through `files`, and the 1024-character YAML description limit; it does not replace the paired benchmark review workflow. +Validation follows the same philosophy: run `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1` locally for the fast feedback loop, and use `pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Full` when the slower DocFX regression suites are part of the gate. GitHub Actions runs full mode on pull requests as the safety net. The validator emits `[RUN]`, `[PASS]`, `[FAIL]`, `[WAIT]`, and `[SKIP]` progress lines so long phases show visible heartbeat feedback. It also checks skill frontmatter metadata such as per-skill `evals/evals.json` files, optional eval fixture paths declared through `files`, and the 1024-character YAML description limit; it does not replace the paired benchmark review workflow. ## Install a skill @@ -82,6 +82,7 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-new-lib-sln npx skills add https://github.com/codebeltnet/agentic --skill git-remote-release npx skills add https://github.com/codebeltnet/agentic --skill dotnet-change-impact npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-digest +npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark # npx skills add https://github.com/codebeltnet/agentic --skill another-skill ``` @@ -111,11 +112,12 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-diges | [git-repo-digest](skills/git-repo-digest/SKILL.md) | Turns any full repository URL into a deterministic digest workspace using the bundled .NET file-based runner `scripts/digest.cs`. Requires explicit `--repo-url`, resolves omitted output paths to `/.bot/digests` and passes that as `--output-root`, maps multiple positional URLs the same way for slash commands, bare pasted URLs, and natural-language requests by treating the first URL as the digest repo and every later URL as repeated `--external-repo-url`, always writes into `{output-root}/{repo-id}/{yyyyMMdd-HHmmssZ}`, accepts repeated curated public consumer repos, derives `{repo-id}`, fixes `result/`, performs shallow git clones, packs local tracked files with the bundled C# packer using `git ls-files`, separates XML evidence into `source.xml`, `tests.xml`, `projects.xml`, editorial `readmes.xml`, and scenario-only `external-usage.xml`, writes package and conceptual overview prompts under `prompts/`, emits public API summaries, engineering signals, evidence indexes, ordered XML chunks, referenced-package evidence maps for aggregate examples, and manifest-backed frontmatter hints, treats previous digest prose as contamination during fresh generation, then guides the agent to fully read the current phase's required evidence before writing package digests and a concept-led `result/Index.md` with YAML frontmatter containing Product-derived overview title metadata, validated documentation URLs resolved from PackageProjectUrl, documentation-host-filtered exact `.nuget//README.md` documentation links including emoji-prefixed Documentation headings and "More documentation..." blocks, DocFX `metadata[].dest` API paths, and source namespace page candidates from `src//**/*.cs`, target frameworks, package/library counts, external links, package-family links, and context glyphs, and validates authored result examples with `--validate-results` as a deterministic API-shape, Codebelt.Extensions.Xunit shape, PascalCase `MethodName_Scenario_ExpectedBehavior` test-method naming, Basic usage quality, and optimized NuGet-backed executable test gate with bounded parallelism. | | [dotnet-new-lib-slnx](skills/dotnet-new-lib-slnx/SKILL.md) | Scaffold a new .NET NuGet library solution following codebeltnet engineering conventions. Dynamic defaults for TFM/repository metadata, latest-stable NuGet package resolution, tuning projects plus a tooling-based benchmark runner, TFM-aware test environments, strong-name signing, NuGet packaging, DocFX documentation, CI/CD pipeline, and code quality tooling. | | [dotnet-new-app-slnx](skills/dotnet-new-app-slnx/SKILL.md) | Scaffold a new .NET standalone application solution following codebeltnet engineering conventions. Supports Console, Web, and Worker host families with Startup or Minimal hosting patterns; Web expands into Empty Web, Web API, MVC, or Web App / Razor, plus functional tests and a simplified CI pipeline. | -| [trunk-first-repo](skills/trunk-first-repo/SKILL.md) | Initialize a git repository following [scaled trunk-based development](https://trunkbaseddevelopment.com/#scaled-trunk-based-development). Seeds an empty `main` branch, creates a versioned feature branch (`v0.1.0/init`), and supports a later `push remote ` mode that pushes `main` before feature branches so new remotes keep the right default branch while content reaches main only through peer-reviewed pull requests. | +| [trunk-first-repo](skills/trunk-first-repo/SKILL.md) | Initialize a git repository following [scaled trunk-based development](https://trunkbaseddevelopment.com/#scaled-trunk-based-development). Seeds an empty `main` branch, creates a versioned feature branch (`v0.1.0/init`), confirms configured remotes in its post-init summary, and supports a guarded later `push remote ` mode that checks the feature-branch/empty-main state before pushing `main` ahead of the first feature branch so content still reaches main only through peer-reviewed pull requests. | | [dotnet-strong-name-signing](skills/dotnet-strong-name-signing/SKILL.md) | Generate a strong name key (`.snk`) file for signing .NET assemblies using pure .NET cryptography — no Visual Studio Developer PowerShell or `sn.exe` required. Works in any terminal. Defaults to 1024-bit RSA (matching `sn.exe`), with 2048 and 4096 available as options. | | [git-remote-release](skills/git-remote-release/SKILL.md) | Generate GitHub release notes by summarizing all commits and pull requests between two Git tags or branches in a remote GitHub repository. Accepts a compare URL or separate owner/repo, previous ref, and current ref values; falls back to comparing the current branch against the upstream default branch when no input is provided. Produces a human-friendly `## What's Changed` summary with optional GitHub alert blocks, a `Sources:` section preserving PR and commit references, and a full changelog compare link. | | [dotnet-change-impact](skills/dotnet-change-impact/SKILL.md) | Classify .NET library or NuGet package changes and recommend the correct release bump — `Major`, `Minor`, or `Patch` — for both Semantic Versioning (`MAJOR.MINOR.PATCH`) and .NET assembly/file versioning (`Major.Minor.Build.Revision`), grounded in Microsoft's official .NET compatibility rules. Uses the current Git branch by default when no explicit change details or compare range are provided, resolving it against the upstream/default base branch with local read-only git state. Always returns structured behavioral/binary/source/design-time/backwards compatibility reasoning with the recommendation, even when the bump is clear. | | [dotnet-docfx-digest](skills/dotnet-docfx-digest/SKILL.md) | Create and maintain developer-friendly DocFX documentation for .NET public APIs, including repo-wide no-input audits that inspect source, tests, DocFX config, DocFX `build.content` and `build.overwrite` Markdown inputs, namespace pages, and availability includes before asking for clarification, while treating bare direct skill invocations as autonomous repo-wide runs rather than human-driven checkpoint sessions. Enforces the workflow with two bundled .NET 10 file-based scripts resolved from the loaded skill directory, falling back to the repo-managed source path only when present: `scripts/agents.cs` writes an idempotent, marker-bounded DocFX maintenance block into the repository `AGENTS.md`; `scripts/docfx.cs` is **fast and build-free by default** — it validates Markdown, prose, DocFX overwrite layout, namespace overview pages, `Extension Members` tables, decorated receiver signatures such as `IDecorator`, generic method displays such as `As`, purpose-first summaries, and required per-type/extension examples without invoking `dotnet`, `msbuild`, `docfx`, or `gh`, discovering the public API from existing DocFX YAML metadata or a conservative source scan and ending every run with a `[processes] dotnet=0 msbuild=0 docfx=0 gh=0` summary plus per-phase timings. Compilation and network access are strictly opt-in: `--validate-samples` compiles each C# sample in an isolated project while batching all sample projects into one temporary `.slnx` graph build with bounded MSBuild parallelism and scoped references, `--build-api-model` (alias `--strict-api-discovery`) does reflection-backed discovery from compiled metadata via `MetadataLoadContext` through a single scoped `.slnx` graph build, `--verify-docfx-build` runs the DocFX CLI in a temp copy, and `--search-examples` runs `gh` code search. Final verification adapts to available processors and memory, overlaps isolated DocFX work on high-capacity machines, uses a 30-minute child timeout, and emits 10-second `stderr` heartbeats with active phase, workload, runner count, PID, elapsed time, last-output age, and current child output while preserving machine-readable JSON on `stdout`. Honors a single DocFX metadata `TargetFramework` when `--framework` is omitted, collapses C# 14 extension-block compiler containers such as `$...` back to the authored outer static class in both fast DocFX-YAML discovery and build-backed reflection discovery, validates namespace fly-ins that explain the problem solved/when to use/where to start plus example fly-ins before every C# fence, the Codebelt namespace-and-type-folder overwrite layout (`.docfx/api/namespaces/**/*.md` and `.docfx/api/types/**/*.md` under `build.overwrite` only), keeps `--changed-only` validation scoped to affected docs and APIs while still including brand-new untracked overwrite Markdown, uses the root Codebelt `.snk` when present and falls back to `-p:SkipSignAssembly=true` for keyless strong-name build verification, drains child stdout and stderr concurrently to avoid verbose-build deadlocks, writes deterministic `--assessment-queue` Markdown work queues for noisy audits, preserves working URL references unless a verified HTTP 404 justifies removal, treats unexpected new repo-root or DocFX-workspace files that are not known `dotnet-docfx-digest` deliverables as blocking cleanup diagnostics, keeps assessment/manifests/captured output/helper scripts in temp or session storage instead of the target repository, requires a namespace-first pass across the active queue before net-new type/example authoring during full audits, keeps deeper `EXTENSION_METHOD_MISSING` and `EXTENSION_METHOD_SIGNATURE_MISSING` follow-on diagnostics in that same namespace-layer table-repair phase when they appear after `EXTENSION_SECTION_MISSING` drops, preserves existing BOM and line-ending state while flagging actual mojibake instead of creating encoding-only diffs, and leaves generated DocFX YAML metadata untouched unless `--clean-generated-metadata` is explicitly requested (which runs only after the API model is built, never deleting metadata the run relied on). Documents public API only, uses bundled reference docs for overwrite rules, workflow details, and script behavior, keeps authored API overwrite Markdown under `.docfx/api/namespaces/` and `.docfx/api/types/`, moves legacy authored `.docfx/api/*.md` overwrite files there instead of widening the glob to `api/**/*.md`, teaches namespace and API prose to orient newcomers around purpose instead of inventorying contents, prefers inline or small sibling-batch prose repairs over slow per-page worker fan-out, makes examples start from package-ID usage evidence before type/member-only searches and requires each example to introduce the consumer task before the code, allows multi-type Microsoft Learn-style scenario samples when they better explain the consumer workflow, keeps extension-method examples on readable declaring-class type pages under `.docfx/api/types/` instead of synthetic method-UID filenames or namespace pages that mix extra `uid:` / `example:` blocks into the overview, flags weak skip-compile reasons, requires deterministic `.docfx/skip-compile-allowlist.json` entries for any pre-existing approved skip waivers, treats newly introduced or unallowlisted skip markers as fail-level diagnostics that do not suppress compilation, establishes reflection-backed packets with `--build-api-model --project-manifest` before full-run authoring, forces mid-audit continuations to name that manifest or the sequential assessment/namespace-first fallback explicitly, requires those continuations to restate the fast `docfx.cs --json` rerun cadence, the exact final `docfx.cs --build-api-model --validate-samples --verify-docfx-build --json` gate, and the clean JSON completion contract instead of generic “verify later” prose, treats batch size only as rerun cadence rather than permission to stop, runs a completion repair loop that treats every diagnostic as active work regardless of age or volume, treats newly surfaced follow-on diagnostics as the next repair queue instead of a stop point, reruns packet discovery with `--build-api-model --project-manifest` when fast source-scan packets are unnamed or zero-project, falls back to sequential namespace-first or assessment work queue order when packet discovery is still unusable, treats `EXAMPLE_MISSING`, `EXAMPLE_LEAD_MISSING`, `EXAMPLE_ADVANCED_LEAD_MISSING`, `FAMILY_ANCHOR_EXAMPLE_MISSING`, `SAMPLE_STRUCTURE_INVALID`, `FAIL_NEW_SKIP_MARKER_INTRODUCED`, `SAMPLE_SKIP_NOT_ALLOWLISTED`, and `INTERIM_ARTIFACT_IN_WORKTREE` queues as core work rather than checkpoints or quality backlog, drives large example and lead queues through a concrete fast-path micro-loop (next item or next 3-5 items → rerun → continue), suppresses progress-table/checkpoint output until the completion contract is clean or a real external blocker is reported, treats premature completion-shaped handoffs as execution-protocol failures while the queue is still dirty, reserves the final `--build-api-model --validate-samples --verify-docfx-build` verification for the real end of the queue, exposes `summary.fullVerificationRan`, `summary.canClaimCompletion`, `summary.remainingWorkItems`, `summary.remainingDiagnosticsByCode`, `summary.newlyIntroducedSkipMarkers`, and `summary.interimArtifacts` as machine-readable final gates, reruns the fast `docfx.cs --json` after edits until the queue is empty, then runs the build-backed verification before completion, preserves manual edits and authored Markdown during cleanup, skips recursive generated-output cleanup when a target directory contains documentation or source files, and returns deterministic exit codes plus `--json` reports (including process counts, phase timings, warning counts, and skip-marker accounting) so CI can gate on real failures instead of AI claims. | +| [dotnet-benchmark](skills/dotnet-benchmark/SKILL.md) | Discovers, prioritizes, and authors trustworthy BenchmarkDotNet experiments for a .NET type following codebelt conventions and using the `Codebelt.Extensions.BenchmarkDotNet.Console` runner. It inspects implementation code, call sites, tests, existing benchmarks, and available profiles instead of benchmarking every public member; ranks likely high-impact operations; selects representative typical, boundary, scaling, and adverse cases; and rejects external-I/O or service-level questions that need profiling, macrobenchmarks, or load tests. It creates fair current-versus-candidate comparisons only when observable work is equivalent, uses baseline-free single-operation characterization when no honest comparator exists, prevents unrelated construction/formatting/equality/hash ratios, requires exact per-case correctness oracles plus a semantic preflight for truthful workload labels, hard-gates interpretation on a complete valid BenchmarkDotNet summary, preserves workload invariants such as selectivity and hit/miss ratios as sizes scale, distinguishes deferred pipeline creation from terminal/materialization work, and performs Release build, discovery listing, and dry execution before any explicit full run. Explicit `yolo` mode auto-accepts routine repo-derived defaults and the proposed plan, then proceeds through build/list/dry validation without confirmation churn; only a separate explicit human instruction can start a full performance run. Its runner preflight recognizes the standard Slim/runtime setup and explains when `SkipBenchmarksWithReports = true` plus a matching `reports/tuning/` artifact deliberately filters a benchmark, preventing needless class renames, disassembly, or tool thrash; after the first valid full result it stops unless deeper diagnostics could change a real engineering decision. Harness setup remains adaptive: it detects `.slnx`/`.sln`, CPM, existing `tuning/` projects, and a reusable `tooling/` runner, onboards only missing pieces, resolves package versions dynamically, and keeps the benchmark class in the SUT namespace. | ### Copyable Install Commands @@ -211,6 +213,12 @@ npx skills add https://github.com/codebeltnet/agentic --skill dotnet-change-impa npx skills add https://github.com/codebeltnet/agentic --skill dotnet-docfx-digest ``` +`dotnet-benchmark` + +```bash +npx skills add https://github.com/codebeltnet/agentic --skill dotnet-benchmark +``` + ### Why git-visual-commits? Commit messages are the most-read documentation in any codebase — yet they're usually an afterthought. "fix stuff", "wip", "address PR feedback" tells you nothing six months later. Writing good commits takes discipline, and when you're in flow, it's the first thing that slips. @@ -508,12 +516,12 @@ Generating a `.snk` file traditionally requires `sn.exe`, which is only availabl Most repositories start with `git init` followed by committing everything directly to `main`. This works — until someone force-pushes to main, or a half-finished feature lands without review. By the time you add branch protection, the history is already messy. -**trunk-first-repo** flips this: main starts empty and stays clean from the very first commit. Every piece of content enters through a pull request, and `push remote ` can be invoked later when the remote is ready so the first remote push sends `main` before any feature branch. This gives you: +**trunk-first-repo** flips this: main starts empty and stays clean from the very first commit. Every piece of content enters through a pull request, and the skill now branches cleanly depending on when `origin` becomes available: configure it during setup and later you only push `HEAD`; add it later with `push remote ` and the skill first verifies the feature-branch/empty-main state before publishing. This gives you: - **Review from day one** — no "we'll add branch protection later" that never happens - **Clean, meaningful history** — main tells the story of reviewed, approved changes - **Version-aware branches** — `v0.0.1/spike-auth` vs `v1.0.0/release-prep` signals project maturity at a glance -- **Safer first push** — invoke `push remote ` to push `main` by ref from the feature branch, then push the feature branch without manually deleting files or checking out `main` +- **Safer first push** — if `origin` is ready during setup, the summary points straight to `git push -u origin HEAD`; if not, invoke `push remote ` to verify the branch state, push `main` by ref from the feature branch, then push the feature branch without manually deleting files or checking out `main` - **Zero-friction setup** — one skill invocation, not a 10-step checklist ### Why git-remote-release? @@ -588,6 +596,32 @@ API documentation rots the moment code changes. A new public type ships without - **Cleanup keeps authored docs and is opt-in** — generated-metadata cleanup runs only when `--clean-generated-metadata` is explicitly passed, and even then only after the API model is built so it never deletes YAML the run relied on; `.docfx/**/*.md` overwrite files, namespace pages, includes, and config are documentation outputs, not disposable build artifacts, and cleanup is limited to known metadata files and safe site-output directories that contain no authored documentation or source files, - **CI-friendly** — deterministic exit codes plus `--json` reports let pipelines fail on actual documentation drift instead of trusting an agent's claim that it checked. +### Why dotnet-benchmark? + +Setting up a benchmark "properly" is only half the problem. A benchmark can compile and still answer the wrong question: public members get measured because they are visible, unrelated operations share a meaningless baseline, random inputs miss real branches, setup leaks into the timed path, or a disk/service bottleneck is disguised as a microbenchmark. The result looks scientific but gives an engineer little trustworthy optimization evidence. + +**dotnet-benchmark** combines codebelt harness conventions with an evidence-driven performance investigation. It inspects the type, callers, tests, existing benchmarks, and available profiles; ranks the operations most likely to matter; selects a small set of representative experiments; and explicitly rejects misleading measurements. When no profile exists, it labels the result as source-informed exploration rather than claiming to have found an application bottleneck. + +- **Works on existing repos** — detects your solution format, package-management style, and any runner you already have (`benchmark-runner`, `bdn-runner`, …) and reuses it instead of forcing a new layout +- **Codebelt convention by default** — `tuning/` benchmark projects, a single `tooling/` runner host, and `reports/` output, mirroring `codebeltnet/cuemon` and `codebeltnet/xunit` +- **Evidence-backed candidate selection** — ranks operations from call-site frequency, input scaling, allocations, contention, optimization leverage, and measurement fitness instead of treating public-member coverage as thoroughness +- **Honest experiment shapes** — creates equivalent current-versus-candidate comparisons, baseline-free single-operation characterization, or profiling/macrobenchmark guidance; unrelated construction, formatting, equality, and hashing never receive misleading ratios +- **Hard validity gate** — reads the complete BenchmarkDotNet summary after dry execution and after any full run, treating `NA`, `Benchmarks with issues`, failed jobs, setup/cleanup exceptions, validation errors, or a partial matrix as invalid rather than letting surviving rows stand in for the whole experiment +- **Representative workloads** — derives typical, boundary, scaling, hit/miss, valid/invalid, and other adverse-but-real cases from repository evidence, uses coupled scenario sources instead of accidental parameter Cartesian products, and keeps selectivity, branch mix, and other workload invariants stable unless they are explicit scenarios +- **Correctness before speed** — validates exact outputs, counts, status, exception behavior, and state transitions for every case outside the measured path before a full performance run, while still allowing intended zero-match boundaries +- **Semantic preflight** — checks the full method/job/parameter/scenario matrix before build/list/dry/full interpretation, proves independently derived exact expected results, verifies workload labels such as selectivity, hit rate, and type mix, and rejects accidental fast paths or fake type-filter workloads +- **Terminal-operation honesty** — distinguishes deferred query creation, terminal operators such as `Count()`/`Any()`/`First()`, explicit enumeration, and materialization, so a `List.Count` fast path is never sold as predicate traversal +- **Specialized investigations** — handles mutation, async, contention, cold start, and exception paths explicitly, routing `ThreadingDiagnoser`, `ExceptionDiagnoser`, disassembly, or EventPipe only when each answers the stated question +- **Namespace-correct** — the `*Benchmark` class lives in the same namespace as the code it measures, via a `RootNamespace` override, so type discovery and reports stay clean +- **Allocations always measured** — `[MemoryDiagnoser]` is on by default +- **Multi-runtime aware** — the runner host runs on .NET 9/10, but its BenchmarkDotNet jobs can compare `net48`, `net8.0`, `net9.0`, and `net10.0` +- **Latest stable packages** — `BenchmarkDotNet`, `BenchmarkDotNet.Diagnostics.Windows`, and `Codebelt.Extensions.BenchmarkDotNet.Console` versions are resolved from NuGet, not hardcoded +- **Layered validation** — verifies the Release build, lists discovered cases, dry-executes lifecycle and correctness wiring, and reruns build/list/dry after benchmark-owned validity fixes; it never turns that smoke check into a performance claim or launches the full machine-sensitive run unless asked +- **Yolo mode without permission creep** — saying `yolo` auto-accepts evidence-backed defaults, skips routine plan/execution confirmations, and continues through build/list/dry validation; only an explicit human instruction can start a full benchmark run, and the mode never implies a commit, push, or unrelated external action +- **Proportional escalation** — after the first valid full result, it asks whether the issue is reproducible, material, and likely to change a real engineering decision before suggesting disassembly, EventPipe/ETW, repeated reruns, or alternative implementations +- **Honest Slim reporting** — reports the active `BenchmarkWorkspaceOptions.Slim` job accurately, including when its one-warmup developer-oriented shape limits runtime- or JIT-sensitive conclusions, instead of silently swapping the runner configuration +- **Report-aware runner preflight** — inspects the canonical `BenchmarkWorkspaceOptions.Slim` runtime jobs, `SkipBenchmarksWithReports`, and matching `reports/tuning/` artifacts before touching benchmark code, so an intentional existing-report skip is explained instead of triggering disassembly, renaming, or speculative rewrites + ## Repository structure ``` diff --git a/scripts/validate-skill-templates.ps1 b/scripts/validate-skill-templates.ps1 index ec32751..1035f56 100644 --- a/scripts/validate-skill-templates.ps1 +++ b/scripts/validate-skill-templates.ps1 @@ -60,6 +60,25 @@ function Get-RepoFileList { return @($output | Where-Object { $_ -like "$prefix*" } | ForEach-Object { $_.Substring($prefix.Length) } | Sort-Object) } +function Get-TrackedRepoPaths { + param( + [string]$RepoRoot, + [string]$GitRef + ) + + if ([string]::IsNullOrWhiteSpace($GitRef)) { + $output = git -C $RepoRoot ls-files 2>$null + } else { + $output = git -C $RepoRoot ls-tree -r --name-only $GitRef 2>$null + } + + if ($LASTEXITCODE -ne 0) { + throw 'Unable to enumerate tracked repository files for validation.' + } + + return @($output | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | Sort-Object) +} + function Get-FileText { param( [string]$RepoRoot, @@ -83,6 +102,109 @@ function Get-FileText { return ($output -join [Environment]::NewLine) } +function Test-IsLocalShellPolicyScanCandidate { + param([string]$RelativePath) + + $normalizedPath = $RelativePath -replace '\\', '/' + if ($normalizedPath.StartsWith('./')) { + $normalizedPath = $normalizedPath.Substring(2) + } + if ($normalizedPath -eq 'CHANGELOG.md') { + return $false + } + + if ($normalizedPath -eq 'scripts/validate-skill-templates.ps1') { + return $false + } + + if ($normalizedPath -like 'skills/skill-creator-agnostic/*') { + return $false + } + + $extension = [System.IO.Path]::GetExtension($normalizedPath).ToLowerInvariant() + return @('.cs', '.json', '.md', '.ps1', '.yml', '.yaml') -contains $extension +} + +function Get-LocalShellPolicyFindingsFromContentItems { + param([object[]]$Items) + + $legacyShell = 'power' + 'shell' + $legacyShellPattern = [regex]::Escape($legacyShell) + + $rules = @( + [pscustomobject]@{ + Pattern = "(?i)\b$legacyShellPattern(?:\.exe)?\s+-" + Message = 'Local command examples must use `pwsh` 7+ instead of the legacy executable.' + } + [pscustomobject]@{ + Pattern = "(?i)\bshell:\s*$legacyShellPattern(?:\.exe)?\b" + Message = 'Workflow steps that explicitly choose a `pwsh`-style shell must use `pwsh`.' + } + ) + + $findings = [System.Collections.Generic.List[object]]::new() + + foreach ($item in @($Items)) { + if ($null -eq $item -or [string]::IsNullOrWhiteSpace([string]$item.Path)) { + continue + } + + $relativePath = [string]$item.Path -replace '\\', '/' + if ($relativePath.StartsWith('./')) { + $relativePath = $relativePath.Substring(2) + } + if (-not (Test-IsLocalShellPolicyScanCandidate -RelativePath $relativePath)) { + continue + } + + $content = if ($null -eq $item.Content) { '' } else { [string]$item.Content } + $lines = [regex]::Split($content, '\r?\n') + + for ($index = 0; $index -lt $lines.Length; $index++) { + $line = $lines[$index] + $message = $null + + foreach ($rule in $rules) { + if ($line -match $rule.Pattern) { + $message = $rule.Message + break + } + } + + if ($null -ne $message) { + $findings.Add([pscustomobject]@{ + Path = $relativePath + LineNumber = $index + 1 + Line = $line + Message = $message + }) + } + } + } + + return @($findings) +} + +function Get-LocalShellPolicyFindings { + param( + [string]$RepoRoot, + [string]$GitRef + ) + + $items = foreach ($path in Get-TrackedRepoPaths -RepoRoot $RepoRoot -GitRef $GitRef) { + if (-not (Test-IsLocalShellPolicyScanCandidate -RelativePath $path)) { + continue + } + + [pscustomobject]@{ + Path = $path + Content = Get-FileText -RepoRoot $RepoRoot -RelativePath $path -GitRef $GitRef + } + } + + return @(Get-LocalShellPolicyFindingsFromContentItems -Items $items) +} + function Assert-Contains { param( [string]$Name, @@ -519,6 +641,133 @@ Add-ValidationResult -Results $results -Name 'All repo-managed skills keep YAML } } +Add-ValidationResult -Results $results -Name 'Active local shell guidance rejects only legacy PowerShell executable use' -Action { + $findings = @(Get-LocalShellPolicyFindings -RepoRoot $repoRoot -GitRef $Ref) + + if ($findings.Count -gt 0) { + $details = @( + $findings | ForEach-Object { + '{0}:{1}: {2}`n {3}' -f $_.Path, $_.LineNumber, $_.Message, $_.Line.Trim() + } + ) + + throw ("Legacy `powershell` executable usage is still present; update these lines:`n" + ($details -join "`n")) + } +} + +Add-ValidationResult -Results $results -Name 'Local shell policy scanner rejects legacy executable invocations and allows valid terminology' -Action { + $legacyShell = 'power' + 'shell' + $legacyExe = $legacyShell + '.exe' + $fence = [string]([char]96) * 3 + + $cases = @( + [pscustomobject]@{ + Name = 'maintained skill legacy command' + Path = 'skills/example/case-01.md' + Content = "$legacyShell -NoProfile -File ./scripts/example.ps1" + ExpectViolation = $true + } + [pscustomobject]@{ + Name = 'case-variant legacy command' + Path = 'skills/example/case-02.md' + Content = ('PoWeR' + 'ShElL -NoProfile -File ./scripts/example.ps1') + ExpectViolation = $true + } + [pscustomobject]@{ + Name = 'legacy executable command' + Path = 'skills/example/case-03.md' + Content = "$legacyExe -Command Get-ChildItem" + ExpectViolation = $true + } + [pscustomobject]@{ + Name = 'pwsh script command' + Path = 'skills/example/case-04.md' + Content = 'pwsh -NoProfile -File ./scripts/example.ps1' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'bash command' + Path = 'skills/example/case-05.md' + Content = 'bash ./scripts/example.sh' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'workflow bash shell' + Path = '.github/workflows/case-06.yml' + Content = 'shell: bash' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'workflow pwsh shell' + Path = '.github/workflows/case-07.yml' + Content = 'shell: pwsh' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'powershell fence' + Path = 'skills/example/case-08.md' + Content = $fence + 'powershell' + [Environment]::NewLine + '$value = 1' + [Environment]::NewLine + $fence + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'PowerShell session prose' + Path = 'skills/example/case-09.md' + Content = 'Use a PowerShell session if that is the host the user already chose.' + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'released changelog exclusion' + Path = 'CHANGELOG.md' + Content = "$legacyShell -File ./scripts/example.ps1" + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'obsolete skill exclusion' + Path = 'skills/skill-creator-agnostic/SKILL.md' + Content = "$legacyShell -File ./scripts/example.ps1" + ExpectViolation = $false + } + [pscustomobject]@{ + Name = 'legacy workflow shell' + Path = '.github/workflows/case-11.yml' + Content = "shell: $legacyShell" + ExpectViolation = $true + } + ) + + $items = foreach ($case in $cases) { + [pscustomobject]@{ + Path = $case.Path + Content = $case.Content + } + } + + $findings = Get-LocalShellPolicyFindingsFromContentItems -Items $items + + foreach ($case in $cases) { + $caseFindings = @($findings | Where-Object { $_.Path -eq $case.Path }) + + if ($case.ExpectViolation -and $caseFindings.Count -eq 0) { + throw "Expected a finding for '$($case.Name)' but none was reported." + } + + if (-not $case.ExpectViolation -and $caseFindings.Count -gt 0) { + throw "Expected no finding for '$($case.Name)' but found: $($caseFindings[0].Message)" + } + } +} + +Add-ValidationResult -Results $results -Name 'Repository docs define the local shell execution policy' -Action { + $agents = Get-FileText -RepoRoot $repoRoot -RelativePath 'AGENTS.md' -GitRef $Ref + $contributing = Get-FileText -RepoRoot $repoRoot -RelativePath 'CONTRIBUTING.md' -GitRef $Ref + + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle '## Local Shell Execution' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'Agents may use any appropriate local shell.' + Assert-Contains -Name 'AGENTS.md' -Content $agents -Needle 'use PowerShell 7+ through `pwsh`; never invoke `powershell` or `powershell.exe`' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1' + Assert-Contains -Name 'CONTRIBUTING.md' -Content $contributing -Needle 'pwsh -NoProfile -File ./scripts/validate-skill-templates.ps1 -Ref HEAD' +} + Add-ValidationResult -Results $results -Name 'App skill collects target framework and conditional web_variant' -Action { $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-new-app-slnx/FORMS.md' -GitRef $Ref Assert-Contains -Name 'dotnet-new-app-slnx/FORMS.md' -Content $forms -Needle '### target_framework' @@ -549,6 +798,8 @@ Add-ValidationResult -Results $results -Name 'App skill documents web-family App Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'Treat the scaffold as a fidelity copy of the documented template set, not a "best effort" approximation.' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle '## Step 3: Resolve Dynamic Dependency Versions' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'scripts/resolve-package-versions.ps1' + Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File "/scripts/resolve-package-versions.ps1" -TargetFramework ' + Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File "/scripts/restore-missing-shared-assets.ps1"' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'If the host does not render native form controls, follow the deterministic plain-text fallback defined in `FORMS.md` instead of improvising your own questioning style.' Assert-Contains -Name 'dotnet-new-app-slnx/SKILL.md' -Content $skill -Needle 'Consistency matters more than creativity during parameter collection.' @@ -659,6 +910,7 @@ Add-ValidationResult -Results $results -Name 'App reference guide uses ROOT_NAME Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Directory.Packages.props` is the authoritative version source for app scaffolds.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Do **not** duplicate `` inside the generated app or test `.csproj` files as a workaround.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'scripts/resolve-package-versions.ps1' + Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'pwsh -NoProfile -File "/scripts/resolve-package-versions.ps1" -TargetFramework {TARGET_FRAMEWORK}' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle '`testenvironments.json` is required output for the scaffold.' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'MinVer may report a bootstrap pre-release such as `0.0.0-alpha.0`' Assert-Contains -Name 'dotnet-new-app-slnx/references/app.md' -Content $guide -Needle 'Where `{AppType}` maps to the emitted project suffix:' @@ -772,6 +1024,8 @@ Add-ValidationResult -Results $results -Name 'Library skill documents PROJECT_NA Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle '{PROJECT_NAME}' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle '{DOCFX_TARGET_FRAMEWORK}' + Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File "/scripts/restore-missing-shared-assets.ps1"' + Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'In PowerShell, prefer .NET file APIs' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'If the host does not render native form controls, follow the deterministic plain-text fallback defined in `FORMS.md` instead of improvising your own questioning style.' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'Consistency matters more than creativity during parameter collection.' Assert-Contains -Name 'dotnet-new-lib-slnx/SKILL.md' -Content $skill -Needle 'treat a blank response as accepting that shown value' @@ -801,6 +1055,7 @@ Add-ValidationResult -Results $results -Name 'Library reference guide uses curre Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'current working directory' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'do not create an extra solution-named wrapper folder' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'src/{PROJECT_NAME}/{PROJECT_NAME}.csproj' + Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'Recommended PowerShell approach for rewritten templates:' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'offer every other generally supported non-preview .NET LTS and STS channel' Assert-Contains -Name 'dotnet-new-lib-slnx/references/library.md' -Content $guide -Needle 'highest selected generally supported non-preview executable TFM' } @@ -832,10 +1087,85 @@ Add-ValidationResult -Results $results -Name 'Benchmark runner wildcard is prese Assert-Match -Name 'benchmark-program.cs' -Content $program -Pattern 'namespace\s+\{BENCHMARK_RUNNER_NAMESPACE\};' } +Add-ValidationResult -Results $results -Name 'dotnet-benchmark enforces valid, proportionate experiments and preserves honest comparison semantics' -Action { + $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/SKILL.md' -GitRef $Ref + $forms = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/FORMS.md' -GitRef $Ref + $candidateSelection = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/candidate-selection.md' -GitRef $Ref + $experimentDesign = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/experiment-design.md' -GitRef $Ref + $benchmarkEssentials = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md' -GitRef $Ref + $runnerPreflight = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/references/runner-preflight.md' -GitRef $Ref + $comparison = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/comparison-benchmark.cs' -GitRef $Ref + $operation = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/assets/operation-benchmark.cs' -GitRef $Ref + $evals = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/evals/evals.json' -GitRef $Ref + $fixtureFiles = Get-RepoFileList -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/evals/files' -GitRef $Ref + $validateSkillScript = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-benchmark/scripts/validate-skill.ps1' -GitRef $Ref + + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'A microbenchmark measures a suspected cost under a defined workload; it does not prove that the type is an application bottleneck.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Read `references/candidate-selection.md`' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Read `references/experiment-design.md`' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--list flat' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '--job dry' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot ""' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle '#### Yolo mode' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Start a full performance run only after an explicit human instruction to run it now.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Yolo never authorizes a full performance run.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'read `references/runner-preflight.md`' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'reports.wouldSkipRequestedBenchmark' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'complete BenchmarkDotNet summary' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'When a parameter is only size or payload' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Semantic preflight' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'independently derived exact expected result' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'Successful execution is not a correctness oracle.' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' + Assert-Contains -Name 'dotnet-benchmark/SKILL.md' -Content $skill -Needle 'After the first valid full result' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'Auto-discover the highest-value performance questions (Recommended)' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '### candidate_plan_confirmation' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle '## Yolo mode override' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'skip `candidate_plan_confirmation`' + Assert-Contains -Name 'dotnet-benchmark/FORMS.md' -Content $forms -Needle 'explicit human instruction to start a full performance run' + Assert-Contains -Name 'candidate-selection.md' -Content $candidateSelection -Needle '## Candidate matrix' + Assert-Contains -Name 'candidate-selection.md' -Content $candidateSelection -Needle '## Profiling-first gate' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Correctness oracle' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Do not compare unrelated operations.' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Workload invariants' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Benchmark validity gate' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Deferred execution and terminal operations' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle '## Semantic preflight' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Successful execution is not a correctness oracle.' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' + Assert-Contains -Name 'experiment-design.md' -Content $experimentDesign -Needle 'Do not benchmark a homogeneous all-match store and call it type filtering unless the all-match path is the actual subject.' + Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle 'one warmup iteration plus controlled iteration counts' + Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Deferred pipelines and terminal operations' + Assert-Contains -Name 'benchmarkdotnet-essentials.md' -Content $benchmarkEssentials -Needle '## Result-validity gate' + Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'SkipBenchmarksWithReports = true' + Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'Anti-thrashing rule' + Assert-Contains -Name 'runner-preflight.md' -Content $runnerPreflight -Needle 'pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "" -BenchmarkType ' + Assert-Contains -Name 'validate-skill.ps1' -Content $validateSkillScript -Needle 'if ([string]::IsNullOrWhiteSpace($SkillRoot))' + Assert-Contains -Name 'validate-skill.ps1' -Content $validateSkillScript -Needle 'Harness detector validation requires pwsh 7+' + Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle '{EQUIVALENCE_CHECK}' + Assert-Contains -Name 'comparison-benchmark.cs' -Content $comparison -Needle 'Baseline = true' + Assert-Contains -Name 'operation-benchmark.cs' -Content $operation -Needle 'Do not add Baseline = true merely to produce a ratio column.' + Assert-NotContains -Name 'operation-benchmark.cs measured method' -Content ($operation -replace '// Do not add Baseline = true merely to produce a ratio column\.', '') -Needle 'Baseline = true' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'RouteMatcher.IsMatch' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'cannot prove whether file I/O or JSON parsing dominates' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'ThreadingDiagnoser' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'YOLO mode:' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'Acme.Core.ParserBenchmark-report-github.md' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'LegacyAliasQuery benchmark and summary' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'InMemoryTestStore benchmark' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'LowSelectivity, HalfMatches, and AllMatch' + Assert-Contains -Name 'dotnet-benchmark/evals/evals.json' -Content $evals -Needle 'TraitFilter helper only runs in test discovery' + if (@($fixtureFiles | Where-Object { $_ -match '(^|/)(obj|bin|BenchmarkDotNet\.Artifacts)(/|$)' }).Count -gt 0) { + throw 'dotnet-benchmark eval fixtures must not include obj/, bin/, or BenchmarkDotNet.Artifacts paths' + } +} + Add-ValidationResult -Results $results -Name 'Strong-name skill matches FORMS summary flow and 1024-bit default' -Action { $skill = Get-FileText -RepoRoot $repoRoot -RelativePath 'skills/dotnet-strong-name-signing/SKILL.md' -GitRef $Ref Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'compute the defaults silently, and present a single summary for confirmation' Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'default: 1024' + Assert-Contains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'Run this PowerShell command block with `pwsh` 7+ in the target directory:' Assert-NotContains -Name 'dotnet-strong-name-signing/SKILL.md' -Content $skill -Needle 'default: 4096' } diff --git a/skills/dotnet-benchmark/FORMS.md b/skills/dotnet-benchmark/FORMS.md new file mode 100644 index 0000000..5678e5a --- /dev/null +++ b/skills/dotnet-benchmark/FORMS.md @@ -0,0 +1,81 @@ +# Parameter Form + +`dotnet-benchmark` derives most decisions from the repository, the target type, and the user's request. Ask only unresolved fields, one at a time. Prefer the host's native structured input controls when available. If the host does not provide them, use the deterministic plain-text fallback below without changing field order or choices. + +## Yolo mode override + +Yolo mode is request syntax, not a field to ask about. Activate it only when the current request explicitly says `yolo`, `yolo mode`, `auto-proceed`, or clearly instructs the agent to use defaults without routine confirmation. + +While yolo mode is active: + +- skip `performance_intent` and use auto-discovery unless the request already specifies another intent; +- skip `workload_context` when repository evidence supports a defensible workload; state the chosen assumption and continue; +- show the candidate plan as a progress update, then skip `candidate_plan_confirmation` and treat its recommended default as accepted; +- skip `execution_depth` and use build, list, and dry execution; +- ask only when missing information creates material correctness/design risk with no safe default; +- require a separate explicit human instruction to start a full performance run. `yolo` by itself never selects the full-run option, and an agent recommendation never supplies that authority. + +The override lasts for the current benchmark task only and does not waive correctness checks, repository policy, external-side-effect restrictions, or blockers. + +## Fields + +### sut_type + +- **type:** text +- **prompt:** "Which type do you want to investigate? Use the namespace-qualified name if it may be ambiguous." +- **placeholder:** "e.g. Cuemon.DateSpan or Acme.Buffers.RingBuffer" +- **required:** true +- **description:** Skip this field when the user already named an unambiguous type. Resolve the declaration in source rather than relying only on the name. + +### performance_intent + +- **type:** single-choice +- **prompt:** "What should the benchmark investigation optimize for?" +- **choices:** + - Auto-discover the highest-value performance questions (Recommended) + - Compare current and candidate implementations + - Characterize one specific member or operation + - Establish a regression benchmark for a known workload +- **default:** Auto-discover the highest-value performance questions (Recommended) +- **required:** true +- **description:** Infer and skip this field when the request already states the operation, comparison, or regression goal. In yolo mode, skip it and use auto-discovery unless another intent is explicit. Auto-discovery ranks candidates from implementation, usage, tests, and any available profiling evidence; it does not claim a measured application bottleneck without profile or telemetry data. + +### workload_context + +- **type:** text +- **prompt:** "I could not infer a representative workload confidently. What inputs, sizes, frequency, and operating conditions matter in production?" +- **required:** false +- **description:** Show this field only when tests, call sites, documentation, or supplied profiling evidence do not establish a representative workload and choosing one would materially affect correctness. Offer the strongest repo-derived workload as a selectable recommended choice when one exists, plus the option to enter a custom value. In yolo mode, state and use the strongest defensible repo-derived workload; ask only when no safe choice exists. + +### candidate_plan_confirmation + +- **type:** single-choice +- **prompt:** "Use the proposed benchmark questions, workloads, and validation strategy?" +- **choices:** + - Use the proposed plan (Recommended) + - Adjust the selected operations or inputs +- **default:** Use the proposed plan (Recommended) +- **required:** true +- **description:** Present the evidence-backed experiment plan immediately before this field. Include selected and rejected candidates, comparable baseline/candidate pairs, parameter cases, correctness oracle, lifecycle risks, and whether the plan is exploratory or profile-backed. In yolo mode, present it as a progress update and skip this field by accepting the recommended choice. + +### execution_depth + +- **type:** single-choice +- **prompt:** "How far should validation run?" +- **choices:** + - Build, list, and dry-execute the benchmark (Recommended) + - Run the full performance benchmark after validation +- **default:** Build, list, and dry-execute the benchmark (Recommended) +- **required:** true +- **description:** A dry execution validates discovery and lifecycle but produces no trustworthy performance conclusion. A full run can be slow and machine-sensitive. In yolo mode, skip this field and select the recommended build/list/dry option. Only an explicit human instruction such as "run it fully now" or "measure it now" selects the full-run option; yolo, plan acceptance, or an agent recommendation is not enough. + +## Presentation rules + +1. Detect the yolo mode override before presenting any field. When active, skip the routine fields described above and continue autonomously; otherwise ask one field at a time and wait for the answer before presenting the next unresolved field. +2. Skip fields already answered by the conversation or reliable repository evidence. Do not ask the user to choose BenchmarkDotNet attributes, a "simple/complex" tier, or extra runtimes unless those choices are part of the user's goal. +3. When native controls are unavailable, start with `Field: `, repeat the prompt verbatim, show numbered choices in declared order, and accept either a number or exact choice text. +4. For a text field with a repo-derived suggestion, show `1. Use "" (Recommended)` and `2. Enter a custom value`. Do not fabricate a suggestion when the evidence is weak. +5. Present each default first and append `(Recommended)` when it is not already included. A blank response accepts the displayed default. +6. After each answer, restate the normalized value in one short line before continuing. +7. Before `candidate_plan_confirmation`, show the experiment plan produced after source inspection. This is the final design confirmation; do not ask a second generic confirmation afterward. +8. If the user chooses to adjust the plan, collect only the disputed operation or workload, revise the plan, and present `candidate_plan_confirmation` again. diff --git a/skills/dotnet-benchmark/SKILL.md b/skills/dotnet-benchmark/SKILL.md new file mode 100644 index 0000000..8d35f41 --- /dev/null +++ b/skills/dotnet-benchmark/SKILL.md @@ -0,0 +1,203 @@ +--- +name: dotnet-benchmark +description: > + Discover, prioritize, and author trustworthy BenchmarkDotNet performance experiments for a .NET type while following codebelt engineering conventions and using the Codebelt.Extensions.BenchmarkDotNet Console runner. Use whenever a user wants to benchmark, micro-benchmark, performance-test, profile, optimize, compare implementations, investigate allocations or contention, or find likely bottlenecks in a .NET type or method. The skill inspects source and usage evidence, ranks high-value operations instead of every public member, selects representative workloads, rejects misleading microbenchmarks, creates or reuses the tuning/ and tooling/ harness, preflights existing-report skips, semantic-preflights workload correctness, validates discovery, and keeps full runs human-initiated. When the user says yolo, it auto-accepts routine defaults and proceeds through safe validation without confirmation churn. +--- + +# Evidence-Driven .NET Benchmarking + +Create the smallest benchmark suite that can answer the most valuable performance questions about the supplied type. Follow the repository's established conventions first, then apply the codebelt `tuning/` benchmark project and `tooling/` runner layout where the repository has no stronger local pattern. + +## Critical benchmark contract + +- Treat a benchmark as an experiment, not as public-member coverage. Do not benchmark every constructor, property, or method merely because it exists. +- A microbenchmark measures a suspected cost under a defined workload; it does not prove that the type is an application bottleneck. Prefer production telemetry or a CPU/allocation/contention profile when the user asks where an application is slow. If only a type is provided, perform source-informed candidate discovery and label the result as an exploratory benchmark plan. +- Rank candidates using evidence from the implementation, call sites, tests, documentation, existing benchmark results, and profiles. Never invent usage frequency, input distributions, or a competing implementation. +- Compare only operations that produce equivalent observable work. Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation. +- Use `Baseline = true` only when at least two benchmark methods form a meaningful comparison group. A single-operation scaling or regression benchmark needs no fabricated baseline. When a class has several comparison groups, assign categories and one baseline inside each category. +- Before accepting `Baseline = true`, answer: do the baseline and candidate perform equivalent consumer-visible work over identical logical input? If not, remove the baseline or split the experiment. +- Before build or dry validation, inspect benchmark attributes for internal coherence: `Baseline = true` only on equivalent observable work, `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]` only with meaningful `[BenchmarkCategory]` values, every baseline category has an equivalent peer, single-operation characterization benchmarks have no fabricated baseline, benchmark descriptions name the real measured terminal operation, and decorative grouping attributes are removed. +- Keep correctness outside the timed path but inside the verification workflow. Every scenario must define the input, intended operation, exact expected observable result, how that result was derived independently of the measured API path, and any promised workload characteristic such as selectivity, hit rate, type mix, or branch distribution. Successful execution is not a correctness oracle. Every parameter or scenario case needs exact observable-result validation for count, value, status, exception behavior, output contents, or mutation unless the domain explicitly defines approximation. Valid zero-match cases stay valid; if a zero result is unintended, fix the workload before measurement. +- When one parameter only scales size, payload, or another magnitude, keep selectivity, hit/miss ratio, valid/invalid ratio, branch distribution, collision rate, string shape/encoding, type mix, entropy, and cache state stable unless one of them is intentionally exposed as a named scenario or parameter. +- A benchmark is invalid for performance interpretation until the complete BenchmarkDotNet summary shows the full intended method/job/parameter matrix without `NA`, `Benchmarks with issues`, setup/cleanup exceptions, validation errors, failed jobs/runtimes, or missing combinations. Report the exact failing method, job, and parameter case, do not treat surviving rows as a completed benchmark, fix only benchmark-owned causes, rerun build/list/dry validation, and require fresh explicit human authority before any replacement full run. +- Treat deferred pipeline creation, terminal operations such as `Count()`, `Any()`, or `First()`, explicit full enumeration, and materialization through `ToArray()` or `ToList()` as different workloads. Inspect fast paths such as `List.Count` before describing a benchmark as enumeration or using it as the baseline for predicate traversal. +- Keep external I/O, network latency, database latency, sleeps, logging, and random data generation out of measured microbenchmark methods. Recommend profiling, a macrobenchmark, or a load test when those effects are the actual question. +- Always distinguish code that was built, smoke-executed, or fully measured. Never report performance numbers from a build, discovery listing, dry run, or unexecuted benchmark. +- Start a full performance run only after an explicit human instruction to run it now. Never infer that authority from yolo mode, defaults, plan acceptance, an agent recommendation, or the existence of a runnable benchmark. +- When a Codebelt runner appears to list or execute nothing, inspect `SkipBenchmarksWithReports` and matching `reports/tuning/` artifacts before changing benchmark code. An existing report is an expected skip condition, not a reason to rename a lean class or add diagnosers. + +## Workflow + +### 1. Resolve intent with minimal questioning + +Read `FORMS.md` and use its one-field-at-a-time interaction only for information that is not already clear. A named type plus a request such as “find the likely bottlenecks” is sufficient to start inspection. Do not make the user choose an implementation tier or BenchmarkDotNet attributes. + +Default to automatic candidate discovery, the runner's existing/default runtime, and build plus discovery and dry execution validation. Ask about extra runtimes only when cross-runtime comparison is part of the request. Require an explicit human instruction before a full benchmark run because it can be long and machine-sensitive. + +#### Yolo mode + +Activate yolo mode only when the current request explicitly says `yolo`, `yolo mode`, `auto-proceed`, or an equally clear instruction to use defaults without routine confirmation. The mode applies to the current benchmark task only; do not persist it into later requests. + +In yolo mode: + +- infer automatic candidate discovery unless the user already supplied a more specific performance intent; +- use the strongest defensible workload and repository conventions from source, call sites, tests, profiles, and existing benchmarks, recording material assumptions instead of asking the user to approve a 99%-certain default; +- present the compact experiment plan as a progress update and continue immediately without asking `candidate_plan_confirmation`; +- select build, discovery listing, and dry execution as the default validation depth without asking `execution_depth`; +- stop and ask only when a missing fact creates material correctness/design risk and no safe repo-derived choice exists, or when new authority is required; +- preserve all benchmark-quality, repository, and safety gates. Yolo is a convenience mode, not permission to invent workloads, ignore blockers, run unrelated operations, commit, push, or make external changes. + +Yolo never authorizes a full performance run. Start the full benchmark only when the human explicitly asks to run or measure it fully now; otherwise stop after build/list/dry validation. An agent must not promote its own recommendation into that authority. + +### 2. Inspect the repository and harness + +Run the bundled read-only detector before changing files: + +```console +pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "" +``` + +If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. + +Also inspect applicable `AGENTS.md`, solution/project files, `Directory.Build.props`, `Directory.Packages.props`, existing `tuning/` and `tooling/` projects, and nearby benchmark styles. Reuse an existing runner and benchmark project when they fit. Read `references/onboarding.md` only when the detector finds missing or partial harness infrastructure. + +When investigating a benchmark that builds but is not listed or executed, read `references/runner-preflight.md` before inspecting or rewriting the benchmark class. Rerun the detector with `-BenchmarkType ` and inspect the reported runner program, `SkipBenchmarksWithReports` setting, slim/runtime jobs, `reports/tuning/` files, and `wouldSkipRequestedBenchmark`. If a matching existing report explains the skip, preserve the runner and benchmark unchanged, report the matching file, and stop diagnostic escalation. Do not add disassembly, rename the class, disable report skipping, or churn through tools to evade the filter. + +The .NET SDK is the only hard harness prerequisite. If detector status is `not-found`, report that blocker instead of generating unverified project files. If the probe is `timed-out`, `start-failed`, or `failed`, report the probe failure distinctly and verify the SDK through a safe direct check before concluding that it is absent. + +### 3. Resolve the type and gather performance evidence + +Locate the exact declaration, owning `src/` project, namespace, interfaces/base types, and target frameworks. Then read the complete implementation and inspect its collaborators, tests, call sites, documentation, existing benchmarks, and any profiling artifacts available in the repository or supplied by the user. + +Read `references/candidate-selection.md` whenever the user has not already specified the exact operation and workload. Build an evidence-backed candidate matrix and select at most three performance questions for one benchmark class; prefer one focused question when it is clearly dominant. + +Look beyond public methods. Private loops, repeated conversions, allocation-heavy helpers, hashing/equality used by collections, reflection, regex construction, buffer copies, parsing branches, locks, task scheduling, exception-heavy paths, and deferred enumeration can dominate the cost exposed by one public operation. Benchmark the public or internal entry point that represents the real consumer operation, not an arbitrary private helper, unless isolating that helper is the explicit experiment. + +### 4. Decide whether BenchmarkDotNet is the right instrument + +Use a microbenchmark when the work is deterministic, repeatable, isolatable, and small enough to execute many times. If the suspected cost is end-to-end I/O, request concurrency across a service, startup of a whole application, distributed latency, or an unknown application-wide hotspot, explain why a microbenchmark would mislead and propose the narrowest useful next instrument instead. + +When profiling evidence exists, use it to choose the benchmark target. When it does not, state that selection is based on source and usage evidence rather than measured hotspot data. Do not silently turn a hypothesis into a claim. + +### 5. Present the experiment plan + +Before authoring code, present a compact plan with: + +- the performance question and metric: latency/throughput, allocated bytes, scaling, contention, cold start, or exception frequency; +- the selected operation and the evidence that made it important; +- the baseline and candidate, if a fair comparison exists; +- representative cases, including typical, boundary, scaling, and adverse-but-valid inputs where relevant, plus any named workload characteristic such as selectivity, hit rate, type mix, or branch distribution; +- setup/reset strategy and correctness oracle, including how each exact expected result is derived independently of the benchmark method; +- candidates deliberately rejected and why; +- whether the result will be exploratory or grounded in profile/telemetry evidence. + +Follow the confirmation flow in `FORMS.md` in interactive mode. In yolo mode, post the plan as a concise progress update and proceed without confirmation. If the user already named exact members, inputs, and implementations, confirm only material corrections or risks rather than repeating settled choices. + +### 6. Design the experiment + +Read `references/experiment-design.md` and `references/benchmarkdotnet-essentials.md`. Choose one of these shapes: + +1. **Equivalent implementation comparison.** Use separate benchmark methods for current/reference and candidate implementations, feed them identical state, check equivalent results in setup, and mark the current production or established reference method as the baseline. Start from `assets/comparison-benchmark.cs`. +2. **Single-operation characterization.** Use one benchmark method across meaningful cases or sizes when the goal is absolute cost, scaling, allocation characterization, or future regression tracking and no honest competing implementation exists. Do not add a baseline solely to obtain a ratio column. Start from `assets/operation-benchmark.cs`. +3. **Independent operation groups.** Split unrelated operations into separate benchmark classes when practical. If a cohesive class contains multiple alternative pairs, use `[BenchmarkCategory]`, group by category, and assign one baseline per category. Never compare ratios across different semantic work. + +Do not copy an asset blindly. The assets are structural examples with placeholders; adapt namespaces, types, cases, lifecycle, return consumption, correctness checks, and attributes to the real API. + +When a parameter is only size or payload, preserve the other workload characteristics unless the experiment names them explicitly. For predicate or filter benchmarks, state the intended selectivity, use deterministic data, prefer fixed-width or otherwise structurally stable inputs when digit length or formatting would change the branch mix, and verify the exact expected match count for every scenario. + +If a case label promises an exact percentage such as `10% selectivity`, make that statement true for every size under a documented rounding rule. Otherwise choose compatible sizes, define explicit scenario objects with exact expected counts, or rename the cases honestly with qualitative names such as `LowSelectivity`, `HalfMatches`, or `AllMatch`. + +For LINQ and other deferred pipelines, decide whether the benchmark measures query or iterator creation, a terminal operation, explicit enumeration, or materialization. Encode that distinction in names, descriptions, baselines, and conclusions; `List.Count` and `Where(...).Count()` are not interchangeable evidence. + +For `QueryFor`-style runtime-type filters, use deterministic heterogeneous input with exact matches, sibling nonmatches, derived types when exact-type versus assignability semantics matter, and nulls only when supported and relevant. Validate the exact expected count and, when it matters, insertion order and exact runtime types. Do not benchmark a homogeneous all-match store and call it type filtering unless that all-match path is the stated subject. + +### 7. Author the benchmark + +Place the class under `tuning/{SutProject}.Benchmarks/`. Name it for the performance question and end the class name with `Benchmark`. Keep it in the SUT namespace rather than adding `.Benchmarks`; the benchmark project `RootNamespace` supports this codebelt convention. + +Every benchmark should use deterministic state and `[MemoryDiagnoser]`. Add other diagnosers only when they answer the question: `[ThreadingDiagnoser]` for lock/thread-pool signals, `[ExceptionDiagnoser]` for intentional exception paths, `[DisassemblyDiagnoser]` for JIT/code-generation investigations, or EventPipe/ETW profiling for a targeted deep dive. Extra diagnosers often cause extra runs and platform constraints, so do not add them decoratively. + +Keep setup outside the measured method unless setup is part of the consumer-visible operation. Return a result or consume it so the JIT cannot eliminate the work. Avoid independent `[Params]` axes that create meaningless Cartesian products; use a scenario object from `[ParamsSource]` or `[ArgumentsSource]` when inputs must vary together. + +For mutating operations, ensure every measurement observes equivalent starting state without allowing reset cost to dominate a tiny operation. For async APIs, await the real `Task`/`ValueTask`; do not substitute `.Result` or benchmark a completed fake. For concurrency, measure a defined worker/contention scenario and avoid accidentally measuring task creation when shared-state throughput is the actual question. + +### 8. Onboard only missing harness pieces + +If the detector found missing infrastructure, follow `references/onboarding.md`. Resolve current stable package versions dynamically from NuGet.org, preserve central package management when present, reuse existing solution and folder conventions, create at most one runner, and avoid restructuring unrelated repository content. + +### 9. Validate in layers + +Before the build, inspect the benchmark attributes for coherence and remove decorative configuration that no longer serves the question. + +Run a semantic preflight across the full Cartesian set of benchmark methods, parameter values, runtime jobs, and scenario objects before build, discovery, dry execution, or any request for a full run. For every case verify that setup succeeds, the exact expected output is known independently, the actual output matches, the named workload characteristic is true, the intended code path is exercised, and the case does not collapse into a trivial fast path unless that fast path is the stated subject. + +Semantic preflight +- Every scenario has an independently derived exact expected result. +- Every parameter combination satisfies that oracle. +- Benchmark names accurately describe the generated workload. +- Selectivity, hit rate, type mix, and other workload distributions are explicit and verified. +- Every benchmark exercises the intended path. +- Baselines compare equivalent observable work. + +Successful execution is not a correctness oracle. After semantic preflight, validate the benchmark's correctness through existing tests or a setup-time oracle for every parameter case. The oracle should normally verify exact observable behavior rather than merely nonzero or approximate success unless that is the real domain contract. Then build the benchmark project in Release: + +```console +dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj +``` + +Before interpreting discovery or execution output, run the report-aware preflight for the exact class: + +```console +pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "" -BenchmarkType +``` + +If `pwsh` 7+ is unavailable, report that blocker instead of falling back to legacy Windows PowerShell. + +If `reports.wouldSkipRequestedBenchmark` is true, the runner is intentionally filtering the type because a prior report exists. Report that as the validation outcome; do not claim the list/dry run exercised the class and do not modify code to force it through. A fresh full run and any report archive/replacement require explicit human direction. + +Verify runner discovery without measuring: + +```console +dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* +``` + +Compare the discovered method, job, and parameter matrix with the intended design. Missing or surprise combinations are validation failures, not report quirks. + +Unless execution is impossible or the user declines, run a dry execution smoke check and inspect all BenchmarkDotNet validation warnings. A dry run proves executable wiring and basic lifecycle, not performance: + +```console +dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* +``` + +Inspect the complete BenchmarkDotNet summary after the dry run and before reading any numbers. If any intended case shows `NA`, appears under `Benchmarks with issues`, throws in setup or cleanup, triggers a validation error, fails a job or runtime, or is missing from the intended matrix, report the exact failing method, job, and parameter case. Do not interpret successful rows as the completed benchmark. Correct the benchmark design only when the cause is within the benchmark, then rerun build, discovery, and dry validation. + +Run the full benchmark only when the human explicitly asks to start it. Use an unplugged laptop, debugger, busy CI worker, VM, or power-throttled environment only if that environment is itself the target; otherwise warn that the results may not be stable or representative. + +After any full run, apply the same full-summary validity gate before interpreting the tables. If rerunning is necessary because of a benchmark-owned issue, preserve the human-authority rule and get explicit approval before starting another full measurement run. + +### 10. Report the outcome + +Summarize the selected and rejected candidates, benchmark question, cases, correctness oracle, diagnosers, generated files, and validation commands/results. If dry or full execution exposed an invalid case, report that invalid experiment instead of a performance conclusion. + +If a full run occurred, report the environment, active job shape, mean/median where relevant, error and standard deviation, ratios only within valid comparison groups, allocations/GC, warnings, and the workload-specific conclusion. When the runner uses `BenchmarkWorkspaceOptions.Slim`, report that shortened job accurately and mention its warmup/iteration limits whenever they matter to interpretation, especially for runtime- or JIT-sensitive comparisons. + +After the first valid full result, answer three questions: is the result reproducible, is the absolute cost material for observed or plausible usage, and would a deeper diagnostic change a concrete engineering decision? Stop when the answer does not justify further work. Do not automatically escalate to repeated full reruns, tiered-PGO variants, disassembly, EventPipe or ETW tracing, alternative implementations, or runtime-source archaeology. Escalate only when the result is reproducible, material, the next diagnostic can distinguish concrete competing explanations, and the user requested it or it is necessary to answer the original decision. For low-microsecond test-support utilities with no credible production change, prefer a concise "no change justified" conclusion. + +## Completion checklist + +- [ ] Candidate selection is supported by implementation, usage, tests, telemetry, or profiling evidence. +- [ ] The suite answers one to three explicit performance questions and excludes low-value member coverage. +- [ ] Baselines compare equivalent work; single operations and unrelated members have no misleading ratio. +- [ ] Cases represent realistic, boundary, scaling, and adverse paths without useless Cartesian products, and size-only sweeps keep other workload invariants stable unless explicitly named. +- [ ] Setup, mutation, async, concurrency, disposal, and result consumption are handled correctly. +- [ ] Exact correctness oracles cover every parameter and scenario case, including valid zero-match boundaries. +- [ ] A semantic preflight covered every method/param/job/scenario combination with independently derived expected results, truthful workload labels, intended code paths, and no accidental fast paths. +- [ ] Baselines, categories, grouping attributes, and descriptions are internally coherent and non-decorative. +- [ ] `[MemoryDiagnoser]` is present and every additional diagnoser has a stated purpose. +- [ ] Harness changes preserve repository conventions and reuse existing projects/runner where possible. +- [ ] Runner preflight accounts for `SkipBenchmarksWithReports`, configured slim/runtime jobs, and any matching `reports/tuning/` artifact before benchmark-code diagnosis. +- [ ] Release build, benchmark discovery, and dry execution succeed, or exact blockers are reported. +- [ ] The complete discovery/dry/full summary covers the full intended matrix with no `NA`, `Benchmarks with issues`, failed jobs, or missing cases before any performance interpretation. +- [ ] Full-run performance claims are made only from an actual full run in a described environment, and any Slim-job limitation or other warning is reported honestly. +- [ ] After the first valid full result, deeper diagnostics or reruns are justified by a reproducible, material, decision-changing question rather than sunk cost. +- [ ] Generated files are UTF-8 without mojibake, and no unrelated files were changed. diff --git a/skills/dotnet-benchmark/assets/benchmark-program.cs b/skills/dotnet-benchmark/assets/benchmark-program.cs new file mode 100644 index 0000000..2a745ef --- /dev/null +++ b/skills/dotnet-benchmark/assets/benchmark-program.cs @@ -0,0 +1,24 @@ +// Emit these runtime-job using directives only when extra AddJob(...) runtimes were selected. +{RUNTIME_USINGS} +using Codebelt.Extensions.BenchmarkDotNet.Console; + +namespace {RUNNER_NAMESPACE}; + +public class Program +{ + public static void Main(string[] args) + { + BenchmarkProgram.Run(args, o => + { + o.AllowDebugBuild = BenchmarkProgram.IsDebugBuild; + o.SkipBenchmarksWithReports = true; + o.ConfigureBenchmarkDotNet(c => + { +{RUNTIME_SETUP} + // If the user chose "Runner default only", leave the next line as `return c;`. + // Otherwise append newline-prefixed chained `.AddJob(...)` calls to the return expression. + return c{RUNTIME_JOBS}; + }); + }); + } +} diff --git a/skills/dotnet-benchmark/assets/benchmark-runner.csproj b/skills/dotnet-benchmark/assets/benchmark-runner.csproj new file mode 100644 index 0000000..13c3ff9 --- /dev/null +++ b/skills/dotnet-benchmark/assets/benchmark-runner.csproj @@ -0,0 +1,16 @@ + + + + {RUNNER_TARGET_FRAMEWORK} + {RUNNER_NAMESPACE} + + + + + + + + + + + diff --git a/skills/dotnet-benchmark/assets/benchmark.csproj b/skills/dotnet-benchmark/assets/benchmark.csproj new file mode 100644 index 0000000..7b2e68a --- /dev/null +++ b/skills/dotnet-benchmark/assets/benchmark.csproj @@ -0,0 +1,11 @@ + + + + {ROOT_NAMESPACE} + + + + + + + diff --git a/skills/dotnet-benchmark/assets/comparison-benchmark.cs b/skills/dotnet-benchmark/assets/comparison-benchmark.cs new file mode 100644 index 0000000..3180ea0 --- /dev/null +++ b/skills/dotnet-benchmark/assets/comparison-benchmark.cs @@ -0,0 +1,37 @@ +using System; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace {SUT_NAMESPACE}; + +// Structural example only. Replace every placeholder and adapt cases, state, calls, return types, +// equivalence checks, and names to the real consumer operation. Remove unused using directives. +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] +public class {BENCHMARK_CLASS} +{ + [Params(64, 4_096, 1_048_576)] + public int Size { get; set; } + + private byte[] _input = null!; + + [GlobalSetup] + public void Setup() + { + _input = new byte[Size]; + new Random(42).NextBytes(_input); + + var expected = {BASELINE_CALL}; + var actual = {CANDIDATE_CALL}; + if (!{EQUIVALENCE_CHECK}) + { + throw new InvalidOperationException("Baseline and candidate results differ for the current benchmark case."); + } + } + + [Benchmark(Baseline = true, Description = "{BASELINE_DESCRIPTION}")] + public {RETURN_TYPE} Current() => {BASELINE_CALL}; + + [Benchmark(Description = "{CANDIDATE_DESCRIPTION}")] + public {RETURN_TYPE} Candidate() => {CANDIDATE_CALL}; +} diff --git a/skills/dotnet-benchmark/assets/operation-benchmark.cs b/skills/dotnet-benchmark/assets/operation-benchmark.cs new file mode 100644 index 0000000..07968ab --- /dev/null +++ b/skills/dotnet-benchmark/assets/operation-benchmark.cs @@ -0,0 +1,28 @@ +using System; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace {SUT_NAMESPACE}; + +// Structural example for one operation with no honest competing implementation. Replace every +// placeholder and adapt the cases, data shape, return consumption, and names to the real workload. +// Do not add Baseline = true merely to produce a ratio column. Remove unused using directives. +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByParams)] +public class {BENCHMARK_CLASS} +{ + [Params(64, 4_096, 1_048_576)] + public int Size { get; set; } + + private byte[] _input = null!; + + [GlobalSetup] + public void Setup() + { + _input = new byte[Size]; + new Random(42).NextBytes(_input); + } + + [Benchmark(Description = "{OPERATION_DESCRIPTION}")] + public {RETURN_TYPE} Measure() => {SUT_CALL}; +} diff --git a/skills/dotnet-benchmark/evals/evals.json b/skills/dotnet-benchmark/evals/evals.json new file mode 100644 index 0000000..3a701a7 --- /dev/null +++ b/skills/dotnet-benchmark/evals/evals.json @@ -0,0 +1,207 @@ +{ + "skill_name": "dotnet-benchmark", + "evals": [ + { + "id": 1, + "prompt": "This repo has src/Acme.Core/Acme.Core.csproj but no benchmark infrastructure. Set up the codebelt BenchmarkDotNet harness without inventing a benchmark target yet.", + "expected_output": "The agent detects repository state first and onboards only the missing tuning project and single tooling runner, preserves CPM/solution conventions, resolves current package versions dynamically, and does not fabricate a target type or benchmark method.", + "expectations": [ + "Runs or references scripts/check-benchmark-requirements.ps1 before changing harness files", + "Creates or proposes exactly one reusable tooling runner and a tuning benchmark project while preserving the detected solution and package-management conventions", + "Resolves BenchmarkDotNet package versions dynamically from NuGet.org rather than copying hardcoded example versions", + "Does not invent a SUT type, member, workload, or benchmark class when none was supplied", + "Explains that candidate discovery starts after a concrete type is available" + ] + }, + { + "id": 2, + "prompt": "Create the most useful BenchmarkDotNet benchmark for Acme.Routing.RouteMatcher from the attached source. I only know that routing gets expensive in large route tables; inspect the type and usages and choose what is worth measuring.", + "files": [ + "evals/files/route-matcher/RouteMatcher.cs", + "evals/files/route-matcher/RouteTable.cs", + "evals/files/route-matcher/RouteMatcherTests.cs" + ], + "expected_output": "The agent selects IsMatch as the evidence-backed hot operation because RouteTable calls it inside the route scan, identifies repeated regex construction as a cost hypothesis, builds structured hit/miss and pattern/path-complexity scenarios, characterizes the current operation without inventing a candidate or fake baseline, and rejects low-value Normalize/ToString member coverage.", + "expectations": [ + "Uses RouteTable call-site evidence to prioritize RouteMatcher.IsMatch rather than enumerating every public member", + "Identifies per-call wildcard-to-regex conversion and Regex.IsMatch construction/caching behavior as hypotheses to measure without claiming they are already proven bottlenecks", + "Uses representative coupled scenarios covering hit and miss plus simple and complex patterns instead of a meaningless Cartesian product or random strings", + "Authors or proposes a single-operation characterization benchmark with MemoryDiagnoser and no Baseline=true because no equivalent candidate implementation exists", + "Keeps input/setup work outside the measured method, returns or consumes the IsMatch result, and includes build, list, and dry-run validation", + "Explicitly rejects Normalize and ToString as lower-value candidates based on the supplied usage evidence" + ] + }, + { + "id": 3, + "prompt": "Benchmark Acme.Text.DecimalParser and tell me whether the span-based implementation is a worthwhile replacement for the legacy implementation. Use the attached type, call site, and tests to choose cases. Measure allocations too, but do not run a full benchmark yet.", + "files": [ + "evals/files/decimal-parser/DecimalParser.cs", + "evals/files/decimal-parser/ImportPipeline.cs", + "evals/files/decimal-parser/DecimalParserTests.cs" + ], + "expected_output": "The agent creates a fair Legacy versus Span comparison using identical UTF-8 inputs derived from realistic valid and invalid import values, treats Legacy as the production baseline, validates equivalent success/value/consumed behavior outside measurement, and stops after Release build, discovery, and dry execution without claiming a speedup.", + "expectations": [ + "Selects ParseLegacy versus TryParseSpan as equivalent implementation candidates and uses ParseLegacy as the production baseline", + "Derives valid typical, boundary, long, and invalid/partially-consumed cases from the source/tests instead of using arbitrary buffer sizes alone", + "Avoids independent parameters that create invalid combinations by using coupled scenario objects or an equivalent explicit case source", + "Checks success, parsed value, and consumed length equivalence for every case outside the timed methods", + "Feeds both methods identical bytes and keeps encoding/input construction outside measurement", + "Uses MemoryDiagnoser, returns observable results, and either performs Release build/list/dry validation or gives those commands while honestly reporting that the source-only fixture lacks a runnable harness; makes no performance claim because the full benchmark was not run" + ] + }, + { + "id": 4, + "prompt": "Add a thorough benchmark for Acme.DateWindow: construction, formatting, equality, and GetHashCode. I want useful optimization evidence, not just a benchmark file.", + "files": [ + "evals/files/date-window/DateWindow.cs", + "evals/files/date-window/DateWindowUsage.cs" + ], + "expected_output": "The agent challenges the request to compare unrelated members, uses dictionary/set call-site evidence to prioritize equality/hash behavior, treats formatting and construction as separate questions only if retained, and never makes constructor timing the baseline for formatting/equality/hashing.", + "expectations": [ + "Inspects supplied usage evidence and prioritizes Equals/GetHashCode or the dictionary lookup consumer operation over equal-weight public-member coverage", + "Does not use construction as the baseline for formatting, equality, or hashing and does not present ratios between unrelated work", + "Uses equal, unequal-early, unequal-late, dictionary hit, or dictionary miss scenarios where they answer the selected question", + "Splits independent questions into focused classes/categories or omits low-value operations with explicit reasons", + "Uses MemoryDiagnoser and an honest baseline only within an equivalent comparison group, or no baseline for a single-operation characterization" + ] + }, + { + "id": 5, + "prompt": "Find and benchmark the bottleneck in Acme.Reporting.ReportLoader. Users say loading large report files is slow. Use the attached source and usage, and make the benchmark as realistic as possible.", + "files": [ + "evals/files/report-loader/ReportLoader.cs", + "evals/files/report-loader/ReportImportJob.cs" + ], + "expected_output": "The agent refuses to claim a bottleneck from source alone and does not put disk I/O in a microbenchmark. It recommends profiling the end-to-end import and, if a useful BenchmarkDotNet artifact is requested, isolates in-memory JSON parsing with representative payload sizes while clearly excluding file-system latency from the conclusion.", + "expectations": [ + "States that source inspection alone cannot prove whether file I/O or JSON parsing dominates and recommends profiling or a macrobenchmark for the end-to-end load", + "Does not call File.ReadAllText, File.ReadAllBytes, or other disk I/O inside a Benchmark method", + "If authoring BenchmarkDotNet code, isolates Parse on preloaded in-memory UTF-8 payloads and labels the result as parsing-only", + "Uses structured small/typical/large report payloads rather than arbitrary random bytes", + "Does not claim that an in-memory parsing result represents end-to-end report-loading latency" + ] + }, + { + "id": 6, + "prompt": "Benchmark Acme.Caching.ConcurrentMemoizer. We suspect lock contention during parallel request bursts. Inspect the attached type and call site, then create the benchmark that would actually answer that question.", + "files": [ + "evals/files/concurrent-memoizer/ConcurrentMemoizer.cs", + "evals/files/concurrent-memoizer/MemoizedEndpoint.cs" + ], + "expected_output": "The agent designs a defined contention experiment rather than a sequential GetOrAdd microbenchmark: fixed worker counts, synchronized start, shared memoizer, realistic warm-hit/miss mix, per-operation normalization, ThreadingDiagnoser plus MemoryDiagnoser, and a clear warning that service tail latency still needs load testing/profiling.", + "expectations": [ + "Uses the Parallel.ForEachAsync call site as evidence that concurrent shared-state behavior is the relevant operation", + "Defines worker count, shared state, hit/miss mix, synchronized start or equivalent coordination, and per-operation normalization", + "Avoids a benchmark that only measures one sequential GetOrAdd call and avoids accidentally making Task creation the unexplained dominant cost", + "Adds ThreadingDiagnoser for lock-contention evidence alongside MemoryDiagnoser", + "Separates contention characterization from service-level throughput/tail-latency claims and recommends load testing or profiling for the latter", + "Defines reset/lifecycle behavior so cache state does not leak unpredictably between cases" + ] + }, + { + "id": 7, + "prompt": "Set up a benchmark for Acme.Core.Parser, but keep the codebelt runner on its default runtime only. I do not want net48/net8/net9/net10 comparison jobs.", + "expected_output": "The runner remains a valid default configuration with return c;, no raw placeholders, no AddJob chain, and no unused runtime-job using directives; benchmark design still waits for inspection of Parser and its workload.", + "expectations": [ + "Keeps ConfigureBenchmarkDotNet as a valid plain return c; expression without AddJob calls", + "Emits no raw runtime placeholders or unused BenchmarkDotNet.Environments, BenchmarkDotNet.Jobs, or Codebelt.Extensions.BenchmarkDotNet runtime-job using directives", + "Does not treat the absence of extra jobs as permission to skip target-type and workload inspection" + ] + }, + { + "id": 8, + "prompt": "YOLO mode: add the most useful benchmark for Acme.Routing.RouteMatcher from the attached source. Use the repo evidence and recommended defaults, proceed without routine confirmation, and do not run a full performance benchmark.", + "files": [ + "evals/files/route-matcher/RouteMatcher.cs", + "evals/files/route-matcher/RouteTable.cs", + "evals/files/route-matcher/RouteMatcherTests.cs" + ], + "expected_output": "The agent recognizes yolo mode, autonomously selects the evidence-backed RouteMatcher.IsMatch characterization, reports the experiment plan as a progress update rather than a question, authors the benchmark, and proceeds through build/list/dry validation or honestly reports a source-only harness blocker. It never treats yolo as authorization for a full run or unrelated side effects.", + "expectations": [ + "Recognizes yolo mode and does not ask for performance intent, workload, candidate-plan confirmation, or execution depth when repository evidence provides safe defaults", + "Uses RouteTable and tests to select RouteMatcher.IsMatch plus representative coupled hit/miss and pattern-complexity cases", + "States the experiment plan and material assumptions as information, then proceeds without asking the user to approve the recommended plan", + "Uses a MemoryDiagnoser single-operation characterization with no fabricated Baseline=true", + "Selects build, list, and dry validation by default, or supplies those commands while honestly reporting that the source-only fixture lacks a runnable harness", + "Does not run or claim a full performance benchmark without an explicit human instruction, and does not treat yolo as authorization for unrelated repository or remote operations" + ] + }, + { + "id": 9, + "prompt": "Acme.Core.ParserBenchmark builds, but the codebelt runner appears to do nothing when I list or run it. Diagnose the attached minimal repository. Keep the benchmark slim and do not start a full performance run.", + "files": [ + "evals/files/runner-skip/tooling/benchmark-runner/Program.cs", + "evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj", + "evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs", + "evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj", + "evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md" + ], + "expected_output": "The agent performs runner preflight before benchmark-code diagnosis, recognizes that SkipBenchmarksWithReports=true plus the matching reports/tuning file deliberately filters ParserBenchmark, verifies the canonical Slim runtime jobs, preserves the lean benchmark and its name, avoids disassembly/tool thrash, and does not remove reports, disable the option, or start a full run without explicit human direction.", + "expectations": [ + "Runs the detector with -BenchmarkType Acme.Core.ParserBenchmark against the canonical repo layout and reports runner.skipBenchmarksWithReports=true, Slim runtime jobs, the matching report, and reports.wouldSkipRequestedBenchmark=true", + "Identifies Acme.Core.ParserBenchmark-report-github.md as a case-insensitive type-name match that deliberately filters ParserBenchmark", + "Explains that an empty list or execution is expected runner behavior rather than evidence that ParserBenchmark is malformed", + "Preserves the lean ParserBenchmark class, method, attributes, and name without adding DisassemblyDiagnoser or unrelated benchmark cases", + "Does not disable SkipBenchmarksWithReports, rename the type, or delete/move/overwrite the report automatically", + "Does not start or claim a full performance run and requires explicit human direction for a fresh run plus report-retention choice" + ] + }, + { + "id": 10, + "prompt": "Review the attached Acme.Search.LegacyAliasQuery benchmark and summary. I tried to compare List.Count with a filtered Where(...).Count() across sizes 8, 256, and 4096, but the filtered Size=8 case reported NA and I still wrote up the larger rows as a win. Fix the benchmark design and tell me whether any performance conclusion is valid.", + "files": [ + "evals/files/selectivity-drift/LegacyAliasQuery.cs", + "evals/files/selectivity-drift/LegacyAliasImport.cs", + "evals/files/selectivity-drift/LegacyAliasQueryTests.cs", + "evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs", + "evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md" + ], + "expected_output": "The agent rejects the benchmark as invalid because the intended matrix is incomplete and the filtered Size=8 case fails in setup, explains that List.Count is an O(1) collection fast path rather than predicate enumeration, replaces the drifting workload with deterministic fixed-width or explicit selectivity scenarios, removes the unrelated baseline and decorative category grouping, and reruns build, discovery, and dry validation before allowing any future full run.", + "expectations": [ + "Detects the valid zero-match boundary or the summary's NA and Benchmarks with issues signals and refuses to treat the surviving rows as a completed benchmark", + "Reports the failing method, job, and parameter case exactly and makes no performance claim from the invalid run", + "Replaces the drifting generator with deterministic fixed-width or explicit selectivity scenarios so size changes do not silently change predicate selectivity, and verifies exact expected match counts for every scenario", + "Removes Baseline = true from the unrelated collection Count fast path comparison or otherwise separates it from filtered enumeration instead of using it as the baseline", + "Removes GroupBenchmarksBy(ByCategory) when categories are absent, or only uses meaningful categories with equivalent comparison groups", + "States that after fixing benchmark-owned issues it must rerun build, discovery, and dry validation, and that another full measurement run still requires explicit human approval" + ] + }, + { + "id": 11, + "prompt": "The attached Acme.Testing.TraitFilter helper only runs in test discovery. I already have the first valid benchmark summary, and the costs are low-microsecond with small fixed allocations. Tell me whether to go deeper and what benchmark, if any, is still justified.", + "files": [ + "evals/files/proportionate-stop/TraitFilter.cs", + "evals/files/proportionate-stop/TraitDiscovery.cs", + "evals/files/proportionate-stop/TraitDiscoveryTests.cs", + "evals/files/proportionate-stop/TraitFilterBenchmark-summary.md" + ], + "expected_output": "The agent keeps the suite to one focused characterization benchmark for the real filter operation, treats the attached first valid full result as enough to answer the current question, reports that no production change is justified for a low-microsecond test-support helper, declines automatic escalation to extra diagnostics or alternative implementations, and names the condition that would justify reopening the investigation.", + "expectations": [ + "Uses the test-discovery call site to keep the scope to one linear filter and materialization benchmark rather than inventing baselines or extra implementations", + "Reads the first valid full result proportionally and answers whether the effect is reproducible, material, and decision-changing before suggesting more work", + "Concludes that no production optimization change is justified for the supplied low-microsecond test-support utility", + "Does not automatically add repeated full reruns, tiered-PGO, disassembly, EventPipe or ETW tracing, runtime-source archaeology, or speculative alternative implementations", + "Identifies a concrete reopen condition such as promotion to a production hot path, materially larger inputs or frequencies, or a new regression with decision-changing cost" + ] + }, + { + "id": 12, + "prompt": "Review the attached Codebelt.Extensions.Xunit InMemoryTestStore benchmark and the first successful summary excerpt. Close the remaining correctness and workload-semantics gap before any further performance interpretation or rerun. Fix the benchmark design only; do not begin another deep performance investigation.", + "files": [ + "evals/files/inmemory-test-store/InMemoryTestStore.cs", + "evals/files/inmemory-test-store/InMemoryTestStoreTests.cs", + "evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs", + "evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md" + ], + "expected_output": "The agent adds a semantic preflight before any further run, derives independently trusted exact expected counts for every ItemCount and scenario, rejects the false 10% label at the smallest size, fixes or relabels the fake type-filter workload, removes the invalid no-predicate baseline, and validates the full case matrix without drifting into a broader performance investigation.", + "expectations": [ + "States that the existing successful benchmark summary is not a correctness oracle and requires semantic preflight before any future full run or interpretation", + "Derives exact expected counts independently of the measured Query path and validates every ItemCount/scenario combination, including the 8-item case whose current 10% label yields zero matches", + "Either chooses sizes or explicit scenario objects that preserve exact percentage semantics, or renames the cases honestly with qualitative labels such as LowSelectivity, HalfMatches, and AllMatch", + "Replaces the homogeneous QueryFor() input with deterministic heterogeneous runtime types spanning matches, sibling nonmatches, and derived types when assignability matters, or explicitly relabels the benchmark as the all-match path instead of type filtering", + "Removes Baseline=true from the no-predicate Count fast path comparison and rejects decorative ByCategory grouping unless an equivalent comparison group remains", + "Checks the full Cartesian matrix of benchmark methods, ItemCount values, runtime jobs, and scenario objects before build/list/dry/full validation, then stops without launching another deep performance investigation" + ] + } + ] +} diff --git a/skills/dotnet-benchmark/evals/files/concurrent-memoizer/ConcurrentMemoizer.cs b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/ConcurrentMemoizer.cs new file mode 100644 index 0000000..419acae --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/ConcurrentMemoizer.cs @@ -0,0 +1,30 @@ +namespace Acme.Caching; + +public sealed class ConcurrentMemoizer where TKey : notnull +{ + private readonly object _gate = new(); + private readonly Dictionary _values = new(); + + public TValue GetOrAdd(TKey key, Func factory) + { + lock (_gate) + { + if (_values.TryGetValue(key, out var value)) + { + return value; + } + + value = factory(key); + _values.Add(key, value); + return value; + } + } + + public void Clear() + { + lock (_gate) + { + _values.Clear(); + } + } +} diff --git a/skills/dotnet-benchmark/evals/files/concurrent-memoizer/MemoizedEndpoint.cs b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/MemoizedEndpoint.cs new file mode 100644 index 0000000..f697665 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/concurrent-memoizer/MemoizedEndpoint.cs @@ -0,0 +1,13 @@ +namespace Acme.Caching; + +public sealed class MemoizedEndpoint(ConcurrentMemoizer memoizer) +{ + public async Task HandleBurst(IEnumerable ids, CancellationToken cancellationToken) + { + await Parallel.ForEachAsync(ids, cancellationToken, (id, cancellation) => + { + _ = memoizer.GetOrAdd(id, static key => key.ToString()); + return ValueTask.CompletedTask; + }); + } +} diff --git a/skills/dotnet-benchmark/evals/files/date-window/DateWindow.cs b/skills/dotnet-benchmark/evals/files/date-window/DateWindow.cs new file mode 100644 index 0000000..c7ec5f8 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/date-window/DateWindow.cs @@ -0,0 +1,6 @@ +namespace Acme; + +public readonly record struct DateWindow(DateOnly Start, DateOnly End) +{ + public override string ToString() => $"{Start:O}/{End:O}"; +} diff --git a/skills/dotnet-benchmark/evals/files/date-window/DateWindowUsage.cs b/skills/dotnet-benchmark/evals/files/date-window/DateWindowUsage.cs new file mode 100644 index 0000000..163ecc9 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/date-window/DateWindowUsage.cs @@ -0,0 +1,8 @@ +namespace Acme; + +public sealed class DateWindowIndex +{ + private readonly Dictionary _values = new(); + + public bool TryGet(DateWindow window, out string? value) => _values.TryGetValue(window, out value); +} diff --git a/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParser.cs b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParser.cs new file mode 100644 index 0000000..3753186 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParser.cs @@ -0,0 +1,29 @@ +using System.Buffers.Text; +using System.Globalization; +using System.Text; + +namespace Acme.Text; + +public static class DecimalParser +{ + public static bool ParseLegacy(ReadOnlySpan utf8, out decimal value, out int consumed) + { + var text = Encoding.UTF8.GetString(utf8); + var success = decimal.TryParse(text, NumberStyles.Number, CultureInfo.InvariantCulture, out value); + consumed = success ? utf8.Length : 0; + return success; + } + + public static bool TryParseSpan(ReadOnlySpan utf8, out decimal value, out int consumed) + { + if (Utf8Parser.TryParse(utf8, out value, out var parsedBytes, 'G') && parsedBytes == utf8.Length) + { + consumed = parsedBytes; + return true; + } + + value = default; + consumed = 0; + return false; + } +} diff --git a/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParserTests.cs b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParserTests.cs new file mode 100644 index 0000000..d2df239 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/decimal-parser/DecimalParserTests.cs @@ -0,0 +1,7 @@ +namespace Acme.Text.Tests; + +public class DecimalParserTests +{ + // Import values include typical integers/decimals, decimal boundaries, and rejected content. + private static readonly string[] Cases = ["42", "1234.56", "-79228162514264337593543950335", "not-a-number", "123.45 trailing"]; +} diff --git a/skills/dotnet-benchmark/evals/files/decimal-parser/ImportPipeline.cs b/skills/dotnet-benchmark/evals/files/decimal-parser/ImportPipeline.cs new file mode 100644 index 0000000..f5901d0 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/decimal-parser/ImportPipeline.cs @@ -0,0 +1,18 @@ +namespace Acme.Text; + +public sealed class ImportPipeline +{ + public decimal Sum(IEnumerable> values) + { + decimal total = 0; + foreach (var value in values) + { + if (DecimalParser.ParseLegacy(value.Span, out var parsed, out _)) + { + total += parsed; + } + } + + return total; + } +} diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStore.cs b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStore.cs new file mode 100644 index 0000000..b2d7655 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStore.cs @@ -0,0 +1,18 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Codebelt.Extensions.Xunit; + +public sealed class InMemoryTestStore +{ + private readonly List _items = new(); + + public void Add(T item) => _items.Add(item); + + public IReadOnlyCollection Query() => _items; + + public IEnumerable Query(Func predicate) => _items.Where(predicate); + + public IEnumerable QueryFor() => _items.OfType(); +} diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md new file mode 100644 index 0000000..617f0eb --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark-summary.md @@ -0,0 +1,21 @@ +# InMemoryTestStoreBenchmark summary excerpt + +The benchmark already executed successfully; the concern is whether the workloads and labels are semantically true. + +| Method | Runtime | ItemCount | Mean | Allocated | +|---|---|---:|---:|---:| +| Query - no predicate | .NET 10.0 | 8 | 0.7339 ns | - | +| Query - 10% selectivity | .NET 10.0 | 8 | 10.7615 ns | 72 B | +| Query - 50% selectivity | .NET 10.0 | 8 | 13.6044 ns | 72 B | +| Query - 100% selectivity | .NET 10.0 | 8 | 9.9140 ns | 72 B | +| QueryFor - filtered by type | .NET 10.0 | 8 | 9.0760 ns | 72 B | +| Query - no predicate | .NET 10.0 | 256 | 0.6446 ns | - | +| Query - 10% selectivity | .NET 10.0 | 256 | 80.3261 ns | 72 B | +| Query - 50% selectivity | .NET 10.0 | 256 | 100.6783 ns | 72 B | +| Query - 100% selectivity | .NET 10.0 | 256 | 96.9278 ns | 72 B | +| QueryFor - filtered by type | .NET 10.0 | 256 | 72.5479 ns | 72 B | +| Query - no predicate | .NET 10.0 | 4096 | 0.6269 ns | - | +| Query - 10% selectivity | .NET 10.0 | 4096 | 953.6987 ns | 72 B | +| Query - 50% selectivity | .NET 10.0 | 4096 | 1,103.5284 ns | 72 B | +| Query - 100% selectivity | .NET 10.0 | 4096 | 1,376.7703 ns | 72 B | +| QueryFor - filtered by type | .NET 10.0 | 4096 | 868.7072 ns | 72 B | diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs new file mode 100644 index 0000000..85795c4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreBenchmark.cs @@ -0,0 +1,80 @@ +using System; +using System.Linq; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace Codebelt.Extensions.Xunit; + +/// +/// Benchmarks for the query operations. +/// +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] +public class InMemoryTestStoreBenchmark +{ + private InMemoryTestStore _store; + private Func _low10PercentSelectivity; + private Func _mid50PercentSelectivity; + private Func _high100PercentSelectivity; + + [Params(8, 256, 4096)] + public int ItemCount { get; set; } + + [GlobalSetup] + public void Setup() + { + _store = new InMemoryTestStore(); + for (int i = 0; i < ItemCount; i++) + { + _store.Add(i); + } + + // Predicates with different selectivity levels. + var threshold10 = (int)(ItemCount * 0.1); + var threshold50 = ItemCount / 2; + + _low10PercentSelectivity = x => x < threshold10; + _mid50PercentSelectivity = x => x < threshold50; + _high100PercentSelectivity = x => x >= 0; + } + + [Benchmark(Baseline = true, Description = "Query - no predicate")] + [BenchmarkCategory("Query")] + public int Query_NoPredicate() + { + var result = _store.Query(); + return result.Count(); + } + + [Benchmark(Description = "Query - 10% selectivity")] + [BenchmarkCategory("Query")] + public int Query_10PercentSelectivity() + { + var result = _store.Query(_low10PercentSelectivity); + return result.Count(); + } + + [Benchmark(Description = "Query - 50% selectivity")] + [BenchmarkCategory("Query")] + public int Query_50PercentSelectivity() + { + var result = _store.Query(_mid50PercentSelectivity); + return result.Count(); + } + + [Benchmark(Description = "Query - 100% selectivity")] + [BenchmarkCategory("Query")] + public int Query_100PercentSelectivity() + { + var result = _store.Query(_high100PercentSelectivity); + return result.Count(); + } + + [Benchmark(Description = "QueryFor - filtered by type")] + [BenchmarkCategory("QueryFor")] + public int QueryFor_TypeFilter() + { + var result = _store.QueryFor(); + return result.Count(); + } +} diff --git a/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreTests.cs b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreTests.cs new file mode 100644 index 0000000..ed775b1 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/inmemory-test-store/InMemoryTestStoreTests.cs @@ -0,0 +1,43 @@ +using System.Linq; +using Xunit; + +namespace Codebelt.Extensions.Xunit; + +public class InMemoryTestStoreTests +{ + [Fact] + public void Query_AllowsValidZeroMatchFilters() + { + var store = new InMemoryTestStore(); + foreach (var value in Enumerable.Range(0, 8)) + { + store.Add(value); + } + + Assert.Empty(store.Query(x => x < 0)); + } + + [Fact] + public void QueryFor_ReturnsAssignableRuntimeTypesInInsertionOrder() + { + var store = new InMemoryTestStore(); + store.Add(new Order(1, "SO-001")); + store.Add(new Draft(2, "draft")); + store.Add(new PriorityOrder(3, "SO-002", 5)); + + var result = store.QueryFor().ToArray(); + + Assert.Collection( + result, + item => Assert.IsType(item), + item => Assert.IsType(item)); + } + + public abstract record StoreItem(int Id); + + public record Order(int Id, string Number) : StoreItem(Id); + + public sealed record PriorityOrder(int Id, string Number, int Priority) : Order(Id, Number); + + public sealed record Draft(int Id, string Name) : StoreItem(Id); +} diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscovery.cs b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscovery.cs new file mode 100644 index 0000000..e06375f --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscovery.cs @@ -0,0 +1,9 @@ +using System.Collections.Generic; + +namespace Acme.Testing; + +public sealed class TraitDiscovery +{ + public IReadOnlyList SelectIntegrationTraits(IReadOnlyList traits) => + TraitFilter.Matching(traits, "integration:"); +} diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscoveryTests.cs b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscoveryTests.cs new file mode 100644 index 0000000..536f0a4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitDiscoveryTests.cs @@ -0,0 +1,22 @@ +using Xunit; + +namespace Acme.Testing; + +public class TraitDiscoveryTests +{ + [Fact] + public void SelectIntegrationTraits_FiltersDeterministically() + { + var traits = new[] + { + "integration:postgres", + "unit:formatter", + "integration:redis", + "unit:validator" + }; + + var selected = new TraitDiscovery().SelectIntegrationTraits(traits); + + Assert.Equal(new[] { "integration:postgres", "integration:redis" }, selected); + } +} diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilter.cs b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilter.cs new file mode 100644 index 0000000..85772fe --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilter.cs @@ -0,0 +1,11 @@ +using System; +using System.Collections.Generic; +using System.Linq; + +namespace Acme.Testing; + +public static class TraitFilter +{ + public static string[] Matching(IReadOnlyList traits, string prefix) => + traits.Where(trait => trait.StartsWith(prefix, StringComparison.Ordinal)).ToArray(); +} diff --git a/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilterBenchmark-summary.md b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilterBenchmark-summary.md new file mode 100644 index 0000000..5d90eb4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/proportionate-stop/TraitFilterBenchmark-summary.md @@ -0,0 +1,9 @@ +# TraitFilterBenchmark summary excerpt + +| Method | Scenario | Mean | Error | StdDev | Allocated | +|---|---|---:|---:|---:|---:| +| Matching | Small (16 traits, 25% hits) | 0.84 us | 0.02 us | 0.02 us | 352 B | +| Matching | Typical (64 traits, 25% hits) | 1.47 us | 0.04 us | 0.04 us | 768 B | +| Matching | Large (256 traits, 25% hits) | 4.18 us | 0.09 us | 0.09 us | 2.4 KB | + +No warnings. The first valid full result came from the repository's `BenchmarkWorkspaceOptions.Slim` job. diff --git a/skills/dotnet-benchmark/evals/files/report-loader/ReportImportJob.cs b/skills/dotnet-benchmark/evals/files/report-loader/ReportImportJob.cs new file mode 100644 index 0000000..dbcf660 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/report-loader/ReportImportJob.cs @@ -0,0 +1,6 @@ +namespace Acme.Reporting; + +public sealed class ReportImportJob(ReportLoader loader) +{ + public IReadOnlyList Import(IEnumerable paths) => paths.Select(loader.Load).ToArray(); +} diff --git a/skills/dotnet-benchmark/evals/files/report-loader/ReportLoader.cs b/skills/dotnet-benchmark/evals/files/report-loader/ReportLoader.cs new file mode 100644 index 0000000..2f15f34 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/report-loader/ReportLoader.cs @@ -0,0 +1,14 @@ +using System.Text.Json; + +namespace Acme.Reporting; + +public sealed class ReportLoader +{ + public Report Load(string path) => Parse(File.ReadAllBytes(path)); + + public Report Parse(ReadOnlySpan utf8) => JsonSerializer.Deserialize(utf8)!; +} + +public sealed record Report(string Name, IReadOnlyList Rows); + +public sealed record ReportRow(string Key, decimal Value, string? Comment); diff --git a/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcher.cs b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcher.cs new file mode 100644 index 0000000..b4c8d1d --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcher.cs @@ -0,0 +1,16 @@ +using System.Text.RegularExpressions; + +namespace Acme.Routing; + +public sealed class RouteMatcher +{ + public bool IsMatch(string pattern, string path) + { + var regexPattern = "^" + Regex.Escape(pattern).Replace("\\*", ".*").Replace("\\?", ".") + "$"; + return Regex.IsMatch(path, regexPattern, RegexOptions.IgnoreCase | RegexOptions.CultureInvariant); + } + + public string Normalize(string path) => path.Trim().Trim('/').ToLowerInvariant(); + + public override string ToString() => nameof(RouteMatcher); +} diff --git a/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcherTests.cs b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcherTests.cs new file mode 100644 index 0000000..470d5f1 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/route-matcher/RouteMatcherTests.cs @@ -0,0 +1,13 @@ +namespace Acme.Routing.Tests; + +public class RouteMatcherTests +{ + // Representative cases: exact hit, wildcard hit, single-character wildcard, and miss. + private static readonly (string Pattern, string Path, bool Expected)[] Cases = + [ + ("api/health", "api/health", true), + ("api/*/orders/*", "api/v2/orders/2026-000042", true), + ("tenant-?/reports/*.json", "tenant-a/reports/monthly-2026-06.json", true), + ("assets/*.css", "assets/app.js", false) + ]; +} diff --git a/skills/dotnet-benchmark/evals/files/route-matcher/RouteTable.cs b/skills/dotnet-benchmark/evals/files/route-matcher/RouteTable.cs new file mode 100644 index 0000000..d89185e --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/route-matcher/RouteTable.cs @@ -0,0 +1,17 @@ +namespace Acme.Routing; + +public sealed class RouteTable(RouteMatcher matcher, IReadOnlyList patterns) +{ + public string? Find(string path) + { + foreach (var pattern in patterns) + { + if (matcher.IsMatch(pattern, path)) + { + return pattern; + } + } + + return null; + } +} diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md b/skills/dotnet-benchmark/evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md new file mode 100644 index 0000000..adf4dfd --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/reports/tuning/Acme.Core.ParserBenchmark-report-github.md @@ -0,0 +1,3 @@ +# Existing ParserBenchmark report + +This fixture represents an earlier completed run. diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs new file mode 100644 index 0000000..71070b6 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/Program.cs @@ -0,0 +1,27 @@ +using Codebelt.Extensions.BenchmarkDotNet; +using Codebelt.Extensions.BenchmarkDotNet.Console; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Environments; +using BenchmarkDotNet.Jobs; + +namespace benchmark_runner; + +public class Program +{ + public static void Main(string[] args) + { + BenchmarkProgram.Run(args, o => + { + o.AllowDebugBuild = BenchmarkProgram.IsDebugBuild; + o.SkipBenchmarksWithReports = true; + o.ConfigureBenchmarkDotNet(c => + { + var slimJob = BenchmarkWorkspaceOptions.Slim; + return c + .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); + }); + }); + } +} diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj new file mode 100644 index 0000000..55ac864 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tooling/benchmark-runner/benchmark-runner.csproj @@ -0,0 +1,10 @@ + + + Exe + net10.0 + + + + + + diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj new file mode 100644 index 0000000..d27d0c8 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj @@ -0,0 +1,8 @@ + + + net10.0 + + + + + diff --git a/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs new file mode 100644 index 0000000..8fc3d15 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/runner-skip/tuning/Acme.Core.Benchmarks/ParserBenchmark.cs @@ -0,0 +1,17 @@ +using BenchmarkDotNet.Attributes; + +namespace Acme.Core; + +[MemoryDiagnoser] +public class ParserBenchmark +{ + private readonly Parser _parser = new(); + + [Benchmark] + public int ParseTypical() => _parser.Parse("42"); +} + +public sealed class Parser +{ + public int Parse(string value) => int.Parse(value, System.Globalization.CultureInfo.InvariantCulture); +} diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md new file mode 100644 index 0000000..f4b0abd --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark-summary.md @@ -0,0 +1,14 @@ +# LegacyAliasCountBenchmark summary excerpt + +| Method | Size | Mean | Error | Allocated | +|---|---:|---:|---:|---:| +| CountAll | 8 | 0.381 ns | 0.02 ns | 0 B | +| CountLegacy | 8 | NA | NA | NA | +| CountAll | 256 | 0.380 ns | 0.02 ns | 0 B | +| CountLegacy | 256 | 1.206 us | 0.03 us | 40 B | +| CountAll | 4096 | 0.381 ns | 0.02 ns | 0 B | +| CountLegacy | 4096 | 10.944 us | 0.19 us | 40 B | + +Benchmarks with issues: + LegacyAliasCountBenchmark.CountLegacy(Size: 8, Job: DefaultJob) + GlobalSetup failed: InvalidOperationException: Workload must have at least one legacy alias. diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs new file mode 100644 index 0000000..8c07ee4 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasCountBenchmark.cs @@ -0,0 +1,37 @@ +using System; +using System.Collections.Generic; +using System.Linq; +using BenchmarkDotNet.Attributes; +using BenchmarkDotNet.Configs; + +namespace Acme.Search; + +[MemoryDiagnoser] +[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)] +public class LegacyAliasCountBenchmark +{ + [Params(8, 256, 4_096)] + public int Size { get; set; } + + private List _aliases = null!; + + [GlobalSetup] + public void Setup() + { + _aliases = Enumerable.Range(0, Size) + .Select(i => $"USR{i}") + .ToList(); + + var expectedMatches = LegacyAliasQuery.CountLegacyAliases(_aliases); + if (expectedMatches == 0) + { + throw new InvalidOperationException("Workload must have at least one legacy alias."); + } + } + + [Benchmark(Baseline = true, Description = "List.Count")] + public int CountAll() => _aliases.Count; + + [Benchmark(Description = "Where(alias.Length == 6).Count()")] + public int CountLegacy() => _aliases.Where(alias => alias.Length == 6).Count(); +} diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasImport.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasImport.cs new file mode 100644 index 0000000..0486ff1 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasImport.cs @@ -0,0 +1,12 @@ +using System.Collections.Generic; +using System.Linq; + +namespace Acme.Search; + +public sealed class LegacyAliasImport +{ + public IReadOnlyList BuildAliases(int count) => + Enumerable.Range(0, count) + .Select(i => $"USR{i:D3}") + .ToArray(); +} diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQuery.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQuery.cs new file mode 100644 index 0000000..d057440 --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQuery.cs @@ -0,0 +1,9 @@ +using System.Collections.Generic; +using System.Linq; + +namespace Acme.Search; + +public static class LegacyAliasQuery +{ + public static int CountLegacyAliases(IEnumerable aliases) => aliases.Count(alias => alias.Length == 6); +} diff --git a/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQueryTests.cs b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQueryTests.cs new file mode 100644 index 0000000..2c3e9da --- /dev/null +++ b/skills/dotnet-benchmark/evals/files/selectivity-drift/LegacyAliasQueryTests.cs @@ -0,0 +1,21 @@ +using System.Linq; +using Xunit; + +namespace Acme.Search; + +public class LegacyAliasQueryTests +{ + [Fact] + public void CountLegacyAliases_ReturnsAllFixedWidthAliases() + { + var aliases = Enumerable.Range(0, 8).Select(i => $"USR{i:D3}"); + Assert.Equal(8, LegacyAliasQuery.CountLegacyAliases(aliases)); + } + + [Fact] + public void CountLegacyAliases_AllowsZeroMatchesWhenScenarioIsNonLegacy() + { + var aliases = new[] { "NEW0001", "NEW0002", "NEW0003" }; + Assert.Equal(0, LegacyAliasQuery.CountLegacyAliases(aliases)); + } +} diff --git a/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md new file mode 100644 index 0000000..e10b8f5 --- /dev/null +++ b/skills/dotnet-benchmark/references/benchmarkdotnet-essentials.md @@ -0,0 +1,142 @@ +# BenchmarkDotNet Essentials + +Use this as a compact API and validation reference after the experiment question is defined. Official documentation: . + +## Core attributes + +| Attribute | Purpose | +|---|---| +| `[Benchmark]` | Marks measured work. Add `Baseline = true` only inside an equivalent comparison group and use `Description` to name the real measured terminal operation or materialization step. | +| `[MemoryDiagnoser]` | Reports managed allocations and GC counts. Always include it for codebelt benchmarks. | +| `[BenchmarkCategory("...")]` | Labels logical comparison groups when one class contains multiple alternative pairs. | +| `[GroupBenchmarksBy(...)]` | Groups report rows by category, params, or a deliberate combination. Use it only when the grouping is meaningful for interpretation; remove decorative grouping. | +| `[Params(...)]` | Sweeps independent compile-time-constant values. Multiple params properties create a Cartesian product. | +| `[ParamsSource(nameof(...))]` | Supplies computed or coupled scenario objects with readable names. | +| `[Arguments(...)]` / `[ArgumentsSource]` | Supplies method arguments, useful for explicit scenario sets. | +| `[GlobalSetup]` / `[GlobalCleanup]` | Prepares and releases per-method/per-parameter state outside measurement. | +| `[IterationSetup]` / `[IterationCleanup]` | Resets per iteration but forces single invocation/unroll; avoid for tiny microbenchmarks. | + +## Baselines + +A method baseline adds a ratio distribution against equivalent methods. BenchmarkDotNet allows category-specific baselines when benchmarks are grouped by category, but every baseline category still needs at least one equivalent peer. Do not attach a baseline to an unrelated member or a lone benchmark just to satisfy a template. + +Runtime comparisons can use a job baseline. Keep method, inputs, and implementation fixed when attributing a difference to runtime. + +## Prevent invalid measurements + +- Use Release builds and run without an attached debugger. +- Return or consume results to prevent dead-code elimination. +- Keep setup and correctness checks outside timed methods. +- Do not rely on execution order or shared mutation between methods. +- Avoid manual loops unless batching is the real workload; BenchmarkDotNet selects invocation counts automatically. +- Inspect all validation and environment warnings before reading result tables. +- Compare the discovered and reported method, job, and parameter matrix with the intended design; missing combinations are validation failures. +- Keep the machine powered and quiet for full runs unless the noisy/throttled environment is the intended target. + +## Validators + +BenchmarkDotNet always validates duplicate baselines. `ExecutionValidator` can smoke-execute cases and `ReturnValueValidator` can compare compatible return values, but domain-specific correctness checks remain necessary for every parameter and scenario case. Prefer exact checks for counts, values, status, exceptions, output contents, or mutations; use approximation only when the domain itself defines approximation. A Release build plus `--job dry` provides practical wiring and lifecycle validation through the codebelt runner, but zero-match boundary cases are still valid when the workload intends them. + +## Diagnosers and profilers + +- `[MemoryDiagnoser]`: allocations and GC, always enabled by this skill. +- `[ThreadingDiagnoser]`: completed thread-pool work items and monitor lock contention on .NET Core 3+. +- `[ExceptionDiagnoser]`: exception frequency for intentional exception-path experiments. +- `[DisassemblyDiagnoser]`: generated code; heavy and subject to toolchain/platform restrictions. +- EventPipe profiler: cross-platform CPU/GC/JIT trace artifacts for a targeted deep dive. +- ETW/native hardware diagnostics: Windows/privilege/toolchain restrictions; opt in only when needed. + +Diagnosers may require separate runs and increase duration. + +## Jobs and runtimes + +The codebelt runner starts from `BenchmarkWorkspaceOptions.Slim`. Add runtime jobs only for an explicit cross-runtime question: + +```csharp +var slimJob = BenchmarkWorkspaceOptions.Slim; +return c + .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core80)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); +``` + +| Target | Job runtime | +|---|---| +| .NET Framework 4.8 | `ClrRuntime.Net48` (Windows only) | +| .NET 8 | `CoreRuntime.Core80` | +| .NET 9 | `CoreRuntime.Core90` | +| .NET 10 | `CoreRuntime.Core10_0` | + +Only add jobs that the SUT and benchmark toolchain can execute. Keep the runner-default-only template as `return c;` with no unused runtime `using` directives. + +`BenchmarkWorkspaceOptions.Slim` is the codebelt repository's deliberately shortened developer-oriented job. Report its active settings accurately instead of saying BenchmarkDotNet chose a fully adaptive warmup or iteration plan. In the current codebelt runner shape, Slim fixes one warmup iteration plus controlled iteration counts; that can characterize ordinary repository workloads, but one warmup may be insufficient for tiered-compilation, tiered-PGO, LINQ, or other JIT-sensitive runtime comparisons. + +Do not silently replace the repository runner configuration. Use a more stable diagnostic job only when the runtime or JIT comparison is itself material and explicitly in scope. Short and dry jobs still validate or iterate quickly; they do not replace a valid full run for performance conclusions. + +## Deferred pipelines and terminal operations + +Distinguish between: + +- query or iterator creation only; +- a terminal operation such as `Count()`, `Any()`, or `First()`; +- explicit full enumeration; +- materialization through `ToArray()` or `ToList()`. + +Name the benchmark and its conclusion accordingly. Inspect collection fast paths before describing a result as enumeration. `List.Count` can be O(1) while `Where(...).Count()` enumerates the filtered pipeline, so the former is not an honest baseline for the latter. + +## Result-validity gate + +After a dry execution and after any full run, inspect the complete BenchmarkDotNet summary before interpreting numbers. Treat the run as invalid for performance interpretation when any intended case has an `NA` measurement, appears under `Benchmarks with issues`, throws in setup or cleanup, triggers a validation error, fails a job or runtime, or is missing from the intended matrix. + +If that happens, report the exact failing method, job, and parameter case. Do not let the surviving rows stand in for the completed benchmark. Correct benchmark-owned causes, rerun build, discovery, and dry validation, and require fresh human authority before any replacement full measurement run. + +## Existing-report filtering + +Codebelt library runners normally set `SkipBenchmarksWithReports = true`. The console runner filters a loaded `*Benchmark` type when a matching report already exists under the artifacts tuning folder, normally `reports/tuning/`. Therefore a successful build followed by an empty list/dry/full invocation can be expected behavior. + +Run the bundled detector with `-BenchmarkType ` and inspect its runner/report fields before changing a benchmark or adding diagnostics. Read `runner-preflight.md` for the matching rule and the no-thrashing workflow. Never rename a class, add disassembly, or disable the option merely to evade an existing report. + +## Runner commands + +Build: + +```console +dotnet build -c Release tuning/{SutProject}.Benchmarks/{SutProject}.Benchmarks.csproj +``` + +List cases without measuring: + +```console +dotnet run -c Release --project tooling/{runner} -- --list flat --filter *{BenchmarkClass}* +``` + +Dry execution smoke: + +```console +dotnet run -c Release --project tooling/{runner} -- --job dry --filter *{BenchmarkClass}* +``` + +Full default run: + +```console +dotnet run -c Release --project tooling/{runner} -- --filter *{BenchmarkClass}* +``` + +Reports are written under `reports/`. + +## Interpreting a full run + +Read warnings before tables. Compare mean, error, standard deviation, median when distributions are skewed, ratio distributions within valid groups, allocated bytes, GC counts, and specialized diagnoser columns for the exact workload and environment. + +After the first valid full result, answer three questions: is the result reproducible, is the absolute cost material for observed or plausible usage, and would a deeper diagnostic change a concrete engineering decision? Stop when the answer is no. Do not escalate automatically to repeated reruns, tiered-PGO variants, disassembly, EventPipe or ETW tracing, alternative implementations, or runtime-source archaeology. Escalate only when the result is reproducible, material, the next diagnostic can distinguish concrete competing explanations, and the user requested it or it is necessary to answer the original decision. + +## Primary sources + +- BenchmarkDotNet good practices: +- Parameterization: +- Setup and cleanup: +- Baselines: +- Diagnosers: +- Validators: +- .NET diagnostics overview: diff --git a/skills/dotnet-benchmark/references/candidate-selection.md b/skills/dotnet-benchmark/references/candidate-selection.md new file mode 100644 index 0000000..5d1da1b --- /dev/null +++ b/skills/dotnet-benchmark/references/candidate-selection.md @@ -0,0 +1,110 @@ +# Selecting High-Value Benchmark Candidates + +Use this reference when the user supplies a type but does not already provide an exact performance hypothesis. The goal is to find a small set of operations whose measurement could plausibly guide an optimization decision. This is source-informed discovery, not proof that the operation dominates a deployed workload. + +## Evidence ladder + +Prefer evidence in this order: + +1. Production telemetry, traces, allocation profiles, contention traces, or an existing performance regression. +2. Representative application benchmarks or load-test results that identify the type or call path. +3. Real call sites that reveal frequency, batching, concurrency, and input shape. +4. Tests and examples that reveal valid, boundary, failure, and compatibility scenarios. +5. Implementation inspection that reveals algorithmic scaling, allocations, synchronization, code generation, or repeated work. +6. Public API shape alone. + +Do not promote weak evidence to a stronger label. When no profile exists, say that the candidate is selected from source and usage evidence. + +## Inspection sequence + +1. Read the complete type, including partial declarations and generated-source inputs where available. +2. Identify the public or internal consumer operations that enter the type. Include inherited/interface operations when call sites use them. +3. Search call sites across `src/`, tests, samples, tooling, and benchmarks. Note loop nesting, batch sizes, collection use, concurrency, and repeated calls. +4. Read focused tests and examples to learn realistic inputs, equivalence rules, error behavior, and boundary cases. +5. Follow the selected entry point into private helpers and important collaborators far enough to understand dominant work. Do not benchmark private helpers merely because they look expensive. +6. Inspect existing benchmarks and reports before adding another suite. Extend a compatible benchmark rather than duplicating it. +7. If profiling artifacts exist, use their hot stacks/allocation types/contention sites to confirm or reorder candidates. + +Useful searches include the exact type name, constructed generic forms, interface/base-type names, factory methods, extension methods, and characteristic member names. Search aliases and static imports when a direct type search is sparse. + +## Cost signals + +Treat these as hypotheses that require measurement, not defects by themselves: + +- Nested loops, repeated scans, recursion, sorting, hashing, or work whose complexity grows with input size. +- Per-call `new` allocations, boxing, closures, iterator state machines, LINQ materialization, array/string copies, interpolation, and repeated buffer growth. +- Repeated parsing, formatting, normalization, encoding/decoding, regular-expression construction, serialization, reflection, expression compilation, or metadata lookup. +- `GetHashCode` and `Equals` on types heavily used as dictionary/set keys, especially when they traverse fields, strings, collections, or normalized forms. +- Locks, concurrent collections, atomics, task scheduling, blocking waits, and shared mutable state used from multiple call sites. +- Exception construction or throwing on a documented/common path. +- Cache misses, lazy initialization, cold initialization, and expensive work that could be hoisted or reused. +- Span/array conversions, pinning, interop, crypto transforms, compression, and buffer pooling. +- Branch-heavy parsing/matching whose cost changes for success/failure, hit/miss, early/late match, or well/ill-shaped input. + +Trivial getters, constant-returning properties, thin wrappers, and rarely used diagnostic formatting are low priority unless call-site evidence shows exceptional frequency or a regression specifically names them. + +## Candidate matrix + +Create a compact matrix before choosing benchmark methods: + +| Candidate operation | Usage evidence | Cost/scaling signal | Representative cases | Comparison available | Measurement fitness | Decision | +|---|---|---|---|---|---|---| +| `TryParse` | Called for each imported record | Scans and allocates normalized strings | short/typical/large; valid/invalid | current vs span candidate | deterministic, in-memory | Select | +| `ToString` | Debug/logging only | one allocation | typical object | none | measurable but low impact | Reject | + +Use a score only as a prioritization aid, never as a performance claim. A practical score is: + +- 0–3 for observed frequency/importance; +- 0–3 for per-call cost or input scaling; +- 0–2 for allocation, contention, or cold-path significance; +- 0–2 for optimization leverage or a credible comparison; +- 0–2 for deterministic measurement fitness; +- subtract 0–3 for external noise, unresettable state, unrealistic isolation, or weak workload evidence. + +Record the evidence behind each score. A high score based only on source appearance is still exploratory. + +## Select the performance questions + +Choose one to three questions that could change an engineering decision. Good questions are specific: + +- Does the span-based parser reduce latency and allocations versus the current string parser for representative valid and invalid inputs? +- How does wildcard matching scale across path length and pattern complexity, and does the current method allocate per call? +- Under a fixed worker count and hit/miss mix, does cache lookup show lock contention? + +Weak questions merely inventory members: + +- How fast are all public methods? +- Is `ToString` faster than `GetHashCode`? +- What happens if every available enum and size is crossed with every method? + +If several unrelated operations are genuinely important, prefer separate benchmark classes or explicit comparison categories so reports do not imply invalid ratios. + +## Workload cases + +Derive cases from call sites and tests. Include only dimensions that can change the conclusion: + +- typical production-like case; +- small/empty boundary when valid; +- a scaling point large enough to expose complexity; +- adverse-but-valid case such as miss, late match, escaped content, collision, invalid parse, or cache miss; +- cold/warm state only when both are meaningful consumer modes. + +Random bytes are appropriate for some codecs, hashes, and raw buffers, but not as a universal workload. Parsers, matchers, compressors, collections, and caches often need structured cases. Use fixed seeds only after choosing a representative distribution. + +Avoid independent parameter properties when values are coupled. `[Params]` creates the Cartesian product of every axis. Use a scenario record/class with a readable `ToString()` supplied through `[ParamsSource]`, or use `[ArgumentsSource]`, to enumerate only meaningful combinations. + +## Profiling-first gate + +Recommend profiling or a macrobenchmark before writing a microbenchmark when: + +- the user asks what makes an application or endpoint slow but provides no hotspot evidence; +- the type mainly coordinates network, disk, database, process, UI, or distributed work; +- the important behavior is request concurrency, queueing, thread-pool starvation, or tail latency across components; +- startup/JIT/module loading for a whole application is the target; +- state cannot be reset reproducibly or isolation changes the behavior under investigation. + +A useful response still narrows the next step: name the entry point, workload, and signal to collect. After profiling identifies a controllable code path, return to BenchmarkDotNet to compare implementations under a reproducible workload. + +## Candidate rejection rules + +Reject or defer a candidate when it has no plausible consumer impact, duplicates a selected operation, cannot be isolated without changing semantics, measures mostly test scaffolding, crosses unrelated work, depends on uncontrolled external systems, or has no representative workload. Report the reason so the user can correct missing context. diff --git a/skills/dotnet-benchmark/references/codebelt-conventions.md b/skills/dotnet-benchmark/references/codebelt-conventions.md new file mode 100644 index 0000000..5ef4bd6 --- /dev/null +++ b/skills/dotnet-benchmark/references/codebelt-conventions.md @@ -0,0 +1,61 @@ +# Codebelt Benchmark Conventions + +Follow these repository conventions so benchmark projects remain discoverable and consistent across codebelt repositories. Experiment validity takes precedence over forcing a template shape. + +## Naming and placement + +- Benchmark projects live under `tuning/` and are named `{SutProject}.Benchmarks`, such as `Cuemon.Core.Benchmarks`. +- A benchmark class name ends with `Benchmark` and names the measured question or type, such as `DateSpanFormattingBenchmark` or `Sha512256ComparisonBenchmark`. +- The benchmark class uses the same namespace as the type it measures; do not append `.Benchmarks`. The project overrides `RootNamespace` to the SUT root. +- Method names distinguish implementations or scenarios and every `[Benchmark]` has a readable `Description`. +- One `tooling/` runner host discovers all `tuning/` projects through the existing wildcard project reference. + +## Default instrumentation + +Every codebelt benchmark class uses `[MemoryDiagnoser]`. Add `[GroupBenchmarksBy]` when the class has multiple methods/params and the grouping makes the report clearer. Use `[GlobalSetup]` when state or correctness checks must be prepared outside measurement; do not add an empty setup merely for visual consistency. + +Baselines follow experiment semantics: + +- A current/reference implementation is `Baseline = true` when one or more equivalent alternatives are compared. +- A single-operation characterization benchmark has no baseline. +- Unrelated operations do not share a baseline. +- A cohesive class with several alternative pairs uses `[BenchmarkCategory]`, grouping by category, and one baseline per category. + +This refines the older “exactly one baseline per class” shortcut. BenchmarkDotNet ratios are useful only when the grouped methods perform comparable work. + +## Experiment shapes + +### Equivalent implementation comparison + +Use `assets/comparison-benchmark.cs` when the same consumer operation has a current/reference and candidate implementation. Keep inputs and wrappers symmetric, validate equivalent results in setup, and sweep only dimensions that can change the comparison. + +### Single-operation characterization + +Use `assets/operation-benchmark.cs` when there is no honest competing implementation. This shape characterizes time, allocations, and scaling for one operation across representative cases without inventing a baseline. + +### Multiple independent questions + +Prefer separate focused classes. If existing repository convention keeps them together, use categories that prevent unrelated ratios. Construction, formatting, equality, hashing, parsing, and matching are not automatically comparable merely because they belong to one type. + +## Deterministic inputs + +Build inputs outside measured methods. Use fixed seeds only when pseudo-random data matches the domain. Prefer structured inputs for parsers, matchers, collections, caches, and serializers. Parameter cases should have readable report labels. + +Do not use network, disk, database, logging, or sleep calls inside a microbenchmark. When external behavior is the point, select profiling, load testing, or a macrobenchmark and preserve codebelt project placement only if it remains useful. + +## Runner and reports + +The runner calls `Codebelt.Extensions.BenchmarkDotNet.Console.BenchmarkProgram.Run`, reuses the repository's existing name such as `benchmark-runner` or `bdn-runner`, and writes artifacts under `reports/`. Codebelt libraries normally keep `SkipBenchmarksWithReports = true`; a matching file under `reports/tuning/` intentionally filters that benchmark type on later invocations. Run the `runner-preflight.md` checks before diagnosing benchmark code. + +Default runtime configuration stays plain `return c;`. For explicit runtime jobs, assign `BenchmarkWorkspaceOptions.Slim` once to `slimJob`, then add `slimJob.WithRuntime(...)` for each supported TFM. The TFM list is the expected repository-specific difference; preserve the surrounding lean runner shape. + +## Reference implementations + +Repository precedents include: + +- `codebeltnet/cuemon/tuning/Cuemon.Core.Benchmarks/DateSpanBenchmark.cs` +- `codebeltnet/cuemon/tuning/Cuemon.Security.Cryptography.Benchmarks/Sha512256Benchmark.cs` +- `codebeltnet/xunit/tuning/Codebelt.Extensions.Xunit.Benchmarks/TestBenchmark.cs` +- the `tooling/bdn-runner` or `tooling/benchmark-runner` hosts in those repositories + +Use these for placement and runner conventions, not as authority to preserve a misleading experiment. Adapt the benchmark design to the actual performance question. diff --git a/skills/dotnet-benchmark/references/experiment-design.md b/skills/dotnet-benchmark/references/experiment-design.md new file mode 100644 index 0000000..d33b98c --- /dev/null +++ b/skills/dotnet-benchmark/references/experiment-design.md @@ -0,0 +1,232 @@ +# Designing Trustworthy BenchmarkDotNet Experiments + +Use this reference after candidate selection. Every benchmark method should answer a stated performance question with representative inputs and controlled conditions. + +## Define the experiment before the attributes + +Write down: + +- the operation and consumer-visible behavior being measured; +- the primary metric: time/throughput, allocations, scaling, contention, cold start, or exception frequency; +- baseline and candidate implementations, if both exist; +- parameter cases and why each can change the result; +- the exact expected observable result for each scenario; +- how each expected result is derived independently of the measured API path; +- workload characteristics that the scenario name promises, such as match count, selectivity, hit rate, type distribution, or branch distribution; +- setup/reset/disposal strategy; +- a correctness oracle; +- environmental variables that must remain fixed. + +If these cannot be defined, more inspection is needed. Attributes do not rescue an ambiguous experiment. Successful execution is not a correctness oracle. + +## Comparison semantics + +A baseline exists to create a meaningful ratio. Use the current production implementation or an established reference implementation as the baseline when all methods in the group perform equivalent observable work on identical input. + +Do not compare unrelated operations. Construction, parsing, formatting, equality, hashing, copying, and validation answer different questions. Give them separate classes, or use explicit `[BenchmarkCategory]` groups with one baseline inside each group only when every category has alternatives. + +For a single current implementation measured across sizes or scenarios, omit `Baseline = true`. The parameter columns and absolute time/allocation results already describe scaling. A fake baseline adds no evidence. + +When comparing runtimes rather than implementations, use a job baseline and keep the measured method the same. Do not mix runtime and algorithm changes in one conclusion unless the full matrix is intentional. + +Before accepting a method baseline, answer: do the baseline and candidate perform equivalent consumer-visible work over identical logical input? Reject baselines where one side returns an existing collection while the other enumerates, uses an O(1) fast path while the other scans, filters while the other does not, materializes while the other stays deferred, or validates/parses/transforms different semantics. + +## Configuration coherence + +Before build, list, or dry validation, read the attributes back as a matrix: + +- `Baseline = true` only where methods do equivalent observable work. +- `[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]` only when meaningful `[BenchmarkCategory]` values exist. +- Every category with a baseline contains at least one equivalent comparison method. +- Single-operation characterization benchmarks do not invent a method baseline. +- `[Benchmark(Description = "...")]` names the real measured terminal operation or materialization step. +- Remove grouping attributes that do not improve interpretation. + +## Representative inputs + +Prefer cases grounded in real usage. Typical input should come first; add boundaries and adverse cases only when they exercise a distinct path. + +Examples: + +- Parsers: valid typical, valid large, invalid early, invalid late, escaped/culture-sensitive when supported. +- Match/search: hit early, hit late, miss, simple pattern, complex pattern, realistic lengths. +- Collections: empty/small/typical/large, hit/miss, collision-heavy only when plausible, pre-sized versus growth only if that is the question. +- Hash/codec/compression: representative size distribution and content entropy, not only zero-filled or arbitrary random buffers. +- Equality/hashing: equal, unequal-early, unequal-late, and actual dictionary/set usage when that is the consumer operation. +- Caches: warm hit, miss/fill, eviction, and contention as separate questions. + +Use `[Params]` for independent scalar dimensions and `[ParamsSource]`/`[ArgumentsSource]` for coupled cases. Give complex case objects stable readable names through `ToString()` so reports are interpretable. + +Keep the case count disciplined. Each parameter combination multiplies methods, jobs, launch count, and diagnoser runs. Three representative sizes usually tell more than a dense power-of-two sweep; add points only when they locate a threshold or crossover. + +## Workload invariants + +When one parameter only scales size, payload length, or another magnitude, keep other workload characteristics stable unless one is intentionally varied as its own named scenario. Common invariants include predicate selectivity or match percentage, hit/miss ratio, valid/invalid ratio, branch distribution, collision rate, string-length or encoding distribution, type distribution, entropy, and cache state. + +If one of those characteristics should vary, make it an explicit scenario or parameter instead of letting it drift accidentally with size. + +For predicate and filter benchmarks: + +- state the intended selectivity; +- use deterministic data; +- prefer fixed-width or structurally stable inputs when digit length or formatting would otherwise move cases between branches; +- verify the exact expected match count for every scenario, including valid zero-match cases. + +If a case name promises an exact percentage such as `10% selectivity`, make that statement true for every size under a documented rounding rule. When that is not possible, choose compatible sizes, define explicit scenario objects with exact expected counts, or rename the cases honestly with qualitative names such as `LowSelectivity`, `HalfMatches`, or `AllMatch`. + +### Runtime-type filtering + +For `QueryFor`-style or other runtime-type filters, use deterministic heterogeneous input: + +- exact matches; +- sibling nonmatches; +- derived types when exact-type versus assignability semantics matter; +- nulls only when supported and relevant. + +Validate the exact expected count and, when it matters, insertion order and exact runtime types. Do not benchmark a homogeneous all-match store and call it type filtering unless the all-match path is the actual subject. + +## Setup, state, and disposal + +Use `[GlobalSetup]` for deterministic state that is not part of the operation: payload creation, parsing expected results, object construction for instance methods, and correctness checks. BenchmarkDotNet runs global setup for each benchmark method and parameter combination, so setup must not rely on another benchmark method having run first. + +Use `[GlobalCleanup]` for resources owned by the benchmark. Avoid external I/O resources in microbenchmarks; cleanup does not remove their measurement noise. + +Mutating operations need equivalent starting state. Options include: + +- create or clone state inside every benchmark method when that creation is genuinely part of the compared operation and both sides pay the same cost; +- benchmark a representative batch/sequence over prebuilt state and use `OperationsPerInvoke` when per-operation normalization remains honest; +- use iteration setup only for macro-scale operations where forced single-invocation behavior will not dominate. + +`[IterationSetup]` forces one invocation/unroll factor and can distort tiny operations. Do not use it reflexively to reset a nanosecond-scale benchmark. + +Avoid state leakage between methods, params, warmup, and measurement. Never depend on benchmark execution order. + +## Correctness oracle + +An optimization benchmark without correctness validation can reward wrong code. For every scenario, define the input, intended operation, exact expected observable result, how that result was derived independently of the measured API path, and any promised workload characteristic such as selectivity, hit rate, or type mix. Successful execution is not a correctness oracle. Before a full run: + +1. Execute baseline and candidate for every scenario outside the timed method. +2. Compare exact observable results with the domain's real equivalence rule, including counts, values, status codes, exceptions, output buffers, mutations, side effects, insertion order, and runtime types when they matter. +3. Ensure the expected value is derived independently of the measured API path; do not turn the implementation's current behavior into the oracle by computing `_expected` with the same call you intend to benchmark. +4. Fail setup or a focused test when results differ or when the generated workload does not match the named scenario. +5. Keep the assertion/check out of the timed path. + +Approximate checks are acceptable only when approximation is part of the domain semantics and the benchmark documents that rule. A valid boundary case that produces zero matches must still pass; only unexpected zero results should fail setup and force workload redesign. + +A check such as `result == 0` is acceptable only when zero versus nonzero is the real domain contract. For collections, filters, parsers, matchers, and type selection, validate exact counts or exact outputs for every parameter case. + +For non-equivalent APIs, do not force a comparison. Characterize them separately and state the semantic difference. + +BenchmarkDotNet's `ReturnValueValidator` can supplement this for compatible return values, but it does not replace domain-aware correctness checks. A reference implementation may act as the oracle only when it is identified explicitly as trusted and is semantically equivalent. + +## Prevent dead-code elimination and accidental work + +Return the computed result whenever practical. For `void`, ref-like, or multi-output work, consume observable outputs with `BenchmarkDotNet.Engines.Consumer` or return a stable derived value. Do not return a precomputed field while discarding the measured call. + +Keep logging, assertions, `Random`, fixture construction, reflection discovery, and string formatting used only to label cases out of timed methods. Avoid closures and LINQ in the benchmark wrapper unless they are the subject under test. + +Do not add a manual loop merely to make a tiny method measurable; BenchmarkDotNet chooses invocation counts and subtracts overhead. Use an in-method loop only when a batch is the real workload or state reset requires a defined sequence, then declare `OperationsPerInvoke` accurately. + +## Deferred execution and terminal operations + +LINQ-style pipelines and iterators can measure different work: + +- query or iterator creation only; +- a terminal operation such as `Count()`, `Any()`, or `First()`; +- explicit full enumeration, such as a `foreach`; +- materialization through `ToArray()` or `ToList()`. + +Benchmark names, descriptions, and conclusions must say which one is measured. Inspect optimized terminal-operation paths before calling a result "enumeration." `List.Count` can be an O(1) collection fast path while `Where(...).Count()` enumerates the filtered pipeline; they are not interchangeable baselines. + +## Fair comparisons + +- Feed identical logical inputs and starting state to every implementation. +- Match API semantics, validation, culture, encoding, comparer, error handling, and output ownership. +- Do not let one side reuse cached/precompiled state while the other recreates it unless the experiment explicitly compares those consumer strategies. +- Keep wrapper overhead symmetric. If one API requires an adapter, decide whether the adapter is part of real consumer cost and document it. +- Do not compare a scalar API with a batched/vectorized API per invocation without normalizing per item and explaining the workload difference. +- Keep compiler settings, runtime, architecture, GC mode, environment variables, and affinity consistent unless one of them is the experimental variable. + +## Specialized workloads + +### Async + +Return and await the real `Task` or `ValueTask`. Never use `.Result`/`.Wait()`. Separate synchronous completion from genuinely asynchronous completion when both occur in production. A fake completed task measures the fake, not the I/O path. Use macro/load testing for external async I/O. + +### Concurrency and contention + +Define worker count, shared state, operation mix, synchronization start, and per-operation normalization. Avoid measuring task creation when lock/collection throughput is the question. Add `[ThreadingDiagnoser]` when completed work items or lock-contention counts help interpret the run. For service-level throughput, queueing, or tail latency, use a load test or profiler instead of a naive BenchmarkDotNet loop. + +### Cold start and initialization + +Keep cold-start experiments separate from steady-state throughput. Use a cold-start job only when process/JIT/initialization cost is the question, and define what is cold: process, type initializer, cache, parser, or application host. + +### Exceptions and invalid inputs + +Benchmark an exception path only when it is part of documented or observed behavior. Separate successful and throwing paths, add `[ExceptionDiagnoser]` when frequency matters, and do not use exceptions as a substitute for invalid-result checks. + +### Extremely fast operations + +Benchmark trivial getters/operators only with evidence of extreme frequency or a known regression. Ensure the result is consumed. Inspect disassembly when the question concerns inlining, bounds-check elimination, vectorization, or code generation. + +## Diagnoser routing + +`[MemoryDiagnoser]` is the codebelt default and belongs on every benchmark class. Add only what answers the question: + +| Question | Diagnoser/tool | Notes | +|---|---|---| +| Managed allocations and GC | `[MemoryDiagnoser]` | Always on; interpret allocated bytes per operation. | +| Lock/thread-pool activity | `[ThreadingDiagnoser]` | .NET Core 3+; useful for contention experiments. | +| Thrown exception frequency | `[ExceptionDiagnoser]` | Use only for intentional exception-path analysis. | +| JIT/code generation | `[DisassemblyDiagnoser]` | Heavy; platform/toolchain limitations apply. | +| CPU/GC/JIT hot stacks | EventPipe profiler | Cross-platform targeted deep dive; creates separate profiling artifacts. | +| Windows ETW/native counters | ETW/hardware counter diagnosers | Platform/privilege restrictions; enable only when required. | + +Diagnosers can create additional runs and change total duration. Do not combine every diagnoser into a default benchmark. + +## Semantic preflight + +Before build, list, or dry validation, inspect the full Cartesian set of benchmark methods, parameter values, runtime jobs, and scenario objects. For every case verify: + +- setup succeeds; +- the exact expected output is known independently; +- the exact actual output matches; +- the named workload characteristic is true; +- the case exercises the intended code path; +- the case does not collapse into a trivial fast path unless that fast path is the stated subject. + +Semantic preflight +- Every scenario has an independently derived exact expected result. +- Every parameter combination satisfies that oracle. +- Benchmark names accurately describe the generated workload. +- Selectivity, hit rate, type mix, and other workload distributions are explicit and verified. +- Every benchmark exercises the intended path. +- Baselines compare equivalent observable work. + +If any item fails, fix the benchmark before build, list, dry validation, or human authorization for a full run. + +## Layered validation + +1. Run the semantic preflight across the full case matrix. +2. Run existing correctness tests for the SUT where feasible. +3. Execute the benchmark's correctness oracle for every case. +4. Build the benchmark project in Release. +5. Run `--list flat` with a filter and confirm the expected case/method combinations without accidental Cartesian products. +6. Run a `--job dry` execution smoke and resolve BenchmarkDotNet validation warnings or runtime failures. +7. Run the full default job only with explicit user intent and a suitable environment. + +A dry job has too few measurements for conclusions. It validates wiring and lifecycle only. + +## Benchmark validity gate + +After a dry execution and after any full run, inspect the complete BenchmarkDotNet summary before interpreting numbers. Treat the benchmark as invalid for performance interpretation when any intended case has an `NA` measurement, appears under `Benchmarks with issues`, throws in setup or cleanup, triggers a validation error, fails a job or runtime, or is missing from the intended matrix. + +When that happens, report the exact failing method, job, and parameter case. Do not treat successful rows as a completed benchmark. Correct the benchmark only when the cause lies within the benchmark, then rerun build, discovery, and dry validation. Another full run still requires explicit human authority. + +## Interpreting a full run + +Report the benchmark environment and exact workload. Read warnings before tables. Compare mean, error, standard deviation, median when distributions are skewed, ratio distributions within valid groups, allocated bytes, GC counts, and specialized diagnoser columns. + +Treat gains within noise as inconclusive. Check whether the result holds across representative cases and whether one case regresses. State the scope precisely: a result applies to the measured runtime, hardware, inputs, and configuration. Optimization value depends on application frequency and the maintenance/correctness cost of the change. + +After the first valid full result, answer three questions: is the result reproducible, is the absolute cost material for observed or plausible usage, and would a deeper diagnostic change a concrete engineering decision? Stop when the answer is no. Do not escalate automatically to repeated reruns, tiered-PGO variants, disassembly, EventPipe or ETW tracing, alternative implementations, or runtime-source archaeology. Escalate only when the result is reproducible, material, the next diagnostic can distinguish concrete competing explanations, and the user requested it or it is necessary to answer the original decision. Low-microsecond test-support utilities often end with "no change justified." diff --git a/skills/dotnet-benchmark/references/onboarding.md b/skills/dotnet-benchmark/references/onboarding.md new file mode 100644 index 0000000..28344e8 --- /dev/null +++ b/skills/dotnet-benchmark/references/onboarding.md @@ -0,0 +1,118 @@ +# Onboarding a Benchmark Harness Into an Existing Repo + +Use this when Step 1 detection shows the harness is missing or partial. The goal is to add **only** +what is missing while matching whatever layout the repo already uses. Never restructure the repo, +rename its solution, or convert `.sln` to `.slnx`. + +## Decision inputs (from `scripts/check-benchmark-requirements.ps1`) + +| Signal | Why it matters | +|--------|----------------| +| `sdk` | Hard prerequisite. If absent, stop and ask the user to install the .NET SDK. | +| `solution` / `solutionFormat` | Determines how you wire new projects (`.slnx` XML vs `dotnet sln add`). | +| `centralPackageManagement` | Chooses `` in `Directory.Packages.props` vs versioned ``. | +| `centralizesBenchmarkConventions` | If the root `Directory.Build.props` defines `IsBenchmarkProject`/`IsToolingProject`, project files stay minimal. | +| `benchmarkProjects` | Existing `tuning/*.Benchmarks` projects to reuse instead of recreating. | +| `runner` | Existing `tooling/` runner host (folder name + whether it references the Console package). Reuse it; do not add a second. | + +## 1. Packages + +Resolve the **latest stable listed** versions from NuGet.org at author time (never hardcode from an +example). Resolve each package ID independently: + +- `BenchmarkDotNet` +- `BenchmarkDotNet.Diagnostics.Windows` +- `Codebelt.Extensions.BenchmarkDotNet.Console` + +Resolution source: the NuGet V3 service index `https://api.nuget.org/v3/index.json`; prefer the +registration resource so you can skip unlisted and prerelease versions. + +- **CPM repo** (`Directory.Packages.props` present): add `` + entries, and reference the packages **without** a version in the project files. +- **Non-CPM repo**: put versioned `` directly in the + project files that need them (benchmark project needs the two `BenchmarkDotNet*` packages; runner + needs the Console package). + +## 2. Benchmark project (`tuning/{SutProject}.Benchmarks/`) + +Create from `assets/benchmark.csproj`. Substitutions: + +| Placeholder | Value | +|-------------|-------| +| `{ROOT_NAMESPACE}` | The SUT's root namespace (e.g. `Cuemon`), so benchmarks compile into the measured namespace. | +| `{SUT_PROJECT}` | The owning `src/` project name (e.g. `Cuemon.Core`). | + +The default `assets/benchmark.csproj` is the **minimal** codebelt form and assumes the root +`Directory.Build.props` injects the benchmark TFMs and `BenchmarkDotNet*` packages (as codebelt repos +do). If `centralizesBenchmarkConventions` is **false**, make the project self-contained by adding: + +```xml + + net10.0;net9.0 + false + + + + + + +``` + +(Use versioned ``s here if the repo is non-CPM.) Pick benchmark TFMs the SUT +actually supports; `net10.0;net9.0` matches `Codebelt.Extensions.BenchmarkDotNet` availability. + +## 3. Runner host (`tooling/{runner}/`) + +If `runner` already exists, **reuse it** — the wildcard `..\..\tuning\**\*.csproj` reference already +picks up new benchmark projects, so you usually change nothing here. Only create a runner when none +exists. + +When creating one, default the folder name to `benchmark-runner` (matches the codebelt library +scaffold and `codebeltnet/xunit`). Copy `assets/benchmark-runner.csproj` and +`assets/benchmark-program.cs`. Substitutions: + +| Placeholder | Value | +|-------------|-------| +| `{RUNNER_TARGET_FRAMEWORK}` | Highest supported non-preview executable TFM (`net10.0` or `net9.0`). | +| `{RUNNER_NAMESPACE}` | Runner folder name converted to a valid C# identifier (`benchmark-runner` -> `benchmark_runner`). | +| `{RUNTIME_USINGS}` | Empty for **Runner default only**. Otherwise emit `using Codebelt.Extensions.BenchmarkDotNet;`, `using BenchmarkDotNet.Configs;`, `using BenchmarkDotNet.Environments;`, and `using BenchmarkDotNet.Jobs;`, each terminated with a newline. The slim job and runtime-specific `AddJob` chain need all four namespaces. | +| `{RUNTIME_SETUP}` | Empty for **Runner default only**. Otherwise emit the indented `var slimJob = BenchmarkWorkspaceOptions.Slim;` declaration used by every configured runtime job. | +| `{RUNTIME_JOBS}` | Empty for **Runner default only** so the method stays `return c;`. Otherwise emit newline-prefixed chained `.AddJob(slimJob.WithRuntime(...))` calls, one per runtime to measure (see `benchmarkdotnet-essentials.md`). | + +If the root `Directory.Build.props` does **not** mark tooling projects as executables, add +`Exe` and `false` to the runner `PropertyGroup`. + +## 4. Solution wiring + +- **`.slnx`**: add `` + under a `` element, and the runner under ``. + Create the folders if absent. Example: + + ```xml + + + + + + + ``` + +- **`.sln`**: run `dotnet sln .sln add ` for each new project. `dotnet` + places them under solution folders automatically. + +- **No solution file**: it is fine to leave projects unlisted; note it for the user. Do not fabricate + a solution unless they ask. + +## 5. `reports/` folder + +Benchmark output is written under `reports/` by the Codebelt workspace. You do not need to pre-create +it; the runner creates it on first run. Mention it so the user knows where results land. + +The standard runner sets `SkipBenchmarksWithReports = true`. Existing matching files under `reports/tuning/` deliberately filter their benchmark type from later list/dry/full invocations. Read `runner-preflight.md` before treating a skipped type as a benchmark-code defect. + +## Guardrails + +- Preserve UTF-8 (no BOM unless the source had one) when writing generated files; watch for mojibake. +- Do not commit or push; leave that to the user. +- If you cannot resolve a package version or cannot determine the SUT project, stop and report rather + than guessing. diff --git a/skills/dotnet-benchmark/references/runner-preflight.md b/skills/dotnet-benchmark/references/runner-preflight.md new file mode 100644 index 0000000..8f3c92d --- /dev/null +++ b/skills/dotnet-benchmark/references/runner-preflight.md @@ -0,0 +1,71 @@ +# Codebelt Runner Preflight + +Use this reference before diagnosing a benchmark that builds but is not listed or executed. Codebelt runners commonly enable `SkipBenchmarksWithReports`, so an existing report can make a healthy benchmark appear to do nothing. + +## Canonical runner shape + +The runtime list varies by repository TFMs; the surrounding setup should stay lean and recognizable: + +```csharp +using Codebelt.Extensions.BenchmarkDotNet; +using Codebelt.Extensions.BenchmarkDotNet.Console; +using BenchmarkDotNet.Configs; +using BenchmarkDotNet.Environments; +using BenchmarkDotNet.Jobs; + +namespace benchmark_runner; + +public class Program +{ + public static void Main(string[] args) + { + BenchmarkProgram.Run(args, o => + { + o.AllowDebugBuild = BenchmarkProgram.IsDebugBuild; + o.SkipBenchmarksWithReports = true; + o.ConfigureBenchmarkDotNet(c => + { + var slimJob = BenchmarkWorkspaceOptions.Slim; + return c + .AddJob(slimJob.WithRuntime(ClrRuntime.Net48)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core90)) + .AddJob(slimJob.WithRuntime(CoreRuntime.Core10_0)); + }); + }); + } +} +``` + +Default-runtime-only runners intentionally omit the runtime/job usings and `slimJob`, leaving `ConfigureBenchmarkDotNet` as `return c;`. + +## Why a benchmark can be skipped + +With `SkipBenchmarksWithReports = true`, the Codebelt console runner enumerates files under the configured BenchmarkDotNet artifacts path plus the tuning folder, normally `reports/tuning/`. For each report, it takes the filename portion before the first `-`, then the final dotted segment, and compares that value case-insensitively with loaded types whose names end in `Benchmark`. A matching report adds a filter that excludes that entire benchmark type. + +For example, `reports/tuning/Acme.Core.ParserBenchmark-report-github.md` causes `ParserBenchmark` to be filtered. This is expected idempotent runner behavior, not evidence that the benchmark class, filter, diagnoser, namespace, or method name is wrong. + +## Preflight sequence + +1. Build the benchmark project in Release. Stop on a compiler error; report it directly. +2. Run the detector with the intended benchmark type: + + ```console + pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "" -BenchmarkType + ``` + + If `pwsh` 7+ is unavailable, report that blocker instead of invoking legacy Windows PowerShell. + +3. Inspect `runner.programPath`, `runner.skipBenchmarksWithReports`, `runner.usesSlimJob`, `runner.configuredRuntimes`, `reports.tuningPath`, `reports.matchingReportFiles`, and `reports.wouldSkipRequestedBenchmark`. +4. Verify that configured runtime jobs match the repository's supported TFMs. The TFM list is the normal variable; do not rewrite the runner merely to make it look different. +5. If `wouldSkipRequestedBenchmark` is true, explain that the existing report deliberately suppresses the type. Treat an empty list/dry/full invocation as accounted for and leave the benchmark class and runner unchanged. +6. Only investigate filters, discovery, class visibility, attributes, toolchains, or diagnosers when the report preflight does not explain the behavior. + +## Rerun boundary + +Do not delete, move, overwrite, or ignore reports automatically. Do not flip `SkipBenchmarksWithReports` to false or rename a benchmark to evade the filter. Those actions discard the runner's intentional idempotency and can create duplicate or misleading result history. + +If the human explicitly requests a fresh full run, report the matching files and ask them to choose the repository's accepted report-retention action, such as archive/replace. After that explicit choice, preserve the canonical runner setup and rerun the same benchmark type. Yolo mode never supplies the human authorization for the full run or report replacement. + +## Anti-thrashing rule + +An existing matching report is a terminal explanation for runner-level skipping. Do not respond by adding `DisassemblyDiagnoser`, changing benchmark attributes, expanding tool calls, renaming a slim class, or rewriting a focused benchmark. Diagnostics and class changes must answer a separate evidence-backed performance question, not work around report filtering. diff --git a/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 new file mode 100644 index 0000000..38ad54b --- /dev/null +++ b/skills/dotnet-benchmark/scripts/check-benchmark-requirements.ps1 @@ -0,0 +1,204 @@ +#requires -Version 5.1 +<# +.SYNOPSIS + Detects the current state of a repository's BenchmarkDotNet harness for the dotnet-benchmark skill. + +.DESCRIPTION + Emits a single JSON object describing what already exists so the skill adds only what is missing. + Read-only: it inspects files and the dotnet CLI but changes nothing. + +.PARAMETER RepoRoot + Repository root to inspect. Defaults to the current directory. + +.PARAMETER BenchmarkType + Optional namespace-qualified or simple benchmark type name used to detect matching reports that would suppress execution. + +.PARAMETER SkipSdkCheck + Skips invoking dotnet --version. Intended for deterministic detector tests; normal skill runs should not use it. + +.EXAMPLE + pwsh -NoProfile -File "/scripts/check-benchmark-requirements.ps1" -RepoRoot "C:\src\myrepo" +#> +[CmdletBinding()] +param( + [string]$RepoRoot = (Get-Location).Path, + [string]$BenchmarkType, + [switch]$SkipSdkCheck +) + +$ErrorActionPreference = 'Stop' + +function Test-CommandExists { + param([string]$Name) + return [bool](Get-Command $Name -ErrorAction SilentlyContinue) +} + +function Get-DotNetSdkVersion { + param([int]$TimeoutMilliseconds = 10000) + + $startInfo = [System.Diagnostics.ProcessStartInfo]::new() + $startInfo.FileName = (Get-Command 'dotnet' -ErrorAction Stop).Source + $startInfo.Arguments = '--version' + $startInfo.UseShellExecute = $false + $startInfo.CreateNoWindow = $true + $startInfo.RedirectStandardOutput = $true + $startInfo.RedirectStandardError = $true + + $process = [System.Diagnostics.Process]::new() + $process.StartInfo = $startInfo + try { + if (-not $process.Start()) { + return [ordered]@{ status = 'start-failed'; version = $null } + } + if (-not $process.WaitForExit($TimeoutMilliseconds)) { + $process.Kill() + return [ordered]@{ status = 'timed-out'; version = $null } + } + if ($process.ExitCode -ne 0) { + return [ordered]@{ status = 'failed'; version = $null } + } + $version = ($process.StandardOutput.ReadToEnd() -split '\r?\n' | Select-Object -First 1).Trim() + return [ordered]@{ status = 'available'; version = $version } + } finally { + $process.Dispose() + } +} + +$RepoRoot = (Resolve-Path -LiteralPath $RepoRoot).Path + +# --- .NET SDK --------------------------------------------------------------- +$sdkAvailable = $false +$sdkVersion = $null +$sdkStatus = if ($SkipSdkCheck) { 'skipped' } else { 'not-found' } +if (-not $SkipSdkCheck -and (Test-CommandExists 'dotnet')) { + try { + $sdkProbe = Get-DotNetSdkVersion + $sdkVersion = $sdkProbe.version + $sdkStatus = $sdkProbe.status + $sdkAvailable = -not [string]::IsNullOrWhiteSpace($sdkVersion) + } catch { + $sdkAvailable = $false + $sdkStatus = 'failed' + } +} + +# --- Solution files --------------------------------------------------------- +$rootFiles = @(Get-ChildItem -LiteralPath $RepoRoot -File -ErrorAction SilentlyContinue) +$slnx = @($rootFiles | Where-Object { $_.Extension -ieq '.slnx' } | Select-Object -ExpandProperty Name -Unique) +$sln = @($rootFiles | Where-Object { $_.Extension -ieq '.sln' } | Select-Object -ExpandProperty Name -Unique) +$solutionFormat = if ($slnx.Count -gt 0) { 'slnx' } elseif ($sln.Count -gt 0) { 'sln' } else { 'none' } +$solutionFiles = @($slnx + $sln) + +# --- Central Package Management -------------------------------------------- +$packagesProps = Join-Path $RepoRoot 'Directory.Packages.props' +$cpm = Test-Path -LiteralPath $packagesProps +$declaredPackages = @() +if ($cpm) { + $packagesText = [System.IO.File]::ReadAllText($packagesProps) + foreach ($id in @('BenchmarkDotNet', 'BenchmarkDotNet.Diagnostics.Windows', 'Codebelt.Extensions.BenchmarkDotNet.Console')) { + if ($packagesText -match [regex]::Escape("Include=`"$id`"")) { $declaredPackages += $id } + } +} + +# --- Root Directory.Build.props conventions -------------------------------- +$buildProps = Join-Path $RepoRoot 'Directory.Build.props' +$centralizesBenchmarkConventions = $false +if (Test-Path -LiteralPath $buildProps) { + $buildText = [System.IO.File]::ReadAllText($buildProps) + $centralizesBenchmarkConventions = ($buildText -match 'IsBenchmarkProject') -and ($buildText -match 'IsToolingProject') +} + +# --- Existing tuning benchmark projects ------------------------------------ +$tuningDir = Join-Path $RepoRoot 'tuning' +$benchmarkProjects = @() +if (Test-Path -LiteralPath $tuningDir) { + $benchmarkProjects = @( + Get-ChildItem -LiteralPath $tuningDir -Recurse -Filter *.Benchmarks.csproj -File -ErrorAction SilentlyContinue | + ForEach-Object { $_.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } + ) +} + +# --- Existing tooling runner host ------------------------------------------ +$toolingDir = Join-Path $RepoRoot 'tooling' +$runner = $null +if (Test-Path -LiteralPath $toolingDir) { + $runnerCsproj = Get-ChildItem -LiteralPath $toolingDir -Recurse -Filter *.csproj -File -ErrorAction SilentlyContinue | + Where-Object { + $text = [System.IO.File]::ReadAllText($_.FullName) + $text -match 'Codebelt\.Extensions\.BenchmarkDotNet\.Console' + } | Select-Object -First 1 + if ($runnerCsproj) { + $programFile = Get-ChildItem -LiteralPath $runnerCsproj.Directory.FullName -Filter Program.cs -File -ErrorAction SilentlyContinue | Select-Object -First 1 + $programText = if ($programFile) { [System.IO.File]::ReadAllText($programFile.FullName) } else { '' } + $configuredRuntimes = @( + [regex]::Matches($programText, '(?:ClrRuntime|CoreRuntime)\.[A-Za-z0-9_]+') | + ForEach-Object { $_.Value } | + Select-Object -Unique + ) + $runner = [ordered]@{ + name = $runnerCsproj.Directory.Name + path = $runnerCsproj.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + programPath = if ($programFile) { $programFile.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } else { $null } + referencesConsole = $true + skipBenchmarksWithReports = $programText -match 'SkipBenchmarksWithReports\s*=\s*true' + usesSlimJob = $programText -match 'BenchmarkWorkspaceOptions\.Slim' + configuredRuntimes = $configuredRuntimes + } + } +} + +# --- reports/ --------------------------------------------------------------- +$reportsPath = Join-Path $RepoRoot 'reports' +$reportsTuningPath = Join-Path $reportsPath 'tuning' +$reportsExists = Test-Path -LiteralPath $reportsPath +$reportFiles = @() +if (Test-Path -LiteralPath $reportsTuningPath) { + $reportFiles = @( + Get-ChildItem -LiteralPath $reportsTuningPath -File -ErrorAction SilentlyContinue | + ForEach-Object { $_.FullName.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' } + ) +} + +$matchingReportFiles = @() +if (-not [string]::IsNullOrWhiteSpace($BenchmarkType)) { + $benchmarkTypeName = ($BenchmarkType -split '\.')[-1] + $matchingReportFiles = @( + $reportFiles | Where-Object { + $filename = [System.IO.Path]::GetFileNameWithoutExtension($_) + $potentialTypeFullName = ($filename -split '-', 2)[0] + $potentialTypeName = ($potentialTypeFullName -split '\.')[-1] + $potentialTypeName.Equals($benchmarkTypeName, [System.StringComparison]::OrdinalIgnoreCase) + } + ) +} + +$wouldSkipRequestedBenchmark = ($runner -ne $null) -and + $runner.skipBenchmarksWithReports -and + (-not [string]::IsNullOrWhiteSpace($BenchmarkType)) -and + ($matchingReportFiles.Count -gt 0) + +$harnessReady = ($runner -ne $null) -and ($benchmarkProjects.Count -gt 0) + +$result = [ordered]@{ + repoRoot = $RepoRoot + sdk = [ordered]@{ available = $sdkAvailable; version = $sdkVersion; status = $sdkStatus } + solutionFormat = $solutionFormat + solutionFiles = $solutionFiles + centralPackageManagement = $cpm + declaredBenchmarkPackages = $declaredPackages + centralizesBenchmarkConventions = $centralizesBenchmarkConventions + benchmarkProjects = $benchmarkProjects + runner = $runner + reportsFolderExists = $reportsExists + reports = [ordered]@{ + path = $reportsPath.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + tuningPath = $reportsTuningPath.Substring($RepoRoot.Length).TrimStart('\', '/') -replace '\\', '/' + files = $reportFiles + requestedBenchmarkType = $BenchmarkType + matchingReportFiles = $matchingReportFiles + wouldSkipRequestedBenchmark = $wouldSkipRequestedBenchmark + } + harnessReady = $harnessReady +} + +$result | ConvertTo-Json -Depth 6 diff --git a/skills/dotnet-benchmark/scripts/validate-skill.ps1 b/skills/dotnet-benchmark/scripts/validate-skill.ps1 new file mode 100644 index 0000000..c53f21f --- /dev/null +++ b/skills/dotnet-benchmark/scripts/validate-skill.ps1 @@ -0,0 +1,259 @@ +#requires -Version 5.1 +<# +.SYNOPSIS + Deterministically validates the dotnet-benchmark skill package and its read-only harness detector. + +.PARAMETER SkillRoot + Root of the dotnet-benchmark skill. Defaults to the parent of this script's directory. +#> +[CmdletBinding()] +param( + [string]$SkillRoot +) + +$ErrorActionPreference = 'Stop' +$failures = [System.Collections.Generic.List[string]]::new() + +if ([string]::IsNullOrWhiteSpace($SkillRoot)) { + $SkillRoot = Split-Path -Parent $PSScriptRoot +} + +function Add-Failure { + param([string]$Message) + $failures.Add($Message) +} + +function Assert-Contains { + param([string]$Name, [string]$Content, [string]$Needle) + if (-not $Content.Contains($Needle)) { + Add-Failure "$Name is missing required content: $Needle" + } +} + +function Assert-File { + param([string]$RelativePath) + $path = Join-Path $SkillRoot $RelativePath + if (-not (Test-Path -LiteralPath $path -PathType Leaf)) { + Add-Failure "Missing required file: $RelativePath" + } +} + +$SkillRoot = (Resolve-Path -LiteralPath $SkillRoot).Path +Write-Verbose "Validating skill root: $SkillRoot" + +$requiredFiles = @( + 'SKILL.md', + 'FORMS.md', + 'assets/benchmark.csproj', + 'assets/benchmark-program.cs', + 'assets/benchmark-runner.csproj', + 'assets/comparison-benchmark.cs', + 'assets/operation-benchmark.cs', + 'references/benchmarkdotnet-essentials.md', + 'references/candidate-selection.md', + 'references/codebelt-conventions.md', + 'references/experiment-design.md', + 'references/onboarding.md', + 'references/runner-preflight.md', + 'scripts/check-benchmark-requirements.ps1', + 'evals/evals.json' +) + +foreach ($file in $requiredFiles) { + Assert-File $file +} +Write-Verbose 'Required file checks completed.' + +if ($failures.Count -eq 0) { + $skillPath = Join-Path $SkillRoot 'SKILL.md' + $skill = [System.IO.File]::ReadAllText($skillPath) + $skillLines = [System.IO.File]::ReadAllLines($skillPath) + $benchmarkEssentials = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/benchmarkdotnet-essentials.md')) + $lineCount = $skillLines.Count + if ($lineCount -gt 500) { + Add-Failure "SKILL.md must stay at or below 500 lines; found $lineCount" + } + + $frontmatterEnd = [Array]::IndexOf($skillLines, '---', 1) + $nameLine = $skillLines | Select-Object -First $frontmatterEnd | Where-Object { $_ -match '^name:\s*' } | Select-Object -First 1 + $descriptionStart = [Array]::IndexOf($skillLines, 'description: >') + if ($skillLines[0] -ne '---' -or $frontmatterEnd -lt 1 -or $nameLine -ne 'name: dotnet-benchmark' -or $descriptionStart -lt 1 -or $descriptionStart -ge $frontmatterEnd) { + Add-Failure 'SKILL.md frontmatter is missing the expected name and folded description' + } else { + $description = [string]::Join(' ', @($skillLines[($descriptionStart + 1)..($frontmatterEnd - 1)] | ForEach-Object { $_.Trim() } | Where-Object { $_ })) + if ($description.Length -gt 1024) { + Add-Failure "SKILL.md description exceeds 1024 characters; found $($description.Length)" + } + } + Write-Verbose 'Frontmatter checks completed.' + + Assert-Contains 'SKILL.md' $skill 'Read `references/candidate-selection.md`' + Assert-Contains 'SKILL.md' $skill 'Read `references/experiment-design.md`' + Assert-Contains 'SKILL.md' $skill 'Do not use construction as the baseline for formatting, equality, hashing, parsing, or another unrelated operation.' + Assert-Contains 'SKILL.md' $skill '--list flat' + Assert-Contains 'SKILL.md' $skill '--job dry' + Assert-Contains 'SKILL.md' $skill 'Never report performance numbers from a build, discovery listing, dry run, or unexecuted benchmark.' + Assert-Contains 'SKILL.md' $skill '#### Yolo mode' + Assert-Contains 'SKILL.md' $skill 'Start a full performance run only after an explicit human instruction to run it now.' + Assert-Contains 'SKILL.md' $skill 'Yolo never authorizes a full performance run.' + Assert-Contains 'SKILL.md' $skill 'read `references/runner-preflight.md`' + Assert-Contains 'SKILL.md' $skill '-BenchmarkType ' + Assert-Contains 'SKILL.md' $skill 'reports.wouldSkipRequestedBenchmark' + Assert-Contains 'SKILL.md' $skill 'complete BenchmarkDotNet summary' + Assert-Contains 'SKILL.md' $skill 'When a parameter is only size or payload' + Assert-Contains 'SKILL.md' $skill 'Semantic preflight' + Assert-Contains 'SKILL.md' $skill 'independently derived exact expected result' + Assert-Contains 'SKILL.md' $skill 'Successful execution is not a correctness oracle.' + Assert-Contains 'SKILL.md' $skill 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' + Assert-Contains 'SKILL.md' $skill 'After the first valid full result' + + $forms = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'FORMS.md')) + Assert-Contains 'FORMS.md' $forms '## Yolo mode override' + Assert-Contains 'FORMS.md' $forms 'skip `candidate_plan_confirmation`' + Assert-Contains 'FORMS.md' $forms 'explicit human instruction to start a full performance run' + + $comparison = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/comparison-benchmark.cs')) + $operation = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/operation-benchmark.cs')) + foreach ($required in @('[MemoryDiagnoser]', '[GlobalSetup]', '[Params(', 'Baseline = true', '{EQUIVALENCE_CHECK}')) { + Assert-Contains 'assets/comparison-benchmark.cs' $comparison $required + } + if (($comparison.Split(@('Baseline = true'), [System.StringSplitOptions]::None).Count - 1) -ne 1) { + Add-Failure 'assets/comparison-benchmark.cs must contain exactly one Baseline = true marker' + } + foreach ($required in @('[MemoryDiagnoser]', '[GlobalSetup]', '[Params(', '{SUT_CALL}')) { + Assert-Contains 'assets/operation-benchmark.cs' $operation $required + } + if ($operation -match '\[Benchmark\([^\]]*Baseline\s*=\s*true') { + Add-Failure 'assets/operation-benchmark.cs must not fabricate a baseline' + } + + $runner = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'assets/benchmark-program.cs')) + Assert-Contains 'assets/benchmark-program.cs' $runner 'return c{RUNTIME_JOBS};' + Assert-Contains 'assets/benchmark-program.cs' $runner '{RUNTIME_USINGS}' + Assert-Contains 'assets/benchmark-program.cs' $runner '{RUNTIME_SETUP}' + Assert-Contains 'assets/benchmark-program.cs' $runner 'public class Program' + + $runnerPreflight = [System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/runner-preflight.md')) + Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'SkipBenchmarksWithReports = true' + Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'reports/tuning/' + Assert-Contains 'references/runner-preflight.md' $runnerPreflight 'Anti-thrashing rule' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Workload invariants' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Benchmark validity gate' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Deferred execution and terminal operations' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) '## Semantic preflight' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) 'Successful execution is not a correctness oracle.' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) 'do the baseline and candidate perform equivalent consumer-visible work over identical logical input?' + Assert-Contains 'references/experiment-design.md' ([System.IO.File]::ReadAllText((Join-Path $SkillRoot 'references/experiment-design.md'))) 'Do not benchmark a homogeneous all-match store and call it type filtering unless the all-match path is the actual subject.' + Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials 'one warmup iteration plus controlled iteration counts' + Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials '## Deferred pipelines and terminal operations' + Assert-Contains 'references/benchmarkdotnet-essentials.md' $benchmarkEssentials '## Result-validity gate' + + try { + $evals = Get-Content -LiteralPath (Join-Path $SkillRoot 'evals/evals.json') -Raw | ConvertFrom-Json + if ($evals.skill_name -ne 'dotnet-benchmark') { + Add-Failure 'evals/evals.json skill_name must be dotnet-benchmark' + } + if ($evals.evals.Count -lt 12) { + Add-Failure 'evals/evals.json must include at least twelve diverse evals' + } + if (-not ($evals.evals | Where-Object { $_.prompt -match '(?i)yolo' })) { + Add-Failure 'evals/evals.json must include a yolo-mode interaction eval' + } + if (-not ($evals.evals | Where-Object { $_.prompt -match 'LegacyAliasQuery' })) { + Add-Failure 'evals/evals.json must include the invalid parameter-matrix and drifting-selectivity eval' + } + if (-not ($evals.evals | Where-Object { $_.prompt -match 'InMemoryTestStore benchmark' })) { + Add-Failure 'evals/evals.json must include the semantic-preflight workload validation eval' + } + if (-not ($evals.evals | Where-Object { $_.prompt -match 'TraitFilter helper only runs in test discovery' })) { + Add-Failure 'evals/evals.json must include the proportionate-stopping eval' + } + $ids = @($evals.evals | ForEach-Object { $_.id }) + if (($ids | Sort-Object -Unique).Count -ne $ids.Count) { + Add-Failure 'evals/evals.json contains duplicate eval IDs' + } + foreach ($eval in $evals.evals) { + if ([string]::IsNullOrWhiteSpace($eval.prompt) -or [string]::IsNullOrWhiteSpace($eval.expected_output) -or $eval.expectations.Count -lt 1) { + Add-Failure "Eval $($eval.id) must include a prompt, expected_output, and expectations" + } + foreach ($fixture in @($eval.files) | Where-Object { -not [string]::IsNullOrWhiteSpace($_) }) { + $fixturePath = Join-Path $SkillRoot $fixture + if (-not (Test-Path -LiteralPath $fixturePath -PathType Leaf)) { + Add-Failure "Eval $($eval.id) references missing fixture: $fixture" + } + } + } + } catch { + Add-Failure "evals/evals.json is invalid: $($_.Exception.Message)" + } + $evalFixtureRoot = Join-Path $SkillRoot 'evals/files' + if (Test-Path -LiteralPath $evalFixtureRoot) { + Get-ChildItem -LiteralPath $evalFixtureRoot -Recurse -Directory -Force | + Where-Object { $_.Name -in @('obj', 'bin', 'BenchmarkDotNet.Artifacts') } | + ForEach-Object { + $relativePath = $_.FullName.Substring($SkillRoot.Length).TrimStart('\') + Add-Failure "Eval fixtures must not contain generated artifact directories: $relativePath" + } + } + Write-Verbose 'Template and eval checks completed.' +} + +$tempRoot = [System.IO.Path]::GetFullPath([System.IO.Path]::GetTempPath()) +$fixtureRoot = Join-Path $tempRoot ("dotnet-benchmark-validator-" + [guid]::NewGuid().ToString('N')) +Write-Verbose "Creating detector fixture: $fixtureRoot" +try { + New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'tuning/Acme.Core.Benchmarks') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'tooling/bdn-runner') -Force | Out-Null + New-Item -ItemType Directory -Path (Join-Path $fixtureRoot 'reports/tuning') -Force | Out-Null + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Acme.sln'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Directory.Packages.props'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'Directory.Build.props'), 'falsefalse') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tuning/Acme.Core.Benchmarks/Acme.Core.Benchmarks.csproj'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/bdn-runner.csproj'), '') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'tooling/bdn-runner/Program.cs'), 'using Codebelt.Extensions.BenchmarkDotNet; using Codebelt.Extensions.BenchmarkDotNet.Console; using BenchmarkDotNet.Environments; public class Program { public static void Main(string[] args) { BenchmarkProgram.Run(args, o => { o.SkipBenchmarksWithReports = true; o.ConfigureBenchmarkDotNet(c => { var slimJob = BenchmarkWorkspaceOptions.Slim; return c.AddJob(slimJob.WithRuntime(CoreRuntime.Core90)); }); }); } }') + [System.IO.File]::WriteAllText((Join-Path $fixtureRoot 'reports/tuning/Acme.Core.WidgetBenchmark-report-github.md'), '# existing report') + + $detectorPath = Join-Path $SkillRoot 'scripts/check-benchmark-requirements.ps1' + if (Test-Path -LiteralPath $detectorPath) { + try { + $pwshExe = Get-Command pwsh -ErrorAction SilentlyContinue + if ($null -eq $pwshExe) { + Add-Failure 'Harness detector validation requires pwsh 7+; do not fall back to legacy Windows PowerShell.' + } else { + $detected = & $pwshExe.Source -NoProfile -File $detectorPath -RepoRoot $fixtureRoot -BenchmarkType Acme.Core.WidgetBenchmark -SkipSdkCheck | ConvertFrom-Json + if ($detected.solutionFormat -ne 'sln' -or -not $detected.centralPackageManagement -or -not $detected.centralizesBenchmarkConventions) { + Add-Failure 'Harness detector did not recognize the fixture solution, CPM, and centralized conventions' + } + if ($detected.sdk.status -ne 'skipped') { + Add-Failure 'Harness detector did not report the intentional skipped SDK probe distinctly' + } + if ($detected.benchmarkProjects.Count -ne 1 -or $detected.runner.name -ne 'bdn-runner' -or -not $detected.harnessReady) { + Add-Failure 'Harness detector did not recognize the existing benchmark project and runner' + } + if (-not $detected.runner.skipBenchmarksWithReports -or -not $detected.runner.usesSlimJob -or @($detected.runner.configuredRuntimes) -notcontains 'CoreRuntime.Core90') { + Add-Failure 'Harness detector did not recognize report skipping, the Slim job, and configured runtime' + } + if (-not $detected.reports.wouldSkipRequestedBenchmark -or @($detected.reports.matchingReportFiles).Count -ne 1) { + Add-Failure 'Harness detector did not identify the matching report that suppresses the requested benchmark type' + } + } + } catch { + Add-Failure "Harness detector failed on the deterministic fixture: $($_.Exception.Message)" + } + } +} finally { + $resolvedFixture = [System.IO.Path]::GetFullPath($fixtureRoot) + if ($resolvedFixture.StartsWith($tempRoot, [System.StringComparison]::OrdinalIgnoreCase) -and (Test-Path -LiteralPath $resolvedFixture)) { + Remove-Item -LiteralPath $resolvedFixture -Recurse -Force + } +} +Write-Verbose 'Harness detector fixture checks completed.' + +if ($failures.Count -gt 0) { + foreach ($failure in $failures) { + Write-Host "[FAIL] $failure" -ForegroundColor Red + } + exit 1 +} + +Write-Host '[PASS] dotnet-benchmark skill structure, templates, eval fixtures, and harness detector are valid.' -ForegroundColor Green diff --git a/skills/dotnet-docfx-digest/SKILL.md b/skills/dotnet-docfx-digest/SKILL.md index 2f8623f..9fa6aad 100644 --- a/skills/dotnet-docfx-digest/SKILL.md +++ b/skills/dotnet-docfx-digest/SKILL.md @@ -18,14 +18,14 @@ Create and maintain developer-friendly DocFX documentation digests for .NET publ - Iterate quickly with the fast path, reading diagnostics as the next work queue: ```bash -dotnet run --file /scripts/docfx.cs -- --repo-root --json +dotnet run --file "/scripts/docfx.cs" -- --repo-root "" --json ``` - Before claiming completion, run `agents.cs`, then a thorough build-backed verification that compiles samples, uses reflection-backed API discovery, and verifies the DocFX build: ```bash -dotnet run --file /scripts/agents.cs -- --repo-root -dotnet run --file /scripts/docfx.cs -- --repo-root --build-api-model --validate-samples --verify-docfx-build +dotnet run --file "/scripts/agents.cs" -- --repo-root "" +dotnet run --file "/scripts/docfx.cs" -- --repo-root "" --build-api-model --validate-samples --verify-docfx-build ``` - `--build-api-model` makes API discovery reflection-precise (the fast source-scan path is conservative and may under-report), `--validate-samples` compiles every C# documentation sample through isolated projects in one temporary `.slnx` graph build with bounded MSBuild parallelism, and `--verify-docfx-build` confirms the DocFX build succeeds. Treat `API_MODEL_SOURCE_SCANNER_LIMITED` in a fast run as a reminder to run the build-backed verification before completion, not as a failure. @@ -34,7 +34,7 @@ dotnet run --file /scripts/docfx.cs -- --repo-root --json --assessment-queue --search-examples +dotnet run --file "docfx.cs" -- --repo-root "" --json --assessment-queue --search-examples ``` Resolve `` outside the target repository working tree (for example, `$env:TEMP\docfx-assessment-queue.md` on Windows or `/tmp/docfx-assessment-queue.md` on Unix). @@ -46,7 +46,7 @@ The assessment work queue includes a "GitHub Example Sources" section with pre-c - For repo-wide or other full authoring runs, establish a bounded write queue before authoring new examples or overwrite rewrites: ```bash -dotnet run --file /scripts/docfx.cs -- --repo-root --build-api-model --project-manifest --json +dotnet run --file "/scripts/docfx.cs" -- --repo-root "" --build-api-model --project-manifest --json ``` Resolve `` outside the target repository working tree (for example, `$env:TEMP\dotnet-docfx-digest-project-manifest.json` on Windows or `/tmp/dotnet-docfx-digest-project-manifest.json` on Unix). When you name this command in a continuation response, replace the placeholder with that concrete temp/session path rather than leaving `` unresolved. diff --git a/skills/dotnet-docfx-digest/references/scripts.md b/skills/dotnet-docfx-digest/references/scripts.md index 4faa64f..ff3ffac 100644 --- a/skills/dotnet-docfx-digest/references/scripts.md +++ b/skills/dotnet-docfx-digest/references/scripts.md @@ -9,15 +9,15 @@ dotnet build