V0.8.0/dotnet benchmark - #31
Conversation
Add the dotnet-benchmark skill to the codebelt agentic repository. Includes SKILL.md with workflow and conventions, FORMS.md for parameter collection, assets with benchmark templates and runner projects, references with BenchmarkDotNet essentials and codebelt conventions, scripts for checking benchmark requirements, and evals/evals.json for testing.
Add dotnet-benchmark to the install commands, available skills table, and Why section. Includes the official skill description and benefits of using the skill for performance testing of .NET types.
Update README.md table entry to reflect the refined push-remote workflow behavior and clearer origin configuration options.
Restructure and expand skill instructions with clearer step-by-step push workflows, conditional summaries for remote configuration states, and more explicit validation for late-remote scenarios. Separate push-remote branching logic from inline feature-push instructions to reduce duplication and improve maintainability.
Align eval test cases with refined skill behavior: configured remotes now appear in post-init summary, late-remote scenarios route through guarded push-remote flow with validation checks, and feature-branch publish simplifies to git push -u origin HEAD when main was already pushed.
Greptile SummaryThis PR introduces the
Confidence Score: 5/5Safe to merge. All previously flagged issues are resolved and the new dotnet-benchmark skill package is self-consistent. The PR is a clean skill addition and standardization pass. The only finding is a missing [Unreleased] CHANGELOG placeholder, a process convenience concern that does not affect any shipped behavior. CHANGELOG.md — no [Unreleased] section was added after closing out v0.8.0, so the next contributor will need to add one manually. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A([User request]) --> B{Mode?}
B -->|push remote| C[Push Remote Workflow]
B -->|otherwise| D[Initialize Workflow]
D --> D1[Collect Parameters]
D1 --> D2[git init + seed commit + feature branch]
D2 --> D3{Remote URL provided?}
D3 -->|Yes| D3a[git remote add + push main:main]
D3 -->|No| D3b[Remind user to add later]
D3a --> D4a[Step 3a: push HEAD only]
D3b --> D4b[Step 3a: route to Push Remote Workflow]
C --> P1[git branch --show-current]
P1 --> P1a{On main?}
P1a -->|Yes| STOP1[Stop]
P1a -->|No| P1b[git ls-tree -r --name-only main]
P1b --> P1c{main empty?}
P1c -->|No| STOP2[Stop]
P1c -->|Yes| P2[check/add origin]
P2 --> P3[push main:main then HEAD]
P3 --> DONE([Open PR to main])
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A([User request]) --> B{Mode?}
B -->|push remote| C[Push Remote Workflow]
B -->|otherwise| D[Initialize Workflow]
D --> D1[Collect Parameters]
D1 --> D2[git init + seed commit + feature branch]
D2 --> D3{Remote URL provided?}
D3 -->|Yes| D3a[git remote add + push main:main]
D3 -->|No| D3b[Remind user to add later]
D3a --> D4a[Step 3a: push HEAD only]
D3b --> D4b[Step 3a: route to Push Remote Workflow]
C --> P1[git branch --show-current]
P1 --> P1a{On main?}
P1a -->|Yes| STOP1[Stop]
P1a -->|No| P1b[git ls-tree -r --name-only main]
P1b --> P1c{main empty?}
P1c -->|No| STOP2[Stop]
P1c -->|Yes| P2[check/add origin]
P2 --> P3[push main:main then HEAD]
P3 --> DONE([Open PR to main])
Reviews (12): Last reviewed commit: "🔧 update .gitignore to include addition..." | Re-trigger Greptile |
Simplify benchmark-program.cs template substitution using {RUNTIME_USINGS} and {RUNTIME_JOBS} placeholders to cleanly handle the runner-default-only case without stray syntax. Refactor params-benchmark.cs to remove the Variant enum pattern and clarify guidance about comparing implementations. Update evals.json with clearer expectations and add eval case 7 for runner-default-only scenario. Improve references with fixed code examples (slimJob -> BenchmarkWorkspaceOptions.Slim), detailed placeholder documentation, and guidance about keeping the runner configuration clean.
Redesign skill to prioritize candidate selection and measurement fitness over tier-based templates. Rewrite SKILL.md workflow to introduce discovery-phase focus: evidence ladder, inspection sequence, cost signals, and candidate matrix guidance. Restructure FORMS.md around the new workflow parameters. Expand references with candidate-selection.md and experiment-design.md; refactor benchmarkdotnet-essentials.md and codebelt-conventions.md to reflect discovery-first approach.
Remove params-benchmark.cs and simple-benchmark.cs single-tier starters. Introduce operation-benchmark.cs and comparison-benchmark.cs as refined structural examples that support the discovery-focused workflow with clearer guidance on placeholder substitution, baseline configuration, and measurement fitness.
Refactor evals.json test cases to validate candidate selection, evidence gathering, cost-signal analysis, and measurement fitness instead of tier-based template logic. Add evals/files/ fixtures supporting discovery workflow scenarios.
Refactor check-benchmark-requirements.ps1 to validate discovery workflow inputs and candidate matrix evidence. Add validate-skill.ps1 for per-skill validation. Update repo-level validate-skill-templates.ps1 to support new template asset structure.
Update skill listing and onboarding guidance to reflect discovery-focused refactor: evidence-driven candidate selection, cost-signal analysis, and measurement fitness over tier-based templates.
…e runner preflight Add workflow sections for yolo mode, explicitly requiring user intent for full performance runs. Add report-aware runner preflight logic to detect when SkipBenchmarksWithReports plus existing reports intentionally filter a benchmark type. Update descriptions and validation checkpoints to support these features safely.
…nostics Add eval test cases 8 and 9 with fixtures. Test 8 validates yolo mode with autonomous candidate selection and progress-update-only planning. Test 9 validates report-aware preflight: skill recognizes SkipBenchmarksWithReports plus matching reports/tuning/ artifacts and preserves benchmark code unchanged.
…are preflight Add SkipBenchmarksWithReports and reports/tuning/ detection to check-benchmark-requirements.ps1. Report runner program, slim/runtime jobs, and wouldSkipRequestedBenchmark status. Add [MemoryDiagnoser] to benchmark-program.cs template to align with skill documentation. Update validate-skill.ps1 to support the expanded detector output.
Enhance validate-skill-templates.ps1 with additional style and structure checks for SKILL.md, FORMS.md, references/, and evals/ files. Ensures consistency across repo-managed skills and enforces conventions for new feature additions.
…ight Update dotnet-benchmark skill table entry with descriptions of new yolo mode capability and report-aware runner preflight. Add feature bullets about yolo mode without permission creep and report-aware runner preflight for intentional benchmark skips.
…non-static Update benchmark-runner Program.cs fixture, runner-preflight.md reference, and validate-skill.ps1 assertions to use 'public class Program' instead of 'public static class Program'. Console app entry points should be non-static to align with .NET templates and best practices.
…tionate-stop & selectivity-drift eval tests Add tests 10 and 11 with fixtures. Test 10 validates benchmark design repair when selectivity drifts across parameters and some cases fail. Test 11 validates proportionate stopping: when measurement is complete and cost does not justify deeper investigation. Update SKILL.md with case studies and decision logic. Refine benchmarkdotnet-essentials.md and experiment-design.md references for rigor and practical guidance.
…updates Update validate-skill.ps1 to discover and validate proportionate-stop and selectivity-drift test fixtures. Add assertions for new eval test cases 10 and 11. Ensure fixture structure and referenced files align with updated skill expectations.
Enhance validate-skill-templates.ps1 with checks for eval test depth, fixture distribution, and coverage-gap detection. Ensures repo-managed skills maintain sufficient test coverage and fixture variety. Add helper functions for tracking eval statistics.
…ortionate-stop guidance Update dotnet-benchmark skill description with new test coverage for selectivity drift, proportionate stopping, and benchmark-design rigor. Add feature bullets about case-study validation and early-stop conditions.
Remove stale obj/ directories and generated project files from benchmark-runner and Acme.Core.Benchmarks eval fixtures. These artifacts should not be committed to source control.
…stStore test case Add eval test case 12 to validate semantic preflight before performance interpretation. Test case covers correctness oracle derivation, workload-semantics gaps (selectivity drift), baseline validation, and fixture preparation. Update experiment-design.md reference with concrete guidance on deterministic input, exact count verification, and semantic preflight requirements. Refine SKILL.md to emphasize semantic preflight as a mandatory gate before accepting full-run results.
Update skills/dotnet-benchmark/scripts/validate-skill.ps1 to discover and validate the new inmemory-test-store eval fixture. Add fixture structure assertions and fixture-file validators. Enhance scripts/validate-skill-templates.ps1 with checks for eval test semantics and correctness oracle validation.
…ss gates Update dotnet-benchmark skill table entry to emphasize semantic preflight as a mandatory validation gate and InMemoryTestStore correctness oracle test case. Add feature bullets about correctness oracle derivation, deterministic workload definition, and false-positive baseline prevention.
Update example commands to use pwsh (PowerShell 7+) and add conditional guidance for Windows environments where PowerShell 7+ is unavailable, falling back to powershell. This improves clarity for users following the skill instructions.
Add mandatory guidance that local PowerShell invocations must use pwsh 7+ runtime. Agents must report missing pwsh as a blocker instead of falling back to legacy Windows PowerShell. Legacy fallback silently uses the wrong runtime; standardizing on pwsh improves both clarity and compatibility across platforms. Validation script now enforces this policy across repo content.
Update all local command examples and contributor guidance to use pwsh 7+ consistently. Clarify that Bash and pwsh are both supported for local development, but every local PowerShell command and .ps1 script must use pwsh, never the legacy executable. GitHub Actions workflows may continue using shell choices appropriate for their context.
Record the standardization of local PowerShell execution to pwsh 7+ as a change in the unreleased section. This clarifies the shift to pwsh as a requirement while preserving Bash and workflow-specific shell choices.
Propagate pwsh 7+ guidance across all skill documentation, evals, and references. Update command examples in SKILL.md files to use pwsh-specific invocations and remove legacy fallback language. Update evals to reflect current guidance.
Remove overly aggressive validation rules for pwsh standardization. Revert code fence labels from ps1 to powershell for better readability. Simplify validator patterns and remove pwsh-specific eval expectations that over-constrained the agent workflow.
Remove overly prescriptive shell language from AGENTS.md, CONTRIBUTING.md, and README.md. Update skill descriptions to soften pwsh 7+ terminology, allowing flexibility while maintaining clear guidance to use pwsh (not powershell.exe) when PowerShell syntax is needed locally. Sync skill documentation to reflect the more permissive posture.
Align validator assertions in validate-skill-templates.ps1 with simplified documentation text. Quote file path arguments in pwsh command examples across skill documentation to prevent issues with paths containing spaces. Remove README.md checks from validator as guidance no longer prescribes specific shell choices there.
This pull request introduces the new
dotnet-benchmarkskill for .NET performance testing, extends thetrunk-first-reposkill with a safer push-remote workflow, and strengthensgit-keep-a-changelogwith mandatory base-commit inspection for concrete releases. It also updates the documentation and installation instructions to reflect these new capabilities and improvements.New Skill:
dotnet-benchmarkdotnet-benchmarkskill, providing a comprehensive BenchmarkDotNet-based benchmarking workflow for .NET types, including project scaffolding, parameter collection, runner infrastructure, onboarding documentation, eval coverage, and README showcase. [1] [2] [3]FORMS.md, benchmark project templates, runner infrastructure, a requirements-checking script, and detailed reference docs fordotnet-benchmark.Improvements to
trunk-first-repomainbefore feature branches, with explicit mode selection and improved guidance in both the README and SKILL.md. [1] [2] [3]originis configured, and highlight the new safer first-push behavior. [1] [2]Strengthening
git-keep-a-changelogDocumentation and Release Management
CHANGELOG.mdwith detailed entries for versions 0.8.0, 0.7.5, and 0.7.4, and adds compare links for the new releases. [1] [2]