Skip to content

V0.8.0/dotnet benchmark - #31

Merged
gimlichael merged 38 commits into
mainfrom
v0.8.0/dotnet-benchmark
Jul 18, 2026
Merged

V0.8.0/dotnet benchmark#31
gimlichael merged 38 commits into
mainfrom
v0.8.0/dotnet-benchmark

Conversation

@gimlichael

Copy link
Copy Markdown
Member

This pull request introduces the new dotnet-benchmark skill for .NET performance testing, extends the trunk-first-repo skill with a safer push-remote workflow, and strengthens git-keep-a-changelog with mandatory base-commit inspection for concrete releases. It also updates the documentation and installation instructions to reflect these new capabilities and improvements.


New Skill: dotnet-benchmark

  • Introduces the dotnet-benchmark skill, providing a comprehensive BenchmarkDotNet-based benchmarking workflow for .NET types, including project scaffolding, parameter collection, runner infrastructure, onboarding documentation, eval coverage, and README showcase. [1] [2] [3]
  • Adds FORMS.md, benchmark project templates, runner infrastructure, a requirements-checking script, and detailed reference docs for dotnet-benchmark.

Improvements to trunk-first-repo

  • Adds a push-remote workflow mode that allows safe first-time remote pushes by sending main before feature branches, with explicit mode selection and improved guidance in both the README and SKILL.md. [1] [2] [3]
  • README and documentation now clarify the branching logic depending on when origin is configured, and highlight the new safer first-push behavior. [1] [2]

Strengthening git-keep-a-changelog

  • Enforces mandatory base-commit inspection for concrete releases, requiring explicit reporting of foundational version bumps and manifest changes before diff and commit analysis. Step 4 is now divided into clear sub-steps (4a–4f) with improved documentation and eval coverage.
  • README and SKILL.md are updated to emphasize the critical importance of base-commit inclusion and the consequences of omission.

Documentation and Release Management

  • Updates the CHANGELOG.md with detailed entries for versions 0.8.0, 0.7.5, and 0.7.4, and adds compare links for the new releases. [1] [2]
  • README is updated to include installation instructions, feature descriptions, and "Why?" sections for the new and improved skills. [1] [2] [3] [4]

Add the dotnet-benchmark skill to the codebelt agentic repository. Includes SKILL.md with workflow and conventions, FORMS.md for parameter collection, assets with benchmark templates and runner projects, references with BenchmarkDotNet essentials and codebelt conventions, scripts for checking benchmark requirements, and evals/evals.json for testing.
Add dotnet-benchmark to the install commands, available skills table, and Why section. Includes the official skill description and benefits of using the skill for performance testing of .NET types.
Update README.md table entry to reflect the refined push-remote workflow behavior and clearer origin configuration options.
Restructure and expand skill instructions with clearer step-by-step push workflows, conditional summaries for remote configuration states, and more explicit validation for late-remote scenarios. Separate push-remote branching logic from inline feature-push instructions to reduce duplication and improve maintainability.
Align eval test cases with refined skill behavior: configured remotes now appear in post-init summary, late-remote scenarios route through guarded push-remote flow with validation checks, and feature-branch publish simplifies to git push -u origin HEAD when main was already pushed.
@gimlichael gimlichael self-assigned this Jul 17, 2026
@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown

Greptile Summary

This PR introduces the dotnet-benchmark skill for evidence-driven BenchmarkDotNet performance testing, standardizes local PowerShell invocations to pwsh 7+ across all skill documentation and validation tooling, and extends trunk-first-repo with a guarded push-remote workflow. It also retroactively documents versions 0.7.4 and 0.7.5 in the CHANGELOG alongside the new 0.8.0 entry.

  • dotnet-benchmark skill: Full skill package including SKILL.md, FORMS.md, two benchmark asset templates (comparison-benchmark.cs, operation-benchmark.cs), check-benchmark-requirements.ps1 detector, validate-skill.ps1 harness, six reference documents, and 12 evals covering discovery, semantic preflight, yolo mode, runner-skip preflight, and proportionate-stopping scenarios.
  • pwsh standardization: A new Get-LocalShellPolicyFindings scanner was added to validate-skill-templates.ps1 to enforce pwsh in all local command examples; CONTRIBUTING.md, all skill SKILL.md files, and AGENTS.md were updated accordingly.
  • trunk-first-repo push-remote workflow: Step 3a now routes late-remote cases through the guarded Push Remote Workflow (branch check + git ls-tree seed verification + main:main first).

Confidence Score: 5/5

Safe to merge. All previously flagged issues are resolved and the new dotnet-benchmark skill package is self-consistent.

The PR is a clean skill addition and standardization pass. The only finding is a missing [Unreleased] CHANGELOG placeholder, a process convenience concern that does not affect any shipped behavior.

CHANGELOG.md — no [Unreleased] section was added after closing out v0.8.0, so the next contributor will need to add one manually.

Important Files Changed

Filename Overview
skills/dotnet-benchmark/SKILL.md New skill with well-structured 10-step workflow, yolo-mode support, semantic preflight gates, runner-skip preflight, and proportionate-stopping guidance.
skills/dotnet-benchmark/scripts/validate-skill.ps1 Deterministic validator with pwsh 7+ harness-detector fixture test. Previously flagged issues (static class assertion, powershell invocation) are resolved.
scripts/validate-skill-templates.ps1 Adds Get-LocalShellPolicyFindings scanner with two patterns and self-test with 11 fixture cases. CHANGELOG.md and the script itself are correctly excluded from scanning.
CHANGELOG.md Adds entries for v0.8.0, v0.7.5, and v0.7.4. The [Unreleased] section is removed without adding a new placeholder.
skills/trunk-first-repo/SKILL.md Step 3a now correctly routes late-remote cases through the guarded Push Remote Workflow.
CONTRIBUTING.md Updated validator invocation from powershell -NoProfile -ExecutionPolicy Bypass to pwsh -NoProfile with cross-platform path.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A([User request]) --> B{Mode?}
    B -->|push remote| C[Push Remote Workflow]
    B -->|otherwise| D[Initialize Workflow]
    D --> D1[Collect Parameters]
    D1 --> D2[git init + seed commit + feature branch]
    D2 --> D3{Remote URL provided?}
    D3 -->|Yes| D3a[git remote add + push main:main]
    D3 -->|No| D3b[Remind user to add later]
    D3a --> D4a[Step 3a: push HEAD only]
    D3b --> D4b[Step 3a: route to Push Remote Workflow]
    C --> P1[git branch --show-current]
    P1 --> P1a{On main?}
    P1a -->|Yes| STOP1[Stop]
    P1a -->|No| P1b[git ls-tree -r --name-only main]
    P1b --> P1c{main empty?}
    P1c -->|No| STOP2[Stop]
    P1c -->|Yes| P2[check/add origin]
    P2 --> P3[push main:main then HEAD]
    P3 --> DONE([Open PR to main])
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A([User request]) --> B{Mode?}
    B -->|push remote| C[Push Remote Workflow]
    B -->|otherwise| D[Initialize Workflow]
    D --> D1[Collect Parameters]
    D1 --> D2[git init + seed commit + feature branch]
    D2 --> D3{Remote URL provided?}
    D3 -->|Yes| D3a[git remote add + push main:main]
    D3 -->|No| D3b[Remind user to add later]
    D3a --> D4a[Step 3a: push HEAD only]
    D3b --> D4b[Step 3a: route to Push Remote Workflow]
    C --> P1[git branch --show-current]
    P1 --> P1a{On main?}
    P1a -->|Yes| STOP1[Stop]
    P1a -->|No| P1b[git ls-tree -r --name-only main]
    P1b --> P1c{main empty?}
    P1c -->|No| STOP2[Stop]
    P1c -->|Yes| P2[check/add origin]
    P2 --> P3[push main:main then HEAD]
    P3 --> DONE([Open PR to main])
Loading

Reviews (12): Last reviewed commit: "🔧 update .gitignore to include addition..." | Re-trigger Greptile

Comment thread skills/dotnet-benchmark/assets/params-benchmark.cs Outdated
Comment thread skills/dotnet-benchmark/assets/params-benchmark.cs Outdated
Comment thread skills/dotnet-benchmark/assets/benchmark-program.cs
aicia-bot added 12 commits July 17, 2026 13:55
Simplify benchmark-program.cs template substitution using {RUNTIME_USINGS} and {RUNTIME_JOBS} placeholders to cleanly handle the runner-default-only case without stray syntax. Refactor params-benchmark.cs to remove the Variant enum pattern and clarify guidance about comparing implementations. Update evals.json with clearer expectations and add eval case 7 for runner-default-only scenario. Improve references with fixed code examples (slimJob -> BenchmarkWorkspaceOptions.Slim), detailed placeholder documentation, and guidance about keeping the runner configuration clean.
Redesign skill to prioritize candidate selection and measurement fitness over tier-based templates. Rewrite SKILL.md workflow to introduce discovery-phase focus: evidence ladder, inspection sequence, cost signals, and candidate matrix guidance. Restructure FORMS.md around the new workflow parameters. Expand references with candidate-selection.md and experiment-design.md; refactor benchmarkdotnet-essentials.md and codebelt-conventions.md to reflect discovery-first approach.
Remove params-benchmark.cs and simple-benchmark.cs single-tier starters. Introduce operation-benchmark.cs and comparison-benchmark.cs as refined structural examples that support the discovery-focused workflow with clearer guidance on placeholder substitution, baseline configuration, and measurement fitness.
Refactor evals.json test cases to validate candidate selection, evidence gathering, cost-signal analysis, and measurement fitness instead of tier-based template logic. Add evals/files/ fixtures supporting discovery workflow scenarios.
Refactor check-benchmark-requirements.ps1 to validate discovery workflow inputs and candidate matrix evidence. Add validate-skill.ps1 for per-skill validation. Update repo-level validate-skill-templates.ps1 to support new template asset structure.
Update skill listing and onboarding guidance to reflect discovery-focused refactor: evidence-driven candidate selection, cost-signal analysis, and measurement fitness over tier-based templates.
…e runner preflight

Add workflow sections for yolo mode, explicitly requiring user intent for full performance runs. Add report-aware runner preflight logic to detect when SkipBenchmarksWithReports plus existing reports intentionally filter a benchmark type. Update descriptions and validation checkpoints to support these features safely.
…nostics

Add eval test cases 8 and 9 with fixtures. Test 8 validates yolo mode with autonomous candidate selection and progress-update-only planning. Test 9 validates report-aware preflight: skill recognizes SkipBenchmarksWithReports plus matching reports/tuning/ artifacts and preserves benchmark code unchanged.
…are preflight

Add SkipBenchmarksWithReports and reports/tuning/ detection to check-benchmark-requirements.ps1. Report runner program, slim/runtime jobs, and wouldSkipRequestedBenchmark status. Add [MemoryDiagnoser] to benchmark-program.cs template to align with skill documentation. Update validate-skill.ps1 to support the expanded detector output.
Enhance validate-skill-templates.ps1 with additional style and structure checks for SKILL.md, FORMS.md, references/, and evals/ files. Ensures consistency across repo-managed skills and enforces conventions for new feature additions.
…ight

Update dotnet-benchmark skill table entry with descriptions of new yolo mode capability and report-aware runner preflight. Add feature bullets about yolo mode without permission creep and report-aware runner preflight for intentional benchmark skips.
Comment thread skills/dotnet-benchmark/scripts/validate-skill.ps1
…non-static

Update benchmark-runner Program.cs fixture, runner-preflight.md reference, and validate-skill.ps1 assertions to use 'public class Program' instead of 'public static class Program'. Console app entry points should be non-static to align with .NET templates and best practices.
…tionate-stop & selectivity-drift eval tests

Add tests 10 and 11 with fixtures. Test 10 validates benchmark design repair when selectivity drifts across parameters and some cases fail. Test 11 validates proportionate stopping: when measurement is complete and cost does not justify deeper investigation. Update SKILL.md with case studies and decision logic. Refine benchmarkdotnet-essentials.md and experiment-design.md references for rigor and practical guidance.
…updates

Update validate-skill.ps1 to discover and validate proportionate-stop and selectivity-drift test fixtures. Add assertions for new eval test cases 10 and 11. Ensure fixture structure and referenced files align with updated skill expectations.
Enhance validate-skill-templates.ps1 with checks for eval test depth, fixture distribution, and coverage-gap detection. Ensures repo-managed skills maintain sufficient test coverage and fixture variety. Add helper functions for tracking eval statistics.
…ortionate-stop guidance

Update dotnet-benchmark skill description with new test coverage for selectivity drift, proportionate stopping, and benchmark-design rigor. Add feature bullets about case-study validation and early-stop conditions.
Remove stale obj/ directories and generated project files from benchmark-runner and Acme.Core.Benchmarks eval fixtures. These artifacts should not be committed to source control.
…stStore test case

Add eval test case 12 to validate semantic preflight before performance interpretation. Test case covers correctness oracle derivation, workload-semantics gaps (selectivity drift), baseline validation, and fixture preparation. Update experiment-design.md reference with concrete guidance on deterministic input, exact count verification, and semantic preflight requirements. Refine SKILL.md to emphasize semantic preflight as a mandatory gate before accepting full-run results.
Update skills/dotnet-benchmark/scripts/validate-skill.ps1 to discover and validate the new inmemory-test-store eval fixture. Add fixture structure assertions and fixture-file validators. Enhance scripts/validate-skill-templates.ps1 with checks for eval test semantics and correctness oracle validation.
…ss gates

Update dotnet-benchmark skill table entry to emphasize semantic preflight as a mandatory validation gate and InMemoryTestStore correctness oracle test case. Add feature bullets about correctness oracle derivation, deterministic workload definition, and false-positive baseline prevention.
Comment thread skills/dotnet-benchmark/scripts/validate-skill.ps1 Outdated
Update example commands to use pwsh (PowerShell 7+) and add conditional guidance for Windows environments where PowerShell 7+ is unavailable, falling back to powershell. This improves clarity for users following the skill instructions.
Comment thread skills/dotnet-benchmark/references/runner-preflight.md Outdated
Add mandatory guidance that local PowerShell invocations must use pwsh 7+ runtime. Agents must report missing pwsh as a blocker instead of falling back to legacy Windows PowerShell. Legacy fallback silently uses the wrong runtime; standardizing on pwsh improves both clarity and compatibility across platforms. Validation script now enforces this policy across repo content.
Update all local command examples and contributor guidance to use pwsh 7+ consistently. Clarify that Bash and pwsh are both supported for local development, but every local PowerShell command and .ps1 script must use pwsh, never the legacy executable. GitHub Actions workflows may continue using shell choices appropriate for their context.
Record the standardization of local PowerShell execution to pwsh 7+ as a change in the unreleased section. This clarifies the shift to pwsh as a requirement while preserving Bash and workflow-specific shell choices.
Propagate pwsh 7+ guidance across all skill documentation, evals, and references. Update command examples in SKILL.md files to use pwsh-specific invocations and remove legacy fallback language. Update evals to reflect current guidance.
Remove overly aggressive validation rules for pwsh standardization. Revert code fence labels from ps1 to powershell for better readability. Simplify validator patterns and remove pwsh-specific eval expectations that over-constrained the agent workflow.
Remove overly prescriptive shell language from AGENTS.md, CONTRIBUTING.md, and README.md. Update skill descriptions to soften pwsh 7+ terminology, allowing flexibility while maintaining clear guidance to use pwsh (not powershell.exe) when PowerShell syntax is needed locally. Sync skill documentation to reflect the more permissive posture.
Align validator assertions in validate-skill-templates.ps1 with simplified documentation text. Quote file path arguments in pwsh command examples across skill documentation to prevent issues with paths containing spaces. Remove README.md checks from validator as guidance no longer prescribes specific shell choices there.
@gimlichael
gimlichael merged commit b5c535a into main Jul 18, 2026
1 check passed
@gimlichael
gimlichael deleted the v0.8.0/dotnet-benchmark branch July 18, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants