Skip to content

Add public API governance instruction evaluation - #68698

Draft
PureWeen wants to merge 1 commit into
pureween-public-api-instructionsfrom
pureween-public-api-instruction-eval
Draft

Add public API governance instruction evaluation#68698
PureWeen wants to merge 1 commit into
pureween-public-api-instructionsfrom
pureween-public-api-instruction-eval

Conversation

@PureWeen

@PureWeen PureWeen commented Aug 21, 2026

Copy link
Copy Markdown
Member

Summary

  • adds a specialized Vally regression suite for the repository-wide public API governance instructions in Add public API governance instructions #68168
  • covers an unapproved new public member, shipped interface signature change, and shipped default/convention change
  • loads .github/copilot-instructions.md into Vally's isolated execution environment and checks the complete API-review gate

This is the top layer over #68168 and targets pureween-public-api-instructions. It follows the evaluation infrastructure merged in #68628 and adds no Vally onboarding infrastructure.

Validation

Validated in a synthesized checkout containing current main's #68628 infrastructure, #68168's instruction changes, and this stack layer:

  • pwsh -NoProfile -File ./eng/skill-evals/run.ps1 Test
  • pwsh -NoProfile -File ./eng/skill-evals/run.ps1 Validate -Eval eng/skill-evals/copilot-instructions/public-api-governance.vally.yaml
  • one current-behavior run with --runs 1 --workers 3: 100%, all graders passed for all three stimuli
  • git diff --check

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PureWeen
PureWeen changed the base branch from main to pureween-public-api-instructions August 21, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants