Skip to content

Support honest metadata-only release evidence in the high-risk evaluator #485

Description

@leehack

Observed behavior

Release-prep PR #482 at 16eaa80fcdb7578599d8757056cfb18536a642cb against main ccc2d35633e3ca6347a697f19fde4944e0a8dbd0 changes eight version/docs/lock files only. The classifier correctly flags pubspec.yaml as high-risk artifactConsumer, but the local evaluator unconditionally requires a changed production test. An empty affected-test inventory returns missingTestPath; citing the existing release-doc regression test returns unchangedEvidencePath.

This makes honest metadata-only release evidence impossible without unrelated test churn, even when canonical full VM/Chrome, strict release-version verification and independent exact-head review pass. This is a pre-existing policy tooling limitation exposed by release preparation, not a runtime regression introduced by #482 or the Apple guard #484.

Reproduction

Run the current classify_high_risk_changes.dart and high_risk_readiness.dart against that exact base/head with an independent accepted audit, zero unresolved threads and zero known PR-caused P1 regressions. Use the actual Git-derived inventory; do not claim unchanged tests are changed.

Acceptance

  • Define a narrow explicit metadata-only release evidence route, based on actual Git diff and release-policy validation, without weakening production artifact-consumer changes.
  • Require meaningful existing release-verifier tests and independent exact-head review while allowing no newly changed production test when production behavior is unchanged.
  • Keep negative tests proving runtime/hook/config/consumer changes cannot exploit the metadata-only route.
  • Preserve the separate unverified external App/auditor/environment/ruleset boundary; this issue does not authorize settings or required-check publication changes.
  • Document that no caller-supplied changed inventory, fake test churn, self-approval or advisory-green shortcut establishes readiness.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:P2Planned next: useful unblocked work or validation after P1 items

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions