diff --git a/AGENTS.md b/AGENTS.md index 34a24c1d..8491e942 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -334,6 +334,10 @@ require issue-specific positive and negative tests that fail if the relevant branch is deleted, bypassed, or miswired. Record zero known PR-caused P1 regressions and zero unresolved review threads in the PR's high-risk block and evidence payload. +For core-patch release metadata only, use the bounded +`release-metadata-verification` evidence route described in +`doc/high_risk_pre_merge_readiness.md`. It checks exact Git blobs and existing +release tests; do not fabricate changed tests or reclassify the PR as standard. Structured-output changes must cover compiled grammar acceptance and rejection, schema-directed scalar and container reconstruction, partial-streaming diff --git a/doc/high_risk_pre_merge_readiness.md b/doc/high_risk_pre_merge_readiness.md index 03385372..22979472 100644 --- a/doc/high_risk_pre_merge_readiness.md +++ b/doc/high_risk_pre_merge_readiness.md @@ -43,7 +43,7 @@ The contract fails closed against: 3. duplicate JSON keys, unknown fields, wrong scalar/container types, and caller-declared decisions; 4. caller-forged changed-file lists; -5. deleted, renamed-old, unchanged, absolute, traversal, wildcard, non-test, or +5. deleted, renamed-old, unauthorized unchanged, absolute, traversal, wildcard, non-test, or phantom evidence paths; 6. boolean-only structured-output attestations without named production tests; 7. PR-authored workflow or evidence execution; @@ -93,7 +93,55 @@ source and destination paths are both classified. Every cited test must: This intentionally rejects unchanged tests as issue-specific proof. Existing coverage can still inform a human audit, but it cannot satisfy the changed -production-test evidence field. +production-test evidence field except through the bounded release route below. + +### Core-patch release metadata evidence + +A release-only PR remains **high-risk / artifactConsumer**. It can additionally +declare the `release-metadata-verification` matrix row, with this exact command +and a `pass` result (never `notApplicable`): + +```bash +dart run tool/testing/verify_release_docs_versions.dart --release-prep && dart test -p vm -j 1 test/unit/tooling/verify_release_docs_companion_pins_test.dart +``` + +For this route, `affected_test_paths` must name exactly that existing test. +The independent audit and both matrix rows remain required and are bound to +the exact head and base. This records an existing release regression suite, +not a fabricated newly changed test. The evaluator does not execute candidate +scripts and cannot authenticate a claimed test run; the independent reviewer +must inspect the exact-head command/log evidence. + +Eligibility comes from literal Git blobs and the complete rename-aware diff, +not a `metadata_only` flag or a caller-provided inventory: + +- `pubspec.yaml` changes only its canonical stable version to the next patch; + every other byte, including dependencies, SDK, hooks and overrides, is fixed. +- Both current changelogs and the four maintained installation README/docs + pages are modified. Historical numbered changelog sections, runtime identity + identities, frontmatter and already-prepared companion constraints remain + unchanged; current core snippets name the new patch. Mutable release prose + must be inactive Markdown with no MDX expressions/imports/exports or HTML + outside inert code examples. Executable `mdx-code-block` fences are rejected. + Historical changelog tails are excluded from that syntax check only after + proving byte equality. Narrative prose still needs independent review. +- The generated `example/chat_app/pubspec.lock` must change only the + matching local `llamadart` version. Path, source, inventory, hashes, SDK and + all other bytes remain identical. Pub must generate the lock normally. +- All changed paths are existing non-executable regular files; additions, + deletions, copies, renames and mode changes are excluded. +- The existing release verifier and named regression test are unchanged + regular blobs at both revisions. + +The exact allowlist lives in `tool/testing/release_metadata_readiness.dart`. +Companion version changes, runtime pins, generated bindings, hooks, SwiftPM, +workflows, security/review policy, other docs/MDX and production changes cannot +use this exception. Broader release changes use the ordinary changed-test +contract instead of expanding this route implicitly. + +An internally consistent release evaluation still returns +`unverifiedPrerequisites` (exit 2), never operational readiness. External +authentication/publication/ruleset boundaries below are unchanged. ### Independent audit diff --git a/test/unit/tooling/classify_high_risk_changes_test.dart b/test/unit/tooling/classify_high_risk_changes_test.dart index fe670eae..dfb2bbe6 100644 --- a/test/unit/tooling/classify_high_risk_changes_test.dart +++ b/test/unit/tooling/classify_high_risk_changes_test.dart @@ -77,6 +77,7 @@ void main() { 'doc/pr_branch_writer_inventory.md', 'doc/high_risk_pre_merge_readiness.md', 'tool/testing/high_risk_readiness.dart', + 'tool/testing/release_metadata_readiness.dart', 'tool/testing/high_risk_readiness_evidence.schema.json', 'test/unit/tooling/high_risk_readiness_test.dart', ]); @@ -102,6 +103,15 @@ void main() { ); }); + test('release metadata validator alone remains regression-policy risk', () { + expect( + assessHighRiskFiles([ + 'tool/testing/release_metadata_readiness.dart', + ]).surfaces, + {HighRiskSurface.regressionPolicy}, + ); + }); + test('normalizes blank input and prints deterministic surfaces', () { final assessment = assessHighRiskFiles([ '', diff --git a/test/unit/tooling/high_risk_readiness_test.dart b/test/unit/tooling/high_risk_readiness_test.dart index 8f770cda..489df22f 100644 --- a/test/unit/tooling/high_risk_readiness_test.dart +++ b/test/unit/tooling/high_risk_readiness_test.dart @@ -8,6 +8,7 @@ import 'package:test/test.dart'; import '../../../tool/testing/classify_high_risk_changes.dart'; import '../../../tool/testing/high_risk_readiness.dart'; +import '../../../tool/testing/release_metadata_readiness.dart'; const baseSha = '1111111111111111111111111111111111111111'; const headSha = '2222222222222222222222222222222222222222'; @@ -30,6 +31,7 @@ class FakeRepositoryState implements RepositoryStateReader { this.commitsExist = true, this.baseIsAncestor = true, this.pathProbeThrows = false, + this.files = const {}, }) : existingPaths = existingPaths ?? changes @@ -42,8 +44,16 @@ class FakeRepositoryState implements RepositoryStateReader { final bool commitsExist; final bool baseIsAncestor; final bool pathProbeThrows; + final Map> files; var repositoryCallCount = 0; + @override + Future fileAt( + String sha, + String path, { + String? workingDirectory, + }) async => files[sha]?[path]; + @override Future commitExists(String sha, {String? workingDirectory}) async { repositoryCallCount++; @@ -160,6 +170,69 @@ Map backendEvidence() { return evidence; } +Map metadataEvidence() { + final evidence = backendEvidence(); + evidence['surfaces'] = ['artifactConsumer']; + (evidence['required_matrix_row_ids'] as List).add(releaseMetadataRow); + (evidence['matrix_row_evidence'] as Map)[releaseMetadataRow] = { + 'row_id': releaseMetadataRow, + 'result': 'pass', + 'command': releaseMetadataCommand, + 'evidence_notes': + 'Strict verifier and existing companion-pin tests passed on exact head.', + }; + evidence['affected_test_paths'] = [releaseMetadataTest]; + return evidence; +} + +Map> metadataFiles() { + ReadinessFile file(String text) => (mode: '100644', contents: text); + final base = { + 'pubspec.yaml': file( + 'name: llamadart\nversion: 0.8.22\ndependencies:\n yaml: ^3.1.3\n', + ), + for (final path in releaseMetadataDocs) + path: file( + path.contains('CHANGELOG') || path.contains('recent-releases') + ? '# Changes\n\n## Unreleased\n\n* Update.\n\n## 0.8.22\n\n* History.\n' + : '# Install\n\n```yaml\ndependencies:\n llamadart: ^0.8.22\n```\n', + ), + releaseMetadataLock: file( + 'packages:\n llamadart:\n dependency: "direct main"\n description:\n path: "../.."\n relative: true\n source: path\n version: "0.8.22"\nsdks:\n dart: ">=3.13.0 <4.0.0"\n', + ), + releaseMetadataVerifier: file('// maintained verifier\n'), + releaseMetadataTest: file( + '// existing negative companion pin regressions\n', + ), + }; + final head = Map.from(base); + for (final path in releaseMetadataPaths) { + final old = base[path]!; + head[path] = file( + path.contains('CHANGELOG') || path.contains('recent-releases') + ? old.contents.replaceFirst('## Unreleased', '## 0.8.23') + : old.contents.replaceAll('0.8.22', '0.8.23'), + ); + } + return {baseSha: base, headSha: head}; +} + +Future evaluateMetadata({ + Map? evidence, + Map>? files, + List? changes, +}) => HighRiskReadinessEvaluator( + repositoryState: FakeRepositoryState( + changes: + changes ?? + [ + for (final path in releaseMetadataPaths) + RepositoryChange(path: path, kind: RepositoryChangeKind.modified), + ], + files: files ?? metadataFiles(), + ), +).evaluate(evidence: evidence ?? metadataEvidence(), context: context); + Map standardEvidence() { final evidence = backendEvidence(); evidence['classification'] = 'standard'; @@ -262,6 +335,475 @@ void mutateIdentityToOtherValidValues(Map evidence) { } void main() { + group('bounded metadata-only release evidence', () { + test( + 'new release prefix cannot duplicate or invent historical sections', + () async { + for (final heading in ['## 0.8.22', '## 9.0.0', '## Migration']) { + final files = metadataFiles(); + final old = files[headSha]!['CHANGELOG.md']!; + files[headSha]!['CHANGELOG.md'] = ( + mode: old.mode, + contents: old.contents.replaceFirst( + '* Update.', + '* Update.\n\n$heading\n\n* Duplicate.', + ), + ); + expectFailure( + await evaluateMetadata(files: files), + ReadinessFailureClassification.invalidReleaseMetadata, + ); + } + }, + ); + test( + 'existing active JSX template attributes and script bodies cannot use exception', + () async { + for (final pair in [ + ( + before: '
', + after: r'
', + ), + ( + before: '', + after: '', + ), + ( + before: '', + after: '', + ), + ]) { + final files = metadataFiles(); + final base = files[baseSha]!['README.md']!; + final head = files[headSha]!['README.md']!; + files[baseSha]!['README.md'] = ( + mode: base.mode, + contents: '${base.contents}\n${pair.before}\n', + ); + files[headSha]!['README.md'] = ( + mode: head.mode, + contents: '${head.contents}\n${pair.after}\n', + ); + expectFailure( + await evaluateMetadata(files: files), + ReadinessFailureClassification.invalidReleaseMetadata, + ); + } + }, + ); + test( + 'omitting a required lock or current document cannot hide stale metadata', + () async { + for (final omitted in releaseMetadataPaths) { + final changes = [ + for (final path in releaseMetadataPaths) + if (path != omitted) + RepositoryChange( + path: path, + kind: RepositoryChangeKind.modified, + ), + ]; + expectFailure( + await evaluateMetadata(changes: changes), + ReadinessFailureClassification.invalidReleaseMetadata, + ); + } + }, + ); + test( + 'real Git blobs authorize only the committed metadata candidate', + () async { + final repo = Directory.systemTemp.createTempSync( + 'release-metadata-git-', + ); + addTearDown(() => repo.deleteSync(recursive: true)); + String git(List args) { + final result = Process.runSync( + 'git', + args, + workingDirectory: repo.path, + ); + expect(result.exitCode, 0, reason: '${result.stderr}'); + return (result.stdout as String).trim(); + } + + git(['init', '--quiet']); + git(['config', 'user.name', 'Metadata test']); + git(['config', 'user.email', 'metadata@example.invalid']); + git(['config', 'core.autocrlf', 'false']); + final files = metadataFiles(); + void write(Map tree) { + for (final entry in tree.entries) { + File('${repo.path}/${entry.key}') + ..parent.createSync(recursive: true) + ..writeAsStringSync(entry.value.contents); + } + } + + write(files[baseSha]!); + git(['add', '.']); + git(['commit', '--quiet', '-m', 'base']); + final base = git(['rev-parse', 'HEAD']); + write(files[headSha]!); + git(['add', '.']); + git(['commit', '--quiet', '-m', 'release']); + final head = git(['rev-parse', 'HEAD']); + final evidence = metadataEvidence() + ..['expected_pr_head_sha'] = head + ..['current_base_sha'] = base; + evidence['independent_audit']['audit_head_sha'] = head; + evidence['independent_audit']['audit_base_sha'] = base; + Future evaluate( + String candidate, + Map input, + ) => const HighRiskReadinessEvaluator().evaluate( + evidence: input, + context: PullRequestContext( + repository: context.repository, + prNumber: context.prNumber, + headSha: candidate, + baseSha: base, + author: context.author, + ), + workingDirectory: repo.path, + ); + // Mutable working-tree content cannot substitute for committed blobs. + File('${repo.path}/README.md').writeAsStringSync('{unsafe()}'); + expect( + (await evaluate(head, evidence)).decision, + ReadinessDecision.unverifiedPrerequisites, + ); + git(['add', 'README.md']); + git(['commit', '--quiet', '-m', 'unsafe MDX']); + final unsafe = git(['rev-parse', 'HEAD']); + expectFailure( + await evaluate(unsafe, evidence), + ReadinessFailureClassification.headMismatch, + ); + evidence['expected_pr_head_sha'] = unsafe; + evidence['independent_audit']['audit_head_sha'] = unsafe; + expectFailure( + await evaluate(unsafe, evidence), + ReadinessFailureClassification.invalidReleaseMetadata, + ); + }, + ); + test( + 'documented grammar and repeated existing runtime identity are inert', + () async { + final files = metadataFiles(); + for (final sha in [baseSha, headSha]) { + final old = files[sha]!['README.md']!; + files[sha]!['README.md'] = ( + mode: old.mode, + contents: '${old.contents}\nNative v0.4.0.\n', + ); + } + final candidate = files[headSha]!['README.md']!; + files[headSha]!['README.md'] = ( + mode: candidate.mode, + contents: + '${candidate.contents}\nKnown v0.4.0 limitation: `root ::= "a"{2000}`.\n', + ); + expect( + (await evaluateMetadata(files: files)).decision, + ReadinessDecision.unverifiedPrerequisites, + ); + }, + ); + test( + 'fence and inline context cannot convert existing fragments to MDX', + () async { + for (final snippet in [ + '{dangerous()}', + 'export const bad = 1;', + '