From e5101ad4333699f8983425480910a1500ceaff4d Mon Sep 17 00:00:00 2001 From: Bruno Vicco Date: Sat, 18 Jul 2026 18:36:12 -0300 Subject: [PATCH] docs: finalize Phase 0-1 validation --- CHANGELOG.md | 6 +++++ VALIDATION.md | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++ docs/LOOPS.md | 27 +++++++++++++-------- 3 files changed, 89 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9fe008a..3433ffc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,12 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and build artifact. It never modifies repository code; its optional agent-interpretation step is disabled by default and defines no credentials. Minimal per-job permissions, `persist-credentials: false`, and full-SHA-pinned actions throughout. + - The shared validator is now consumed as the deterministic `engineering-loop-schemas v0.1.2` + bundle pinned to `0459d61b7b1d4e7b46709e6d3895770553e6fab0`. Its `manifest.json` records + source provenance, file sizes, SHA-256 hashes, and the declared package-import adaptation. + The mandatory `loop-schema-vendor` quality check validates the bundle offline, and regression + tests prove that manual tampering is rejected. The earlier `75a63eef...` integration described + below was an intermediate pre-release state and is superseded by this published bundle. ### Fixed diff --git a/VALIDATION.md b/VALIDATION.md index 4e6ca3f..ecd84bb 100644 --- a/VALIDATION.md +++ b/VALIDATION.md @@ -1,5 +1,71 @@ # Validation record +## Current validation — Phase 0-1 + +Validated on 2026-07-18 for the report-only Evidence-Gated Engineering Loop foundation. + +### Identity and provenance + +- Published repository commit: + `59b29429c4b50984e2ceffd9f411ef2028a72e8b`. +- Harness generator version: `1.2.1` (`bootstrap.py` `HARNESS_VERSION`). +- Shared schemas: `v0.1.2` at + `0459d61b7b1d4e7b46709e6d3895770553e6fab0`. +- Final integration: pull request + [#8](https://github.com/brunovicco/codex-python-engineering-harness/pull/8). +- The pull-request quality run validated source head + `ed0f5bd6827bba8cfef7948bb339b694dbfb62d5` before GitHub recreated the commit through squash + merge: + . + +### Results + +- Repository quality gate: passed. +- Regression tests: 70 passed. +- Ruff lint and format check: passed. +- Strict Mypy: passed. +- Bandit medium/high findings: none. +- `pip-audit`: no known vulnerabilities. +- `loop-schema-vendor`: passed for `v0.1.2`. +- Positive bundle-integrity test: passed. +- Manual-tampering detection test: passed. +- Legacy provider-reference regression test: passed. +- Harness self-evaluation: `Overall: PASS`. +- Complete quality gates passed for all six profile/governance combinations: + - `service-none`; + - `service-agentic`; + - `library-none`; + - `library-agentic`; + - `workspace-none`; + - `workspace-agentic`. + +### Scope and limitations + +This validation covers Phase 0-1 only. The repository can validate contracts, execute existing +quality gates, verify vendored-schema integrity, render temporary profiles, and produce reports. +It does not provide a loop runner, state machine, evaluator runtime, autonomous candidate creation, +candidate promotion, merge, or deployment. + +The builder's report remains non-authoritative. A quality gate is the technical authority, and +human review remains required before any promotion decision. + +The self-evaluation workflow still uses the previously pinned `setup-uv` action while the main +quality workflow uses the newer approved pin. Aligning those pins is CI maintenance and should be +performed in a separate change from this documentation record. + +### Reproduce the Phase 0-1 validation + +```bash +uv lock --check +uv sync --frozen --all-groups +uv run python scripts/quality_gate.py +uv run python scripts/loop_self_evaluation.py \ + --output-dir build/loop-self-evaluation +git diff --check +``` + +## Previous validation — 2026-07-16 + Validated on 2026-07-16 (America/Sao_Paulo) with Python 3.12 for the harness and Python 3.13 for generated profiles. diff --git a/docs/LOOPS.md b/docs/LOOPS.md index fcb30f4..cb45b72 100644 --- a/docs/LOOPS.md +++ b/docs/LOOPS.md @@ -51,10 +51,13 @@ Every loop run belongs to one of three levels of scrutiny, mirrored from - A builder never certifies its own result. Only a mechanically-derived `verdict` can. - A hard gate is default-FAIL and must reduce to a command with an exit - code. The set of hard gates a contract may reference - (`acceptance.hard_gates`) is exactly the named checks this harness's own + code. The contract-addressable hard gates + (`acceptance.hard_gates`) are exactly the named checks this harness's own `quality_gate.py` implements: `lock`, `lint`, `format`, `typing`, `tests`, `security`, `dependencies`, `architecture`, `mcp`, `governance`. + Separate mandatory infrastructure checks, including `loop-schema-vendor` and + `loop-contracts`, always protect provenance, integrity, and contract validation; they are not + arbitrary commands selectable by a builder. - Evidence is bound to exact commits (`baseline_sha`, `candidate_sha`) and a hashed environment (`uv_lock_sha256`), so a verdict can always be traced back to exactly what ran against exactly what code. @@ -81,14 +84,18 @@ Every completed run resolves to exactly one final state ## Vendoring -`scripts/_vendor_loop_schemas/` (and its `template/scripts/` copy shipped -to generated projects) is a verbatim vendored copy of -`engineering-loop-schemas`' `src/loop_schemas/` at a pinned commit, recorded -in a header comment in each vendored file. Re-vendor from the source -repository rather than hand-editing; the one intentional deviation (the -package directory is named `_vendor_loop_schemas`, not `loop_schemas`, so -it does not collide with the `scripts/loop_*` denylist above) is documented -in that same header. +`template/scripts/_vendor_loop_schemas/` is a deterministic bundle rendered from +`engineering-loop-schemas v0.1.2` at the full commit +`0459d61b7b1d4e7b46709e6d3895770553e6fab0`. Its `manifest.json` records the source repository, +version, commit, file sizes, SHA-256 hashes, and the declared import adaptation. + +The bundle is not a byte-for-byte copy. During rendering, the package import in +`validate_contract.py` changes from `loop_schemas` to `_vendor_loop_schemas`; this keeps the +vendored package isolated and avoids collision with the protected `scripts/loop_*` namespace. The +adaptation is explicit in the manifest and covered by deterministic-rendering, integrity, and +tampering tests. `loop-schema-vendor` verifies the bundle offline on every generated-project +quality run. Fix the canonical schemas repository and render a new version instead of editing the +bundle manually. `validate_contract.py` is stdlib-only. Reading a YAML contract requires PyYAML to be importable in the environment `scripts/quality_gate.py` runs