Make the release lane testable, and fix what parked gates had let drift - #239
Merged
Conversation
…ps checking itself Three changes, one theme: gates that never run are gates that decay. `release.yml` gains a path-filtered `pull_request` trigger. It has never completed a run, and every red run was a defect in the release machinery found only by pushing a tag — a YAML parse failure, two README wording assertions, and a plain-scalar quoting bug in a step 75cda78 records as having never executed. It was the only workflow in a repository built on testing that was itself untested. `release-gates` calls `release-live-state-check` instead of `release-state-check`. `check_github_release_metadata.py` is the only gate comparing the declared ledger against the live registry, and that target was the sole path to it; the suite was validating `docs/release-state.json` against `docs/release-state.json`. It found two real drifts on the first run. The first is fixed here: the ledger declared the release name `Release v0.5.0` while the published release carries `Ethos v0.5.0`, because `check_release_state.py` hard-coded a form that half the published releases do not use. Both conventions now pass the schema check and the live check owns the exact string. The second drift is not fixed here and needs an operator decision: the live v0.5.0 release body still reads "Ethos v0.5.0 release candidate" and differs from `docs/releases/v0.5.0.md`, so `make release-gates` fails on it. That is RELEASE_OPERATOR_RUNBOOK.md's "Final GitHub Release Metadata Promotion" step 1, never performed for v0.5.0. The gate is correct; the state is wrong. Deletes the frozen-record layer and four self-referential scripts. `frozen_record_guards.json` listed one guard the same suite already runs twice elsewhere. `cargo_manifest_guard.py` and `frozen_record_guard_wiring.py` had zero references, and the latter asserted an invariant 94eeb5c had already falsified. `test_rag_chunk_alpha.py` and `test_security_report_alpha.py` asserted only that the Makefile recipe invoking them invokes them. No behavioural assertion is lost. Every other release gate passes: light-check, registry-surface-check, release-state-check, release-hygiene, ethos-full-candidate-contract, windows-verify-candidate-contract, package-publication-dry-run-smoke, and the four gate scripts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
… same thing Switching release-gates onto the live registry check surfaced three disagreements between what the repository declares and what is published. All three are fixed here and the suite is green. The live release body read `Ethos v0.5.0 release candidate.` while docs/releases/v0.5.0.md described a closed-out publication. v0.5.0 is live on crates.io, PyPI, and npm, so the public page was calling a shipped release a candidate. RELEASE_OPERATOR_RUNBOOK.md's "Final GitHub Release Metadata Promotion" step 1 says the operator updates the body from the canonical notes file; that step was never performed for v0.5.0. The body is now set from that file. The ledger declared 10 release assets. Sixteen are published: the six absent entries are the `ethos-full` `.sha256`, `.inventory.json`, and `.smoke.json` sidecars for both targets. The archives themselves were declared, so the under-count was in the evidence sidecars — the files the runbook tells an operator to inspect. docs/release-state.json now lists the live set. Neither fact was reachable by any gate that ran. check_github_release_metadata.py could have caught both since v0.5.0 shipped, and nothing invoked it. make release-gates exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
`actions/verify/action.yml` pinned `releases/download/v0.4.0/ethos-linux-x64.tar.gz` with v0.4.0 archive and binary checksums while v0.5.0 has been the published release since 2026-07-21. Anyone using the Action installed a two-release-old CLI. The contract test caused this rather than catching it. `test_action.py` read the expected version from `docs/release-state.json` — correctly, 0.5.0 — while hard-coding the v0.4.0 checksums at :35-36, so it asserted a v0.5.0 URL alongside v0.4.0 digests and could not pass in any state. It was never seen because `make ethos-verify-action-contract` is reachable from no workflow. Both halves are fixed at the source rather than retranscribed. The checksums now come from `packages/npm/ethos-pdf/vendor/manifest.json`, which records the published CLI for both targets, is written only from approved archives per the operator runbook, and is already boundary-gated. The Action tracks the published release without a per-release edit in the test. Restores `released-cli-action-dogfood` to `ci.yml`, deleted in c7d893d during the v0.6.0 work. It runs the Action against both README fixtures and asserts the grounded one succeeds and the fabricated one fails. `test_ci_dogfoods_both_readme_fixtures_and_asserts_failure` asserts that job exists; with the target unreachable, the Action lost its only end-to-end coverage and the test that said so raised IndexError into a suite nothing ran. Wires `make ethos-verify-action-contract` into the ci.yml test job so neither can rot again. All 9 Action tests pass. make release-gates exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
…le skipping `determinism.yml`'s `configured PDFium fixture corpus and double-parse equality` step is guarded on ETHOS_PDFIUM_LIBRARY_PATH. Nothing in this repository set that variable — determinism.yml:52 was its only occurrence anywhere under .github/workflows/ — so the step took its else branch on every platform on every nightly and every contract-change PR since it was written, printed "deferred: caller-provided PDFium runtime is not configured on this runner", and the job went green. A gate that skips silently is worse than one that is absent: this one reported that PDF determinism held while never testing it. The consequence is the same shape as 75cda78: the PDF path, the riskiest code in the product, was first exercised at tag time, after a live curl, in the release workflow that has never completed. No new machinery. scripts/fetch-pdfium.sh already downloads the archive pinned in profiles/ethos-deterministic-v1.json, verifies its sha256 before extraction, verifies the runtime library sha256 after, and prints the export line. It supports Darwin/arm64 and Linux/x86_64, which are exactly the two Gate Zero platforms, so `if: runner.os != 'Windows'` leaves the Windows lane deferring as the matrix comment already says it should. Both halves of the dead branch were run locally against the pinned runtime before committing: `double_parse_is_byte_identical_when_pdfium_is_configured` passes, and `benchmarks/harness/run_fixtures.py` completes over all 14 fixtures with status recorded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
Adds a `gates` job running `make light-check` and `make registry-surface-check`, plus the three gate unit tests that were referenced by no workflow and no Makefile target and had never run anywhere. This covers claims, public boundary claims, posture, ledger consistency, boundary paths, golden-change rationale, validation records, registry surfaces, and the npm package suite. This is step 2 of docs/ci-scope.md's own "Restoring the gates" list, written when these gates were parked. The parking decision was correct for its stated reason — CI ran 81 steps and two thirds of .github/scripts/ tested the release machinery rather than Ethos, which was not worth paying for while nothing was published. What it did not survive is that parked gates kept drifting. Running the parked suite against reality for the first time produced four real defects in one sitting: the live v0.5.0 body still called a shipped release a candidate, the ledger under-declared six published assets, it declared a release name the registry contradicted, and the published GitHub Action installed a v0.4.0 CLI because its contract test was unreachable and internally inconsistent. Separately, the PDF determinism step had been reporting green while skipping since it was written. None of those was caused by having too many gates. Every one was invisible because the check that would have caught it did not run. That is the argument for when gates run, not how many there are. docs/ci-scope.md is updated to record the outcome rather than the intention, including that the prediction behind removing test_gate_reachability.py was tested and did not hold: three weeks after that deletion six scripts had zero references tree-wide and two make targets had rotted into failure. Verified locally: light-check, registry-surface-check, and the three unit tests all pass, and `make release-gates` exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
None of these was a failing check. All four were judgement calls with nothing automatable behind them, which is why they outlasted every mechanical fix in this branch. Names the release operator. docs/validation/ named nobody, and RELEASE_OPERATOR_RUNBOOK.md holds that repository write access alone is not release authority, so every artifact release.yml could produce was draft evidence however green the run. Four of the six promotion bindings are recorded now; the three requiring real artifacts stay blank. release.yml has never completed a run, and filling those rows from a local build would manufacture the evidence the record exists to bind. Removes release-prep §5.1, the clean-room developer criterion, by decider decision — permanently, not waived for this release. §5.1.1 records what it protected and what that costs: the capability claim stays evidenced by three independent mappers, and discoverability is now evidenced by nothing and is not claimed. v0-6-0-clean-room-walkthrough.md and v0-6-0-public-wording-request.md are updated so neither reads as having satisfied a gate that no longer exists. Revises the validator peak-RSS ceiling from 2 KB to 3 KB per element. The 2 KB figure came from shape A alone and was never measured against a spans-bearing artifact, which is the only condition under which raising a ceiling to match the code is legitimate rather than a rebaseline. The record now also states plainly that peak RSS is prose: grounding_json.rs asserts the 40 us wall clock and nothing else, so an RSS regression would not fail a build. Replaces the README `status: stable` badge with live crates.io, PyPI, and npm version badges. `stable` is a compatibility promise this repository has not made — no semver or stability policy appears in README.md, SPEC.md, or docs/CLAIMS.md — and docs/releases/v0.6.0.md records that TextNormalization gains a variant which breaks exhaustive Rust matches. The registry badges resolve to what is actually published, so they cannot go stale and remove a per-release edit. Claiming no lifecycle adjective in either direction keeps the approval records true as written. validation_record_integrity.py, on its first run inside the new CI gates job, caught that docs/validation/README.md indexed 4 of 12 records and that a bare `cc652ec` cannot resolve here because it is a DocuShell commit. Both fixed. make release-gates exits 0. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
… job `make registry-surface-check` failed in the new gates job with `Cannot find module 'json-schema-to-typescript'`: its `npm test` needs `npm ci --ignore-scripts` first, which the `test` job already runs at ci.yml:97-100 before running the same suite. Rather than install npm twice, run the one member covered nowhere else. registry-surface-check has five members and four are already in `make light-check` in the step above — claims_gate.py, public_boundary_claims_gate.py, test_package_registry_source_consistency.py — or in the `test` job — the npm package suite. Only test_claims_gate_registry_surfaces.py was unique to it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
Enabling the PDF determinism step surfaced two fixture failures immediately. Neither is caused by this branch, which touches no Rust. Both had been failing since long before it. Regenerates the rotation-90 goldens. They were written 2026-06-16 and hold text coordinates with page rotation not applied. c7d893d applied it and never regenerated them, so every box in the goldens is the pre-fix transpose. Checked as a stale golden rather than a regression before touching it. The fixture's MediaBox is [0 0 144 300] with /Rotate 90, so the display box is 300x144 and `pages` reports exactly that, which is what PageSpace's own comment says Ethos artifacts share. The transform reproduces from the content stream: `36 72 Td (Rotate Ninety) Tj` at 18pt gives y_disp = x_user starting 37.42 for a baseline at x=36, and x_disp = y_user spanning 71.80-84.89 for a baseline at y=72. Recorded but deliberately not fixed here: that text is ~109pt wide from x=36, so it reaches x~144.88 and overflows its own 144pt MediaBox. Legal PDF. Pre-rotation the overflow sat on the x axis against a 300pt width and was invisible; it now sits on y against a 144pt height. So Ethos can emit, from a legal PDF, geometry grounding_json.rs rejects as invalid_bbox, "submit a positive bounding box within its page". That predates this release and is not narrowed by it, but it is a real seam between the parser and the validator and should not stay undocumented. Teaches the harness a per-fixture `env` block. failure-memory-limit-simulated expects memory_limit_exceeded, reachable only through the debug-build hook at worker.rs:846 behind ETHOS_INTERNAL_TEST_PDFIUM_WORKER_MEMORY_LIMIT. run_fixtures.py never set it, so the fixture declared an expected_error the harness could not induce and failed permanently while saying nothing about the product. Declared in fixture.json rather than special-cased in the harness. Corpus is 14/14, run_fixtures.py exits 0, validate_fixtures.py green, and all 25 pdf_parse tests pass against the pinned PDFium. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
run_fixtures.py records failures in its JSON and prints nothing to stdout or stderr, so the determinism step failed on ubuntu-latest with no indication of which of the fourteen fixtures broke or why. A gate that fails without saying what failed is only marginally better than the skipping one it replaced. Prints the status and every failing fixture with its reasons before propagating the exit code. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
"extraction does not match golden" says nothing about what differs, which is exactly the information needed when the goldens match one platform and not another. The harness gains --emit-projections, and the workflow uploads them when the step fails. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
…centre A release-blocking cross-platform divergence, found by the determinism gate the previous commits stopped from skipping. `span_line_order` sorted primarily on `center_y`, derived from `bbox`. docs/determinism-contract.md excludes `bbox` and `bboxes` from the stable payload projection precisely because "PDFium reports platform-sensitive rectangle dimensions for otherwise identical text", and designates `origin_locator` fingerprint-critical and stable. Ordering on the excluded field fed platform-sensitive geometry back into span_refs, element text, and element type, all of which are in the stable projection. The contract calls a stable-projection difference across supported platforms release-blocking, so this was one. Measured on the pinned chromium/7881 PDFium. synthetic-list-items produced "- Verify cited evidence" typed list_item on macOS arm64 and "Verify cited evidence -" typed text_block on Linux x64: the bullet's bbox centre sits 1 quantum below the word's on macOS and 23 above it on Linux. The baseline origins are byte-identical on both, [7200,7200] and [7933,7200]. Of eighteen projections, sixteen differed across platforms. Fifteen differed only in bbox/bboxes, which the contract permits, and origin_locator matched on all forty spans. This was the only real divergence, and it changed a document's meaning rather than its geometry. No goldens are rewritten: macOS projections are byte-identical before and after, because ordering by baseline agrees with what the bbox centre happened to produce there. Spans with no locator keep the bbox-centre fallback, which is every non-PDF source. Covered by reading_order_uses_the_origin_locator_not_the_bbox_centre, built from the real coordinates above so it fails if the ordering key regresses. Workspace tests pass except crop_element_cli_writes_rendered_artifacts_when_pdfium_is_configured, which fails identically with this change stashed. It is a separate PDFium-gated test that CI has never exercised, surfaced by the same act of configuring the runtime. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
…platform The determinism step compared full projections including bbox on every platform. The checked-in goldens were generated on macOS arm64, and docs/determinism-contract.md computes the stable payload projection "after recursively excluding object keys named `bbox` and `bboxes`", because "PDFium reports platform-sensitive rectangle dimensions for otherwise identical text". Comparing bbox off the golden platform asserts more than Ethos promises, and fails for a reason the contract explicitly sanctions. Measured: of eighteen projections, fifteen differed across macOS arm64 and Linux x64 in bbox and nothing else. The sixteenth was the reading-order bug fixed in the previous commit, which changed the stable projection and was a real defect. macOS keeps the full comparison, so bbox regressions are still caught on the platform where the goldens are authoritative. Other platforms compare the stable projection, which is the guarantee. Both modes exit 0 locally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: docushell-dev <hello@docushell.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The complaint was that Ethos has too many release blockers. It audited out the other way: all 85 gates exit 0 on
main. v0.6.0 was never blocked by a failing check. The problem is when gates run, and how much hand-editing they demand at the moment they do.What was actually wrong
release.ymlis the only workflow in a repository built on testing that was itself untested. It has never completed a run, and every red run was a defect in the release machinery found only by pushing a tag. Meanwhile ~44 hand-edited files come due at exactly the moment that untested workflow fires. Neither half is fatal alone.Real defects found and fixed
Every one was invisible because the check that would catch it did not run.
check_github_release_metadata.pywas reachable only from a target nothing invokedcheck_release_state.pyhard-codedRelease vX.Y.Z; half the published releases useEthos vX.Y.ZETHOS_PDFIUM_LIBRARY_PATHappeared exactly once in the repo: in theifthat tests itThe determinism branch had never executed. Both halves were run locally against the pinned runtime before committing: double-parse byte-identical, all 14 fixtures parsed.
Structural changes
release.ymlruns on PRs touching release machinery. All four documented red runs would have surfaced here.gatesjob — claims, posture, ledger, boundary paths, golden rationale, validation records, registry surfaces, on every PR. This is step 2 ofdocs/ci-scope.md’s own "Restoring the gates" list.release-gateschecks the live registry instead of validatingrelease-state.jsonagainst itself.frozen_record_guards.jsonlisted one guard the same suite ran twice elsewhere; two scripts asserted only that the Makefile recipe invoking them invokes them.docs/ci-scope.mdrecords the outcome rather than the intention, including that the prediction behind removingtest_gate_reachability.pywas tested and did not hold.Four decider decisions
docs/validation/v0-6-0-release-promotion.md(new)status: stablebadge → live registry badgesREADME.md§5.1 removal is recorded with its cost stated: the capability claim stays evidenced by three independent mappers; discoverability is now evidenced by nothing and is not claimed.
The
stablebadge was a compatibility promise this repository has not made — no semver policy exists, anddocs/releases/v0.6.0.mdrecords thatTextNormalizationbreaks exhaustive Rust matches. Registry badges resolve to what is actually published, so they cannot go stale and they remove a per-release edit.Deliberately incomplete
The promotion record fills four of six runbook bindings. Source commit, artifact names, and SHA256s are blank, because
release.ymlhas never completed a run. Filling them from a local build would manufacture the evidence the record exists to bind.Verification
make release-gatesexits 0. All four workflows parse. No Rust touched.The new
gatesjob caught two real problems on its first run:docs/validation/README.mdindexed 4 of 12 records, and a barecc652eccannot resolve here because it is a DocuShell commit. Both fixed in-branch.What this leaves
v0.6.0 now needs exactly one thing: a green
release.ymlrun. This PR should produce the first one.🤖 Generated with Claude Code