ci(test): e2e and live filename tiers move to a nightly run on main

[claude]

Fixes #16455
Fixes #16491

What

The e2e and live test tiers leave the per-PR and merge-queue runs and run nightly on main. Selection is by the existing filename tiers only (*.e2e.test.*, *.live.test.*); no test file is renamed, deleted or edited.

• One switch, OS_TEST_TIERS (queue | nightly; unset reads as queue; anything else is refused loudly), read once in scripts/nightly-tiers.mjs and applied in packages/cli/vitest-tiers.ts's walk — the one place both of that package's vitest projects derive their population from, so unit / integration stay a partition of whatever the setting selects. The unit project becomes an explicit include list (population minus integration) instead of exclude: INTEGRATION_FILES, because an exclude-shaped project falls back to vitest's default include and would collect the tier files the queue must not run.
• turbo.json hashes OS_TEST_TIERS into every test task (env, not passThroughEnv): turbo 2.10 runs in strict env mode and strips undeclared variables, so without this the switch never reaches vitest — and a per-package <pkg>#test override does not inherit the base task's env (measured with --dry-run=json: identical hash across modes until each override named it), so all 27 overrides name it. A nightly can therefore never replay a queue-mode cache entry.
• .github/workflows/ci.yml: only the "Run this shard's tests" step gains OS_TEST_TIERS: queue in its env:. Nothing else in the file moves.
• .github/workflows/test-nightly-tiers.yml (new, coverage-nightly.yml's shape: schedule + workflow_dispatch, node 22, pnpm, restore-only caches): enumerates the tier-owning packages (nightly-tiers.mjs --packages, turbo ls document shape), refuses loudly if any of them does not honour the switch (--check, judged by what vitest collects under each setting), shards them with scripts/partition-test-shards.mjs across 2 shards, runs the same leg loop as Test Core under OS_TEST_TIERS=nightly with vitest's JSON reporter beside the default one, and a single report job files or refreshes one card on red — title nightly-tiers: red on main, labels bug · domain:devx · priority:p1 on create only, body with the run link and the failing files (from the JSON reports, never a log regex), de-duplicated by scanning open issues for the title prefix through the repo-scoped REST API with the workflow's token (bounded pages; a truncated scan refuses to file). On green it does nothing; on any ref but main it writes nothing and publishes the would-be card to the run summary.

Deviation from the card: one test edit, ruled by the seat

The card's no-test-edit clause is lifted for exactly one file, packages/cli/src/commands/secret/orphans.driver-contract.test.ts, by the seat's ruling on the rider #16491 (its claim, first sentence, verbatim): "Claim: PM loop round 9 — rider on flight X (#16455, PR #16481), ruled by the seat at X's delivery 04:3xZ: the card's ⛔ no-test-edit clause is lifted for exactly this one file." — #16491 (comment)

The edit (commit 07464ab227, +7/−1): beforeAll saves and sets OS_SECRET_KEY to randomBytes(32).toString('hex') beside NODE_ENV = 'production' — the variable serve-process.ts already applies to its spawns — and afterAll restores it beside NODE_ENV; one node:crypto import. Nothing is written under $HOME, nothing is depended on from a sibling, no other test file is touched. Measured after the edit: the file alone with OS_SECRET_KEY unset and with it set both Test Files 1 passed (1) / Tests 5 passed (5), and $HOME/.objectstack/dev-crypto-key still absent afterwards.

Measurement first (ruling 1)

At 6eba38f5a3: 72 vitest.config.ts files, every one a standalone defineConfig, none extending a shared config; no root vitest.config.*, no vitest.workspace. Tier-named files: 60 *.e2e.test.ts, all in packages/cli; 0 *.live.test.* anywhere (the brief's "23 *live* files" are substring hits — live-mysql, live-dialect-matrix, lint-liveness, queue-delivery — not tier-named, and not selected). So the count of configs that would have to read the switch to exclude the two tiers is 1 (≤ 10 ⇒ continue), and that config already partitions its suite through vitest-tiers.ts, which is where the switch is applied.

Collection, read from vitest list --filesOnly in packages/cli (272 test files on disk): queue → 212 (180 unit + 32 integration); nightly → 60 (1 unit + 59 integration) == the tier files on disk; 212 + 60 = 272, no file in both projects under either setting; unset == queue; OS_TEST_TIERS=night → exit 1 naming the two legal values.

Sharding (ruling 4)

Two shards, derived rather than picked: cli is the only tier-owning package (~75% of its 1231 s CI suite is e2e ⇒ ≈ 15 min), FILE_SHARDED_PACKAGES already cuts it into two vitest slices, and the partitioner refuses one shard (assertSlicesSpread: "bin 1 holds more than one slice of @objectstack/cli … Refusing to shard"), so 2 is the smallest legal count; each shard carries one slice (@objectstack/cli 1/2 / 2/2, 229 s predicted each from the dataset), well under 60 min.

Ablation (ruling 5)

Scratch commit 8ff289aba0 on top of 770dd18205 added packages/cli/test/zz-nightly-tiers-ablation.e2e.test.ts (its marker line confirmed on disk by grep -c = 1 before the run).

• OS_TEST_TIERS=queue: vitest list --filesOnly → 212 files, scratch absent (0 hits); the narrowed vitest run <scratch> → No test files found, exiting with code 1; the whole package → Test Files 1 failed | 211 passed (212) / Tests 2719 passed | 6 expected fail | 5 skipped (2730) — the one failure is not the scratch file (0 mentions in the log) but src/commands/secret/orphans.driver-contract.test.ts refusing to boot without a crypto key (see "Findings" below).
• OS_TEST_TIERS=nightly: list → 61 files, scratch present (1 hit); the narrowed run → FAIL |unit| test/zz-nightly-tiers-ablation.e2e.test.ts … AssertionError: expected 'collected under OS_TEST_TIERS=nightly' to be 'NIGHTLY_TIERS_ABLATION_MARKER_RED', exit 1.
• The tier partition pin under queue with the scratch file on disk: Test Files 1 passed (1) / Tests 22 passed (22).
• Restore: git reset --hard 770dd18205 → HEAD 770dd18205, git status --porcelain empty, scratch file absent (ls: No such file or directory). The scratch commit was never pushed.

The partition pin itself, on the committed tree: queue → 22/22 green; unset → 22/22 green; nightly → No test files found, exiting with code 1 — the pin carries no tier name, so under the nightly setting it is not collected, which is the ruled behaviour (the nightly runs exactly the two tiers), not a red.

Nightly smoke run (ruling 5)

Two real-runner runs on 770dd18205, both green, both with the board write skipped (not main):

• PR-triggered (pull_request, the paths-filtered trigger the dispatch-gates pin requires): https://github.com/objectstack-ai/objectstack/actions/runs/34080518326 — Nightly tiers (1/2) 17m11s, Nightly tiers (2/2) 15m53s, File or refresh the red card 0m18s (success; summary: "Board write skipped: this run is on refs/pull/16481/merge, not main").
• workflow_dispatch on the branch (REST answered 403 Resource not accessible by integration for the container token; the MCP route queued it): https://github.com/objectstack-ai/objectstack/actions/runs/34081750890 — Nightly tiers (1/2) 16m50s, Nightly tiers (2/2) 13m07s, report job 0m15s, success.

So the whole e2e tier (60 files, two slices) finishes in under 18 minutes per shard on a hosted runner; no real e2e suite is red on this head.

Serial with flight V (ruling 7)

V's branch claude/issue-16453-merge-group-affected-set was pushed (head e6fe28e9) before this branch's first content push; git merge-tree --write-tree HEAD refs/flight/16453 → exit 0, tree 2a13194c9a; the same with -c merge.os-regen.driver=false → exit 0, same tree. V's ci.yml hunks (lines 350 / 415) and this PR's (line 575) do not overlap. V's PR #16471 has since merged; origin/main (4500bc5d79) is merged into this branch as 2fe914aa48 (merge commit, no rebase) and the shard step composes: OS_TEST_TIERS: queue sits in the "Run this shard's tests" env: beside V's scripts/ci/select-shard-packages.sh step. Flight C3 (#16466, claude/issue-16466-repo-conformance-task) also touches turbo.json: git merge-tree --write-tree HEAD refs/flight/16466 against its pushed head is recorded in the report (twice, the second with -c merge.os-regen.driver=false).

Gates (ruling 6)

All on 770dd18205 unless stated. node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack derived 77 commands (from git, off the merge base); every one ran with its exit captured before any pipe (one log per command).

• 73 exit 0 on the previous commit d2fab7ab77; the four non-zero and their disposition:
  • check:declared-population-live (1) — a real finding on scripts/nightly-tiers.mjs (a vitest/package.json literal read as a population declaration): fixed in 770dd18205 (resolve vitest's root by walking up; no-path-population marker) → re-run exit 0 (223 of 272 families … every one reaches this tree's 8088 tracked files).
  • check:pm-dispatch-gates (1) — a real finding: a family reachable only from a scheduled workflow is refused; fixed by the pull_request paths trigger + the marker → re-run in progress on 770dd18205 at PR time (its self-test drives ~1,500 cases and takes ~14 min); result in the report.
  • check:dual-build-cjs-loads (3, PREREQUISITE NOT MET: packages outside cli's closure have no dist/) — NOT MEASURED locally; CI builds fresh. This diff touches no package source or build config.
  • check:type-check-debt (3, node heap crash under a neighbour's concurrent gate batch) — re-run alone under the lock with 6144 MB: exit 0 (5 ledger entries re-measured … none above its recorded number; surplus: none).
• test/vitest-tiers-partition.test.ts: queue 22/22, unset 22/22, nightly not collected (above).
• node scripts/check-shard-attestation.mjs --self-test → exit 0 (144 assertions).
• node scripts/pm/check-governed-merges.mjs --test <final file list> → exit 0: 0 of 7 path(s) hit the register … ✅ NOT governed.
• pnpm check:nul-bytes → exit 0 (scanned 8081 text file(s) … no raw ASCII control bytes); own control-char grep over the seven files: clean.
• Whole-repo pnpm lint (eslint . --no-inline-config) through scripts/pm/os-verify-lock.sh (slot issue-16455, one acquisition, waited 27 s): LINT-EXIT=0, held 2m37s, on 770dd18205 with a clean tree.
• pnpm --filter @objectstack/cli typecheck → exit 0 (check:test-typecheck: OK — 3 file(s) / 28 error(s) … held in test-typecheck-debt.json, unchanged).
• node scripts/nightly-tiers.mjs --self-test → 46 cases; --check → 1 tier-owning package(s) honour OS_TEST_TIERS (judged by what vitest collects); node scripts/partition-test-shards.mjs --self-test → OK; check-declaration-mirrors → OK (pairs scripts/nightly-tiers.d.mts).
• turbo: --dry-run=json shows @objectstack/cli#test specified.env: ['OS_TEST_TIERS'] and a different hash under queue vs nightly (035285c9dd71f80c vs 902cf67d871aad09); @objectstack/types#build hash unchanged across modes (builds are not rehashed).

Findings

• src/commands/secret/orphans.driver-contract.test.ts reds under queue on this box (the only red of 212). It sets NODE_ENV=production and boots a real kernel; LocalCryptoProvider derives its posture from NODE_ENV alone and, in production, needs OS_SECRET_KEY, OS_DEV_CRYPTO_KEY or a persisted $HOME/.objectstack/dev-crypto-key; the test provides none. Alone with OS_DEV_CRYPTO_KEY set it passes 5/5; alone without, it refuses (exit 1). No file in the queue population writes that key (absent on this box after all 212 files ran) — whether main's green Test Core today rests on an e2e spawn minting it on the runner's $HOME is what this PR's Test Core measures. ⛔ Not edited here (the ruling forbids editing any test file); resolved by the ruled rider edit above (test(cli): orphans.driver-contract.test.ts's production-posture boot depends on a sibling e2e spawn minting the dev crypto key — deterministic red once the e2e tier leaves the queue #16491).

Confirmed on this PR's CI: Test Core (5/6) (cli slice 1/2) https://github.com/objectstack-ai/objectstack/actions/runs/34080518324/job/101614831504 — Test Files 1 failed | 106 passed (107), the one failure exactly this test with the same refusal; the completeness guard is OK (12 of 12 packages, 8118 tests accounted for). Every other check on the head is green (43 check runs: 2 failed — this shard and its aggregate — 5 skipped, the rest success). The mechanism is documented in packages/cli/test/helpers/serve-process.ts's header: a dev-mode boot persists a minted key to $HOME/.objectstack/dev-crypto-key, "put there by any earlier dev-mode boot in the same run … a cross-test ordering coupling through the runner's home directory, and under vitest's parallel workers it is nondeterministic". The helper's own spawns pass OS_SECRET_KEY for that reason; this test's in-process production-posture boot never did, and only an e2e sibling that still spawns bin/run-dev.js without a key (e.g. test/start-port-banner-agreement.e2e.test.ts, test/migrate-meta.e2e.test.ts spawn os start) minted the file for it. Moving the e2e tier out of the queue turns that nondeterministic green into a deterministic red. That is the one-file edit recorded under the deviation heading above; on this branch the test now declares its own key and passes with or without the variable in the environment.

Out of scope / open questions

• Per-PR drift guard. nightly-tiers.mjs --check makes a package that adopts a tier without reading the switch red on the nightly; nothing makes it red on the pull request that introduces it (that would be a check: script wired into lint.yml, outside this card's file surface). The seat filed that follow-up as ci(test-tiers): per-PR gate for a package that adopts a *.e2e.test.* or *.live.test.* file without reading OS_TEST_TIERS #16494; this PR is not widened.
• .claude/agents/os-dev.md still names cli's tiers as CI's integration layer; flight Y owns that file and it is untouched here.

Session: https://claude.ai/code/session_019RfFHiRCSs3JXLK4cwcfox (body edited after the nightly runs completed and again for the seat's rider ruling; the footer below is the platform's edited form)

🤖 Generated with Claude Code

https://claude.ai/code/session_019RfFHiRCSs3JXLK4cwcfox

---

Generated by Claude Code

[github-actions]

📓 Docs Drift Check

3 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

What this run could not see

• the SDK route bridge reached 61 of 219 client-bound route-ledger rows — the other 158 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 158: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 22 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 21c5dcbb3d073716c7d15354a6f9b312cf2205ff → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 9ea72e5e05c817d2360e42bef104d8f5a87c393a — the merge of head 07464ab2279e4f5e18ccb3bb682359d14a706f67 into base 21c5dcbb3d073716c7d15354a6f9b312cf2205ff, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 9ea72e5e05c817d2360e42bef104d8f5a87c393a && git checkout 9ea72e5e05c817d2360e42bef104d8f5a87c393a
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 21c5dcbb3d073716c7d15354a6f9b312cf2205ff 07464ab2279e4f5e18ccb3bb682359d14a706f67 && git checkout -B drift-repro 21c5dcbb3d073716c7d15354a6f9b312cf2205ff && git merge --no-ff 07464ab2279e4f5e18ccb3bb682359d14a706f67

node scripts/docs-audit/affected-docs.mjs --json 21c5dcbb3d073716c7d15354a6f9b312cf2205ff

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.