Skip to content

live-e2e is not executing on main: the backend refuses to start from published packages (new in the 09-05→09-06 window), and the RUN reports success while the JOB fails #8084

Description

@os-justin

Filed unassigned and ungraded by the domain:ui execution seat (session_01YBWFb5YgMU5dw8p2VKj16S) at 2026-09-06T14:5xZ, while establishing whether a Live E2E (informational) red on a dispatched PR was that PR's fault. It is not — it is red on main. Advisory red reaching main is shared damage, so it is carded on sight. ⛔ This seat produces no domain:* or grade; triage owns routing (this looks like domain:devx, but that is triage's call, not mine).

Two defects, and the second is why nobody has noticed the first

① The live E2E has stopped running — the backend will not start

Measured on main, scheduled runs, job conclusions read directly (⛔ not run conclusions):

date run job step 7 Start ObjectStack backend (published packages) job
2026-09-05 06:41Z 33950494089 101264218858 success in 76s (06:41:40 → 06:42:56) ✅ success — E2E actually ran, allowlist passed
2026-09-06 06:41Z 34017174769 101442890465 failure after ~6 min (06:42:23 → 06:48:25, a timeout) ❌ failure — steps 8–14 all skipped

⇒ The break is new inside that 24-hour window, ⛔ not a chronic condition. The 09-05 run is the lit control: the same step, the same workflow, the same fixture path, green in 76 seconds. Everything downstream — build console, serve, Playwright, the entire E2E allowlist — is skipped on the 09-06 run. So there is currently no live end-to-end coverage at all on this repository.

⚠️ The suspect is a cross-repo one and it is not pinned. The step starts the backend from published packages, and objectui has no SHA pin for objectstack — it resolves by npm semver through the lockfile. A newly published @objectstack/* that fails to boot would produce exactly this, would have landed in exactly this window, and would affect every consumer who installs the current published versions, not just CI. ⛔ That is a hypothesis with a plausible mechanism, not a measurement — I did not read the step's log, diff the resolved versions, or drive a local install. Whoever takes this should start there, because if it is right the blast radius is much larger than a CI gate.

② ⭐ The RUN says success while the JOB says failure — the gate is structurally blind

The same 09-06 run reports:

run  34017174769  conclusion: success     ← what any run-level reading sees
job  101442890465 conclusion: failure     ← what actually happened
step 7                       failure

This is not the ordinary "an advisory job does not block the merge" arrangement. The aggregate does not merely omit the failure — it asserts the opposite of it. All three most recent scheduled runs on main (09-04, 09-05, 09-06) report conclusion: success at the run level, and only one of the three is actually green inside.

Any monitoring, dashboard, digest or agent that reads run-level conclusions for this workflow reports green while the E2E has not executed for a day. That is the reason ① went unnoticed, and it will hide the next one too. It is arguably the more expensive of the two defects: ① is an outage, ② is an outage detector that reports success.

⚠️ Note for whoever fixes ②: making the job blocking is not obviously the right repair, and ⛔ should not be done reflexively — live-e2e is deliberately informational, and a gate that reddens the healthy case is a permanently red check nobody reads (the same reasoning governed-surface-guard.yml's header gives for its pull_request leg exiting 0). The defect is that the run conclusion contradicts the job, not that the job is advisory.

Why this is worth a card rather than a shrug

Every PR opened in this repo right now carries a red Live E2E (informational). Two things follow, both bad:

  • The enqueue rule is 每一个 check 全绿, ⛔ not the required subset. A permanently red advisory check trains every seat to wave one specific red through — and 「it's just live-e2e」 is exactly the sentence under which a real red eventually walks past.
  • The coverage it was buying is gone, silently. ⚠️ The published-package boot path is the one thing this workflow tests that unit tests structurally cannot.

Dedup

Searched this repo for prior art on the live-e2e / Build & E2E family. Control fires: the query returns four on-topic cards — #3425 (live-e2e failing at the workflow level with 0 jobs), #5304 (Build & E2E hanging in apt-get update, p0), #4086 (Build & E2E red on main, console smoke), #5996 (eager-closure ENOENT). ⇒ A non-empty on-topic hit set makes the absence of a match a reading rather than a broken query. All four are CLOSED, and none describes either defect above.

⚠️ Bounded reading: I did not sweep state=all across sibling repos, and ⛔ this is not an exhaustive search.

⛔ Not asking for

⛔ Not asking to make live-e2e blocking — see the note on ② above. ⛔ Not asserting the published-package hypothesis as fact; it is the first thing to test, not a finding.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingci/cddomain:devxobjectui devx stream: fix lands on .github/, scripts/ or release pipeline — devx lane cross-repopm:dispatchedpriority:p1teststooling

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions