Skip to content

Notify demo maintainers when their demo breaks on main - #124

Draft
abernier wants to merge 5 commits into
mainfrom
broken-demos-notify
Draft

Notify demo maintainers when their demo breaks on main#124
abernier wants to merge 5 commits into
mainfrom
broken-demos-notify

Conversation

@abernier

Copy link
Copy Markdown
Member

What

Every demo's pmndrs.json now declares maintainers (bare GitHub logins, required, [] allowed). On pushes to main, CI plays all e2e tests (--continue=dependencies-successful) and opens/maintains one GitHub issue per broken demo, @mentioning its maintainers. PR behavior is unchanged (fail-fast).

How

  • Schema: maintainers added to schemas/pmndrs.schema.json + the hand-rolled validator (login regex, uniqueness). All 158 pmndrs.json migrated with "maintainers": [] and normalized to prettier style.
  • CI on main: the e2e step is continue-on-error so build + Pages deploy always run — one broken demo no longer freezes the site. A final e2e-status-job still turns the run red. Snapshot-regen dispatch runs never notify.
  • Notifier (bin/notify-broken-demos.mjs, pure logic unit-tested in bin/lib/broken-demos.mjs, pnpm test:unit): reads the turbo --summarize run summary (broken = build2 or test exit ≠ 0); each issue is tied to its demo by a hidden body marker; the set of open broken-demos issues is the only state store. Delta-only pings: body mentions on creation, comment on reopen, silence while still broken, auto-close with a note once green.
  • Flakes: playwright retries: 2 in CI — a demo is only broken after 3 straight failures.
  • Docs: docs/agents/broken-demos.md + pointer in AGENTS.md.

Trade-off (deliberate)

A visual regression caught by snapshots is deployed until its maintainer fixes it: monitoring over gating.

Test plan

  • pnpm test:unit (11 tests on the pure logic, also runs in CI)
  • pnpm lint:metadata over the 158 migrated files
  • DRY_RUN=1 node bin/notify-broken-demos.mjs against a real local turbo summary
  • Full pnpm test e2e suite locally
  • After merge: watch the first main run (label + issues created on a real breakage)

🤖 Generated with Claude Code

https://claude.ai/code/session_016zSPJcxKxiSPKX1hkejGGC

abernier and others added 5 commits July 30, 2026 10:17
Each demo's pmndrs.json now declares its maintainers: the GitHub logins
notified when the demo breaks on main. Empty list = nobody to ping.
pmndrs.json files are normalized to prettier style along the way.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zSPJcxKxiSPKX1hkejGGC
Reads the turbo run summary, maps failing build2/test tasks back to demos
and their pmndrs.json maintainers, and maintains one issue per broken demo
(hidden body marker ties issue to demo; the set of open issues is the only
state store). Delta-only mentions: body on creation, comment on reopen,
silence while still broken, auto-close with a note once green again.

Pure logic lives in bin/lib/broken-demos.mjs, covered by node --test
(pnpm test:unit).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zSPJcxKxiSPKX1hkejGGC
- PRs keep the fail-fast behavior
- main runs turbo test --continue=dependencies-successful --summarize with
  continue-on-error: one broken demo no longer holds the deploy of the
  other demos hostage
- bin/notify-broken-demos.mjs reconciles the per-demo issues (issues: write)
- e2e-status-job mirrors the e2e outcome so the run still goes red

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016zSPJcxKxiSPKX1hkejGGC
@abernier
abernier requested review from krispya and removed request for krispya July 30, 2026 10:23
@abernier
abernier marked this pull request as draft July 30, 2026 11:04
@abernier

Copy link
Copy Markdown
Member Author

or maybe the only realistic maintainer should be @claude ...

@abernier abernier self-assigned this Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant