fix(flaky): promote dialog confirm creates machine and promotes guest to member by timothyfroehlich · Pull Request #1602 · timothyfroehlich/PinPoint

timothyfroehlich · 2026-07-05T16:20:30Z

Summary

Weekly flaky-test scan of CI runs from the last 7 days on main (and PR branches feeding into it). Full ranked report below. This PR fixes the #1 flakiest test.

Root-cause analysis (the fix)

Test: e2e/full/machine-owner-picker.spec.ts:126 → Machine Owner Picker — promote-dialog journeys (PP-6oi) › promote dialog confirm creates machine and promotes guest to member

Evidence it's the worst offender this week:

Run 28718046579 (main, 913f7fff) — Playwright's own summary reports 4 flaky, including [Mobile Safari] machine-owner-picker.spec.ts:126:3 (failed once, passed on Playwright's internal retry).
Run 28709215184 (main, fcf31c30) — 6 flaky, including both [Mobile Chrome] and [Mobile Safari] variants of the same test.
Run 28714816089 (dependabot ratchet PR, 9b12478c) — hard, non-recovering failure: original attempt + Retry feat: Setup CI with GitHub Actions #1 + Retry chore: Configure Dependabot for automated dependency updates #2 all failed identically on [chromium]:
```
Error: expect(locator).toBeVisible() failed
Locator: getByRole('heading', { name: /Promote to member and assign/i })
Expected: visible / Timeout: 15000ms
```
This PR only touched GitHub Actions SHA-pinning — unrelated to the test — confirming this is collateral flakiness, not a real regression.

Root cause: playwright.config.ts sets a CI-aware expect.timeout (30s in CI, 10s locally) and gives the Mobile Safari project extra retries + a longer overall test timeout specifically because mobile emulation + parallel-worker contention are slower. But machine-owner-picker.spec.ts hardcoded { timeout: 15000 } on exactly the assertions that flake — the promote-dialog heading wait and the post-confirm redirect wait. That override undercut the environment's own default by half, right where the extra margin was needed most. This explains both classes of failure:

Transient slowness under ~15s but under 30s → Playwright's automatic retry recovers it (the "N flaky" runs).
Sustained CI load pushing every attempt past even the retries → the one hard 3/3 failure.

Fix: removed the three { timeout: 15000 } overrides so these assertions inherit the config's own environment-aware default instead of a fixed value that quietly shrank it.

Verification: typecheck (tsgo), ESLint, Prettier, and the full unit suite (1306 tests) all pass. I could not exercise this Playwright spec live in this sandbox — no Docker daemon is available here for the local Supabase stack (docker info fails to reach dockerd), so there's no way to spin up the app + DB for an E2E run. Flagging this explicitly rather than claiming a local E2E pass that didn't happen; CI (which already runs this spec on every PR) is the first real exercise of the fix, and worth watching once this PR's CI runs.

Other recurring flaky signals seen this week (not fixed in this PR — same undercut-timeout pattern, logged as follow-up)

The same three main-branch "E2E Comprehensive Tests (Post-Merge)" runs above each report a nonzero Playwright flaky count, with several other specs recurring across commits/browsers, all navigation/element-timeout races on Mobile Safari / Mobile Chrome:

e2e/full/machine-timeline.spec.ts:139 — admin reassign journey — TimeoutError waiting for issue-actions-menu-trigger
e2e/full/rich-text.spec.ts:24 — rich text + mentions — toHaveURL never settles on /m/TUV/i/1
e2e/full/technician-role.spec.ts:60 — Technician can create a machine — toHaveURL never settles on /m/TC7283
Also seen: machine-with-invite.spec.ts:29, public-reporting-extended.spec.ts:121, issue-list-extended.spec.ts:175, invite-signup.spec.ts:130, form-resets.spec.ts, status-overhaul.spec.ts:24

These look like the same anti-pattern (hardcoded per-assertion timeouts below the CI default) repeated across specs, but I didn't diff every file this week — filing a follow-up bead to sweep the E2E suite for the same undercut-timeout pattern is worth doing separately rather than folding into this PR.

CI infra flake (not a test): supabase/setup-cli hit a GitHub API rate limit twice this week (runs 28714316725, 28633842520) — Failed to resolve latest Supabase CLI release: rate limit exceeded. Not a test flake, but worth knowing about if CI setup jobs start failing more often.

Dependabot failures this week were not genuine breaks: both the ratchet PR and a TS7-nightly-bump PR failed due to the flakes above (owner-picker E2E, and the Supabase CLI rate limit respectively) — neither dependency bump actually caused an incompatibility.

Cancelled main-branch runs this week (4 of them) were confirmed as concurrency-group cancellations from rapid successive pushes, not test failures — no assertion failed before cancellation in any of them.

ci-legacy.yml is dormant — only 2 runs total, both from 2025-07-20, not a factor in current CI.

Test plan

pnpm exec eslint e2e/full/machine-owner-picker.spec.ts
pnpm exec prettier --check e2e/full/machine-owner-picker.spec.ts
pnpm run typecheck
pnpm run test (full unit suite, 1306 tests)
Watch this PR's own CI run of machine-owner-picker.spec.ts across all projects (chromium/firefox/Mobile Chrome/Mobile Safari) — first live exercise of the fix

Generated by Claude Code

… to member Weekly CI flaky-test scan (2026-07-05) flagged this as the #1 flakiest test: it showed up as Playwright "flaky" (fail-then-pass-on-retry) on two main-branch runs (Mobile Safari + Mobile Chrome), and as a hard 3/3 non-recovering failure (original + 2 retries) on a dependabot PR run — all at the same assertion. Root cause: playwright.config.ts sets a CI-aware `expect.timeout` (30s in CI, 10s locally), but this spec hardcoded `{ timeout: 15000 }` on exactly the assertions that were flaking (the promote-dialog heading and the post-confirm redirect). That override undercut the environment's own default by half, right where Mobile Safari/Mobile Chrome plus parallel-worker contention need the most margin — explaining both the retry-recoverable flakes (transient slowness under ~15s but under 30s) and the one run where sustained CI load pushed every attempt past even the retries. Fix: drop the hardcoded overrides so these assertions inherit the config's own CI-aware timeout instead of a fixed value that quietly shrinks it. Verified: typecheck, eslint, prettier, and the full unit suite (1306 tests) pass. Could not exercise the spec live in this sandbox (no Docker daemon available for the local Supabase stack) — CI is the first live run of the fix; flagging that limitation rather than claiming an E2E run that didn't happen. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013LzqEJCLCBB3Nk2HndA4gd

vercel · 2026-07-05T16:20:36Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
pin-point	Ready	Preview, Comment	Jul 5, 2026 4:21pm

Copilot

Pull request overview

This PR reduces CI flakiness in the “Machine Owner Picker — promote-dialog journeys (PP-6oi)” E2E spec by removing per-assertion hardcoded timeouts that were shorter than the repo’s CI-aware Playwright expect.timeout, allowing the assertions to inherit the configured defaults.

Changes:

Removed { timeout: 15000 } overrides from promote-dialog visibility assertions so they inherit the CI-aware expect.timeout.
Removed { timeout: 15000 } override from the post-confirm redirect (toHaveURL) assertion for the same reason.
Expanded inline comments explaining why the assertions should rely on global Playwright timeouts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    // Promote dialog should appear after the server action round-trip +
+    // React useEffect cycle. Use the config's own CI-aware `expect.timeout`
+    // (30s in CI, 10s locally) instead of a hardcoded value — a fixed 15s
+    // override here previously undercut the CI default by half, right where
+    // Mobile Safari/Mobile Chrome + parallel-worker contention need the most
+    // margin (PP flaky-test report 2026-07-05: this assertion was the #1
+    // flakiest test in CI over the prior week, including one run where it
+    // failed all 3 attempts — original + 2 retries — at the 15s mark).


Copilot AI review requested due to automatic review settings July 5, 2026 16:20

Copilot started reviewing on behalf of timothyfroehlich July 5, 2026 16:20 View session

vercel Bot deployed to Preview July 5, 2026 16:21 View deployment

Copilot AI reviewed Jul 5, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(flaky): promote dialog confirm creates machine and promotes guest to member#1602

fix(flaky): promote dialog confirm creates machine and promotes guest to member#1602
timothyfroehlich wants to merge 1 commit into
mainfrom
claude/vigilant-franklin-bccoi5

timothyfroehlich commented Jul 5, 2026

Uh oh!

vercel Bot commented Jul 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

timothyfroehlich commented Jul 5, 2026

Summary

Root-cause analysis (the fix)

Other recurring flaky signals seen this week (not fixed in this PR — same undercut-timeout pattern, logged as follow-up)

Test plan

Uh oh!

vercel Bot commented Jul 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vercel Bot commented Jul 5, 2026 •

edited

Loading