Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 13 additions & 11 deletions .github/workflows/fix-flakes-prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ bots/PRs it disrupts — **not** by "has a tidy error message".
Pick a candidate whose failing `bot_name` OS matches yours so you can reproduce it.
Keep the ranked list — you fall back down it in step 2. Say why you picked the top one.
Once you have the target, use the "Generate a linked emoji run history" recipe in
`.claude/skills/playwright-test-results/SKILL.md`. Keep the Markdown output for the PR body.
`.claude/skills/playwright-test-results/SKILL.md`. Include the complete output in the PR body:
never trim it or remove green runs, and check the square count matches the per-run row count.

## 2. Check nobody's on it — and that it isn't already fixed

Expand All @@ -33,9 +34,8 @@ down your step-1 ranking, re-checking each; only stop once the whole shortlist i
before a fix landed, so a test fixed mid-window looks maximally bimodal (old fails + new
passes), which is exactly what floats it up step 1. Check whether a fix has landed since it
last flaked: find the commit its most recent failing run ran on, and look at what's changed
in that test's file since. A commit that plausibly targets the test or its feature — a
browser roll, or a fix to the test or the code it exercises — means the remaining DB fails
are from before the fix, so move on.
in that test's file since. Only move on if enough subsequent runs show the failures stopped;
a browser roll alone is not evidence of a fix.

## 3. Reproduce on this OS

Expand All @@ -59,15 +59,17 @@ are project-locked, so use the one matching the failing bot:
Other suites (electron `etest`, etc.): see `package.json` scripts.
Add `--repeat-each=N` to force a flake's flip.

**You can only reproduce what this OS reaches.** A failure you can't reach here is itself
evidence it's environment-specific — a skip candidate (step 6), not a blind-fix.
Keep any OS handling keyed on the *current* OS, never hardcoded.
**You can only reproduce what this OS and architecture reaches.** Match both from `bot_name`;
non-reproduction under lighter local load or on another architecture is not itself a skip candidate.
Keep any OS/architecture handling keyed on the *current* environment, never hardcoded.
Broader OS coverage comes from different agent runs on other OSes, not from you.

## 4. Fix — root cause or scoped skip

- **Tractable → fix the source.** Usually a test-side race: a missing `await`, waiting on the
wrong signal, an under-specified locator, leaked state. Fix the test (or the product bug).
- **Timeout → check the budget before skipping.** Compare passing durations and neighboring tests;
if working tests share an expensive operation near the timeout, prefer a scoped `it.slow()`.
- **Engine/OS-specific, unreachable here, or genuinely hard → scope a skip**, narrowed to the
exact failing condition — never a whole file or browser:

Expand All @@ -81,9 +83,9 @@ Broader OS coverage comes from different agent runs on other OSes, not from you.
flake-fighter rarely is). Never `skip` just to turn a red green.

**Verify locally — your PR gets no CI (step 6), so this is the only proof:** flake → re-run with
`--repeat-each` and confirm it's stable; skip → confirm it's skipped on the target config and
**still runs elsewhere**. Then run `npm run flint`. Record exactly what you ran — it goes in the
PR body.
`--repeat-each` and confirm it's stable; slow → confirm it still runs and passes; skip → confirm
it's skipped on the target config and **still runs elsewhere**. Then run `npm run flint`. Record
exactly what you ran — it goes in the PR body.

## 5. Pick a reviewer

Expand Down Expand Up @@ -121,4 +123,4 @@ Report what you committed, the reviewer, and why.

- **One target, one commit.** Don't batch.
- **Scoped skips only** — narrow to the failing condition, never a whole file or browser matrix.
- **No unverified PRs** — if you couldn't run it locally, prefer a documented fixme.
- **No unverified PRs** — if you cannot validate a scoped mitigation, make no commit.
11 changes: 6 additions & 5 deletions .github/workflows/fix-flakes.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,12 @@ jobs:
via the playwright-test-results skill. \`bot_name\` encodes the OS.

Write your answer to output/runner.txt, and nothing else there: exactly one of these
six runner labels, or the single word \`none\` if no OS is worth a run.
seven runner labels, or the single word \`none\` if no platform is worth a run.
ubuntu-22.04 ubuntu-22.04-arm ubuntu-24.04
macos-14-xlarge macos-15-xlarge windows-latest
Map a candidate's \`bot_name\` to the closest of those (e.g. \`*-macos-15-*\` ->
macos-15-xlarge). Explain your reasoning briefly in your final message.
macos-14-xlarge macos-15-large macos-15-xlarge windows-latest
Map \`*-macos-15-large\` to \`macos-15-large\` and \`*-macos-15-xlarge\` to
\`macos-15-xlarge\`; never substitute one architecture for another. Explain your
reasoning briefly in your final message.

${HINT:+Focus hint from the operator: $HINT}
EOF
Expand All @@ -102,7 +103,7 @@ jobs:
exit 0
fi
case "$runner" in
ubuntu-22.04|ubuntu-22.04-arm|ubuntu-24.04|macos-14-xlarge|macos-15-xlarge|windows-latest) ;;
ubuntu-22.04|ubuntu-22.04-arm|ubuntu-24.04|macos-14-xlarge|macos-15-large|macos-15-xlarge|windows-latest) ;;
*) echo "::error::Triage chose a non-allowlisted runner: '$runner'." >&2; exit 1 ;;
esac
echo "Triage chose runner: $runner"
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# 🎭 Playwright

[![npm version](https://img.shields.io/npm/v/playwright.svg)](https://www.npmjs.com/package/playwright) <!-- GEN:chromium-version-badge -->[![Chromium version](https://img.shields.io/badge/chromium-151.0.7922.19-blue.svg?logo=google-chrome)](https://www.chromium.org/Home)<!-- GEN:stop --> <!-- GEN:firefox-version-badge -->[![Firefox version](https://img.shields.io/badge/firefox-152.0.4-blue.svg?logo=firefoxbrowser)](https://www.mozilla.org/en-US/firefox/new/)<!-- GEN:stop --> <!-- GEN:webkit-version-badge -->[![WebKit version](https://img.shields.io/badge/webkit-26.5-blue.svg?logo=safari)](https://webkit.org/)<!-- GEN:stop --> [![Join Discord](https://img.shields.io/badge/join-discord-informational)](https://aka.ms/playwright/discord)
[![npm version](https://img.shields.io/npm/v/playwright.svg)](https://www.npmjs.com/package/playwright) <!-- GEN:chromium-version-badge -->[![Chromium version](https://img.shields.io/badge/chromium-151.0.7922.34-blue.svg?logo=google-chrome)](https://www.chromium.org/Home)<!-- GEN:stop --> <!-- GEN:firefox-version-badge -->[![Firefox version](https://img.shields.io/badge/firefox-152.0.4-blue.svg?logo=firefoxbrowser)](https://www.mozilla.org/en-US/firefox/new/)<!-- GEN:stop --> <!-- GEN:webkit-version-badge -->[![WebKit version](https://img.shields.io/badge/webkit-26.5-blue.svg?logo=safari)](https://webkit.org/)<!-- GEN:stop --> [![Join Discord](https://img.shields.io/badge/join-discord-informational)](https://aka.ms/playwright/discord)

## [Documentation](https://playwright.dev) | [API reference](https://playwright.dev/docs/api/class-playwright)

Expand Down Expand Up @@ -296,7 +296,7 @@ The [Playwright VS Code extension](https://marketplace.visualstudio.com/items?it

| | Linux | macOS | Windows |
| :--- | :---: | :---: | :---: |
| Chromium<sup>1</sup> <!-- GEN:chromium-version -->151.0.7922.19<!-- GEN:stop --> | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| Chromium<sup>1</sup> <!-- GEN:chromium-version -->151.0.7922.34<!-- GEN:stop --> | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| WebKit <!-- GEN:webkit-version -->26.5<!-- GEN:stop --> | :white_check_mark: | :white_check_mark: | :white_check_mark: |
| Firefox <!-- GEN:firefox-version -->152.0.4<!-- GEN:stop --> | :white_check_mark: | :white_check_mark: | :white_check_mark: |

Expand Down
Loading
Loading