👷 Add pinned Firefox 119 + WebKit 17.4 e2e + Chromium 120 test runner without BrowserStack by thomas-lebeau · Pull Request #4560 · DataDog/browser-sdk

thomas-lebeau · 2026-05-06T05:50:42Z

Motivation

Replicate the BrowserStack Firefox 119 + WebKit 17.4 coverage locally so we can keep running pinned-browser e2e tests without BrowserStack, and lay the groundwork for a CI eval loop that iterates on GitLab jobs autonomously.

Changes

Add e2e-pinned GitLab job and test:e2e:pinned script driven by a new playwright.pinned.config.ts.
Run the pinned matrix via Playwright 1.40.1 run-server plus a small translation proxy (test/e2e/scripts/pinned-proxy.mjs) so the current 1.58 client can talk to the older server.
Pre-install Playwright browsers (current Chromium + pinned FF 119 / WK 17.4) in the CI image; bump CURRENT_CI_IMAGE to 106.
Use the test page's own origin in flushEvents so the flush works for pages served from custom domains (WebKit fix).
Skip iframe-without-src session cookie tests on Firefox.
Document the autonomous CI eval loop approach in APPROACH.md.

Test instructions

yarn test:e2e:pinned:init then yarn test:e2e:pinned locally to run Firefox 119 + WebKit 17.4.
Verify the e2e-pinned job runs green in GitLab on this branch.

Checklist

Tested locally
Tested on staging
Added unit tests for this change.
Added e2e/integration tests for this change.
Updated documentation and/or relevant AGENTS.md file

datadog-official · 2026-05-06T06:11:26Z

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 1 Test failed

microfrontend › RUM › with beforeSend › expose handling stack for xhr requests › async from microfrontend.scenario.ts

(Fix with Cursor)

microfrontend.scenario.ts:58:17 async

[firefox-pinned] › microfrontend.scenario.ts:58:17 › microfrontend › RUM › with beforeSend › expose handling stack for xhr requests › async 

    TypeError: expect(received).toMatch(expected)

    Matcher error: received value must be a string

    Received has value: undefined

...

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 77.49% (+0.01%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: af7e99a | Docs | Datadog PR Page | Give us feedback!}

cit-pr-commenter-54b7da · 2026-05-06T06:12:05Z

Bundles Sizes Evolution

📦 Bundle Name	Base Size	Local Size	𝚫	𝚫%	Status
Rum	N/A	169.22 KiB	N/A	N/A	N/A
Rum Profiler	N/A	6.10 KiB	N/A	N/A	N/A
Rum Recorder	N/A	21.23 KiB	N/A	N/A	N/A
Logs	N/A	54.56 KiB	N/A	N/A	N/A
Rum Slim	N/A	127.56 KiB	N/A	N/A	N/A
Worker	N/A	22.99 KiB	N/A	N/A	N/A

🚀 CPU Performance

Action Name	Base CPU Time (ms)	Local CPU Time (ms)	𝚫%
addglobalcontext	N/A	0.0018	N/A
addaction	N/A	0.0106	N/A
adderror	N/A	0.0106	N/A
addtiming	N/A	0.0005	N/A
startview	N/A	0.0115	N/A
startstopsessionreplayrecording	N/A	0.0007	N/A
logmessage	N/A	0.0146	N/A

🧠 Memory Performance

Action Name	Base Memory Consumption	Local Memory Consumption	𝚫
addglobalcontext	N/A	38.32 KiB	N/A
addaction	N/A	62.45 KiB	N/A
addtiming	N/A	38.05 KiB	N/A
adderror	N/A	69.23 KiB	N/A
startstopsessionreplayrecording	N/A	44.78 KiB	N/A
startview	N/A	482.18 KiB	N/A
logmessage	N/A	53.89 KiB	N/A

🔗 RealWorld

…rStack - Add `playwright.pinned.config.ts` that boots a Playwright 1.40.1 `run-server` (which bundles FF 119 / WK 17.4) plus a translation proxy, and connects the current 1.58 client over wsEndpoint. - Add `scripts/pinned-proxy.mjs` to bridge the JSON-RPC schema gap between Playwright 1.58 and 1.40 (UA spoofing for the version check, patches to __create__ initializers and a few command params). - Add `test:e2e:pinned:init` / `test:e2e:pinned` scripts to install the pinned browsers locally and run the pinned matrix. - Ignore the local `test/e2e/.pinned-browsers/` install directory.

In WebKit, `beforeunload` doesn't always fire when navigating to a different domain. Specifically, navigating from `localhost` to `foo.localhost` or between `localhost` and `127.0.0.1` silently skips the event, so the SDK never flushes. Tests using `withHostName()` were affected. Fix by navigating to `/flush` on the same origin when the current page is served by the base server, and falling back to the base server origin otherwise.

Firefox does not allow setting cookies from iframes without src, so the SDK won't start and these tests are not meaningful there.

- rename test/e2e/scripts/pinned-proxy.mjs to pinnedProxy.ts - add ws and @types/ws to root devDependencies (was an unlisted import) - add types for JSON-RPC messages, RawData handling, and header forwarding - update playwright.pinned.config.ts to invoke `node pinnedProxy.ts`

Without this directive, GitLab relies on the project-level "Auto-cancel redundant pipelines" setting, which is currently not effective for this project. Explicitly enabling `auto_cancel.on_new_commit: interruptible` ensures interruptible jobs (e.g. unit-bs, e2e-bs queued behind the browserstack resource group) are cancelled when a new commit lands on the same branch.

The previous check allowed a margin of 2 for Firefox, but Firefox async stack traces can exceed that margin. Instead of continuously adjusting the margin, we drop the upper bound assertion entirely.

thomas-lebeau · 2026-05-07T12:26:55Z


        expect((await findSessionCookie(browserContext))?.aid).toEqual(anonymousId)
-
-        expect(true).toBeTruthy()


In WebKit, the dd_cookie_test_* probe cookie (written by areCookiesAuthorized) lingers in the browser's in-memory store after deletion, appearing before the real _dd_s_v2 session cookie. Tests reading cookies[0] or asserting the full cookie array shape would fail because they hit the probe cookie instead.

Co-authored-by: Benoit Zugmeyer <benoit.zugmeyer@datadoghq.com>

Firefox renders the inner setTimeout callback as `window.RUM_INIT/<` instead of `<anonymous>` with the npm setup, because the test's rumInit is wrapped in a `window.RUM_INIT` arrow.

- install Chromium via the pinned Playwright 1.40.1 in Dockerfile and test:e2e:init - add chromium-pinned project (Chrome 120) to playwright config

Two RUM tests (rum/actions: "associate a long tasks to its action", microfrontend: "LOAf should have service and version from source code context") rely on PerformanceLongAnimationFrameTiming, which shipped in Chrome 123 and is not available in chromium-pinned (Chrome 120). Replace the previous "browserName !== 'chromium'" gate with a runtime feature check via PerformanceObserver.supportedEntryTypes — also covers non-Chromium browsers without an explicit version assumption.

…vert after)" This reverts commit f942810.

The Playwright 1.40 server used by the pinned Chromium runtime never emits service-worker-sourced console events on any channel, so the NO_SESSION_WARNING check would always fail there. Detect the pinned runtime via project metadata (presence of `version`) and skip the assertion, mirroring the existing Firefox carve-out. - add `isPinnedRuntime` helper based on project metadata - make `os`/`osVersion` optional in BrowserConfiguration (pinned projects don't set them) and apply `satisfies BrowserConfiguration` to project metadata for type safety

- A bare `yarn test:e2e` now runs only the four non-pinned projects, so a fresh checkout that hasn't run `yarn test:e2e:init` doesn't fail trying to launch pinned browsers it doesn't have. - Pinned projects (and their `run-server` + proxy web servers) only surface when `--project=...` is passed; CI already does this. - Tighten the SW-console comment in the no-session warning assertion.

Earlier on this branch yarn.lock drifted with many unintended downgrades (esbuild 0.27.7→0.27.3, @babel/helpers 7.29.2→7.28.6, @emnapi/* 1.10.0→1.9.2, @1natsu/wait-element 4.2.0→4.1.2, and others) — none of which match a package.json change in this PR. Restore yarn.lock from origin/v7 and rerun yarn install so the lockfile reflects only the real additions on this branch (ws, @types/ws). Net diff vs v7 is now +18/-16.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d34c6ba0ae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-05-08T08:11:54Z

+RUN npx -y playwright@${PLAYWRIGHT_VERSION} install --with-deps chromium
+
+# Pinned Playwright browsers: Chromium 120 + Firefox 119 + WebKit 17.4 (used by the e2e-pinned job)
+ARG PINNED_PLAYWRIGHT_VERSION=1.40.1
+RUN npx -y playwright@${PINNED_PLAYWRIGHT_VERSION} install --with-deps chromium firefox webkit


Replace npx with Yarn in Docker build

The root AGENTS.md package-manager rule says this repository uses Yarn workspaces and to never use npm or npx; these new Docker build steps invoke npx twice, so rebuilding the CI image now bypasses the repo's required package-manager workflow and can fail policy/reproducibility checks. Use the Yarn equivalent for invoking the pinned Playwright packages instead.

Useful? React with 👍 / 👎.

yes! but this is fine for now. we'll find an alternative in a separate PR

…config Filtering projects via process.argv broke worker processes: Error: Project "chromium-pinned" not found in the worker process. Make sure project name does not change. Worker processes don't inherit the test runner's argv, so getSelectedProjects() returned [] inside the worker, getProjects() stripped the pinned entries, and the worker couldn't find the project name assigned to it. ~700 retries later the job died. Project list must be deterministic across processes. Restore the original shape: always return all 7 projects; only the webServer boot decision (which runs only in the main process) gates on argv. The 'fresh-checkout boots pinned servers' UX concern from the review is best handled in docs, not config.

thomas-lebeau changed the title ~~test(e2e): pinned Firefox 119 + WebKit 17.4 runner without BrowserStack~~ 👷 Add pinned Firefox 119 + WebKit 17.4 e2e test runner without BrowserStack May 6, 2026

BenoitZugmeyer force-pushed the thomas.lebeau/no-bs branch 2 times, most recently from 3eb95df to 2a09222 Compare May 6, 2026 16:57

thomas-lebeau and others added 6 commits May 7, 2026 08:35

add APPROACH.md

861154f

👷 add test job + cache browsers

0670a79

✅ skip iframe-without-src session cookie tests on Firefox

4cd3a15

Firefox does not allow setting cookies from iframes without src, so the SDK won't start and these tests are not meaningful there.

REMOVEME don't stop at the first error

ca787b3

thomas-lebeau force-pushed the thomas.lebeau/no-bs branch from b0eef36 to df42b80 Compare May 7, 2026 06:40

thomas-lebeau force-pushed the thomas.lebeau/no-bs branch from df42b80 to 6918c2e Compare May 7, 2026 07:09

thomas-lebeau and others added 3 commits May 7, 2026 09:12

✅ remove upper bound assertion on stack trace line count

add5625

The previous check allowed a margin of 2 for Firefox, but Firefox async stack traces can exceed that margin. Instead of continuously adjusting the margin, we drop the upper bound assertion entirely.

✅ skip partitioned cookie test on webkit browsers

10d1b27

thomas-lebeau commented May 7, 2026

View reviewed changes

BenoitZugmeyer force-pushed the thomas.lebeau/no-bs branch 2 times, most recently from c857101 to 6767271 Compare May 7, 2026 13:01

BenoitZugmeyer force-pushed the thomas.lebeau/no-bs branch from 6767271 to 044b204 Compare May 7, 2026 13:17

thomas-lebeau and others added 2 commits May 7, 2026 15:40

👷 parallelize e2e job across browsers via matrix (#4572)

5cfb9ff

Co-authored-by: Benoit Zugmeyer <benoit.zugmeyer@datadoghq.com>

✅ accept Firefox-named frames in pre-init error stack assertion

c1f8226

Firefox renders the inner setTimeout callback as `window.RUM_INIT/<` instead of `<anonymous>` with the npm setup, because the test's rumInit is wrapped in a `window.RUM_INIT` arrow.

BenoitZugmeyer force-pushed the thomas.lebeau/no-bs branch from 8f31454 to c1f8226 Compare May 7, 2026 14:37

👷 add pinned Chromium 120 to e2e browser matrix

37289e1

- install Chromium via the pinned Playwright 1.40.1 in Dockerfile and test:e2e:init - add chromium-pinned project (Chrome 120) to playwright config

thomas-lebeau force-pushed the thomas.lebeau/no-bs branch from f2665a4 to 37289e1 Compare May 7, 2026 14:50

thomas-lebeau changed the title ~~👷 Add pinned Firefox 119 + WebKit 17.4 e2e test runner without BrowserStack~~ 👷 Add pinned Firefox 119 + WebKit 17.4 e2e + Chromium 120 test runner without BrowserStack May 7, 2026

thomas-lebeau added 2 commits May 7, 2026 18:44

👷 bump CI image to 107

aa1c3bc

thomas-lebeau added 5 commits May 8, 2026 08:12

🔍 DEBUG: trace pinned-proxy SW traffic in chromium-pinned (revert after)

f942810

Revert "🔍 DEBUG: trace pinned-proxy SW traffic in chromium-pinned (re…

491888f

…vert after)" This reverts commit f942810.

thomas-lebeau marked this pull request as ready for review May 8, 2026 08:08

thomas-lebeau requested review from a team as code owners May 8, 2026 08:08

chatgpt-codex-connector Bot reviewed May 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

👷 Add pinned Firefox 119 + WebKit 17.4 e2e + Chromium 120 test runner without BrowserStack#4560

👷 Add pinned Firefox 119 + WebKit 17.4 e2e + Chromium 120 test runner without BrowserStack#4560
thomas-lebeau wants to merge 22 commits intov7from
thomas.lebeau/no-bs

thomas-lebeau commented May 6, 2026

Uh oh!

datadog-official Bot commented May 6, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

Uh oh!

cit-pr-commenter-54b7da Bot commented May 6, 2026 •

edited

Loading

Uh oh!

thomas-lebeau May 7, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Uh oh!

thomas-lebeau May 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		expect((await findSessionCookie(browserContext))?.aid).toEqual(anonymousId)

		expect(true).toBeTruthy()

Conversation

thomas-lebeau commented May 6, 2026

Motivation

Changes

Test instructions

Checklist

Uh oh!

datadog-official Bot commented May 6, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

cit-pr-commenter-54b7da Bot commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundles Sizes Evolution

Uh oh!

thomas-lebeau May 7, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot May 8, 2026

Choose a reason for hiding this comment

Uh oh!

thomas-lebeau May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-official Bot commented May 6, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

cit-pr-commenter-54b7da Bot commented May 6, 2026 •

edited

Loading