Skip to content

CONSOLE-5196: Refresh admin auth mid-run to prevent session expiry cascade - #16877

Open
shahsahil264 wants to merge 1 commit into
openshift:mainfrom
shahsahil264:CONSOLE-5196-auth-refresh
Open

CONSOLE-5196: Refresh admin auth mid-run to prevent session expiry cascade#16877
shahsahil264 wants to merge 1 commit into
openshift:mainfrom
shahsahil264:CONSOLE-5196-auth-refresh

Conversation

@shahsahil264

@shahsahil264 shahsahil264 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Analysis / Root cause:

Playwright CI runs all ~300 e2e tests using a single storageState (saved OAuth session cookies) created once at the start of the run. The OpenShift OAuth session has a fixed TTL, and with 300+ tests running across 2 workers (~50 min total runtime), the session can expire mid-run — typically around test ~220 when dev-console tests begin.

When the session expires:

  1. warmupSPA navigates to / but the console backend rejects the stale cookie
  2. The browser is redirected to oauth-openshift.../oauth/authorize then auth/callback
  3. The page lands on the last-visited page (Topology) instead of a page with page-heading
  4. page-heading is never found so the test fails
  5. The storageState file on disk is never updated, so every subsequent test loads the same stale cookies causing a cascade failure (10-20+ tests fail in a row)

This has caused repeated CI failures across multiple migration PRs (#16741, #16751), where all dev-console, topology, and webterminal tests fail with identical warmupSPA OAuth redirect errors — despite the test code being correct (proven by passing on runs where auth holds).

Solution description:

Split the test suite into two phases with a re-authentication step between them:

  • Phase 1 (early packages): smoke, console — depend on admin-auth (original login)
  • Re-auth: admin-auth-refresh — depends on Phase 1 completing, re-runs the login flow and overwrites kubeadmin.json with a fresh session
  • Phase 2 (late packages): dev-console, helm, knative, olm, topology, webterminal — depend on admin-auth-refresh

This ensures that late-running test packages always start with a fresh OAuth session, eliminating the cascade failure pattern.

Changes:

  • playwright.config.ts: Split packages array into earlyPackages and latePackages, added admin-auth-refresh project between them with correct dependency ordering
  • e2e/setup/admin-auth-refresh.setup.ts: New setup file that performs the same login as admin-auth.setup.ts to refresh the storageState

Screenshots / screen recording:
N/A (CI infrastructure change, no UI changes)

Test setup:
Tested locally against a live OCP 5.0 cluster. Verified:

  • admin-auth creates initial kubeadmin.json
  • admin-auth-refresh re-logs in and overwrites kubeadmin.json
  • dev-console tests run successfully with the refreshed session (10/10 passed, zero auth redirect failures)

Test cases:

  • CI run completes without auth expiry cascade in late test packages
  • All existing tests continue to pass (no behavioral change to test execution, only dependency ordering)

Browser conformance:
N/A (no UI changes)

Additional info:
This complements PR #16780 which reduced runtime by setting WORKERS=2. That fix brought runtime from ~2h to ~50min, but session expiry still occurs non-deterministically depending on the ephemeral cluster OAuth token lifetime. This PR eliminates the root cause by refreshing auth before the second half of the suite runs.

…scade

Auth storageState is created once at test start. On CI with 300+ tests
and 2 workers (~50 min runtime), the OAuth session can expire mid-run,
causing every subsequent test to fail with an OAuth redirect in warmupSPA.

Split test packages into early (smoke, console) and late (dev-console,
helm, knative, olm, topology, webterminal) phases with a re-login
between them. The admin-auth-refresh project re-authenticates and
overwrites kubeadmin.json before late packages start.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 29, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@shahsahil264: This pull request references CONSOLE-5196 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set.

Details

In response to this:

…scade

Auth storageState is created once at test start. On CI with 300+ tests and 2 workers (~50 min runtime), the OAuth session can expire mid-run, causing every subsequent test to fail with an OAuth redirect in warmupSPA.

Split test packages into early (smoke, console) and late (dev-console, helm, knative, olm, topology, webterminal) phases with a re-login between them. The admin-auth-refresh project re-authenticates and overwrites kubeadmin.json before late packages start.

Analysis / Root cause:

Solution description:

Screenshots / screen recording:

Test setup:

Test cases:

Browser conformance:

  • Chrome
  • Firefox
  • Safari (or Epiphany on Linux)

Additional info:

Reviewers and assignees:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2026
@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Changes

Playwright authentication flow

Layer / File(s) Summary
Kubeadmin storage state refresh
frontend/e2e/setup/admin-auth-refresh.setup.ts
Adds conditional kubeadmin login using environment-configured console settings and saves the authenticated storage state.
Early and late project dependencies
frontend/playwright.config.ts
Splits package projects into early and late groups and routes late projects through the new admin-auth-refresh setup project.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Playwright
  participant AuthRefreshSetup
  participant WebConsole
  participant StorageState
  participant LatePackages
  Playwright->>AuthRefreshSetup: run admin-auth-refresh setup
  AuthRefreshSetup->>WebConsole: perform kubeadmin login
  WebConsole-->>AuthRefreshSetup: authenticated browser context
  AuthRefreshSetup->>StorageState: save kubeadmin storage state
  StorageState-->>LatePackages: provide adminStorageState
  LatePackages->>WebConsole: run package tests
Loading

Suggested reviewers: rhamilto

🚥 Pre-merge checks | ✅ 15
✅ Passed checks (15 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The PR adds one static Playwright title and no Ginkgo It/Describe/Context/When names with dynamic data in the touched files.
Test Structure And Quality ✅ Passed No Ginkgo test code was changed; this PR only adds/updates Playwright setup/config, so the Ginkgo-specific quality check is not applicable.
Microshift Test Compatibility ✅ Passed PASS: The PR only adds Playwright setup/config changes; no new Ginkgo It/Describe/Context/When tests or MicroShift-unsafe APIs/features were added.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo tests were added; the PR only changes Playwright setup/config and contains no multi-node or SNO-specific assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed PASS: Only Playwright test setup/config changed; no deployment manifests, controllers, or topology/node-scheduling logic were added.
Ote Binary Stdout Contract ✅ Passed PR only changes Playwright setup/config; no main/init/TestMain/suite-level stdout writes or console/log calls were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed No new Ginkgo tests or IPv4/external-network assumptions found; the added Playwright setup logs into the cluster console only.
No-Weak-Crypto ✅ Passed The diff only adds Playwright auth/setup flow; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons appear.
Container-Privileges ✅ Passed No container/K8s manifests were changed, and the edited TS files contain none of the flagged privilege settings.
No-Sensitive-Data-In-Logs ✅ Passed No logging in the changed files; credentials are used for login only and not printed or emitted.
Title check ✅ Passed The title is concise, Jira-prefixed, and accurately summarizes the auth-refresh change.
Description check ✅ Passed All required sections are present and substantive; only the reviewers/assignees section is left blank.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: shahsahil264
Once this PR has been reviewed and has the lgtm label, please assign vojtechszocs for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@shahsahil264
shahsahil264 marked this pull request as ready for review July 29, 2026 14:48
@shahsahil264

Copy link
Copy Markdown
Contributor Author

/pipeline required

@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-gcp-console
/test e2e-playwright

@shahsahil264

Copy link
Copy Markdown
Contributor Author

@rhamilto Can you PTAL!
Thanks

@openshift-ci
openshift-ci Bot requested review from jhadvig and rhamilto July 29, 2026 14:53
@shahsahil264 shahsahil264 changed the title CONSOLE-5196: Refresh admin auth mid-run to prevent session expiry ca… CONSOLE-5196: Refresh admin auth mid-run to prevent session expiry cascade Jul 29, 2026
@rhamilto

Copy link
Copy Markdown
Member

/test e2e-playwright

@openshift-ci

openshift-ci Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

@shahsahil264: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants