CONSOLE-5196: Refresh admin auth mid-run to prevent session expiry cascade - #16877
CONSOLE-5196: Refresh admin auth mid-run to prevent session expiry cascade#16877shahsahil264 wants to merge 1 commit into
Conversation
…scade Auth storageState is created once at test start. On CI with 300+ tests and 2 workers (~50 min runtime), the OAuth session can expire mid-run, causing every subsequent test to fail with an OAuth redirect in warmupSPA. Split test packages into early (smoke, console) and late (dev-console, helm, knative, olm, topology, webterminal) phases with a re-login between them. The admin-auth-refresh project re-authenticates and overwrites kubeadmin.json before late packages start. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
@shahsahil264: This pull request references CONSOLE-5196 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Skipping CI for Draft Pull Request. |
WalkthroughChangesPlaywright authentication flow
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant Playwright
participant AuthRefreshSetup
participant WebConsole
participant StorageState
participant LatePackages
Playwright->>AuthRefreshSetup: run admin-auth-refresh setup
AuthRefreshSetup->>WebConsole: perform kubeadmin login
WebConsole-->>AuthRefreshSetup: authenticated browser context
AuthRefreshSetup->>StorageState: save kubeadmin storage state
StorageState-->>LatePackages: provide adminStorageState
LatePackages->>WebConsole: run package tests
Suggested reviewers: 🚥 Pre-merge checks | ✅ 15✅ Passed checks (15 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: shahsahil264 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/pipeline required |
|
Scheduling tests matching the |
|
@rhamilto Can you PTAL! |
|
/test e2e-playwright |
|
@shahsahil264: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Analysis / Root cause:
Playwright CI runs all ~300 e2e tests using a single
storageState(saved OAuth session cookies) created once at the start of the run. The OpenShift OAuth session has a fixed TTL, and with 300+ tests running across 2 workers (~50 min total runtime), the session can expire mid-run — typically around test ~220 when dev-console tests begin.When the session expires:
warmupSPAnavigates to/but the console backend rejects the stale cookieoauth-openshift.../oauth/authorizethenauth/callbackpage-headingpage-headingis never found so the test failsstorageStatefile on disk is never updated, so every subsequent test loads the same stale cookies causing a cascade failure (10-20+ tests fail in a row)This has caused repeated CI failures across multiple migration PRs (#16741, #16751), where all dev-console, topology, and webterminal tests fail with identical
warmupSPAOAuth redirect errors — despite the test code being correct (proven by passing on runs where auth holds).Solution description:
Split the test suite into two phases with a re-authentication step between them:
smoke,console— depend onadmin-auth(original login)admin-auth-refresh— depends on Phase 1 completing, re-runs the login flow and overwriteskubeadmin.jsonwith a fresh sessiondev-console,helm,knative,olm,topology,webterminal— depend onadmin-auth-refreshThis ensures that late-running test packages always start with a fresh OAuth session, eliminating the cascade failure pattern.
Changes:
playwright.config.ts: Splitpackagesarray intoearlyPackagesandlatePackages, addedadmin-auth-refreshproject between them with correct dependency orderinge2e/setup/admin-auth-refresh.setup.ts: New setup file that performs the same login asadmin-auth.setup.tsto refresh thestorageStateScreenshots / screen recording:
N/A (CI infrastructure change, no UI changes)
Test setup:
Tested locally against a live OCP 5.0 cluster. Verified:
admin-authcreates initialkubeadmin.jsonadmin-auth-refreshre-logs in and overwriteskubeadmin.jsondev-consoletests run successfully with the refreshed session (10/10 passed, zero auth redirect failures)Test cases:
Browser conformance:
N/A (no UI changes)
Additional info:
This complements PR #16780 which reduced runtime by setting
WORKERS=2. That fix brought runtime from ~2h to ~50min, but session expiry still occurs non-deterministically depending on the ephemeral cluster OAuth token lifetime. This PR eliminates the root cause by refreshing auth before the second half of the suite runs.