Skip to content

test(e2e): raise full-e2e onboard budget to 205s for cold-build variance (#6002) #6663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cv merged 2 commits into from
Jul 10, 2026
Merged

test(e2e): raise full-e2e onboard budget to 205s for cold-build variance (#6002) #6663

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e26907e
test(e2e): raise full-e2e onboard budget to 205s for cold-build varia…
prekshivyas Jul 10, 2026
eb5e70f
test(e2e): register full-e2e mock-parity entry (#6002)
prekshivyas Jul 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion test/e2e/live/full-e2e.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,15 @@ import { extractOpenClawAgentPayloadText } from "./agent-turn-latency-helpers.ts
const SANDBOX_NAME = process.env.NEMOCLAW_SANDBOX_NAME ?? "e2e-full";
const LIVE_TIMEOUT_MS = 50 * 60_000;
const FIRST_TURN_TIMEOUT_MS = 240_000;
const ONBOARD_BUDGET_SECS = 180;
// Cold-path acceptance budget for install + BuildKit image build + gateway
// health + first hosted agent turn (#6002, #6265). The dominant term is the
// cold BuildKit image build, which swings run-to-run with Docker Hub pull
// speed and hosted-runner I/O. Observed post-#6265 onboard phase alone ranged
// 163s (pass) to 173s (fail) on identical `main` commits — a ~10s swing — so
// the original 180s cap left only ~7s of headroom and tipped over on slow
// builds. Raised to 205s to absorb build-phase variance while still catching
// gross onboard regressions (first hosted turn itself is only ~5-8s).
const ONBOARD_BUDGET_SECS = 205;
Comment on lines +36 to +44

@coderabbitai coderabbitai Bot Jul 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Synchronize the documented cold-onboard contract.

test/e2e/README.md still describes the hard full-e2e acceptance limit as 180 seconds, while this test now enforces and reports 205 seconds. Update the documentation and any related CI references in this change so operators do not receive conflicting limits.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/e2e/live/full-e2e.test.ts` around lines 36 - 44, Update
test/e2e/README.md and related CI configuration or documentation references for
the full-e2e cold-onboard acceptance limit from 180 seconds to 205 seconds,
matching the ONBOARD_BUDGET_SECS constant in full-e2e.test.ts. Search for all
references to the 180-second full-e2e limit and synchronize them with the new
documented contract.

Source: Path instructions

const MAX_SILENCE_SECS = 60;
const EXPECTED_FIRST_REPLY = "NEMOCLAW_E2E_READY_6002";
const MEASURE_COLD_ONBOARD = process.env.E2E_TARGET_ID === "full-e2e";
Expand Down
6 changes: 6 additions & 0 deletions test/e2e/mock-parity.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,12 @@
"$comment": "SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: Apache-2.0",
"version": 1,
"entries": [
{
"live": "test/e2e/live/full-e2e.test.ts",
"fast": [
"test/e2e/support/onboard-performance.test.ts"
]
},
{
"live": "test/e2e/live/onboard-repair.test.ts",
"fast": [
Expand Down