Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/skills/nemoclaw-maintainer-cut-release-tag/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,7 @@ npm run release:e2e-evidence -- \
The preflight derives every required execution and these dispatch groups from the candidate workflow:

- `defaultSuite`: full mode, which includes the default-enabled suite and `Exact staging Brev Launchable`;
- `parallelExplicit`: explicit-only selectors that require neither the protected Launchable E2E job nor runner confirmation; and
- `parallelExplicit`: explicit-only selectors that require neither the Launchable E2E job nor runner confirmation; and
- `conditional`: Jetson or another lane that must not queue until its authoritative runner inventory is confirmed online.

First feed applicable existing runs for the candidate SHA into the ledger.
Expand Down Expand Up @@ -368,7 +368,7 @@ If the Announcement is valid, return its URL with the release artifacts and mark

- Plan generation fails: fix the named precondition, then regenerate the plan.
- Planned changelog entry is missing or malformed: stop before plan generation and run the pre-tag `nemoclaw-contributor-update-docs` workflow. Use post-release recovery only when the tag already exists.
- Full-mode E2E waits for protected-environment approval: keep the run pending until a maintainer approves or rejects the job.
- Full-mode E2E waits in the Launchable concurrency queue: keep the run pending until the earlier Launchable E2E job finishes.
- Full-mode E2E ran for another SHA or skipped `Exact staging Brev Launchable`: reject the run and dispatch full mode for the plan candidate SHA.
- Launchable E2E or cleanup evidence is missing or invalid: reject the run. Do not infer Launchable E2E success from the workflow conclusion.
- `origin/main` moved after plan generation: regenerate the plan and ask for the new confirmation phrase.
Expand Down
9 changes: 6 additions & 3 deletions .agents/skills/nemoclaw-maintainer-e2e/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Do not substitute local `npm run test:live-e2e` unless the maintainer explicitly
| “run pre-tag full E2E” | Full | empty | `true` |
| “run release-candidate E2E” | Full | empty | `true` |

A generic E2E request must not authorize the protected Brev path.
A generic E2E request must not authorize the Brev Launchable path.
Do not infer full mode from words such as “all” or “complete.”
Ask for clarification only when the request contains conflicting mode phrases.

Expand Down Expand Up @@ -97,7 +97,10 @@ gh workflow run .github/workflows/e2e.yaml \
Do not set `jobs=staging-brev-launchable` for full mode.
Empty `jobs` and `targets` select the default suite.
The boolean input adds the Launchable E2E job to that same run.
The protected environment can require approval for Launchable and full runs.
The trusted `main` workflow verifies that the dispatching and rerunning actors have
repository `maintain` or `admin` permission before the Launchable path's source
checkout. That role check is the authorization.
Launchable and full runs do not require separate environment approval.

### Release Coverage Dispatch Group

Expand Down Expand Up @@ -173,7 +176,7 @@ Wait for completion:
gh run watch "$RUN_ID" --repo NVIDIA/NemoClaw --exit-status
```

Launchable and full modes can wait for protected-environment approval.
Launchable and full modes can wait in the non-cancelling Launchable concurrency queue.
Queued, waiting, or accepted dispatch state is not success.

## Verify the Result
Expand Down
44 changes: 41 additions & 3 deletions .github/workflows/e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,47 @@ jobs:
' <<< "$check_json" >/dev/null ||
{ echo "::error::controller check does not authorize this exact E2E run"; exit 1; }

- name: Authorize Launchable E2E maintainer dispatch
if: ${{ github.event_name == 'workflow_dispatch' && ((inputs.jobs == 'staging-brev-launchable' && inputs.targets == '') || (inputs.include_staging_brev_launchable && inputs.jobs == '' && inputs.targets == '')) }}
env:
ACTOR: ${{ github.actor }}
GITHUB_TOKEN: ${{ github.token }}
TRIGGERING_ACTOR: ${{ github.triggering_actor }}
shell: bash
run: |
set -euo pipefail

require_maintainer() {
local maintainer="$1"
if [[ ! "$maintainer" =~ ^[A-Za-z0-9-]{1,39}$ || "$maintainer" == -* || "$maintainer" == *- ]]; then
echo "::error::Launchable E2E actor is invalid" >&2
exit 1
fi

local permission_json
permission_json="$(curl --fail --silent --show-error --proto '=https' \
--header "Authorization: Bearer ${GITHUB_TOKEN}" \
--header "Accept: application/vnd.github+json" \
--header "X-GitHub-Api-Version: 2022-11-28" \
"https://api.github.com/repos/${GITHUB_REPOSITORY}/collaborators/${maintainer}/permission")"
if [[ "$(jq -r '.user.login // ""' <<< "$permission_json" | tr '[:upper:]' '[:lower:]')" != "$(tr '[:upper:]' '[:lower:]' <<< "$maintainer")" ]]; then
echo "::error::Launchable E2E permission response did not match the actor" >&2
exit 1
fi
case "$(jq -r '.role_name // ""' <<< "$permission_json")" in
maintain | admin) ;;
*)
echo "::error::Launchable E2E requires a repository maintainer or administrator" >&2
exit 1
;;
esac
}

require_maintainer "$ACTOR"
if [[ "${TRIGGERING_ACTOR,,}" != "${ACTOR,,}" ]]; then
require_maintainer "$TRIGGERING_ACTOR"
fi

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
repository: ${{ inputs.checkout_repository || github.repository }}
Expand Down Expand Up @@ -423,9 +464,6 @@ jobs:
if: ${{ github.repository == 'NVIDIA/NemoClaw' && github.ref == 'refs/heads/main' && github.event_name == 'workflow_dispatch' && ((inputs.jobs == 'staging-brev-launchable' && inputs.targets == '') || (inputs.include_staging_brev_launchable && inputs.jobs == '' && inputs.targets == '')) }}
runs-on: ubuntu-latest
timeout-minutes: 180
environment:
name: approve-brev-launchable-e2e
deployment: false
permissions:
contents: read
concurrency:
Expand Down
10 changes: 7 additions & 3 deletions test/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,9 +217,13 @@ A manual run with `include_staging_brev_launchable=true` and empty `jobs` and
`targets` selectors runs the default suite and the Launchable E2E job.
This is the full run required for pre-tag evidence. Each full dispatch uses
`github.run_id` in its workflow concurrency identity, so another full dispatch
cannot supersede it while it waits. The protected `staging-brev-launchable` job
uses the non-cancelling `staging-brev-launchable-cpu` group with `queue: max`,
so pending Launchable E2E runs remain queued instead of replacing one another.
cannot supersede it while it waits. The trusted `main` workflow dispatch
verifies that the dispatching and rerunning actors have repository `maintain` or
`admin` permission before the Launchable path's source checkout. That automatic
role check authorizes `staging-brev-launchable`; the job does not use GitHub
environment approval. The job uses the non-cancelling
`staging-brev-launchable-cpu` group with `queue: max`, so pending Launchable E2E
runs remain queued instead of replacing one another.

### Hosted-runner recovery

Expand Down
21 changes: 18 additions & 3 deletions test/e2e/support/e2e-workflow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,25 +39,40 @@ describe("e2e workflow boundary", () => {
expect(validateE2eWorkflowBoundary()).toEqual([]);
});

it("rejects staging Launchable protected-environment and secret-guard drift", () => {
it("rejects a Launchable environment gate, authorization drift, and secret-guard drift", () => {
const workflow = readWorkflow() as {
jobs: Record<
string,
{
if?: string;
environment?: Record<string, unknown>;
steps?: Array<{ env?: Record<string, string>; name?: string }>;
steps?: Array<{
env?: Record<string, string>;
name?: string;
run?: string;
uses?: string;
}>;
}
>;
};
const job = workflow.jobs["staging-brev-launchable"]!;
job.environment = { name: "unprotected" };
const prepare = job.steps!.find((step) => step.name === "Prepare the trusted lane")!;
prepare.env!.BREV_API_KEY = "${{ secrets.BREV_API_KEY }}";
const generateSteps = workflow.jobs["generate-matrix"]!.steps!;
const authorization = generateSteps.find(
(step) => step.name === "Authorize Launchable E2E maintainer dispatch",
)!;
delete authorization.env!.TRIGGERING_ACTOR;
authorization.run = authorization.run!.replace("maintain | admin", "write");
generateSteps.push(...generateSteps.splice(generateSteps.indexOf(authorization), 1));

expect(validateE2eWorkflow(workflow)).toEqual(
expect.arrayContaining([
"staging-brev-launchable must use its protected non-deployment environment",
"staging-brev-launchable must not use a GitHub environment",
"Launchable E2E maintainer authorization must bind TRIGGERING_ACTOR",
"step 'Authorize Launchable E2E maintainer dispatch' run script must include maintain | admin",
"Launchable E2E maintainer authorization must run before generate-matrix checkout",
"staging-brev-launchable BREV_API_KEY must use the trusted-run secret guard",
]),
);
Expand Down
8 changes: 7 additions & 1 deletion test/maintainer-e2e-skill.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,13 @@ describe("nemoclaw-maintainer-e2e workflow routing", () => {
expect(skill).toContain("deploy pre-release full E2E");
expect(skill).toContain("run pre-tag full E2E");
expect(skill).toContain("run release-candidate E2E");
expect(skill).toContain("must not authorize the protected Brev path");
expect(skill).toContain("must not authorize the Brev Launchable path");
expect(skill).toContain(
"Launchable and full runs do not require separate environment approval",
);
expect(skill).toContain(
"repository `maintain` or `admin` permission before the Launchable path's source",
);
expect(skill).not.toMatch(/variable (?:set|delete) NEMOCLAW_BREV_LAUNCHABLE_E2E_ENABLED/u);
});

Expand Down
54 changes: 50 additions & 4 deletions tools/e2e/workflow-boundary.mts
Original file line number Diff line number Diff line change
Expand Up @@ -4105,16 +4105,15 @@ function validateFullE2eConcurrency(errors: string[], workflow: WorkflowRecord):

function validateStagingBrevLaunchableJob(errors: string[], jobs: WorkflowRecord): void {
const job = asRecord(jobs["staging-brev-launchable"]);
const environment = asRecord(job.environment);
if (environment.name !== "approve-brev-launchable-e2e" || environment.deployment !== false) {
errors.push("staging-brev-launchable must use its protected non-deployment environment");
if (Object.hasOwn(job, "environment")) {
errors.push("staging-brev-launchable must not use a GitHub environment");
}
const trustedRun = "github.repository == 'NVIDIA/NemoClaw' && github.ref == 'refs/heads/main'";
if (
!stringValue(job.if).includes(trustedRun) ||
stringValue(job.if).includes("checkout_sha == ''")
) {
errors.push("staging-brev-launchable must allow only protected trusted-main dispatches");
errors.push("staging-brev-launchable must allow only trusted-main dispatches");
}
const expectedSelector =
"${{ github.repository == 'NVIDIA/NemoClaw' && github.ref == 'refs/heads/main' && github.event_name == 'workflow_dispatch' && ((inputs.jobs == 'staging-brev-launchable' && inputs.targets == '') || (inputs.include_staging_brev_launchable && inputs.jobs == '' && inputs.targets == '')) }}";
Expand All @@ -4123,6 +4122,53 @@ function validateStagingBrevLaunchableJob(errors: string[], jobs: WorkflowRecord
"staging-brev-launchable must run for its exact Launchable-only selection or an empty-selector full dispatch",
);
}
const generateMatrix = asRecord(jobs["generate-matrix"]);
const generateSteps = asSteps(generateMatrix.steps);
const authorization = requireJobStep(
errors,
"generate-matrix",
generateSteps,
"Authorize Launchable E2E maintainer dispatch",
);
const expectedAuthorizationSelector =
"${{ github.event_name == 'workflow_dispatch' && ((inputs.jobs == 'staging-brev-launchable' && inputs.targets == '') || (inputs.include_staging_brev_launchable && inputs.jobs == '' && inputs.targets == '')) }}";
if (authorization?.if !== expectedAuthorizationSelector) {
errors.push("Launchable E2E maintainer authorization must cover exact and full dispatches");
}
if (authorization?.shell !== "bash") {
errors.push("Launchable E2E maintainer authorization must use bash");
}
const authorizationEnv = asRecord(authorization?.env);
for (const [key, expected] of [
["ACTOR", "${{ github.actor }}"],
["GITHUB_TOKEN", "${{ github.token }}"],
["TRIGGERING_ACTOR", "${{ github.triggering_actor }}"],
] as const) {
if (authorizationEnv[key] !== expected) {
errors.push(`Launchable E2E maintainer authorization must bind ${key}`);
}
}
for (const required of [
"/collaborators/${maintainer}/permission",
"'.user.login // \"\"'",
"'.role_name // \"\"'",
"maintain | admin",
'require_maintainer "$ACTOR"',
'require_maintainer "$TRIGGERING_ACTOR"',
"requires a repository maintainer or administrator",
]) {
requireRunContains(errors, authorization, required);
}
const generateCheckout = generateSteps.find((step) =>
stringValue(step.uses).startsWith("actions/checkout@"),
);
if (
authorization &&
generateCheckout &&
generateSteps.indexOf(authorization) >= generateSteps.indexOf(generateCheckout)
) {
errors.push("Launchable E2E maintainer authorization must run before generate-matrix checkout");
}
const concurrency = asRecord(job.concurrency);
if (
concurrency.group !== "staging-brev-launchable-cpu" ||
Expand Down
Loading