Skip to content

Add hack/trigger-qe-tests.sh to trigger WMCO QE z-stream tests - #4324

Open
rrasouli wants to merge 1 commit into
openshift:masterfrom
rrasouli:add-trigger-qe-script
Open

Add hack/trigger-qe-tests.sh to trigger WMCO QE z-stream tests#4324
rrasouli wants to merge 1 commit into
openshift:masterfrom
rrasouli:add-trigger-qe-script

Conversation

@rrasouli

@rrasouli rrasouli commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Manual process to trigger WMCO QE z-stream/y-stream tests

Periodic CI jobs cannot be triggered on demand. The workaround is to rename the job as: field in the openshift/release config, which makes Prow see them as "new" jobs eligible for /pj-rehearse.

  1. Clone or locate a local copy of openshift/release
  2. Identify the config file:
    ci-operator/config/openshift/openshift-tests-private/openshift-openshift-tests-private-release-<OCP_VER>__amd64-nightly.yaml
  3. Find all winc periodic jobs (grep 'as:.*winc.*-f[0-9]')
  4. Rename the as: field by inserting zstream (e.g. aws-ipi-ovn-winc-f14 -> aws-ipi-ovn-winc-zstream-f14)
    • Z-stream: rename only aws, gcp, vsphere
    • Y-stream: rename all winc jobs
  5. Run make update to regenerate Prow job configs
  6. Commit, push, create a non-draft PR titled DEBUG Do not merge: <WMCO_VER> Z stream triggered jobs
  7. Wait for REHEARSALNOTIFIER bot comment listing the rehearsable jobs
  8. Post /pj-rehearse (or /pj-rehearse max if more than 5 jobs) as a comment
  9. After tests complete, close the PR without merging

What this script automates

hack/trigger-qe-tests.sh performs all of the above steps:

  • Maps WMCO version to OCP version (10.xx -> 4.xx, 11.x -> 5.x)
  • Supports -s y (y-stream, all jobs) and -s z (z-stream, subset) with -j/-x flags for overrides
  • Shows an interactive toggle menu to add/remove individual jobs before proceeding
  • Creates a worktree from upstream/main, renames jobs, runs make update
  • Creates the PR and polls for REHEARSALNOTIFIER (10 min timeout)
  • Posts /pj-rehearse to trigger the tests
  • Cleans up leftover branches/worktrees on re-runs

Reference: PR openshift/release#81646 (original manual 4.21 z-stream trigger)

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 48d29d85-2e1c-4bfe-98c8-5545c91ce3a7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds hack/trigger-qe-tests.sh, which validates inputs, locates or clones the OpenShift release repository, identifies WMCO periodic jobs, renames selected jobs with the zstream prefix, regenerates YAML, and verifies the result. The script then commits and pushes a debug branch, creates a pull request, waits for a REHEARSALNOTIFIER comment, and posts the appropriate rehearsal command after confirmation.


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error validate_release_repo() logs the full upstream remote URL on error, which can leak embedded credentials or internal hostnames. Redact the remote before logging, or print only the expected repo name/host without the full URL.
✅ Passed checks (19 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Go Best Practices & Build Tags ✅ Passed Only hack/trigger-qe-tests.sh changed; it’s a Bash script with no Go code or build tags, so this Go/build-tag check is not applicable.
Security: Secrets, Ssh & Csr ✅ Passed PASS: the new script only does repo/PR automation; no secret, SSH, CSR, cert, or key handling appears, and error paths expose only paths/URLs.
Kubernetes Controller Patterns ✅ Passed Check not applicable: PR adds a bash automation script for CI test triggering, not a Kubernetes controller. Controller pattern validation (reconciliation, status conditions, watch predicates, final...
Windows Service Management ✅ Passed PR only adds a bash automation script for QE triggers; no Windows service SCM, dependencies, reboot, or cleanup logic exists in the diff.
Platform-Specific Requirements ✅ Passed PR adds only a QE-triggering release script; it doesn't change vSphere/AWS/Azure/GCP platform logic or docs, so the platform-specific checklist is inapplicable.
Stable And Deterministic Test Names ✅ Passed PR only adds a bash helper; no Ginkgo test titles or dynamic test-name strings were introduced.
Test Structure And Quality ✅ Passed No Ginkgo test code was modified; the PR only adds a bash helper script, so these test-structure criteria are not applicable.
Microshift Test Compatibility ✅ Passed This PR only adds a bash helper script; it introduces no Ginkgo specs, OpenShift API use, or MicroShift-relevant test logic.
Single Node Openshift (Sno) Test Compatibility ✅ Passed PASS: PR adds only a bash automation script; there are no new Ginkgo e2e tests or multi-node/SNO assumptions to review.
Topology-Aware Scheduling Compatibility ✅ Passed The PR adds a bash automation script (hack/trigger-qe-tests.sh) for triggering test workflows, not deployment manifests, operator code, or controllers with scheduling constraints. No topology-aware...
Ote Binary Stdout Contract ✅ Passed PR only adds a bash helper script; no OTE binary main/suite code or stdout-contract changes were touched.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PR adds a bash helper script, not new Ginkgo/e2e tests; no It/Describe/Context/When or IPv4/network assumptions to flag.
No-Weak-Crypto ✅ Passed The new bash script contains no weak crypto, no custom crypto, and no secret/token comparisons.
Container-Privileges ✅ Passed PR only adds a bash helper script; no container/K8s manifests were changed and the diff contains no privileged settings.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly states the new script and its purpose of triggering WMCO QE z-stream tests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rrasouli
Once this PR has been reviewed and has the lgtm label, please assign mansikulkarni96 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rrasouli
rrasouli marked this pull request as draft July 14, 2026 04:34
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 14, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
hack/trigger-qe-tests.sh (2)

117-121: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicated rename logic; consider a helper (also addresses shellcheck SC2001).

The winc-winc-zstream- substitution via sed 's/winc-/winc-zstream-/' is duplicated at Line 120 (display) and Line 154 (actual rename). Static analysis also flags both as SC2001, suggesting bash parameter expansion instead of sed for a simple substitution. Extracting a small helper avoids the two call sites drifting apart and satisfies the lint hint:

♻️ Proposed refactor
+to_zstream_name() {
+  echo "${1/winc-/winc-zstream-}"
+}
+
 echo "Will rename:"
 echo -e "$JOBS_TO_RENAME" | while read -r job; do
   [[ -z "$job" ]] && continue
-  echo "  $job -> $(echo "$job" | sed 's/winc-/winc-zstream-/')"
+  echo "  $job -> $(to_zstream_name "$job")"
 done
 echo -e "$JOBS_TO_RENAME" | while read -r job; do
   [[ -z "$job" ]] && continue
-  new_name=$(echo "$job" | sed 's/winc-/winc-zstream-/')
+  new_name=$(to_zstream_name "$job")
   sed -i.bak "s/^- as: ${job}$/- as: ${new_name}/" "$WORKTREE_CONFIG"
 done

Also applies to: 150-157

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/trigger-qe-tests.sh` around lines 117 - 121, Extract the winc- to
winc-zstream- transformation into a small helper in hack/trigger-qe-tests.sh,
using Bash parameter expansion instead of sed to address SC2001. Update both the
rename preview loop and the actual rename logic near the existing JOBS_TO_RENAME
handling to call this helper, preserving the current substitution behavior.

Source: Linters/SAST tools


98-115: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Fragile newline-in-string pattern for job lists.

JOBS_TO_RENAME/SKIPPED are built by concatenating literal \n inside double quotes and later re-expanded with echo -e (Lines 118, 152, 239). This works but is a brittle idiom (breaks silently if any call site forgets -e, and doesn't handle job names containing \ well). A bash array (JOBS_TO_RENAME=(); JOBS_TO_RENAME+=("$job")) would be more robust and idiomatic for accumulating a list of job names.

Also applies to: 152-156, 239-244

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hack/trigger-qe-tests.sh` around lines 98 - 115, Replace the literal-newline
string accumulation for JOBS_TO_RENAME and SKIPPED with Bash arrays, appending
each job as a distinct quoted element in the loop. Update the related output and
emptiness checks at the shown call sites to use array-safe length and iteration,
removing reliance on echo -e while preserving the existing skip and rename
messages.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hack/trigger-qe-tests.sh`:
- Around line 46-51: Extend cleanup_temp_repo to handle failure cleanup for the
branch and worktree created later in the script: when WORKTREE_DIR exists,
remove it with the appropriate git worktree cleanup, then remove the local
BRANCH only after confirming it exists. Install an ERR/exit failure trap that
invokes this cleanup after those resources are created, while preserving normal
cleanup of TEMP_RELEASE_REPO and avoiding removal of resources that were never
created.
- Around line 34-44: Ensure the temporary clone flow creates an upstream remote
before calling validate_release_repo, or allow that flow to validate using its
clone remote. In validate_release_repo, tighten the remote URL check to match
the openshift/release repository exactly, preventing similarly named
repositories such as openshift/release-tools from passing.

---

Nitpick comments:
In `@hack/trigger-qe-tests.sh`:
- Around line 117-121: Extract the winc- to winc-zstream- transformation into a
small helper in hack/trigger-qe-tests.sh, using Bash parameter expansion instead
of sed to address SC2001. Update both the rename preview loop and the actual
rename logic near the existing JOBS_TO_RENAME handling to call this helper,
preserving the current substitution behavior.
- Around line 98-115: Replace the literal-newline string accumulation for
JOBS_TO_RENAME and SKIPPED with Bash arrays, appending each job as a distinct
quoted element in the loop. Update the related output and emptiness checks at
the shown call sites to use array-safe length and iteration, removing reliance
on echo -e while preserving the existing skip and rename messages.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7576baf5-9bb5-4896-ac35-35464ba679e9

📥 Commits

Reviewing files that changed from the base of the PR and between d6aac66 and 059b0c3.

📒 Files selected for processing (1)
  • hack/trigger-qe-tests.sh

Comment thread hack/trigger-qe-tests.sh
Comment thread hack/trigger-qe-tests.sh Outdated
@rrasouli
rrasouli force-pushed the add-trigger-qe-script branch 4 times, most recently from 4956e8c to baeaa4d Compare July 14, 2026 14:30
@rrasouli
rrasouli marked this pull request as ready for review July 14, 2026 14:31
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 14, 2026
@rrasouli
rrasouli force-pushed the add-trigger-qe-script branch from baeaa4d to 884f4fe Compare July 14, 2026 14:47
@rrasouli
rrasouli force-pushed the add-trigger-qe-script branch from 884f4fe to aa57bf3 Compare July 15, 2026 06:37
@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@rrasouli: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/vsphere-proxy-e2e-operator aa57bf3 link true /test vsphere-proxy-e2e-operator
ci/prow/vsphere-disconnected-e2e-operator aa57bf3 link false /test vsphere-disconnected-e2e-operator
ci/prow/nutanix-e2e-operator aa57bf3 link true /test nutanix-e2e-operator
ci/prow/platform-none-vsphere-e2e-operator aa57bf3 link true /test platform-none-vsphere-e2e-operator

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant