Skip to content

MCO-2275: Part2 Migrate MCO OCB - #6340

Open
ptalgulk01 wants to merge 2 commits into
openshift:mainfrom
ptalgulk01:migrate-mco-ocb-part2
Open

MCO-2275: Part2 Migrate MCO OCB#6340
ptalgulk01 wants to merge 2 commits into
openshift:mainfrom
ptalgulk01:migrate-mco-ocb-part2

Conversation

@ptalgulk01

@ptalgulk01 ptalgulk01 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

MCO-2275: Migrate OCB longduration test cases (Part 2)

Completes the OCB test migration from openshift-tests-private to machine-config-operator. This is Part 2, following PR #6080 which migrated 11 OCB test cases and all shared helpers/infrastructure.

This PR adds the remaining 6 longduration OCB test cases:

  • 83137: OCB use OutputImage with CurrentImagePullSecret
  • 79137: OCB respect maxUnavailable setting for Workers
  • 83139: OCB build images in many MCPs at the same time
  • 83755: OCL no new image applied on node after ssh/password/file MC
  • 85843: Internal Registry new nodes boot directly with OCL image
  • 82536: Internal Registry image removed triggers rebuild

Tests 83755 and 82536 are excluded via g.Label("Exclude: ...") until OCPBUGS-85094 is resolved. Both tests fail consistently across 4.23 and 5.0 on AWS shards due to an upstream bug, not a test migration issue.

Also includes:

  • Improved MCP wait-time estimation for proxy environments
  • Increased MOSC build validation timeout for concurrent builds
  • Pod readiness polling that excludes transient Job pods
  • Broader usbguard permission error handling

Summary by CodeRabbit

  • New Features
    • Added helpers to manage and read rendered image push settings and detect internal registry usage.
    • Added pod readiness polling that excludes transient Job pods.
    • Introduced a new long-duration, serial disruptive E2E suite covering OCB workflows, image rebuilds, node boot behavior, and MCP update ordering.
  • Bug Fixes
    • Improved MachineConfigPool wait-time estimation for clusters using HTTP proxying.
    • Increased MOSC build validation timeout to better handle longer concurrent builds.
    • Broadened expected usbguard permission error handling during validation.

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Walkthrough

The PR updates extended-privilege MCP and MOSC helpers, adds proxy-aware wait timing and non-Job pod readiness polling, expands OCB validation tolerance, and introduces serial long-duration tests for builds, MCP updates, node boot images, and rendered-image deletion.

Changes

OCB validation and helper updates

Layer / File(s) Summary
MCP helper consolidation and proxy-aware waits
test/extended-priv/machineconfigpool.go
MCP wait estimates now account for HTTP proxies, while node update, maxUnavailable, and node membership helpers are consolidated without changing their core behavior.
MOSC image and pod readiness helpers
test/extended-priv/machineosconfig.go, test/extended-priv/util/pods.go
MOSC rendered-image push specifications can be read and updated, internal-registry usage can be detected, and readiness polling excludes Job pods.
Long-duration OCB scenarios
test/extended-priv/mco_ocb_longduration.go
New serial disruptive tests validate OCL image usage, MCP ordering, concurrent builds, no-image-update cases, and direct OCL boot for new nodes.
Rendered-image rebuild and assertion updates
test/extended-priv/mco_ocb_longduration.go, test/extended-priv/mco_ocb.go
Rendered images can be deleted through ImageStreams or Quay, MOSB rebuild behavior is checked, build waits increase to 35 minutes, and an additional usbguard output variant is accepted.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant OCBTest
  participant MachineOSConfig
  participant MachineOSBuild
  participant MachineConfigPool
  participant NewNodes
  OCBTest->>MachineOSConfig: read or update rendered image push spec
  OCBTest->>MachineOSBuild: poll build conditions
  MachineOSBuild-->>OCBTest: report build state
  OCBTest->>MachineConfigPool: wait for MCP update
  MachineConfigPool->>NewNodes: apply OCL image
  NewNodes-->>OCBTest: report boot image state
Loading

Suggested labels: verified

Suggested reviewers: proietfb, sergiordlr


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name Status Explanation Resolution
No-Sensitive-Data-In-Logs ❌ Error New logs print image pullspecs and node names (e.g. OCL image, digested image, node.GetName()), which can expose internal hostnames. Redact registry/host portions and avoid logging node names or image pullspecs; log only non-sensitive resource identifiers.
Microshift Test Compatibility ⚠️ Warning mco_ocb_longduration.go adds unguarded Ginkgo tests using MachineConfigPool, MachineOSConfig, MachineSet, and internal-registry flows; no MicroShift skip/apigroup tag found. Add [apigroup:machineconfiguration.openshift.io] (and other needed tags) or a g.Skip()/[Skipped:MicroShift] guard so MicroShift jobs skip these tests.
✅ Passed checks (13 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All Ginkgo titles are static strings; dynamic/generated values appear only in test bodies and helpers, not in titles.
Test Structure And Quality ✅ Passed The new Ginkgo tests use bounded Eventually/Consistently calls and clean up created MCP/MOSC/MachineSet resources via defers or helper cleanup; no clear multi-responsibility or missing-timeout issu...
Single Node Openshift (Sno) Test Compatibility ✅ Passed All new multi-node tests are guarded by SkipIfCompactOrSNO, and the remaining test (83137) uses single-node-safe compact-compatible helpers.
Topology-Aware Scheduling Compatibility ✅ Passed Only E2E/test helpers changed; no manifests/controllers or new scheduling constraints were added, and topology-sensitive tests already skip SNO/compact.
Ote Binary Stdout Contract ✅ Passed No stdout writes were added in init/main/TestMain/suite setup; the new logger routes to GinkgoWriter, not stdout.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed New Ginkgo tests only use cluster-internal registry/DNS and no IPv4 literals or IPv4-only URL construction; no external internet dependencies found.
No-Weak-Crypto ✅ Passed Touched files import no crypto packages and targeted search found no md5/sha1/des/rc4/3DES/blowfish/ECB, bytes.Equal, or subtle comparisons.
Container-Privileges ✅ Passed PASS: The changed Go files add test/helpers only; searches found no privileged/hostPID/hostNetwork/hostIPC/SYS_ADMIN/allowPrivilegeEscalation/root securityContext settings.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: migrating the second part of MCO OCB coverage.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ptalgulk01
ptalgulk01 force-pushed the migrate-mco-ocb-part2 branch from 57785af to 2b55131 Compare July 24, 2026 14:04
@openshift-ci
openshift-ci Bot requested review from proietfb and sergiordlr July 24, 2026 14:05
@openshift-ci

openshift-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ptalgulk01

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 24, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
test/extended-priv/mco_ocb_longduration.go (1)

450-450: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Rename the unused removeQuayImageUsingSkepo helper or wire it into the test path.

removeQuayImageUsingSkepo is defined in test/extended-priv/mco_ocb_longduration.go, but there is no call to it in the Go codebase; only removeImageStream is used. Remove the dead helper, or if it is intended to run Skopeo, rename the name from SkepoSkopeo and wire it in.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended-priv/mco_ocb_longduration.go` at line 450, Remove the unused
removeQuayImageUsingSkepo helper, or, if it is required for the test flow,
rename it to removeQuayImageUsingSkopeo and add a call from the appropriate test
path. Ensure the final code has no dead helper and that only the intended
image-removal implementation is used.
test/extended-priv/util/pods.go (1)

205-205: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Use the canonical Job label selector for consistency.

job-name is still added by the Kubernetes Job controller for backward compatibility, so this selector still excludes Job pods, but the sibling readiness gate uses !batch.kubernetes.io/job-name. Align the selector with the canonical form to keep the test helpers consistent.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended-priv/util/pods.go` at line 205, Update the pod selector in the
Run call to exclude Job pods using the canonical batch.kubernetes.io/job-name
label, matching the selector used by the sibling readiness gate while preserving
the existing namespace and template behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended-priv/machineconfigpool.go`:
- Line 588: Update the node-diagnostic command in the surrounding
polling/error-handling flow to capture the error returned by CLI.Execute() for
mcp.oc.Run("get").Args("nodes").Execute(). Preserve that error in the test
context or include it in the final assertion so failures from oc get nodes are
surfaced alongside the polling error.

---

Nitpick comments:
In `@test/extended-priv/mco_ocb_longduration.go`:
- Line 450: Remove the unused removeQuayImageUsingSkepo helper, or, if it is
required for the test flow, rename it to removeQuayImageUsingSkopeo and add a
call from the appropriate test path. Ensure the final code has no dead helper
and that only the intended image-removal implementation is used.

In `@test/extended-priv/util/pods.go`:
- Line 205: Update the pod selector in the Run call to exclude Job pods using
the canonical batch.kubernetes.io/job-name label, matching the selector used by
the sibling readiness gate while preserving the existing namespace and template
behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1e62a24b-94f0-4d4e-bc28-b8da47079c63

📥 Commits

Reviewing files that changed from the base of the PR and between 76953dd and 2b55131.

📒 Files selected for processing (5)
  • test/extended-priv/machineconfigpool.go
  • test/extended-priv/machineosconfig.go
  • test/extended-priv/mco_ocb.go
  • test/extended-priv/mco_ocb_longduration.go
  • test/extended-priv/util/pods.go

}
if totalUpdating > maxUnavailable {
// print nodes for debug
mcp.oc.Run("get").Args("nodes").Execute()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C2 --type go 'func\s*\([^)]*\)\s*Execute\s*\(' .
rg -n -C2 --type go '\.Run\("get"\)\.Args\("nodes"\)\.Execute\(\)' test/extended-priv/machineconfigpool.go

Repository: openshift/machine-config-operator

Length of output: 8796


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "## client.go around CLI Execute"
sed -n '820,885p' test/extended-priv/util/client.go

echo
echo "## machineconfigpool.go around diagnostic call"
sed -n '560,605p' test/extended-priv/machineconfigpool.go

echo
echo "## same repository calls to CLI Execute without error check"
rg -n --type go '\.+Execute\(\)' test/extended-priv -g '*.go'

Repository: openshift/machine-config-operator

Length of output: 10952


Handle the diagnostic command failure.

CLI.Execute() returns the underlying oc get error, but line 588 discards it and the test still asserts the polling error. Capture the error, keep it in the context or prepend it to the final assertion, so node-dump failures surface useful diagnostics instead of being absorbed.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended-priv/machineconfigpool.go` at line 588, Update the
node-diagnostic command in the surrounding polling/error-handling flow to
capture the error returned by CLI.Execute() for
mcp.oc.Run("get").Args("nodes").Execute(). Preserve that error in the test
context or include it in the final assertion so failures from oc get nodes are
surfaced alongside the polling error.

Source: Path instructions

@ptalgulk01

ptalgulk01 commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

● 4.23 Jobs (image: ci-ln-dj702mt/release:latest)

5.0 Jobs (image: ci-ln-ydkgvmt/release:latest)

● PR #6340 — Test Case Results (6 tests × 10 jobs)

Legend:  ✅ = PASS    ❌ = FAIL    ─ = Not in this shard

                                                    │      4.23                    │      5.0
 #   OCP ID   Description                           │ a1   a2   a3   v1   v2      │ a1   a2   a3   v1   v2
──────────────────────────────────────────────────────┼──────────────────────────────┼──────────────────────────
 1   83137    OCB OutputImage CurrentImagePullSecret  │ ✅    ─    ─   ✅    ─      │ ✅    ─    ─   ✅    ─
 2   79137    OCB respect maxUnavailable Workers      │  ─   ✅    ─   ✅    ─      │  ─   ✅    ─   ✅    ─
 3   83139    OCB build images in many MCPs           │  ─   ✅    ─   ✅    ─      │  ─   ✅    ─   ✅    ─
 4   83755    OCL no new image after MC changes       │  ─   ❌    ─    ─   ✅      │  ─   ❌    ─    ─   ✅
 5   85843    IntReg new nodes boot with OCL image    │  ─   ✅    ─   ❌    ─      │  ─   ✅    ─   ✅    ─
 6   82536    IntReg image removed triggers rebuild   │ ❌    ─    ─    ─   ✅      │ ❌    ─    ─    ─   ✅
──────────────────────────────────────────────────────┴──────────────────────────────┴──────────────────────────

@ptalgulk01

Copy link
Copy Markdown
Contributor Author

/test unit

- Add tests: 77497, 77498, 77576, 77977, 78001, 78196, 79137, 79172, 82536, 83136, 83137, 83139, 83755, 85843, 85980, 87176, 88801
- Add supporting functions: checkUpdatedLists, SkipIfCompactOrSNO, GetAllApplicableExtensionsToMCPOrFail, AssertAllNonJobPodsToBeReadyWithPollerParams
- Add MCP methods: SetMaxUnavailable, RemoveMaxUnavailable, GetSortedUpdatedNodes
- Add MOSC methods: SetRenderedImagePushspec, GetRenderedImagePushspec, IsUsingInternalRegistry
- Preserve function ordering to match otp3 source
@ptalgulk01
ptalgulk01 force-pushed the migrate-mco-ocb-part2 branch from 2b55131 to 1f393eb Compare July 27, 2026 06:25
@ptalgulk01

ptalgulk01 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor Author

4.23 Jobs (image: ci-ln-wqm0vh2/release:latest)

5.0 Jobs (image: ci-ln-9j50f32/release:latest)

  #6340 — OCB Test Results (6 tests × 10 jobs)

  ┌─────┬────────┬────────────────────────────────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬────────┬────────┬────────┬────────┬────────┐
  │  #  │ OCP ID │              Description               │ 4.23 a1 │ 4.23 a2 │ 4.23 a3 │ 4.23 v1 │ 4.23 v2 │ 5.0 a1 │ 5.0 a2 │ 5.0 a3 │ 5.0 v1 │ 5.0 v2 │
  ├─────┼────────┼────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼────────┼────────┼────────┼────────┤
  │ 1   │ 83137  │ OCB OutputImage CurrentImagePullSecret │ ✅      │ ─       │ ─       │ ✅      │ ─       │ ✅     │ ─      │ ─      │ ∅      │ ⏭️      │
  ├─────┼────────┼────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼────────┼────────┼────────┼────────┤
  │ 2   │ 79137  │ OCB respect maxUnavailable Workers     │ ─       │ ✅      │ ─       │ ✅      │ ─       │ ─      │ ✅     │ ─      │ ∅      │ ⏭️      │
  ├─────┼────────┼────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼────────┼────────┼────────┼────────┤
  │ 3   │ 83139  │ OCB build images in many MCPs          │ ─       │ ✅      │ ─       │ ✅      │ ─       │ ─      │ ✅     │ ─      │ ∅      │ ⏭️      │
  ├─────┼────────┼────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼────────┼────────┼────────┼────────┤
  │ 4   │ 83755  │ OCL no new image after MC changes      │ ─       │ ❌      │ ─       │ ─       │ ✅      │ ─      │ ❌     │ ─      │ ∅      │ ⏭️      │
  ├─────┼────────┼────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼────────┼────────┼────────┼────────┤
  │ 5   │ 85843  │ IntReg new nodes boot with OCL image   │ ─       │ ✅      │ ─       │ ✅      │ ─       │ ─      │ ✅     │ ─      │ ∅      │ ⏭️      │
  ├─────┼────────┼────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼────────┼────────┼────────┼────────┼────────┤
  │ 6   │ 82536  │ IntReg image removed triggers rebuild  │ ❌      │ ─       │ ─       │ ─       │ ✅      │ ❌     │ ─      │ ─      │ ∅      │ ⏭️      │
  └─────┴────────┴────────────────────────────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴────────┴────────┴────────┴────────┴────────┘

  Legend: ✅ PASS · ❌ FAIL · ⏭️  SKIP · ─ Not in shard · ∅ Infra failure

The TC 83755 and TC 82536 is failing because of bug https://redhat.atlassian.net/browse/OCPBUGS-85094 hence, excluding it.

@ptalgulk01 ptalgulk01 changed the title Migrate mco ocb part2 MCO-2275: Part2 Migrate MCO OCB Jul 28, 2026
@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 28, 2026
@openshift-ci-robot

openshift-ci-robot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@ptalgulk01: This pull request references MCO-2275 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Part 2

Part 2 (migrate-mco-ocb-part2 branch — uncommitted): 6 tests needing verification (83137, 79137, 83139, 83755, 85843, 82536) + all shared helpers/infrastructure. Build passes.

Summary by CodeRabbit

  • New Features
  • Added helpers to manage and read rendered image push settings and detect internal registry usage.
  • Added pod readiness polling that excludes transient Job pods.
  • Introduced a new long-duration, serial disruptive E2E suite covering OCB workflows, image rebuilds, node boot behavior, and MCP update ordering.
  • Bug Fixes
  • Improved MachineConfigPool wait-time estimation for clusters using HTTP proxying.
  • Increased MOSC build validation timeout to better handle longer concurrent builds.
  • Broadened expected usbguard permission error handling during validation.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@ptalgulk01
ptalgulk01 force-pushed the migrate-mco-ocb-part2 branch from 1f393eb to 8c3aa0e Compare July 28, 2026 06:34
@ptalgulk01
ptalgulk01 force-pushed the migrate-mco-ocb-part2 branch from 8c3aa0e to 61fc8ec Compare July 28, 2026 06:36
@openshift-ci

openshift-ci Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@ptalgulk01: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@isabella-janssen

Copy link
Copy Markdown
Member

/hold

Holding to allow the Kube rebase to land in #6321. Please ensure this will not cause merge conflicts for the Kube rebase before unholding this PR.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants