OCPBUGS-86704: Fix MaxParallelUpgrades bypass in userData deletion path - #4155
OCPBUGS-86704: Fix MaxParallelUpgrades bypass in userData deletion path#4155ranjithrajaram wants to merge 1 commit into
Conversation
openshift-ci-robot
added
jira/severity-moderate
|
@ranjithrajaram: This pull request references Jira Issue OCPBUGS-86704, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR introduces cluster-wide upgrade-limit gating at two critical junctures in the Windows Machine Config Operator's reconciliation loop. The secret controller now checks upgrade capacity before clearing node public-key-hash annotations, and the windowsmachine controller gates machine deletion on public-key mismatch. Both paths reuse a shared 🚥 Pre-merge checks | ✅ 5 | ❌ 15❌ Failed checks (15 inconclusive)
✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: ranjithrajaram The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
needs-ok-to-test
|
Hi @ranjithrajaram. Thanks for your PR. I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/ok-to-test |
openshift-ci
Bot
added
ok-to-test
2c1bafc to
bd640c9
Compare
|
/test vsphere-e2e-operator |
|
@code-rabbit full review |
|
@code-rabbit review |
| // Requires at least 2 Machine nodes to test parallel behavior | ||
| require.NoError(t, tc.loadExistingNodes(), "error loading existing nodes") | ||
| if len(gc.machineNodes) < 2 { | ||
| t.Skip("Test requires at least 2 Machine nodes to verify MaxParallelUpgrades enforcement") |
There was a problem hiding this comment.
this test is a no-op, as you can see in the logs. all the current jobs uses 1 machine and 1 BYOH instance. to achieve this scenario need to configure a job for two instances, so that we can hit this test.
as a pre-req; we need a PR in the release repo adjusting one of the jobs to have two machines. Azure job is a good candidate.
| } | ||
|
|
||
| // deployParallelUpgradesChecker deploys the monitoring job that checks for MaxParallelUpgrades violations | ||
| func (tc *testContext) deployParallelUpgradesChecker() error { |
There was a problem hiding this comment.
no need to reinvent the wheel, with createParallelUpgradeCheckerResources, can and achive the same behavior, and the code is wired to check it in testParallelUpgradesChecker
There was a problem hiding this comment.
@ranjithrajaram thanks for working on this. The operator changes looks good, just need to adjust the testing suite to actually test this case.
| return ctrl.Result{Requeue: true}, nil | ||
| } | ||
|
|
||
| deletionAllowed, err := r.isAllowedDeletion(ctx, machine) |
There was a problem hiding this comment.
this may introduce a race condition with duplicate/competing throttles that create inconsistent behavior. PR adds cluster-wide gate in WindowsMachineReconciler, but still keeps per-MachineSet isAllowedDeletion() gate.
two different limiters (cluster-wide + per-machineset unhealthy limit) may conflict and cause hard-to-predict requeue/defer behavior.
another reason for the e2e test as a must have.
There was a problem hiding this comment.
if annotation patch fails after marking upgrading, consider explicitly remove the upgrading label (best-effort) before returning error.
bd640c9 to
d003126
Compare
The secret controller's updateUserData method clears the pub-key-hash annotation on Machine-backed nodes to trigger Machine recreation when the userData secret changes. This bypasses MaxParallelUpgrades because it does not acquire an upgrade slot before clearing the annotation. Add markNodeAsUpgrading() checks in secret_controller.go and windowsmachine_controller.go before operations that trigger Machine recreation. Nodes that exceed the upgrade limit are deferred. Best-effort rollback of the upgrading label is performed if the subsequent annotation patch fails. Add E2E test wiring to validate the MaxParallelUpgrades constraint during userData-triggered Machine recreation by reusing the existing parallel upgrade checker infrastructure.
d003126 to
e34fe90
Compare
|
Apologies for mixing updates for openshift/release pull request 81847 in here. i have reverted it. It was an oversight |
|
@code-rabbit full review |
|
@ranjithrajaram: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
4 participants
Trying to address JIRA OCPBUGS-86704
The MaxParallelUpgrades=1 protection is bypassed during userData template changes, causing simultaneous machine deletions across MachineSets during normal WMCO upgrades.
This adds cluster-wide upgrade limit checking at two layers:
Root cause: When userData template changes (e.g., AWS routes, SSH retry logic, firewall rules), SecretReconciler cleared pub-key-hash annotations on ALL Machine nodes simultaneously. WindowsMachine Reconciler then used per-MachineSet isAllowedDeletion() check instead of cluster-wide markNodeAsUpgrading() lock, allowing N MachineSets = N simultaneous deletions.
Defense in depth approach:
Summary by CodeRabbit