Skip to content

WIP: OCPBUGS-10431: clear staticPodPath template default when user sets it to empty - #6319

Closed
harche wants to merge 1 commit into
openshift:mainfrom
harche:staticpodpath-empty-render
Closed

WIP: OCPBUGS-10431: clear staticPodPath template default when user sets it to empty#6319
harche wants to merge 1 commit into
openshift:mainfrom
harche:staticpodpath-empty-render

Conversation

@harche

@harche harche commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

- What I did

Follow-up to #5724. That PR relaxed validation so a KubeletConfig CR with staticPodPath: "" is accepted for worker and custom pools, but the empty value never reached the rendered kubelet.conf: mergo.Merge(..., mergo.WithOverride) in generateKubeletIgnFiles does not overwrite a destination field with a zero-value source, so the template default /etc/kubernetes/manifests survived the merge.

This change detects an explicit staticPodPath: "" in the raw KubeletConfig (presence check on the raw JSON, since after decoding an explicit empty value is indistinguishable from unset) and clears the template default before merging, mirroring the existing protectKernelDefaults workaround a few lines above.

Also adds a render-path unit test asserting that:

  • an explicit {"staticPodPath":""} renders a kubelet.conf without /etc/kubernetes/manifests
  • an unset staticPodPath keeps the template default

- How to verify it

  1. On a cluster with this change, apply a KubeletConfig CR targeting the worker pool with staticPodPath: ""
  2. Check the rendered MachineConfig (oc get mc 99-worker-generated-kubelet -o yaml) or /etc/kubernetes/kubelet.conf on a worker node: the staticPodPath entry should be absent
  3. Without this change, the rendered config still contains staticPodPath: /etc/kubernetes/manifests even though the CR is accepted

- Description for the changelog

Fix staticPodPath: "" in a KubeletConfig CR not taking effect in the rendered kubelet configuration.

WIP: pending cluster testing.

Summary by CodeRabbit

  • Bug Fixes

    • Preserved an explicitly provided empty staticPodPath so it’s rendered as intended, rather than falling back to the template default.
  • Tests

    • Added new test coverage to confirm correct staticPodPath behavior when the value is omitted or explicitly empty, validating handling across supported configuration input formats.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 20, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@harche: This pull request references Jira Issue OCPBUGS-10431, which is invalid:

  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did

Follow-up to #5724. That PR relaxed validation so a KubeletConfig CR with staticPodPath: "" is accepted for worker and custom pools, but the empty value never reached the rendered kubelet.conf: mergo.Merge(..., mergo.WithOverride) in generateKubeletIgnFiles does not overwrite a destination field with a zero-value source, so the template default /etc/kubernetes/manifests survived the merge.

This change detects an explicit staticPodPath: "" in the raw KubeletConfig (presence check on the raw JSON, since after decoding an explicit empty value is indistinguishable from unset) and clears the template default before merging, mirroring the existing protectKernelDefaults workaround a few lines above.

Also adds a render-path unit test asserting that:

  • an explicit {"staticPodPath":""} renders a kubelet.conf without /etc/kubernetes/manifests
  • an unset staticPodPath keeps the template default

- How to verify it

  1. On a cluster with this change, apply a KubeletConfig CR targeting the worker pool with staticPodPath: ""
  2. Check the rendered MachineConfig (oc get mc 99-worker-generated-kubelet -o yaml) or /etc/kubernetes/kubelet.conf on a worker node: the staticPodPath entry should be absent
  3. Without this change, the rendered config still contains staticPodPath: /etc/kubernetes/manifests even though the CR is accepted

- Description for the changelog

Fix staticPodPath: "" in a KubeletConfig CR not taking effect in the rendered kubelet configuration.

WIP: pending cluster testing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01F9QZAvsyRgd8KagUdBxaAZ

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 20, 2026
@openshift-ci

openshift-ci Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Walkthrough

Kubelet configuration generation now preserves an explicitly empty staticPodPath from raw JSON or YAML input while retaining template defaults when the field is omitted. Tests decode rendered Ignition content and verify both behaviors.

Changes

Static Pod Path Handling

Layer / File(s) Summary
Detect explicit empty staticPodPath
pkg/controller/kubelet-config/helpers.go
Raw kubelet configuration is checked for an explicitly empty staticPodPath, clearing the original value before merging.
Validate rendered configuration
pkg/controller/kubelet-config/staticpodpath_render_test.go
A helper decodes rendered Ignition content, and tests verify struct-marshaled, raw JSON, and YAML inputs produce the expected staticPodPath output.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (14 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo titles were added; the only new test name is the static Go test function TestStaticPodPathEmptyRender.
Test Structure And Quality ✅ Passed Unit test is self-contained and matches nearby package style; no cluster resources, waits, or cleanup concerns were introduced.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests or MicroShift-unsupported APIs; the added test is a standard Go unit test with no MicroShift-specific concerns.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The added test is a plain Go unit test in pkg/controller/kubelet-config, not a Ginkgo e2e test, and it has no multi-node/HA assumptions.
Topology-Aware Scheduling Compatibility ✅ Passed Only kubelet-config merge/render logic and tests changed; no pod specs, affinities, replicas, node selectors, or topology assumptions were added.
Ote Binary Stdout Contract ✅ Passed No new main/init/TestMain/BeforeSuite stdout writes were added; the PR only changes helper logic and test-body code, with no process-level stdout output.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Only a Go unit test was added; no Ginkgo/e2e test, IPv4-only logic, or external connectivity is present.
No-Weak-Crypto ✅ Passed Changed files only adjust kubelet config merging/tests; scans found no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons.
Container-Privileges ✅ Passed No container/K8s manifest changes or privilege flags were introduced; touched files only adjust kubelet config merge logic and tests.
No-Sensitive-Data-In-Logs ✅ Passed The PR adds no production logging, and the new test messages don't include passwords, tokens, PII, or hostnames.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: clearing the staticPodPath template default when users set it to empty.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci

openshift-ci Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: harche
Once this PR has been reviewed and has the lgtm label, please assign isabella-janssen for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
pkg/controller/kubelet-config/helpers.go (1)

6-6: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Remove unused import.

If you apply the suggested fix below to use yaml.NewYAMLOrJSONDecoder instead of json.Unmarshal, this encoding/json import will become unused and can be removed.

♻️ Proposed refactor
-	"encoding/json"
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/kubelet-config/helpers.go` at line 6, Remove the unused
encoding/json import from the imports in helpers.go after replacing
json.Unmarshal with yaml.NewYAMLOrJSONDecoder.
pkg/controller/kubelet-config/staticpodpath_render_test.go (1)

69-69: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Avoid reusing a mutable configuration object without copying.

generateKubeletIgnFiles mutates the originalKubeConfig object through mergo.Merge and direct field assignments. The subsequent original2 := originalKubeConfig.DeepCopy() at line 86 copies the already-mutated state from this first call. To ensure the second test case runs against a pristine configuration, deep copy originalKubeConfig before passing it to the first function call.

♻️ Proposed fix
-	kubeletIgn, _, _, err := generateKubeletIgnFiles(kc, originalKubeConfig)
+	kubeletIgn, _, _, err := generateKubeletIgnFiles(kc, originalKubeConfig.DeepCopy())
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/kubelet-config/staticpodpath_render_test.go` at line 69,
Deep-copy originalKubeConfig before the first generateKubeletIgnFiles call so
that function receives an isolated mutable configuration. Preserve the existing
originalKubeConfig object for the later original2 copy, ensuring both test cases
start from pristine state.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/kubelet-config/helpers.go`:
- Around line 556-561: Update the raw kubelet configuration parsing near
DecodeKubeletConfig to use yaml.NewYAMLOrJSONDecoder, preserving support for
both YAML and JSON payloads. Handle and explicitly return the decoder error
instead of silently skipping the staticPodPath override, while retaining the
existing override behavior for an explicitly empty value.

---

Nitpick comments:
In `@pkg/controller/kubelet-config/helpers.go`:
- Line 6: Remove the unused encoding/json import from the imports in helpers.go
after replacing json.Unmarshal with yaml.NewYAMLOrJSONDecoder.

In `@pkg/controller/kubelet-config/staticpodpath_render_test.go`:
- Line 69: Deep-copy originalKubeConfig before the first generateKubeletIgnFiles
call so that function receives an isolated mutable configuration. Preserve the
existing originalKubeConfig object for the later original2 copy, ensuring both
test cases start from pristine state.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b8b438f0-eb46-4464-9b89-23b7534cbe3b

📥 Commits

Reviewing files that changed from the base of the PR and between cb061e1 and af14db4.

📒 Files selected for processing (2)
  • pkg/controller/kubelet-config/helpers.go
  • pkg/controller/kubelet-config/staticpodpath_render_test.go

Comment thread pkg/controller/kubelet-config/helpers.go
… to empty

mergo.Merge never overwrites a destination field with a zero-value
source, so an explicit staticPodPath "" in a KubeletConfig CR passes
validation (after openshift#5724) but is dropped during the merge, and the
rendered kubelet.conf still contains the template default
/etc/kubernetes/manifests. Detect the explicit empty value in the raw
config with a YAMLOrJSON presence check and clear the default before
merging, mirroring the existing protectKernelDefaults workaround.

Add a render-path test asserting that an explicit empty staticPodPath
(JSON or YAML) clears the default and that an unset field keeps it.
@harche
harche force-pushed the staticpodpath-empty-render branch 2 times, most recently from 3625b20 to 460b78c Compare July 21, 2026 16:25
@harche

harche commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Folding this into #6325 so the staticPodPath work is reviewable as one unit — the mergo empty-value fix only makes sense together with moving the CRI-O metrics proxy off static pods. Closing in favor of #6325.

@harche harche closed this Jul 22, 2026
@openshift-ci-robot

Copy link
Copy Markdown
Contributor

@harche: This pull request references Jira Issue OCPBUGS-10431. The bug has been updated to no longer refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did

Follow-up to #5724. That PR relaxed validation so a KubeletConfig CR with staticPodPath: "" is accepted for worker and custom pools, but the empty value never reached the rendered kubelet.conf: mergo.Merge(..., mergo.WithOverride) in generateKubeletIgnFiles does not overwrite a destination field with a zero-value source, so the template default /etc/kubernetes/manifests survived the merge.

This change detects an explicit staticPodPath: "" in the raw KubeletConfig (presence check on the raw JSON, since after decoding an explicit empty value is indistinguishable from unset) and clears the template default before merging, mirroring the existing protectKernelDefaults workaround a few lines above.

Also adds a render-path unit test asserting that:

  • an explicit {"staticPodPath":""} renders a kubelet.conf without /etc/kubernetes/manifests
  • an unset staticPodPath keeps the template default

- How to verify it

  1. On a cluster with this change, apply a KubeletConfig CR targeting the worker pool with staticPodPath: ""
  2. Check the rendered MachineConfig (oc get mc 99-worker-generated-kubelet -o yaml) or /etc/kubernetes/kubelet.conf on a worker node: the staticPodPath entry should be absent
  3. Without this change, the rendered config still contains staticPodPath: /etc/kubernetes/manifests even though the CR is accepted

- Description for the changelog

Fix staticPodPath: "" in a KubeletConfig CR not taking effect in the rendered kubelet configuration.

WIP: pending cluster testing.

Summary by CodeRabbit

  • Bug Fixes

  • Preserved an explicitly provided empty staticPodPath so it’s rendered as intended, rather than falling back to the template default.

  • Tests

  • Added new test coverage to confirm correct staticPodPath behavior when the value is omitted or explicitly empty, validating handling across supported configuration input formats.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

harche added a commit to harche/machine-config-operator that referenced this pull request Jul 23, 2026
The kube-rbac-proxy-crio static pod (added in openshift#4175 / OCPNODE-2098) is the
only static pod that runs on every node role, which blocks disabling
staticPodPath on worker pools to satisfy STIG V-242397: removing
staticPodPath kills the proxy, port 9637 goes dead, and CRI-O metrics
disappear for those nodes.

Run the proxy as a container in the machine-config-daemon daemonset
instead. The daemonset is hostNetwork, so the container still binds
node:9637 and forwards to CRI-O's loopback-only metrics endpoint at
127.0.0.1:9537 using the same kubelet serving certificate and client CA,
keeping the cluster-monitoring-operator kubelet ServiceMonitor scrape
chain (job="crio") unchanged.

The static authorizer config moves from a MachineConfig-written host file
to the kube-rbac-proxy-crio ConfigMap, and the proxy now uses in-cluster
service account credentials (the MCD service account already holds
tokenreview/subjectaccessreview permissions) instead of the kubelet
kubeconfig. The kube-rbac-proxy-crio Service selects the
machine-config-daemon pods so its endpoints keep pointing at port 9637.

Upgrade ordering: the operator waits for the MCD daemonset rollout before
it syncs the node configs that delete the static pod manifest, so the new
container waits for the manifest to disappear and the port to be released
before starting the proxy rather than crashlooping on a bind failure and
deadlocking the rollout. The container also declares no ports: with
hostNetwork a declared containerPort acts as a hostPort, and the kubelet
would reject the pod at admission while the legacy static pod still
declares port 9637.

Also clear the staticPodPath template default when a KubeletConfig sets
it to empty (previously in openshift#6319): the rendered config is produced by
mergo.Merge, which never overwrites a field with an empty value, so
staticPodPath: "" was silently ignored while the CR reported Success —
the same empty-value problem already worked around for
protectKernelDefaults. With both changes, disabling staticPodPath on
worker pools resolves STIG V-242397 without losing CRI-O metrics.

Verified on two live 4.22.6 clusters: daemonset rolls out Ready while the
static pods still exist, hands off port 9637 as each node updates,
up{job="crio"} stays 1 on all nodes throughout, and staticPodPath: "" on
workers removes the setting from kubelet.conf with metrics uninterrupted.

Claude-Session: https://claude.ai/code/session_01ChGPtsm6ZxReyA3oZjq79o
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants