Skip to content

feat: Add selector and bootstrap observability metrics#286

Open
rawadhossain wants to merge 1 commit into
kubernetes-sigs:mainfrom
rawadhossain:feat/add-observability-metrics
Open

feat: Add selector and bootstrap observability metrics#286
rawadhossain wants to merge 1 commit into
kubernetes-sigs:mainfrom
rawadhossain:feat/add-observability-metrics

Conversation

@rawadhossain

@rawadhossain rawadhossain commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Description

This PR adds three new metrics. Two metrics are fully implemented, while the third is left with TODOs pending discussion.

What each metric does

node_readiness_selector_matched_nodes_total

Tracks how many nodes currently match a rule's spec. If a rule's NodeSelector matches no nodes, controller performs no work and produces no other signal. This metric makes those misconfigurations immediately visible.

node_readiness_bootstrap_completion_errors_total

Counts failures writing the bootstrap completion annotation. If this write fails, the node continues to be re-evaluated even though bootstrap completed. This metric makes those failures visible. This metric surfaces those silent failures with a rule and reason label.

Related to Issue #182

Type of Change

/kind feature

Testing

  • Added tests coverages.

Checklist

  • make test passes
  • make lint passes

@kubernetes-prow kubernetes-prow Bot added the kind/feature Categorizes issue or PR as related to a new feature. label Jul 2, 2026
@netlify

netlify Bot commented Jul 2, 2026 •
edited
Loading

Copy link
Copy Markdown

✅ Deploy Preview for node-readiness-controller canceled.

Name Link
🔨 Latest commit 3051634
🔍 Latest deploy log https://app.netlify.com/projects/node-readiness-controller/deploys/6a4e2459a22d2e00082bb99a

@kubernetes-prow

kubernetes-prow Bot commented Jul 2, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rawadhossain
Once this PR has been reviewed and has the lgtm label, please assign ajaysundark for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

kubernetes-prow Bot commented Jul 2, 2026

Copy link
Copy Markdown

Hi @rawadhossain. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow kubernetes-prow Bot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Jul 2, 2026
@rawadhossain
rawadhossain force-pushed the feat/add-observability-metrics branch from 406dde6 to 11900ae Compare July 3, 2026 15:07
@kubernetes-prow kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 3, 2026
@ajaysundark
ajaysundark self-requested a review July 3, 2026 21:22
AvineshTripathi
AvineshTripathi reviewed Jul 4, 2026
View reviewed changes

@AvineshTripathi AvineshTripathi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Left a few comments.

Comment thread internal/controller/nodereadinessrule_controller.go

// Update selector_matched_nodes_total from the node list we already fetched.
var matchedCount int
for i := range nodeList.Items {

@AvineshTripathi AvineshTripathi Jul 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we already do the same thing inside processAllNodesForRule here

@rawadhossain rawadhossain Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I intentionally kept it before the dry-run check so the gauge gets updated for both dry-run and normal rules in one place. If I move it into processAllNodesForRule, dryrun rules would never update the metric as they go through processDryRun. Open to refactor it if you think splitting it across both paths is cleaner.

@AvineshTripathi AvineshTripathi Jul 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you think a cleanup would be better? We could filter nodes before the evaluating them in dryRun or regular run

@rawadhossain rawadhossain Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, that seems cleaner. I'll filter the nodes once in Reconcile(), set the gauge there, and pass the filtered list to both paths. Thanks.

Comment thread internal/metrics/metrics.go
Comment thread internal/metrics/metrics.go Outdated
Comment thread internal/controller/nodereadinessrule_controller.go Outdated
@rawadhossain
rawadhossain force-pushed the feat/add-observability-metrics branch from 11900ae to 12ab0d9 Compare July 5, 2026 07:32
@rawadhossain

rawadhossain commented Jul 5, 2026

Copy link
Copy Markdown
Contributor Author

@AvineshTripathi PTAL. I'll commit the changes once you approve.
Thanks.

@AvineshTripathi

AvineshTripathi commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

@rawadhossain I left one comment PHAL and maybe remove the Bootstrap duration metrics till we merge #224

@kubernetes-prow kubernetes-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 7, 2026
@ajaysundark
ajaysundark requested review from AvineshTripathi and removed request for dchen1107 and haircommander July 7, 2026 21:33
@rawadhossain
rawadhossain force-pushed the feat/add-observability-metrics branch from 12ab0d9 to 8fcbeca Compare July 8, 2026 10:01
@kubernetes-prow kubernetes-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 8, 2026
@rawadhossain
rawadhossain force-pushed the feat/add-observability-metrics branch from 8fcbeca to 3051634 Compare July 8, 2026 10:20
@rawadhossain

rawadhossain commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

@AvineshTripathi I have made the changes. PTAL.
Thanks.

@AvineshTripathi

AvineshTripathi commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

/ok-to-test

@kubernetes-prow kubernetes-prow Bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ajaysundark ajaysundark Awaiting requested review from ajaysundark
@AvineshTripathi AvineshTripathi Awaiting requested review from AvineshTripathi

Assignees

No one assigned

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rawadhossain @AvineshTripathi