Skip to content

Refuse <initlabels> which cannot fit into max_series_per_metric - #266

Merged
Watson1978 merged 1 commit into
fluent:masterfrom
kenhys:fix-initlabels-guard
Aug 31, 2026
Merged

Refuse <initlabels> which cannot fit into max_series_per_metric#266
Watson1978 merged 1 commit into
fluent:masterfrom
kenhys:fix-initlabels-guard

Conversation

@kenhys

@kenhys kenhys commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

<initlabels> label sets take their slots at startup, so a smaller limit
fills the metric before any record arrives and drops every one of them.
Stop at startup instead. A limit equal to the number of label sets is
fine: every label set is known in advance.

@kenhys

kenhys commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

This PR is follow-up of #265 , need rebase.

<initlabels> label sets take their slots at startup, so a limit smaller
than their number is already exceeded before any record arrives: the
metric could never take a new label set. Stop at startup instead. A
record on one of those label sets is still counted, since the metric
already holds it. A limit equal to the number of label sets is fine:
every label set is known in advance.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
@kenhys
kenhys force-pushed the fix-initlabels-guard branch from 26a162f to 396ec8f Compare August 28, 2026 07:26
@kenhys
kenhys marked this pull request as ready for review August 28, 2026 07:26
@kenhys
kenhys requested a review from Watson1978 August 28, 2026 07:27
def check_initlabels_fit_series_limit!
return if @max_series_per_metric <= 0
# two <initlabels> blocks with the same values make one label set
return if @series_set.size <= @max_series_per_metric

@Watson1978 Watson1978 Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check_initlabels_fit_series_limit! compares this section's limit against
@series_set.size, but that SeriesSet is shared by every <metric> section
with the same name (SeriesSet.of stores it on the client metric). So the
left side counts other sections' label sets while the right side is this
section's own limit, and the result depends on the order the sections are
configured in.

<metric>
  name shared
  max_series_per_metric 10
  initialized true
  <labels>
    path $.path
  </labels>
  <initlabels> path /a </initlabels>
  ... /b /c /d /e
</metric>

<metric>
  name shared          # same name -> same SeriesSet
  max_series_per_metric 3
  initialized true
  <labels>
    path $.path
  </labels>
  <initlabels> path /z </initlabels>
</metric>

Configured in this order, the second section aborts startup:

A: @series_set.size=5  @max_series_per_metric=10 -> pass
B: @series_set.size=6  @max_series_per_metric=3  -> ConfigError

The 6 includes A's five label sets; B has a single <initlabels>. Swapping the
two sections starts fine, so the same configuration behaves differently
depending on section order, and the message points at the wrong section
("metric shared holds 6 label sets from " for a section that has
one).

Suggestion

Counting this section's own <initlabels> fixes it and matches what the method
name, the comment and the README already describe:

diff --git a/lib/fluent/plugin/prometheus.rb b/lib/fluent/plugin/prometheus.rb
index ae64c2d..55e1b68 100644
--- a/lib/fluent/plugin/prometheus.rb
+++ b/lib/fluent/plugin/prometheus.rb
@@ -472,7 +472,8 @@ def bind_series_set(client_metric)
         def check_initlabels_fit_series_limit!
           return if @max_series_per_metric <= 0
           # two <initlabels> blocks with the same values make one label set
-          return if @series_set.size <= @max_series_per_metric
+          size = @base_initlabels.map { |l| normalize_label_set(l) }.uniq.size
+          return if size <= @max_series_per_metric
 
           raise ConfigError, "metric #{@name} holds #{@series_set.size} label sets from <initlabels>, " \
                              "but max_series_per_metric is #{@max_series_per_metric}: " \

uniq keeps the "two <initlabels> blocks with the same values make one label
set" behaviour, and normalize_label_set is needed because ${worker_id} is
stored as an Integer in @base_initlabels. The existing suite passes with this
change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though suggested fix resolves order dependency, but it might add one more edge case (pass on startup, drop on runtime)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thus, it should be also fixed in follow-up.

@Watson1978

Watson1978 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Are these two edge cases in scope for this PR, or should we handle them in a follow-up?

Both involve a second <metric> section with the same name, which shares one SeriesSet. Startup passes, and the second section then drops every record that brings a new label set (records on the label sets the client already holds still go through).

  1. Uninitialized sibling: the check runs inside if @initialized, so a section that is not initialized true is never checked, even though the shared count is already over its limit.
  2. Sibling with the limit off (max_series_per_metric 0): confirm_series returns early, so its <initlabels> never reach the shared SeriesSet and the check undercounts.

Neither is a regression, and neither is covered by the change I suggested on the other thread, since both are about the shared count rather than the per-section one. I'm fine keeping this PR focused and addressing them later — what do you prefer?

@kenhys

kenhys commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Are these two edge cases in scope for this PR, or should we handle them in a follow-up?

It might be better to handle in follow-up.

@Watson1978
Watson1978 merged commit 4643fc4 into fluent:master Aug 31, 2026
9 checks passed
@kenhys
kenhys deleted the fix-initlabels-guard branch August 31, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants