Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,12 @@ not a number, does not consume the limit. A pre-initialized label set
(`initialized` and `<initlabels>`) consumes it from the start, since the metric
holds it before any record arrives.

A `<metric>` section is refused at startup with a configuration error when its
limit is smaller than the number of its own `<initlabels>` label sets: the limit
is already exceeded before any record arrives, so the metric could never take a
new label set. A limit equal to that number is fine, since every label set of
the metric is known in advance.

##### Observing what the limit leaves out

A dropped label set is not routed to `@ERROR`, because it is what the
Expand Down
16 changes: 16 additions & 0 deletions lib/fluent/plugin/prometheus.rb
Original file line number Diff line number Diff line change
Expand Up @@ -460,9 +460,25 @@ def bind_series_set(client_metric)
@base_initlabels.each do |initlabels|
confirm_series(normalize_label_set(initlabels))
end
check_initlabels_fit_series_limit!
end
end

# The client is given these label sets at startup, so a limit which
# does not fit them is already exceeded before any record arrives and
# the metric could never take a new one. Stop instead of running that
# way. A record on one of them is still counted, since the metric
# already holds its label set.
def check_initlabels_fit_series_limit!
return if @max_series_per_metric <= 0
# two <initlabels> blocks with the same values make one label set
return if @series_set.size <= @max_series_per_metric

@Watson1978 Watson1978 Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

check_initlabels_fit_series_limit! compares this section's limit against
@series_set.size, but that SeriesSet is shared by every <metric> section
with the same name (SeriesSet.of stores it on the client metric). So the
left side counts other sections' label sets while the right side is this
section's own limit, and the result depends on the order the sections are
configured in.

<metric>
  name shared
  max_series_per_metric 10
  initialized true
  <labels>
    path $.path
  </labels>
  <initlabels> path /a </initlabels>
  ... /b /c /d /e
</metric>

<metric>
  name shared          # same name -> same SeriesSet
  max_series_per_metric 3
  initialized true
  <labels>
    path $.path
  </labels>
  <initlabels> path /z </initlabels>
</metric>

Configured in this order, the second section aborts startup:

A: @series_set.size=5  @max_series_per_metric=10 -> pass
B: @series_set.size=6  @max_series_per_metric=3  -> ConfigError

The 6 includes A's five label sets; B has a single <initlabels>. Swapping the
two sections starts fine, so the same configuration behaves differently
depending on section order, and the message points at the wrong section
("metric shared holds 6 label sets from " for a section that has
one).

Suggestion

Counting this section's own <initlabels> fixes it and matches what the method
name, the comment and the README already describe:

diff --git a/lib/fluent/plugin/prometheus.rb b/lib/fluent/plugin/prometheus.rb
index ae64c2d..55e1b68 100644
--- a/lib/fluent/plugin/prometheus.rb
+++ b/lib/fluent/plugin/prometheus.rb
@@ -472,7 +472,8 @@ def bind_series_set(client_metric)
         def check_initlabels_fit_series_limit!
           return if @max_series_per_metric <= 0
           # two <initlabels> blocks with the same values make one label set
-          return if @series_set.size <= @max_series_per_metric
+          size = @base_initlabels.map { |l| normalize_label_set(l) }.uniq.size
+          return if size <= @max_series_per_metric
 
           raise ConfigError, "metric #{@name} holds #{@series_set.size} label sets from <initlabels>, " \
                              "but max_series_per_metric is #{@max_series_per_metric}: " \

uniq keeps the "two <initlabels> blocks with the same values make one label
set" behaviour, and normalize_label_set is needed because ${worker_id} is
stored as an Integer in @base_initlabels. The existing suite passes with this
change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though suggested fix resolves order dependency, but it might add one more edge case (pass on startup, drop on runtime)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thus, it should be also fixed in follow-up.


raise ConfigError, "metric #{@name} holds #{@series_set.size} label sets from <initlabels>, " \
"but max_series_per_metric is #{@max_series_per_metric}: " \
"the limit is already exceeded before any record arrives"
end

# The SeriesSet keys a label set by its values, so the same value has to
# look the same whether a RecordAccessor or <initlabels> produced it.
def normalize_label_value(value)
Expand Down
29 changes: 29 additions & 0 deletions spec/fluent/plugin/prometheus/series_limit_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -270,6 +270,16 @@ def instrument_other(path, value = 1)
)
end

context 'with a limit below the number of <initlabels> label sets' do
# 3 label sets exist at startup, so a limit of 1 would drop every record
let(:max_series_per_metric) { 1 }

it 'stops at startup instead of dropping every record' do
expect { metric }.to raise_error(Fluent::ConfigError,
/holds 3 label sets from <initlabels>.*max_series_per_metric is 1/)
end
end

context 'with a limit equal to the number of <initlabels> label sets' do
# every label set is known in advance, so the limit is reached but no
# record is dropped
Expand All @@ -289,6 +299,25 @@ def instrument_other(path, value = 1)
expect { instrument('/d') }.to raise_error(Fluent::Plugin::Prometheus::LabelSetLimitError)
end
end

context 'with two <initlabels> holding the same values' do
# both make the same label set, so they take one slot
let(:initlabels) { ['/a', '/a'] }
let(:max_series_per_metric) { 1 }

it 'counts the label sets and not the <initlabels> blocks' do
expect { metric }.not_to raise_error
end
end

context 'without a limit' do
let(:max_series_per_metric) { 0 }

it 'accepts any number of <initlabels> label sets' do
expect { metric }.not_to raise_error
expect { instrument('/d') }.not_to raise_error
end
end
end

describe '<metric> overriding the plugin limit' do
Expand Down