Refuse <initlabels> which cannot fit into max_series_per_metric - #266
Conversation
|
This PR is follow-up of #265 , need rebase. |
473ec5c to
0b78a1d
Compare
0b78a1d to
26a162f
Compare
<initlabels> label sets take their slots at startup, so a limit smaller than their number is already exceeded before any record arrives: the metric could never take a new label set. Stop at startup instead. A record on one of those label sets is still counted, since the metric already holds it. A limit equal to the number of label sets is fine: every label set is known in advance. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
26a162f to
396ec8f
Compare
| def check_initlabels_fit_series_limit! | ||
| return if @max_series_per_metric <= 0 | ||
| # two <initlabels> blocks with the same values make one label set | ||
| return if @series_set.size <= @max_series_per_metric |
There was a problem hiding this comment.
check_initlabels_fit_series_limit! compares this section's limit against
@series_set.size, but that SeriesSet is shared by every <metric> section
with the same name (SeriesSet.of stores it on the client metric). So the
left side counts other sections' label sets while the right side is this
section's own limit, and the result depends on the order the sections are
configured in.
<metric>
name shared
max_series_per_metric 10
initialized true
<labels>
path $.path
</labels>
<initlabels> path /a </initlabels>
... /b /c /d /e
</metric>
<metric>
name shared # same name -> same SeriesSet
max_series_per_metric 3
initialized true
<labels>
path $.path
</labels>
<initlabels> path /z </initlabels>
</metric>
Configured in this order, the second section aborts startup:
A: @series_set.size=5 @max_series_per_metric=10 -> pass
B: @series_set.size=6 @max_series_per_metric=3 -> ConfigError
The 6 includes A's five label sets; B has a single <initlabels>. Swapping the
two sections starts fine, so the same configuration behaves differently
depending on section order, and the message points at the wrong section
("metric shared holds 6 label sets from " for a section that has
one).
Suggestion
Counting this section's own <initlabels> fixes it and matches what the method
name, the comment and the README already describe:
diff --git a/lib/fluent/plugin/prometheus.rb b/lib/fluent/plugin/prometheus.rb
index ae64c2d..55e1b68 100644
--- a/lib/fluent/plugin/prometheus.rb
+++ b/lib/fluent/plugin/prometheus.rb
@@ -472,7 +472,8 @@ def bind_series_set(client_metric)
def check_initlabels_fit_series_limit!
return if @max_series_per_metric <= 0
# two <initlabels> blocks with the same values make one label set
- return if @series_set.size <= @max_series_per_metric
+ size = @base_initlabels.map { |l| normalize_label_set(l) }.uniq.size
+ return if size <= @max_series_per_metric
raise ConfigError, "metric #{@name} holds #{@series_set.size} label sets from <initlabels>, " \
"but max_series_per_metric is #{@max_series_per_metric}: " \
uniq keeps the "two <initlabels> blocks with the same values make one label
set" behaviour, and normalize_label_set is needed because ${worker_id} is
stored as an Integer in @base_initlabels. The existing suite passes with this
change.
There was a problem hiding this comment.
Though suggested fix resolves order dependency, but it might add one more edge case (pass on startup, drop on runtime)
There was a problem hiding this comment.
Thus, it should be also fixed in follow-up.
|
Are these two edge cases in scope for this PR, or should we handle them in a follow-up? Both involve a second
Neither is a regression, and neither is covered by the change I suggested on the other thread, since both are about the shared count rather than the per-section one. I'm fine keeping this PR focused and addressing them later — what do you prefer? |
It might be better to handle in follow-up. |
<initlabels>label sets take their slots at startup, so a smaller limitfills the metric before any record arrives and drops every one of them.
Stop at startup instead. A limit equal to the number of label sets is
fine: every label set is known in advance.