Introduce max_series_per_metric limitation

[kenhys]

Before:
a metric held one label set per distinct label value a record
carried. About 8 million records took the RSS from 64MB to 582MB, which
is an OOM DoS a client can drive.

After:
max_series_per_metric drops a new label set once the metric holds
that many of them, and the same records take the RSS from 64MB to 84MB.
Set it on the plugin, or per . It defaults to 0 (no limit),
because a drop loses the record for good.

[kenhys]

It should be rebased after #264 was merged.

[kenhys]

Consistency with initlabels should be fixed in follow-up #266

[Watson1978]

with_label_set gives the reserved slot back whenever the block raises, on the
assumption that the client call is all-or-nothing. Summary#observe breaks that
assumption:

def observe(value, labels: {})
  base_label_set = label_set_for(labels)

  @store.synchronize do
    @store.increment(labels: base_label_set.merge(quantile: "count"), by: 1)
    @store.increment(labels: base_label_set.merge(quantile: "sum"), by: value)
  end
end

https://github.com/prometheus/client_ruby/blob/9c452538f4c9e06fb32a4fa5acc903652227bc52/lib/prometheus/client/summary.rb#L20-L27

Suggestion:

if value
  value = Float(value)
  with_label_set(record, expander) do |label|
    @summary.observe(value, labels: label)
  end
end

Float() raises before with_label_set, so a bad record never takes a slot and
still reaches @ERROR. I would not use to_f here: it turns "-", "" and
"abc" into 0.0, which silently pollutes the observations instead of
surfacing them.

[kenhys]

As value=Float(value) passthough numeric string, it is behavior change. so changed to validate it before entering with_label_set.

[Watson1978]

Could this section also note that the limit is per worker process? @registry is
::Prometheus::Client.registry, which is process local, so with workers N a
metric can hold up to N times max_series_per_metric label sets in total.

[Watson1978]

👍🏻