Unexpected metric export increase

[phbnf]

Between

We, and other log operators running on GCP have observed increased metric exports since the v0.1.2 release on GCP, correlated with cardinality increase on rpc.client metrics. Thanks @Hayden-IO for the initial report.

Impact: elevated bills

Before:

After:

[phbnf]

rpc.client.call.duration is exported by default by underlying client libraries.

Due to a series of changes in these libraries and the otel libraries, the server_address metric field is now populated. This field is populated with individual IP addresses, which have a high turnover for services such as Spanner and SCS. In a ideal world, server_address would be hostname:port, and not a resolved IP address, but this is out of our control.

Until this is an issue, and to prevent elevated bills due, one can either:

• disable this metric export with --drop_metrics=rpc.client.call.duration (which will now be the default value)
• exclude collection of this metric in the metrics management dashboard (kudos to @Hayden-IO and @roger2hk for the tip!)