The Go client for prometheus aggregates summary metrics over 10
minutes by default, in 5 buckets. This is not always the behaviour we
want.
Allow tweaking those settings in `statsd_exporter`, so we can
aggregate summary metrics over more or less time, with more or fewer
buckets, and set the cap for the bucket as well.
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
Currently the Buckets and Quantiles settings are top level settings per metric
in the yaml. In a subsequent commit we're going to allow adding more such
options for summaries, at which point having them all at the top level gets
confusing.
Split the options out into separate hierarchies to allow adding more options,
without adding confusion. We preserve backwards compatibility by still
accepting the old option, but warning when it is present.
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
lastRegistredAt element was only set at creation of a time series.
Now it is updated every time a new event is seen for that time series.
Signed-off-by: Cedric Den Haese <cedric.den.haese@gmail.com>
This was attempting to clean up any vectors that no longer have metrics
associated with them. Unfortunately this isn't really possible because
the prometheus client registry allows the second registration of the
same metric, and then throws an error when gathering the metrics.
Another options would be to unregister the metric with the client
library, but that's not implemented for unchecked collectors.
Unfortunately this means that if we have a lot of metrics with differing
label sets appearing and disappearing, that we'll leak some amount of
memory for vectors that are never used again, but this is the way the
old metric tracking code worked also.
Signed-off-by: Clayton O'Neill <claytono@github.com>
This reworks/rewrites the way that metric registration and tracking is
handled across all of statsd_exporter. The goal here is to reduce
memory and cpu usage, but also to reduce complexity by unifying metric
registration with the ttl tracking for expiration.
Some high level notes:
* Previously metric names and labels were being hashed three times for
every event accepted: in the container code, the save label set code and
again in the prometheus client libraries. This unifies the first two
and caches the results of `GetMetricWith` to avoid the third.
* This optimizes the label hashing to reduce cpu overhead and memory
allocations. The label hashing code previously showed up high on all
profiling done for both CPU and memory allocations
Using the BenchmarkExporterListener benchmark, the improvement looks
like this.
Before:
cpu: 11,341,797 ns/op
memory allocated: 1,731,119 B/op
memory allocations: 58,028 allocs/op
After:
cpu: 7,084,651 ns/op
memory allocated: 906,556 B/op
memory allocations: 42,026 allocs/op
Signed-off-by: Clayton O'Neill <claytono@github.com>
