Commit graph

464 commits

Author SHA1 Message Date
Matthias Rampke
95e376a40b
Merge pull request #230 from prometheus/makefile_common
Synchronize Makefile.common from prometheus/prometheus
2019-06-07 08:49:14 +02:00
prombot
1d21cdcc1e makefile: update Makefile.common with newer version
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-06-07 00:02:26 +00:00
Matthias Rampke
59056c097a
update changelog for #229 2019-06-06 12:34:05 +00:00
Matthias Rampke
8551a65827
Merge pull request #229 from andy-paine/make-mapper-cache-respect-metric-type
Make mapper cache respect metric type
2019-06-06 14:31:58 +02:00
Andy Paine
0135b40c08 Make mapper cache respect metric type
- Statsd allows users to provide a metric with the same name but
  differing types (counter, gauge, timer)
- The exporter allows this by letting users specify a
  "match_metric_type" in the mapping config
- However the mapper cache does not look at the metric type so it would
  return a MetricMapperCacheResult for the type of the first metric with
that name that the exporter saw
- Add MetricType to the signature for the cache and format the metric
  name with the type to provide unique keys for a metric with the same
name but differing type

Signed-off-by: Andy Paine <andy.paine@digital.cabinet-office.gov.uk>
2019-06-06 11:33:58 +01:00
Clayton O'Neill
c7e76967c8
Add internal event queuing and flushing
At high traffic levels, the locking around sending on channels can cause
a large amount of blocking and CPU usage.  These adds an event queue
mechanism so that events are queued for short period of time, and
flushed in batches to the main exporter goroutine periodically.

The default is is to flush every 1000 events, or every 200ms, whichever
happens first.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-06-04 06:57:48 -04:00
Matthias Rampke
50d5932124
Merge pull request #223 from claytono/rework-registry
Rework metric registration and tracking
2019-06-04 10:34:13 +02:00
Clayton O'Neill
f6f1d7f071
Remove redundant code in counter inc test
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:16 -04:00
Clayton O'Neill
a241eb0b69
Remove vector cleanup from registry
This was attempting to clean up any vectors that no longer have metrics
associated with them.  Unfortunately this isn't really possible because
the prometheus client registry allows the second registration of the
same metric, and then throws an error when gathering the metrics.

Another options would be to unregister the metric with the client
library, but that's not implemented for unchecked collectors.

Unfortunately this means that if we have a lot of metrics with differing
label sets appearing and disappearing, that we'll leak some amount of
memory for vectors that are never used again, but this is the way the
old metric tracking code worked also.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:16 -04:00
Clayton O'Neill
680eb0e826
Add additional tests
This adds two tests.

The first test is to validate that two successive counter events will
increment a counter.  This is more about ensuring we can look up the
same metric twice in a row than checking increment functionality.

The second test verifies that the hashLabels function returns different
results when labels are changed.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:16 -04:00
Clayton O'Neill
f6c0dd965b
Remove dead code
This removes all the code that the new registry code replaces.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:16 -04:00
Clayton O'Neill
7c30120dbc
Rework metric registration and tracking
This reworks/rewrites the way that metric registration and tracking is
handled across all of statsd_exporter.  The goal here is to reduce
memory and cpu usage, but also to reduce complexity by unifying metric
registration with the ttl tracking for expiration.

Some high level notes:

* Previously metric names and labels were being hashed three times for
every event accepted: in the container code, the save label set code and
again in the prometheus client libraries.  This unifies the first two
and caches the results of `GetMetricWith` to avoid the third.

* This optimizes the label hashing to reduce cpu overhead and memory
allocations.  The label hashing code previously showed up high on all
profiling done for both CPU and memory allocations

Using the BenchmarkExporterListener benchmark, the improvement looks
like this.

Before:
cpu: 11,341,797 ns/op
memory allocated: 1,731,119 B/op
memory allocations: 58,028 allocs/op

After:
cpu: 7,084,651 ns/op
memory allocated: 906,556 B/op
memory allocations: 42,026 allocs/op

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:15 -04:00
Clayton O'Neill
7a107f899f
Add benchmark for exporter listener
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:15 -04:00
Matthias Rampke
d74922e0fd
Merge pull request #226 from simonpasquier/use-org-context
Use Circle CI's org context
2019-05-31 08:47:05 +00:00
Simon Pasquier
83cd28067a Use Circle CI's org context
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-05-29 18:00:06 +02:00
Matthias Rampke
bfa30a94b2
Merge pull request #225 from jeamland/add_label_test
Add a test for adding labels from mapper configuration.
2019-05-28 08:26:08 +00:00
Benno Rice
f12451ee5b Add a test for adding labels from mapper configuration.
Signed-off-by: Benno Rice <benno@jeamland.net>
2019-05-28 10:06:17 +10:00
Matthias Rampke
09c2603f4b
Update changelog for #224, release 0.10.5
Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-27 09:09:59 +00:00
Matthias Rampke
56a091a693
Merge pull request #224 from ptqa/master
Fix inconsistent label cardinality error
2019-05-27 09:06:29 +00:00
Tony Nyurkin
7d6244987a Fix inconsistent label cardinality error
Signed-off-by: Tony Nyurkin <ptqa@users.noreply.github.com>
2019-05-24 14:26:17 +03:00
Matthias Rampke
b333ecaacc
Add changelog for #221 and release 0.10.4
Reverting #218 due to race condition issue #220

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-20 08:35:36 +00:00
Matthias Rampke
3abf21e053
Merge pull request #221 from prometheus/revert-218-handlepacket-optimization
Revert "Reduce memory allocations in handlePacket"
2019-05-20 08:34:09 +00:00
Matthias Rampke
26e9d482db
Revert "Reduce memory allocations in handlePacket" 2019-05-20 08:20:19 +00:00
Matthias Rampke
471b28dc21
Update changelog for #217, #218, #219 and release
Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-17 13:51:13 +00:00
Matthias Rampke
c9004f8f3f
Merge pull request #219 from claytono/optimize-label-handling
Optimize label sorting
2019-05-17 13:47:03 +00:00
Matthias Rampke
4e64da2a41
Merge pull request #218 from claytono/handlepacket-optimization
Reduce memory allocations in handlePacket
2019-05-17 13:38:56 +00:00
Matthias Rampke
27d9273107
Merge pull request #217 from claytono/emn-optimization
Convert escapeMetricName to use strings.Builder
2019-05-17 13:38:09 +00:00
Matthias Rampke
cf4290bf7e
Update CircleCI config for prometheus/prometheus#5031
multi-arch \m/

Release 0.10.2 with this. Hopefully.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-17 08:39:50 +00:00
Matthias Rampke
df81d15e84
Bump to 0.10.1
I never finished releasing 0.10.0 because the build was broken, but I
don't want to re-use that tag.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-17 08:04:55 +00:00
Clayton O'Neill
d143398343
Optimize label sorting
Previously we were sorting labels in every container to build a map key,
but also when generating the hash key by calling the `LabelsToSignature`
method.  This moves the sorting to be done early in the event processing
then passes the sorted label array around.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-16 17:17:09 -04:00
Clayton O'Neill
732be39b4b
Add benchmark for hashNameAndLabels
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-16 17:17:09 -04:00
Clayton O'Neill
4cf6f74dc3
Reduce memory allocations in handlePacket
This converts the byte buffer to a string by casting and parses the
string by looping over it instead of calling Split.

This usage of unsafe is taken from the strings.Builder package here:
cb5c82bc3d/src/strings/builder.go (L47)

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-16 16:35:32 -04:00
Clayton O'Neill
98da6f7057
Convert escapeMetricName to use strings.Builder
This converts escapeMetricName to use strings.Builder instead of
allocating a byte array, filling it and then converting it to a string.
This also optimizes for the case where the metricName is already valid,
and in that case, it just returns the original string.

This reduces memory allocations from 2 per call to 1 per call in the
case when the string does need to be escaped, and reduces it to zero
memory allocations when the string is already valid.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-16 09:28:35 -04:00
Matthias Rampke
a294491e0b
Resolve merge conflicts of #198 with #212 and #213
In #198 the signature for setting up the exporter changed slightly, a
change that #212 and #213 didn't have. This doesn't change anything
substantial.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-15 13:21:04 +00:00
Matthias Rampke
f502171bdc
Merge pull request #216 from claytono/conflicting-histogram-fix
Check for histogram conflict on main name
2019-05-15 13:20:11 +00:00
Matthias Rampke
8cc291f419
Release v0.10.0
* [CHANGE] Do not run as root in the Docker container by default ([#202](https://github.com/prometheus/statsd_exporter/pull/202))
* [FEATURE] Add metric for count of events by action ([#193](https://github.com/prometheus/statsd_exporter/pull/193))
* [FEATURE] Add metric for count of distinct metric names ([#200](https://github.com/prometheus/statsd_exporter/pull/200))
* [FEATURE] Add UNIX socket listener support ([#199](https://github.com/prometheus/statsd_exporter/pull/199))
* [FEATURE] Accept Datadog [distributions](https://docs.datadoghq.com/graphing/metrics/distributions/) ([#211](https://github.com/prometheus/statsd_exporter/pull/211))
* [ENHANCEMENT] Add a health check to the Docker container ([#182](https://github.com/prometheus/statsd_exporter/pull/182))
* [ENHANCEMENT] Allow inconsistent label sets ([#194](https://github.com/prometheus/statsd_exporter/pull/194))
* [ENHANCEMENT] Speed up sanitization of metric names ([#197](https://github.com/prometheus/statsd_exporter/pull/197))
* [ENHANCEMENT] Enable pprof endpoints ([#205](https://github.com/prometheus/statsd_exporter/pull/205))
* [ENHANCEMENT] DogStatsD tag parsing is faster ([#210](https://github.com/prometheus/statsd_exporter/pull/210))
* [ENHANCEMENT] Cache mapped metrics ([#198](https://github.com/prometheus/statsd_exporter/pull/198))
* [BUGFIX] Fix panic if a mapping resulted in an empty name ([#192](https://github.com/prometheus/statsd_exporter/pull/192))
* [BUGFIX] Ensure that there are always default quantiles if using summaries ([#212](https://github.com/prometheus/statsd_exporter/pull/212))
* [BUGFIX] Prevent ingesting conflicting metric types that would make scraping fail ([#213](https://github.com/prometheus/statsd_exporter/pull/213))

With #192, the count of events rejected because of negative counter increments has moved into the `statsd_exporter_events_error_total` metric, instead of being lumped in with the different kinds of successful events.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-15 13:09:41 +00:00
Matthias Rampke
228a969b1a
Add changelog entry for #198
Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-15 13:06:04 +00:00
Matthias Rampke
698bcdf8c3
Merge pull request #198 from SpencerMalone/mapping-cache
Adding mapping cache
2019-05-15 13:05:00 +00:00
Clayton O'Neill
cce95f6980
Check for histogram conflict on main name
While it doesn't report it as a metric, when collecting, the registry
considers the base metric name to below to the histogram that's
registered.  This means that we need to prevent other metrics from
registering anything under this name, in addition to checking for the
bucket, sum and count suffixes.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-15 09:04:19 -04:00
Matthias Rampke
a42de85289
Add changelog entry for #213
Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-15 11:22:17 +00:00
Matthias Rampke
04b7b71a2a
Merge pull request #213 from claytono/conflicting-metrics-test
Add checking for conflicting metrics
2019-05-15 11:20:56 +00:00
Clayton O'Neill
3dcad090b3
Add test cases for conflicting metrics with labels
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-15 06:46:11 -04:00
Matthias Rampke
24e288a3a4
Merge pull request #214 from prometheus/makefile_common
Synchronize Makefile.common from prometheus/prometheus
2019-05-15 07:40:06 +00:00
Ben Kochie
0eadae3ca7
Remove obsolete release tool.
Signed-off-by: Ben Kochie <superq@gmail.com>
2019-05-15 02:07:49 +02:00
prombot
7514e37e5b makefile: update Makefile.common with newer version
Signed-off-by: prombot <prometheus-team@googlegroups.com>
2019-05-15 00:01:34 +00:00
Clayton O'Neill
9f1c6b81a5
Add checking for conflicting metrics
This adds sanity checking before registering a metric to ensure that the
metric isn't already registered under an existing name.  This prevents
the exporter from getting into a state where it has accepted and
registered conflicting metrics, and then cannot report the metrics it
has collected.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-14 18:33:49 -04:00
Matthias Rampke
f08bf14965
Update changelog for #210, #211, #212
Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-14 16:24:33 +00:00
Matthias Rampke
bee73cbb9e
Merge pull request #211 from claytono/add-distributions
Add support for Datadog distribution type
2019-05-14 16:23:07 +00:00
Matthias Rampke
d7eb1edeed
Merge pull request #210 from claytono/tag-parsing-optimizations
Tag parsing optimizations
2019-05-14 16:18:37 +00:00
Matthias Rampke
f1c0052ce7
Merge pull request #212 from vsakhart/quantile_fix
Add default quantiles for summaries if no mapping file exists
2019-05-14 16:14:15 +00:00