Commit graph

41 commits

Author SHA1 Message Date
Clayton O'Neill
c7e76967c8
Add internal event queuing and flushing
At high traffic levels, the locking around sending on channels can cause
a large amount of blocking and CPU usage.  These adds an event queue
mechanism so that events are queued for short period of time, and
flushed in batches to the main exporter goroutine periodically.

The default is is to flush every 1000 events, or every 200ms, whichever
happens first.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-06-04 06:57:48 -04:00
Clayton O'Neill
f6f1d7f071
Remove redundant code in counter inc test
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:16 -04:00
Clayton O'Neill
680eb0e826
Add additional tests
This adds two tests.

The first test is to validate that two successive counter events will
increment a counter.  This is more about ensuring we can look up the
same metric twice in a row than checking increment functionality.

The second test verifies that the hashLabels function returns different
results when labels are changed.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:16 -04:00
Clayton O'Neill
f6c0dd965b
Remove dead code
This removes all the code that the new registry code replaces.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:16 -04:00
Clayton O'Neill
7c30120dbc
Rework metric registration and tracking
This reworks/rewrites the way that metric registration and tracking is
handled across all of statsd_exporter.  The goal here is to reduce
memory and cpu usage, but also to reduce complexity by unifying metric
registration with the ttl tracking for expiration.

Some high level notes:

* Previously metric names and labels were being hashed three times for
every event accepted: in the container code, the save label set code and
again in the prometheus client libraries.  This unifies the first two
and caches the results of `GetMetricWith` to avoid the third.

* This optimizes the label hashing to reduce cpu overhead and memory
allocations.  The label hashing code previously showed up high on all
profiling done for both CPU and memory allocations

Using the BenchmarkExporterListener benchmark, the improvement looks
like this.

Before:
cpu: 11,341,797 ns/op
memory allocated: 1,731,119 B/op
memory allocations: 58,028 allocs/op

After:
cpu: 7,084,651 ns/op
memory allocated: 906,556 B/op
memory allocations: 42,026 allocs/op

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-31 08:38:15 -04:00
Benno Rice
f12451ee5b Add a test for adding labels from mapper configuration.
Signed-off-by: Benno Rice <benno@jeamland.net>
2019-05-28 10:06:17 +10:00
Clayton O'Neill
d143398343
Optimize label sorting
Previously we were sorting labels in every container to build a map key,
but also when generating the hash key by calling the `LabelsToSignature`
method.  This moves the sorting to be done early in the event processing
then passes the sorted label array around.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-16 17:17:09 -04:00
Clayton O'Neill
732be39b4b
Add benchmark for hashNameAndLabels
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-16 17:17:09 -04:00
Matthias Rampke
a294491e0b
Resolve merge conflicts of #198 with #212 and #213
In #198 the signature for setting up the exporter changed slightly, a
change that #212 and #213 didn't have. This doesn't change anything
substantial.

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2019-05-15 13:21:04 +00:00
Matthias Rampke
f502171bdc
Merge pull request #216 from claytono/conflicting-histogram-fix
Check for histogram conflict on main name
2019-05-15 13:20:11 +00:00
Matthias Rampke
698bcdf8c3
Merge pull request #198 from SpencerMalone/mapping-cache
Adding mapping cache
2019-05-15 13:05:00 +00:00
Clayton O'Neill
cce95f6980
Check for histogram conflict on main name
While it doesn't report it as a metric, when collecting, the registry
considers the base metric name to below to the histogram that's
registered.  This means that we need to prevent other metrics from
registering anything under this name, in addition to checking for the
bucket, sum and count suffixes.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-15 09:04:19 -04:00
Clayton O'Neill
3dcad090b3
Add test cases for conflicting metrics with labels
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-15 06:46:11 -04:00
Clayton O'Neill
9f1c6b81a5
Add checking for conflicting metrics
This adds sanity checking before registering a metric to ensure that the
metric isn't already registered under an existing name.  This prevents
the exporter from getting into a state where it has accepted and
registered conflicting metrics, and then cannot report the metrics it
has collected.

Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-14 18:33:49 -04:00
Matthias Rampke
d7eb1edeed
Merge pull request #210 from claytono/tag-parsing-optimizations
Tag parsing optimizations
2019-05-14 16:18:37 +00:00
Vitaliy Sakhartchouk
cddeb87405 Add default quantiles for summaries if no mapping file exists
Signed-off-by: Vitaliy Sakhartchouk <vsakhart@github.com>
2019-05-13 17:14:42 -07:00
Clayton O'Neill
9cd711ed3e
Add benchmark for tag parsing
Signed-off-by: Clayton O'Neill <claytono@github.com>
2019-05-13 12:46:49 -04:00
SpencerMalone
35d1a99592 Adding mapping cache
Signed-off-by: SpencerMalone <malone.spencer@gmail.com>
2019-04-26 18:00:39 -04:00
Matthias Rampke
ece9385f22
Merge pull request #197 from bakins/escapeMetricName-performance
Increase escapeMetricName performance
2019-04-10 08:47:49 +00:00
Brian Akins
7f70979120 Add Benchmarks for escapeMetricName
Signed-off-by: Brian Akins <brian@akins.org>
2019-04-09 11:58:18 -04:00
Vitaliy Sakhartchouk
2025b47cb1 Allow support for inconsistent label sets by marking metrics
registered as unchecked collectors

Signed-off-by: Vitaliy Sakhartchouk <vsakhart@github.com>
2019-03-29 13:18:31 -07:00
Ivan Izaguirre
51e735c878 Fix test error message
Signed-off-by: Ivan Izaguirre <ivanizag@gmail.com>
2019-03-26 01:02:38 +01:00
Ivan Izaguirre
c9e5e94ed2 Adds a separate counter for events discarded due to errors
Signed-off-by: Ivan Izaguirre <ivanizag@gmail.com>
2019-03-26 00:44:17 +01:00
Ivan Izaguirre
c477c7703f Fix format to comply with go1.11 rules
Signed-off-by: Ivan Izaguirre <ivanizag@gmail.com>
2019-03-23 19:47:25 +01:00
Ivan Izaguirre
1c71191aaf Fixes #191: crash on empty metric name
Signed-off-by: Ivan Izaguirre <ivanizag@gmail.com>
2019-03-23 19:38:27 +01:00
SpencerMalone
c73a7b2c9a Scale summaries from milliseconds to seconds.
Fixes #177

Updating exporter tests for new summary unit scaling behavior

Signed-off-by: Spencer Malone <malone.spencer@gmail.com>
2019-01-09 12:37:38 -05:00
Simon Pasquier
8f56cc811d *: add staticcheck target back
Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2019-01-04 15:26:57 +01:00
Ivan Mikheykin
699c11ca11 Rework tests to not depend on actual wall clocks
Signed-off-by: Ivan Mikheykin <ivan.mikheykin@flant.com>
2018-12-19 08:24:25 +03:00
Ivan Mikheykin
d1b2dd47a8 TTL expiration tests, README update
Signed-off-by: Ivan Mikheykin <ivan.mikheykin@flant.com>
2018-12-18 09:46:05 +03:00
Ivan Mikheykin
b638b9d808 Replace Metrics with Collectors
- use MetricVec family instead of Metric
- dynamic label values instead of ConstLabels
- use dto.Metric to gain histrogram value in exporter_test
- remove hash calculations

Signed-off-by: Ivan Mikheykin <ivan.mikheykin@flant.com>
2018-12-18 08:39:23 +03:00
Matthias Rampke
40a13e604f
Move mapper -> pkg/mapper
to make clear that this is meant to be used elsewhere

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2018-08-14 09:20:00 +00:00
Matthias Rampke
124c5b88d0
Move mapper to a package
This allows the mapping logic and configuration format to be re-used by
the graphite exporter.

Enables prometheus/graphite_exporter#37, cf. https://github.com/prometheus/graphite_exporter/pull/52#issuecomment-412324849

Signed-off-by: Matthias Rampke <mr@soundcloud.com>
2018-08-12 11:14:18 +00:00
Matthias Rampke
0ac908603a Update documentation of metric name escaping.
The escaping changed in 728bdc52ae, before
0.1.0. Add a test for the escape function, and test various cases,
including those mentioned in the README.

This reveals that the README is inaccurate, adjust it to what has been
the reality for many years. Fixes #97.
2017-11-10 20:34:10 +00:00
Matthias Rampke
ea77554dc4 Force IPv4 for the mock TCP listener.
Travis CI [does not have IPv6][0] in any environment. It could be
[enabled][1] if we ran the build in a VM, but that is much slower to
start. Instead, we just force the test connection to use IPv4.

[0]: https://blog.travis-ci.com/2015-11-27-moving-to-a-more-elastic-future#ipv6-no-longer-present
[1]: https://github.com/travis-ci/travis-ci/issues/5200#issuecomment-162539556
2017-11-10 14:21:38 +00:00
Dave Rawks
0478c40ab0 Removes -statsd.add-suffix option flag
* Remove option flag
* Update NewExporter call signature to not take suffix boolean
* Update tests to reflect new behavior and signature
* Update documentation to reflect option flag removal
2017-09-28 11:30:17 -07:00
Dave Rawks
06444ed5c0 Minor fix-ups per PR discussion
* Removed extraneous newline
  * Reformatted/ordered imports
  * Used hash function to derive index of `Elements` instead of
    hardcoding hash result
2017-08-15 08:51:13 -07:00
Dave Rawks
b2082eda2b Added test for histogram unit conversion 2017-08-14 15:20:45 -07:00
jwfang
07543ac557 Add TCP StatsD listener support (#71)
* add TCP StatsD listener support

* add listen-tcp flag to control UDP/TCP mode on same port

* statsdListenUDP/statsdListenTCP as string, and alias listen-address to listen-udp

* add stats for tcp error/line_too_long

* add test for TCP listener
2017-08-01 12:21:00 +02:00
Ilya Margolin
bb4e42068f Skip metrics with invalid utf8 2016-07-20 20:27:23 +02:00
Johannes 'fish' Ziemke
4d22a721eb Fix parameter to NewExporter in test 2016-05-03 23:45:48 +02:00
Jonathon Klobucar
494cc38bf8 Don't panic when negative number counter is submitted.
Instead now we will send a warning message to the logs
and continue on with our day.

Also a guard has been added when we detect a channel has been
closed and the `Exporter.Listen` function will return out of
its loop vs always running. When a channel is closed there will
be no more data to consume, so this seems correct. It also allows
for a test to be written for the panic.
2016-04-01 10:53:20 -07:00