mirrors/statsd_exporter
1
0
Fork 1
mirror of https://github.com/prometheus/statsd_exporter.git synced 2024-05-21 18:48:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' of github.com:prometheus/statsd_exporter into signalfx

Browse source
This commit is contained in:
glightfoot 2020-06-19 09:05:10 -04:00
parent aa591e3e8e f1de97dbdd
commit 4f7abe5226
15 changed files with 507 additions and 260 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

16
CHANGELOG.md
View file

@ -1,6 +1,18 @@
## 0.17.0 / unreleased
## 0.17.0 / Unreleased
* [FEATURE] Offline configuration check ([#312](https://github.com/prometheus/statsd_exporter/pull/312))
* [CHANGE] Support non-timer distributions without unit conversion ([#314](https://github.com/prometheus/statsd_exporter/pull/314))
* [ENHANCEMENT] Offline configuration check ([#312](https://github.com/prometheus/statsd_exporter/pull/312))
* [BUGFIX] Allow matching single-letter metric name components ([#309](https://github.com/prometheus/statsd_exporter/pull/309))
Distribution and histogram events (type `d`, `h`) are now treated as distinct from timer events (type `ms`).
Their values are observed as they are, while timer events are converted from milliseconds to seconds.
To reflect this generalization, the `observer_type` mapping option replaces `timer_type`.
Similary, change `match_metric_type: timer` to `match_metric_type: observer`.
The old name remains available for compatibility.
For users of the mapper library, the `ObserverEvent` replaces `TimerEvent`.
For timer metrics, it is emitted by the mapper already converted to seconds.
## 0.16.0 / 2020-05-29

65
README.md
View file

@ -162,9 +162,7 @@ In general, the different metric types are translated as follows:
StatsD counter -> Prometheus counter
StatsD timer -> Prometheus summary <-- indicates timer quantiles
-> Prometheus counter (suffix `_total`) <-- indicates total time spent
-> Prometheus counter (suffix `_count`) <-- indicates total number of timer events
StatsD timer, histogram, distribution -> Prometheus summary or histogram
An example mapping configuration:
@ -246,17 +244,18 @@ mappings:
code: "$1"
```
### StatsD timers
### StatsD timers and distributions
By default, statsd timers are represented as a Prometheus summary with
quantiles. You may optionally configure the [quantiles and acceptable
error](https://prometheus.io/docs/practices/histograms/#quantiles), as
well as adjusting how the summary metric is aggregated:
By default, statsd timers and distributions (collectively "observers") are
represented as a Prometheus summary with quantiles. You may optionally
configure the [quantiles and acceptable
error](https://prometheus.io/docs/practices/histograms/#quantiles), as well
as adjusting how the summary metric is aggregated:
```yaml
mappings:
- match: "test.timing.*.*.*"
timer_type: summary
observer_type: summary
name: "my_timer"
labels:
provider: "$2"
@ -280,19 +279,17 @@ mappings:
The default quantiles are 0.99, 0.9, and 0.5.
The default summary age is 10 minutes, the default number of buckets
is 5 and the default buffer size is 500. See also the
[`golang_client` docs](https://godoc.org/github.com/prometheus/client_golang/prometheus#SummaryOpts).
The `max_summary_age` corresponds to `SummaryOptions.MaxAge`, `summary_age_buckets`
to `SummaryOptions.AgeBuckets` and `stream_buffer_size` to `SummaryOptions.BufCap`.
is 5 and the default buffer size is 500.
See also the [`golang_client` docs](https://godoc.org/github.com/prometheus/client_golang/prometheus#SummaryOpts).
The `max_summary_age` corresponds to `SummaryOptions.MaxAge`, `summary_age_buckets` to `SummaryOptions.AgeBuckets` and `stream_buffer_size` to `SummaryOptions.BufCap`.
In the configuration, one may also set the timer type to "histogram". The
default is "summary" as in the plain text configuration format. For example,
to set the timer type for a single metric:
In the configuration, one may also set the observer type to "histogram". For example,
to set the observer type for a single timer metric:
```yaml
mappings:
- match: "test.timing.*.*.*"
timer_type: histogram
observer_type: histogram
histogram_options:
buckets: [ 0.01, 0.025, 0.05, 0.1 ]
name: "my_timer"
@ -302,18 +299,18 @@ mappings:
job: "${1}_server"
```
Note that timers will be accepted with the `ms`, `h`, and `d` statsd types. The first two are timers and histograms and the `d` type is for DataDog's "distribution" type. The distribution type is treated identically to timers and histograms.
Timers will be accepted with the `ms` statsd type.
Statsd timer data is transmitted in milliseconds, while Prometheus expects the unit to be seconds.
The exporter converts all timer observations to seconds.
It should be noted that whereas timers in statsd expects the unit of timing data to be in milliseconds,
prometheus expects the unit to be seconds. Hence, the exporter converts all timers to seconds
before exporting them.
Histogram and distribution events (`h` and `d` metric type) are not subject to unit conversion.
### DogStatsD Client Behavior
#### `timed()` decorator
If you are using the DogStatsD client's [timed](https://datadogpy.readthedocs.io/en/latest/#datadog.threadstats.base.ThreadStats.timed) decorator,
it emits the metric in seconds, set [use_ms](https://datadogpy.readthedocs.io/en/latest/index.html?highlight=use_ms) to `True` to fix this.
The DogStatsD client's [timed](https://datadogpy.readthedocs.io/en/latest/#datadog.threadstats.base.ThreadStats.timed) decorator emits the metric in seconds but uses the `ms` type.
Set [`use_ms=True`](https://datadogpy.readthedocs.io/en/latest/index.html?highlight=use_ms) to send the correct units.
### Regular expression matching
@ -339,20 +336,20 @@ Note, that one may also set the histogram buckets. If not set, then the default
[Prometheus client values](https://godoc.org/github.com/prometheus/client_golang/prometheus#pkg-variables) are used: `[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]`. `+Inf` is added
automatically.
`timer_type` is only used when the statsd metric type is a timer. `buckets` is
only used when the statsd metric type is a timerand the `timer_type` is set to
"histogram."
`observer_type` is only used when the statsd metric type is a timer, histogram, or distribution.
`buckets` is only used when the statsd metric type is one of these, and the `observer_type` is set to `histogram`.
### Global defaults
One may also set defaults for the timer type, buckets or quantiles, and match_type. These will be used
by all mappings that do not define these.
One may also set defaults for the observer type, buckets or quantiles, and match type.
These will be used by all mappings that do not define them.
An option that can only be configured in `defaults` is `glob_disable_ordering`, which is `false` if omitted. By setting this to `true`, `glob` match type will not honor the occurance of rules in the mapping rules file and always treat `*` as lower priority than a general string.
An option that can only be configured in `defaults` is `glob_disable_ordering`, which is `false` if omitted.
By setting this to `true`, `glob` match type will not honor the occurance of rules in the mapping rules file and always treat `*` as lower priority than a concrete string.
```yaml
defaults:
timer_type: histogram
observer_type: histogram
buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5 ]
match_type: glob
glob_disable_ordering: false
@ -366,9 +363,9 @@ mappings:
outcome: "$3"
job: "${1}_server"
# This will be a summary timer.
- match: "other.timing.*.*.*"
timer_type: summary
name: "other_timer"
- match: "other.distribution.*.*.*"
observer_type: summary
name: "other_distribution"
labels:
provider: "$2"
outcome: "$3"
@ -437,7 +434,7 @@ mappings:
provider: "$1"
```
Possible values for `match_metric_type` are `gauge`, `counter` and `timer`.
Possible values for `match_metric_type` are `gauge`, `counter` and `observer`.
### Mapping cache size and cache replacement policy

184
bridge_test.go
View file

@ -84,60 +84,60 @@ func TestHandlePacket(t *testing.T) {
name: "simple timer",
in: "foo:200|ms",
out: event.Events{
&event.TimerEvent{
TMetricName: "foo",
TValue: 200,
TLabels: map[string]string{},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.2,
OLabels: map[string]string{},
},
},
}, {
name: "simple histogram",
in: "foo:200|h",
out: event.Events{
&event.TimerEvent{
TMetricName: "foo",
TValue: 200,
TLabels: map[string]string{},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 200,
OLabels: map[string]string{},
},
},
}, {
name: "simple distribution",
in: "foo:200|d",
out: event.Events{
&event.TimerEvent{
TMetricName: "foo",
TValue: 200,
TLabels: map[string]string{},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 200,
OLabels: map[string]string{},
},
},
}, {
name: "distribution with sampling",
in: "foo:0.01|d|@0.2|#tag1:bar,#tag2:baz",
out: event.Events{
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
},
}, {
@ -300,30 +300,30 @@ func TestHandlePacket(t *testing.T) {
name: "histogram with sampling",
in: "foo:0.01|h|@0.2|#tag1:bar,#tag2:baz",
out: event.Events{
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.01,
OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
},
},
}, {
@ -356,15 +356,15 @@ func TestHandlePacket(t *testing.T) {
name: "combined multiline metrics",
in: "foo:200|ms:300|ms:5|c|@0.1:6|g\nbar:1|c:5|ms",
out: event.Events{
&event.TimerEvent{
TMetricName: "foo",
TValue: 200,
TLabels: map[string]string{},
&event.ObserverEvent{
OMetricName: "foo",
OValue: .200,
OLabels: map[string]string{},
},
&event.TimerEvent{
TMetricName: "foo",
TValue: 300,
TLabels: map[string]string{},
&event.ObserverEvent{
OMetricName: "foo",
OValue: .300,
OLabels: map[string]string{},
},
&event.CounterEvent{
CMetricName: "foo",
@ -381,26 +381,26 @@ func TestHandlePacket(t *testing.T) {
CValue: 1,
CLabels: map[string]string{},
},
&event.TimerEvent{
TMetricName: "bar",
TValue: 5,
TLabels: map[string]string{},
&event.ObserverEvent{
OMetricName: "bar",
OValue: .005,
OLabels: map[string]string{},
},
},
}, {
name: "timings with sampling factor",
in: "foo.timing:0.5|ms|@0.1",
out: event.Events{
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
},
}, {
name: "bad line",
@ -457,6 +457,36 @@ func TestHandlePacket(t *testing.T) {
CLabels: map[string]string{},
},
},
}, {
name: "ms timer with conversion to seconds",
in: "foo:200|ms",
out: event.Events{
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.2,
OLabels: map[string]string{},
},
},
}, {
name: "histogram with no unit conversion",
in: "foo:200|h",
out: event.Events{
&event.ObserverEvent{
OMetricName: "foo",
OValue: 200,
OLabels: map[string]string{},
},
},
}, {
name: "distribution with no unit conversion",
in: "foo:200|d",
out: event.Events{
&event.ObserverEvent{
OMetricName: "foo",
OValue: 200,
OLabels: map[string]string{},
},
},
},
}
@ -589,9 +619,9 @@ mappings:
GValue: 200,
},
// event with ttl = 2s from a mapping
&event.TimerEvent{
TMetricName: "bazqux.main",
TValue: 42000,
&event.ObserverEvent{
OMetricName: "bazqux.main",
OValue: 42,
},
}

12
exporter_benchmark_test.go
View file

@ -87,13 +87,13 @@ func BenchmarkExporterListener(b *testing.B) {
GMetricName: "gauge",
GValue: 10,
},
&event.TimerEvent{ // simple timer
TMetricName: "timer",
TValue: 200,
&event.ObserverEvent{ // simple timer
OMetricName: "timer",
OValue: 200,
},
&event.TimerEvent{ // simple histogram
TMetricName: "histogram.test",
TValue: 200,
&event.ObserverEvent{ // simple histogram
OMetricName: "histogram.test",
OValue: 200,
},
&event.CounterEvent{ // simple_tags
CMetricName: "simple_tags",

16
pkg/event/event.go
View file

@ -52,16 +52,16 @@ func (g *GaugeEvent) Value() float64 { return g.GValue }
func (c *GaugeEvent) Labels() map[string]string { return c.GLabels }
func (c *GaugeEvent) MetricType() mapper.MetricType { return mapper.MetricTypeGauge }
type TimerEvent struct {
TMetricName string
TValue float64
TLabels map[string]string
type ObserverEvent struct {
OMetricName string
OValue float64
OLabels map[string]string
}
func (t *TimerEvent) MetricName() string { return t.TMetricName }
func (t *TimerEvent) Value() float64 { return t.TValue }
func (c *TimerEvent) Labels() map[string]string { return c.TLabels }
func (c *TimerEvent) MetricType() mapper.MetricType { return mapper.MetricTypeTimer }
func (t *ObserverEvent) MetricName() string { return t.OMetricName }
func (t *ObserverEvent) Value() float64 { return t.OValue }
func (c *ObserverEvent) Labels() map[string]string { return c.OLabels }
func (c *ObserverEvent) MetricType() mapper.MetricType { return mapper.MetricTypeObserver }
type Events []Event

28
pkg/exporter/exporter.go
View file

@ -140,38 +140,38 @@ func (b *Exporter) handleEvent(thisEvent event.Event) {
b.ConflictingEventStats.WithLabelValues("gauge").Inc()
}
case *event.TimerEvent:
t := mapper.TimerTypeDefault
case *event.ObserverEvent:
t := mapper.ObserverTypeDefault
if mapping != nil {
t = mapping.TimerType
t = mapping.ObserverType
}
if t == mapper.TimerTypeDefault {
t = b.Mapper.Defaults.TimerType
if t == mapper.ObserverTypeDefault {
t = b.Mapper.Defaults.ObserverType
}
switch t {
case mapper.TimerTypeHistogram:
case mapper.ObserverTypeHistogram:
histogram, err := b.Registry.GetHistogram(metricName, prometheusLabels, help, mapping, b.MetricsCount)
if err == nil {
histogram.Observe(thisEvent.Value() / 1000) // prometheus presumes seconds, statsd millisecond
b.EventStats.WithLabelValues("timer").Inc()
histogram.Observe(thisEvent.Value())
b.EventStats.WithLabelValues("observer").Inc()
} else {
level.Debug(b.Logger).Log("msg", regErrF, "metric", metricName, "error", err)
b.ConflictingEventStats.WithLabelValues("timer").Inc()
b.ConflictingEventStats.WithLabelValues("observer").Inc()
}
case mapper.TimerTypeDefault, mapper.TimerTypeSummary:
case mapper.ObserverTypeDefault, mapper.ObserverTypeSummary:
summary, err := b.Registry.GetSummary(metricName, prometheusLabels, help, mapping, b.MetricsCount)
if err == nil {
summary.Observe(thisEvent.Value() / 1000) // prometheus presumes seconds, statsd millisecond
b.EventStats.WithLabelValues("timer").Inc()
summary.Observe(thisEvent.Value())
b.EventStats.WithLabelValues("observer").Inc()
} else {
level.Debug(b.Logger).Log("msg", regErrF, "metric", metricName, "error", err)
b.ConflictingEventStats.WithLabelValues("timer").Inc()
b.ConflictingEventStats.WithLabelValues("observer").Inc()
}
default:
level.Error(b.Logger).Log("msg", "unknown timer type", "type", t)
level.Error(b.Logger).Log("msg", "unknown observer type", "type", t)
os.Exit(1)
}

98
pkg/exporter/exporter_test.go
View file

@ -228,25 +228,25 @@ func TestInconsistentLabelSets(t *testing.T) {
GValue: 1,
GLabels: secondLabelSet,
},
&event.TimerEvent{
TMetricName: "histogram.test",
TValue: 1,
TLabels: firstLabelSet,
&event.ObserverEvent{
OMetricName: "histogram.test",
OValue: 1,
OLabels: firstLabelSet,
},
&event.TimerEvent{
TMetricName: "histogram.test",
TValue: 1,
TLabels: secondLabelSet,
&event.ObserverEvent{
OMetricName: "histogram.test",
OValue: 1,
OLabels: secondLabelSet,
},
&event.TimerEvent{
TMetricName: "summary_test",
TValue: 1,
TLabels: firstLabelSet,
&event.ObserverEvent{
OMetricName: "summary_test",
OValue: 1,
OLabels: firstLabelSet,
},
&event.TimerEvent{
TMetricName: "summary_test",
TValue: 1,
TLabels: secondLabelSet,
&event.ObserverEvent{
OMetricName: "summary_test",
OValue: 1,
OLabels: secondLabelSet,
},
}
events <- c
@ -427,9 +427,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test1",
CValue: 1,
},
&event.TimerEvent{
TMetricName: "histogram.test1",
TValue: 2,
&event.ObserverEvent{
OMetricName: "histogram.test1",
OValue: 2,
},
},
},
@ -441,9 +441,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test1_sum",
CValue: 1,
},
&event.TimerEvent{
TMetricName: "histogram.test1",
TValue: 2,
&event.ObserverEvent{
OMetricName: "histogram.test1",
OValue: 2,
},
},
},
@ -455,9 +455,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test2_count",
CValue: 1,
},
&event.TimerEvent{
TMetricName: "histogram.test2",
TValue: 2,
&event.ObserverEvent{
OMetricName: "histogram.test2",
OValue: 2,
},
},
},
@ -469,9 +469,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test3_bucket",
CValue: 1,
},
&event.TimerEvent{
TMetricName: "histogram.test3",
TValue: 2,
&event.ObserverEvent{
OMetricName: "histogram.test3",
OValue: 2,
},
},
},
@ -483,9 +483,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "cvsq_test",
CValue: 1,
},
&event.TimerEvent{
TMetricName: "cvsq_test",
TValue: 2,
&event.ObserverEvent{
OMetricName: "cvsq_test",
OValue: 2,
},
},
},
@ -497,9 +497,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "cvsc_count",
CValue: 1,
},
&event.TimerEvent{
TMetricName: "cvsc",
TValue: 2,
&event.ObserverEvent{
OMetricName: "cvsc",
OValue: 2,
},
},
},
@ -511,9 +511,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "cvss_sum",
CValue: 1,
},
&event.TimerEvent{
TMetricName: "cvss",
TValue: 2,
&event.ObserverEvent{
OMetricName: "cvss",
OValue: 2,
},
},
},
@ -672,9 +672,9 @@ func TestSummaryWithQuantilesEmptyMapping(t *testing.T) {
name := "default_foo"
c := event.Events{
&event.TimerEvent{
TMetricName: name,
TValue: 300,
&event.ObserverEvent{
OMetricName: name,
OValue: 300,
},
}
events <- c
@ -711,7 +711,7 @@ func TestHistogramUnits(t *testing.T) {
testMapper := mapper.MetricMapper{}
testMapper.InitCache(0)
ex := NewExporter(&testMapper, log.NewNopLogger(), eventsActions, eventsUnmapped, errorEventStats, eventStats, conflictingEventStats, metricsCount)
ex.Mapper.Defaults.TimerType = mapper.TimerTypeHistogram
ex.Mapper.Defaults.ObserverType = mapper.ObserverTypeHistogram
ex.Listen(events)
}()
@ -719,9 +719,9 @@ func TestHistogramUnits(t *testing.T) {
// Then close events channel to stop a listener.
name := "foo"
c := event.Events{
&event.TimerEvent{
TMetricName: name,
TValue: 300,
&event.ObserverEvent{
OMetricName: name,
OValue: .300,
},
}
events <- c
@ -737,9 +737,7 @@ func TestHistogramUnits(t *testing.T) {
if value == nil {
t.Fatal("Histogram value should not be nil")
}
if *value == 300 {
t.Fatalf("Histogram observations not scaled into Seconds")
} else if *value != .300 {
if *value != .300 {
t.Fatalf("Received unexpected value for histogram observation %f != .300", *value)
}
}
@ -869,9 +867,9 @@ mappings:
GValue: 200,
},
// event with ttl = 2s from a mapping
&event.TimerEvent{
TMetricName: "bazqux.main",
TValue: 42000,
&event.ObserverEvent{
OMetricName: "bazqux.main",
OValue: 42,
},
}

16
pkg/line/line.go
View file

@ -42,11 +42,17 @@ func buildEvent(statType, metric string, value float64, relative bool, labels ma
GRelative: relative,
GLabels: labels,
}, nil
case "ms", "h", "d":
return &event.TimerEvent{
TMetricName: metric,
TValue: float64(value),
TLabels: labels,
case "ms":
return &event.ObserverEvent{
OMetricName: metric,
OValue: float64(value) / 1000, // prometheus presumes seconds, statsd millisecond
OLabels: labels,
}, nil
case "h", "d":
return &event.ObserverEvent{
OMetricName: metric,
OValue: float64(value),
OLabels: labels,
}, nil
case "s":
return nil, fmt.Errorf("no support for StatsD sets")

10
pkg/mapper/fsm/README.md
View file

@ -40,7 +40,7 @@ At first, the FSM only contains three states, representing three possible metric
/
(start)---- [counter]
\
'--- [ timer ]
'--- [observer]
Adding a rule `client.*.request.count` with type `counter` will make the FSM to be:
@ -50,7 +50,7 @@ Adding a rule `client.*.request.count` with type `counter` will make the FSM to
/
(start)---- [counter] -- [client] -- [*] -- [request] -- [count] -- {R1}
\
'--- [timer]
'--- [observer]
`{R1}` is short for result 1, which is the match result for `client.*.request.count`.
@ -60,7 +60,7 @@ Adding a rule `client.*.*.size` with type `counter` will make the FSM to be:
/ /
(start)---- [counter] -- [client] -- [*]
\ \__ [*] -- [size] -- {R2}
'--- [timer]
'--- [observer]
### Finding a result state in FSM
@ -76,7 +76,7 @@ FSM, the `^1` to `^7` symbols indicate how FSM will traversal in its tree:
/ / ^5 ^6 ^7
(start)---- [counter] -- [client] -- [*]
^1 \ ^2 ^3 \__ [*] -- [size] -- {R2}
'--- [timer] ^4
'--- [observer] ^4
To map `client.bbb.request.size`, FSM will do a backtracking:
@ -86,7 +86,7 @@ To map `client.bbb.request.size`, FSM will do a backtracking:
/ / ^5 ^6
(start)---- [counter] -- [client] -- [*]
^1 \ ^2 ^3 \__ [*] -- [size] -- {R2}
'--- [timer] ^4
'--- [observer] ^4
^7 ^8 ^9

45
pkg/mapper/mapper.go
View file

@ -27,7 +27,7 @@ import (
)
var (
statsdMetricRE = `[a-zA-Z_](-?[a-zA-Z0-9_])+`
statsdMetricRE = `[a-zA-Z_](-?[a-zA-Z0-9_])*`
templateReplaceRE = `(\$\{?\d+\}?)`
metricLineRE = regexp.MustCompile(`^(\*\.|` + statsdMetricRE + `\.)+(\*|` + statsdMetricRE + `)$`)
@ -35,15 +35,6 @@ var (
labelNameRE = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]+$`)
)
type mapperConfigDefaults struct {
TimerType TimerType `yaml:"timer_type"`
Buckets []float64 `yaml:"buckets"`
Quantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
GlobDisableOrdering bool `yaml:"glob_disable_ordering"`
Ttl time.Duration `yaml:"ttl"`
}
type MetricMapper struct {
Defaults mapperConfigDefaults `yaml:"defaults"`
Mappings []MetricMapping `yaml:"mappings"`
@ -56,26 +47,6 @@ type MetricMapper struct {
MappingsCount prometheus.Gauge
}
type MetricMapping struct {
Match string `yaml:"match"`
Name string `yaml:"name"`
nameFormatter *fsm.TemplateFormatter
regex *regexp.Regexp
Labels prometheus.Labels `yaml:"labels"`
labelKeys []string
labelFormatters []*fsm.TemplateFormatter
TimerType TimerType `yaml:"timer_type"`
LegacyBuckets []float64 `yaml:"buckets"`
LegacyQuantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
HelpText string `yaml:"help"`
Action ActionType `yaml:"action"`
MatchMetricType MetricType `yaml:"match_metric_type"`
Ttl time.Duration `yaml:"ttl"`
SummaryOptions *SummaryOptions `yaml:"summary_options"`
HistogramOptions *HistogramOptions `yaml:"histogram_options"`
}
type SummaryOptions struct {
Quantiles []metricObjective `yaml:"quantiles"`
MaxAge time.Duration `yaml:"max_age"`
@ -119,7 +90,7 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
remainingMappingsCount := len(n.Mappings)
n.FSM = fsm.NewFSM([]string{string(MetricTypeCounter), string(MetricTypeGauge), string(MetricTypeTimer)},
n.FSM = fsm.NewFSM([]string{string(MetricTypeCounter), string(MetricTypeGauge), string(MetricTypeObserver)},
remainingMappingsCount, n.Defaults.GlobDisableOrdering)
for i := range n.Mappings {
@ -181,8 +152,8 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
n.doRegex = true
}
if currentMapping.TimerType == "" {
currentMapping.TimerType = n.Defaults.TimerType
if currentMapping.ObserverType == "" {
currentMapping.ObserverType = n.Defaults.ObserverType
}
if currentMapping.LegacyQuantiles != nil &&
@ -207,9 +178,9 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
return fmt.Errorf("cannot use buckets in both the top level and histogram options at the same time in %s", currentMapping.Match)
}
if currentMapping.TimerType == TimerTypeHistogram {
if currentMapping.ObserverType == ObserverTypeHistogram {
if currentMapping.SummaryOptions != nil {
return fmt.Errorf("cannot use histogram timer and summary options at the same time")
return fmt.Errorf("cannot use histogram observer and summary options at the same time")
}
if currentMapping.HistogramOptions == nil {
currentMapping.HistogramOptions = &HistogramOptions{}
@ -222,9 +193,9 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
}
}
if currentMapping.TimerType == TimerTypeSummary {
if currentMapping.ObserverType == ObserverTypeSummary {
if currentMapping.HistogramOptions != nil {
return fmt.Errorf("cannot use summary timer and histogram options at the same time")
return fmt.Errorf("cannot use summary observer and histogram options at the same time")
}
if currentMapping.SummaryOptions == nil {
currentMapping.SummaryOptions = &SummaryOptions{}

51
pkg/mapper/mapper_defaults.go Normal file
View file

@ -0,0 +1,51 @@
// Copyright 2020 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package mapper
import "time"
type mapperConfigDefaults struct {
ObserverType ObserverType `yaml:"observer_type"`
TimerType ObserverType `yaml:"timer_type,omitempty"` // DEPRECATED - field only present to preserve backwards compatibility in configs. Always empty
Buckets []float64 `yaml:"buckets"`
Quantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
GlobDisableOrdering bool `yaml:"glob_disable_ordering"`
Ttl time.Duration `yaml:"ttl"`
}
// UnmarshalYAML is a custom unmarshal function to allow use of deprecated config keys
// observer_type will override timer_type
func (d *mapperConfigDefaults) UnmarshalYAML(unmarshal func(interface{}) error) error {
type mapperConfigDefaultsAlias mapperConfigDefaults
var tmp mapperConfigDefaultsAlias
if err := unmarshal(&tmp); err != nil {
return err
}
// Copy defaults
d.ObserverType = tmp.ObserverType
d.Buckets = tmp.Buckets
d.Quantiles = tmp.Quantiles
d.MatchType = tmp.MatchType
d.GlobDisableOrdering = tmp.GlobDisableOrdering
d.Ttl = tmp.Ttl
// Use deprecated TimerType if necessary
if tmp.ObserverType == "" {
d.ObserverType = tmp.TimerType
}
return nil
}

117
pkg/mapper/mapper_test.go
View file

@ -441,12 +441,39 @@ mappings:
},
},
},
// Config with good timer type.
// Config with good observer type.
{
config: `---
mappings:
- match: test.*.*
timer_type: summary
observer_type: summary
name: "foo"
labels: {}
quantiles:
- quantile: 0.42
error: 0.04
- quantile: 0.7
error: 0.002
`,
mappings: mappings{
{
statsdMetric: "test.*.*",
name: "foo",
labels: map[string]string{},
quantiles: []metricObjective{
{Quantile: 0.42, Error: 0.04},
{Quantile: 0.7, Error: 0.002},
},
},
},
},
// Config with good observer type and unused timer type
{
config: `---
mappings:
- match: test.*.*
observer_type: summary
timer_type: histogram
name: "foo"
labels: {}
quantiles:
@ -470,6 +497,28 @@ mappings:
{
config: `---
mappings:
- match: test1.*.*
observer_type: summary
name: "foo"
labels: {}
`,
mappings: mappings{
{
statsdMetric: "test1.*.*",
name: "foo",
labels: map[string]string{},
quantiles: []metricObjective{
{Quantile: 0.5, Error: 0.05},
{Quantile: 0.9, Error: 0.01},
{Quantile: 0.99, Error: 0.001},
},
},
},
},
// Config with good deprecated timer type
{
config: `---
mappings:
- match: test1.*.*
timer_type: summary
name: "foo"
@ -488,7 +537,18 @@ mappings:
},
},
},
// Config with bad timer type.
// Config with bad observer type.
{
config: `---
mappings:
- match: test.*.*
observer_type: wrong
name: "foo"
labels: {}
`,
configBad: true,
},
// Config with bad deprecated timer type.
{
config: `---
mappings:
@ -504,7 +564,7 @@ mappings:
config: `---
mappings:
- match: test.*.*
timer_type: summary
observer_type: summary
name: "foo"
labels: {}
summary_options:
@ -531,7 +591,7 @@ mappings:
config: `---
mappings:
- match: test.*.*
timer_type: summary
observer_type: summary
name: "foo"
labels: {}
summary_options:
@ -564,7 +624,7 @@ mappings:
config: `---
mappings:
- match: test.*.*
timer_type: summary
observer_type: summary
name: "foo"
labels: {}
quantiles:
@ -584,6 +644,26 @@ mappings:
- match: test.*.*
match_metric_type: counter
name: "foo"
labels: {}
`,
},
// Config with good metric type observer.
{
config: `---
mappings:
- match: test.*.*
match_metric_type: observer
name: "foo"
labels: {}
`,
},
// Config with good metric type timer.
{
config: `---
mappings:
- match: test.*.*
match_metric_type: timer
name: "foo"
labels: {}
`,
},
@ -638,7 +718,7 @@ mappings:
config: `---
mappings:
- match: foo.*.*
timer_type: summary
observer_type: summary
name: "foo"
labels: {}
`,
@ -662,6 +742,29 @@ mappings:
`,
configBad: true,
},
{
config: `---
mappings:
- match: p.*.*.c.*
match_type: glob
name: issue_256
labels:
one: $1
two: $2
three: $3
`,
mappings: mappings{
{
statsdMetric: "p.one.two.c.three",
name: "issue_256",
labels: map[string]string{
"one": "one",
"two": "two",
"three": "three",
},
},
},
},
// Example from the README.
{
config: `

76
pkg/mapper/mapping.go Normal file
View file

@ -0,0 +1,76 @@
// Copyright 2020 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either xpress or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package mapper
import (
"regexp"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/statsd_exporter/pkg/mapper/fsm"
)
type MetricMapping struct {
Match string `yaml:"match"`
Name string `yaml:"name"`
nameFormatter *fsm.TemplateFormatter
regex *regexp.Regexp
Labels prometheus.Labels `yaml:"labels"`
labelKeys []string
labelFormatters []*fsm.TemplateFormatter
ObserverType ObserverType `yaml:"observer_type"`
TimerType ObserverType `yaml:"timer_type,omitempty"` // DEPRECATED - field only present to preserve backwards compatibility in configs. Always empty
LegacyBuckets []float64 `yaml:"buckets"`
LegacyQuantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
HelpText string `yaml:"help"`
Action ActionType `yaml:"action"`
MatchMetricType MetricType `yaml:"match_metric_type"`
Ttl time.Duration `yaml:"ttl"`
SummaryOptions *SummaryOptions `yaml:"summary_options"`
HistogramOptions *HistogramOptions `yaml:"histogram_options"`
}
// UnmarshalYAML is a custom unmarshal function to allow use of deprecated config keys
// observer_type will override timer_type
func (m *MetricMapping) UnmarshalYAML(unmarshal func(interface{}) error) error {
type MetricMappingAlias MetricMapping
var tmp MetricMappingAlias
if err := unmarshal(&tmp); err != nil {
return err
}
// Copy defaults
m.Match = tmp.Match
m.Name = tmp.Name
m.Labels = tmp.Labels
m.ObserverType = tmp.ObserverType
m.LegacyBuckets = tmp.LegacyBuckets
m.LegacyQuantiles = tmp.LegacyQuantiles
m.MatchType = tmp.MatchType
m.HelpText = tmp.HelpText
m.Action = tmp.Action
m.MatchMetricType = tmp.MatchMetricType
m.Ttl = tmp.Ttl
m.SummaryOptions = tmp.SummaryOptions
m.HistogramOptions = tmp.HistogramOptions
// Use deprecated TimerType if necessary
if tmp.ObserverType == "" {
m.ObserverType = tmp.TimerType
}
return nil
}

11
pkg/mapper/metric_type.go
View file

@ -18,9 +18,10 @@ import "fmt"
type MetricType string
const (
MetricTypeCounter MetricType = "counter"
MetricTypeGauge MetricType = "gauge"
MetricTypeTimer MetricType = "timer"
MetricTypeCounter MetricType = "counter"
MetricTypeGauge MetricType = "gauge"
MetricTypeObserver MetricType = "observer"
MetricTypeTimer MetricType = "timer" // DEPRECATED
)
func (m *MetricType) UnmarshalYAML(unmarshal func(interface{}) error) error {
@ -34,8 +35,10 @@ func (m *MetricType) UnmarshalYAML(unmarshal func(interface{}) error) error {
*m = MetricTypeCounter
case MetricTypeGauge:
*m = MetricTypeGauge
case MetricTypeObserver:
*m = MetricTypeObserver
case MetricTypeTimer:
*m = MetricTypeTimer
*m = MetricTypeObserver
default:
return fmt.Errorf("invalid metric type '%s'", v)
}

22
pkg/mapper/timer.go → pkg/mapper/observer.go
View file

@ -15,27 +15,27 @@ package mapper
import "fmt"
type TimerType string
type ObserverType string
const (
TimerTypeHistogram TimerType = "histogram"
TimerTypeSummary TimerType = "summary"
TimerTypeDefault TimerType = ""
ObserverTypeHistogram ObserverType = "histogram"
ObserverTypeSummary ObserverType = "summary"
ObserverTypeDefault ObserverType = ""
)
func (t *TimerType) UnmarshalYAML(unmarshal func(interface{}) error) error {
func (t *ObserverType) UnmarshalYAML(unmarshal func(interface{}) error) error {
var v string
if err := unmarshal(&v); err != nil {
return err
}
switch TimerType(v) {
case TimerTypeHistogram:
*t = TimerTypeHistogram
case TimerTypeSummary, TimerTypeDefault:
*t = TimerTypeSummary
switch ObserverType(v) {
case ObserverTypeHistogram:
*t = ObserverTypeHistogram
case ObserverTypeSummary, ObserverTypeDefault:
*t = ObserverTypeSummary
default:
return fmt.Errorf("invalid timer type '%s'", v)
return fmt.Errorf("invalid observer type '%s'", v)
}
return nil
}