Merge branch 'master' of github.com:prometheus/statsd_exporter into signalfx

This commit is contained in:
glightfoot 2020-06-19 09:05:10 -04:00
commit 4f7abe5226
15 changed files with 507 additions and 260 deletions

View file

@ -1,6 +1,18 @@
## 0.17.0 / unreleased ## 0.17.0 / Unreleased
* [FEATURE] Offline configuration check ([#312](https://github.com/prometheus/statsd_exporter/pull/312)) * [CHANGE] Support non-timer distributions without unit conversion ([#314](https://github.com/prometheus/statsd_exporter/pull/314))
* [ENHANCEMENT] Offline configuration check ([#312](https://github.com/prometheus/statsd_exporter/pull/312))
* [BUGFIX] Allow matching single-letter metric name components ([#309](https://github.com/prometheus/statsd_exporter/pull/309))
Distribution and histogram events (type `d`, `h`) are now treated as distinct from timer events (type `ms`).
Their values are observed as they are, while timer events are converted from milliseconds to seconds.
To reflect this generalization, the `observer_type` mapping option replaces `timer_type`.
Similary, change `match_metric_type: timer` to `match_metric_type: observer`.
The old name remains available for compatibility.
For users of the mapper library, the `ObserverEvent` replaces `TimerEvent`.
For timer metrics, it is emitted by the mapper already converted to seconds.
## 0.16.0 / 2020-05-29 ## 0.16.0 / 2020-05-29

View file

@ -162,9 +162,7 @@ In general, the different metric types are translated as follows:
StatsD counter -> Prometheus counter StatsD counter -> Prometheus counter
StatsD timer -> Prometheus summary <-- indicates timer quantiles StatsD timer, histogram, distribution -> Prometheus summary or histogram
-> Prometheus counter (suffix `_total`) <-- indicates total time spent
-> Prometheus counter (suffix `_count`) <-- indicates total number of timer events
An example mapping configuration: An example mapping configuration:
@ -246,17 +244,18 @@ mappings:
code: "$1" code: "$1"
``` ```
### StatsD timers ### StatsD timers and distributions
By default, statsd timers are represented as a Prometheus summary with By default, statsd timers and distributions (collectively "observers") are
quantiles. You may optionally configure the [quantiles and acceptable represented as a Prometheus summary with quantiles. You may optionally
error](https://prometheus.io/docs/practices/histograms/#quantiles), as configure the [quantiles and acceptable
well as adjusting how the summary metric is aggregated: error](https://prometheus.io/docs/practices/histograms/#quantiles), as well
as adjusting how the summary metric is aggregated:
```yaml ```yaml
mappings: mappings:
- match: "test.timing.*.*.*" - match: "test.timing.*.*.*"
timer_type: summary observer_type: summary
name: "my_timer" name: "my_timer"
labels: labels:
provider: "$2" provider: "$2"
@ -280,19 +279,17 @@ mappings:
The default quantiles are 0.99, 0.9, and 0.5. The default quantiles are 0.99, 0.9, and 0.5.
The default summary age is 10 minutes, the default number of buckets The default summary age is 10 minutes, the default number of buckets
is 5 and the default buffer size is 500. See also the is 5 and the default buffer size is 500.
[`golang_client` docs](https://godoc.org/github.com/prometheus/client_golang/prometheus#SummaryOpts). See also the [`golang_client` docs](https://godoc.org/github.com/prometheus/client_golang/prometheus#SummaryOpts).
The `max_summary_age` corresponds to `SummaryOptions.MaxAge`, `summary_age_buckets` The `max_summary_age` corresponds to `SummaryOptions.MaxAge`, `summary_age_buckets` to `SummaryOptions.AgeBuckets` and `stream_buffer_size` to `SummaryOptions.BufCap`.
to `SummaryOptions.AgeBuckets` and `stream_buffer_size` to `SummaryOptions.BufCap`.
In the configuration, one may also set the timer type to "histogram". The In the configuration, one may also set the observer type to "histogram". For example,
default is "summary" as in the plain text configuration format. For example, to set the observer type for a single timer metric:
to set the timer type for a single metric:
```yaml ```yaml
mappings: mappings:
- match: "test.timing.*.*.*" - match: "test.timing.*.*.*"
timer_type: histogram observer_type: histogram
histogram_options: histogram_options:
buckets: [ 0.01, 0.025, 0.05, 0.1 ] buckets: [ 0.01, 0.025, 0.05, 0.1 ]
name: "my_timer" name: "my_timer"
@ -302,18 +299,18 @@ mappings:
job: "${1}_server" job: "${1}_server"
``` ```
Note that timers will be accepted with the `ms`, `h`, and `d` statsd types. The first two are timers and histograms and the `d` type is for DataDog's "distribution" type. The distribution type is treated identically to timers and histograms. Timers will be accepted with the `ms` statsd type.
Statsd timer data is transmitted in milliseconds, while Prometheus expects the unit to be seconds.
The exporter converts all timer observations to seconds.
It should be noted that whereas timers in statsd expects the unit of timing data to be in milliseconds, Histogram and distribution events (`h` and `d` metric type) are not subject to unit conversion.
prometheus expects the unit to be seconds. Hence, the exporter converts all timers to seconds
before exporting them.
### DogStatsD Client Behavior ### DogStatsD Client Behavior
#### `timed()` decorator #### `timed()` decorator
If you are using the DogStatsD client's [timed](https://datadogpy.readthedocs.io/en/latest/#datadog.threadstats.base.ThreadStats.timed) decorator, The DogStatsD client's [timed](https://datadogpy.readthedocs.io/en/latest/#datadog.threadstats.base.ThreadStats.timed) decorator emits the metric in seconds but uses the `ms` type.
it emits the metric in seconds, set [use_ms](https://datadogpy.readthedocs.io/en/latest/index.html?highlight=use_ms) to `True` to fix this. Set [`use_ms=True`](https://datadogpy.readthedocs.io/en/latest/index.html?highlight=use_ms) to send the correct units.
### Regular expression matching ### Regular expression matching
@ -339,20 +336,20 @@ Note, that one may also set the histogram buckets. If not set, then the default
[Prometheus client values](https://godoc.org/github.com/prometheus/client_golang/prometheus#pkg-variables) are used: `[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]`. `+Inf` is added [Prometheus client values](https://godoc.org/github.com/prometheus/client_golang/prometheus#pkg-variables) are used: `[.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10]`. `+Inf` is added
automatically. automatically.
`timer_type` is only used when the statsd metric type is a timer. `buckets` is `observer_type` is only used when the statsd metric type is a timer, histogram, or distribution.
only used when the statsd metric type is a timerand the `timer_type` is set to `buckets` is only used when the statsd metric type is one of these, and the `observer_type` is set to `histogram`.
"histogram."
### Global defaults ### Global defaults
One may also set defaults for the timer type, buckets or quantiles, and match_type. These will be used One may also set defaults for the observer type, buckets or quantiles, and match type.
by all mappings that do not define these. These will be used by all mappings that do not define them.
An option that can only be configured in `defaults` is `glob_disable_ordering`, which is `false` if omitted. By setting this to `true`, `glob` match type will not honor the occurance of rules in the mapping rules file and always treat `*` as lower priority than a general string. An option that can only be configured in `defaults` is `glob_disable_ordering`, which is `false` if omitted.
By setting this to `true`, `glob` match type will not honor the occurance of rules in the mapping rules file and always treat `*` as lower priority than a concrete string.
```yaml ```yaml
defaults: defaults:
timer_type: histogram observer_type: histogram
buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5 ] buckets: [.005, .01, .025, .05, .1, .25, .5, 1, 2.5 ]
match_type: glob match_type: glob
glob_disable_ordering: false glob_disable_ordering: false
@ -366,9 +363,9 @@ mappings:
outcome: "$3" outcome: "$3"
job: "${1}_server" job: "${1}_server"
# This will be a summary timer. # This will be a summary timer.
- match: "other.timing.*.*.*" - match: "other.distribution.*.*.*"
timer_type: summary observer_type: summary
name: "other_timer" name: "other_distribution"
labels: labels:
provider: "$2" provider: "$2"
outcome: "$3" outcome: "$3"
@ -437,7 +434,7 @@ mappings:
provider: "$1" provider: "$1"
``` ```
Possible values for `match_metric_type` are `gauge`, `counter` and `timer`. Possible values for `match_metric_type` are `gauge`, `counter` and `observer`.
### Mapping cache size and cache replacement policy ### Mapping cache size and cache replacement policy

View file

@ -84,60 +84,60 @@ func TestHandlePacket(t *testing.T) {
name: "simple timer", name: "simple timer",
in: "foo:200|ms", in: "foo:200|ms",
out: event.Events{ out: event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 200, OValue: 0.2,
TLabels: map[string]string{}, OLabels: map[string]string{},
}, },
}, },
}, { }, {
name: "simple histogram", name: "simple histogram",
in: "foo:200|h", in: "foo:200|h",
out: event.Events{ out: event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 200, OValue: 200,
TLabels: map[string]string{}, OLabels: map[string]string{},
}, },
}, },
}, { }, {
name: "simple distribution", name: "simple distribution",
in: "foo:200|d", in: "foo:200|d",
out: event.Events{ out: event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 200, OValue: 200,
TLabels: map[string]string{}, OLabels: map[string]string{},
}, },
}, },
}, { }, {
name: "distribution with sampling", name: "distribution with sampling",
in: "foo:0.01|d|@0.2|#tag1:bar,#tag2:baz", in: "foo:0.01|d|@0.2|#tag1:bar,#tag2:baz",
out: event.Events{ out: event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
}, },
}, { }, {
@ -300,30 +300,30 @@ func TestHandlePacket(t *testing.T) {
name: "histogram with sampling", name: "histogram with sampling",
in: "foo:0.01|h|@0.2|#tag1:bar,#tag2:baz", in: "foo:0.01|h|@0.2|#tag1:bar,#tag2:baz",
out: event.Events{ out: event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 0.01, OValue: 0.01,
TLabels: map[string]string{"tag1": "bar", "tag2": "baz"}, OLabels: map[string]string{"tag1": "bar", "tag2": "baz"},
}, },
}, },
}, { }, {
@ -356,15 +356,15 @@ func TestHandlePacket(t *testing.T) {
name: "combined multiline metrics", name: "combined multiline metrics",
in: "foo:200|ms:300|ms:5|c|@0.1:6|g\nbar:1|c:5|ms", in: "foo:200|ms:300|ms:5|c|@0.1:6|g\nbar:1|c:5|ms",
out: event.Events{ out: event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 200, OValue: .200,
TLabels: map[string]string{}, OLabels: map[string]string{},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "foo", OMetricName: "foo",
TValue: 300, OValue: .300,
TLabels: map[string]string{}, OLabels: map[string]string{},
}, },
&event.CounterEvent{ &event.CounterEvent{
CMetricName: "foo", CMetricName: "foo",
@ -381,26 +381,26 @@ func TestHandlePacket(t *testing.T) {
CValue: 1, CValue: 1,
CLabels: map[string]string{}, CLabels: map[string]string{},
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "bar", OMetricName: "bar",
TValue: 5, OValue: .005,
TLabels: map[string]string{}, OLabels: map[string]string{},
}, },
}, },
}, { }, {
name: "timings with sampling factor", name: "timings with sampling factor",
in: "foo.timing:0.5|ms|@0.1", in: "foo.timing:0.5|ms|@0.1",
out: event.Events{ out: event.Events{
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
&event.TimerEvent{TMetricName: "foo.timing", TValue: 0.5, TLabels: map[string]string{}}, &event.ObserverEvent{OMetricName: "foo.timing", OValue: 0.0005, OLabels: map[string]string{}},
}, },
}, { }, {
name: "bad line", name: "bad line",
@ -457,6 +457,36 @@ func TestHandlePacket(t *testing.T) {
CLabels: map[string]string{}, CLabels: map[string]string{},
}, },
}, },
}, {
name: "ms timer with conversion to seconds",
in: "foo:200|ms",
out: event.Events{
&event.ObserverEvent{
OMetricName: "foo",
OValue: 0.2,
OLabels: map[string]string{},
},
},
}, {
name: "histogram with no unit conversion",
in: "foo:200|h",
out: event.Events{
&event.ObserverEvent{
OMetricName: "foo",
OValue: 200,
OLabels: map[string]string{},
},
},
}, {
name: "distribution with no unit conversion",
in: "foo:200|d",
out: event.Events{
&event.ObserverEvent{
OMetricName: "foo",
OValue: 200,
OLabels: map[string]string{},
},
},
}, },
} }
@ -589,9 +619,9 @@ mappings:
GValue: 200, GValue: 200,
}, },
// event with ttl = 2s from a mapping // event with ttl = 2s from a mapping
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "bazqux.main", OMetricName: "bazqux.main",
TValue: 42000, OValue: 42,
}, },
} }

View file

@ -87,13 +87,13 @@ func BenchmarkExporterListener(b *testing.B) {
GMetricName: "gauge", GMetricName: "gauge",
GValue: 10, GValue: 10,
}, },
&event.TimerEvent{ // simple timer &event.ObserverEvent{ // simple timer
TMetricName: "timer", OMetricName: "timer",
TValue: 200, OValue: 200,
}, },
&event.TimerEvent{ // simple histogram &event.ObserverEvent{ // simple histogram
TMetricName: "histogram.test", OMetricName: "histogram.test",
TValue: 200, OValue: 200,
}, },
&event.CounterEvent{ // simple_tags &event.CounterEvent{ // simple_tags
CMetricName: "simple_tags", CMetricName: "simple_tags",

View file

@ -52,16 +52,16 @@ func (g *GaugeEvent) Value() float64 { return g.GValue }
func (c *GaugeEvent) Labels() map[string]string { return c.GLabels } func (c *GaugeEvent) Labels() map[string]string { return c.GLabels }
func (c *GaugeEvent) MetricType() mapper.MetricType { return mapper.MetricTypeGauge } func (c *GaugeEvent) MetricType() mapper.MetricType { return mapper.MetricTypeGauge }
type TimerEvent struct { type ObserverEvent struct {
TMetricName string OMetricName string
TValue float64 OValue float64
TLabels map[string]string OLabels map[string]string
} }
func (t *TimerEvent) MetricName() string { return t.TMetricName } func (t *ObserverEvent) MetricName() string { return t.OMetricName }
func (t *TimerEvent) Value() float64 { return t.TValue } func (t *ObserverEvent) Value() float64 { return t.OValue }
func (c *TimerEvent) Labels() map[string]string { return c.TLabels } func (c *ObserverEvent) Labels() map[string]string { return c.OLabels }
func (c *TimerEvent) MetricType() mapper.MetricType { return mapper.MetricTypeTimer } func (c *ObserverEvent) MetricType() mapper.MetricType { return mapper.MetricTypeObserver }
type Events []Event type Events []Event

View file

@ -140,38 +140,38 @@ func (b *Exporter) handleEvent(thisEvent event.Event) {
b.ConflictingEventStats.WithLabelValues("gauge").Inc() b.ConflictingEventStats.WithLabelValues("gauge").Inc()
} }
case *event.TimerEvent: case *event.ObserverEvent:
t := mapper.TimerTypeDefault t := mapper.ObserverTypeDefault
if mapping != nil { if mapping != nil {
t = mapping.TimerType t = mapping.ObserverType
} }
if t == mapper.TimerTypeDefault { if t == mapper.ObserverTypeDefault {
t = b.Mapper.Defaults.TimerType t = b.Mapper.Defaults.ObserverType
} }
switch t { switch t {
case mapper.TimerTypeHistogram: case mapper.ObserverTypeHistogram:
histogram, err := b.Registry.GetHistogram(metricName, prometheusLabels, help, mapping, b.MetricsCount) histogram, err := b.Registry.GetHistogram(metricName, prometheusLabels, help, mapping, b.MetricsCount)
if err == nil { if err == nil {
histogram.Observe(thisEvent.Value() / 1000) // prometheus presumes seconds, statsd millisecond histogram.Observe(thisEvent.Value())
b.EventStats.WithLabelValues("timer").Inc() b.EventStats.WithLabelValues("observer").Inc()
} else { } else {
level.Debug(b.Logger).Log("msg", regErrF, "metric", metricName, "error", err) level.Debug(b.Logger).Log("msg", regErrF, "metric", metricName, "error", err)
b.ConflictingEventStats.WithLabelValues("timer").Inc() b.ConflictingEventStats.WithLabelValues("observer").Inc()
} }
case mapper.TimerTypeDefault, mapper.TimerTypeSummary: case mapper.ObserverTypeDefault, mapper.ObserverTypeSummary:
summary, err := b.Registry.GetSummary(metricName, prometheusLabels, help, mapping, b.MetricsCount) summary, err := b.Registry.GetSummary(metricName, prometheusLabels, help, mapping, b.MetricsCount)
if err == nil { if err == nil {
summary.Observe(thisEvent.Value() / 1000) // prometheus presumes seconds, statsd millisecond summary.Observe(thisEvent.Value())
b.EventStats.WithLabelValues("timer").Inc() b.EventStats.WithLabelValues("observer").Inc()
} else { } else {
level.Debug(b.Logger).Log("msg", regErrF, "metric", metricName, "error", err) level.Debug(b.Logger).Log("msg", regErrF, "metric", metricName, "error", err)
b.ConflictingEventStats.WithLabelValues("timer").Inc() b.ConflictingEventStats.WithLabelValues("observer").Inc()
} }
default: default:
level.Error(b.Logger).Log("msg", "unknown timer type", "type", t) level.Error(b.Logger).Log("msg", "unknown observer type", "type", t)
os.Exit(1) os.Exit(1)
} }

View file

@ -228,25 +228,25 @@ func TestInconsistentLabelSets(t *testing.T) {
GValue: 1, GValue: 1,
GLabels: secondLabelSet, GLabels: secondLabelSet,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "histogram.test", OMetricName: "histogram.test",
TValue: 1, OValue: 1,
TLabels: firstLabelSet, OLabels: firstLabelSet,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "histogram.test", OMetricName: "histogram.test",
TValue: 1, OValue: 1,
TLabels: secondLabelSet, OLabels: secondLabelSet,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "summary_test", OMetricName: "summary_test",
TValue: 1, OValue: 1,
TLabels: firstLabelSet, OLabels: firstLabelSet,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "summary_test", OMetricName: "summary_test",
TValue: 1, OValue: 1,
TLabels: secondLabelSet, OLabels: secondLabelSet,
}, },
} }
events <- c events <- c
@ -427,9 +427,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test1", CMetricName: "histogram_test1",
CValue: 1, CValue: 1,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "histogram.test1", OMetricName: "histogram.test1",
TValue: 2, OValue: 2,
}, },
}, },
}, },
@ -441,9 +441,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test1_sum", CMetricName: "histogram_test1_sum",
CValue: 1, CValue: 1,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "histogram.test1", OMetricName: "histogram.test1",
TValue: 2, OValue: 2,
}, },
}, },
}, },
@ -455,9 +455,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test2_count", CMetricName: "histogram_test2_count",
CValue: 1, CValue: 1,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "histogram.test2", OMetricName: "histogram.test2",
TValue: 2, OValue: 2,
}, },
}, },
}, },
@ -469,9 +469,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "histogram_test3_bucket", CMetricName: "histogram_test3_bucket",
CValue: 1, CValue: 1,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "histogram.test3", OMetricName: "histogram.test3",
TValue: 2, OValue: 2,
}, },
}, },
}, },
@ -483,9 +483,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "cvsq_test", CMetricName: "cvsq_test",
CValue: 1, CValue: 1,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "cvsq_test", OMetricName: "cvsq_test",
TValue: 2, OValue: 2,
}, },
}, },
}, },
@ -497,9 +497,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "cvsc_count", CMetricName: "cvsc_count",
CValue: 1, CValue: 1,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "cvsc", OMetricName: "cvsc",
TValue: 2, OValue: 2,
}, },
}, },
}, },
@ -511,9 +511,9 @@ func TestConflictingMetrics(t *testing.T) {
CMetricName: "cvss_sum", CMetricName: "cvss_sum",
CValue: 1, CValue: 1,
}, },
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "cvss", OMetricName: "cvss",
TValue: 2, OValue: 2,
}, },
}, },
}, },
@ -672,9 +672,9 @@ func TestSummaryWithQuantilesEmptyMapping(t *testing.T) {
name := "default_foo" name := "default_foo"
c := event.Events{ c := event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: name, OMetricName: name,
TValue: 300, OValue: 300,
}, },
} }
events <- c events <- c
@ -711,7 +711,7 @@ func TestHistogramUnits(t *testing.T) {
testMapper := mapper.MetricMapper{} testMapper := mapper.MetricMapper{}
testMapper.InitCache(0) testMapper.InitCache(0)
ex := NewExporter(&testMapper, log.NewNopLogger(), eventsActions, eventsUnmapped, errorEventStats, eventStats, conflictingEventStats, metricsCount) ex := NewExporter(&testMapper, log.NewNopLogger(), eventsActions, eventsUnmapped, errorEventStats, eventStats, conflictingEventStats, metricsCount)
ex.Mapper.Defaults.TimerType = mapper.TimerTypeHistogram ex.Mapper.Defaults.ObserverType = mapper.ObserverTypeHistogram
ex.Listen(events) ex.Listen(events)
}() }()
@ -719,9 +719,9 @@ func TestHistogramUnits(t *testing.T) {
// Then close events channel to stop a listener. // Then close events channel to stop a listener.
name := "foo" name := "foo"
c := event.Events{ c := event.Events{
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: name, OMetricName: name,
TValue: 300, OValue: .300,
}, },
} }
events <- c events <- c
@ -737,9 +737,7 @@ func TestHistogramUnits(t *testing.T) {
if value == nil { if value == nil {
t.Fatal("Histogram value should not be nil") t.Fatal("Histogram value should not be nil")
} }
if *value == 300 { if *value != .300 {
t.Fatalf("Histogram observations not scaled into Seconds")
} else if *value != .300 {
t.Fatalf("Received unexpected value for histogram observation %f != .300", *value) t.Fatalf("Received unexpected value for histogram observation %f != .300", *value)
} }
} }
@ -869,9 +867,9 @@ mappings:
GValue: 200, GValue: 200,
}, },
// event with ttl = 2s from a mapping // event with ttl = 2s from a mapping
&event.TimerEvent{ &event.ObserverEvent{
TMetricName: "bazqux.main", OMetricName: "bazqux.main",
TValue: 42000, OValue: 42,
}, },
} }

View file

@ -42,11 +42,17 @@ func buildEvent(statType, metric string, value float64, relative bool, labels ma
GRelative: relative, GRelative: relative,
GLabels: labels, GLabels: labels,
}, nil }, nil
case "ms", "h", "d": case "ms":
return &event.TimerEvent{ return &event.ObserverEvent{
TMetricName: metric, OMetricName: metric,
TValue: float64(value), OValue: float64(value) / 1000, // prometheus presumes seconds, statsd millisecond
TLabels: labels, OLabels: labels,
}, nil
case "h", "d":
return &event.ObserverEvent{
OMetricName: metric,
OValue: float64(value),
OLabels: labels,
}, nil }, nil
case "s": case "s":
return nil, fmt.Errorf("no support for StatsD sets") return nil, fmt.Errorf("no support for StatsD sets")

View file

@ -40,7 +40,7 @@ At first, the FSM only contains three states, representing three possible metric
/ /
(start)---- [counter] (start)---- [counter]
\ \
'--- [ timer ] '--- [observer]
Adding a rule `client.*.request.count` with type `counter` will make the FSM to be: Adding a rule `client.*.request.count` with type `counter` will make the FSM to be:
@ -50,7 +50,7 @@ Adding a rule `client.*.request.count` with type `counter` will make the FSM to
/ /
(start)---- [counter] -- [client] -- [*] -- [request] -- [count] -- {R1} (start)---- [counter] -- [client] -- [*] -- [request] -- [count] -- {R1}
\ \
'--- [timer] '--- [observer]
`{R1}` is short for result 1, which is the match result for `client.*.request.count`. `{R1}` is short for result 1, which is the match result for `client.*.request.count`.
@ -60,7 +60,7 @@ Adding a rule `client.*.*.size` with type `counter` will make the FSM to be:
/ / / /
(start)---- [counter] -- [client] -- [*] (start)---- [counter] -- [client] -- [*]
\ \__ [*] -- [size] -- {R2} \ \__ [*] -- [size] -- {R2}
'--- [timer] '--- [observer]
### Finding a result state in FSM ### Finding a result state in FSM
@ -76,7 +76,7 @@ FSM, the `^1` to `^7` symbols indicate how FSM will traversal in its tree:
/ / ^5 ^6 ^7 / / ^5 ^6 ^7
(start)---- [counter] -- [client] -- [*] (start)---- [counter] -- [client] -- [*]
^1 \ ^2 ^3 \__ [*] -- [size] -- {R2} ^1 \ ^2 ^3 \__ [*] -- [size] -- {R2}
'--- [timer] ^4 '--- [observer] ^4
To map `client.bbb.request.size`, FSM will do a backtracking: To map `client.bbb.request.size`, FSM will do a backtracking:
@ -86,7 +86,7 @@ To map `client.bbb.request.size`, FSM will do a backtracking:
/ / ^5 ^6 / / ^5 ^6
(start)---- [counter] -- [client] -- [*] (start)---- [counter] -- [client] -- [*]
^1 \ ^2 ^3 \__ [*] -- [size] -- {R2} ^1 \ ^2 ^3 \__ [*] -- [size] -- {R2}
'--- [timer] ^4 '--- [observer] ^4
^7 ^8 ^9 ^7 ^8 ^9

View file

@ -27,7 +27,7 @@ import (
) )
var ( var (
statsdMetricRE = `[a-zA-Z_](-?[a-zA-Z0-9_])+` statsdMetricRE = `[a-zA-Z_](-?[a-zA-Z0-9_])*`
templateReplaceRE = `(\$\{?\d+\}?)` templateReplaceRE = `(\$\{?\d+\}?)`
metricLineRE = regexp.MustCompile(`^(\*\.|` + statsdMetricRE + `\.)+(\*|` + statsdMetricRE + `)$`) metricLineRE = regexp.MustCompile(`^(\*\.|` + statsdMetricRE + `\.)+(\*|` + statsdMetricRE + `)$`)
@ -35,15 +35,6 @@ var (
labelNameRE = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]+$`) labelNameRE = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]+$`)
) )
type mapperConfigDefaults struct {
TimerType TimerType `yaml:"timer_type"`
Buckets []float64 `yaml:"buckets"`
Quantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
GlobDisableOrdering bool `yaml:"glob_disable_ordering"`
Ttl time.Duration `yaml:"ttl"`
}
type MetricMapper struct { type MetricMapper struct {
Defaults mapperConfigDefaults `yaml:"defaults"` Defaults mapperConfigDefaults `yaml:"defaults"`
Mappings []MetricMapping `yaml:"mappings"` Mappings []MetricMapping `yaml:"mappings"`
@ -56,26 +47,6 @@ type MetricMapper struct {
MappingsCount prometheus.Gauge MappingsCount prometheus.Gauge
} }
type MetricMapping struct {
Match string `yaml:"match"`
Name string `yaml:"name"`
nameFormatter *fsm.TemplateFormatter
regex *regexp.Regexp
Labels prometheus.Labels `yaml:"labels"`
labelKeys []string
labelFormatters []*fsm.TemplateFormatter
TimerType TimerType `yaml:"timer_type"`
LegacyBuckets []float64 `yaml:"buckets"`
LegacyQuantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
HelpText string `yaml:"help"`
Action ActionType `yaml:"action"`
MatchMetricType MetricType `yaml:"match_metric_type"`
Ttl time.Duration `yaml:"ttl"`
SummaryOptions *SummaryOptions `yaml:"summary_options"`
HistogramOptions *HistogramOptions `yaml:"histogram_options"`
}
type SummaryOptions struct { type SummaryOptions struct {
Quantiles []metricObjective `yaml:"quantiles"` Quantiles []metricObjective `yaml:"quantiles"`
MaxAge time.Duration `yaml:"max_age"` MaxAge time.Duration `yaml:"max_age"`
@ -119,7 +90,7 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
remainingMappingsCount := len(n.Mappings) remainingMappingsCount := len(n.Mappings)
n.FSM = fsm.NewFSM([]string{string(MetricTypeCounter), string(MetricTypeGauge), string(MetricTypeTimer)}, n.FSM = fsm.NewFSM([]string{string(MetricTypeCounter), string(MetricTypeGauge), string(MetricTypeObserver)},
remainingMappingsCount, n.Defaults.GlobDisableOrdering) remainingMappingsCount, n.Defaults.GlobDisableOrdering)
for i := range n.Mappings { for i := range n.Mappings {
@ -181,8 +152,8 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
n.doRegex = true n.doRegex = true
} }
if currentMapping.TimerType == "" { if currentMapping.ObserverType == "" {
currentMapping.TimerType = n.Defaults.TimerType currentMapping.ObserverType = n.Defaults.ObserverType
} }
if currentMapping.LegacyQuantiles != nil && if currentMapping.LegacyQuantiles != nil &&
@ -207,9 +178,9 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
return fmt.Errorf("cannot use buckets in both the top level and histogram options at the same time in %s", currentMapping.Match) return fmt.Errorf("cannot use buckets in both the top level and histogram options at the same time in %s", currentMapping.Match)
} }
if currentMapping.TimerType == TimerTypeHistogram { if currentMapping.ObserverType == ObserverTypeHistogram {
if currentMapping.SummaryOptions != nil { if currentMapping.SummaryOptions != nil {
return fmt.Errorf("cannot use histogram timer and summary options at the same time") return fmt.Errorf("cannot use histogram observer and summary options at the same time")
} }
if currentMapping.HistogramOptions == nil { if currentMapping.HistogramOptions == nil {
currentMapping.HistogramOptions = &HistogramOptions{} currentMapping.HistogramOptions = &HistogramOptions{}
@ -222,9 +193,9 @@ func (m *MetricMapper) InitFromYAMLString(fileContents string, cacheSize int, op
} }
} }
if currentMapping.TimerType == TimerTypeSummary { if currentMapping.ObserverType == ObserverTypeSummary {
if currentMapping.HistogramOptions != nil { if currentMapping.HistogramOptions != nil {
return fmt.Errorf("cannot use summary timer and histogram options at the same time") return fmt.Errorf("cannot use summary observer and histogram options at the same time")
} }
if currentMapping.SummaryOptions == nil { if currentMapping.SummaryOptions == nil {
currentMapping.SummaryOptions = &SummaryOptions{} currentMapping.SummaryOptions = &SummaryOptions{}

View file

@ -0,0 +1,51 @@
// Copyright 2020 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package mapper
import "time"
type mapperConfigDefaults struct {
ObserverType ObserverType `yaml:"observer_type"`
TimerType ObserverType `yaml:"timer_type,omitempty"` // DEPRECATED - field only present to preserve backwards compatibility in configs. Always empty
Buckets []float64 `yaml:"buckets"`
Quantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
GlobDisableOrdering bool `yaml:"glob_disable_ordering"`
Ttl time.Duration `yaml:"ttl"`
}
// UnmarshalYAML is a custom unmarshal function to allow use of deprecated config keys
// observer_type will override timer_type
func (d *mapperConfigDefaults) UnmarshalYAML(unmarshal func(interface{}) error) error {
type mapperConfigDefaultsAlias mapperConfigDefaults
var tmp mapperConfigDefaultsAlias
if err := unmarshal(&tmp); err != nil {
return err
}
// Copy defaults
d.ObserverType = tmp.ObserverType
d.Buckets = tmp.Buckets
d.Quantiles = tmp.Quantiles
d.MatchType = tmp.MatchType
d.GlobDisableOrdering = tmp.GlobDisableOrdering
d.Ttl = tmp.Ttl
// Use deprecated TimerType if necessary
if tmp.ObserverType == "" {
d.ObserverType = tmp.TimerType
}
return nil
}

View file

@ -441,12 +441,39 @@ mappings:
}, },
}, },
}, },
// Config with good timer type. // Config with good observer type.
{ {
config: `--- config: `---
mappings: mappings:
- match: test.*.* - match: test.*.*
timer_type: summary observer_type: summary
name: "foo"
labels: {}
quantiles:
- quantile: 0.42
error: 0.04
- quantile: 0.7
error: 0.002
`,
mappings: mappings{
{
statsdMetric: "test.*.*",
name: "foo",
labels: map[string]string{},
quantiles: []metricObjective{
{Quantile: 0.42, Error: 0.04},
{Quantile: 0.7, Error: 0.002},
},
},
},
},
// Config with good observer type and unused timer type
{
config: `---
mappings:
- match: test.*.*
observer_type: summary
timer_type: histogram
name: "foo" name: "foo"
labels: {} labels: {}
quantiles: quantiles:
@ -470,6 +497,28 @@ mappings:
{ {
config: `--- config: `---
mappings: mappings:
- match: test1.*.*
observer_type: summary
name: "foo"
labels: {}
`,
mappings: mappings{
{
statsdMetric: "test1.*.*",
name: "foo",
labels: map[string]string{},
quantiles: []metricObjective{
{Quantile: 0.5, Error: 0.05},
{Quantile: 0.9, Error: 0.01},
{Quantile: 0.99, Error: 0.001},
},
},
},
},
// Config with good deprecated timer type
{
config: `---
mappings:
- match: test1.*.* - match: test1.*.*
timer_type: summary timer_type: summary
name: "foo" name: "foo"
@ -488,7 +537,18 @@ mappings:
}, },
}, },
}, },
// Config with bad timer type. // Config with bad observer type.
{
config: `---
mappings:
- match: test.*.*
observer_type: wrong
name: "foo"
labels: {}
`,
configBad: true,
},
// Config with bad deprecated timer type.
{ {
config: `--- config: `---
mappings: mappings:
@ -504,7 +564,7 @@ mappings:
config: `--- config: `---
mappings: mappings:
- match: test.*.* - match: test.*.*
timer_type: summary observer_type: summary
name: "foo" name: "foo"
labels: {} labels: {}
summary_options: summary_options:
@ -531,7 +591,7 @@ mappings:
config: `--- config: `---
mappings: mappings:
- match: test.*.* - match: test.*.*
timer_type: summary observer_type: summary
name: "foo" name: "foo"
labels: {} labels: {}
summary_options: summary_options:
@ -564,7 +624,7 @@ mappings:
config: `--- config: `---
mappings: mappings:
- match: test.*.* - match: test.*.*
timer_type: summary observer_type: summary
name: "foo" name: "foo"
labels: {} labels: {}
quantiles: quantiles:
@ -584,6 +644,26 @@ mappings:
- match: test.*.* - match: test.*.*
match_metric_type: counter match_metric_type: counter
name: "foo" name: "foo"
labels: {}
`,
},
// Config with good metric type observer.
{
config: `---
mappings:
- match: test.*.*
match_metric_type: observer
name: "foo"
labels: {}
`,
},
// Config with good metric type timer.
{
config: `---
mappings:
- match: test.*.*
match_metric_type: timer
name: "foo"
labels: {} labels: {}
`, `,
}, },
@ -638,7 +718,7 @@ mappings:
config: `--- config: `---
mappings: mappings:
- match: foo.*.* - match: foo.*.*
timer_type: summary observer_type: summary
name: "foo" name: "foo"
labels: {} labels: {}
`, `,
@ -662,6 +742,29 @@ mappings:
`, `,
configBad: true, configBad: true,
}, },
{
config: `---
mappings:
- match: p.*.*.c.*
match_type: glob
name: issue_256
labels:
one: $1
two: $2
three: $3
`,
mappings: mappings{
{
statsdMetric: "p.one.two.c.three",
name: "issue_256",
labels: map[string]string{
"one": "one",
"two": "two",
"three": "three",
},
},
},
},
// Example from the README. // Example from the README.
{ {
config: ` config: `

76
pkg/mapper/mapping.go Normal file
View file

@ -0,0 +1,76 @@
// Copyright 2020 The Prometheus Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either xpress or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package mapper
import (
"regexp"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/statsd_exporter/pkg/mapper/fsm"
)
type MetricMapping struct {
Match string `yaml:"match"`
Name string `yaml:"name"`
nameFormatter *fsm.TemplateFormatter
regex *regexp.Regexp
Labels prometheus.Labels `yaml:"labels"`
labelKeys []string
labelFormatters []*fsm.TemplateFormatter
ObserverType ObserverType `yaml:"observer_type"`
TimerType ObserverType `yaml:"timer_type,omitempty"` // DEPRECATED - field only present to preserve backwards compatibility in configs. Always empty
LegacyBuckets []float64 `yaml:"buckets"`
LegacyQuantiles []metricObjective `yaml:"quantiles"`
MatchType MatchType `yaml:"match_type"`
HelpText string `yaml:"help"`
Action ActionType `yaml:"action"`
MatchMetricType MetricType `yaml:"match_metric_type"`
Ttl time.Duration `yaml:"ttl"`
SummaryOptions *SummaryOptions `yaml:"summary_options"`
HistogramOptions *HistogramOptions `yaml:"histogram_options"`
}
// UnmarshalYAML is a custom unmarshal function to allow use of deprecated config keys
// observer_type will override timer_type
func (m *MetricMapping) UnmarshalYAML(unmarshal func(interface{}) error) error {
type MetricMappingAlias MetricMapping
var tmp MetricMappingAlias
if err := unmarshal(&tmp); err != nil {
return err
}
// Copy defaults
m.Match = tmp.Match
m.Name = tmp.Name
m.Labels = tmp.Labels
m.ObserverType = tmp.ObserverType
m.LegacyBuckets = tmp.LegacyBuckets
m.LegacyQuantiles = tmp.LegacyQuantiles
m.MatchType = tmp.MatchType
m.HelpText = tmp.HelpText
m.Action = tmp.Action
m.MatchMetricType = tmp.MatchMetricType
m.Ttl = tmp.Ttl
m.SummaryOptions = tmp.SummaryOptions
m.HistogramOptions = tmp.HistogramOptions
// Use deprecated TimerType if necessary
if tmp.ObserverType == "" {
m.ObserverType = tmp.TimerType
}
return nil
}

View file

@ -20,7 +20,8 @@ type MetricType string
const ( const (
MetricTypeCounter MetricType = "counter" MetricTypeCounter MetricType = "counter"
MetricTypeGauge MetricType = "gauge" MetricTypeGauge MetricType = "gauge"
MetricTypeTimer MetricType = "timer" MetricTypeObserver MetricType = "observer"
MetricTypeTimer MetricType = "timer" // DEPRECATED
) )
func (m *MetricType) UnmarshalYAML(unmarshal func(interface{}) error) error { func (m *MetricType) UnmarshalYAML(unmarshal func(interface{}) error) error {
@ -34,8 +35,10 @@ func (m *MetricType) UnmarshalYAML(unmarshal func(interface{}) error) error {
*m = MetricTypeCounter *m = MetricTypeCounter
case MetricTypeGauge: case MetricTypeGauge:
*m = MetricTypeGauge *m = MetricTypeGauge
case MetricTypeObserver:
*m = MetricTypeObserver
case MetricTypeTimer: case MetricTypeTimer:
*m = MetricTypeTimer *m = MetricTypeObserver
default: default:
return fmt.Errorf("invalid metric type '%s'", v) return fmt.Errorf("invalid metric type '%s'", v)
} }

View file

@ -15,27 +15,27 @@ package mapper
import "fmt" import "fmt"
type TimerType string type ObserverType string
const ( const (
TimerTypeHistogram TimerType = "histogram" ObserverTypeHistogram ObserverType = "histogram"
TimerTypeSummary TimerType = "summary" ObserverTypeSummary ObserverType = "summary"
TimerTypeDefault TimerType = "" ObserverTypeDefault ObserverType = ""
) )
func (t *TimerType) UnmarshalYAML(unmarshal func(interface{}) error) error { func (t *ObserverType) UnmarshalYAML(unmarshal func(interface{}) error) error {
var v string var v string
if err := unmarshal(&v); err != nil { if err := unmarshal(&v); err != nil {
return err return err
} }
switch TimerType(v) { switch ObserverType(v) {
case TimerTypeHistogram: case ObserverTypeHistogram:
*t = TimerTypeHistogram *t = ObserverTypeHistogram
case TimerTypeSummary, TimerTypeDefault: case ObserverTypeSummary, ObserverTypeDefault:
*t = TimerTypeSummary *t = ObserverTypeSummary
default: default:
return fmt.Errorf("invalid timer type '%s'", v) return fmt.Errorf("invalid observer type '%s'", v)
} }
return nil return nil
} }