-
Notifications
You must be signed in to change notification settings - Fork 542
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add influx push endpoint to mimir (#10153)
* olegs base commits from #1971 Signed-off-by: alexgreenbank <[email protected]> * move top level influx files Signed-off-by: alexgreenbank <[email protected]> * latest wip Signed-off-by: alexgreenbank <[email protected]> * still WIP but better, still need to move to parserFunc() style Signed-off-by: alexgreenbank <[email protected]> * it builds! Signed-off-by: alexgreenbank <[email protected]> * tweaks and add span logging Signed-off-by: alexgreenbank <[email protected]> * more todo Signed-off-by: alexgreenbank <[email protected]> * further tweaks Signed-off-by: alexgreenbank <[email protected]> * some fixes to tests Signed-off-by: alexgreenbank <[email protected]> * rejigged error handling, tests passing Signed-off-by: alexgreenbank <[email protected]> * add vendored influxdb code Signed-off-by: alexgreenbank <[email protected]> * lint Signed-off-by: alexgreenbank <[email protected]> * go mod sum vendor/modules.txt Signed-off-by: alexgreenbank <[email protected]> * add a metric, add tenant info, other tweaks Signed-off-by: alexgreenbank <[email protected]> * various rework, still WIP Signed-off-by: alexgreenbank <[email protected]> * propagate bytesRead down to caller and log and histogram Signed-off-by: alexgreenbank <[email protected]> * remove comment now dealt with Signed-off-by: alexgreenbank <[email protected]> * add defaults in error handling Signed-off-by: alexgreenbank <[email protected]> * Add note to docs about experimental Influx flag Signed-off-by: alexgreenbank <[email protected]> * Note influx endpoint as experimental too Signed-off-by: alexgreenbank <[email protected]> * test for specific errors received Signed-off-by: alexgreenbank <[email protected]> * bolster parser tests Signed-off-by: alexgreenbank <[email protected]> * Use literal chars rather than ascii codes Signed-off-by: alexgreenbank <[email protected]> * remove unnecessary cast to int() Signed-off-by: alexgreenbank <[email protected]> * use mimirpb.PreallocTimeseries in influx parser Signed-off-by: alexgreenbank <[email protected]> * remove unnecessary tryUnwrap() Signed-off-by: alexgreenbank <[email protected]> * Work on byteslice rather than chars Signed-off-by: alexgreenbank <[email protected]> * yoloString for label value as push code does not keep references to strings from LabelAdapter Signed-off-by: alexgreenbank <[email protected]> * update go.sum Signed-off-by: alexgreenbank <[email protected]> * gah go.sum Signed-off-by: alexgreenbank <[email protected]> * oops, missed removal of paramter to InfluxHandler() Signed-off-by: alexgreenbank <[email protected]> * wrong metrics incremented Signed-off-by: alexgreenbank <[email protected]> * lint Signed-off-by: alexgreenbank <[email protected]> * lint Signed-off-by: alexgreenbank <[email protected]> * go mod tidy && go mod vendor Signed-off-by: alexgreenbank <[email protected]> * go.sum conflict Signed-off-by: alexgreenbank <[email protected]> * make doc Signed-off-by: alexgreenbank <[email protected]> * make influx config hidden/experimental Signed-off-by: alexgreenbank <[email protected]> * fix byteslice handling in replaceInvalidChars() Signed-off-by: alexgreenbank <[email protected]> * remove unnecessary TODOs Signed-off-by: alexgreenbank <[email protected]> * influx: happy path e2e test Signed-off-by: alexgreenbank <[email protected]> * lint Signed-off-by: alexgreenbank <[email protected]> * consolidate logging Signed-off-by: alexgreenbank <[email protected]> * CHANGELOG Signed-off-by: alexgreenbank <[email protected]> * about-versioning.md Signed-off-by: alexgreenbank <[email protected]> * Update pkg/distributor/influxpush/parser.go Co-authored-by: Oleg Zaytsev <[email protected]> * Update pkg/distributor/influxpush/parser.go Co-authored-by: Oleg Zaytsev <[email protected]> * Update pkg/distributor/influxpush/parser.go Co-authored-by: Oleg Zaytsev <[email protected]> * fix parsing string replacing code Signed-off-by: alexgreenbank <[email protected]> * fix nits Signed-off-by: alexgreenbank <[email protected]> --------- Signed-off-by: alexgreenbank <[email protected]> Co-authored-by: Oleg Zaytsev <[email protected]>
- Loading branch information
1 parent
bd6e14b
commit 1836ff1
Showing
27 changed files
with
4,343 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| // SPDX-License-Identifier: AGPL-3.0-only | ||
| //go:build requires_docker | ||
|
|
||
| package integration | ||
|
|
||
| import ( | ||
| "math" | ||
| "testing" | ||
| "time" | ||
|
|
||
| "github.com/grafana/e2e" | ||
| e2edb "github.com/grafana/e2e/db" | ||
| "github.com/prometheus/common/model" | ||
| "github.com/prometheus/prometheus/model/labels" | ||
| "github.com/prometheus/prometheus/prompb" | ||
| v1 "github.com/prometheus/prometheus/web/api/v1" | ||
| "github.com/stretchr/testify/assert" | ||
| "github.com/stretchr/testify/require" | ||
|
|
||
| "github.com/grafana/mimir/integration/e2emimir" | ||
| ) | ||
|
|
||
| func TestInfluxIngestion(t *testing.T) { | ||
| t.Helper() | ||
|
|
||
| s, err := e2e.NewScenario(networkName) | ||
| require.NoError(t, err) | ||
| defer s.Close() | ||
|
|
||
| // Start dependencies. | ||
| minio := e2edb.NewMinio(9000, blocksBucketName) | ||
| require.NoError(t, s.StartAndWaitReady(minio)) | ||
|
|
||
| // Start Mimir components. | ||
| require.NoError(t, copyFileToSharedDir(s, "docs/configurations/single-process-config-blocks.yaml", mimirConfigFile)) | ||
|
|
||
| // Start Mimir in single binary mode, reading the config from file and overwriting | ||
| // the backend config to make it work with Minio. | ||
| flags := mergeFlags( | ||
| DefaultSingleBinaryFlags(), | ||
| BlocksStorageFlags(), | ||
| BlocksStorageS3Flags(), | ||
| map[string]string{ | ||
| "-distributor.influx-endpoint-enabled": "true", | ||
| }, | ||
| ) | ||
|
|
||
| mimir := e2emimir.NewSingleBinary("mimir-1", flags, e2emimir.WithConfigFile(mimirConfigFile), e2emimir.WithPorts(9009, 9095)) | ||
| require.NoError(t, s.StartAndWaitReady(mimir)) | ||
|
|
||
| c, err := e2emimir.NewClient(mimir.HTTPEndpoint(), mimir.HTTPEndpoint(), "", "", "user-1") | ||
| require.NoError(t, err) | ||
|
|
||
| // Push some series to Mimir. | ||
| now := time.Now() | ||
|
|
||
| series, expectedVector, expectedMatrix := generateFloatSeries("series_f1", now, prompb.Label{Name: "foo", Value: "bar"}) | ||
| // Fix up the expectation as Influx values seem to be rounded to millionths | ||
| for _, s := range expectedVector { | ||
| s.Metric[model.LabelName("__mimir_source__")] = model.LabelValue("influx") | ||
| s.Value = model.SampleValue(math.Round(float64(s.Value)*1000000) / 1000000.0) | ||
| } | ||
| // Fix up the expectation as Influx values seem to be rounded to millionths | ||
| for im, s := range expectedMatrix { | ||
| for iv, v := range s.Values { | ||
| expectedMatrix[im].Values[iv].Value = model.SampleValue(math.Round(float64(v.Value)*1000000) / 1000000.0) | ||
| } | ||
| } | ||
|
|
||
| res, err := c.PushInflux(series) | ||
| require.NoError(t, err) | ||
| require.Equal(t, 204, res.StatusCode) | ||
|
|
||
| // Check metric to track Influx requests | ||
| require.NoError(t, mimir.WaitSumMetricsWithOptions(e2e.Equals(1), []string{"cortex_distributor_influx_requests_total"}, e2e.WithLabelMatchers( | ||
| labels.MustNewMatcher(labels.MatchEqual, "user", "user-1")))) | ||
|
|
||
| // Query the series. | ||
| result, err := c.Query("series_f1", now) | ||
| require.NoError(t, err) | ||
| require.Equal(t, model.ValVector, result.Type()) | ||
| assert.Equal(t, expectedVector, result.(model.Vector)) | ||
|
|
||
| labelValues, err := c.LabelValues("foo", v1.MinTime, v1.MaxTime, nil) | ||
| require.NoError(t, err) | ||
| require.Equal(t, model.LabelValues{"bar"}, labelValues) | ||
|
|
||
| labelNames, err := c.LabelNames(v1.MinTime, v1.MaxTime, nil) | ||
| require.NoError(t, err) | ||
| require.Equal(t, []string{"__mimir_source__", "__name__", "foo"}, labelNames) | ||
|
|
||
| rangeResult, err := c.QueryRange("series_f1", now.Add(-15*time.Minute), now, 15*time.Second) | ||
| require.NoError(t, err) | ||
| require.Equal(t, model.ValMatrix, rangeResult.Type()) | ||
| require.Equal(t, expectedMatrix, rangeResult.(model.Matrix)) | ||
|
|
||
| // No metadata to query, but we do the query anyway. | ||
| _, err = c.GetPrometheusMetadata() | ||
| require.NoError(t, err) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.