feat: Detect fields based on per-tenant configuration and put them into structured metadata at ingest time

[chaudum]

What this PR does / why we need it:

This PR introduces a new feature that allows for extraction of "fields" into structured metadata at ingest time.

Fields can either be regular labels, structured metadata keys, or keys from logfmt or json formatted log lines.

The fields are defined in a per-tenant configuration as map[string][]string, where the key is the target key of the structured metadata, and the value is the list of source fields in given order and the order given above.

Example configuration:

limits_config:
  discover_generic_fields:
    trace_id:
      - "trace_id"
      - "TRACE_ID"
      - "traceID"
      - "TraceID"
    org_id:
      - "org_id"
      - "tenant_id"
      - "user_id"

Update: The fields moved into a "fields" objects:

limits_config:
  discover_generic_fields:
    fields:
      trace_id:
        - "trace_id"
        - "TRACE_ID"
        - "traceID"
        - "TraceID"
      org_id:
        - "org_id"
        - "tenant_id"
        - "user_id"

While parsing of log lines comes with a certain penalty at ingest time (increased latency and CPU usage on distributors), the idea is to extract certain fields once to avoid parsing the log lines every single time at query time. This is mainly useful in combination with bloom filters.

Discussion

Should the value of the config map support jsonpath expression, such as

limits_config:
  discover_generic_fields:
    ticket_id:
      - "message.ticket.id"

Where the log line looks like this:

{"timestamp": 1733128051000, "message": {"ticket": {"id": "2024-d95f87018cdb1f10"}}}

✔️ jsonpath support has been implemented with d216856

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[shantanualsi]

eventually I think we can deprecate the explicit discover_service_name and merge it to discover_generic_fields itself.. levels is a bit different as it explicitly uses detected_level as the SM label name..

[JoaoBraveCoding]

Overall lgtm, just one comment

[trevorwhitney]

@shantanualsi but discover_service_name creates service_name as an indexed label, this only populates structured metadata, correct?

@chaudum why put this in Loki rather than have this sort of thing be the responsibility of the agent?

[Nachtfalkeaw]

Hello,
I like this idea however I have an open issue where I like to automatically parse akll fields and then define the known fields to labels or structured_metadata:

grafana/alloy#2156

[chaudum]

@chaudum why put this in Loki rather than have this sort of thing be the responsibility of the agent?

Yes, ideally this is done on the agent.
However, there are cases where the operator of Loki is not in control of the agents and updating/changing client configuration is not easy to achieve.

[trevorwhitney]

overall looks good, a few small nits.

[trevorwhitney]

no longer doing the work of a levelDetecter, let's rename this variable

[trevorwhitney]

what about multiple matches (ie. org_id and tenant_id)? a test documenting that behavior would be nice

[chaudum]

This was a good call! The behaviour was different between json and logfmt lines. json matches the first field from the config, logfmt matches the first matching field in the log line.

[chaudum]

Fixed with 167f350

[periklis]

Considering the fact that this detection of fields happens on both push paths (Push, OTLP) how does this feature interact with the otlp_config where users can and I would dare say should actually list the OTLP resource/scope/log attributes to be extracted as structured metadata?

Note: Leaving the resource atributes to labels out here because the presented feature is only about structured metadata.

[chaudum]

Considering the fact that this detection of fields happens on both push paths (Push, OTLP) how does this feature interact with the otlp_config where users can and I would dare say should actually list the OTLP resource/scope/log attributes to be extracted as structured metadata?

Note: Leaving the resource atributes to labels out here because the presented feature is only about structured metadata.

I see it as an addition. The otlp_config is used to transform OTel push payloads into Loki push payloads, and the discover_generic_fields is used to extract parsed fields from JSON or logfmt logs lines into structured metadata (if not already present in logs labels or structured metadata).

[JoaoBraveCoding]

from a code POV lgtm

[chaudum]

There was a suggestion to slightly change the configuration schema to allow for possible future extension to extract all fields of a log line, e.g.

limits_config:
  discover_generic_fields:
    auto:
      format: json | logfmt | pattern
      pattern: "<_> FOO:<foo> <_>"
    fields:
      trace_id:
        - "trace_id"
        - "TRACE_ID"
        - "traceID"
        - "TraceID"
      org_id:
        - "org_id"
        - "tenant_id"
        - "user_id"

The auto object is subject for future implementations. The field definitions that are currently under discover_generic_fields move one level down into a fields object.

[github-actions]

💻 Deploy preview deleted.

[slim-bean]

LGTM!