Originally published on kuryzhev.cloud
The scenario
A checkout request fails, you find the error in Grafana, and the log line has no trace link. The trace exists in Tempo, but nothing connects the two, so you copy IDs by hand. This is a common outcome when you ship logs from the OpenTelemetry Collector to Loki through a path that flattens the log record into a plain string and drops its trace fields.
The timing is good for a cleanup. Grafana 13.3 is the current release cycle (October 2026). Read its what's new page before you upgrade, because log and trace correlation lives in the Loki data source and Explore. The configuration below uses documented provisioning fields, not features specific to 13.3.
Short answer: use Loki's native OTLP endpoint (/otlp) with the Collector's otlphttp exporter, not the deprecated loki exporter, which has been removed from current contrib builds. Keep trace_id and span_id as structured metadata, never as labels. Then add a derived field in the Loki data source that points at Tempo. This applies when you run Loki 3.x and want one pipeline for logs, metrics and traces. It does not pay off if you only grep logs and never open traces.
Prerequisites
Check these before you change anything:
-
Loki 3.x with a TSDB index and schema v13. Loki keeps trace IDs in structured metadata, and structured metadata requires both. Verify your
schema_configand theallow_structured_metadatalimit in the Loki OTLP documentation. -
An OpenTelemetry Collector contrib distribution. The
filelogreceiver andk8sattributesprocessor ship in contrib, not in the minimal core build. See the Collector docs. - A Tempo data source in Grafana (13.3.x here) and applications that already emit traces.
-
Network access from the Collector to Loki, for example a
lokiService on port 3100. If Loki is multi-tenant, you also need the tenant ID.
Accept one cost up front: structured metadata adds storage and query-time work per log line. For high-volume debug logs, sample or filter them in the Collector before they reach Loki.
Step 1: Preserve trace context at the source
Trace context survives only if it reaches the Collector as fields on the log record, not as text inside the body. There are two paths.
Path A (preferred): the application uses an OpenTelemetry SDK log bridge or appender and sends OTLP to the Collector. When a log is emitted inside an active span, the SDK attaches that span's trace and span IDs to the record. You change nothing in the pipeline.
Path B: the application writes JSON lines to stdout that include trace_id and span_id, and the Collector tails the files. The Collector then has to promote those values into the record's real trace fields. The trace_parser operator in the filelog receiver does that. Without it, the IDs stay as ordinary attributes and the correlation features never see them.
Watch the ID formats. The documented expectation is a 32-hex-character trace ID and a 16-hex-character span ID. If your logger prints a different format, the parser cannot turn it into a valid trace context, so normalize it in the application's logger.
Step 2: Configure the Collector pipeline
This config covers both paths: OTLP in, plus Kubernetes container logs with trace parsing. It exports to Loki over native OTLP. Adjust paths, namespaces and the Loki address to your environment.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317 # recent releases default to localhost only
http:
endpoint: 0.0.0.0:4318
filelog:
include: [/var/log/pods/*/*/*.log]
# exclude the Collector's own pods to avoid a feedback loop, e.g.:
# exclude: [/var/log/pods/observability_otel-collector-*/*/*.log]
include_file_path: true
operators:
- type: container # parses the CRI/containerd line format
- type: json_parser # only for JSON bodies
parse_from: body
if: 'body matches "^\\{"'
- type: trace_parser # promotes IDs into real trace fields
trace_id:
parse_from: attributes.trace_id
span_id:
parse_from: attributes.span_id
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 80
k8sattributes: {} # adds k8s.namespace.name, k8s.pod.name, ...
batch: {}
exporters:
otlphttp/loki:
endpoint: http://loki:3100/otlp # the exporter appends /v1/logs itself
# headers:
# X-Scope-OrgID: team-a # only if auth_enabled: true in Loki
service:
pipelines:
logs:
receivers: [otlp, filelog]
processors: [memory_limiter, k8sattributes, batch]
exporters: [otlphttp/loki]
Gotcha: do not append /v1/logs to the endpoint. The exporter adds it, and a doubled path produces 404 errors that look like a Loki outage. Keep batch after memory_limiter and k8sattributes so you batch the final, enriched records. Binding to 0.0.0.0 exposes the receivers on the pod network, so restrict access with a NetworkPolicy if needed.
The container operator and its options have changed across Collector releases. Confirm the field names against the filelog receiver README for the version you deploy.
Step 3: Decide what becomes a label
Loki's OTLP ingestion converts a documented default list of resource attributes into index labels, such as service.name, k8s.namespace.name and k8s.pod.name. Dots become underscores, so you query service_name. Everything else, including trace_id and span_id, is stored as structured metadata. That is the behavior you want.
Avoid the cardinality trap: never promote trace_id to a label. Every request would create a new stream, and Loki's performance and cost would degrade quickly. If you need different index labels than the defaults, set them on the Loki side. ignore_defaults: true makes your list replace the defaults instead of adding to them:
# loki.yaml (excerpt)
limits_config:
allow_structured_metadata: true # required for trace_id/span_id storage
otlp_config:
resource_attributes:
ignore_defaults: true # use only the list below as index labels
attributes_config:
- action: index_label # keep this list short and low-cardinality
attributes:
- service.name
- k8s.namespace.name
Check the exact key layout against the Loki version you run, because otlp_config options are version-dependent.
Step 4: Link logs to traces in Grafana
Provision a derived field on the Loki data source so each log line with a trace ID gets a link into Tempo. Provisioning from files keeps this reviewable, and the same idea applies to any other dashboards-as-code setup you keep in Git.
# provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
uid: loki
url: http://loki:3100
jsonData:
derivedFields:
- name: TraceID
matcherType: label # read from structured metadata, not a regex on the line
matcherRegex: trace_id
url: '$${__value.raw}' # $$ escapes env expansion in provisioning files
datasourceUid: tempo # UID of your Tempo data source
If your Grafana version does not offer matcherType: label, fall back to a regex derived field that matches the ID in the line. Check the Loki data source docs for the supported options.
On the Tempo side, a "logs for this span" link needs the Loki data source configured under Tempo's trace-to-logs settings. Keep the filter on service_name and k8s_namespace_name, and avoid tag mappings that don't exist in your labels.
Verify and test
First, send a synthetic OTLP log with a known trace ID straight to the Collector's HTTP port:
curl -sS -X POST http://localhost:4318/v1/logs \
-H 'Content-Type: application/json' \
-d '{"resourceLogs":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"otlp-smoke"}}]},
"scopeLogs":[{"logRecords":[{"timeUnixNano":"'"$(date +%s)"'000000000",
"severityText":"ERROR","body":{"stringValue":"smoke test"},
"traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174"}]}]}]}'
# then query Loki: label for the stream, structured metadata filter for the ID
# add --org-id=<tenant-id> if Loki runs with auth_enabled: true
logcli query --addr=http://loki:3100 \
'{service_name="otlp-smoke"} | trace_id="5b8efff798038103d269b633813fc60c"' --since=10m
A healthy result is one log line returned for that filter. A bare 200 from the Collector only means it accepted the record, so the Loki query is the real check. In Grafana Explore, the same line should show trace_id under structured metadata and a TraceID link that opens Tempo.
Next, watch the Collector's own metrics:
-
otelcol_exporter_sent_log_recordsshould climb. -
otelcol_exporter_send_failed_log_recordsshould stay flat.
Recent Collector versions expose these with a _total suffix on the Prometheus endpoint, and names can differ by version. Confirm them on the Collector's internal telemetry endpoint.
If lines arrive but links are missing, check these in order:
- The log record had a trace ID when it left the Collector. Path B usually fails here.
-
allow_structured_metadatais on. - The derived field points at the right Tempo UID.
Where this leaves you
The reliable pattern has four parts: native OTLP into Loki, trace and span IDs kept as structured metadata, a short list of low-cardinality labels, and one derived field into Tempo. Do not adopt it if your logs are unstructured and uninstrumented. There is no trace context to preserve, and the extra storage buys you nothing. Start with one service on Path A, confirm the smoke test above, and only then roll the filelog pipeline out cluster-wide.
Top comments (0)