Oleksandr Kuryzhev

Posted on Oct 20 Originally published at kuryzhev.cloud

Rust game server logs in Loki: avoid label and parsing traps

#devops

Originally published on kuryzhev.cloud

A wipe-day restart goes badly, players report rubber-banding, and the only evidence is a large text file on one host that the next restart may overwrite. Centralizing Rust game server logs in Loki and Grafana fixes that, but only if you decide up front three things: what becomes a label, what becomes a searchable line, and what never leaves the box. This article covers the Facepunch game's dedicated server (RustDedicated), not the programming language.

What this actually does

Loki does not index log text. It indexes a small set of labels, such as server or env, and stores the log lines compressed in chunks. A query picks streams by label, then scans the matching chunks with filters. Grafana is the query and dashboard layer on top.

For a Rust server, the pipeline has three parts:

The game process writes to a file (Unity's -logfile argument) or to stdout.
A collector reads that output and attaches labels.
Loki stores it.

As of 2026 the collector should be Grafana Alloy. Promtail is deprecated and reached end of life in early 2026 (verify the exact date in the Grafana documentation).

The useful sources are:

The main server log.
The plugin framework logs. Oxide/uMod or Carbon write their own files, and their locations depend on the install.
Optionally, RCON output.

Each has a different shape, so treat them as separate job values rather than one blob.

How people use it wrong

The most common failure is turning player data into labels. A steamid, player name or IP address looks like a natural thing to filter by, so people promote them with a labels stage. Every distinct value creates a new stream, and on a populated community server that number keeps growing without bound. The documented consequences are:

A large index.
Many tiny chunks.
Slow queries.
Ingestion errors once per-tenant stream limits are reached.

Watch out for: labels should describe where a log came from, not what it says. If the set of possible values is unbounded, it does not belong in a label.

Other typical mistakes:

Tailing a file that the game truncates. If the server reuses one log filename across restarts, collectors can lose their position or re-read lines. Give each run a timestamped filename and match with a glob.
Shipping everything. Rust consoles are chatty. Paying to store repetitive status lines you never query is the easiest cost to avoid.
Ignoring privacy. Join lines typically contain IP addresses and Steam IDs, and chat logs contain player speech. Under GDPR-style rules that is personal data, so decide on retention and access before the first push.
Treating stack traces as separate events. Unity exceptions span many lines. Without multiline handling, one error becomes many unrelated entries.
Running Loki with defaults and no retention. Compactor retention is disabled by default (retention_enabled: false). Unless you enable it, nothing is ever deleted.

The correct approach

Keep labels to a handful of bounded values: job, server, env, and optionally level.

Extract high-cardinality fields like Steam IDs as structured metadata, which Loki 3.x supports. They stay attached to each line and can be filtered without creating streams. Structured metadata requires the TSDB index with schema v13. Verify this in the Loki documentation for your version.

The following Alloy configuration does four things:

Tails timestamped log files.
Drops blank lines.
Pulls a Steam ID out of join-style lines.
Pushes to Loki.

The regex is an example, so check it against your server's actual output.

// Discover log files; labels here are the ONLY stream labels
local.file_match "rust" {
  path_targets = [{
    __path__ = "/srv/rust/logs/*.log",  // one file per server run
    job      = "rust-server",
    server   = "eu-main",
    env      = "prod",
  }]
}

loki.source.file "rust" {
  targets    = local.file_match.rust.targets
  forward_to = [loki.process.rust.receiver]
}

loki.process "rust" {
  // Drop empty lines before they cost storage
  stage.drop {
    expression = "^\\s*$"
  }

  // Example pattern: 17-digit SteamID64 followed by a "joined" message
  stage.regex {
    expression = "(?P<steamid>\\d{17})/.*joined"
  }

  // Structured metadata: searchable, but NOT a stream label
  stage.structured_metadata {
    values = { steam_id = "steamid" }
  }

  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

On the Grafana side, add Loki as a data source. Start with queries that select by label and then filter. Line filters (|=) are cheap, so put them before any parser.

# Errors on one server, line filter first, then count per 5 minutes
sum by (server) (
  count_over_time({job="rust-server", server="eu-main"} |~ "(?i)exception|error" [5m])
)

# Everything one player did, using structured metadata (no regex needed)
{job="rust-server", server="eu-main"} | steam_id = "76561198000000000"

Advanced patterns

Logs for events, metrics for numbers. Player count and server FPS are time series. RustDedicated does not expose a native Prometheus endpoint, but an RCON-based exporter can collect them. Scraping that into Prometheus is cheaper and more accurate than counting join and leave lines. Use Loki for the question "what happened at 21:14", and Prometheus for "how did it trend". Grafana can show both on one dashboard with a shared time range.

Recording and alert rules. Loki's ruler evaluates LogQL on a schedule. You can alert on an exception rate above a threshold, or on the absence of any log line for several minutes (for example with absent_over_time), which usually means a hung process. Alert on rates over a window instead of single lines, otherwise one noisy plugin pages you all night.

Multiline stack traces. Alloy has a stage.multiline stage that merges continuation lines into one entry. It needs a reliable "first line" pattern. Whether Rust output has one depends on how you launch the server and which plugins add timestamps. Verify against real files before relying on it.

Multiple servers. With several game instances, run Alloy on each host (or as a sidecar) and set server per instance. Keep the label set identical across them so one dashboard variable can switch between servers.

If you run them in Kubernetes, use apps/v1 workloads with discovery.kubernetes. Pod metadata is not turned into labels automatically: you map the fields you want with discovery.relabel. Promote only bounded fields such as namespace or app, not pod names or UIDs. Other infrastructure topics are on kuryzhev.cloud.

Plugin logs as JSON. If you write or configure plugins, emitting JSON lines lets you use | json at query time and avoids brittle regexes. Parse at query time first. Move extraction to ingestion only for fields you query constantly.

Performance notes

Cost and speed in Loki follow stream count and bytes scanned. Few streams with large chunks compress well and query fast. Many small streams do the opposite.

Check the active stream count through Loki's own metrics (for example loki_ingester_memory_streams) and in Grafana's label browser. A label with hundreds of values is a warning sign.

Reduce volume at the source. Dropping noise in Alloy, before the push, saves network, storage and query time. Do it deliberately:

Keep errors, warnings, joins, leaves, kills and admin commands.
Drop only patterns you have confirmed are repetitive.

Watch out for ingestion limits. Loki enforces per-tenant and per-stream rate limits (ingestion_rate_mb, per_stream_rate_limit), and the defaults are modest. A server dumping a large backlog after a restart can hit HTTP 429 responses.

Alloy's loki.write retries with backoff, but only up to its retry limit (max_backoff_retries). After that the batch is dropped, and a long backlog also delays recent logs. Verify the current defaults in the limits_config reference for your Loki version. Raise them only after you understand the burst.

For queries:

Narrow the time range and the label selector before anything else.
Use line filters before parsers.
Avoid unbounded regexes over multi-day ranges.

For automatic deletion of old wipes, set retention_period in limits_config. Then enable it on the compactor with retention_enabled: true and a configured delete_request_store. Thirty days is often plenty for operations. Anything containing IP addresses should expire sooner unless you have a documented reason to keep it.

Finally, rotate and prune the source files on the host too. Centralized storage does not remove the need to stop a game server from filling its own disk.

DEV Community