Self-Hosted Loki vs Grafana Cloud for Rust Server Logs

#devops #monitoring

Originally published on kuryzhev.cloud

Your Carbon plugin's debug logging just quietly turned a free Grafana Cloud tier into a $200/mo surprise. That's not hypothetical — it happened to a community I consult for, and the root cause was a single label they never should have added. If you're trying to centralize Rust server logs across a growing fleet of Facepunch servers, the stack you pick and how you configure it matters more than most people assume before they hit the bill.

When you face this choice

You know the moment. You're running one Rust server, `journalctl -u rustserver.service -f` is good enough, and grep does the rest. Then you add a second server for a different wipe schedule, then a third with Carbon instead of Oxide, and suddenly you're SSH-ing into three boxes trying to correlate an RCON timeout on server 2 with a plugin crash on server 1 during the same wipe-day traffic spike. That's the trigger point. Raw console logs and journald don't scale across a fleet, and you need something that lets you query "show me every RCON error across all servers in the last hour" in one place.

Once you've decided Loki + Grafana is the right stack for this — and I think it is, LogQL is genuinely good for this use case — you hit a second decision immediately: run the pipeline yourself, or push everything to Grafana Cloud's managed Loki. There's also a smaller sub-decision baked into the self-hosted path: Promtail or Alloy. Promtail went into maintenance mode in early 2025, so if you're starting fresh, don't build on it — Grafana Alloy is the supported path now, and its `.alloy` config syntax replaces Promtail's YAML `scrape_configs` entirely.

Option A: Self-Hosted Loki Stack

This means running Loki, Grafana, and Alloy on your existing dedi or a small monitoring VM sitting next to your game servers. The pros are real: you control retention without paying per-GB, it runs fine on a $5-10/mo VM alongside the actual RustDedicated process, and no player data — SteamIDs, IPs — leaves your network. If your community cares about privacy or you're in a jurisdiction with data residency concerns, that last point isn't optional.

LogQL queries run against storage you own, so there's no cardinality billing surprise waiting for you. You can tune chunk size and retention specifically for wipe-cycle log volume patterns — Rust logs spike hard on wipe day and go quiet for the rest of the cycle, and self-hosted lets you plan storage around that shape instead of a flat monthly quota.

The cons are the usual self-hosted tax. You own upgrades, and this isn't trivial — Loki 3.1 made TSDB the default index store, deprecating the `boltdb-shipper`-only configs that most 2022-era tutorials still show. Copy-pasting an old config into a 3.x deployment will bite you. You also own backup of the object storage or filesystem, and alerting infra — either Alertmanager or Grafana OnCall — is on you to run and patch. And single-node Loki has no HA by default: if the VM reboots mid-wipe-day spike, you lose the ingest path until it comes back, and you'll have a gap in your dashboards exactly when you needed them most.

Option B: Grafana Cloud (Managed Loki)

Here you ship Rust and Carbon logs straight to Grafana Cloud's hosted Loki via Alloy, using one API key and a push endpoint. Zero infra to patch, built-in HA, dashboards and alerting live in the same UI you're already using. The free tier — 50GB of logs per month as of 2024 pricing — is often plenty for a handful of vanilla-ish Rust servers. Onboarding is genuinely fast: one Alloy config block, one API key, and you're querying logs in under 15 minutes. For a small community where the admin has zero ops experience, that's a legitimate win.

The cons show up the moment plugin logging gets verbose. Oxide and Carbon plugins — especially anti-cheat plugins that log every hit registration — can produce serious volume, and that free tier disappears fast. Budget roughly $0.50/GB beyond the free allotment as a planning number; it adds up quicker than people expect once a fleet is fully instrumented.

There's also a data-transit concern: logs go over the public internet to Grafana's endpoints, so you need to think about what's actually in those log lines — SteamIDs and player IPs show up more than admins realize. And Grafana Cloud enforces stricter label and cardinality limits than self-hosted defaults, which brings me to the mistake that drove this whole comparison.

Decision matrix

Score your situation against a few concrete criteria instead of guessing:

Monthly log volume: under ~2GB/day and fewer than 5 servers — either option works fine.
Volume at scale: over ~10GB/day, or you need 30+ day retention for ban appeal investigations — self-hosted becomes cheaper fast, often within the first month.
Existing infra: if you already run a Docker or Kubernetes host for the game servers, self-hosted's marginal cost is close to zero — you're adding containers, not a new box.
Ops tolerance: if you're on a bare VPS with no monitoring stack today and no appetite to build one, Grafana Cloud avoids weeks of yak-shaving you probably don't have time for.
Data sensitivity: logs containing SteamIDs and player IPs staying in-network pushes toward self-hosted, especially for EU-based communities.

My pick

I pick self-hosted Loki + Grafana Alloy for anything beyond a single hobby server. The cost curve on Grafana Cloud gets ugly fast once Carbon's debug logging is enabled fleet-wide, and I've watched it happen. The exception: a single small community server run by an admin with no ops background — there, optimize for their time, not dollars per gigabyte, and Grafana Cloud's free tier is the right call.

Here's the actual gotcha that shaped this recommendation. That $200/mo surprise I mentioned at the top came from labeling Loki streams by `steamid`. High-cardinality labels like SteamID or player_name create thousands of distinct streams, which destroys query performance and, on Cloud, gets billed per stream. A 500MB/day workload turned into a multi-GB nightmare because every unique player created a new stream. On self-hosted, that mistake costs you disk space and slow queries. On Grafana Cloud, it costs you actual money, every month, until someone notices.

Here's a working self-hosted setup. This docker-compose.yml runs Loki, Alloy, and Grafana together, assuming RustDedicated writes to ./rust-server/logs and Carbon writes to ./rust-server/carbon/logs:


# docker-compose.yml — self-hosted Loki + Grafana + Alloy for a Rust dedicated server host
# Assumes RustDedicated writes logs to ./rust-server/logs and Carbon logs to ./rust-server/carbon/logs
version: "3.8"

services:
  loki:
    image: grafana/loki:3.1.0
    container_name: loki
    ports:
      - "3100:3100"  # NOTE: bind to 127.0.0.1:3100 in prod, never 0.0.0.0 without auth
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml:ro
      - loki-data:/loki
    command: -config.file=/etc/loki/local-config.yaml
    restart: unless-stopped

  alloy:
    image: grafana/alloy:1.2.0
    container_name: alloy
    volumes:
      - ./alloy-config.alloy:/etc/alloy/config.alloy:ro
      - ./rust-server/logs:/var/logs/rust:ro         # RustDedicated console log
      - ./rust-server/carbon/logs:/var/logs/carbon:ro # Carbon plugin logs
    command: run /etc/alloy/config.alloy
    depends_on:
      - loki
    restart: unless-stopped

  grafana:
    image: grafana/grafana:11.2.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme_use_secrets  # rotate before going live
      - GF_INSTALL_PLUGINS=
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - loki
    restart: unless-stopped

volumes:
  loki-data:
  grafana-data:

And the Alloy config that tails both log paths, tags each with a server_name label (so you can filter per-server without touching Loki itself), and forwards to the local Loki instance. This is the piece most people skip — see the Loki docs for the full label reference before you deploy across a real fleet:


// alloy-config.alloy — tails Rust + Carbon logs, adds server_name label, ships to Loki
// Common mistake: forgetting to add a per-server label here breaks fleet-wide filtering later

local.file_match "rust_console" {
  path_targets = [{
    __path__    = "/var/logs/rust/*.log",
    job         = "rust-server",
    server_name = "wipe-server-01", // set per host to avoid label collisions across the fleet
  }]
}

loki.source.file "rust_console" {
  targets    = local.file_match.rust_console.targets
  forward_to = [loki.write.default.receiver]
}

local.file_match "carbon_plugins" {
  path_targets = [{
    __path__    = "/var/logs/carbon/*.log",
    job         = "carbon-plugins",
    server_name = "wipe-server-01",
  }]
}

loki.source.file "carbon_plugins" {
  targets    = local.file_match.carbon_plugins.targets
  forward_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

// Example LogQL query to run in Grafana Explore afterward:
// {job="rust-server", server_name="wipe-server-01"} |= "RCON" |= "error"

Two things I'd flag before you copy this. First, rotate the RustDedicated console log with logrotate using a copytruncate policy — without it, RustDedicated can grow a single log file to multiple GB per wipe cycle, and Alloy or Promtail re-tailing that on restart will hammer your CPU. Second, Carbon logs to file and RCON simultaneously by default; if you're shipping the file path already, disable duplicate RCON forwarding or you'll double your ingest volume for nothing.

Whichever path you pick, don't expose Loki's push API on port 3100 directly to the internet — it has no built-in multi-tenant auth in single-binary mode, so put it behind a reverse proxy with basic auth or TLS at minimum. For more on wiring monitoring stacks into container setups like this, check the DevOps_DayS archive — there's overlap with how we handle similar log pipelines for other services.

Centralizing Rust server logs isn't complicated once the pipeline is right — the complexity is almost entirely in the label design and the retention math, not in Loki itself. Get the labels right on day one and this scales to a dozen servers without drama.