Originally published on kuryzhev.cloud
Your Prometheus dashboard says everything is green while players are rage-quitting in Discord over lag. That disconnect happens because node_exporter and blackbox_exporter have no idea what's happening inside your game server's UDP query protocol. Building a proper prometheus game server exporter is the only way to close that gap, and it's easy to get wrong in ways that only show up under real player load.
Why this checklist
Every gaming infra team I've worked with eventually hits the same wall: CPU and memory graphs look fine, but support tickets say "the game feels laggy." That's because player_count and query_latency are the actual SLIs players feel — not the generic host metrics your exporters already scrape. A server can sit at 20% CPU and still have a broken matchmaking queue or a query protocol timing out under load.
The problem is that most game engines don't expose Prometheus-friendly metrics out of the box. Source engine servers, Unreal dedicated servers, and custom UDP-based protocols speak A2S, RCON, or proprietary JSON — none of which Prometheus understands natively. You need glue code: something that polls the game server's own query protocol and translates it into metric types Prometheus can scrape.
I've reviewed enough of these exporters, written by different teams at different studios, to notice a pattern: the mistakes are rarely about missing knowledge. They're small oversights — a Counter where a Gauge belongs, a missing timeout, a label that quietly explodes cardinality. That's exactly the kind of failure a checklist catches better than a tutorial does. You don't need to relearn Prometheus fundamentals every time you ship a new exporter; you need something to run down before merging the PR. That's what follows.
The checklist (numbered)
-
Pick the right metric type. Use
Gaugeforgameserver_player_count— it goes up and down. UseHistogramforgameserver_query_latency_secondswith buckets matched to your game genre (tight buckets like[0.05, 0.1, 0.25]for FPS, wider ones for MMO tick rates). Never useSummaryfor latency if you plan to aggregate across instances in PromQL — quantiles from a Summary can't be averaged mathematically across servers. -
Follow naming conventions. Stick to
gameserver_<noun>_<unit>, e.g.gameserver_query_latency_seconds. This matches the Prometheus metric naming guidelines and keeps your Grafana queries predictable. -
Bound your label set. Label by
region,shard, ormap— never byplayer_id,session_id, or IP. Labels must be a known, finite set decided at design time, not something that grows with player count. -
Make the query layer non-blocking with a hard timeout. Poll the game server's query protocol on a background thread or async loop, with a timeout shorter than Prometheus's
scrape_timeout. The HTTP handler serving/metricsshould only read from an in-memory cache, never make a live network call. -
Separate
/healthzfrom/metrics. A liveness probe checking exporter process health shouldn't depend on the game server being reachable — otherwise a game server outage takes down your monitoring too. -
Package it properly. Run as a non-root user, bind to
127.0.0.1:9105by default (9105+ is the unofficial exporter port range), and let a sidecar or reverse proxy handle external exposure. -
Self-instrument. Expose
gameserver_exporter_build_infoandgameserver_exporter_scrape_duration_secondsso you can tell "game server is down" from "exporter is down." -
Validate output in CI. Run
promtool check metricsagainst a live curl of/metricsbefore shipping — this catches missing#TYPE/#HELPlines that a plain curl check would miss.
Here's a minimal exporter that satisfies most of the checklist above — it polls a Source-style query protocol on a background thread and exposes the metrics on a local-only port. Pin prometheus-client==0.20.0; pre-0.9 versions had registry behavior that broke silently on re-registration.
#!/usr/bin/env python3
# gameserver_exporter.py - polls a UDP query protocol and exposes Prometheus metrics
import socket
import time
import threading
from prometheus_client import start_http_server, Gauge, Histogram, Counter
# --- Metric definitions (checklist item: correct types + labels) ---
PLAYER_COUNT = Gauge(
"gameserver_player_count",
"Current number of connected players",
["region", "shard"]
)
QUERY_LATENCY = Histogram(
"gameserver_query_latency_seconds",
"Latency of the query protocol round-trip",
["region", "shard"],
buckets=[0.05, 0.1, 0.25, 0.5, 1, 2, 5]
)
QUERY_FAILURES = Counter(
"gameserver_query_failures_total",
"Number of failed queries to the game server",
["region", "shard"]
)
SERVERS = [
{"host": "10.0.1.10", "port": 27015, "region": "eu-west", "shard": "shard-1"},
{"host": "10.0.1.11", "port": 27015, "region": "eu-west", "shard": "shard-2"},
]
QUERY_TIMEOUT = 3.0 # must stay well below Prometheus scrape_timeout (10s)
def query_server(host, port):
"""Send a minimal A2S-style query, return (player_count, latency_seconds)."""
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.settimeout(QUERY_TIMEOUT)
start = time.monotonic()
try:
sock.sendto(b"\xFF\xFF\xFF\xFFTSource Engine Query\x00", (host, port))
data, _ = sock.recvfrom(1024)
latency = time.monotonic() - start
# NOTE: real parsing depends on protocol; this is illustrative
players = data[-1] if data else 0
return players, latency
finally:
sock.close()
def poll_loop():
while True:
for srv in SERVERS:
labels = {"region": srv["region"], "shard": srv["shard"]}
try:
players, latency = query_server(srv["host"], srv["port"])
PLAYER_COUNT.labels(**labels).set(players)
QUERY_LATENCY.labels(**labels).observe(latency)
except (socket.timeout, OSError):
QUERY_FAILURES.labels(**labels).inc()
time.sleep(15) # matches scrape_interval to avoid stale-vs-fresh mismatch
if __name__ == "__main__":
threading.Thread(target=poll_loop, daemon=True).start()
# bind to localhost only; expose via reverse proxy/service mesh in prod
start_http_server(9105, addr="127.0.0.1")
while True:
time.sleep(3600)
And the matching scrape config, plus what a healthy scrape should actually look like when you validate it with promtool:
# prometheus.yml snippet - scrape config for the exporter above
scrape_configs:
- job_name: game_servers
scrape_interval: 15s
scrape_timeout: 10s # must exceed QUERY_TIMEOUT * number of servers polled synchronously
static_configs:
- targets: ["10.0.2.5:9105"]
labels:
env: production
# --- expected /metrics output (validate with promtool check metrics) ---
# HELP gameserver_player_count Current number of connected players
# TYPE gameserver_player_count gauge
gameserver_player_count{region="eu-west",shard="shard-1"} 42
# HELP gameserver_query_latency_seconds Latency of the query protocol round-trip
# TYPE gameserver_query_latency_seconds histogram
gameserver_query_latency_seconds_bucket{region="eu-west",shard="shard-1",le="0.1"} 12
gameserver_query_latency_seconds_bucket{region="eu-west",shard="shard-1",le="0.5"} 30
gameserver_query_latency_seconds_bucket{region="eu-west",shard="shard-1",le="+Inf"} 34
gameserver_query_latency_seconds_sum{region="eu-west",shard="shard-1"} 5.32
gameserver_query_latency_seconds_count{region="eu-west",shard="shard-1"} 34
Run promtool check metrics < <(curl -s localhost:9105/metrics) before every deploy. It's a five-second check that catches malformed output long before Grafana starts mis-rendering panels.
Commonly missed items
Cardinality is the big one. I've seen a 50-server fleet turn into millions of active time series because someone added a player_id or session_id label "for debugging." Prometheus doesn't age out label combinations gracefully — it just keeps ingesting until memory runs out. Watch out for this: it doesn't show up in dev with 3 test players, only in prod with real traffic.
Metric type confusion is the second-most common bug. Using Counter for player_count means every server restart looks like a reset, and your alerting rules — which usually assume Counters only go up — will fire false "anomaly" alerts every time you deploy. Gauge is correct here; the value can legitimately go down when players leave.
Blocking scrapes are a quieter failure. If the /metrics handler queries the game server synchronously with no timeout, a single hung UDP request blocks the entire scrape past the 10s scrape_timeout. The result is up == 0 flapping exactly when you need visibility most — during a load spike or DDoS attempt.
And don't skip the security angle. An unauthenticated /metrics endpoint leaks player counts and internal server topology to anyone scanning ports. I stopped exposing exporters directly to the internet after a griefer group used exposed metrics to time raids on low-population shards. Put it behind mTLS, a firewall rule, or federate it through a Prometheus instance with basic_auth configured in the scrape job — see the Prometheus scrape configuration docs for the auth options.
Automation ideas
Once the exporter pattern works for one game server, the goal is fleet-wide repeatability, not another hand-deployed binary. Bake the exporter into the same pod as the game server as a sidecar, and auto-register it with a Prometheus Operator PodMonitor. Use relabel_configs to drop noisy Kubernetes metadata like __meta_kubernetes_pod_uid before it ever hits your TSDB — that alone can cut series count meaningfully across a large fleet.
Generate dashboards and alert rules as code — jsonnet or Terraform templated per game title — so spinning up a new shard doesn't mean someone manually clones a Grafana dashboard and forgets to update a variable. We covered a similar templated-alerting pattern in our DevOps_DayS monitoring notes if you want the broader pattern beyond game servers.
Finally, add a CI job that spins up the exporter against a mock UDP server, curls /metrics, runs promtool check metrics, and asserts histogram bucket boundaries with a Python unit test. Fail the build if #HELP or #TYPE lines are missing — that's the cheapest bug to catch in CI and the most expensive one to catch in a 3am page. And never tag exporter images as latest; a silent metric rename in a new build breaks every downstream dashboard without a visible diff in your deploy log.
Getting a prometheus game server exporter right isn't about clever engineering — it's about running down the same short list every time before it hits production. The checklist above is exactly what I run before merging any exporter PR now, and it's caught real bugs every single time.
Top comments (0)