How I built a Steam VoC engine: LLM assigns meaning, code counts numbers

Three months into running findindiegame.com, I kept bumping into the same limitation: every VoC analysis I ran was a one-shot snapshot. The jp-en-divergence script tracked English vs. Japanese review ratios over time, but it counted review counts — it never stored the actual text. When I tried to compare how players talked about a game two weeks ago versus today, I had nothing to compare.

This post is about building the persistent corpus and counting layer that fixes that. The core architectural decision I reached is something I'm calling the TERUS rule: LLMs assign meaning, code counts numbers. If the model touches counting, the numbers drift between runs. If code assigns meaning, the categories are too rigid to handle natural language. Each layer does what it's actually good at.

Why one-shot snapshots break for time-series VoC

The naive approach to VoC is: fetch reviews, send them to a model, get a summary. The problem is the result has no memory and can't be verified. When the model says "38 reviews mention performance issues," I can't audit that number the next day because I don't have the review texts stored. I can't tell if a topic is rising or falling without re-running the analysis over historical data I never saved.

Three specific failures this produced before I rebuilt the pipeline:

1. I couldn't measure whether a complaint about a specific game was a new surge or a known baseline
2. Repeating the analysis over the same window of time often produced different topic names because the model would cluster things differently
3. Any claim about "what players say" was unverifiable — I had no raw source to point to

The fix is structural: save the raw text first, run deterministic code second, let the model only do what code genuinely can't. The LLM step becomes a mapping function from text to structured labels, not a summarizer with its own arithmetic.

The corpus layer: append-only JSONL, one file per game

The collector (scripts/voc/collect-reviews.mjs) fetches Steam reviews from the public storefront endpoint — no API key, no authentication:

https://store.steampowered.com/appreviews/<appid>
  ?json=1&language=<lang>&filter=recent&num_per_page=50&purchase_type=all

Fifty reviews per language per game per day, fetched at one request per second to stay polite. Each line appended to data/voc/reviews/<appid>.jsonl looks like this:

{
  "rid": 139847362,
  "appid": 413150,
  "lang": "english",
  "date": "2026-08-23",
  "voted_up": true,
  "playtime_h": 47,
  "text": "Still crashing on the new GPU drivers since the August patch.",
  "fetched": "2026-08-24"
}

The key design decision: append-only, never update. If Steam edits a review, we don't rewrite the stored entry. The rid (Steam's integer recommendationid) is the dedup key — a second fetch seeing the same ID skips it. This is intentional. The corpus is a historical record of what players published at the time we first saw it, which is what you actually want for trend analysis. A player who edits their negative review after a bug is patched shouldn't disappear from the historical spike data.

Text is truncated at 1000 characters. Steam review medians are far shorter, but the cap keeps the corpus bounded — at 100 reviews/day, a year of collection is roughly 35 MB of raw JSONL.

Game selection (candidate appids) comes from three sources in priority order: games named in the market trends document's "Hot subjects" list, the top 10 games by JP-EN review gap (≥5pp), and games used in video matchups uploaded in the last 14 days. Hard cap: 20 appids per day. At 2 languages × 1 request per second, that's about 40 seconds of total fetch time per daily run.

One edge case I had to handle explicitly: the corpus can't be bootstrapped on day one. If the JSONL file for an appid doesn't exist, the collector treats all fetched reviews as new — a normal "first day" state, not an error. But if the file exists and fails to read for any other reason (permissions, corruption), the collector skips that appid entirely and adds it to the error list rather than treating it as a first-day corpus. Silent ETL failures that look like successes are the hardest to diagnose later.

The topic registry: claims, not keywords

Before building the counting layer, I needed a schema for what the LLM should assign. The naive approach is a keyword list: "performance," "price," "story." The problem is that "performance" covers both "the game runs flawlessly at 120fps" and "this is completely unplayable on my hardware" — opposite claims, same keyword.

The registry stores full assertions: "起動・ロード遅延" (slow startup/loading), "価格が高い" (price is high), "ストーリーが薄い" (thin story). Not bare nouns. Two reviews with opposite experiences of game performance get assigned to different topics even if they both mention "performance." The distinction is what the player is claiming, not what they're mentioning.

Each topic entry in data/voc/topics.json has a stable ID (t_0001, t_0002, …), the claim text, the date the topic first appeared, a status (active or merged), and optionally a merged_into pointer. Merge chains matter: when I clean up redundant topics, old assignments follow the merge chain to the canonical form. The counting layer always resolves through merge chains before tallying, so history stays consistent through registry cleanups.

I built the initial registry by hand, which I'd do differently in retrospect. Running the daily routine on a pilot corpus of 200 reviews first, letting the model propose topic labels, then distilling recurring clusters into initial entries would have produced a more grounded schema than top-down taxonomy design.

The counting layer: what the LLM never touches

This is the core of the TERUS pattern. The daily Cloud Routine reads new review bodies, maps each review to one or more topics in the registry, and appends one JSONL line per (review, topic) pair to data/voc/assignments.jsonl. That's the only thing the LLM does.

Everything numeric afterward is handled by scripts/voc/topics.mjs --count. No model invocations, no judgment calls. The script reads assignments.jsonl, resolves merge chains, deduplicates (rid, canonical_topic) pairs, and produces a topic × date × language matrix:

{
  "generated": "2026-08-24",
  "topics": {
    "t_0003": {
      "claim": "起動・ロード遅延",
      "counts": {
        "2026-08-22": { "english": 0, "japanese": 3 },
        "2026-08-23": { "english": 1, "japanese": 4 },
        "2026-08-24": { "english": 2, "japanese": 7 }
      }
    }
  }
}

The matrix is recomputable from scratch at any time from the two source files (reviews/ JSONL files + assignments.jsonl). Every claim about "X reviews said Y" traces to a specific rid in a specific appid file. If someone challenges a number, I can grep the assignments file and count manually.

I run the counter with --count to rebuild the matrix, --alerts to check for negative spikes, and --render to produce the daily report section. These are separate flags precisely because each stage of an ETL pipeline should be independently re-runnable.

Negative-spike alerts: 3× the 7-day median, minimum n=5

A count matrix is useful for charts. It becomes operationally useful when it tells me something changed. The alert rule: if today's negative-polarity count for a topic exceeds 3× the 7-day median for that topic, and today's count is at least 5, fire an alert.

The 3× multiplier filters noise while catching real shifts. The minimum-n=5 prevents a 0→2 jump on a low-volume game from generating alerts — a jump from zero to two is not a crisis. Both constants are exported from the module so tests can override them without monkey-patching.

One refinement I haven't built yet: a rising-trend detector that compares the last 7 days against the 7 days before that. The current spike detector only catches same-day anomalies; a game where negative sentiment rises by 1 extra mention per day for 10 consecutive days wouldn't trigger until day 8. The --trend flag is in the codebase but not yet wired into the daily alert channel.

The daily pipeline in practice

The full flow runs at 16:30 UTC via a GitHub Actions workflow:

1. collect-reviews.mjs fetches and appends new review bodies to the per-appid JSONL files
2. A Claude routine reads new review texts and assigns topic IDs, appending to assignments.jsonl
3. topics.mjs --count rebuilds the count matrix from assignments
4. topics.mjs --render adds the VoC section to the daily analytics report

The LLM is involved in exactly one step. If I need to reanalyze the corpus with a revised topic registry, I delete and rebuild assignments.jsonl from the stored review bodies — the texts are already there, no re-fetching needed. The corpus is the record; assignments are derived from it; counts are derived from assignments. Each layer recomputable from the one beneath.

What worked cleanly: the rid-dedup append pattern. Simple, predictable, with an explicit model of what "first seen" means. The same pattern works well for other append-only corpora like Bluesky follow logs, and it avoids the silent double-count failure you get when dedup logic sits in the writer rather than the reader.

What didn't: I initially asked the LLM to output a confidence score alongside the topic assignment (0.0–1.0). The scores were inconsistent across runs — the same review would get 0.7 on Monday and 0.4 on Thursday. I removed confidence scoring entirely. The binary assignment is what matters; filtering by polarity already provides a softer quality signal.

What I'd do differently: the 3×-median alert threshold was chosen by intuition rather than calibration. I'll recalibrate it in 30 days when the corpus has a real 7-day history and I can see how many alerts fire versus how many represent actual news worth reading.

FAQ

Why not use an existing sentiment library instead of building this?

Off-the-shelf sentiment classifiers (VADER, TextBlob) classify polarity — positive/negative/neutral — but not topic. Knowing a review is negative doesn't tell me what it's negative about. The topic layer is what makes the counts actionable for product decisions.

Why Steam specifically?

Steam provides a free documented JSON endpoint with no authentication required. The recommendationid integers are stable, which makes dedup clean and unambiguous. Google Play and App Store both require credentials and have tighter rate limits.

What happens when a player deletes their review?

We keep the first-seen text. The corpus is a historical record, not a live mirror. A player who complains about a crash bug and deletes the review after a patch shouldn't vanish from the historical trend — the spike that happened still happened.

Can this work on non-Steam review sources?

The appid-selection logic is Steam-specific, but the topic registry, assignments format, and counting layer are generic. If you can build a collector that appends {rid, source, lang, date, voted_up, playtime_h, text, fetched} lines to a JSONL, the semantic and counting layers apply unchanged.

How do you handle topics the LLM invents that aren't in the registry?

Unknown topic IDs in assignments.jsonl are counted, printed to stderr, and written into the counts artifact as an error. The process exits with code 3 after finishing all work. This makes unknown-topic assignments visible and repairable rather than silently excluded.

---

Related: Pull-based JSON handoff ledger for Claude-Codex coordination · Four filters I apply when pulling HuggingFace models into a directory

Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.