Flat review counts are a bad signal for bug priority. A topic with 54 complaints looks less urgent than one with 148 positive mentions. But if the 54 come from players who have each put in a median 152 hours, and the 148 come from players with a median of 88 hours, the priority ordering reverses completely.
That is what the playtime data in my VoC engine showed when I joined it to the topic assignments.
What the engine collects and how
The engine currently holds 4,888 Steam review bodies across the titles I track, with 753 assignments across 46 topics. Each review body is passed to a language model that assigns it to a topic from a fixed taxonomy. The model decides meaning; a script tallies the assignments. I wrote about the architecture of this split when I shipped the first version — the core rule is that you never ask the model how many, because models count badly. Every number in the output comes from a database query over the assignments table.
The collection script pulls playtime_forever from the Steam Web API alongside the review text. Each assignment row is joined to the reviewer's total playtime via review ID. That join is what makes the playtime analysis possible.
The result that changed the priority ordering
Three topics, three different playtime profiles:
| Topic | Voices | Median playtime | 50h+ share |
|---|---|---|---|
| Gameplay is addictive, you lose track of time | 148 | 88.1 h | 68% |
| More fun than expected | 58 | 36.1 h | 41% |
| Bugs and technical problems hurt the experience | 54 | 152.2 h | 74% |
The bug topic has the lowest voice count and the highest median playtime. Three quarters of the bug reporters are past 50 hours. These are not people who bounced from the tutorial; they are players who have seen the edge cases that appear after dozens of sessions. They are reporting bugs that matter to the most invested part of the player base.
A flat count says "54 complaints, lower priority than 148 positive mentions." The playtime join says "54 veteran reports, highest priority."
Why this matters more than better model accuracy
The insight is not about the language model. The topic taxonomy and the assignment model are the same before and after you add playtime. What changes is the interpretation layer: the code that uses the assignment counts to make a recommendation.
I've been noticing this pattern in other projects too. A second system I'm working on — an edge-AI shelf detector running on a Raspberry Pi 3 — has a similar structure: a YOLO11n model with 0.844 mAP on the held-out test set that wasn't reliably useful until three post-processing layers in code were added. The model's job was to find candidates. Everything else — ROI masking, baseline subtraction, temporal vote — was interpretation. Improving the model accuracy from 0.844 to some higher number would not have solved the problem that code solved.
The VoC engine has the same shape. Getting a better topic assignment model would not have surfaced the 152-hour playtime median. That required a different join.
What I changed downstream
Before the playtime join, I was sorting topics by voice count and treating all complaints equally. After, I weight bug-class topics by the median playtime of their reporters. A bug with 40 reporters averaging 200 hours ranks ahead of a bug with 80 reporters averaging 15 hours, even though the second has twice the voice count.
This is not a complex formula. It is a single multiplier applied before sorting. The complexity was deciding what the multiplier should reflect — and the answer was that the product's failure modes mostly manifest after sustained play, so the reporters who have sustained play are the most credible reporters.
The 50h+ share column is a secondary signal. If median playtime is high but the distribution is bimodal — some very long-time players mixed with short sessions — the 50h+ share tells a different story than the median alone. In the bug topic's case, both agree: 152.2 hours median and 74% past 50 hours.
The number I don't have yet
Playtime tells me who is complaining. It doesn't tell me whether the bugs they report are actually present in the current build. The VoC engine collects reviews across a rolling window; some of those bug reports may be for issues that were fixed months ago. Filtering by review date would help, but recent review counts are small enough that the playtime stats would become noisy.
For now, the output is a ranked list of bug topics weighted by reporter investment, not a confirmed bug list. The distinction matters: the output is a triage input, not a ground truth. A developer looking at the 54 veteran bug reports still needs to go read the actual reviews and verify whether the issues are reproducible today.
That caveat should travel with every presentation of this data.
Part of an ongoing 6-month experiment running three AI-curated directory sites. The technical claims here are real; this article was AI-assisted.
Top comments (0)