There's only one source of truth for what was promised in my Sunday night D&D game, and it's my memory of it.
The recording doesn't have it and neither do my notes. Someone said they'd go back for the body, and someone else agreed to keep quiet about the letter, and I don't think either of those was ever said in a sentence you could point at. They were arrived at, over about twenty minutes, mostly sideways.
Why this particular file has to be right
Scout listens to a tabletop session and turns it into structured campaign facts. Who was there, what got promised, what the party now knows that it didn't at teatime.
To find out whether it's any good at that, you need a labelled set: a file saying, for each session, what was really committed to. Then you run the extraction, compare, and get a number.
The number is only worth what the file is worth. And realistically the file can only come from a person who was in the room. In this case that's me, going back through my own campaign transcripts, writing down what actually happened.
That's a slow, tedious, undelegatable job... and I did of course try to delegate it!
The terms I set for the first pass
I asked a model to take a first pass, to save myself the worst of the typing (there's a lot of typing). Then I set the terms.
The first is unremarkable and I'd have said it about anything. I told the session before it started that I definitely wouldn't be accepting any of it wholesale, as this needed to be right and I'm the only person who knows what actually happened in those sessions.
The second is what made the exercise work: where you cannot fill a field, leave it empty, and do not put something plausible there.
A blank is a piece of work with a location attached: I can see it, I know what it's asking me, and I can answer it from memory in about thirty seconds. A confident wrong answer is a piece of work I might never know to do, sitting inside the file every other number depends on.
That's what I meant when I told it a gap is better than a guess. It makes my job easier, and it stops the file lying to me later without my knowing.
A guess in the ground truth ratifies itself
In most places in a system, a model's guess is a bug. It's wrong, something downstream behaves oddly, and with luck a test catches it.
In the reference set it's worse than a bug, because it's self-ratifying. The instrument I built to detect wrong answers now contains one. Extraction comes up with that wrong answer too, the harness marks it correct, and my score goes up. A machine that rewards agreement with a mistake, and it'll carry on doing that for as long as I let it without once looking wrong.
You can't test your way out of a corrupted reference set when the reference set is the test... so here, of all places, I won't take a plausible answer, however tidy it looks.
The extraction side of this is the other end of this programme: three models on one job, reconciled by majority, because a bad write poisons the record. It would be a daft kind of carelessness to take that much trouble over the writes and then score them against something I'd let a model invent.
The same argument, pointed at a schema
The instinct turned up again about an hour later, in a place I wasn't expecting it.
The schema had a boolean for whether something was stated. True or false. So I stopped and typed back at the session that it isn't a boolean at all:
we have true or false but this is not boolean, it may not be cleanly stated but may be implied across multiple utterances, how do we capture that? is there even value in capturing that?
A field that only offers yes and no will get a yes or a no. The labeller needn't be careless for that to happen, those are simply the two options on offer, and a promise assembled out of four half-sentences across twenty minutes has to become one of them.
That's manufacturing certainty in the data model instead of in the answers, and I suspect it's harder to spot later, since it looks like schema design... a person forced to pick a side will pick one just as readily as a machine.
It's slow, and the fix isn't a better model
I wouldn't call any of this elegant. It's laborious, it doesn't scale, and I said at the time that if it turned into a repetitive job I'd end up building a small web interface just to speed the labelling up (I still might).
If I do build it, it'll be a web form... better tooling for me rather than more confidence from the machine.
The part that couldn't be handed over
The fashionable version of this is that AI takes the grunt work off you and frees you for the interesting bit.
What happened here was the reverse. The interesting bit, extraction, reconciliation, the whole measurement instrument, went to the machines quite happily. The grunt work is the only part that couldn't go anywhere. It's the part where being right matters more than being fast, and the only qualified labeller is the person who was at the table when someone agreed to keep quiet about the letter.
Originally published at phpboyscout.uk on 13 November 2026.
Top comments (0)