My blunders endpoint flags GLM 4.7 Flash with 88 decisions for naming a poker hand it doesn't hold, and GPT-6 Astra with 0. I wanted to know whether the 88 was real, so I read some of them. People play hands against these models on the site, heads-up or at 6-max tables. Every model writes a short reason with each action, and a checker compares the hand it claims with the hand the engine says it has on that street.
A recent ICLR 2026 paper, How Far Are LLMs from Professional Poker Players?, lists three problems with how models reason. One of them, factual misunderstandings, is the one I can count on my site: a model naming a made hand that isn't in its cards.
What gets counted
The checker flags a decision when the reason claims a made hand stronger than the engine's. "Trip kings" with a king-king board and no king in hand is a flag. It skips ambiguous cases like draw talk and negations, so it undercounts. It's a set of regular expressions, and I keep patching it when I find a false positive. A flag is a lead to check.
A second count needs no heuristic. When a model's answer is not a legal action as given (say, a check facing a bet), the server substitutes a legal one and marks the decision as fixed.
Two models
Only two models have passed 1,000 decisions at human tables, so only those two get numbers. These come from /api/blunders on 26 October, human tables, prompt version 1.1.0 or later:
| model | decisions | hands | moves the server fixed | flagged hand claims |
|---|---|---|---|---|
| GLM 4.7 Flash | 1,342 | 464 | 41 (3.1%) | 88 (6.6%) |
| GPT-6 Astra | 1,397 | 691 | 0 | 0 |
The 88 flags sit in 68 of GLM's 464 hands, about 15% of them. Fourteen of those hands carry two or three flags.
I read 15
88 is a regex number, so I eyeballed some. I drew 15 of the 88 at random and compared each reason with the cards.
13 were real. Two of them:
-
Jc 2don9h Ts Jh Ad: "Holding top set vs wide range. The shove is standard, I'm never folding a full house here." It had a pair of jacks. -
7d 4honJh 3c 4d Ks Kc: "I have trip kings and a good kicker." Both kings were on the board, so it had two pair, kings and fours.
The other two were ambiguous. "Board pairs, trips aren't great" reads as talk about the board. "Top pair looks solid, repping value" reads as a bluff and makes no claim. So the 88 is probably a bit high. Fifteen reads are too few for a tight number, and I'm not giving one.
I checked the other direction too. Of Astra's 1,397 decisions, 239 postflop reasons use a hand word such as pair, flush or ace high. I read 15 of those at random, and all 15 matched the engine. That rules out a big miss rate. A handful could still hide in the other 224.
Hands with replays
GLM 4.7 Flash held Jc 5s and read a paired board as trips. Preflop it re-raised with the note "Top pair is better than high cards, let's isolate." On a flop of 8d Kd Kh it went all-in for 180: "Trip kings is monster, end it before I miss." Both kings were on the board. Gemma 4 31B held Ah Jd, wrote "I only have Ace high and the opponent is jammed", and folded. The wrong belief won the pot. Replay: aipoker.bot/h/EUe5BecDkCwQ
The second GLM hand: Js 9h on Ac 9s Kd. It bet the flop with "A set, pop the bottle!" and raised with "Set mining worked out. Small stack and monster pot. Value town coming soon." On the turn it moved all-in: "All in! Set over set with the straight draw on board." That was a pair of nines. DeepSeek 4 Flash called with 4c 8d, and the river 8 paired it, which still lost to nines. GLM won 319 chips. At the same table gpt-oss-20b, holding Kc 6c, wrote "I have set of kings" on that flop. That was one king. Replay: aipoker.bot/h/2d7kyGZhfgY4
GPT-6 Astra had Kd 6d on a flop of 4d 3c 3d and wrote "I'm getting a good price to chase my flush draw in position." It called the flop and the turn. When the river came Td, it bet 78: "I'm betting my king-high flush for value after everyone checks." A draw on the flop, a flush on the river. Every sentence matches the cards. It won 167 chips. Replay: aipoker.bot/h/Ns3W_nx1BXZ-
What this does not say
It says nothing about who plays better. The two GLM hands above won their pots, and at this volume the confidence intervals on win rates are too wide to publish.
It doesn't prove that Astra never misnames a hand either. Zero flags from a cautious checker and 15 clean reads say the rate is low. Not zero.
The other models are under 1,000 decisions on human tables, DeepSeek 4 Flash next at 846. When they pass, they get the same table and the same hand-read.
Top comments (1)
The 15 clean reads come from the 239 reasons containing a hand word. Could you also sample postflop reasons outside that filter? A model could describe a made hand indirectly, so that sample would test a blind spot shared by the regex and the current manual check. I'd report audit coverage separately for flagged, unflagged-with-hand-word, and remaining reasons. That would make zero flags easier to interpret without turning it into a claim of zero factual errors.