This article is a result of almost 2 years of experimenting. Yeah, it took me some... It will be in two parts: this one a bit on the technical side...
For further actions, you may consider blocking this person and/or reporting abuse
I came here to make that obvious, sophmoric comment, not expecting much from the article, but started reading ...
Fascinating stuff! It just kept evolving. Thanks for sharing this!
Yeah, this is why it took so much time from me. A simple project appeared to be not that simple. Thank you for reading.
Interesting, I am just surprised that an active user can cost around $5 per day. I would expect more, because even with compaction and caching, the total context and number of model calls must still grow quite a lot during a game. I saw that you track the cost of each request. Did you use this metric to optimize the costs?
That's a great question. I sent Simona to grab the some statistics from the database.
Per game (67 games currently stored, all with cost > 0; I age out games after no activity for 30 days):
Per user, all-time (231 users, 179 ever spent anything):
I have 7 users who spent more than $3. So, not really a problem at the moment.
How I optimize. I decide how much free users can have based on the statistics. Today, it's quite a lot, I only restrict the most expensive models like Fable, Sol, Opus, K3, Fugu Ultra. Here is the breakdown. I don't restrict any features like voice or ability to delete messages and basically replay a day or a night again and again.
When (and if) free users become a problem, I'll have to rethink the free tier.
Another comment here pointed to suboptimal caching - I can improve that. However, not sure it will save me much. Games are slow, so the cache usually expires faster than a new request hits it. Unless it's a 1h cache, but it's usually more expensive. I'm researching this. I'll probably make another post just about that. Or maybe caching + cost of running this app.
How's the cache performance? Putting the game state in the system prompt would mean cache is busted every turn?
We implemented "Progress Report" user messages after certain checkpoints in our app and now keep the system prompt static for better cache behavior.
Wow, that's a good question. My system prompt stays static through the whole game day and the night phase, so caching works fine there. When it's time to begin a new day, the bots summarize the past events and the alive/dead player list in the system prompt gets updated. That's where their cache gets busted. Not too bad, because it's basically a compaction event.
But I've been thinking about it and I'll optimize that. I can move the alive/dead list into the
reminder, and keep the past day summaries strictly append-only. That leaves the system prompt and the summaries as a static prefix. Thanks for the idea.
My AI also discovered we're not setting cache breakpoints for the Claude models at all - so no
caching there whatsoever. Something to fix as well.
The section on the JSON schema is the point I would emphasize to anyone developing agents, and your presentation of it is better than mine. You describe what happens in case of failure: a vague hallucination becomes a validation error, and a validation error is something you can retry. It’s the same mechanism, but your explanation shows why it’s worth tackling.
The router does something I don’t have. Selecting two to five responders instead of a broadcast limits the spread at the source. You solved this structurally, at the level of the conversation’s form, rather than with a downstream counter. That’s the counter I was looking for. 🙏
Here’s why I realized this. My system refuses to act unless two independent endpoints in the chain agree on the chain’s state : different operators, different companies, a quorum recorded in a hash-chained log. Last week, I finally asked what this agreement entailed: both endpoints were running the same execution client, of the same version, right down to the commit hash. Every line in the log was accurate, the quorum was two out of two, there were no discrepancies, and yet this didn’t certify anything, because the field that would have indicated that these two witnesses were using the same software didn’t exist. This week, in another discussion thread, someone observed the same configuration in fourteen tags from two sources they believed to be distinct, both of which traced back to their own judgment.
You have the property I had to add, and it’s located where it’s most useful: a human can see it without having to ask any questions. If you ever want the "paranoid" version, the only thing I’d log alongside each vote wouldn’t just be which model cast it, but what that model replied when asked to identify itself at that moment : providers hide version information behind stable names, and a list that was heterogeneous when you wrote it can quietly cease to be so. This is the flaw I couldn’t see until I asked the question, and my version had a version string to query. 🫴
That's interesting, thanks for sharing.
In my case, every bot is prompted to be a character, so asking one to identify itself gets you "Frodo" despite the model.
I don't think the witness analogy fits my situation though. My models are not certifying anything by agreeing, they are actors, and the personality lives in the prompt rather than the weights. I can swap a model mid-game and the character carries on as the same character. A new model will bring in a new style and its replies will be different from the previous one, but it's fine. The identity will survive that. It runs the other way too: nine companies, different weights, and they all still pile onto the same lynch target until one line in the reminder tells them to question mob consensus. Which model answered does not decide how the game plays.
I've spent a lot of time on the same context-and-attention games, and I recognize it as an art.
The reminder at the end - I needed the same thing. The facts pile up in the timeline as they happen and stay there. And the signals and the focus - I keep that in the tail, where the model always looks.
The play-style "why" - the motivation, not just the rule - a powerful insight which it greatly improved my own level of attention control too. Sharing the setup the model is acting in, and giving it the meaning behind what it does - described, high-level, not a list of wants that would flatten its own reasoning. That's what lets it generalize where no rule spells things out.
And running it on 20-something models - that's huge 🔥 it proves generalization across different training strategies.
The timeline-and-tail split is exactly it, and I would add one mechanism: the reminder is never written to the history at all. It exists only in the last message of each request and gets rebuilt every turn. So the timeline accumulates and the tail cannot, which is what stops the focus block from turning into more sediment.
On the play-style "why", agreed, and the thing that made it land for me was writing each style twice. Once for when that character is a villager, once for when the same character is a wolf. Same personality, opposite reason for doing it. That doubling did more than any amount of behavioral detail.
The 20-model part I would temper though. It generalizes less cleanly than it looks from outside. Small models still fail the structured picks, and play styles decay: a protective character holds for 30 or 40 messages and then quietly rejoins the mob. Motivation buys you more than a rule does, but it has a half-life.
i did not finish yet im busy but i will read the rest at home verry intersting
Thank you. That's one of the best projects I've even done. I didn't plan making this article so long but yeah... looks like I've collected a lot of things to share.