DEV Community: John Polacek

Turning a Chat Thread into a Product

John Polacek — Tue, 11 Aug 2026 13:06:00 +0000

ShowReco started as a chat thread. I was trying to find movies and shows to watch that weren't just the same old junk that streaming platforms kept recommending. It worked really well, you can see the chat here for yourself.

The problem with a chat thread is it eventually runs out of context. It forgets everything I told it. All the ratings, and the nuance of everything we discussed gets lost.

So I built the chat thread into a product.

The Taste Profile

I worked on the memory system first. Every 5 ratings the app re-reads everything you've rated and rewrites a profile document: one sentence that sums you up, the ways you differ from other fans of the same genres, your lanes, your turn-offs, your dealbreakers.

The model rewrites its own memory, but there are guards on it. It snapshots the old profile before every rewrite. It can add nuance but it can't delete an established lane on little evidence.

The first real bug was a queue of 29 dramas. I had rated about nine things, most of them drama, so the profile over prioritized. The generator was told to ground every pick in the profile, so it read "not in the profile" as "exclude everything not in this narrow lane for this user".

The adjustment was to have the profile assess its own confidence now and enforce that a genre you were never shown is unknown, not rejected. The model is encouraged to ask for ratings on content where it lacks info.

Asking Better Questions

Getting the app to ask me things to figure out my taste was the key. Most people like The Bear, so a good rating tells you almost nothing. On ShowReco, every rating is lined up against the IMDb score instead, and when the gap hits 1.5 the app opens a conversation about why. That's the good stuff.

There are still always going to be gotchas. A real user (me) had rated five horror titles, but somehow with horror appearing nowhere in my profile. Every rating had been filed under a lane with no genre in the name, the lane list was full, yada yada. The bug was that it was never going to ask about or recommend a horror movie or show because of this.

So we did a second probe. It finds a genre your ratings are loud about and your profile is silent about, has the model write the question, picks titles to settle it, and requires the next profile rewrite to answer.

Working With the Agent

I built almost all of ShowReco with a coding agent, much of it on my phone, shipped to production, and tested it myself. A tight feedback loop that was both fun and effective.

A model's training data is older than your project. It will write confidently against the version of a library it learned instead of the one you installed. A rule at the top of my AGENTS.md is to read the real docs for the actual version before writing any code. That one line takes out a whole category of confident wrong answers.

For anything bigger than a small change we write a plan into a markdown first, starting with the real data from production, and I read the plan instead of the diff.

Use the smartest model you have as the orchestrator and let it hand building to others. Fable holds the plan and reviews what comes back, and it spins up Opus and Sonnet subagents to do the implementation.

Two things I never had to ask for. It comments as it goes, and the comments are nearly all why rather than what, which matters because the next session has no memory of the evening that produced some odd looking guard clause. It also writes tests (there are 806 of them now).

Simple Auth and Users

Just use Clerk. Don't overthink.

Flat-File Databases

There is no database. Every user is a folder of JSON files in S3. Again, keep it simple. If you don't need realtime, store data in S3. Cheap. Reliable. Fast.

For ShowReco, reading your taste profile is one GET. Writing it is one PUT, conditional on the S3 ETag (so if two things hit the same file at once the loser retries instead of overwriting).

This works because everything is scoped to one person. Nothing in the app queries across users. If I needed leaderboards or search this would be a bad idea. Then I'd probably reach for Convex.

Tracking Every Token

Every model call writes its token counts to a usage file, split by day, by operation, and by model. Chat, queue rebuild, profile distill, calibration, web search, all counted separately.

I put that in early, before I needed it. A great decision. When it comes to price, don't guess. Track everything.

Knowing cost per operation can shape a free tier around real numbers. For ShowReco, rating is unlimited because it is virtually free. Queue rebuilds run on the cheap model until a profile has enough signal to be worth the expensive one, which is also what makes every new signup cheap to serve (my first seven signups cost between 0.8 and 7.7 cents each, median about 2 cents).

Admin Panel

I have admin pages (protected routes just for me) with analytics, funnel, retention, attribution, finance, user activity, etc. There is nothing about this app I can't go look up and figure out how to make better.

I would never have hand built all these admin pages for a product with so few users, but it's just so damn easy and useful.

Will ShowReco grow? I don't know. I built an app to serve my own needs and that was fun. Not too worried about it! I am shipping features to make it better all the time.

Give it a try at showreco.com and tell me what it gets wrong about you!

Training an AI GameMaster to run D20 Adventures

John Polacek — Wed, 04 Jun 2025 13:25:44 +0000

I've been thinking about making a multiplayer RPG web platform since my days as a Flash dev. There are plenty of online TTRPG options out there (many great!) but none of them felt quite right for how I'd like to play.

Have you seen the video where Deborah Ann Woll shows Jon Bernthal how to play D&D? (it has 2M views!)

That was so cool. What if I could make something that feels like that?

The Idea

A narrative-driven, multiplayer online rpg platform, light on rules but high on shared storytelling, with outcomes determined by the roll of a D20.

I've always loved D&D modules. The art. The exciting introduction. The detailed encounters and epic story plans for the players to uncover as their characters make their way through to a climactic finale.

What if I could create a way for people to make their own adventure modules, for any RPG genre, then let players run their characters through those adventures?

Who would run the games though? Game masters are hard to find. What if I could train an AI to be a game master? To actually run a well structured adventure plan (designed by a human!) that would be fun for players and not just a bunch of AI slop?

Getting Started

I've been having fun building lots of things with AI (see my project starter). I'll be using my preferred stack to make D20Adventures.com, a Next.js web app with Tailwind for UI (need it even be said?) powered by the AI SDK utilizing Gemini and a Convex database deployed to Vercel.

And I've decided to build it in the open, publishing the code on Github.

The Prototype

For a prototype, I'm literally starting from the scenario laid out on the podcast, a ranger walking through the woods at night hears a crack in the distance, which turns out to be an owlbear.

My aim is to build a short one-shot adventure and see if I can train an AI DM to actually run a small series of turns and encounters for this adventure. I fleshed out the scenario a bit with a backstory:

The Midnight Summons
A mysterious summons from an old druid friend draws a reclusive ranger into the wilds of the Valkarr forest.
After a lot of trial and error, I was finally able to do a full play through and published it to YouTube:

How It Works

Landing Page

The landing page is a big hero image (generated on Midjourney with a prompt of “The Power of the D20”). I added some simple fade in animation using the new CSS @starting-style rule:

.fade-in {
  @apply opacity-100 transition-opacity duration-1000 ease-in-out;
  @starting-style {
    opacity: 0;
  }
}

Authentication

To play the Quick Start, I require a user account. This is to avoid getting charged lots of money because of anonymous people or bots using my APIs. Clerk makes it very simple to add user management, and I use them on all my projects.

Additionally, I have token usage tracking where I will limit usage with a token system, where you start out with enough tokens to do one play through of the demo, then can buy more as you go.

The First Turn

When the user lands on the adventure page for the quick start, the first thing that happens is we load the data for the demo adventure, which is just a simple JSON file (like this) on S3. An adventure module or plan in my system is made up of a series of encounters, that are linked to each other with instructions for the LLM:

"encounters": [
  {
    "id": "broken-silence",
    "title": "Broken Silence",
    "intro": "Thalbern, a solitary ranger of the Valkarr woods, has always trusted the silence of the wilds more than the promises of men. Orphaned by border raiders and raised by the elves of the Valkrarr Forest, he has spent years living on the edge of Kordavos, guiding travelers, hunting for his own survival, and keeping his distance from the tangled politics of the city.\n\nYet on this night, a message delivered by a red squirrel bearing the unmistakable script of Wollandora, a trusted elven friend and druid, has drawn him from his hidden home. The note was simple and urgent: Meet me at the Old Standing Stones at midnight. The balance of the forest could depend on it.\n\nNow, as midnight approaches, Thalbern moves quietly through the dense undergrowth, guided by memory and instinct. It is dark with almost no moonlight coming through the forest canopy.\n\nSuddenly, the hush of the night is broken by a sharp crack. Something large has just stepped on a branch somewhere off in the distance.",
    "instructions": "A perception check is appropriate if Thalbern investigates (low difficulty with a plus 3 modifier). If successful, he will determine it is a large creature that is approaching quickly. With a high roll (18+), he will determine it is an Owlbear. If combat ensues and Thalbern is below 25% health, Wollandora will intervene. If Thalbern avoids or defeats the Owlbear, or if Wollandora saves him, he proceeds to the Old Standing Stones.",
    "image": "images/settings/realm-of-myr/the-midnight-summons/broken-silence-2.png",
    "transitions": [
      {
        "condition": "If Thalbern successfully uses stealth to evade and proceeds cautiously towards the Standing Stones, go to meeting-at-stones.",
        "encounter": "meeting-at-stones"
      },
      {
        "condition": "If Thalbern fails a perception check, advance to owlbear-confrontation.",
        "encounter": "owlbear-confrontation"
      },
      {
        "condition": "If Thalbern fails any dice roll (including stealth, perception, or any other check), advance to owlbear-confrontation.",
        "encounter": "owlbear-confrontation"
      },
      {
        "condition": "If Thalbern does NOT successfully use stealth to evade, go to owlbear-confrontation.",
        "encounter": "owlbear-confrontation"
      },
      {
        "condition": "If Thalbern does nothing or takes no action, go to owlbear-confrontation.",
        "encounter": "owlbear-confrontation"
      },
      {
        "condition": "If Thalbern has a healthPercent of less than 50%, go to wollandora-intervention.",
        "encounter": "wollandora-intervention"
      }
    ]
  },
  {
    "id": "owlbear-confrontation",
    "title": "Owlbear Confrontation",
    "intro": "From the direction of the sound, a little bit of eye shine glints in the shadows of the tree line. A hulking fifteen foot tall monster with the body of a giant bear and the head of an owl. As it crashes out from the undergrowth, it lets out a guttural squawk, clearly agitated and territorial.",
    "instructions": "The Owlbear will attack. If Thalbern attempts an animal handling check (high difficulty) and succeeds, he can move past the Owlbear. If Thalbern wins initiative and attempts to hide, he can move past the Owlbear if he passes a medium difficulty stealth check. If Thalbern's health drops to a critical level, Wollandora appears and drives off the Owlbear, transitioning to 'wollandora-intervention'. If Thalbern defeats the Owlbear, describe his victory and transition to 'meeting-at-stones'.",
    "image": "images/settings/realm-of-myr/the-midnight-summons/owlbear-confrontation.png",
    "npc": [
      {
        "id": "owlbear",
        "behavior": "Aggressively attacks any perceived threat. Will fight until heavily wounded or driven off.",
        "initialInitiative": 1
      }
    ],
    "transitions": [
      {
        "condition": "Thalbern defeats the Owlbear, manages to evade it, successfully uses Animal Handling to pacify and move past it, or successfully rolls any other way to move past it.",
        "encounter": "meeting-at-stones"
      },
      {
        "condition": "Thalbern is reduced to critical health by the Owlbear.",
        "encounter": "timely-rescue"
      }
    ]
  },
  ...
]

The First Reply

Since we are in demo mode, there is no actual adventure created yet on the backend. That happens when the player makes the first reply.
With every reply, we run it though a formatNarrativeAction function that uses AI to evaluate the reply to make sure it is grammatically correct, in the third person and adds dialogue or other prose to make it work well in a literary narrative style.

After the formatting is applied, the reply is sent to a server action. Because this is a demo, we are creating the adventure in the database when this first reply is received. After this is done, it will follow the normal flow for processing a player reply.

Processing Player Responses

The processTurnReply function loads the current turn from Convex, the adventure data from S3, and identifies the specific encounter and the character performing the action.

With this context, it then uses AI to determine whether the action is plausible (my son when play testing had the ranger launch a nuke at the owlbear) and, if so, whether a dice roll is mechanically required (e.g., an "Attack Roll" or a "Stealth Check"), including the type of roll and its difficulty.

We can do this with a function call, where we can specify to the AI that we want structured data returned, in this case a rollRequirementSchema:

import { z } from "zod";

export const rollRequirementSchema = z.union([
  z.object({
    rollType: z.string().describe("The type of roll required, e.g. 'Stealth Check'"),
    difficulty: z.number().describe("The difficulty class (DC) for the roll"),
    modifier: z.number().optional().describe("Bonus or penalty to the roll, e.g. +2 or -1"),
  }),
  z.null()
]);

export type RollRequirement = z.infer<typeof rollRequirementSchema>;

Then we have a function that sends a detailed prompt and the schema to generateObject:

export async function getRollRequirementForAction(action: string) {
  const prompt = `
Given the following player or NPC action, determine if a dice roll is required for the character to attempt the action. If a roll is required, return a JSON object with "rollType" (choose the most appropriate from the list below) and "difficulty" (a number between 5 and 25). If no roll is required, return the JSON value null (not a string).

Possible roll types:
- Perception Check
- Investigation Check
- Insight Check
- Stealth Check
...

Examples:
Action: "Try to sneak past the guards."
Result: { "rollType": "Stealth Check", "difficulty": 15 }

Action: "Attack the goblin."
Result: { "rollType": "Attack Roll", "difficulty": 12 }

Action: "Try to determine what the sound is."
Result: { "rollType": "Perception Check", "difficulty": 10 }

Action: "Say hello."
Result: null

Now, given the following action, determine the roll requirement.

Action: "${action}"
`;
  try {
    const result = await generateObject({
      schema: rollRequirementSchema,
      prompt,
    });
    if (
      result.object &&
      typeof result.object === "object" &&
      "rollType" in result.object &&
      (result.object.rollType === "null" || result.object.rollType === "none" || result.object.rollType === "")
    ) {
      return null;
    }
    return result.object ?? null;
  } catch (error) {
    throw error;
  }
}

After appending the the player’s response to the turn's story, if a dice roll is required we update the character's state in the Convex turn data to reflect that they've replied but their turn isn't complete. This update causes a realtime update to the UI showing the details of the roll and give the player a button to roll a D20.

Once the user rolls, we have a resolvePlayerRollResult server action that updates the narrative with a visual display of the result and writes prose describing the outcome. We also have another AI function call to update health and status for all the characters in the turn.

NPC Actions

When it is an NPCs turn to reply, we follow a similar pattern as a player, except in this case the AI will write the response. Every encounter includes information about the NPCs motivation and when combined with the context of the adventure narrative so far, hopefully the AI can generate a good reply, then generate its own dice roll and outcome update.

Training AI to Run RPGs

If you've ever tried to do a game session with AI in a chat, you know how quickly it can go off the rails.

Hopefully by providing the appropriate context, specific prompt instructions and structured data function calls, we can get an experience that is enjoyable.

There is definitely a lot of trial and error, console logging and prompt tweaking. For example, when asking why the AI did not follow the encounter instructions in one of my test runs, this is what Cursor chat had to say to me:

In summary, the LLM did not follow its explicit instructions to transition when a character fails a perception check, despite being provided with clear evidence of such a failure and a transition rule for that specific scenario. This seems to be a misapplication of the provided logic by the LLM.

I guess the randomness of the AI not doing exactly what is expected could be part of the fun. I suppose we'll find out!

Here is an example of one of the play sessions.

You can see the full source code for this project at github.com/johnpolacek/d20adventures.com