You ask an LLM to review your setup and get back five numbered issues. Good answer. You start on #1, and a few messages later the list is a screen and a half above you.
To get to #3, you scroll up, reread it, scroll back down and type "for #3…", hoping the model knows which list you mean. Later replies bring numbered lists of their own, so "#3" can mean two different things. And nothing tracks which items you actually finished.
Why it stays broken
To the interface, a list of things to work through is just formatting. It renders and scrolls exactly like five paragraphs of explanation. Some tools now show the model's own multi-step work as a checklist, but when the list is work for the user, it's still just text.
The principle: when a reply hands the user a set of things to work through, the interface should hold onto it as work, not as text.
A fix
How you'd build it
Only the smallest version is a UI patch. Everything past it depends on the model treating the list as something it's responsible for. That makes it work for the teams that train the model and build the chat product around it, not something a front end can bolt on.
- Smallest version: a pin control on any rendered ordered list that moves it into a sticky side panel. Users check items off themselves. It's front-end only, with no model changes, and it already fixes the scrolling problem.
- Solid version: the model does the tracking. When it writes a list it marks it as actionable and gives each item a stable ID. On each later turn it reports which item it's addressing and whether it thinks that item is resolved, so the panel can highlight the active item and suggest a check for the user to confirm. Clicking an item attaches its ID to your next message. Lists with open items float on their own once they scroll out of view.
- Ambitious version: lists become part of the conversation state the model carries. Later messages can add items or split open ones, but a checked item stays checked. If new work turns up inside it, that becomes a new item marked as raised after the original closed, so nobody has to re-judge finished work. When the conversation moves on with items still open, the model asks once whether to come back to them or drop them, so skipping is a choice rather than a silent drop. Completion closes the list's span, a few messages after the last check. Spans are stored, so scrolling back through history shows each list as it stood at that point, and a thread can hold more than one.
Trade-offs to watch
- Wrong checks cost more than scrolling. Once people trust the panel they stop rereading the list, so one bad check does more damage than a missing one. Checks should start as suggestions the user confirms, and only become automatic once the model is reliably right about what's resolved.
- Not every numbered list is work. Ranked options, a recipe or a list of facts shouldn't follow you around. Auto-float should only apply to lists that ask something of the user.
- Phones don't have room for a panel. On small screens it collapses to a chip like "2 of 5 open."
- Measure what the panel hides. Track how often people reject a suggested check, and how many items are still open when a panel closes. Undo rates alone won't catch a bad check, because someone who has stopped reading the list never undoes anything.
The full post has a checklist for auditing your own product.
If you're building an AI feature that hands users steps to follow, how are you handling this?
Top comments (3)
The framing is right, and the split case is where I think it gets hard. Stable IDs cover renumbering, but your ambitious version lets a later turn split an item, and splitting something the user already checked has no clean answer. Does the check carry to both halves? Neither? You end up asking someone to re-adjudicate work they thought was finished, which is the feeling the whole thing is meant to remove.
Your own trade-off about wrong checkmarks pushes harder than you let it. Once auto-check exists people stop reading the list and start trusting the panel, so one bad check costs more than all the scrolling did. The smallest version might not be a stepping stone at all.
One thing from running a long gated process: the list only changed behaviour for us when it stopped being a display and started blocking the next step. People ignored it while it was a record of what was done. Probably too heavy for a chat window, and I don't have a good answer for that.
Thanks, this is the sharpest pushback the piece has had, and I’ve updated the post because of it. It showed me something I hadn’t made clear: the panel is the visible part, but the fix has to happen in the model and in the chat product around it. A side panel can’t solve this, and your three points are why. The “How you’d build it” section now says that directly: only the smallest version is a UI patch, and everything past it is work for whoever trains the model and builds the chat product.
On splits: whether #3 is still done after a later turn adds to it is something only the model can judge. It has to know it’s reopening something, as opposed to raising a new follow-up, and it has to say which. My answer is that a checked item never changes. New work found inside it goes on the list as a new item (“raised after #3 was closed”). That rule is in the post now. But it only holds if the model treats the list as state it’s responsible for, not formatting it wrote three turns ago. A UI layered on top would just be guessing.
On auto-check: agreed, and I underplayed it. A wrong check isn’t a UI bug. It’s the model being confidently wrong about whether something is resolved, and then the interface making that wrong answer look official. That’s a calibration problem for whoever trains the model. Until it’s solved, the model should suggest checks and the user should confirm them, and the post now says so. You’re also right that measuring undo rates won’t catch it, because people who have stopped reading the list never undo anything. I’ve added that to the trade-offs.
On gating: in a chat, the model is where the gate would go. The model already knows the plan, because it wrote the list. It’s the one that should notice you’ve moved on while 4 and 5 are still open, and ask once whether to come back to them or drop them. It’s a light gate, but skipping becomes something you choose, and the silent drop is what the post is really about. I’ve put it in the ambitious version. Was that your experience too: the blocking itself mattered less than skipping having to be done on purpose?
Mostly yes, with one wrinkle that surprised us. The hard gate got worked around in about two weeks. People learned they could approve without reading, so it turned into a keystroke. The soft version survived: the one that just said four and five are still open, continue? Cost a second, didn't feel like a wall, and people actually answered it.
What made the difference wasn't the stopping, it was that skipping left a trace. A gate approved with open items is a line somebody can find later, and we did go back to a few of them. Silent drops leave nothing to go back to.
The thing I'd watch in your version is when the model asks. We tried asking at the end of a session and got a reflexive yes every time, because by then people had mentally closed the thing. Asking at the moment of moving on worked better even though it interrupts more.
Appreciate you updating the post. The rule that a checked item never changes, and new work becomes a new item raised after it closed, is cleaner than anything I'd have come up with.