We built the AI chat in Sybill by hand, with ChatGPT as our reference.
In January 2025 we started building Ask Sybill, the AI chat inside Sybill. Sybill makes AI tools for sales teams, and we wanted the chat to feel like ChatGPT, with the answer appearing word by word.
AI chat frameworks were still young, and we did not use one. Instead, we opened ChatGPT in the browser and watched in the Network tab how an answer arrives: a stream of small events (Server-Sent Events), one after another. Then we built our own version of that stream.
Today Ask Sybill keeps the answer when the connection drops, and you can run many chats at the same time. Getting there took much longer than we planned, and for a long time it felt like there was always one more thing to fix.
Finally, a real use for async generators
At the end of January I wrote the code that reads those events while they arrive. For this I used an async generator: a function that hands out values one at a time and waits in between for more data. Most of us almost never write one. But a stream of events is exactly what it was made for, and it was fun to finally use one in real code.
An async generator reads the stream. The chat just loops over what it hands out.
You may notice that the reader never throws. Early on, I decided that our code would return errors as values, like any other result. So a network error, a message from the backend that we cannot read, and the user pressing Stop all come out of the same loop. The chat can then handle each one in its own way. A stop, for example, should never look like an error.
Every feature became a new event
Each event the reader hands out has a type, and almost every type exists because of a feature in the chat. chat_title gives a new chat its name. task_progress shows the steps while the AI is working. mcp_tool_approval asks you before the AI uses an outside tool. And delta carries the words of the answer, a few at a time.
One answer in Ask Sybill, as a stream of events.
The number of events grew with the product. Each new event meant work on every side: the backend had to send it, the shared types had to describe it, and the frontend had to read it and show it. With a framework, the common events come ready, and you only add the ones that are really yours.
Streaming text and keeping the page fast
Everyone expects an AI chat to write its answer word by word, and we wanted that from the start. But it took a while. Without a framework, we had to turn the stream of each AI model into our own events, and the page had to stay fast while new words arrived many times a second. There was always something more urgent, so we kept putting streaming off. Until then, the full answer appeared at the end.
When we finally turned it on, the text did not feel right. Words do not arrive at a steady speed. Sometimes many arrive at once, then nothing for a moment. Each new piece of text made React draw the chat again (a re-render). Those re-renders were expensive, so the page slowed down, and the text came in jumps. We tried throttling the updates, so React rendered less often, but the text still came in jumps. We tried delaying the stream and typing it out at a steady speed, but then the AI was already done while the page was still typing. What worked was a small buffer between the stream and the screen. It collects the new text and shows it on every frame, about 60 times a second on most screens. That is often enough to look smooth, and the text keeps up with the AI.
There was one more slow part, in how we stored a chat. All the state of a chat lived in one object, so each new word created a new copy of it, and every part of the page that used the chat re-rendered. I moved the streaming text into its own small slice of the store, so now only the message that is being written updates. In a test chat with 15 turns, React spent about 3.3 seconds rendering while one answer streamed in. After the change, it spent about 0.35 seconds.
When chats get huge
Streaming was not our only speed problem. Chats grow, and some get huge. Some people had more than 100 turns in one chat. Opening one froze the page, and the newest messages took about 24 seconds to appear. React drew every message in one go and could not stop in the middle, so until it was done, you only saw the empty start screen.
React has a tool called startTransition. It marks an update as not urgent, so React can stop that work when something more important comes in, like a key press, and start it again after. The React docs show it mostly for async work, like sending a form, and for a slow screen, like switching tabs. Drawing a long list in pieces is not one of their examples, but I used it for that, and it fits well. The newest 10 messages appear at once. Then the older ones are added 10 at a time, each step inside startTransition, until the whole chat is there.
The newest 10 messages first, then 10 older ones at a time, when React is free.
Now the newest messages appear in less than a second instead of about 24 seconds, and the older ones keep loading while the page stays usable.
Every message still ends up on the page, though. Nothing is removed later. For the long chats we tested, that worked well. If chats grow much larger, you will also need virtualization: the page draws only the messages near the screen, and adds and removes the others as you scroll.
Native popovers to the rescue
Huge chats hid one more cost, in a place nobody looks: the tooltips. Each message has a row of buttons, like copy, feedback, and retry, and each button has a tooltip. A long chat has hundreds of them. With many component libraries, every tooltip lives in React, even while it is hidden. So React spends time on all of them, even though you see at most one. When you hover a button, showing its tooltip goes through React too, and JavaScript places it by measuring the button and the page. The library we used worked like that. In a test chat with 124 turns, hovering the buttons 30 times blocked the page for about 11.5 seconds in total, almost 0.4 seconds per hover.
Browsers can now do this job themselves. The Popover API puts the tooltip on top of the page, and CSS anchor positioning places it next to its button. I rewrote our tooltip to use these two and moved its hover logic to plain JavaScript, outside React. So the hidden tooltips stay cheap, and showing one makes React draw only that tooltip. A JavaScript library still places the tooltip in a few cases, like when it does not fit on the screen or the browser is too old.
The browser shows the tooltip on top of the page and places it next to its button.
In the same test, the 30 hovers now block the page for about 2.5 seconds instead of 11.5.
Keeping the answer when the connection drops
Sometimes an answer got lost halfway: the connection dropped, or someone reloaded the page while the AI was still writing. We wanted the answer to survive both.
So the backend saves every event of an answer in Redis Streams, a Redis feature for ordered messages. Each event gets an id. The browser remembers the id of the last event it received. When the connection comes back, it asks for everything after that id, and the answer goes on from where it stopped. If you open a chat while its answer is still being written, the page joins that answer in the same way.
When the connection drops, the browser asks for the rest.
Resume alone was not enough, though. At first, the connection to the browser kept the answer alive: when it closed, the answer from the AI model stopped too. That happened when you closed the page, but also when the Wi-Fi dropped. Now the backend reads the answer from the model on its own. It sends each event to the open connection and also saves it in Redis. If the connection closes, the backend keeps going and keeps saving, and the browser can come back for the rest.
One thing surprised us. Sometimes the AI thinks or works for a long time without sending anything. This is a known weak spot of Server-Sent Events: a proxy, a server between the browser and our backend, may close a connection that stays quiet for too long. So while the backend has nothing else to say, it sends a heartbeat every 15 seconds, and the connection stays open.
What I would choose today
All of this works well. But if I built a chat today, I would not write the stream myself. AI chat frameworks like Vercel's AI SDK are common now, and they already solve much of what we built by hand, like streaming text, stop, and tool approval.
Looking back, I think we should have moved to a framework once we understood the problem well. That is easy to say now. At the time, the chat was already live, it had features of its own, and moving would have paused product work for a while. Building it ourselves also taught us how every part of a chat works.
We started by opening ChatGPT and watching its answers arrive. If you start today, open a framework's docs first. Most of what took us so long is already in there.
Top comments (0)