We built an AI assistant for our support team so that agents wouldn't have to answer the same questions day after day. Users write to us on Telegram and Discord, and the team handles those conversations in Chatwoot. At first, the assistant left a suggestion that only the agent could see. Later, we let it answer routine questions on its own and hand the conversation over to a person when needed. I wrote about that transition in part one of this series.
A support agent has the whole conversation on screen. When a user replies, “I've already tried that,” the agent can glance up a line to see what they suggested. A language model has no conversation window. On each call it sees only what our service put in the request. Give it just the latest message, and it may politely ask someone to repeat the very thing they just did.
Here's how we put that context together: which messages we take from the conversation, what we add from the product documentation, and where we tell the assistant when to call in a human agent. We'll follow an ordinary text request from someone whose email with a sign-in code hasn't arrived. Messages with attachments have additional sources of context; we won't need them here.
Two lines that change the answer
Imagine this exchange. A user writes, “I'm not getting the sign-in code by email.” The assistant replies, “Have you checked your spam folder?” Now comes our line: “I've already tried that.”
Nothing about the product or the sign-in instructions has changed. But we do know something useful now: someone has already suggested checking spam, and the user has responded. We still don't know whether the email reached the mail server. If the assistant writes, “We checked and the email was delivered,” it has just added a check that nobody performed.
Our service gets the conversation history from Chatwoot. It selects incoming user messages and public outgoing replies. If the last entry is a user message and its text matches the one we're processing, the service removes it from the end of the history: the current message goes into the request separately. It then takes the last few messages. The default is six messages, not six question-and-answer pairs.
Six is a setting. There's no guarantee that everything that matters will fit. Imagine a long exchange about a different sign-in attempt slipping in between the spam question and the user's latest message. The connection is lost. It can also disappear if the request for history from Chatwoot fails: the function returns an empty list. Processing continues, so the model can receive the current message without its backstory. When reviewing suggestions, I like a simple test: if I can't tell from these lines what the person tried, the assistant has no grounds for talking as though it can. One short clarifying question would serve the user better.
The part of a conversation the user never sees
An agent looking at a support ticket sees more than the user does. Suppose a private note sits between two public messages: “The emails might be delayed. Worth checking the queue.” That's a useful lead for the next person on duty. For a model preparing a reply, there are two traps: nobody has checked the queue, and the user has never seen the hypothesis.
That's why we leave private outgoing notes out of the history. Early on, those notes included suggestions from the assistant itself. Imagine that yesterday it suggested checking spam in a note, but the agent sent a completely different reply. Feed both entries to the model today as conversation history and it might ask, “Did yesterday's suggestion work?” The user will be looking for advice they never received.
There's another subtlety. An agent's public reply goes into the history with the role assistant. The label identifies the support side of the conversation. It doesn't prove that a model wrote the reply. What matters for continuity is preserving what the user was actually told, whoever wrote it. The service does check whether the bot should continue the conversation before it calls the model. In live mode, if it sees that a human agent has already replied, the bot steps aside. That's why the assistant itself asked about spam in our example.
The documentation knows how the product works. It doesn't know what happened to the email.
Back to our sign-in code. Suppose the help page says that when someone requests another code, only the latest one is valid. That's a good reason to ask whether the user requested several emails. But even a perfectly written help page can't tell you how many times this particular person pressed the button. It describes how the product works; it isn't an event log for their account.
We use Confluence for this knowledge. The service starts from a configured page and gathers material from its page tree, from linked pages, and from relevant attachments on the root page. This path doesn't run a separate semantic search for the passage closest to the question. It uses the assembled knowledge base text to build the request. When investigating a bad answer, I'd start with a fairly ordinary check: what exactly did the version of the instructions that reached the model say?
“Version” matters here. The service keeps the downloaded text in process memory. By default, it uses it for an hour before trying to refresh it. If Confluence is completely unavailable at that point, the old text can remain in use. One hour tells us when to check for an update; it doesn't set a maximum age for the instructions. That explains a familiar situation: an agent is reading the corrected page while the assistant is still giving yesterday's advice. Both consulted the documentation, just at different times. If the service can't fetch the knowledge base and has no cached copy, live mode hands the request to a human agent without calling the model. In shadow mode, it leaves the team a private note that the knowledge base is unavailable.
Who can take the next step?
In our example, someone still needs to find out what happened to the email. Suppose that requires an internal delivery log that only an agent can access. “Check the delivery log” makes perfect sense in a staff guide and is useless in an email to the user. A bot saying “I checked the log” would be worse.
Alongside the product material, the service loads two separate sets of rules for the assistant. One sets the limits on when it can answer independently and takes precedence if it conflicts with the documentation. The other describes when to hand the request to a human agent. When these texts are available, the service puts them in separate sections of the system message. We can then ask two different questions: “How would we normally solve this problem?” and “Which part can the assistant handle in this conversation?” The instructions guide the model's choice. The code reads the action it chose and carries out the handoff.
What does that look like to the user? If there's nothing useful to explain yet, the model selects the handoff action. The service sends the notification, leaves the team a private note with the reason, and opens the conversation for an agent to pick up. If there is a useful explanation, a separate action sends it and then performs the same handoff. When reviewing the result, I'd look beyond whether the assistant wrote “I'll pass this to a colleague.” Which action did it choose, and what actually happened to the conversation?
How to check whether the model has enough context
For a first test, I'd take one conversation like this and show it to the assistant several times in shadow mode, where replies stay in private notes. I'd keep the model and knowledge base fixed and change one detail at a time. That makes it easier to see what influenced the answer.
| What we change in the example | What to check in the assistant's suggestion |
|---|---|
| Leave only “I've already tried that” | It asks what the user did instead of assuming they checked spam |
| Restore the assistant's previous public reply asking about spam | It takes the user's answer into account and doesn't send them back to the same folder |
| Add a private note to the ticket suggesting a queue delay | The note doesn't appear in the model request, and the user isn't told there's a confirmed outage |
| Add a rule that only a human agent can access the delivery log | It selects a handoff without claiming it has already checked delivery |
This is a proposed test, not a report of four successful runs. What I like about it is that we know in advance what counts as an error. In shadow mode the answer stays in a note; this test won't bring a person into the conversation. First we can check what context the model received and which action it proposed. Whether the service actually notified the user and handed over the request deserves a separate test in a live-mode test conversation. Even a polished reply fails if it credits the user with doing something they didn't do or promises a check without handing the request over.
From the user's side, the work is almost invisible. All they see is that nobody has told them to look in spam a third time, and that the next person knows where to pick up. That's a useful goal for me: preserve just enough of the thread for someone else to follow it.
In the next part, I'll look at what an answer like this costs: when the first call is enough, what changes at the next processing tier, and why it can sometimes be cheaper to bring in a human agent straight away.
Top comments (0)