I spend a good part of my day working with the community at CopilotKit, answering questions in Discord, GitHub, and Slack, helping people get unstuck, and pointing them to the right resources.
I started building answerLoops to help me with that work. As it took shape, I added integrations for more platforms so other teams could use it in their communities too.
It’s open source, and I’m building it in public.
Explore answerLoops on ⭐️ GitHub
A big part of the work has been deciding how knowledge gets into the system. Some answers are in the docs. Others come out of a support conversation or are things someone on the team just knows.
For an agent to use those answers, the team needs a straightforward way to save them, review them, and keep them current. That’s what I’ve been building into the knowledge base.
Answering the question is only part of the work
A question comes in. You find the relevant documentation or resource, figure out how it applies to that person’s setup, and write a reply.
During that conversation, you might discover something the docs don’t cover. A missing setup step, a confusing default, or a workaround that deserves its own article.
You resolve the question, but that useful detail stays buried in the conversation. When someone else asks, the team has to find it again.
I want answerLoops to help with both parts: preparing a useful answer now and saving what the team learns for future questions.
What answerLoops does
answerLoops is an open-source AI support tool that brings questions from Discord, Slack, GitHub, Telegram, and other connected channels into one dashboard. It uses your knowledge base to help prepare answers.
For knowledge-based replies in the ticket pipeline, a Mastra agent generates a draft and a separate agent scores its confidence and whether it fully answers the question. Automatic replies are off by default. When enabled, answers that meet the configured criteria can be sent automatically; other drafts remain available for human review.
There’s also an embeddable website chat widget built with CopilotKit and connected to a Mastra support agent. Visitors can ask questions directly, with published knowledge-base articles supplied as reference material.
The widget supports richer responses, including step-by-step guides, quick replies, callouts, link cards, and contact options.
But the quality of those answers depends on the information available to the agent. That’s why much of the recent work has been on the knowledge base.
Add the answer that isn’t in your docs
Sometimes you don’t need to import a documentation site. You just need to write down something the agent should know.
An owner or admin can open Knowledge Base → Manage and use Write an answer by hand:
- Enter a question or title.
- Write the answer or context.
- Click Add to knowledge base.
For example, a customer might ask whether changing a configuration requires a restart. Someone on your team knows the answer, but it isn’t documented anywhere.
The article is published and indexed when it’s saved, making it available for subsequent questions. There’s no draft step in this form, so the answer should be suitable for customers to read.
Adding an article doesn’t retrain the model. It gives the agent another reference to retrieve when a relevant question comes in.
Articles added this way appear under Added by hand in the Knowledge coverage tab.
It’s a small feature, but it gives the person with the answer a direct way to improve what the agent knows.
Bring in the knowledge you already have
You can also populate the knowledge base from existing content:
- Documentation URLs: import a page or crawl a documentation site.
- Files: upload PDF, DOCX, Markdown, text, or CSV files.
- GitHub: sync Markdown and MDX files, with optional syncs when changes are pushed to the default branch.
- Notion: connect pages and databases your team wants to use.
Notion content imports unpublished. Your team can inspect it before making it available for customer-facing answers, and that publishing choice is preserved across syncs.
The manual-entry form sits alongside these sources. If an answer already exists in your docs, import it. If it only exists in someone’s head, write it down.
Turn a resolved ticket into reusable knowledge
Closing a ticket helps the person who asked. Turning that resolution into a knowledge-base article makes the explanation available for future questions.
The Promote to KB action lets an owner or admin turn that ticket’s question and answer into a knowledge-base article.
Before promoting it, check that the answer applies beyond the original conversation. Remove account-specific details and make the explanation useful to someone encountering the same problem later.
This matters for the website widget in particular. Its retrieved reference material comes from published KB articles, rather than raw resolved tickets.
Promoting an answer gives it a place in the shared knowledge base where the widget can find it.
Let customer questions help you decide what to write
You won’t know every gap in your documentation ahead of time.
The Review tab captures questions that weren’t covered by the available knowledge. On hosted plans, this queue is available on Pro and above.
A usable draft can come along as a starting point. If there isn’t one, the item is marked No usable answer, and someone can write it.
Owners and admins can edit the question and answer, then choose Approve and add to knowledge base. If the item shouldn’t become an article, they can reject it instead.
Nothing in the queue becomes searchable customer-facing knowledge until it’s approved.
When the same question comes in again, its counter increases instead of creating another copy. The most-asked questions appear first.
If several people ask about the same setup step, that’s a useful signal about what your knowledge base is missing. The queue gives your team a question to work from and a place to review the answer before publishing it.
Go from a knowledge gap to a new article
The Knowledge Gaps page provides another view of where support needs attention.
It surfaces tickets with low-confidence answers, tickets that need a person, and resolved questions without a covering KB article.
Repeated questions are grouped with an Asked N times badge, so you can see recurring topics without reading every ticket separately.
The + KB action opens the manual-entry form with the ticket’s question and any existing answer filled in. Review the text, add the article, and it’s linked back to the ticket to close that gap.
You don’t have to leave the question behind and start a blank document somewhere else.
Check what the agent can actually find
Importing a document doesn’t tell you whether it will help with a particular question.
The Knowledge coverage tab shows articles grouped by source, including whether they’re live, unpublished, or not searchable.
There’s also a Try a customer question field. It runs the same lookup the website widget uses and shows which articles match—or that nothing covers the question.
After adding an answer, try the original customer question and a few variations. Does the lookup find the article you just wrote? If it doesn’t, you have something concrete to investigate before judging the generated reply.
This checks retrieval, not whether every generated reply will be correct. You still need to review the answers themselves. But it gives you a way to investigate whether the intended information is being found.
The KB is the source of product knowledge your team maintains for the agent. If no article matches, the assistant can fall back to general knowledge, which may not fit your product. Finding those gaps and adding the right material is part of maintaining it.
See which articles are being used
answerLoops also tracks article usage.
It counts when an article is supplied as context for an answer that is actually produced, including website replies and ticket drafts.
The distinction matters: this means the article was given to the agent, not necessarily that the final reply quoted it.
Searches and simulations don’t increase the count.
Together with the coverage view and repeated-question counts, this helps you understand what information is being retrieved and where more work might be useful.
Where CopilotKit and Mastra fit
I work in DevRel at CopilotKit, so part of building answerLoops is putting the tools I work with into a product and sharing what I learn.
Mastra runs the agents behind the support drafts and website conversation, including the chat agent’s thread memory.
CopilotKit provides the foundation for the custom widget interface. The application connects that interface to the agent and supplies the relevant knowledge-base context.
The product also has its own workflows around those pieces: importing sources, reviewing unanswered questions, publishing articles, and handling staff actions on tickets.
Those workflows are where the team gets to improve the information the agent uses. Someone can spot a missing answer, write it, publish it, and check whether a future question retrieves it.
That’s become a much bigger part of answerLoops as I’ve kept building.
Try it, I would love your feedback
I started this to help with my own community work. I’m sharing it so other developers and support teams can try it in theirs.
The code and setup instructions are on GitHub. You can explore how retrieval, the review queue, and the integrations work, or run answerLoops yourself. The knowledge-base documentation walks through the workflows in this article.
If it looks useful, give the repo a ⭐️. If you try it, tell me what worked and what got in your way.
New to contributing to open source? Reach out—I’d be happy to help you find a place to start.
Where does your team keep the useful answers that haven’t made it into the docs yet?






Top comments (59)
"The article was supplied as context, not necessarily that the final reply quoted it" is the most honest line in this article. Most RAG tools claim "this article was used" when they really mean "it was retrieved", and those are very different things.
Curious how you'd handle the case where an article is retrieved but the agent ignores it. Does the usage tracking show "retrieved but not used", or does it just count the retrieval? Asking because that's the difference between "this article is valuable" and "this article is noise", and it's exactly what a support team needs to know when deciding what to write next.
@alexcodebytes, it currently counts articles supplied as context when an answer is generated. If three articles go in and the answer only draws on one, all three get counted. Searches and simulations don’t increment it, but there isn’t a “retrieved but not used” distinction yet.
You’re right that this isn’t enough to judge an article’s value. I opened an issue to track answer-level sources and separate context exposure from evidence that an article contributed to the reply.
If you’d like to try answerLoops, reach out directly. I’d love your feedback on what would make that useful for your team.
Appreciate the honesty and the issue. I'll take you up on the offer — I'll try answerLoops this week and send you notes.
One thing I'd already flag: the "retrieved but not used" distinction is only useful if it's visible to the person writing the knowledge base. If the dashboard shows "this article was retrieved 100 times" but doesn't show "and the agent used it 20 times", the writer will keep optimizing the wrong metric.
Looking forward to seeing how the issue lands.
Thanks, Oleksandr. I appreciate you giving it a try, and I’d love to hear what you find.
That’s a useful point about the person writing the KB. They need to see the distinction alongside the article, with a way to inspect the answers behind the numbers.
I’d want to show “supplied as context” alongside “evidence found in the answer,” and keep unassessed answers separate. Otherwise, we risk replacing one misleading count with another. “Not assessed” shouldn’t look like “not used.”
Your 100-versus-20 example is a good test for the dashboard: curious, if you were maintaining your KB, would you understand those numbers and see enough detail to decide whether to revise the article?
Message me when you’re ready to try it. Happy to help you get set up.
"'Not assessed' shouldn't look like 'not used'" is the exact distinction I was worried about.
If the dashboard shows "this article was retrieved 100 times, used 20 times", the writer will assume the other 80 were wasted. But maybe 40 of those were "not assessed" (the model didn't cite it, but it also didn't need to), and only 40 were genuinely ignored.
Without the "not assessed" bucket, the numbers lie by omission.
To answer your question: yes, I'd understand 100 vs 20, but only if the tool also showed me why the 80 weren't used. Was it because the article was irrelevant, or because the model found a better source, or because the answer didn't need a citation at all?
Thanks, Oleksandr. This conversation has helped me get much more specific about what would actually help someone maintain their KB. I appreciate your continued feedback.
I’ve opened a detailed follow-up issue for the assessment and dashboard behavior.
The approach I want to explore is comparing the finished answer with the articles supplied to it, then showing a short explanation alongside the relevant passages. That could identify supporting guidance, conflicting information, or overlapping sources where attribution is unclear.
There’s an important distinction in how we explain it: we can assess the evidence, but asking the model why it ignored an article wouldn’t prove that was its actual reason. I don’t want to turn a plausible explanation into another misleading metric.
The dashboard would separate:
You’d be able to open the examples behind those counts. An answer without citations could still have supporting evidence; “not assessed” would mean the check hasn’t happened or couldn’t be completed.
I’ll need to test those assessments against human-reviewed examples before treating the numbers as useful guidance. Your feedback has helped shape both what to build and how to tell whether it’s working. Looking forward to your notes when you try it.
"Asking the model why it ignored an article wouldn't prove that was its actual reason" is the most important sentence here, and it's the one most AI observability tools get wrong.
A model can always generate a plausible explanation for its behavior. That doesn't mean it's the real one. If you build a dashboard on top of "the model said it ignored this because X", you're just adding another layer of hallucination.
The four-bucket split (supporting / no evidence / inconclusive / not assessed) is much better. It's honest about what you actually know.
Hey @alexcodebytes, I'd love to get your feedback on the architecture I'm putting together. Your point about numbers lying by omission is why "not assessed" is its own bucket now.
Rough plan: after an answer goes out, a separate check compares it against the articles it was given and shows the matching resources side by side. I'm not asking the model why it skipped something, for the obvious purpose. Nothing gets shown as a number until I've tested it against examples people have labeled, and if the check can't tell, it says "inconclusive".
I want your take on a few things, since you'd be the interfacing with this kind of system.
When you open one of 100 answers, is the answer sentence next to the article passage enough? Or would you want to see the other articles that went into the same answer?
Sometimes two articles say the same thing. I can say both support the answer, but not which one it leaned on. Would you want that shown as "shared support", or does that feel like dodging?
What would you do with the numbers first? I can list articles that get supplied a lot but rarely support an answer, or list questions that keep coming back unsupported, which usually means a missing article. Which would you look at?
On your point about why the other 80 weren't used: I can show "this article is about something else" or "another article says the same thing". I can't show that the model preferred one source. Is the first kind still worth having?
Last one. To show evidence, I'd have to keep short excerpts of answers for something like 90 days. Right now it's not written in the code yet. Would that bother you?
Your feedback really helps. And when you try it, tell me wherever the numbers confuse you.
Thanks for the detailed questions — this is the right way to design it. Let me go one by one.
"Is the answer sentence next to the article passage enough?"
For the common case, yes. If I'm looking at "answer says X, article says Y", that's enough to judge whether the support is real. But if two articles say the same thing and the answer contradicts both, I'd want to see both. The side-by-side is good; a "see all sources" link is enough for the rest.
"Shared support" vs "dodging"
"Shared support" is honest. "Dodging" would be hiding the fact that two articles say the same thing. Showing both with a note that they're redundant is the right answer. If the model leaned on one, you can't know, so don't pretend to.
Which number first?
"Questions that keep coming back unsupported" is the one that changes what I write next. "Articles supplied but rarely supporting" tells me an article might be useless, but "questions with no support" tells me what's missing entirely. That's the actionable one.
"This article is about something else" vs "another article says the same thing"
Both are useful. The first one tells me the article is irrelevant; the second tells me the article is redundant. Those are different problems, and both are worth showing. The "model preferred one source" is unknowable, so don't show it.
90-day excerpts
That would bother me if I didn't know about it. If it's opt-in, or if the excerpts are short enough to be non-sensitive (e.g. one sentence, not the whole answer), it's fine. If it's on by default with full answers, that's a privacy problem.
One last thing: the "not assessed" bucket is the right call. If the check can't tell, "unknown" is honest, and "not used" is a lie. Keep it.
Hey Oleksandr, thank you for taking the time to go through each question. It was really helpful, and I've already updated the plan to match.
Showing both articles when they say the same thing, with a note that they're redundant, is how it will work, and we won't claim to know which one the model leaned on. I'm also adding the "see all sources" link so you can see every article when an answer contradicts more than one.
When an article doesn't help, we'll show two separate reasons: it's about something else, or another article says the same thing. The "not assessed" bucket stays.
Your point about the first number was the biggest change. I'd been focused on how each article performs, but questions that keep coming back with no supporting article are what tell a writer what's missing, so I opened a separate item for that view: github.com/answerLoops/answerLoops...
On the 90-day excerpts, you're right that it would be a problem if people didn't know. Storing them will be opt-in per workspace, kept to about a sentence, and the setting will say how long they're kept.
Thanks again!
This is exactly the kind of iteration speed that separates good products from great ones. The shift to tracking unanswered questions as the primary metric (the "first number") is a massive product insight. It turns the tool from a simple auditing dashboard into a genuine content strategy engine—showing writers exactly what they should be writing next, rather than just grading what they've already written.
Also, great call on the opt-in, sentence-level retention for the 90-day excerpts. That completely removes the privacy and compliance friction for agency or enterprise workspaces while still enabling the core feature to work.
Excited to see the new view in action. Keep up the great work, and thanks for actually listening to feedback!
Hey Oleksandr, thanks for the kind words. Your insights are helping me redesign the KB so it's own insights paint a clear picture.
If you ever want to compare notes, or something about the numbers confuses you, I thought I'd pass along my email: nathan@answerloops.com.
I'd be glad to hear from you. I'll also post here when the new view lands.
Thanks, Nathan! I'll definitely shoot you an email soon to compare notes on the metrics. Really looking forward to seeing how the new "unanswered questions" view turns out—it's rare to see a builder implement feedback that fast. Best of luck with the redesign, and I'll keep an eye out for your update when it lands!
Amazing!
I'll be reaching out soon with an update!
Thanks for building this in the open, and for the "retrieved but not used" distinction, that is the honest version of most RAG claims.
Your closing question is where I live. I spent three weeks measuring which parts of a guide still change what a coding agent does: 438 runs, nine agent setups, each task with a guide and without. A guide that retells the docs did nothing in 32 of 32 runs. What moved the result was exactly the kind of answer you describe, the one that lives in someone's head: a fact the docs don't state, a fix the agent can diagnose but not know, a check it skips on its own. Write-up with the numbers: dev.to/yahhi/when-coding-agent-gui...
Where my answers live: I keep them as short stories on worklore.dev, a small site I run. Each one is one developer's case (what I tried, what failed, what worked) with a Verify section, and the agent that reproduces it reports back, so a story carries a count of how many other people's agents it worked for. It is the same idea as your promoted ticket, from the other side: not the support team's answer to a user, but the developer's own case, shared across teams, with the review done by reproduction instead of approval.
One question from your Knowledge Gaps view: when a gap gets an answer, do you keep who answered it and when? In my runs the answers aged fast; one story didn't have answer in the docs when it was written. But now it makes no sense in improvement the agent's reply.
Thanks, @yahhi. Keeping the verification alongside the answer makes a lot of sense, especially when the docs eventually catch up.
In answerLoops, it’s partially covered today. Adding an answer from Knowledge Gaps keeps the originating ticket link and the article’s creation/update dates, but it doesn’t yet record who added that knowledge. The separate review queue does record who approved an answer and when.
What’s missing is a consistent history across those paths, plus a separate record of when someone last verified the answer. An edit or sync shouldn’t count as “this still works.”
I’ve opened an issue to track that, including who added/reviewed the knowledge, which revision was verified, and a way to mark answers that newer documentation has superseded. Thanks for pointing this out, it's super helpful.
If you’d like to try answerLoops, reach out directly. I’d love your feedback on how this holds up with the cases you’ve collected.
Since we're close to competitors, it would help both of us to try each other's tools and compare pros and cons. I'd like to see how answerLoops holds up on the cases I've collected, and you're welcome to read the stories and see what the format does and doesn't give you.
Your issue about marking answers that newer documentation has superseded touches something I've settled on differently. Worklore can work as a documentation centre, but it's also something to read when you want to burn some time wandering the web. So I'm fine with stories that matter only to a human and add nothing to an agent's exact instructions. A person can read one and get inspired to try a new idea, while the agent may already know all those ideas. The PostHog widget story is the example: in the test it did nothing, 9 of 9 either way, now that the docs cover it, but it's still a true record of what someone did.
So a "superseded" mark could say the answer is no longer needed by the agent, without implying it should disappear for people. Worklore is at worklore.dev/ if you'd like to try it.
Hey Valentina, yes, let's do it. I read your 438-run post and checked out a few other stories. They are extremely interesting.
The note vs. "follow this guide" result caught my attention. 7 of 7 agents rewrote the token exchange when told to follow the guide, and only 3 of 7 did when it was framed as a colleague's note. We hand KB answers to the agent as instructions in answerLoops, so I'm going to try framing them as a case and see what changes.
Your "last reproduced" date is basically what I meant in my issue. An edit or a sync shouldn't count as "this still works." You get it from the agent re-running the story, and I get it from a person reviewing. I can't say from historical data which works 100% yet.
You're right that "superseded" can mean two things. I'm still mapping out the architecture in this area of the KB at the moment.
Send over a few cases, and I'll run them. It would be an interesting experiment.
Four cases, each with the question a user would actually send, so you can run it as a ticket. The story itself is the KB article; the raw text is at the same URL with .md at the end.
Intervals.icu OAuth quirks, by radikkhaziev. Ticket: "Connect works, then the token stops after an hour, and the authorize URL fails with Duplicate scope." The facts are on a forum, not in the docs. Without the story 3 of 7 agents got it, with it 6 of 7.
worklore.dev/s/2026-04-15-interval...
Exact HR corridors on Garmin, by radikkhaziev. Ticket: "Workouts I push through the API show one HR value on the watch instead of a zone." A fact the docs don't state, the strongest effect in my runs: 3 of 8 without, 16 of 16 with.
worklore.dev/s/2026-05-12-exact-hr...
PostHog event per keystroke, mine. Ticket: "Our PostHog free tier was gone in days after we added a Flutter registration form." Nothing is hidden here, the agent just walks into the trap: 3 of 7 fell in without the story, 0 with.
worklore.dev/s/2026-10-07-posthog-...
PostHogWidget, mine, as the control. Ticket: "Session replay records nothing in our Flutter app." The docs cover this now, so the story changed nothing in 9 of 9 runs. If your "superseded" mark works, this is the one it should catch.
worklore.dev/s/2026-10-07-posthog-...
Two things I'd love to see back: whether the draft reply used the fact from the article (not just retrieved it), and what changed when you handed the article over as a case instead of instructions. Each story has a Verify section if you want a pass/fail rather than a reading.
Nice, I'm a huge open source advocate, Nathan, and I love that you created an issue to close the gap. That's awesome!
Following!
Thanks Ethan :)
the manual form having no draft step is the line that stuck with me. one hand written answer that goes live straight into retrieval and is wrong gets quoted by the widget forever, and nobody notices because it reads confident.
what bit us in a similar kb was staleness. we stamped each article with the product version it was written against and downweighted older ones at retrieval, which killed most of the confident but outdated answers.
does answerLoops record which article an answer came from, so a bad one can be traced and pulled?
Hey, @mudassirworks, the manual form is a direct publishing action for owners and admins, so it assumes the person adding the answer has checked it.
On tracing: answerLoops currently records daily usage counts per article when it’s supplied as context for a generated answer. It doesn’t yet store a per-answer record linking that specific reply to its source articles, and a usage count doesn’t prove the reply quoted the article.
You can delete an article to remove it from future KB retrieval, but that doesn’t retract answers already sent or remove them from an existing conversation’s memory.
Your product-version approach is useful feedback. Semantic retrieval currently ranks by relevance; it doesn’t downweight articles by age or product version. The coverage tab helps inspect what a question retrieves, but it doesn’t replace that historical trace.
If you’re interested, reach out to me directly-I’d love to help you try answerLoops and get your feedback, especially given what you’ve already learned working with a KB in the past. It would be extremely valuable.
thanks for the straight answer. the gap you described, usage count per article but no per answer record, is the one that hurts most in practice. once a bad article gets pulled you still can't tell which replies already sent quoted it.
what worked for us was logging the retrieved chunk ids next to each reply at generation time. even without proving the quote, it gave us a list of conversations to review after a pull.
would a per reply source log be cheap to add on your side, or does the conversation memory make it messy?
Hey Mudassir, thank you for coming back to this, and for the concrete detail about logging retrieved ids at generation time. That's close to what we're building, and your feedback is helping shape it.
A per-reply source log is lightweight to run, and the conversation memory doesn't make it messy. It's one small write per answer and doesn't need another model call, though it does mean new storage and a cleanup job that removes old records. For each generated answer we log only identifiers: which articles went into the final context, their rank, and which version of each article it was at that moment. We don't store the question, the answer text or the visitor, so there's no conversation memory to untangle. Because the log records the version, it stays accurate after someone edits the article. If an article is deleted, the log keeps that version information, so you can still tell which answers were built on the one you pulled.
It's the same idea as your retrieved-chunk log: it doesn't prove the reply quoted the article, but it gives you a list of answers to review after you pull one. We're in the process of implementing it now, along with a view of an article's recent answers, so you can go straight from a bad article to the replies it was supplied to. We expect to release it within the next couple of days. It's tracked here: github.com/answerLoops/answerLoops...
One thing we're still deciding is how an answer links back to the specific conversation, since we don't want to store conversation content just to make that possible. Your point about needing a list of conversations to review is useful input on that.
Your version-stamp idea is now its own item, because we think it's worth getting right: github.com/answerLoops/answerLoops.... We're wary of lowering an article's ranking just because it's old. An old article can still be correct, and the article would look unchanged in the dashboard while customers quietly stopped getting answers from it. So we're leaning toward flagging articles for review when the product version moves, so a person decides, and showing the team which often-used articles haven't been checked in a while. Since you've done this for real, a few questions would help us a lot:
Again, I really appreciate your feedback.
version came from our releases, not hand stamping. every article carried the product version it was written against, and a release flagged anything more than two minors behind. flagged for a person, never auto downweighted. we tried downweighting once and it buried an old billing article that was still correct. support noticed because answers got vague, the dashboard showed nothing. so review queue over ranking penalty for us. when the version moves, how will you decide which flagged articles get looked at first?
The point that answering is only half the work matches what I have seen with support teams. The useful detail comes out in the conversation, the ticket closes, and three weeks later someone digs for the same answer again.
Mudassir's comment about the manual form is worth taking seriously. A hand written answer that goes straight into retrieval gets quoted with full confidence until someone notices, and the person who notices is usually a customer. Stamping each article with the product version it was written against gives you a cheap trigger: when the version moves, the article drops to review instead of staying live.
I would also add a per answer record linking each reply to the articles it used. You mentioned that is not stored yet. Once it is, Luis's question mostly answers itself, because a corrected article can show every past reply that relied on the old version. Is that on the roadmap after the coverage tab?
Hi @ascendra_ventures_92c21f2, thanks for this. We’ve already implemented several of the earlier suggestions for the next release, which were very helpful: recording who added an article, keeping a history of changes and review decisions, and recording who last verified the answer and when.
Verification applies to the content that was checked. If the answer changes afterward, it shows as “Changed since last verified.” There’s also an optional verification note for details like the product version someone checked against.
Your suggestion to trigger review when the product version changes would be a useful next step. The current implementation detects changes to the article itself; it doesn’t automatically move articles into review when a new product version ships.
The per-answer source history is tracked in Issue #422. That would let a team inspect which article revisions were supplied to a reply and find other replies associated with those sources. We’ll keep “supplied as context” separate from evidence that the reply actually drew on them.
Thanks for taking the time to think through the maintenance side. If you’d like to try answerLoops and share feedback, I would be happy to help you get started, just reach out directly.
The supplied-as-context vs actually-quoted distinction is the right one to track. The part I would add: once a wrong answer ships, what matters most is being able to search which articles informed it, not just the usage counts. That is what turns the KB from append-only into something a team can actually maintain.
Thanks for this @argumentmoney8117. When the agents answer is off, a usage count doesn’t tell you what to fix. Being able to open that answer and see the articles supplied at the time would be a useful upgrade.
Today, answerLoops tracks article exposure by day. I’ve opened #422 to add that answer-to-article history, including looking up recent answers associated with an article.
Keeping the revision matters too: editing an article today shouldn’t change the record of what was supplied yesterday. And that history should still distinguish “supplied as context” from evidence that the answer actually drew on it.
If you’d like to try answerLoops and share feedback on that workflow, reach out directly. I’d appreciate it.
We've been doing all of this manually, and I can see the value of having a system like this.
Most of our community lives in GitHub. Do you support GitHub Discussions or does it just register the Issue that is submitted and turns that into a ticket?
Hey Shashra, thanks for checking it out. Yes, both GitHub Issues, and Discussions are both supported in answerLoops. You can check out the docs here: answerloops.com/docs/integrations/...
One failure mode this thread hasn't touched yet: language. The manual-entry form publishes straight to retrieval with no draft step, which is already flagged above as risky for accuracy, but even a correct, reviewed answer can become subtly wrong once it's the match returned for a question asked in a different language than it was written in, if retrieval doesn't account for how a technical term or product name gets paraphrased by a non-native speaker. We hit this building a support KB for a product with a large non-English user base: retrieval worked fine, the quoted answer didn't fully apply to that user's setup, and nothing in the usage metrics flagged it until a customer pointed it out manually. Does answerLoops have any visibility into which language a question/answer pair actually serves well, or is that still something a human has to notice after the fact?
Hey @techanalysisinsights, it's a good one to learn from. I appreciate your question.
answerLoops replies in the language the user writes in, and retrieval works by meaning rather than exact wording, so a question in one language can match an article written in another.
Usage tracking records which articles were supplied to each answer, so you can trace any reply back to exactly what it was built from. That's how a case like yours gets diagnosed: you see the match, then check it against the user's setup.
The best way to pressure-test the paraphrased-term problem is with real cases. If you're willing, I would be happy to run an experiment and share the results with you. If you're open to sending me two or three of the non-English questions (mock data is fine) that you noticed didn't line up, along with the terms those users used, and I'll run them against English articles and share what comes back.
Your feedback on the results would be highly valuable, and I'm open to communicating via email @ nathan@answerloops.com
Thanks for sharing. How difficult is it to connect to Discord and start replying to support queries? My support lives in Discord, and we are a very small team but it tends to be time consuming, if I have to find the answers in the source code.
I designed it so it's only one click to connect to your Discord. This authenticates, then you choose the server you want the answerLoops bot added to, then choose the channels you want it to listen, and respond in. I'd love to get your feedback if you try it out.
I just found this, and I'm intrigued. I'm curious, If you’ve connected an AI assistant to your docs, how do you check that it’s finding the right information?
Hey James, one important detail is adding the correct info to your KB. How I validate is that I start with questions people have actually asked and check which articles come back, not just whether the generated answer sounds right. I want to make sure it's accurate.
In answerLoops, the Knowledge coverage tab runs the same lookup as the website widget, so I can see the matching articles or spot a question with no coverage. Then I review the generated reply separately: did it use that information correctly and actually answer the question?
If something’s missing, I can add the answer directly to the KB and try the question again.
Ok, so on the platform, someone can manually add to the Knowledge Source, then?
That's correct, you can manually add to the KB, so let's say someone on your team is the Product Expert, you can add notes from a meeting notetaker or just type them out, etc. Here's the docs reference: answerloops.com/docs/product/knowl...
Love this!
Some comments may only be visible to logged-in visitors. Sign in to view all comments.