I spend a good part of my day working with the community at CopilotKit, answering questions in Discord, GitHub, and Slack, helping people get unstu...
For further actions, you may consider blocking this person and/or reporting abuse
"The article was supplied as context, not necessarily that the final reply quoted it" is the most honest line in this article. Most RAG tools claim "this article was used" when they really mean "it was retrieved", and those are very different things.
Curious how you'd handle the case where an article is retrieved but the agent ignores it. Does the usage tracking show "retrieved but not used", or does it just count the retrieval? Asking because that's the difference between "this article is valuable" and "this article is noise", and it's exactly what a support team needs to know when deciding what to write next.
@alexcodebytes, it currently counts articles supplied as context when an answer is generated. If three articles go in and the answer only draws on one, all three get counted. Searches and simulations don’t increment it, but there isn’t a “retrieved but not used” distinction yet.
You’re right that this isn’t enough to judge an article’s value. I opened an issue to track answer-level sources and separate context exposure from evidence that an article contributed to the reply.
If you’d like to try answerLoops, reach out directly. I’d love your feedback on what would make that useful for your team.
Appreciate the honesty and the issue. I'll take you up on the offer — I'll try answerLoops this week and send you notes.
One thing I'd already flag: the "retrieved but not used" distinction is only useful if it's visible to the person writing the knowledge base. If the dashboard shows "this article was retrieved 100 times" but doesn't show "and the agent used it 20 times", the writer will keep optimizing the wrong metric.
Looking forward to seeing how the issue lands.
Thanks, Oleksandr. I appreciate you giving it a try, and I’d love to hear what you find.
That’s a useful point about the person writing the KB. They need to see the distinction alongside the article, with a way to inspect the answers behind the numbers.
I’d want to show “supplied as context” alongside “evidence found in the answer,” and keep unassessed answers separate. Otherwise, we risk replacing one misleading count with another. “Not assessed” shouldn’t look like “not used.”
Your 100-versus-20 example is a good test for the dashboard: curious, if you were maintaining your KB, would you understand those numbers and see enough detail to decide whether to revise the article?
Message me when you’re ready to try it. Happy to help you get set up.
"'Not assessed' shouldn't look like 'not used'" is the exact distinction I was worried about.
If the dashboard shows "this article was retrieved 100 times, used 20 times", the writer will assume the other 80 were wasted. But maybe 40 of those were "not assessed" (the model didn't cite it, but it also didn't need to), and only 40 were genuinely ignored.
Without the "not assessed" bucket, the numbers lie by omission.
To answer your question: yes, I'd understand 100 vs 20, but only if the tool also showed me why the 80 weren't used. Was it because the article was irrelevant, or because the model found a better source, or because the answer didn't need a citation at all?
Thanks, Oleksandr. This conversation has helped me get much more specific about what would actually help someone maintain their KB. I appreciate your continued feedback.
I’ve opened a detailed follow-up issue for the assessment and dashboard behavior.
The approach I want to explore is comparing the finished answer with the articles supplied to it, then showing a short explanation alongside the relevant passages. That could identify supporting guidance, conflicting information, or overlapping sources where attribution is unclear.
There’s an important distinction in how we explain it: we can assess the evidence, but asking the model why it ignored an article wouldn’t prove that was its actual reason. I don’t want to turn a plausible explanation into another misleading metric.
The dashboard would separate:
You’d be able to open the examples behind those counts. An answer without citations could still have supporting evidence; “not assessed” would mean the check hasn’t happened or couldn’t be completed.
I’ll need to test those assessments against human-reviewed examples before treating the numbers as useful guidance. Your feedback has helped shape both what to build and how to tell whether it’s working. Looking forward to your notes when you try it.
Thanks for building this in the open, and for the "retrieved but not used" distinction, that is the honest version of most RAG claims.
Your closing question is where I live. I spent three weeks measuring which parts of a guide still change what a coding agent does: 438 runs, nine agent setups, each task with a guide and without. A guide that retells the docs did nothing in 32 of 32 runs. What moved the result was exactly the kind of answer you describe, the one that lives in someone's head: a fact the docs don't state, a fix the agent can diagnose but not know, a check it skips on its own. Write-up with the numbers: dev.to/yahhi/when-coding-agent-gui...
Where my answers live: I keep them as short stories on worklore.dev, a small site I run. Each one is one developer's case (what I tried, what failed, what worked) with a Verify section, and the agent that reproduces it reports back, so a story carries a count of how many other people's agents it worked for. It is the same idea as your promoted ticket, from the other side: not the support team's answer to a user, but the developer's own case, shared across teams, with the review done by reproduction instead of approval.
One question from your Knowledge Gaps view: when a gap gets an answer, do you keep who answered it and when? In my runs the answers aged fast; one story didn't have answer in the docs when it was written. But now it makes no sense in improvement the agent's reply.
Thanks, @yahhi. Keeping the verification alongside the answer makes a lot of sense, especially when the docs eventually catch up.
In answerLoops, it’s partially covered today. Adding an answer from Knowledge Gaps keeps the originating ticket link and the article’s creation/update dates, but it doesn’t yet record who added that knowledge. The separate review queue does record who approved an answer and when.
What’s missing is a consistent history across those paths, plus a separate record of when someone last verified the answer. An edit or sync shouldn’t count as “this still works.”
I’ve opened an issue to track that, including who added/reviewed the knowledge, which revision was verified, and a way to mark answers that newer documentation has superseded. Thanks for pointing this out, it's super helpful.
If you’d like to try answerLoops, reach out directly. I’d love your feedback on how this holds up with the cases you’ve collected.
Since we're close to competitors, it would help both of us to try each other's tools and compare pros and cons. I'd like to see how answerLoops holds up on the cases I've collected, and you're welcome to read the stories and see what the format does and doesn't give you.
Your issue about marking answers that newer documentation has superseded touches something I've settled on differently. Worklore can work as a documentation centre, but it's also something to read when you want to burn some time wandering the web. So I'm fine with stories that matter only to a human and add nothing to an agent's exact instructions. A person can read one and get inspired to try a new idea, while the agent may already know all those ideas. The PostHog widget story is the example: in the test it did nothing, 9 of 9 either way, now that the docs cover it, but it's still a true record of what someone did.
So a "superseded" mark could say the answer is no longer needed by the agent, without implying it should disappear for people. Worklore is at worklore.dev/ if you'd like to try it.
Hey Valentina, yes, let's do it. I read your 438-run post and checked out a few other stories. They are extremely interesting.
The note vs. "follow this guide" result caught my attention. 7 of 7 agents rewrote the token exchange when told to follow the guide, and only 3 of 7 did when it was framed as a colleague's note. We hand KB answers to the agent as instructions in answerLoops, so I'm going to try framing them as a case and see what changes.
Your "last reproduced" date is basically what I meant in my issue. An edit or a sync shouldn't count as "this still works." You get it from the agent re-running the story, and I get it from a person reviewing. I can't say from historical data which works 100% yet.
You're right that "superseded" can mean two things. I'm still mapping out the architecture in this area of the KB at the moment.
Send over a few cases, and I'll run them. It would be an interesting experiment.
Four cases, each with the question a user would actually send, so you can run it as a ticket. The story itself is the KB article; the raw text is at the same URL with .md at the end.
Intervals.icu OAuth quirks, by radikkhaziev. Ticket: "Connect works, then the token stops after an hour, and the authorize URL fails with Duplicate scope." The facts are on a forum, not in the docs. Without the story 3 of 7 agents got it, with it 6 of 7.
worklore.dev/s/2026-04-15-interval...
Exact HR corridors on Garmin, by radikkhaziev. Ticket: "Workouts I push through the API show one HR value on the watch instead of a zone." A fact the docs don't state, the strongest effect in my runs: 3 of 8 without, 16 of 16 with.
worklore.dev/s/2026-05-12-exact-hr...
PostHog event per keystroke, mine. Ticket: "Our PostHog free tier was gone in days after we added a Flutter registration form." Nothing is hidden here, the agent just walks into the trap: 3 of 7 fell in without the story, 0 with.
worklore.dev/s/2026-10-07-posthog-...
PostHogWidget, mine, as the control. Ticket: "Session replay records nothing in our Flutter app." The docs cover this now, so the story changed nothing in 9 of 9 runs. If your "superseded" mark works, this is the one it should catch.
worklore.dev/s/2026-10-07-posthog-...
Two things I'd love to see back: whether the draft reply used the fact from the article (not just retrieved it), and what changed when you handed the article over as a case instead of instructions. Each story has a Verify section if you want a pass/fail rather than a reading.
Nice, I'm a huge open source advocate, Nathan, and I love that you created an issue to close the gap. That's awesome!
Following!
the manual form having no draft step is the line that stuck with me. one hand written answer that goes live straight into retrieval and is wrong gets quoted by the widget forever, and nobody notices because it reads confident.
what bit us in a similar kb was staleness. we stamped each article with the product version it was written against and downweighted older ones at retrieval, which killed most of the confident but outdated answers.
does answerLoops record which article an answer came from, so a bad one can be traced and pulled?
Hey, @mudassirworks, the manual form is a direct publishing action for owners and admins, so it assumes the person adding the answer has checked it.
On tracing: answerLoops currently records daily usage counts per article when it’s supplied as context for a generated answer. It doesn’t yet store a per-answer record linking that specific reply to its source articles, and a usage count doesn’t prove the reply quoted the article.
You can delete an article to remove it from future KB retrieval, but that doesn’t retract answers already sent or remove them from an existing conversation’s memory.
Your product-version approach is useful feedback. Semantic retrieval currently ranks by relevance; it doesn’t downweight articles by age or product version. The coverage tab helps inspect what a question retrieves, but it doesn’t replace that historical trace.
If you’re interested, reach out to me directly-I’d love to help you try answerLoops and get your feedback, especially given what you’ve already learned working with a KB in the past. It would be extremely valuable.
thanks for the straight answer. the gap you described, usage count per article but no per answer record, is the one that hurts most in practice. once a bad article gets pulled you still can't tell which replies already sent quoted it.
what worked for us was logging the retrieved chunk ids next to each reply at generation time. even without proving the quote, it gave us a list of conversations to review after a pull.
would a per reply source log be cheap to add on your side, or does the conversation memory make it messy?
Hey Mudassir, thank you for coming back to this, and for the concrete detail about logging retrieved ids at generation time. That's close to what we're building, and your feedback is helping shape it.
A per-reply source log is lightweight to run, and the conversation memory doesn't make it messy. It's one small write per answer and doesn't need another model call, though it does mean new storage and a cleanup job that removes old records. For each generated answer we log only identifiers: which articles went into the final context, their rank, and which version of each article it was at that moment. We don't store the question, the answer text or the visitor, so there's no conversation memory to untangle. Because the log records the version, it stays accurate after someone edits the article. If an article is deleted, the log keeps that version information, so you can still tell which answers were built on the one you pulled.
It's the same idea as your retrieved-chunk log: it doesn't prove the reply quoted the article, but it gives you a list of answers to review after you pull one. We're in the process of implementing it now, along with a view of an article's recent answers, so you can go straight from a bad article to the replies it was supplied to. We expect to release it within the next couple of days. It's tracked here: github.com/answerLoops/answerLoops...
One thing we're still deciding is how an answer links back to the specific conversation, since we don't want to store conversation content just to make that possible. Your point about needing a list of conversations to review is useful input on that.
Your version-stamp idea is now its own item, because we think it's worth getting right: github.com/answerLoops/answerLoops.... We're wary of lowering an article's ranking just because it's old. An old article can still be correct, and the article would look unchanged in the dashboard while customers quietly stopped getting answers from it. So we're leaning toward flagging articles for review when the product version moves, so a person decides, and showing the team which often-used articles haven't been checked in a while. Since you've done this for real, a few questions would help us a lot:
Again, I really appreciate your feedback.
version came from our releases, not hand stamping. every article carried the product version it was written against, and a release flagged anything more than two minors behind. flagged for a person, never auto downweighted. we tried downweighting once and it buried an old billing article that was still correct. support noticed because answers got vague, the dashboard showed nothing. so review queue over ranking penalty for us. when the version moves, how will you decide which flagged articles get looked at first?
The point that answering is only half the work matches what I have seen with support teams. The useful detail comes out in the conversation, the ticket closes, and three weeks later someone digs for the same answer again.
Mudassir's comment about the manual form is worth taking seriously. A hand written answer that goes straight into retrieval gets quoted with full confidence until someone notices, and the person who notices is usually a customer. Stamping each article with the product version it was written against gives you a cheap trigger: when the version moves, the article drops to review instead of staying live.
I would also add a per answer record linking each reply to the articles it used. You mentioned that is not stored yet. Once it is, Luis's question mostly answers itself, because a corrected article can show every past reply that relied on the old version. Is that on the roadmap after the coverage tab?
Hi @ascendra_ventures_92c21f2, thanks for this. We’ve already implemented several of the earlier suggestions for the next release, which were very helpful: recording who added an article, keeping a history of changes and review decisions, and recording who last verified the answer and when.
Verification applies to the content that was checked. If the answer changes afterward, it shows as “Changed since last verified.” There’s also an optional verification note for details like the product version someone checked against.
Your suggestion to trigger review when the product version changes would be a useful next step. The current implementation detects changes to the article itself; it doesn’t automatically move articles into review when a new product version ships.
The per-answer source history is tracked in Issue #422. That would let a team inspect which article revisions were supplied to a reply and find other replies associated with those sources. We’ll keep “supplied as context” separate from evidence that the reply actually drew on them.
Thanks for taking the time to think through the maintenance side. If you’d like to try answerLoops and share feedback, I would be happy to help you get started, just reach out directly.
Thanks for sharing. How difficult is it to connect to Discord and start replying to support queries? My support lives in Discord, and we are a very small team but it tends to be time consuming, if I have to find the answers in the source code.
I designed it so it's only one click to connect to your Discord. This authenticates, then you choose the server you want the answerLoops bot added to, then choose the channels you want it to listen, and respond in. I'd love to get your feedback if you try it out.
The supplied-as-context vs actually-quoted distinction is the right one to track. The part I would add: once a wrong answer ships, what matters most is being able to search which articles informed it, not just the usage counts. That is what turns the KB from append-only into something a team can actually maintain.
Thanks for this @argumentmoney8117. When the agents answer is off, a usage count doesn’t tell you what to fix. Being able to open that answer and see the articles supplied at the time would be a useful upgrade.
Today, answerLoops tracks article exposure by day. I’ve opened #422 to add that answer-to-article history, including looking up recent answers associated with an article.
Keeping the revision matters too: editing an article today shouldn’t change the record of what was supplied yesterday. And that history should still distinguish “supplied as context” from evidence that the answer actually drew on it.
If you’d like to try answerLoops and share feedback on that workflow, reach out directly. I’d appreciate it.
We've been doing all of this manually, and I can see the value of having a system like this.
Most of our community lives in GitHub. Do you support GitHub Discussions or does it just register the Issue that is submitted and turns that into a ticket?
Hey Shashra, thanks for checking it out. Yes, both GitHub Issues, and Discussions are both supported in answerLoops. You can check out the docs here: answerloops.com/docs/integrations/...
One failure mode this thread hasn't touched yet: language. The manual-entry form publishes straight to retrieval with no draft step, which is already flagged above as risky for accuracy, but even a correct, reviewed answer can become subtly wrong once it's the match returned for a question asked in a different language than it was written in, if retrieval doesn't account for how a technical term or product name gets paraphrased by a non-native speaker. We hit this building a support KB for a product with a large non-English user base: retrieval worked fine, the quoted answer didn't fully apply to that user's setup, and nothing in the usage metrics flagged it until a customer pointed it out manually. Does answerLoops have any visibility into which language a question/answer pair actually serves well, or is that still something a human has to notice after the fact?
Hey @techanalysisinsights, it's a good one to learn from. I appreciate your question.
answerLoops replies in the language the user writes in, and retrieval works by meaning rather than exact wording, so a question in one language can match an article written in another.
Usage tracking records which articles were supplied to each answer, so you can trace any reply back to exactly what it was built from. That's how a case like yours gets diagnosed: you see the match, then check it against the user's setup.
The best way to pressure-test the paraphrased-term problem is with real cases. If you're willing, I would be happy to run an experiment and share the results with you. If you're open to sending me two or three of the non-English questions (mock data is fine) that you noticed didn't line up, along with the terms those users used, and I'll run them against English articles and share what comes back.
Your feedback on the results would be highly valuable, and I'm open to communicating via email @ nathan@answerloops.com
Love this!
I just found this, and I'm intrigued. I'm curious, If you’ve connected an AI assistant to your docs, how do you check that it’s finding the right information?
Hey James, one important detail is adding the correct info to your KB. How I validate is that I start with questions people have actually asked and check which articles come back, not just whether the generated answer sounds right. I want to make sure it's accurate.
In answerLoops, the Knowledge coverage tab runs the same lookup as the website widget, so I can see the matching articles or spot a question with no coverage. Then I review the generated reply separately: did it use that information correctly and actually answer the question?
If something’s missing, I can add the answer directly to the KB and try the question again.
Ok, so on the platform, someone can manually add to the Knowledge Source, then?
That's correct, you can manually add to the KB, so let's say someone on your team is the Product Expert, you can add notes from a meeting notetaker or just type them out, etc. Here's the docs reference: answerloops.com/docs/product/knowl...
I'm also curious if the KB reveals the source because, in my experience, over time the resources become outdated. How does answerLoops handle this?
John, the team can see where articles came from in the Knowledge coverage tab-imported pages, files, synced sources, promoted tickets, or manually added answers. The website widget doesn’t currently display article citations with its replies.
To keep content current, GitHub sources can auto-sync when you push changes to the default branch, and Notion sources can be re-synced. You can also remove outdated articles from the KB.
That handles source updates, but detecting stale information is still a separate problem.
answerLoops doesn’t currently expire articles or rank them by product version. The coverage tab lets you test a question and inspect which articles it retrieves, so your team can check whether it’s finding the right material.
I’d love to hear how you’ve handled this. Feel free to reach out directly if you’d like to try answerLoops-I’d be happy to help you get started and get your feedback.
What about local LLMs - Do you support those, and if so, is it recommended?
Yes, you can use local LLMs with answerLoops. I would just do your research because the answer quality might vary if you're using a small light weight model.
Saving this for later. Looks like a really useful tool for those managing communities.
Thanks Alex, I appreciate the feedback.
Nice article, thanks!
Love that this supports Circle. I will definitely try this out. Can I self host it?
Thanks for the feedback. Yes, it's open source, and you can self host it for free. Nothing paid, ever.
Nice one, I'm definitely checking this out. I would also love to contribute.
Thanks Zender, I appreciate that. Let me know if you need help with any GitHub issues. I've labeled them from
frontendtobackendif that helps.github.com/answerLoops/answerLoops...
I'm always worries of AI slop so when you fix an incorrect AI answer, do you have a way to make that correction useful for the next person?
Hey Luis, in AnswerLoops, automatic replies are off by default, and that's by design. I’d start by reviewing drafts against the KB and seeing how they handle actual community questions before enabling them for an integration or GitHub repo. The important part is trusting that the answer is going to be correct and accurate.
For knowledge-based replies, one agent writes the draft and a second assesses its confidence and whether it fully answers the question. The default threshold is 0.8, and both conditions have to pass. Otherwise, the draft gets escalated for human review.
That score isn’t a guarantee of correctness. I’d use the Knowledge coverage tab to check what the agent can retrieve, fix missing information, and review the resulting answers before trusting it with more automatic replies.
Ok, it sounds like it's worth giving it a shot. Can I self host it?
Yep, and bring your own LLM, even local models.
I've been needing something like this for a while. Really appreciate the share.
Thanks for the feedback Steve. I really appreciate it!