In the spirit of Hacktoberfest, with it being the last day of October, I wanted to share my story about open source and building in public.
I'm a...
For further actions, you may consider blocking this person and/or reporting abuse
Separating the drafting agent from the verification check against retrieved sources keeps hallucinations from slipping into the auto-reply path. The part that usually trips this pipeline in production is the feedback loop for resolutions. When a resolution gets saved back into knowledge, stale answers or outdated workarounds from old issues can poison later retrievals. Pinning a freshness timestamp or tying each saved chunk to its git commit sha helps purge obsolete answers when documentation updates.
Hey @reidmarlow, thanks for this. Stale answers coming back through retrieval is something we are actively working on.
Here's what we have today. Each team's knowledge base is seperated by org, and a connected GitHub repo is one source feeding it. When a team re-syncs that repo, answerLoops rebuilds that repo's content from the current files and swaps it in for the old copy. So if a doc was deleted or rewritten, the old answer from it is gone after the next sync. We don't track the commit SHA, though. It reads the latest version and stores when it last synced, so your SHA idea is a real improvement.
Saved resolutions are also on the list. A person has to review and save them, and each one keeps a link back to its ticket. An owner or admin can also mark an article verified, and if anyone edits the text afterward, it shows as changed since it was verified. But nothing flags one that has simply aged, and a repo re-sync never touches them, so an old workaround can sit there until someone notices.
What we do have is tracing. Every answer records which articles it used, and every article shows which answers used it, so a bad answer leads back to the article behind it.
Your comment turned into an issue. It covers recording the commit SHA with each sync, showing how old each saved resolution is, and flagging ones saved before the latest docs sync: github.com/answerLoops/answerLoops...
If you have a better rule for tying a resolution to the docs it came from, I'd like to hear it. That's the open question on the issue.
Labeling issues by skill, not just type, is a small change with a clear payoff. A newcomer can tell at a glance whether the work fits them.
That is so true @robertpaschal!
Appreciate that, Nathan. Skill labels make the first step a lot less guesswork for a newcomer.
Glad to hear your project is getting contributors. That's a great validator when someone lands on your GitHub, and it shows that it's clearly maintained and has people contributing to it.
Nice job!
Thanks Fliin, I am very open-source-centric, and that is my thought exactly, plus I love building in public with other developers who have great ideas, and add feature requests.
Nice walkthrough of how you started boosting your project on GitHub. Thanks for sharing!
Thanks David, yeah, it's something I'm really passionate about. I appreciate the feedback!
Nice article.
What happens if the second AI agent fails or returns nothing? Could an unchecked answer go out automatically?
Hey Andrew, No. Auto-reply only happens when two things are true. Auto-reply has to be turned on for that channel / integration, and the checker(2nd AI Mastra agent) has to score the draft at or above the confidence bar. The default bar is 80%, and a team can set its own per channel.
If the check itself fails or comes back empty, the draft is scored as zero confidence and goes to a person to review. It never goes out unchecked.