In the spirit of Hacktoberfest, with it being the last day of October, I wanted to share my story about open source and building in public.
I'm a DevRel. In my day job, I help people who've never contributed to open source get their first code shipped. So when I started building answerLoops, an open source, self-hostable AI support tool, one thing bugged me. Was my own project actually easy to jump into?
What is answerLoops?
answerLoops turns your knowledge source into answers for your community. It's self-hostable, so you can run it on your own infrastructure with the AI provider you choose, or use the hosted app.
Here's the workflow:
- A question comes in from a community channel like Discord, Slack, Discourse, GitHub, or website chat.
- answerLoops searches your knowledge: your docs, files, GitHub repos, and Notion pages.
- It drafts a reply with the sources attached.
- A second AI agent checks the draft against those sources.
- Your team approves the reply, or you turn on auto-reply for the questions that pass the check.
- Useful resolutions are saved as knowledge, and knowledge gaps show what your docs are missing.
That workflow is why the labels below make sense. frontend work touches the dashboard where a team reviews and approves replies. backend work touches the pipeline and the channel integrations behind it.
Explore answerLoops on ⭐️ GitHub
The project is new, so the hard part right now is getting off the ground and getting people to find it. I posted about it on Reddit and X. It didn't bring in many stars. So I changed where I put my effort. Instead of trying to reach everyone, I focused on developers who want to build something and on making it easy for them to start.
Over the last week, developers started showing up. They sent pull requests, and I merged them. Some built something from an open issue. Others found a problem on their own and fixed it.
I can't prove any of this is cause and effect. I have a handful of data points, and some of them didn't pick up an issue at all. But I did set a few things up beforehand, and I think they're worth writing down for anyone starting out.
1. I labeled issues by skill, not just by type
At first my issues only had the usual labels: enhancement, bug, documentation. That tells a contributor what kind of change it is, but not whether it's for them.
I added two labels: frontend and backend. Right now I have 58 open issues, 41 tagged frontend and 33 tagged backend. Someone can click one label and see only the work that matches their skills, without reading all 58.
I also use good first issue and help wanted. good first issue is the one most people already know. The skill labels are the new part.
Most of the PRs I merged closed an issue with one of these labels, so they look like they're doing something. I'm still careful about how much I claim.
One side effect: only 8 good first issue tickets are open now. Labels only work while there's something behind them, so I'm writing more.
2. I wrote issues like specs
I'd have undersold this one if I'd only written about labels.
The issue my first contributor picked up was tagged frontend, good first issue, and help wanted. It was also written so that someone who had never seen the codebase could start. It had:
- A Problem section: the data was already being collected and stored, but nothing in the UI showed it.
- A Scope section: exactly what to show, where to start looking in the code, and what the wording should be.
- A Not in this issue section: what to leave alone, so nobody drifts into a bigger project.
- Acceptance criteria as a checklist.
It also said it was part of a larger tracking issue, so a contributor could see where it fit.
A vague issue like "show usage on the article page" leaves a newcomer guessing about scope, wording, and what "done" means. A scoped one lets them start. If I had to pick one thing to copy, it's this.
Every issue also ends with a link to the contributing guide, and it's now built into the Issues template.
3. I made the project easy to run
You can't contribute to something you can't start. I rewrote the README to be a short overview, and added a CONTRIBUTING guide plus issue forms for bugs, features, integrations, and docs, so new reports come in with the details I need.
The first version of the setup steps wasn't good enough. It ran the Docker path, then said "or run it natively," and I couldn't tell you which one to pick. If I was confused, a newcomer was too. I rewrote it as three steps: clone the app, set the values in one .env, and run it. Then I ran the native steps myself against a fresh database to make sure they work.
Contributors have already improved the docs from there, including a diagram of the install flow in the self-hosting quickstart.
4. Review was fast, and the PRs made it easy
The PRs came with a clear description of what changed, why, and how to try it. That made them easy to review, and I could merge quickly.
Fast feedback matters. A first-time contributor who waits two weeks for a reply rarely comes back.
5. I said thank you in public
The contributors are credited in the release notes and in the docs, and I thanked them publicly. Open source is mostly volunteers, so it's the least I can do. It also shows the next person what happens when they contribute.
I also posted on LinkedIn and X.
Thanks Aaqibhafeez Khan for the kind words
What I got wrong, or am not sure of
- I don't know if the labels worked 100%. Nothing else about the project changed much, but I also posted about it in a few places, which brings people in on its own.
- Not everyone took a labeled issue. Some contributors found a problem nobody had filed, and the labels can't explain that.
- Return visits are the real test. What matters is whether people come back for a second PR.
-
The supply runs out. Contributors finish
good first issuetickets quickly. Writing good ones is now a regular chore for me.
If you want to try this
- Label issues by the skill needed, not just the type of change.
- Write one issue as a spec: problem, scope, what's out of scope, and acceptance criteria.
- Make sure someone can run the project, and test those steps yourself.
- Review fast.
- Thank people in public.
If you're new to open source and want help with a first PR, leave a comment or message me. I'm happy to walk you through it. The open issues are here: https://github.com/answerLoops/answerLoops/issues
And if you maintain a project: what got your first outside contributors to show up?
P.S. If you're reading this, starring ⭐️ the repo also really helps people find it :)
Thanks for reading!
Top comments (11)
Separating the drafting agent from the verification check against retrieved sources keeps hallucinations from slipping into the auto-reply path. The part that usually trips this pipeline in production is the feedback loop for resolutions. When a resolution gets saved back into knowledge, stale answers or outdated workarounds from old issues can poison later retrievals. Pinning a freshness timestamp or tying each saved chunk to its git commit sha helps purge obsolete answers when documentation updates.
Hey @reidmarlow, thanks for this. Stale answers coming back through retrieval is something we are actively working on.
Here's what we have today. Each team's knowledge base is seperated by org, and a connected GitHub repo is one source feeding it. When a team re-syncs that repo, answerLoops rebuilds that repo's content from the current files and swaps it in for the old copy. So if a doc was deleted or rewritten, the old answer from it is gone after the next sync. We don't track the commit SHA, though. It reads the latest version and stores when it last synced, so your SHA idea is a real improvement.
Saved resolutions are also on the list. A person has to review and save them, and each one keeps a link back to its ticket. An owner or admin can also mark an article verified, and if anyone edits the text afterward, it shows as changed since it was verified. But nothing flags one that has simply aged, and a repo re-sync never touches them, so an old workaround can sit there until someone notices.
What we do have is tracing. Every answer records which articles it used, and every article shows which answers used it, so a bad answer leads back to the article behind it.
Your comment turned into an issue. It covers recording the commit SHA with each sync, showing how old each saved resolution is, and flagging ones saved before the latest docs sync: github.com/answerLoops/answerLoops...
If you have a better rule for tying a resolution to the docs it came from, I'd like to hear it. That's the open question on the issue.
Labeling issues by skill, not just type, is a small change with a clear payoff. A newcomer can tell at a glance whether the work fits them.
That is so true @robertpaschal!
Appreciate that, Nathan. Skill labels make the first step a lot less guesswork for a newcomer.
Glad to hear your project is getting contributors. That's a great validator when someone lands on your GitHub, and it shows that it's clearly maintained and has people contributing to it.
Nice job!
Thanks Fliin, I am very open-source-centric, and that is my thought exactly, plus I love building in public with other developers who have great ideas, and add feature requests.
Nice walkthrough of how you started boosting your project on GitHub. Thanks for sharing!
Thanks David, yeah, it's something I'm really passionate about. I appreciate the feedback!
Nice article.
What happens if the second AI agent fails or returns nothing? Could an unchecked answer go out automatically?
Hey Andrew, No. Auto-reply only happens when two things are true. Auto-reply has to be turned on for that channel / integration, and the checker(2nd AI Mastra agent) has to score the draft at or above the confidence bar. The default bar is 80%, and a team can set its own per channel.
If the check itself fails or comes back empty, the draft is scored as zero confidence and goes to a person to review. It never goes out unchecked.