The answer was already in another thread, so I built ThreadSherpa

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

I got the idea from the MLH Discord.

People were asking about account names, missing offer codes and student benefits. Different people, same questions. The useful answer was often already there, just lurking in another thread.

I built ThreadSherpa to take over that bit of repetitive work. A moderator answers a question in the usual way, then right-clicks their reply and selects Apps → Save this answer. When someone asks a similar question in a connected support forum, the bot uses those saved facts to write a short reply for that person, with a link back to the original.

For the new captures, I used a Minecraft support question: which version can join the server? A moderator saves the server's Java version and console restrictions. The next person describes an "Outdated client" error, and ThreadSherpa uses that answer to explain what to change.

Demo

Website and walkthrough · Setup instructions

These are actual interactions in my test server, using separate moderator and participant accounts. Paper Instruments generated the reply and MongoDB Atlas stored the saved answer. ThreadSherpa is an independent project inspired by the MLH community. It has not been deployed in MLH's server.

First, I answer the participant as a moderator, then save that message:

In a new post, the participant describes the same problem in different words. This time, the bot replies automatically and links back to the moderator's answer:

Code

sea-deep / threadsherpa

A Discord support bot that saves moderator answers and writes short, sourced replies for repeat questions.

Built solo. The code is Apache-2.0 licensed, with credits and separate font licenses in the README. There's also a GitLab mirror.

The README covers setting it up locally with Node.js 22+, a Discord bot token, MongoDB and Ollama. The setup guide includes the required permissions and forum setup.

How I Built It

The bot is TypeScript using discord.js. Answers, versions, cases and evidence are all stored in MongoDB. Ollama is used to run the local models and Paper Instruments is an optional reply provider.

The flow is:

Moderator saves a reply
  → preserve their wording, check duplicates, store it in MongoDB

New forum question + thread context
  → retrieve current answers for this server
  → give the relevant evidence to the model
  → check cited sources and links
  → reply in the thread, or leave it in the moderator queue

Retrieval is a mix of exact match, words and local embeddings. Atlas Vector Search can do the ranking of candidates from the database; smaller deployments can also rank the stored vectors within the app. Each server has its own saved answers. Expired entries are excluded from replies.

Saving the answer should be simple

An earlier version of this was way over the top in terms of display. It had the titles repeated, the review details, the various versions and the explanatory labels all around a very brief answer. It also simply re-used the old message too much.

So saving and answering became two separate jobs. The moderator's wording is preserved when it's saved. Generation happens at a later stage, with the new question and its context. In the learning pipeline, the assignment is simply:

canonical_answer: options.adminMessage,

The model generates a new response when a future question matches. Duplicate saves reuse the existing entry.

Moderators need to see what the bot uses

Servers also have their own existing FAQs and the rules are subject to change.

I added /memory, a private panel for administrators and configured moderators. They can search saved answers, view them, change the question wording or answer, update the source, set conditions and expiry, or remove an entry.

The same answer appears in the private panel, where a moderator can read or edit it:

Import FAQs accepts pasted Markdown:

## Can I submit by myself?
Yes, solo projects are fine.

## Do I need a video?
A video is optional.

The moderator previews the import, then hits Save answers. It accepts up to 10 answers and 4,000 characters, skips duplicates and can pick up where it left off after a partial failure.

Editing an answer changes what future questions use. It doesn't send a correction out into all the old threads.

A failed check should not become another message

During testing, the bot posted:

I couldn’t check this. A moderator can help.

In one instance the earlier moderator response had the README guidance that the person required. It was taking up space without actually doing any good.

I changed automatic replies so a queued result stays quiet. The Discord handler only sends the case card if the result is not queue:

if (result.policyResult.action !== 'queue') {
  await thread.send(await this.card(c.case_id));
}

Moderators can inspect waiting questions with /queue. Explicit /ask requests still get a response. A conditional answer that is very narrowly relevant can trigger a clarification and sensitive account actions are directed to human support.

Follow-ups matter too. A clear 'that worked' can close the case without any further intervention. 'That didn't help' is passed to the moderator and the unsuccessful advice is added to the context so that the bot doesn't just keep re-suggesting it.

One limitation from this capture session: an Xbox follow-up stayed queued even though the saved answer covered consoles. The first reply worked, but follow-up matching still needs work.

What I checked

The latest model-free test run passed 140 tests, with 3 live-provider tests skipped. The tests cover routing, approval, duplicate saves, expiry, source checks, FAQ confirmation and private panel access.

An earlier development smoke test used 12 synthetic support cases (paraphrases, a different event with a similar name, expired guidance and a correction). Both the local Ollama run and the Paper run passed those 12 checks. The evaluation script is reproducible via npm run eval:local or npm run eval:paper.

These are developer-written smoke cases, not a community accuracy benchmark. The reports predate the memory panel, which has its own interaction tests.

The Paper evaluation recorded 10 generation calls, 5,408 input tokens and 1,109 output tokens. At the rates recorded on 23 October, that comes to an estimated $0.0013657 for that run. This excludes local embeddings and duplicate checks, hosting and other development requests. It's a usage-based estimate, not an invoice or a cost per user.

I also checked Atlas persistence across separate Node processes, an actual vector query and cross-server filtering with temporary synthetic records, then removed them.

Citation checks establish where an answer came from; they don't prove every sentence is supported. A moderator can save an incorrect answer, and the model can misread a match. The bot can still be wrong.

Why Does Open Innovation Matter?

The default path is Gemma via Ollama with a local embedding model. The prompts and retrieval code are in the repo so someone running the bot could look at an answer, tweak the prompt or pick a different model.

Local generation uses the host machine's resources (no metered inference API bill) and gives the operator the option of whether or not to send context to a hosted model.

I used Paper's Ultramarine for the Discord testing above. That mode sends the selected evidence and question context over to Paper for synthesis, with the embeddings and duplicate checking staying local. The provider interface keeps that choice separate from saving answers and handling Discord interactions.

The bot as a whole is online software (Discord requires the network and this deployment has answers stored in Atlas). Running the model locally doesn't make those parts offline.

The Lazy Payoff

The moderator still decides what the answer is. They write it once and save it. ThreadSherpa then deals with locating it and adapting it when the next person comes along and asks.

I've run that loop on a test server. I haven't measured hours saved in a live community yet. The next test is with a small server's moderators, to see which replies help and which should have waited.

For this week's theme, that's the payoff I'm after: less time spent digging through old threads and less time re-writing the same advice.

Prize Categories

• Best Use of Gemma: the local Ollama path generates replies and handles duplicate comparisons, with local embeddings for retrieval.
• Best Use of MongoDB Atlas: durable saved answers, versions, cases and server-scoped retrieval. Restart persistence and real vector queries were tested separately.
• Best Use of Ultramarine: Paper Instruments generates replies from the selected evidence. The development run's token usage and estimated cost are reported above.
• Best Use of Render: Render hosts the project's explanatory frontend and setup pages at threadsherpa.onrender.com, defined by the repository's Blueprint. The Discord worker runs separately.
• Best Use of GitHub Copilot: the category includes GitHub Actions automation. The Checks workflow checks bot types, runs tests, builds the bot and verifies the fixture flow. It also checks and builds the website. This entry is for Actions; I'm not claiming Copilot wrote the code.

I used AI coding assistants while building ThreadSherpa, and ChatGPT helped draft this post.