DEV Community

Serge Levin
Serge Levin

Posted on • Edited on

Classifying 33K Reddit posts on a laptop: anchor first, exclude nothing

#ai #opensource #llm #machinelearning

Anchor-first prompts: the fix for noisy local-LLM classifiers

If you're using a local LLM as a YES/NO classifier and seeing too many false positives, don't fix it by enumerating off-domain categories to exclude. Flip the prompt to require an explicit named anchor from the target domain - answer YES only when a specific in-domain term appears, otherwise NO. Same prompt shape transfers across model sizes. I've applied it on phi3.5, qwen2.5:7b, and phi4:14b.

Setup

I'm building an agent that watches Reddit (and eventually HN, Lobsters, Mastodon) for posts and comments matching specific signals. Not for lead gen - more for community intelligence. One signal I'm working on: someone comparing or migrating between S3-compatible object storage providers.

Pipeline 

Reddit RSS/JSON  →  keyword pre-filter  →  Bayes classifier  →  LLM (Ollama)
Enter fullscreen mode Exit fullscreen mode

Most posts get rejected at the pre-filter. The Bayes classifier handles the bulk of obvious YES/NO. The LLM calculates Bayes weights on the first run; on later runs it only sees ambiguous cases.

Naive approach

I started with phi3.5:latest (~2 GB on disk, snappy in Ollama). Almost immediately, two false-positive classes appeared with hundreds of posts:

  1. Kubernetes infra posts. Threads like "comparing Kafka deployment options on K8s" got flagged YES even though there's no object-storage angle.
  2. Microsoft Fabric / Copilot / data warehouse posts. "Snowflake vs Fabric for our team".

The smaller model was pattern-matching on the shape of "user comparing options" and dropping the domain anchor entirely.

The first attempt to fix was to add exclusions to the prompt.

Answer YES if the post is about object storage migration.
Answer NO if the post is about Kubernetes.
Answer NO if the post is about data warehouses.
Answer NO if the post is about AI tooling.
Enter fullscreen mode Exit fullscreen mode

The negative categories started acting as their own relevance signal. The model treated "is this Kubernetes-y?" as a classification axis - meaning a Kafka-on-K8s post was now being classified along the wrong dimension entirely. Started to get even more noise. Hard to describe the world by what it isn't.

Next, instead of NO if {growing list of off-domain things}, structure the prompt as: YES only if {short positive list of in-domain anchors} AND {intent clause}. Otherwise, NO. No exclusions.

Also, changed the model to qwen2.5:7b as such prompt appeared to be too heavy for phi3.5. What that looks like for my signal:

Answer YES only if the text explicitly names:
  - S3, or an S3-compatible provider (AWS S3, MinIO, Ceph, Garage,
    SeaweedFS, Backblaze B2, Cloudflare R2, Wasabi, Storj),
  - or a tool for moving data between them (rclone, s5cmd, mc mirror,
    AWS DataSync, Cyberduck, boto3, aws cli),
AND the author is comparing options or planning to migrate.
Otherwise answer NO.
Do not infer. If no such name appears, answer NO.
Output a single token: YES or NO.
Enter fullscreen mode Exit fullscreen mode

No "ignore Kubernetes." Nothing about what NOT to match. The prompt only describes what counts. As a result, false positives dropped significantly.

Model comparison

Model VRAM Verdict
phi3.5:latest ~2 GB Too small to hold the domain anchor reliably even with the positive gate. Dropped.
qwen2.5:7b ~5 GB Large step up. Honored the gate. Fast enough to experiment with the prompt.
phi4:14b ~9 GB Settled here for production. Needed one more prompt iteration.

Bayes classifier in front of the LLM

Most posts get a high-confidence YES or NO and never see the LLM. LLM as an initial classifier saves time compared to manual classification. And as soon as Bayesian filter weights are good enough, only the ambiguous cases are passed to it.

I pushed roughly 33,000 Reddit records through the full pipeline in under an hour on a laptop GPU. LLM-only would've taken so long that manual filtering would've been faster. If a post doesn't name an in-domain thing AND express comparison/migration intent, it's rejected before the LLM is even invoked. This helps save tokens and mirrors the prompt's logic.

Every new signal I've written since follows this shape - anchor first, everything else after.

Seeding

Running experiments on the data needs a lot of seeding. I tried Google's and Bing's search APIs first - both have been shut down. Ended up with the Brave Search API. The free tier was enough to pull more than 30K seed posts and comments.

Code

All of this is part of an open-source project I'm building - AGPL, runs on a laptop. Some restrictions baked in on purpose: no auto-posting and no DM outreach.

The current approach with the Bayes filter is lean. It does incremental retraining, not drift handling. Every 50 LLM labels, the system pulls the entire historical dataset from DB and calculates the weights from scratch. That's the whole adaptation mechanism for now.

What's missing:

  • A label from six months ago counts the same as yesterday's.
  • No distribution-shift detection.
  • No automatic relabeling of old data when prompts change. I added the --reset command to call full retrain, not sure if that's enough in the long run.

What's coming (and more):

  1. Recency-weighted Bayes retraining. Time-decay on label weights so vocabulary drift follows automatically, plus optional pruning of stale examples that disagree with current labels.
  2. Cross-source aggregation. Same signal hitting three different subreddits is stronger than three hits in one sub. Cross-channel multiplier on the score.
  3. HN, Lobsters, Mastodon as additional sources. The same pain showing up across different cultures, wording, and blind spots is a much stronger signal than three hits in one subreddit.

This post was drafted by the author and refined with AI for clarity and structure.

Top comments (0)