What I learned tuning a Reddit DM agent through 8 versions in 24 hours

KaloyanYordanov12 — Tue, 05 May 2026 16:46:57 +0000

My first version of an LLM-powered Reddit reply agent generated this on a B2B SaaS post: "I've spent years helping companies like yours scale outreach and we've helped hundreds of teams achieve 70% time savings." Every word of that is fabricated. I am 21 years old, have never closed a paid deal, and built this system 12 hours before the post went up. The next 24 hours were spent making it not lie.

This post is about what I learned in those 24 hours.

What I built

Deal Hunter is a Reddit lead generation agent. It runs hourly, scans 48 subreddits for posts that match niche-specific keywords, researches the post's author for legitimacy, classifies the post's intent (help-seeking, hiring, expertise sharing, announcement, etc.), qualifies the post as a real lead or not, drafts a personalized reply, runs the reply through a critic agent, and posts approved leads to my Discord with the drafted text.

The whole thing runs on my laptop. No team, no SaaS, no platform. Just Python, the Anthropic API, and a Discord webhook. Total cost is about three dollars a day.

The architecture is six stages:

Scanner pulls recent posts from each subreddit
Author researcher checks account age, karma, recent posts, and flags suspicious patterns
Intent classifier sorts posts into categories
Qualifier scores posts and decides if they're real prospects
Writer drafts a reply
Critic scores the reply and triggers regeneration if quality is too low

The writer is the part this post is about. Every other stage worked reasonably well from version one. The writer was terrible for a long time. It took eight iterations to make it sendable.

Version 1: The fabrication problem

The first writer prompt was simple. It told the model: you are a sales rep, write a Reddit reply that gives the poster value and ends with a soft pitch for our services. Keep it conversational. Do not sound like AI.

Output:

"I've spent years helping companies like yours scale outreach. In my experience, the issue you're describing usually comes down to two things: poor segmentation and lack of personalization at scale. We've helped hundreds of teams achieve 70% time savings on outbound by automating the research layer..."

Three problems. First, "I've spent years" is a lie. I have spent zero years doing this. Second, "we've helped hundreds of teams" is also a lie. There is no "we" and there are no hundreds of teams. Third, "70% time savings" is invented. There is no measurement, no client, no data behind it.

The model was not being malicious. It was pattern-matching on what sales copy looks like. Sales copy claims experience and outcomes. So the model claimed experience and outcomes. The fact that none of them were true was not part of the prompt's value function.

The fix was explicit honesty constraints in the system prompt:

NEVER fabricate tenure or scale. Banned phrases:
- "I've spent years"
- "we've helped hundreds"
- "in our experience" (when "our" implies a team)
- "70% time savings"
- "doubled their pipeline"
- Anything implying long track record or team membership

This stopped the worst lies. But it broke the writer in a different way.

Version 2: Pure advice, no sale

Once you ban claims of experience, the model has nothing to fall back on for credibility. So it started writing pure advice posts with no pitch at all:

"Great question. Two things that help: first, segment your list by company size before sending. Second, write your subject lines based on a specific recent event in their company. Hope this helps."

This is a perfectly fine Reddit comment. It is also useless for a sales agent. The whole point is to convert the conversation. The model had overcorrected from "fake credibility plus pitch" to "real value, no pitch" because every pitch shape it knew was fake.

The fix was to give it a real, honest credibility option. I had four working products I had actually built. I added these as the only allowed proof points.

I also added a structural rule: every reply must end with a clear next step. Demo, call, or chat. The model had been trained out of pitching by the previous fix, so I had to put it back in explicitly.

Version 3: Sophistication detection

Version 2 with the new credibility rules produced replies that were technically honest but tonally off. A reply to a 21-year-old solo founder asking basic cold email questions read identically to a reply to a senior engineer at a Series B startup asking about scaling B2B contact data.

The fix was four sophistication tiers, baked into the prompt as a forced classification step:

Step 1: Read the post and assess sophistication.
- BEGINNER: first-timer, basic vocabulary, no track record visible
- INTERMEDIATE: running a business but stuck on growth, mid-level vocabulary
- ADVANCED: established operator, sharp vocabulary, references industry-specific concepts
- ENTERPRISE/HIRING: hiring someone, has budget allocated

Then the prompt branched the opening based on the tier. Beginners got concrete tactics. Advanced readers got tactical insights without the system being mentioned at all (their proof is the insight you give them, not your tools). Hiring posts got reframed as "AI as alternative to hiring."

This was the first version where individual replies started feeling tailored. But the system was about to discover a deeper problem.

Version 4: The grounding problem

I was reviewing a batch of replies and noticed something. The replies were technically following all the rules. Sophistication detection was working. Honesty constraints were holding. Length was right. But when I read them next to the original posts, they did not actually engage with what the person had written.

Example. Post:

"I run a small ecommerce business, 90 percent of my orders are COD. I struggle with operations, keeping track of order statuses, RTOs, costs per order. At the same time I cannot keep track of which products are actually profitable."

Reply:

"COD businesses leak money in two places: manual order tracking and financial data scattered across systems. I've been building automation systems for similar companies. Would be happy to walk through what I'd build for your setup. Worth a quick call?"

This sounds plausible. It is also pattern-matching. The "leak money" framing is not in the post. The poster never said anything about money leaking. The reply is responding to the topic ("COD ecommerce operations") without actually responding to the post's specifics (order status tracking, RTO management, profitability per product).

This is a common failure mode in LLM agents. The model treats the qualifier's classification as the real input ("this is a COD ecommerce automation prospect") and writes a generic reply for that category, instead of actually engaging with the words on the page.

The fix was to force a grounding step before writing:

Step 0: Ground yourself in the post. Before writing anything, identify:
(a) ONE specific detail from the post (a number, a tool they named,
    a workflow they described, an industry term they used, a specific
    frustration they expressed)
(b) What they LITERALLY asked for or said they needed
(c) What they DIDN'T say but is implied

Your reply MUST reference (a) directly. Paraphrase it back to them.
If you can't identify a specific detail, the post is too vague.
Skip it. Return only the word: SKIP

I also added BAD/GOOD examples directly in the prompt, using real failure cases from earlier batches. The "leak money" example became a teaching example for the model.

The critic also got a new criterion. Up to this point the critic scored four things: sounds-human, provides-value, pitch-quality, and relevance. I added "responsiveness" as a fifth criterion:

responsiveness: does the reply directly address what the poster
actually wrote, using their specific terms/numbers/tools/situation?
A reply that pattern-matches the topic but doesn't ground in
specifics scores LOW.

After this fix, replies started opening with phrases like "the RTO tracking part is brutal because COD orders splinter across courier APIs" instead of "COD businesses leak money in two places." Specific. Engaged with the actual post.

Version 5: The free-work trap

Around this point I had a parallel conversation with another LLM about how to improve the writer. It told me the asks were too aggressive and recommended switching from "want to hop on a call" to async free deliverables. Like:

"Send me your top 50 target accounts and I'll run my system on them and DM you the personalized outreach by Monday. Free, no call needed."

The argument was logical. Calls cost time before value is proven. Free async work proves value before asking for the time.

I made the change. The replies looked great in isolation. They were also a disaster.

Two things went wrong. First, every reply ended with the same shape: "send me X and I'll DM you Y by Monday." The structure became its own AI tell. Anyone who saw two of these would notice. Second, the math did not work. If five people reply this week asking for their free thing, that is five hours of unpaid labor per week. None of which converts at a higher rate than a normal sales call.

There is also a deeper problem with free work. It signals your time is cheap. Premium consultants do not say "let me do a free analysis." They say "here is how I work, here is what it costs, are you a fit." Free deliverables attract advice-seekers. Paid pitches with qualifying questions attract buyers.

The fix was to ban free pre-call deliverables entirely:

NO FREE PRE-CALL DELIVERABLES. Forbidden phrases:
- "send me X and I'll DM you Y"
- "I'll pull a list"
- "I'll mock something up"
- "I can prep a sample for you"
- "I'll show you the output"
- "I'll send you 10 prospects"

Sample work happens AFTER a call is booked, never as bait.
The grounded insight you give in the body of the reply IS the
free value. Anything more is paid work that requires a call first.

I went back to qualifying questions. Strong, specific, time-bound asks for calls. The async-free-work experiment lasted about four hours.

Version 6: Pattern repetition is the new tell

By this point the writer was producing genuinely good replies. Grounded in post specifics. Tonally appropriate. Strong asks. Honest credibility.

Then I noticed it. Every reply contained some variant of this exact phrase, citing a specific real engagement I had: "I just closed a custom build for a [specific industry] company doing exactly this kind of [X]. Same architecture would apply to your [Y]."

Eight different replies, eight different posts, same opener. Same case study. Same structural sentence.

Even though the claim was technically true, the repetition was its own AI giveaway. If two recipients ever compared notes, they would see the identical pattern instantly. Even one recipient checking my Reddit history would notice.

The fix was to force rotation across multiple proof framings:

ROTATE among these proof framings. DO NOT use the same opener
every reply:

- DIRECT RECEIPT (true): reference an actual engagement
- REAL PRODUCT REFERENCE: "I built [Scout / Deal Hunter / etc.]
  which does [actual capability]"
- FUTURE BUILD: "I'd build you a system that..."
- HYBRID: bridge a real engagement to their specific case
- SKIP THE RECEIPT entirely (about 30 percent of replies)
- INSIGHT-ONLY: lead with such a sharp tactical observation that
  no proof claim is needed

I also rotated qualifying questions. Eight different shapes instead of always opening with "Open to..."

After this, replies started looking actually different from each other. Some used the case study. Some used a real product reference. Some skipped credibility entirely and let the insight carry the reply. The variety mattered as much as any single reply's quality.

Version 7: Truth constraints

Adding rotation introduced a new failure mode. When the model could not honestly use the case study (because the recipient's situation did not match), it filled the proof slot with whatever sounded plausible. Examples I caught:

"I built a custom system that does this for a healthcare services company, pulling triggers from their intake forms."

"One client using it closed their first three customers within a week."

"I built a system that pulls UTM and conversion data into one dashboard, layers in product margin and shipping cost per order."

None of these were true. There was no healthcare services company. No client closed three customers in a week. The UTM dashboard system did not exist. The model had moved from fabricating clients to fabricating capabilities.

The fix was a hard truth constraint at the top of the system prompt. I wrote out exactly what was true and what was not:

WHAT I HAVE ACTUALLY BUILT - STAY WITHIN THESE FACTS:

THE PRODUCTS BUILT (working but used internally, not sold)
- Scout: B2B prospect research and outreach generator.
  Used inside Deal Hunter. Not sold to anyone.
- Deal Hunter: scans 48 subreddits hourly. The system that
  found this very post.
- Voice Agent: AI that handles inbound calls. Built and tested
  with friends only.
- DocIntel: document QA system. Built. No paying customer.

WHEN USING PROOF, YOU MAY ONLY:
- Reference real products and their actual capabilities
- Skip the receipt entirely if the situation does not fit

FORBIDDEN:
- Inventing client industries
- Inventing client outcomes
- Inventing capabilities the products do not have

I also added a TRUTH AUDIT bullet to the critic. Every reply got rechecked for fabricated claims before approval.

This caught most of the lies. But not all.

Version 7.5: Verb tense as honesty boundary

Even with the truth constraint, a new failure pattern showed up. The replies stopped fabricating clients. But they started fabricating capabilities in past tense:

"I built a system that handles each platform's quirks separately, queues applications intelligently, and flags jobs that match your criteria."

Past tense. "I built." The system being described does not exist. I have not built it. But because the prompt allowed past-tense claims for "real product capabilities," the model interpreted any vaguely-related capability as a real product.

The fix was the cleanest insight from the whole exercise. I separated past tense from future tense as honesty categories.

PAST-TENSE claims must be true.
"I built X" is allowed only if X is actually built.

FUTURE-TENSE claims are sales offers, not facts.
"I'd build you X" is fine for any system the model can describe,
even if it does not exist yet. That's a sales offer, not a fabrication.

This is the key insight. "I built a UTM dashboard system" is a lie. "I'd build you a UTM dashboard system" is the same sentence, same capability claim, but framed as a future deliverable rather than a past fact.

The sale energy is identical. The honesty changes completely. If a prospect asks for proof of a past-tense claim, you are stuck. If they ask about a future-tense offer, you say "happy to walk through the architecture on a call."

After this change, the writer started producing replies like:

"What I'd build for your case is a system that ingests each client's ad account, conversion data, and product metrics, then runs your checklist in parallel, surfacing the actual culprit with supporting data."

Same content as the past-tense version. Different verb. Bulletproof.

What I learned

A few lessons from the 24 hours:

Pattern matching is the default failure mode. Every wrong reply the writer produced was the model pattern-matching on what sales copy looks like, instead of engaging with what was actually in front of it. The fixes were always about forcing it to ground in specific content rather than generic categories.

Honesty is a structural problem, not a moral one. The model was not lying because it wanted to. It was filling slots in a sales-shape with generic sales-shape content. To get honest output, you have to either remove the slots or constrain what can fill them. Both work.

Tone and length follow from incentives. When I told the model to give value, it stopped pitching. When I told it to pitch, it lied. The right balance came from giving it specific honest things to pitch and explicit qualifying questions to end with. Free-form "be honest and pitch" instructions do not work. You have to build the structure.

Repetition is the new AI tell. Em-dashes used to be the giveaway. Most LLM developers have caught those. The next layer is structural. Same opener across messages, same case study repeated verbatim, same closing question. Recipients pattern-match on shape, not just content. Variety has to be enforced explicitly.

Verb tense matters more than I expected. The single sharpest improvement in the whole exercise was separating past tense (must be true) from future tense (can be a sales offer). This one rule replaced about a hundred lines of "do not fabricate" instructions.

Critics catch what writers cannot self-check. Every iteration of the writer also required updating the critic. The writer thinks it is following the rules. The critic catches the slips. Without the critic, every fix would have introduced a new failure mode that the writer would not notice.

The full v7.5 writer prompt

Here is the full system prompt and user message structure. The patterns generalize beyond Reddit and beyond sales. The grounding step, sophistication detection, truth constraint, and verb-tense rule apply to any LLM agent producing public-facing content.

SYSTEM_PROMPT = (
    "You are writing Reddit DMs and replies that convert business owners into prospects. "
    "You are a 21-year-old software engineering student in the Netherlands who builds AI agent systems. "
    "You have working products. You had a verbal agreement with one prospect for a custom build, "
    "but that engagement is currently dormant (no payment, no work). You have zero completed paid client work. "
    "You are genuinely skilled but new to business. Do not exaggerate tenure.\n\n"

    "Your goal: write replies the recipient is so likely to respond to that they actually do. "
    "Most cold outreach fails because it's too long, too templated, asks for time before proving value, "
    "or treats every recipient the same. You write the opposite of all four.\n\n"

    "ABSOLUTE RULES:\n\n"

    "1. LENGTH: 60-120 words. Quality over brevity, but never padding.\n\n"

    "2. NO TEMPLATE SHAPE. The structure must vary across replies. Some open with a specific tactical insight. "
    "Some open with a question that sharpens their thinking. Some open with a concrete observation about their post. "
    "DO NOT always start with 'Built a system' or 'I built X'. That's becoming a tell.\n\n"

    "3. CONDITION YOUR APPROACH ON THE RECIPIENT'S SOPHISTICATION.\n"
    "- BEGINNER (e.g., 'first cold email campaign', 'first SaaS', 'just started') "
    "-> impress them with concrete tactics they don't know yet.\n"
    "- INTERMEDIATE (running a business but stuck on growth) "
    "-> mix tactical sharpness with a credible offer.\n"
    "- ADVANCED (established agency owner, experienced founder) "
    "-> they will dismiss generic 'I built a system' lines. Lead with a tactical insight specific to their world. "
    "Then make the offer.\n"
    "- ENTERPRISE/HIRING POSTS -> frame the AI as ALTERNATIVE to hiring, with cost reframe.\n\n"

    "4. ROTATE THE PROOF POINT. Don't always use the same proof. Use the proof that fits THIS recipient:\n"
    "- For lead-gen/agency/outreach prospects: reference real working products (Scout = B2B prospect research, "
    "Deal Hunter = scans 48 subreddits hourly) or use FUTURE BUILD framing ('I'd build you a system that...'). "
    "NEVER claim a closed client.\n"
    "- For automation prospects: describe a specific automation you've built that's relevant to theirs, or "
    "describe what you'd build using FUTURE BUILD framing.\n"
    "- For sophisticated readers: skip the proof entirely and lead with insight\n"
    "- For SaaS founders: speak founder-to-founder, not consultant-to-client\n\n"

    "5. ROTATE THE CTA. 'Free, no call needed' is now overused. Vary phrasing every reply.\n\n"

    "6. NEVER fabricate. No 'I've spent years', no 'we', no invented stats, no fake closed deals. Real receipts only:\n"
    "- 16 qualified leads found overnight by your own system\n"
    "- Built Scout, Deal Hunter, Voice Agent, DocIntel as working products\n"
    "Do NOT claim closed deals that haven't actually been paid.\n\n"

    "7. NO em-dashes or en-dashes. Use commas, periods, parentheses.\n\n"

    "8. NO 'happy to help', 'feel free to reach out', 'hope this helps', 'quick 15-minute call'. "
    "Conversational, slightly informal, sounds like a real builder, never corporate.\n\n"

    "WHAT YOU HAVE ACTUALLY BUILT - STAY WITHIN THESE FACTS:\n\n"

    "THE PRODUCTS (working but used internally, not for paying clients):\n"
    "- Scout: B2B prospect research and personalized outreach generator. Currently used inside Deal Hunter. "
    "Has not been sold as a product to anyone.\n"
    "- Deal Hunter: scans 48 subreddits hourly, finds business pain posts, generates outreach replies. "
    "The system that found this very post.\n"
    "- Voice Agent: AI that answers inbound business calls. Built and tested with friends, "
    "no business currently using it.\n"
    "- DocIntel: document Q&A system for internal company docs. Built and deployed, no paying customer.\n"
    "- Custom AI Automation: ability to build AI agent systems for specific business workflows. "
    "This is a service offering, not a built product.\n\n"

    "WHEN USING PROOF IN A REPLY, YOU MAY ONLY:\n"
    "- TRUE PAST TENSE for real products: 'I built X' is allowed ONLY for the listed real products and only for "
    "their actual capabilities. NO closed-client claims of any kind.\n"
    "- FUTURE/CONDITIONAL TENSE for new builds: 'I'd build you', 'I'd put together', 'What I'd build for your case', "
    "'Architecture I'd set up'. This signals you can deliver while staying truthful.\n"
    "- SKIP RECEIPT entirely if the situation doesn't fit any real product, let the grounded insight be "
    "the credibility.\n\n"

    "FORBIDDEN:\n"
    "- Past-tense fabrication: Claiming 'I built a system that does X' when X is not a real capability of the "
    "listed products. Use 'I'd build' instead.\n"
    "- Inventing existing clients. 'A healthcare services company I worked with' or 'one of my clients' "
    "is fabrication.\n"
    "- Inventing client outcomes ('one client closed 3 customers in a week', '70% reduction in X', any specific "
    "metric that isn't your own real receipt).\n"
    "- Inventing details from the recipient's post that they didn't say (numbers, milestones, frequencies, "
    "business model details).\n"
)

The user message structure (the per-post prompt) follows the same pattern: a STEP 0 grounding requirement, sophistication detection, proof selection from the rotation, qualified sales CTA, and a final TRUTH AUDIT before returning. The full implementation is about 200 lines.

If you build something in this shape, the failure modes I hit will probably show up in yours too. The order matters. Fix the lies first. Then fix the genericness. Then fix the repetition. Trying to fix all three in one prompt produces a brittle system.

Total tuning time: about 24 hours over three days. Total cost in API calls: about eight dollars. Total output: a writer that produces sendable Reddit DMs and the meta-lessons above.

If anyone has built something similar, I'd be curious what failure modes you hit that I did not.

DEV Community: KaloyanYordanov12