B2B Lead Gen on X: My Bot's 68% False Positive Rate

Originally published on AIdeazz — cross-posted here with canonical link.

My B2B lead generation agent, Algom Alpha, currently operates with a 68% false positive rate on X (formerly Twitter). This isn't a boast; it's a hard number from our last 30-day audit, processing 1,240 potential leads. The goal was to find companies actively hiring for AI/ML roles, signaling budget and a need for our production AI agent services. We built it because traditional lead sources were too expensive for a bootstrapped operation on Oracle Cloud.

The Signal: Hiring-Adjacent Keywords

The core hypothesis was simple: companies hiring for specific AI/ML roles are potential clients. They have budget, a problem, and a team. We focused on X because of its public API access (at the time of initial build, before the drastic changes) and the real-time nature of announcements. Our initial keyword list included terms like "hiring AI engineer," "seeking ML specialist," "AI team growth," "data scientist opening," and specific tech stacks like "PyTorch," "TensorFlow," "LLM architect." This list, now refined to 47 terms, is the first filter.

The agent monitors X for these keywords. When a tweet matches, it's not immediately flagged as a lead. That's where the multi-agent system kicks in.

The Scoring Logic: A Three-Agent Pipeline

Algom Alpha isn't a single monolithic agent. It's a pipeline of three specialized agents, orchestrated by a custom Python framework running on Oracle Container Engine for Kubernetes (OKE).

1. Scout Agent (Groq/Mixtral 8x7B): This agent's job is initial tweet classification. It receives the raw tweet and a small surrounding context (up to 5 previous tweets from the same user, if available). Its prompt instructs it to identify if the tweet actually indicates a hiring intent, rather than just mentioning AI. For example, "Our AI team is growing!" is a positive signal. "AI is transforming industries" is not. We use Groq for this stage due to its speed and cost-efficiency for high-volume, low-complexity classification. The output is a JSON object with is_hiring_intent: true/false and confidence_score: 0-1. Tweets below a 0.7 confidence score are discarded.

2. Qualifier Agent (Claude 3 Haiku): If the Scout Agent flags a tweet as hiring intent, the Qualifier Agent takes over. Its task is to determine if the company behind the tweet is a potential B2B client for production AI agents. This involves:

   • Extracting the company name from the tweet or user profile.
   • Searching Google (via SerpAPI) for the company's website and LinkedIn profile.
   • Visiting the website (using requests and BeautifulSoup) to look for "About Us," "Services," "Careers" pages.
   • Analyzing the company's stated industry and size. We're looking for mid-market to enterprise, not solo founders or direct competitors.
   • Checking for existing AI products or services on their website. If they already have a mature AI product suite, they're less likely to need our foundational production AI agent services.
   • The prompt for Claude 3 Haiku includes a detailed rubric: "Is this a B2B company? Is their primary business model compatible with needing custom AI agents for internal operations or new product lines? Do they appear to have an existing AI department or are they just starting to build one? Score 1-10." We use Claude 3 Haiku here for its stronger reasoning capabilities and larger context window compared to Mixtral, allowing for more nuanced website analysis.

3. Synthesizer Agent (Claude 3 Sonnet): For tweets that pass both Scout and Qualifier, the Synthesizer Agent creates a concise lead profile. It aggregates all data points: original tweet, company name, website, LinkedIn, inferred industry, and the scores from previous agents. It then generates a 3-sentence summary of why this company is a good fit and suggests a potential entry point (e.g., "Company X is expanding its ML ops team, indicating a need for scalable deployment solutions. Their recent product launch in [industry] could benefit from an automated customer support agent. Contact their Head of Engineering."). This output is then pushed to a Telegram channel for manual review. Claude 3 Sonnet is used for its superior summarization and synthesis capabilities, ensuring a high-quality, actionable output for the human sales team.

The entire pipeline runs asynchronously. Each agent's output is stored in an Oracle Autonomous Database, allowing for retries and auditing.

The 68% False Positive Rate: Where It Breaks

The 68% false positive rate is a direct consequence of the Qualifier Agent's limitations and the inherent noise on X.

• Misinterpretation of "Hiring": Many tweets use "hiring" in a broader sense ("We're always hiring great talent!"). The Scout Agent, despite its prompt, occasionally misclassifies these. This accounts for about 20% of the false positives.
• Company Irrelevance: The biggest chunk, roughly 40%, comes from the Qualifier Agent. It struggles with:
  • Small Businesses/Startups: Identifying company size from a website is hard. Many small agencies or early-stage startups use enterprise-level language. Our target is mid-market ($10M-$1B revenue). The agent often flags companies far below this threshold.
  • Indirect AI Use: A company might be hiring an "AI specialist" but their core business is, say, graphic design, and the AI role is for internal tools, not a core product. While still a potential lead, it's a lower priority and often a poor fit for our specific production AI agent offerings.
  • Competitors/Partners: Sometimes the agent flags companies that are direct competitors or potential partners, not clients. This requires human discernment.
• Stale Information: X is real-time, but company websites aren't always. A hiring tweet might be for a role filled months ago, or the company's strategic direction has shifted. This contributes about 8% to the false positive rate.

We've tried refining prompts, adding more negative keywords, and increasing the confidence thresholds. Pushing the Scout Agent's confidence threshold to 0.8 reduced volume by 30% but only dropped the overall false positive rate by 5 percentage points, indicating the problem is deeper in the Qualifier stage.

What 'Qualified Lead' Means to a Bot (and to Me)

For Algom Alpha, a "qualified lead" is a JSON object with is_hiring_intent: true, company_is_b2b: true, company_size_inferred: mid-market_or_enterprise, industry_compatible: true, and a synthesized_summary with a score above 7/10.

For me, a human, a qualified lead means a company that:

1. Has a demonstrated need for production AI agents (e.g., actively building an AI team, struggling with manual processes that AI could automate).
2. Has the budget to invest in custom AI solutions (inferred from size, recent funding, or existing tech investments).
3. Is in an industry where our expertise is relevant (e.g., logistics, e-commerce, content creation, not deep-tech AI research).
4. Has a clear point of contact or department that would benefit.

The bot's definition is a proxy for mine. The 68% false positive rate shows the gap. The bot is good at identifying signals, but poor at inferring intent and fit with the nuance a human brings. Each "qualified" lead from the bot still requires 5-10 minutes of human review to confirm genuine interest and fit before any outreach.

The Cost of a "Free" Lead

Running Algom Alpha isn't free, even with Oracle's generous free tier and my own infrastructure.

• LLM Costs:

  • Groq (Scout Agent): ~$0.00000027 per token for Mixtral. Processing ~100k tweets/day (average 20 tokens input + 50 tokens output per tweet) costs about $0.00189/day. Negligible.
  • Claude 3 Haiku (Qualifier Agent): ~$0.00025/1k input tokens, $0.00125/1k output tokens. Each qualification involves ~5k input tokens (tweet, website content, search results) and ~500 output tokens. With 300 qualified tweets/day, this is ~$0.45/day.
  • Claude 3 Sonnet (Synthesizer Agent): ~$0.003/1k input tokens, $0.015/1k output tokens. Each synthesis involves ~1k input tokens and ~200 output tokens. With 100 synthesized leads/day, this is ~$0.003 * 100 * 1 + $0.015 * 100 * 0.2 = $0.30 + $0.30 = $0.60/day.
  • Total LLM cost: ~$1.05/day.

• API Costs:

  • SerpAPI (Google Search): $50 for 5,000 searches. We average 300 searches/day, so ~9,000 searches/month. This is ~$90/month.
  • X API: We use the free tier for now, but this is a constant risk.

• Infrastructure (Oracle Cloud):

  • OKE cluster: Running on 2 OCPUs, 16GB RAM. Free tier for now, but scaling will incur costs.
  • Autonomous Database: Free tier.
  • Object Storage: Free tier.
  • Total infrastructure cost: Currently $0.

Total monthly operational cost: ~$31.50 (LLMs) + ~$90 (SerpAPI) = ~$121.50.

This generates ~100 "qualified" leads per day, or ~3,000 per month. With a 68% false positive rate, that's ~960 genuinely relevant leads per month. Each genuine lead costs ~$0.12. This is significantly cheaper than LinkedIn Sales Navigator ($100+/month for limited leads) or buying lead lists (often $1-$5 per lead). The human review time is the real cost.

Next Steps: Reducing Noise and Improving Fit

My immediate focus is on reducing the false positive rate, specifically in the Qualifier Agent.

1. Fine-tuning: Instead of pure zero-shot prompting, I'm collecting a dataset of ~500 manually classified companies (good fit/bad fit) to fine-tune a smaller, domain-specific model (perhaps Llama 3 8B) for the Qualifier Agent. This should improve accuracy and potentially reduce Claude 3 Haiku reliance.
2. Website Content Analysis: Instead of just looking for keywords, I'll implement more sophisticated text embeddings (e.g., from OpenAI's text-embedding-3-small) to compare the semantic similarity of a company's website content against a corpus of "ideal client" websites. This should help filter out irrelevant industries or business models.
3. LinkedIn Scraper: Integrating a dedicated LinkedIn scraper (with appropriate rate limits and ethical considerations) to get more accurate company size, employee count, and specific role information. This is a higher-risk, higher-reward endeavor due to LinkedIn's aggressive anti-scraping measures.
4. Feedback Loop: Building a direct feedback loop from the human review process back into the agent's training data. Every manual classification (good/bad lead) becomes a new training example.

The goal isn't 0% false positives — that's unrealistic and would likely filter out too many good leads. But getting it under 40% would significantly reduce human review time and make the entire process even more efficient.

Frequently Asked Questions

Q: Why not just use a commercial lead generation platform?
A: Commercial platforms like ZoomInfo or Apollo.io are expensive, often $500-$1000+/month. For a bootstrapped operation, this cost is prohibitive. My current setup costs ~$120/month for a comparable volume of leads, albeit with higher manual review.

Q: How do you handle X API rate limits and changes?
A: We currently operate within the free tier limits, which are restrictive. For higher volume, we'd need to pay for X API access, which is a significant cost. We also abstract the X API calls, so if it changes drastically, we can swap out the data source (e.g., to LinkedIn, Reddit, or job boards) with minimal impact on the agent pipeline.

Q: What's your strategy for avoiding legal issues with scraping?
A: We only scrape publicly available information (tweets, public websites, public LinkedIn profiles). We adhere to robots.txt directives for websites. We do not attempt to bypass authentication or access private data. The primary risk is X's terms of service, which are constantly changing.

Q: How do you prevent your agents from getting stuck in loops or generating irrelevant content?
A: Each agent has strict token limits for output and a timeout. We also implement guardrails in the prompt engineering, explicitly telling the LLM what not to do or say. For example, "Do not invent company names or make assumptions about their business model."

Q: Why Oracle Cloud for a small operation?
A: Oracle Cloud Infrastructure (OCI) offers a very generous Always Free tier, including OKE, Autonomous Database, and object storage. This allows me to build and scale complex multi-agent systems without upfront infrastructure costs, which is critical for a bootstrapped business.

— Elena Revicheva · AIdeazz · Portfolio