DEV Community: Larry Johnson

How to scrape business contact details and verify the emails in one step

Larry Johnson — Mon, 15 Jun 2026 12:53:48 +0000

If you do cold outreach, you have hit this wall: you scrape a list of company sites, get a pile of emails, load them into your sequencer, and a third of them bounce. Your sender reputation drops, and now even the good ones land in spam. The usual fix is two tools: one to scrape, one to verify. Here is how to do both in a single pass so the list that comes out is already usable.

Why the two-step way leaks

A plain contact scraper returns whatever matches an email pattern on the page. On a real business site that means you also collect placeholders from the template (info@example.com), demo addresses, a Sentry error-tracking key that looks like an email (key@o0.ingest.sentry.io), git@github.com from a code snippet, and third-party addresses smuggled into a mailto:?cc= link. Then you pay a second service to verify the whole dirty pile, including the garbage. The waste compounds.

The better approach is to validate as you extract, and only keep what survives.

Step 1: extract cleanly, not greedily

The trick that removes most of the noise is to stop trusting raw HTML attributes. Real contact emails live in visible text or in JSON-LD structured data. Tracking params, image filenames, and dev keys live in attributes and scripts. So strip executable scripts and blank out attribute values before you run the email regex, and take mailto: addresses only up to the ? (never the query string):

import re

# blank href/src/content attribute VALUES so ?cc=, ?ref=, data-url junk can't match
ATTR = re.compile(r'(href|src|content|data-[\w-]+)\s*=\s*("[^"]*"|\'[^\']*\')', re.I)
clean = ATTR.sub(r"\1=''", html_without_scripts)
emails = set(re.findall(r"\b[A-Za-z0-9._%+\-]+@[A-Za-z0-9.\-]+\.[A-Za-z]{2,}\b", clean))

Then drop known service domains (sentry.io, github.com, googleapis.com) and placeholder locals (name@, you@, jane.doe@). What is left is candidate contacts, not soup.

Step 2: verify each one without sending

You do not need to send an email or even open an SMTP connection to catch most bad addresses. Resolve the domain's MX records. No MX (and no fallback A record) means mail cannot be delivered. Flag disposable domains by suffix match, and flag role inboxes (info@, sales@). Score each address 0 to 100 and assign a verdict:

import dns.resolver
def deliverable(domain):
    try:
        mx = dns.resolver.resolve(domain, "MX")
        return any(str(r.exchange) not in (".", "") for r in mx)
    except Exception:
        return False

Cache the result per domain. A list of 5,000 emails often has only a few hundred unique domains, so you resolve each once.

The state most tools get wrong

Two-state verification (valid or invalid) quietly corrupts your list. A DNS timeout is not proof an address is dead, it is proof you could not check right now. Keep a third state: unknown. Mark it, do not drop it, and never charge for it. The three buckets are valid (deliverable, keep), invalid (no MX, drop), and unknown (retry later). That third state is the difference between a tool that cleans your list and one that silently deletes real leads.

What good output looks like

Run the combined pipeline on a real site and you get the deliverable contact surfaced on top, scored, with the noise already gone. For example, scanning basecamp.com returns jason@basecamp.com with a valid verdict and a 100 score, while the demo and service addresses that a naive scraper would have mixed in are dropped before they ever reach you. One pass, one clean row per company.

One honest caveat

There is no SMTP handshake here, and that is on purpose. From shared datacenter IPs, SMTP probes get greylisted or blocked and return unreliable answers, and the attempt can get your IP flagged. So this is DNS-tier confidence: it proves the domain accepts mail, not that one specific person's mailbox exists. A real but fictional demo address on a real domain can still pass. That is the honest ceiling of verify-without-sending, and it still removes the large majority of bounces.

The shortcut

If you would rather not wire up the crawler, the noise filters, the DNS cache, and the three-state logic, I packaged the whole find-and-verify pipeline into one tool. You feed it a list of business sites and it returns deliverable, scored contacts in one run, with validation on by default and failed fetches never billed. It is the Lead-Gen Pipeline Pro on Apify. Either way, validate before you send, and make sure your tool keeps that third state.

Giving an AI agent live prediction-market data with an MCP server

Larry Johnson — Mon, 15 Jun 2026 12:30:55 +0000

If you are building an AI agent, the hardest part is rarely the model. It is getting the model trustworthy, current data at the moment it needs to act. A language model trained months ago has no idea what Polymarket is pricing a question at right now. The Model Context Protocol (MCP) is how you close that gap: it lets an agent call external tools mid-reasoning. Here is how that works for prediction-market data, and why it matters.

What an MCP server actually is

An MCP server exposes a set of tools, each with a name, a description, and typed inputs and outputs. An MCP-aware client (Claude, Cursor, an n8n flow, a custom agent) reads that list and can decide, on its own, to call a tool when it needs something it does not know. The agent sends the inputs, your server runs, and the result comes back into the model's context as a fact it can reason over.

So instead of hardcoding an API call into your app, you publish a tool like get_market_odds(question) and any agent that connects can use it. The agent figures out when to call it.

Why prediction markets are a good fit

Prediction markets are a live, numeric, fast-moving signal about real-world questions: elections, rate decisions, sports, company events. That is exactly the kind of thing an agent gets wrong from training data alone. A few patterns this unlocks:

A research agent answering "how likely is X" can cite the current market-implied probability instead of guessing.
A trading or monitoring agent can watch for a market repricing and act on the move.
A writing agent can ground a claim in a live number rather than a stale one.

The point is the agent stays current without you rebuilding the integration every time.

What a good prediction-market MCP tool returns

The tools that are actually useful to an agent are small and composable. A sensible set:

list or search active markets by keyword
get the current price and implied probability for a market
get recent large trades or volume for a market
get a wallet or trader's history for a market

Each returns clean structured JSON the model can read directly. The design rule is to keep each tool doing one thing, so the agent can chain them.

The honesty constraint that matters for billing

If you charge per tool call (which is the natural model for an agent buyer), there is one rule that builds trust: never charge for a call that failed to return real data. If the upstream is rate-limited or unreachable, the call should fail loudly, not return a plausible-but-empty result that the agent treats as fact and you bill for. An agent acting on fabricated empty data is worse than an agent that knows the call failed. Bill for delivered answers, never for the attempt.

How agents connect

An MCP server published in a registry (the official MCP registry, Glama, Smithery) is discoverable by agent builders, and connecting is usually a one-line config: point the client at the server URL or run command, and the tools appear. No SDK to learn, no per-app integration. That low friction is the whole reason the protocol is spreading.

The shortcut

If you want live prediction-market data in your agent without building and hosting the server yourself, I published one that exposes Polymarket market data as native MCP tools, billed per tool call, with the honest-billing rule above baked in (a call that cannot return real data is not charged). It is in the official MCP registry and on Apify as the Polymarket MCP Server. Point your agent at it and it can read the markets the same way it reads anything else you give it.

The billing state most APIs get wrong: "unknown" is not "no"

Larry Johnson — Sun, 14 Jun 2026 13:55:12 +0000

If you bill per result, there is one design decision that quietly decides whether customers trust you: what you do when you could not get an answer.

Most usage-billed APIs collapse the world into two states. The call worked, or it failed. Valid or invalid. Found or not found. That binary is where the trust leaks out, because it hides a third state that happens constantly in the real world: you asked, and the system genuinely could not tell you right now.

A concrete example

Say you run an email verification endpoint. A customer sends an address, you check it, you bill for the answer. Easy.

Then a DNS lookup times out. Or the domain returns SERVFAIL. Or the mail server greylists your probe and hangs. What do you return?

The lazy answer is invalid, because it is not valid, and now the row is "done" and billable. That single shortcut does two bad things at once. It deletes a real lead from your customer's list, because a timeout is not proof the address is dead. And it charges them for the deletion. They paid you to damage their own data.

The honest version keeps three states:

valid        -> we confirmed it. billable.
invalid      -> we confirmed it is not deliverable (no MX, etc). billable.
unknown      -> we could not determine it right now. NOT billable.

unknown is the whole game. It says "you asked, we tried, we could not be sure, so we are not going to charge you or pretend." It is the difference between a tool that cleans a list and a tool that silently corrupts it.

This generalizes past email

The same gap shows up anywhere you bill per result:

A scraper hits a page that 404s. Is that "no contact found" (a real answer) or "we could not load the page" (an unknown)? Bill the first, never the second.
A profile lookup gets rate-limited. The lazy path returns an empty profile that looks exactly like a real "this person has no data." Now your customer cannot tell a true empty from a failure, and you billed for both.
A price checker can not reach a vendor. Returning "no price" reads as "free or unavailable." Returning unknown reads as "try again." Very different downstream.

The rule I have settled on across a handful of pay-per-event tools: bill for delivered answers, never for attempts, and make "we could not tell" a first-class, free, clearly-labeled output.

Why this is worth the lost revenue

It costs you money on paper. Every unknown and every no-result is a row you did not charge for. Your per-run number looks worse than a competitor who bills for everything including the garbage.

It is also the only version of the number that survives the customer opening their own logs. The competitor who billed for failures looks cheaper for exactly one cycle, until someone notices their bounce rate went up after a "cleaning" or their lead count dropped for no reason. Trust is the actual product. The billing model is just where you prove you have it.

How to implement it without overthinking

Every record carries its own status. Do not infer success from "the batch finished."
Use at least three buckets: answer, no-result, unknown. Add more if your domain needs them (fetch-error, rate-limited, etc), but never fewer.
Only "answer" (and a definitive negative answer, if that is what was asked for) is billable. Everything else is free and labeled.
Say it out loud in your docs. "We do not bill for failed fetches or unknowns" converts better than any feature list, because it tells buyers you understand the failure modes they have already been burned by.

If you are building anything usage-billed, the binary will tempt you because it is simpler and it makes more money this week. Resist it. The third state is cheap insurance against the one outcome you can not recover from, which is a customer deciding your numbers can not be trusted.

I build a few data tools on this rule (the public ones are at apify.com/mrlarryjohnson). Happy to compare notes if you are designing a pay-per-result model. What is your third state?

Give your AI agent live prediction-market data in 30 seconds (MCP, no keys, no installs)

Larry Johnson — Fri, 12 Jun 2026 13:42:40 +0000

Most "connect your AI to data" tutorials end with you managing API keys, running a local server, and debugging someone's half-maintained npm package. Here's the 30-second version instead, using the Model Context Protocol and a hosted server.

What you get

Your agent (Claude, Cursor, n8n, Make — anything that speaks MCP) gains native tools for live Polymarket data:

search_markets / get_market — find any prediction market + live prices
get_orderbook — current bids/asks
whale_trades — recent large trades with wallet addresses
top_holders — who's positioned where

So you can ask things like "What's the implied probability SpaceX closes above $2T today, and which whales disagree?" and the agent pulls real numbers instead of hallucinating stale ones.

The 30 seconds

Add this to your MCP config (Claude Desktop, Claude Code, or Cursor):

{
  "mcpServers": {
    "polymarket": {
      "url": "https://mrlarryjohnson--polymarket-mcp-server.apify.actor/mcp",
      "headers": { "Authorization": "Bearer <your-apify-token>" }
    }
  }
}

The token is a free Apify account token. No Polymarket account, no wallet, no keys for the data source — the server is hosted and always warm (Apify Standby mode), and you pay fractions of a cent per tool call instead of a subscription.

There's a matching Web Search MCP server if you also want multi-engine web/news search tools next to it.

Why hosted MCP beats local for data tools

Nothing to install or babysit — no node/python process dying on your laptop mid-session.
Read-only by design — the server holds no wallet and can't trade. Your agent gets market data, not the ability to spend your money.
Pay-per-call — idle costs nothing; heavy research days cost cents.

The billing rule I wish more APIs had

One decision I'd recommend to anyone building paid agent tools: never bill a failed call. An upstream timeout or rate limit raises a clear error and is not charged — never disguised as an empty result. Agents retry blindly, so silent-empty responses both corrupt the agent's reasoning AND bill the user for nothing. Fail loud, bill only verdicts.

(Same rule across the rest of my catalog — whale trackers, an email validator honest about not doing SMTP theater from datacenter IPs, a website change monitor that ships actual diffs: apify.com/mrlarryjohnson. Thanks to @apify for the Standby-mode hosting.)

Questions welcome — happy to share implementation details.