<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: 0xGollum</title>
    <description>The latest articles on DEV Community by 0xGollum (@0xgollum).</description>
    <link>https://dev.to/0xgollum</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4026311%2F3a2e5ec0-3483-4cdf-894c-cee31825ccbb.jpg</url>
      <title>DEV Community: 0xGollum</title>
      <link>https://dev.to/0xgollum</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/0xgollum"/>
    <language>en</language>
    <item>
      <title>I cloned my equity options scanner for crypto — the same 3 rules, a completely different data source</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Tue, 28 Jul 2026 19:26:51 +0000</pubDate>
      <link>https://dev.to/0xgollum/i-cloned-my-equity-options-scanner-for-crypto-the-same-3-rules-a-completely-different-data-source-4o6c</link>
      <guid>https://dev.to/0xgollum/i-cloned-my-equity-options-scanner-for-crypto-the-same-3-rules-a-completely-different-data-source-4o6c</guid>
      <description>&lt;p&gt;A while back I built a scanner that flags "unusual options activity" on US equities — the kind of signal where a contract suddenly trades on way more volume than its standing open interest, backed by real dollar premium, not just noise. Three gates: liquidity, freshness (volume vs open interest ratio), and money behind the trade.&lt;/p&gt;

&lt;p&gt;Someone asked me: could the same idea work for crypto options? My first instinct was "sure, just swap the data source." That instinct was mostly right, but not entirely — here's what actually changed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The data source is a different shape
&lt;/h2&gt;

&lt;p&gt;Equities options data (I was pulling from a public CDN) comes back per-ticker: one call, one full chain for that symbol. Crypto options don't work that way — there's really one dominant venue for BTC/ETH options, and its public API returns every strike and expiry for a whole currency in a single call. So the "loop over tickers" became a "loop over currencies" — structurally similar, but the unit of iteration changed.&lt;/p&gt;

&lt;h2&gt;
  
  
  No fixed contract multiplier
&lt;/h2&gt;

&lt;p&gt;US equity options are always 1 contract = 100 shares. That "x100" was hardcoded in my premium calculation. Crypto options are 1 contract = 1 unit of the underlying (1 BTC, 1 ETH) — no multiplier. I ended up parameterizing the premium function instead of hardcoding the multiplier, which also made the pure signal-logic module reusable as-is between both versions. Same detection math, same tests, different constant.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prices aren't in USD
&lt;/h2&gt;

&lt;p&gt;Crypto options are quoted in the base currency (a BTC option's price is itself a fraction of a BTC), not in dollars. So every price has to be converted using the current index price before any of the USD-premium thresholds make sense. Easy to get right, easy to silently get wrong if you forget it in one code path.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug that actually mattered
&lt;/h2&gt;

&lt;p&gt;While writing tests for the instrument-name parser (something like &lt;code&gt;BTC-28AUG26-59000-P&lt;/code&gt;), I'd assumed the date chunk was always a fixed length. It's not — a 1-digit day ("7AUG26") is shorter than a 2-digit day ("28AUG26"). My first length check silently rejected every single-digit-day contract as unparseable. That's roughly 10% of live instruments gone, quietly, with no error — and they skew toward near-dated (often the most actively traded) expiries.&lt;/p&gt;

&lt;p&gt;Nothing caught it until I wrote a dedicated test for the parser itself, separate from the end-to-end "does it produce output" test. The lesson isn't really about crypto or options — it's that a parsing function deserves its own unit tests with real edge-case inputs, not just a smoke test that happens to pass because your sample data avoided the edge case.&lt;/p&gt;

&lt;p&gt;Same three rules, same pure detection logic, ported across a genuinely different market — most of the actual work was in the boring conversion layer, not the "signal" part.&lt;/p&gt;




&lt;p&gt;The crypto version is live on Apify if you want to poke at it: &lt;a href="https://apify.com/0xgollum/unusual-crypto-options-activity" rel="noopener noreferrer"&gt;https://apify.com/0xgollum/unusual-crypto-options-activity&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>crypto</category>
      <category>api</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Turning a Public Market-Transparency Dataset Into a Volume-Anomaly Scanner</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Tue, 28 Jul 2026 12:30:24 +0000</pubDate>
      <link>https://dev.to/0xgollum/turning-a-public-market-transparency-dataset-into-a-volume-anomaly-scanner-4h8c</link>
      <guid>https://dev.to/0xgollum/turning-a-public-market-transparency-dataset-into-a-volume-anomaly-scanner-4h8c</guid>
      <description>&lt;p&gt;Every week, publicly listed stocks in the US have their off-exchange ("dark pool") trading volume disclosed per symbol, per venue, under a market-transparency rule that predates most people's interest in market data APIs. The catch: it's published as raw, paginated compliance data, not a friendly product. Nobody looks at it directly.&lt;/p&gt;

&lt;p&gt;I spent an afternoon turning that into a signal: is this week's off-exchange volume for a given ticker unusual compared to its own recent history?&lt;/p&gt;

&lt;h2&gt;
  
  
  The interesting engineering bits
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Partition-key gotchas.&lt;/strong&gt; The API enforces that any sort operation must filter ALL its partition keys with an exact-match comparator, or you get a 400 - though the error message thankfully tells you exactly which keys are missing. Once you know that, filtering instead of sorting sidesteps the whole problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Comparator naming isn't obvious.&lt;/strong&gt; "greater or equal" isn't spelled the way you'd guess - it's a short enum name you basically have to discover by trial. A handful of curl calls with different guesses got there faster than reading the docs end to end.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;JSON vs CSV by header, not URL param.&lt;/strong&gt; Same endpoint, same body - send an &lt;code&gt;Accept: application/json&lt;/code&gt; header and you get typed JSON back instead of a quoted-CSV text blob you'd otherwise have to parse by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  The actual signal
&lt;/h2&gt;

&lt;p&gt;Rather than an absolute threshold (which punishes small-caps and ignores mega-caps equally), each ticker's most recent reported week gets compared against its OWN trailing average over the prior weeks. A stock trading 3x its own normal off-exchange volume is a much more useful signal than "over 1 million shares," which means something completely different depending on the ticker.&lt;/p&gt;

&lt;p&gt;Only tickers that clear both a minimum volume floor and a minimum spike ratio get returned. Empty runs are never billed, so there's no incentive to pad results with noise - a design pattern I try to keep across everything I ship.&lt;/p&gt;

&lt;p&gt;Packaged it as an Apify actor (pay-per-result): Dark Pool Pulse - &lt;a href="https://apify.com/0xgollum/dark-pool-pulse" rel="noopener noreferrer"&gt;https://apify.com/0xgollum/dark-pool-pulse&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Always curious what data-anomaly patterns other people are scanning for - drop them in the comments.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>opendata</category>
      <category>showdev</category>
    </item>
    <item>
      <title>Signal Hub MCP: Plugging Trading Signals Directly Into AI Agents</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Mon, 27 Jul 2026 18:15:53 +0000</pubDate>
      <link>https://dev.to/0xgollum/signal-hub-mcp-plugging-trading-signals-directly-into-ai-agents-dd5</link>
      <guid>https://dev.to/0xgollum/signal-hub-mcp-plugging-trading-signals-directly-into-ai-agents-dd5</guid>
      <description>&lt;p&gt;If you're building an autonomous trading or betting agent, you've probably hit this friction: your data source is a dashboard, but your agent lives in a chat loop. You end up writing glue code to bridge the two.&lt;/p&gt;

&lt;p&gt;I just shipped Signal Hub MCP, a small Apify Actor that closes that gap for two data sources I already maintain: unusual options activity (volume/open-interest anomalies on US equities) and horse racing odds-movement signals.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it does
&lt;/h2&gt;

&lt;p&gt;It's a standard MCP server, deployed as an Apify Actor in Standby mode (Streamable HTTP, not stdio). Two tools exposed:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;get_unusual_options_activity(tickers, min_vol_oi_ratio, max_results)&lt;/li&gt;
&lt;li&gt;get_horse_racing_signals(max_races, only_signals)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Your agent just calls the tool mid-conversation. No dashboard, no manual actor runs, no dataset parsing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Standby mode + Pay-Per-Event
&lt;/h2&gt;

&lt;p&gt;Apify's Actor Standby mode keeps the server warm and reachable over HTTP instead of spinning up per-request. Billing is Pay-Per-Event, charged per tool call, not a subscription. That mapped cleanly onto the MCP tool-call model: one Actor.charge() per tool invocation.&lt;/p&gt;

&lt;p&gt;Internally, each tool call triggers a run of one of my existing standalone actors (Unusual Options Activity Scanner, Horse Racing Pulse) and returns the dataset. No duplicated scraping logic — the MCP layer is purely a thinner, agent-friendly interface on top of what already existed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://apify.com/0xgollum/signal-hub-mcp" rel="noopener noreferrer"&gt;https://apify.com/0xgollum/signal-hub-mcp&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;More tools from the same portfolio (crypto funding rates, prediction markets) are next in line for the same treatment.&lt;/p&gt;

</description>
      <category>mcp</category>
      <category>ai</category>
      <category>apify</category>
    </item>
    <item>
      <title>Why I only track one bookmaker for my odds-movement signal (and it's not the popular one)</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Fri, 24 Jul 2026 18:29:38 +0000</pubDate>
      <link>https://dev.to/0xgollum/why-i-only-track-one-bookmaker-for-my-odds-movement-signal-and-its-not-the-popular-one-1713</link>
      <guid>https://dev.to/0xgollum/why-i-only-track-one-bookmaker-for-my-odds-movement-signal-and-its-not-the-popular-one-1713</guid>
      <description>&lt;p&gt;Most odds-comparison tools I looked at blend prices across 20, 30, sometimes 40+ bookmakers. More books, more data, right?&lt;/p&gt;

&lt;p&gt;For a movement-detection signal specifically, I don't think so — and building an odds-movement tracker for soccer taught me why.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem with averaging books
&lt;/h2&gt;

&lt;p&gt;A "steamer" (implied probability jumping fast) or a "drifter" (fading) is supposed to mean one thing: real money is moving the market. But most retail-facing bookmakers move their lines reactively — copying the sharper books, adjusting for their own promotional exposure, or just lagging by minutes to hours. Blend 30 of those together and you average out the signal along with the noise. A steamer detected on a consensus feed is often just one popular book catching up, not new information.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why one specific bookmaker
&lt;/h2&gt;

&lt;p&gt;Pinnacle is built differently: low margins, no betting limits for winners, and a business model that depends on the line staying accurate rather than protecting the house. Professional bettors treat it as the closest thing to a "true" market price precisely because it reacts fastest and cleanest to real money. If you want a movement signal that means something, you want the reaction, not the consensus.&lt;/p&gt;

&lt;h2&gt;
  
  
  What that means in practice
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;One source, key-less, no proxy — the bookmaker's own guest API is public&lt;/li&gt;
&lt;li&gt;Leagues aren't hardcoded: every run pulls whatever's actively being priced, so it tracks domestic leagues most of the year and auto-picks up World Cup / Euro matches during a tournament without a code change&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;steamer&lt;/code&gt;/&lt;code&gt;drifter&lt;/code&gt; rows fire only on the delta between two runs — no history table to maintain yourself&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;I'm 0xGollum, I build small data actors on Apify — mostly scrapers and monitors for signals I found annoying to track by hand. More of what I ship: linktr.ee/0xGollum&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>api</category>
      <category>buildinpublic</category>
    </item>
    <item>
      <title>Getting past Binance and Bybit's datacenter IP blocks without overpaying for proxies</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Fri, 24 Jul 2026 18:08:54 +0000</pubDate>
      <link>https://dev.to/0xgollum/getting-past-binance-and-bybits-datacenter-ip-blocks-without-overpaying-for-proxies-5dce</link>
      <guid>https://dev.to/0xgollum/getting-past-binance-and-bybits-datacenter-ip-blocks-without-overpaying-for-proxies-5dce</guid>
      <description>&lt;p&gt;I've been building a small actor that compares perpetual futures funding rates across exchanges to flag arbitrage spreads. Most exchanges (OKX, Gate.io, KuCoin, Bitget, MEXC) have clean, key-less public APIs that just work from any server, including cloud/datacenter IPs.&lt;/p&gt;

&lt;p&gt;Binance is not one of them. Hit its public funding rate endpoint from a datacenter IP and you get a flat HTTP 451 — geo-blocked, no way around it with a direct connection.&lt;/p&gt;

&lt;p&gt;I'd shipped without Binance for months, figuring that wall was permanent. Then, while extending the actor to more exchanges, I noticed something I hadn't planned for: Bybit — which had worked fine from day one — started returning HTTP 403 from the exact same server. Same code, same IP range, different day. No announcement, no changelog entry I could find. It just started blocking.&lt;/p&gt;

&lt;p&gt;That's the actual lesson here: an API block you hit once isn't necessarily permanent, and a block you've never hit isn't necessarily safe either. Both directions can change without warning.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: proxy only what actually needs it
&lt;/h2&gt;

&lt;p&gt;The obvious fix is "route everything through a residential proxy." The problem: residential proxy bandwidth costs real money, and 5 of my 7 exchanges don't need it at all. Paying proxy rates for every single request when only two exchanges are blocking would be wasteful — and slower, since residential circuits add latency direct connections don't have.&lt;/p&gt;

&lt;p&gt;So the actual fix is per-exchange:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Detect the blocked ones by status code (451 for a hard geo-block, 403 for the softer anti-bot flavor)&lt;/li&gt;
&lt;li&gt;Only those two calls get an explicit proxy URL&lt;/li&gt;
&lt;li&gt;Everything else keeps hitting the exchange directly, fast and free
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_get_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;proxy_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;proxy_url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;httpx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;AsyncClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;proxy_url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;proxied&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;proxied&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...)&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...)&lt;/span&gt;
    &lt;span class="bp"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One conditional client, one proxy URL passed down only where it's needed. The other 5 exchanges never touch the proxy at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Worth re-testing what you gave up on
&lt;/h2&gt;

&lt;p&gt;The bigger habit change: I'd marked Binance as "permanently excluded" in my own notes and never looked back. That was fine for Binance (still 451, still needs the proxy). But I would have shipped Bybit broken silently if I hadn't happened to be touching that code again for an unrelated reason.&lt;/p&gt;

&lt;p&gt;If you maintain anything that scrapes or calls third-party APIs long enough, the honest move is to periodically re-run the sources you've already integrated, not just the ones you're adding — blocks appear on a schedule you don't control.&lt;/p&gt;




&lt;p&gt;I'm 0xGollum, I build small data actors on Apify — mostly scrapers and monitors for signals I found annoying to track by hand. If you're curious what else I'm shipping: linktr.ee/0xGollum&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>webscraping</category>
      <category>crypto</category>
    </item>
    <item>
      <title>I added a new feature to a live paid actor — the data was already there, I just wasn't reading it</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Wed, 22 Jul 2026 12:14:17 +0000</pubDate>
      <link>https://dev.to/0xgollum/i-added-a-new-feature-to-a-live-paid-actor-the-data-was-already-there-i-just-wasnt-reading-it-2dnc</link>
      <guid>https://dev.to/0xgollum/i-added-a-new-feature-to-a-live-paid-actor-the-data-was-already-there-i-just-wasnt-reading-it-2dnc</guid>
      <description>&lt;p&gt;Horse Racing Pulse has been running for a few weeks now, tracking live PMU odds and flagging steamers/drifters as money moves the market. It found a real paying user early on, which made me nervous about touching it — the classic "don't fix what isn't broken" problem.&lt;/p&gt;

&lt;p&gt;But a user having a working tool doesn't mean the tool can't get more useful without breaking what already works. Here's what I did.&lt;/p&gt;

&lt;h2&gt;
  
  
  The realization
&lt;/h2&gt;

&lt;p&gt;The actor calls one PMU endpoint per race to get live odds. I was only reading a handful of fields off each participant: name, number, jockey, odds. Out of curiosity I dumped the full raw response for one horse, and it turned out I'd been ignoring most of the payload the whole time — career starts, wins, places, this-year vs last-year earnings, whether the jockey changed for today's race, whether this is the horse's debut. All of it was already coming back on every single call I was already making. Zero extra requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turning raw fields into something worth reading
&lt;/h2&gt;

&lt;p&gt;Raw numbers aren't a feature. "12 starts, 0 wins" only means something if 12 is enough starts for that to be a real pattern and not noise. I picked a minimum sample size (10+ starts) before flagging a winless streak, and a minimum swing (50%+) before flagging a year-over-year earnings change, specifically to keep the signal-to-noise ratio honest.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug that would have shipped wrong
&lt;/h2&gt;

&lt;p&gt;While testing against live races, the earnings numbers looked absurd — a modest PMU trotter apparently earning several million euros a year. Turns out PMU returns these amounts in centimes, not euros. A silent off-by-100 that would have made every earnings message wrong, caught only because I ran it against real live data instead of trusting the field name.&lt;/p&gt;

&lt;p&gt;Lesson I keep re-learning: field names lie, or at least don't tell you the unit. Test against production data before you ship.&lt;/p&gt;

&lt;h2&gt;
  
  
  Not touching what works
&lt;/h2&gt;

&lt;p&gt;The new patterns live behind one boolean input, off by default. The existing paying user sees zero change unless they opt in. Old tests still pass unmodified — I added new ones for the new behavior, didn't touch the old ones.&lt;/p&gt;

&lt;p&gt;If you're sitting on a scraper or API integration that's "done," it might be worth checking what you're actually parsing out of the response versus what's available in it. Sometimes the next feature isn't a new source — it's a field you never read.&lt;/p&gt;

&lt;p&gt;Actor: apify.com/0xgollum/horse-racing-pulse&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>apify</category>
      <category>buildinpublic</category>
    </item>
    <item>
      <title>I scraped live betting odds from 9 countries and never once hit a real anti-bot wall</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Mon, 20 Jul 2026 14:28:45 +0000</pubDate>
      <link>https://dev.to/0xgollum/i-scraped-live-betting-odds-from-9-countries-and-never-once-hit-a-real-anti-bot-wall-57ng</link>
      <guid>https://dev.to/0xgollum/i-scraped-live-betting-odds-from-9-countries-and-never-once-hit-a-real-anti-bot-wall-57ng</guid>
      <description>&lt;p&gt;I just shipped a horse-racing odds tracker across 9 countries. Four totally different site architectures, zero paid proxies, zero headless browsers, zero CAPTCHAs solved. Every "protected" data source I hit turned out to be a public API hiding in plain sight. Here's what actually worked.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Curl before you Playwright
&lt;/h2&gt;

&lt;p&gt;The instinct when you see a modern-looking racing site is to reach for a headless browser. Don't. First move is always a plain &lt;code&gt;curl&lt;/code&gt; with a real User-Agent and a look at how much of the page is actual text vs markup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;
&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;[^&amp;gt;]+&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A high ratio usually means server-side rendering — the data's already in the HTML, a JS engine was never going to be necessary. This alone ruled out headless browsers for 3 of the 4 sources I ended up using.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Official public APIs are more common than you'd think
&lt;/h2&gt;

&lt;p&gt;One source runs on a pari-mutuel model, and it turns out the operator ships a fully public, key-less JSON API for its own web/app clients — no login, no proxy, nothing. If a market has a state-run or monopoly tote operator, check whether their own site calls a JSON backend before assuming you need to scrape HTML at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Read the &lt;strong&gt;NEXT_DATA&lt;/strong&gt; / JSON-LD blob before you write a single CSS selector
&lt;/h2&gt;

&lt;p&gt;Two of my four sources are Next.js apps. Both ship the entire page's data as a JSON blob in a &lt;code&gt;&amp;lt;script&amp;gt;&lt;/code&gt; tag — &lt;code&gt;__NEXT_DATA__&lt;/code&gt; on one, schema.org &lt;code&gt;SportsEvent&lt;/code&gt; JSON-LD on the other. No DOM parsing needed for the core fields; just find the tag and &lt;code&gt;json.loads()&lt;/code&gt; it.&lt;/p&gt;

&lt;p&gt;Gotcha: if the JSON has nested objects, don't reach for a regex like &lt;code&gt;\{.*?\}&lt;/code&gt; to extract it — non-greedy matching stops at the first inner &lt;code&gt;}&lt;/code&gt;, not the real end. I burned twenty minutes on this exact bug before switching to a proper brace-counting extractor:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;extract_json_object&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  4. Client-side config files sometimes ship real API keys
&lt;/h2&gt;

&lt;p&gt;The most surprising find: one source is a GraphQL app, and the frontend's own JS config file contained a working AWS AppSync API key — meant for the public site to call directly. Introspection was even enabled, so I could enumerate the whole schema and find the exact query I needed (turned out to be &lt;code&gt;getRaceForm&lt;/code&gt;, not the more obvious-looking &lt;code&gt;getOdds&lt;/code&gt;, which silently returned nothing for reasons I never fully nailed down).&lt;/p&gt;

&lt;p&gt;Lesson: view-source a site's own JS bundles before assuming an API is closed. If the frontend can call it with a static key, so can you.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Country/region filters on aggregator sites can lie to you
&lt;/h2&gt;

&lt;p&gt;One aggregator site shows "next races" for a country page even when nothing's actually happening in that country right now — it silently falls back to showing whatever's soonest site-wide. First pass of my scraper mislabeled tracks from one country as belonging to another because of this. Fix was trivial once caught (cross-check against the actual structured-data field for country) but the failure mode was quiet — no error, just wrong data. Worth an explicit assertion for any "listing page defaults to something unexpected" pattern.&lt;/p&gt;




&lt;p&gt;Built as a new Apify actor, &lt;a href="https://apify.com/0xgollum/world-turf-pulse" rel="noopener noreferrer"&gt;World Turf Pulse&lt;/a&gt; — tracks steamer/drifter odds movement across nine countries. Happy to answer questions on any of the four scraping approaches above.&lt;/p&gt;

</description>
      <category>webscraping</category>
      <category>python</category>
      <category>api</category>
      <category>buildinpublic</category>
    </item>
    <item>
      <title>What breaks when you scrape Steam Community Market at scale</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Mon, 20 Jul 2026 08:38:22 +0000</pubDate>
      <link>https://dev.to/0xgollum/what-breaks-when-you-scrape-steam-community-market-at-scale-200c</link>
      <guid>https://dev.to/0xgollum/what-breaks-when-you-scrape-steam-community-market-at-scale-200c</guid>
      <description>&lt;p&gt;CS2 Skin Pulse tracks price and volume signals on CS2 skins from the Steam Community Market. No official API, no key required — but the public JSON endpoints behind it were built for a browser making occasional requests, not a cloud worker hitting them on a schedule. Here's what actually broke, and what fixed it.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Datacenter IPs get rate-limited hard
&lt;/h2&gt;

&lt;p&gt;Steam doesn't just slow you down. On Apify's shared IP ranges, priceoverview starts returning 429s on nearly every request, even at low volume — direct requests basically never succeed from there. The fix is routing through Apify's residential proxy, which resolves as normal consumer traffic instead of a flagged datacenter block. On top of that, requests are throttled deliberately (a semaphore capped at 4 concurrent calls, ~350ms sleep between price fetches) instead of relying on the proxy alone to absorb the load.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. There's no historical price endpoint if you're not logged in
&lt;/h2&gt;

&lt;p&gt;Steam has a pricehistory endpoint, but it requires an authenticated session — not viable for a public actor. So instead of asking "what's the trend," the actor asks "what changed since last time": every run persists each item's price and volume to a named key-value store, then diffs against the previous run to compute price_spike / price_drop / volume_spike signals. Same pattern already used in Bitcoin Pulse for funding-rate history — once you build a "diff against last run" primitive, it works for any source that has no history API at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Retry logic has to know the difference between "try again" and "give up"
&lt;/h2&gt;

&lt;p&gt;429/500/502/503/504 get retried with backoff. A 403 is a hard block — hammering the same IP again wastes the run's time budget for nothing. And when even priceoverview (rate-limited harder than plain item search) comes back empty after retries, there's a fallback to a lighter search/render call that still returns a current sell price. No median, no volume, but enough to keep the signal alive instead of silently dropping the item for that run.&lt;/p&gt;

&lt;p&gt;None of this is specific to CS2 skins — it's the shape of any public JSON endpoint sitting behind a site built for a browser, not a script. Source's public on the Apify Store if you want to see the actual code: &lt;a href="https://apify.com/0xgollum/cs2-skin-pulse" rel="noopener noreferrer"&gt;https://apify.com/0xgollum/cs2-skin-pulse&lt;/a&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>apify</category>
      <category>buildinpublic</category>
    </item>
    <item>
      <title>What it actually takes to turn a Python scraper into a real Apify actor</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Thu, 16 Jul 2026 19:02:50 +0000</pubDate>
      <link>https://dev.to/0xgollum/what-it-actually-takes-to-turn-a-python-scraper-into-a-real-apify-actor-404l</link>
      <guid>https://dev.to/0xgollum/what-it-actually-takes-to-turn-a-python-scraper-into-a-real-apify-actor-404l</guid>
      <description>&lt;p&gt;A scraper that works on your machine and an actor that survives in production are two different things. I found this out the hard way shipping ten of them on the Apify Store. Here's what actually breaks, and what fixes it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pinned dependencies, or the build silently breaks
&lt;/h2&gt;

&lt;p&gt;Crawlee and Apify's SDK move fast enough that an unpinned &lt;code&gt;apify&lt;/code&gt; or &lt;code&gt;pydantic&lt;/code&gt; version can break the import at build time, with an error message that points nowhere useful. The fix is boring: pin everything (&lt;code&gt;apify==2.7.3&lt;/code&gt;, &lt;code&gt;pydantic==2.10.6&lt;/code&gt;, &lt;code&gt;browserforge==1.2.3&lt;/code&gt;, &lt;code&gt;httpx==0.27.0&lt;/code&gt;), and build on Python 3.11, not whatever your local machine runs. I lost an afternoon to this before pinning became a habit.&lt;/p&gt;

&lt;h2&gt;
  
  
  The default key-value store does not persist
&lt;/h2&gt;

&lt;p&gt;If your actor needs to remember anything between runs, deduplicated items, a rolling window for trend detection, whatever, the default KV store gets wiped every run. You need &lt;code&gt;Actor.open_key_value_store(name="something-persistent")&lt;/code&gt; explicitly. Easy to miss once, annoying to debug when your "new item" detector fires on the same item every single run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bare except blocks hide the real failure
&lt;/h2&gt;

&lt;p&gt;Early on, a source going down looked identical to a source returning zero results: both produced an empty dataset, silently. The only fix is boring discipline: log the HTTP status on every request, and retry on 429/5xx with backoff instead of swallowing the exception. Once I added this everywhere, actors that used to fail invisibly started failing loudly and correctly, which sounds worse but is much better.&lt;/p&gt;

&lt;h2&gt;
  
  
  Datacenter IPs get blocked even when residential IPs don't
&lt;/h2&gt;

&lt;p&gt;One actor scraping a big marketplace worked fine from my laptop and returned 429 on every single request from Apify's cloud IP range, on two different endpoints. Not a bug in the code, a block on the IP class. The fix was routing through Apify's own residential proxy (&lt;code&gt;Actor.create_proxy_configuration()&lt;/code&gt;), which resolved it completely. If a scraper mysteriously works locally and dies in the cloud, check the IP class before you touch the code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sometimes the obvious source blocks you outright
&lt;/h2&gt;

&lt;p&gt;I built an actor around Yahoo Finance's options data, then discovered its crumb-authentication endpoint flatly rejects datacenter IPs, no workaround, no proxy fix. The move was pivoting to a different public source entirely (CBOE's delayed-quotes endpoint, no auth required) rather than fighting a wall that wasn't going to move. Delayed data that actually loads beats real-time data that never does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Resource defaults quietly eat your margin
&lt;/h2&gt;

&lt;p&gt;Apify actors default to fairly generous RAM and timeout settings. On a pay-per-result actor, that default can mean you're burning more in compute than you're earning per run, silently, for weeks. Tuning RAM down (in one case from 4096MB to 512MB) cut the cost per run by 4-8x and turned a marginal actor profitable without changing a line of scraping logic. Worth checking before you assume a niche "doesn't work."&lt;/p&gt;

&lt;h2&gt;
  
  
  Every input field needs a description, or the build fails
&lt;/h2&gt;

&lt;p&gt;Small one, but it costs real time the first time you hit it: Apify's input schema validation requires a &lt;code&gt;description&lt;/code&gt; on every property. Skip one and the build fails with a message that doesn't always point at which field.&lt;/p&gt;

&lt;p&gt;None of this is exotic. It's the difference between "I wrote a script that scrapes a thing" and "I shipped something that keeps working when the site changes, the IP gets flagged, or the traffic spikes." That gap is most of what people are actually paying for when they hire someone to build a scraper instead of writing one themselves.&lt;/p&gt;

&lt;p&gt;If you'd rather not deal with any of the above yourself, I take on custom Apify actor builds. My existing actors are live on the Apify Store, not just a portfolio of screenshots: &lt;a href="https://apify.com/0xgollum" rel="noopener noreferrer"&gt;check my profile&lt;/a&gt;. Custom build requests go through my &lt;a href="https://www.fiverr.com/crabe666/develop-a-custom-apify-actor-or-web-scraper-for-you" rel="noopener noreferrer"&gt;Fiverr gig&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>apify</category>
      <category>buildinpublic</category>
    </item>
    <item>
      <title>How I verify B2B emails without sending a single message</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Wed, 15 Jul 2026 13:13:00 +0000</pubDate>
      <link>https://dev.to/0xgollum/how-i-verify-b2b-emails-without-sending-a-single-message-3o06</link>
      <guid>https://dev.to/0xgollum/how-i-verify-b2b-emails-without-sending-a-single-message-3o06</guid>
      <description>&lt;p&gt;Most "verified" B2B lead lists still bounce on the first send. Here is the actual technique behind a real deliverability check — without ever sending an email — and why honest confidence labels beat a fake "100% verified" badge.&lt;/p&gt;

&lt;h2&gt;
  
  
  The core trick: talk to the mail server, then hang up
&lt;/h2&gt;

&lt;p&gt;Every domain that receives email publishes MX records. To check whether a mailbox exists, you:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Resolve the domain's MX hosts (DNS).&lt;/li&gt;
&lt;li&gt;Open an SMTP connection to the top MX host on port 25.&lt;/li&gt;
&lt;li&gt;Walk the handshake: HELO -&amp;gt; MAIL FROM -&amp;gt; RCPT TO for the target address.&lt;/li&gt;
&lt;li&gt;Read the server's response code to RCPT TO, then QUIT &lt;strong&gt;before&lt;/strong&gt; DATA.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;You never send a message. You just ask "would you accept mail for this mailbox?" and listen — the same no-send probe the commercial verifiers use.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;smtplib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SMTP&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;smtp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;smtp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mx_host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;smtp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;helo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checker.local&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;smtp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;probe@checker.local&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;    &lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;smtp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rcpt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# the answer lives here
&lt;/span&gt;    &lt;span class="n"&gt;smtp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;quit&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;                    &lt;span class="c1"&gt;# QUIT before DATA: nothing is sent
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Why "valid" is never fully ironclad
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Catch-all domains&lt;/strong&gt; accept every local part, so a 250 on RCPT TO does not prove that exact mailbox exists. Detect it by probing a random bogus address on the same domain — if that is accepted too, it is catch-all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Gmail / Outlook / M365&lt;/strong&gt; often return an ambiguous accept up front and only reject at final delivery. "The server did not reject it" is not "guaranteed deliverable."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Greylisting&lt;/strong&gt; returns a temporary 4xx — not a no, just "try again later."&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The honest move: tiers, not a badge
&lt;/h2&gt;

&lt;p&gt;Instead of stamping everything "100% verified," label each address by how confident the check really is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;valid&lt;/code&gt; — MX exists, RCPT accepted, not catch-all&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;catch_all&lt;/code&gt; — accepts anything, cannot confirm this exact mailbox&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;published&lt;/code&gt; — the address is published on the company's own website (a real human contact) even when SMTP stays inconclusive&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;unknown&lt;/code&gt; — the server blocked or greylisted the probe&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A buyer who sees the label knows exactly what they are getting. That honesty outperforms an inflated number that bounces.&lt;/p&gt;

&lt;h2&gt;
  
  
  One speed lesson
&lt;/h2&gt;

&lt;p&gt;If a domain's mail server never answers the probe at all, do not retry five guessed patterns against it — each one eats a full timeout for nothing. Detect "unreachable domain" once and bail. That single change cut a run from minutes to seconds on niches whose mail hosts refuse port-25 probes.&lt;/p&gt;




&lt;p&gt;I build data and automation tools, and I just opened a Fiverr gig doing exactly this — verified B2B lead lists with honest confidence labels, filtered by industry and location. If dead-inbox lists annoy you as much as they annoy me: &lt;a href="https://www.fiverr.com/crabe666/generate-verified-b2b-leads-with-valid-email-addresses" rel="noopener noreferrer"&gt;it is right here&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>python</category>
      <category>emailverification</category>
      <category>showdev</category>
      <category>webdev</category>
    </item>
    <item>
      <title>I shipped 10 paid data scrapers in one week: what actually worked (and what quietly lost money)</title>
      <dc:creator>0xGollum</dc:creator>
      <pubDate>Sun, 12 Jul 2026 18:27:27 +0000</pubDate>
      <link>https://dev.to/0xgollum/i-shipped-10-paid-data-scrapers-in-one-week-what-actually-worked-and-what-quietly-lost-money-3k1k</link>
      <guid>https://dev.to/0xgollum/i-shipped-10-paid-data-scrapers-in-one-week-what-actually-worked-and-what-quietly-lost-money-3k1k</guid>
      <description>&lt;p&gt;A week ago I had zero products. Today I have 10 paid scrapers ("actors") live on the Apify Store, all pay-per-result. Total revenue so far: modest. Total lessons: enormous. Here are the ones I wish someone had told me on day one.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. "Zero competitors" is not a green light
&lt;/h2&gt;

&lt;p&gt;My first instinct was to hunt for empty niches. Zero competitors means zero proof: it is not evidence of an untapped market, it is the absence of evidence of any market. A niche actor I studied had 2 users after 7 weeks.&lt;/p&gt;

&lt;p&gt;The niche that actually performs for me (crypto funding rate arbitrage) had two established competitors with 20+ paying users each. That was the real signal: money already flows here, and the pie is not locked up.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Your platform defaults are eating your margin
&lt;/h2&gt;

&lt;p&gt;This one hurt. On pay-per-event pricing, YOU pay the platform usage of every run out of your earnings. The default memory allocation was 4096 MB. My actors are simple HTTP fetchers that finish in 5 seconds.&lt;/p&gt;

&lt;p&gt;One of my actors was literally losing money: $0.0161 platform cost per run vs $0.015 revenue. Negative margin on every single sale, invisible unless you check run stats via the API.&lt;/p&gt;

&lt;p&gt;Dropping memory from 4096 MB to 512 MB cut per-run cost by a factor of 4 to 8. One API call. Biggest ROI of the whole week.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Sources will block you, and your datacenter IP is the problem
&lt;/h2&gt;

&lt;p&gt;What works from your laptop will not necessarily work from a cloud IP:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Binance geo-blocks datacenter IPs with HTTP 451. OKX serves them fine. Same data.&lt;/li&gt;
&lt;li&gt;Steam rate-limits its price endpoint so hard from cloud IPs that every request 429s. Fix: route through a proxy and keep a fallback endpoint.&lt;/li&gt;
&lt;li&gt;Yahoo Finance blocks its crumb endpoint from datacenter ranges. CBOE serves the same options data with no auth at all.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The lesson: pick your data source for its reachability from cloud infra, not just for its data quality. Always have a plan B endpoint.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Silent failure handling is a trap you set for yourself
&lt;/h2&gt;

&lt;p&gt;Every fetch helper I wrote originally swallowed exceptions and returned None. Clean, defensive, and completely blind. When a source started failing, my actor shipped empty datasets with a green SUCCEEDED status. A paying customer gets zero rows and you get zero idea why.&lt;/p&gt;

&lt;p&gt;Now every helper retries on 429/5xx with backoff and logs the status code on final failure. The first day this was live, the logs immediately told me exactly which source was blocking which endpoint. Debugging went from "rerun and guess" to "read one log line".&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Never ship an empty dataset
&lt;/h2&gt;

&lt;p&gt;If your scraper found no signals today, that is information. Push a baseline snapshot row, or a top-N fallback, anything structured. Empty datasets get your actor flagged as "under maintenance" and kill buyer trust. A customer who pays for 5 rows of "nothing unusual today, here is the current state" is happier than one who pays for nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the portfolio looks like
&lt;/h2&gt;

&lt;p&gt;10 actors, all pay-per-result ($0.002 to $0.005): funding rate arbitrage across 4 exchanges, a Bitcoin market pulse score, new-token momentum with anti-rug filters, news trend acceleration, odds movement for soccer and horse racing, CS2 skin prices, prediction market comparison (Polymarket vs Kalshi), unusual options activity from CBOE, and an address-history lookup.&lt;/p&gt;

&lt;p&gt;All of them: &lt;a href="https://apify.com/0xgollum" rel="noopener noreferrer"&gt;https://apify.com/0xgollum&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Happy to answer questions about any of this, especially the pricing/margin math, which nobody seems to write about.&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>sidehustle</category>
      <category>buildinpublic</category>
    </item>
  </channel>
</rss>
