<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Vlad Oso</title>
    <description>The latest articles on DEV Community by Vlad Oso (@vladoso).</description>
    <link>https://dev.to/vladoso</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4195287%2Fe7b311e6-92b9-4998-b80f-65bf0b96ffb6.png</url>
      <title>DEV Community: Vlad Oso</title>
      <link>https://dev.to/vladoso</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/vladoso"/>
    <language>en</language>
    <item>
      <title>How to Check If a WhatsApp Number Is Active — Without Sending It a Message</title>
      <dc:creator>Vlad Oso</dc:creator>
      <pubDate>Wed, 21 Oct 2026 23:20:57 +0000</pubDate>
      <link>https://dev.to/vladoso/how-to-check-if-a-whatsapp-number-is-active-without-sending-it-a-message-22c1</link>
      <guid>https://dev.to/vladoso/how-to-check-if-a-whatsapp-number-is-active-without-sending-it-a-message-22c1</guid>
      <description>&lt;p&gt;Every team that sends WhatsApp messages at scale eventually asks the same question: &lt;em&gt;can I tell whether this number is actually on WhatsApp before I try to message it?&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Sending a probe message is the obvious answer and the worst one. It costs money, it can hit a stranger who inherited a recycled number, and enough of those will drag your sender quality rating down until Meta cuts your messaging limits. This post walks through what you can learn about a number &lt;em&gt;without&lt;/em&gt; contacting it, where each signal stops being useful, and how to wire it into a list-cleaning job.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost of not knowing
&lt;/h2&gt;

&lt;p&gt;Phone lists decay fast. Numbers get disconnected, SIMs churn, and in prepaid-heavy markets a number you collected eighteen months ago may belong to a different person today. A realistic decay figure for an e-commerce customer list is 20–30% bad or unreachable entries.&lt;/p&gt;

&lt;p&gt;Run the math on a 100,000-number list with 25% decay:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Direct send costs.&lt;/strong&gt; WhatsApp template messages are billed per delivered message, and SMS fallback (which many stacks trigger automatically when WhatsApp delivery fails) is billed per segment whether the number is alive or not. At $0.01–0.05 per segment, one campaign can burn $250–$1,250 on numbers that were never reachable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wrong-person costs.&lt;/strong&gt; Recycled numbers that &lt;em&gt;are&lt;/em&gt; on WhatsApp receive your message — and strangers block and report. Blocks and reports feed your quality rating, and a downgraded rating means lower throughput tiers for &lt;em&gt;all&lt;/em&gt; your traffic, not just the bad rows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Operational costs.&lt;/strong&gt; Every failed or misdirected conversation lands in a CRM as a dead lead that sales keeps following up on.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A pre-flight check that costs fractions of a cent per number pays for itself if it removes even a few percent of the dead weight. The question is which check.&lt;/p&gt;

&lt;h2&gt;
  
  
  Signal 1: Line status (HLR-style lookups)
&lt;/h2&gt;

&lt;p&gt;The oldest tool in this space is the HLR lookup. The Home Location Register is the carrier-side database of subscribers, and lookup services query it (directly or through aggregators) to answer questions about the &lt;em&gt;line&lt;/em&gt;, not the apps on it.&lt;/p&gt;

&lt;p&gt;An HLR/line-status check can typically tell you:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Whether the number is allocated and currently active on a mobile network&lt;/li&gt;
&lt;li&gt;The current network (MCC/MNC) and whether the number has been ported&lt;/li&gt;
&lt;li&gt;In some implementations, a coarse reachability signal for the device&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What it cannot tell you is the thing you actually care about: &lt;strong&gt;whether that line has a WhatsApp account.&lt;/strong&gt; A number can be a perfectly healthy, active SIM in a drawer, owned by someone who has never installed WhatsApp. Conversely the mapping between "line active" and "app registered" is loose enough in both directions that HLR alone is a weak predictor of messageability.&lt;/p&gt;

&lt;p&gt;Use line status for what it's good at: killing clearly dead numbers cheaply, and detecting porting/country mismatches. Don't treat "active line" as "reachable on WhatsApp."&lt;/p&gt;

&lt;h2&gt;
  
  
  Signal 2: Registration status
&lt;/h2&gt;

&lt;p&gt;Registration is a different layer. WhatsApp accounts are keyed by phone number in the service's own account registry, so "is this number registered on WhatsApp" is a question about the app's database, not the carrier's. That's why it needs a different kind of check than an HLR query — and why the result is more actionable: it maps almost one-to-one to "a message to this number can be delivered to a WhatsApp client."&lt;/p&gt;

&lt;p&gt;A registration check returns a discrete status per number rather than a score. Used well:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;wa_registered&lt;/code&gt; → safe to include in WhatsApp sends&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;dead&lt;/code&gt; → line is disconnected; drop it everywhere, including SMS&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;inactive&lt;/code&gt; → line exists but shows no recent app activity; segment separately, don't lead with it&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;unknown&lt;/code&gt; → couldn't be determined; fall back to your default policy&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two honest caveats. First, registration state is a snapshot — a number checked last month can churn this month, so re-validate lists you keep for a long time. Second, neither signal verifies &lt;em&gt;identity&lt;/em&gt;: a registered number tells you nothing about whether it still belongs to the customer who gave it to you. Recycled-number risk shrinks with list hygiene, it doesn't disappear.&lt;/p&gt;

&lt;h3&gt;
  
  
  A note on scope
&lt;/h3&gt;

&lt;p&gt;Everything above assumes you're verifying &lt;strong&gt;your own first-party data&lt;/strong&gt; — numbers your users gave you at checkout or signup, with consent to be contacted. That's list hygiene. Running registration checks against scraped or purchased lists to build outreach targets is a different activity, it's what gets senders banned and providers shut down, and it's out of scope for this approach entirely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trying it: one number, one curl
&lt;/h2&gt;

&lt;p&gt;TrueReach exposes both checks through a single endpoint. A real-time lookup looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-X&lt;/span&gt; POST https://truereach.one/v1/verify &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &amp;lt;redacted&amp;gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Content-Type: application/json"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-d&lt;/span&gt; &lt;span class="s1"&gt;'{"phone": "5511987654321"}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"phone"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"5511987654321"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"active"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"checked_at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-21T02:22:38Z"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The response is deliberately boring: the number, a status, and a timestamp. Boring is what you want from infrastructure you call in a loop.&lt;/p&gt;

&lt;h2&gt;
  
  
  Batch: cleaning a list before a campaign
&lt;/h2&gt;

&lt;p&gt;Real-time calls are for ones and twos. For a list, the pattern is upload → poll → download the same rows back with a status column attached. The shape of the job:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://truereach.one&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;H&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Authorization&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Bearer YOUR_API_KEY&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# 1. Upload the list (CSV with a phone column)
&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customers.csv&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;task&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/v1/verify/batch&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                         &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;file&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;}).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="c1"&gt;# 2. Poll until done
&lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/v1/verify/tasks/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;task_id&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;done&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 3. Download results and keep only sendable rows
&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;API&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/v1/verify/tasks/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;task_id&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/download&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;H&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;csv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;DictReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sendable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verification_status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wa_registered&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sendable&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; numbers cleared to send&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In practice you'd persist the statuses back into your CRM rather than re-checking the whole list every campaign — check once at intake, re-check on a decay schedule.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which mode when
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use real-time single checks when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A number is arriving &lt;em&gt;right now&lt;/em&gt; — checkout, signup, lead form — and you want to validate before storing it or triggering an OTP&lt;/li&gt;
&lt;li&gt;Volume is low (hundreds a day) and latency matters more than unit cost&lt;/li&gt;
&lt;li&gt;You're gating something downstream: no point sending a confirmation flow to a dead line&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Use batch when:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You're about to run a campaign against an existing list (this is the main event)&lt;/li&gt;
&lt;li&gt;You're doing periodic hygiene on a CRM that's accumulated months of decay&lt;/li&gt;
&lt;li&gt;Volume is in the thousands-plus, where per-call overhead dominates&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Pricing follows the same split: batch checks run $0.002 per number, real-time singles $0.0025 — cheap enough that the decision should be driven by workflow, not cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  Closing
&lt;/h2&gt;

&lt;p&gt;"Is this number on WhatsApp" sounds like one question but it's two — one about the carrier's line, one about the app's registry — and conflating them is how teams end up paying to message drawer SIMs. Check line status to kill dead numbers, check registration status to decide who gets a message, and do both before you send rather than learning from your block rate after.&lt;/p&gt;

&lt;p&gt;If you want to run the checks above against your own list, &lt;a href="https://truereach.one" rel="noopener noreferrer"&gt;TrueReach&lt;/a&gt; gives new accounts 2,000 free checks, and the full API surface is in the &lt;a href="https://truereach.one/api-docs" rel="noopener noreferrer"&gt;API docs&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>whatsapp</category>
      <category>api</category>
      <category>webdev</category>
      <category>saas</category>
    </item>
    <item>
      <title>I Tested 5 Proxy Types for Web Scraping — Here's What Actually Matters</title>
      <dc:creator>Vlad Oso</dc:creator>
      <pubDate>Tue, 20 Oct 2026 04:09:38 +0000</pubDate>
      <link>https://dev.to/vladoso/i-tested-5-proxy-types-for-web-scraping-heres-what-actually-matters-5flb</link>
      <guid>https://dev.to/vladoso/i-tested-5-proxy-types-for-web-scraping-heres-what-actually-matters-5flb</guid>
      <description>&lt;p&gt;Last month I rebuilt a scraping pipeline that had been limping along on a bargain-bin datacenter proxy plan. The symptoms were familiar to anyone who's done this at any scale: success rate hovering around 60%, CAPTCHAs on every third session, and a monthly bill that made no sense relative to the data we were actually collecting.&lt;/p&gt;

&lt;p&gt;So I did what any reasonable developer would do: I spent two weeks testing every proxy type I could get my hands on against the same set of targets, with the same code, and wrote down what actually happened.&lt;/p&gt;

&lt;p&gt;This post is the write-up. No affiliate nonsense, no "top 10 providers" list — just what each proxy type is, when it works, when it fails, and the code patterns that matter regardless of which type you pick.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Setup
&lt;/h2&gt;

&lt;p&gt;Test workload: product pages from a handful of mid-size e-commerce sites, a couple of search-heavy directories, and one notoriously aggressive social platform. Roughly 50k requests per proxy type, spread across a week, using Python + &lt;code&gt;requests&lt;/code&gt;. Same headers, same throttling (2–4s between requests per host), same retry logic. The only variable was the proxy layer.&lt;/p&gt;

&lt;p&gt;Here's the baseline code everything else builds on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;PROXY&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://user:pass@proxy.example.com:8080&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="n"&gt;proxies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;PROXY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;PROXY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0 (Windows NT 10.0; Win64; x64) &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;AppleWebKit/537.36 (KHTML, like Gecko) &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Chrome/128.0.0.0 Safari/537.36&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
            &lt;span class="p"&gt;),&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Accept-Language&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;en-US,en;q=0.9&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you're copy-pasting proxy examples from blog posts, stop here and internalize one thing: &lt;strong&gt;most scraping failures are not proxy failures&lt;/strong&gt;. They're fingerprint failures, rate failures, or session failures that a better proxy happens to mask. Keep that in mind as we go through the types, because the marketing for each type will happily take credit for all three.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. Datacenter Proxies
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What they are:&lt;/strong&gt; IPs allocated to hosting providers and cloud ranges (AWS, Hetzner, OVH, and thousands of smaller hosts). Cheap, fast, and trivially identifiable — the ASN alone gives them away.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Results:&lt;/strong&gt; ~55–65% success on the e-commerce targets, near-zero on the social platform. Latency was excellent (often under 300ms to first byte). Cost per GB was the lowest of anything tested by a wide margin.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When to use them:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Targets with weak or no bot protection (public APIs, small sites, RSS-like endpoints)&lt;/li&gt;
&lt;li&gt;High-volume, low-value requests where a 40% failure rate is acceptable because retries are cheap&lt;/li&gt;
&lt;li&gt;Geo-checking and uptime monitoring, where you just need &lt;em&gt;an&lt;/em&gt; IP in a country, not a trustworthy one&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;When to avoid:&lt;/strong&gt; Anything behind Cloudflare Bot Management, DataDome, PerimeterX, or Akamai in a strict configuration. You'll burn your retry budget and learn nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Residential Proxies (Rotating)
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What they are:&lt;/strong&gt; IPs belonging to real consumer ISPs, routed through devices on home connections. Sold by the GB, usually with per-request or timed rotation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Results:&lt;/strong&gt; ~92–96% success on e-commerce, ~80% on the social platform with careful session handling. Latency varied wildly — 800ms on a good exit node, 4s+ on a bad one. This variance is the price of admission.&lt;/p&gt;

&lt;p&gt;The single biggest improvement in my whole test came from rotation &lt;em&gt;strategy&lt;/em&gt;, not from switching providers. Two patterns matter:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Per-request rotation&lt;/strong&gt; — fresh IP every request. Good for stateless, high-parallelism workloads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;string&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;session_id&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;string&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ascii_lowercase&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;string&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;digits&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rotating_proxy&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# Most rotating residential gateways encode the session in the username.
&lt;/span&gt;    &lt;span class="c1"&gt;# A new session ID = a new exit IP.
&lt;/span&gt;    &lt;span class="n"&gt;user&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;customer-&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;session_id&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;proxy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:password@gateway.example.com:8080&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Sticky sessions&lt;/strong&gt; — same IP for N minutes. Mandatory for anything with login state, carts, or multi-step flows:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;StickySession&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;One exit IP per logical task; rotate only when the task ends or fails.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ttl_requests&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ttl&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ttl_requests&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_new_session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_new_session&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Session&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;sid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;session_id&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;proxy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://customer-&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;sid&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:password@gateway.example.com:8080&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;proxies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ttl&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_new_session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Honest trade-off:&lt;/strong&gt; you're paying per GB for traffic that transits someone's home connection. That makes residential the wrong tool for downloading large payloads, and the &lt;em&gt;right&lt;/em&gt; tool when the cost of a blocked request (a missed price change, a dead account) exceeds the cost of the bandwidth.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Mobile Proxies
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What they are:&lt;/strong&gt; IPs from carrier networks (4G/5G). The magic here is CGNAT: hundreds or thousands of real subscribers share a single public IP, so aggressive blocking hurts the site's real users — which means sites are &lt;em&gt;very&lt;/em&gt; reluctant to block mobile IPs outright.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Results:&lt;/strong&gt; Highest success rate of the test (~97% even on the social platform). Also the slowest and most expensive per GB, typically 3–10× residential pricing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When to use them:&lt;/strong&gt; Social platforms, account-adjacent automation, anywhere a block is catastrophic rather than merely annoying.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When to avoid:&lt;/strong&gt; Bulk scraping. You'll go bankrupt feeding a price monitor through mobile bandwidth. Also note that mobile IPs rotate on their own schedule (tower handoffs, NAT timeouts), so long sticky sessions are not guaranteed no matter what the sales page says.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. ISP Proxies
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;What they are:&lt;/strong&gt; Datacenter-hosted IPs that are &lt;em&gt;registered&lt;/em&gt; to consumer ISPs. You get the speed and stability of a datacenter connection with an ASN that looks residential. Typically sold per-IP, per-month rather than per-GB.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Results:&lt;/strong&gt; ~90% success on e-commerce, stable latency (~400ms), and — uniquely in this test — the same IPs stayed fast and unblocked for the entire week. No rotation lottery.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When to use them:&lt;/strong&gt; Long-running jobs against a fixed set of targets, monitoring tasks, anything where you want residential-grade trust with datacenter-grade predictability and a flat monthly bill.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When to avoid:&lt;/strong&gt; Workloads that need thousands of distinct IPs. Per-IP pricing means breadth costs money, and if you hammer one ISP IP hard enough, it &lt;em&gt;will&lt;/em&gt; get flagged eventually — the pool you can fall back to is much smaller than a rotating residential pool.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Static (Dedicated) Proxies
&lt;/h2&gt;

&lt;p&gt;A quick taxonomy note, because vendors abuse these terms: "static" describes &lt;em&gt;behavior&lt;/em&gt; (the IP doesn't change), not a network type. A static proxy can be a datacenter IP or an ISP IP that you keep for the duration of your plan. In my test setup, static ISP was the workhorse configuration for the monitoring jobs — one IP per target site, requests spread politely, zero surprises across two weeks.&lt;/p&gt;

&lt;p&gt;The general rule: &lt;strong&gt;rotating for breadth, static for depth.&lt;/strong&gt; If a job revisits the same target repeatedly and failure has consequences, static wins. If a job fans out across many targets and individual failures are cheap, rotate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Detecting Blocks (and Not Lying to Yourself About It)
&lt;/h2&gt;

&lt;p&gt;The most dangerous scraping system is one that reports 100% success while quietly ingesting garbage. Sites defend themselves on a spectrum:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Hard blocks:&lt;/strong&gt; 403, 429, connection resets. Easy to detect, easy to handle.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Soft blocks:&lt;/strong&gt; HTTP 200 with a CAPTCHA page, a "verify you're human" interstitial, or an empty-but-valid HTML shell. &lt;em&gt;This is where pipelines go to die.&lt;/em&gt; Your parser returns empty results and nobody notices for a week.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Honeypots and poisoned data:&lt;/strong&gt; 200 with subtly wrong data — prices off by a digit, shuffled listings, fake out-of-stock flags. Rare, but they exist specifically to corrupt datasets like yours.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Detection needs to be structural, not status-code-based:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;BLOCK_MARKERS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;captcha&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;verify you are human&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;access denied&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;are you a robot&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unusual traffic&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;px-captcha&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expected_selector_found&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;403&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;429&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;503&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;hard_block&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;BLOCK_MARKERS&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;soft_block&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;expected_selector_found&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;suspect&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;  &lt;span class="c1"&gt;# 200 but the data isn't there — treat as failure
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Track these categories separately in your metrics. If your dashboard only shows HTTP status codes, you're flying blind.&lt;/p&gt;

&lt;p&gt;And the retry logic that ties it together — retry on a &lt;em&gt;different&lt;/em&gt; IP for hard/soft blocks, back off on 429s, and never retry a poisoned response:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch_with_recovery&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;StickySession&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RequestException&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;verdict&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;classify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;expected_selector_found&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;has_data&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verdict&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;verdict&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;suspect&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="c1"&gt;# Possible honeypot — log it, don't ingest, don't blindly retry
&lt;/span&gt;                &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;SuspiciousResponse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="c1"&gt;# Hard block, soft block, or network error: new exit IP, then back off
&lt;/span&gt;        &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_new_session&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;task&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;BlockedError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; failed after &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; attempts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Cost Optimization: Stop Burning Residential GB on JPEGs
&lt;/h2&gt;

&lt;p&gt;Per-GB pricing means every byte counts, and the default behavior of most scraping stacks is a bonfire. In my first pipeline version, &lt;strong&gt;over 60% of residential traffic was images, fonts, and CSS&lt;/strong&gt; — assets I was never going to parse.&lt;/p&gt;

&lt;p&gt;The fixes, in order of impact:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Block static assets.&lt;/strong&gt; If you're parsing HTML, you need HTML. In &lt;code&gt;requests&lt;/code&gt; this means not following asset URLs at all; in browser automation (Playwright/Selenium), intercept and abort image/font/media requests. This alone halved my bill.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Route by target hardness.&lt;/strong&gt; Send the easy 80% of targets through datacenter proxies at a fraction of the cost, and reserve residential for the hosts that actually block you. A simple per-host success-rate table in your code can automate this routing after a day of data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cache aggressively at the URL level.&lt;/strong&gt; Product pages rarely change minute-to-minute. A 15-minute in-memory cache on listing pages eliminated a third of my requests entirely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compress and shrink.&lt;/strong&gt; Send &lt;code&gt;Accept-Encoding: gzip, br&lt;/code&gt; explicitly (some proxy gateways strip it), and use &lt;code&gt;HEAD&lt;/code&gt;/conditional requests where the target supports ETags.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Right-size sessions.&lt;/strong&gt; Sticky sessions that live too long waste the "freshness" you're paying for; sessions that rotate too often waste connection setup and look bot-like. 10–30 minutes is the sweet spot for most of my workloads.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;After all five, the same crawl that cost me a painful amount in week one ran at roughly &lt;strong&gt;one third of the cost&lt;/strong&gt; in week two — with a &lt;em&gt;higher&lt;/em&gt; success rate, because blocked-target routing meant residential IPs were only spent where they mattered.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Summary Table
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Trust level&lt;/th&gt;
&lt;th&gt;Speed&lt;/th&gt;
&lt;th&gt;Pricing model&lt;/th&gt;
&lt;th&gt;Best for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Datacenter&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;Fastest&lt;/td&gt;
&lt;td&gt;Per IP / per GB (cheap)&lt;/td&gt;
&lt;td&gt;Weakly defended targets, monitoring&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Residential (rotating)&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Variable&lt;/td&gt;
&lt;td&gt;Per GB&lt;/td&gt;
&lt;td&gt;General-purpose scraping, breadth&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mobile&lt;/td&gt;
&lt;td&gt;Highest&lt;/td&gt;
&lt;td&gt;Slowest&lt;/td&gt;
&lt;td&gt;Per GB (expensive)&lt;/td&gt;
&lt;td&gt;Social platforms, high-stakes requests&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ISP&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Fast + stable&lt;/td&gt;
&lt;td&gt;Per IP / month&lt;/td&gt;
&lt;td&gt;Long-running jobs, fixed targets&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Static (ISP/DC)&lt;/td&gt;
&lt;td&gt;Depends on base&lt;/td&gt;
&lt;td&gt;Stable&lt;/td&gt;
&lt;td&gt;Per IP / month&lt;/td&gt;
&lt;td&gt;Sessions, accounts, depth over breadth&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What I'd Do Differently
&lt;/h2&gt;

&lt;p&gt;Two lessons stuck. First, &lt;strong&gt;log the block taxonomy from day one&lt;/strong&gt; — retrofitting detection onto a pipeline that assumed 200 = success cost me more time than the proxy testing itself. Second, &lt;strong&gt;match the proxy to the failure cost, not to the target's reputation.&lt;/strong&gt; "This site is hard, use mobile" is worse advice than "this request is expensive to lose, use mobile."&lt;/p&gt;

&lt;p&gt;For the residential and static ISP portions of this test I used &lt;a href="https://ipstation.cc" rel="noopener noreferrer"&gt;IPStation&lt;/a&gt; ($2.50/GB dynamic residential, $3.99/month static ISP, and the 200MB free trial was enough to validate the setup before spending anything) — but honestly, every pattern in this post is provider-agnostic. Pick any reputable vendor, steal the detection and routing code, and you'll get 90% of the benefit.&lt;/p&gt;

&lt;p&gt;The proxies were never the hard part. Knowing &lt;em&gt;why&lt;/em&gt; a request failed — and what that failure should make your code do next — is the whole game.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Questions or better war stories? Drop them in the comments — especially if you've caught a target serving poisoned data in the wild. I want to hear how you detected it.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>webscraping</category>
      <category>proxy</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
