<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Muhammad Imran</title>
    <description>The latest articles on DEV Community by Muhammad Imran (@mimranaslam500).</description>
    <link>https://dev.to/mimranaslam500</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4171944%2F1d757bfb-d6a0-401f-bb5a-08aae46102f2.png</url>
      <title>DEV Community: Muhammad Imran</title>
      <link>https://dev.to/mimranaslam500</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mimranaslam500"/>
    <language>en</language>
    <item>
      <title>Math.random() Is Not Enough: Generating Truly Secure Passwords in JavaScript</title>
      <dc:creator>Muhammad Imran</dc:creator>
      <pubDate>Thu, 15 Oct 2026 07:31:44 +0000</pubDate>
      <link>https://dev.to/mimranaslam500/mathrandom-is-not-enough-generating-truly-secure-passwords-in-javascript-3oaa</link>
      <guid>https://dev.to/mimranaslam500/mathrandom-is-not-enough-generating-truly-secure-passwords-in-javascript-3oaa</guid>
      <description>&lt;p&gt;Most password generators you find in tutorials share the same flaw. They call &lt;code&gt;Math.random()&lt;/code&gt;, pick characters from a string, and call it a day. The passwords look random. They are not random enough.&lt;/p&gt;

&lt;p&gt;This matters because a predictable generator is barely better than no generator at all. If someone can guess the internal state of your random source, they can reproduce every "random" password it ever produced.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Math.random() is not safe
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;Math.random()&lt;/code&gt; is a pseudorandom number generator. It starts from a seed and produces a long but fully deterministic sequence. In V8 (the engine behind Chrome and Node.js) the algorithm is xorshift128+. It is fast and fine for games, shuffling a playlist, or picking a random quote.&lt;/p&gt;

&lt;p&gt;It is not fine for security, for three reasons:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;It is predictable.&lt;/strong&gt; Given enough consecutive outputs, the internal state can be reconstructed, and every future value becomes known. Researchers have demonstrated this repeatedly against real-world token generators.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is not seeded by you.&lt;/strong&gt; You cannot give it a strong seed, and you cannot verify the quality of the seed the engine chose.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MDN says so directly.&lt;/strong&gt; The documentation warns that &lt;code&gt;Math.random()&lt;/code&gt; does not provide cryptographically secure random numbers and must not be used for anything security related.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Passwords, reset tokens, session IDs, and API keys are all security related.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to use instead
&lt;/h2&gt;

&lt;p&gt;Browsers and Node.js ship a cryptographic random source: the Web Crypto API.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;array&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Uint32Array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getRandomValues&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;array&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;array&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// four unpredictable 32-bit numbers&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;crypto.getRandomValues()&lt;/code&gt; pulls from the operating system's entropy pool, the same source used for TLS keys. It is available in every modern browser and in Node.js 19+ as a global (earlier via &lt;code&gt;require('crypto').webcrypto&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;For single IDs, &lt;code&gt;crypto.randomUUID()&lt;/code&gt; gives you a v4 UUID in one call:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;randomUUID&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="c1"&gt;// "3f6d4a2e-9b1c-4d5f-8a2e-7c9d1b3f5a6e"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The math that actually matters: entropy
&lt;/h2&gt;

&lt;p&gt;A password's strength is measured in bits of entropy. The formula is simple:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;entropy = length x log2(charset size)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Some concrete numbers:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;8 lowercase letters: 8 x 4.7 = ~38 bits. Crackable in hours on a single GPU.&lt;/li&gt;
&lt;li&gt;12 characters from letters, digits, and symbols (94 chars): 12 x 6.55 = ~79 bits. Out of reach for brute force.&lt;/li&gt;
&lt;li&gt;16 characters from the same set: ~105 bits. Comfortable margin for years.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Length beats cleverness. A 16-character password from a plain generator is stronger than an 8-character one full of symbols that you struggle to remember. This is also why passphrases work: four random words from a 7,776-word list give about 51 bits, and they are easy to type.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building a secure generator
&lt;/h2&gt;

&lt;p&gt;Here is a compact generator that does it right:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;securePassword&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;charset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
    &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;abcdefghijklmnopqrstuvwxyz&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
    &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;ABCDEFGHIJKLMNOPQRSTUVWXYZ&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
    &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;0123456789&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
    &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;!@#$%^&amp;amp;*()-_=+[]{};:,.&amp;lt;&amp;gt;?&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;randomValues&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Uint32Array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nx"&gt;crypto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getRandomValues&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;randomValues&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;password&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// modulo bias is negligible here for typical lengths,&lt;/span&gt;
    &lt;span class="c1"&gt;// but rejection sampling removes it entirely&lt;/span&gt;
    &lt;span class="nx"&gt;password&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nx"&gt;charset&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;randomValues&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="nx"&gt;charset&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;password&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;securePassword&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt; &lt;span class="c1"&gt;// e.g. "kQ7$mZ2@pL9#vX4!"&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;securePassword&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few notes on this code:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;crypto.getRandomValues()&lt;/code&gt; fills the whole array in one call, which is faster than calling it per character.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;% charset.length&lt;/code&gt; introduces a tiny modulo bias because 2^32 is not a multiple of 91. For password generation this bias is negligible, but if you want it gone, discard values above the largest multiple of the charset size and draw again.&lt;/li&gt;
&lt;li&gt;The function runs entirely client-side. The password never leaves the browser, which is exactly how a generator should behave.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you want to go further, add options users actually ask for: exclude ambiguous characters (&lt;code&gt;0&lt;/code&gt;, &lt;code&gt;O&lt;/code&gt;, &lt;code&gt;l&lt;/code&gt;, &lt;code&gt;1&lt;/code&gt;), guarantee at least one character from each group, or generate a memorable passphrase from a word list instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mistakes to avoid
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Seeding with &lt;code&gt;Date.now()&lt;/code&gt;.&lt;/strong&gt; Timestamps are guessable to the millisecond. An attacker who knows roughly when a password was created can brute force the seed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Using &lt;code&gt;Math.random()&lt;/code&gt; for reset tokens or API keys.&lt;/strong&gt; Same predictability problem, worse consequences, because tokens are often transmitted and logged.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Short lengths with big charsets.&lt;/strong&gt; Eight characters is short no matter how exotic the symbols are. Default to 16 or more.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rolling your own crypto.&lt;/strong&gt; You do not need a custom shuffling algorithm. The platform already gives you a vetted one.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The one-line version
&lt;/h2&gt;

&lt;p&gt;If you just need a strong password right now and do not want to write code, there is a free client-side password generator at &lt;a href="https://handynest.live/tools/password-generator.html" rel="noopener noreferrer"&gt;https://handynest.live/tools/password-generator.html&lt;/a&gt; that uses this same approach. Nothing is sent to a server.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;Random-looking is not random. &lt;code&gt;Math.random()&lt;/code&gt; is a fast PRNG for non-security tasks. For passwords, tokens, and keys, reach for &lt;code&gt;crypto.getRandomValues()&lt;/code&gt;, make the output long enough to carry real entropy, and keep the whole thing client-side. It is about ten lines of code, and it is the difference between a password that looks strong and one that actually is.&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>security</category>
      <category>webdev</category>
      <category>tutorial</category>
    </item>
    <item>
      <title>HTTP vs SOCKS5 Proxies: Which One Should You Actually Use?</title>
      <dc:creator>Muhammad Imran</dc:creator>
      <pubDate>Mon, 12 Oct 2026 09:15:06 +0000</pubDate>
      <link>https://dev.to/mimranaslam500/http-vs-socks5-proxies-which-one-should-you-actually-use-i6h</link>
      <guid>https://dev.to/mimranaslam500/http-vs-socks5-proxies-which-one-should-you-actually-use-i6h</guid>
      <description>&lt;h1&gt;
  
  
  HTTP vs SOCKS5 Proxies: Which One Should You Actually Use?
&lt;/h1&gt;

&lt;p&gt;You grabbed a proxy from a free list, pasted it into your scraper, and it half-works. Pages load but stall. Connections drop. Your code is fine. The proxy is alive. So what broke?&lt;/p&gt;

&lt;p&gt;Probably the protocol. HTTP and SOCKS5 proxies look interchangeable on a proxy list, but they behave very differently under the hood. Pick the wrong one and you get weird failures that are painful to debug. Here is how they differ and how to choose.&lt;/p&gt;

&lt;h2&gt;
  
  
  What an HTTP proxy actually does
&lt;/h2&gt;

&lt;p&gt;An HTTP proxy understands HTTP traffic. When you send a request through it, the proxy reads your request line and headers, forwards them to the target server, and passes the response back. It can cache responses, filter content, and log what passes through.&lt;/p&gt;

&lt;p&gt;That is also its limit. It only speaks HTTP (and HTTPS via the CONNECT method, which just opens a blind tunnel). If your traffic is not web traffic, an HTTP proxy cannot carry it.&lt;/p&gt;

&lt;p&gt;For most web scraping this is fine. Your Python requests code only ever makes HTTP calls, so an HTTP proxy slots right in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;proxies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://203.0.113.10:8080&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http://203.0.113.10:8080&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://example.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you work with &lt;a href="https://proxynest.live/protocols/http" rel="noopener noreferrer"&gt;HTTP proxies&lt;/a&gt; on a daily basis, you will notice they are the most common type on every &lt;a href="https://proxynest.live/proxy-list" rel="noopener noreferrer"&gt;free proxy list&lt;/a&gt;. That abundance is their main selling point: easy to find, easy to replace when one dies.&lt;/p&gt;

&lt;h2&gt;
  
  
  What SOCKS5 does differently
&lt;/h2&gt;

&lt;p&gt;SOCKS5 does not care what your traffic is. It works at a lower level, shuttling raw TCP (and UDP) packets between you and the destination without reading or interpreting them. Your application asks for a connection, SOCKS5 opens it, and then it gets out of the way.&lt;/p&gt;

&lt;p&gt;Three practical differences follow from that:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;UDP support.&lt;/strong&gt; SOCKS5 can carry UDP traffic. HTTP proxies cannot. If your use case involves anything beyond plain web requests, DNS-over-UDP quirks, VoIP, or gaming traffic, SOCKS5 is your only option of the two.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Authentication built in.&lt;/strong&gt; SOCKS5 has native username/password auth in the protocol. With HTTP proxies, auth is bolted on via headers and behaves inconsistently across servers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No header meddling.&lt;/strong&gt; A SOCKS proxy does not touch your headers. With misconfigured HTTP proxies you sometimes get rewritten headers or stripped cookies, which breaks login flows and fingerprint-sensitive scraping.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;In Python, SOCKS5 needs a small helper because requests does not speak SOCKS natively:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="c1"&gt;# pip install requests[socks]
&lt;/span&gt;&lt;span class="n"&gt;proxies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;socks5://203.0.113.20:1080&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;socks5://203.0.113.20:1080&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://example.com&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The code looks identical. The transport underneath is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest comparison
&lt;/h2&gt;

&lt;p&gt;HTTP proxy vs SOCKS5 proxy:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Traffic types: HTTP/HTTPS only vs any TCP or UDP traffic&lt;/li&gt;
&lt;li&gt;Speed for web requests: slightly faster vs slightly slower&lt;/li&gt;
&lt;li&gt;Authentication: via headers, inconsistent vs native in-protocol auth&lt;/li&gt;
&lt;li&gt;Header handling: can modify or strip headers vs passes everything through untouched&lt;/li&gt;
&lt;li&gt;Availability on free lists: very common vs less common, dies faster&lt;/li&gt;
&lt;li&gt;Best for: web scraping, browsing vs bots, streaming, gaming, anything with UDP&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Neither is "better" in general. The right question is always: what traffic are you sending?&lt;/p&gt;

&lt;h2&gt;
  
  
  When to use which
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Use HTTP when:&lt;/strong&gt; you are scraping websites or doing anything that is purely web traffic. HTTP proxies are faster for this, far easier to find for free, and your existing code already supports them. If you just need a quick proxy for a script, grab one from a free list and go.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Use SOCKS5 when:&lt;/strong&gt; you need UDP, you need reliable authentication, or you are running into weird issues that smell like header interference. Scrapers that log in to accounts, tools that tunnel non-web protocols, and anything latency-sensitive over UDP belong on &lt;a href="https://proxynest.live/protocols/socks5" rel="noopener noreferrer"&gt;SOCKS5&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;One more scenario: &lt;strong&gt;rotation pools.&lt;/strong&gt; If you rotate proxies per request, mixing protocols in one pool is a classic source of silent failures. Keep HTTP and SOCKS5 proxies in separate pools and route traffic accordingly. A scraper that randomly gets a SOCKS5 proxy for an HTTP-only code path (or vice versa) will produce the exact kind of intermittent breakage that eats your afternoon.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before you trust any proxy, test it
&lt;/h2&gt;

&lt;p&gt;Protocol choice only matters if the proxy itself works, and most free proxies do not. Roughly half of freshly listed free proxies never respond at all, and of the ones that do, about half die within hours.&lt;/p&gt;

&lt;p&gt;So test first, code second. A &lt;a href="https://proxynest.live/tools/proxy-checker" rel="noopener noreferrer"&gt;proxy checker&lt;/a&gt; that reports protocol, anonymity level, and response time tells you in seconds whether a proxy is worth your time. Check the protocol field specifically: plenty of lists label everything "HTTP" when it is not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;Scraping websites? HTTP is fine and plentiful. Need UDP, clean headers, or real auth? SOCKS5. Test every proxy before you trust it, keep your protocol pools separate, and stop debugging code that was never broken. The proxy was the problem all along.&lt;/p&gt;

</description>
      <category>proxy</category>
      <category>python</category>
      <category>networking</category>
      <category>webdev</category>
    </item>
    <item>
      <title>How to Build a Proxy Rotator in Python (That Actually Works)</title>
      <dc:creator>Muhammad Imran</dc:creator>
      <pubDate>Thu, 08 Oct 2026 18:38:14 +0000</pubDate>
      <link>https://dev.to/mimranaslam500/how-to-build-a-proxy-rotator-in-python-that-actually-works-3nhh</link>
      <guid>https://dev.to/mimranaslam500/how-to-build-a-proxy-rotator-in-python-that-actually-works-3nhh</guid>
      <description>&lt;p&gt;If you have ever run a web scraper for more than a few hours, you know the pattern. It works great at first. Then the requests start failing. Then your IP gets blocked entirely. The site was not even angry at you, you just looked like a bot because every request came from the same address.&lt;/p&gt;

&lt;p&gt;The fix is rotating proxies: send each request (or every few requests) through a different IP address. It sounds complicated, but the core idea fits in about 40 lines of Python. Here is how I built mine.&lt;/p&gt;

&lt;h2&gt;
  
  
  What you need
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Python 3.8+&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;requests&lt;/code&gt; library (&lt;code&gt;pip install requests&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;A list of proxies (I will show you where to get free ones)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 1: Get a proxy list
&lt;/h2&gt;

&lt;p&gt;You need a pool of proxies to rotate through. Free proxy lists work fine for learning and light jobs. I maintain &lt;a href="https://proxynest.live/proxy-list" rel="noopener noreferrer"&gt;ProxyNest&lt;/a&gt;, a free proxy list that retests its entries continuously, and it has a simple JSON API:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;GET https://proxynest.live/api/proxies?limit=50
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That returns a list of live proxies with their IP, port, protocol (http/socks4/socks5), and country. For this tutorial you can also paste proxies from any list into a text file, one per line in &lt;code&gt;ip:port&lt;/code&gt; format.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: Build the rotator
&lt;/h2&gt;

&lt;p&gt;The idea is simple. Keep a list of proxies, pick one at random (or round-robin) for each request, and drop the ones that fail so they do not poison your pool.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;

&lt;span class="n"&gt;PROXY_API&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://proxynest.live/api/proxies?limit=100&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch_proxies&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;PROXY_API&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;proxies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;protocol&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;://&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ip&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;port&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;proxies&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ProxyRotator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;proxies&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch_proxies&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;good&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;proxies&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;good&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Proxy pool exhausted, refetching...&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;choice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;good&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;mark_bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;http&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

&lt;span class="n"&gt;rotator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ProxyRotator&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the whole rotator. Ten lines of real logic.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: Use it in your scraper
&lt;/h2&gt;

&lt;p&gt;Now wrap your requests so a dead proxy does not kill the run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;proxy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;rotator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;proxies&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Mozilla/5.0 (compatible; my-scraper/1.0)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RequestException&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;rotator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mark_bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Failed to fetch &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; after &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; retries&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://example.com/products&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;characters fetched&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few things to notice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dead proxies get removed.&lt;/strong&gt; &lt;code&gt;mark_bad&lt;/code&gt; takes failing proxies out of the pool instead of retrying them forever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Random selection beats round-robin&lt;/strong&gt; for small pools, because it spreads load unevenly and looks less mechanical.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The User-Agent matters.&lt;/strong&gt; Rotating proxies but sending the same default &lt;code&gt;python-requests&lt;/code&gt; user agent is like wearing a disguise and then shouting your name. Set a realistic one.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Step 4: Do not be rude
&lt;/h2&gt;

&lt;p&gt;Rotation is not a license to hammer a site. Add delays between requests (&lt;code&gt;time.sleep(1)&lt;/code&gt; at minimum), respect &lt;code&gt;robots.txt&lt;/code&gt;, and keep your concurrency sane. Most blocks happen because of request rate, not because of proxies. The proxy just buys you a second identity; it does not make aggressive scraping polite.&lt;/p&gt;

&lt;p&gt;If you need serious throughput, free proxies will eventually frustrate you. They die fast and they are slow. But for learning, prototypes, and light scheduled jobs, this setup works surprisingly well.&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaway
&lt;/h2&gt;

&lt;p&gt;Proxy rotation is one of those things that sounds like infrastructure but is really just a list and a random choice. Fetch a pool, pick one per request, drop the failures, and be polite about your request rate. That is 90 percent of it.&lt;/p&gt;




&lt;h2&gt;
  
  
  &lt;em&gt;Muhammad Imran works on network tooling and runs ProxyNest, a free directory of public proxies with a live API and a proxy checker that re-verifies any proxy on demand.&lt;/em&gt;
&lt;/h2&gt;

</description>
      <category>python</category>
      <category>webdev</category>
      <category>webscraping</category>
      <category>tutorial</category>
    </item>
  </channel>
</rss>
