<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mary</title>
    <description>The latest articles on DEV Community by Mary (@mary112233).</description>
    <link>https://dev.to/mary112233</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4240542%2Ffbe5a52b-82a4-4f48-8072-b0b8e9e295e9.png</url>
      <title>DEV Community: Mary</title>
      <link>https://dev.to/mary112233</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mary112233"/>
    <language>en</language>
    <item>
      <title>A Search API Is Not a Data Pipeline: Building Beyond the First JSON Response</title>
      <dc:creator>Mary</dc:creator>
      <pubDate>Tue, 17 Nov 2026 08:02:20 +0000</pubDate>
      <link>https://dev.to/mary112233/a-search-api-is-not-a-data-pipeline-building-beyond-the-first-json-response-47h4</link>
      <guid>https://dev.to/mary112233/a-search-api-is-not-a-data-pipeline-building-beyond-the-first-json-response-47h4</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F56i6bgndl9jj7csf0w8x.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F56i6bgndl9jj7csf0w8x.png" alt=" "&gt;&lt;/a&gt;Getting search results into a Python script is a useful milestone.&lt;/p&gt;

&lt;p&gt;It is not the same thing as building a reliable search-data pipeline.&lt;/p&gt;

&lt;p&gt;Once those results feed a dashboard, a scheduled report, or an AI application, a different set of questions matters:&lt;/p&gt;

&lt;p&gt;What exactly did we search for?&lt;/p&gt;

&lt;p&gt;Which location and device settings did we use?&lt;/p&gt;

&lt;p&gt;When did we collect the results?&lt;/p&gt;

&lt;p&gt;Does an empty list mean “no results,” or did our integration break?&lt;/p&gt;

&lt;p&gt;Can we change providers without rewriting the application?&lt;/p&gt;

&lt;p&gt;This article uses Thordata’s SERP API as a concrete example. Its documentation describes an HTTP interface for retrieving search results, with options such as search engine, location, and device. The interesting engineering work starts after that request returns.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Define the observation before writing the request&lt;/strong&gt;
Consider a small application that tracks which pages appear for a set of product-related searches.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The tempting data model is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;text
keyword → list of URLs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That leaves out the context needed to interpret the list.&lt;/p&gt;

&lt;p&gt;A more useful model is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;text
query + search context + collection time → observed results
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A search observation should preserve the settings that produced it. Thordata’s own tutorial highlights geography, device settings, and collection timestamps as important parts of interpreting SERP data.&lt;/p&gt;

&lt;p&gt;For an application like this, I would start with an internal record shaped roughly like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;"query"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"project management software"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"engine"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"google"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"location"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"United States"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"device"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"desktop"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"requested_at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-17T10:00:00Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"received_at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-11-17T10:00:02Z"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"provider"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"thordata"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nl"&gt;"results"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a proposed application schema, not a copy of Thordata’s response format.&lt;/p&gt;

&lt;p&gt;That distinction matters: your storage model should describe what your application needs to remember, rather than inherit every detail of a vendor’s JSON structure.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Keep provider-specific parsing in one place&lt;/strong&gt;
The shortest integration passes the API response directly to the rest of the application:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;python&lt;/span&gt;
&lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch_search_results&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;update_dashboard&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a prototype, that can be enough.&lt;/p&gt;

&lt;p&gt;For a longer-lived application, I would introduce a boundary:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;text
Provider response
       ↓
Provider-specific adapter
       ↓
Application-owned search record
       ↓
Database, dashboard, or downstream consumer
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here is a minimal internal model:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;dataclasses&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;dataclass&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;typing&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Optional&lt;/span&gt;


&lt;span class="nd"&gt;@dataclass&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;frozen&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;SearchResult&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;position&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;snippet&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Optional&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The adapter is responsible for mapping the provider’s documented fields into this model.&lt;/p&gt;

&lt;p&gt;It is also where I would decide what happens when a required field is missing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;UnexpectedSearchResponse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important design choice is not the exception’s name. It is refusing to quietly convert every unexpected response into an empty list.&lt;/p&gt;

&lt;p&gt;“No organic results were returned” and “our parser no longer understands the response” should be different states.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Treat pagination as a collection problem&lt;/strong&gt;
Pagination looks simple until the underlying results change between requests.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Thordata’s Python tutorial explicitly calls out shifting result sets and duplicate URLs across search-result pages. That makes a multi-page collection different from reading slices of a fixed database snapshot.&lt;/p&gt;

&lt;p&gt;For this application, I would use three rules:&lt;/p&gt;

&lt;p&gt;Use the provider’s documented pagination mechanism.&lt;/p&gt;

&lt;p&gt;Preserve the page and position associated with each result.&lt;/p&gt;

&lt;p&gt;Deduplicate only in the downstream view that needs unique URLs.&lt;/p&gt;

&lt;p&gt;That last rule prevents a subtle loss of information.&lt;/p&gt;

&lt;p&gt;A research tool may want one entry per URL. A ranking-analysis tool may need to retain every observed position. Those are different views of the same collected data.&lt;/p&gt;

&lt;p&gt;Repeatedly sending the same request is not pagination. The adapter must actually advance the page or offset according to the current API documentation.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Make retries a budget decision&lt;/strong&gt;
A retry policy should answer two questions:&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Is another attempt likely to help?&lt;/p&gt;

&lt;p&gt;How much additional work are we willing to pay for?&lt;/p&gt;

&lt;p&gt;Thordata’s tutorial recommends bounded retries, backoff, and measured rather than automatic concurrency for per-request-billed API usage.&lt;/p&gt;

&lt;p&gt;For this project, I would handle failures according to their cause:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Invalid credentials: Stop the job and surface a configuration error. Retrying with the same credentials will not fix the problem.&lt;/li&gt;
&lt;li&gt;Invalid request parameters: Review and correct the request before trying again.&lt;/li&gt;
&lt;li&gt;Rate-limit responses: Follow the provider’s documented retry guidance and set a maximum number of attempts.&lt;/li&gt;
&lt;li&gt;Temporary upstream failures: Retry within a limited attempt count and an overall time budget.&lt;/li&gt;
&lt;li&gt;Unexpected response structures: Preserve relevant diagnostic metadata and investigate instead of silently treating the response as an empty result.
These are application-design choices, not a claim about Thordata’s exact error contract. The implementation should use its current documentation to map actual responses to these categories.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I would also avoid letting every layer retry independently. A scheduler, worker, and HTTP client should not each assume they own the full retry budget.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Measure usable output, not just successful requests&lt;/strong&gt;
For this application, I would track:&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Whether the HTTP request succeeded.&lt;/p&gt;

&lt;p&gt;Whether the response passed validation.&lt;/p&gt;

&lt;p&gt;How many usable results were extracted.&lt;/p&gt;

&lt;p&gt;How many attempts the collection required.&lt;/p&gt;

&lt;p&gt;How long the complete collection took.&lt;/p&gt;

&lt;p&gt;Whether the requested search context was recorded.&lt;/p&gt;

&lt;p&gt;A request returning successfully is only one part of the job.&lt;/p&gt;

&lt;p&gt;The application needs an observation it can interpret, store, and compare later.&lt;/p&gt;

&lt;p&gt;During development, I would retain a small, access-controlled sample of raw responses alongside the normalized records. That gives the adapter something concrete to test against when a field disappears or a response shape changes. Retention and redaction would be deliberate choices, not an instruction to log everything.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Decide what you are outsourcing&lt;/strong&gt;
A SERP API moves search-result retrieval and parsing behind a service boundary. Thordata is one option in that category; its overview describes returning structured search data for queries with contextual parameters such as country, language, or device.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;For this project, I would evaluate it with a representative set of queries rather than a single successful demo.&lt;/p&gt;

&lt;p&gt;The evaluation would ask:&lt;/p&gt;

&lt;p&gt;Does it support the search contexts the application needs?&lt;/p&gt;

&lt;p&gt;Are the required fields consistently usable?&lt;/p&gt;

&lt;p&gt;Is pagination behavior clear enough for our collection strategy?&lt;/p&gt;

&lt;p&gt;Can we distinguish valid empty results from failures?&lt;/p&gt;

&lt;p&gt;What does a complete collection cost after retries and pagination?&lt;/p&gt;

&lt;p&gt;I would also keep the adapter small enough that testing another provider does not require changing the dashboard, database schema, and business logic at the same time.&lt;/p&gt;

&lt;p&gt;That is not an argument against using a managed API. It is a way to make the integration easier to reason about.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The useful boundary&lt;/strong&gt;&lt;br&gt;
For a prototype, the first JSON response may be the finish line.&lt;/p&gt;

&lt;p&gt;For a maintained application, I would treat it as the input boundary.&lt;/p&gt;

&lt;p&gt;Let the search API handle retrieval. Keep search context, validation, retry budgets, storage, and downstream meaning explicit in your own code.&lt;/p&gt;

&lt;p&gt;That is the difference between a script that fetches results and a pipeline whose results you can explain.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
      <category>webdev</category>
      <category>dataengineering</category>
    </item>
    <item>
      <title>How I Set Up Rotating Proxies for Python Web Scraping with Thordata</title>
      <dc:creator>Mary</dc:creator>
      <pubDate>Thu, 12 Nov 2026 08:44:13 +0000</pubDate>
      <link>https://dev.to/mary112233/how-i-set-up-rotating-proxies-for-python-web-scraping-with-thordata-4h4j</link>
      <guid>https://dev.to/mary112233/how-i-set-up-rotating-proxies-for-python-web-scraping-with-thordata-4h4j</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwxchpt7ykhkcyl9ku3hh.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwxchpt7ykhkcyl9ku3hh.jpg" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;br&gt;
When you build a web scraper, sooner or later you hit the same wall: your requests work locally, but fail at scale because of rate limits, IP blocks, or geo-restrictions.&lt;/p&gt;

&lt;p&gt;A proxy layer helps you separate the scraping logic from the network access layer. In this post, I’ll show a simple Python setup for rotating residential proxies, plus a few practical tips for keeping the scraper stable.&lt;/p&gt;

&lt;p&gt;Thordata is one option for this kind of workflow: it offers residential, mobile, static ISP, and datacenter proxies, with residential IP coverage across 190+ countries.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why use a proxy for scraping?&lt;/strong&gt;&lt;br&gt;
Proxies are useful when you need to:&lt;/p&gt;

&lt;p&gt;Collect publicly available data at scale&lt;/p&gt;

&lt;p&gt;Test how a page behaves from different locations&lt;/p&gt;

&lt;p&gt;Avoid having all requests originate from a single IP&lt;/p&gt;

&lt;p&gt;Handle geo-specific content, pricing, or search results&lt;/p&gt;

&lt;p&gt;Build more resilient automation workflows&lt;/p&gt;

&lt;p&gt;For most scraping projects, I start with rotating residential proxies because they are suitable for targets that apply stricter anti-bot controls. Thordata states that its residential pool includes over 100 million real-user IPs and supports rotation as well as sticky sessions.&lt;br&gt;
&lt;strong&gt;Python example: rotating proxy requests&lt;/strong&gt;&lt;br&gt;
Here’s a minimal example using requests. Replace the credentials and endpoint with the values from your own proxy dashboard.&lt;br&gt;
python&lt;br&gt;
import os&lt;br&gt;
import requests&lt;br&gt;
import time&lt;br&gt;
import random&lt;/p&gt;

&lt;p&gt;PROXY_HOST = os.getenv("PROXY_HOST")&lt;br&gt;
PROXY_PORT = os.getenv("PROXY_PORT")&lt;br&gt;
PROXY_USER = os.getenv("PROXY_USER")&lt;br&gt;
PROXY_PASS = os.getenv("PROXY_PASS")&lt;/p&gt;

&lt;p&gt;PROXY_URL = (&lt;br&gt;
    f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"&lt;br&gt;
)&lt;/p&gt;

&lt;p&gt;proxies = {&lt;br&gt;
    "http": PROXY_URL,&lt;br&gt;
    "https": PROXY_URL,&lt;br&gt;
}&lt;/p&gt;

&lt;p&gt;headers = {&lt;br&gt;
    "User-Agent": (&lt;br&gt;
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "&lt;br&gt;
        "AppleWebKit/537.36 (KHTML, like Gecko) "&lt;br&gt;
        "Chrome/124.0.0.0 Safari/537.36"&lt;br&gt;
    ),&lt;br&gt;
    "Accept-Language": "en-US,en;q=0.9",&lt;br&gt;
}&lt;/p&gt;

&lt;p&gt;def fetch(url: str, retries: int = 3):&lt;br&gt;
    for attempt in range(1, retries + 1):&lt;br&gt;
        try:&lt;br&gt;
            response = requests.get(&lt;br&gt;
                url,&lt;br&gt;
                proxies=proxies,&lt;br&gt;
                headers=headers,&lt;br&gt;
                timeout=30,&lt;br&gt;
            )&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;        if response.status_code == 200:
            return response.text

        print(f"Attempt {attempt}: HTTP {response.status_code}")

    except requests.RequestException as error:
        print(f"Attempt {attempt}: {error}")

    time.sleep(random.uniform(1, 3))

raise RuntimeError(f"Failed to fetch {url}")
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;if &lt;strong&gt;name&lt;/strong&gt; == "&lt;strong&gt;main&lt;/strong&gt;":&lt;br&gt;
    html = fetch("&lt;a href="https://example.com%22" rel="noopener noreferrer"&gt;https://example.com"&lt;/a&gt;)&lt;br&gt;
    print(html[:500])&lt;br&gt;
&lt;strong&gt;A few important details:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Store credentials in environment variables, not in your source code.&lt;br&gt;
Set a realistic User-Agent and request timeout.&lt;br&gt;
Add retries with randomized delays; scraping systems should degrade gracefully.&lt;br&gt;
Only scrape data you are legally permitted to collect, and respect the target site’s terms, robots directives, and rate limits.&lt;br&gt;
Rotating vs. sticky sessions&lt;br&gt;
Rotating sessions give you a different IP for requests, which is useful for broad collection tasks. Sticky sessions keep the same IP for a period of time, which can help when a flow requires continuity.&lt;/p&gt;

&lt;p&gt;Thordata lists both rotating and sticky session options, and its residential product page describes configurable sessions. In practice, I use rotation for independent page requests and sticky sessions when a sequence of requests needs to look like it comes from the same client.&lt;br&gt;
&lt;strong&gt;A simple scraping checklist&lt;/strong&gt;&lt;br&gt;
Before running a scraper in production, I check:&lt;/p&gt;

&lt;p&gt;Does the site allow this kind of access?&lt;/p&gt;

&lt;p&gt;Am I collecting only public data?&lt;/p&gt;

&lt;p&gt;Can I reduce request frequency without breaking the task?&lt;/p&gt;

&lt;p&gt;Do I handle HTTP errors, timeouts, and malformed responses?&lt;/p&gt;

&lt;p&gt;Are credentials stored securely?&lt;/p&gt;

&lt;p&gt;Can I cache responses to avoid repeated requests?&lt;/p&gt;

&lt;p&gt;Do I log failures without exposing proxy credentials?&lt;/p&gt;

&lt;p&gt;A proxy improves network resilience, but it does not replace good scraping engineering. Rate limiting, caching, retries, and respectful crawling practices still matter.&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
