<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: gresiuseo</title>
    <description>The latest articles on DEV Community by gresiuseo (@gresiuseo).</description>
    <link>https://dev.to/gresiuseo</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1135341%2F2aaaffc7-ca96-4f93-8b30-6a9e1e94829b.png</url>
      <title>DEV Community: gresiuseo</title>
      <link>https://dev.to/gresiuseo</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/gresiuseo"/>
    <language>en</language>
    <item>
      <title>Test a New Balance Dashboard Against the Old One Before Switching</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:05:13 +0000</pubDate>
      <link>https://dev.to/gresiuseo/test-a-new-balance-dashboard-against-the-old-one-before-switching-4ccp</link>
      <guid>https://dev.to/gresiuseo/test-a-new-balance-dashboard-against-the-old-one-before-switching-4ccp</guid>
      <description>&lt;p&gt;Changing a dashboard’s rules can move an alert even when the underlying account hasn’t changed. The problem is especially awkward when the old dashboard packed “no sample,” “zero balance,” and “we haven’t refreshed recently” into two or three colours. I’d run the old and new interpretations on the same observations before replacing the view anyone relies on.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This tutorial was generated with AI. The code and results were checked locally with synthetic inputs.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A read-only response such as the &lt;a href="https://2captcha.com/api-docs/get-balance" rel="noopener noreferrer"&gt;2Captcha account balance&lt;/a&gt; is one input a dashboard may display. The comparison here is entirely local. It uses the same stored records twice and does not add a second API request.&lt;/p&gt;

&lt;h2&gt;
  
  
  Compare decisions on identical inputs
&lt;/h2&gt;

&lt;p&gt;I’d define a tiny comparison interface. A record includes a balance in integer microdollars and its age in seconds. The classifiers each return a category. The legacy classifier defaults missing values to zero and ignores age. The proposed classifier has explicit unknown and stale states, with a synthetic freshness window of 300 seconds. That makes the comparison easy to explain in an incident review.&lt;/p&gt;

&lt;p&gt;If I were comparing two running systems, I’d capture the observation once and feed the same record to both. Comparing two live reads can make the dashboard logic look different when the balance changed between the requests. A stored corpus also makes the investigation repeatable.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;

&lt;span class="no"&gt;LOW&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5_000_000&lt;/span&gt;
&lt;span class="no"&gt;MAX_AGE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;legacy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;balance&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"low"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;balance&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="no"&gt;LOW&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;candidate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;balance&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;balance&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"unknown"&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="no"&gt;MAX_AGE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"stale"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"low"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;balance&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="no"&gt;LOW&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;

&lt;span class="n"&gt;records&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9_000_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;   &lt;span class="c1"&gt;# Healthy reading&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;          &lt;span class="c1"&gt;# Genuine zero&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;     &lt;span class="c1"&gt;# Missing balance&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6_000_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;301&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# Stale normal balance&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4_000_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;600&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# Stale low balance&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;9_000_000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;   &lt;span class="c1"&gt;# Clock or timestamp problem&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;pairs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;legacy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;candidate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;records&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;pairs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"low"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"low"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;pairs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"low"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"unknown"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;pairs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"stale"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nb"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;before&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;after&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;pairs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
           &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;before&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;after&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;pair&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pairs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pair&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fixture reports four changed classifications in six records. That number is chosen by the makeup of the fixture. It says nothing about the frequency of stale data in an actual deployment. The value of this example is that I can name each changed record and decide whether its new category fits the desired product behavior.&lt;/p&gt;

&lt;h2&gt;
  
  
  Look beyond the disagreement rate
&lt;/h2&gt;

&lt;p&gt;An overall disagreement rate is the first number I’d calculate, and it’s rarely enough to approve a rollout. A change from stale low to stale may be expected. A change from genuine low to healthy would need a deeper investigation. I’d report the transition pairs separately and inspect examples where either implementation would page a person.&lt;/p&gt;

&lt;p&gt;I also need a way to separate intentional policy changes from formatting mistakes. If I deliberately introduce a freshness limit, I expect more stale observations. If everything becomes unknown, I’d suspect an input mapping or timestamp conversion error. A representative sample should include accounts that haven’t returned a valid balance, timestamps near the cutoff, and short gaps with a confirmed balance of zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decide what the shadow run is allowed to do
&lt;/h2&gt;

&lt;p&gt;The candidate classifier can emit metrics or test logs. I’d keep it out of the route used for notices until someone reviews the differences. That avoids two conflicting notifications from the same data. If logging example mismatches is useful, I’d record an internal observation reference and the two categories. The raw account credential and response body are unnecessary for this decision.&lt;/p&gt;

&lt;p&gt;Once the expected transitions are documented, I’d roll the candidate out to a small audience and keep the old interpretation available for comparison. The same examples that supported the change become a regression suite. If an unexpected category appears, I can reproduce the reasoning with one observation instead of trying to reconstruct it from a screenshot.&lt;/p&gt;

</description>
      <category>python</category>
    </item>
    <item>
      <title>A Smaller CI Matrix Can Still Spend the Same Amount</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:05:11 +0000</pubDate>
      <link>https://dev.to/gresiuseo/a-smaller-ci-matrix-can-still-spend-the-same-amount-9ne</link>
      <guid>https://dev.to/gresiuseo/a-smaller-ci-matrix-can-still-spend-the-same-amount-9ne</guid>
      <description>&lt;p&gt;Two different questions can collapse into one setting during CI design. How many jobs can run at once? How much work can a run create in total? Limiting the number of parallel jobs answers the first. If the same jobs still run, there’s no reason for the total to change.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This tutorial was generated with AI. The code and results were checked locally with synthetic inputs.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Consider account-monitor tests for an integration with a read-only endpoint such as the &lt;a href="https://2captcha.com/api-docs/get-balance" rel="noopener noreferrer"&gt;2Captcha balance API&lt;/a&gt;. The unit tests can use synthetic fixtures on every pull request. Any credential-bearing smoke check belongs in a separately controlled job. Here I’m looking at the arithmetic of jobs and attempts. The examples make no network requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Count the matrix cells first
&lt;/h2&gt;

&lt;p&gt;A matrix with three operating systems and two Python versions creates six cells before exclusions. If each cell performs one external check and permits one retry, the upper bound is twelve external checks for a single workflow run. Running just two cells concurrently can make the traffic less concentrated. It doesn’t change the upper bound of twelve. Repeated workflow runs add another factor that the single-run calculation doesn’t capture.&lt;/p&gt;

&lt;p&gt;I’d record which combinations actually run and which steps are permitted to call the dependency. In a stage where credentials are never present and the loader has been replaced by a fixture, the correct external budget is zero. That remains true even if the test matrix has a dozen cells.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;itertools&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os_names&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;skipped&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;retries&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;cells&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;product&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os_names&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;skipped&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;per_cell&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;calls&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;retries&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cells&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cells&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;per_cell&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;systems&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"linux"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"windows"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"macos"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;versions&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"3.12"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"3.13"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;systems&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;systems&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"macos"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"3.12"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;budget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;systems&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;versions&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Six cells with one retry can make twelve calls; offline cells make none."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This local check makes the assumptions visible. It uses an upper bound that assumes every initial call is followed by its allowed retries. Real counts may be lower. I’d compare the bound with observed totals after a run and investigate any number above it, including retries inside an HTTP client that the workflow didn’t count.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep pull-request checks offline
&lt;/h2&gt;

&lt;p&gt;This GitHub Actions fragment runs the published budget example on two Python versions without credentials. Save the Python example as &lt;code&gt;matrix_budget.py&lt;/code&gt; at the repository root. In an application repository, I’d add the actual fixture-based tests alongside it. The matrix setting controls concurrency within this job strategy. &lt;a href="https://docs.github.com/en/actions/how-tos/write-workflows/choose-what-workflows-do/run-job-variations" rel="noopener noreferrer"&gt;GitHub Actions matrix documentation&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Offline account-monitor checks&lt;/span&gt;
&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;pull_request&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="na"&gt;permissions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;contents&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;read&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;fixtures&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;ubuntu-latest&lt;/span&gt;
    &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;max-parallel&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
      &lt;span class="na"&gt;matrix&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;python&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'3.12'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'3.13'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="na"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/checkout@v7&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;persist-credentials&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;actions/setup-python@v7&lt;/span&gt;
        &lt;span class="na"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
          &lt;span class="na"&gt;python-version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ matrix.python }}&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;python matrix_budget.py&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Python example passed locally. The workflow is a configuration example and was not run on GitHub. For an authenticated check, I’d choose one runner and record a request cap with the job owner. If several branches or workflows share an account, the budget should identify which runs can overlap and who owns the shared limit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Track consumption at the boundary
&lt;/h2&gt;

&lt;p&gt;I’d count attempted requests where they leave the client, including the ones that time out, and keep “requests attempted” separate from any confirmed billable work. A timeout doesn’t by itself establish a charge. If the monitor retries, log the attempt count and the final outcome against the same internal check ID. That gives the person reviewing the run a way to distinguish a growing workload from a lookup that got slower.&lt;/p&gt;

&lt;p&gt;Give the next workflow review two numbers. Calculate the largest number of concurrent jobs under that configuration, then calculate the maximum number of external attempts across the whole run. If a proposed change only improves the first number, the budget for total work stays as it was.&lt;/p&gt;

</description>
      <category>python</category>
    </item>
    <item>
      <title>Stop Parallel Dashboard Readers From Refreshing the Same Account</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:05:01 +0000</pubDate>
      <link>https://dev.to/gresiuseo/stop-parallel-dashboard-readers-from-refreshing-the-same-account-48je</link>
      <guid>https://dev.to/gresiuseo/stop-parallel-dashboard-readers-from-refreshing-the-same-account-48je</guid>
      <description>&lt;p&gt;An account dashboard may look inexpensive when it has one viewer. The problem shows up when everyone opens it during an incident and the cached value has just expired. Each request sees the expiry and starts its own external lookup. A slow dependency gets more work at exactly the wrong time.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This tutorial was generated with AI. The code and results were checked locally with synthetic inputs.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I’d use a similar rule for any low-frequency account lookup, including a read-only &lt;a href="https://2captcha.com/api-docs/get-balance" rel="noopener noreferrer"&gt;balance observation&lt;/a&gt;. Let one reader refresh the cached value and give the others an existing observation if it is still within an acceptable age. After that age, return an unavailable state. A balance that has never been read is unavailable from the start.&lt;/p&gt;

&lt;h2&gt;
  
  
  Separate freshness from the maximum stale age
&lt;/h2&gt;

&lt;p&gt;In the example, an observation is fresh for five seconds and may still be shown for 30 seconds. These short thresholds keep the fixture easy to read. In a deployed system I’d choose intervals appropriate for the monitor. I’d also show the age and status in the dashboard so an operator can tell that the value they see is from an earlier observation.&lt;/p&gt;

&lt;p&gt;I use two locks for distinct purposes. A small lock guards the cached snapshot. A separate refresh lock identifies the one reader allowed to invoke the loader. The loader runs outside the snapshot lock, so a follower can obtain the previous value without waiting for the refresh. The example uses a monotonic clock for elapsed age within the process. &lt;a href="https://docs.python.org/3/library/threading.html" rel="noopener noreferrer"&gt;Python threading primitives&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;concurrent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;futures&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ThreadPoolExecutor&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;threading&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Lock&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Cache&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;clock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;clock&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;snapshot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data_lock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;refresh_lock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Lock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;current&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data_lock&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;snapshot&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;snapshot&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"unavailable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;age&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;clock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"unavailable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"fresh"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;age&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="s2"&gt;"stale"&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;snapshot&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;load&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;current&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"fresh"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;refresh_lock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;acquire&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;blocking&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;current&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;current&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"fresh"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;amount&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;TimeoutError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;current&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data_lock&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;snapshot&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;clock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"fresh"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;refresh_lock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;release&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;cache&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Cache&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"fresh"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;
&lt;span class="n"&gt;started&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;release&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;controlled_refresh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;started&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;release&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Test did not release refresh"&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;1100&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;unexpected_refresh&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AssertionError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Second reader called the loader"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nc"&gt;ThreadPoolExecutor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_workers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;owner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;submit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;controlled_refresh&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;started&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"Refresh did not start"&lt;/span&gt;
        &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;unexpected_refresh&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"stale"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;release&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="kt"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;result&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"fresh"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;TimeoutError&lt;/span&gt;

&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;15&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"stale"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;37&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"unavailable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"One refresh owner, stale fallback, and expiry verified."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The two events in the test control when the loader starts and finishes. That lets the test place another reader inside the refresh window without a sleep. While the first reader is held, the second gets the stale value and does not call the loader. Once the held refresh finishes, a later timeout can use the new value until it passes the maximum age.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep the scope of the guarantee visible
&lt;/h2&gt;

&lt;p&gt;The locks coordinate threads inside one Python process. They don’t coordinate multiple service replicas. I’d use a shared cache with an atomic lease, or a dedicated monitoring job that writes observations for the dashboard to read, if many processes need the same value. In the dedicated-job design, dashboard traffic can’t multiply external requests at all.&lt;/p&gt;

&lt;p&gt;I’d also add a pause after a failed refresh. This example limits the number of simultaneous calls, but sequential visitors could still trigger repeated timeouts while a dependency is down. In production, a short recovery interval tied to the monitor’s cadence can protect the dependency and make alerts easier to interpret. Test that behavior with a controlled clock and a loader that fails repeatedly before putting a live account behind it.&lt;/p&gt;

</description>
      <category>python</category>
    </item>
    <item>
      <title>Make a Low-Balance Alert Survive a Process Crash</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:04:59 +0000</pubDate>
      <link>https://dev.to/gresiuseo/make-a-low-balance-alert-survive-a-process-crash-26ja</link>
      <guid>https://dev.to/gresiuseo/make-a-low-balance-alert-survive-a-process-crash-26ja</guid>
      <description>&lt;p&gt;A balance monitor can detect a low account correctly and still lose the alert. Suppose it records the account as low, then crashes before sending the message. On restart it sees the account was already marked low and decides it has nothing new to send. A small persistence decision determines whether the operator ever hears about the incident.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This tutorial was generated with AI. The code and results were checked locally with synthetic inputs.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The same issue applies to read-only monitoring of an account service such as the &lt;a href="https://2captcha.com/api-docs/get-balance" rel="noopener noreferrer"&gt;2Captcha balance API&lt;/a&gt;. The example starts after the monitor has already decided to open an incident. It makes no API request and sends no notification. The goal is to make the handoff between monitor and delivery worker durable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Commit the decision and the pending work together
&lt;/h2&gt;

&lt;p&gt;I’d give each transition a stable identifier when the monitor accepts it. The delivery worker gets a separate pending record with that identifier. If SQLite commits the transition, it commits the pending record in the same transaction. When the transaction rolls back, both changes are undone. The worker can pick up the pending record after a restart without rereading an external account. &lt;a href="https://www.sqlite.org/lang_transaction.html" rel="noopener noreferrer"&gt;SQLite transaction semantics&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The example uses a single account row and one alert type. A production ledger would store a row per account and include the observation identity. The worker would read only events that are actually ready to send.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sqlite3&lt;/span&gt;

&lt;span class="n"&gt;db&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sqlite3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;":memory:"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;executescript&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"""
    CREATE TABLE account_status (id INTEGER PRIMARY KEY, state TEXT NOT NULL);
    INSERT INTO account_status VALUES (1, 'normal');
    CREATE TABLE outbox (
        id TEXT PRIMARY KEY CHECK(length(id) &amp;gt; 0),
        message TEXT NOT NULL,
        delivered INTEGER NOT NULL DEFAULT 0
    );
"""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;open_alert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"UPDATE account_status SET state = 'low' WHERE id = 1"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"""
            INSERT INTO outbox(id, message) VALUES (?, 'Balance below threshold')
            ON CONFLICT(id) DO NOTHING
        """&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;open_alert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;sqlite3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;IntegrityError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;pass&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AssertionError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Invalid event unexpectedly committed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SELECT state FROM account_status"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"normal"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SELECT count(*) FROM outbox"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;open_alert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"qa-low-41"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;open_alert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"qa-low-41"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;pending&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"SELECT id FROM outbox WHERE delivered = 0"&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"qa-low-41"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="c1"&gt;# Simulate a sender crash before acknowledgement.&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"qa-low-41"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"UPDATE outbox SET delivered = 1 WHERE id = ?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"qa-low-41"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Rollback, replay, pending recovery, and acknowledgement verified."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fixture deliberately fails an insert after the account update. The account remains normal and the outbox remains empty. Then the same valid transition is processed twice. Both copies have the same event identifier, so the outbox contains one event. The pending record remains eligible until the worker records successful delivery.&lt;/p&gt;

&lt;h2&gt;
  
  
  Decide what happens after a successful send and a failed acknowledgement
&lt;/h2&gt;

&lt;p&gt;This design doesn’t provide exactly-once delivery to an external destination. If sending succeeds and the worker crashes before marking the row delivered, the next worker may send the same event. If the destination supports idempotency keys or incident deduplication, I’d send the stored event identifier. If it has neither, I’d document when duplicates can occur.&lt;/p&gt;

&lt;p&gt;I’d also put retry counts and the oldest pending event age on the worker’s dashboard. Retrying forever can hide a broken connection to the destination. An event that has been waiting for an hour should tell an operator something specific about the delivery path. It should not create a second low-balance incident.&lt;/p&gt;

&lt;p&gt;For a practical failure drill, stop the worker after the transaction commits and before it acknowledges delivery. Start it again and trace the event identifier to the destination. This drill checks a different risk than replaying the threshold logic alone. You learn whether an incident survives a real interruption between the decision and the notification.&lt;/p&gt;

</description>
      <category>python</category>
    </item>
    <item>
      <title>Keep Money Exact When a JSON API Returns a Balance</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Thu, 19 Nov 2026 13:04:41 +0000</pubDate>
      <link>https://dev.to/gresiuseo/keep-money-exact-when-a-json-api-returns-a-balance-4bgg</link>
      <guid>https://dev.to/gresiuseo/keep-money-exact-when-a-json-api-returns-a-balance-4bgg</guid>
      <description>&lt;p&gt;A monitor that observes the same account twice can report a tiny change even when the balance stayed the same. That makes a discrepancy awkward to diagnose. The bug may be in parsing or conversion, well before a ledger or chart sees the number. I’d start by deciding exactly what a stored balance represents.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This tutorial was generated with AI. The code and results were checked locally with synthetic inputs.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Preserve the distinction between zero and missing
&lt;/h2&gt;

&lt;p&gt;The example assumes a JSON account endpoint whose amount arrives as a number. The read-only &lt;a href="https://2captcha.com/api-docs/get-balance" rel="noopener noreferrer"&gt;2Captcha balance API&lt;/a&gt; provides one reference for this response shape, with a balance in the successful response. I’ll use synthetic values throughout and treat the parsing rules as application choices.&lt;/p&gt;

&lt;p&gt;There are three cases I care about. A successful zero observation means the account reported a balance of zero. A successful-looking response with no balance is incomplete. An API error should be reported through the error path even when it includes a number somewhere in the response. If every path eventually becomes a numeric zero, the monitor sends an operator to investigate an apparent depleted balance instead of a broken read.&lt;/p&gt;

&lt;h2&gt;
  
  
  Parse the JSON token before turning it into a storage value
&lt;/h2&gt;

&lt;p&gt;I use Python’s JSON decoder with a decimal parser for non-integer numbers. Calling Decimal on a float after parsing preserves the float’s representation, including any binary approximation that already entered the process. In this example, the storage unit is one millionth of a dollar. That unit is useful when calculating differences smaller than a cent. It’s a contract for this ledger, independent of what a provider may support. &lt;a href="https://docs.python.org/3/library/json.html" rel="noopener noreferrer"&gt;Python’s JSON decoding options&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;decimal&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;

&lt;span class="no"&gt;QUANTUM&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"0.000001"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="no"&gt;MAX_AMOUNT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"1000000"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;reject_constant&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Non-finite JSON number"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;balance_micros&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;parse_float&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;parse_constant&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;reject_constant&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nb"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"errorId"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Invalid response shape"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"errorId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"API rejected the observation"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"balance"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Balance must be a JSON number"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;amount&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Decimal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_finite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;copy_abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="no"&gt;MAX_AMOUNT&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Amount outside ledger range"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;normalized&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;quantize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;QUANTUM&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;normalized&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Amount exceeds ledger precision"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;normalized&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1_000_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;balance_micros&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'{"errorId":0,"balance":0}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;balance_micros&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'{"errorId":0,"balance":12.123456}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;12_123_456&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;balance_micros&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'{"errorId":0,"balance":0.000001}'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s1"&gt;'{"errorId":0}'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;'{"errorId":0,"balance":false}'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;'{"errorId":0,"balance":0.0000001}'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;'{"errorId":0,"balance":NaN}'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s1"&gt;'{"errorId":1,"balance":10}'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;balance_micros&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;pass&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;AssertionError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"Accepted {body}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Three valid amounts preserved. Five invalid responses rejected."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The successful cases include a genuine zero, an amount with six fractional digits, and the smallest positive unit in this ledger. A value below that unit is rejected rather than rounded into an apparent zero. The maximum amount in the example is a deliberately chosen ledger bound. If your account can have a larger balance, choose a different bound and verify the storage type can represent it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keep presentation rounding at the edge
&lt;/h2&gt;

&lt;p&gt;A dashboard for humans can still display two decimal places. I’d render an amount such as 1.234567 as 1.23 on a daily summary, while keeping 1,234,567 millionths in the underlying record. When reconciling several operations I’d subtract the two stored balances first and format the result afterward. Rounding both endpoints before subtraction can hide a real sub-cent movement.&lt;/p&gt;

&lt;p&gt;I’d test the export as well as the parser. A CSV user may open the file in a spreadsheet and format an integer as currency without scaling it. I’d name the column with its unit and keep the human-readable amount in a distinct display column if both are needed. The next step is to send the smallest supported amount, a genuine zero, and a missing response through the complete export pipeline and compare what reaches the dashboard.&lt;/p&gt;

</description>
      <category>python</category>
      <category>api</category>
    </item>
    <item>
      <title>xCaptcha Review: My Experience with Modern CAPTCHA Protection After Years of Working with Websites and Automation</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Mon, 16 Nov 2026 15:54:52 +0000</pubDate>
      <link>https://dev.to/gresiuseo/xcaptcha-review-my-experience-with-modern-captcha-protection-after-years-of-working-with-websites-308f</link>
      <guid>https://dev.to/gresiuseo/xcaptcha-review-my-experience-with-modern-captcha-protection-after-years-of-working-with-websites-308f</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F03wtvpb0ujcqf5htgjf1.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F03wtvpb0ujcqf5htgjf1.jpg" alt=" " width="800" height="447"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you have been working with websites, web scraping, browser automation, or online services for several years, you probably understand how frustrating automated traffic can become.&lt;/p&gt;

&lt;p&gt;When I first started developing and managing web projects, CAPTCHA was something I considered a relatively simple security feature. Add a verification widget, make users prove they are human, and the problem should be solved.&lt;/p&gt;

&lt;p&gt;At least, that was the theory.&lt;/p&gt;

&lt;p&gt;In reality, things are considerably more complicated.&lt;/p&gt;

&lt;p&gt;Traditional CAPTCHA systems can create unnecessary friction for legitimate visitors while still leaving opportunities for increasingly sophisticated automated attacks. Some bots can interact with websites in ways that look surprisingly similar to normal human behavior.&lt;/p&gt;

&lt;p&gt;And this creates an interesting problem for website owners.&lt;/p&gt;

&lt;p&gt;You want to stop automated abuse, but you do not want to make every genuine customer solve puzzles, select traffic lights, or repeatedly click pictures just to access your website.&lt;/p&gt;

&lt;p&gt;Over time, I became increasingly interested in modern bot protection technologies that approach this problem differently.&lt;/p&gt;

&lt;p&gt;One of the platforms I came across during my research was xCaptcha.&lt;/p&gt;

&lt;p&gt;After looking into its architecture, features, pricing structure, and available documentation, I found that it approaches website protection from a different perspective than traditional CAPTCHA providers.&lt;/p&gt;

&lt;p&gt;Instead of focusing exclusively on presenting challenges to visitors, &lt;a href="https://xcaptcha.com/" rel="noopener noreferrer"&gt;xCaptcha&lt;/a&gt; emphasizes intelligent bot detection, adaptive verification, and risk analysis.&lt;/p&gt;

&lt;p&gt;In this article, I want to explain why this approach is interesting, what the platform offers, where it makes sense to use it, and what I would pay attention to before introducing it into a production environment.&lt;/p&gt;

&lt;p&gt;This is not another generic list of marketing features. It is a practical look at xCaptcha from the perspective of someone familiar with website development, automation, and online security.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. First Impressions: xCaptcha Is Not Just Another CAPTCHA Widget&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The first thing worth understanding about xCaptcha is that it is not simply another service that displays distorted letters or asks visitors to identify objects in photographs.&lt;/p&gt;

&lt;p&gt;Its main purpose is to protect websites, applications, and APIs from malicious automated traffic.&lt;/p&gt;

&lt;p&gt;That distinction is important.&lt;/p&gt;

&lt;p&gt;Traditional CAPTCHA products generally concentrate on verifying whether a particular interaction comes from a human.&lt;/p&gt;

&lt;p&gt;Modern bot protection platforms try to evaluate a much broader picture.&lt;/p&gt;

&lt;p&gt;For example: &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Does the incoming request look legitimate?&lt;/li&gt;
&lt;li&gt;Does the visitor's behavior resemble normal browsing?&lt;/li&gt;
&lt;li&gt;Is the traffic associated with suspicious activity?&lt;/li&gt;
&lt;li&gt;Is the request part of an automated attack?&lt;/li&gt;
&lt;li&gt;Does the interaction require additional verification?&lt;/li&gt;
&lt;li&gt;Should the request be allowed or blocked?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;xCaptcha describes its technology as an adaptive, risk-based protection system that uses machine learning to analyze interactions.&lt;/p&gt;

&lt;p&gt;The idea is to reduce the number of unnecessary challenges presented to legitimate visitors while making automated abuse more difficult.&lt;/p&gt;

&lt;p&gt;This is exactly the direction I believe modern website security should be moving toward.&lt;/p&gt;

&lt;p&gt;A security system should not automatically assume that every visitor is suspicious.&lt;/p&gt;

&lt;p&gt;It should evaluate risk and apply an appropriate response.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Task Rotation: An Interesting Approach to Fighting Automated Attacks&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One of the features that caught my attention was xCaptcha's task rotation mechanism.&lt;/p&gt;

&lt;p&gt;Traditional CAPTCHA systems often use recognizable challenge formats.&lt;/p&gt;

&lt;p&gt;This creates a problem because once an automated system understands a particular challenge, it may become easier to build a solution specifically designed for that format.&lt;/p&gt;

&lt;p&gt;The challenge becomes predictable.&lt;/p&gt;

&lt;p&gt;xCaptcha approaches this differently by regularly changing verification scenarios.&lt;/p&gt;

&lt;p&gt;The company describes this mechanism as task rotation.&lt;/p&gt;

&lt;p&gt;In practical terms, the verification scenario does not remain permanently identical. Different tasks can be presented over time, making it more difficult for automated systems to adapt to a single predictable pattern.&lt;/p&gt;

&lt;p&gt;Why does this matter?&lt;/p&gt;

&lt;p&gt;Because automation is extremely good at repeating known instructions.&lt;/p&gt;

&lt;p&gt;If a bot can recognize the same interface and perform the same actions thousands of times, the protection mechanism gradually becomes less effective.&lt;/p&gt;

&lt;p&gt;Changing the verification scenario increases the complexity of maintaining such automation.&lt;/p&gt;

&lt;p&gt;I particularly like the general idea behind this approach.&lt;/p&gt;

&lt;p&gt;Security should not depend entirely on hiding one static challenge or making one puzzle more complicated.&lt;/p&gt;

&lt;p&gt;It should continuously adapt to changing traffic patterns.&lt;/p&gt;

&lt;p&gt;Of course, task rotation alone cannot stop every sophisticated bot. It is most useful as one component of a broader detection strategy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The Biggest Advantage: Reducing Friction for Real Users&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;From my perspective, this is one of the most important aspects of modern CAPTCHA protection.&lt;/p&gt;

&lt;p&gt;A website owner can have the strongest security system in the world, but if legitimate visitors constantly experience annoying verification screens, the user experience suffers.&lt;/p&gt;

&lt;p&gt;Think about an online store.&lt;/p&gt;

&lt;p&gt;A potential customer finds a product, adds it to the shopping cart, and proceeds to checkout.&lt;/p&gt;

&lt;p&gt;Suddenly, the website asks them to identify bicycles, traffic lights, buses, or crosswalks.&lt;/p&gt;

&lt;p&gt;Sometimes the first attempt fails.&lt;/p&gt;

&lt;p&gt;Then another challenge appears.&lt;/p&gt;

&lt;p&gt;Eventually, the customer may simply leave.&lt;/p&gt;

&lt;p&gt;This is not a hypothetical concern. Additional friction can affect conversion rates, especially during important interactions such as registration, login, and checkout.&lt;/p&gt;

&lt;p&gt;xCaptcha promotes a frictionless approach in which legitimate visitors can continue using a website while suspicious traffic receives additional scrutiny.&lt;/p&gt;

&lt;p&gt;Its security materials also describe invisible verification options.&lt;/p&gt;

&lt;p&gt;This is particularly interesting for websites where user experience directly affects revenue.&lt;/p&gt;

&lt;p&gt;I would rather have a system that intelligently evaluates traffic than one that treats every visitor as a potential attacker.&lt;/p&gt;

&lt;p&gt;However, no bot protection product can guarantee that every legitimate user will always pass without friction. False positives and unusual browser configurations should always be considered during testing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Adaptive Bot Detection and Machine Learning&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Another major component of xCaptcha is its risk-based detection technology.&lt;/p&gt;

&lt;p&gt;The platform describes using machine learning to analyze customer and bot interactions.&lt;/p&gt;

&lt;p&gt;Instead of relying exclusively on fixed rules, the system aims to identify suspicious patterns and adapt to changing behavior.&lt;/p&gt;

&lt;p&gt;This is important because automated traffic has become increasingly sophisticated.&lt;/p&gt;

&lt;p&gt;Older bot protection methods often rely on relatively simple indicators:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Request frequency.&lt;/li&gt;
&lt;li&gt;IP reputation.&lt;/li&gt;
&lt;li&gt;Browser information.&lt;/li&gt;
&lt;li&gt;User-agent strings.&lt;/li&gt;
&lt;li&gt;Repeated access patterns.&lt;/li&gt;
&lt;li&gt;Known malicious signatures.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These signals are still useful, but they are not sufficient on their own.&lt;/p&gt;

&lt;p&gt;Modern automation can rotate IP addresses, modify browser fingerprints, use real browsers, and introduce random delays between interactions.&lt;/p&gt;

&lt;p&gt;This makes simplistic detection less reliable.&lt;/p&gt;

&lt;p&gt;A more advanced system needs to evaluate multiple signals together.&lt;/p&gt;

&lt;p&gt;xCaptcha describes its approach as continuous risk analysis supported by machine learning.&lt;/p&gt;

&lt;p&gt;The company also advertises extremely fast detection, claiming that its engine can distinguish legitimate users from malicious actors in less than two milliseconds.&lt;/p&gt;

&lt;p&gt;That is an interesting performance claim, although I would want to validate actual end-to-end latency under the specific traffic conditions of a production website.&lt;/p&gt;

&lt;p&gt;There is a difference between the processing time of a detection engine and the total time added to a real visitor's request.&lt;/p&gt;

&lt;p&gt;For developers, both measurements matter.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Protection Against Web Scraping&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Anyone who has managed a website containing valuable content or product information understands the scraping problem.&lt;/p&gt;

&lt;p&gt;Web scraping is not automatically malicious. It is widely used for research, price monitoring, search indexing, data analysis, and many legitimate business activities.&lt;/p&gt;

&lt;p&gt;However, uncontrolled automated extraction can create serious problems.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Competitors may collect proprietary content.&lt;/li&gt;
&lt;li&gt;Automated systems may overload servers.&lt;/li&gt;
&lt;li&gt;Product information may be copied at scale.&lt;/li&gt;
&lt;li&gt;Search and advertising content may be harvested.&lt;/li&gt;
&lt;li&gt;Valuable data may be redistributed without permission.&lt;/li&gt;
&lt;li&gt;xCaptcha specifically lists scraping protection among its use cases.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This makes it relevant to websites that need to distinguish ordinary visitors from suspicious automated traffic.&lt;/p&gt;

&lt;p&gt;I find this particularly important for e-commerce websites, online marketplaces, ticketing platforms, and content-heavy projects.&lt;/p&gt;

&lt;p&gt;Imagine running an online store with thousands of products.&lt;/p&gt;

&lt;p&gt;A competitor could potentially collect product names, prices, descriptions, and availability information through repeated automated requests.&lt;/p&gt;

&lt;p&gt;If the scraping volume becomes significant, it can consume infrastructure resources and affect the experience of genuine customers.&lt;/p&gt;

&lt;p&gt;A bot protection system can help identify suspicious patterns and reduce this type of abuse.&lt;/p&gt;

&lt;p&gt;However, it is important to understand that bot detection is not the same as complete scraping prevention.&lt;/p&gt;

&lt;p&gt;A determined attacker may use distributed infrastructure, browser automation, or human-assisted workflows.&lt;/p&gt;

&lt;p&gt;For this reason, I would consider xCaptcha one layer in a broader security strategy rather than a replacement for every other protection mechanism.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Protecting Login Pages and User Accounts&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Account security is another area where xCaptcha's capabilities become particularly relevant.&lt;/p&gt;

&lt;p&gt;One of the most common automated attacks against websites is credential stuffing.&lt;/p&gt;

&lt;p&gt;This happens when attackers use previously leaked username and password combinations to attempt access to accounts on other platforms.&lt;/p&gt;

&lt;p&gt;The problem is that users frequently reuse passwords.&lt;/p&gt;

&lt;p&gt;A successful attack can therefore expose customer accounts, personal information, and business data.&lt;/p&gt;

&lt;p&gt;Another common threat is brute-force login activity.&lt;/p&gt;

&lt;p&gt;Automated systems repeatedly attempt different credentials until they discover a valid combination or trigger an account protection mechanism.&lt;/p&gt;

&lt;p&gt;xCaptcha lists several relevant security use cases:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Account takeover prevention.&lt;/li&gt;
&lt;li&gt;Credential stuffing protection.&lt;/li&gt;
&lt;li&gt;Brute-force attack mitigation.&lt;/li&gt;
&lt;li&gt;Fake account creation prevention.&lt;/li&gt;
&lt;li&gt;Fraudulent transaction prevention.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I would pay particular attention to login and registration forms when evaluating any bot protection platform.&lt;/p&gt;

&lt;p&gt;These endpoints are often targeted because they provide direct access to valuable functionality.&lt;/p&gt;

&lt;p&gt;A good implementation should protect them without making normal account access unnecessarily complicated.&lt;/p&gt;

&lt;p&gt;It should also work alongside established security practices such as multi-factor authentication, rate limiting, secure password storage, and account recovery controls.&lt;/p&gt;

&lt;p&gt;No CAPTCHA solution should be treated as a replacement for proper authentication security.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. The Dashboard and Reporting Tools&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One thing I always look for in security software is visibility.&lt;/p&gt;

&lt;p&gt;It is not enough for a platform to claim that it blocks malicious traffic.&lt;/p&gt;

&lt;p&gt;I want to understand what is happening.&lt;/p&gt;

&lt;p&gt;How many suspicious requests are being detected?&lt;/p&gt;

&lt;p&gt;Are attacks increasing?&lt;/p&gt;

&lt;p&gt;Which pages are receiving the most automated traffic?&lt;/p&gt;

&lt;p&gt;Are legitimate visitors being affected?&lt;/p&gt;

&lt;p&gt;Can I identify changes after adjusting security settings?&lt;/p&gt;

&lt;p&gt;xCaptcha promotes an integrated reporting dashboard that provides information about user retention and attack volume.&lt;/p&gt;

&lt;p&gt;Its wider bot protection offering also describes real-time threat insights and automated reports.&lt;/p&gt;

&lt;p&gt;This is useful because security decisions should be based on actual observations rather than assumptions.&lt;/p&gt;

&lt;p&gt;For example, imagine noticing an unusual increase in suspicious traffic after publishing a new landing page.&lt;/p&gt;

&lt;p&gt;A reporting dashboard can help determine whether the change is associated with automated activity.&lt;/p&gt;

&lt;p&gt;Similarly, if you introduce additional protection to a checkout process, you should monitor whether legitimate users are experiencing unexpected friction.&lt;/p&gt;

&lt;p&gt;I would particularly value the following measurements in a production environment:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Total incoming requests.&lt;/li&gt;
&lt;li&gt;Suspicious request volume.&lt;/li&gt;
&lt;li&gt;Blocked requests.&lt;/li&gt;
&lt;li&gt;Verification frequency.&lt;/li&gt;
&lt;li&gt;False positive rate.&lt;/li&gt;
&lt;li&gt;Traffic distribution by endpoint.&lt;/li&gt;
&lt;li&gt;Changes in attack volume over time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The more visibility a security platform provides, the easier it becomes to tune its configuration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Integration and Developer Experience&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For me, integration is one of the most important factors when evaluating any third-party security service.&lt;/p&gt;

&lt;p&gt;A platform can have impressive technology, but if implementation requires rewriting half of an existing application, its practical value decreases.&lt;/p&gt;

&lt;p&gt;xCaptcha advertises a relatively straightforward deployment process using a code snippet.&lt;/p&gt;

&lt;p&gt;Its website also provides a dedicated API and quick-start entry point.&lt;/p&gt;

&lt;p&gt;The platform is intended to support protection across:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Websites.&lt;/li&gt;
&lt;li&gt;Mobile applications.&lt;/li&gt;
&lt;li&gt;APIs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is important because modern applications rarely consist of a single frontend page.&lt;/p&gt;

&lt;p&gt;A typical project may include a public website, a backend API, an administration panel, mobile clients, and several third-party integrations.&lt;/p&gt;

&lt;p&gt;Security should be consistent across these components.&lt;/p&gt;

&lt;p&gt;What I Would Check Before Integration&lt;/p&gt;

&lt;p&gt;Before deploying xCaptcha, I would investigate several technical details.&lt;/p&gt;

&lt;p&gt;Frontend compatibility&lt;/p&gt;

&lt;p&gt;Check whether the integration works correctly with the website's existing JavaScript framework and rendering architecture.&lt;/p&gt;

&lt;p&gt;Backend verification&lt;/p&gt;

&lt;p&gt;Client-side verification alone should never be considered sufficient.&lt;/p&gt;

&lt;p&gt;The backend should validate security responses using the documented server-side integration.&lt;/p&gt;

&lt;p&gt;API requirements&lt;/p&gt;

&lt;p&gt;Understand the authentication process, request formats, response structure, and error handling.&lt;/p&gt;

&lt;p&gt;Failure behavior&lt;/p&gt;

&lt;p&gt;Determine what happens if the verification service becomes temporarily unavailable.&lt;/p&gt;

&lt;p&gt;Should the application fail closed, fail open, or apply a different fallback policy?&lt;/p&gt;

&lt;p&gt;This decision depends on the security requirements of the application.&lt;/p&gt;

&lt;p&gt;Performance&lt;/p&gt;

&lt;p&gt;Measure the actual impact on page loading, interaction time, and backend response latency.&lt;/p&gt;

&lt;p&gt;Privacy&lt;/p&gt;

&lt;p&gt;Review which information is collected, how it is processed, and whether the implementation meets applicable privacy requirements.&lt;/p&gt;

&lt;p&gt;These checks are important for any production security integration.&lt;br&gt;
**&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Pricing: Is xCaptcha Economical?**&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Pricing is another reason I find xCaptcha interesting.&lt;/p&gt;

&lt;p&gt;According to its published pricing page, the platform provides several usage tiers.&lt;/p&gt;

&lt;p&gt;The current published structure includes Essentials, Standard, and Enterprise.&lt;/p&gt;

&lt;p&gt;These figures are based on the published pricing table and should be checked again before purchasing because commercial terms can change.&lt;/p&gt;

&lt;p&gt;The first thing I notice is the free allowance.&lt;/p&gt;

&lt;p&gt;Ten thousand checks can be useful for evaluating a service on a smaller website before committing to a paid plan.&lt;/p&gt;

&lt;p&gt;For developers, this makes experimentation easier.&lt;/p&gt;

&lt;p&gt;You can investigate whether the integration fits your application and whether the security features provide practical value.&lt;/p&gt;

&lt;p&gt;The Standard plan is also interesting because the published price is relatively accessible for projects that need more usage.&lt;/p&gt;

&lt;p&gt;However, I would not choose a security provider based on price alone.&lt;/p&gt;

&lt;p&gt;A cheap system that incorrectly blocks legitimate customers can become considerably more expensive than a more advanced solution.&lt;/p&gt;

&lt;p&gt;The real calculation should include security effectiveness, conversion impact, infrastructure costs, and maintenance.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;10. xCaptcha Compared with Traditional CAPTCHA Solutions&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One of the most useful ways to understand xCaptcha is to compare its general approach with conventional CAPTCHA systems.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Characteristic&lt;/li&gt;
&lt;li&gt;Traditional CAPTCHA&lt;/li&gt;
&lt;li&gt;xCaptcha approach&lt;/li&gt;
&lt;li&gt;Main purpose&lt;/li&gt;
&lt;li&gt;Human verification&lt;/li&gt;
&lt;li&gt;Bot and fraud protection&lt;/li&gt;
&lt;li&gt;Detection approach&lt;/li&gt;
&lt;li&gt;Challenge-based&lt;/li&gt;
&lt;li&gt;Risk-based and adaptive&lt;/li&gt;
&lt;li&gt;User interaction&lt;/li&gt;
&lt;li&gt;Often visible&lt;/li&gt;
&lt;li&gt;Designed to reduce friction&lt;/li&gt;
&lt;li&gt;Challenge behavior&lt;/li&gt;
&lt;li&gt;Often predictable formats&lt;/li&gt;
&lt;li&gt;Rotating verification scenarios&lt;/li&gt;
&lt;li&gt;Machine learning&lt;/li&gt;
&lt;li&gt;Depends on provider&lt;/li&gt;
&lt;li&gt;Central part of positioning&lt;/li&gt;
&lt;li&gt;Reporting&lt;/li&gt;
&lt;li&gt;Depends on provider&lt;/li&gt;
&lt;li&gt;Integrated reporting dashboard&lt;/li&gt;
&lt;li&gt;Protection scope&lt;/li&gt;
&lt;li&gt;Usually individual verification points&lt;/li&gt;
&lt;li&gt;Websites, applications, and APIs&lt;/li&gt;
&lt;li&gt;Main focus&lt;/li&gt;
&lt;li&gt;Distinguishing humans from bots&lt;/li&gt;
&lt;li&gt;Detecting and preventing automated abuse&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This comparison is about the general product approaches, not a controlled performance benchmark.&lt;/p&gt;

&lt;p&gt;Traditional CAPTCHA still has its place.&lt;/p&gt;

&lt;p&gt;For smaller projects, a conventional verification widget may be perfectly adequate.&lt;/p&gt;

&lt;p&gt;But for applications that face sophisticated automated traffic, a more comprehensive bot protection platform can offer additional capabilities.&lt;/p&gt;

&lt;p&gt;I see xCaptcha as an option worth evaluating when basic CAPTCHA protection is no longer sufficient.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;What I Like About the xCaptcha Approach&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There are several aspects that make the platform interesting to me.&lt;/p&gt;

&lt;p&gt;Less Friction for Legitimate Visitors&lt;/p&gt;

&lt;p&gt;This is probably my favorite concept.&lt;/p&gt;

&lt;p&gt;Security should not make a website unpleasant to use.&lt;/p&gt;

&lt;p&gt;Reducing unnecessary verification steps is valuable, especially for e-commerce and customer-facing applications.&lt;/p&gt;

&lt;p&gt;Adaptive Detection&lt;/p&gt;

&lt;p&gt;The idea of analyzing traffic patterns rather than depending exclusively on static rules makes sense in an environment where automated systems constantly evolve.&lt;/p&gt;

&lt;p&gt;Task Rotation&lt;/p&gt;

&lt;p&gt;Changing verification scenarios makes the protection strategy less predictable.&lt;/p&gt;

&lt;p&gt;Multiple Protection Scenarios&lt;/p&gt;

&lt;p&gt;The platform addresses scraping, account takeover, fraudulent transactions, fake account creation, and other forms of automated abuse.&lt;/p&gt;

&lt;p&gt;Reporting and Visibility&lt;/p&gt;

&lt;p&gt;Having access to attack statistics and security reports makes it easier to understand what is happening on a website.&lt;/p&gt;

&lt;p&gt;Accessible Entry-Level Pricing&lt;/p&gt;

&lt;p&gt;The published free allowance provides an opportunity to evaluate the platform before moving to a larger plan.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;12. Things I Would Improve or Investigate Further&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I also believe it is important to discuss areas that deserve additional attention.&lt;/p&gt;

&lt;p&gt;No security product should be evaluated only through its marketing materials.&lt;/p&gt;

&lt;p&gt;There are several things I would investigate before deploying xCaptcha on an important production website.&lt;/p&gt;

&lt;p&gt;More Public Technical Documentation&lt;/p&gt;

&lt;p&gt;Developers benefit from detailed documentation, practical examples, SDKs, and troubleshooting guides.&lt;/p&gt;

&lt;p&gt;I would like to see comprehensive examples for popular technologies such as Node.js, Python, PHP, Laravel, React, and other commonly used frameworks.&lt;/p&gt;

&lt;p&gt;Independent Performance Benchmarks&lt;/p&gt;

&lt;p&gt;Claims about extremely fast detection and low false positive rates are interesting.&lt;/p&gt;

&lt;p&gt;However, independent testing under different workloads would make these claims easier to evaluate.&lt;/p&gt;

&lt;p&gt;Detailed False Positive Reporting&lt;/p&gt;

&lt;p&gt;Blocking malicious requests is important, but incorrectly blocking genuine visitors can be equally damaging.&lt;/p&gt;

&lt;p&gt;I would like to see particularly detailed reporting in this area.&lt;/p&gt;

&lt;p&gt;Flexible Configuration&lt;/p&gt;

&lt;p&gt;Different websites have different security requirements.&lt;/p&gt;

&lt;p&gt;A news website, an online store, and a financial application should not necessarily use identical protection settings.&lt;/p&gt;

&lt;p&gt;More granular configuration is always valuable.&lt;/p&gt;

&lt;p&gt;Clear Failure and Recovery Procedures&lt;/p&gt;

&lt;p&gt;Developers need to understand exactly what happens when a security service experiences an outage.&lt;/p&gt;

&lt;p&gt;Reliable fallback behavior is essential for production systems.&lt;/p&gt;

&lt;p&gt;These are not necessarily weaknesses in the product itself. They are areas I would examine as part of a responsible technical evaluation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;13. Who Would Benefit Most from xCaptcha?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Based on its positioning and feature set, I see several groups that could benefit from evaluating the platform.&lt;/p&gt;

&lt;p&gt;E-commerce businesses&lt;/p&gt;

&lt;p&gt;Online stores need to protect registration, login, checkout, and product inventory from automated abuse.&lt;/p&gt;

&lt;p&gt;SaaS companies&lt;/p&gt;

&lt;p&gt;Subscription-based platforms often deal with fake registrations, account abuse, and automated requests against public APIs.&lt;/p&gt;

&lt;p&gt;Content websites&lt;/p&gt;

&lt;p&gt;Websites with valuable articles, images, or proprietary information may want additional protection against unauthorized scraping.&lt;/p&gt;

&lt;p&gt;Ticketing platforms&lt;/p&gt;

&lt;p&gt;Automated ticket purchasing and inventory manipulation can create serious operational problems.&lt;/p&gt;

&lt;p&gt;Financial services&lt;/p&gt;

&lt;p&gt;Platforms dealing with transactions and customer accounts need additional layers of protection against suspicious automated behavior.&lt;/p&gt;

&lt;p&gt;Developers managing multiple websites&lt;/p&gt;

&lt;p&gt;A centralized security platform can simplify the management of protection across different projects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;14. My Recommended Evaluation Process&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If I were introducing xCaptcha into a new production project, I would not immediately enable every available security feature.&lt;/p&gt;

&lt;p&gt;I would start with a controlled evaluation.&lt;/p&gt;

&lt;p&gt;Here is the approach I would recommend.&lt;/p&gt;

&lt;p&gt;Phase 1: Understand Existing Traffic&lt;/p&gt;

&lt;p&gt;Before changing anything, collect baseline information.&lt;/p&gt;

&lt;p&gt;Identify important endpoints, normal request volumes, and existing automated activity.&lt;/p&gt;

&lt;p&gt;Phase 2: Choose a Suitable Test Environment&lt;/p&gt;

&lt;p&gt;Start with a staging environment or a low-risk website.&lt;/p&gt;

&lt;p&gt;This allows you to evaluate integration behavior without unnecessarily affecting customers.&lt;/p&gt;

&lt;p&gt;Phase 3: Integrate the Verification System&lt;/p&gt;

&lt;p&gt;Follow the official documentation and verify the integration on both the frontend and backend.&lt;/p&gt;

&lt;p&gt;Phase 4: Test Different Scenarios&lt;/p&gt;

&lt;p&gt;Check normal browsing, registration, login, form submissions, and API requests.&lt;/p&gt;

&lt;p&gt;Pay attention to unusual browsers, mobile devices, and accessibility requirements.&lt;/p&gt;

&lt;p&gt;Phase 5: Review Reporting&lt;/p&gt;

&lt;p&gt;Examine the available statistics and determine whether the protection system identifies suspicious traffic correctly.&lt;/p&gt;

&lt;p&gt;Phase 6: Measure Performance&lt;/p&gt;

&lt;p&gt;Compare response times and user experience before and after integration.&lt;/p&gt;

&lt;p&gt;Phase 7: Gradually Expand Protection&lt;/p&gt;

&lt;p&gt;Once the implementation behaves as expected, introduce it to additional pages and services.&lt;/p&gt;

&lt;p&gt;This gradual approach reduces the risk of accidentally disrupting legitimate users.&lt;/p&gt;




&lt;p&gt;After examining the platform, I think xCaptcha represents an interesting direction in website security.&lt;/p&gt;

&lt;p&gt;The traditional approach to CAPTCHA was relatively straightforward: display a challenge and ask visitors to solve it.&lt;/p&gt;

&lt;p&gt;But the internet has changed.&lt;/p&gt;

&lt;p&gt;Automated systems are becoming more sophisticated, websites are increasingly interconnected, and businesses need to protect more than just individual forms.&lt;/p&gt;

&lt;p&gt;They need to protect entire user journeys, APIs, accounts, transactions, and valuable content.&lt;/p&gt;

&lt;p&gt;This is why adaptive bot detection is becoming increasingly relevant.&lt;/p&gt;

&lt;p&gt;What I find most interesting about xCaptcha is its emphasis on reducing friction for legitimate visitors while making automated abuse more difficult.&lt;/p&gt;

&lt;p&gt;Task rotation, machine-learning-based risk analysis, reporting tools, and multiple protection scenarios make it worth investigating for websites that need more than a basic CAPTCHA widget.&lt;/p&gt;

&lt;p&gt;Would I recommend immediately replacing every existing security system with xCaptcha?&lt;/p&gt;

&lt;p&gt;No.&lt;/p&gt;

&lt;p&gt;I would first evaluate the platform in a controlled environment, check its documentation, test its behavior with real application traffic, and measure the impact on legitimate users.&lt;/p&gt;

&lt;p&gt;But would I include it in a shortlist when researching modern bot protection solutions?&lt;/p&gt;

&lt;p&gt;Absolutely.&lt;/p&gt;

&lt;p&gt;For developers and businesses looking for an alternative approach to traditional CAPTCHA protection, xCaptcha is a platform worth exploring.&lt;/p&gt;

&lt;p&gt;The most important thing is to remember that effective security is not about making life difficult for every visitor.&lt;/p&gt;

&lt;p&gt;It is about identifying suspicious activity, reducing unnecessary friction, and allowing legitimate users to interact with a website without constantly fighting verification screens.&lt;/p&gt;

&lt;p&gt;That is the direction I want to see more security platforms taking.&lt;/p&gt;

&lt;p&gt;Website: &lt;a href="https://xcaptcha.com/" rel="noopener noreferrer"&gt;https://xcaptcha.com/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Final recommendation: If you manage a website, SaaS platform, online store, or API that experiences automated abuse, take a look at xCaptcha, explore its free allowance, and evaluate whether its adaptive protection approach fits your infrastructure.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>How to Spy on Amazon Competitors and Find Profitable Products Without Manually Checking Thousands of Listings</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Thu, 12 Nov 2026 18:20:08 +0000</pubDate>
      <link>https://dev.to/gresiuseo/how-to-spy-on-amazon-competitors-and-find-profitable-products-without-manually-checking-thousands-3me8</link>
      <guid>https://dev.to/gresiuseo/how-to-spy-on-amazon-competitors-and-find-profitable-products-without-manually-checking-thousands-3me8</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9xkq9irifsmb2p4ow1y7.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9xkq9irifsmb2p4ow1y7.jpg" alt=" " width="800" height="447"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Selling on Amazon is competitive. A product that looks profitable today might become uncompetitive tomorrow because another seller has lowered the price, a new product has entered the market, or customer feedback has changed purchasing behavior.&lt;/p&gt;

&lt;p&gt;The biggest challenge is getting enough reliable information to make decisions quickly.&lt;/p&gt;

&lt;p&gt;Manually checking competitor listings, copying prices into spreadsheets, counting reviews, and comparing products is exhausting. And when you have hundreds of products to monitor, it becomes practically impossible.&lt;/p&gt;

&lt;p&gt;Fortunately, you don't need to spend your entire day browsing Amazon.&lt;/p&gt;

&lt;p&gt;With the open-source &lt;a href="https://github.com/2scraper/amazon-scraper" rel="noopener noreferrer"&gt;2Scraper Amazon Scraper&lt;/a&gt;, you can automate product data collection and turn Amazon listings into structured datasets ready for analysis.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Here is how to put it to work.&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Stop Guessing What Your Competitors Are Doing&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Imagine that you sell wireless headphones on Amazon.&lt;/p&gt;

&lt;p&gt;You want to understand:&lt;/p&gt;

&lt;p&gt;What prices competing sellers are offering.&lt;/p&gt;

&lt;p&gt;Which products have the highest customer ratings.&lt;/p&gt;

&lt;p&gt;How many reviews popular products receive.&lt;/p&gt;

&lt;p&gt;Which products appear in search results.&lt;/p&gt;

&lt;p&gt;Which listings are sponsored.&lt;/p&gt;

&lt;p&gt;How prices and product positions change over time.&lt;/p&gt;

&lt;p&gt;Checking these details manually across dozens of listings takes time and makes comparisons difficult.&lt;/p&gt;

&lt;p&gt;Automated data collection gives you a much clearer picture.&lt;/p&gt;

&lt;p&gt;The 2Scraper Amazon Scraper is a Python-based, open-source tool designed specifically for extracting Amazon product information. It supports search results, best-seller pages, individual product pages, and publicly displayed customer reviews.&lt;/p&gt;

&lt;p&gt;Instead of copying information one listing at a time, you can collect structured records and work with them in your preferred data analysis tools.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Collect the Amazon Data That Actually Matters&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Not every product detail is equally useful for market research.&lt;/p&gt;

&lt;p&gt;The scraper provides several extraction modes, each designed for a different task.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Search results and best sellers&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Collect product listings and compare:&lt;/p&gt;

&lt;p&gt;ASIN identifiers.&lt;/p&gt;

&lt;p&gt;Product titles.&lt;/p&gt;

&lt;p&gt;Prices and currencies.&lt;/p&gt;

&lt;p&gt;Ratings and review counts.&lt;/p&gt;

&lt;p&gt;Product positions.&lt;/p&gt;

&lt;p&gt;Sponsored placements.&lt;/p&gt;

&lt;p&gt;Discounts and coupons.&lt;/p&gt;

&lt;p&gt;Availability information.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Individual product pages&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When you need more detailed information, product mode can extract:&lt;/p&gt;

&lt;p&gt;Brand and seller.&lt;/p&gt;

&lt;p&gt;Product descriptions and bullet points.&lt;/p&gt;

&lt;p&gt;Images.&lt;/p&gt;

&lt;p&gt;Availability.&lt;/p&gt;

&lt;p&gt;Product variations.&lt;/p&gt;

&lt;p&gt;Original prices and discounts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Customer reviews&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Review mode collects publicly displayed reviews, including ratings, authors, dates, verified-purchase indicators, helpful votes, and review text.&lt;/p&gt;

&lt;p&gt;This is particularly useful when researching customer complaints, identifying popular product features, or understanding why competing products receive negative feedback.&lt;/p&gt;

&lt;p&gt;The important advantage is that you can collect different types of information without building an entirely separate scraper for every task.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Compare Competitor Prices Automatically&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Price monitoring is one of the most practical applications of Amazon scraping.&lt;/p&gt;

&lt;p&gt;Let's say you have identified 30 competing products.&lt;/p&gt;

&lt;p&gt;You collect their prices today, repeat the process tomorrow, and compare the results.&lt;/p&gt;

&lt;p&gt;Your dataset might look like this:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn8g3s9peu90bal1fjvx1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fn8g3s9peu90bal1fjvx1.png" alt=" " width="741" height="150"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The repository includes diff_runs.py, a utility that compares two scraping runs and identifies product-level changes, including price differences.&lt;/p&gt;

&lt;p&gt;It also supports price tolerance settings to avoid treating tiny currency-conversion fluctuations as meaningful price changes.&lt;/p&gt;

&lt;p&gt;This makes it easier to distinguish actual pricing movements from minor noise.&lt;/p&gt;

&lt;p&gt;For sellers, this information can help with pricing decisions, competitor research, and identifying products that deserve closer attention.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Discover Potentially Profitable Products&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Finding a profitable product is not simply about finding something with a high price.&lt;/p&gt;

&lt;p&gt;You need to understand the competitive environment.&lt;/p&gt;

&lt;p&gt;For example, consider two products:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Product A&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Price: $25&lt;/li&gt;
&lt;li&gt;Rating: 4.8&lt;/li&gt;
&lt;li&gt;Reviews: 8,000&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Product B&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Price: $35&lt;/li&gt;
&lt;li&gt;Rating: 4.5&lt;/li&gt;
&lt;li&gt;Reviews: 250&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Product A has stronger social proof, but Product B might represent an interesting opportunity if demand, competition, sourcing costs, and margins support it.&lt;/p&gt;

&lt;p&gt;By collecting information across hundreds of listings, you can build filters to identify products with characteristics worth investigating.&lt;/p&gt;

&lt;p&gt;Useful research criteria include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Products within your target price range.&lt;/li&gt;
&lt;li&gt;Listings with relatively few reviews.&lt;/li&gt;
&lt;li&gt;Products with strong ratings.&lt;/li&gt;
&lt;li&gt;Categories with manageable competition.&lt;/li&gt;
&lt;li&gt;Listings with frequent price changes.&lt;/li&gt;
&lt;li&gt;Products with multiple variations.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Remember that Amazon listing data alone cannot establish actual sales volume or profitability. You still need to calculate sourcing costs, Amazon fees, advertising expenses, shipping, and expected demand.&lt;/p&gt;

&lt;p&gt;However, having structured product information makes the initial research considerably easier.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Get Started Without Building a Scraper from Scratch&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One of the best things about this project is that it is open source and released under the MIT license.&lt;/p&gt;

&lt;p&gt;You can inspect the code, modify the extraction logic, and run it on your own infrastructure.&lt;/p&gt;

&lt;p&gt;The basic setup is straightforward if you already have Python and Git installed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 1. Clone the repository&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;git clone &lt;a href="https://github.com/2scraper/amazon-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/amazon-scraper&lt;/a&gt;&lt;br&gt;
cd amazon-scraper&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 2. Install the dependencies&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;pip install -r requirements.txt&lt;br&gt;
pip install -r requirements-playwright.txt&lt;br&gt;
playwright install chromium&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 3. Collect Amazon search results&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;python3 playwright_scraper.py \&lt;br&gt;
  --url "&lt;a href="https://www.amazon.com/s?k=wireless+headphones" rel="noopener noreferrer"&gt;https://www.amazon.com/s?k=wireless+headphones&lt;/a&gt;" \&lt;br&gt;
  --pages 3 \&lt;br&gt;
  --format csv&lt;/p&gt;

&lt;p&gt;This example collects up to three pages of search results and exports the records into CSV format.&lt;/p&gt;

&lt;p&gt;You can then open the file in Excel, import it into Google Sheets, or process it using Python.&lt;/p&gt;

&lt;p&gt;For additional extraction modes, consult the repository's README.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Work Across Multiple Amazon Marketplaces&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Another useful feature is marketplace support.&lt;/p&gt;

&lt;p&gt;The marketplace is determined by the hostname in the URL.&lt;/p&gt;

&lt;p&gt;This is particularly useful for sellers researching international opportunities or comparing product positioning across different countries.&lt;/p&gt;

&lt;p&gt;One important detail: prices and currencies can depend on the visitor's delivery country and currency conversion. Always check the extracted currency rather than assuming it from the marketplace domain.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Don't Lose Your Data When a Scraping Run Fails&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A useful scraping tool should do more than collect information. It should also help you understand whether the results are complete.&lt;/p&gt;

&lt;p&gt;The 2Scraper Amazon Scraper generates a metadata file alongside successful exports.&lt;/p&gt;

&lt;p&gt;This file includes information such as:&lt;/p&gt;

&lt;p&gt;Number of records collected.&lt;/p&gt;

&lt;p&gt;Pages requested and completed.&lt;/p&gt;

&lt;p&gt;Failed pages.&lt;/p&gt;

&lt;p&gt;Extraction status.&lt;/p&gt;

&lt;p&gt;Source marketplace.&lt;/p&gt;

&lt;p&gt;Collection timestamp.&lt;/p&gt;

&lt;p&gt;The scraper also distinguishes between complete, partial, and failed runs.&lt;/p&gt;

&lt;p&gt;Why does this matter?&lt;/p&gt;

&lt;p&gt;Imagine collecting 500 products and encountering a temporary website restriction halfway through the process. Without proper status information, you might accidentally treat an incomplete dataset as a complete market analysis.&lt;/p&gt;

&lt;p&gt;Metadata makes it easier to detect such problems before using the results.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Build Your Own Amazon Research System&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The real value appears when you combine automated extraction with your own analysis.&lt;/p&gt;

&lt;p&gt;A simple workflow could look like this:&lt;/p&gt;

&lt;p&gt;Amazon Search Results&lt;br&gt;
        |&lt;br&gt;
        v&lt;br&gt;
2Scraper Amazon Scraper&lt;br&gt;
        |&lt;br&gt;
        v&lt;br&gt;
Product Data (CSV / JSON)&lt;br&gt;
        |&lt;br&gt;
        v&lt;br&gt;
Python / Excel / Google Sheets&lt;br&gt;
        |&lt;br&gt;
        v&lt;br&gt;
Price Comparison&lt;br&gt;
        |&lt;br&gt;
        v&lt;br&gt;
Competitor Analysis&lt;br&gt;
        |&lt;br&gt;
        v&lt;br&gt;
Product Research Decisions&lt;/p&gt;

&lt;p&gt;You can extend this workflow with scheduled runs, databases, dashboards, and custom reports.&lt;/p&gt;

&lt;p&gt;For example, a seller could collect competitor prices every day, compare them with previous records, and generate a report highlighting products with significant price changes.&lt;/p&gt;

&lt;p&gt;Developers can also integrate the exported data into existing applications and analytics systems.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is It Worth Trying?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you regularly research Amazon products, monitor competitors, or analyze marketplace trends, manually collecting information is an unnecessary bottleneck.&lt;/p&gt;

&lt;p&gt;The 2Scraper Amazon Scraper on GitHub offers a practical starting point:&lt;/p&gt;

&lt;p&gt;Free and open-source Python implementation.&lt;/p&gt;

&lt;p&gt;Three extraction modes: listings, products, and reviews.&lt;/p&gt;

&lt;p&gt;Support for 21 Amazon marketplaces.&lt;/p&gt;

&lt;p&gt;CSV and JSON exports.&lt;/p&gt;

&lt;p&gt;Price comparison between scraping runs.&lt;/p&gt;

&lt;p&gt;Run metadata for checking extraction completeness.&lt;/p&gt;

&lt;p&gt;Multiple browser engines.&lt;/p&gt;

&lt;p&gt;The project also documents its infrastructure requirements. Its measured baseline demonstrates that ordinary local Chromium can handle a three-page search run without an API key or proxy from a residential European connection. Larger workloads or different access conditions may require additional infrastructure.&lt;/p&gt;

&lt;p&gt;Successful Amazon selling requires good decisions, and good decisions require useful information.&lt;/p&gt;

&lt;p&gt;You don't need to manually inspect thousands of listings to understand your competitors. You need a repeatable way to collect product information, compare prices, examine customer feedback, and identify opportunities worth investigating.&lt;/p&gt;

&lt;p&gt;That is exactly the kind of workflow the 2Scraper Amazon Scraper is designed to support.&lt;/p&gt;

&lt;p&gt;Stop spending hours copying Amazon listings into spreadsheets. Automate the collection, analyze the results, and spend more time making smarter product decisions.&lt;/p&gt;

&lt;p&gt;Get started here: &lt;a href="https://github.com/2scraper/amazon-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/amazon-scraper&lt;/a&gt;&lt;/p&gt;

</description>
      <category>automation</category>
      <category>startup</category>
      <category>tools</category>
      <category>webscraping</category>
    </item>
    <item>
      <title>2Scraper — How to Collect Website Data Without Copying Pages by Hand</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Tue, 10 Nov 2026 16:58:57 +0000</pubDate>
      <link>https://dev.to/gresiuseo/2scraper-how-to-collect-website-data-without-copying-pages-by-hand-921</link>
      <guid>https://dev.to/gresiuseo/2scraper-how-to-collect-website-data-without-copying-pages-by-hand-921</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz3q3ssk5l1b70yeahis1.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fz3q3ssk5l1b70yeahis1.jpg" alt=" " width="800" height="447"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h1&gt;
  
  
  2Scraper: How to Collect Website Data Without Copying Pages by Hand
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Stop spending hours copying information from websites. Discover how web scraping can automate data collection, organize information, and turn repetitive browser tasks into useful workflows.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem With Collecting Data Manually
&lt;/h2&gt;

&lt;p&gt;Imagine that you need information about 200 products from an online marketplace.&lt;/p&gt;

&lt;p&gt;You open the first product page, copy its name, price, rating, and availability, then paste everything into a spreadsheet. Next, you repeat the process for the second product, the third, and eventually all 200.&lt;/p&gt;

&lt;p&gt;Now imagine doing the same thing every morning.&lt;/p&gt;

&lt;p&gt;It is repetitive, time-consuming, and surprisingly easy to get wrong. A missed price, an incorrect product title, or a value copied into the wrong column can make the entire spreadsheet unreliable.&lt;/p&gt;

&lt;p&gt;The problem becomes even bigger when information changes frequently.&lt;/p&gt;

&lt;p&gt;Product prices fluctuate. New listings appear. Reviews accumulate. Job vacancies disappear. Property listings are updated.&lt;/p&gt;

&lt;p&gt;Manually checking all these changes is not a practical long-term strategy.&lt;/p&gt;

&lt;p&gt;Fortunately, there is a better approach: &lt;strong&gt;web scraping&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Instead of copying information page by page, you can use software to visit websites, extract selected information, and save it in a structured format.&lt;/p&gt;

&lt;p&gt;One project designed to make this process easier is &lt;strong&gt;2Scraper&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Explore the collection at &lt;a href="https://2scraper.com/" rel="noopener noreferrer"&gt;https://2scraper.com/&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What Exactly Is Web Scraping?
&lt;/h2&gt;

&lt;p&gt;Web scraping is the automated extraction of information from websites.&lt;/p&gt;

&lt;p&gt;A scraper visits a webpage, identifies the elements containing useful information, extracts their values, and organizes the results.&lt;/p&gt;

&lt;p&gt;For example, an online product page may contain:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Product name.&lt;/li&gt;
&lt;li&gt;Current price.&lt;/li&gt;
&lt;li&gt;Original price.&lt;/li&gt;
&lt;li&gt;Discount.&lt;/li&gt;
&lt;li&gt;Customer rating.&lt;/li&gt;
&lt;li&gt;Number of reviews.&lt;/li&gt;
&lt;li&gt;Product image.&lt;/li&gt;
&lt;li&gt;Availability.&lt;/li&gt;
&lt;li&gt;Product identifier.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A scraper can extract supported fields and save them as individual records.&lt;/p&gt;

&lt;p&gt;Instead of manually copying information into a spreadsheet, you receive a dataset that can be processed by other software.&lt;/p&gt;

&lt;p&gt;A simplified example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Product&lt;/th&gt;
&lt;th&gt;Price&lt;/th&gt;
&lt;th&gt;Rating&lt;/th&gt;
&lt;th&gt;Availability&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Wireless Headphones&lt;/td&gt;
&lt;td&gt;$79.99&lt;/td&gt;
&lt;td&gt;4.5&lt;/td&gt;
&lt;td&gt;In stock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mechanical Keyboard&lt;/td&gt;
&lt;td&gt;$59.99&lt;/td&gt;
&lt;td&gt;4.7&lt;/td&gt;
&lt;td&gt;In stock&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gaming Mouse&lt;/td&gt;
&lt;td&gt;$39.99&lt;/td&gt;
&lt;td&gt;4.4&lt;/td&gt;
&lt;td&gt;Out of stock&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The table is illustrative, but it demonstrates the main idea: information from different webpages becomes easier to compare once it is organized.&lt;/p&gt;

&lt;h2&gt;
  
  
  Meet 2Scraper: Ready-to-Use Tools for Different Websites
&lt;/h2&gt;

&lt;p&gt;Building a web scraper from scratch is not always easy.&lt;/p&gt;

&lt;p&gt;Every website has its own page structure, loading behavior, pagination system, and data formats. Some pages render content dynamically, while others expose structured information through public endpoints.&lt;/p&gt;

&lt;p&gt;A scraper built for one website may not work on another without significant modifications.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2Scraper&lt;/strong&gt; addresses this problem with a collection of open-source Python scrapers, each designed for a particular website.&lt;/p&gt;

&lt;p&gt;Instead of starting with an empty Python file, developers can browse the available projects, choose a supported platform, and follow its installation instructions.&lt;/p&gt;

&lt;p&gt;The collection covers multiple categories:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;E-commerce and retail.&lt;/li&gt;
&lt;li&gt;Classifieds and property.&lt;/li&gt;
&lt;li&gt;Travel.&lt;/li&gt;
&lt;li&gt;Social media and video.&lt;/li&gt;
&lt;li&gt;Publishing and online communities.&lt;/li&gt;
&lt;li&gt;Jobs and business directories.&lt;/li&gt;
&lt;li&gt;Finance and market information.&lt;/li&gt;
&lt;li&gt;Sports.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Examples include Amazon, Etsy, YouTube, TikTok, Facebook Marketplace, Google Play, Goodreads, Binance, and Transfermarkt.&lt;/p&gt;

&lt;p&gt;Each repository documents its supported pages, extracted fields, dependencies, and execution requirements.&lt;/p&gt;

&lt;p&gt;The project currently provides dozens of public scraper repositories, making it a useful starting point for different data collection projects.&lt;/p&gt;

&lt;p&gt;You can browse the complete directory here:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://2scraper.com/" rel="noopener noreferrer"&gt;https://2scraper.com/&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Five Everyday Tasks You Can Automate With 2Scraper
&lt;/h2&gt;

&lt;p&gt;The value of web scraping becomes much easier to understand when you look at practical examples.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Monitor Competitor Prices
&lt;/h3&gt;

&lt;p&gt;For an online store, pricing information is important.&lt;/p&gt;

&lt;p&gt;A competitor might reduce the price of a popular product overnight. Another store might introduce a discount or start selling a new model.&lt;/p&gt;

&lt;p&gt;Checking every product manually is inefficient.&lt;/p&gt;

&lt;p&gt;With an appropriate scraper, you can collect supported product information and store it in a spreadsheet or database.&lt;/p&gt;

&lt;p&gt;A simple automated workflow could:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Collect product information from selected public pages.&lt;/li&gt;
&lt;li&gt;Save prices and product identifiers.&lt;/li&gt;
&lt;li&gt;Compare the latest results with previous records.&lt;/li&gt;
&lt;li&gt;Identify price changes.&lt;/li&gt;
&lt;li&gt;Generate a report.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The result is a basic competitor monitoring system.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Research Products Before Entering a Market
&lt;/h3&gt;

&lt;p&gt;Imagine that you want to launch a new product.&lt;/p&gt;

&lt;p&gt;Before investing money, you need to understand what is already available, how competing products are positioned, and what customers are saying.&lt;/p&gt;

&lt;p&gt;Scraped product listings and publicly available reviews can help organize this research.&lt;/p&gt;

&lt;p&gt;You could compare:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Product categories.&lt;/li&gt;
&lt;li&gt;Advertised prices.&lt;/li&gt;
&lt;li&gt;Ratings.&lt;/li&gt;
&lt;li&gt;Review counts.&lt;/li&gt;
&lt;li&gt;Product specifications.&lt;/li&gt;
&lt;li&gt;Public customer feedback.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Instead of opening dozens of tabs and taking notes manually, you can work with structured information.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Collect Public Content for Research
&lt;/h3&gt;

&lt;p&gt;Content creators, marketers, and researchers often need to analyze what people are discussing online.&lt;/p&gt;

&lt;p&gt;Some 2Scraper projects support collecting public content and metadata from platforms such as YouTube and Medium.&lt;/p&gt;

&lt;p&gt;Depending on the repository, this may include video information, comments, replies, articles, authors, or publication details.&lt;/p&gt;

&lt;p&gt;This can help with:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Topic research.&lt;/li&gt;
&lt;li&gt;Content planning.&lt;/li&gt;
&lt;li&gt;Public discussion analysis.&lt;/li&gt;
&lt;li&gt;Competitor research.&lt;/li&gt;
&lt;li&gt;Identifying frequently discussed questions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, a marketing team could analyze public comments to discover which questions customers repeatedly ask about a product category.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Find Business and Employment Opportunities
&lt;/h3&gt;

&lt;p&gt;Job platforms and business directories contain information that can be useful for market research.&lt;/p&gt;

&lt;p&gt;A suitable scraper can help organize supported public listings into a dataset.&lt;/p&gt;

&lt;p&gt;For employment research, relevant fields might include job titles, salaries, skills, seniority, and company information.&lt;/p&gt;

&lt;p&gt;For business research, the available fields depend on the specific target and repository.&lt;/p&gt;

&lt;p&gt;Once the information is collected, it becomes easier to filter, compare, and analyze.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Track Market and Financial Information
&lt;/h3&gt;

&lt;p&gt;Some projects in the 2Scraper collection focus on financial platforms and market-related information.&lt;/p&gt;

&lt;p&gt;Depending on the supported source, developers can collect publicly available market data, company information, financial indicators, or other structured records.&lt;/p&gt;

&lt;p&gt;This can be useful for historical research, dashboards, and analytical experiments.&lt;/p&gt;

&lt;p&gt;Always verify data timestamps, source definitions, and update frequency before relying on scraped information for financial decisions.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Start Collecting Data With 2Scraper
&lt;/h2&gt;

&lt;p&gt;You do not need to build a complete data platform to get started.&lt;/p&gt;

&lt;p&gt;The easiest approach is to choose one website and run one scraper.&lt;/p&gt;

&lt;p&gt;Here is a practical example using the Amazon scraper.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 1: Choose a Scraper
&lt;/h3&gt;

&lt;p&gt;Open the official directory:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://2scraper.com/" rel="noopener noreferrer"&gt;https://2scraper.com/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Find the website you want to collect information from.&lt;/p&gt;

&lt;p&gt;For this example, we will use the Amazon project:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/amazon-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/amazon-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Before running it, read the repository README. It explains supported pages, extraction modes, dependencies, and output formats.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 2: Install the Project
&lt;/h3&gt;

&lt;p&gt;The Amazon scraper is written in Python.&lt;/p&gt;

&lt;p&gt;First, clone the repository:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/2scraper/amazon-scraper
&lt;span class="nb"&gt;cd&lt;/span&gt; amazon-scraper
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Install its dependencies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install -r requirements.txt
pip install -r requirements-playwright.txt
playwright install chromium
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact setup may change, so follow the current repository instructions if they differ.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 3: Run Your First Collection
&lt;/h3&gt;

&lt;p&gt;For example, you can collect results for a product search:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 playwright_scraper.py \
  --url &lt;span class="s2"&gt;"https://www.amazon.com/s?k=wireless+headphones"&lt;/span&gt; \
  --pages 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is an example of the documented listing mode.&lt;/p&gt;

&lt;p&gt;The scraper visits the supported search pages and extracts available product information.&lt;/p&gt;

&lt;p&gt;Depending on the mode, the project can collect fields such as product titles, ASINs, prices, ratings, review counts, and availability.&lt;/p&gt;

&lt;h3&gt;
  
  
  Step 4: Export the
&lt;/h3&gt;

</description>
    </item>
    <item>
      <title>2scraper: How to Build Practical Web Data Pipelines Without Writing Every Scraper from Scratch</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Fri, 06 Nov 2026 14:44:36 +0000</pubDate>
      <link>https://dev.to/gresiuseo/2scraper-how-to-build-practical-web-data-pipelines-without-writing-every-scraper-from-scratch-4k6d</link>
      <guid>https://dev.to/gresiuseo/2scraper-how-to-build-practical-web-data-pipelines-without-writing-every-scraper-from-scratch-4k6d</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft7uxmyz2u5dqodx8kh59.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft7uxmyz2u5dqodx8kh59.png" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Web scraping is one of those engineering tasks that looks deceptively simple.&lt;/p&gt;

&lt;p&gt;You open a website, inspect its HTML, find a few CSS selectors, write a Python script, and export the information into a CSV file.&lt;/p&gt;

&lt;p&gt;The first version usually takes less than an hour.&lt;/p&gt;

&lt;p&gt;Then the real problems begin.&lt;/p&gt;

&lt;p&gt;The website changes its HTML structure. Product cards load dynamically. Pagination behaves differently from what you expected. Some fields are missing. A browser session gets blocked. A request returns an unexpected response. Your parser silently starts producing incomplete records.&lt;/p&gt;

&lt;p&gt;Suddenly, a small scraping script becomes an application that needs maintenance, monitoring, testing, retries, browser management, and data validation.&lt;/p&gt;

&lt;p&gt;This is exactly the kind of repetitive infrastructure work that &lt;strong&gt;&lt;a href="https://github.com/2scraper" rel="noopener noreferrer"&gt;2scraper&lt;/a&gt;&lt;/strong&gt; aims to reduce.&lt;/p&gt;

&lt;p&gt;2scraper is an open-source collection of Python web scrapers, organized into separate repositories for individual websites. Instead of providing one generic scraping framework that tries to understand every website, it provides focused implementations designed around specific platforms and their data structures.&lt;/p&gt;

&lt;p&gt;The repositories cover different categories, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;E-commerce and product catalogs.&lt;/li&gt;
&lt;li&gt;Social media and video platforms.&lt;/li&gt;
&lt;li&gt;Classified advertisements.&lt;/li&gt;
&lt;li&gt;Real estate listings.&lt;/li&gt;
&lt;li&gt;Financial information.&lt;/li&gt;
&lt;li&gt;Job listings.&lt;/li&gt;
&lt;li&gt;Publishing platforms.&lt;/li&gt;
&lt;li&gt;Marketplaces and auction websites.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each scraper has its own documented capabilities, command-line interface, supported extraction modes, and output schema.&lt;/p&gt;

&lt;p&gt;The idea is straightforward: start with an existing implementation, run it on your own infrastructure, inspect the extracted data, and adapt it to your application.&lt;/p&gt;

&lt;p&gt;In this tutorial, we will explore how to use the collection in practical development scenarios.&lt;/p&gt;

&lt;p&gt;We will:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Understand the architecture of 2scraper.&lt;/li&gt;
&lt;li&gt;Select the right scraper for a project.&lt;/li&gt;
&lt;li&gt;Install and configure a real repository.&lt;/li&gt;
&lt;li&gt;Collect structured product data with Python.&lt;/li&gt;
&lt;li&gt;Work with JSON and CSV output.&lt;/li&gt;
&lt;li&gt;Validate scraping results programmatically.&lt;/li&gt;
&lt;li&gt;Handle incomplete runs and errors.&lt;/li&gt;
&lt;li&gt;Automate recurring extraction jobs.&lt;/li&gt;
&lt;li&gt;Integrate the output into a data pipeline.&lt;/li&gt;
&lt;li&gt;Decide when to use a local browser, direct HTTP requests, or a hosted browser.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Our main example will use the Amazon scraper, but the same general approach applies to other repositories, provided you follow their individual README instructions.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. What Exactly Is 2scraper?
&lt;/h2&gt;

&lt;p&gt;Start with the main GitHub organization:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The organization contains multiple independent repositories.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Repository&lt;/th&gt;
&lt;th&gt;Main purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;amazon-scraper&lt;/td&gt;
&lt;td&gt;Product listings, product pages, reviews&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;youtube-scraper&lt;/td&gt;
&lt;td&gt;Video metadata, comments, search&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;catawiki-scraper&lt;/td&gt;
&lt;td&gt;Auction lots, bids, sellers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;stockx-scraper&lt;/td&gt;
&lt;td&gt;Sneaker prices and marketplace data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;transfermarkt-scraper&lt;/td&gt;
&lt;td&gt;Football players, squads, transfers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;google-play-scraper&lt;/td&gt;
&lt;td&gt;App listings and reviews&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;hackernews-scraper&lt;/td&gt;
&lt;td&gt;Stories, comments, user profiles&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;medium-scraper&lt;/td&gt;
&lt;td&gt;Articles, authors, publications&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;screener-scraper&lt;/td&gt;
&lt;td&gt;Financial and stock-market data&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This repository-per-website approach is important.&lt;/p&gt;

&lt;p&gt;Websites have different structures, different pagination mechanisms, and different ways of exposing information.&lt;/p&gt;

&lt;p&gt;A universal scraper must make many assumptions. A site-specific scraper can instead implement the extraction logic around the actual website.&lt;/p&gt;

&lt;p&gt;For example, an e-commerce scraper may need to understand:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Product identifiers.&lt;/li&gt;
&lt;li&gt;Price containers.&lt;/li&gt;
&lt;li&gt;Discount labels.&lt;/li&gt;
&lt;li&gt;Availability indicators.&lt;/li&gt;
&lt;li&gt;Product variations.&lt;/li&gt;
&lt;li&gt;Seller information.&lt;/li&gt;
&lt;li&gt;Search result pagination.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A social media scraper has completely different requirements.&lt;/p&gt;

&lt;p&gt;Keeping these implementations separate makes the code easier to understand, test, and maintain.&lt;/p&gt;

&lt;h3&gt;
  
  
  Open source and local execution
&lt;/h3&gt;

&lt;p&gt;The 2scraper repositories use the MIT license, according to the organization and repository documentation.&lt;/p&gt;

&lt;p&gt;This is useful for developers who want to inspect the implementation, modify extraction logic, or integrate the code into their own systems.&lt;/p&gt;

&lt;p&gt;The scrapers are designed to run on your own infrastructure.&lt;/p&gt;

&lt;p&gt;That means you can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Run them on a development computer.&lt;/li&gt;
&lt;li&gt;Execute them on a Linux server.&lt;/li&gt;
&lt;li&gt;Package them with Docker.&lt;/li&gt;
&lt;li&gt;Schedule them with cron.&lt;/li&gt;
&lt;li&gt;Integrate them into Python applications.&lt;/li&gt;
&lt;li&gt;Store their output in your own database.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You are not required to send all collected data to a centralized scraping dashboard.&lt;/p&gt;

&lt;p&gt;The individual repository is the source of truth for supported fields, required infrastructure, and available execution modes.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. When Should You Use 2scraper?
&lt;/h2&gt;

&lt;p&gt;Not every web scraping project needs an existing scraper.&lt;/p&gt;

&lt;p&gt;If you need to extract five elements from a simple static website, a short BeautifulSoup script may be sufficient.&lt;/p&gt;

&lt;p&gt;However, an existing site-specific implementation becomes much more attractive when the extraction process involves several interconnected problems.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scenario A: Product price monitoring
&lt;/h3&gt;

&lt;p&gt;Imagine that you operate an online store and want to monitor publicly displayed prices across several marketplaces.&lt;/p&gt;

&lt;p&gt;You need to collect:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Product names.&lt;/li&gt;
&lt;li&gt;Product identifiers.&lt;/li&gt;
&lt;li&gt;Current prices.&lt;/li&gt;
&lt;li&gt;Original prices.&lt;/li&gt;
&lt;li&gt;Ratings.&lt;/li&gt;
&lt;li&gt;Review counts.&lt;/li&gt;
&lt;li&gt;Availability.&lt;/li&gt;
&lt;li&gt;Product URLs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Writing a scraper for every marketplace independently requires considerable work.&lt;/p&gt;

&lt;p&gt;With 2scraper, you can inspect the available repositories and start from an implementation that already understands the target platform.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scenario B: Competitor research
&lt;/h3&gt;

&lt;p&gt;Suppose you want to compare products from several retailers.&lt;/p&gt;

&lt;p&gt;Your application needs structured records rather than screenshots or raw HTML.&lt;/p&gt;

&lt;p&gt;A scraper that produces predictable JSON or CSV output is much easier to integrate into a comparison engine.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scenario C: Market research
&lt;/h3&gt;

&lt;p&gt;You might want to analyze:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Used vehicle listings.&lt;/li&gt;
&lt;li&gt;Real estate prices.&lt;/li&gt;
&lt;li&gt;Auction activity.&lt;/li&gt;
&lt;li&gt;Job offers.&lt;/li&gt;
&lt;li&gt;Financial indicators.&lt;/li&gt;
&lt;li&gt;Public product reviews.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The same principle applies: find the repository that already targets the platform and examine its extraction capabilities.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scenario D: AI and data pipelines
&lt;/h3&gt;

&lt;p&gt;Modern AI applications often need external information as input.&lt;/p&gt;

&lt;p&gt;For example, you might want to collect product descriptions and reviews, normalize them, and then pass selected records to an LLM for classification or summarization.&lt;/p&gt;

&lt;p&gt;A structured scraping output is a much better starting point than a collection of inconsistent HTML documents.&lt;/p&gt;

&lt;h3&gt;
  
  
  When should you avoid it?
&lt;/h3&gt;

&lt;p&gt;There are also situations where 2scraper may not be the right solution.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The target website is not supported.&lt;/li&gt;
&lt;li&gt;You need authenticated access that the repository does not implement.&lt;/li&gt;
&lt;li&gt;You need a completely different data schema.&lt;/li&gt;
&lt;li&gt;Your project requires an officially supported API with a formal SLA.&lt;/li&gt;
&lt;li&gt;The repository's last live test is too old for your requirements.&lt;/li&gt;
&lt;li&gt;The target's terms or applicable laws do not permit your intended collection.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In those cases, you may need a custom implementation or an official data API.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Choosing the Right Repository
&lt;/h2&gt;

&lt;p&gt;Before cloning anything, inspect the repository directory.&lt;/p&gt;

&lt;p&gt;Start here:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;For each target, check four things.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First: What data does it actually extract?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A repository might support product listings but not complete product details.&lt;/p&gt;

&lt;p&gt;Another might support public comments but not private or authenticated content.&lt;/p&gt;

&lt;p&gt;Do not assume that the presence of a website in the directory means every page on that website is supported.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Second: Which execution engines are supported?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Many repositories include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Playwright.&lt;/li&gt;
&lt;li&gt;Selenium.&lt;/li&gt;
&lt;li&gt;Puppeteer.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Some targets also have a browserless HTTP implementation or use the website's own public data endpoints.&lt;/p&gt;

&lt;p&gt;Playwright is generally the recommended starting point across the collection.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Third: Does the scraper require additional infrastructure?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Some repositories work with a normal local browser and no API credentials.&lt;/p&gt;

&lt;p&gt;Others may require a residential proxy, CAPTCHA processing, or a hosted browser.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fourth: How recent is the validation?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Look for test badges, dated measurements, sample output, and troubleshooting documentation.&lt;/p&gt;

&lt;p&gt;A scraper that worked six months ago is not automatically guaranteed to work today.&lt;/p&gt;

&lt;h3&gt;
  
  
  Example: Amazon scraper
&lt;/h3&gt;

&lt;p&gt;Repository:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/amazon-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/amazon-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The documented implementation supports three main extraction modes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;Extracted information&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Listing&lt;/td&gt;
&lt;td&gt;Search results and best-seller products&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Product&lt;/td&gt;
&lt;td&gt;Detailed product information and variations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reviews&lt;/td&gt;
&lt;td&gt;Reviews displayed on the product page&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;It supports multiple Amazon marketplaces and exports structured data.&lt;/p&gt;

&lt;p&gt;The repository also includes a comparison utility for analyzing differences between scraping runs.&lt;/p&gt;

&lt;p&gt;This makes it a useful starting point for price-monitoring applications.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. Installing Your First Scraper
&lt;/h2&gt;

&lt;p&gt;Let's build a real project.&lt;/p&gt;

&lt;p&gt;We will use Python and the Amazon scraper.&lt;/p&gt;

&lt;p&gt;You need:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Python 3.9 or newer.&lt;/li&gt;
&lt;li&gt;Git.&lt;/li&gt;
&lt;li&gt;A terminal.&lt;/li&gt;
&lt;li&gt;An internet connection.&lt;/li&gt;
&lt;li&gt;Chromium installed through Playwright.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Start by cloning the repository:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/2scraper/amazon-scraper.git

&lt;span class="nb"&gt;cd &lt;/span&gt;amazon-scraper
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Create a virtual environment:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python &lt;span class="nt"&gt;-m&lt;/span&gt; venv .venv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Activate it on Linux or macOS:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;source&lt;/span&gt; .venv/bin/activate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Windows:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight powershell"&gt;&lt;code&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;venv&lt;/span&gt;&lt;span class="n"&gt;\Scripts\activate&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Install the core dependencies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; requirements.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then install the Playwright engine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; requirements-playwright.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Install Chromium:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;playwright &lt;span class="nb"&gt;install &lt;/span&gt;chromium
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The project now has the main dependencies required to execute the Playwright scraper.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why use a virtual environment?
&lt;/h3&gt;

&lt;p&gt;Python scraping projects often depend on several libraries with specific version requirements.&lt;/p&gt;

&lt;p&gt;Playwright, Selenium, and Puppeteer-related packages can have conflicting dependencies.&lt;/p&gt;

&lt;p&gt;Installing everything globally may cause one project to break another.&lt;/p&gt;

&lt;p&gt;A dedicated virtual environment isolates the scraper's dependencies and makes deployment much easier.&lt;/p&gt;

&lt;p&gt;If you need to run multiple browser engines, the repository documentation recommends separate environments rather than combining incompatible dependency sets.&lt;/p&gt;




&lt;h2&gt;
  
  
  5. Your First Scraping Task
&lt;/h2&gt;

&lt;p&gt;Let's start with a small extraction job.&lt;/p&gt;

&lt;p&gt;The Amazon scraper supports listing extraction using a URL.&lt;/p&gt;

&lt;p&gt;For example, to collect wireless headphone listings:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 playwright_scraper.py &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--url&lt;/span&gt; &lt;span class="s2"&gt;"https://www.amazon.com/s?k=wireless+headphones"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--pages&lt;/span&gt; 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This command:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Opens the specified search page.&lt;/li&gt;
&lt;li&gt;Processes the requested pagination.&lt;/li&gt;
&lt;li&gt;Extracts product records.&lt;/li&gt;
&lt;li&gt;Removes duplicate products where applicable.&lt;/li&gt;
&lt;li&gt;Writes structured output.&lt;/li&gt;
&lt;li&gt;Generates run metadata.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The default output is JSON.&lt;/p&gt;

&lt;p&gt;You can also request CSV:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 playwright_scraper.py &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--url&lt;/span&gt; &lt;span class="s2"&gt;"https://www.amazon.com/s?k=wireless+headphones"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--pages&lt;/span&gt; 3 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--format&lt;/span&gt; csv &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--out&lt;/span&gt; headphones.csv
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is particularly useful when the next step is data analysis in Excel, pandas, or another reporting tool.&lt;/p&gt;

&lt;h3&gt;
  
  
  Keep the first run small
&lt;/h3&gt;

&lt;p&gt;Do not start with hundreds of pages.&lt;/p&gt;

&lt;p&gt;A good initial workflow is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 page
   |
   v
Inspect output
   |
   v
Validate fields
   |
   v
Run 3 pages
   |
   v
Check pagination
   |
   v
Increase collection size if appropriate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This reduces unnecessary requests and helps you identify problems before they affect a large dataset.&lt;/p&gt;




&lt;h2&gt;
  
  
  6. Extracting Detailed Product Information
&lt;/h2&gt;

&lt;p&gt;Search results and product detail pages are not the same thing.&lt;/p&gt;

&lt;p&gt;A search result might contain:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Product title.&lt;/li&gt;
&lt;li&gt;ASIN.&lt;/li&gt;
&lt;li&gt;Current price.&lt;/li&gt;
&lt;li&gt;Rating.&lt;/li&gt;
&lt;li&gt;Review count.&lt;/li&gt;
&lt;li&gt;Product image.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A product page can provide additional fields, such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Brand.&lt;/li&gt;
&lt;li&gt;Seller.&lt;/li&gt;
&lt;li&gt;Product description bullets.&lt;/li&gt;
&lt;li&gt;Availability.&lt;/li&gt;
&lt;li&gt;Images.&lt;/li&gt;
&lt;li&gt;Variation information.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Amazon scraper provides a separate product mode.&lt;/p&gt;

&lt;p&gt;Example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 playwright_scraper.py &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--mode&lt;/span&gt; product &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--url&lt;/span&gt; &lt;span class="s2"&gt;"https://www.amazon.com/dp/B07K5214NZ"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replace the example ASIN with a product you are authorized to analyze.&lt;/p&gt;

&lt;p&gt;This distinction matters when designing a data pipeline.&lt;/p&gt;

&lt;p&gt;For example, if your goal is to monitor the market and compare hundreds of products, collecting complete product pages for every item may be unnecessarily expensive.&lt;/p&gt;

&lt;p&gt;A better approach is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Collect search listings.&lt;/li&gt;
&lt;li&gt;Identify products of interest.&lt;/li&gt;
&lt;li&gt;Fetch details only for selected products.&lt;/li&gt;
&lt;li&gt;Store the additional fields separately.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;This is a practical way to reduce processing time and resource consumption.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Understanding the JSON Output
&lt;/h2&gt;

&lt;p&gt;Structured output is one of the biggest advantages of using a prepared scraper.&lt;/p&gt;

&lt;p&gt;Instead of receiving a raw HTML document and writing another parser, you receive records with defined fields.&lt;/p&gt;

&lt;p&gt;A simplified product record may look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"amazon.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"sku"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"B07K5214NZ"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Example Wireless Headphones"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"price"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;49.99&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"currency"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"USD"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"rating"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;4.5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"review_count"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1250&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"in_stock"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"https://www.amazon.com/dp/B07K5214NZ"&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is an illustrative example of the data structure, not a copied live result.&lt;/p&gt;

&lt;p&gt;The actual repository documents its complete schema and provides sample output files.&lt;/p&gt;

&lt;p&gt;Some fields are specific to individual extraction modes.&lt;/p&gt;

&lt;p&gt;For example, brand information is available in the product mode, while it may be null in listing mode because the search result itself does not expose a separate brand field.&lt;/p&gt;

&lt;p&gt;That is an important distinction.&lt;/p&gt;

&lt;p&gt;A missing field does not necessarily indicate a parser failure.&lt;/p&gt;

&lt;p&gt;Sometimes the source page simply does not provide that information.&lt;/p&gt;

&lt;h3&gt;
  
  
  Load the output with Python
&lt;/h3&gt;

&lt;p&gt;Create a separate analysis script:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;

&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;amazon_products.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nb"&gt;file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

    &lt;span class="n"&gt;products&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;file&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Products collected:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;currency&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is enough to begin building a basic price comparison application.&lt;/p&gt;

&lt;h3&gt;
  
  
  Calculate average product price
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;prices&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;prices&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

    &lt;span class="n"&gt;average_price&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prices&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prices&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Average price: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;average_price&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;No valid prices found&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notice the null check.&lt;/p&gt;

&lt;p&gt;Real-world scraping data is rarely perfectly complete.&lt;/p&gt;

&lt;p&gt;Your application should always handle missing values instead of assuming every product contains every field.&lt;/p&gt;




&lt;h2&gt;
  
  
  8. Validate Data Before Using It
&lt;/h2&gt;

&lt;p&gt;A successful process exit does not automatically mean that the dataset is suitable for your application.&lt;/p&gt;

&lt;p&gt;You should validate the output.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;validate_product&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;

    &lt;span class="n"&gt;required_fields&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;sku&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;title&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;required_fields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;

            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;

    &lt;span class="n"&gt;price&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;price&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;price&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;price&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;


&lt;span class="n"&gt;valid_products&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="n"&gt;product&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;product&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;validate_product&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;product&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Valid records: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;valid_products&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Rejected records: &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;valid_products&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a production pipeline, I would also validate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Duplicate product identifiers.&lt;/li&gt;
&lt;li&gt;Currency consistency.&lt;/li&gt;
&lt;li&gt;URL validity.&lt;/li&gt;
&lt;li&gt;Price types.&lt;/li&gt;
&lt;li&gt;Timestamp formats.&lt;/li&gt;
&lt;li&gt;Unexpected schema changes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Data validation should happen before records enter your main database.&lt;/p&gt;

&lt;p&gt;Otherwise, a small extraction problem can become a much larger analytics problem.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. The Most Important File: Run Metadata
&lt;/h2&gt;

&lt;p&gt;One of the most useful design decisions in the 2scraper collection is the use of metadata sidecar files.&lt;/p&gt;

&lt;p&gt;A successful extraction can generate two files:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;amazon_products.json
amazon_products.meta.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first contains the extracted records.&lt;/p&gt;

&lt;p&gt;The second describes how the run finished.&lt;/p&gt;

&lt;p&gt;For example, metadata can contain:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"source"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"amazon.com"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"mode"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"listing"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"complete"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"stop_reason"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"completed"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pages_requested"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pages_completed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"pages_failed"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[],&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"records"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;77&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a simplified example of the documented metadata structure.&lt;/p&gt;

&lt;p&gt;Why is this important?&lt;/p&gt;

&lt;p&gt;Imagine your application runs every morning.&lt;/p&gt;

&lt;p&gt;On Monday, it collects 500 products.&lt;/p&gt;

&lt;p&gt;On Tuesday, the website changes its markup and the scraper collects only 12 products.&lt;/p&gt;

&lt;p&gt;If your automation checks only whether a JSON file exists, it may assume that Tuesday's run was successful.&lt;/p&gt;

&lt;p&gt;A metadata sidecar gives your application additional information about the extraction process.&lt;/p&gt;

&lt;p&gt;You can distinguish between:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Complete runs.&lt;/li&gt;
&lt;li&gt;Partial runs.&lt;/li&gt;
&lt;li&gt;Failed runs.&lt;/li&gt;
&lt;li&gt;Blocked sessions.&lt;/li&gt;
&lt;li&gt;Genuine zero-result searches.&lt;/li&gt;
&lt;li&gt;Pages that could not be parsed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The repositories also use a separate last-attempt metadata file for attempts that do not produce a normal result file.&lt;/p&gt;

&lt;p&gt;This helps preserve the last successful dataset instead of replacing it with an empty file after a failed run.&lt;/p&gt;

&lt;p&gt;For automated data pipelines, this is extremely valuable.&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Understanding Exit Codes
&lt;/h2&gt;

&lt;p&gt;The collection uses a consistent exit-code convention across its scrapers.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Exit code&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;Complete&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Application crash&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Invalid command usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Blocked&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;Genuine zero results&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Content could not be obtained&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;Partial result&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These distinctions are important.&lt;/p&gt;

&lt;p&gt;Consider the following cases:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exit code 4&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The page loaded correctly, but no products matched the search.&lt;/p&gt;

&lt;p&gt;This may be a valid result.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exit code 3&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The website refused the session before the scraper could collect the requested data.&lt;/p&gt;

&lt;p&gt;This is not equivalent to an empty search.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exit code 5&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The page was not successfully fetched, a remote service failed, or the expected content could not be obtained.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exit code 6&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Some data was collected, but the run did not finish cleanly.&lt;/p&gt;

&lt;p&gt;A production application should handle these cases differently.&lt;/p&gt;

&lt;p&gt;For example, a genuine empty search might update a database, while a failed extraction should preserve the previous successful data.&lt;/p&gt;




&lt;h2&gt;
  
  
  11. Automating a Scraper from Another Python Application
&lt;/h2&gt;

&lt;p&gt;You do not have to run the scraper manually.&lt;/p&gt;

&lt;p&gt;A useful approach is to execute its command-line interface from a Python application.&lt;/p&gt;

&lt;p&gt;The following example demonstrates how to run a scraper safely using &lt;code&gt;subprocess&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_scraper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;

    &lt;span class="n"&gt;command&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;python3&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;playwright_scraper.py&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--url&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;--pages&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;command&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;capture_output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Exit code:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;returncode&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Output:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stdout&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Errors:&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;returncode&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;__main__&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

    &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;run_scraper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://www.amazon.com/s?k=wireless+headphones&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Extraction completed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;code&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Partial extraction&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Extraction needs investigation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Using a list of command arguments instead of building a shell command string is preferable.&lt;/p&gt;

&lt;p&gt;It avoids shell interpolation issues and makes it easier to handle URLs containing special characters.&lt;/p&gt;

&lt;h3&gt;
  
  
  Add scheduled execution
&lt;/h3&gt;

&lt;p&gt;For a recurring data collection job, you can use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Linux cron.&lt;/li&gt;
&lt;li&gt;Windows Task Scheduler.&lt;/li&gt;
&lt;li&gt;GitHub Actions.&lt;/li&gt;
&lt;li&gt;Docker.&lt;/li&gt;
&lt;li&gt;A dedicated job queue.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For example, a daily Linux cron entry might look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0 6 * * * cd /opt/amazon-scraper &amp;amp;&amp;amp; /opt/amazon-scraper/.venv/bin/python playwright_scraper.py --url "https://www.amazon.com/s?k=wireless+headphones" --pages 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This runs the collection every day at 06:00.&lt;/p&gt;

&lt;p&gt;For a production deployment, add logging, monitoring, metadata validation, and alerting around the command.&lt;/p&gt;




&lt;h2&gt;
  
  
  12. Comparing Two Scraping Runs
&lt;/h2&gt;

&lt;p&gt;Collecting data once is useful.&lt;/p&gt;

&lt;p&gt;Collecting it repeatedly is where interesting applications begin.&lt;/p&gt;

&lt;p&gt;Imagine that you run a product scraper every day.&lt;/p&gt;

&lt;p&gt;You want to know:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which products have changed prices?&lt;/li&gt;
&lt;li&gt;Which products disappeared?&lt;/li&gt;
&lt;li&gt;Which products are new?&lt;/li&gt;
&lt;li&gt;Which products changed availability?&lt;/li&gt;
&lt;li&gt;Which products received new ratings?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Amazon repository includes &lt;code&gt;diff_runs.py&lt;/code&gt; for comparing scraping runs.&lt;/p&gt;

&lt;p&gt;Example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 diff_runs.py &lt;span class="se"&gt;\&lt;/span&gt;
  previous.json &lt;span class="se"&gt;\&lt;/span&gt;
  current.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact supported options are available through:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python3 diff_runs.py &lt;span class="nt"&gt;--help&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The comparison tool is especially useful for price monitoring because it helps identify changes without requiring you to compare two large JSON files manually.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why stable identifiers matter
&lt;/h3&gt;

&lt;p&gt;Product titles are not reliable identifiers.&lt;/p&gt;

&lt;p&gt;A title can change because of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Updated descriptions.&lt;/li&gt;
&lt;li&gt;Formatting changes.&lt;/li&gt;
&lt;li&gt;Marketing text.&lt;/li&gt;
&lt;li&gt;Different language versions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Amazon scraper uses ASIN as the product identifier.&lt;/p&gt;

&lt;p&gt;This allows your application to compare records across different runs.&lt;/p&gt;

&lt;p&gt;The same principle applies to other platforms.&lt;/p&gt;

&lt;p&gt;Whenever possible, use the target website's stable identifier rather than a display name.&lt;/p&gt;




&lt;h2&gt;
  
  
  13. Choosing the Right Execution Engine
&lt;/h2&gt;

&lt;p&gt;One of the interesting aspects of 2scraper is that repositories can provide several execution paths.&lt;/p&gt;

&lt;p&gt;The available options depend on the individual project.&lt;/p&gt;

&lt;h3&gt;
  
  
  Playwright
&lt;/h3&gt;

&lt;p&gt;Playwright is generally the recommended starting point.&lt;/p&gt;

&lt;p&gt;It provides:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Browser automation.&lt;/li&gt;
&lt;li&gt;Page navigation.&lt;/li&gt;
&lt;li&gt;JavaScript execution.&lt;/li&gt;
&lt;li&gt;Dynamic-content handling.&lt;/li&gt;
&lt;li&gt;Chromium integration.&lt;/li&gt;
&lt;li&gt;Remote browser connections through CDP.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is particularly useful when the data is rendered dynamically.&lt;/p&gt;

&lt;h3&gt;
  
  
  Selenium
&lt;/h3&gt;

&lt;p&gt;Selenium is a familiar choice for developers who already maintain browser automation projects.&lt;/p&gt;

&lt;p&gt;Some repositories provide a Selenium implementation with the same site-specific extraction logic.&lt;/p&gt;

&lt;p&gt;However, the supported features may differ.&lt;/p&gt;

&lt;h3&gt;
  
  
  Puppeteer
&lt;/h3&gt;

&lt;p&gt;Puppeteer is another browser automation option.&lt;/p&gt;

&lt;p&gt;Depending on the repository, the implementation may use Node Puppeteer or pyppeteer.&lt;/p&gt;

&lt;p&gt;Check the repository's README before selecting this engine.&lt;/p&gt;

&lt;h3&gt;
  
  
  Direct HTTP requests
&lt;/h3&gt;

&lt;p&gt;Sometimes a browser is unnecessary.&lt;/p&gt;

&lt;p&gt;If the target exposes the required information in the initial HTML or through a documented public endpoint, a direct HTTP implementation can be significantly lighter.&lt;/p&gt;

&lt;p&gt;You avoid the overhead of launching Chromium.&lt;/p&gt;

&lt;p&gt;For example, a simple request-based workflow may look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;

&lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://example.com/public-page&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;User-Agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;MyResearchBot/1.0 (contact: dev@example.com)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;

&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is only a generic HTTP example, not an implementation of any particular 2scraper repository.&lt;/p&gt;

&lt;p&gt;Use the browserless path only when the site's public response actually contains the information you need and your use is permitted.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which engine should you choose?
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Requirement&lt;/th&gt;
&lt;th&gt;Recommended starting point&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Dynamic JavaScript pages&lt;/td&gt;
&lt;td&gt;Playwright&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Existing Selenium infrastructure&lt;/td&gt;
&lt;td&gt;Selenium&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Existing Puppeteer application&lt;/td&gt;
&lt;td&gt;Puppeteer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Static HTML or accessible public endpoint&lt;/td&gt;
&lt;td&gt;Direct HTTP&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Managed remote browser&lt;/td&gt;
&lt;td&gt;Playwright or Puppeteer over CDP&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The most important rule is to use the execution mode tested and documented by the repository.&lt;/p&gt;




&lt;h2&gt;
  
  
  14. Understanding Proxies, CAPTCHA Processing, and Hosted Browsers
&lt;/h2&gt;

&lt;p&gt;Some websites treat automated requests differently depending on the connection and browser environment.&lt;/p&gt;

&lt;p&gt;This is where the optional infrastructure documented by individual 2scraper repositories becomes relevant.&lt;/p&gt;

&lt;p&gt;The collection integrates with several 2Captcha products where a specific target requires them.&lt;/p&gt;

&lt;p&gt;These products serve different purposes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Residential proxies
&lt;/h3&gt;

&lt;p&gt;A residential proxy provides an exit IP associated with a residential network.&lt;/p&gt;

&lt;p&gt;This can be relevant when a website refuses datacenter IP ranges.&lt;/p&gt;

&lt;p&gt;A scraper that works from a home connection may behave differently when executed on a cloud server.&lt;/p&gt;

&lt;p&gt;However, a proxy is not a guarantee of access.&lt;/p&gt;

&lt;p&gt;It also does not authorize activity that would otherwise violate a website's terms or applicable law.&lt;/p&gt;

&lt;h3&gt;
  
  
  CAPTCHA processing
&lt;/h3&gt;

&lt;p&gt;Some sites present CAPTCHA challenges during normal page access.&lt;/p&gt;

&lt;p&gt;Where a repository documents a CAPTCHA integration, it may use an external solving service.&lt;/p&gt;

&lt;p&gt;Such processing should be explicitly enabled and capped.&lt;/p&gt;

&lt;p&gt;Never assume that every CAPTCHA needs to be solved.&lt;/p&gt;

&lt;p&gt;Some browser challenges clear automatically, while other failures may be caused by a rate limit, a network error, or a page-layout change.&lt;/p&gt;

&lt;p&gt;Buying a CAPTCHA solution for every failed request is a poor debugging strategy.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scraping Browser API
&lt;/h3&gt;

&lt;p&gt;A hosted browser provides a remote browser session that can be controlled through the Chrome DevTools Protocol.&lt;/p&gt;

&lt;p&gt;Instead of managing Chromium yourself, your application connects to the remote browser.&lt;/p&gt;

&lt;p&gt;This can be useful when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your server cannot maintain browser infrastructure.&lt;/li&gt;
&lt;li&gt;You need a managed browser environment.&lt;/li&gt;
&lt;li&gt;A repository documents support for a remote CDP endpoint.&lt;/li&gt;
&lt;li&gt;You want to centralize browser management.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The exact setup depends on the repository and its environment variables.&lt;/p&gt;

&lt;p&gt;One important rule from the 2scraper documentation is that a hosted CDP browser should not be combined with local proxy, user-agent, or fingerprint configuration.&lt;/p&gt;

&lt;p&gt;Choose one coherent browser configuration rather than mixing incompatible settings.&lt;/p&gt;




&lt;h2&gt;
  
  
  15. Building a Reliable Production Pipeline
&lt;/h2&gt;

&lt;p&gt;A scraping script is not yet a complete data pipeline.&lt;/p&gt;

&lt;p&gt;For a production application, I recommend separating the system into several components.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                 Scheduler
                     |
                     v
               Scraping Job
                     |
                     v
                2scraper
                     |
                     v
              JSON / CSV Output
                     |
                     v
              Metadata Validation
                     |
                     v
               Data Validation
                     |
                     v
                Database
                     |
                     v
              Analytics / Reports
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each stage has a clear responsibility.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scheduler
&lt;/h3&gt;

&lt;p&gt;Starts extraction jobs at predefined intervals.&lt;/p&gt;

&lt;h3&gt;
  
  
  Scraping job
&lt;/h3&gt;

&lt;p&gt;Executes the appropriate repository and records its exit status.&lt;/p&gt;

&lt;h3&gt;
  
  
  Output storage
&lt;/h3&gt;

&lt;p&gt;Stores the extracted JSON or CSV data.&lt;/p&gt;

&lt;h3&gt;
  
  
  Metadata validation
&lt;/h3&gt;

&lt;p&gt;Checks whether the run completed successfully.&lt;/p&gt;

&lt;h3&gt;
  
  
  Data validation
&lt;/h3&gt;

&lt;p&gt;Checks the fields, identifiers, prices, and record counts.&lt;/p&gt;

&lt;h3&gt;
  
  
  Database
&lt;/h3&gt;

&lt;p&gt;Stores normalized records for historical comparison.&lt;/p&gt;

&lt;h3&gt;
  
  
  Analytics
&lt;/h3&gt;

&lt;p&gt;Calculates price changes, availability trends, and other useful information.&lt;/p&gt;

&lt;p&gt;This architecture makes the scraper replaceable.&lt;/p&gt;

&lt;p&gt;If a website changes its layout, you can update the extraction component without rewriting the analytics layer.&lt;/p&gt;




&lt;h2&gt;
  
  
  16. Testing and Maintenance
&lt;/h2&gt;

&lt;p&gt;Websites change frequently.&lt;/p&gt;

&lt;p&gt;A scraper should therefore be treated as software that needs testing, not as a script that can be forgotten after the first successful run.&lt;/p&gt;

&lt;p&gt;The 2scraper repositories include offline test or smoke-test entry points, depending on the project.&lt;/p&gt;

&lt;p&gt;These tests help verify parsing against saved, sanitized fixtures without requiring a live website request.&lt;/p&gt;

&lt;p&gt;Some repositories also include scheduled GitHub Actions canaries that test the scraper against the live website.&lt;/p&gt;

&lt;p&gt;A green canary is useful evidence that the last scheduled check succeeded, but it is not a guarantee that every query, marketplace, or page type works.&lt;/p&gt;

&lt;h3&gt;
  
  
  Recommended maintenance routine
&lt;/h3&gt;

&lt;p&gt;After updating a repository:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git pull

python3 smoke_test.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the repository uses pytest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pytest tests/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check the README for the actual test command.&lt;/p&gt;

&lt;p&gt;Then run a small live test using a limited number of pages.&lt;/p&gt;

&lt;p&gt;Finally, compare the output schema against the version expected by your application.&lt;/p&gt;

&lt;h3&gt;
  
  
  Pin dependencies
&lt;/h3&gt;

&lt;p&gt;For production deployments, avoid relying on an unchanging set of loosely specified dependencies.&lt;/p&gt;

&lt;p&gt;Use a tested lock file when the repository provides one.&lt;/p&gt;

&lt;p&gt;This makes builds more reproducible and reduces the risk of an unrelated package update breaking your scraping environment.&lt;/p&gt;




&lt;h2&gt;
  
  
  17. A Few Practical Mistakes to Avoid
&lt;/h2&gt;

&lt;p&gt;After working with web scraping systems, several recurring mistakes become obvious.&lt;/p&gt;

&lt;h3&gt;
  
  
  Mistake 1: Assuming every run is complete
&lt;/h3&gt;

&lt;p&gt;Always inspect exit codes and metadata.&lt;/p&gt;

&lt;p&gt;A partial result is not the same as a complete dataset.&lt;/p&gt;

&lt;h3&gt;
  
  
  Mistake 2: Collecting too much data too early
&lt;/h3&gt;

&lt;p&gt;Start with a small sample.&lt;/p&gt;

&lt;p&gt;Validate the extraction before increasing the number of pages.&lt;/p&gt;

&lt;h3&gt;
  
  
  Mistake 3: Treating missing values as zero
&lt;/h3&gt;

&lt;p&gt;A missing price is not a price of zero.&lt;/p&gt;

&lt;p&gt;Keep null values distinct from real numeric values.&lt;/p&gt;

&lt;h3&gt;
  
  
  Mistake 4: Ignoring pagination
&lt;/h3&gt;

&lt;p&gt;Some pages load additional products only after scrolling.&lt;/p&gt;

&lt;p&gt;Check the actual number of records and whether all requested pages were processed.&lt;/p&gt;

&lt;h3&gt;
  
  
  Mistake 5: Using the wrong browser configuration
&lt;/h3&gt;

&lt;p&gt;Do not combine incompatible proxy, fingerprint, and hosted-browser settings.&lt;/p&gt;

&lt;p&gt;Follow the specific repository instructions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Mistake 6: Assuming a CAPTCHA is the reason for every failure
&lt;/h3&gt;

&lt;p&gt;A timeout, an HTTP 429 response, an invalid URL, and a page-layout change require different fixes.&lt;/p&gt;

&lt;p&gt;Inspect the stop reason before changing infrastructure.&lt;/p&gt;

&lt;h3&gt;
  
  
  Mistake 7: Ignoring data provenance
&lt;/h3&gt;

&lt;p&gt;Always record where the data came from and when it was collected.&lt;/p&gt;

&lt;p&gt;Without timestamps and source identifiers, historical analysis quickly becomes unreliable.&lt;/p&gt;




&lt;h2&gt;
  
  
  18. Responsible Web Scraping
&lt;/h2&gt;

&lt;p&gt;Before deploying any scraping workflow, consider the website's terms, applicable laws,&lt;/p&gt;

</description>
      <category>2scraper</category>
      <category>scraper</category>
    </item>
    <item>
      <title>2scraper: A Collection of Open-Source Web Scraping Tools for Almost Any Website</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Tue, 03 Nov 2026 19:34:08 +0000</pubDate>
      <link>https://dev.to/gresiuseo/2scraper-a-collection-of-open-source-web-scraping-tools-for-almost-any-website-305f</link>
      <guid>https://dev.to/gresiuseo/2scraper-a-collection-of-open-source-web-scraping-tools-for-almost-any-website-305f</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feegr599m5g5g156a48g9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Feegr599m5g5g156a48g9.png" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. What Is 2scraper?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2scraper&lt;/strong&gt; is an open-source organization on GitHub that develops ready-to-use web scraping tools for collecting publicly available information from popular websites.&lt;/p&gt;

&lt;p&gt;Official GitHub organization: &lt;a href="https://github.com/2scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Instead of building a custom scraper from scratch for every website, developers can choose an existing project, install it, and start collecting structured data.&lt;/p&gt;

&lt;p&gt;The organization maintains individual Python-based scraping projects, each designed for a particular website or platform.&lt;/p&gt;

&lt;p&gt;The collected information can be exported into structured formats such as JSON and CSV, making it easy to analyze, store, process, or integrate into other applications.&lt;/p&gt;

&lt;p&gt;The collection covers a wide range of industries, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;E-commerce and online marketplaces.&lt;/li&gt;
&lt;li&gt;Social media and video platforms.&lt;/li&gt;
&lt;li&gt;Financial markets and cryptocurrency.&lt;/li&gt;
&lt;li&gt;Real estate and classified advertisements.&lt;/li&gt;
&lt;li&gt;Travel and accommodation.&lt;/li&gt;
&lt;li&gt;Job boards and recruitment.&lt;/li&gt;
&lt;li&gt;Publishing platforms and online communities.&lt;/li&gt;
&lt;li&gt;Sports statistics and player information.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Official repository: &lt;a href="https://github.com/2scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. How Does 2scraper Work?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The main idea behind 2scraper is simple: one website, one dedicated scraper.&lt;/p&gt;

&lt;p&gt;Each repository contains the code and documentation required to collect information from a particular website.&lt;/p&gt;

&lt;p&gt;The general workflow looks like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Choose a website from the 2scraper directory.&lt;/li&gt;
&lt;li&gt;Open the corresponding GitHub repository.&lt;/li&gt;
&lt;li&gt;Read its installation instructions.&lt;/li&gt;
&lt;li&gt;Download the source code.&lt;/li&gt;
&lt;li&gt;Install the required dependencies.&lt;/li&gt;
&lt;li&gt;Configure optional settings if necessary.&lt;/li&gt;
&lt;li&gt;Run the scraper with a target URL or search query.&lt;/li&gt;
&lt;li&gt;Receive structured data in JSON or CSV format.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Depending on the target website, the scraper may use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Playwright for browser automation.&lt;/li&gt;
&lt;li&gt;Selenium for browser control.&lt;/li&gt;
&lt;li&gt;Puppeteer or pyppeteer for browser automation.&lt;/li&gt;
&lt;li&gt;The website's own public or accessible data endpoints.&lt;/li&gt;
&lt;li&gt;A remote browser through the 2Captcha Scraping Browser API.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Not every scraper supports every execution method. The individual repository's README explains which engines and features are available.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. What Can You Scrape with 2scraper?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One of the biggest advantages of this organization is its variety of supported websites.&lt;/p&gt;

&lt;p&gt;Below are some of the most interesting categories and projects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3.1. E-commerce and Online Shopping&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Amazon Scraper&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/amazon-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/amazon-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Collect information from Amazon marketplaces, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Product names and ASINs.&lt;/li&gt;
&lt;li&gt;Prices and discounts.&lt;/li&gt;
&lt;li&gt;Customer ratings.&lt;/li&gt;
&lt;li&gt;Review counts.&lt;/li&gt;
&lt;li&gt;Product images.&lt;/li&gt;
&lt;li&gt;Product descriptions.&lt;/li&gt;
&lt;li&gt;Publicly displayed customer reviews.&lt;/li&gt;
&lt;li&gt;Best Seller rankings.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Useful for product research, competitor analysis, and price monitoring.&lt;/p&gt;

&lt;p&gt;StockX Scraper&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/stockx-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/stockx-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Collect sneaker and streetwear market information:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Product listings.&lt;/li&gt;
&lt;li&gt;Asking prices.&lt;/li&gt;
&lt;li&gt;Bids.&lt;/li&gt;
&lt;li&gt;Last sale prices.&lt;/li&gt;
&lt;li&gt;Market statistics.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Useful for resale market research and tracking price movements.&lt;/p&gt;

&lt;p&gt;Etsy Scraper&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/etsy-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/etsy-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Collect information about products, shops, and listings on Etsy.&lt;/p&gt;

&lt;p&gt;Farfetch Scraper&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/farfetch-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/farfetch-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Collect fashion product listings, prices, and product information.&lt;/p&gt;

&lt;p&gt;Other supported shopping platforms include SHEIN, MediaMarkt, Home Depot, Rakuten, LG, Woolworths, and Catawiki.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3.2. Social Media and Video Platforms&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;YouTube Scraper&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/youtube-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/youtube-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Collect publicly accessible YouTube information, including:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Video metadata.&lt;/li&gt;
&lt;li&gt;Comments.&lt;/li&gt;
&lt;li&gt;Comment replies.&lt;/li&gt;
&lt;li&gt;Search results.&lt;/li&gt;
&lt;li&gt;Video information and engagement data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Useful for content research, audience analysis, and monitoring discussions.&lt;/p&gt;

&lt;p&gt;TikTok Scrapers&lt;/p&gt;

&lt;p&gt;The organization provides separate tools for different TikTok data sources:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Public profiles.&lt;/li&gt;
&lt;li&gt;Public videos.&lt;/li&gt;
&lt;li&gt;TikTok Shop.&lt;/li&gt;
&lt;li&gt;EU TikTok Ad Library.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These tools can collect information such as follower counts, video engagement, captions, hashtags, product information, and advertising data, depending on the scraper.&lt;/p&gt;

&lt;p&gt;Instagram Scraper&lt;/p&gt;

&lt;p&gt;Collect publicly accessible information about Instagram profiles and posts, including followers, likes, comments, captions, and media URLs.&lt;/p&gt;

&lt;p&gt;Facebook Scrapers&lt;/p&gt;

&lt;p&gt;Separate projects cover Facebook Pages, Facebook Marketplace, and the Meta Ad Library.&lt;/p&gt;

&lt;p&gt;These are useful for researching businesses, public advertisements, and marketplace listings.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3.3. Finance and Cryptocurrency&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;2scraper also includes tools for collecting financial and market information.&lt;/p&gt;

&lt;p&gt;Examples include:&lt;/p&gt;

&lt;p&gt;Binance: P2P advertisements, copy-trading portfolios, and announcements.&lt;/p&gt;

&lt;p&gt;Google Finance: Quotes, financial information, analyst ratings, foreign exchange, and historical market data.&lt;/p&gt;

&lt;p&gt;Polymarket: Prediction market prices, order books, and token information.&lt;/p&gt;

&lt;p&gt;OpenSea: NFT floor prices, offers, sales history, and rankings.&lt;/p&gt;

&lt;p&gt;Screener: Indian stock market screening data.&lt;/p&gt;

&lt;p&gt;TipRanks: Analyst ratings, Smart Scores, and price targets.&lt;/p&gt;

&lt;p&gt;These tools can be useful for market research, financial data collection, and building analytical dashboards.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3.4. Real Estate, Classifieds, and Travel&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Other projects focus on collecting listings and property information.&lt;/p&gt;

&lt;p&gt;Examples include:&lt;/p&gt;

&lt;p&gt;Facebook Marketplace: Product listings, prices, categories, and locations.&lt;/p&gt;

&lt;p&gt;Avito: Classified advertisements, item pages, and seller profiles.&lt;/p&gt;

&lt;p&gt;Craigslist: Classified listings and advertisements.&lt;/p&gt;

&lt;p&gt;dubizzle: Cars, property, and job advertisements in the UAE.&lt;/p&gt;

&lt;p&gt;Vrbo: Vacation rental listings and property information.&lt;/p&gt;

&lt;p&gt;Skyscanner: Flight itineraries, prices, airlines, and departure times.&lt;/p&gt;

&lt;p&gt;Spinny: Used-car listings and vehicle information.&lt;/p&gt;

&lt;p&gt;Autotrader: Used-car listings, prices, VINs, and dealer information.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3.5. Jobs and Business Information&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For recruitment and business research, 2scraper provides projects for:&lt;/p&gt;

&lt;p&gt;Wellfound: Startup job listings, salaries, and equity ranges.&lt;/p&gt;

&lt;p&gt;Just Join IT: IT job advertisements, salaries, and required skills.&lt;/p&gt;

&lt;p&gt;Mercor: Contract opportunities, rates, and eligibility.&lt;/p&gt;

&lt;p&gt;Better Business Bureau: Business profiles, ratings, and complaints.&lt;/p&gt;

&lt;p&gt;G2: Software reviews, ratings, and pricing information.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3.6. Publishing, Communities, and Other Data&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Additional projects include:&lt;/p&gt;

&lt;p&gt;Medium: Articles, authors, publications, and full article text.&lt;/p&gt;

&lt;p&gt;Goodreads: Books, author pages, lists, and reviews.&lt;/p&gt;

&lt;p&gt;Hacker News: Stories, comments, and user profiles.&lt;/p&gt;

&lt;p&gt;Quora: Public answers, profiles, and topics.&lt;/p&gt;

&lt;p&gt;Transfermarkt: Football player profiles, market values, squads, and transfers.&lt;/p&gt;

&lt;p&gt;The complete and continuously maintained directory is available on the official GitHub organization page.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. How to Find the Right Scraper&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You do not need to download the entire 2scraper organization.&lt;/p&gt;

&lt;p&gt;Each scraper is an independent project.&lt;/p&gt;

&lt;p&gt;Follow these steps:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 1. Open the official organization:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 2. Browse the repository directory.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 3. Find the website you want to collect information from.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Amazon: amazon-scraper.&lt;/li&gt;
&lt;li&gt;YouTube: youtube-scraper.&lt;/li&gt;
&lt;li&gt;StockX: stockx-scraper.&lt;/li&gt;
&lt;li&gt;Transfermarkt: transfermarkt-scraper.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Step 4. Open the selected repository.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 5. Read its README file.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Pay particular attention to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Supported website pages.&lt;/li&gt;
&lt;li&gt;Available data fields.&lt;/li&gt;
&lt;li&gt;Installation requirements.&lt;/li&gt;
&lt;li&gt;Browser engine support.&lt;/li&gt;
&lt;li&gt;Proxy requirements.&lt;/li&gt;
&lt;li&gt;CAPTCHA handling.&lt;/li&gt;
&lt;li&gt;Example commands.&lt;/li&gt;
&lt;li&gt;Sample output.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Important: Not every scraper has identical capabilities. Always follow the documentation of the specific project you choose.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. How to Install a 2scraper Project&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Most projects follow a similar installation process.&lt;/p&gt;

&lt;p&gt;The following example demonstrates how to install the Amazon scraper on a Windows, Linux, or macOS computer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 1. Install Python&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Download Python:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.python.org/downloads/" rel="noopener noreferrer"&gt;https://www.python.org/downloads/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Check the installation:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;python --version&lt;/p&gt;

&lt;p&gt;Python 3.9 or newer is the general family baseline, although individual projects may require a newer version.&lt;/p&gt;

&lt;p&gt;*&lt;em&gt;Step 2. Install Git&lt;br&gt;
*&lt;/em&gt;&lt;br&gt;
Download Git:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://git-scm.com/downloads" rel="noopener noreferrer"&gt;https://git-scm.com/downloads&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Check:&lt;/p&gt;

&lt;p&gt;git --version&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 3. Download a Scraper&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For example, to download the Amazon scraper:&lt;/p&gt;

&lt;p&gt;git clone &lt;a href="https://github.com/2scraper/amazon-scraper.git" rel="noopener noreferrer"&gt;https://github.com/2scraper/amazon-scraper.git&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Open its directory:&lt;/p&gt;

&lt;p&gt;cd amazon-scraper&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 4. Create a Virtual Environment&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;On Windows:&lt;/p&gt;

&lt;p&gt;python -m venv .venv&lt;br&gt;
.venv\Scripts\activate&lt;/p&gt;

&lt;p&gt;On Linux or macOS:&lt;/p&gt;

&lt;p&gt;python3 -m venv .venv&lt;br&gt;
source .venv/bin/activate&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 5. Install Dependencies&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For projects using Playwright:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;pip install -r requirements-playwright.txt&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Install the Chromium browser:&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;playwright install chromium&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Step 6. Configure the Project&lt;/p&gt;

&lt;p&gt;Check whether the repository contains an .env.example file.&lt;/p&gt;

&lt;p&gt;If it does, copy it to .env and configure any required settings.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;p&gt;cp .env.example .env&lt;/p&gt;

&lt;p&gt;On Windows:&lt;/p&gt;

&lt;p&gt;copy .env.example .env&lt;/p&gt;

&lt;p&gt;Not every project requires an API key or proxy.&lt;/p&gt;

&lt;p&gt;Some scrapers can run immediately after installation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 7. Run the Scraper&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Each project has its own command-line interface.&lt;/p&gt;

&lt;p&gt;For example, the Amazon scraper can be launched using:&lt;/p&gt;

&lt;p&gt;python3 playwright_scraper.py --url "&lt;a href="https://www.amazon.com/s?k=wireless+headphones" rel="noopener noreferrer"&gt;https://www.amazon.com/s?k=wireless+headphones&lt;/a&gt;" --pages 3&lt;/p&gt;

&lt;p&gt;Always check the individual README for the correct command and available parameters.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;JSON and CSV: How the Collected Data Is Stored&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Most 2scraper projects support structured data export.&lt;/p&gt;

&lt;p&gt;JSON Format&lt;/p&gt;

&lt;p&gt;JSON is useful for developers and applications that need structured information.&lt;/p&gt;

&lt;p&gt;Example:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;{&lt;br&gt;
  "product_name": "Wireless Headphones",&lt;br&gt;
  "price": 49.99,&lt;br&gt;
  "currency": "USD",&lt;br&gt;
  "rating": 4.6,&lt;br&gt;
  "review_count": 1250,&lt;br&gt;
  "asin": "B000000000"&lt;br&gt;
}&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;This example illustrates a possible product record, not a guaranteed output schema.&lt;/p&gt;

&lt;p&gt;CSV Format&lt;/p&gt;

&lt;p&gt;CSV is particularly useful for working with spreadsheets.&lt;/p&gt;

&lt;p&gt;You can open CSV files using:&lt;/p&gt;

&lt;p&gt;Microsoft Excel.&lt;/p&gt;

&lt;p&gt;Google Sheets.&lt;/p&gt;

&lt;p&gt;LibreOffice Calc.&lt;/p&gt;

&lt;p&gt;Python pandas.&lt;/p&gt;

&lt;p&gt;Business intelligence software.&lt;/p&gt;

&lt;p&gt;For example, you can collect hundreds of products and compare their prices, ratings, and review counts in one spreadsheet.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Advanced Features and Integrations&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Although many projects can run locally, the 2scraper ecosystem also supports optional services for more demanding scraping tasks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7.1. Playwright, Selenium, and Puppeteer&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;These browser automation frameworks allow scrapers to interact with dynamic websites.&lt;/p&gt;

&lt;p&gt;Playwright is generally the recommended starting point because it receives the most testing across the project family.&lt;/p&gt;

&lt;p&gt;Selenium and Puppeteer are available where documented by individual repositories.&lt;/p&gt;

&lt;p&gt;Use separate Python virtual environments when installing different browser engines, since their dependencies can conflict.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7.2. Proxy Support&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Some websites restrict requests coming from datacenter IP addresses or apply geographic restrictions.&lt;/p&gt;

&lt;p&gt;Residential proxies can provide connections through residential IP addresses in different countries.&lt;/p&gt;

&lt;p&gt;This is useful when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Collecting information from geographically restricted pages.&lt;/li&gt;
&lt;li&gt;Testing regional product availability.&lt;/li&gt;
&lt;li&gt;Running larger research workflows.&lt;/li&gt;
&lt;li&gt;Working with websites that restrict datacenter traffic.&lt;/li&gt;
&lt;li&gt;Proxy support and configuration vary between repositories.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;7.3. CAPTCHA Solving&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Some websites display CAPTCHA challenges before allowing access to their pages.&lt;/p&gt;

&lt;p&gt;The ecosystem supports optional 2Captcha integrations for compatible challenges.&lt;/p&gt;

&lt;p&gt;Depending on the scraper, this may include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CAPTCHA token solving.&lt;/li&gt;
&lt;li&gt;CAPTCHA-aware browser sessions.&lt;/li&gt;
&lt;li&gt;Remote browser execution.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Not every restriction can be solved with CAPTCHA recognition. Sign-in walls, HTTP 503 throttling, and other access restrictions require different approaches.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7.4. 2Captcha Scraping Browser API&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The Scraping Browser API allows a scraper to connect to a remotely hosted browser using Chrome DevTools Protocol (CDP).&lt;/p&gt;

&lt;p&gt;This can be useful when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A local browser is frequently blocked.&lt;/li&gt;
&lt;li&gt;You need a particular geographic exit location.&lt;/li&gt;
&lt;li&gt;You want managed browser infrastructure.&lt;/li&gt;
&lt;li&gt;You need persistent browser profiles.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is an optional paid service, not a mandatory requirement for every project.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7.5. Run-to-Run Data Comparison&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Some repositories include a utility called diff_runs.py.&lt;/p&gt;

&lt;p&gt;This tool compares the results of two scraping sessions.&lt;/p&gt;

&lt;p&gt;It can identify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;New products or listings.&lt;/li&gt;
&lt;li&gt;Removed products.&lt;/li&gt;
&lt;li&gt;Price changes.&lt;/li&gt;
&lt;li&gt;Changes in product information.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is particularly useful for building price-monitoring systems and competitor tracking workflows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Are the 2scraper Projects Free?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The scraper repositories are open-source and distributed under the MIT License.&lt;/p&gt;

&lt;p&gt;This means you can generally:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Download the source code.&lt;/li&gt;
&lt;li&gt;Run projects on your own computer.&lt;/li&gt;
&lt;li&gt;Modify the code.&lt;/li&gt;
&lt;li&gt;Build custom integrations.&lt;/li&gt;
&lt;li&gt;Use the software in commercial projects, subject to the license terms.&lt;/li&gt;
&lt;li&gt;However, some optional services have separate costs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These may include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Residential proxies.&lt;/li&gt;
&lt;li&gt;CAPTCHA-solving services.&lt;/li&gt;
&lt;li&gt;Managed scraping browsers.&lt;/li&gt;
&lt;li&gt;External infrastructure.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The important distinction is that the scraper source code can be free even when optional infrastructure costs money.&lt;/p&gt;

&lt;p&gt;Always check the license file and the requirements of the individual repository before using a project.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;9. Real-World Use Cases&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The 2scraper collection can be useful in many different industries.&lt;/p&gt;

&lt;p&gt;E-commerce Businesses&lt;/p&gt;

&lt;p&gt;Collect product prices from competing online stores, monitor discounts, and identify products with strong customer ratings.&lt;/p&gt;

&lt;p&gt;Market Researchers&lt;/p&gt;

&lt;p&gt;Collect large datasets of public listings, advertisements, reviews, and product information to identify market trends.&lt;/p&gt;

&lt;p&gt;Digital Marketing Agencies&lt;/p&gt;

&lt;p&gt;Research public social media content, advertising campaigns, audience engagement, and competitor activity.&lt;/p&gt;

&lt;p&gt;Developers&lt;/p&gt;

&lt;p&gt;Build applications that require structured information from public websites without writing an entirely new scraping system.&lt;/p&gt;

&lt;p&gt;Investors and Analysts&lt;/p&gt;

&lt;p&gt;Collect publicly available financial information, market statistics, and analyst data for further research.&lt;/p&gt;

&lt;p&gt;Recruitment Agencies&lt;/p&gt;

&lt;p&gt;Monitor job listings, salaries, skills requirements, and recruitment trends across different platforms.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;10. Important Limitations and Responsible Usage&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Web scraping is not a universal solution that guarantees access to every page.&lt;/p&gt;

&lt;p&gt;Keep the following limitations in mind:&lt;/p&gt;

&lt;p&gt;Websites can change their HTML and internal APIs.&lt;/p&gt;

&lt;p&gt;Some pages require authentication.&lt;/p&gt;

&lt;p&gt;Certain websites use anti-bot protection.&lt;/p&gt;

&lt;p&gt;Data availability can vary by country and location.&lt;/p&gt;

&lt;p&gt;Some fields may be missing or incomplete.&lt;/p&gt;

&lt;p&gt;High-volume collection can trigger rate limits.&lt;/p&gt;

&lt;p&gt;Scraped information should be validated before being used in business decisions.&lt;/p&gt;

&lt;p&gt;Always respect the target website's terms, access restrictions, privacy requirements, and applicable laws.&lt;/p&gt;

&lt;p&gt;Only collect information you are authorized to access and use.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;11. Official Links and Resources&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Main GitHub organization:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Amazon Scraper:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/amazon-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/amazon-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;YouTube Scraper:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/youtube-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/youtube-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;StockX Scraper:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/stockx-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/stockx-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Transfermarkt Scraper:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/transfermarkt-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/transfermarkt-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Catawiki Scraper:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/catawiki-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/catawiki-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Medium Scraper:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/2scraper/medium-scraper" rel="noopener noreferrer"&gt;https://github.com/2scraper/medium-scraper&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;2Captcha Scraping Browser documentation:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://2captcha.com/" rel="noopener noreferrer"&gt;https://2captcha.com/&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;2scraper is more than a single web scraping tool. It is a growing collection of specialized open-source projects designed to make public web data easier to collect and use.&lt;/p&gt;

&lt;p&gt;Whether you need Amazon product prices, YouTube comments, StockX market information, job listings, financial data, or public social media information, there may already be a dedicated scraper available.&lt;/p&gt;

&lt;p&gt;The biggest advantage is that you do not have to reinvent the wheel for every website.&lt;/p&gt;

&lt;p&gt;Choose a repository, follow its documentation, install the dependencies, and start collecting structured data.&lt;/p&gt;

&lt;p&gt;For developers, researchers, e-commerce businesses, and data analysts, 2scraper provides a practical starting point for building custom web data collection workflows.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>Recaptcha Auto Solver</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Tue, 03 Oct 2023 12:35:12 +0000</pubDate>
      <link>https://dev.to/gresiuseo/recaptcha-auto-solver-3ega</link>
      <guid>https://dev.to/gresiuseo/recaptcha-auto-solver-3ega</guid>
      <description>&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--O89-kQXl--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f8kxswrtbm3t269lok2r.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--O89-kQXl--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/f8kxswrtbm3t269lok2r.png" alt="Image description" width="640" height="400"&gt;&lt;/a&gt;&lt;br&gt;
&lt;a href="https://chrome.google.com/webstore/detail/recaptcha-solver-auto-cap/infdcenbdoibcacogknkjleclhnjdmfh"&gt;https://chrome.google.com/webstore/detail/recaptcha-solver-auto-cap/infdcenbdoibcacogknkjleclhnjdmfh&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a security measure used by websites to prevent automated bots from accessing their services. CAPTCHAs are designed to be difficult for computers to solve but easy for humans. However, sometimes CAPTCHAs can be too difficult for humans to solve, and that's where CAPTCHA auto solver extensions come in. In this article, we will focus on the reCAPTCHA Solver Auto Captcha Bypass Chrome extension.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What is reCAPTCHA Solver Auto Captcha Bypass?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;reCAPTCHA Solver Auto Captcha Bypass is a Chrome extension that helps users solve CAPTCHAs automatically. The extension can bypass various types of CAPTCHAs, including reCAPTCHA v2, reCAPTCHA v3, and reCAPTCHA Enterprise. The extension works by sending a request to the solving service to recognize the CAPTCHA. To use the extension, you need to create an account on the solving service and obtain an API key. Without an API key, the extension will not work.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How does reCAPTCHA Solver Auto Captcha Bypass work?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;To use reCAPTCHA Solver Auto Captcha Bypass, you need to follow these steps:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Add the extension to your Chrome browser.&lt;/li&gt;
&lt;li&gt;Register on the solving service website.&lt;/li&gt;
&lt;li&gt;Add funds to your account balance.&lt;/li&gt;
&lt;li&gt;Enter your API key in the extension.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The extension will only deduct funds from your account balance for automatically solved CAPTCHAs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Other CAPTCHA Auto Solver Extensions&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Apart from reCAPTCHA Solver Auto Captcha Bypass, there are other CAPTCHA auto solver extensions available. Here are some of them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;Captcha Solver: Auto Recognition and Bypass**: This official 2Captcha solver browser extension finds CAPTCHAs on a page and bypasses them automatically. The price is $0.03 cents per 1 CAPTCHA bypass.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;Buster: This extension helps users solve difficult CAPTCHAs by completing reCAPTCHA audio challenges using speech recognition. The extension is also available for Firefox and Edge.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;2Captcha Extension: This extension is an open-source project that helps users solve any CAPTCHAs on any sites. The extension is regularly used by more than 60,000 users.&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Conclusion&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;CAPTCHA auto solver extensions can be useful for users who have difficulty solving CAPTCHAs. reCAPTCHA Solver Auto Captcha Bypass is one such extension that can bypass various types of reCAPTCHAs automatically. However, it requires a paid API key from the solving service. There are also other CAPTCHA auto solver extensions available, such as Captcha Solver: Auto Recognition and Bypass, Buster, and 2Captcha Extension.&lt;/p&gt;

</description>
      <category>recaptcha</category>
    </item>
    <item>
      <title>Math bot captcha solving</title>
      <dc:creator>gresiuseo</dc:creator>
      <pubDate>Mon, 02 Oct 2023 13:19:32 +0000</pubDate>
      <link>https://dev.to/gresiuseo/math-bot-captcha-solving-13jh</link>
      <guid>https://dev.to/gresiuseo/math-bot-captcha-solving-13jh</guid>
      <description>&lt;p&gt;&lt;a href="https://2captcha.com/?from=10075179"&gt;2captcha.com&lt;/a&gt; is a captcha solving service that can help solve math bot captchas. The purpose of captchas is to prevent automated bots from accessing or spamming websites. Captchas usually contain a distorted text within an image, but math bot captchas require the user to solve a simple math problem to gain access to a website. &lt;/p&gt;

&lt;p&gt;The process of solving a math bot captcha using 2Captcha is as follows: we take the captcha image from the page and send it to the 2Captcha service, where an employee solves it by solving the math problem, then the answer is returned to us, which must be entered in the appropriate field to solve the captcha. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Here are the steps to solve a math bot captcha using 2Captcha:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Take the captcha image from the page.&lt;/li&gt;
&lt;li&gt;Send the captcha image to the 2Captcha service.&lt;/li&gt;
&lt;li&gt;Wait for the result, which will be the answer to the math problem.&lt;/li&gt;
&lt;li&gt;Enter the answer in the appropriate field to solve the captcha.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;2Captcha's API works via a two-step process where you submit CAPTCHA data in and then poll for results with the returned request ID. The price for solving captchas varies depending on the type of captcha and the number of captchas you need to solve. For normal, basic captchas, you can earn anywhere between $0.25 and $0.60 for every 1,000 captchas you solve. &lt;/p&gt;

&lt;p&gt;In conclusion, 2Captcha is a captcha solving service that can help solve math bot captchas. To solve a math bot captcha, you need to take the captcha image from the page and send it to the 2Captcha service. The recognition price varies depending on the type of captcha and the number of captchas you need to solve.&lt;/p&gt;

</description>
      <category>captchasolving</category>
    </item>
  </channel>
</rss>
