<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: yanmoheluo</title>
    <description>The latest articles on DEV Community by yanmoheluo (@yanmoheluo).</description>
    <link>https://dev.to/yanmoheluo</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3945843%2F8257b244-b24b-46d1-87a7-f613d44ec349.jpeg</url>
      <title>DEV Community: yanmoheluo</title>
      <link>https://dev.to/yanmoheluo</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/yanmoheluo"/>
    <language>en</language>
    <item>
      <title>Laravel Exchange Rate Service: How We Solved Multi-Currency Refund Disputes</title>
      <dc:creator>yanmoheluo</dc:creator>
      <pubDate>Wed, 10 Jun 2026 09:09:24 +0000</pubDate>
      <link>https://dev.to/yanmoheluo/laravel-exchange-rate-service-how-we-solved-multi-currency-refund-disputes-44lb</link>
      <guid>https://dev.to/yanmoheluo/laravel-exchange-rate-service-how-we-solved-multi-currency-refund-disputes-44lb</guid>
      <description>&lt;p&gt;"I paid 102.30, and you returned 99.85? That’s not a refund, that’s a penalty."&lt;/p&gt;

&lt;p&gt;Alex had ordered sneakers from our cross-border platform — ¥16,500, which came out to €102.30 at the time of payment. A few days later the size didn’t fit, so he requested a refund. Everything went smoothly from an operations standpoint … until the money hit his card.&lt;/p&gt;

&lt;p&gt;We sent back ¥16,500, but the yen-to-euro rate had moved since his purchase. He got €99.85 — €2.45 less than he paid. The root cause wasn’t a calculation error. It was our naive exchange rate handling: we were using whatever rate was current at the time of the refund, not the rate at the time of the original payment.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Real Problem: A Drifting Exchange Rate
&lt;/h2&gt;

&lt;p&gt;Our platform is a Laravel monolith that powers a marketplace connecting European buyers with Japanese sneaker sellers. Order amounts are captured in the seller’s local currency — yen. But we charge customers in their preferred currency (euros, dollars, etc.), and all the financial reporting is in yet another base currency. That meant every money‑moving operation crossed a rate boundary.&lt;/p&gt;

&lt;p&gt;For a purchase, we would fetch the live rate from an external provider, convert the total, and charge the customer. The order record only stored the final converted amount and the currency, &lt;em&gt;not the rate itself&lt;/em&gt;. When a refund was triggered, the system simply fetched the &lt;em&gt;current&lt;/em&gt; rate again and used it to calculate the refund amount in the customer’s currency. If the yen had weakened against the euro between purchase and refund, the customer lost money. If it had strengthened, we lost money. Either way, someone was unhappy.&lt;/p&gt;

&lt;p&gt;The fix was obvious in hindsight: we needed to pin the exchange rate to the order at the moment of creation, and then reuse that exact same rate for any subsequent financial event — refunds, partial refunds, chargebacks, everything.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building a Laravel Exchange Rate Service That Actually Works
&lt;/h2&gt;

&lt;p&gt;We introduced an &lt;code&gt;OrderExchangeRate&lt;/code&gt; model and a dedicated service class to handle rate capture and retrieval. The service’s job is simple:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;When an order is placed, fetch the live rate and store it together with the order.&lt;/li&gt;
&lt;li&gt;When a refund is initiated, retrieve that stored rate — avoid hitting the API again.&lt;/li&gt;
&lt;li&gt;Fall back gracefully if no rate is found (which shouldn’t happen, but paranoia pays).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here’s the core of the service:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="kn"&gt;namespace&lt;/span&gt; &lt;span class="nn"&gt;App\Services&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kn"&gt;use&lt;/span&gt; &lt;span class="nc"&gt;App\Models\OrderExchangeRate&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kn"&gt;use&lt;/span&gt; &lt;span class="nc"&gt;App\Services\ExchangeRateProvider&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kn"&gt;use&lt;/span&gt; &lt;span class="nc"&gt;Illuminate\Support\Facades\Log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ExchangeRateService&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;captureRateForOrder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="nv"&gt;$from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="nv"&gt;$to&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt;

&lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="nv"&gt;$rate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;app&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;ExchangeRateProvider&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="n"&gt;class&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getRate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$to&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;\Throwable&lt;/span&gt; &lt;span class="nv"&gt;$e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="nc"&gt;Log&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Failed to fetch rate for order'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;

&lt;span class="s1"&gt;'order_id'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="s1"&gt;'exception'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$e&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getMessage&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;

&lt;span class="p"&gt;]);&lt;/span&gt;

&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nv"&gt;$e&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nc"&gt;OrderExchangeRate&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;

&lt;span class="s1"&gt;'order_id'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="s1"&gt;'from_currency'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$from&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="s1"&gt;'to_currency'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$to&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="s1"&gt;'rate'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$rate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="s1"&gt;'captured_at'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;

&lt;span class="p"&gt;]);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="nc"&gt;The&lt;/span&gt; &lt;span class="sb"&gt;`ExchangeRateProvider`&lt;/span&gt; &lt;span class="n"&gt;is&lt;/span&gt; &lt;span class="n"&gt;just&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="n"&gt;thin&lt;/span&gt; &lt;span class="n"&gt;wrapper&lt;/span&gt; &lt;span class="n"&gt;around&lt;/span&gt; &lt;span class="n"&gt;the&lt;/span&gt; &lt;span class="n"&gt;actual&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt; &lt;span class="nf"&gt;API&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;we&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="nc"&gt;Fixer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;later&lt;/span&gt; &lt;span class="n"&gt;switched&lt;/span&gt; &lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;the&lt;/span&gt; &lt;span class="no"&gt;ECB&lt;/span&gt; &lt;span class="n"&gt;feed&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;cost&lt;/span&gt; &lt;span class="n"&gt;reasons&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="nc"&gt;The&lt;/span&gt; &lt;span class="n"&gt;important&lt;/span&gt; &lt;span class="n"&gt;part&lt;/span&gt; &lt;span class="n"&gt;is&lt;/span&gt; &lt;span class="n"&gt;that&lt;/span&gt; &lt;span class="n"&gt;the&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt; &lt;span class="n"&gt;is&lt;/span&gt; &lt;span class="n"&gt;captured&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;exactly&lt;/span&gt; &lt;span class="n"&gt;once&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;and&lt;/span&gt; &lt;span class="n"&gt;stored&lt;/span&gt; &lt;span class="n"&gt;immutably&lt;/span&gt;&lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="nc"&gt;No&lt;/span&gt; &lt;span class="n"&gt;drifting&lt;/span&gt;&lt;span class="mf"&gt;.&lt;/span&gt;

&lt;span class="nc"&gt;Then&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;when&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="n"&gt;refund&lt;/span&gt; &lt;span class="n"&gt;is&lt;/span&gt; &lt;span class="n"&gt;initiated&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;we&lt;/span&gt; &lt;span class="n"&gt;look&lt;/span&gt; &lt;span class="n"&gt;up&lt;/span&gt; &lt;span class="n"&gt;that&lt;/span&gt; &lt;span class="n"&gt;stored&lt;/span&gt; &lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;&lt;br&gt;
php&lt;br&gt;
public function getRateForOrder(int $orderId): float&lt;br&gt;
{&lt;/p&gt;

&lt;p&gt;$rate = OrderExchangeRate::where('order_id', $orderId)&lt;/p&gt;

&lt;p&gt;-&amp;gt;latest('captured_at')&lt;/p&gt;

&lt;p&gt;-&amp;gt;value('rate');&lt;/p&gt;

&lt;p&gt;if (!$rate) {&lt;/p&gt;

&lt;p&gt;Log::warning('No stored rate found for order, falling back', [&lt;/p&gt;

&lt;p&gt;'order_id' =&amp;gt; $orderId,&lt;/p&gt;

&lt;p&gt;]);&lt;/p&gt;

&lt;p&gt;// Emergency fallback – fetch live, but not for refund calculation&lt;/p&gt;

&lt;p&gt;// This path should be extremely rare and trigger an alert&lt;/p&gt;

&lt;p&gt;return app(ExchangeRateProvider::class)-&amp;gt;getRate('JPY', 'EUR');&lt;/p&gt;

&lt;p&gt;}&lt;/p&gt;

&lt;p&gt;return $rate;&lt;br&gt;
}&lt;br&gt;
Notice the fallback – it’s deliberately kept as a last resort. In production we haven’t hit it because the capture happens in the same database transaction as the order creation. But having it there, with a loud log, saved us one night when a deployment script accidentally dropped a handful of rate records. Knowing exactly which orders were affected let us manually recalculate the refunds before any customer noticed.&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;


## A Note on Caching

A common temptation is to cache rates in Redis to avoid the API call on every order. We did that too – until we realized that a 5‑minute cache window could still mean two orders placed seconds apart would use slightly different rates. That’s fine for display purposes, but for money‑locking, every order needs its own point‑in‑time snapshot. So we kept the rate capture per‑order, and used a short‑lived cache only for the order‑creation page’s currency conversion preview. The actual payment captured the rate afresh and stored it.

In a project I reviewed later — Taocarts, a cross‑border e‑commerce SaaS — they handle this similarly. Each order carries its own exchange rate, and refunds consistently reference that original rate. The lesson is the same everywhere: exchange rates are not a global variable.

## The Impact: From Angry Tickets to Boring Books

After rolling out the exchange rate service, the number of refund‑related disputes dropped to practically zero. Before, we averaged 3–5 complaints per month about refund amounts being off. In the eight months since, we’ve had one — and that one turned out to be a double‑charge caused by a payment gateway callback retry, not a rate issue.

The other win was on the accounting side. With every order pinned to a known rate, our finance team stopped spending hours reconciling currency‑conversion discrepancies at month‑end. The "books don’t match" meetings became 10‑minute check‑ins rather than multi‑day investigations.

## Lessons Learned (the Hard Way)

* **Don’t use a live rate for money that already moved.** The moment a payment is captured, the exchange rate becomes part of the transaction’s DNA. Storing it in the order record is cheap; customer trust isn’t.
* **Store the rate, not just the converted amount.** If you only store the final amount, you can’t reverse‑engineer the correct rate for partial refunds, and you lose the ability to provide transparency to the customer.
* **Fallbacks are dangerous, but necessary.** Design for the impossible. A missing rate record is an edge case that shouldn’t exist, but when it does, a clear log entry and an alert are worth more than an automatic “fix.”
* **The exchange rate service doesn’t need to be complex.** A single model, two methods, and a database column. The complexity isn’t in the code — it’s in realising you need it in the first place.

Looking back, the €2.45 that sparked all this work was the best unexpected investment. Alex eventually got his full refund — manually, while we were fixing the system — and stayed a customer. That’s the part that stuck with me: a €2.45 rounding error nearly lost us a customer, and fixing it was cheaper than any ad campaign.

These days our refund disputes are close to zero — one incident in eight months, and that was a gateway retry. One model, two methods, and a database column.

The key takeaway: capture the rate once and trust that snapshot. Have you faced similar refund issues?

DESCRIPTION: Storing exchange rates at order creation prevents multi-currency refund disputes.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>php</category>
      <category>ecommerce</category>
    </item>
    <item>
      <title>Daigou Code: How We Solved Payment Reconciliation Nightmares in Cross-border Order Management</title>
      <dc:creator>yanmoheluo</dc:creator>
      <pubDate>Thu, 04 Jun 2026 10:18:23 +0000</pubDate>
      <link>https://dev.to/yanmoheluo/daigou-code-how-we-solved-payment-reconciliation-nightmares-in-cross-border-order-management-3df7</link>
      <guid>https://dev.to/yanmoheluo/daigou-code-how-we-solved-payment-reconciliation-nightmares-in-cross-border-order-management-3df7</guid>
      <description>&lt;p&gt;Last month, a client pulled me into their finance Slack channel. Their operations lead had spent four nights reconciling orders from the previous quarter. The ledger showed total revenue around $47,000, but the payment gateway settlement came in at just over $45,200. Almost two grand missing.&lt;/p&gt;

&lt;p&gt;The usual suspects were innocent: gateway fees matched the contract, refunds were accounted for, exchange rates didn't explain that gap. Three days later we found it — a cron job that retried failed payment callbacks, and the system processed the same webhook twice for about 120 orders over two months. Each double‑processed callback didn't charge the customer again, but it &lt;em&gt;did&lt;/em&gt; insert duplicate reconciliation records that inflated the ledger.&lt;/p&gt;

&lt;p&gt;That's when I stopped blaming Excel and started fixing the code.&lt;/p&gt;

&lt;h2&gt;
  
  
  Solution
&lt;/h2&gt;

&lt;p&gt;The root cause was simple: no idempotency layer in payment callbacks. Every external webhook — from PayPal, Stripe, or any local gateway — was handled by a controller that verified signature, updated order status, and logged the transaction. If the same payload arrived twice (network retry, gateway resend), the system inserted duplicate records.&lt;/p&gt;

&lt;p&gt;We redesigned the callback handler around a &lt;strong&gt;request idempotency key&lt;/strong&gt;. The pattern is now standard in our &lt;code&gt;daigou code&lt;/code&gt; base.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Callback controller snippet&lt;/span&gt;
&lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;handlePaymentCallback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;Request&lt;/span&gt; &lt;span class="nv"&gt;$request&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="nv"&gt;$payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$request&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getContent&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="nv"&gt;$idempotencyKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$request&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nb"&gt;header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'X-Idempotency-Key'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nb"&gt;md5&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$payload&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// Try to acquire lock with Redis&lt;/span&gt;

&lt;span class="nv"&gt;$lockKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"payment:lock:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nv"&gt;$idempotencyKey&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nv"&gt;$lock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;setnx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$lockKey&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;time&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nv"&gt;$lock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="c1"&gt;// Another process is handling the same callback&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Processing'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;202&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;beginTransaction&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="c1"&gt;// Check if already processed&lt;/span&gt;

&lt;span class="nv"&gt;$processed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'payment_idempotency'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'key'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$processed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'OK'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// Extract order_id from payload&lt;/span&gt;

&lt;span class="nv"&gt;$orderId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'order_id'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;\Exception&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Missing order reference'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// Update order status with state machine check&lt;/span&gt;

&lt;span class="nv"&gt;$order&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Order&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$order&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s1"&gt;'paid'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="c1"&gt;// Already paid, still record idempotency but skip double action&lt;/span&gt;

&lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;recordIdempotency&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'OK'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;$order&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'paid'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nv"&gt;$order&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;paid_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="nv"&gt;$order&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="c1"&gt;// Record idempotency entry&lt;/span&gt;

&lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;recordIdempotency&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;del&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$lockKey&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'OK'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;\Exception&lt;/span&gt; &lt;span class="nv"&gt;$e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;rollBack&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;del&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$lockKey&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nc"&gt;Log&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Payment callback failed'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'key'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'error'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$e&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getMessage&lt;/span&gt;&lt;span class="p"&gt;()]);&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Failed'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;recordIdempotency&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;

&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;table&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'payment_idempotency'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;insert&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;

&lt;span class="s1"&gt;'key'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="s1"&gt;'order_id'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$orderId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="s1"&gt;'created_at'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;

&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;payment_idempotency&lt;/code&gt; table has a unique index on &lt;code&gt;key&lt;/code&gt;, so even if Redis lock fails (rare), the database insertion prevents the second write.&lt;/p&gt;

&lt;p&gt;We also locked the exchange rate at order creation time. Before, the system used a live rate at payment confirmation — which could be hours or days after the customer clicked "buy". Now the &lt;code&gt;orders&lt;/code&gt; table stores &lt;code&gt;exchange_rate_used&lt;/code&gt; and &lt;code&gt;base_currency_amount&lt;/code&gt;. The customer sees the final price immediately, and the finance team reconciles against that frozen rate.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;exchange_rate_used&lt;/span&gt; &lt;span class="nb"&gt;DECIMAL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;base_currency_amount&lt;/span&gt; &lt;span class="nb"&gt;DECIMAL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Order creation flow:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nv"&gt;$rate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'cny_to_usd_rate'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mf"&gt;6.45&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// cached, refreshed every 5 min&lt;/span&gt;
&lt;span class="nv"&gt;$order&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;exchange_rate_used&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$rate&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nv"&gt;$order&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;base_currency_amount&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$cartTotalUSD&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nv"&gt;$rate&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nv"&gt;$order&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;save&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Lessons Learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Idempotency is not just about deduplication — it's about defining what "once" means.&lt;/strong&gt; Our first implementation used the raw JSON payload as the idempotency key. That broke when gateways added a timestamp field that changed between retries. We switched to a composite key: gateway name + gateway transaction ID. That survived retries.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The lock + database index combination saved us from deadlocks.&lt;/strong&gt; Early on we tried using only a &lt;code&gt;UNIQUE&lt;/code&gt; index on the idempotency table. Under high concurrency (about 30 callbacks per second during flash sales), the database threw duplicate key exceptions, and our error handler retried the whole request, causing a cascade. Adding the Redis lock reduced duplicate insert attempts by about 95%.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Exchange rate caching has a hidden trade‑off.&lt;/strong&gt; We refresh the Redis cache every 15 minutes. During extreme volatility (like JPY moving 1.5% in an hour), the customer pays a rate that's already stale. We added a safety valve: if the external rate deviates from cached rate by more than 2%, the system rejects the order and asks the customer to retry. This happens on maybe 1% of orders, but it saved us from losing money on another 3-4% of transactions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Reconciliation still needs a human in the loop.&lt;/strong&gt; We built a daily diff report that compares gateway settlement files (CSV) with our &lt;code&gt;payment_idempotency&lt;/code&gt; and &lt;code&gt;orders&lt;/code&gt; tables. Any mismatch over 0.5% triggers an alert. The report cut monthly reconciliation time from about 15 hours to under 2 hours.&lt;/p&gt;

&lt;p&gt;One thing we didn't anticipate: partial refunds. When a customer returns two of five items, which exchange rate do you use for the refund? The original locked rate or the current rate? We settled on original rate to avoid customer complaints, but that means you eat the currency risk on refunds. Our solution now is to hold a small buffer (around 0.6% of order value) in a separate account specifically for refund exchange losses.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Payment reconciliation in cross-border daigou isn't a math problem — it's a design problem. Idempotency keys, rate locking, and daily diffs turn a "where did the money go" mystery into a routine check. Start with the database unique index, add Redis locks when concurrency grows, and never assume a webhook arrives exactly once.&lt;/p&gt;

&lt;p&gt;If your team still reconciles with Excel and coffee at midnight, you're one flash sale away from a very bad week. The code above is production‑tested on a system handling a few hundred orders daily. Full example with migration files and test webhook simulator is in our internal repo — adapt the pattern to your stack.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;About the Author&lt;/strong&gt;: I'm a CTO who's built and scaled cross‑border purchasing platforms for the past decade. Currently working with Taocarts — a daigou system that handles order management, warehouse coordination, and international shipping for 1688/Taobao sourcing. The idempotency pattern described here runs in production on Taocarts deployments.&lt;/p&gt;

</description>
      <category>php</category>
      <category>logistics</category>
    </item>
    <item>
      <title>Bulk Downloading 1688 Product Images: A Lesson in Maxing Out Bandwidth</title>
      <dc:creator>yanmoheluo</dc:creator>
      <pubDate>Wed, 27 May 2026 03:32:46 +0000</pubDate>
      <link>https://dev.to/yanmoheluo/bulk-downloading-1688-product-images-a-lesson-in-maxing-out-bandwidth-1gbm</link>
      <guid>https://dev.to/yanmoheluo/bulk-downloading-1688-product-images-a-lesson-in-maxing-out-bandwidth-1gbm</guid>
      <description>&lt;p&gt;ur purchasing system suddenly went down. Monitoring showed that outbound bandwidth was maxed out at 500Mbps, causing all external API requests to timeout. The culprit was a script for bulk downloading 1688 product images—it launched 200 concurrent download threads without any rate limiting, completely saturating our shared bandwidth.&lt;/p&gt;

&lt;h2&gt;
  
  
  Problem Scenario: A Brutal Approach to Image Downloading
&lt;/h2&gt;

&lt;p&gt;We needed to sync approximately 3,000 1688 products daily, including main images and detail images, averaging 5 images per product. The initial implementation was straightforward but crude:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Old brute-force download script&lt;/span&gt;
&lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;downloadAllImages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$productIds&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$productIds&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nv"&gt;$images&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get1688ProductImages&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// Call 1688 API to get image URL list&lt;/span&gt;
        &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$images&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nv"&gt;$content&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;file_get_contents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// Synchronous blocking download&lt;/span&gt;
            &lt;span class="nb"&gt;file_put_contents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/images/&lt;/span&gt;&lt;span class="nv"&gt;$id&lt;/span&gt;&lt;span class="s2"&gt;/"&lt;/span&gt;&lt;span class="mf"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nv"&gt;$content&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This script had 3 critical issues:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;No concurrency control&lt;/strong&gt;: While &lt;code&gt;file_get_contents&lt;/code&gt; is synchronous, the outer loop had no limits, resulting in massive HTTP requests fired simultaneously&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No retry mechanism&lt;/strong&gt;: If an image download failed (e.g., network jitter), the script simply skipped it, leaving product images missing&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No bandwidth limiting&lt;/strong&gt;: 200 concurrent requests downloading simultaneously, each averaging 2MB, instantly consumed 400MB of bandwidth&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The immediate consequence: all other business operations (including order processing and logistics queries) were interrupted for 18 minutes. We had to manually kill the process and spend 2 hours re-downloading the failed images.&lt;/p&gt;

&lt;h2&gt;
  
  
  Solution: A Downloader with Rate Limiting and Queue
&lt;/h2&gt;

&lt;p&gt;We redesigned the downloader using Guzzle's async capabilities, adding bandwidth control and retry mechanisms.&lt;/p&gt;

&lt;p&gt;Step one: Use Guzzle's concurrent request pool with a maximum concurrency limit.&lt;br&gt;
Step two: Implement a simple token bucket algorithm for bandwidth control.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="c1"&gt;// New rate-limited downloader&lt;/span&gt;
&lt;span class="kn"&gt;use&lt;/span&gt; &lt;span class="nc"&gt;GuzzleHttp\Client&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kn"&gt;use&lt;/span&gt; &lt;span class="nc"&gt;GuzzleHttp\Pool&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kn"&gt;use&lt;/span&gt; &lt;span class="nc"&gt;GuzzleHttp\Psr7\Request&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;ThrottledImageDownloader&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="nv"&gt;$client&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="nv"&gt;$concurrency&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// Maximum concurrency&lt;/span&gt;
    &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="nv"&gt;$bandwidthLimit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// 50MB/s bandwidth limit&lt;/span&gt;
    &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="nv"&gt;$tokens&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="nv"&gt;$lastRefillTime&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;__construct&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="s1"&gt;'timeout'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
        &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;bandwidthLimit&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;lastRefillTime&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;microtime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="c1"&gt;// Token bucket algorithm for bandwidth control&lt;/span&gt;
    &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;consumeBandwidth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nv"&gt;$now&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;microtime&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nv"&gt;$elapsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$now&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;lastRefillTime&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;bandwidthLimit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nv"&gt;$elapsed&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;bandwidthLimit&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;lastRefillTime&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$now&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nv"&gt;$bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nv"&gt;$sleepTime&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$bytes&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;bandwidthLimit&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="nb"&gt;usleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$sleepTime&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;e6&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt; &lt;span class="o"&gt;-=&lt;/span&gt; &lt;span class="nv"&gt;$bytes&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;downloadBatch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;array&lt;/span&gt; &lt;span class="nv"&gt;$imageUrls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="nv"&gt;$requests&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$urls&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'GET'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;};&lt;/span&gt;

        &lt;span class="nv"&gt;$pool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Pool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$requests&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$imageUrls&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="s1"&gt;'concurrency'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;concurrency&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s1"&gt;'fulfilled'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$response&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;use&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$imageUrls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="nv"&gt;$content&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$response&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getBody&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getContents&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
                &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;consumeBandwidth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;strlen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$content&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
                &lt;span class="c1"&gt;// Save image logic&lt;/span&gt;
                &lt;span class="nv"&gt;$filename&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;basename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$imageUrls&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$index&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
                &lt;span class="nb"&gt;file_put_contents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/images/&lt;/span&gt;&lt;span class="nv"&gt;$filename&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$content&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="s1"&gt;'rejected'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$reason&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;use&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$imageUrls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="c1"&gt;// Retry on failure, up to 3 times&lt;/span&gt;
                &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;retryDownload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$imageUrls&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$index&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
            &lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;]);&lt;/span&gt;

        &lt;span class="nv"&gt;$pool&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;promise&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;wait&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;retryDownload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$maxRetries&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;$i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nv"&gt;$maxRetries&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;$i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="nv"&gt;$response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$this&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="nv"&gt;$content&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$response&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getBody&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="nf"&gt;getContents&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
                &lt;span class="nv"&gt;$filename&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;basename&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="nb"&gt;file_put_contents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/images/&lt;/span&gt;&lt;span class="nv"&gt;$filename&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$content&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;\Exception&lt;/span&gt; &lt;span class="nv"&gt;$e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$i&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nv"&gt;$maxRetries&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                    &lt;span class="c1"&gt;// Log failure&lt;/span&gt;
                    &lt;span class="nb"&gt;error_log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Failed to download &lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="s2"&gt; after &lt;/span&gt;&lt;span class="nv"&gt;$maxRetries&lt;/span&gt;&lt;span class="s2"&gt; attempts"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
                &lt;span class="p"&gt;}&lt;/span&gt;
                &lt;span class="nb"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;pow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$i&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt; &lt;span class="c1"&gt;// Exponential backoff&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Key improvements:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Concurrency control&lt;/strong&gt;: &lt;code&gt;concurrency&lt;/code&gt; set to 10, preventing instant bandwidth saturation&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Token bucket rate limiting&lt;/strong&gt;: The &lt;code&gt;consumeBandwidth&lt;/code&gt; method ensures downloads don't exceed 50MB per second&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exponential backoff retry&lt;/strong&gt;: Wait 2^i seconds after failure, with a maximum of 3 attempts&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Lessons Learned: From Bandwidth Disaster to Stable Sync
&lt;/h2&gt;

&lt;p&gt;After deploying the new downloader, we ran A/B tests. The old script took 12 minutes to download images for 3,000 products (~15,000 images), but consumed 500Mbps of bandwidth. The new script took 18 minutes for the same task, but bandwidth remained stable at 45-50Mbps with zero impact on other services.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Further optimization: Incremental downloads and caching&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;We also added a simple file hash check to avoid re-downloading existing images:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Incremental check - only download new images&lt;/span&gt;
&lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;needsDownload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$localPath&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nb"&gt;file_exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$localPath&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="c1"&gt;// Check if remote file has been updated via HEAD request&lt;/span&gt;
    &lt;span class="nv"&gt;$headers&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;get_headers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nv"&gt;$remoteSize&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'Content-Length'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nv"&gt;$localSize&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;filesize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$localPath&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nv"&gt;$remoteSize&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="nv"&gt;$localSize&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This optimization reduced daily incremental sync time from 18 minutes to 3-5 minutes, since only about 10% of product images are updated daily.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Summary:&lt;/strong&gt; When bulk downloading third-party resources, never assume that "faster is better." Brute-force concurrent downloads may seem efficient, but they often sacrifice system stability. Rate limiting, retry mechanisms, and incremental checks are the three core elements of a reliable download system. If your image sync script is still running &lt;code&gt;file_get_contents&lt;/code&gt; without protection, it's time for an upgrade.&lt;/p&gt;

&lt;p&gt;Has your system encountered similar issues when handling large volumes of external resource downloads? Feel free to share your solutions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;About the Author&lt;/strong&gt;: Building cross-border purchasing solutions with taocarts — a daigou system for 1688/Taobao purchasing, order management, and international shipping.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>ai</category>
    </item>
    <item>
      <title>Syncing 1688 Orders with Webhooks: A Production Incident</title>
      <dc:creator>yanmoheluo</dc:creator>
      <pubDate>Wed, 27 May 2026 03:22:47 +0000</pubDate>
      <link>https://dev.to/yanmoheluo/syncing-1688-orders-with-webhooks-a-production-incident-4i87</link>
      <guid>https://dev.to/yanmoheluo/syncing-1688-orders-with-webhooks-a-production-incident-4i87</guid>
      <description>&lt;p&gt;Our order processing queue hit a backlog of over 5,000 messages—compared to a normal baseline under 100. Even worse, customer order statuses were stuck at "Paid" for over 4 hours with no updates. Investigation revealed that our scheduled polling script for syncing 1688 order statuses had exhausted the database connection pool under high concurrency, causing the entire system to hang.&lt;/p&gt;

&lt;h2&gt;
  
  
  Background: The Cost of Polling
&lt;/h2&gt;

&lt;p&gt;We were running a Cron Job every 5 minutes that called the 1688 Open API to fetch order status changes from the past 24 hours. The code looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;myapp.models&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Order&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;myapp.db&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;get_db_connection&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;poll_1688_orders&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;get_db_connection&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;cursor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="c1"&gt;# Fetch 100 records per page, no incremental markers
&lt;/span&gt;    &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.1688.com/order/list&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pageSize&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pageNum&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
        &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;UPDATE orders SET status = %s WHERE order_id = %s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This script had three critical issues:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Full fetch&lt;/strong&gt;: Retrieved all orders on each run, with no incremental ID or timestamp markers, causing duplicate processing&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Synchronous blocking&lt;/strong&gt;: &lt;code&gt;requests.get&lt;/code&gt; was blocking—when the API responded slowly, the entire process stalled&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Connection leaks&lt;/strong&gt;: Under high concurrency, connections created by &lt;code&gt;get_db_connection()&lt;/code&gt; weren't properly released, eventually exhausting the connection pool&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The direct consequence of that incident: database connections spiked to 200+, exceeding MySQL's default maximum of 151, causing all write operations to fail. We spent 45 minutes manually restarting services and clearing the queue, during which 120 order shipping status updates were delayed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Solution: Replacing Polling with Webhooks
&lt;/h2&gt;

&lt;p&gt;We decided to migrate to Webhook mode. The 1688 Open API supports configuring a callback URL for each application. When order statuses change, it proactively pushes a JSON payload to our server. This fundamentally eliminates the resource waste and latency caused by polling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 1&lt;/strong&gt;: Configure the Webhook endpoint in the 1688 Open Platform.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Step 2&lt;/strong&gt;: Write a lightweight Webhook receiver using an async framework to process requests without blocking.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;fastapi&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;HTTPException&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hmac&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;myapp.services&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;update_order_status&lt;/span&gt;

&lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FastAPI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;OrderWebhookPayload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;new_status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;
    &lt;span class="n"&gt;timestamp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;
    &lt;span class="n"&gt;signature&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;

&lt;span class="nd"&gt;@app.post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/webhook/1688/order&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handle_1688_webhook&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;OrderWebhookPayload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# 1. Verify signature - prevent forged requests
&lt;/span&gt;    &lt;span class="n"&gt;secret&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;your_1688_webhook_secret&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;raw_body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;body&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;expected_signature&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hmac&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;secret&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;raw_body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sha256&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;signature&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;expected_signature&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;HTTPException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status_code&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;403&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;detail&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Invalid signature&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# 2. Async database update - use connection pool
&lt;/span&gt;    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;update_order_status&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;new_status&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="c1"&gt;# 3. Return 200 quickly to avoid timeout retries
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Key improvements:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Async processing&lt;/strong&gt;: &lt;code&gt;async def&lt;/code&gt; allows FastAPI to handle thousands of concurrent Webhook requests simultaneously without blocking I/O&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Signature verification&lt;/strong&gt;: Uses HMAC-SHA256 to verify payload origin, preventing malicious requests&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Connection pool&lt;/strong&gt;: &lt;code&gt;update_order_status&lt;/code&gt; internally uses SQLAlchemy's connection pool, keeping maximum connections under 20&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Lessons Learned: Refactoring from the Incident
&lt;/h2&gt;

&lt;p&gt;This incident prompted us to redesign the entire order sync architecture. Here are the specific optimizations:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Decouple with a Message Queue&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The Webhook receiver only handles verification and enqueuing—it doesn't write directly to the database. We use Redis as a lightweight queue:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;aioredis&lt;/span&gt;

&lt;span class="n"&gt;redis&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;aioredis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_url&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;redis://localhost:6379&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;enqueue_order_update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;new_status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;message&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;order_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;new_status&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lpush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order_updates&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. Batch Processing in Consumer&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Another async worker pulls messages from the queue in batches, updating the database every 10 messages or every 5 seconds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;batch_update_worker&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;messages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rpop&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order_updates&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
            &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;break&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;db_pool&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;acquire&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;executemany&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;UPDATE orders SET status = $1 WHERE order_id = $2&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                    &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order_id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
                &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After deploying this architecture, our order sync latency dropped from an average of 5 minutes to under 10 seconds, and database connections stabilized at 15-20. More importantly, we haven't experienced any more cascading failures caused by polling.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Summary:&lt;/strong&gt; Don't use polling to solve real-time problems, especially when a third-party API provides Webhooks. The Webhook + message queue combination not only reduces resource consumption but also makes systems more stable when handling burst traffic. If your order system is still using Cron Job polling, now is the time to refactor.&lt;/p&gt;

&lt;p&gt;What do you think of this approach? Has your order sync system encountered similar bottlenecks? Let's discuss in the comments.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;About the Author&lt;/strong&gt;: Building cross-border purchasing solutions with taocarts — a daigou system for 1688/Taobao purchasing, order management, and international shipping.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>backend</category>
      <category>performance</category>
      <category>python</category>
    </item>
    <item>
      <title>reverse Haitao Development's 3 Common Pitfalls</title>
      <dc:creator>yanmoheluo</dc:creator>
      <pubDate>Sat, 23 May 2026 04:19:53 +0000</pubDate>
      <link>https://dev.to/yanmoheluo/reverse-haitao-developments-3-common-pitfalls-4g7n</link>
      <guid>https://dev.to/yanmoheluo/reverse-haitao-developments-3-common-pitfalls-4g7n</guid>
      <description>&lt;p&gt;When I received this demand, my first reaction was, "Can't this thing work with a ready-made solution?" Later, after a careful analysis of the business scenario, it was found that there were indeed several special constraints.&lt;/p&gt;

&lt;p&gt;A three-layer defense system against oversold items: the first layer of Redis Lua atomic withholding inventory (traffic peak shaving), the second layer of Laravel Queue asynchronous processing (fast response), and the third layer of MySQL transactions + optimistic lock-up. Tested to support an average daily order volume of over 100,000 orders with 100% inventory accuracy, Sneaker's代购website development is also a point that many users frequently inquire about.&lt;/p&gt;

&lt;p&gt;First, let's see what options are available. There are roughly three options available on the market: developing from scratch, using open-source systems, and directly using off-the-shelf SaaS systems like taocarts. The applicable scenarios and hidden costs for each scheme vary significantly.&lt;/p&gt;

&lt;p&gt;It's not scary for a technical decision to turn into technical debt three years later; what's scary is forgetting why it was chosen in the first place.&lt;/p&gt;

&lt;p&gt;The final selection primarily considered three constraints: limited server budget (2C4G lightweight cloud), a team of just me working on the backend, and customers needing to go online within two weeks. Under this constraint, taocarts' out-of-the-box solution is more appropriate. When ChatGPT was first introduced, people were still questioning whether 'AI can work.' Now, they no longer question it.&lt;/p&gt;

&lt;p&gt;The overall architecture adopts a separation of frontend and backend. PHP's self-developed framework provides RESTful APIs, Vue. JavaScript builds frontend interfaces for authentication using HMAC signatures. File caching is used for hot data caching, while MySQL is used for persistent storage.&lt;/p&gt;

&lt;p&gt;Below is a key code snippet:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;
&lt;span class="c1"&gt;// Order Status Machine: The Evolution from 5-State to 8-State&lt;/span&gt;
&lt;span class="c1"&gt;// Initially, only 5 states were defined, but it was later discovered that there was a gap between 'pending procurement' and 'purchased'.&lt;/span&gt;
&lt;span class="c1"&gt;// Missing a 'purchase in progress' status — placing an order in 1688 might take 3-5 seconds during this period.&lt;/span&gt;
&lt;span class="c1"&gt;// If users repeatedly click, it will trigger repeat purchases. After adding this intermediate state, the problem is solved.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ORDER_STATES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="na"&gt;PENDING&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// Pending payment&lt;/span&gt;
&lt;span class="na"&gt;PAID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;paid&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// Paid&lt;/span&gt;
&lt;span class="na"&gt;PURCHASING&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;purchasing&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// Purchasing (for weight protection)&lt;/span&gt;
&lt;span class="na"&gt;PURCHASED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;purchased&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// 已采购&lt;/span&gt;
&lt;span class="na"&gt;ARRIVED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;arrived&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// Already in Stock&lt;/span&gt;
&lt;span class="na"&gt;PACKED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;packed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// Packaged&lt;/span&gt;
&lt;span class="na"&gt;SHIPPED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;shipped&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// Shipped&lt;/span&gt;
&lt;span class="na"&gt;COMPLETED&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;completed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// 已完成&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Of course, this plan also has limitations. Single-machine deployment limits scalability, and service splitting may be necessary if the number of future tenants doubles. Additionally, file caching is less stable than Redis in high-concurrency scenarios, which is also something that needs to be improved in the future.&lt;/p&gt;

&lt;p&gt;Technical selection doesn't have a silver bullet, but it follows principles—first clarify business constraints, then assess team capabilities, and finally choose the most suitable one. Do not rely on technology just for the sake of it.&lt;/p&gt;

&lt;p&gt;I wrote this article because I couldn't find complete information online during my previous trip, and I hope it can help those who encountered similar issues. Welcome to share your implementation in the comments.&lt;/p&gt;

&lt;p&gt;The COD signature rate in the Middle East market exceeds 90%, while the return rate is below 5%. The consumption peak occurs around Ramadan, so it's important to plan 60 days in advance to seize the traffic window.&lt;/p&gt;

&lt;p&gt;Platform 1688 saw a 45% year-over-year increase in cross-border purchase orders in 2025, with an increasing number of overseas merchants purchasing directly from the 1688 source.&lt;/p&gt;

&lt;p&gt;In 2025, China's cross-border e-commerce exports exceeded 15 trillion yuan, marking a 18% year-over-year growth.&lt;/p&gt;

&lt;p&gt;In 2026, cross-border e-commerce growth in Southeast Asia is expected to exceed 25%, making it the fastest-growing regional market.&lt;/p&gt;

&lt;p&gt;The GMV of WeChat mini-program e-commerce is set to exceed 5 trillion yuan by 2025, with mini-programs becoming an important entry point for cross-border shopping agents.&lt;/p&gt;

&lt;p&gt;The average unit price in the Middle East market (United Arab Emirates, Saudi Arabia) is three times that of Southeast Asia, with luxury goods代购growth exceeding 50%.&lt;/p&gt;

&lt;p&gt;The data shows that after integrating logistics tracking, customer inquiries decreased by 60%, as buyers could check the logistics progress themselves.&lt;/p&gt;

&lt;p&gt;A solution for optimizing images from 5MB to 200KB: WebP format (30-50% smaller than JPEG) + multi-size thumbnails (list 200px, details 800px) + CDN distribution + lazy loading (Intersection Observer). Optimized first-screen image loading time has been reduced from 4.5 seconds to 1.2 seconds.&lt;/p&gt;

&lt;p&gt;Automatic database backup solution: Fully back up to OSS via mysqldump at 3 AM every day, retaining data for the past 7 days. binlog backups are performed every 2 hours. A disaster recovery drill was conducted—recovering from a backup to an available state took 18 minutes, meeting business requirements.&lt;/p&gt;

&lt;p&gt;By 2025, global independent station transactions exceeded $1.2 trillion, growing at a rate of 28.7%, far outpacing platform e-commerce's 16.3%. The cost of acquiring customers for independent stations is 32% lower than for platforms, and the user repeat purchase rate is 27%.&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>backend</category>
      <category>laravel</category>
      <category>performance</category>
    </item>
    <item>
      <title>Taobao agent system development pitfalls recorded</title>
      <dc:creator>yanmoheluo</dc:creator>
      <pubDate>Sat, 23 May 2026 04:17:54 +0000</pubDate>
      <link>https://dev.to/yanmoheluo/taobao-agent-system-development-pitfalls-recorded-5d5d</link>
      <guid>https://dev.to/yanmoheluo/taobao-agent-system-development-pitfalls-recorded-5d5d</guid>
      <description>&lt;p&gt;he product synchronization module processes over 100,000 product entries daily. The initial plan was to run for 4 hours at a time, which nearly affected the next day's business. Optimized for 20 minutes, there's no time to do more.&lt;/p&gt;

&lt;p&gt;Off-topic: From 2013 to the present, toolchains have changed, but the初。of doing technology remains the same—solve the problem well, write the documentation clearly, and let the next successor not scold you.&lt;/p&gt;

&lt;p&gt;First, let's look at the data before the optimization. Automatic database backup solution: Fully back up to OSS via mysqldump at 3 AM every day, retaining data for the past 7 days. binlog backups are performed every 2 hours. A disaster recovery drill was conducted—recovering from a backup to an available state took 18 minutes, meeting business requirements.&lt;/p&gt;

&lt;p&gt;The first step in optimization is to identify the bottleneck. Using MySQL slow query logs + Chrome DevTools Performance analysis, it was found that the bottleneck distribution is as follows: database 60%, front-end rendering 25%, and network transmission 15%. First, let's address the largest piece.&lt;/p&gt;

&lt;p&gt;The optimization plan involves several steps. 1688/Taobao API Callback Processing: The most frustrating aspect is the order status callback for 1688—not real-time push, but batch push every so often. Sometimes users have already paid, but the 1688 callback doesn't arrive until 30 minutes later. The solution is a dual-channel active polling + callback: proactively querying the order status changes of 1688's last 30 minutes every 5 minutes as a supplement to the callback. Every step involves specific modifications and verification methods.&lt;/p&gt;

&lt;p&gt;Optimized Real-Time Comparison —— Using Apache Bench to pressure the same interface and the same number of concurrent instances ensures that the data does not lie. The response time was reduced from XX before optimization to YY, and the error rate was zero.&lt;/p&gt;

&lt;p&gt;Trudging isn't scary; it's scary if you don't write it down. Each pit was created using real money for the production environment.&lt;/p&gt;

&lt;p&gt;Summarize a few experiences. Performance optimization follows a principle: measure first, then optimize, and finally measure. Optimization without data support is esoteric, and shopping agent websites are also a point that many users often ask about.&lt;/p&gt;

&lt;p&gt;The core lesson from this optimization: performance bottlenecks are often not where you think they are. I initially doubted the database, but the issue was with the frontend stuffing the entire product list into the DOM at once.&lt;/p&gt;

&lt;p&gt;That's the core idea and key implementation of this solution. Welcome to share your implementation in the comments.&lt;/p&gt;

&lt;p&gt;The shipping cost estimation feature is very practical, allowing customers to see the estimated shipping costs after selecting the item, without needing to ask me every time.&lt;/p&gt;

&lt;p&gt;Merchants using automated代。 systems saw their average unit price increase by 30% due to support for multiple languages and currencies, which made orders smoother for overseas customers.&lt;/p&gt;

&lt;p&gt;Multi-tenant data isolation solution: database-level isolation (with each tenant having its own independent database), rather than the traditional &lt;code&gt;tenant_id&lt;/code&gt; field isolation. Reasons:一是）。 data security issues (physical isolation of data from different merchants), and二是灵活 flexible backup and recovery (a problem with one tenant does not affect others).&lt;/p&gt;

&lt;p&gt;The average unit price in the Middle East market (United Arab Emirates, Saudi Arabia) is three times that of Southeast Asia, with luxury goods代购growth exceeding 50%.&lt;/p&gt;

&lt;p&gt;N+1 queries are the most common performance killer in代。 systems. A product detail page has been optimized from 127 SQL queries to 6, and the database time has been reduced from 1.8 seconds to 120ms. The key is Eager Loading (preloading associated data), which uses Laravel's &lt;code&gt;with()&lt;/code&gt; method to identify all associations at once, rather than checking each one in a loop.&lt;/p&gt;

&lt;p&gt;The biggest feeling is that customer trust has increased. Because the system automatically generates logistics tracking numbers and updates the status in real time, customers don't need to keep urging them.&lt;/p&gt;

&lt;p&gt;The most commonly used service is the inspection and photography service. Upon arrival at the warehouse, staff will take photos and upload them, allowing customers to see the physical photos before deciding whether to ship.&lt;/p&gt;

&lt;p&gt;API Layer HMAC Signature Authentication Mechanism: Each tenant is assigned a separate App Key and App Secret, and request signatures are protected against replay attacks using sha256 (request body + timestamp + secret). Timestamp error allows ±5 minutes, and Redis records show that Nonce has been used to prevent replay.&lt;/p&gt;

&lt;p&gt;The profit statistics report for代。is very intuitive. Daily, weekly, and monthly revenues and costs are clearly visible, with data determining which categories make money and which ones incur losses.&lt;/p&gt;

</description>
      <category>backend</category>
      <category>database</category>
      <category>performance</category>
      <category>systemdesign</category>
    </item>
  </channel>
</rss>
