<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Sergei Solod</title>
    <description>The latest articles on DEV Community by Sergei Solod (@sergeisolod).</description>
    <link>https://dev.to/sergeisolod</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3252935%2Ffee66b0c-1071-49bf-aac1-0f83ff48c2b4.jpg</url>
      <title>DEV Community: Sergei Solod</title>
      <link>https://dev.to/sergeisolod</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/sergeisolod"/>
    <language>en</language>
    <item>
      <title>AVA Hosting Said “No Sharing.” Linux Reported 32.73% CPU Steal</title>
      <dc:creator>Sergei Solod</dc:creator>
      <pubDate>Tue, 01 Sep 2026 22:05:27 +0000</pubDate>
      <link>https://dev.to/sergeisolod/ava-hosting-said-no-sharing-linux-reported-3273-cpu-steal-4gfc</link>
      <guid>https://dev.to/sergeisolod/ava-hosting-said-no-sharing-linux-reported-3273-cpu-steal-4gfc</guid>
      <description>&lt;p&gt;I expected to find a bug in my application.&lt;/p&gt;

&lt;p&gt;Maybe Node.js was overloaded. Maybe Nginx was queuing requests. Maybe memory was running out. Maybe the disk was stalling.&lt;/p&gt;

&lt;p&gt;Instead, &lt;code&gt;mpstat&lt;/code&gt; showed this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Average CPU steal: 32.73%
Average CPU idle:   0.00%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That changed the investigation completely.&lt;/p&gt;

&lt;p&gt;This is not a negative review of AVA Hosting, and I am not claiming that every VPS they operate behaves like mine did.&lt;/p&gt;

&lt;p&gt;This is simply what happened to one KVM VPS I used under a real production workload, how I diagnosed it, and why I eventually cancelled it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The VPS
&lt;/h2&gt;

&lt;p&gt;The configuration was small:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 vCPU
2 GB RAM
25 GB NVMe
KVM
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The workload was not synthetic.&lt;/p&gt;

&lt;p&gt;Nginx was running.&lt;/p&gt;

&lt;p&gt;The backend was running.&lt;/p&gt;

&lt;p&gt;The server still had available RAM.&lt;/p&gt;

&lt;p&gt;The filesystem was not full.&lt;/p&gt;

&lt;p&gt;But requests were accumulating and the backend was not keeping up.&lt;/p&gt;

&lt;p&gt;Instead of immediately upgrading the server, I wanted to know what resource was actually exhausted.&lt;/p&gt;

&lt;p&gt;So I monitored it for 60 seconds.&lt;/p&gt;

&lt;p&gt;No &lt;code&gt;stress-ng&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;No benchmark.&lt;/p&gt;

&lt;p&gt;No artificial HTTP flood.&lt;/p&gt;

&lt;p&gt;Just the real workload.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 60-second CPU profile
&lt;/h2&gt;

&lt;p&gt;The summary was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;vCPU count:             1

Average CPU user:       52.52%
Average CPU system:      8.07%
Average CPU softirq:     6.69%
Average CPU steal:      32.73%
Average CPU iowait:      0.00%
Average CPU idle:        0.00%

Maximum runnable queue: 11
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important part is how those percentages fit together.&lt;/p&gt;

&lt;p&gt;About 52% was normal userspace work.&lt;/p&gt;

&lt;p&gt;About 8% was kernel work.&lt;/p&gt;

&lt;p&gt;Almost 7% was &lt;code&gt;softirq&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;And then:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;steal: 32.73%
idle:   0.00%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The VM had no spare CPU.&lt;/p&gt;

&lt;p&gt;But roughly one third of CPU accounting was also being reported as &lt;strong&gt;steal&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What &lt;code&gt;%steal&lt;/code&gt; means
&lt;/h2&gt;

&lt;p&gt;On a virtual machine, high CPU utilization and high CPU steal are not the same problem.&lt;/p&gt;

&lt;p&gt;Suppose I had measured:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;user + system: ~100%
steal:            0%
idle:             0%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That would be straightforward.&lt;/p&gt;

&lt;p&gt;The workload consumed the CPU it was given.&lt;/p&gt;

&lt;p&gt;I would either optimize the workload or buy more CPU.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;steal&lt;/code&gt; means something different.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;guest has runnable work
        ↓
vCPU wants to run
        ↓
vCPU is not being executed
        ↓
Linux accounts the time as steal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;KVM provides steal-time accounting specifically so a guest can observe time in which its virtual CPU was not running.&lt;/p&gt;

&lt;p&gt;That does not tell me &lt;em&gt;why&lt;/em&gt; the hypervisor did not schedule the vCPU.&lt;/p&gt;

&lt;p&gt;From inside the VM I cannot see:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;physical host utilization
number of neighbouring VMs
CPU overcommit ratio
scheduler policy
vCPU pinning
host-side quotas
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But I can observe the effect from the guest.&lt;/p&gt;

&lt;p&gt;And in this case the effect was large.&lt;/p&gt;

&lt;h2&gt;
  
  
  It wasn't one bad sample
&lt;/h2&gt;

&lt;p&gt;A momentary &lt;code&gt;30% steal&lt;/code&gt; value would not have been enough for me.&lt;/p&gt;

&lt;p&gt;Scheduler noise happens.&lt;/p&gt;

&lt;p&gt;So I looked at the per-second samples.&lt;/p&gt;

&lt;p&gt;They repeatedly looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%usr    %sys   %soft   %steal   %idle

53.54   10.10   5.05    31.31     0.00
56.44    6.93   5.94    30.69     0.00
51.52    7.07   6.06    35.35     0.00
55.45    6.93   5.94    31.68     0.00
51.49    8.91   5.94    33.66     0.00
52.53    7.07   7.07    33.33     0.00
52.48    6.93   7.92    32.67     0.00
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Near the end:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;50.98   10.78   6.86    31.37     0.00
48.48    8.08   9.09    34.34     0.00
55.00    5.00   7.00    33.00     0.00
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The pattern persisted throughout the observation.&lt;/p&gt;

&lt;p&gt;This was the average:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;steal: 32.73%
idle:   0.00%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For intuition:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;60 × 0.3273 = 19.64 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That does &lt;strong&gt;not&lt;/strong&gt; mean the VPS froze continuously for 19.64 seconds.&lt;/p&gt;

&lt;p&gt;It means the smaller periods accounted as steal added up to roughly 19.6 seconds during that minute.&lt;/p&gt;

&lt;p&gt;For a busy one-vCPU server, that is a huge amount of unavailable execution time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Linux PSI showed the CPU was under constant pressure
&lt;/h2&gt;

&lt;p&gt;I also checked Linux Pressure Stall Information:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;some avg10=99.00
some avg60=99.12
some avg300=98.92
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Later:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;some avg10=99.19
some avg60=99.16
some avg300=98.98
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So for almost the entire observation window, at least some runnable work was stalled waiting for CPU.&lt;/p&gt;

&lt;p&gt;The load average was also around:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3.5 – 4.0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;on a machine with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 vCPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the runnable queue reached:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;11
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;None of those measurements alone proves a hypervisor problem.&lt;/p&gt;

&lt;p&gt;Together with &lt;code&gt;32.73% steal&lt;/code&gt;, though, they described a machine with persistent CPU contention.&lt;/p&gt;

&lt;h2&gt;
  
  
  The application queue confirmed that this wasn't just an ugly metric
&lt;/h2&gt;

&lt;p&gt;The backend process was alive.&lt;/p&gt;

&lt;p&gt;That is an important distinction.&lt;/p&gt;

&lt;p&gt;This was not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;backend crashed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;backend running
but unable to keep up
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before the 60-second observation, its listening queue showed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Recv-Q: 168
Send-Q: 511
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After one minute:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Recv-Q: 166
Send-Q: 511
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The interesting part is not &lt;code&gt;168&lt;/code&gt; versus &lt;code&gt;166&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It is that the queue essentially &lt;strong&gt;did not drain&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The server spent a full minute processing traffic and remained just as far behind.&lt;/p&gt;

&lt;p&gt;That connected the CPU metrics with an actual application symptom.&lt;/p&gt;

&lt;h2&gt;
  
  
  I checked memory
&lt;/h2&gt;

&lt;p&gt;Before blaming the virtualization layer, I wanted to eliminate the boring explanations.&lt;/p&gt;

&lt;p&gt;Memory looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;RAM total:      ~1.9 GiB
RAM available:  ~959 MiB

Swap total:     2 GiB
Swap used:      ~33 MiB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nearly a gigabyte was still available.&lt;/p&gt;

&lt;p&gt;There was no obvious OOM event explaining the behavior.&lt;/p&gt;

&lt;p&gt;So this did not look like memory exhaustion.&lt;/p&gt;

&lt;h2&gt;
  
  
  I checked the disk
&lt;/h2&gt;

&lt;p&gt;The filesystem looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;24 GB total
14 GB used
9.4 GB available
60% used
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And, more importantly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Average CPU iowait: 0.00%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server was not sitting around waiting for storage.&lt;/p&gt;

&lt;p&gt;The useful combination was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;iowait:  0.00%
idle:    0.00%
steal:  32.73%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At this point, adding RAM or cleaning the disk would have been solving a different problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then I looked at AVA Hosting's CPU claims
&lt;/h2&gt;

&lt;p&gt;This is where the incident became particularly interesting.&lt;/p&gt;

&lt;p&gt;AVA currently describes its VPS resources as:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;“Guaranteed resources — no sharing”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;and says each VPS receives guaranteed vCPU resources whose performance is not affected by other customers.&lt;/p&gt;

&lt;p&gt;Its Linux VPS page is more specific: AVA says that a CPU-intensive workload from another tenant cannot introduce steal time into another instance and describes its vCPU allocation as dedicated.&lt;/p&gt;

&lt;p&gt;Its unmanaged VPS page goes further again, saying:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;“CPU steal time is eliminated at the hypervisor level”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;and describing CPU resources as isolated rather than a shared or burstable pool.&lt;/p&gt;

&lt;p&gt;That is a much stronger claim than simply advertising:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 vCPU
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A vCPU does &lt;strong&gt;not&lt;/strong&gt; automatically mean one private physical CPU core.&lt;/p&gt;

&lt;p&gt;Virtualization does not work that way by default.&lt;/p&gt;

&lt;p&gt;But AVA's own wording specifically addresses resource sharing and CPU steal.&lt;/p&gt;

&lt;p&gt;My VM reported:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Average CPU steal: 32.73%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the contradiction that mattered to me.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this does and does not prove
&lt;/h2&gt;

&lt;p&gt;I want to keep this technically narrow.&lt;/p&gt;

&lt;p&gt;My measurements establish what happened &lt;strong&gt;inside my VM&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I observed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 vCPU
32.73% average CPU steal
0% CPU idle
~99% CPU pressure
runnable queue up to 11
backend queue that would not drain
0% average CPU iowait
substantial RAM still available
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I cannot use those measurements to prove:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;intentional overselling
exact host occupancy
number of neighbouring VMs
specific neighbour activity
AVA's physical CPU topology
exact hypervisor configuration
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I also tested one VPS, not AVA Hosting's entire infrastructure.&lt;/p&gt;

&lt;p&gt;So my conclusion is deliberately narrower:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The VPS I received exhibited severe CPU steal under a real workload, and that behavior was difficult to reconcile with AVA's explicit claims about dedicated CPU allocation and the absence of neighbour-induced steal time.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That's enough.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I didn't upgrade to more vCPUs
&lt;/h2&gt;

&lt;p&gt;At this point I could have bought a larger plan.&lt;/p&gt;

&lt;p&gt;But I wanted to answer a basic question first:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Was my application simply consuming the CPU I purchased?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If the answer had been yes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;steal ≈ 0%
CPU saturated
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;then buying another vCPU would have been reasonable.&lt;/p&gt;

&lt;p&gt;Instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;steal = 32.73%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Buying additional vCPUs might still have improved throughput.&lt;/p&gt;

&lt;p&gt;But I did not want to pay more before understanding why so much of the first vCPU's requested execution time was already being recorded as steal.&lt;/p&gt;

&lt;p&gt;So I cancelled the VPS.&lt;/p&gt;

&lt;h2&gt;
  
  
  AVA refunded me in full
&lt;/h2&gt;

&lt;p&gt;This part deserves to be included because it was handled well.&lt;/p&gt;

&lt;p&gt;I requested a full refund.&lt;/p&gt;

&lt;p&gt;AVA Hosting returned the entire amount quickly.&lt;/p&gt;

&lt;p&gt;They did not spend days arguing with me about the measurements or trying to turn the issue into a long support exchange.&lt;/p&gt;

&lt;p&gt;I explained the problem, requested the refund, and received it.&lt;/p&gt;

&lt;p&gt;So my experience has two separate parts:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The VPS I received had a serious CPU-availability problem.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;AVA handled the refund well.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Both are true.&lt;/p&gt;

&lt;h2&gt;
  
  
  The VPS check I use now
&lt;/h2&gt;

&lt;p&gt;The practical lesson was not “never use one vCPU.”&lt;/p&gt;

&lt;p&gt;It was not “KVM is bad.”&lt;/p&gt;

&lt;p&gt;And it was not “every AVA VPS must behave like this.”&lt;/p&gt;

&lt;p&gt;The lesson was that I should validate CPU behavior under the actual workload instead of trusting the specification alone.&lt;/p&gt;

&lt;p&gt;These are now some of my first checks on a new Linux VPS:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mpstat 1 60
vmstat 1 60

&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/pressure/cpu
&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/pressure/memory
&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/pressure/io

ss &lt;span class="nt"&gt;-ltnp&lt;/span&gt;

free &lt;span class="nt"&gt;-h&lt;/span&gt;
&lt;span class="nb"&gt;df&lt;/span&gt; &lt;span class="nt"&gt;-h&lt;/span&gt;

iostat &lt;span class="nt"&gt;-xz&lt;/span&gt; 1 60
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I am looking for relationships between metrics.&lt;/p&gt;

&lt;h3&gt;
  
  
  High CPU, almost no steal
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;steal ≈ 0%
idle ≈ 0%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The workload probably needs more CPU.&lt;/p&gt;

&lt;h3&gt;
  
  
  High iowait
&lt;/h3&gt;

&lt;p&gt;Investigate storage.&lt;/p&gt;

&lt;h3&gt;
  
  
  Memory pressure and swap activity
&lt;/h3&gt;

&lt;p&gt;Investigate RAM.&lt;/p&gt;

&lt;h3&gt;
  
  
  High steal + zero idle + high CPU PSI + growing queues
&lt;/h3&gt;

&lt;p&gt;Now I start looking beyond the application.&lt;/p&gt;

&lt;p&gt;That combination is much more useful than simply saying:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;“The VPS feels slow.”&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  A running service is not a healthy service
&lt;/h2&gt;

&lt;p&gt;This incident also reinforced something easy to forget during production debugging.&lt;/p&gt;

&lt;p&gt;These checks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemctl status nginx
systemctl status my-backend
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;can both say:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;active (running)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;while the system is already failing operationally.&lt;/p&gt;

&lt;p&gt;A process can exist.&lt;/p&gt;

&lt;p&gt;A TCP socket can listen.&lt;/p&gt;

&lt;p&gt;Nginx can accept requests.&lt;/p&gt;

&lt;p&gt;And the machine can still be unable to process incoming work fast enough.&lt;/p&gt;

&lt;p&gt;That is why I now care much more about:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU scheduling
pressure
queues
latency
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;than whether a process merely exists.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three numbers I remember from this incident
&lt;/h2&gt;

&lt;p&gt;If I had to reduce the whole debugging session to three lines, they would be:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU steal:    32.73%
CPU idle:      0.00%
CPU pressure: ~99%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;AVA Hosting says its VPS CPU resources are guaranteed without sharing and explicitly describes protection from neighbour-induced CPU steal.&lt;/p&gt;

&lt;p&gt;My VPS reported otherwise.&lt;/p&gt;

&lt;p&gt;I cancelled it.&lt;/p&gt;

&lt;p&gt;AVA refunded everything quickly and without an argument.&lt;/p&gt;

&lt;p&gt;And I added &lt;code&gt;%steal&lt;/code&gt; to the list of metrics I check before trusting a new VPS with production traffic.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>devops</category>
      <category>performance</category>
      <category>vps</category>
    </item>
    <item>
      <title>My REGXA VPS Had 94% CPU Steal With Zero Traffic</title>
      <dc:creator>Sergei Solod</dc:creator>
      <pubDate>Mon, 31 Aug 2026 21:45:15 +0000</pubDate>
      <link>https://dev.to/sergeisolod/my-regxa-vps-had-94-cpu-steal-with-zero-traffic-p71</link>
      <guid>https://dev.to/sergeisolod/my-regxa-vps-had-94-cpu-steal-with-zero-traffic-p71</guid>
      <description>&lt;p&gt;I am not writing this as a negative review of REGXA.&lt;/p&gt;

&lt;p&gt;I am not trying to tell anyone whether they should or should not use the provider.&lt;/p&gt;

&lt;p&gt;This is simply one of my days as a developer.&lt;/p&gt;

&lt;p&gt;I moved a normal workload to a KVM VPS with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 vCPU
2 GB RAM
60 GB NVMe
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nginx was running.&lt;/p&gt;

&lt;p&gt;The backend was running.&lt;/p&gt;

&lt;p&gt;The machine was reachable.&lt;/p&gt;

&lt;p&gt;But it behaved as if it were completely overloaded.&lt;/p&gt;

&lt;p&gt;Requests accumulated. TLS operations became slow. Connections stayed open for far too long. Some requests eventually returned HTTP 504 after more than two minutes.&lt;/p&gt;

&lt;p&gt;My first assumption was simple:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;something inside my VPS was consuming all the CPU.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Then I ran &lt;code&gt;mpstat&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Average CPU steal:   92.58%

CPU 0 steal:         90.62%
CPU 1 steal:         94.57%

CPU user:             4.02%
CPU system:           1.47%
CPU iowait:           0.41%
CPU idle:             0.53%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That changed the investigation completely.&lt;/p&gt;

&lt;p&gt;My application was not consuming 90% of the processor.&lt;/p&gt;

&lt;p&gt;It was barely getting CPU time at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  CPU usage and CPU steal are very different things
&lt;/h2&gt;

&lt;p&gt;If an application is genuinely CPU-bound, I expect something like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;user:    80%
system:  10%
steal:    0%
idle:    10%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The VM is receiving CPU time and spending it on application or kernel work.&lt;/p&gt;

&lt;p&gt;My VPS looked roughly like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;user:     4%
system:   1%
steal:   93%
idle:     1%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;%steal&lt;/code&gt; is time during which a virtual CPU is ready to run but the hypervisor does not schedule it.&lt;/p&gt;

&lt;p&gt;So when I say that CPU time was being taken away from the VPS, I mean it in that specific virtualization sense.&lt;/p&gt;

&lt;p&gt;I cannot prove that REGXA deliberately took physical cores assigned exclusively to me and gave them to another customer.&lt;/p&gt;

&lt;p&gt;Guest metrics cannot prove that.&lt;/p&gt;

&lt;p&gt;They also cannot prove intent.&lt;/p&gt;

&lt;p&gt;What they can show is much simpler:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;the VM had runnable work, but its vCPUs were repeatedly not being scheduled.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Individual samples kept landing in the same range:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;89.85% steal
92.16% steal
96.48% steal
94.87% steal
95.69% steal
95.95% steal
94.57% steal
93.75% steal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;CPU pressure pointed in the same direction:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU PSI some avg10:   79.47
CPU PSI some avg60:   75.32
CPU PSI some avg300:  76.31

Load average:
5.85 / 5.75 / 5.73
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This was a two-vCPU machine.&lt;/p&gt;

&lt;p&gt;There was clearly runnable work waiting for processor time.&lt;/p&gt;

&lt;p&gt;But I still wanted to rule out the most obvious explanation:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;my own workload.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  I removed all production traffic
&lt;/h2&gt;

&lt;p&gt;I moved the active workload to another server.&lt;/p&gt;

&lt;p&gt;Then I stopped all production traffic to the affected VPS.&lt;/p&gt;

&lt;p&gt;The request queues cleared.&lt;/p&gt;

&lt;p&gt;I repeated the CPU test.&lt;/p&gt;

&lt;p&gt;The result:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Average CPU steal:   94.17%

CPU 0 steal:         95.56%
CPU 1 steal:         92.83%

CPU user:             1.95%
CPU system:           0.59%
CPU iowait:           0.59%
CPU idle:             2.18%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;CPU steal actually became worse.&lt;/p&gt;

&lt;p&gt;The important combination was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;user:    1.95%
system:  0.59%
steal:  94.17%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;My applications were almost idle.&lt;/p&gt;

&lt;p&gt;Yet the VPS was still spending roughly 94% of measured CPU time waiting for the hypervisor.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;vmstat&lt;/code&gt; samples were still showing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;91% steal
98% steal
98% steal
97% steal
97% steal
95% steal
96% steal
95% steal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At some points, as many as 15 runnable processes were waiting for CPU.&lt;/p&gt;

&lt;p&gt;That was the strongest experiment in the entire incident.&lt;/p&gt;

&lt;p&gt;Once the workload was gone, application CPU consumption could no longer explain what I was seeing.&lt;/p&gt;

&lt;h2&gt;
  
  
  localhost was taking seconds
&lt;/h2&gt;

&lt;p&gt;Next I wanted to remove the external network from the equation.&lt;/p&gt;

&lt;p&gt;So I tested HTTPS through:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;127.0.0.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That removes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;public DNS;&lt;/li&gt;
&lt;li&gt;my ISP;&lt;/li&gt;
&lt;li&gt;geographic distance;&lt;/li&gt;
&lt;li&gt;internet routing;&lt;/li&gt;
&lt;li&gt;upstream transit;&lt;/li&gt;
&lt;li&gt;the public network path to the server.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The client and server were inside the same VM.&lt;/p&gt;

&lt;p&gt;Under production traffic, I ran ten localhost HTTPS requests.&lt;/p&gt;

&lt;p&gt;Four failed during the TLS handshake.&lt;/p&gt;

&lt;p&gt;The successful ones took:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;29.30 s
22.77 s
12.25 s
11.87 s
11.12 s
 9.40 s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Some TLS handshakes alone took around nine seconds.&lt;/p&gt;

&lt;p&gt;For localhost.&lt;/p&gt;

&lt;p&gt;After removing production traffic, localhost improved, but remained extremely inconsistent:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0.061 s
0.745 s
0.830 s
0.873 s
1.010 s
1.117 s
1.121 s
2.188 s
3.355 s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same local operation could take:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;61 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and then more than:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3.3 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;on another attempt.&lt;/p&gt;

&lt;p&gt;That is not the kind of result I would explain with public networking.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rest of the stack started backing up
&lt;/h2&gt;

&lt;p&gt;The application-level symptoms were also visible in socket state.&lt;/p&gt;

&lt;p&gt;At one point I had approximately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;450 established connections
122 orphaned connections
110 FIN-WAIT-1
33 CLOSE-WAIT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The backend listen queue on localhost reached around:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;14–15
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The HTTPS listen queue reached around:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;40
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nginx logged requests like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;142.857 s  -&amp;gt; HTTP 504
138.902 s  -&amp;gt; HTTP 504
135.064 s  -&amp;gt; HTTP 504
129.819 s  -&amp;gt; HTTP 504
128.657 s  -&amp;gt; HTTP 504
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Other requests remained open for roughly 67–130 seconds.&lt;/p&gt;

&lt;p&gt;I also saw:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;secure-connect timeout: ~35 s
secure-connect timeout: ~41 s
database network timeouts
delayed TLS operations
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Looking at these individually could have sent me in completely different directions.&lt;/p&gt;

&lt;p&gt;A 504?&lt;/p&gt;

&lt;p&gt;Maybe Nginx.&lt;/p&gt;

&lt;p&gt;Database timeout?&lt;/p&gt;

&lt;p&gt;Maybe MongoDB.&lt;/p&gt;

&lt;p&gt;Slow TLS?&lt;/p&gt;

&lt;p&gt;Maybe networking.&lt;/p&gt;

&lt;p&gt;Growing queues?&lt;/p&gt;

&lt;p&gt;Maybe backend concurrency.&lt;/p&gt;

&lt;p&gt;But all of those systems depend on the process being scheduled.&lt;/p&gt;

&lt;p&gt;Increasing a timeout does not create CPU time.&lt;/p&gt;

&lt;p&gt;It only allows the request to wait longer.&lt;/p&gt;

&lt;h2&gt;
  
  
  RAM and disk were not the bottleneck
&lt;/h2&gt;

&lt;p&gt;Memory looked fine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Total RAM:       ~1.9 GiB
Available RAM:   ~1.0–1.1 GiB
Swap used:       ~1–2 MiB

OOM events:      none
OOM killer:      none
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Storage looked fine too:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Filesystem:      ~58 GB
Used:            ~11 GB
Available:       ~44 GB
Usage:           ~20%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;During the important CPU measurements, I/O wait remained below 1%.&lt;/p&gt;

&lt;p&gt;So I did not have:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;exhausted memory;&lt;/li&gt;
&lt;li&gt;heavy swapping;&lt;/li&gt;
&lt;li&gt;OOM activity;&lt;/li&gt;
&lt;li&gt;a full filesystem;&lt;/li&gt;
&lt;li&gt;enough disk wait to explain the behavior.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The number that kept dominating the system was still:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%steal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  I compared it with another KVM VPS
&lt;/h2&gt;

&lt;p&gt;A useful diagnostic needs a control.&lt;/p&gt;

&lt;p&gt;So I ran the same style of test on another KVM VPS that was actively handling normal traffic.&lt;/p&gt;

&lt;p&gt;It reported:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Average CPU steal:   0.02%
CPU idle:           87.86%

CPU PSI avg10:       0.29
CPU PSI avg60:       0.63
CPU PSI avg300:      0.49

Load average:
0.47 / 0.33 / 0.14
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten localhost HTTPS requests completed between roughly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;37 ms
and
69 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Later, I tested another production VPS that genuinely was CPU-busy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU user:     61.71%
CPU system:    5.08%
CPU idle:     24.09%
CPU steal:     0.13%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That comparison made the difference very clear.&lt;/p&gt;

&lt;h3&gt;
  
  
  Busy because my application is running
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;user:   high
system: some
steal:  low
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Busy because the guest is not getting scheduled
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;user:   low
system: low
steal:  extremely high
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both can result in a slow website.&lt;/p&gt;

&lt;p&gt;But they require completely different debugging.&lt;/p&gt;

&lt;h2&gt;
  
  
  REGXA later confirmed resource contention
&lt;/h2&gt;

&lt;p&gt;Until this point, I only had guest-side evidence.&lt;/p&gt;

&lt;p&gt;I could see what Linux was reporting.&lt;/p&gt;

&lt;p&gt;I could not see:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the physical host;&lt;/li&gt;
&lt;li&gt;total host CPU load;&lt;/li&gt;
&lt;li&gt;CPU quotas;&lt;/li&gt;
&lt;li&gt;scheduler weights;&lt;/li&gt;
&lt;li&gt;neighbouring VMs;&lt;/li&gt;
&lt;li&gt;the provider's oversubscription level.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;REGXA could.&lt;/p&gt;

&lt;p&gt;Support eventually told me the VPS was running on:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;shared CPU infrastructure&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;They explained that CPU resources were shared across multiple virtual machines and that performance could vary with physical-node load.&lt;/p&gt;

&lt;p&gt;More importantly, they explicitly said that the elevated CPU steal I was observing resulted from:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;resource contention on the underlying infrastructure&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;They also said the Frankfurt infrastructure was experiencing particularly high demand.&lt;/p&gt;

&lt;p&gt;According to support, they could not change the CPU quotas or scheduling policy or provide additional dedicated CPU resources for that shared VPS.&lt;/p&gt;

&lt;p&gt;The proposed solution was to move the VPS to a less-loaded location.&lt;/p&gt;

&lt;p&gt;At that point, resource contention was no longer only my interpretation of &lt;code&gt;mpstat&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It was also the provider's explanation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The CPU wording was difficult to reconcile
&lt;/h2&gt;

&lt;p&gt;This part bothered me.&lt;/p&gt;

&lt;p&gt;The REGXA VPS material I had seen used wording around:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Dedicated CPU Cores&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;and guaranteed resources.&lt;/p&gt;

&lt;p&gt;But support described my VPS as using shared CPU infrastructure whose performance depended on physical-node load.&lt;/p&gt;

&lt;p&gt;I do not know the exact host configuration.&lt;/p&gt;

&lt;p&gt;I cannot tell whether the immediate mechanism was:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CPU overcommitment;&lt;/li&gt;
&lt;li&gt;CPU quotas;&lt;/li&gt;
&lt;li&gt;scheduler weighting;&lt;/li&gt;
&lt;li&gt;throttling;&lt;/li&gt;
&lt;li&gt;another hypervisor configuration;&lt;/li&gt;
&lt;li&gt;or some combination.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And again, I cannot prove intent.&lt;/p&gt;

&lt;p&gt;What I can say is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Under workload:
CPU steal = 92.58%

With production traffic removed:
CPU steal = 94.17%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And REGXA itself later attributed the elevated steal to resource contention on its shared infrastructure.&lt;/p&gt;

&lt;p&gt;That is enough evidence for me to document what happened without pretending I know more than I do.&lt;/p&gt;

&lt;h2&gt;
  
  
  The refund became another part of the incident
&lt;/h2&gt;

&lt;p&gt;Once the infrastructure issue was clear, I did not want to keep moving the VPS around and testing different nodes.&lt;/p&gt;

&lt;p&gt;I wanted to cancel it.&lt;/p&gt;

&lt;p&gt;Initially, I was offered only a &lt;strong&gt;partial refund&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;And the proposed refund was not back to the card I had used.&lt;/p&gt;

&lt;p&gt;It was REGXA account credit.&lt;/p&gt;

&lt;p&gt;For me, provider credit is not the same thing as a refund if I am trying to stop using the provider.&lt;/p&gt;

&lt;p&gt;So I kept replying.&lt;/p&gt;

&lt;p&gt;I asked for:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;100% of the payment
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;to be returned to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;the original payment method
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It took additional back-and-forth.&lt;/p&gt;

&lt;p&gt;Eventually, REGXA agreed and refunded the &lt;strong&gt;full amount to the original payment method&lt;/strong&gt;, describing the full refund as an exception.&lt;/p&gt;

&lt;p&gt;I appreciate that they eventually returned everything.&lt;/p&gt;

&lt;p&gt;But having to keep pushing for the financial resolution after the infrastructure problem had already been acknowledged is also part of the experience.&lt;/p&gt;

&lt;h2&gt;
  
  
  The debugging lesson was more useful than the hosting story
&lt;/h2&gt;

&lt;p&gt;The easiest mistake here would have been to keep optimizing my application.&lt;/p&gt;

&lt;p&gt;I could have:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;increased Nginx timeouts;&lt;/li&gt;
&lt;li&gt;changed keepalive settings;&lt;/li&gt;
&lt;li&gt;reduced concurrency;&lt;/li&gt;
&lt;li&gt;added retries;&lt;/li&gt;
&lt;li&gt;tuned connection pools;&lt;/li&gt;
&lt;li&gt;investigated MongoDB;&lt;/li&gt;
&lt;li&gt;rewritten backend code.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Some of those changes might have altered individual symptoms.&lt;/p&gt;

&lt;p&gt;None would have answered the real question:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Why is the guest spending more than 90% of its CPU time waiting for the hypervisor?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That problem lives below the application layer.&lt;/p&gt;

&lt;h2&gt;
  
  
  My VPS smoke test now includes &lt;code&gt;%steal&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;I no longer consider a new VPS healthy just because:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SSH works
Nginx starts
/health returns 200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I also check:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mpstat &lt;span class="nt"&gt;-P&lt;/span&gt; ALL 1 20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;vmstat 1 20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/pressure/cpu
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And for a web server, I test localhost latency.&lt;/p&gt;

&lt;p&gt;The metrics I care about are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;%user
%system
%iowait
%idle
%steal
CPU PSI
run queue
localhost latency
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And when the result looks suspicious, I do one more test:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;remove the workload and measure again.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That was the most useful step in this incident.&lt;/p&gt;

&lt;p&gt;The final numbers were hard to misread:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CPU user:     1.95%
CPU system:   0.59%
CPU steal:   94.17%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I am not claiming that every REGXA VPS behaves like this.&lt;/p&gt;

&lt;p&gt;I tested one VPS and documented what happened on that machine.&lt;/p&gt;

&lt;p&gt;But in this case, the evidence was unusually clear:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;my applications were barely using CPU;&lt;/li&gt;
&lt;li&gt;Linux reported 92–94% steal;&lt;/li&gt;
&lt;li&gt;localhost HTTPS became multi-second and sometimes failed;&lt;/li&gt;
&lt;li&gt;the condition remained after production traffic was removed;&lt;/li&gt;
&lt;li&gt;another KVM VPS behaved normally under comparable testing;&lt;/li&gt;
&lt;li&gt;REGXA later attributed the elevated steal to resource contention on its shared infrastructure.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is why I am writing this.&lt;/p&gt;

&lt;p&gt;Not as a rating.&lt;/p&gt;

&lt;p&gt;Not as a recommendation.&lt;/p&gt;

&lt;p&gt;Just as one of my days as a developer — and one Linux metric I will never ignore on a VPS again.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>devops</category>
      <category>performance</category>
      <category>vps</category>
    </item>
    <item>
      <title>I Used Bunny Storage Behind Nginx for Two Months. The Cache Was Fast, the Cold Path Wasn't</title>
      <dc:creator>Sergei Solod</dc:creator>
      <pubDate>Fri, 28 Aug 2026 19:12:42 +0000</pubDate>
      <link>https://dev.to/sergeisolod/i-used-bunny-storage-behind-nginx-for-two-months-the-cache-was-fast-the-cold-path-wasnt-451g</link>
      <guid>https://dev.to/sergeisolod/i-used-bunny-storage-behind-nginx-for-two-months-the-cache-was-fast-the-cold-path-wasnt-451g</guid>
      <description>&lt;p&gt;For almost two months, my media delivery path looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser
   ↓
Application / working server
   ↓
Nginx
   ↓
local proxy_cache
   ↓ cache MISS
Bunny Storage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Images, videos, and audio files lived in Bunny Storage.&lt;/p&gt;

&lt;p&gt;Users did &lt;strong&gt;not&lt;/strong&gt; access Bunny directly.&lt;/p&gt;

&lt;p&gt;My Nginx server requested objects from Bunny Storage, authenticated to Storage on the server side, cached the response on local SSD, and then served the public URL itself.&lt;/p&gt;

&lt;p&gt;On paper, this was a perfectly reasonable architecture.&lt;/p&gt;

&lt;p&gt;And when the requested file was already in the Nginx cache, it was extremely fast.&lt;/p&gt;

&lt;p&gt;The problem was everything that happened when it wasn't.&lt;/p&gt;

&lt;p&gt;After two months of production logs, connection measurements, Nginx configuration changes, MP4 Range experiments, and some very specific &lt;code&gt;ETag&lt;/code&gt; errors, I eventually replaced Bunny Storage as the runtime origin with a private media server connected over WireGuard.&lt;/p&gt;

&lt;p&gt;The biggest lesson was not that self-hosting is universally better.&lt;/p&gt;

&lt;p&gt;It was this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;A fast cache can hide a slow or unpredictable cold path for a surprisingly long time.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And for a media-heavy site, the cold path matters much more than I originally thought.&lt;/p&gt;




&lt;h2&gt;
  
  
  First, an important distinction: this was not Bunny CDN
&lt;/h2&gt;

&lt;p&gt;I want to make this explicit because otherwise the comparison would be misleading.&lt;/p&gt;

&lt;p&gt;I was &lt;strong&gt;not using Bunny CDN&lt;/strong&gt; for this experiment.&lt;/p&gt;

&lt;p&gt;I was using Bunny Storage directly as the storage origin behind my own Nginx proxy cache.&lt;/p&gt;

&lt;p&gt;So my architecture was essentially:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;My Nginx
   ↓
Bunny Storage API / storage endpoint
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;User
   ↓
Bunny CDN
   ↓
Bunny Storage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are different architectures.&lt;/p&gt;

&lt;p&gt;Bunny's normal Storage + Pull Zone/CDN setup adds a globally distributed delivery layer designed specifically for serving content to users.&lt;/p&gt;

&lt;p&gt;That was not what I was testing.&lt;/p&gt;

&lt;p&gt;I still think Bunny CDN is a very good product, and this article should not be read as "Bunny is bad."&lt;/p&gt;

&lt;p&gt;This is about a much narrower question:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What happens when Bunny Storage itself becomes the runtime origin behind your own Nginx cache?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For my workload, the answer became increasingly complicated.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why I built it this way in the first place
&lt;/h2&gt;

&lt;p&gt;My workload is mostly immutable media:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;AVIF&lt;/li&gt;
&lt;li&gt;JPEG&lt;/li&gt;
&lt;li&gt;PNG&lt;/li&gt;
&lt;li&gt;MP4&lt;/li&gt;
&lt;li&gt;WebM&lt;/li&gt;
&lt;li&gt;audio&lt;/li&gt;
&lt;li&gt;other static files&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There are a lot of files, with very different sizes.&lt;/p&gt;

&lt;p&gt;Some images are tiny.&lt;/p&gt;

&lt;p&gt;Some videos are much larger.&lt;/p&gt;

&lt;p&gt;I wanted storage separated from the working server, but I also wanted my own Nginx to control:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;public URLs&lt;/li&gt;
&lt;li&gt;cache headers&lt;/li&gt;
&lt;li&gt;cache eviction&lt;/li&gt;
&lt;li&gt;missing-file behavior&lt;/li&gt;
&lt;li&gt;redirects&lt;/li&gt;
&lt;li&gt;media fallbacks&lt;/li&gt;
&lt;li&gt;byte ranges&lt;/li&gt;
&lt;li&gt;video seeking&lt;/li&gt;
&lt;li&gt;HTTP status codes&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So I built a classic reverse-proxy cache:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    ┌── HIT ── local SSD
                    │
Browser → Nginx cache
                    │
                    └── MISS ── Bunny Storage
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The design had an obvious advantage.&lt;/p&gt;

&lt;p&gt;A popular file only had to come from Bunny once.&lt;/p&gt;

&lt;p&gt;After that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser
   ↓
Nginx
   ↓
local SSD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No external storage connection.&lt;/p&gt;

&lt;p&gt;No remote object transfer.&lt;/p&gt;

&lt;p&gt;Very fast.&lt;/p&gt;

&lt;p&gt;And that worked exactly as expected.&lt;/p&gt;

&lt;p&gt;The mistake was judging the architecture mostly by what happened on a HIT.&lt;/p&gt;




&lt;h1&gt;
  
  
  A cache HIT tells you almost nothing about your origin
&lt;/h1&gt;

&lt;p&gt;This sounds obvious now.&lt;/p&gt;

&lt;p&gt;It did not feel obvious while the system was running.&lt;/p&gt;

&lt;p&gt;A popular image gets requested frequently:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;request
↓
cache HIT
↓
local SSD
↓
fast
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A rarely visited image behaves differently:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;old page
↓
file was evicted
↓
cache MISS
↓
connect to Bunny Storage
↓
retrieve object
↓
populate cache
↓
respond
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This made older and less popular pages an accidental stress test of the real origin path.&lt;/p&gt;

&lt;p&gt;And that is exactly where I started noticing something strange.&lt;/p&gt;

&lt;p&gt;The popular pages felt fine.&lt;/p&gt;

&lt;p&gt;Older pages could feel dramatically worse.&lt;/p&gt;

&lt;p&gt;Sometimes one image would take much longer than everything around it.&lt;/p&gt;

&lt;p&gt;Sometimes an image would appear to hang.&lt;/p&gt;

&lt;p&gt;That pattern eventually made sense once I looked closely at the Nginx error logs.&lt;/p&gt;




&lt;h1&gt;
  
  
  The production logs showed that some requests were stalling before the file even started downloading
&lt;/h1&gt;

&lt;p&gt;The most important errors looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;upstream timed out
while connecting to upstream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and, even more interestingly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;upstream timed out ... while SSL handshaking to upstream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That distinction matters.&lt;/p&gt;

&lt;p&gt;This wasn't:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;large file
↓
slow download
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It was sometimes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cache MISS
↓
try to establish upstream connection
↓
TCP / TLS
↓
timeout
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The actual media body had barely entered the picture yet.&lt;/p&gt;

&lt;p&gt;Nginx was having trouble completing the connection to the remote storage endpoint.&lt;/p&gt;

&lt;p&gt;That matched what I was seeing in the browser.&lt;/p&gt;

&lt;p&gt;An image wasn't necessarily loading slowly.&lt;/p&gt;

&lt;p&gt;Sometimes the useful transfer had not really started.&lt;/p&gt;




&lt;h2&gt;
  
  
  This was not a one-request anomaly
&lt;/h2&gt;

&lt;p&gt;One diagnostic snapshot from July 30 contained:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;317
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;upstream-timeout matches in the last:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;5,000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;lines of the media error log.&lt;/p&gt;

&lt;p&gt;A snapshot from the previous day contained:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;578
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;matches.&lt;/p&gt;

&lt;p&gt;Those numbers are &lt;strong&gt;log matches&lt;/strong&gt;, not unique users or unique failed objects.&lt;/p&gt;

&lt;p&gt;I don't want to turn them into a metric they aren't.&lt;/p&gt;

&lt;p&gt;But they do establish something useful:&lt;/p&gt;

&lt;p&gt;This wasn't one unlucky request that happened to fail during debugging.&lt;/p&gt;

&lt;p&gt;The failure pattern was occurring repeatedly in production.&lt;/p&gt;




&lt;h1&gt;
  
  
  It also wasn't one permanently bad Bunny Storage IP
&lt;/h1&gt;

&lt;p&gt;During that period, the storage hostname resolved to addresses including:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;109.61.89.53
109.61.89.54
109.61.89.55
109.61.89.57
79.127.226.193
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I saw connection or TLS-related failures across different addresses in that set.&lt;/p&gt;

&lt;p&gt;That does &lt;strong&gt;not&lt;/strong&gt; tell me what the underlying root cause inside the network was.&lt;/p&gt;

&lt;p&gt;From my logs alone, I cannot honestly say whether it was:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a storage backend&lt;/li&gt;
&lt;li&gt;routing&lt;/li&gt;
&lt;li&gt;peering&lt;/li&gt;
&lt;li&gt;my hosting provider's path to Bunny&lt;/li&gt;
&lt;li&gt;balancing behavior&lt;/li&gt;
&lt;li&gt;another network component&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I didn't isolate those variables.&lt;/p&gt;

&lt;p&gt;What I &lt;em&gt;can&lt;/em&gt; say is much narrower:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;From my server, the external path to Bunny Storage occasionally failed to establish an upstream connection within the expected time.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That distinction matters.&lt;/p&gt;

&lt;p&gt;I don't need to know which router or backend caused the delay to know that it existed in my application's critical path.&lt;/p&gt;




&lt;h1&gt;
  
  
  One measurement showed exactly why averages were misleading
&lt;/h1&gt;

&lt;p&gt;I tested individual resolved Bunny Storage addresses.&lt;/p&gt;

&lt;p&gt;Several paths looked completely reasonable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;109.61.89.53     total ≈ 29.8 ms
109.61.89.54     total ≈ 28.5 ms
109.61.89.57     total ≈ 42.7 ms
79.127.226.193   total ≈ 43.6 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing alarming there.&lt;/p&gt;

&lt;p&gt;But one measured path looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;109.61.89.55

TCP    1.017782 s
TLS    1.048306 s
TOTAL  1.054474 s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That doesn't mean:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Bunny Storage has 1-second latency.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It clearly didn't.&lt;/p&gt;

&lt;p&gt;Most of my measured paths were tens of milliseconds.&lt;/p&gt;

&lt;p&gt;The interesting part was the variance.&lt;/p&gt;

&lt;p&gt;I could get something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;30 ms
29 ms
43 ms
1,054 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;from paths behind the same storage hostname.&lt;/p&gt;

&lt;p&gt;That one-second measurement happened &lt;strong&gt;before considering normal object-transfer time&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;And production logs also showed cases that went beyond "slow" and reached actual connection or TLS timeouts.&lt;/p&gt;

&lt;p&gt;This is where average latency becomes a dangerous metric.&lt;/p&gt;




&lt;h1&gt;
  
  
  For a gallery, p99 matters more than the average image
&lt;/h1&gt;

&lt;p&gt;Imagine a page loading 80 images.&lt;/p&gt;

&lt;p&gt;Suppose 79 of them load in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;30–50 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and one takes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1,000 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The average can still look respectable.&lt;/p&gt;

&lt;p&gt;The page does not.&lt;/p&gt;

&lt;p&gt;The user experiences:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;████ image
████ image
████ image
████ image

     [empty]

████ image
████ image
████ image
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If another object hits a multi-second timeout, the experience becomes even worse.&lt;/p&gt;

&lt;p&gt;The browser doesn't render "average latency."&lt;/p&gt;

&lt;p&gt;It renders individual objects.&lt;/p&gt;

&lt;p&gt;That's why media-heavy pages are especially sensitive to long-tail latency.&lt;/p&gt;

&lt;p&gt;What matters is often not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;mean request time
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;but:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;p95
p99
slowest cold object on the page
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This became one of the most useful performance lessons from the entire migration.&lt;/p&gt;




&lt;h1&gt;
  
  
  Why older content exposed the problem more often
&lt;/h1&gt;

&lt;p&gt;My old local Nginx media cache became large.&lt;/p&gt;

&lt;p&gt;One July snapshot showed roughly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cache size: ~36 GB
cache files: &amp;gt;355,000
configured limit: ~35 GB
root filesystem: ~93% used
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nearby snapshots approached:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~400,000 cache entries
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exact numbers changed as the cache evolved, but the important part was structural:&lt;/p&gt;

&lt;p&gt;The cache could not hold everything forever.&lt;/p&gt;

&lt;p&gt;So:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;popular media
↓
frequent access
↓
likely still cached
↓
fast HIT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;while:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;old / rare media
↓
less frequently accessed
↓
more likely evicted
↓
MISS
↓
remote origin exposed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That explains something I found confusing at first.&lt;/p&gt;

&lt;p&gt;Why could a site with a huge local cache still feel bad on random old pages?&lt;/p&gt;

&lt;p&gt;Because the cache was making the good cases &lt;em&gt;very&lt;/em&gt; good while doing nothing to eliminate the cost of the remaining cold misses.&lt;/p&gt;




&lt;h1&gt;
  
  
  Then video made the architecture much more complicated
&lt;/h1&gt;

&lt;p&gt;Images were only half the story.&lt;/p&gt;

&lt;p&gt;Video introduced a completely different problem.&lt;/p&gt;

&lt;p&gt;Browsers don't necessarily download an MP4 from byte zero to the end.&lt;/p&gt;

&lt;p&gt;They can request byte ranges.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;Range: bytes=0-1048575
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the user seeks somewhere else:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;Range: bytes=50000000-51048575
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server can reply:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;206 Partial Content
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and return only the requested bytes.&lt;/p&gt;

&lt;p&gt;This is essential for normal video seeking.&lt;/p&gt;

&lt;p&gt;It also makes caching much more interesting.&lt;/p&gt;




&lt;h1&gt;
  
  
  My first approach: include the browser Range in the cache key
&lt;/h1&gt;

&lt;p&gt;One early configuration effectively worked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;proxy_set_header&lt;/span&gt; &lt;span class="s"&gt;Range&lt;/span&gt; &lt;span class="nv"&gt;$http_range&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;proxy_set_header&lt;/span&gt; &lt;span class="s"&gt;If-Range&lt;/span&gt; &lt;span class="nv"&gt;$http_if_range&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_cache_key&lt;/span&gt; &lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$scheme&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$request_uri&lt;/span&gt;&lt;span class="s"&gt;|range=&lt;/span&gt;&lt;span class="nv"&gt;$http_range&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_cache_valid&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="mi"&gt;206&lt;/span&gt; &lt;span class="mi"&gt;301&lt;/span&gt; &lt;span class="mi"&gt;302&lt;/span&gt; &lt;span class="s"&gt;30d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This prevented two different byte ranges from incorrectly sharing one cache entry.&lt;/p&gt;

&lt;p&gt;But there was an obvious downside.&lt;/p&gt;

&lt;p&gt;Browser ranges are arbitrary.&lt;/p&gt;

&lt;p&gt;One client may request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;bytes=0-1048575
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;another:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;bytes=0-999999
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;another may seek and request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;bytes=58321473-59370048
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So one physical MP4 could become:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;video.mp4 + Range A → cache object A
video.mp4 + Range B → cache object B
video.mp4 + Range C → cache object C
video.mp4 + Range D → cache object D
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I didn't measure how many gigabytes this fragmentation consumed, so I'm not going to invent a number.&lt;/p&gt;

&lt;p&gt;But the fragmentation mechanism was directly visible in the cache key.&lt;/p&gt;

&lt;p&gt;This made me look for something more deterministic.&lt;/p&gt;




&lt;h1&gt;
  
  
  Nginx Slice looked like the right solution
&lt;/h1&gt;

&lt;p&gt;Nginx has a Slice module specifically for caching large resources in fixed-size segments.&lt;/p&gt;

&lt;p&gt;Instead of arbitrary browser ranges:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0–734129
917283–1500000
2000000–2768129
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I could normalize the cache into predictable pieces:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0–1 MB
1–2 MB
2–3 MB
3–4 MB
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The relevant configuration looked roughly like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;slice&lt;/span&gt; &lt;span class="mi"&gt;1m&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_set_header&lt;/span&gt; &lt;span class="s"&gt;Range&lt;/span&gt; &lt;span class="nv"&gt;$slice_range&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_cache_key&lt;/span&gt; &lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$scheme&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$uri&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$slice_range&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_cache_valid&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="mi"&gt;206&lt;/span&gt; &lt;span class="s"&gt;30d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This architecture is much cleaner for reuse.&lt;/p&gt;

&lt;p&gt;A user asks for bytes somewhere inside:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1–2 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nginx fetches and caches that normalized slice.&lt;/p&gt;

&lt;p&gt;Another user later requests a different sub-range inside the same chunk.&lt;/p&gt;

&lt;p&gt;The same cached slice can potentially satisfy it.&lt;/p&gt;

&lt;p&gt;Great.&lt;/p&gt;

&lt;p&gt;Except production started showing this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;etag mismatch in slice response while reading response header from upstream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That error became one of the most interesting parts of the whole investigation.&lt;/p&gt;




&lt;h1&gt;
  
  
  The video problem was ETag consistency, not "different signatures"
&lt;/h1&gt;

&lt;p&gt;When I first tried to describe the issue informally, I thought of the pieces as having different "signatures."&lt;/p&gt;

&lt;p&gt;That's not the correct technical description.&lt;/p&gt;

&lt;p&gt;The relevant HTTP value was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ETag
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AccessKey
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Not a signed URL.&lt;/p&gt;

&lt;p&gt;Not a cryptographic signature.&lt;/p&gt;

&lt;p&gt;An ETag is a validator for a particular representation of a resource.&lt;/p&gt;

&lt;p&gt;Conceptually, Nginx wants something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;slice 1
bytes 0–1 MB
ETag: A

slice 2
bytes 1–2 MB
ETag: A
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both pieces appear to belong to the same representation.&lt;/p&gt;

&lt;p&gt;Now imagine this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;slice 1
ETag: A

slice 2
ETag: B
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nginx has a very good reason not to blindly combine those pieces.&lt;/p&gt;

&lt;p&gt;They may represent different versions of the underlying file.&lt;/p&gt;

&lt;p&gt;If it continued anyway, the final response could theoretically become:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;first part of version A
+
second part of version B
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's corrupt data.&lt;/p&gt;

&lt;p&gt;So Nginx aborting the sliced transaction is protective behavior.&lt;/p&gt;




&lt;h1&gt;
  
  
  And this wasn't theoretical — I had the exact error in production
&lt;/h1&gt;

&lt;p&gt;On July 30, repeated slice subrequests for one MP4 generated:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;etag mismatch in slice response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;while requests were reaching storage addresses including:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;109.61.89.53
109.61.89.57
79.127.226.193
109.61.89.55
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On July 31, the same class of error appeared again for another MP4 while slice requests moved among addresses from the same storage pool.&lt;/p&gt;

&lt;p&gt;What does that prove?&lt;/p&gt;

&lt;p&gt;It proves:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Nginx received slice responses for the same MP4 whose validators were not consistent enough for Nginx to safely assemble them.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;What does it &lt;strong&gt;not&lt;/strong&gt; prove?&lt;/p&gt;

&lt;p&gt;It does not prove exactly &lt;em&gt;why&lt;/em&gt; the ETags differed.&lt;/p&gt;

&lt;p&gt;At the time I did not log the literal ETag returned by every individual subrequest.&lt;/p&gt;

&lt;p&gt;So I cannot honestly reconstruct the exact internal mechanism.&lt;/p&gt;

&lt;p&gt;The movement between different upstream addresses makes backend-response inconsistency a plausible explanation.&lt;/p&gt;

&lt;p&gt;But that's an inference.&lt;/p&gt;

&lt;p&gt;The production fact is the Nginx error itself.&lt;/p&gt;




&lt;h1&gt;
  
  
  This does NOT mean Bunny Storage cannot serve Range requests
&lt;/h1&gt;

&lt;p&gt;This distinction is important.&lt;/p&gt;

&lt;p&gt;Range requests themselves worked.&lt;/p&gt;

&lt;p&gt;I was receiving and caching:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;206 Partial Content
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So the lesson is not:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Bunny Storage does not support byte ranges.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The narrower issue was the combination of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Nginx Slice
+
multiple upstream responses
+
ETag consistency required by Nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And video could simultaneously suffer from the other problem:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;upstream timed out while connecting to upstream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So video had two independent failure modes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Origin connection / long-tail latency&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Slice / ETag consistency&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That made the overall system much harder to reason about.&lt;/p&gt;




&lt;h1&gt;
  
  
  I eventually stopped using Slice for MP4
&lt;/h1&gt;

&lt;p&gt;I later moved MP4 back toward ordinary browser Range behavior.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;proxy_set_header&lt;/span&gt; &lt;span class="s"&gt;Range&lt;/span&gt; &lt;span class="nv"&gt;$http_range&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;proxy_set_header&lt;/span&gt; &lt;span class="s"&gt;If-Range&lt;/span&gt; &lt;span class="nv"&gt;$http_if_range&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_cache_key&lt;/span&gt; &lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$scheme&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$uri&lt;/span&gt;&lt;span class="s"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_no_cache&lt;/span&gt; &lt;span class="nv"&gt;$http_range&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;proxy_cache_valid&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt; &lt;span class="s"&gt;30d&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Partial responses would no longer be allowed to masquerade as complete cached files.&lt;/p&gt;

&lt;p&gt;That removed the problematic sliced MP4 assembly.&lt;/p&gt;

&lt;p&gt;But it exposed a trade-off.&lt;/p&gt;

&lt;p&gt;A cold video Range might now require:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser
↓
Nginx
↓
MISS
↓
external storage origin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;again.&lt;/p&gt;

&lt;p&gt;I had fixed one layer of complexity by becoming more dependent on another.&lt;/p&gt;

&lt;p&gt;At some point I had to ask a more fundamental question.&lt;/p&gt;




&lt;h1&gt;
  
  
  I realized I was building part of a CDN in front of a storage service
&lt;/h1&gt;

&lt;p&gt;Look at the mechanisms I had accumulated:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;proxy_cache&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;cache locking&lt;/li&gt;
&lt;li&gt;stale responses&lt;/li&gt;
&lt;li&gt;background updates&lt;/li&gt;
&lt;li&gt;custom cache keys&lt;/li&gt;
&lt;li&gt;Range handling&lt;/li&gt;
&lt;li&gt;&lt;code&gt;206 Partial Content&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Nginx Slice&lt;/li&gt;
&lt;li&gt;normalized 1 MB chunks&lt;/li&gt;
&lt;li&gt;upstream keepalive&lt;/li&gt;
&lt;li&gt;TLS session reuse&lt;/li&gt;
&lt;li&gt;connection retries&lt;/li&gt;
&lt;li&gt;timeout tuning&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these are bad features.&lt;/p&gt;

&lt;p&gt;They are useful tools.&lt;/p&gt;

&lt;p&gt;But together they made me ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;What do I actually need from my origin?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;My answer was surprisingly boring.&lt;/p&gt;

&lt;p&gt;I needed something that could:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;store immutable files
+
return their bytes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's it.&lt;/p&gt;

&lt;p&gt;I wasn't trying to build a globally distributed public edge network between my two servers.&lt;/p&gt;

&lt;p&gt;I had one working server that needed media from one storage server.&lt;/p&gt;

&lt;p&gt;So I tried the simplest possible version of that architecture.&lt;/p&gt;




&lt;h1&gt;
  
  
  The replacement: a boring private media origin
&lt;/h1&gt;

&lt;p&gt;The new design is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser
   ↓
Working server
   ↓
Nginx + local media cache
   │
   ├── HIT
   │
   └── MISS
          ↓
      WireGuard
          ↓
     Media server
          ↓
        Nginx
          ↓
         SSD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The media server does almost nothing.&lt;/p&gt;

&lt;p&gt;It stores files.&lt;/p&gt;

&lt;p&gt;Nginx serves them.&lt;/p&gt;

&lt;p&gt;The origin listener is private.&lt;/p&gt;

&lt;p&gt;The working server connects to a fixed private address over WireGuard.&lt;/p&gt;

&lt;p&gt;Public HTTPS still terminates on the working server.&lt;/p&gt;

&lt;p&gt;Inside the encrypted WireGuard tunnel, the origin connection can be plain HTTP:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;http://private-ip:port
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I don't need a second TLS negotiation inside a tunnel that is already encrypted.&lt;/p&gt;




&lt;h1&gt;
  
  
  Why a tiny server is enough for this kind of job
&lt;/h1&gt;

&lt;p&gt;For a pure static-file origin, the application logic is basically nonexistent.&lt;/p&gt;

&lt;p&gt;There is no:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;SSR&lt;/li&gt;
&lt;li&gt;database&lt;/li&gt;
&lt;li&gt;application runtime&lt;/li&gt;
&lt;li&gt;authentication system&lt;/li&gt;
&lt;li&gt;template rendering&lt;/li&gt;
&lt;li&gt;expensive API logic&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The critical resources are much more boring:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;enough SSD capacity&lt;/li&gt;
&lt;li&gt;enough sequential/random disk performance&lt;/li&gt;
&lt;li&gt;enough network throughput&lt;/li&gt;
&lt;li&gt;enough file descriptors&lt;/li&gt;
&lt;li&gt;sensible Nginx connection limits&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For this kind of role, I'd consider a machine in the class of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~1 vCPU
~1 GB RAM
hundreds of GB of SSD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;as a reasonable starting point for a small static origin, assuming the expected bandwidth, concurrency, and disk behavior fit.&lt;/p&gt;

&lt;p&gt;That's not a universal sizing rule.&lt;/p&gt;

&lt;p&gt;My current validation host has more headroom, so the latency numbers below should &lt;strong&gt;not&lt;/strong&gt; be interpreted as a scientific benchmark of exactly a 1-vCPU / 1-GB machine.&lt;/p&gt;

&lt;p&gt;The point is architectural:&lt;/p&gt;

&lt;p&gt;Static file origins are cheap workloads compared with many application workloads.&lt;/p&gt;




&lt;h1&gt;
  
  
  The new cold path has far fewer moving parts
&lt;/h1&gt;

&lt;p&gt;The old MISS path looked approximately like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Working server
↓
public DNS
↓
one of several storage addresses
↓
public network path
↓
TCP connection
↓
TLS handshake
↓
Storage request
↓
object
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The new path looks like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Working server
↓
WireGuard
↓
fixed private IP
↓
Nginx
↓
SSD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The network still exists.&lt;/p&gt;

&lt;p&gt;Encryption still exists.&lt;/p&gt;

&lt;p&gt;Failure is still possible.&lt;/p&gt;

&lt;p&gt;But I removed variables that did not add much value to this specific one-to-one origin relationship.&lt;/p&gt;

&lt;p&gt;Most importantly, the cold path became predictable.&lt;/p&gt;




&lt;h1&gt;
  
  
  The difference was immediately visible in the measurements
&lt;/h1&gt;

&lt;p&gt;After the migration, small real-file Range requests from the working server to the media origin were generally around:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;CONNECT ≈ 9.5–12.5 ms
TTFB    ≈ 19–23 ms
TOTAL   ≈ 19–23 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I ran sequential and parallel checks.&lt;/p&gt;

&lt;p&gt;Results included:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10/10 successful
20/20 sequential successful
20/20 parallel successful
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the media origin itself, the same small local request typically took roughly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0.5–0.9 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That tells me something useful about where the ~20 ms comes from.&lt;/p&gt;

&lt;p&gt;The origin isn't spending 20 ms finding the file.&lt;/p&gt;

&lt;p&gt;Most of that time is the network path between the two machines.&lt;/p&gt;

&lt;p&gt;Which is exactly what I want from a boring static origin.&lt;/p&gt;




&lt;h2&gt;
  
  
  Important: this does not mean a full video downloads in 20 ms
&lt;/h2&gt;

&lt;p&gt;This distinction is easy to lose when discussing latency numbers.&lt;/p&gt;

&lt;p&gt;My:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~20 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;measurements were small Range/TTFB-style tests.&lt;/p&gt;

&lt;p&gt;They measure the cost of reaching the origin and starting to receive useful data.&lt;/p&gt;

&lt;p&gt;A:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;500 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;video obviously does not transfer completely in 20 ms.&lt;/p&gt;

&lt;p&gt;Full transfer time depends on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;file size
÷
available throughput
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The win here was not infinite bandwidth.&lt;/p&gt;

&lt;p&gt;The win was making the &lt;strong&gt;cold connection path cheap and stable&lt;/strong&gt;.&lt;/p&gt;




&lt;h1&gt;
  
  
  Comparing one observed old outlier with the new path
&lt;/h1&gt;

&lt;p&gt;Remember the old measurement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1.054474 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the new cold request:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~20 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mathematically:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1054 ms / 20 ms ≈ 52×
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That comparison is striking.&lt;/p&gt;

&lt;p&gt;But it needs to be stated correctly.&lt;/p&gt;

&lt;p&gt;I am &lt;strong&gt;not&lt;/strong&gt; claiming:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;My media server is 52× faster than Bunny Storage.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That would be a terrible benchmark.&lt;/p&gt;

&lt;p&gt;I am saying:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;One real old origin path I measured took about 1.054 seconds, while my current cold origin requests are around 20 ms.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Those are two observed network paths.&lt;/p&gt;

&lt;p&gt;The normal old Bunny measurements were much better:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;~29–44 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important difference was not merely the best case.&lt;/p&gt;

&lt;p&gt;It was the tail.&lt;/p&gt;

&lt;p&gt;I cared about eliminating this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;30 ms
31 ms
43 ms
1,054 ms
timeout
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and getting something much closer to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;20 ms
20 ms
21 ms
19 ms
22 ms
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For perceived web performance, reducing variance can matter more than improving an already-good average.&lt;/p&gt;




&lt;h1&gt;
  
  
  Something surprising happened: I no longer needed a huge cache to make old pages feel fast
&lt;/h1&gt;

&lt;p&gt;My old media cache had grown to roughly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;35–36 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and hundreds of thousands of cache entries.&lt;/p&gt;

&lt;p&gt;The new cache did not need to immediately grow anywhere near that size.&lt;/p&gt;

&lt;p&gt;At my latest check it was only around:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3.9 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;max_size = 25 GB
min_free = 8 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Yet old pages felt dramatically faster.&lt;/p&gt;

&lt;p&gt;That initially sounds backwards.&lt;/p&gt;

&lt;p&gt;The old cache was much larger.&lt;/p&gt;

&lt;p&gt;Shouldn't it have been better?&lt;/p&gt;

&lt;p&gt;The answer is that the two systems optimize different things.&lt;/p&gt;

&lt;p&gt;Old system:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HIT  = very fast
MISS = unpredictable
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;New system:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HIT  = very fast
MISS = also fast
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A larger cache reduces how frequently you miss.&lt;/p&gt;

&lt;p&gt;A good origin reduces the &lt;strong&gt;cost of every miss&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I had spent a lot of time optimizing the first variable.&lt;/p&gt;

&lt;p&gt;The second one turned out to matter more.&lt;/p&gt;




&lt;h1&gt;
  
  
  MP4 became simpler after the origin moved too
&lt;/h1&gt;

&lt;p&gt;With a private, predictable origin, I can use a much simpler MP4 strategy.&lt;/p&gt;

&lt;p&gt;For some MP4 files, I can let a cold request trigger a full-object cache fill:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cold MISS
↓
request full MP4 from private origin
↓
store one complete local cache object
↓
serve browser byte ranges locally
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once the complete MP4 exists locally, Nginx can satisfy normal browser Range requests from that cached file.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Browser
   ↓ Range
Working Nginx
   ↓
complete MP4 in local cache
   ↓
206 Partial Content
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No remote slice assembly.&lt;/p&gt;

&lt;p&gt;No arbitrary remote Range fragments becoming separate cache objects.&lt;/p&gt;

&lt;p&gt;No need to combine pieces fetched from different storage responses.&lt;/p&gt;

&lt;p&gt;One file.&lt;/p&gt;

&lt;p&gt;One cache entry.&lt;/p&gt;

&lt;p&gt;Normal local byte-range serving.&lt;/p&gt;




&lt;h1&gt;
  
  
  But full-object caching is not free either
&lt;/h1&gt;

&lt;p&gt;There is a trade-off here.&lt;/p&gt;

&lt;p&gt;Suppose the MP4 is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;500 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first viewer watches:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If my cache strategy fills the entire object, the origin may still send all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;500 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;to the working server.&lt;/p&gt;

&lt;p&gt;A pure Range request could have transferred far less.&lt;/p&gt;

&lt;p&gt;For my file sizes and access patterns, I accept that trade-off because subsequent seeking and subsequent viewers can reuse one simple local object.&lt;/p&gt;

&lt;p&gt;But I would not blindly use this design for every video workload.&lt;/p&gt;

&lt;p&gt;For a library of mostly cold, multi-gigabyte videos, I would evaluate things such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;stable cache slicing&lt;/li&gt;
&lt;li&gt;HLS&lt;/li&gt;
&lt;li&gt;DASH&lt;/li&gt;
&lt;li&gt;a real video CDN&lt;/li&gt;
&lt;li&gt;Bunny Stream&lt;/li&gt;
&lt;li&gt;another dedicated video platform&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Architecture follows workload.&lt;/p&gt;

&lt;p&gt;The important change is that I now get to make that decision based on video behavior, instead of also fighting an unpredictable origin path underneath it.&lt;/p&gt;




&lt;h1&gt;
  
  
  Nginx Slice is not the villain either
&lt;/h1&gt;

&lt;p&gt;I want to be equally careful here.&lt;/p&gt;

&lt;p&gt;Nginx Slice is a useful feature.&lt;/p&gt;

&lt;p&gt;It exists specifically because caching huge files as fixed byte ranges can be much more efficient than waiting for complete cache fills.&lt;/p&gt;

&lt;p&gt;For an immutable file and stable origin:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0–1 MB
1–2 MB
2–3 MB
...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;can be an excellent model.&lt;/p&gt;

&lt;p&gt;My problem wasn't:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Slice is broken.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;My problem was:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;My production slice requests encountered ETag inconsistencies across upstream responses.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Now, if I use slicing between my two servers, every slice comes from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;one origin
one Nginx
one filesystem
one file representation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is a much simpler system to debug.&lt;/p&gt;




&lt;h1&gt;
  
  
  One URL benchmark is not enough
&lt;/h1&gt;

&lt;p&gt;Another lesson from this migration is how easily origin testing can become unrealistic.&lt;/p&gt;

&lt;p&gt;Consider:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;curl same-file.avif
curl same-file.avif
curl same-file.avif
curl same-file.avif
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After the first request, you're often benchmarking:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cache
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;origin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Even &lt;code&gt;proxy_cache_lock&lt;/code&gt; only helps requests competing for the &lt;strong&gt;same cache key&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;If a page loads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;image-1.avif
image-2.avif
image-3.avif
...
image-50.avif
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and all 50 are cold, those are still 50 different objects.&lt;/p&gt;

&lt;p&gt;A realistic media-origin test should therefore include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;different files&lt;/li&gt;
&lt;li&gt;cold objects&lt;/li&gt;
&lt;li&gt;parallel requests&lt;/li&gt;
&lt;li&gt;small and large objects&lt;/li&gt;
&lt;li&gt;Range requests&lt;/li&gt;
&lt;li&gt;seeking patterns&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One cold URL and twenty distinct cold URLs are very different tests.&lt;/p&gt;




&lt;h1&gt;
  
  
  What I would monitor now
&lt;/h1&gt;

&lt;p&gt;If I were building a similar system again, my Nginx access log would include at least:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="nv"&gt;$upstream_addr&lt;/span&gt;
&lt;span class="nv"&gt;$upstream_connect_time&lt;/span&gt;
&lt;span class="nv"&gt;$upstream_header_time&lt;/span&gt;
&lt;span class="nv"&gt;$upstream_response_time&lt;/span&gt;
&lt;span class="nv"&gt;$upstream_cache_status&lt;/span&gt;
&lt;span class="nv"&gt;$request_time&lt;/span&gt;
&lt;span class="nv"&gt;$status&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then I would analyze:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HIT latency
MISS latency
MISS p95
MISS p99
upstream connect p95
upstream connect p99
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;separately.&lt;/p&gt;

&lt;p&gt;Do not average HIT and MISS together.&lt;/p&gt;

&lt;p&gt;They are different execution paths.&lt;/p&gt;

&lt;p&gt;A site can have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;95% extremely fast HITs
+
5% terrible MISSes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and still feel broken whenever a user happens to open the wrong page.&lt;/p&gt;




&lt;h1&gt;
  
  
  For video, I would test actual seeking behavior
&lt;/h1&gt;

&lt;p&gt;A successful request for:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;Range: bytes=0-1048575
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;does not prove that video delivery is healthy.&lt;/p&gt;

&lt;p&gt;I would test at least:&lt;/p&gt;

&lt;h3&gt;
  
  
  Beginning of the file
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;Range: bytes=0-1048575
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Somewhere in the middle
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;Range: bytes=50000000-51048575
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  End of the file
&lt;/h3&gt;

&lt;p&gt;A suffix or final-byte range.&lt;/p&gt;

&lt;h3&gt;
  
  
  Random seeking
&lt;/h3&gt;

&lt;p&gt;Several non-sequential ranges.&lt;/p&gt;

&lt;p&gt;And for each, inspect:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HTTP status
206 Partial Content
Content-Range
Content-Length
Accept-Ranges
ETag
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If using Slice, I would explicitly compare validators between multiple slices.&lt;/p&gt;

&lt;p&gt;That test would have exposed my MP4 problem much earlier.&lt;/p&gt;




&lt;h1&gt;
  
  
  Production traffic also showed what a warm cache should look like
&lt;/h1&gt;

&lt;p&gt;One useful sanity check was comparing public traffic from the working server with traffic arriving from the media origin.&lt;/p&gt;

&lt;p&gt;In one 60-second production snapshot:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;public traffic to users: ~96.84 Mbit/s
traffic from media origin: ~7.07 Mbit/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;During that measurement window there were no new:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;502
503
504
upstream timeout
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;errors.&lt;/p&gt;

&lt;p&gt;In another 30-second snapshot:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;public TX:      61.36 Mbit/s
origin RX:       1.40 Mbit/s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The byte ratio in that window was approximately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2.28%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;origin traffic relative to public TX.&lt;/p&gt;

&lt;p&gt;I would &lt;strong&gt;not&lt;/strong&gt; call the inverse of that number a cache-hit ratio.&lt;/p&gt;

&lt;p&gt;That would be incorrect.&lt;/p&gt;

&lt;p&gt;Public traffic contains more than media, and a byte ratio is not a request ratio.&lt;/p&gt;

&lt;p&gt;But it does demonstrate the architectural behavior I wanted:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;users consume a lot of media
↓
working server serves mostly cached bytes
↓
origin only supplies a much smaller cold-fill stream
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The media server is acting like an origin, not like a second copy of all public traffic.&lt;/p&gt;




&lt;h1&gt;
  
  
  Self-hosting did not magically remove infrastructure problems
&lt;/h1&gt;

&lt;p&gt;Moving the media origin to my own server gives me control.&lt;/p&gt;

&lt;p&gt;It also gives me responsibility.&lt;/p&gt;

&lt;p&gt;Now I have to care about:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;disk health&lt;/li&gt;
&lt;li&gt;backups&lt;/li&gt;
&lt;li&gt;restore procedures&lt;/li&gt;
&lt;li&gt;free space&lt;/li&gt;
&lt;li&gt;filesystem health&lt;/li&gt;
&lt;li&gt;Nginx updates&lt;/li&gt;
&lt;li&gt;firewall rules&lt;/li&gt;
&lt;li&gt;connection limits&lt;/li&gt;
&lt;li&gt;monitoring&lt;/li&gt;
&lt;li&gt;redundancy&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A single private media origin is also a single point of failure for uncached objects.&lt;/p&gt;

&lt;p&gt;If it disappears:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cached media → may continue working
new MISS     → cannot fill
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If I need high availability, I need another origin, replication, failover, or another layer entirely.&lt;/p&gt;

&lt;p&gt;So this is not:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;VPS good, managed storage bad.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It's a trade-off.&lt;/p&gt;

&lt;p&gt;For my workload, I preferred operational responsibility over origin unpredictability.&lt;/p&gt;

&lt;p&gt;Another team may reasonably make the opposite choice.&lt;/p&gt;




&lt;h1&gt;
  
  
  When I would still choose Bunny
&lt;/h1&gt;

&lt;p&gt;There are several situations where I would absolutely consider Bunny again.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;globally distributed users&lt;/li&gt;
&lt;li&gt;managed edge delivery&lt;/li&gt;
&lt;li&gt;managed redundancy&lt;/li&gt;
&lt;li&gt;rapidly increasing storage&lt;/li&gt;
&lt;li&gt;minimal infrastructure administration&lt;/li&gt;
&lt;li&gt;large geographic distance between users and my servers&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;And in that case, I would evaluate Bunny the way its platform is intended to be used for delivery:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Storage
+
Pull Zone / CDN
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;rather than assuming my:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Nginx
↓
Storage API
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;architecture represents Bunny's best delivery model.&lt;/p&gt;

&lt;p&gt;It doesn't.&lt;/p&gt;

&lt;p&gt;That's why I don't think "Bunny vs VPS" is the right framing.&lt;/p&gt;

&lt;p&gt;The more accurate comparison is:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Bunny Storage as my private Nginx origin vs a private server as my Nginx origin.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For that very specific role, my private origin fits my workload better.&lt;/p&gt;




&lt;h1&gt;
  
  
  The real lesson wasn't about Bunny
&lt;/h1&gt;

&lt;p&gt;Over those two months I tuned:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cache size
cache keys
Range behavior
206 handling
Nginx Slice
1 MB chunks
keepalive
TLS reuse
retry behavior
stale responses
connection timeouts
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;All of those were legitimate engineering changes.&lt;/p&gt;

&lt;p&gt;But the change that improved the system the most was much simpler:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;I changed the origin.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That changed the question I now ask whenever I design a cache.&lt;/p&gt;

&lt;p&gt;I used to start with:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;How do I maximize my cache-hit ratio?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Now I start with:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;What happens when the object is not in cache?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If the answer is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;stable origin
predictable TTFB
correct Range behavior
consistent validators
understandable failure modes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;then cache is an optimization.&lt;/p&gt;

&lt;p&gt;Great.&lt;/p&gt;

&lt;p&gt;But if the architecture only feels fast because users usually avoid the cold path, the cache may be hiding the real problem.&lt;/p&gt;




&lt;h1&gt;
  
  
  A cache should make a good origin cheaper, not make a bad cold path invisible
&lt;/h1&gt;

&lt;p&gt;That is the mental model I kept from this migration.&lt;/p&gt;

&lt;p&gt;A good architecture can look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HIT
↓
very fast

MISS
↓
still fast enough
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Instead of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HIT
↓
excellent

MISS
↓
please don't happen
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Today, I can open old pages containing media that almost certainly wasn't requested recently, and the experience is still fast.&lt;/p&gt;

&lt;p&gt;Sometimes the object is cached.&lt;/p&gt;

&lt;p&gt;Sometimes it isn't.&lt;/p&gt;

&lt;p&gt;From the browser, the difference is often difficult to notice.&lt;/p&gt;

&lt;p&gt;That is exactly what I wanted.&lt;/p&gt;

&lt;p&gt;The private origin isn't a CDN.&lt;/p&gt;

&lt;p&gt;It isn't magic.&lt;/p&gt;

&lt;p&gt;It doesn't eliminate operational responsibility.&lt;/p&gt;

&lt;p&gt;It's just:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WireGuard
↓
Nginx
↓
SSD
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And after two months of tracing connection timeouts, inspecting cold cache misses, experimenting with MP4 byte ranges, and staring at:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;etag mismatch in slice response
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;making the cold path boring turned out to be the biggest performance improvement of all.&lt;/p&gt;

</description>
      <category>nginx</category>
      <category>webperf</category>
      <category>devops</category>
      <category>mp4</category>
    </item>
    <item>
      <title>I Raised My WebP AVIF SSIMULACRA2 Target to 65. The Encoder Still Passed at 60</title>
      <dc:creator>Sergei Solod</dc:creator>
      <pubDate>Thu, 27 Aug 2026 08:49:29 +0000</pubDate>
      <link>https://dev.to/sergeisolod/i-raised-my-webp-avif-ssimulacra2-target-to-65-the-encoder-still-passed-at-60-4c8p</link>
      <guid>https://dev.to/sergeisolod/i-raised-my-webp-avif-ssimulacra2-target-to-65-the-encoder-still-passed-at-60-4c8p</guid>
      <description>&lt;p&gt;I thought I was changing one threshold.&lt;/p&gt;

&lt;p&gt;Instead, I found two different problems:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;I was measuring a second-generation encode against a reference that had already been compressed.&lt;/li&gt;
&lt;li&gt;My code could report the stricter threshold correctly while the adaptive encoder still made its decision using the old global one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The first problem changed how I think about perceptual metrics.&lt;/p&gt;

&lt;p&gt;The second changed how I design validation logic.&lt;/p&gt;

&lt;p&gt;My original AVIF policy was simple:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;target = 60
single allowed worst sample = 58
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For small image sets, every representative sample had to reach 60.&lt;/p&gt;

&lt;p&gt;For larger sets, I allowed one sample to fall as low as 58 while requiring every other sample to remain at 60 or above.&lt;/p&gt;

&lt;p&gt;That worked well enough when I was encoding from the source I actually cared about.&lt;/p&gt;

&lt;p&gt;Then the input started looking like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;higher-quality original: ~2 MB
          ↓
      lossy WebP: ~100 KB
          ↓
          AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The original was gone.&lt;/p&gt;

&lt;p&gt;The WebP was all I had.&lt;/p&gt;

&lt;p&gt;The obvious question was:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;If SSIMULACRA2 60 is acceptable for source → AVIF, why shouldn't the same 60 be acceptable for lossy WebP → AVIF?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The answer depends on what SSIMULACRA2 can actually see.&lt;/p&gt;




&lt;h2&gt;
  
  
  SSIMULACRA2 measures the reference you give it
&lt;/h2&gt;

&lt;p&gt;SSIMULACRA2 is a full-reference perceptual image metric.&lt;/p&gt;

&lt;p&gt;Conceptually, it receives:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;reference
candidate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and estimates the perceptual difference between those two images.&lt;/p&gt;

&lt;p&gt;Its published quality scale includes approximate anchors around:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;50  ≈ medium / fair
70  ≈ high / good
80  ≈ very high
85  ≈ excellent
90  ≈ visually lossless
100 = mathematically lossless
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These are useful orientation points, not universal encoding thresholds.&lt;/p&gt;

&lt;p&gt;The critical word is &lt;strong&gt;reference&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Suppose I still have the real source:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original
   ↓
AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The comparison is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original ↔ AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the AVIF scores 60, the metric is describing the perceptual change between the source I care about and the output I am about to ship.&lt;/p&gt;

&lt;p&gt;That is exactly the question I want answered.&lt;/p&gt;

&lt;p&gt;Now consider this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original
   ↓ lossy encode
WebP
   ↓ lossy encode
AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the original no longer exists, the metric sees only:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WebP ↔ AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first lossy conversion is no longer part of the comparison.&lt;/p&gt;

&lt;p&gt;That loss has not disappeared from the pixels.&lt;/p&gt;

&lt;p&gt;It has disappeared from the &lt;strong&gt;measurement history&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That distinction is the foundation of the whole problem.&lt;/p&gt;




&lt;h2&gt;
  
  
  Once the derivative becomes the reference, its defects become reference content
&lt;/h2&gt;

&lt;p&gt;Imagine the original contains a perfectly smooth gradient.&lt;/p&gt;

&lt;p&gt;The first lossy encoder introduces a little banding.&lt;/p&gt;

&lt;p&gt;The WebP still looks acceptable.&lt;/p&gt;

&lt;p&gt;Now that WebP becomes the reference for the second conversion.&lt;/p&gt;

&lt;p&gt;From SSIMULACRA2's perspective, the existing banding is no longer an error.&lt;/p&gt;

&lt;p&gt;It is part of the reference.&lt;/p&gt;

&lt;p&gt;If the AVIF introduces more banding, more blur, ringing, edge damage, or other distortion, the metric can react to that &lt;strong&gt;additional&lt;/strong&gt; change.&lt;/p&gt;

&lt;p&gt;But it cannot say:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This region already lost information one generation ago.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The earlier image is not available.&lt;/p&gt;

&lt;p&gt;So a high score has different meanings depending on the reference.&lt;/p&gt;

&lt;p&gt;For a direct encode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;high score
≈
close to the real source
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a second-generation encode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;high score
≈
close to the already-compressed derivative
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those statements are not equivalent.&lt;/p&gt;

&lt;p&gt;This does not mean WebP→AVIF transcoding is inherently bad.&lt;/p&gt;

&lt;p&gt;It does not mean AVIF automatically amplifies existing artifacts.&lt;/p&gt;

&lt;p&gt;It means the second encoder is operating on pixels that may already contain irreversible decisions made by the first encoder.&lt;/p&gt;




&lt;h2&gt;
  
  
  Source provenance matters more than the extension
&lt;/h2&gt;

&lt;p&gt;This led me away from rules based purely on &lt;code&gt;.webp&lt;/code&gt;, &lt;code&gt;.jpg&lt;/code&gt;, or &lt;code&gt;.png&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The property I really care about is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;What happened to these pixels before they reached me?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A simplistic implementation would be:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;extension&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.webp&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;target&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I no longer like that rule.&lt;/p&gt;

&lt;p&gt;A better model is closer to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;isKnownLossyDerivative&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;target&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;target&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why?&lt;/p&gt;

&lt;p&gt;Because WebP itself can be lossless.&lt;/p&gt;

&lt;p&gt;And there is an important subtlety here:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;lossless WebP does not automatically mean clean provenance.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Lossless WebP means that the WebP encoding itself can reproduce its input pixels exactly.&lt;/p&gt;

&lt;p&gt;But imagine this chain:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original
   ↓
lossy JPEG
   ↓
lossless WebP
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The WebP step did not introduce new loss.&lt;/p&gt;

&lt;p&gt;But the pixels inside the WebP already came from a lossy derivative.&lt;/p&gt;

&lt;p&gt;So the strongest rule is not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;lossless WebP = clean source
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;judge the image by the best provenance information available
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For my operational policy, that means roughly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;canonical / trusted source
→ 60 / 58

known already-lossy derivative
→ 65 / 63
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A lossless WebP created directly from trusted source pixels can stay on the normal policy.&lt;/p&gt;

&lt;p&gt;A lossless WebP wrapping pixels that are already known to be a downstream lossy derivative should not magically erase that history.&lt;/p&gt;

&lt;p&gt;The container is not the provenance.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why I use 65 instead of 60
&lt;/h2&gt;

&lt;p&gt;This part needs a disclaimer.&lt;/p&gt;

&lt;p&gt;There is no official SSIMULACRA2 rule saying:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;second lossy generation
=
add exactly 5 points
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is no formula where:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;60 + previous compression = 65
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;SSIMULACRA2 scores are not a linear distortion budget that can be added across generations.&lt;/p&gt;

&lt;p&gt;So 65 is not a scientific constant.&lt;/p&gt;

&lt;p&gt;It is an engineering policy.&lt;/p&gt;

&lt;p&gt;My reasoning is simpler.&lt;/p&gt;

&lt;p&gt;A target of 60 is already more aggressive than the published high/good anchor around 70.&lt;/p&gt;

&lt;p&gt;For a first encode from a trusted source, I am willing to spend that quality budget because reducing bytes is part of the objective.&lt;/p&gt;

&lt;p&gt;For a source that I know has already gone through lossy compression, I want the next encoder to have &lt;strong&gt;less room to introduce additional distortion&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I could use 70.&lt;/p&gt;

&lt;p&gt;That would be more conservative.&lt;/p&gt;

&lt;p&gt;But it would also push more outputs toward higher AVIF qualities and larger files.&lt;/p&gt;

&lt;p&gt;Without controlled evidence that 70 produces enough additional visual benefit for my workload to justify those bytes, I do not want to force it globally.&lt;/p&gt;

&lt;p&gt;So I currently use:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;trusted source
target = 60

known lossy derivative
target = 65
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I think of 65 as a guardrail.&lt;/p&gt;

&lt;p&gt;Not a law.&lt;/p&gt;




&lt;h2&gt;
  
  
  Why 65/63 instead of 65/62
&lt;/h2&gt;

&lt;p&gt;The original rule was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;target = 60
floor  = 58
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The one permitted outlier therefore had a two-point allowance:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;60 - 58 = 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If I raise the normal target to 65 while preserving the same semantics:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;65 - 63 = 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So the corresponding policy becomes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;target = 65
floor  = 63
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Using 62 would change the exception to three points:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;65 - 62 = 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That would be odd.&lt;/p&gt;

&lt;p&gt;I would be saying:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This source is already lossy, so I want to be stricter.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;while also saying:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The worst sample is now allowed to miss its target by more than before.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;I could not justify that.&lt;/p&gt;

&lt;p&gt;Neither 65 nor 63 is magical.&lt;/p&gt;

&lt;p&gt;The useful property is preserving the meaning of the existing policy.&lt;/p&gt;




&lt;h2&gt;
  
  
  File size cannot tell me how much perceptual damage already exists
&lt;/h2&gt;

&lt;p&gt;A transformation like this looks dramatic:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2 MB → 100 KB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is roughly a 20× reduction.&lt;/p&gt;

&lt;p&gt;It is tempting to turn that number into a quality judgment.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;compressionRatio&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;target&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I do not think that is defensible.&lt;/p&gt;

&lt;p&gt;Compression efficiency depends on the content.&lt;/p&gt;

&lt;p&gt;A clean illustration with flat regions can sometimes compress extremely well.&lt;/p&gt;

&lt;p&gt;A noisy photograph may behave very differently.&lt;/p&gt;

&lt;p&gt;Gradients, line art, texture, resolution, chroma, entropy, encoder settings, and source format all matter.&lt;/p&gt;

&lt;p&gt;So:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;compression ratio
!=
perceptual quality
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The question I care about is not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;How impressive was the byte reduction?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Do I know that this image has already passed through a lossy generation?
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;






&lt;h2&gt;
  
  
  Then I found the bug that mattered more than 60 vs. 65
&lt;/h2&gt;

&lt;p&gt;Once I decided on the stricter policy, I expected the implementation change to be trivial.&lt;/p&gt;

&lt;p&gt;The code already had the concept of source-specific targets.&lt;/p&gt;

&lt;p&gt;Conceptually:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;SOURCE_TARGET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;LOSSY_DERIVATIVE_TARGET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There was also logic capable of reporting the correct target for an individual sample.&lt;/p&gt;

&lt;p&gt;At first glance, everything looked fine.&lt;/p&gt;

&lt;p&gt;But the adaptive search that actually chose the AVIF quality still used a global acceptance condition equivalent to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;GLOBAL_TARGET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;GLOBAL_FLOOR&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;58&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nx"&gt;passed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="nx"&gt;worstScore&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;GLOBAL_FLOOR&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
  &lt;span class="nx"&gt;secondWorstScore&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;GLOBAL_TARGET&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The stricter target appeared later in the reporting layer.&lt;/p&gt;

&lt;p&gt;So the system could effectively produce this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sample:
known lossy derivative

required target:
65

actual score:
61.2

log:
BELOW TARGET
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;while the adaptive search itself still concluded:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;61.2 &amp;gt;= 60

PASS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The logs knew about 65.&lt;/p&gt;

&lt;p&gt;The encoder was still optimizing against 60.&lt;/p&gt;

&lt;p&gt;That was the real bug.&lt;/p&gt;

&lt;p&gt;Nothing crashed.&lt;/p&gt;

&lt;p&gt;There was no exception.&lt;/p&gt;

&lt;p&gt;The build remained green.&lt;/p&gt;

&lt;p&gt;The output image looked plausible.&lt;/p&gt;

&lt;p&gt;The system could even print the correct stricter threshold.&lt;/p&gt;

&lt;p&gt;But the code responsible for selecting the output was solving a different problem.&lt;/p&gt;




&lt;h2&gt;
  
  
  A threshold that only exists in reporting is not a threshold
&lt;/h2&gt;

&lt;p&gt;That bug gave me a rule I now use beyond image compression:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A configuration value only matters if it reaches the decision that chooses the output.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Defining this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;LOSSY_TARGET&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;does not enforce anything.&lt;/p&gt;

&lt;p&gt;Logging this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;required target: 65
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;does not enforce anything either.&lt;/p&gt;

&lt;p&gt;The value has to participate in the actual PASS/FAIL predicate used by the adaptive search.&lt;/p&gt;

&lt;p&gt;So instead of keeping the quality policy far away from the measurement, I prefer attaching it directly to each sample.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;policyForSample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;isKnownLossyDerivative&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;63&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;58&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then every measured result can be assessed against its own policy:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;assessSample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;policy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;policyForSample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;target&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;target&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;targetMargin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;score&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;target&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;floorMargin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;score&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;meetsTarget&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;score&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;target&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

    &lt;span class="na"&gt;meetsFloor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;score&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the measurement carries its own context:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;score
target
floor
target margin
floor margin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That makes the next step much safer.&lt;/p&gt;




&lt;h2&gt;
  
  
  The one-outlier rule should be expressed directly
&lt;/h2&gt;

&lt;p&gt;Suppose the policy is:&lt;/p&gt;

&lt;p&gt;For smaller collections:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;every sample must reach its own target
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For larger collections:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;at most one sample may miss its target

but

no sample may fall below its own floor
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then I would implement exactly those semantics instead of reconstructing them indirectly from global score ordering.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;evaluateCandidate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;allowSingleOutlier&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;assessed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
    &lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;assessSample&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;belowFloor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
    &lt;span class="nx"&gt;assessed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;meetsFloor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;belowFloor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;passed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;below-floor&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nx"&gt;assessed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;belowTarget&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
    &lt;span class="nx"&gt;assessed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;item&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;meetsTarget&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;allowedMisses&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
    &lt;span class="nx"&gt;allowSingleOutlier&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;passed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
    &lt;span class="nx"&gt;belowTarget&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nx"&gt;allowedMisses&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;passed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;passed&lt;/span&gt;
      &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;passed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;
      &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;too-many-target-misses&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nx"&gt;assessed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rule is visible in the code.&lt;/p&gt;

&lt;p&gt;There is no hidden assumption that every sample shares one target.&lt;/p&gt;




&lt;h2&gt;
  
  
  Mixed thresholds break raw "worst score" ordering
&lt;/h2&gt;

&lt;p&gt;This was another important consequence.&lt;/p&gt;

&lt;p&gt;Imagine:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Sample A

score  = 63
target = 60
margin = +3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Sample B

score  = 64
target = 65
margin = -1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If I sort by raw SSIMULACRA2 score:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;63 &amp;lt; 64
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Sample A looks worse.&lt;/p&gt;

&lt;p&gt;But relative to the actual requirements:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Sample A
PASS

Sample B
FAIL
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The meaningful quantity is no longer just:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;score
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is closer to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;score - target
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A normalized target margin makes heterogeneous policies comparable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;targetMargin&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="nx"&gt;score&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;target&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;margin &amp;gt;= 0
→ target passed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;margin &amp;lt; 0
→ target missed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The floor can be represented the same way:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;floorMargin&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
  &lt;span class="nx"&gt;score&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The one permitted outlier then has a very clear state:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;targetMargin &amp;lt; 0
floorMargin &amp;gt;= 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is much safer than sorting raw scores and comparing the two lowest values against global constants.&lt;/p&gt;




&lt;h2&gt;
  
  
  The adaptive search must use the same predicate
&lt;/h2&gt;

&lt;p&gt;Fixing final validation is not enough.&lt;/p&gt;

&lt;p&gt;Suppose the encoder searches for the minimum passing AVIF quality:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;q=35
q=43
q=39
q=41
q=40
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The definition of &lt;strong&gt;passing&lt;/strong&gt; drives the entire search.&lt;/p&gt;

&lt;p&gt;The correct flow is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;encode candidate
      ↓
measure samples
      ↓
apply per-sample policies
      ↓
PASS / FAIL
      ↓
choose next quality
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The dangerous version is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;encode candidate
      ↓
compare everything to global 60
      ↓
choose next quality
      ↓
later report that some samples wanted 65
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the search predicate is wrong, correct reporting afterward cannot repair the quality that was already selected.&lt;/p&gt;




&lt;h2&gt;
  
  
  Historical quality predictions need policy versioning too
&lt;/h2&gt;

&lt;p&gt;There is another subtle issue if the encoder learns from previous runs.&lt;/p&gt;

&lt;p&gt;Suppose historical data says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;similar images usually pass around q=38
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That prediction only makes sense together with the policy under which those images passed.&lt;/p&gt;

&lt;p&gt;If the historical runs used:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;60 / 58
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the current source now requires:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;65 / 63
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;the old history is solving an easier problem.&lt;/p&gt;

&lt;p&gt;A correct final acceptance predicate still protects correctness.&lt;/p&gt;

&lt;p&gt;The search may simply start too low and climb.&lt;/p&gt;

&lt;p&gt;But efficiency suffers, and debugging becomes harder.&lt;/p&gt;

&lt;p&gt;So I prefer making the policy part of the history version.&lt;/p&gt;

&lt;p&gt;For example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sim-v1-60-58
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;becomes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sim-v2-source60-58-lossy65-63
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is not glamorous.&lt;/p&gt;

&lt;p&gt;It makes optimization results reproducible.&lt;/p&gt;




&lt;h2&gt;
  
  
  If the real original exists, I skip the second lossy generation
&lt;/h2&gt;

&lt;p&gt;Whenever I still have the true source, the best solution is simpler.&lt;/p&gt;

&lt;p&gt;Suppose I have:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original: 2 MB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WebP derivative: 100 KB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I would rather do:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original
   ↓
AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;than:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;original
   ↓
WebP
   ↓
AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The original becomes the reference again.&lt;/p&gt;

&lt;p&gt;A stricter second-generation threshold cannot recover information that disappeared during the first lossy encode.&lt;/p&gt;

&lt;p&gt;This distinction matters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WebP → AVIF at target 65
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;means:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;preserve this WebP more faithfully
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It does not mean:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;reconstruct the image that existed
before the WebP was created
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once information is gone from the pixels, SSIMULACRA2 cannot bring it back.&lt;/p&gt;

&lt;p&gt;Neither can a higher AVIF quality setting.&lt;/p&gt;




&lt;h2&gt;
  
  
  Passing the quality gate does not mean transcoding is worth it
&lt;/h2&gt;

&lt;p&gt;There is another decision SSIMULACRA2 cannot make for me.&lt;/p&gt;

&lt;p&gt;Suppose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;source WebP = 100 KB
AVIF        = 96 KB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the AVIF passes 65/63.&lt;/p&gt;

&lt;p&gt;The quality requirement is satisfied.&lt;/p&gt;

&lt;p&gt;But I saved only 4 KB.&lt;/p&gt;

&lt;p&gt;For that saving I added another encoding operation, another lossy generation, processing cost, and another transformation path to maintain.&lt;/p&gt;

&lt;p&gt;Maybe the correct optimization is to keep the WebP.&lt;/p&gt;

&lt;p&gt;Now consider:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;source WebP = 100 KB
AVIF        = 65 KB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;with the same perceptual policy still passing.&lt;/p&gt;

&lt;p&gt;That is a much more interesting trade-off.&lt;/p&gt;

&lt;p&gt;So I separate two questions:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quality gate:&lt;/strong&gt; Is the additional distortion acceptable?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Size gate:&lt;/strong&gt; Is the byte saving large enough to justify another encode?&lt;/p&gt;

&lt;p&gt;A candidate can pass the first and still fail the second.&lt;/p&gt;




&lt;h2&gt;
  
  
  What would actually validate the 65 threshold?
&lt;/h2&gt;

&lt;p&gt;The strongest experiment requires preserved originals.&lt;/p&gt;

&lt;p&gt;I would build a representative corpus and compare these chains:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A
original → AVIF
target 60

B
original → lossy WebP → AVIF
target 60

C
original → lossy WebP → AVIF
target 63

D
original → lossy WebP → AVIF
target 65

E
original → lossy WebP → AVIF
target 70
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For each final image I would record:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;final byte size

selected AVIF quality

SSIMULACRA2:
WebP → final AVIF

SSIMULACRA2:
true original → final AVIF
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last comparison is the important one.&lt;/p&gt;

&lt;p&gt;It restores the context that disappears when the WebP derivative becomes the only available reference.&lt;/p&gt;

&lt;p&gt;I would also manually inspect difficult content: smooth gradients, fine line art, small text, dense texture, dark gradients, saturated transitions, and sources that already contain visible compression artifacts.&lt;/p&gt;

&lt;p&gt;I have &lt;strong&gt;not&lt;/strong&gt; run that controlled experiment across a sufficiently representative corpus of preserved originals.&lt;/p&gt;

&lt;p&gt;So I cannot honestly claim:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;65 is scientifically optimal
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What I can defend is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;For a known already-lossy derivative,
I want a stricter additional-loss budget
than I use for a trusted source.

My current policy is 65/63.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is an engineering policy.&lt;/p&gt;

&lt;p&gt;Not a universal SSIMULACRA2 rule.&lt;/p&gt;




&lt;h2&gt;
  
  
  The rule I use now
&lt;/h2&gt;

&lt;p&gt;My current mental model is:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Trusted / canonical source:&lt;/strong&gt; &lt;code&gt;60 / 58&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Known already-lossy derivative:&lt;/strong&gt; &lt;code&gt;65 / 63&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lossless encoding does not erase earlier provenance:&lt;/strong&gt; classify the pixels by their known history&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Per-sample policy must participate in the real PASS/FAIL predicate&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Mixed thresholds should be evaluated by policy margin, not raw score ordering&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Adaptive search and historical prediction must understand the same policy&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;If the true original exists, encode from it&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;If AVIF barely saves bytes, consider keeping the current format&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The exact thresholds may change as better controlled data becomes available.&lt;/p&gt;

&lt;p&gt;The architecture should not.&lt;/p&gt;




&lt;h2&gt;
  
  
  The metric was not the bug
&lt;/h2&gt;

&lt;p&gt;SSIMULACRA2 was doing exactly what it was supposed to do.&lt;/p&gt;

&lt;p&gt;It compared the reference I gave it with the candidate I gave it.&lt;/p&gt;

&lt;p&gt;My first mistake was expecting that score to describe quality loss that happened before the reference existed.&lt;/p&gt;

&lt;p&gt;My second mistake was adding a stricter threshold without tracing whether that value actually reached the acceptance condition used by the adaptive search.&lt;/p&gt;

&lt;p&gt;Both mistakes came from the same habit:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;looking at a number without tracing the decision path around it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A perceptual score has context.&lt;/p&gt;

&lt;p&gt;A reference has history.&lt;/p&gt;

&lt;p&gt;A threshold has semantics.&lt;/p&gt;

&lt;p&gt;And a configuration value only matters if the code that chooses the output actually obeys it.&lt;/p&gt;

&lt;p&gt;For me, that lesson ended up being more important than whether the threshold was 60, 63, or 65.&lt;/p&gt;

</description>
      <category>webperf</category>
      <category>performance</category>
      <category>javascript</category>
      <category>webdev</category>
    </item>
    <item>
      <title>DeepSeek V4 Flash Got Expensive. I Kept the Model and Cut the API Cost Anyway</title>
      <dc:creator>Sergei Solod</dc:creator>
      <pubDate>Mon, 17 Aug 2026 13:31:20 +0000</pubDate>
      <link>https://dev.to/sergeisolod/deepseek-v4-flash-got-expensive-i-kept-the-model-and-cut-the-api-cost-anyway-125i</link>
      <guid>https://dev.to/sergeisolod/deepseek-v4-flash-got-expensive-i-kept-the-model-and-cut-the-api-cost-anyway-125i</guid>
      <description>&lt;p&gt;&lt;em&gt;DeepSeek changed its API pricing, and my spending jumped almost overnight. Instead of replacing the model, I tried something simpler: keep the exact same V4 Flash checkpoint and move the inference somewhere else.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;On August 17, I checked my API usage and initially assumed something in my application had gone wrong.&lt;/p&gt;

&lt;p&gt;The balance was disappearing much faster than usual.&lt;/p&gt;

&lt;p&gt;Not a little faster.&lt;/p&gt;

&lt;p&gt;Roughly &lt;strong&gt;five times faster&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I already knew DeepSeek had announced a pricing change. I had seen the announcement a few days earlier, but I did not expect it to have such a large effect on my actual usage.&lt;/p&gt;

&lt;p&gt;A 20% increase would have been annoying.&lt;/p&gt;

&lt;p&gt;A 50% increase would have been noticeable.&lt;/p&gt;

&lt;p&gt;Even doubling the price would not have surprised me that much.&lt;/p&gt;

&lt;p&gt;But this was enough to make me stop and look at the numbers properly.&lt;/p&gt;

&lt;p&gt;There was nothing wrong with my billing integration.&lt;/p&gt;

&lt;p&gt;The new prices were simply live.&lt;/p&gt;

&lt;p&gt;DeepSeek introduced peak and off-peak pricing for V4 Flash and V4 Pro starting at 16:00 UTC on August 16. Reuters had already reported that some DeepSeek prices were increasing anywhere from 50% to 1,100%, depending on the model, token type and time of day.&lt;/p&gt;

&lt;p&gt;That was when I remembered an important detail.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;DeepSeek V4 Flash is available as an open-weight model.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;And that changes the problem completely.&lt;/p&gt;

&lt;p&gt;Instead of asking:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Should I replace DeepSeek with another model?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;I started asking:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Can I keep exactly the same model and just stop buying the inference from DeepSeek?&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That turned out to be the more interesting question.&lt;/p&gt;

&lt;h2&gt;
  
  
  The old DeepSeek pricing was extremely cheap
&lt;/h2&gt;

&lt;p&gt;Before the change, V4 Flash had pricing that was difficult to complain about.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Token type&lt;/th&gt;
&lt;th&gt;Old price per 1M tokens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Input, cache miss&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.14&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cached input&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.0028&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.28&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That is the pricing I had become used to.&lt;/p&gt;

&lt;p&gt;The current official &lt;code&gt;deepseek-v4-flash&lt;/code&gt;, which corresponds to &lt;code&gt;DeepSeek-V4-Flash-0731&lt;/code&gt;, now has separate off-peak and peak rates.&lt;/p&gt;

&lt;h3&gt;
  
  
  Off-peak
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Token type&lt;/th&gt;
&lt;th&gt;Price per 1M tokens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Input, cache miss&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.22&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cached input&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.007&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.66&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Peak
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Token type&lt;/th&gt;
&lt;th&gt;Price per 1M tokens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Input, cache miss&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.44&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cached input&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.014&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$1.32&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;DeepSeek currently lists the peak windows as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;01:00 to 04:00 UTC&lt;/li&gt;
&lt;li&gt;06:00 to 10:00 UTC&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;My own spending being close to five times higher does not mean every possible DeepSeek request is exactly five times more expensive.&lt;/p&gt;

&lt;p&gt;The actual increases depend on what kind of tokens you are paying for.&lt;/p&gt;

&lt;p&gt;For example:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;cache-miss input went from $0.14 to $0.44 at peak, about &lt;strong&gt;3.14x&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;output went from $0.28 to $1.32 at peak, about &lt;strong&gt;4.71x&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;cached input went from $0.0028 to $0.014 at peak, exactly &lt;strong&gt;5x&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your application generates a lot of output, a bill that feels close to five times larger is not difficult to explain.&lt;/p&gt;

&lt;p&gt;I did not want to keep paying those rates while spending several days slowly evaluating alternatives.&lt;/p&gt;

&lt;p&gt;So I started searching immediately.&lt;/p&gt;

&lt;h2&gt;
  
  
  Changing providers is much easier than changing models
&lt;/h2&gt;

&lt;p&gt;There are plenty of models I could have tested instead.&lt;/p&gt;

&lt;p&gt;Qwen.&lt;/p&gt;

&lt;p&gt;GLM.&lt;/p&gt;

&lt;p&gt;Kimi.&lt;/p&gt;

&lt;p&gt;MiniMax.&lt;/p&gt;

&lt;p&gt;Mistral.&lt;/p&gt;

&lt;p&gt;And many more.&lt;/p&gt;

&lt;p&gt;But changing an LLM in a real application is not always a one-line configuration change.&lt;/p&gt;

&lt;p&gt;Even when the API format is similar, behavior can move in ways that matter.&lt;/p&gt;

&lt;p&gt;Prompts may need adjustment.&lt;/p&gt;

&lt;p&gt;Answers may become longer or shorter.&lt;/p&gt;

&lt;p&gt;The tone can change.&lt;/p&gt;

&lt;p&gt;Sampling settings that worked well before may no longer be ideal.&lt;/p&gt;

&lt;p&gt;Long-context behavior can differ.&lt;/p&gt;

&lt;p&gt;Edge cases appear.&lt;/p&gt;

&lt;p&gt;If the current model already works well, replacing it creates a new testing problem.&lt;/p&gt;

&lt;p&gt;I wanted to avoid that problem.&lt;/p&gt;

&lt;p&gt;My goal became very specific:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Keep DeepSeek V4 Flash 0731. Change only the infrastructure serving it.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Open weights create a separate inference market
&lt;/h2&gt;

&lt;p&gt;This is the part I think developers sometimes overlook.&lt;/p&gt;

&lt;p&gt;DeepSeek published the &lt;code&gt;deepseek-ai/DeepSeek-V4-Flash-0731&lt;/code&gt; checkpoint publicly on Hugging Face under the MIT License.&lt;/p&gt;

&lt;p&gt;The weights are available.&lt;/p&gt;

&lt;p&gt;Deployment information is available.&lt;/p&gt;

&lt;p&gt;DeepSeek even documents a deployment example using vLLM on a single node with four GB300 GPUs.&lt;/p&gt;

&lt;p&gt;So an inference company does not necessarily need to purchase API tokens from DeepSeek and resell those tokens to you.&lt;/p&gt;

&lt;p&gt;It can operate the model itself.&lt;/p&gt;

&lt;p&gt;Conceptually, I used to think about DeepSeek like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Application
    |
    v
DeepSeek API
    |
    v
DeepSeek V4 Flash
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But with public weights, the picture is closer to this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                  DeepSeek V4 Flash weights
                           |
          +----------------+----------------+
          |                |                |
          v                v                v
    DeepSeek API        Runware       Other providers
                                            |
                                            v
                                      Self-hosting too
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;DeepSeek created the model.&lt;/p&gt;

&lt;p&gt;That does not mean DeepSeek has to be the only company selling inference for it.&lt;/p&gt;

&lt;p&gt;Those are now two separate things.&lt;/p&gt;

&lt;p&gt;And once I started looking at the market that way, the pricing became much more interesting.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I searched for providers
&lt;/h2&gt;

&lt;p&gt;I did not start with a list of famous inference companies.&lt;/p&gt;

&lt;p&gt;I wanted to search broadly because smaller providers are often where the interesting prices are.&lt;/p&gt;

&lt;p&gt;My process was basically this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Ask an AI to help me create a very detailed research prompt.&lt;/li&gt;
&lt;li&gt;Run that research repeatedly, usually around &lt;strong&gt;15 to 20 separate searches&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Combine the results into one conversation.&lt;/li&gt;
&lt;li&gt;Compare the providers.&lt;/li&gt;
&lt;li&gt;Remove obviously unsuitable options.&lt;/li&gt;
&lt;li&gt;Build a shortlist.&lt;/li&gt;
&lt;li&gt;Verify the finalists manually.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I do not treat the AI's final ranking as truth.&lt;/p&gt;

&lt;p&gt;That would be a mistake.&lt;/p&gt;

&lt;p&gt;The useful part is breadth.&lt;/p&gt;

&lt;p&gt;It can surface providers I would probably never discover by manually searching through the same five well-known companies.&lt;/p&gt;

&lt;p&gt;Once I get to the shortlist, I check everything myself.&lt;/p&gt;

&lt;p&gt;I open the provider's website.&lt;/p&gt;

&lt;p&gt;I read the documentation.&lt;/p&gt;

&lt;p&gt;I verify the model identifier.&lt;/p&gt;

&lt;p&gt;I check the actual pricing.&lt;/p&gt;

&lt;p&gt;And if it looks promising, I send real API requests.&lt;/p&gt;

&lt;p&gt;That process eventually led me to &lt;strong&gt;Runware&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Runware was serving the exact checkpoint
&lt;/h2&gt;

&lt;p&gt;The important part was not merely that Runware supported "DeepSeek."&lt;/p&gt;

&lt;p&gt;It offered:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DeepSeek-V4-Flash-0731
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is what I wanted.&lt;/p&gt;

&lt;p&gt;I was specifically trying to avoid introducing another model variable into the test.&lt;/p&gt;

&lt;p&gt;At the time I checked, Runware's pricing for the checkpoint was:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Token type&lt;/th&gt;
&lt;th&gt;Runware per 1M tokens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Input&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.076&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cached input&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.014&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$0.153&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The first time I saw those numbers, I checked them again.&lt;/p&gt;

&lt;p&gt;Not because Runware was cheaper than the new DeepSeek pricing.&lt;/p&gt;

&lt;p&gt;I expected that.&lt;/p&gt;

&lt;p&gt;What surprised me was that it was also substantially cheaper than the &lt;strong&gt;old&lt;/strong&gt; DeepSeek pricing.&lt;/p&gt;

&lt;h2&gt;
  
  
  It was cheaper than DeepSeek before the increase
&lt;/h2&gt;

&lt;p&gt;Take input first.&lt;/p&gt;

&lt;p&gt;Old DeepSeek:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.14 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Runware:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.076 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That makes Runware roughly &lt;strong&gt;45.7% cheaper&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Or viewed from the other direction, the old DeepSeek price was around &lt;strong&gt;1.84x higher&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Now output.&lt;/p&gt;

&lt;p&gt;Old DeepSeek:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.28 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Runware:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.153 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is approximately &lt;strong&gt;45.4% cheaper&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The old DeepSeek rate was about &lt;strong&gt;1.83x higher&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This was the part I did not expect.&lt;/p&gt;

&lt;p&gt;Finding a provider cheaper than DeepSeek after a major price increase is not surprising.&lt;/p&gt;

&lt;p&gt;Finding the exact same public checkpoint for around 45% less than DeepSeek charged &lt;strong&gt;before&lt;/strong&gt; the increase is much more interesting.&lt;/p&gt;

&lt;h2&gt;
  
  
  Against the new DeepSeek prices, the gap gets huge
&lt;/h2&gt;

&lt;p&gt;For normal cache-miss input and output, the difference is now much larger.&lt;/p&gt;

&lt;h3&gt;
  
  
  DeepSeek off-peak vs Runware
&lt;/h3&gt;

&lt;p&gt;Input:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.22 / $0.076 = ~2.9x
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.66 / $0.153 = ~4.3x
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  DeepSeek peak vs Runware
&lt;/h3&gt;

&lt;p&gt;Input:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.44 / $0.076 = ~5.8x
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$1.32 / $0.153 = ~8.6x
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is one important exception.&lt;/p&gt;

&lt;p&gt;DeepSeek's off-peak cached input currently costs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.007 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Runware's cached input costs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.014 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So DeepSeek is cheaper for that particular category during off-peak hours.&lt;/p&gt;

&lt;p&gt;This is why comparing only one headline number can be misleading.&lt;/p&gt;

&lt;p&gt;Your real cost depends on your own traffic.&lt;/p&gt;

&lt;p&gt;An application with very high cache-hit rates may reach a different conclusion from an application generating large amounts of output.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cheap pricing means nothing if the API is unreliable
&lt;/h2&gt;

&lt;p&gt;At this point I still had no reason to trust Runware.&lt;/p&gt;

&lt;p&gt;A pricing page is not a benchmark.&lt;/p&gt;

&lt;p&gt;And cheap tokens are worthless if requests fail constantly.&lt;/p&gt;

&lt;p&gt;So I funded the account, connected the API and started sending actual traffic.&lt;/p&gt;

&lt;p&gt;One early dashboard snapshot looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;790 requests
56 success
734 errors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Which looks awful.&lt;/p&gt;

&lt;p&gt;Later I checked again:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1,570 requests
836 success
734 errors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The difference between those snapshots was:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;+780 requests
+780 successful
+0 new errors
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I want to be precise about what I am claiming here.&lt;/p&gt;

&lt;p&gt;This does &lt;strong&gt;not&lt;/strong&gt; prove some ridiculous five-nines reliability number.&lt;/p&gt;

&lt;p&gt;It also does not tell us why the original 734 requests failed.&lt;/p&gt;

&lt;p&gt;What it does tell me is very simple:&lt;/p&gt;

&lt;p&gt;During the next batch of 780 requests, the success counter increased by 780 and the error counter did not increase at all.&lt;/p&gt;

&lt;p&gt;I continued testing after that.&lt;/p&gt;

&lt;p&gt;At the time I am writing this, I have already passed &lt;strong&gt;1,000 successful AI requests&lt;/strong&gt; through the Runware endpoint.&lt;/p&gt;

&lt;p&gt;So far, the experience has been good.&lt;/p&gt;

&lt;h2&gt;
  
  
  The latency surprised me too
&lt;/h2&gt;

&lt;p&gt;I was mostly looking for price differences.&lt;/p&gt;

&lt;p&gt;Latency was secondary.&lt;/p&gt;

&lt;p&gt;But the successful requests I checked were generally fast.&lt;/p&gt;

&lt;p&gt;Most completed in somewhere between a fraction of a second and roughly 1.5 seconds.&lt;/p&gt;

&lt;p&gt;Some of the smaller requests were costing around:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.000005
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;to:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$0.000018
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;per request.&lt;/p&gt;

&lt;p&gt;After hundreds of successful calls, the total displayed spending was still around one cent.&lt;/p&gt;

&lt;p&gt;That is when the comparison stopped feeling theoretical.&lt;/p&gt;

&lt;p&gt;This was no longer:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Provider A says it costs X and Provider B says it costs Y.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The endpoint was connected to my application.&lt;/p&gt;

&lt;p&gt;Requests were completing.&lt;/p&gt;

&lt;p&gt;Responses were coming back quickly.&lt;/p&gt;

&lt;p&gt;And the cost was tiny.&lt;/p&gt;

&lt;h2&gt;
  
  
  A necessary disclaimer about Runware
&lt;/h2&gt;

&lt;p&gt;I want to be explicit about this because I am naming one company repeatedly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;This article is not sponsored.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Runware did not contact me.&lt;/p&gt;

&lt;p&gt;I did not contact them before writing this.&lt;/p&gt;

&lt;p&gt;Nobody asked me to publish anything.&lt;/p&gt;

&lt;p&gt;I have no affiliate agreement.&lt;/p&gt;

&lt;p&gt;No referral deal.&lt;/p&gt;

&lt;p&gt;No discount code.&lt;/p&gt;

&lt;p&gt;No free credits.&lt;/p&gt;

&lt;p&gt;I found Runware while researching inference providers, created an account, deposited my own money and tested the API.&lt;/p&gt;

&lt;p&gt;That is the entire relationship.&lt;/p&gt;

&lt;p&gt;There is another important disclaimer.&lt;/p&gt;

&lt;p&gt;I have been testing it for &lt;strong&gt;one day&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That is nowhere near enough time to declare a provider the best option on the market.&lt;/p&gt;

&lt;p&gt;Maybe I discover reliability issues later.&lt;/p&gt;

&lt;p&gt;Maybe its pricing changes.&lt;/p&gt;

&lt;p&gt;Maybe performance gets worse.&lt;/p&gt;

&lt;p&gt;Maybe another provider appears next week with better infrastructure and an even lower price.&lt;/p&gt;

&lt;p&gt;If that happens, I will switch again.&lt;/p&gt;

&lt;p&gt;I am not loyal to an inference endpoint.&lt;/p&gt;

&lt;p&gt;I moved because DeepSeek's new pricing made me reconsider where I was buying compute.&lt;/p&gt;

&lt;p&gt;Runware happened to be the provider that looked good enough to test immediately.&lt;/p&gt;

&lt;p&gt;So far:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;more than &lt;strong&gt;1,000 requests have completed successfully&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;latency has been good in my testing&lt;/li&gt;
&lt;li&gt;pricing is extremely low&lt;/li&gt;
&lt;li&gt;the first day has gone well&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is the extent of my claim.&lt;/p&gt;

&lt;h2&gt;
  
  
  DeepSeek is now competing with companies that run DeepSeek
&lt;/h2&gt;

&lt;p&gt;This is the most interesting part of the whole situation to me.&lt;/p&gt;

&lt;p&gt;DeepSeek built the model.&lt;/p&gt;

&lt;p&gt;Then it published the weights under a permissive license.&lt;/p&gt;

&lt;p&gt;That means infrastructure companies can take the same model and compete on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;GPU utilization&lt;/li&gt;
&lt;li&gt;batching&lt;/li&gt;
&lt;li&gt;serving software&lt;/li&gt;
&lt;li&gt;capacity planning&lt;/li&gt;
&lt;li&gt;margins&lt;/li&gt;
&lt;li&gt;geographic infrastructure&lt;/li&gt;
&lt;li&gt;latency&lt;/li&gt;
&lt;li&gt;reliability&lt;/li&gt;
&lt;li&gt;pricing&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In other words, DeepSeek is not only competing with Qwen, OpenAI, Anthropic, Mistral or other model developers.&lt;/p&gt;

&lt;p&gt;For inference revenue, it can also end up competing with companies serving &lt;strong&gt;DeepSeek's own model&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;When official V4 Flash inference cost $0.14 for input and $0.28 for output, that competition was easy to ignore.&lt;/p&gt;

&lt;p&gt;The official API was already extremely cheap.&lt;/p&gt;

&lt;p&gt;But when peak pricing becomes $0.44 input and $1.32 output while another provider offers the same checkpoint at $0.076 and $0.153, the infrastructure market starts to matter a lot more.&lt;/p&gt;

&lt;p&gt;This is also why I do not think "Runware is reselling DeepSeek" is necessarily the right way to think about it.&lt;/p&gt;

&lt;p&gt;If a provider is running public weights on its own hardware, then what you are purchasing is its &lt;strong&gt;inference infrastructure&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The model came from DeepSeek.&lt;/p&gt;

&lt;p&gt;The compute came from someone else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Did DeepSeek make a mistake by releasing the weights?
&lt;/h2&gt;

&lt;p&gt;I do not think the answer is obvious.&lt;/p&gt;

&lt;p&gt;It is tempting to look at these price differences and say:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;DeepSeek created competitors for its own API business.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That is true in one narrow sense.&lt;/p&gt;

&lt;p&gt;But open-weight models also gain advantages that closed models do not.&lt;/p&gt;

&lt;p&gt;More developers can experiment with them.&lt;/p&gt;

&lt;p&gt;More companies can integrate them.&lt;/p&gt;

&lt;p&gt;Cloud providers can support them.&lt;/p&gt;

&lt;p&gt;Inference platforms can optimize them.&lt;/p&gt;

&lt;p&gt;Researchers can use them.&lt;/p&gt;

&lt;p&gt;Businesses can deploy them privately.&lt;/p&gt;

&lt;p&gt;The model can spread into places where a single proprietary API never would.&lt;/p&gt;

&lt;p&gt;That adoption has value.&lt;/p&gt;

&lt;p&gt;So I am not arguing that releasing V4 Flash was a bad business decision.&lt;/p&gt;

&lt;p&gt;The claim I am comfortable making is much narrower:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Once a model's weights are publicly available under a permissive license, the company that created the model is no longer the only company capable of selling access to its inference.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;When the creator's API is the cheapest and easiest option, nobody cares.&lt;/p&gt;

&lt;p&gt;When the creator raises prices significantly, people start looking around.&lt;/p&gt;

&lt;p&gt;I certainly did.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why not just self-host V4 Flash?
&lt;/h2&gt;

&lt;p&gt;I also considered the obvious next step.&lt;/p&gt;

&lt;p&gt;If Runware can download the model and run it, why not do the same thing myself?&lt;/p&gt;

&lt;p&gt;Because running a model like this is not the same thing as deploying a Node.js service onto a $20 VPS.&lt;/p&gt;

&lt;p&gt;The model repository itself is large.&lt;/p&gt;

&lt;p&gt;DeepSeek's own deployment example mentions &lt;strong&gt;four GB300 GPUs on one node&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;And GPUs are only one part of the system.&lt;/p&gt;

&lt;p&gt;You also need to think about:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;enough VRAM&lt;/li&gt;
&lt;li&gt;inference engines&lt;/li&gt;
&lt;li&gt;model loading&lt;/li&gt;
&lt;li&gt;batching&lt;/li&gt;
&lt;li&gt;KV cache management&lt;/li&gt;
&lt;li&gt;concurrency&lt;/li&gt;
&lt;li&gt;networking&lt;/li&gt;
&lt;li&gt;monitoring&lt;/li&gt;
&lt;li&gt;capacity planning&lt;/li&gt;
&lt;li&gt;failover&lt;/li&gt;
&lt;li&gt;power&lt;/li&gt;
&lt;li&gt;cooling&lt;/li&gt;
&lt;li&gt;maintenance&lt;/li&gt;
&lt;li&gt;keeping expensive GPUs utilized&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Self-hosting can absolutely make sense at sufficient scale.&lt;/p&gt;

&lt;p&gt;But there is a huge difference between:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I can download the weights.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;and:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I can serve this model economically in production.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;An inference provider can spread the cost of its hardware across many customers.&lt;/p&gt;

&lt;p&gt;For my current usage, I cannot.&lt;/p&gt;

&lt;p&gt;If somebody is willing to serve the model to me for &lt;strong&gt;$0.076 per million input tokens&lt;/strong&gt;, buying the API can be much more rational than building the infrastructure myself.&lt;/p&gt;

&lt;p&gt;If I had enough GPUs and capital, though, I would absolutely be interested in this market.&lt;/p&gt;

&lt;p&gt;Running open models efficiently and selling inference is an interesting infrastructure business.&lt;/p&gt;

&lt;p&gt;I just do not need to own that infrastructure today.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I am changing after this
&lt;/h2&gt;

&lt;p&gt;The main lesson for me is not specifically "use Runware."&lt;/p&gt;

&lt;p&gt;That could change next week.&lt;/p&gt;

&lt;p&gt;The bigger change is how I evaluate open-weight APIs.&lt;/p&gt;

&lt;p&gt;Before this, my mental model was basically:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DeepSeek built the model
        ↓
DeepSeek sells the API
        ↓
That is the price of DeepSeek
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now I think about it more like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Model creator
     ↓
Public weights
     ↓
Many companies compete to serve them
     ↓
Different prices, latency and reliability
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the numbers make the distinction hard to ignore.&lt;/p&gt;

&lt;p&gt;Old DeepSeek V4 Flash:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Input:  $0.14 / 1M
Output: $0.28 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;New DeepSeek peak pricing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Input:  $0.44 / 1M
Output: $1.32 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Runware when I tested it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Input:  $0.076 / 1M
Output: $0.153 / 1M
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And I have already sent more than &lt;strong&gt;1,000 successful AI requests&lt;/strong&gt; through that endpoint.&lt;/p&gt;

&lt;p&gt;Will I still use it a month from now?&lt;/p&gt;

&lt;p&gt;Maybe.&lt;/p&gt;

&lt;p&gt;Maybe not.&lt;/p&gt;

&lt;p&gt;What I will definitely keep doing is this:&lt;/p&gt;

&lt;p&gt;When a model has public weights, I will no longer assume that the model creator automatically has the best inference API.&lt;/p&gt;

&lt;p&gt;I will identify the exact checkpoint.&lt;/p&gt;

&lt;p&gt;I will find out who else serves it.&lt;/p&gt;

&lt;p&gt;I will compare the real token pricing.&lt;/p&gt;

&lt;p&gt;And then I will send actual requests before making a decision.&lt;/p&gt;

&lt;p&gt;That is how I found a cheaper way to keep using DeepSeek V4 Flash without changing the model at all.&lt;/p&gt;

&lt;h1&gt;
  
  
  deepseek #ai #llm #api
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>api</category>
      <category>llm</category>
    </item>
    <item>
      <title>My Next.js Blog: Global in a Week (21 Languages)</title>
      <dc:creator>Sergei Solod</dc:creator>
      <pubDate>Tue, 17 Jun 2025 13:04:35 +0000</pubDate>
      <link>https://dev.to/sergeisolod/my-multilingual-blog-experiment-from-idea-to-global-seo-in-7-days-5f23</link>
      <guid>https://dev.to/sergeisolod/my-multilingual-blog-experiment-from-idea-to-global-seo-in-7-days-5f23</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1a5evjp1mt993x3ksldr.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1a5evjp1mt993x3ksldr.jpg" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A little while ago, I decided to try a simple experiment: build a &lt;strong&gt;Next.js blog in 21 languages&lt;/strong&gt; and see how search engines would handle it.&lt;/p&gt;

&lt;p&gt;There was no big business plan behind it. I mostly wanted to test multilingual routing, localization, indexing, and SEO on a real project instead of another demo app.&lt;/p&gt;

&lt;p&gt;After the first week, Google Search Console showed &lt;strong&gt;439 impressions and 5 clicks&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Yandex had also started picking up some of the pages.&lt;/p&gt;

&lt;p&gt;The numbers are obviously small, but for a brand-new personal blog, I was happy to see that Google was already discovering pages and showing them in search.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I used Next.js
&lt;/h2&gt;

&lt;p&gt;The blog is built with &lt;strong&gt;Next.js&lt;/strong&gt;, and every article has a separate version for each of the 21 supported languages.&lt;/p&gt;

&lt;p&gt;I chose Next.js because it gave me a convenient setup for:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;internationalized routing&lt;/li&gt;
&lt;li&gt;server-side rendering&lt;/li&gt;
&lt;li&gt;static generation&lt;/li&gt;
&lt;li&gt;reusable layouts and metadata&lt;/li&gt;
&lt;li&gt;generating a large number of localized pages&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For this kind of project, that made the implementation much easier.&lt;/p&gt;

&lt;p&gt;Of course, using Next.js does not automatically mean better rankings or faster indexing. It just gives you the tools to build pages that are easy for search engines to crawl and understand.&lt;/p&gt;

&lt;p&gt;Google still decides what gets indexed and what gets shown in search.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first search results
&lt;/h2&gt;

&lt;p&gt;After roughly one week, I had:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;439 Google impressions&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;5 clicks&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;about &lt;strong&gt;1.1% CTR&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Five clicks is not much traffic, but traffic was not really the interesting part yet.&lt;/p&gt;

&lt;p&gt;What interested me was that the site had already started getting impressions across a large multilingual structure.&lt;/p&gt;

&lt;p&gt;One article on my blog can exist as 21 separate URLs.&lt;/p&gt;

&lt;p&gt;That creates a lot more possible search entry points than publishing only in English.&lt;/p&gt;

&lt;p&gt;It does not mean 21 languages will give you 21 times more traffic. Each page still has to earn its own visibility.&lt;/p&gt;

&lt;p&gt;But it does make the site much more interesting to monitor.&lt;/p&gt;

&lt;h2&gt;
  
  
  Indexing took longer
&lt;/h2&gt;

&lt;p&gt;The next thing I started watching was indexing.&lt;/p&gt;

&lt;p&gt;My &lt;code&gt;sitemap.xml&lt;/code&gt; is generated automatically, and I submit it through &lt;strong&gt;Google Search Console&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A few weeks later, I had &lt;strong&gt;471 pages&lt;/strong&gt; in the group I was tracking.&lt;/p&gt;

&lt;p&gt;Out of those:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;436 pages were indexed&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;35 pages were still not indexed&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So around 93% of those pages had made it into Google's index.&lt;/p&gt;

&lt;p&gt;That was a useful reminder that discovery, indexing, impressions, and clicks are all different things.&lt;/p&gt;

&lt;p&gt;Submitting a sitemap does not mean Google will immediately index every page.&lt;/p&gt;

&lt;p&gt;An indexed page also does not mean it will rank or get traffic.&lt;/p&gt;

&lt;p&gt;And if a new page has not been indexed yet, that does not automatically mean there is a serious SEO problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Multilingual sites scale very quickly
&lt;/h2&gt;

&lt;p&gt;One thing that became obvious pretty fast is how quickly the number of pages grows.&lt;/p&gt;

&lt;p&gt;If I publish one article, I am not really publishing one page.&lt;/p&gt;

&lt;p&gt;I am publishing &lt;strong&gt;21 localized pages&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Ten articles can become 210 URLs.&lt;/p&gt;

&lt;p&gt;Twenty articles can become 420 URLs.&lt;/p&gt;

&lt;p&gt;That means small technical mistakes can also get duplicated very quickly.&lt;/p&gt;

&lt;p&gt;A problem with metadata, internal links, canonical tags, &lt;code&gt;hreflang&lt;/code&gt;, translations, or URL structure can affect dozens or hundreds of pages instead of just one.&lt;/p&gt;

&lt;p&gt;So multilingual publishing gives you more opportunities, but it also gives you more things to check.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I learned so far
&lt;/h2&gt;

&lt;p&gt;The biggest lesson for me is that multilingual SEO is not simply about creating as many translated pages as possible.&lt;/p&gt;

&lt;p&gt;The technical side matters, but there are several separate questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Can Google discover the URLs?&lt;/li&gt;
&lt;li&gt;Can Google crawl them correctly?&lt;/li&gt;
&lt;li&gt;Which pages actually get indexed?&lt;/li&gt;
&lt;li&gt;Which languages receive impressions?&lt;/li&gt;
&lt;li&gt;Which pages receive clicks?&lt;/li&gt;
&lt;li&gt;Which localized versions eventually start getting consistent search traffic?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;At first, I mostly looked at the total number of impressions.&lt;/p&gt;

&lt;p&gt;Now I think it is much more useful to look at each part separately.&lt;/p&gt;

&lt;h2&gt;
  
  
  Current result
&lt;/h2&gt;

&lt;p&gt;So far, the experiment looks like this:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;First week:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;439 Google impressions&lt;/li&gt;
&lt;li&gt;5 clicks&lt;/li&gt;
&lt;li&gt;Yandex started showing some pages&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;A few weeks later:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;471 pages tracked&lt;/li&gt;
&lt;li&gt;436 indexed&lt;/li&gt;
&lt;li&gt;35 still not indexed&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I would not call that proof that multilingual SEO automatically works, or that Next.js gives some special SEO advantage.&lt;/p&gt;

&lt;p&gt;But it does show that a small multilingual project can get discovered across a surprisingly large number of pages fairly quickly.&lt;/p&gt;

&lt;p&gt;For me, the more interesting question now is not whether Google can find the site.&lt;/p&gt;

&lt;p&gt;It is which languages and articles will eventually turn those impressions into consistent organic traffic.&lt;/p&gt;

&lt;p&gt;Blog: &lt;a href="https://jsvar.com/en/blog" rel="noopener noreferrer"&gt;https://jsvar.com/en/blog&lt;/a&gt;&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>google</category>
      <category>seo</category>
      <category>ssr</category>
    </item>
  </channel>
</rss>
