<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Isaque Silva Chaves</title>
    <description>The latest articles on DEV Community by Isaque Silva Chaves (@isaque_silvachaves).</description>
    <link>https://dev.to/isaque_silvachaves</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F1988015%2Fece1aaf6-5f4e-4efc-91d6-bda99d013afb.jpeg</url>
      <title>DEV Community: Isaque Silva Chaves</title>
      <link>https://dev.to/isaque_silvachaves</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/isaque_silvachaves"/>
    <language>en</language>
    <item>
      <title>Your Claude Plan Could Be Worth $11,000 a Month</title>
      <dc:creator>Isaque Silva Chaves</dc:creator>
      <pubDate>Wed, 04 Nov 2026 01:24:14 +0000</pubDate>
      <link>https://dev.to/isaque_silvachaves/your-claude-plan-could-be-worth-11000-a-month-3mn3</link>
      <guid>https://dev.to/isaque_silvachaves/your-claude-plan-could-be-worth-11000-a-month-3mn3</guid>
      <description>&lt;p&gt;The most expensive Claude plan costs $200 a month. At API prices, the usage it allows could be worth &lt;strong&gt;about $11,000 a month&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;I did not measure that plan. I measured mine. I pay $100 a month for Max 5x, and in the last 30 days my Claude Code usage would have cost &lt;strong&gt;about $5,000&lt;/strong&gt; at API prices. The weekly limit stopped me twice, and those stops let me put a dollar value on the plan: about $5,500 a month. The $200 plan allows twice as much usage per week, so it is worth about twice as much.&lt;/p&gt;

&lt;p&gt;Anthropic sells its plans in "times Pro usage", not in dollars, so the plan page does not show these numbers. I calculated them from the transcripts that Claude Code keeps on my disk.&lt;/p&gt;

&lt;p&gt;In this article I'll show how I measured it, what each plan is worth in API dollars, where the tokens went, and why your own number is probably smaller.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The case:&lt;/strong&gt; one developer (me), one machine, Claude Code on the Max 5x plan, from October 4 to November 3, 2026. Most of the work was coding with agents on Claude Opus 5.5. &lt;strong&gt;Every number in this article is an estimate from my own usage, and the $11,000 is an extrapolation from my plan to the $200 one.&lt;/strong&gt; Anthropic does not publish its limits in dollars, and I have no data on what my usage costs Anthropic.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The Result
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Plan&lt;/th&gt;
&lt;th&gt;Price per month&lt;/th&gt;
&lt;th&gt;Usage it allows, at API prices&lt;/th&gt;
&lt;th&gt;Ratio&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Max 5x (mine, measured)&lt;/td&gt;
&lt;td&gt;$100&lt;/td&gt;
&lt;td&gt;~$5,500&lt;/td&gt;
&lt;td&gt;~55×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ultra (extrapolated)&lt;/td&gt;
&lt;td&gt;$200&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~$11,000&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~55×&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;And my real month on Max 5x: I paid $100 and used &lt;strong&gt;~$5,000&lt;/strong&gt; at API prices. That is $3.30 a day against $160 a day, a 98% discount.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Measured It
&lt;/h2&gt;

&lt;p&gt;Claude Code writes every session to a JSONL file under &lt;code&gt;~/.claude/projects/&lt;/code&gt;. Each answer from the model carries a &lt;code&gt;usage&lt;/code&gt; block with four counts: input tokens, cache writes, cache reads and output tokens. I summed every answer of the 30 days, from the main sessions and from the subagents, and priced each one with the public API price table:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Input&lt;/th&gt;
&lt;th&gt;Cache write (5 min / 1 h)&lt;/th&gt;
&lt;th&gt;Cache read&lt;/th&gt;
&lt;th&gt;Output&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Claude Opus 5.5&lt;/td&gt;
&lt;td&gt;$4&lt;/td&gt;
&lt;td&gt;$5 / $8&lt;/td&gt;
&lt;td&gt;$0.20&lt;/td&gt;
&lt;td&gt;$20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Sonnet 5.5&lt;/td&gt;
&lt;td&gt;$2&lt;/td&gt;
&lt;td&gt;$2.50 / $4&lt;/td&gt;
&lt;td&gt;$0.10&lt;/td&gt;
&lt;td&gt;$10&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The prices are in US$ per million tokens, from Anthropic's pricing page on November 3, 2026. Opus 5.5 was 96% of my cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  What My Plan Is Worth
&lt;/h2&gt;

&lt;p&gt;The Max 5x plan has two limits: one per 5 hours and one per week. I hit both. Each stop shows a message with the time of the reset, so I summed the API-equivalent usage inside each window, up to the stop.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Limit&lt;/th&gt;
&lt;th&gt;Times I hit it&lt;/th&gt;
&lt;th&gt;API-equivalent at the stop&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Weekly&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;~$1,250&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5 hours&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;~$150 to ~$180&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A week has 33.6 windows of 5 hours, and the weekly limit runs out after 7 or 8 full ones. So for heavy work, the weekly limit is the one that stops you. Both times, it stopped me on a Saturday, about one day before the weekly reset.&lt;/p&gt;

&lt;p&gt;From the weekly limit, the monthly ceiling is about $1,250 × 4.35 ≈ &lt;strong&gt;$5,500 of API usage for $100, about 55 times the price&lt;/strong&gt;. In these 30 days, I used about 90% of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the $200 Plan Is Worth
&lt;/h2&gt;

&lt;p&gt;The Ultra plan costs $200. It gives "10x Pro usage" per week instead of "5x", and it has no 5-hour limit for verified accounts.&lt;/p&gt;

&lt;p&gt;If 10x is exactly twice 5x, its weekly ceiling is about $2,500 of API usage, and its monthly ceiling is about &lt;strong&gt;$11,000&lt;/strong&gt;. Both plans also include API credits for the Claude Platform: $100 a month on Max and $200 on Ultra. I did not count them here.&lt;/p&gt;

&lt;p&gt;The ratio stays at about 55×. The $200 plan does not make a token cheaper. It buys twice the room, and it removes the 5-hour stop. I don't have Ultra, so this number is an extrapolation from my measurements, not a measurement.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the Tokens Went
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Token type&lt;/th&gt;
&lt;th&gt;Tokens&lt;/th&gt;
&lt;th&gt;API cost&lt;/th&gt;
&lt;th&gt;Share of cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cache reads&lt;/td&gt;
&lt;td&gt;13.9 billion&lt;/td&gt;
&lt;td&gt;~$2,700&lt;/td&gt;
&lt;td&gt;54%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cache writes&lt;/td&gt;
&lt;td&gt;0.2 billion&lt;/td&gt;
&lt;td&gt;~$1,400&lt;/td&gt;
&lt;td&gt;28%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Output&lt;/td&gt;
&lt;td&gt;0.05 billion&lt;/td&gt;
&lt;td&gt;~$900&lt;/td&gt;
&lt;td&gt;18%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Input without cache&lt;/td&gt;
&lt;td&gt;almost zero&lt;/td&gt;
&lt;td&gt;~$0&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~14 billion&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~$5,000&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;98% of my tokens were cache reads.&lt;/strong&gt; An agent sends the whole conversation again on every call: the instructions, the files it read, the tool results. I made about 56,000 calls, and each one read a large context again.&lt;/p&gt;

&lt;p&gt;The prompt cache makes that cheap. On Opus 5.5, a cache read costs $0.20 per million tokens, 5% of the input price. Even so, the context is most of the bill: cache reads and cache writes together are 82% of the cost. The output, which includes all the code the model wrote, is only 18%.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Your Number Is Probably Smaller
&lt;/h2&gt;

&lt;p&gt;A plan is worth what you actually use, and I am a heavy user.&lt;/p&gt;

&lt;p&gt;Anthropic's Claude Code documentation says that across enterprise deployments, the average developer costs about $13 per active day and $150 to $250 per month at API prices. On the $100 plan, that is a discount of &lt;strong&gt;about 35% to 60%&lt;/strong&gt;. That is still a good deal, but it is far from 98%.&lt;/p&gt;

&lt;p&gt;To use $11,000 a month on the $200 plan, you would need about 45 to 75 times the usage of that average developer. My usage was 20 to 30 times that average, and it filled about 90% of my $100 plan.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is Anthropic Losing Money on Me?
&lt;/h2&gt;

&lt;p&gt;I don't know. The API price is a list price, not a cost. Anthropic itself sells the same tokens at different prices: the Batch API costs 50% of the normal price, and a cache read costs 5% of an input token. 98% of my tokens were the cheapest kind. Many subscribers also use much less than the ceiling.&lt;/p&gt;

&lt;p&gt;I found analyst estimates of Anthropic's margins, but they disagree with each other and I could not verify them, so I don't quote them here. What I can say is narrower: my subscription price is not a cost, and the API price is not a cost either.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure Your Own
&lt;/h2&gt;

&lt;p&gt;Here is the script I used, cut to the essentials. It reads your transcripts, removes duplicates and prints the API-equivalent cost since a date:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;What would my Claude Code usage cost at API prices?

Reads the transcripts under ~/.claude/projects and prices every assistant
message with the public API price table. Usage: python3 claude_api_equivalent.py 2026-10-04
&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;

&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;glob&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;

&lt;span class="c1"&gt;# US$ per million tokens: input, 5-minute cache write, 1-hour cache write, cache read, output.
&lt;/span&gt;&lt;span class="n"&gt;PRICES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;claude-opus-5-5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;4.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;5.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;8.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;20.00&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;claude-sonnet-5-5&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;2.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;4.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;10.00&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;claude-fable-5-1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;10.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;12.50&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;20.00&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.25&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;50.00&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;since&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;0000&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dollars&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;unpriced&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;expanduser&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;~/.claude/projects/**/*.jsonl&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;recursive&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isfile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;  &lt;span class="c1"&gt;# skip broken links
&lt;/span&gt;        &lt;span class="k"&gt;continue&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;replace&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="s"&gt;usage&lt;/span&gt;&lt;span class="sh"&gt;"'&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;entry&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;JSONDecodeError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="n"&gt;message&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;message&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
            &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;message_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;usage&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;model&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="c1"&gt;# A resumed session copies old messages into a new file, so dedupe across files.
&lt;/span&gt;            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;assistant&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;message_id&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;timestamp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;since&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;synthetic&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;message_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;PRICES&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="n"&gt;unpriced&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
                &lt;span class="k"&gt;continue&lt;/span&gt;
            &lt;span class="n"&gt;cache&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache_creation&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="p"&gt;{}&lt;/span&gt;
            &lt;span class="n"&gt;write_1h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ephemeral_1h_input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
            &lt;span class="n"&gt;write_5m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache_creation_input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;write_1h&lt;/span&gt;
            &lt;span class="n"&gt;counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache write&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;write_5m&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;write_1h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache read&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache_read_input_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;usage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;output_tokens&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
            &lt;span class="n"&gt;p_in&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p_5m&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p_1h&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p_read&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p_out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;PRICES&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
            &lt;span class="n"&gt;dollars&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p_in&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;
            &lt;span class="n"&gt;dollars&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache write&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;write_5m&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p_5m&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;write_1h&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p_1h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;
            &lt;span class="n"&gt;dollars&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache read&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache read&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p_read&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;
            &lt;span class="n"&gt;dollars&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;p_out&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;
            &lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;total&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dollars&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;seen&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; messages, &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; billion tokens, US$ &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;total&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;,.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; at API prices&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache read&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cache write&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;output&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;input&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e6&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; M tokens   US$ &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;dollars&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;unpriced&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Not priced (add them to PRICES):&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;unpriced&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it with a start date: &lt;code&gt;python3 claude_api_equivalent.py 2026-10-04&lt;/code&gt;. It takes a few seconds. It prices only the models in &lt;code&gt;PRICES&lt;/code&gt;, and it lists the other models it finds, so you can add their prices. It can only measure the transcripts that are still on your disk: my oldest ones were already gone.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Limitations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The $11,000 is an extrapolation.&lt;/strong&gt; I measured only Max 5x. The Ultra number assumes that "10x Pro" is exactly twice "5x Pro".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One user, one machine.&lt;/strong&gt; My usage on claude.ai also counts toward the limit, and the script cannot see it. The real ceiling may be a little higher.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The limit is not in dollars.&lt;/strong&gt; My dollar ceiling depends on my mix: 96% Opus and a large cache. The 5-hour window that was mostly Sonnet stopped at ~$150, and the Opus windows at ~$175. So the limit does not weigh tokens exactly by API price.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small samples.&lt;/strong&gt; I have two weekly stops and three 5-hour stops.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The API price is not Anthropic's cost.&lt;/strong&gt; The value is against a list price.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prices change.&lt;/strong&gt; I used the prices of November 3, 2026.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;My $100 plan is worth about $5,500 a month at API prices.&lt;/strong&gt; Last month, I used about $5,000 of it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The $200 plan, by the same math, could be worth about $11,000 a month.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Both plans give about 55 times their price&lt;/strong&gt;, if you use them to the limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Most people use much less.&lt;/strong&gt; For an average developer, the $100 plan is a discount of 35% to 60%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;98% of my tokens were cache reads.&lt;/strong&gt; The context, not the code the model writes, is most of the bill.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measure your own number.&lt;/strong&gt; It takes one script and a few seconds.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Your Claude plan could be worth $11,000 a month. Mine was worth $5,000 last month, and I paid $100 for it.&lt;/p&gt;

</description>
    </item>
    <item>
      <title>From 10 Seconds to Under 1: How I Made a Site 10x Faster</title>
      <dc:creator>Isaque Silva Chaves</dc:creator>
      <pubDate>Fri, 30 Oct 2026 11:50:00 +0000</pubDate>
      <link>https://dev.to/isaque_silvachaves/from-10-seconds-to-under-1-how-i-made-a-site-10x-faster-5hgd</link>
      <guid>https://dev.to/isaque_silvachaves/from-10-seconds-to-under-1-how-i-made-a-site-10x-faster-5hgd</guid>
      <description>&lt;h1&gt;
  
  
  How I Cut 60% of the Bundle and 76% of the Boot CPU of a Large Web App
&lt;/h1&gt;

&lt;p&gt;In production, the home page of the app took more than 10 seconds. Now it takes &lt;strong&gt;under 1 second&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;No single trick did that. I cut the JavaScript and CSS that every page downloads from 6,070 KB to 2,434 KB gzip, and the code that runs before the first paint from 6.1 s to 1.5 s of CPU. I moved more than 1,000 templates out of the main bundle and loaded each one with its route. Then I served the lazy code from a CDN I designed, in chunks built from the module graph.&lt;/p&gt;

&lt;p&gt;In this article I'll walk through how I found where the time went, the tools I built to measure it, each cut with its numbers, the lazy-loading patterns I kept, the traps that broke features, and what the CDN changed.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;The case:&lt;/strong&gt; a large web app with about 1,150 client templates, React panels inside them, and about 9,000 lazy modules. It runs on Meteor 2.16. Some limitations of this work come from Meteor, and I'll name them. The method does not depend on Meteor.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The Result
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Home page in production&lt;/th&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;th&gt;After&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Time until the page starts to show&lt;/td&gt;
&lt;td&gt;12.6 s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.8 s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;I measured the "after" number in production, after the CDN release, logged in: 0.89 s at the median of five loads.&lt;/p&gt;

&lt;p&gt;The other numbers in this article come from the lab, and they are higher than the production number. That is expected. The lab starts every load with an empty cache, so the browser downloads and compiles all the code each time. It runs on a local server, with no CDN and no HTTP/2. Its timings count until the largest element of the page appears. A second lab profile also slows the network to Fast 3G and the CPU by 4x, close to what a slow phone gets. I used the lab only to compare two versions of the code with the same test.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What every page downloads.&lt;/strong&gt; The measurement is deterministic, so these numbers are exact.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Eager bundle, gzip&lt;/th&gt;
&lt;th&gt;Start&lt;/th&gt;
&lt;th&gt;End&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;JavaScript&lt;/td&gt;
&lt;td&gt;5,666 KB&lt;/td&gt;
&lt;td&gt;2,042 KB&lt;/td&gt;
&lt;td&gt;−64.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JavaScript + CSS&lt;/td&gt;
&lt;td&gt;6,070 KB&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2,434 KB&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;−59.9%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;A cold load of &lt;code&gt;/home&lt;/code&gt; in the lab.&lt;/strong&gt; Median of five loads. I measured both columns on the same day.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Measurement&lt;/th&gt;
&lt;th&gt;Start&lt;/th&gt;
&lt;th&gt;After the view tree&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Modules in the main bundle&lt;/td&gt;
&lt;td&gt;11,177&lt;/td&gt;
&lt;td&gt;5,041&lt;/td&gt;
&lt;td&gt;−54.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CPU spent running modules before the first paint&lt;/td&gt;
&lt;td&gt;6,113 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1,454 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;−76.2% (4.2x)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Longest task&lt;/td&gt;
&lt;td&gt;6,476 ms&lt;/td&gt;
&lt;td&gt;2,053 ms&lt;/td&gt;
&lt;td&gt;−68.3% (3.2x)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;First Contentful Paint&lt;/td&gt;
&lt;td&gt;8,704 ms&lt;/td&gt;
&lt;td&gt;2,900 ms&lt;/td&gt;
&lt;td&gt;−66.7% (3.0x)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Largest Contentful Paint&lt;/td&gt;
&lt;td&gt;8,988 ms&lt;/td&gt;
&lt;td&gt;4,068 ms&lt;/td&gt;
&lt;td&gt;−54.7% (2.2x)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LCP, 5 core routes, one load each&lt;/td&gt;
&lt;td&gt;9.6–10.8 s&lt;/td&gt;
&lt;td&gt;3.9–4.8 s&lt;/td&gt;
&lt;td&gt;−53% to −62%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The timings come from the view-tree commit. Three later cuts took another 352 KB gzip out of the JavaScript, and I did not measure their timings in the lab. The bundle table includes them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the Time Went
&lt;/h2&gt;

&lt;p&gt;Before I had a profile, I did two rounds of bundle work: the error-tracking SDK, a second copy of pdf.js, a timezone library, and a few feature routes. They took 622 KB gzip out of the JavaScript. On Fast 3G with a 4x slower CPU, the median LCP of the five core routes fell by 11 s. On &lt;code&gt;localhost&lt;/code&gt;, the LCP of &lt;code&gt;/home&lt;/code&gt; went from 8,684 to 8,172 ms, and the ranges of the two builds overlapped. Fewer bytes helped the slow network. They did not shorten the wait on a fast one.&lt;/p&gt;

&lt;p&gt;So I measured where the time went. The first runtime measurement was a timeline of one cold load of &lt;code&gt;/home&lt;/code&gt;, on the build after those two rounds:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Moment&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;The HTML arrives&lt;/td&gt;
&lt;td&gt;21 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The CSS arrives&lt;/td&gt;
&lt;td&gt;181 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The JavaScript (5,043 KB gzip, 23,189 KB raw) downloads&lt;/td&gt;
&lt;td&gt;59–1,322 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;One long task: the bundle parses and runs&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1,432–6,747 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The WebSocket opens&lt;/td&gt;
&lt;td&gt;6,878 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;First Contentful Paint&lt;/td&gt;
&lt;td&gt;6,884 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Largest Contentful Paint&lt;/td&gt;
&lt;td&gt;7,724 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the five core routes of that build, the LCP was 7.7 to 9.8 s, and the shape was the same on every route. The network delivered the bundle in 1.3 s. Then the tab spent &lt;strong&gt;5.3 s in one long task&lt;/strong&gt;, with nothing on the screen. The first paint came 6 ms after that task ended. No data request could start before it, because the code that opens the socket was inside the same bundle.&lt;/p&gt;

&lt;p&gt;Two more numbers explained the size of the problem:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Only a third of the code ran.&lt;/strong&gt; V8 coverage on &lt;code&gt;/home&lt;/code&gt; showed 7.7 MB of the 23.1 MB of eager JavaScript in use. The browser downloaded and decompressed the other 15.4 MB, and V8 still pre-parsed it, but none of it ran.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A slow network made it extreme.&lt;/strong&gt; On Fast 3G with a 4x slower CPU, the LCP of the five core routes was 51 to 55 s.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the problem had two parts: &lt;strong&gt;too many bytes, and too much CPU at boot&lt;/strong&gt;. They turned out to be separate problems, with separate fixes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measure First: Three Small Tools
&lt;/h2&gt;

&lt;p&gt;The usual tools gave me wrong answers, so I wrote three small scripts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Exact bytes per module.&lt;/strong&gt; A bundle visualizer that reads the source map puts the bytes of generated code (the compiled templates, for example) on a neighboring file, because the map has no entry for them. &lt;code&gt;measureEagerBundle.js&lt;/code&gt; parses the module tree of the built bundle instead, so each module gets its exact byte span. It is deterministic: two builds of the same code give the same numbers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. CPU per module.&lt;/strong&gt; A sampling profile said that one file, the one that creates the app's 112 database collections, cost 2,160 ms. That was wrong. The profile reads positions through the same source map. &lt;code&gt;measureModuleTime.js&lt;/code&gt; replaces the module installer before the bundle runs and wraps every module function with a timer. The real cost of that file was about 280 ms. This is the core of it, simplified:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Runs before the bundle (Playwright addInitScript).&lt;/span&gt;
&lt;span class="c1"&gt;// Meteor gives every module tree to one global installer.&lt;/span&gt;
&lt;span class="c1"&gt;// The hook wraps each module function with a timer.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;stack&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
&lt;span class="nb"&gt;window&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;__moduleTimes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;wrapTree&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isArray&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{};&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;modulePath&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;prefix&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;function&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;object&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nf"&gt;wrapTree&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;modulePath&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;continue&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;timedModule&lt;/span&gt;&lt;span class="p"&gt;(...&lt;/span&gt;&lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;frame&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;child&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
      &lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;frame&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;started&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
      &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;finally&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;elapsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;performance&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;started&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
        &lt;span class="c1"&gt;// The parent's own time does not include the time of its children.&lt;/span&gt;
        &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;child&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nx"&gt;elapsed&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="nb"&gt;window&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;__moduleTimes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;modulePath&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;self&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;elapsed&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;frame&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;child&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
      &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;wrapped&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;defineProperty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;window&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;meteorInstall&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;configurable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;get&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;wrapped&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;realInstall&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;wrapped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;options&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;realInstall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;wrapTree&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tree&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nx"&gt;options&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;assign&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;wrapped&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;realInstall&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No source map is involved. One rule made the report useful: &lt;strong&gt;a large number means "this module is the door to an expensive package"&lt;/strong&gt;. npm packages go through a separate copy of the installer, so they have no row of their own. Their cost goes to the first app module that requires them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The doors.&lt;/strong&gt; &lt;code&gt;edges.js&lt;/code&gt; walks the static import graph from every file that loads on every page, and stops at each dynamic &lt;code&gt;import()&lt;/code&gt;. It answers two questions: which npm packages a page can reach, and which eager file is the first import on the chain to a module. I call that file a &lt;strong&gt;door&lt;/strong&gt;. Close the door, and everything behind it leaves the bundle.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;node scripts/perf/edges.js &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PWD&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; npm:material-react-table   &lt;span class="c"&gt;# the eager importers of one package&lt;/span&gt;
node scripts/perf/edges.js &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PWD&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &amp;lt;directory&amp;gt;                &lt;span class="c"&gt;# the chains that reach a folder&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For page timings, one run was useless. Five cold loads of the same build varied by up to 940 ms on &lt;code&gt;localhost&lt;/code&gt;. I took the median of five loads per side, and I used Fast 3G with a slower CPU when I needed to separate two results.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Cuts, One Door at a Time
&lt;/h2&gt;

&lt;p&gt;First, every step on one table. The bytes are exact. The timings are cold loads of &lt;code&gt;/home&lt;/code&gt; on the same machine, with the same script and the same account.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Step&lt;/th&gt;
&lt;th&gt;Eager JS, gzip&lt;/th&gt;
&lt;th&gt;Saved&lt;/th&gt;
&lt;th&gt;Module evaluation&lt;/th&gt;
&lt;th&gt;Longest task&lt;/th&gt;
&lt;th&gt;LCP&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Start&lt;/td&gt;
&lt;td&gt;5,666 KB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;6,113 ms&lt;/td&gt;
&lt;td&gt;6,476 ms&lt;/td&gt;
&lt;td&gt;8,988 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1. Error-tracking SDK, second pdf.js, timezone library, feature routes&lt;/td&gt;
&lt;td&gt;5,044 KB&lt;/td&gt;
&lt;td&gt;−622 KB&lt;/td&gt;
&lt;td&gt;5,319 ms&lt;/td&gt;
&lt;td&gt;5,772 ms&lt;/td&gt;
&lt;td&gt;8,172 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2. Video recorder&lt;/td&gt;
&lt;td&gt;5,014 KB&lt;/td&gt;
&lt;td&gt;−30 KB&lt;/td&gt;
&lt;td&gt;4,290 ms&lt;/td&gt;
&lt;td&gt;4,438 ms&lt;/td&gt;
&lt;td&gt;7,028 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3. Table library&lt;/td&gt;
&lt;td&gt;4,712 KB&lt;/td&gt;
&lt;td&gt;−302 KB&lt;/td&gt;
&lt;td&gt;3,516 ms&lt;/td&gt;
&lt;td&gt;3,534 ms&lt;/td&gt;
&lt;td&gt;5,708 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4. Server code, date pickers, AI chat panels&lt;/td&gt;
&lt;td&gt;4,496 KB&lt;/td&gt;
&lt;td&gt;−216 KB&lt;/td&gt;
&lt;td&gt;2,712 ms&lt;/td&gt;
&lt;td&gt;3,364 ms&lt;/td&gt;
&lt;td&gt;5,828 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5. Content catalogue, video-call SDK&lt;/td&gt;
&lt;td&gt;4,398 KB&lt;/td&gt;
&lt;td&gt;−164 KB¹&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;3,457 ms²&lt;/td&gt;
&lt;td&gt;5,716 ms²&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6. View tree&lt;/td&gt;
&lt;td&gt;2,394 KB&lt;/td&gt;
&lt;td&gt;−2,004 KB³&lt;/td&gt;
&lt;td&gt;1,454 ms&lt;/td&gt;
&lt;td&gt;2,053 ms&lt;/td&gt;
&lt;td&gt;4,068 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;7. Server-only reporting helpers&lt;/td&gt;
&lt;td&gt;2,345 KB&lt;/td&gt;
&lt;td&gt;−49 KB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8. Material UI off the first paint&lt;/td&gt;
&lt;td&gt;2,107 KB&lt;/td&gt;
&lt;td&gt;−238 KB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;9. jQuery UI&lt;/td&gt;
&lt;td&gt;2,042 KB&lt;/td&gt;
&lt;td&gt;−65 KB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;¹ A merge of new code from the main branch added 66 KB between steps 4 and 5, so the column moves by 98 KB, not 164 KB.&lt;br&gt;
² Median of three loads. The other LCP and longest-task cells are medians of five.&lt;br&gt;
³ Step 6 holds 24 commits. The move of the view tree is the largest.&lt;/p&gt;

&lt;p&gt;I measured the start row and step 6 on the same day. I measured steps 1 to 5 on two earlier days, when the start build had an LCP of 8,684 ms. Read the timing columns as a trend, and the byte column as exact. The table counts JavaScript only. Step 1 also cut the CSS from 404 KB to 271 KB, a revert of that CSS work came back with step 6, and the CSS is now 392 KB.&lt;/p&gt;

&lt;p&gt;Two things stand out. Step 2 saved 30 KB and took 1 s off the boot CPU. Step 6 saved 2 MB, more than all the other steps together.&lt;/p&gt;
&lt;h3&gt;
  
  
  Before the profile: an error-tracking SDK (step 1)
&lt;/h3&gt;

&lt;p&gt;The bundler cannot tree-shake, so a static import of &lt;code&gt;@sentry/browser&lt;/code&gt; brought the SDK with replay, feedback and canvas recording: 916 KB minified. Twelve client files imported it.&lt;/p&gt;

&lt;p&gt;I made one module the only door. It queues every call, and it loads the SDK in &lt;code&gt;requestIdleCallback&lt;/code&gt; after the first route paints, with a 4 s limit. Errors raised before that go into a 20-entry buffer and reach Sentry when it starts. The Sentry packages left the bundle: &lt;strong&gt;−276 KB gzip&lt;/strong&gt;.&lt;/p&gt;
&lt;h3&gt;
  
  
  A recorder that cost 1 second of CPU (step 2)
&lt;/h3&gt;

&lt;p&gt;The CPU report put one module at the top: the app's video recorder component, at 1,032 ms. It was the first file to require &lt;code&gt;react-media-recorder&lt;/code&gt;, which pulls &lt;code&gt;standardized-audio-context&lt;/code&gt;. That library builds a class hierarchy when it loads. It is cheap to download and expensive to run.&lt;/p&gt;

&lt;p&gt;Exactly one eager file imported the recorder, for a panel that renders behind a condition. I made that import lazy:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Cold load of &lt;code&gt;/home&lt;/code&gt;, median of 5&lt;/th&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;th&gt;After&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;FCP&lt;/td&gt;
&lt;td&gt;7,276 ms&lt;/td&gt;
&lt;td&gt;6,256 ms (−14.0%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LCP&lt;/td&gt;
&lt;td&gt;8,172 ms&lt;/td&gt;
&lt;td&gt;7,028 ms (−14.0%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Longest task&lt;/td&gt;
&lt;td&gt;5,772 ms&lt;/td&gt;
&lt;td&gt;4,438 ms (−23.1%)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The bundle barely moved: 5,044 KB to 5,014 KB gzip. &lt;strong&gt;Boot CPU and bundle size are separate problems.&lt;/strong&gt; A byte budget alone would never have found this.&lt;/p&gt;
&lt;h3&gt;
  
  
  A table library behind 22 doors (step 3)
&lt;/h3&gt;

&lt;p&gt;The next module was the app's table wrapper, at 795 ms, the door to &lt;code&gt;material-react-table&lt;/code&gt;. Making that one file lazy did not help: 37 modules imported the package, through 22 different doors. Most of them were template helpers that returned a React component at module scope. Each one became a lazy component.&lt;/p&gt;

&lt;p&gt;Result: −302 KB gzip, −774 ms of module evaluation. After the recorder and the table, the longest task on &lt;code&gt;/home&lt;/code&gt; was 42.7% shorter than at the start.&lt;/p&gt;
&lt;h3&gt;
  
  
  Server code that ran in the browser (step 4)
&lt;/h3&gt;

&lt;p&gt;Shared code had &lt;code&gt;if (Meteor.isClient) return;&lt;/code&gt; at the top, but it still imported the server's event store. Those modules built a validation schema when they loaded: 234 ms of CPU in the browser, for code that never runs there. A function-level &lt;code&gt;require()&lt;/code&gt; stopped the CPU cost, but the bundler still found the literal and kept the bytes.&lt;/p&gt;

&lt;p&gt;The fix that removes the bytes too is a registry: the shared code has an empty slot, and a server-only file fills it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Shared code: the slot.&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;eventStore&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;EventStore&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;registerEventStore&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;store&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;EventStore&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="k"&gt;void&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;eventStore&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;store&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;getEventStore&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt; &lt;span class="nx"&gt;EventStore&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;eventStore&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;The event store is not registered.&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;eventStore&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// server/registerEventStore.ts: the only import of the real module.&lt;/span&gt;
&lt;span class="nf"&gt;registerEventStore&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;append&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  A catalogue that the header loaded for one field (step 5)
&lt;/h3&gt;

&lt;p&gt;The header called a hook, the hook called a helper, and the helper read a content catalogue of 43 items with all their text. It used one field of each item: the &lt;code&gt;_id&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;I moved the 43 ids to a small file, and I kept the full catalogue lazy. A unit test compares the two lists, so a new item that reaches only one file fails the build. Result: &lt;strong&gt;−160 KB gzip&lt;/strong&gt;, and a feature folder fell from 60 eager modules to 12.&lt;/p&gt;

&lt;h3&gt;
  
  
  A video-call SDK that loaded on every page (step 5)
&lt;/h3&gt;

&lt;p&gt;The chat panel sits in the default layout, and its &lt;code&gt;onCreated&lt;/code&gt; hook loaded the video-call SDK: an 843 KB gzip chunk, on every page, for every user. Nobody had started a call.&lt;/p&gt;

&lt;p&gt;Now the SDK loads when a chat window holds a live video. For one screen that needs it right after a click, the download starts at the click, while the modal opens and the server creates the session. &lt;strong&gt;843 KB gzip&lt;/strong&gt; stopped loading on every page.&lt;/p&gt;

&lt;h3&gt;
  
  
  Smaller doors
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Door&lt;/th&gt;
&lt;th&gt;What it cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A date picker package imported through its root barrel&lt;/td&gt;
&lt;td&gt;73 KB gzip, 260 ms of CPU across 3 files&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AI chat panels in 5 template helpers&lt;/td&gt;
&lt;td&gt;the door to a charting library, 97 KB gzip&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A PDF viewer package that listed pdf.js twice&lt;/td&gt;
&lt;td&gt;406 KB minified, one copy ran and exported nothing&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;moment-timezone&lt;/code&gt; from npm next to a global &lt;code&gt;moment&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;763 KB minified, for one component&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  The big one: templates that load with their route (step 6)
&lt;/h3&gt;

&lt;p&gt;Each fix above closed one door. The bundle was still about 4.4 MB gzip, because the framework loads every file under &lt;code&gt;client/&lt;/code&gt; on every page, whatever the imports say. More than 1,000 templates lived there.&lt;/p&gt;

&lt;p&gt;The key observation: &lt;strong&gt;the templating layer finds a template by name when it renders, not when the file loads.&lt;/strong&gt; So a route can wait for a chunk, and then render any template inside it. Only the templates that render on every page must load with the page: 135 of 1,148.&lt;/p&gt;

&lt;p&gt;I moved the view tree to a lazy folder. A list of 155 files loads with the page. The other 604 files load in a route chunk. Each route group waits for its chunk:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;adminArea&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;createRouteChunkLoader&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;importChunk&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/imports/features/admin/ui&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;adminRoutes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;FlowRouter&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;adminRoutes&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;waitOn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;adminArea&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;waitOn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;whileWaiting&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;adminArea&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;whileWaiting&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The helper exists for one reason. The router calls &lt;code&gt;whileWaiting&lt;/code&gt; on every route run, and a query parameter change is a route run. A plain &lt;code&gt;waitOn&lt;/code&gt; showed the loading screen on each change, so the page lost its state. The helper shows the loading screen only while the chunk is on the way.&lt;/p&gt;

&lt;p&gt;This table compares the start with step 6:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Measurement&lt;/th&gt;
&lt;th&gt;Start&lt;/th&gt;
&lt;th&gt;After the view tree&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Eager JavaScript, gzip&lt;/td&gt;
&lt;td&gt;5,666 KB&lt;/td&gt;
&lt;td&gt;2,394 KB (−57.8%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modules in the bundle&lt;/td&gt;
&lt;td&gt;11,177&lt;/td&gt;
&lt;td&gt;5,041 (−54.9%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;npm packages that a page can reach&lt;/td&gt;
&lt;td&gt;57&lt;/td&gt;
&lt;td&gt;40&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Module evaluation on &lt;code&gt;/home&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;6,113 ms&lt;/td&gt;
&lt;td&gt;1,454 ms (−76.2%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Longest task on &lt;code&gt;/home&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;6,476 ms&lt;/td&gt;
&lt;td&gt;2,053 ms (−68.3%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LCP on &lt;code&gt;/home&lt;/code&gt;, median of 5&lt;/td&gt;
&lt;td&gt;8,988 ms&lt;/td&gt;
&lt;td&gt;4,068 ms (−54.7%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Share of the eager code that runs on &lt;code&gt;/home&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;34.6%&lt;/td&gt;
&lt;td&gt;50.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the five core routes, LCP fell between 53% and 62% in the lab, and between 42% and 46% on Fast 3G with a 4x slower CPU. The longest task fell between 68% and 75%.&lt;/p&gt;

&lt;h3&gt;
  
  
  Three more doors after the view tree (steps 7 to 9)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Reporting helpers that only the server calls.&lt;/strong&gt; 48 files moved to a server folder, which never enters the client bundle: −49 KB gzip.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Material UI on the first paint.&lt;/strong&gt; Four static imports still carried it: a barrel file of the component library that 37 files read for the theme, the entry of the confirmation dialog, a menu with its own table, and a demo page. The barrel went away. The dialog entry now loads the dialog on its first call, so its 131 call sites did not change. Result: −238 KB gzip, and the Material UI components in the bundle fell from 73 to 33. The 33 that stay belong to the header, which paints on every page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;jQuery UI.&lt;/strong&gt; The file (247 KB raw) sat in an eager folder, and no file on the first-paint path called a widget. It moved to a lazy folder, and the 67 modules that call a widget import it: −65 KB gzip.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  After the Bundle: the Data Waterfall
&lt;/h2&gt;

&lt;p&gt;A smaller bundle moved the first paint earlier, and then the data became visible. After the long task, the page opened its socket and sent 28 subscriptions for 109 documents. The payload was small. The cost was the number of round trips and their order.&lt;/p&gt;

&lt;p&gt;Only 3 of the 28 held the first paint. That fact told me which fixes would move LCP and which would not:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Duplicate subscriptions.&lt;/strong&gt; Two call sites asked for the same data, so the server ran two publications twice. I gave each subscription one owner: 28 became 26. LCP did not move. I had predicted that, because the duplicates did not hold the paint. The page still did less work.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A 3-second timer from 2018.&lt;/strong&gt; Six subscriptions waited for a &lt;code&gt;setTimeout&lt;/code&gt; of 3,000 ms that no longer had a purpose. They held 80 of the 109 documents. Without the timer, notifications were ready 506 ms after the first subscription message, instead of 3,042 ms. LCP did not move here either: none of the six held the paint.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A blocking &lt;code&gt;waitOn&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;/&lt;/code&gt; and &lt;code&gt;/home&lt;/code&gt; waited for a full publication before they rendered anything, and the same data was already on its way through another subscription. I removed the wait.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A publication that read a whole user document.&lt;/strong&gt; One of the 3 that hold the paint read the full user document, which grows with every login session, to get one field. With a projection it went from 126 ms to 18 ms. The 3 run in parallel, so the gate went from 126 ms to 56 ms, a gain too small for the lab to show in LCP.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Cutting requests that do not hold the paint does not move LCP.&lt;/strong&gt; Find the requests that do, and make them shorter.&lt;/p&gt;

&lt;h2&gt;
  
  
  Patterns I Kept
&lt;/h2&gt;

&lt;h3&gt;
  
  
  A React panel that loads on its first render
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Rendered only when its panel opens, so it loads on demand.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;SecurityPanelLazy&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;lazyComponent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/imports/features/account/ui/SecurityPanel&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;SecurityPanel&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;lazyComponent&lt;/code&gt; shows a spinner until the module arrives. When the download fails, it renders nothing, reports the failure, and shows a toast that offers a reload.&lt;/p&gt;

&lt;h3&gt;
  
  
  Code that runs after a click
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;click button[data-action="create-project"]&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;template&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// The handler stays synchronous: the template layer drops the promise&lt;/span&gt;
  &lt;span class="c1"&gt;// of an async handler, so a failed chunk would leave the button dead.&lt;/span&gt;
  &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/imports/features/projects/ui/CreateProjectDialog&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;then&lt;/span&gt;&lt;span class="p"&gt;(({&lt;/span&gt; &lt;span class="nx"&gt;showCreateProjectDialog&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;showCreateProjectDialog&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;catch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nf"&gt;reportError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;context&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;CreateProjectDialogLoadFailed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;},&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  A library that loads at the moment of use
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;chartPromise&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;loadChart&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;chartPromise&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;chartPromise&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;chart.js&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;then&lt;/span&gt;&lt;span class="p"&gt;(({&lt;/span&gt; &lt;span class="na"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Chart&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;Chart&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;defaults&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;global&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;pointHitDetectionRadius&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;Chart&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
    &lt;span class="c1"&gt;// A failed download clears the promise, so the next chart tries again.&lt;/span&gt;
    &lt;span class="nx"&gt;chartPromise&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;chartPromise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;catch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;chartPromise&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;chartPromise&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every caller waits for the same download, and the global setup of the library runs once. After the &lt;code&gt;await&lt;/code&gt;, check that the component still exists.&lt;/p&gt;

&lt;h3&gt;
  
  
  A small index of large data
&lt;/h3&gt;

&lt;p&gt;Keep the part that eager code reads in a small eager file, keep the full data lazy, and add a test that fails when the two differ. I used this for the content catalogue and for a list of program definitions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Traps That Broke Features
&lt;/h2&gt;

&lt;p&gt;A lazy load changes &lt;strong&gt;when&lt;/strong&gt; code runs. Each of these broke a feature during the work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A side-effect module stopped running.&lt;/strong&gt; A file that registers custom validators moved to a lazy folder, and only the server imported it. Eight forms failed validation with &lt;code&gt;Expected undefined, got string&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A global stopped existing.&lt;/strong&gt; Seven files assigned helpers to a global namespace and moved with no importer. Creating a form failed with &lt;code&gt;... is not a function&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The CSS cascade changed.&lt;/strong&gt; A lazy stylesheet loads last, so its rules win. One component stylesheet imported the global styles and restyled every button.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A route waited for its own chunk.&lt;/strong&gt; A redirect module sat inside the chunk that waits for a route, and the route waited for the redirect module. The page stayed on the loading screen.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Two modules changed order.&lt;/strong&gt; Both read a value that clears when it is read. A deep link after a login landed on the home page.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A pre-commit hook rewrote a vendor file.&lt;/strong&gt; &lt;code&gt;eslint --fix&lt;/code&gt; changed &lt;code&gt;var&lt;/code&gt; to &lt;code&gt;let&lt;/code&gt; in a minified jQuery UI file, and every view that used it failed to load.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A moved file left an import on the old path.&lt;/strong&gt; Neither ESLint nor the build reported it. The browser threw &lt;code&gt;Cannot find module&lt;/code&gt; only when the import ran.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The last one is the most dangerous, because nothing fails before production. I test every changed page three ways: a cold deep link in a private window, a hard reload, and an in-app navigation, then again on Fast 3G.&lt;/p&gt;

&lt;h2&gt;
  
  
  The CDN
&lt;/h2&gt;

&lt;p&gt;Lazy loading moved cost from boot to navigation. After the view tree left the bundle, &lt;code&gt;/home&lt;/code&gt; asked for its route code in one request to the app server: 1,021 KB, between the first paint at 2,900 ms and the LCP at 4,068 ms. The same request had a second problem. After a deploy, the server answered with the code of its new build, so an open tab could receive modules from another build.&lt;/p&gt;

&lt;p&gt;I moved the lazy modules to a CDN with versioned names, so every tab reads the modules of its own build. The full design will be in my next article. These are the parts that matter for speed:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Chunks from the module graph.&lt;/strong&gt; The first version made one request per module. &lt;code&gt;/home&lt;/code&gt; needs 2,591 lazy modules, so it made about 2,600 requests. In a sandbox, a public route with about 2,000 modules took 9.4 to 9.9 s through CloudFront, against 1.7 to 2.1 s through the app server. So I grouped the modules with the rules of Rollup, webpack and Turbopack. On a production build, &lt;code&gt;/home&lt;/code&gt; now loads its 2,591 modules in 24 chunks, plus one index. The chunks carry 1.0% more bytes than the modules that &lt;code&gt;/home&lt;/code&gt; needs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A cache per module.&lt;/strong&gt; Meteor's dynamic-import package already stores each module in IndexedDB under its version. With the CDN, that cache holds only modules of the right build. A second visit to the same build makes one request: the index of the build.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Small deploys stay small.&lt;/strong&gt; After a deploy, the browser plans each chunk: it downloads the whole chunk only when the requests that it saves cost more than its unused bytes. The modules that did not change come from the local cache.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Edges and HTTP/2.&lt;/strong&gt; CloudFront serves the chunks from an edge near the user, with Brotli and up to 128 parallel streams on one connection.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A preload that I removed.&lt;/strong&gt; Preloading the chunks from the HTML gained 0.31 s, and cost about 12 s when the preload failed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Not in production yet.&lt;/strong&gt; A 30 s total timeout per request aborted the largest chunk of &lt;code&gt;/home&lt;/code&gt; (4.3 MB before compression) on a 3G profile while its data was still arriving. The fix aborts a request only after 10 s with no data on any CDN request.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping It Fast
&lt;/h2&gt;

&lt;p&gt;A bundle that shrinks by 60% grows back if nobody watches it. So I wrote down &lt;strong&gt;13 ways the bundle grew back&lt;/strong&gt;, each with an example from the app's history: a new file in an eager folder, one static import into a large tree, a barrel import, a download from code that runs on every page, server code in shared folders, large data in an eager file, a second copy of a library, generated CSS, and others.&lt;/p&gt;

&lt;p&gt;The contribution rules of the project now point to this list. A change that adds a file to an eager folder, or a dependency to client code, starts with a read of it. A release gate in CI also measures the JavaScript that &lt;code&gt;/&lt;/code&gt; downloads on a staging server, and it fails above a budget of 2,042 KiB.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Limitations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Lab and production are different setups.&lt;/strong&gt; The lab numbers come from &lt;code&gt;localhost&lt;/code&gt; with an empty cache and no CDN. The production "after" number is my own measurement for a returning user, on one account and one machine, not a p75 of all users.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A first visit to a route waits for its chunk.&lt;/strong&gt; Lazy loading moves cost from boot to navigation. A failed chunk needs a recovery path, and every page needs a test on a slow network.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Meteor 2 loads every file under &lt;code&gt;client/&lt;/code&gt; and &lt;code&gt;both/&lt;/code&gt; on every page.&lt;/strong&gt; The &lt;code&gt;meteor.mainModule&lt;/code&gt; option turns that off, and this app does not use it. In an app of this size, every file needs an importer before that option can work, which is the same work as the view-tree move. There is no tree shaking either, so every cut needed a move, not only an &lt;code&gt;import()&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A &lt;code&gt;require()&lt;/code&gt; inside a function still bundles the module.&lt;/strong&gt; The bundler reads the literal. Only a registry slot removes the bytes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Each lazy chunk grows the version manifest.&lt;/strong&gt; The main bundle carries the version of every lazy module. It grew from 152 KB to 556 KB raw during this work, and it is now 288 KB gzip, 14% of the eager JavaScript.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Some weight is still on every page.&lt;/strong&gt; The header keeps 33 Material UI components, 128 KB gzip of Material UI in all. The icon font stylesheet blocks the first render, and &lt;code&gt;/home&lt;/code&gt; downloads 453 KB of fonts. About 495 method stubs in a shared folder still reach the browser.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The home chunk is still large.&lt;/strong&gt; &lt;code&gt;/home&lt;/code&gt; asks for the chunk of the whole authenticated area, not only for the templates that it renders.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;In production, the home page went from more than 10 seconds to &lt;strong&gt;under 1 second&lt;/strong&gt; for a returning user. In the lab, the same work cut 60% of the bytes and 76% of the boot CPU. These are the lessons I keep:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Measure bytes and CPU separately.&lt;/strong&gt; A 1 s CPU cost hid behind a library that was cheap to download.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Find the doors, not the files.&lt;/strong&gt; One static import from an eager file can bring a whole package. Close the door, and the package leaves.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Build your own measurements when the usual tools are wrong.&lt;/strong&gt; Source maps put bytes and CPU on the wrong files. Five runs per side, not one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The large win was structural.&lt;/strong&gt; The fixes to single doors saved tens to hundreds of KB each. Moving the view tree to its routes saved about 2 MB gzip.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fix what holds the paint.&lt;/strong&gt; Requests that do not hold the first paint do not move LCP.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Give the lazy code a short path too.&lt;/strong&gt; Versioned chunks on a CDN, a cache that matches the build, and a few well-sized requests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Guard it.&lt;/strong&gt; A written list of the known regressions and a budget in CI keep the bundle where you left it.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>webdev</category>
      <category>javascript</category>
      <category>performance</category>
      <category>frontend</category>
    </item>
    <item>
      <title>Monitoring AI Quality: Evaluation Pipelines for Non-Deterministic Products</title>
      <dc:creator>Isaque Silva Chaves</dc:creator>
      <pubDate>Tue, 20 Oct 2026 19:12:21 +0000</pubDate>
      <link>https://dev.to/isaque_silvachaves/monitoring-ai-quality-evaluation-pipelines-for-non-deterministic-products-355j</link>
      <guid>https://dev.to/isaque_silvachaves/monitoring-ai-quality-evaluation-pipelines-for-non-deterministic-products-355j</guid>
      <description>&lt;p&gt;Shipping an AI feature is the easy part. Knowing that it still works next month is the hard one.&lt;/p&gt;

&lt;p&gt;In a traditional application, the same input always gives you the same output. That's why tests work so well: you write &lt;code&gt;expect(sum(2, 2)).toBe(4)&lt;/code&gt; once, and it protects you forever. AI features break that contract. Send the same input to an LLM twice and you get two different answers. Both may be good. One may be a bit worse. And one day, after a prompt tweak or a silent model update, all of them may be wrong, with no error in the logs.&lt;/p&gt;

&lt;p&gt;In this article we'll walk through how to monitor the quality of an AI feature: a pipeline that generates real outputs, asks a second model to grade them, and tells the team in Slack when quality drops. Then we'll go one step further and run it on a schedule with cron jobs, so you also catch the problems that don't come from your own deploys.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;How to read this article:&lt;/strong&gt; if you're on the product or leadership side, the first sections explain the problem and what to measure, with no code required. If you're a developer, the implementation and the cron examples are in "Building the Pipeline" and "Beyond CI".&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;To keep things concrete, we'll use one example throughout the article: a &lt;strong&gt;meeting assistant&lt;/strong&gt;. It reads the transcript of a recorded call and writes a structured summary: key decisions, action items with owners, and highlights tied to moments in the recording. Each team picks a summary template (sales call, standup, job interview, customer support and so on), so the same call should produce a different summary depending on that choice. That's a lot of ways to be almost right.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Should Product Leaders Care?
&lt;/h2&gt;

&lt;p&gt;With AI, quality is not a value, it's a distribution. Your feature doesn't produce "the right answer"; it produces a range of answers, and most of them are good. The job is to keep that range where you want it.&lt;/p&gt;

&lt;p&gt;Think of it like a restaurant chain. The same dish is cooked by different cooks, in different kitchens, with ingredients from different suppliers. Nobody tastes every plate, but someone tastes a sample against the recipe on a regular schedule. That's what an evaluation pipeline does for an AI feature.&lt;/p&gt;

&lt;p&gt;The tricky part is that quality can change even when nobody touches the code. There are three common sources of change:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Your prompts:&lt;/strong&gt; the team improves a prompt for one case and, without noticing, makes another case worse.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your model provider:&lt;/strong&gt; models get updated, deprecated and replaced. A new version can be better on average and still worse for your specific task.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your inputs:&lt;/strong&gt; customers start using the feature in ways you didn't plan, with longer calls, other languages or new templates.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of these throws an error. The output just gets a little worse, and your users notice before you do. For an AI feature, that's the real cost: not a crash, but &lt;strong&gt;trust&lt;/strong&gt;. A feature that is great 90% of the time and strange the other 10% teaches users to double-check everything, and then it stops saving them time.&lt;/p&gt;

&lt;p&gt;An evaluation pipeline gives you three things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A number you can track.&lt;/strong&gt; "Is the AI good?" becomes "is accuracy still above the level we agreed on?".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A gate before release.&lt;/strong&gt; A prompt change that lowers quality is caught on staging, not by a customer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An early warning.&lt;/strong&gt; A scheduled run catches the changes that come from outside your team.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  How Do You Test Something That Never Gives the Same Answer?
&lt;/h2&gt;

&lt;p&gt;You stop checking for &lt;em&gt;the&lt;/em&gt; answer and start checking for &lt;strong&gt;properties&lt;/strong&gt; of a good answer. Instead of &lt;code&gt;expect(output).toBe(...)&lt;/code&gt;, you ask: is it well-formed? Does it cover the whole call? Is it faithful to the transcript? Each question needs a different kind of check, and they get more expensive as they get smarter.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;What it answers&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Deterministic&lt;/td&gt;
&lt;td&gt;Is the output well-formed?&lt;/td&gt;
&lt;td&gt;Free, instant&lt;/td&gt;
&lt;td&gt;The summary is valid JSON with every required section&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Metric (code)&lt;/td&gt;
&lt;td&gt;Does it have a measurable property?&lt;/td&gt;
&lt;td&gt;Cheap&lt;/td&gt;
&lt;td&gt;Highlights are spread across the whole recording&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LLM-as-a-judge&lt;/td&gt;
&lt;td&gt;Is it accurate, complete and useful?&lt;/td&gt;
&lt;td&gt;One extra LLM call per sample&lt;/td&gt;
&lt;td&gt;A 1 to 100 score for each metric you define, like accuracy or clarity&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All of them run on top of the same foundation:&lt;/p&gt;

&lt;h3&gt;
  
  
  1. A Golden Dataset
&lt;/h3&gt;

&lt;p&gt;A &lt;strong&gt;golden dataset&lt;/strong&gt; is a fixed set of realistic inputs that represent how your customers use the feature. For our meeting assistant, that could be 20 fixtures: 15 call transcripts and 5 written meeting notes, covering every summary template. A second set of 5 timestamped recordings is used for the coverage check. Use synthetic or anonymized data, never raw customer content.&lt;/p&gt;

&lt;p&gt;Because the inputs never change, the scores are comparable over time. If last week's average was higher than today's, something on our side or the provider's side changed.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Metric Checks
&lt;/h3&gt;

&lt;p&gt;Some qualities can be measured with plain code, no AI needed. A good one for our example is &lt;strong&gt;coverage&lt;/strong&gt;. If the summary only cites moments from the first ten minutes of a one-hour call, the reader gets half the picture. So we split the recording into bins of about four minutes and measure how evenly the summary's timestamps spread across them. A score of 0 means everything is bunched in one place; 1 means perfectly even.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. LLM-as-a-Judge
&lt;/h3&gt;

&lt;p&gt;For the things code can't measure, like "was this decision really made in the call?", you use a second model as a &lt;strong&gt;judge&lt;/strong&gt;. It receives the source, the generated output and a rubric, and grades the output.&lt;/p&gt;

&lt;p&gt;The most important part is the rubric, and it should be &lt;strong&gt;yours&lt;/strong&gt;. Start by writing down the metrics that define a good output for your product. For our meeting assistant, they could be:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Accuracy:&lt;/strong&gt; every decision and action item really happened in the call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Completeness:&lt;/strong&gt; no decision or action item was left out.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Owners:&lt;/strong&gt; every action item has a person responsible for it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clarity:&lt;/strong&gt; someone who missed the meeting understands the summary.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A support bot would have different ones, like tone, policy compliance or whether the issue was solved. Then ask the judge to return a &lt;strong&gt;score from 1 to 100 for each metric&lt;/strong&gt;, with a one-sentence reason for each score. Run it with temperature 0 to keep it as stable as possible.&lt;/p&gt;

&lt;p&gt;Why a separate score per metric instead of one grade? Because it lets you decide what "good enough" means for each metric on its own, as we'll see in the thresholds.&lt;/p&gt;

&lt;p&gt;The judge is also an LLM, so treat its scores as measurements with some noise, like a thermometer that can be off by a degree. Before trusting the numbers, compare a few judge scores with a human expert's opinion, and adjust the rubric until they mostly agree.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Thresholds
&lt;/h3&gt;

&lt;p&gt;A score is only useful when it's compared against something. With a 1 to 100 score per metric, you set a &lt;strong&gt;threshold for each metric&lt;/strong&gt;, based on how much that metric matters to your product:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Threshold&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Accuracy&lt;/td&gt;
&lt;td&gt;90&lt;/td&gt;
&lt;td&gt;An invented decision destroys trust&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Owners&lt;/td&gt;
&lt;td&gt;85&lt;/td&gt;
&lt;td&gt;Action items without an owner don't get done&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Completeness&lt;/td&gt;
&lt;td&gt;80&lt;/td&gt;
&lt;td&gt;A missed item is bad, but easy to add by hand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Clarity&lt;/td&gt;
&lt;td&gt;70&lt;/td&gt;
&lt;td&gt;Style matters, but less than the facts&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These numbers are only an example. Choosing them is a product decision, not an engineering one: a healthcare product may want accuracy at 98, while a brainstorming tool may be fine with 70. Together, the thresholds are the quality level you expect from the feature, written down.&lt;/p&gt;

&lt;p&gt;There are two ways to apply them, and you'll probably want both:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;On the average across the golden set&lt;/strong&gt;, for the run as a whole. One strange sample doesn't fail the run, but a general drop does.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On each sample&lt;/strong&gt;, for the metrics where a single bad output is unacceptable. One invented decision may be enough to fail the run.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Code metrics, like coverage, work the same way: decide how even the spread must be for your product, and fail the run below that.&lt;/p&gt;

&lt;p&gt;A natural next step for &lt;strong&gt;consistency&lt;/strong&gt; is to run the same input several times and measure the spread of the scores. A feature with an average of 80 and a small spread is a better product than one with the same average that swings between 60 and 95.&lt;/p&gt;

&lt;h2&gt;
  
  
  Building the Pipeline
&lt;/h2&gt;

&lt;p&gt;The pipeline itself doesn't need to be complex. It's a script that runs the golden set against a real environment, calling the same endpoints the product uses, so it tests the whole stack at once: prompts, model, parsing and response format. Run it as a CI job, like a GitHub Action, every time something that can change quality changes: a prompt, a model version or the code around them.&lt;/p&gt;

&lt;p&gt;![Evaluation pipeline: a prompt or model change, or a scheduled run, runs the golden set; a rubric check and a coverage check run in parallel; the results go to the job summary and to a Slack message that is green only when both checks pass]&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsm4on1y5yfz5hqoxzqss.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsm4on1y5yfz5hqoxzqss.png" alt=" " width="800" height="775"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The two checks run side by side, and a failure in either one turns the Slack message red.&lt;/p&gt;
&lt;h3&gt;
  
  
  Generate, Then Grade
&lt;/h3&gt;

&lt;p&gt;The core of the pipeline is a Python script, &lt;code&gt;run_evals.py&lt;/code&gt;. For each fixture it makes two calls to OpenAI: one to generate the summary with the same prompt and model the product uses, and one to ask the judge to grade it. Here is a simplified version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;openai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;AsyncOpenAI&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AsyncOpenAI&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;  &lt;span class="c1"&gt;# reads OPENAI_API_KEY from the environment
&lt;/span&gt;
&lt;span class="n"&gt;MODEL&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;gpt-6-luna&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;  &lt;span class="c1"&gt;# the same model the product uses
&lt;/span&gt;&lt;span class="n"&gt;THRESHOLDS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;accuracy&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;owners&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;85&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;completeness&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;clarity&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;


&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;generate_summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# Step 1: generate the summary with the product's own prompt
&lt;/span&gt;    &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;system&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;SUMMARY_PROMPT&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;user&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;


&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;judge_summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="c1"&gt;# Step 2: ask the judge for a 1-100 score per metric
&lt;/span&gt;    &lt;span class="n"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;chat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;completions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;MODEL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;reasoning_effort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;none&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;  &lt;span class="c1"&gt;# temperature only works with reasoning off
&lt;/span&gt;        &lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;response_format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;type&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;json_object&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="n"&gt;messages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;system&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;JUDGE_PROMPT&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;role&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;user&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;content&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Source:&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="se"&gt;\n\n&lt;/span&gt;&lt;span class="s"&gt;Summary:&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="p"&gt;],&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# JudgeResult validates the answer, we'll see it in the next section
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;JudgeResult&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;model_validate_json&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;choices&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;message&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;content&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;process_fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;source&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;fixture&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;source&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;  &lt;span class="c1"&gt;# a transcript or written notes
&lt;/span&gt;    &lt;span class="n"&gt;summary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;generate_summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;judge_summary&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;source&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;run_all&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;fixtures&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load_fixtures&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;gather&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;process_fixture&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;fixtures&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;

    &lt;span class="n"&gt;failed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;threshold&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;THRESHOLDS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="n"&gt;average&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;PASS&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;average&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;threshold&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;average&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; (threshold &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;threshold&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;) &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;average&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;threshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;failed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;failed&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;run_all&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you read our article on &lt;a href="https://dev.to/quave/improving-javascript-performance-understanding-async-tasks-in-arrays-4el4"&gt;async tasks in arrays&lt;/a&gt;, &lt;code&gt;asyncio.gather&lt;/code&gt; will look familiar: it's Python's &lt;code&gt;Promise.all&lt;/code&gt;. With the SDK's async client, all fixtures run in parallel. LLM calls are slow, so running them one by one would make the pipeline far slower.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;SUMMARY_PROMPT&lt;/code&gt; and &lt;code&gt;MODEL&lt;/code&gt; should be exactly what the product uses, and &lt;code&gt;JUDGE_PROMPT&lt;/code&gt; is the rubric from the next section. To keep the example short, step 1 calls OpenAI directly. If your feature runs behind your own API, call that API instead, so each run also tests what's around the model, like how the prompt is built and how the answer is parsed.&lt;/p&gt;

&lt;p&gt;A complete version should also keep failed requests apart from low scores. A request that errors is reported by name, and any failure fails the run, because "the AI service returned a 500" and "the AI got worse" are different problems.&lt;/p&gt;

&lt;h3&gt;
  
  
  Asking the Judge
&lt;/h3&gt;

&lt;p&gt;The judge's prompt lists your metrics and asks for a strict JSON answer. A trimmed version for our example:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;You are evaluating a meeting summary against the call transcript.
Score each metric from 1 to 100 and explain each score in one sentence.

- accuracy: every decision and action item really happened in the call
- completeness: no decision or action item was left out
- owners: every action item has a person responsible for it
- clarity: someone who missed the meeting understands the summary

Answer only with JSON:
{"scores": {"accuracy": &amp;lt;1-100&amp;gt;, ...}, "reasons": {"accuracy": "&amp;lt;one sentence&amp;gt;", ...}}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Validate the answer before using it. The judge is an LLM too, so sometimes it returns a score out of range, or something that isn't JSON at all. With Pydantic, that becomes an error instead of a made-up number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;typing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Annotated&lt;/span&gt;

&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pydantic&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Field&lt;/span&gt;

&lt;span class="n"&gt;Score&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Annotated&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Field&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ge&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;le&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;


&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;JudgeResult&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;BaseModel&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Score&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;  &lt;span class="c1"&gt;# one 1-100 score per metric
&lt;/span&gt;    &lt;span class="n"&gt;reasons&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;  &lt;span class="c1"&gt;# one sentence per metric
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A score per metric tells you &lt;strong&gt;what kind&lt;/strong&gt; of problem you have. Low accuracy usually means the AI is inventing things that didn't happen in the call. Low completeness means items were skipped. Low clarity means the summary is vague. Each one points to a different fix, and the reasons tell you where to look.&lt;/p&gt;

&lt;p&gt;Version the judge's prompt too, just like the prompts it evaluates; a prompt management tool like Langfuse makes that easy. And if you want one headline number for a dashboard, the average of your metrics works well.&lt;/p&gt;

&lt;h3&gt;
  
  
  Reporting Where People Look
&lt;/h3&gt;

&lt;p&gt;A check that nobody sees is a check that doesn't exist. Every run publishes its results in three places:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The CI job summary&lt;/strong&gt;, with each metric's score next to its threshold.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Artifacts&lt;/strong&gt; with every generated output. A score tells you something is wrong; the output tells you what.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A Slack message&lt;/strong&gt;, green or red, so the whole team sees the result without opening the CI.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  What Usually Goes Wrong
&lt;/h3&gt;

&lt;p&gt;Evaluation pipelines are not immune to the problem they solve. They call slow, non-deterministic services, so they time out, the judge occasionally returns something unparseable, and a deploy may not be live yet when the run starts. When that happens often enough, teams tend to switch the automatic run to manual, and a manual check is one that people forget to run.&lt;/p&gt;

&lt;p&gt;The lesson: &lt;strong&gt;separate "the pipeline broke" from "quality dropped"&lt;/strong&gt;. They need different alerts and often different owners. If every red message might be noise, people stop reading them, and then the pipeline protects nobody.&lt;/p&gt;

&lt;h2&gt;
  
  
  Beyond CI: Running Evaluations on a Schedule
&lt;/h2&gt;

&lt;p&gt;A pipeline that runs after a deploy answers one question: &lt;em&gt;did we break something?&lt;/em&gt; But remember the three sources of change. Two of them, the model provider and your customers' inputs, never come with a deploy. A model can change on a Tuesday afternoon while your team is in a meeting. To catch that, the evaluation has to run on a clock, not on a commit.&lt;/p&gt;

&lt;p&gt;There are two kinds of scheduled runs, and they answer different questions:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scheduled run&lt;/th&gt;
&lt;th&gt;Inputs&lt;/th&gt;
&lt;th&gt;Question it answers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Golden set on a cron&lt;/td&gt;
&lt;td&gt;The same fixtures every time&lt;/td&gt;
&lt;td&gt;Did something change under us?&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Production sampling&lt;/td&gt;
&lt;td&gt;A random sample of real outputs&lt;/td&gt;
&lt;td&gt;How good is what customers actually get?&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The golden set is your &lt;strong&gt;control group&lt;/strong&gt;: the inputs never change, so any change in the score comes from the system. Production sampling is your &lt;strong&gt;reality check&lt;/strong&gt;: the inputs change every day, and that's the point.&lt;/p&gt;

&lt;h3&gt;
  
  
  Option 1: The Golden Set on a Cron
&lt;/h3&gt;

&lt;p&gt;The easiest step is to reuse what you already have. Take the same validation script and add a &lt;code&gt;schedule&lt;/code&gt; trigger:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cron&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;11&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;1-5"&lt;/span&gt; &lt;span class="c1"&gt;# weekdays at 11:00 UTC&lt;/span&gt;
  &lt;span class="na"&gt;workflow_dispatch&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few things to keep in mind with scheduled GitHub workflows:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The cron expression is always in &lt;strong&gt;UTC&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Scheduled runs only start from the &lt;strong&gt;default branch&lt;/strong&gt;, and they can be delayed when GitHub is under heavy load.&lt;/li&gt;
&lt;li&gt;In public repositories, scheduled workflows are disabled after 60 days without activity in the repository.&lt;/li&gt;
&lt;li&gt;If you point it at production, run the fixtures under a dedicated test account or tenant, so evaluation data never mixes with real customer data and is easy to filter out.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Option 2: Sampling Production With a Cron Job
&lt;/h3&gt;

&lt;p&gt;The golden set tells you if the system changed. It doesn't tell you how the AI is doing with the inputs customers send today. For that, you sample real outputs and grade them with the same judge.&lt;/p&gt;

&lt;p&gt;Here's what that job can look like in Node, using &lt;code&gt;node-cron&lt;/code&gt; for the schedule and &lt;code&gt;p-map&lt;/code&gt; to limit how many judge calls run at once:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;cron&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node-cron&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;pMap&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;p-map&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;SAMPLE_SIZE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;MAX_DROP&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// points below the 7-day average&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;SIX_HOURS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;6&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;average&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;values&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="nx"&gt;values&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reduce&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;sum&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;value&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;values&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nx"&gt;cron&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;schedule&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;0 */6 * * *&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// 1. Sample real outputs generated since the last run&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;samples&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;AiOutputs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;$match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;createdAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;$gte&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;SIX_HOURS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;$sample&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;size&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;SAMPLE_SIZE&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;]).&lt;/span&gt;&lt;span class="nf"&gt;toArray&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="c1"&gt;// 2. Grade them with the same judge and metrics used in CI, 4 at a time&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;pMap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sample&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;concurrency&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="c1"&gt;// 3. Store the run, so we can see trends and not only pass/fail&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;metrics&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;keys&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;averages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fromEntries&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nx"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nf"&gt;average&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;])),&lt;/span&gt;
    &lt;span class="p"&gt;])&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;EvalRuns&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;insertOne&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;createdAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="nx"&gt;averages&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;promptVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;promptVersion&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;samples&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(({&lt;/span&gt; &lt;span class="nx"&gt;outputId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;scores&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;outputId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;scores&lt;/span&gt; &lt;span class="p"&gt;})),&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="c1"&gt;// 4. Alert on any metric that dropped below its recent baseline&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;baseline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;getRollingAverages&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;days&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;dropped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;metrics&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;averages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;MAX_DROP&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;dropped&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;notifySlack&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="nx"&gt;dropped&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
          &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
            &lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;averages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;toFixed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;&lt;span class="s2"&gt; vs 7-day average &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;baseline&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;toFixed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;AiOutputs&lt;/code&gt;, &lt;code&gt;EvalRuns&lt;/code&gt;, &lt;code&gt;judge&lt;/code&gt;, &lt;code&gt;getRollingAverages&lt;/code&gt; and &lt;code&gt;notifySlack&lt;/code&gt; stand for your own collections and helpers. The shape is what matters: &lt;strong&gt;sample, grade, store, compare&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Some points that are easy to miss:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Run it once per cluster.&lt;/strong&gt; If your app runs on several instances, &lt;code&gt;node-cron&lt;/code&gt; fires on every one of them, and you pay for the same evaluation several times. Use a scheduler that runs once, like a Kubernetes CronJob, or a package with a shared lock, like SyncedCron in Meteor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Privacy comes first.&lt;/strong&gt; Production sampling sends real customer data to the judge. Keep the judge inside the same compliance boundary as the feature itself (same provider, same data agreements) and store scores and IDs, not copies of the content. This matters even more with health, financial or children's data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sample size is your cost control.&lt;/strong&gt; Each sample is one more LLM call. Twenty samples every six hours is 80 judge calls a day; pick a number your budget can live with.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Store the versions.&lt;/strong&gt; Saving the prompt version and the model with each run is what lets you answer the first question everyone asks when the score drops: &lt;em&gt;what changed?&lt;/em&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Alert on Trends, Not Only Thresholds
&lt;/h3&gt;

&lt;p&gt;A fixed threshold catches big drops. But if accuracy goes from 97 to 91, it still passes a threshold of 90, and something clearly got worse. Comparing each run against a rolling baseline, like the 7-day average in the example above, catches the slow slides that a fixed number misses.&lt;/p&gt;

&lt;p&gt;And remember the lesson from the previous section: give "the job failed to run" its own alert. A monitor that fails silently looks exactly like a monitor that says everything is fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to Track
&lt;/h2&gt;

&lt;p&gt;One number is enough to start, but a few more turn a red light into a diagnosis. These are the metrics we find useful, and who usually reads each one:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;What it tells you&lt;/th&gt;
&lt;th&gt;Who reads it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Average judge score&lt;/td&gt;
&lt;td&gt;Overall quality, the headline number&lt;/td&gt;
&lt;td&gt;Product, leadership&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Score vs. 7-day baseline&lt;/td&gt;
&lt;td&gt;Whether quality is sliding, even above the threshold&lt;/td&gt;
&lt;td&gt;Product, leadership&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Score per scenario (e.g. per summary template)&lt;/td&gt;
&lt;td&gt;Where the AI is weak, and which customers feel it&lt;/td&gt;
&lt;td&gt;Product&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Score per metric (e.g. accuracy, clarity)&lt;/td&gt;
&lt;td&gt;What kind of problem to fix&lt;/td&gt;
&lt;td&gt;Engineering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spread across repeated runs&lt;/td&gt;
&lt;td&gt;Consistency: how much the answer varies for the same input&lt;/td&gt;
&lt;td&gt;Product, engineering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Coverage and other code metrics&lt;/td&gt;
&lt;td&gt;Measurable properties the judge can miss&lt;/td&gt;
&lt;td&gt;Engineering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Latency per call&lt;/td&gt;
&lt;td&gt;How long users wait for the AI&lt;/td&gt;
&lt;td&gt;Product, engineering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failed runs and request errors&lt;/td&gt;
&lt;td&gt;Whether the monitor itself is healthy&lt;/td&gt;
&lt;td&gt;Engineering&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens per run&lt;/td&gt;
&lt;td&gt;What the feature costs, and what monitoring it costs&lt;/td&gt;
&lt;td&gt;Leadership&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Put the numbers where people already look. For leadership, that can be a single daily line in Slack: the average, the trend against the baseline, passed or failed. For engineers, it's the job summary with each metric's score and a link to the generated outputs. The goal is that nobody has to remember to go check a dashboard.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;AI features don't fail like regular code. They don't crash; they drift. A prompt change, a model update or a new kind of input can make the output a little worse, and no log will tell you. The only way to know is to keep measuring.&lt;/p&gt;

&lt;p&gt;The good news is that the tools for this are the ones you already use: a test script, a CI workflow, a cron job and a Slack channel. What changes is what you assert. Not the exact answer, but the properties of a good one: well-formed, complete, faithful to the source, and good enough by a rubric that your team agreed on.&lt;/p&gt;

&lt;p&gt;If you want to start this week, here's the path we'd follow again:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Collect 10 to 20 realistic, anonymized inputs that cover your main use cases.&lt;/li&gt;
&lt;li&gt;Add the cheap checks first: valid format and required fields.&lt;/li&gt;
&lt;li&gt;Write down the metrics that define "good" for your product, and ask a judge for a 1 to 100 score on each.&lt;/li&gt;
&lt;li&gt;Run it once, read the scores next to the outputs, and set a threshold for each metric from that baseline.&lt;/li&gt;
&lt;li&gt;Run it after every deploy that touches prompts or models.&lt;/li&gt;
&lt;li&gt;Run it on a schedule, and sample production when you're ready.&lt;/li&gt;
&lt;li&gt;Send the results where people already look, and store every run to see the trend.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Your AI feature will never give the same answer twice. With an evaluation pipeline watching it, it can still give the same quality.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>devops</category>
    </item>
    <item>
      <title>Improving JavaScript Performance: Understanding Async Tasks in Arrays</title>
      <dc:creator>Isaque Silva Chaves</dc:creator>
      <pubDate>Tue, 27 Aug 2024 17:24:31 +0000</pubDate>
      <link>https://dev.to/quave/improving-javascript-performance-understanding-async-tasks-in-arrays-4el4</link>
      <guid>https://dev.to/quave/improving-javascript-performance-understanding-async-tasks-in-arrays-4el4</guid>
      <description>&lt;p&gt;When working with JavaScript, understanding how arrays work is crucial. Arrays are one of the most basic data structures, but when you're dealing with asynchronous operations, their behavior can significantly impact your application's performance. This article explores the inner workings of JavaScript arrays and provides helpful advice for performance optimization.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Event&amp;nbsp;Loop
&lt;/h2&gt;

&lt;p&gt;Before diving into arrays, it's essential to understand the concept of the event loop in JavaScript. JavaScript is single-threaded, meaning it can execute one operation at a time.&lt;br&gt;
The event loop continually checks the call stack to see if there's any function that needs to run. When an asynchronous operation is initiated, like an API request or a timer, JavaScript offloads this operation to the web APIs (or Node.js APIs), which handle them separately. Once the operation completes, a callback is queued in the event loop to be executed when the call stack is empty.&lt;/p&gt;
&lt;h2&gt;
  
  
  Arrays in the Event&amp;nbsp;Loop
&lt;/h2&gt;

&lt;p&gt;Arrays in JavaScript are often used in conjunction with asynchronous operations. For example, you might have an array of URLs you need to fetch or a list of files you need to process. The way you loop through these arrays can have a significant impact on performance, especially when dealing with large datasets or slow I/O operations.&lt;/p&gt;
&lt;h3&gt;
  
  
  forEach vs. for…of: Async operations perspective
&lt;/h3&gt;

&lt;p&gt;Consider the two most common ways to loop through arrays: &lt;strong&gt;forEach&lt;/strong&gt; and &lt;strong&gt;for…of&lt;/strong&gt;. At first glance, they don't seem to have any difference, but they behave differently, especially in the context of asynchronous operations.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;forEach Loop&lt;/strong&gt;: The &lt;strong&gt;forEach&lt;/strong&gt; loop is often used for its simplicity and readability. However, it does not play well with asynchronous operations. When using forEach with asynchronous functions like fetch or some request to a database, the loop continues execution without waiting for the asynchronous operation to complete. This can lead to unexpected behavior, where all requests are initiated simultaneously, and the loop finishes before any of the requests are resolved.
&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
 &lt;span class="nx"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;forEach&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
   &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
 &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;In this example, &lt;strong&gt;forEach&lt;/strong&gt; will not wait for the fetch call to complete before moving on to the next iteration. This can cause performance issues, especially if you're relying on the order of execution, also it can overload to where this request is being sent, such as a database and/or a external API service with rate limiting&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;for…of Loop&lt;/strong&gt;: In contrast, the &lt;strong&gt;for…of loop&lt;/strong&gt; works better with asynchronous operations. When used with await, the loop pauses at each iteration until the promise resolves. This ensures that the operations are completed in sequence, which can be crucial when order matters or when dealing with rate-limited APIs.
&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;url&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
   &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;The &lt;strong&gt;for…of loop&lt;/strong&gt; is more predictable and often better suited for scenarios where you need to ensure each operation completes before moving to the next one.&lt;/p&gt;
&lt;h2&gt;
  
  
  Impact on Performance
&lt;/h2&gt;

&lt;p&gt;The difference between &lt;strong&gt;forEach&lt;/strong&gt; and &lt;strong&gt;for…of&lt;/strong&gt; becomes more evident when dealing with large arrays or slow asynchronous tasks. Using forEach might seem faster initially because all tasks are initiated almost simultaneously. However, this can lead to issues like overwhelming the system with too many concurrent operations, causing memory spikes, exceeding API rate limits or overloading your database.&lt;br&gt;
On the other hand, while &lt;strong&gt;for…of&lt;/strong&gt; might be slower because it waits for each operation to complete before moving on, it often leads to more stable and predictable performance. This approach ensures that your application does not exceed available resources and can handle errors more gracefully.&lt;/p&gt;
&lt;h2&gt;
  
  
  Using Generators for Efficient Data Processing
&lt;/h2&gt;

&lt;p&gt;Generators in JavaScript provide a powerful way to manage asynchronous tasks more efficiently. Unlike traditional functions, generators can yield multiple values over time, making them ideal for processing large datasets or streams of data.&lt;br&gt;
By using generators, you can create a pipeline that processes data as it becomes available, rather than waiting for the entire dataset to be loaded. This can significantly improve performance, especially in scenarios where you need to process data on demand.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nf"&gt;fetchUrls&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
 &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;url&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
   &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;then&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
 &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;generator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetchUrls&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;request&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;generator&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
 &lt;span class="nx"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;then&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In this example, the generator fetches URLs one at a time, processing each response as soon as it arrives. This method ensures that your application remains responsive and can handle large datasets without consuming excessive memory since you're not storing all the data in memory. Additionally, you're able to provide an answer to the user as soon as possible. This is similar to what streaming services do; they don't wait to load the entire video before making it available to the user. Instead, they deliver the video on-demand, which provides a better user experience and prevents overloading your server's memory or the user's memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Using Array.map and Promise.all for Async Operations
&lt;/h2&gt;

&lt;p&gt;When you use &lt;code&gt;map&lt;/code&gt; in conjunction with an async function in JavaScript, it's essential to understand what happens under the hood. The &lt;code&gt;map&lt;/code&gt; method is designed to transform each element of an array based on the function provided to it. If this function is asynchronous it returns a promise, then &lt;code&gt;map&lt;/code&gt; itself will return an array of promises, not the resolved values of those promises.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;url3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;fetchData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// Applying the async function with map&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;promises&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;fetchData&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;promises&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;// Output: [Promise, Promise, Promise]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;These promises are placeholders for the eventual results, which will only be available after the asynchronous operations are complete.&lt;br&gt;
To actually obtain the resolved values, you need to use &lt;code&gt;Promise.all&lt;/code&gt;, which waits for all promises in the array to resolve, returning a new promise that fulfills with an array of the resolved values:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;promises&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="cm"&gt;/* Output: [data1, data2, data3]
By using Array.map, you can apply an asynchronous function to each URL and gather all results simultaneously.
Here's how it works: */&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/data1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/data2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/data3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;fetchData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// Initiate all fetch requests concurrently&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;promises&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;fetchData&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;// Wait for all fetch requests to complete&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;promises&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In this example, &lt;code&gt;urls.map(fetchData)&lt;/code&gt; creates an array of promises, where each promise represents the asynchronous fetchData operation applied to a URL. These promises are then passed to &lt;code&gt;Promise.all&lt;/code&gt;, which runs them in parallel. Promise.all waits for all the promises to resolve and returns their results as an array. This approach leverages JavaScript's ability to handle asynchronous tasks concurrently, making it possible to fetch data from all URLs at the same time.&lt;br&gt;
However, while this method is powerful and efficient, it has some drawbacks. Because all fetch requests are initiated at once, this can result in a large number of simultaneous network requests, potentially overwhelming the server, exceeding API rate limits, or consuming too much memory, especially when dealing with a large array of URLs or slow network conditions. There's no built-in way to limit the number of concurrent operations, which can lead to resource exhaustion and degrade performance.&lt;br&gt;
To manage such scenarios more effectively, it's often better to use a tool like &lt;code&gt;p-map&lt;/code&gt;, which allows you to control the concurrency level, ensuring that only a limited number of requests are processed at any given time. This can help maintain application stability and prevent overwhelming your system's resources.&lt;/p&gt;
&lt;h2&gt;
  
  
  Using &lt;a href="https://github.com/sindresorhus/p-map" rel="noopener noreferrer"&gt;p-map&lt;/a&gt; for Efficient Async Operations
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://github.com/sindresorhus/p-map" rel="noopener noreferrer"&gt;p-map&lt;/a&gt; is a lightweight library that helps you handle asynchronous operations on arrays with controlled concurrency. It allows you to map over an array of promises and specify a limit on the number of concurrent operations. This is particularly useful when you need to perform tasks like fetching data from multiple URLs, processing files, or interacting with APIs, as it helps prevent overwhelming your server or exceeding API rate limits. By using p-map, you can strike a balance between speed and resource management, ensuring that your application remains stable and responsive under load.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;pMap&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;p-map&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;urls&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/data1&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/data2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;https://api.example.com/data3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;];&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;fetchData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;response&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// Limit the number of concurrent fetch requests to 2&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;pMap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;urls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;fetchData&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;concurrency&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In this example, &lt;a href="https://github.com/sindresorhus/p-map" rel="noopener noreferrer"&gt;p-map&lt;/a&gt; processes the URLs with a concurrency limit of 2, meaning only two fetch requests are in progress at any given time. This approach helps you avoid overloading the server with too many simultaneous requests while still taking advantage of concurrency to improve performance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusion
&lt;/h2&gt;

&lt;p&gt;Understanding how arrays interact with the event loop and asynchronous operations is key to optimizing JavaScript performance. While &lt;strong&gt;forEach&lt;/strong&gt; may seem convenient, it can lead to performance bottlenecks in asynchronous tasks. On the other hand, &lt;strong&gt;for…of loops&lt;/strong&gt;, combined with generators, provide a more robust solution for handling asynchronous operations in a more controlled way.&lt;/p&gt;

</description>
    </item>
  </channel>
</rss>
