<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mike Vu</title>
    <description>The latest articles on DEV Community by Mike Vu (@mikevu).</description>
    <link>https://dev.to/mikevu</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3940235%2F59d061e5-cbf4-40bf-a87e-10406a6d93b0.jpg</url>
      <title>DEV Community: Mike Vu</title>
      <link>https://dev.to/mikevu</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mikevu"/>
    <language>en</language>
    <item>
      <title>Parsing money the way Vietnamese people type it: 2,5tr, 500k and 1tr+500k</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Wed, 18 Nov 2026 21:32:38 +0000</pubDate>
      <link>https://dev.to/mikevu/parsing-money-the-way-vietnamese-people-type-it-25tr-500k-and-1tr500k-1d46</link>
      <guid>https://dev.to/mikevu/parsing-money-the-way-vietnamese-people-type-it-25tr-500k-and-1tr500k-1d46</guid>
      <description>&lt;p&gt;In Vietnam nobody types "2500000" for an electricity bill. People write "2,5tr", "2.5 triệu", "500k", "0,1 tỷ" or "2.500.000", and in a spreadsheet they happily type "=250k*3". &lt;a href="https://finance.vuminhhoa.com/" rel="noopener noreferrer"&gt;Financial Plan&lt;/a&gt;, my budgeting app, is almost entirely amount fields, so it had to read all of these. This post covers the small parser behind them, the dot that means two different things, sums typed like in Excel, keeping the caret in place while formatting, and the missing + key on phone keyboards.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a number input can't hold Vietnamese amounts
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;&amp;lt;input type="number"&amp;gt;&lt;/code&gt; rejects letters, so "2,5tr" is out, and it reads separators by the browser's rules, not the user's. Vietnamese also uses the opposite marks to English: &lt;code&gt;vi-VN&lt;/code&gt; groups thousands with a dot and uses a comma for decimals, so "2.500.000" is two million five hundred thousand. Yet the same people will write "2.5tr" with a dot as the decimal point, because that is how phone keyboards and English-language apps taught them.&lt;/p&gt;

&lt;p&gt;So every amount in the app is a text field with &lt;code&gt;inputmode="decimal"&lt;/code&gt;, which asks phones for a numeric keypad, and a parser that turns the text into a number when the field is saved.&lt;/p&gt;

&lt;h2&gt;
  
  
  One figure: a number plus an optional unit
&lt;/h2&gt;

&lt;p&gt;A single figure is digits and separators, optionally followed by a unit. The units are the ones people actually type: &lt;code&gt;k&lt;/code&gt;, &lt;code&gt;n&lt;/code&gt;, &lt;code&gt;nghìn&lt;/code&gt;, &lt;code&gt;ngàn&lt;/code&gt; for thousands; &lt;code&gt;tr&lt;/code&gt;, &lt;code&gt;triệu&lt;/code&gt;, &lt;code&gt;trieu&lt;/code&gt;, &lt;code&gt;m&lt;/code&gt; for millions; &lt;code&gt;tỷ&lt;/code&gt;, &lt;code&gt;tỉ&lt;/code&gt;, &lt;code&gt;ty&lt;/code&gt;, &lt;code&gt;b&lt;/code&gt; for billions.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/^&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;\&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;ngh&lt;/span&gt;ì&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;ng&lt;/span&gt;à&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;tr&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;tri&lt;/span&gt;ệ&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;trieu&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;ỷ&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;ỉ&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;ty&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;$&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before matching, the input is lowercased, spaces are removed, and so are the currency's code, symbol and common spellings (&lt;code&gt;VND&lt;/code&gt;, &lt;code&gt;₫&lt;/code&gt;, &lt;code&gt;vnđ&lt;/code&gt;, &lt;code&gt;đ&lt;/code&gt;), plus a leading &lt;code&gt;=&lt;/code&gt;. "100.000đ", "100.000 VND" and "100000" all reach the parser as the same string.&lt;/p&gt;

&lt;p&gt;The unit decides what the separators mean:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;With a unit, the separator is a decimal point.&lt;/strong&gt; "2,5tr" and "2.5tr" are both 2,500,000; the comma is replaced by a dot and the figure is multiplied by the unit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Without a unit, in a currency with no decimals, every dot and comma is grouping.&lt;/strong&gt; "2.500.000", "2,500,000" and "2500000" are the same number. The Vietnamese đồng has no subunit in use, so a bare figure is always whole.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;In a currency with decimals,&lt;/strong&gt; the currency's own locale decides: its grouping mark is dropped and its decimal mark becomes the point. "$2,500.50" and "2.500,50 €" each read correctly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The result is rounded to the currency's decimal places, so a sum like "100k/3" saves as 33,333 đ, not 33,333.333.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwamdcsrzoejfusscy1qw.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fwamdcsrzoejfusscy1qw.png" alt="Decision flow for one typed figure: clean up the input, then if there is a unit (k, tr, triệu, tỷ, m, b) the comma or dot is a decimal point and the figure is multiplied by the unit; otherwise, if the currency has decimals, its own locale decides both marks; otherwise every dot and comma is grouping." width="800" height="683"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The unit decides what a separator means: with a unit it is a decimal point, without one, in đồng, it only groups digits.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The dot that means two things
&lt;/h2&gt;

&lt;p&gt;Formatting while typing is where the ambiguity bites. Showing "2.500.000" as the user types "2500000" is helpful; turning "2.5" into "25" because the dot looked like a grouping mark is not.&lt;/p&gt;

&lt;p&gt;The rule the app settled on: only reformat text that is unambiguously a plain whole figure, meaning digits only, or digits in groups of three separated by dots.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="no"&gt;PLAIN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;/^-?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;\&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;+|&lt;/span&gt;\&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;\&lt;span class="p"&gt;.&lt;/span&gt;\&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;$&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;"2500000" and "2.500.000" match and get grouped. "2.5tr", "0,1 tỷ" and "14.31" don't, so they are left exactly as typed and only converted to the full figure when the field loses focus. One exception keeps editing pleasant: if the previous value was a figure the formatter produced, deleting a digit from "2.500.000" leaves "2.50.000", which no longer matches the pattern but should still be regrouped.&lt;/p&gt;

&lt;p&gt;What this does not solve is a bare decimal in đồng. With no unit every dot is grouping, so "2.5" saves as 25 đ and "14.31" as 1,431 đ. The parser can't tell a slip from a real figure, so whoever means two and a half million has to type the unit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Keeping the caret where it was
&lt;/h2&gt;

&lt;p&gt;Reformatting a field moves its text around, and a naïve update sends the caret to the end, which makes fixing a digit in the middle impossible. The fix is to count digits rather than characters: before formatting, count how many digits sit before the caret; after formatting, walk the new string and place the caret after the same number of digits. Inserted dots don't count, so the caret stays after the digit the user just typed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sums, as in a spreadsheet
&lt;/h2&gt;

&lt;p&gt;People budget with arithmetic: "1tr+500k" for two payments, "=250k*3" for three of something, "(3tr+2tr)/2" for an average. Any amount field accepts this.&lt;/p&gt;

&lt;p&gt;The input is split on operators, including the ones phones and Excel users type: &lt;code&gt;×&lt;/code&gt; and &lt;code&gt;x&lt;/code&gt; for multiplication, &lt;code&gt;÷&lt;/code&gt; for division, and &lt;code&gt;−&lt;/code&gt; (the real minus sign) as well as &lt;code&gt;-&lt;/code&gt;. Then a small recursive-descent parser evaluates it with the usual precedence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;sum&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;product&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"+"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"-"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tokens&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"+"&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nf"&gt;product&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;product&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;product&lt;/code&gt; handles &lt;code&gt;*&lt;/code&gt; and &lt;code&gt;/&lt;/code&gt; the same way, and &lt;code&gt;factor&lt;/code&gt; handles unary minus, brackets and single figures, each parsed with the rules above, so "1,5tr*2" mixes units and arithmetic. Anything that doesn't fully parse returns &lt;code&gt;NaN&lt;/code&gt;: tokens left over, an unclosed bracket, or a result that isn't finite, such as a division by zero. The field then keeps its previous value and a message shows what it couldn't read, with examples of what it can, instead of saving a wrong number.&lt;/p&gt;

&lt;p&gt;Two details make it usable:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A leading minus is a figure, not a sum.&lt;/strong&gt; "-500k" is a negative amount; "1tr-500k" is a sum. The check strips one leading minus before looking for operators.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The result shows under the field while typing&lt;/strong&gt;, and what gets saved is the figure, not the expression. Reopening the field shows "1.500.000", which is what the rest of the app works with.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Phones have no + key on the number pad
&lt;/h2&gt;

&lt;p&gt;On iOS, &lt;code&gt;inputmode="decimal"&lt;/code&gt; brings up a keypad of digits and a decimal separator, with no plus or minus at all. So on touch devices, detected with &lt;code&gt;matchMedia("(pointer: coarse)")&lt;/code&gt;, a focused amount field shows its own row of &lt;code&gt;+ − × ÷&lt;/code&gt; keys.&lt;/p&gt;

&lt;p&gt;Those keys have to type into a React-controlled input as if the user had pressed them. Setting &lt;code&gt;input.value&lt;/code&gt; directly doesn't trigger React's &lt;code&gt;onChange&lt;/code&gt;, so the code goes through the native setter and dispatches an &lt;code&gt;input&lt;/code&gt; event:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getOwnPropertyDescriptor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;HTMLInputElement&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prototype&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"value"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;el&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;el&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dispatchEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"input"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;bubbles&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nf"&gt;requestAnimationFrame&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;el&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setSelectionRange&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;caret&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;caret&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The caret is restored on the next frame, after React has re-rendered the field.&lt;/p&gt;

&lt;p&gt;These fields hit one more iOS quirk: the page zooms into any field with text under 16 px. That fix sits with the other iPhone details in &lt;a href="https://vuminhhoa.com/en/blog/installed-pwa-iphone-updates-deploys" rel="noopener noreferrer"&gt;keeping an installed PWA usable on iPhone across updates and deploys&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Displaying amounts: let Intl decide
&lt;/h2&gt;

&lt;p&gt;Output uses &lt;a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/NumberFormat" rel="noopener noreferrer"&gt;&lt;code&gt;Intl.NumberFormat&lt;/code&gt;&lt;/a&gt; rather than hand-written rules. The app asks &lt;code&gt;formatToParts&lt;/code&gt; how the currency's home locale writes 1234.5, and reads off the symbol, which side it goes on, whether there's a space, the grouping and decimal marks and the number of decimals. That gives "2.500.000 ₫", "$2,500.50" and "2.500,50 €" from the same code; supporting another currency is one line naming its locale. Short forms follow the interface language: "2,5 tr" in Vietnamese and "2.5M" in English.&lt;/p&gt;

&lt;p&gt;Negative amounts use the real minus sign "−" rather than a hyphen, which lines up with "+" and reads correctly in screen readers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Tests pin down what people type
&lt;/h2&gt;

&lt;p&gt;The parser is easy to break with a well-meant change, so its tests are a table of real inputs:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Typed&lt;/th&gt;
&lt;th&gt;Read as&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;5.500.000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;5,500,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;5.5tr&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;5,500,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;5,5 triệu&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;5,500,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;500k&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;500,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;0,1 tỷ&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;100,000,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;-2tr&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;−2,000,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;=250k*3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;750,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;abc&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;not a number&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;empty&lt;/td&gt;
&lt;td&gt;no value&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;What the app does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Units and Vietnamese separators&lt;/td&gt;
&lt;td&gt;Text field with &lt;code&gt;inputmode="decimal"&lt;/code&gt;; a unit makes the separator a decimal point, no unit makes it grouping&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Formatting while typing&lt;/td&gt;
&lt;td&gt;Only plain whole figures are regrouped; anything else waits for blur&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Caret jumping to the end&lt;/td&gt;
&lt;td&gt;Place it after the same number of digits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sums like "1tr+500k"&lt;/td&gt;
&lt;td&gt;Tokenizer plus a recursive-descent parser; result shown live, figure saved&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No + key on phone keypads&lt;/td&gt;
&lt;td&gt;A row of &lt;code&gt;+ − × ÷&lt;/code&gt; keys on touch devices, typed through the native value setter&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Several currencies&lt;/td&gt;
&lt;td&gt;Symbol, side, marks and decimals read from &lt;code&gt;Intl.NumberFormat&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;None of this is large, a few hundred lines with the tests, but it is the part of a budgeting app that users touch most. An amount field that understands "1tr+500k" is one fewer reason to keep a calculator open next to it.&lt;/p&gt;

&lt;p&gt;If you've built an amount field that reads "2,5tr" or "500k" in your own way, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>parsing</category>
      <category>intlnumberformat</category>
      <category>forms</category>
    </item>
    <item>
      <title>An installed PWA on iPhone: safe updates, deploys and iOS details</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Tue, 17 Nov 2026 21:53:12 +0000</pubDate>
      <link>https://dev.to/mikevu/an-installed-pwa-on-iphone-safe-updates-deploys-and-ios-details-7a2</link>
      <guid>https://dev.to/mikevu/an-installed-pwa-on-iphone-safe-updates-deploys-and-ios-details-7a2</guid>
      <description>&lt;p&gt;A PWA added to an iPhone's home screen looks like a native app, and people treat it like one: they open it from the icon, leave it in the background for days, and expect it to keep working when a new version ships. &lt;a href="https://finance.vuminhhoa.com/" rel="noopener noreferrer"&gt;Financial Plan&lt;/a&gt;, my budgeting app, is used mostly this way. This post collects what it took to make updates safe, survive deploys that replace files under an open tab, and get the iOS details right: a logo that jumped on launch, fields that zoomed the page, and push notifications.&lt;/p&gt;

&lt;p&gt;How the app syncs its data offline is a separate topic, covered in &lt;a href="https://vuminhhoa.com/en/blog/local-first-pwa-offline-sync" rel="noopener noreferrer"&gt;local-first PWA sync&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ship updates when the user is ready, not mid-sentence
&lt;/h2&gt;

&lt;p&gt;The service worker precaches the whole app, so it opens offline and while the servers are down. The other side of that is that a new version doesn't appear by itself: the browser downloads it in the background, and it waits until the open pages are done with the old one.&lt;/p&gt;

&lt;p&gt;The app decides when to switch:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A visible tab gets a toast&lt;/strong&gt;, "a new version is ready", with a Reload button that stays until used. Reloading on its own could throw away a half-typed form.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A hidden tab switches at once.&lt;/strong&gt; Nobody is looking, so there is nothing to interrupt.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open tabs check for updates every 30 minutes and whenever they become visible again&lt;/strong&gt;, because an installed app may stay open for days without a navigation that would trigger the browser's own check.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is also a manual path in Settings › Version &amp;amp; updates. Every build writes a &lt;code&gt;/version.json&lt;/code&gt; that is never cached, so the page can compare its own build ID with the latest one. That page distinguishes three states that are easy to merge by mistake: &lt;em&gt;up to date&lt;/em&gt;, &lt;em&gt;ready&lt;/em&gt; (a newer build is downloaded and applying it is one reload), and &lt;em&gt;failed&lt;/em&gt; (a newer build exists, but it couldn't be downloaded, for example offline). Showing "up to date" when the download failed would be a lie.&lt;/p&gt;

&lt;p&gt;Finally, the app calls &lt;a href="https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist" rel="noopener noreferrer"&gt;&lt;code&gt;navigator.storage.persist()&lt;/code&gt;&lt;/a&gt; so the browser doesn't evict its IndexedDB data under storage pressure. For a local-first app, that data may be the only copy of changes not yet synced.&lt;/p&gt;

&lt;h2&gt;
  
  
  A deploy can break the tabs that are already open
&lt;/h2&gt;

&lt;p&gt;The bug: a tab running yesterday's version navigates to a page whose code is loaded lazily. That chunk's file name contains a content hash from yesterday's build, and today's deploy no longer has it. The host's single-page-app rewrite answered &lt;em&gt;every&lt;/em&gt; unknown path with &lt;code&gt;index.html&lt;/code&gt;, so the browser received HTML where it expected JavaScript, failed to parse it, and the page broke.&lt;/p&gt;

&lt;p&gt;Three changes fixed it:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Missing assets are a real 404.&lt;/strong&gt; The rewrite now skips &lt;code&gt;/assets/&lt;/code&gt;, so a request for a chunk that no longer exists fails cleanly instead of returning HTML.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Caching follows what can change.&lt;/strong&gt; Hashed files under &lt;code&gt;/assets/&lt;/code&gt; are cached for a year as &lt;code&gt;immutable&lt;/code&gt;; &lt;code&gt;index.html&lt;/code&gt;, the service worker scripts and &lt;code&gt;version.json&lt;/code&gt; are &lt;code&gt;no-cache&lt;/code&gt;, so the next load always asks for the current ones.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A failed chunk moves the tab to the latest version.&lt;/strong&gt; Vite fires a &lt;a href="https://vite.dev/guide/build#load-error-handling" rel="noopener noreferrer"&gt;&lt;code&gt;vite:preloadError&lt;/code&gt;&lt;/a&gt; event when a dynamic import fails. The app handles it by activating the waiting service worker if there is one, or reloading otherwise.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The recovery has a guard, because a real outage also makes chunks fail to load, and reloading forever would make it worse:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;last&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Number&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sessionStorage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getItem&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;RELOAD_KEY&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;sessionStorage&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setItem&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;RELOAD_KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;last&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;30_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;toast&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;t&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pwa.loadFailed"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"stale-version"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At most one automatic reload every 30 seconds; after that, the user sees an error and decides.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Figymmda0l0g3uizpades.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Figymmda0l0g3uizpades.png" alt="Sequence diagram between an open tab running yesterday's build and the host serving today's deploy. Before: the tab requests a chunk with an old hash, the SPA rewrite answers 200 with index.html, and the page breaks because it got HTML instead of JavaScript. After: the same request gets a 404 because the rewrite skips /assets/, and the vite:preloadError handler activates the waiting service worker or reloads, at most once every 30 seconds." width="800" height="809"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A clean 404 turns a broken page into an error the app can recover from.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The launch logo that moved, and measuring instead of guessing
&lt;/h2&gt;

&lt;p&gt;The app shows its logo from the very first paint, in plain HTML outside React, and fades it out once the first page is ready. On an installed iPhone app, the logo visibly jumped upward a moment after launch.&lt;/p&gt;

&lt;p&gt;The first fix assumed the cause: the status bar appears a moment after the first paint and changes the page's top edge, so the logo was positioned from the bottom edge instead. It still moved. So a temporary build logged the real sizes on the device, and the numbers said something else entirely: the page is &lt;strong&gt;844 px tall at the first paint and 797 px tall 74 ms later&lt;/strong&gt;. Its &lt;em&gt;bottom&lt;/em&gt; edge moves up past the home indicator; its top edge and &lt;code&gt;100vh&lt;/code&gt; stay where they are.&lt;/p&gt;

&lt;p&gt;With that known, the fix was simple. In standalone mode, the splash is a fixed box &lt;code&gt;100vh&lt;/code&gt; tall from the top, and the logo sits in the middle of that box, which doesn't change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight css"&gt;&lt;code&gt;&lt;span class="k"&gt;@media&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nt"&gt;display-mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nt"&gt;standalone&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nc"&gt;.splash&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;bottom&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100vh&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The logo still shifted on the screen as the page resized, so two days later one more step went in: in the installed app, a small classic script (&lt;code&gt;public/splash.js&lt;/code&gt;) keeps the logo hidden until the page has its final size, on that resize or after 300 ms without one, and then fades it in over 150 ms.&lt;/p&gt;

&lt;p&gt;The measuring code came out again in the commit right after the one that added it. The lesson is the useful part: two reasonable explanations of a layout shift were both wrong, and logging &lt;code&gt;innerHeight&lt;/code&gt; frame by frame for the first 4 seconds on the real device took minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fields that zoomed the page
&lt;/h2&gt;

&lt;p&gt;iOS zooms the page into any text field whose font size is under 16 px when it gets focus, in Safari and in the installed app alike. The app's fields were 14 px, so every tap on an amount zoomed in and left the layout shifted.&lt;/p&gt;

&lt;p&gt;Fields are now 16 px on phones and 14 px from the &lt;code&gt;sm&lt;/code&gt; breakpoint up, and the viewport sets &lt;code&gt;maximum-scale=1&lt;/code&gt;. On iOS that stops the automatic zoom while still allowing pinch zoom; Chrome on Android disables pinch zoom with that setting unless the user forces zoom in its accessibility options, which is worth knowing before copying the meta tag.&lt;/p&gt;

&lt;h2&gt;
  
  
  A tab bar that minimizes like iOS's
&lt;/h2&gt;

&lt;p&gt;On phones the app has a bottom tab bar that shrinks while scrolling down, as Safari's own toolbar does. The first version also grew back at the end of short pages, and iOS's rubber-band bounce past the end counted as scrolling up. The rules that made it feel native:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Only scrolling &lt;em&gt;up&lt;/em&gt; brings the full bar back, never reaching the end.&lt;/li&gt;
&lt;li&gt;The bounce past the top or the bottom is ignored.&lt;/li&gt;
&lt;li&gt;It takes 24 px of movement in one direction to change state.&lt;/li&gt;
&lt;li&gt;Pages that scroll less than 240 px always keep the full bar.&lt;/li&gt;
&lt;li&gt;Every page starts with the full bar; restoring its scroll position isn't scrolling down.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Push notifications from a local-first app
&lt;/h2&gt;

&lt;p&gt;Web Push works on iPhone only for web apps added to the home screen, since &lt;a href="https://webkit.org/blog/13878/web-push-for-web-apps-on-ios-and-ipados/" rel="noopener noreferrer"&gt;iOS 16.4&lt;/a&gt;. The app's reminders are things like a credit card bill due in three days, last month not closed yet, or a term deposit maturing.&lt;/p&gt;

&lt;p&gt;The data lives on the device, but a phone can't run code on a schedule from a web app. So the server works the reminders out from the synced records, with the &lt;strong&gt;same calculation engine&lt;/strong&gt; the app uses (it lives in a package shared by both). That avoids a second, slightly different implementation of "when is this bill due". A few details:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One notification a day&lt;/strong&gt;, around 8:00 Vietnam time, with everything due. Each reminder is claimed in a table &lt;em&gt;before&lt;/em&gt; it is sent, so a retry or a second server instance can't send it twice.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Amounts are hidden by default&lt;/strong&gt;, because notifications show on the lock screen.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://webkit.org/blog/16535/meet-declarative-web-push/" rel="noopener noreferrer"&gt;Declarative Web Push&lt;/a&gt;&lt;/strong&gt;, available since iOS 18.4, lets the system show a notification from the payload alone, without waking a service worker. Other browsers get the same payload through a small push service worker that shows it and opens the right page on tap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Subscriptions are cleaned up.&lt;/strong&gt; Endpoints are accepted only on the browsers' push services, dropped on HTTP 404 or 410, and dropped if the app hasn't renewed them in 30 days. Signing out removes the device's subscription.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Opening the app clears the icon's badge through the &lt;a href="https://developer.mozilla.org/en-US/docs/Web/API/Badging_API" rel="noopener noreferrer"&gt;Badging API&lt;/a&gt;, so the number on the icon is only the reminders not yet seen.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;What the app does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Updates interrupting work&lt;/td&gt;
&lt;td&gt;Toast with Reload in visible tabs, silent switch in hidden ones, checks every 30 min and on focus&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;"Up to date" when it isn't&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/version.json&lt;/code&gt; never cached; up to date, ready and failed shown apart&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Deploys breaking open tabs&lt;/td&gt;
&lt;td&gt;404 for missing assets, immutable hashed files, &lt;code&gt;vite:preloadError&lt;/code&gt; recovery with a 30 s guard&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Logo jumping on launch&lt;/td&gt;
&lt;td&gt;Measured on the device: the bottom edge moves; logo centred in a &lt;code&gt;100vh&lt;/code&gt; box, shown once the page has resized&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Page zooming into fields&lt;/td&gt;
&lt;td&gt;16 px fields on phones, &lt;code&gt;maximum-scale=1&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reminders for an offline app&lt;/td&gt;
&lt;td&gt;Computed on the server with the shared engine; claimed before sending; amounts hidden&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Most of these were found by using the app every day on an iPhone, not by testing it in a desktop browser with a phone-sized window. An installed PWA is a different environment, and it has to be tested as one.&lt;/p&gt;

&lt;p&gt;If you've shipped updates to a PWA that people keep installed on their iPhones, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>pwa</category>
      <category>serviceworker</category>
      <category>ios</category>
      <category>webpush</category>
    </item>
    <item>
      <title>A large tree in IndexedDB with RxDB: indexes, trigram search and lazy loading</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Tue, 17 Nov 2026 14:00:16 +0000</pubDate>
      <link>https://dev.to/mikevu/a-large-tree-in-indexeddb-with-rxdb-indexes-trigram-search-and-lazy-loading-1lpd</link>
      <guid>https://dev.to/mikevu/a-large-tree-in-indexeddb-with-rxdb-indexes-trigram-search-and-lazy-loading-1lpd</guid>
      <description>&lt;p&gt;&lt;a href="https://plan.vuminhhoa.com/" rel="noopener noreferrer"&gt;Day planner&lt;/a&gt;, my time-management app, keeps a library of &lt;em&gt;blocks&lt;/em&gt;: reusable activities that nest inside each other ("Weekend study" contains "Focus session 1", "Break", "Reading", and "Reading" has its own chapters). Everything lives in the browser, in IndexedDB through &lt;a href="https://rxdb.info/" rel="noopener noreferrer"&gt;RxDB&lt;/a&gt;, and the library can grow without limit. This post covers what made a tree in IndexedDB slow, and the changes that made reading, searching and drawing it depend on what's on screen rather than on the size of the whole table: an index that RxDB couldn't build, a trigram search that ignores Vietnamese diacritics, lazy loading, virtualization, and tests that count queries instead of milliseconds.&lt;/p&gt;

&lt;h2&gt;
  
  
  A query that took 19.6 seconds for 61 nodes
&lt;/h2&gt;

&lt;p&gt;Each block stores its parent in &lt;code&gt;parent_id&lt;/code&gt;, which is &lt;code&gt;null&lt;/code&gt; for top-level blocks. Loading a subtree meant asking for the children of one level at a time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;snapshot_nodes&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;selector&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;parent_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nv"&gt;$in&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;parentIds&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That looks like an index lookup. It wasn't. A day's plan with two levels and 61 nodes, in a table of 2,061 nodes, took &lt;strong&gt;19.6 seconds&lt;/strong&gt; to load, because every level was a full scan of the table.&lt;/p&gt;

&lt;p&gt;The reason is in &lt;a href="https://rxdb.info/rx-schema.html" rel="noopener noreferrer"&gt;RxDB's schema rules&lt;/a&gt;: indexes are only allowed on fields of type &lt;code&gt;string&lt;/code&gt;, &lt;code&gt;integer&lt;/code&gt; or &lt;code&gt;number&lt;/code&gt;. A nullable field is declared as &lt;code&gt;["string", "null"]&lt;/code&gt;, which is neither, so &lt;code&gt;parent_id&lt;/code&gt; could never be indexed. Without an index, &lt;code&gt;$in&lt;/code&gt; falls back to reading every document.&lt;/p&gt;

&lt;h2&gt;
  
  
  A non-null shadow column, filled by a hook
&lt;/h2&gt;

&lt;p&gt;The obvious fix, storing a sentinel like &lt;code&gt;""&lt;/code&gt; in &lt;code&gt;parent_id&lt;/code&gt; instead of &lt;code&gt;null&lt;/code&gt;, would have broken dozens of places that check &lt;code&gt;if (!block.parent_id)&lt;/code&gt; to mean "this is a root". So the fix adds a second column instead:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;parent_key&lt;/code&gt;&lt;/strong&gt; is a non-null copy of &lt;code&gt;parent_id&lt;/code&gt;, equal to it, or &lt;code&gt;""&lt;/code&gt; for a root. It exists only to be indexed; &lt;code&gt;parent_id&lt;/code&gt; keeps its meaning everywhere else.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A &lt;code&gt;preInsert&lt;/code&gt; and &lt;code&gt;preSave&lt;/code&gt; hook fills it on every write&lt;/strong&gt; (repository calls, bulk inserts, upserts, migrations), so no code path can forget it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A schema migration backfills it&lt;/strong&gt; for blocks that already exist on users' devices.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;With that, loading a subtree is one indexed lookup per level, and the number of round trips is the depth of the tree (two or three in practice), not the number of nodes.&lt;/p&gt;

&lt;p&gt;The same trick was later needed for &lt;code&gt;category_id&lt;/code&gt;, for the same reason. A shadow column also has consequences for sync, which are a story of their own in &lt;a href="https://vuminhhoa.com/en/blog/rxdb-sync-silent-failures" rel="noopener noreferrer"&gt;four bugs that silently broke RxDB sync&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read one branch at a time, not the whole table
&lt;/h2&gt;

&lt;p&gt;Even with the index, the library page originally &lt;em&gt;observed the whole table&lt;/em&gt; and derived the tree in JavaScript: build a map, group by parent, count children. Everything on the page cost time proportional to the total number of blocks.&lt;/p&gt;

&lt;p&gt;Now the tree loads lazily:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Roots and the children of each expanded branch&lt;/strong&gt; are separate reactive queries on &lt;code&gt;parent_key&lt;/code&gt;. Collapsing a branch drops its subscription.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Breadcrumbs and detail panels&lt;/strong&gt; walk up by &lt;code&gt;findById&lt;/code&gt; instead of searching an in-memory copy of the table.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deleting or duplicating a subtree&lt;/strong&gt; walks it breadth-first through the index.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A test checks the property that matters. It reads the 20 children of one branch in a table of 120 blocks, then again in a table of 2,020 blocks (16.8 times larger), and compares the times. When the page watched the whole table, the ratio would be close to 16.8; with the index it came out at about &lt;strong&gt;0.95×&lt;/strong&gt;, i.e. constant. The test fails if the ratio goes above 6, a generous margin because it runs on &lt;code&gt;fake-indexeddb&lt;/code&gt; in CI.&lt;/p&gt;

&lt;h2&gt;
  
  
  Substring search without a full-text index
&lt;/h2&gt;

&lt;p&gt;Search is harder. Users expect "contains" matching: typing "họp" should find "Cuộc họp nhóm". An index on &lt;code&gt;title&lt;/code&gt; can only help with prefixes, and RxDB has no full-text search, so the first version loaded every block and filtered in JavaScript.&lt;/p&gt;

&lt;p&gt;The fix is a small inverted index in the style of a search engine, kept in its own &lt;strong&gt;local-only&lt;/strong&gt; collection that is never synced:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Normalize the text&lt;/strong&gt;: strip Vietnamese diacritics, lowercase, collapse spaces. NFD decomposition removes most marks, but &lt;code&gt;đ&lt;/code&gt; is its own letter in Unicode, not &lt;code&gt;d&lt;/code&gt; plus a mark, so it needs an explicit rule.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Split into trigrams&lt;/strong&gt;, overlapping three-character pieces: "cuoc hop" gives &lt;code&gt;cuo&lt;/code&gt;, &lt;code&gt;uoc&lt;/code&gt;, &lt;code&gt;oc&lt;/code&gt;, &lt;code&gt;c h&lt;/code&gt;, &lt;code&gt;ho&lt;/code&gt;, &lt;code&gt;hop&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Store one row per (block, trigram)&lt;/strong&gt; with an index on the trigram.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;To search&lt;/strong&gt;, split the query the same way, fetch the rows matching any of its trigrams with one indexed &lt;code&gt;$in&lt;/code&gt;, then &lt;strong&gt;verify&lt;/strong&gt; each candidate by checking that its normalized title really contains the normalized query. Results are ranked by how many trigrams matched.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvjvol3ogusxel1vhs2df.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvjvol3ogusxel1vhs2df.png" alt="Flow of the trigram search: the title Cuộc họp nhóm is normalized to cuoc hop nhom, split into overlapping three-character trigrams, and stored as one indexed row per block and trigram in a local-only collection. A query such as hop is split the same way and fetched with one indexed $in, then each candidate is kept only if its normalized title really contains the query, ranked by how many trigrams matched." width="800" height="870"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The index only narrows the candidates; the final check on the normalized title decides.&lt;/em&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;normalizeForSearch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normalize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'NFD'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;̀&lt;span class="o"&gt;-&lt;/span&gt;ͯ&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;đ&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'d'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;Đ&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'d'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toLowerCase&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;\&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="o"&gt;+/&lt;/span&gt;&lt;span class="n"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;' '&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because both sides are normalized, typing without diacritics ("hop") finds "họp", which is how many Vietnamese users type on a laptop.&lt;/p&gt;

&lt;p&gt;A few practical details:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The index is updated explicitly by the commands&lt;/strong&gt; that create, rename, duplicate, delete or undo blocks, not by a generic observer. A rename deletes the block's old trigrams before inserting new ones, since a shorter title loses some.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Primary keys can't contain leading or trailing spaces in RxDB&lt;/strong&gt;, and trigrams often do (&lt;code&gt;"oc "&lt;/code&gt;), so the row ID encodes the trigram while the indexed field keeps its real value.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Existing libraries are backfilled in the background&lt;/strong&gt; on startup, page by page with an ID cursor, so the whole table is never loaded into memory at once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Trigrams need at least three characters.&lt;/strong&gt; Queries shorter than that can't use the index, which is a general limit of the technique worth planning for.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Draw only the rows on screen
&lt;/h2&gt;

&lt;p&gt;The tree used to render recursively: every expanded node was in the DOM. A library with thousands of open blocks meant thousands of rows.&lt;/p&gt;

&lt;p&gt;The tree is now flattened into a list in pre-order, descending only into expanded nodes, so every visible row is one array element with its depth. That list goes to &lt;a href="https://tanstack.com/virtual/latest" rel="noopener noreferrer"&gt;TanStack Virtual&lt;/a&gt;, which mounts only the rows in the viewport plus a small overscan.&lt;/p&gt;

&lt;p&gt;One view deliberately isn't virtualized: the day editor. Its drag and drop between nesting levels needs every sortable item in the DOM, and a day has a few dozen nodes, so the benefit is small and the risk real.&lt;/p&gt;

&lt;h2&gt;
  
  
  A depth limit, for usability rather than speed
&lt;/h2&gt;

&lt;p&gt;Blocks can nest at most &lt;strong&gt;8 levels deep&lt;/strong&gt;. The real costs come from fan-out and total size, which the changes above handle; depth is limited because a pathological tree is unusable and recursive code paths deserve a bound. Moving a whole subtree checks the parent's depth plus one plus the height of the moved subtree, since a node being dragged can be fine while its grandchildren would cross the limit.&lt;/p&gt;

&lt;p&gt;There is deliberately no limit on how many children a block can have: a day with many blocks is a valid use, and the expensive paths for wide trees (rendering and deletion) are handled directly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Count queries in tests, not milliseconds
&lt;/h2&gt;

&lt;p&gt;Timing tests on CI machines are noisy. Most of the performance tests in this codebase count instead:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;How many queries an operation makes.&lt;/strong&gt; Copying a week forward used to call &lt;code&gt;count()&lt;/code&gt; once per day for every week ahead, about 364 queries for a year; it now makes one range query. Importing a file did one &lt;code&gt;findOne&lt;/code&gt; per row and now does one &lt;code&gt;find&lt;/code&gt; plus one &lt;code&gt;bulkUpsert&lt;/code&gt;. The tests assert the query counts, so an N+1 pattern coming back fails the build.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Which plan the database uses.&lt;/strong&gt; On the server, tests run &lt;code&gt;EXPLAIN&lt;/code&gt; on a real PostgreSQL with 50,000 rows and assert that the sync pull uses an index scan, not a sequential scan. If someone drops or changes the index, the test turns red, independently of how fast the CI machine is.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;Change&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;$in&lt;/code&gt; on a nullable &lt;code&gt;parent_id&lt;/code&gt; scanned the table&lt;/td&gt;
&lt;td&gt;Non-null &lt;code&gt;parent_key&lt;/code&gt; shadow column, filled by write hooks, indexed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Page derived the tree from the whole table&lt;/td&gt;
&lt;td&gt;Lazy queries per expanded branch; constant cost per branch&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Substring search loaded every block&lt;/td&gt;
&lt;td&gt;Local-only trigram index, diacritics stripped, candidates verified&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Thousands of rows in the DOM&lt;/td&gt;
&lt;td&gt;Flattened tree plus virtualization; drag-and-drop view left alone&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pathological nesting&lt;/td&gt;
&lt;td&gt;Depth limit of 8, checked against subtree height on moves&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Performance regressions&lt;/td&gt;
&lt;td&gt;Tests that count queries and check query plans&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These changes were made and tested at the scale of thousands of blocks, with the goal that nothing on screen depends on the total size. The common thread is simple: in IndexedDB as in a server database, check that the query you think is indexed actually is.&lt;/p&gt;

&lt;p&gt;If you've kept a large tree fast in IndexedDB or RxDB, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>rxdb</category>
      <category>indexeddb</category>
      <category>performance</category>
      <category>search</category>
    </item>
    <item>
      <title>Four bugs that silently broke RxDB sync, and how they were fixed</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Thu, 12 Nov 2026 21:02:17 +0000</pubDate>
      <link>https://dev.to/mikevu/four-bugs-that-silently-broke-rxdb-sync-and-how-they-were-fixed-29pc</link>
      <guid>https://dev.to/mikevu/four-bugs-that-silently-broke-rxdb-sync-and-how-they-were-fixed-29pc</guid>
      <description>&lt;p&gt;&lt;a href="https://note.vuminhhoa.com/" rel="noopener noreferrer"&gt;Personal OS&lt;/a&gt; keeps its data in the browser with &lt;a href="https://rxdb.info/" rel="noopener noreferrer"&gt;RxDB&lt;/a&gt; and syncs it to a Spring Boot and PostgreSQL backend through RxDB's &lt;a href="https://rxdb.info/replication.html" rel="noopener noreferrer"&gt;custom replication&lt;/a&gt;. The bugs that cost the most time there had one thing in common: nothing reported an error. Edits disappeared, a collection stopped syncing, or every write failed after a reload, and the logs looked normal. This post walks through four of them, what caused each, and the fix.&lt;/p&gt;

&lt;p&gt;How changes reach other devices in real time, with Server-Sent Events, is covered in &lt;a href="https://vuminhhoa.com/en/blog/spring-boot-sse-redis-pubsub" rel="noopener noreferrer"&gt;SSE with Spring Boot and Redis Pub/Sub&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The protocol in one paragraph
&lt;/h2&gt;

&lt;p&gt;Every successful write on the server gets a &lt;code&gt;version&lt;/code&gt; from a per-user sequence that only goes up. That number is both the replication checkpoint (a device pulls "everything after version N") and the way to detect conflicts: a push says which version it assumed, and the server rejects it if the row has moved on. When two devices really did edit the same row, the conflict handler keeps the copy with the higher server version, which in practice is the one already on the server; the client's &lt;code&gt;updated_at&lt;/code&gt; only breaks ties between copies at the same version. An edit made offline still goes through on reconnect, as long as no other device changed that row in the meantime. The cost is that when one did, the offline edit is dropped whole, not merged. For one person's data on a few devices that's rare, and it's the trade-off this design accepts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug 1: rows created on this device lost every later edit
&lt;/h2&gt;

&lt;p&gt;The symptom: create a block, then rename it. The rename looked saved, then reverted after the next sync. Deleting it did nothing either.&lt;/p&gt;

&lt;p&gt;What happened, step by step:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A new row is created locally with &lt;code&gt;version = 0&lt;/code&gt; and pushed. The push succeeds, but the server's response lists only conflicts; it doesn't send back the version it assigned.&lt;/li&gt;
&lt;li&gt;The next pull brings the row with its real version, say 1. RxDB first asks the conflict handler's &lt;code&gt;isEqual(pulled, local)&lt;/code&gt; whether anything changed.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;isEqual&lt;/code&gt; had a shortcut: if either side had &lt;code&gt;version === 0&lt;/code&gt;, return &lt;code&gt;false&lt;/code&gt;. RxDB treated the documents as different, but in that step it didn't write the pulled one locally either.&lt;/li&gt;
&lt;li&gt;The local row stayed at &lt;code&gt;version = 0&lt;/code&gt; forever. Every later edit was pushed with "assumed version 0", the server was at 1, so every edit was a conflict, and the conflict handler discarded the user's change.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp920404i1ujw74qw1j8m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fp920404i1ujw74qw1j8m.png" alt="Sequence diagram between this device and the server: the device pushes a new row at version 0 and gets OK without the assigned version; the next pull brings the row at version 1, isEqual returns different because one side has version 0, and the pulled row is not written. A rename is then pushed assuming version 0, the server answers conflict because the row is at version 1, and the conflict handler drops the rename." width="800" height="727"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The row never learns its real version, so every later edit loses a conflict it should not have.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The fix was to delete the shortcut and compare the three fields that matter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;isEqual&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;version&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_deleted&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_deleted&lt;/span&gt;
    &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;updated_at&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;updated_at&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A pulled row at version 1 now differs from the local row at version 0, so RxDB writes the real version down. A local edit not yet pushed has the same version as the server but a newer &lt;code&gt;updated_at&lt;/code&gt;, so it still isn't overwritten by a pull.&lt;/p&gt;

&lt;p&gt;RxDB calls &lt;code&gt;isEqual&lt;/code&gt; very often and requires it to be synchronous, so comparing three scalar fields is also the cheap option; a deep comparison of whole documents isn't.&lt;/p&gt;

&lt;p&gt;The bug was found and pinned with an &lt;strong&gt;integration test that runs a full sync loop with real RxDB against a mock server that follows the backend's contract&lt;/strong&gt;, plus backend tests for the server side of the same scenario. Unit tests of the conflict handler alone had passed throughout.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug 2: a column that only exists on the client
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://vuminhhoa.com/en/blog/large-tree-indexeddb-rxdb" rel="noopener noreferrer"&gt;large tree post&lt;/a&gt; describes &lt;code&gt;parent_key&lt;/code&gt;: a non-null copy of &lt;code&gt;parent_id&lt;/code&gt; that exists only so the browser can index it. The server has no such column. That broke sync in two directions.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Push.&lt;/strong&gt; The client sent &lt;code&gt;parent_key&lt;/code&gt; along with every block. The server validates column names and rejects unknown ones with HTTP 400 rather than ignoring them. Every push of that collection failed, the change stayed in RxDB's queue, and was retried forever. The fix strips local-only fields at the last moment before a document leaves the device:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;stripLocalOnly&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="cm"&gt;/* removes parent_key, category_key */&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Pull.&lt;/strong&gt; Locally, &lt;code&gt;parent_key&lt;/code&gt; is filled by &lt;code&gt;preInsert&lt;/code&gt; and &lt;code&gt;preSave&lt;/code&gt; hooks. But replication writes pulled documents through RxDB's storage layer directly, &lt;strong&gt;not through the collection&lt;/strong&gt;, so those hooks never run on the pull path. Pulled blocks arrived without &lt;code&gt;parent_key&lt;/code&gt;; in development that failed schema validation, and in production the blocks were saved but invisible to every query on &lt;code&gt;parent_key&lt;/code&gt;, so freshly synced blocks were missing from the library tree. The fix adds the derived fields in the pull handler's modifier.&lt;/p&gt;

&lt;p&gt;That fix had its own trap. RxDB &lt;a href="https://rxdb.info/rx-schema.html" rel="noopener noreferrer"&gt;always sets &lt;code&gt;additionalProperties: false&lt;/code&gt;&lt;/a&gt; at the top level of a schema, so adding a field the collection doesn't declare makes &lt;em&gt;every&lt;/em&gt; pull of that collection fail. &lt;code&gt;snapshot_nodes&lt;/code&gt; has &lt;code&gt;category_id&lt;/code&gt; but no &lt;code&gt;category_key&lt;/code&gt;, so a hard-coded "add both keys" would have broken it. The derivation now reads which derived columns each collection's schema declares, instead of naming collections in code.&lt;/p&gt;

&lt;p&gt;The general rule: &lt;strong&gt;a local-only derived column needs handling at three places&lt;/strong&gt;, the write hooks, the push modifier and the pull modifier, and only the first one is obvious.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0zh82eh5v7z67yqiyiao.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0zh82eh5v7z67yqiyiao.png" alt="Diagram of a local-only derived column: on the device, preInsert and preSave hooks add parent_key when the app writes a block; the push modifier strips local-only fields before data goes to the server, which has no parent_key column; the pull modifier adds the derived fields the schema declares, because replication writes straight to storage and the hooks never run." width="800" height="700"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Hooks only cover writes made through the collection; sync needs its own two places.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug 3: a migration that finished but never said so
&lt;/h2&gt;

&lt;p&gt;After a schema change, a real device ended up with every write to one collection failing with RxDB error &lt;code&gt;COL25&lt;/code&gt;, thrown because the collection was flagged as still migrating. The data showed up fine; reordering, editing or deleting anything in that collection failed. Reloading didn't help.&lt;/p&gt;

&lt;p&gt;The state on disk, captured from a real device, explained it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;rx-migration-status|library_blocks-v-4  → DONE, 1704/1704, 100%
collection|library_blocks-3             → still alive (_deleted = false)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After moving the data, RxDB's migration has two more steps: delete the metadata document of the old collection version, &lt;em&gt;then&lt;/em&gt; clear its "migration in progress" flag. If the process dies between those two steps (a closed tab, a power cut, iOS killing a background tab), the old metadata survives. On the next start, RxDB sees it, decides a migration is needed and sets the flag again, but the migration has nothing left to do, so the flag is never cleared. Awaiting &lt;code&gt;migratePromise()&lt;/code&gt; doesn't help either: it waits for work that is already done and never reports it.&lt;/p&gt;

&lt;p&gt;The fix performs the step RxDB missed, marking the old metadata document as deleted, but only when &lt;strong&gt;both&lt;/strong&gt; conditions hold: a status record says the migration is &lt;code&gt;DONE&lt;/code&gt;, and the old metadata is still alive. A migration that is genuinely &lt;code&gt;RUNNING&lt;/code&gt; is left alone, because deleting its metadata would lose the data not yet moved. The repair touches only RxDB's bookkeeping documents, never user data, and runs before &lt;code&gt;migratePromise()&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug 4: a safety net that tore itself
&lt;/h2&gt;

&lt;p&gt;A device that comes back after a long time can't trust its checkpoint: the server purges deletion records (tombstones) after a while, so an old checkpoint might miss deletes. The safety net: if the last successful sync was more than &lt;strong&gt;60 days&lt;/strong&gt; ago, drop the checkpoint and pull the whole account again.&lt;/p&gt;

&lt;p&gt;Three details made it worse than no safety net:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Never synced" was treated as "too old".&lt;/strong&gt; A full resync clears the last-sync timestamp, so the next start found no timestamp, resynced again, and so on: every app start downloaded the whole account. A missing timestamp now means "not stale"; the checkpoint is 0 anyway, so the first pull runs normally.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The timestamp was only written on some code paths.&lt;/strong&gt; It is now written after every successful pull.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A clock moved backwards&lt;/strong&gt; produces a timestamp in the future; the age can't be known, so it is reset to now instead of forcing a resync.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is also an invariant between client and server: the client's 60 days must stay &lt;strong&gt;shorter than the server's tombstone retention&lt;/strong&gt;, 90 days. If it were longer, a device returning after 75 days would skip the full resync while the server had already purged the deletes it missed, and deleted rows would come back to life on that device. The client reads the server's retention from a config endpoint at startup and logs a loud error if the invariant is broken. It only warns: a deployment mistake shouldn't stop sync entirely.&lt;/p&gt;

&lt;h2&gt;
  
  
  Not a sync bug, but just as quiet: a white screen from chunk splitting
&lt;/h2&gt;

&lt;p&gt;To improve caching, RxDB and RxJS were moved into their own vendor chunk with a manual chunks rule. Moving RxJS pulled in modules that React's chunk also needed, which created a &lt;strong&gt;circular dependency between two chunks&lt;/strong&gt;. Module initialization order broke, the RxDB chunk called an export of the React chunk before it existed (&lt;code&gt;Cm is not a function&lt;/code&gt; in the minified build), and production showed a white screen. The development server, which doesn't bundle, was fine.&lt;/p&gt;

&lt;p&gt;The split was reverted and the production build checked in preview. The caching gain wasn't worth an app that might not open, and the lesson is to test chunking changes on a production build, never only on the dev server.&lt;/p&gt;

&lt;h2&gt;
  
  
  Less load while nobody is looking
&lt;/h2&gt;

&lt;p&gt;One more change that didn't fix a bug but cut server load: only the leader tab opens the Server-Sent Events stream, a tab hidden for more than five minutes closes it and reopens on return, and each reconnection is followed by one check against the server's latest version. While the stream is healthy, the periodic safety pull runs every 30 minutes instead of more often.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Cause&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Edits to new rows reverted&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;isEqual&lt;/code&gt; shortcut kept local &lt;code&gt;version = 0&lt;/code&gt; forever&lt;/td&gt;
&lt;td&gt;Compare version, deleted flag and timestamp directly&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Collection stopped pushing&lt;/td&gt;
&lt;td&gt;Local-only column rejected by the server (400)&lt;/td&gt;
&lt;td&gt;Strip local-only fields before push&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Synced blocks missing from the tree&lt;/td&gt;
&lt;td&gt;Write hooks don't run on the pull path&lt;/td&gt;
&lt;td&gt;Derive the columns in the pull modifier, from the schema&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Every write fails with &lt;code&gt;COL25&lt;/code&gt; after reload&lt;/td&gt;
&lt;td&gt;Migration killed between its last two steps&lt;/td&gt;
&lt;td&gt;Repair only when status is &lt;code&gt;DONE&lt;/code&gt; and old metadata is alive&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Full download on every start&lt;/td&gt;
&lt;td&gt;"Never synced" treated as stale&lt;/td&gt;
&lt;td&gt;Missing timestamp means not stale; invariant with tombstone retention&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;White screen in production&lt;/td&gt;
&lt;td&gt;Circular dependency between manual chunks&lt;/td&gt;
&lt;td&gt;Revert the split; preview production builds&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The common lesson: when sync goes wrong, it rarely throws. Each of these was found by looking at the state on a real device or by a test that ran the whole loop, not by reading error logs.&lt;/p&gt;

&lt;p&gt;If you've tracked down sync bugs in RxDB or another local-first stack that never threw an error, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>rxdb</category>
      <category>offlinesync</category>
      <category>springboot</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Local-first PWA sync: last-writer-wins done carefully</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Thu, 12 Nov 2026 14:00:14 +0000</pubDate>
      <link>https://dev.to/mikevu/local-first-pwa-sync-last-writer-wins-done-carefully-5e5e</link>
      <guid>https://dev.to/mikevu/local-first-pwa-sync-last-writer-wins-done-carefully-5e5e</guid>
      <description>&lt;p&gt;Most web apps treat the server as the source of truth: every action sends a request, waits for the reply, then updates the UI. Lose the network and the app freezes. For &lt;a href="https://finance.vuminhhoa.com/" rel="noopener noreferrer"&gt;Financial Plan&lt;/a&gt; (a monthly budgeting PWA), I went the other way: &lt;strong&gt;local-first&lt;/strong&gt;. Data lives on the device, and the app works offline and even when the server is down. The server is only there to sync between devices and keep a backup.&lt;/p&gt;

&lt;p&gt;Making an app work offline isn't the hard part. The hard part is &lt;strong&gt;syncing&lt;/strong&gt; when the same person edits on a laptop and a phone, sometimes online and sometimes not. This post covers the main decisions and the edge cases I ran into.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdqnzqtoyh2p79saaqzfi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdqnzqtoyh2p79saaqzfi.png" alt="Architecture of the app: in the browser, the React UI reads and writes IndexedDB, which is the source of truth and marks records with updatedAt and dirty, while a service worker caches the whole app. Only when online, IndexedDB syncs with the server through one POST /v1/sync request that pushes dirty records and pulls changes after the cursor; the server gives each change a seq in PostgreSQL." width="800" height="675"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The UI never talks to the network; only the sync step does.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The UI only reads and writes IndexedDB&lt;/strong&gt; (through &lt;a href="https://dexie.org/" rel="noopener noreferrer"&gt;Dexie&lt;/a&gt;) and never waits on the network. Every record has an &lt;code&gt;updatedAt&lt;/code&gt; and a &lt;code&gt;dirty&lt;/code&gt; flag meaning "not on the server yet".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A service worker caches the whole app&lt;/strong&gt; (every route, the fonts), so it opens offline. A new version only takes over when the user taps "Reload", so the UI never changes on its own.&lt;/li&gt;
&lt;li&gt;The app calls &lt;code&gt;navigator.storage.persist()&lt;/code&gt; so the browser doesn't evict its data when the device runs low on space. Without it, IndexedDB is only "best-effort" storage.&lt;/li&gt;
&lt;li&gt;The data schema is written in zod in a shared package. Client and server validate with &lt;strong&gt;the same schema&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The sync protocol: one request, both directions
&lt;/h2&gt;

&lt;p&gt;Each sync is a single request: push the &lt;code&gt;dirty&lt;/code&gt; records and, in the same round trip, pull everything changed since the last &lt;code&gt;cursor&lt;/code&gt;. The server gives every change an increasing &lt;code&gt;seq&lt;/code&gt; number, and the next pull only fetches what has a &lt;code&gt;seq&lt;/code&gt; greater than the &lt;code&gt;cursor&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Sync runs in the background:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;1.5 seconds after each change (so a burst of typing becomes one request),&lt;/li&gt;
&lt;li&gt;when the network comes back, and when the user returns to the tab,&lt;/li&gt;
&lt;li&gt;and once a minute.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On failure (server down, timeout, 5xx, 429), the app retries with &lt;strong&gt;exponential backoff&lt;/strong&gt; from 5 seconds up to 5 minutes. The changes stay safely queued on the device.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Multiple tabs&lt;/strong&gt; are an easy case to forget: two tabs syncing at once push duplicates and fight over the &lt;code&gt;cursor&lt;/code&gt;. The &lt;a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API" rel="noopener noreferrer"&gt;Web Locks API&lt;/a&gt; solves it neatly: whichever tab gets the lock syncs, the others skip. In simplified form:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;navigator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;locks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;request&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"finance-sync"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;ifAvailable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;lock&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="c1"&gt;// another tab is syncing&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;syncOnce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Conflicts: last writer wins, per record
&lt;/h2&gt;

&lt;p&gt;Conflict resolution ranges from simple (last-writer-wins) to complex (CRDTs). For personal finance data, where two devices rarely edit &lt;em&gt;the same field&lt;/em&gt; at the same moment, &lt;strong&gt;last-writer-wins per record&lt;/strong&gt; is enough, as long as a few details are right:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The server only overwrites when &lt;code&gt;updatedAt&lt;/code&gt; is newer.&lt;/strong&gt; That makes re-sending a change harmless (idempotent). It matters because a request can succeed but lose its response, and the client will send it again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pushing a version older than the server's&lt;/strong&gt; gets the current version sent back so the client can catch up.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deletes are tombstones&lt;/strong&gt; (&lt;code&gt;data: null&lt;/code&gt;), not real deletes. With a real delete, other devices have no way to learn the record is gone, and would push it right back.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;seq&lt;/code&gt; order must match commit order
&lt;/h3&gt;

&lt;p&gt;This one is subtle. Say two syncs for the same user run concurrently: request A takes &lt;code&gt;seq = 10&lt;/code&gt; but commits slowly, request B takes &lt;code&gt;seq = 11&lt;/code&gt; and commits first. A pull in between sees &lt;code&gt;seq = 11&lt;/code&gt;, moves its &lt;code&gt;cursor&lt;/code&gt; to 11, and &lt;strong&gt;misses change 10 forever&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftyag0il8fanhadz30muc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftyag0il8fanhadz30muc.png" alt="Sequence diagram: sync A takes seq 10, sync B takes seq 11 and commits first, a pull sees seq 11 and sets its cursor to 11, then A commits seq 10 late. The next pull asks for seq greater than 11, so change 10 never arrives." width="800" height="670"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A seq handed out before commit can become visible after a higher one.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The fix: only one sync per user at a time, using a PostgreSQL advisory lock inside the transaction:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="nf"&gt;pg_try_advisory_xact_lock&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;hashtextextended&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$userId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the &lt;code&gt;try&lt;/code&gt;. If a sync is already running, the new request gets an immediate &lt;strong&gt;429&lt;/strong&gt; and the client retries, instead of waiting. Waiting requests hold the pool's connections, and every other user queues up behind them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Clocks: don't trust each device's time
&lt;/h2&gt;

&lt;p&gt;Last-writer-wins relies on &lt;code&gt;updatedAt&lt;/code&gt;, and &lt;code&gt;updatedAt&lt;/code&gt; comes from the device's clock. A phone running 10 minutes fast "wins" every edit made elsewhere during those 10 minutes.&lt;/p&gt;

&lt;p&gt;Two layers of protection:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Correct to the server's clock, NTP-style.&lt;/strong&gt; Every sync response includes &lt;code&gt;serverTime&lt;/code&gt;. The client computes its offset against the &lt;em&gt;middle&lt;/em&gt; of the round trip and stamps changes with the corrected time:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;   &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;clockOffset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;serverTime&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sentAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;receivedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kt"&gt;number&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
     &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;serverTime&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sentAt&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;receivedAt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The server never stores a time more than 1 minute ahead of its own.&lt;/strong&gt; Anything stamped further ahead is pulled back to that limit, so a device with a wrong clock can't keep winning.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One more small but important detail: &lt;strong&gt;saving a record unchanged doesn't count as a change.&lt;/strong&gt; Many forms save every field they display. If each "Save" stamped a new time, a form on the laptop would silently overwrite an edit just made on the phone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Design the data to avoid conflicts
&lt;/h2&gt;

&lt;p&gt;The best conflict resolution is not having the conflict. Originally a month was one record, holding both the plan and the actual figures. Editing the plan on a laptop while entering actuals on a phone meant one side lost.&lt;/p&gt;

&lt;p&gt;The fix was to &lt;strong&gt;split a month into three records&lt;/strong&gt;: the plan, the actuals when the month is closed, and the actuals being entered. These are usually edited at different times on different devices, so once split, conflicts all but disappear. Old single-record months are still readable and get split once the device has synced.&lt;/p&gt;

&lt;p&gt;The general rule: &lt;strong&gt;the unit of sync should match the unit a user edits at one time.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Other edge cases
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;An old app meets new data.&lt;/strong&gt; A user may open an old (cached) version of the app while another device has synced a new kind of record. Records of an unknown kind are kept untouched on the device, don't break the sync, and get used once the app updates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Signing in is optional.&lt;/strong&gt; If the auth server is down, the app keeps working. A token that can't be refreshed because the network is gone only pauses syncing; it doesn't sign the user out.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data created before signing in.&lt;/strong&gt; If the account is empty, the data goes up automatically. If the account already has data, the app asks: use the account's data, or merge this device's in. Never decide that for the user.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A newly signed-in device&lt;/strong&gt; waits for its first sync before showing any figures. Otherwise it could create an empty plan and push it over the real data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Signing out clears local data&lt;/strong&gt; (the device may be shared), but only after pushing pending changes. If some can't be sent, the app warns first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Knowing who changed what.&lt;/strong&gt; Every record a device pushes is written to an audit log whose own history can't be quietly rewritten. How that works is in &lt;a href="https://vuminhhoa.com/en/blog/tamper-evident-audit-log-postgresql" rel="noopener noreferrer"&gt;a tamper-evident audit log in PostgreSQL&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  When local-first is worth it
&lt;/h2&gt;

&lt;p&gt;Local-first pays off when users need the app to be fast and work offline, and the data mostly belongs to &lt;em&gt;one person&lt;/em&gt; using a few devices. It fits less well when many people edit the same data in real time (look at CRDTs or a server as arbiter then), or when the data is too large to keep on the device.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The UI only reads and writes IndexedDB and never waits on the network; the server syncs between devices and keeps a backup.&lt;/li&gt;
&lt;li&gt;One request pushes dirty records and pulls everything after the cursor, with backoff on failure and a Web Lock so only one tab syncs.&lt;/li&gt;
&lt;li&gt;Last-writer-wins per record is enough for one person's data, if overwrites are idempotent and deletes are tombstones.&lt;/li&gt;
&lt;li&gt;A &lt;code&gt;seq&lt;/code&gt; handed out before commit can become visible after a higher one, so allow one sync per user at a time with a non-blocking advisory lock.&lt;/li&gt;
&lt;li&gt;Don't trust device clocks: correct them to the server's time, cap timestamps on the server, and don't count an unchanged save as a change.&lt;/li&gt;
&lt;li&gt;Shape the data so the unit of sync matches the unit a user edits at one time; splitting a month into three records removed most conflicts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To go deeper, Ink &amp;amp; Switch's essay &lt;a href="https://www.inkandswitch.com/local-first/" rel="noopener noreferrer"&gt;Local-first software&lt;/a&gt; is a great place to start. And if you're building an offline app too and want to compare notes, my email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>localfirst</category>
      <category>pwa</category>
      <category>indexeddb</category>
      <category>postgres</category>
    </item>
    <item>
      <title>Self-hosting side projects: principles and silent failures</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Tue, 10 Nov 2026 20:34:35 +0000</pubDate>
      <link>https://dev.to/mikevu/self-hosting-side-projects-principles-and-silent-failures-2di9</link>
      <guid>https://dev.to/mikevu/self-hosting-side-projects-principles-and-silent-failures-2di9</guid>
      <description>&lt;p&gt;My side projects used to each be deployed their own way: some were cloned onto the server and built there, some ran from a &lt;code&gt;docker-compose&lt;/code&gt; file that only existed on that machine, and environment variables lived wherever they happened to land. It all ran, but every change meant first remembering how I'd done it last time.&lt;/p&gt;

&lt;p&gt;This post covers how I brought it all under one roof, and the real mistakes I made along the way. If you run a server for your own projects, I hope a few of these save you an evening or two of debugging.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put everything in git
&lt;/h2&gt;

&lt;p&gt;The foundation: &lt;strong&gt;no hand edits on the server&lt;/strong&gt;. Compose files, config, alert rules and dashboards all live in one repo. To change something, change it in the repo, push, and let CI apply it. Images are built in each app's CI, not on the server.&lt;/p&gt;

&lt;p&gt;That gives you three things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;History&lt;/strong&gt;: every change has a commit, a reason, and can be reverted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rebuildability&lt;/strong&gt;: if the server is lost, you rebuild from the repo and a backup, not from memory.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Drift detection&lt;/strong&gt;: the rebuild script, run in &lt;code&gt;--dry-run&lt;/code&gt; mode against the live server, should report "nothing to change". If it doesn't, git and reality have drifted apart.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One small mistake shows why versions must live somewhere readable: for a while, the database versions sat inside an encrypted secrets file. Diffs were unreadable, the dependency update tool couldn't see them, and some containers ended up running tags like &lt;code&gt;latest&lt;/code&gt; that differed from what the file said, without anyone noticing. Now every image is pinned to an exact version right in the compose file.&lt;/p&gt;

&lt;h2&gt;
  
  
  Expose as little as possible to the internet
&lt;/h2&gt;

&lt;p&gt;Little effort, large payoff: every service exposed to the internet is something to patch, monitor and worry about. A few ways to cut that down:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Use a tunnel instead of opening ports.&lt;/strong&gt; Services such as Cloudflare Tunnel let the server connect outward, so you don't need to open web ports on the machine. DDoS filtering and rate limiting happen before traffic reaches you.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Separate the admin path from the user path.&lt;/strong&gt; Admin access should go over a private network (a VPN), not the public internet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Don't route what's only used internally.&lt;/strong&gt; Databases, caches and &lt;code&gt;/metrics&lt;/code&gt; endpoints only need to talk to each other on Docker's internal network.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Give each app its own database user&lt;/strong&gt;, with rights on its own database only, instead of a shared admin account.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this makes a system "secure" in any absolute sense. It just reduces the number of places to worry about, so you can worry about the rest more carefully.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deploys you can back out of
&lt;/h2&gt;

&lt;p&gt;The current flow:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The app's repo runs its tests and builds an image.&lt;/li&gt;
&lt;li&gt;The running version of each app is recorded in a file in the infrastructure repo, so the deploy history is simply that file's &lt;code&gt;git log&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The new version first runs as a &lt;strong&gt;canary&lt;/strong&gt;: a separate container that takes no traffic and has to pass its health check. Only then does it replace the running version; otherwise it rolls back on its own.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The biggest win isn't automation, it's that &lt;strong&gt;rolling back becomes boring&lt;/strong&gt;: revert a commit and you're done.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcakassbnjrdyz4n9pghh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcakassbnjrdyz4n9pghh.png" alt="Flow diagram: the app repository’s CI runs tests and builds an image; the new version is recorded in a file in the infrastructure repository, whose git log is the deploy history; the deploy starts a canary container that takes no traffic and must pass its health check; if healthy it replaces the running version, otherwise it rolls back on its own and the old version keeps running." width="800" height="720"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A failed canary never touches the running version, and undoing a release is reverting one commit.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Two real bugs I hit while building this, neither of which raised an error:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;cmd | grep -q&lt;/code&gt; in a script with &lt;code&gt;set -o pipefail&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;grep -q&lt;/code&gt; exits as soon as it finds a match, the command before it gets SIGPIPE, and the whole pipeline counts as failed. The deploy script therefore reported "already up to date" instead of deploying. The fix: capture the output in a variable first, then &lt;code&gt;grep&lt;/code&gt; the variable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sharing a &lt;code&gt;concurrency&lt;/code&gt; group between GitHub Actions workflows.&lt;/strong&gt; As the &lt;a href="https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/control-the-concurrency-of-workflows-and-jobs" rel="noopener noreferrer"&gt;GitHub Actions concurrency docs&lt;/a&gt; describe, GitHub keeps only &lt;em&gt;one&lt;/em&gt; pending run per group by default, and a newer run cancels the pending one before it. Some deploys simply vanished. Each workflow now has its own group.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Secrets: no plain text in git
&lt;/h2&gt;

&lt;p&gt;There are a few options, depending on scale:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Encrypt secrets before committing them, with a tool such as &lt;a href="https://github.com/getsops/sops" rel="noopener noreferrer"&gt;sops&lt;/a&gt;, and decrypt only where they're used.&lt;/li&gt;
&lt;li&gt;Or use the secret store of the CI or hosting platform you're on.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Whichever you choose, three habits are worth keeping: give each component exactly the access it needs; write down how to rotate each secret &lt;em&gt;before&lt;/em&gt; you need to; and rotate a secret as soon as it shows up somewhere it shouldn't, even if it's "probably fine".&lt;/p&gt;

&lt;h2&gt;
  
  
  Alerts: few, but right
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Alert on symptoms&lt;/strong&gt; users feel (errors, slowness, unreachable), not causes. High CPU that hurts nobody isn't worth bothering anyone about.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check from the user's point of view.&lt;/strong&gt; A probe that calls the public URLs along the same path users take (including TLS certificate expiry) catches failures that metrics inside the server can't see.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat alert rules as code&lt;/strong&gt;, with unit tests (Prometheus has &lt;a href="https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/" rel="noopener noreferrer"&gt;&lt;code&gt;promtool test rules&lt;/code&gt;&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check metric names against the real exporter before writing a rule.&lt;/strong&gt; This is the mistake I find most dangerous: a rule that uses a metric that &lt;em&gt;doesn't exist&lt;/em&gt; never fires and never reports an error. I had an "Elasticsearch is down" alert like that. It sat there looking reassuring, and could never have gone off.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monitor the monitoring.&lt;/strong&gt; If monitoring runs on the server it watches, it goes quiet when the server dies. A regular "I'm alive" signal sent to an outside service solves that.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Backups only count if they restore
&lt;/h2&gt;

&lt;p&gt;The rule I follow: &lt;strong&gt;a backup only counts once it has been restored successfully.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Test-restore automatically and regularly into a temporary environment, using &lt;strong&gt;the same database versions&lt;/strong&gt; as production, and check the data.&lt;/li&gt;
&lt;li&gt;Alert when a backup fails, when the latest one is too old, and when a test restore fails.&lt;/li&gt;
&lt;li&gt;Keep at least one copy off the server, and encrypt it.&lt;/li&gt;
&lt;li&gt;For upgrades that change a schema one way (such as Keycloak upgrades), try them first on a copy loaded from a backup, and take a fresh backup right before upgrading.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Other silent failures worth knowing
&lt;/h2&gt;

&lt;p&gt;The mistakes that cost me the most time all had one thing in common: no error message.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bind-mount a file that doesn't exist yet&lt;/strong&gt; and Docker creates a &lt;em&gt;directory&lt;/em&gt; with that name, and the container runs as if nothing happened (&lt;a href="https://docs.docker.com/engine/storage/bind-mounts/" rel="noopener noreferrer"&gt;Docker's bind mount docs&lt;/a&gt; say that with &lt;code&gt;-v&lt;/code&gt; a missing source is always created as a directory, while &lt;code&gt;--mount&lt;/code&gt; refuses to start). A custom MySQL config of mine was "applied" that way for 8 months, which is to say never.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;rsync --filter=':- .gitignore'&lt;/code&gt;&lt;/strong&gt; only applies the rules on the receiving side if a &lt;code&gt;.gitignore&lt;/code&gt; &lt;em&gt;already exists&lt;/em&gt; there. A sync nearly deleted data and &lt;code&gt;.env&lt;/code&gt; files because of it. Every sync with deletes now goes through a script that does a dry run first and refuses when it's about to delete anything that looks like data or a secret.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Before moving an app to a new setup, compare the old and new configuration.&lt;/strong&gt; One missing environment variable can switch off a feature without any error showing up.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Keep compose files, config, alert rules and dashboards in one git repo and never edit the server by hand; a &lt;code&gt;--dry-run&lt;/code&gt; rebuild that reports "nothing to change" shows git and the server still agree.&lt;/li&gt;
&lt;li&gt;Pin every image to an exact version in a file people and tools can read, not inside an encrypted secrets file.&lt;/li&gt;
&lt;li&gt;Expose as little as possible: a tunnel instead of open ports, admin access over a VPN, internal services only on Docker's internal network, and one database user per app.&lt;/li&gt;
&lt;li&gt;Deploy through a canary that must pass its health check, and record versions in git so rolling back is reverting one commit.&lt;/li&gt;
&lt;li&gt;Alert on symptoms users feel, test alert rules, check metric names against the real exporter, and monitor the monitoring from outside the server.&lt;/li&gt;
&lt;li&gt;A backup only counts once a test restore has succeeded.&lt;/li&gt;
&lt;li&gt;None of this takes much money: Cloudflare Tunnel and the Prometheus and Grafana stack have free tiers that are plenty for side projects. The hard part is discipline: no hand edits "just this once", and always asking "if this broke, would I know?".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you've run into silent failures like these on your own server, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>cloudflaretunnel</category>
      <category>gitops</category>
      <category>prometheus</category>
    </item>
    <item>
      <title>Modeling hospital procurement bidding: lessons from my graduation project</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Tue, 10 Nov 2026 14:00:14 +0000</pubDate>
      <link>https://dev.to/mikevu/modeling-hospital-procurement-bidding-lessons-from-my-graduation-project-4i9l</link>
      <guid>https://dev.to/mikevu/modeling-hospital-procurement-bidding-lessons-from-my-graduation-project-4i9l</guid>
      <description>&lt;p&gt;My graduation project at Hanoi University of Science and Technology (2024) was a system for a hospital's &lt;strong&gt;medical supplies department&lt;/strong&gt;: clinical departments propose purchases, the supplies department takes each proposal through public procurement bidding, and the equipment that wins the tender goes into the managed inventory. The running version is at &lt;a href="https://bid.vuminhhoa.com/" rel="noopener noreferrer"&gt;bid.vuminhhoa.com&lt;/a&gt; (the interface is in Vietnamese).&lt;/p&gt;

&lt;p&gt;The hardest part wasn't the code but &lt;strong&gt;understanding and modeling the process&lt;/strong&gt;. Public procurement has many steps, each step produces paperwork, and the people doing the work think in steps, not in tables. This post covers the main modeling decisions, which ones held up, and what I would do differently.&lt;/p&gt;

&lt;h2&gt;
  
  
  The process to digitize
&lt;/h2&gt;

&lt;p&gt;Simplified, a purchase goes through three stages:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fldmn4c35jtusyj0tq4z1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fldmn4c35jtusyj0tq4z1.png" alt="Flow of a purchase: a clinical department proposes, the supplies department approves, procurement runs through six phases from price quotation to contract with about 30 steps that each have a date and a document, then acceptance imports the equipment into the managed inventory." width="799" height="712"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Three stages, and the middle one holds about 30 steps of the same shape: a date and a document.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The six procurement phases follow the Vietnamese contractor selection sequence, where each "publish" step means posting to the &lt;a href="https://muasamcong.mof.gov.vn/" rel="noopener noreferrer"&gt;national e-procurement system&lt;/a&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Phase&lt;/th&gt;
&lt;th&gt;Example steps&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Price quotation&lt;/td&gt;
&lt;td&gt;Post the request for quotation, quotation deadline&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Expert team&lt;/td&gt;
&lt;td&gt;Procurement council meeting, budget approval, forming the expert and appraisal teams&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contractor selection plan&lt;/td&gt;
&lt;td&gt;Draft, appraise, approve and publish the plan&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bidding documents&lt;/td&gt;
&lt;td&gt;Draft, preparation report, approval rounds, publish the invitation to bid&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Selection result&lt;/td&gt;
&lt;td&gt;Evaluation report, negotiation, appraisal, approval of the result&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Contract&lt;/td&gt;
&lt;td&gt;Notify bidders, performance guarantee, sign, hand over, warranty guarantee&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;About 30 steps in total, and nearly every one has the same shape: &lt;strong&gt;a date and a document&lt;/strong&gt;. Spotting that early makes everything downstream simpler (the last section covers what happened when I spotted it late).&lt;/p&gt;

&lt;h2&gt;
  
  
  Proposal and procurement are one record
&lt;/h2&gt;

&lt;p&gt;A department's proposal and the procurement activity share one table, told apart by an &lt;code&gt;isProposal&lt;/code&gt; flag:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A department creates a proposal: a new row, &lt;code&gt;isProposal = true&lt;/code&gt;, proposal status "Pending".&lt;/li&gt;
&lt;li&gt;The supplies department approves it: &lt;strong&gt;the same row&lt;/strong&gt; flips to &lt;code&gt;isProposal = false&lt;/code&gt; and starts the activity status at "Waiting to start".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The main benefit is that a procurement activity keeps its whole history, from who proposed and who approved on which date down to every bidding step, without joining two tables or copying data between stages. The cost is one table with columns that only mean something in one stage (proposal approver, activity approver...). For a process where each stage always inherits from the previous one, I think that trade-off is worth it.&lt;/p&gt;

&lt;p&gt;Status is split into two fields, because the two stages have separate life cycles:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F416zaamxb5ijv2fmxatk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F416zaamxb5ijv2fmxatk.png" alt="State diagram of one database row: as a proposal (isProposal = true) it goes from Pending to Approved or Rejected; on approval the same row sets isProposal to false and its activity status goes from Waiting to start, to In progress, to Awaiting approval, then to Completed or Cancelled." width="800" height="891"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One row, two status fields: approving a proposal flips the same row into a procurement activity.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;One lesson: &lt;strong&gt;write the allowed transitions down as a table and enforce it on the server&lt;/strong&gt;, with each transition as its own action ("submit", "approve", "cancel") rather than a field anyone can write. The business rules then live in one place, and the UI only asks "what can be done from this state?".&lt;/p&gt;

&lt;h2&gt;
  
  
  Each phase is a JSON column
&lt;/h2&gt;

&lt;p&gt;Thirty steps, each with a date and a document, would be some sixty columns. Instead each phase is &lt;strong&gt;one JSON column&lt;/strong&gt;, and its default shape lives in code:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;defaultContractorPlan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;     &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;review&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;   &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;approval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;decision&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;publish&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On read, the server merges the stored value over the default:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="n"&gt;contractorPlan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;defaultContractorPlan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bidding&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;contractorPlan&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Adding a step then means editing a constant and a form, with no migration. Older activities get the new step, empty.&lt;/p&gt;

&lt;p&gt;The trade-off to know up front: data in JSON is &lt;strong&gt;harder to query and to constrain&lt;/strong&gt;. The database can't check types or index it as easily as a regular column. This works when the steps are mostly &lt;em&gt;recorded and displayed&lt;/em&gt;; if you need reports like "how many tenders are stuck at bidding document appraisal", pull at least the progress state out into its own column.&lt;/p&gt;

&lt;h2&gt;
  
  
  Documents must follow the record's life cycle
&lt;/h2&gt;

&lt;p&gt;Each step has an attached file in a file storage service. Orphaned files come easily: a user replaces a file and the old one stays; an activity is deleted and its files remain.&lt;/p&gt;

&lt;p&gt;The fix is to have the form track changes by &lt;strong&gt;field path&lt;/strong&gt; and send three lists on save:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;createdFields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'technical_bid.draft.document'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;updatedFields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'contract_signing.contract.document'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;deletedFields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'expert_team.purchase_meeting.document'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The server walks each list: &lt;code&gt;created&lt;/code&gt; uploads; &lt;code&gt;updated&lt;/code&gt; deletes the old file and uploads the new one; &lt;code&gt;deleted&lt;/code&gt; removes the file and sets the field to &lt;code&gt;null&lt;/code&gt;. Deleting a whole activity walks all six JSON columns to clean up its documents.&lt;/p&gt;

&lt;p&gt;The general lesson: &lt;strong&gt;don't make the server guess what changed by diffing the whole object&lt;/strong&gt;; with nested data that's both costly and error-prone. The client knows exactly what the user just did, and sending "what changed" lets the server do the right thing. Also, calls to file storage aren't part of the database transaction, so order the steps so that a failure halfway leaves a stray file (which can be cleaned up), never a record pointing to a file that's gone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Acceptance is one transaction
&lt;/h2&gt;

&lt;p&gt;The last step, approving the activity, is also when the winning equipment enters the inventory. The equipment list is prepared beforehand as an Excel file, the format the supplies department already works in, and kept on the activity as a "preview list". The equipment only really exists once the activity is approved.&lt;/p&gt;

&lt;p&gt;Duplicate &lt;strong&gt;model numbers&lt;/strong&gt; are checked at two levels:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Within the file&lt;/strong&gt;, in the browser as soon as the Excel is read: duplicate rows are highlighted to fix before sending.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Against existing data&lt;/strong&gt;, on the server, inside the approval transaction.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Shortened from &lt;code&gt;biddingService.ts&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;sequelize&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;Bidding&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;approval&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;duplicates&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;Equipment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;where&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;modelNumber&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;Op&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;modelNumbers&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;duplicates&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ConflictError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'duplicateEquipment'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;duplicates&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;Equipment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;bulkCreate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;equipmentList&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nc"&gt;Activity&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;action&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sb"&gt;`imported ${equipmentList.length} items...`&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;transaction&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;// After commit: update the search index.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three things matter here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The status change, the new equipment and the log entry succeed or fail together.&lt;/strong&gt; An activity can't be "Completed" while the inventory is missing its equipment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A duplicate error returns the duplicate rows themselves&lt;/strong&gt;, so the UI can show them for the user to fix, instead of "Something went wrong".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The search index is updated after commit.&lt;/strong&gt; The database is the source of truth; the index is a copy that a reindex script can rebuild at any time. Updating the index inside the transaction would leave it holding equipment that doesn't exist if the transaction rolls back. (The Vietnamese search setup for this inventory has &lt;a href="https://vuminhhoa.com/en/blog/vietnamese-search-elasticsearch" rel="noopener noreferrer"&gt;its own post&lt;/a&gt;.)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It's also worth knowing the general limit of "check, then write": two concurrent transactions can both see "no duplicate". The reliable guard for this is a &lt;strong&gt;unique constraint in the database&lt;/strong&gt;; the check in code is still worth keeping to return a readable message.&lt;/p&gt;

&lt;h2&gt;
  
  
  Permissions in the department's own words
&lt;/h2&gt;

&lt;p&gt;Permissions are named the way the supplies department talks: "Create purchase proposal", "Approve procurement activity", "View all equipment"... An administrator creates roles and ticks permissions, so a change in who does what needs no code change.&lt;/p&gt;

&lt;p&gt;Two axes are worth keeping apart:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Permission&lt;/strong&gt; answers "which &lt;em&gt;actions&lt;/em&gt; can you take": create, edit, approve.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Scope&lt;/strong&gt; answers "whose &lt;em&gt;data&lt;/em&gt; can you see": staff of one department only see their department's proposals and equipment, unless they hold "view all".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Merging the two (say, a role called "Internal Medicine staff") multiplies roles by the number of departments. Kept apart, one role serves every department and the scope comes from the user's department.&lt;/p&gt;

&lt;p&gt;To test permissions, the project has a script that simulates the whole supplies department: five roles, five staff members tied to departments, all six bidding phases through to acceptance. The part I find most worth copying: &lt;strong&gt;every user tries both what they're allowed to do and what they aren't&lt;/strong&gt;. A permission test that only exercises the allowed path proves half of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  An activity log for people to read
&lt;/h2&gt;

&lt;p&gt;Every significant action writes one line with &lt;em&gt;who&lt;/em&gt;, &lt;em&gt;did what&lt;/em&gt;, &lt;em&gt;to which object&lt;/em&gt;, like "… imported 12 items from procurement activity …". The dashboard shows this timeline so the whole department can see where things stand.&lt;/p&gt;

&lt;p&gt;This log is for following the work, not for evidence that can't be denied. If you need history that even a database administrator can't quietly rewrite, see &lt;a href="https://vuminhhoa.com/en/blog/tamper-evident-audit-log-postgresql" rel="noopener noreferrer"&gt;a tamper-evident audit log in PostgreSQL&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two years later
&lt;/h2&gt;

&lt;p&gt;The project ran from March to July 2024. I've come back to it a few times since: moving to TypeScript, adding equipment search with Elasticsearch, caching, Google sign-in and single sign-on.&lt;/p&gt;

&lt;p&gt;The data model's skeleton is the same as in 2024: one table with an &lt;code&gt;isProposal&lt;/code&gt; flag, six JSON columns for six phases, a preview equipment list. What had to be redone was the &lt;strong&gt;shape inside&lt;/strong&gt; the JSON columns. The 2024 version used flat field names in unaccented Vietnamese, two separate fields per step:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// 2024&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;ngayHopHoiDongMuaSam&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;taiLieuHopHoiDongMuaSam&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;ngayPheDuyetDuToan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;   &lt;span class="n"&gt;taiLieuPheDuyetDuToan&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// later&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;purchaseMeeting&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;budgetApproval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With each step as a &lt;code&gt;{ date, document }&lt;/code&gt; object, the form needs one component for every step, and the file cleanup only looks for &lt;code&gt;document&lt;/code&gt; keys instead of remembering sixty field names. That's the price of noticing "every step is a date plus a document" late.&lt;/p&gt;

&lt;p&gt;If you're building software for a multi-step administrative process, my advice is to sit with the people who do the work and &lt;strong&gt;draw the process before drawing the tables&lt;/strong&gt;: list the steps, the paperwork each one produces, who approves it, then look for the shape the steps share. The database usually falls out of that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Most of the roughly 30 procurement steps share one shape, a date and a document. Spotting that early simplifies the forms, the storage and the file cleanup; I spotted it late and had to reshape the JSON.&lt;/li&gt;
&lt;li&gt;A proposal and its procurement activity are one row with an &lt;code&gt;isProposal&lt;/code&gt; flag, so the whole history stays in one record, at the cost of columns that only mean something in one stage.&lt;/li&gt;
&lt;li&gt;Write the allowed status transitions down as a table, enforce them on the server, and expose each transition as its own action.&lt;/li&gt;
&lt;li&gt;One JSON column per phase, merged over a default shape in code, lets you add steps without a migration, but it is harder to query and constrain than regular columns.&lt;/li&gt;
&lt;li&gt;Have the client send which fields were created, updated and deleted, so the server can manage attached files without diffing nested objects.&lt;/li&gt;
&lt;li&gt;Approval, new equipment and the log entry go in one transaction; the search index is updated after commit, and a unique constraint is the reliable guard against duplicates.&lt;/li&gt;
&lt;li&gt;Keep permissions (which actions) separate from scope (whose data), and test both the allowed and the forbidden paths for every user.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you've modeled a similar process differently, I'd love to hear about it. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>domainmodeling</category>
      <category>node</category>
      <category>sequelize</category>
      <category>mysql</category>
    </item>
    <item>
      <title>Tracking Vietnamese news trends with embeddings, HDBSCAN and an LLM</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Thu, 05 Nov 2026 20:47:28 +0000</pubDate>
      <link>https://dev.to/mikevu/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm-h</link>
      <guid>https://dev.to/mikevu/tracking-vietnamese-news-trends-with-embeddings-hdbscan-and-an-llm-h</guid>
      <description>&lt;p&gt;A big event is usually covered by dozens of outlets at once, under different headlines. Reading article by article, it's hard to see the overall picture. The more useful question isn't "what's in the news today", but: &lt;strong&gt;which stories are heating up, which are cooling off, and why?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;To answer that, I built &lt;a href="https://news.vuminhhoa.com/" rel="noopener noreferrer"&gt;Tin tức hôm nay có gì?&lt;/a&gt; ("What's in the news today?"), a system that reads about 95 Vietnamese RSS feeds, groups articles about the same thing into &lt;em&gt;topics&lt;/em&gt;, tracks each topic's lifecycle, and has an LLM summarize what is driving it. This post covers how it works and the numbers I picked along the way.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architecture at a glance
&lt;/h2&gt;

&lt;p&gt;It's a single TypeScript service running on &lt;a href="https://bun.sh" rel="noopener noreferrer"&gt;Bun&lt;/a&gt;, with its data in one SQLite file (&lt;code&gt;bun:sqlite&lt;/code&gt;). The pipeline runs in passes, and each pass goes through these stages:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Ingest&lt;/strong&gt;: read the RSS feeds and drop duplicates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Embed&lt;/strong&gt;: turn each article's title and description into a vector.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cluster&lt;/strong&gt;: group nearby vectors with HDBSCAN.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lifecycle&lt;/strong&gt;: update each topic's velocity, acceleration and stage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Label and insight&lt;/strong&gt;: an LLM names the topic and explains it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Digest&lt;/strong&gt;: put together an email summary.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The same image runs as &lt;code&gt;web&lt;/code&gt; (dashboard and API), &lt;code&gt;pipeline&lt;/code&gt;, or both. In production the two run separately, so the dashboard isn't affected while the pipeline is busy.&lt;/p&gt;

&lt;h2&gt;
  
  
  A schedule that follows the news cycle
&lt;/h2&gt;

&lt;p&gt;News doesn't arrive evenly through the day, so the pipeline doesn't run evenly either. The schedule is computed in Vietnam time:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time of day&lt;/th&gt;
&lt;th&gt;Every&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;6–9 am and 5–10 pm&lt;/td&gt;
&lt;td&gt;15 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;9 am–5 pm&lt;/td&gt;
&lt;td&gt;30 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Overnight&lt;/td&gt;
&lt;td&gt;60 minutes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Passes are aligned to round times (7:00, 7:15, 7:30…), so the dashboard can always say when the next one will run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Dropping duplicates
&lt;/h2&gt;

&lt;p&gt;The same article often shows up in several feeds of one outlet (front page, section, latest news). I hash the URL into a &lt;code&gt;content_hash&lt;/code&gt; and skip anything already in the database. It's simple and good enough, since one article nearly always keeps one URL. Two &lt;em&gt;different&lt;/em&gt; outlets covering the same event are not duplicates, though: that's exactly the signal clustering needs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Embeddings for Vietnamese
&lt;/h2&gt;

&lt;p&gt;By default I use &lt;code&gt;jina-embeddings-v3&lt;/code&gt; through Jina AI's API; it's multilingual and handles Vietnamese well. Without an API key the service falls back to running &lt;strong&gt;LaBSE&lt;/strong&gt; locally (via &lt;code&gt;@xenova/transformers&lt;/code&gt;). It's slower, but the pipeline doesn't depend on an outside service to keep working.&lt;/p&gt;

&lt;p&gt;Vectors are stored with the article, and each pass only embeds new articles, up to 1,000 at a time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why HDBSCAN
&lt;/h2&gt;

&lt;p&gt;With news, I don't know in advance how many topics there are, and most articles are &lt;strong&gt;noise&lt;/strong&gt;: one-off stories that don't belong to anything bigger. K-means makes you pick the number of clusters and forces every point into one. HDBSCAN finds the number of clusters on its own and labels points that don't fit anywhere as &lt;code&gt;-1&lt;/code&gt;. For this problem, that's exactly the behavior I want.&lt;/p&gt;

&lt;p&gt;It runs in a small Python process (the &lt;code&gt;hdbscan&lt;/code&gt; library); Bun pipes the vectors in on stdin and reads the labels from stdout. The current settings:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nc"&gt;HDBSCAN&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;min_cluster_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;min_samples&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;euclidean&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;cluster_selection_method&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;leaf&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few of these choices are worth explaining:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;L2-normalize first.&lt;/strong&gt; For unit vectors, Euclidean distance and cosine similarity are directly related (d² = 2 − 2·cos), so &lt;code&gt;euclidean&lt;/code&gt; ranks neighbors exactly as cosine would, and HDBSCAN is faster with it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;leaf&lt;/code&gt; rather than &lt;code&gt;eom&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;eom&lt;/code&gt; prefers large, stable clusters, so it tends to merge related but separate stories (say a storm, the floods after it and a landslide) into one. &lt;code&gt;leaf&lt;/code&gt; gives smaller, more specific clusters, which suits tracking individual events.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;min_cluster_size=8&lt;/code&gt;.&lt;/strong&gt; A topic needs at least 8 articles in a pass. Lower, and the dashboard fills up with trivia; higher, and new stories get missed.&lt;/li&gt;
&lt;li&gt;Each pass clusters at most 3,000 articles. Articles marked as noise get retried for the next 2 hours, in case their story is only just starting.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  From clusters to topics that live across passes
&lt;/h2&gt;

&lt;p&gt;HDBSCAN only knows about the articles in the current pass. For a topic to &lt;em&gt;live&lt;/em&gt; for hours, each topic stores a &lt;strong&gt;centroid&lt;/strong&gt; (the mean of its vectors), and every new cluster is compared against the centroids of active topics:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rule&lt;/th&gt;
&lt;th&gt;Cosine threshold&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;New cluster joins an existing topic&lt;/td&gt;
&lt;td&gt;≥ 0.75&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Merge two clusters in the same pass&lt;/td&gt;
&lt;td&gt;≥ 0.82&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Merge two topics across passes&lt;/td&gt;
&lt;td&gt;≥ 0.78&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Split a topic that got too big&lt;/td&gt;
&lt;td&gt;from 30 articles&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The merge threshold is higher than the join threshold on purpose: a wrong merge is hard to undo, while leaving two clusters apart for one more pass costs almost nothing. After each merge the centroid is recomputed right away, which stops cascades (A absorbs B, then AB absorbs C only because it grew).&lt;/p&gt;

&lt;h2&gt;
  
  
  A topic's lifecycle
&lt;/h2&gt;

&lt;p&gt;Each topic moves through a simple state machine driven by &lt;strong&gt;velocity&lt;/strong&gt; (how fast new articles arrive, measured over a 2-hour window) and &lt;strong&gt;acceleration&lt;/strong&gt; (how velocity is changing):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Signal → Emerging&lt;/strong&gt;: at least 5 articles and positive velocity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Emerging → Growing&lt;/strong&gt;: positive acceleration.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Growing → Hot&lt;/strong&gt;: at least 10 articles and velocity above 1.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hot → Peak&lt;/strong&gt;: acceleration close to zero relative to velocity (under 10%).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Declining&lt;/strong&gt;: negative velocity for 2 windows in a row.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dead&lt;/strong&gt;: no new articles for 3 windows in a row. If new ones arrive later, the topic comes back as Emerging.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fem3t5iqx6k3xd005sk5m.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fem3t5iqx6k3xd005sk5m.png" alt="State diagram of a topic: Signal, Emerging, Growing, Hot, Peak, Declining and Dead, with the condition for each step: at least 5 articles and positive velocity, positive acceleration, at least 10 articles and velocity above 1, acceleration under 10% of velocity, negative velocity for 2 windows, no new articles for 3 windows. Signal through Peak are marked active and considered for LLM analysis. New articles bring a dead topic back to Emerging." width="800" height="839"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Only Signal through Peak count as active, which is also what limits the LLM calls.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Every transition is recorded with its reason (for example "Count &amp;gt;= 10 and velocity &amp;gt; 1.0"), so when a topic jumps around I can read its history and see why.&lt;/p&gt;

&lt;h2&gt;
  
  
  Scoring how hot a topic is
&lt;/h2&gt;

&lt;p&gt;For ranking, each topic gets a score from 0 to 100, the sum of six parts normalized to 0–1:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Part&lt;/th&gt;
&lt;th&gt;Weight&lt;/th&gt;
&lt;th&gt;Maxes out at&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Velocity (last 2 hours)&lt;/td&gt;
&lt;td&gt;25&lt;/td&gt;
&lt;td&gt;10 articles&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Volume (24 hours)&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;150 articles&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Acceleration&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Distinct sources&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;8 sources&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Velocity relative to the average&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;3×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Topic freshness&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;decays to 0 over 7 days&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Distinct sources get their own weight because 20 articles from one outlet are very different from 8 outlets covering the same story. The score is then multiplied by a decay factor based on the time since the latest article: no decay in the first hour, then 20% after 1 hour, 40% after 6, 60% after 12, 80% after 24, and zero after 48 hours.&lt;/p&gt;

&lt;h2&gt;
  
  
  The LLM: naming and explaining
&lt;/h2&gt;

&lt;p&gt;The LLM is called through an OpenAI-compatible API (DeepSeek by default), for two kinds of requests:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Naming&lt;/strong&gt;: at most 5 words, in Vietnamese, sent in batches of many topics, each with at most 10 article headlines.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Analysis&lt;/strong&gt;: JSON with &lt;code&gt;driving_force&lt;/code&gt; (the main cause), &lt;code&gt;trend_rationale&lt;/code&gt; (why it's heating up or calming down), &lt;code&gt;opportunity&lt;/code&gt; (a concrete opportunity or risk, and for which industry), &lt;code&gt;watch_next&lt;/code&gt; (what to watch for) and an &lt;code&gt;opportunity_score&lt;/code&gt; from 0 to 10.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;LLM calls cost money, so choosing which topics to analyze matters as much as the prompt:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Only &lt;strong&gt;active&lt;/strong&gt; topics (Signal through Peak) are considered; Declining and Dead ones aren't.&lt;/li&gt;
&lt;li&gt;A topic is re-analyzed when it has never been analyzed, has &lt;strong&gt;5 or more new articles&lt;/strong&gt; since last time, has been flagged for a redo, or its last analysis is over 24 hours old.&lt;/li&gt;
&lt;li&gt;Candidates are ordered by hotness score, and each pass is capped (500 topics for naming, 300 for analysis).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Content-based caching&lt;/strong&gt;: each analysis stores a hash of the articles it was based on. If that set hasn't changed, the topic is skipped and no LLM call is made.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Token usage is recorded per pass to keep an eye on cost. Setting &lt;code&gt;LLM_ENABLED=false&lt;/code&gt; keeps the whole pipeline running, just without names and analysis.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it, and what I'd improve
&lt;/h2&gt;

&lt;p&gt;Day to day, the service needs little attention:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Data&lt;/strong&gt;: the SQLite database is about 3.8 GB. Dead topics, orphan articles and logs older than 30 days are deleted daily.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Measuring&lt;/strong&gt;: each pass records the time of every stage (embed, cluster, label, insight, lifecycle, digest) and the tokens it used. The app's dashboard shows them, and my shared monitoring tracks them too (see &lt;a href="https://vuminhhoa.com/en/blog/self-hosted-vps-side-projects" rel="noopener noreferrer"&gt;self-hosting side projects&lt;/a&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is also room for improvement:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Recluster over a sliding window&lt;/strong&gt; instead of clustering only new articles and comparing centroids. The current approach is cheap, but centroids can drift as long-lived topics grow.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A labeled evaluation set.&lt;/strong&gt; The 0.75 / 0.78 / 0.82 thresholds are currently fixed values in the code. A few hundred article pairs labeled "same story / different story" would make those choices less of a guess.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deduplicate by content&lt;/strong&gt;, not just by URL, to catch republished articles.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A trend tracker answers "which stories are heating up, which are cooling off, and why", not just "what's in the news today".&lt;/li&gt;
&lt;li&gt;HDBSCAN fits news because it finds the number of clusters on its own and labels one-off stories as noise; &lt;code&gt;leaf&lt;/code&gt; keeps related but separate events apart.&lt;/li&gt;
&lt;li&gt;Topics live across passes through stored centroids and cosine thresholds, with merging set stricter than joining because a wrong merge is hard to undo.&lt;/li&gt;
&lt;li&gt;Velocity and acceleration drive a simple lifecycle from Signal to Dead, and every transition records its reason so a topic's history can be read later.&lt;/li&gt;
&lt;li&gt;The hotness score gives distinct sources their own weight, since 8 outlets on one story mean more than 20 articles from one outlet.&lt;/li&gt;
&lt;li&gt;LLM cost is controlled by choosing what to analyze: only active topics, only when their articles change, ordered by score and capped per pass.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The live version is at &lt;a href="https://news.vuminhhoa.com/" rel="noopener noreferrer"&gt;news.vuminhhoa.com&lt;/a&gt;. If you're working on something with Vietnamese news too, I'd like to hear about it: my email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>embeddings</category>
      <category>hdbscan</category>
      <category>llm</category>
      <category>bunjs</category>
    </item>
    <item>
      <title>Vietnamese search in Elasticsearch: diacritics, n-grams and filter order</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Wed, 04 Nov 2026 20:30:54 +0000</pubDate>
      <link>https://dev.to/mikevu/vietnamese-search-in-elasticsearch-diacritics-n-grams-and-filter-order-ccg</link>
      <guid>https://dev.to/mikevu/vietnamese-search-in-elasticsearch-diacritics-n-grams-and-filter-order-ccg</guid>
      <description>&lt;p&gt;Vietnamese users type searches every which way: with diacritics, without them, half a word, part of a device code. A search box has to understand that "may sieu am" means "Máy siêu âm" (ultrasound machine), and that "SAMS" is part of the code "SA-SAMSUNG".&lt;/p&gt;

&lt;p&gt;This post walks through the Elasticsearch setup for the equipment inventory in my &lt;a href="https://vuminhhoa.com/en/blog/hospital-procurement-bidding-workflow" rel="noopener noreferrer"&gt;hospital procurement system&lt;/a&gt;: analyzers, the query, deep pagination, rebuilding the index, and a filter-order mistake I only caught while reviewing the config for this post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Removing diacritics with asciifolding
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://www.elastic.co/docs/reference/text-analysis/analysis-asciifolding-tokenfilter" rel="noopener noreferrer"&gt;&lt;code&gt;asciifolding&lt;/code&gt;&lt;/a&gt; filter turns accented characters into ASCII: "Máy siêu âm" becomes &lt;code&gt;may&lt;/code&gt;, &lt;code&gt;sieu&lt;/code&gt;, &lt;code&gt;am&lt;/code&gt;. It handles &lt;code&gt;đ&lt;/code&gt; too, which becomes &lt;code&gt;d&lt;/code&gt;. Use the same analysis when indexing and searching, and accented and unaccented input both match:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;vietnamese_analyzer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;custom&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;tokenizer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;standard&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;lowercase&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;asciifolding&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;vietnamese_stop&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;},&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cost is losing the difference between words that differ only by their marks: "lá" (leaf), "là" (is) and "la" (to shout) become one. For equipment names that's almost harmless. If you want exact-accent matches ranked higher, keep an extra subfield without folding and boost matches on it.&lt;/p&gt;

&lt;p&gt;Another limit of the &lt;code&gt;standard&lt;/code&gt; tokenizer for Vietnamese: it splits on &lt;strong&gt;syllables&lt;/strong&gt;, not words. "Máy siêu âm" is three tokens, not the two words "máy" and "siêu âm". For an equipment list, adding phrase-style matching (below) is enough. For real word segmentation there are community &lt;a href="https://github.com/duydo/elasticsearch-analysis-vietnamese" rel="noopener noreferrer"&gt;Vietnamese analysis plugins&lt;/a&gt; built on Cốc Cốc's tokenizer, at the cost of installing a plugin into the cluster.&lt;/p&gt;

&lt;h2&gt;
  
  
  Filter order: stopwords go before asciifolding
&lt;/h2&gt;

&lt;p&gt;The config above also has a stopword filter with words like "và" (and), "của" (of), "cho" (for), "có" (have), "là" (is). It looks reasonable, but &lt;strong&gt;it never matches anything&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Filters run &lt;strong&gt;in order&lt;/strong&gt;. By the time a token reaches &lt;code&gt;vietnamese_stop&lt;/code&gt;, &lt;code&gt;asciifolding&lt;/code&gt; has already turned "và" into &lt;code&gt;va&lt;/code&gt; and "của" into &lt;code&gt;cua&lt;/code&gt;. The stopword list is still accented, so every comparison misses. Nothing errors, search keeps working, the filter just does nothing.&lt;/p&gt;

&lt;p&gt;The naive fix is to rewrite the list without accents. But once folded, &lt;code&gt;co&lt;/code&gt; is both "có" (have) and "cơ" (muscle) or "cổ" (neck); &lt;code&gt;la&lt;/code&gt; is both "là" and "lá"; &lt;code&gt;cho&lt;/code&gt; is both "cho" (for) and "chó" (dog). Dropping &lt;code&gt;co&lt;/code&gt; deletes real words. The right order is to &lt;strong&gt;remove stopwords while the accents are still there, then fold&lt;/strong&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;lowercase&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;vietnamese_stop&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;asciifolding&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmkssso69m7zt6e76jule.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmkssso69m7zt6e76jule.png" alt="The token Của through two filter orders. Old order: lowercase gives của, asciifolding gives cua, and vietnamese_stop keeps cua because the list has của, not cua. Fixed order: lowercase gives của, and vietnamese_stop removes it before asciifolding runs." width="800" height="675"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Filters run in order: an accented stopword list has to see the token before asciifolding strips the marks.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Two general lessons:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;For any list-based filter (stopwords, synonyms), ask: &lt;strong&gt;what does the token look like by the time it reaches this filter?&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Check analyzers with the &lt;code&gt;_analyze&lt;/code&gt; API instead of guessing. One call shows which tokens survive:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;POST /equipments/_analyze
{ "analyzer": "vietnamese_analyzer", "text": "Máy thở và monitor của ICU" }
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Partial codes with n-grams
&lt;/h2&gt;

&lt;p&gt;Device codes like "SA-SAMSUNG" or "ECG-FUKUDA" are often typed partially: "sams", "fuku". A regular analyzer only matches whole tokens, so add an &lt;strong&gt;&lt;a href="https://www.elastic.co/docs/reference/text-analysis/analysis-ngram-tokenizer" rel="noopener noreferrer"&gt;n-gram&lt;/a&gt; subfield&lt;/strong&gt;: at index time, each token is cut into every substring of 2 to 10 characters.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;ngram_tokenizer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ngram&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;min_gram&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;max_gram&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;token_chars&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;letter&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;digit&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;

&lt;span class="nx"&gt;modelNumber&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nl"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;text&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;analyzer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;vietnamese_analyzer&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nl"&gt;keyword&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;keyword&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="nx"&gt;ngram&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;text&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;analyzer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ngram_analyzer&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;search_analyzer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;ngram_search&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;},&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key detail is that &lt;code&gt;search_analyzer&lt;/code&gt; differs from &lt;code&gt;analyzer&lt;/code&gt;: indexing cuts n-grams, searching does &lt;strong&gt;not&lt;/strong&gt;, it only lowercases and folds. Cut the query into n-grams too and "sams" becomes &lt;code&gt;sa&lt;/code&gt;, &lt;code&gt;am&lt;/code&gt;, &lt;code&gt;ms&lt;/code&gt;... and matches a lot of unrelated things.&lt;/p&gt;

&lt;p&gt;Three things to know about n-grams:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The index grows.&lt;/strong&gt; A 10-character token produces 45 n-grams. Use them on a few short fields that need partial matching (names, codes), not on long descriptions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A query token longer than &lt;code&gt;max_gram&lt;/code&gt; won't match&lt;/strong&gt; the n-gram field, since no n-gram is that long. That's one reason n-grams shouldn't be the only way to match.&lt;/li&gt;
&lt;li&gt;By default Elasticsearch allows &lt;code&gt;max_gram - min_gram&lt;/code&gt; of at most 1; a wider range needs a higher &lt;code&gt;index.max_ngram_diff&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  One query, three ways to match
&lt;/h2&gt;

&lt;p&gt;The query combines three ways to match in one &lt;code&gt;bool.should&lt;/code&gt;, and any one of them is enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;should&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="c1"&gt;// 1. Every word, spread across fields: "siêu âm samsung"&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;multi_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;cross_fields&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;and&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;equipmentName^3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;modelNumber^2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;manufacturer^2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;unit&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;origin&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="c1"&gt;// 2. Last word half typed: "máy siêu â"&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;multi_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;phrase_prefix&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;equipmentName^3&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;modelNumber^2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;manufacturer^2&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="c1"&gt;// 3. Part of a name or code: "sams"&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;multi_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;operator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;and&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;equipmentName.ngram&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;modelNumber.ngram&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="nx"&gt;minimum_should_match&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;cross_fields&lt;/code&gt; treats the fields as &lt;strong&gt;one big field&lt;/strong&gt;, so "siêu âm samsung" matches even though "siêu âm" is in the name and "samsung" in the manufacturer.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;phrase_prefix&lt;/code&gt; covers input that's still being typed, which suits search-as-you-type boxes.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;^3&lt;/code&gt;, &lt;code&gt;^2&lt;/code&gt; boosts rank matches on the name above matches on origin or unit.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Department and user-scope filtering goes in &lt;code&gt;filter&lt;/code&gt;, not &lt;code&gt;must&lt;/code&gt;: &lt;code&gt;filter&lt;/code&gt; doesn't affect scoring and can be cached by Elasticsearch, which suits yes-or-no conditions.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deep pagination with search_after
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://www.elastic.co/docs/reference/elasticsearch/rest-apis/paginate-search-results" rel="noopener noreferrer"&gt;Paging&lt;/a&gt; with &lt;code&gt;from&lt;/code&gt; + &lt;code&gt;size&lt;/code&gt; is capped by &lt;code&gt;index.max_result_window&lt;/code&gt;, 10,000 results by default, because each shard has to fetch and sort &lt;code&gt;from + size&lt;/code&gt; documents. Past that, Elasticsearch returns an error.&lt;/p&gt;

&lt;p&gt;The alternative is &lt;code&gt;search_after&lt;/code&gt;: each page returns the sort values of its last hit, and the next page starts from there.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;sortKey&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;order&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;order&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;asc&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;   &lt;span class="c1"&gt;// tiebreaker, so the order is always deterministic&lt;/span&gt;
&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="nx"&gt;search_after&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;lastSort&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;       &lt;span class="c1"&gt;// from the previous page&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details that are easy to miss:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Always end &lt;code&gt;sort&lt;/code&gt; with a unique tiebreaker&lt;/strong&gt; such as &lt;code&gt;id&lt;/code&gt;. When several documents share a sort value, without one some can repeat or be skipped between pages.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;search_after&lt;/code&gt; only moves &lt;strong&gt;forward&lt;/strong&gt;: to reach page 3 you need page 2's cursor. In this project the UI remembers the cursor of each page it has visited, uses &lt;code&gt;from&lt;/code&gt; for the early pages and switches to cursors further in. If users need to jump to a far page, the answer is usually a filter, not deeper pagination.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The database is the source of truth, the index is rebuildable
&lt;/h2&gt;

&lt;p&gt;Equipment is written to MySQL first and pushed to Elasticsearch after the transaction commits. Dual writes like this are simple, but if the push fails the index falls behind. So you need a &lt;strong&gt;reindex script&lt;/strong&gt; that rebuilds the whole index from the database. A few techniques from that script worth reusing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Keyset pagination&lt;/strong&gt; (&lt;code&gt;WHERE id &amp;gt; lastId ORDER BY id LIMIT n&lt;/code&gt;) instead of &lt;code&gt;OFFSET&lt;/code&gt;: each batch stays fast however deep into the table you are.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Preload small tables&lt;/strong&gt; (departments, activities) into a &lt;code&gt;Map&lt;/code&gt; to add their names to documents, instead of joining per batch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Turn off refresh during bulk loading&lt;/strong&gt; (&lt;code&gt;refresh_interval: -1&lt;/code&gt;) and turn it back on at the end, so Elasticsearch doesn't refresh after every batch.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The current script deletes the old index and creates a new one, so search is empty while it runs. For a system that has to stay up, the standard approach is to &lt;strong&gt;build the new index alongside and switch an &lt;a href="https://www.elastic.co/docs/manage-data/data-store/aliases" rel="noopener noreferrer"&gt;alias&lt;/a&gt;&lt;/strong&gt; to it in one operation: users always search through the alias and never see a half-built index. And if the index must never drift from the database, the next steps are the outbox pattern or CDC (reading the binlog), at the cost of one more component to run.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Fold diacritics with &lt;code&gt;asciifolding&lt;/code&gt;, but put list-based filters &lt;strong&gt;before&lt;/strong&gt; it.&lt;/li&gt;
&lt;li&gt;Use n-grams for partial matching, with a separate &lt;code&gt;search_analyzer&lt;/code&gt;, on a few short fields only.&lt;/li&gt;
&lt;li&gt;Combine &lt;code&gt;cross_fields&lt;/code&gt;, &lt;code&gt;phrase_prefix&lt;/code&gt; and n-grams to cover different ways of typing.&lt;/li&gt;
&lt;li&gt;Page deeply with &lt;code&gt;search_after&lt;/code&gt; and a tiebreaker.&lt;/li&gt;
&lt;li&gt;Treat the index as a rebuildable copy, and check analyzers with &lt;code&gt;_analyze&lt;/code&gt; instead of guessing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There are many other approaches to Vietnamese search, especially for word segmentation. If you've found one that works well, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>elasticsearch</category>
      <category>search</category>
      <category>tingvit</category>
      <category>node</category>
    </item>
    <item>
      <title>Time in Apache Flink: watermarks, idle partitions, state TTL and replay</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Tue, 03 Nov 2026 20:15:33 +0000</pubDate>
      <link>https://dev.to/mikevu/time-in-apache-flink-watermarks-idle-partitions-state-ttl-and-replay-4mcm</link>
      <guid>https://dev.to/mikevu/time-in-apache-flink-watermarks-idle-partitions-state-ttl-and-replay-4mcm</guid>
      <description>&lt;p&gt;In stream processing, most of the hard-to-find bugs aren't in the computation but in &lt;strong&gt;time&lt;/strong&gt;: events that arrive late, a partition that goes quiet, state expiring by the wrong clock, and whether a rerun produces the same result.&lt;/p&gt;

&lt;p&gt;This post collects what I learned about time in Apache Flink from an event analytics pipeline I never finished: events from GitHub, Jira and deploy systems, grouped into windows to compute metrics like PR review time or deploy frequency. Where data can be lost or duplicated in the same pipeline is covered in &lt;a href="https://vuminhhoa.com/en/blog/exactly-once-kafka-flink-clickhouse" rel="noopener noreferrer"&gt;exactly-once doesn't come for free&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Event time and watermarks
&lt;/h2&gt;

&lt;p&gt;A PR opened at 10:00 may have its webhook arrive at 10:03. To count "PRs opened during the 10 o'clock hour" correctly, windows must use &lt;strong&gt;event time&lt;/strong&gt; (when the thing happened), not when Flink received it.&lt;/p&gt;

&lt;p&gt;The catch is that Flink can't know when it has seen every event for the 10 o'clock hour. A &lt;a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/event-time/generating_watermarks/" rel="noopener noreferrer"&gt;&lt;strong&gt;watermark&lt;/strong&gt;&lt;/a&gt; is how it says "from now on, I won't wait for anything older than time T". The usual setup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;WatermarkStrategy&lt;/span&gt;
    &lt;span class="o"&gt;.&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="n"&gt;forBoundedOutOfOrderness&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofMinutes&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;withTimestampAssigner&lt;/span&gt;&lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ts&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;eventTime&lt;/span&gt;&lt;span class="o"&gt;().&lt;/span&gt;&lt;span class="na"&gt;toEpochMilli&lt;/span&gt;&lt;span class="o"&gt;())&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;withIdleness&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Duration&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;ofSeconds&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The watermark trails the newest event by 5 minutes. The 10 o'clock window only closes once the watermark passes 11:00, that is, once events from 11:05 onward have been seen. The trade-off is plain: waiting longer misses fewer late events but delivers results later.&lt;/p&gt;

&lt;h2&gt;
  
  
  Events that are too late are dropped, silently
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/operators/windows/#allowed-lateness" rel="noopener noreferrer"&gt;&lt;code&gt;allowedLateness&lt;/code&gt;&lt;/a&gt; gives a window extra time after it closes: events arriving within it update the window's result. But what about events arriving &lt;strong&gt;after&lt;/strong&gt; that?&lt;/p&gt;

&lt;p&gt;By default, Flink &lt;strong&gt;drops them without a word&lt;/strong&gt;. The project's design doc said "events that are too late go to a side output and then to the DLQ", but the code never had that line. Doing it right takes a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;OutputTag&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Event&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;lateTag&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;OutputTag&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;gt;(&lt;/span&gt;&lt;span class="s"&gt;"late-events"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{};&lt;/span&gt;

&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;stream&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;keyBy&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nl"&gt;Event:&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;teamId&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;window&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;TumblingEventTimeWindows&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;of&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;hours&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;)))&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;allowedLateness&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;minutes&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;sideOutputLateData&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lateTag&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;aggregate&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;CountAgg&lt;/span&gt;&lt;span class="o"&gt;());&lt;/span&gt;

&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;getSideOutput&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lateTag&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;sinkTo&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lateEventsSink&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;   &lt;span class="c1"&gt;// and count them in a metric&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You don't have to process late events right away, but you &lt;strong&gt;do have to count them&lt;/strong&gt;. That number tells you whether 5 minutes is enough, instead of guessing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fowdcbegexl7z8peoo1bp.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fowdcbegexl7z8peoo1bp.png" alt="Diagram of a watermark axis for an event belonging to the 10:00 to 11:00 window. If it arrives before the watermark reaches 11:00, it is counted in the first result. Between 11:00 and 11:05 it is late but within allowedLateness, so the result is updated. After 11:05 it is dropped silently unless a side output catches it. The watermark trails the newest event time by 5 minutes." width="799" height="462"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;What decides an event's fate is the watermark when it arrives, not its own timestamp.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  One quiet partition can stall the whole job
&lt;/h2&gt;

&lt;p&gt;With a multi-partition Kafka source, the source's watermark is the &lt;strong&gt;minimum&lt;/strong&gt; across partitions. If one partition has no data, its watermark doesn't move, and it holds the whole job back: no window ever closes.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/event-time/generating_watermarks/#dealing-with-idle-sources" rel="noopener noreferrer"&gt;&lt;code&gt;withIdleness(30s)&lt;/code&gt;&lt;/a&gt; marks a partition "idle" after 30 seconds of silence, and Flink leaves it out of the combined watermark.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc6fywc5soc3olh47ttnh.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fc6fywc5soc3olh47ttnh.png" alt="Diagram of three partitions feeding one source watermark, which is their minimum. Partitions 0 and 1 are advancing, partition 2 has no data. Without withIdleness, partition 2 stays at Long.MIN_VALUE and no window fires. With withIdleness of 30 seconds, partition 2 is marked idle and left out, so windows close again." width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;One empty partition is enough to hold the minimum, and with it every window in the job.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The original design doc only applied &lt;code&gt;withIdleness&lt;/code&gt; to live mode, while replay mode (rereading history) just used ascending timestamps, since historical data is already ordered. The code has a comment correcting that: during replay only a few of the many partitions have data, the empty ones hold the watermark at &lt;code&gt;Long.MIN_VALUE&lt;/code&gt;, and &lt;strong&gt;no window ever fires&lt;/strong&gt;. The job runs, reads everything, and produces nothing. It's the kind of bug you find by running the job, not by reading the design.&lt;/p&gt;

&lt;p&gt;Know the downside of &lt;code&gt;withIdleness&lt;/code&gt; too: when an idle partition comes back, its first events may already be behind the combined watermark and count as late. Set the idle timeout too short and ordinary delays turn into lost data.&lt;/p&gt;

&lt;h2&gt;
  
  
  State TTL: three details that are easy to get wrong
&lt;/h2&gt;

&lt;p&gt;State such as the set of deduplicated keys needs to expire so it doesn't grow forever. The project's config:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="nc"&gt;StateTtlConfig&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;newBuilder&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;hours&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setUpdateType&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;StateTtlConfig&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;UpdateType&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;OnCreateAndWrite&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;setStateVisibility&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;StateTtlConfig&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;StateVisibility&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;NeverReturnExpired&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;cleanupInRocksdbCompactFilter&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;build&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;1. &lt;code&gt;OnCreateAndWrite&lt;/code&gt;, not &lt;code&gt;OnReadAndWrite&lt;/code&gt;.&lt;/strong&gt; For dedup state, every event &lt;em&gt;reads&lt;/em&gt; the state to check it. If reads extended the TTL, a key that keeps getting duplicated would never expire. Extending only on write is the intent: the key lives 24 hours from when it was first seen.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. The &lt;code&gt;cleanupInRocksdbCompactFilter&lt;/code&gt; parameter isn't a sampling rate.&lt;/strong&gt; A comment in the code says "1000 means check one in every 1000 keys during compaction". In fact every key is checked; the number is &lt;strong&gt;how many entries are processed before the filter asks Flink for the current time again&lt;/strong&gt;. A smaller number cleans up more precisely but slows compaction, since each query is a JNI call. Read the API docs rather than trusting comments, your own included.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. TTL runs on wall-clock time, not event time.&lt;/strong&gt; Per &lt;a href="https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/datastream/fault-tolerance/state/#state-time-to-live-ttl" rel="noopener noreferrer"&gt;Flink's documentation&lt;/a&gt;, state TTL currently only supports processing time. As a result, replaying 30 days of data in an hour, 24-hour state won't expire the way it does in production. Which leads to the next section.&lt;/p&gt;

&lt;h2&gt;
  
  
  "Deterministic" replay, and what quietly breaks it
&lt;/h2&gt;

&lt;p&gt;One of event time's most attractive promises: &lt;strong&gt;same input, same output&lt;/strong&gt;, whenever and however fast you run it. That's what lets you fix the logic and rerun history to recompute metrics. The project's design doc made that promise too, on the condition of "no processing-time-dependent logic".&lt;/p&gt;

&lt;p&gt;Rereading the code turned up three things to watch, two of which quietly break that promise:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Processing-time timers.&lt;/strong&gt; The pipeline computing 30-day metrics sets a timer for "the start of the next day" after each event with &lt;code&gt;registerProcessingTimeTimer&lt;/code&gt;. In production, the timer fires once the day is over, after all its events are in. During replay, that moment is already in the past by the wall clock, so the timer fires &lt;strong&gt;almost immediately&lt;/strong&gt;, when only the day's first few events have arrived. The result depends on how fast the job reads. The fix is &lt;code&gt;registerEventTimeTimer&lt;/code&gt;: the timer fires when the &lt;em&gt;watermark&lt;/em&gt; passes it, so fast or slow runs yield the same sequence of results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Instant.now()&lt;/code&gt; in the output.&lt;/strong&gt; Output records carry a field taken from the wall clock. Here it's only "computed at" metadata, so it's harmless, but it's a spot to watch: the moment such a value becomes part of a key or a version, every rerun produces a different row.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The replay flag isn't part of the key.&lt;/strong&gt; The ClickHouse tables use &lt;code&gt;ReplacingMergeTree&lt;/code&gt; with an &lt;code&gt;is_replay&lt;/code&gt; column to separate rerun data from live data. But &lt;code&gt;is_replay&lt;/code&gt; isn't in the &lt;code&gt;ORDER BY&lt;/code&gt;. With this engine, rows with the same sorting key count as one, so on merge a replay row can &lt;strong&gt;replace&lt;/strong&gt; a live row. To keep two kinds of data apart in one table, the distinguishing column must be part of the key, or the data must go in a separate table.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The rule I took away: inside any operator, &lt;strong&gt;only use time that comes from the event or the watermark&lt;/strong&gt;. Any call to the system clock needs a clear reason.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bonus: after hours for whom?
&lt;/h2&gt;

&lt;p&gt;The same pipeline computes an "after-hours work" signal in UTC:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;hour&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;eventTime&lt;/span&gt;&lt;span class="o"&gt;().&lt;/span&gt;&lt;span class="na"&gt;atZone&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;ZoneOffset&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;UTC&lt;/span&gt;&lt;span class="o"&gt;).&lt;/span&gt;&lt;span class="na"&gt;getHour&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hour&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;9&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="n"&gt;hour&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For someone in Hanoi (UTC+7), 9:00–18:00 UTC is 16:00 to 1:00 at night. A whole normal morning of work counts as "after hours". Event time tells you &lt;em&gt;when&lt;/em&gt; something happened; "office hours" belong to &lt;strong&gt;each person&lt;/strong&gt;, and must be computed in their time zone. It's the same kind of trap as &lt;a href="https://vuminhhoa.com/en/blog/vietnamese-lunar-calendar-javascript" rel="noopener noreferrer"&gt;the Vietnamese lunar calendar using UTC+7&lt;/a&gt;: an instant is absolute, but "day" and "hour" are not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Window on event time, with a watermark trailing the newest event by just enough.&lt;/li&gt;
&lt;li&gt;Always &lt;code&gt;sideOutputLateData&lt;/code&gt; and count late events; by default Flink drops them silently.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;withIdleness&lt;/code&gt; is needed in both live and replay modes, and shouldn't be too short.&lt;/li&gt;
&lt;li&gt;State TTL: extend on write for dedup state, read what cleanup parameters actually mean, and remember TTL runs on the wall clock.&lt;/li&gt;
&lt;li&gt;For deterministic replay: event-time timers, no system clock in the logic, and distinguishing flags belong in the key.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Time in stream processing has many more corners I haven't hit yet. If you have good experience to share, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>apacheflink</category>
      <category>streamprocessing</category>
      <category>watermarks</category>
      <category>statettl</category>
    </item>
    <item>
      <title>Server-Sent Events in Spring Boot: Redis Pub/Sub and slow clients</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Tue, 03 Nov 2026 14:00:14 +0000</pubDate>
      <link>https://dev.to/mikevu/server-sent-events-in-spring-boot-redis-pubsub-and-slow-clients-1m5f</link>
      <guid>https://dev.to/mikevu/server-sent-events-in-spring-boot-redis-pubsub-and-slow-clients-1m5f</guid>
      <description>&lt;p&gt;&lt;a href="https://note.vuminhhoa.com/" rel="noopener noreferrer"&gt;Personal OS&lt;/a&gt; is a time-management app used on several devices at once: edit the schedule on a laptop and the phone should see it almost immediately. The backend is Spring Boot (Java 21), and the "real-time" part uses &lt;strong&gt;Server-Sent Events (SSE)&lt;/strong&gt; with Spring MVC's &lt;code&gt;SseEmitter&lt;/code&gt;, plus Redis Pub/Sub when running several instances.&lt;/p&gt;

&lt;p&gt;SSE is quick to set up, but running it reliably takes attention to a few things the docs rarely mention. This post covers the design, and the first thing that breaks as connections grow. It isn't the database, and it isn't the connection pool.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why SSE rather than WebSockets
&lt;/h2&gt;

&lt;p&gt;The app only needs one direction: the server telling the client "something changed". Clients send data up through ordinary REST calls. For that, SSE has some advantages:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It's plain HTTP, so it passes through proxies and load balancers easily.&lt;/li&gt;
&lt;li&gt;Browsers ship &lt;code&gt;EventSource&lt;/code&gt;, which &lt;strong&gt;reconnects automatically&lt;/strong&gt; when the connection drops.&lt;/li&gt;
&lt;li&gt;On the Spring MVC side you just return an &lt;code&gt;SseEmitter&lt;/code&gt;; no new stack required.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The principle: the stream carries hints, not data
&lt;/h2&gt;

&lt;p&gt;This is the most important decision. Each SSE event only contains &lt;strong&gt;the name of the table that changed&lt;/strong&gt;, for example &lt;code&gt;changed: tasks&lt;/code&gt;. When the client gets one, it calls the sync API (REST) to pull the actual data. How a sync API like this resolves conflicts is covered in &lt;a href="https://vuminhhoa.com/en/blog/local-first-pwa-offline-sync" rel="noopener noreferrer"&gt;local-first PWA sync&lt;/a&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;record&lt;/span&gt; &lt;span class="nf"&gt;SyncEvent&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It looks like an extra request, but in exchange:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Authentication, per-user scoping and conflict resolution live in one place&lt;/strong&gt;: the sync API. The SSE stream doesn't need to know about any of it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Losing an event is harmless.&lt;/strong&gt; An event only means "go pull". Clients also pull periodically to reconcile, so a missed hint delays things by at most one cycle and never loses data.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That second point is what saves the whole design later on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Authenticating when &lt;code&gt;EventSource&lt;/code&gt; can't send headers
&lt;/h2&gt;

&lt;p&gt;The browser's &lt;code&gt;EventSource&lt;/code&gt; &lt;strong&gt;can't set headers&lt;/strong&gt; (the &lt;a href="https://developer.mozilla.org/en-US/docs/Web/API/EventSource/EventSource" rel="noopener noreferrer"&gt;&lt;code&gt;EventSource()&lt;/code&gt; constructor&lt;/a&gt; only takes a &lt;code&gt;withCredentials&lt;/code&gt; option), so it can't send &lt;code&gt;Authorization: Bearer ...&lt;/code&gt;. A few options:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Put the token in the query string: simple, but the token ends up in proxy logs and browser history. Best avoided.&lt;/li&gt;
&lt;li&gt;Use a polyfill instead of &lt;code&gt;EventSource&lt;/code&gt; so you can send headers: works, but you lose the built-in reconnection.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An &lt;code&gt;HttpOnly&lt;/code&gt; cookie&lt;/strong&gt;: the client calls an endpoint (authenticated with a Bearer token like every other API call) that sets an &lt;code&gt;HttpOnly&lt;/code&gt; cookie holding a token, then opens the &lt;code&gt;EventSource&lt;/code&gt;. On the server, a filter reads the cookie and validates the token with &lt;strong&gt;the same &lt;code&gt;JwtDecoder&lt;/code&gt;&lt;/strong&gt; as the rest of the API.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Personal OS uses the third. The filter only applies when the request isn't already authenticated by a header, so the standard Bearer flow still takes precedence.&lt;/p&gt;

&lt;p&gt;One trade-off to be aware of: the token is only checked &lt;strong&gt;when the stream opens&lt;/strong&gt;. The longer a stream stays open, the longer a revoked session can keep its connection. That's the second reason the "no data" principle matters: the stream only carries table names, and data still goes through REST, where every request is authenticated. The stream's lifetime (60 minutes here) then becomes just a balance between how often clients reconnect and how quickly a revocation takes effect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Several instances: Redis Pub/Sub
&lt;/h2&gt;

&lt;p&gt;Each instance only knows the SSE connections open &lt;em&gt;to itself&lt;/em&gt;. A user edits data through instance A while their phone is connected to instance B. The answer is one interface with two implementations:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="kd"&gt;interface&lt;/span&gt; &lt;span class="nc"&gt;SyncEventBus&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;publish&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;LocalSyncEventBus&lt;/code&gt;: a single instance, calling straight into the in-memory registry.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;RedisSyncEventBus&lt;/code&gt;: publishes to a Redis channel. Every instance subscribes and delivers to its own connections.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuzxja80kkqx0k83qr8tl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuzxja80kkqx0k83qr8tl.png" alt="Sequence diagram with five participants: laptop, instance A, Redis, instance B and phone. The laptop writes through REST to instance A. A publishes the user and table to Redis. Redis delivers it to every subscriber, including instance B. B sends the SSE hint changed: tasks to the phone, and the phone pulls the data through the REST sync API." width="800" height="622"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The stream only says which table changed. The data itself still goes through the authenticated REST sync.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;If publishing to Redis fails, it's logged as a warning. Following the principle above, a lost hint is fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first thing to break: one slow client slows everyone
&lt;/h2&gt;

&lt;p&gt;A common misconception is that "each SSE connection holds a thread". With Spring MVC that's not quite true: &lt;code&gt;SseEmitter&lt;/code&gt; runs on async Servlet processing, so the request thread goes back to the pool as soon as the controller returns the emitter. An open connection that isn't sending anything costs no thread.&lt;/p&gt;

&lt;p&gt;The problem is &lt;strong&gt;sending&lt;/strong&gt;. &lt;code&gt;SseEmitter.send()&lt;/code&gt; writes to the socket and &lt;strong&gt;blocks until the write completes&lt;/strong&gt;. A client on a slow network (with a full TCP buffer) makes that call block for a long time. And in the original design, both sending paths ran on shared threads:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Heartbeats&lt;/strong&gt; ran on the &lt;code&gt;@Scheduled&lt;/code&gt; thread. By default Spring has &lt;strong&gt;one&lt;/strong&gt; thread for all &lt;code&gt;@Scheduled&lt;/code&gt; jobs (see &lt;a href="https://docs.spring.io/spring-boot/reference/features/task-execution-and-scheduling.html" rel="noopener noreferrer"&gt;Spring Boot's task scheduling defaults&lt;/a&gt;), shared with every other job.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Change hints&lt;/strong&gt; were sent on Redis's listener thread, also a single thread per instance.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So one slow client stalled &lt;strong&gt;the whole send loop&lt;/strong&gt;. Other clients missed heartbeats, proxies assumed the connections were dead and cut them, everyone reconnected at once (a reconnect storm), and every reconnect triggered a pull. That's the &lt;strong&gt;first&lt;/strong&gt; limit you hit as devices grow, before the database or the connection pool.&lt;/p&gt;

&lt;h3&gt;
  
  
  The fix: a separate, bounded pool for writes
&lt;/h3&gt;

&lt;p&gt;Shortened from &lt;code&gt;SyncEventRegistry.java&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kd"&gt;final&lt;/span&gt; &lt;span class="nc"&gt;ThreadPoolExecutor&lt;/span&gt; &lt;span class="n"&gt;deliverExecutor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ThreadPoolExecutor&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;
    &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;TimeUnit&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;SECONDS&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;LinkedBlockingQueue&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;gt;(&lt;/span&gt;&lt;span class="mi"&gt;10_000&lt;/span&gt;&lt;span class="o"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;namedDaemonThreadFactory&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"sse-deliver-"&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;

&lt;span class="kd"&gt;private&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nf"&gt;enqueueSend&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;SseEmitter&lt;/span&gt; &lt;span class="n"&gt;emitter&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;SseEventBuilder&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;deliverExecutor&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;execute&lt;/span&gt;&lt;span class="o"&gt;(()&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;send&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;userId&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;emitter&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="o"&gt;));&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;RejectedExecutionException&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// Queue full: drop this hint; the client still reconciles periodically.&lt;/span&gt;
    &lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three choices worth noting:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A small pool (4 threads).&lt;/strong&gt; Writing is I/O-bound, not CPU-bound, so a few threads serve many connections. A slow client holds just one thread and never blocks the caller.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A bounded queue (10,000).&lt;/strong&gt; Without a bound, slow clients grow the queue until memory runs out.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Drop when full, not &lt;code&gt;CallerRunsPolicy&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;CallerRunsPolicy&lt;/code&gt; is the familiar "safe" choice, but here the "caller" is the heartbeat thread or the Redis listener thread. Having them run the write themselves rebuilds exactly the blocking I set out to avoid. Dropping a hint is safe, thanks to the principle at the top.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Also, &lt;code&gt;SseEmitter&lt;/code&gt; &lt;strong&gt;isn't safe for concurrent sends&lt;/strong&gt;, so all sends to the same emitter must be serialized.&lt;/p&gt;

&lt;h2&gt;
  
  
  Coalescing hints within 300 ms
&lt;/h2&gt;

&lt;p&gt;A large action on the client (copying a whole week, importing data) pushes dozens of consecutive batches into the same table. Every batch commit publishes a hint, which means dozens of SSE writes to &lt;em&gt;each&lt;/em&gt; device, each one triggering a pull.&lt;/p&gt;

&lt;p&gt;The fix: hints for the same &lt;code&gt;(user, table)&lt;/code&gt; arriving within &lt;strong&gt;300 ms&lt;/strong&gt; are merged into one send. That's long enough to cover back-to-back batches and short enough that other devices still see the change almost instantly. One subtle detail: the "send pending" marker is removed &lt;strong&gt;before&lt;/strong&gt; sending, so a hint that arrives mid-send schedules a new send instead of being swallowed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Other small details
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A heartbeat every 25 seconds&lt;/strong&gt; (an SSE comment the client never sees), so proxies and load balancers don't cut "silent" connections. Many proxies drop idle connections after about a minute; Nginx's default &lt;a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout" rel="noopener noreferrer"&gt;&lt;code&gt;proxy_read_timeout&lt;/code&gt;&lt;/a&gt;, for example, is 60 seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A &lt;code&gt;connected&lt;/code&gt; event&lt;/strong&gt; is sent synchronously as soon as the stream opens, so the client knows the connection is ready. It's the only write that skips the pool, since it runs on the request's own thread for a brand-new emitter.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Clean up&lt;/strong&gt; emitters in all three callbacks: &lt;code&gt;onCompletion&lt;/code&gt;, &lt;code&gt;onTimeout&lt;/code&gt; and &lt;code&gt;onError&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  When to move to WebFlux
&lt;/h2&gt;

&lt;p&gt;For a few hundred devices, the design above is plenty. At thousands of concurrent connections, other limits show up: Tomcat's maximum connections, file descriptors, and the cost of blocking writes on threads. There are two directions worth considering then:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;WebFlux&lt;/strong&gt; with &lt;code&gt;Flux&amp;lt;ServerSentEvent&amp;gt;&lt;/code&gt;: an event-loop model, non-blocking writes, backpressure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Java 21 virtual threads&lt;/strong&gt; (&lt;code&gt;spring.threads.virtual.enabled=true&lt;/code&gt; since Spring Boot 3.2): keep the blocking code style while threads become very cheap.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Because of the &lt;code&gt;SyncEventBus&lt;/code&gt; interface and a stream that only carries table names, either move &lt;strong&gt;needs no frontend changes&lt;/strong&gt;: the &lt;code&gt;text/event-stream&lt;/code&gt; protocol and the endpoint stay the same.&lt;/p&gt;

&lt;p&gt;Rather than guessing when to switch, measure. Metrics worth watching: live JVM threads, Tomcat's current connections, the send pool's queue size, and the number of dropped hints.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;SSE fits when the server only needs to tell clients that something changed: it's plain HTTP, and &lt;code&gt;EventSource&lt;/code&gt; reconnects on its own.&lt;/li&gt;
&lt;li&gt;Send hints, not data. The stream carries only the table name, and clients pull through the authenticated sync API, so a lost hint costs at most one reconciliation cycle.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;EventSource&lt;/code&gt; can't send headers. An &lt;code&gt;HttpOnly&lt;/code&gt; cookie validated by the same &lt;code&gt;JwtDecoder&lt;/code&gt; keeps the token out of URLs, but the token is checked only when the stream opens.&lt;/li&gt;
&lt;li&gt;With several instances, one &lt;code&gt;SyncEventBus&lt;/code&gt; interface and a Redis Pub/Sub implementation get each hint to whichever instance holds the connection.&lt;/li&gt;
&lt;li&gt;The first thing to break is a blocking &lt;code&gt;send()&lt;/code&gt; on shared threads: one slow client stalls heartbeats for everyone. Move writes to a small, bounded pool and drop hints when it's full.&lt;/li&gt;
&lt;li&gt;Coalesce hints per user and table within 300 ms, and send a heartbeat every 25 seconds so proxies keep idle connections open.&lt;/li&gt;
&lt;li&gt;Measure live threads, Tomcat connections, the send queue and dropped hints before moving to WebFlux or virtual threads.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you've built real-time updates with Spring and learned different lessons, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>java</category>
      <category>springboot</category>
      <category>sse</category>
      <category>redis</category>
    </item>
    <item>
      <title>AI and the October 2026 vulnerability storm: what the numbers show</title>
      <dc:creator>Mike Vu</dc:creator>
      <pubDate>Fri, 30 Oct 2026 09:54:11 +0000</pubDate>
      <link>https://dev.to/mikevu/ai-and-the-october-2026-vulnerability-storm-what-the-numbers-show-4bbg</link>
      <guid>https://dev.to/mikevu/ai-and-the-october-2026-vulnerability-storm-what-the-numbers-show-4bbg</guid>
      <description>&lt;p&gt;On 21 October 2026, PHP patched two flaws that allow remote code execution without logging in, and several European government agencies took websites offline to patch. Through October, critical bugs kept arriving in quick succession. &lt;strong&gt;Is AI making software less secure, or just making old bugs surface faster?&lt;/strong&gt; The short answer: disclosed vulnerabilities have doubled this year and AI-credited finds are up more than 20 times, but exploitation grew far more slowly: about 20% by CISA's count. What changed most is speed, from disclosure to attack, and the bottleneck has moved to the people who patch.&lt;/p&gt;

&lt;p&gt;I read the primary reports from Google Threat Intelligence Group (GTIG), VulnCheck, CrowdStrike and Anthropic, an independent analysis of the full CVE list, and coverage in the Vietnamese press. This post puts them in one place: what happened, what the numbers say, the ways AI is involved, and what the numbers don't prove.&lt;/p&gt;

&lt;h2&gt;
  
  
  October 2026: PHP2Shell and a month full of vulnerabilities
&lt;/h2&gt;

&lt;p&gt;The month's biggest event was PHP2Shell: two memory bugs in PHP itself (CVE-2026-109618 and CVE-2026-109619), each scored CVSS 9.2. According to &lt;a href="https://blog.cloudflare.com/php-vulnerabilities-waf-protection/" rel="noopener noreferrer"&gt;Cloudflare&lt;/a&gt;, they affect every web server running PHP 5.4 or later, and were disclosed jointly by Calif, Wiz, OpenAI and Quarkslab. The timeline below combines &lt;a href="https://www.recordedfuture.com/blog/php2shell-vulnerability-exploitation" rel="noopener noreferrer"&gt;Recorded Future's analysis&lt;/a&gt;, the &lt;a href="https://www.php.net/archive/2026.php" rel="noopener noreferrer"&gt;php.net news archive&lt;/a&gt; and the Vietnamese press:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;When&lt;/th&gt;
&lt;th&gt;What&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Early Aug&lt;/td&gt;
&lt;td&gt;Ryan Austin of Calif finds and reports a bug to PHP (&lt;a href="https://vnexpress.net/mot-tuan-hack-ca-the-gioi-5136727.html" rel="noopener noreferrer"&gt;per Duong Ngoc Thai&lt;/a&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;19–24 Aug&lt;/td&gt;
&lt;td&gt;Wiz Research finds a PHP bug and reports the vulnerabilities to PHP (Recorded Future)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20 Oct&lt;/td&gt;
&lt;td&gt;PHP pre-announces an emergency security release for the next day&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;21 Oct, 02:29 UTC&lt;/td&gt;
&lt;td&gt;An archive with vulnerability details leaks to VirusTotal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;21 Oct, 07:34 UTC&lt;/td&gt;
&lt;td&gt;Patchstack gets reports of real exploitation from hosting partners&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;21 Oct, 13:52 UTC&lt;/td&gt;
&lt;td&gt;PHP ships 8.5.12, 8.4.27, 8.3.36 and 8.2.35&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;21–22 Oct&lt;/td&gt;
&lt;td&gt;France and the Netherlands take some government sites and services offline (&lt;a href="https://tuoitre.vn/lo-hong-bao-mat-do-ai-phat-hien-khien-loat-website-chinh-phu-nhieu-nuoc-ngung-hoat-dong-100261022095015218.htm" rel="noopener noreferrer"&gt;Tuoi Tre&lt;/a&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;22 Oct&lt;/td&gt;
&lt;td&gt;Vietnam's Department of Cybersecurity and National Cybersecurity Association advise a review and "not worrying excessively" (&lt;a href="https://vietnamnet.vn/cuc-an-ninh-mang-khuyen-cao-doanh-nghiep-ra-soat-lo-hong-php-2567511.html" rel="noopener noreferrer"&gt;VietNamNet&lt;/a&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;23 Oct&lt;/td&gt;
&lt;td&gt;PHP announces CVE-2026-115560, narrower, not yet patched&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Vietnam is not on the sidelines. &lt;a href="https://vnexpress.net/ngay-toi-te-cua-internet-5135404.html" rel="noopener noreferrer"&gt;VnExpress&lt;/a&gt;, citing BuiltWith, counts about 143,000 websites in Vietnam running PHP. Calif, one of the teams that found the bugs, is run by Vietnamese engineer Duong Ngoc Thai. On the same 21 October, CrowdStrike published four threat actors using AI agents (more below), and LiteLLM, a proxy that routes requests to many language models, had malware slipped into its release packages.&lt;/p&gt;

&lt;h2&gt;
  
  
  Disclosed vulnerabilities in 2026 have doubled
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;From 1 January to 23 October 2026, 86,292 CVEs were published, 2.2 times the same window of 2025 and more than all of 2025 (48,151).&lt;/strong&gt; These figures come from &lt;a href="https://cyber-hire.com/blog/cve-volume-2026" rel="noopener noreferrer"&gt;CyberHire's analysis&lt;/a&gt; of a full CVE list download taken on 23 October. A few of them:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric (1 Jan–23 Oct)&lt;/th&gt;
&lt;th&gt;2025&lt;/th&gt;
&lt;th&gt;2026&lt;/th&gt;
&lt;th&gt;Growth&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;All CVEs&lt;/td&gt;
&lt;td&gt;38,569&lt;/td&gt;
&lt;td&gt;86,292&lt;/td&gt;
&lt;td&gt;2.2×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Critical CVEs&lt;/td&gt;
&lt;td&gt;3,348&lt;/td&gt;
&lt;td&gt;9,599&lt;/td&gt;
&lt;td&gt;2.9×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CVEs in September alone&lt;/td&gt;
&lt;td&gt;4,319&lt;/td&gt;
&lt;td&gt;14,934&lt;/td&gt;
&lt;td&gt;3.5×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Google Chrome CVEs&lt;/td&gt;
&lt;td&gt;110&lt;/td&gt;
&lt;td&gt;3,593&lt;/td&gt;
&lt;td&gt;33×&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;GTIG sees the same trend month by month: monthly CVE disclosures went from 5,045 in January to 10,740 in August, per &lt;a href="https://www.helpnetsecurity.com/2026/10/01/google-ai-discovered-vulnerabilities-remote-code-execution/" rel="noopener noreferrer"&gt;Help Net Security's summary of the report&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI is doing the finding: from 34 to 769 credited CVEs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;CVEs that name an AI system or an AI lab's research team as the finder went from 34 to 769 over the same window.&lt;/strong&gt; Per CyberHire, the most came from Anthropic/Claude (335), AISLE (296) and OpenAI (112), and almost half (376) are rated high or critical. Affected projects include OpenSSL, curl, PostgreSQL, FreeBSD, Mozilla and GitLab.&lt;/p&gt;

&lt;p&gt;769 is still under 1% of this year's CVEs, and it is a floor. GTIG explains why the public count is low: CVE records have no standard tag for "found by AI", and cloud and SaaS providers often fix such bugs without requesting a CVE ID. GTIG also found that &lt;strong&gt;50% of likely AI-discovered flaws lead to remote code execution, against 26% of flaws found other ways&lt;/strong&gt;, and attributes this to research programs pointing their AI agents at infrastructure and privilege boundaries.&lt;/p&gt;

&lt;p&gt;The bugs themselves are not new. PHP2Shell has been in the code since PHP 5.4, released in 2012. In &lt;a href="https://vnexpress.net/mot-tuan-hack-ca-the-gioi-5136727.html" rel="noopener noreferrer"&gt;a column on VnExpress&lt;/a&gt;, Duong Ngoc Thai writes that it belongs to a bug class hacker Stefan Esser found back in 2004. &lt;strong&gt;AI didn't create these vulnerabilities. It made finding them cheap&lt;/strong&gt;, and that cheapness is available to everyone: defenders, researchers and attackers.&lt;/p&gt;

&lt;h2&gt;
  
  
  More bugs found has not meant proportionally more bugs exploited
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Vulnerabilities doubled, but exploitation grew much more slowly: about 20% by CISA's count, more by GTIG's.&lt;/strong&gt; In all three sources below, the exploited share stays far below the growth in disclosures:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GTIG:&lt;/strong&gt; only 0.23% of 2026 CVEs (about 1 in 431) were observed exploited in the wild. January–August 2026 saw 141 exploited vulnerabilities, more than the 127 in all of 2025. Exploited zero-days rose from an average of 8 to 11 per month.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://www.vulncheck.com/blog/state-of-exploitation-1h-2026" rel="noopener noreferrer"&gt;VulnCheck&lt;/a&gt;:&lt;/strong&gt; of 1,061 vulnerabilities attributed to AI-assisted discovery, 14 (1.3%) have been confirmed exploited, close to the rate across all vulnerabilities in the first half of 2026.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CyberHire:&lt;/strong&gt; CISA flagged 195 of the 2026 CVEs as actively exploited, against 163 in 2025. By CyberHire's figures, the exploited share fell from about 0.45% to 0.25%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs4x46w6imoykro4xuum0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fs4x46w6imoykro4xuum0.png" alt="Horizontal bar chart of growth from 2025 to 2026 (1 Jan–23 Oct): CVEs CISA flags as exploited 1.2×, all CVEs 2.2×, critical CVEs 2.9×, CVEs in AI software 6.3×, CVEs credited to AI 22.6×." width="800" height="339"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Bugs being found are growing far faster than bugs being exploited.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The chart puts the growth rates side by side. AI-credited bugs grew more than 22 times, all CVEs 2.2 times, exploited bugs only 1.2 times. None of the sources explains the gap in detail. Likely reasons are that many new bugs are hard to exploit, sit on code paths few people reach, or get patched before anyone uses them. Both PHP2Shell CVEs have "high" attack complexity in their CVSS vectors (&lt;a href="https://github.com/php/php-src/security/advisories/GHSA-6wmf-4rv5-p3rp" rel="noopener noreferrer"&gt;&lt;code&gt;GHSA-6wmf-4rv5-p3rp&lt;/code&gt;&lt;/a&gt;, &lt;a href="https://github.com/php/php-src/security/advisories/GHSA-w8hw-w9fx-j79w" rel="noopener noreferrer"&gt;&lt;code&gt;GHSA-w8hw-w9fx-j79w&lt;/code&gt;&lt;/a&gt;).&lt;/p&gt;

&lt;h2&gt;
  
  
  What really changed: the time from disclosure to attack
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The gap between disclosure and exploitation is shrinking from weeks to days or hours, and that is the part wearing operations teams down.&lt;/strong&gt; GTIG's example is CVE-2026-1731, an unauthenticated OS command injection in BeyondTrust's remote access products, found by the Hacktron AI research agent. One threat cluster exploited it within four days of disclosure, five more within seven days. GTIG suggests attackers may be using large language models to rapidly weaponize already disclosed bugs (n-days).&lt;/p&gt;

&lt;p&gt;PHP2Shell moved even faster: details leaked at 02:29 UTC, real exploitation was reported at 07:34, and the patch shipped at 13:52 the same day. For several hours before the patch, a WAF was the main protection available: Cloudflare's rules went live at 03:02 UTC. A public patch is also a map: diff the code before and after and you know where the bug is, and that is exactly the kind of work AI does quickly.&lt;/p&gt;

&lt;h2&gt;
  
  
  Attackers are already using AI agents
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;AI agents are no longer something only seen in defenders' demos.&lt;/strong&gt; On 21 October, &lt;a href="https://www.crowdstrike.com/en-us/blog/crowdstrike-uncovers-new-agentic-ai-campaigns/" rel="noopener noreferrer"&gt;CrowdStrike&lt;/a&gt; published four previously unreported threat actors that used AI agents to make decisions and run commands during intrusions between May and August 2026. It assesses with moderate confidence that agentic AI turns baseline offensive operations into a commodity, and with high confidence that agent-supported intrusions will almost certainly increase in volume and sophistication.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://www.anthropic.com/threat-intelligence-report-september-2026" rel="noopener noreferrer"&gt;Anthropic's September 2026 report&lt;/a&gt; on misuse it disrupted shows the speed in concrete terms. One financially motivated group dumped more than 2,100 Azure AD token sets across 40+ tenants in about 34 hours. Another case went from a single stolen developer token to full cloud admin in about three hours. The report also notes that a campaign's sophistication is no longer a reliable clue to who is behind it, because AI has narrowed the gap between well-resourced state operations and individual operators.&lt;/p&gt;

&lt;h2&gt;
  
  
  AI software itself is the new target
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;CVEs in AI software went from 505 to 3,191, 3.7% of all 2026 CVEs&lt;/strong&gt; (CyberHire). GTIG counted 2,076 vulnerabilities in AI-related software from January 2025 to August 2026, over 1,500 of them disclosed in 2026. About half sit in agent orchestration frameworks such as Flowise and Langflow, where code execution nodes can be abused. Inference and serving software (vLLM, Ollama, LiteLLM) had 212 disclosures in 2026, nearly a quarter involving unauthenticated APIs or SSRF.&lt;/p&gt;

&lt;p&gt;The attackers' interest is practical. A compromised AI gateway holds the API keys of every provider behind it. Anthropic's report records AI API keys and session tokens becoming loot in their own right: free compute plus a layer of attribution cover. If you run LiteLLM, Langflow or a self-hosted agent framework, that is internet-facing software like any other, and it needs patching and least privilege like any other.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bottleneck has moved to the people who patch
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Finding bugs now runs at machine speed, while fixing them still runs at human speed.&lt;/strong&gt; PHP runs about 70% of websites (&lt;a href="https://vnexpress.net/lo-hong-php2shell-khi-ai-lam-dao-lon-gioi-an-ninh-mang-5135794.html" rel="noopener noreferrer"&gt;W3Techs, via VnExpress&lt;/a&gt;), but according to &lt;a href="https://www.thestandard.com.hk/world/article/346373/PHP-Foundation-under-funded-with-one-full-time-staff-member-director-says" rel="noopener noreferrer"&gt;Reuters via The Standard&lt;/a&gt;, the PHP Foundation has one full-time staff member, its executive director, plus 14 mostly part-time contractors. The project receives nearly twice as many proposed code changes as a year ago, while the number of contributors hasn't doubled.&lt;/p&gt;

&lt;p&gt;Junk reports make the bottleneck worse. When it set up its &lt;a href="https://thephp.foundation/blog/2026/05/18/announcing-ecosystem-security-team/" rel="noopener noreferrer"&gt;ecosystem security team&lt;/a&gt; in May, the PHP Foundation wrote that many projects report a drastic increase in vulnerability reports, many bearing the hallmarks of lazy AI generation. The curl project went further and &lt;a href="https://www.theregister.com/2026/01/21/curl_ends_bug_bounty/" rel="noopener noreferrer"&gt;ended its bug bounty&lt;/a&gt; at the end of January 2026 to remove the incentive for low-quality reports.&lt;/p&gt;

&lt;p&gt;For operators, the bottleneck is a simpler question: &lt;strong&gt;what am I running, and which version?&lt;/strong&gt; The four steps Vietnam's Department of Cybersecurity recommended for PHP2Shell start exactly there: list every server, website and container using PHP with its version, and only then patch, add WAF rules and watch logs.&lt;/p&gt;

&lt;h2&gt;
  
  
  What these numbers don't prove
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;None of the sources above proves that AI caused CVE volume to double.&lt;/strong&gt; CyberHire's author says so plainly: most large vendors don't disclose how bugs were found, so the data can't separate out AI's share. Numbering has changed too: some issuers moved to one CVE per bug instead of one per patch, and 1,662 organizations issued CVEs this year that issued none last year. And 2026 isn't over.&lt;/p&gt;

&lt;p&gt;The figures on AI-agent attacks come from companies that sell security or AI products. They are real evidence about specific cases, not a representative statistic. And this post can't answer the question you care about most: is your website safe? Only a list of what you run, with versions, answers that.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;From 1 January to 23 October 2026 there were 86,292 CVEs, 2.2 times the same window of 2025. AI-credited CVEs went from 34 to 769, and that is a floor.&lt;/li&gt;
&lt;li&gt;Exploitation grew far more slowly: about 20% by CISA's count. The exploited share fell to about 0.23–0.25%.&lt;/li&gt;
&lt;li&gt;What really changed is speed: PHP2Shell was exploited before its patch shipped, and an AI-found BeyondTrust bug was exploited within four days.&lt;/li&gt;
&lt;li&gt;Attackers already use AI agents, and AI software (gateways, agent frameworks, API keys) is a new target.&lt;/li&gt;
&lt;li&gt;The bottleneck is patching and knowing what you run. Start from your software and version list, not from CVSS scores.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you run systems and changed how you triage and prioritize vulnerabilities this year, I'd love to compare notes. My email is on the &lt;a href="https://vuminhhoa.com/en#contact" rel="noopener noreferrer"&gt;home page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>vulnerabilitymanagement</category>
      <category>aisecurity</category>
      <category>cve</category>
      <category>php2shell</category>
    </item>
  </channel>
</rss>
