<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Hoonjae Lee</title>
    <description>The latest articles on DEV Community by Hoonjae Lee (@newmean-ai).</description>
    <link>https://dev.to/newmean-ai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4247035%2F6aa466cc-92ef-4b49-9e21-6272dda96f63.png</url>
      <title>DEV Community: Hoonjae Lee</title>
      <link>https://dev.to/newmean-ai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/newmean-ai"/>
    <language>en</language>
    <item>
      <title>The runtime was as big as my 2.6 MB PII model, so I wrote a 13 KB inference engine</title>
      <dc:creator>Hoonjae Lee</dc:creator>
      <pubDate>Mon, 16 Nov 2026 06:37:40 +0000</pubDate>
      <link>https://dev.to/newmean-ai/the-runtime-was-as-big-as-my-26-mb-pii-model-so-i-wrote-a-13-kb-inference-engine-ahj</link>
      <guid>https://dev.to/newmean-ai/the-runtime-was-as-big-as-my-26-mb-pii-model-so-i-wrote-a-13-kb-inference-engine-ahj</guid>
      <description>&lt;p&gt;I wanted to remove personal data from survey answers before they left the user's browser. Names, phone numbers, addresses, ID numbers: found and masked on the device, so the server only ever sees &lt;code&gt;[PERSON]&lt;/code&gt; and &lt;code&gt;[PHONE]&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That needs a model small enough to ship to a web page. This post is about how small it got, and about the part I did not expect to write: my own inference engine.&lt;/p&gt;

&lt;p&gt;The result is &lt;a href="https://github.com/newmean-ai/pii" rel="noopener noreferrer"&gt;newmean-pii&lt;/a&gt;: four models from 2.6 MB to 15.5 MB, 54 languages, Apache-2.0. You can try it at &lt;a href="https://pii.newmean.ai" rel="noopener noreferrer"&gt;pii.newmean.ai&lt;/a&gt; without installing anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: shrink the model
&lt;/h2&gt;

&lt;p&gt;All four models start from mMiniLMv2 (6 layers, hidden size 384) and are distilled from two much larger private teachers. Then they are cut down:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Vocabulary pruning.&lt;/strong&gt; The XLM-R vocabulary has about 250,000 pieces. Only 35,298 occur in my training data, so the rest go.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Low-rank embeddings.&lt;/strong&gt; After pruning, the embedding table is still most of the model. The smaller tiers factor it into a narrow table and a projection.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;4-bit weights.&lt;/strong&gt; Every weight matrix is stored in 4 bits. For the two larger tiers the embeddings are too, which changes the miss rate by at most 0.1 points and saves 2.7 to 5.2 MB.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Layer sharing.&lt;/strong&gt; The smallest tier has one transformer layer and applies it four times.&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;tier&lt;/th&gt;
&lt;th&gt;structure&lt;/th&gt;
&lt;th&gt;parameters&lt;/th&gt;
&lt;th&gt;download&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;large&lt;/td&gt;
&lt;td&gt;6 layers, full pruned embeddings&lt;/td&gt;
&lt;td&gt;24.4M&lt;/td&gt;
&lt;td&gt;15.5 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;medium (default)&lt;/td&gt;
&lt;td&gt;6 layers, embeddings rank 192&lt;/td&gt;
&lt;td&gt;17.7M&lt;/td&gt;
&lt;td&gt;11.3 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;small&lt;/td&gt;
&lt;td&gt;4 layers, rank 64&lt;/td&gt;
&lt;td&gt;9.6M&lt;/td&gt;
&lt;td&gt;7.1 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;nano&lt;/td&gt;
&lt;td&gt;one shared layer applied 4 times, rank 32&lt;/td&gt;
&lt;td&gt;3.1M&lt;/td&gt;
&lt;td&gt;2.6 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Step 2: notice the runtime
&lt;/h2&gt;

&lt;p&gt;With the model at 2.6 MB, I looked at what else the page downloads. onnxruntime-web is about 2.4 MB after brotli. The runtime was as large as the model it ran.&lt;/p&gt;

&lt;p&gt;I tried a minimal build of onnxruntime-web with only the operators my models use. That came to 0.34–0.43 MB, which is better but still a general runtime doing a specific job.&lt;/p&gt;

&lt;p&gt;My models are all the same architecture: an XLM-R style encoder, a 4-bit matrix multiply, an embedding lookup, a token classifier. A program that runs only that does not need a graph interpreter, an operator registry or a memory planner.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: 483 lines of C
&lt;/h2&gt;

&lt;p&gt;The engine is one file of freestanding C, compiled to WebAssembly with SIMD. No libc and no imports, so the module asks nothing of the host:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;clang --target=wasm32-unknown-unknown -O3 -msimd128 \
  -ffreestanding -nostdlib -Wl,--no-entry -Wl,--strip-all \
  -o pii.wasm pii.c
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No libc means writing the few things I needed. The allocator is a bump pointer that never frees, because the model is loaded once and the buffers live as long as the page:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight c"&gt;&lt;code&gt;&lt;span class="kd"&gt;extern&lt;/span&gt; &lt;span class="kt"&gt;unsigned&lt;/span&gt; &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="n"&gt;__heap_base&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;uintptr_t&lt;/span&gt; &lt;span class="n"&gt;heap_top&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uintptr_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;__heap_base&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="nf"&gt;pii_alloc&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uint32_t&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kt"&gt;uintptr_t&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;heap_top&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;63&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uintptr_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="mi"&gt;63&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kt"&gt;uintptr_t&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kt"&gt;uintptr_t&lt;/span&gt; &lt;span class="n"&gt;have&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;__builtin_wasm_memory_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;65536u&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;have&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;uintptr_t&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;end&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;have&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;65535&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;65536&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;__builtin_wasm_memory_grow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;uintptr_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="n"&gt;heap_top&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;end&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rest is the model itself: dequantize a panel of 4-bit weights, multiply, layer norm, attention, GELU. Each is written once, for the one shape it will ever see.&lt;/p&gt;

&lt;p&gt;There are two builds. The second uses fused multiply-add through WebAssembly relaxed SIMD, and the loader picks it where the browser supports it (Chrome, Edge, Firefox).&lt;/p&gt;

&lt;p&gt;JavaScript glue and WebAssembly together are about 13 KB after brotli.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it bought
&lt;/h2&gt;

&lt;p&gt;Desktop, headless Chrome, one thread, the 11.3 MB model:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;runtime&lt;/th&gt;
&lt;th&gt;download&lt;/th&gt;
&lt;th&gt;load&lt;/th&gt;
&lt;th&gt;40 tokens&lt;/th&gt;
&lt;th&gt;252 tokens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;built-in engine&lt;/td&gt;
&lt;td&gt;13 KB&lt;/td&gt;
&lt;td&gt;63 ms&lt;/td&gt;
&lt;td&gt;23 ms&lt;/td&gt;
&lt;td&gt;147 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;built-in engine, relaxed SIMD&lt;/td&gt;
&lt;td&gt;13 KB&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;td&gt;14 ms&lt;/td&gt;
&lt;td&gt;91 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;onnxruntime-web 1.31&lt;/td&gt;
&lt;td&gt;2.4 MB&lt;/td&gt;
&lt;td&gt;230 ms&lt;/td&gt;
&lt;td&gt;31 ms&lt;/td&gt;
&lt;td&gt;146 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On four phones (one iPhone, three Android), the built-in engine is about 1.9× faster on short texts and about 24% faster on long ones. First load takes 0.77 s instead of 2.85 s. A 40-token message takes about 26 ms with the default model.&lt;/p&gt;

&lt;p&gt;Where it does not win:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Without relaxed SIMD, long texts are a tie (147 ms against 146 ms).&lt;/li&gt;
&lt;li&gt;onnxruntime-web can use several threads on cross-origin isolated pages. Mine is single-threaded.&lt;/li&gt;
&lt;li&gt;WebGPU runs long texts in 15 ms. Its runtime is 28.7 MB, so I left it out.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;onnxruntime-web is still in the package as an option. Each engine is a separate chunk, so a page downloads only the one it picks.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I know it is right
&lt;/h2&gt;

&lt;p&gt;A hand-written engine is easy to get subtly wrong, so it is checked against the reference rather than trusted:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Both engines return the same spans as the Python evaluator on a 412-text parity set (410 for one nano file, the same two texts for both engines).&lt;/li&gt;
&lt;li&gt;On every phone, model size and engine, the spans matched a desktop reference on 108 texts in 54 languages.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The Python, iOS, Android and Flutter packages run the same models through one C core, with token ids, offsets and logits equal to the npm package's.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;The engine runs exactly one architecture. If I change the model's structure, I change the C. That is a fair price for four models that share a design. It would be a bad one for a project that swaps architectures every month.&lt;/p&gt;

&lt;h2&gt;
  
  
  The model is not perfect either
&lt;/h2&gt;

&lt;p&gt;Share of personal-data characters left unmasked, on datasets the models were not trained on: 4.9–6.3% on OpenPII (30 languages), 6.1–9.9% on Nemotron-PII (English), 9.5–11.5% on KDPII (Korean). Finance documents are the weak spot at about 21%. The two smallest models also mask too much in short table cells.&lt;/p&gt;

&lt;p&gt;So it is a strong first pass, not a guarantee. The &lt;a href="https://github.com/newmean-ai/pii" rel="noopener noreferrer"&gt;README&lt;/a&gt; has the full table against 8 other detectors and a regex baseline, and the &lt;a href="https://github.com/newmean-ai/pii/blob/main/docs/report/REPORT.md" rel="noopener noreferrer"&gt;technical report&lt;/a&gt; has the method.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm install @newmean-ai/pii
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;createRedactor&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="s2"&gt;"@newmean-ai/pii"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;redactor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;createRedactor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"medium"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;redactor&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;redact&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"My email is kim@example.com"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;// "My email is [EMAIL]"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;Demo: &lt;a href="https://pii.newmean.ai" rel="noopener noreferrer"&gt;pii.newmean.ai&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Code: &lt;a href="https://github.com/newmean-ai/pii" rel="noopener noreferrer"&gt;github.com/newmean-ai/pii&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Engine source: &lt;a href="https://github.com/newmean-ai/pii/blob/main/js/engine/pii.c" rel="noopener noreferrer"&gt;js/engine/pii.c&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I would like to know where it fails. If you speak one of the 54 languages, try names and addresses the way people really write them, and &lt;a href="https://github.com/newmean-ai/pii/issues" rel="noopener noreferrer"&gt;open an issue&lt;/a&gt; with what it missed or masked by mistake.&lt;/p&gt;

</description>
      <category>webassembly</category>
      <category>machinelearning</category>
      <category>privacy</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
