<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Diego Resendez</title>
    <description>The latest articles on DEV Community by Diego Resendez (@chrnx).</description>
    <link>https://dev.to/chrnx</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F254763%2F3f14b1a7-65c4-400e-bb63-319c17d380fe.png</url>
      <title>DEV Community: Diego Resendez</title>
      <link>https://dev.to/chrnx</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/chrnx"/>
    <language>en</language>
    <item>
      <title>I hated `req.user` enough to write a framework</title>
      <dc:creator>Diego Resendez</dc:creator>
      <pubDate>Thu, 06 Aug 2026 05:37:26 +0000</pubDate>
      <link>https://dev.to/chrnx/i-hated-requser-enough-to-write-a-framework-306e</link>
      <guid>https://dev.to/chrnx/i-hated-requser-enough-to-write-a-framework-306e</guid>
      <description>&lt;p&gt;&lt;em&gt;Or: what happens when the ground moves and you decide to chase the hare.&lt;/em&gt;&lt;/p&gt;




&lt;h2&gt;
  
  
  The line that broke me
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look at it. Nothing wrong with it. It compiles; TypeScript is happy, because three files away somebody wrote &lt;code&gt;declare global { namespace Express { interface Request { user?: User } } }&lt;/code&gt; and made &lt;code&gt;req.user&lt;/code&gt; real by decree.&lt;/p&gt;

&lt;p&gt;It's a prayer with a type annotation.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;req.user&lt;/code&gt; exists if some middleware ran before this handler. Which middleware? The one in &lt;code&gt;app.use()&lt;/code&gt; on line 41 of &lt;code&gt;server.ts&lt;/code&gt;. Does it run for &lt;em&gt;this&lt;/em&gt; route? Depends where the router got mounted. Before the body parser or after? Depends on the line number. Did that analytics package you installed last sprint call &lt;code&gt;app.use()&lt;/code&gt; at import time and land somewhere in the middle of your chain? It did. Sleep well.&lt;/p&gt;

&lt;p&gt;We have a name for the activity of figuring this out. We call it "reading the code." What we're doing is archaeology: digging down through layers of &lt;code&gt;app.use()&lt;/code&gt; to reconstruct what the request looked like by the time it reached the thing that just returned a 500.&lt;/p&gt;

&lt;p&gt;I did that for years without complaining much. Then I did it at 3am with a pager going off, because &lt;code&gt;req.user&lt;/code&gt; was &lt;code&gt;undefined&lt;/code&gt; on exactly one route out of sixty, the one where somebody had mounted the router two lines above the auth middleware instead of two lines below.&lt;/p&gt;

&lt;p&gt;The fix was moving one line. That's the part that got me. Not that it broke, but that the fix was a line number. Nothing was wrong with the code. Every function did what it said. The app was still wrong.&lt;/p&gt;

&lt;p&gt;That's when I stopped blaming myself and started blaming the model.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your API is already a graph. You just wrote it down as a list.
&lt;/h2&gt;

&lt;p&gt;Here's the thing nobody says out loud: a middleware chain is a dependency graph with the edges deleted.&lt;/p&gt;

&lt;p&gt;Think about what you actually know when you write a pipeline:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;auth &lt;strong&gt;needs&lt;/strong&gt; a database and the raw request, and &lt;strong&gt;produces&lt;/strong&gt; a user&lt;/li&gt;
&lt;li&gt;billing &lt;strong&gt;needs&lt;/strong&gt; a user, and &lt;strong&gt;produces&lt;/strong&gt; a subscription&lt;/li&gt;
&lt;li&gt;the handler &lt;strong&gt;needs&lt;/strong&gt; a subscription&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's a graph. Those are facts about your app, and they stay true no matter what order the file is in, no matter where you called &lt;code&gt;app.use()&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;And then we throw all of it away and write this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;use&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;use&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;auth&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;use&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;billing&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;We flatten the graph into a line, drop the edges, and keep the one thing that was never the point: the order we happened to type it in. Then we spend the rest of the project defending that order in code review. "Don't move this." "This has to come after that." There are comments like &lt;code&gt;// IMPORTANT: must run before cors&lt;/code&gt; holding up production apps right now, today, in codebases making real money. That comment is infrastructure. We just don't call it that.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjqfezf2pezofis67wfya.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjqfezf2pezofis67wfya.png" alt="What you wrote, next to what you meant: three app.use() calls whose order is the line you typed them on, versus a dependency graph where the order is derived from needs and provides." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The order isn't information. The order is a &lt;em&gt;consequence&lt;/em&gt;. &lt;code&gt;needs&lt;/code&gt; and &lt;code&gt;provides&lt;/code&gt; are the inputs, ordering is the output, and the algorithm has been sitting there since Kahn published it in 1962. Topological sort. It's about twelve lines.&lt;/p&gt;

&lt;p&gt;So what if you wrote down the edges and let the machine do the sorting?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="p"&gt;@&lt;/span&gt;&lt;span class="nd"&gt;Step&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;provides&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;user&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;needs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;db&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;req&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Authenticate&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;x-token&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Unauthorized&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;invalid token&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No &lt;code&gt;app.use()&lt;/code&gt;. No line number. No "put this before that." You said what it needs and what it makes, and that's the entire contract. Three things fall out of it for free.&lt;/p&gt;

&lt;p&gt;The app refuses to boot if nothing provides &lt;code&gt;user&lt;/code&gt;. Not a 500 at 3am on the one route nobody tested. A crash at startup, on your laptop, with the missing key printed by name. &lt;code&gt;req.user&lt;/code&gt; stops being a prayer, because there's no universe where the app is running and &lt;code&gt;user&lt;/code&gt; is missing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjbgnhmz1xrrtlwic7q1l.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjbgnhmz1xrrtlwic7q1l.png" alt="Terminal showing boot failed: MissingDependencyError, nothing provides 'user', required by UserController.getUser, plus the list of keys that are available." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Each route runs only its own slice. Your auth step doesn't execute on &lt;code&gt;/health&lt;/code&gt;, because that handler doesn't need a user, so it isn't in that route's subgraph. You didn't configure that anywhere. It's just what the graph says.&lt;/p&gt;

&lt;p&gt;And you can print the thing. This is the part I didn't expect to like as much as I do:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/api/users/:id&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdk8mhbpzaclsejewp95a.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fdk8mhbpzaclsejewp95a.png" alt="Output of app.explain for GET /api/users/:id, listing four ordered entries: db from a provider, req built in, the user step from Authenticate needing db and req, and the getUser route needing user and param id." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The ordered chain, every step, where each one came from. There's also &lt;code&gt;app.graph()&lt;/code&gt;, a live diagram at &lt;code&gt;GET /__graph__&lt;/code&gt;, a Mermaid export, and an OpenAPI 3.1 spec, all projected out of the same metadata. Once you've written the graph down you may as well render it four ways. (NestJS has a graph too. It's behind a paid Devtools plan. I'll leave that there.)&lt;/p&gt;

&lt;p&gt;Onboarding stops being archaeology and turns into reading, which is all I ever wanted from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bill: depth costs
&lt;/h2&gt;

&lt;p&gt;Now the part it would be dishonest to skip, since I just spent 600 words selling you the good half.&lt;/p&gt;

&lt;p&gt;A graph doesn't delete work. It orders it. Every step is still a function that runs, and steps aren't free: on my machine each one costs somewhere around 4,000 req/s, so a five-step chain lands about 22% under the zero-step number. Roughly linear, entirely unsurprising once you say it out loud.&lt;/p&gt;

&lt;p&gt;The axis that hurts is depth, not size. Width is cheap. A graph with forty steps where nothing needs anything is forty independent things, and any given route only runs the handful it depends on. Depth is the other story. If &lt;code&gt;d&lt;/code&gt; needs &lt;code&gt;c&lt;/code&gt; needs &lt;code&gt;b&lt;/code&gt; needs &lt;code&gt;a&lt;/code&gt;, that's a chain, and a chain gets walked in order because you asked for it to be. Topological sorting finds the order. It can't invent parallelism your dependencies forbid.&lt;/p&gt;

&lt;p&gt;Worth being clear, though: this isn't a green-tea tax. Those same four steps cost you the same four steps in Express. You'd just be paying without an itemized bill. The work was always there; what changes is whether you can see it.&lt;/p&gt;

&lt;p&gt;And that's the actual mitigation, not a disclaimer wearing a mitigation's clothes. &lt;code&gt;explain()&lt;/code&gt; shows you a route's depth before it turns into a latency chart you're squinting at in Grafana. Per-route slicing means a deep subgraph only bills the routes that genuinely need it, instead of every request paying for the deepest path in the app. And when a chain does get too long, "too long" becomes something you can point at and refactor, instead of a vague feeling you have about &lt;code&gt;server.ts&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;If you build a thirty-step chain where every step needs the one before it, you've built a thirty-step chain, and no framework is going to save you from that. What green-tea gives you is knowing on day one, from a &lt;code&gt;console.log&lt;/code&gt;, rather than from a postmortem.&lt;/p&gt;

&lt;h2&gt;
  
  
  "So you wrote another framework." Yeah. Let me defend that.
&lt;/h2&gt;

&lt;p&gt;I know. There's a special ring of hell for people who publish JavaScript frameworks in 2026 and I have a reserved seat. I sat on this idea for about a year before writing a line of it, mostly because I knew exactly how it sounds.&lt;/p&gt;

&lt;p&gt;My defense isn't "the existing ones are bad." I've shipped production on all three and I'd do it again tomorrow.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Express&lt;/strong&gt; made HTTP in Node feel like fifteen lines, which is why it won and why it'll outlive all of us. The price of that simplicity is &lt;code&gt;req&lt;/code&gt;: a bag anything can write to, any time, from anywhere. That's not a bug nobody got around to fixing. That &lt;em&gt;is&lt;/em&gt; the contract. Which is why Express 5 spent years in the oven and still couldn't fix &lt;code&gt;req.user&lt;/code&gt;. Nothing can, without changing what &lt;code&gt;req&lt;/code&gt; is.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fastify&lt;/strong&gt; is excellent engineering and I don't say that to be polite. Encapsulated plugins were the right instinct; scoping a plugin's blast radius beat "everyone shares one chain" by a mile. But ordering is still hook phase plus registration order, and schemas still validate at runtime. You're maintaining the order by hand. You just have much better tools for doing it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;NestJS&lt;/strong&gt; gave a whole generation of Node teams structure, and I don't think that's a small thing at all. The tradeoff is that its DI resolves tokens at runtime, so a missing provider is a startup error if you're lucky and a mystery if you're not, and every new capability shows up as its own subsystem: a WebSocket Gateway with its own adapter, a Microservices transport with its own message patterns. You end up learning "how Nest does this" three or four separate times.&lt;/p&gt;

&lt;p&gt;None of that is a failure of taste or effort. They're all downstream of one decision that was made before any of us were around to argue about it: the pipeline is a sequence, and a sequence can't tell you what it depends on.&lt;/p&gt;

&lt;p&gt;You also can't patch that from the outside. I know because I tried.&lt;/p&gt;

&lt;p&gt;Years ago I wrote &lt;a href="https://github.com/Expressive-Tea/expresive-tea" rel="noopener noreferrer"&gt;expressive-tea&lt;/a&gt;: decorators, DI through InversifyJS, boot stages, all sitting on top of Express. The sane move. Don't rewrite the world, add some structure to the world that already exists. And it worked. People used it. I used it, on real projects, happily.&lt;/p&gt;

&lt;p&gt;But when you build on someone else's chain you inherit their model, all of it. I could put decorators over the middleware chain. I could not make the middleware chain stop being a chain. I could inject dependencies at runtime. I could not make a missing dependency fail at boot, because the thing underneath was still perfectly willing to hand you a mutable bag and wish you luck. Every good idea I had ended in the same sentence: &lt;em&gt;"...but Express won't let me."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;And then the floor moved. Then it kept moving.&lt;/p&gt;

&lt;p&gt;ESM happened. Deno happened, then Bun, then edge runtimes where there's no &lt;code&gt;listen()&lt;/code&gt;, no filesystem, and your whole mental model of "a server" is quietly wrong. Web-standard &lt;code&gt;Request&lt;/code&gt;/&lt;code&gt;Response&lt;/code&gt; turned into the portable interface while nobody was announcing it. Standard Schema showed up and made "bring your own validator" a real option instead of a configuration nightmare. TC39 decorators reached Stage 3 and, in a twist I'll come back to, left parameter decorators out entirely.&lt;/p&gt;

&lt;p&gt;You can't plugin your way out of that. A plugin is a guest in someone else's house. You can move the furniture, you can't move a load-bearing wall. Eventually the honest answer was that the abstraction I wanted lived &lt;em&gt;below&lt;/em&gt; the one I was standing on, and no amount of clever decorating from up here reaches down there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Si te mueven el piso, tienes que corretear a la liebre.&lt;/strong&gt; If they move the ground under you, go chase the hare. Don't stand there decorating the spot where the floor used to be.&lt;/p&gt;

&lt;p&gt;So green-tea keeps the ideas from expressive-tea and drops the foundation. No Express, no Inversify. The graph is the core, not a coat of paint over a chain. One runtime dependency, &lt;code&gt;reflect-metadata&lt;/code&gt;, plus two optional peers you install only if you use them (&lt;code&gt;ws&lt;/code&gt; and &lt;code&gt;busboy&lt;/code&gt;). And nothing underneath assumes Node anymore, which turned out to matter more than I expected. More on that in a minute.&lt;/p&gt;

&lt;p&gt;That's the whole justification. Not "the others are bad." Just: the thing I wanted was one floor down, and you can't get there from up here.&lt;/p&gt;

&lt;h2&gt;
  
  
  Real-time, without learning a second framework
&lt;/h2&gt;

&lt;p&gt;Quick tangent, because this is where the model paid off in a way I hadn't planned for.&lt;/p&gt;

&lt;p&gt;Most stacks treat "push data over time" as a bolt-on. Express: go find a &lt;code&gt;ws&lt;/code&gt; library. Fastify: a plugin. NestJS: an entire WebSocket Gateway with its own adapter, its own decorators, its own lifecycle. A second mental model glued to your first one, with a second error surface to go with it.&lt;/p&gt;

&lt;p&gt;green-tea has one primitive, an &lt;code&gt;AsyncIterable&lt;/code&gt;. A function that produces values over time already &lt;em&gt;is&lt;/em&gt; a stream. All you declare is how it gets framed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="p"&gt;@&lt;/span&gt;&lt;span class="nd"&gt;Route&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/live&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="kd"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Live&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="p"&gt;@&lt;/span&gt;&lt;span class="nd"&gt;Sse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/prices&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                   &lt;span class="c1"&gt;// one iterable out — each yield is an event&lt;/span&gt;
  &lt;span class="nf"&gt;prices&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;btc&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;getPrice&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;})();&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="p"&gt;@&lt;/span&gt;&lt;span class="nd"&gt;Ws&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/echo&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                      &lt;span class="c1"&gt;// duplex: consume @inbound, return the outbound stream&lt;/span&gt;
  &lt;span class="nf"&gt;echo&lt;/span&gt;&lt;span class="p"&gt;(@&lt;/span&gt;&lt;span class="nd"&gt;inbound&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="nx"&gt;incoming&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;AsyncIterable&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;channel&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;await &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;incoming&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`echo: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="p"&gt;})();&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same &lt;code&gt;@Route&lt;/code&gt;, same handler shape. &lt;code&gt;@Sse&lt;/code&gt; frames it as &lt;code&gt;text/event-stream&lt;/code&gt;. &lt;code&gt;@Ws&lt;/code&gt; gives you a duplex pair. &lt;code&gt;@Stream&lt;/code&gt; negotiates off the client's &lt;code&gt;Accept&lt;/code&gt;/&lt;code&gt;Upgrade&lt;/code&gt; headers, so one handler serves SSE or ndjson or WebSocket without a single branch in your code. Backpressure, cleanup and disconnects are handled for you.&lt;/p&gt;

&lt;p&gt;The transport is also whatever you &lt;em&gt;declared&lt;/em&gt;, never whatever you happened to return. A buffered route that returns an iterable throws &lt;code&gt;TransportMismatchError&lt;/code&gt; instead of quietly turning into a stream. Refactoring the inside of a handler can't change how it talks to the wire. I'm still a bit proud of that one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same app, on Node, Deno and Bun
&lt;/h2&gt;

&lt;p&gt;This is the other half of chasing the hare, and it only worked &lt;em&gt;because&lt;/em&gt; I got off Express.&lt;/p&gt;

&lt;p&gt;Nothing in the core assumes Node. The request model is web-standard &lt;code&gt;Request&lt;/code&gt;/&lt;code&gt;Response&lt;/code&gt;, so an app is really a graph plus a &lt;code&gt;fetch&lt;/code&gt; handler, and the runtime becomes a detail you pick on the last line of the file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Node&lt;/span&gt;
&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;listen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3000&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// Deno&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;serveDeno&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@green-tea/core/deno&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nf"&gt;serveDeno&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// Bun&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;serveBun&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@green-tea/core/bun&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nf"&gt;serveBun&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// Cloudflare Workers&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;edgeHandler&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@green-tea/core/edge&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;edgeHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v86i8fsx0arhizo6c18.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5v86i8fsx0arhizo6c18.png" alt="The four entry points side by side: app.listen on Node, serveDeno on Deno, serveBun on Bun, and edgeHandler on Cloudflare Workers. HTTP, SSE, WebSocket, rooms and channels behave identically on all four." width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That's the diff. Not a port, not a fork, not an &lt;code&gt;#ifdef&lt;/code&gt;. Same modules, same steps, same controllers, one import swapped. &lt;code&gt;matcha new&lt;/code&gt; scaffolds you into any of them and &lt;code&gt;matcha run&lt;/code&gt; works out which one you're on.&lt;/p&gt;

&lt;p&gt;It's also not the usual "runs on Deno" asterisk, where HTTP works and everything interesting quietly doesn't. HTTP, SSE and WebSocket, rooms and channels included, behave the same on all four, because WebSocket support lives in a runtime-neutral core with thin adapters rather than a Node-shaped abstraction wearing a Deno hat. The suite runs separately on each one (&lt;code&gt;test:deno&lt;/code&gt;, &lt;code&gt;test:bun&lt;/code&gt;, &lt;code&gt;test:edge&lt;/code&gt;) specifically so "the same" stays a fact instead of a hope.&lt;/p&gt;

&lt;p&gt;Why I care about this beyond the demo value: the graph model isn't a bet on one runtime's future. JavaScript's floor has moved three times in five years and it isn't finished moving. If Bun wins, fine. If Deno wins, fine. If it all ends up at the edge, mostly fine, and I mean &lt;em&gt;mostly&lt;/em&gt;, so here's the fine print rather than letting you find it yourself: Workers have no &lt;code&gt;listen()&lt;/code&gt; and no filesystem, so file-mode &lt;code&gt;@Html&lt;/code&gt; and static serving are out, mesh doesn't run there at all, and you'll need the &lt;code&gt;nodejs_compat&lt;/code&gt; flag. Node 18+, Deno and Bun run everything.&lt;/p&gt;

&lt;p&gt;I'd rather tell you which door is locked than let you discover it during a deploy.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it doesn't have
&lt;/h2&gt;

&lt;p&gt;Better I say this than someone in the comments.&lt;/p&gt;

&lt;p&gt;It's beta, heading for an RC. Express has a decade of ecosystem behind it and NestJS has enterprise tooling plus a plugin catalog you could get lost in for a week. Pick green-tea for the model and the ergonomics, not for the ecosystem. Not yet.&lt;/p&gt;

&lt;p&gt;You bring your own auth. It ships transport security (TLS and wss, secure-by-default headers, CORS, size caps, path-traversal guards) but no authentication, authorization, rate limiting, CSRF or sessions. You compose those as steps, which is sort of the point, but there's nothing off the shelf the way there is on Express. If that's a blocker for you today, it's a blocker, and I'd rather you know now.&lt;/p&gt;

&lt;p&gt;mesh is alpha and I mean alpha. Distributed DI does work, and it's the piece I had the most fun building, which is exactly why I don't trust myself about it: &lt;code&gt;@needs('billing')&lt;/code&gt; resolves the same whether &lt;code&gt;billing&lt;/code&gt; is in this process or on another node, no gRPC layer, no message-pattern DSL to learn. But discovery, load balancing and failover aren't built, and the wire protocol may still change. It's gated behind &lt;code&gt;experimental: true&lt;/code&gt; and &lt;code&gt;createApp&lt;/code&gt; throws if you forget the flag. Don't put it in production.&lt;/p&gt;

&lt;p&gt;Legacy decorators, so you'll be setting &lt;code&gt;experimentalDecorators: true&lt;/code&gt;. This one is a decision rather than inertia, and I still get asked about it weekly. The whole argument-injection API (&lt;code&gt;@param&lt;/code&gt;, &lt;code&gt;@query&lt;/code&gt;, &lt;code&gt;@body&lt;/code&gt;, &lt;code&gt;@needs&lt;/code&gt;, &lt;code&gt;@inbound&lt;/code&gt;) runs on parameter decorators, and the TC39 Stage 3 proposal deliberately leaves parameter decorators out. There is no standards-track way to write &lt;code&gt;handler(@param('id') id: string)&lt;/code&gt; today. Stage 3 also means not finalized. If a viable standard path shows up, I'll take it.&lt;/p&gt;

&lt;p&gt;Route matching is a linear scan. Fine for normal route tables, and a radix tree is post-beta work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm &lt;span class="nb"&gt;install&lt;/span&gt; @green-tea/core@beta reflect-metadata
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The only thing I'd actually ask you to do: wire up two steps, then run&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;explain&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;/your/route&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If seeing your own request printed out as an ordered chain, with origins, with nothing hidden, doesn't do anything for you, then this isn't your framework and that's a completely fine outcome. Express will be here forever and it's a good tool.&lt;/p&gt;

&lt;p&gt;But if you've ever run &lt;code&gt;grep -rn "app.use" src/&lt;/code&gt; at 3am trying to work out why &lt;code&gt;req.user&lt;/code&gt; was &lt;code&gt;undefined&lt;/code&gt;, I wrote this one for you.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Docs: &lt;a href="https://green-tea.expressive-tea.io/docs" rel="noopener noreferrer"&gt;green-tea.expressive-tea.io/docs&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CLI: &lt;a href="https://github.com/Expressive-Tea/matcha" rel="noopener noreferrer"&gt;matcha&lt;/a&gt;. &lt;code&gt;matcha new&lt;/code&gt; scaffolds Node, Deno or Bun. Standalone Rust binary, no JS runtime needed to install it.&lt;/li&gt;
&lt;li&gt;Benchmarks, with every caveat spelled out and &lt;code&gt;npm run bench&lt;/code&gt; if you'd rather redo them yourself: &lt;code&gt;BENCHMARKS.md&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Less to hold in your head. That's the tea. 🍵&lt;/p&gt;

</description>
      <category>node</category>
      <category>typescript</category>
      <category>webdev</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Processing 1M Chess Games in 15 Seconds with Rust</title>
      <dc:creator>Diego Resendez</dc:creator>
      <pubDate>Tue, 31 Mar 2026 04:24:01 +0000</pubDate>
      <link>https://dev.to/chrnx/processing-1m-chess-games-in-15-seconds-with-rust-pe3</link>
      <guid>https://dev.to/chrnx/processing-1m-chess-games-in-15-seconds-with-rust-pe3</guid>
      <description>&lt;p&gt;I train self-supervised models on chess game data. My Python pipeline using python-chess took 25 minutes to parse and tokenize 1M games from Lichess PGN dumps. I rewrote it in Rust. It now takes 15 seconds.&lt;/p&gt;

&lt;p&gt;This post covers the architecture, why Rust was the right choice, and what I learned.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;Training a chess move predictor requires converting PGN (Portable Game Notation) files into tokenized sequences — arrays of integer IDs that a neural network can consume. A typical Lichess monthly dump has 5M+ games in a zstd-compressed PGN file.&lt;/p&gt;

&lt;p&gt;My Python pipeline had three bottlenecks:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;PGN parsing&lt;/strong&gt; — python-chess parses SAN notation, validates moves on a board, handles edge cases. Correct, but slow. ~15 minutes for 1M games.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tokenization&lt;/strong&gt; — converting validated UCI moves to token IDs, tracking piece types and turns. ~10 minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Memory&lt;/strong&gt; — all games loaded into a Python list of dicts. 1M games = ~4GB RAM.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The Rust rewrite
&lt;/h2&gt;

&lt;p&gt;The tool is called &lt;a href="https://github.com/Ailed-AI/ailed-soulsteal" rel="noopener noreferrer"&gt;ailed-soulsteal&lt;/a&gt; (named after a Castlevania ability — the project has a theme).&lt;/p&gt;

&lt;h3&gt;
  
  
  Architecture
&lt;/h3&gt;

&lt;p&gt;Three layers, each a clean boundary:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Input Layer → Filter Layer → Output Layer
(PGN parser)   (ELO, result)  (.somabin binary)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything streams — games are parsed, filtered, tokenized, and written one at a time. Memory usage stays constant regardless of input size.&lt;/p&gt;

&lt;h3&gt;
  
  
  PGN parsing
&lt;/h3&gt;

&lt;p&gt;PGN is a messy format. Tags, comments, variations, NAGs, move numbers, results — all interleaved. I wrote a simple streaming parser that yields one &lt;code&gt;RawGame&lt;/code&gt; at a time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="n"&gt;PgnIterator&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;line_buf&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;impl&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;BufRead&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;Iterator&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;PgnIterator&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;R&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;Item&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;RawGame&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;next&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt; &lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;Option&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;RawGame&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="c1"&gt;// Read tags until blank line, then movetext until next blank line&lt;/span&gt;
        &lt;span class="c1"&gt;// Strip comments, NAGs, variations, move numbers&lt;/span&gt;
        &lt;span class="c1"&gt;// Return (tags, moves, result)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No allocations per game beyond the reused line buffer. The &lt;code&gt;RawGame&lt;/code&gt; struct holds tags as a &lt;code&gt;HashMap&amp;lt;String, String&amp;gt;&lt;/code&gt; and moves as &lt;code&gt;Vec&amp;lt;String&amp;gt;&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Move validation with shakmaty
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://crates.io/crates/shakmaty" rel="noopener noreferrer"&gt;shakmaty&lt;/a&gt; is a pure Rust chess library. It handles SAN parsing, move validation, and piece type lookup — the same things python-chess does, but at native speed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;san&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nn"&gt;shakmaty&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nn"&gt;san&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;San&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;san_str&lt;/span&gt;&lt;span class="nf"&gt;.parse&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="nf"&gt;.ok&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;san&lt;/span&gt;&lt;span class="nf"&gt;.to_move&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="nf"&gt;.ok&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;uci&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;uci_string&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;category&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;role_to_category&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="nf"&gt;.role&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
&lt;span class="n"&gt;pos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="nf"&gt;.play&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="nf"&gt;.ok&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is where most of the speedup comes from. shakmaty's &lt;code&gt;play()&lt;/code&gt; is essentially a few bitboard operations — no Python overhead, no GC pressure.&lt;/p&gt;

&lt;h3&gt;
  
  
  Binary output format
&lt;/h3&gt;

&lt;p&gt;Instead of writing JSON or CSV, I designed a binary format (&lt;code&gt;.somabin&lt;/code&gt;) optimized for ML training:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Header (64 bytes): magic, version, vocab_size, num_games, ...
Index Table:       byte offset for each game (enables random access)
Data Section:      per game: [seq_len, token_ids, turn_ids, category_ids, outcome]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The index table is the key insight. A PyTorch &lt;code&gt;Dataset.__getitem__(i)&lt;/code&gt; can seek directly to game &lt;code&gt;i&lt;/code&gt; via mmap without scanning the file. Loading 50K games takes 20ms. Random access runs at 500K games/sec.&lt;/p&gt;

&lt;p&gt;The Python reader is 30 lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;SomabinDataset&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_file&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_mm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mmap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;mmap&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fileno&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;access&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;mmap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ACCESS_READ&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;read_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_mm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_index&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;read_index&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_mm&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__getitem__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;offset&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_index&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;read_game&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_mm&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;offset&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Zstd decompression
&lt;/h3&gt;

&lt;p&gt;Lichess distributes PGN files as &lt;code&gt;.pgn.zst&lt;/code&gt;. The &lt;code&gt;zstd&lt;/code&gt; crate handles decompression transparently:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="nf"&gt;.extension&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="nf"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"zst"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;let&lt;/span&gt; &lt;span class="n"&gt;decoder&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nn"&gt;zstd&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nn"&gt;Decoder&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;?&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nf"&gt;Ok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;BufReader&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;with_capacity&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;decoder&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;Ok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;BufReader&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;with_capacity&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Auto-detected from the file extension. No separate decompression step needed.&lt;/p&gt;

&lt;h3&gt;
  
  
  Filtering
&lt;/h3&gt;

&lt;p&gt;Games are filtered by metadata before tokenization — no wasted work:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight rust"&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt; &lt;span class="k"&gt;trait&lt;/span&gt; &lt;span class="n"&gt;GameFilter&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;fn&lt;/span&gt; &lt;span class="nf"&gt;accept&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;game&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;RawGame&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// Chain of filters — all must pass&lt;/span&gt;
&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="nf"&gt;.add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;EloFilter&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1800&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="nf"&gt;.add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;MovesFilter&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nb"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
&lt;span class="n"&gt;filters&lt;/span&gt;&lt;span class="nf"&gt;.add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nn"&gt;ResultFilter&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Decisive&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Filters operate on PGN tags (strings), not board positions. Checking &lt;code&gt;WhiteElo &amp;gt;= "1000"&lt;/code&gt; is effectively free compared to move validation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Benchmarks
&lt;/h2&gt;

&lt;p&gt;Processing Lichess monthly dumps (zstd compressed) on an M1 MacBook:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Month&lt;/th&gt;
&lt;th&gt;Input&lt;/th&gt;
&lt;th&gt;Games (1000-1800)&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Rate&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;2016-01&lt;/td&gt;
&lt;td&gt;831 MB .zst&lt;/td&gt;
&lt;td&gt;2,060,197&lt;/td&gt;
&lt;td&gt;45s&lt;/td&gt;
&lt;td&gt;46K/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2016-02&lt;/td&gt;
&lt;td&gt;866 MB .zst&lt;/td&gt;
&lt;td&gt;2,071,332&lt;/td&gt;
&lt;td&gt;46s&lt;/td&gt;
&lt;td&gt;45K/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2016-03&lt;/td&gt;
&lt;td&gt;994 MB .zst&lt;/td&gt;
&lt;td&gt;2,399,234&lt;/td&gt;
&lt;td&gt;54s&lt;/td&gt;
&lt;td&gt;45K/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2016-04&lt;/td&gt;
&lt;td&gt;1.0 GB .zst&lt;/td&gt;
&lt;td&gt;2,438,621&lt;/td&gt;
&lt;td&gt;55s&lt;/td&gt;
&lt;td&gt;44K/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2016-07&lt;/td&gt;
&lt;td&gt;1.0 GB .zst&lt;/td&gt;
&lt;td&gt;2,598,733&lt;/td&gt;
&lt;td&gt;59s&lt;/td&gt;
&lt;td&gt;44K/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;11.6M games in 4.3 minutes. The equivalent Python pipeline would take roughly 5 hours.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Streaming wins.&lt;/strong&gt; The biggest architectural decision was making everything an iterator. Games flow through parse → filter → tokenize → write without buffering. Memory usage is constant at ~10MB regardless of input size.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Binary formats beat JSON for ML.&lt;/strong&gt; My first version wrote JSONL. A 1M-game JSONL file was 2GB and took 30 seconds to load in Python. The &lt;code&gt;.somabin&lt;/code&gt; binary for the same data is 550MB and loads in 20ms via mmap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;shakmaty is excellent.&lt;/strong&gt; Chess move validation is the bottleneck in any PGN pipeline. shakmaty's bitboard implementation made this a non-issue. The crate is well-documented and the API maps cleanly to what you need for tokenization.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rust's type system caught real bugs.&lt;/strong&gt; The &lt;code&gt;GameParser&lt;/code&gt; and &lt;code&gt;GameTokenizer&lt;/code&gt; traits enforce separation between parsing (text → structured data) and tokenization (structured data → integers). When I mixed them up during development, the compiler told me immediately.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cargo &lt;span class="nb"&gt;install &lt;/span&gt;ailed-soulsteal

&lt;span class="c"&gt;# Generate vocabulary&lt;/span&gt;
soulsteal vocab &lt;span class="nt"&gt;--generate&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; vocab.json

&lt;span class="c"&gt;# Tokenize a Lichess dump&lt;/span&gt;
soulsteal tokenize lichess_2016-02.pgn.zst &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-o&lt;/span&gt; train.somabin &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--vocab&lt;/span&gt; vocab.json &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;--elo&lt;/span&gt; 1000:1800

&lt;span class="c"&gt;# Inspect&lt;/span&gt;
soulsteal info train.somabin
soulsteal stats train.somabin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pre-tokenized datasets are available on &lt;a href="https://huggingface.co/datasets/Ailed-AI/lichess-chess-1000-1800" rel="noopener noreferrer"&gt;Hugging Face&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The tool is designed to support any turn-based game — Go (SGF), Shogi (KIF), etc. Chess is the v1 implementation, but the &lt;code&gt;GameParser&lt;/code&gt; and &lt;code&gt;GameTokenizer&lt;/code&gt; traits are game-agnostic.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;This article was drafted with assistance from Claude (Anthropic), an AI language model. The code, benchmarks, and technical decisions described are entirely my own work.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Resources:&lt;/strong&gt;&lt;br&gt;
Source: &lt;a href="https://github.com/Ailed-AI/ailed-soulsteal" rel="noopener noreferrer"&gt;github.com/Ailed-AI/ailed-soulsteal&lt;/a&gt;&lt;br&gt;
crates.io: &lt;a href="https://crates.io/crates/ailed-soulsteal" rel="noopener noreferrer"&gt;ailed-soulsteal&lt;/a&gt;&lt;br&gt;
License: MIT&lt;/p&gt;

</description>
      <category>rust</category>
      <category>chess</category>
      <category>machinelearning</category>
      <category>gamedev</category>
    </item>
    <item>
      <title>Introducing Expressive Tea</title>
      <dc:creator>Diego Resendez</dc:creator>
      <pubDate>Wed, 26 May 2021 22:52:46 +0000</pubDate>
      <link>https://dev.to/expressive-tea/introducing-expressive-tea-26k7</link>
      <guid>https://dev.to/expressive-tea/introducing-expressive-tea-26k7</guid>
      <description>&lt;h2&gt;
  
  
  Introduction
&lt;/h2&gt;

&lt;p&gt;When I started coding &lt;a href="https://expressive-tea.io"&gt;Expressive Tea&lt;/a&gt; almost two years ago, it was just a tool to increase my laziness (as most engineers) avoid setting up a project on &lt;a href="https://expressjs.com"&gt;ExpressJS&lt;/a&gt;. I was thinking about combining a language that I started to learn (Typescript) and making easier a configuration for &lt;strong&gt;ExpressJS&lt;/strong&gt;; by that moment, I did not know about &lt;a href="https://nestjs.com"&gt;NestJS&lt;/a&gt;, a very mature and fantastic Framework. I did not have a chance to research, just because I got the idea to start working in this new (at that moment) personal library.&lt;/p&gt;

&lt;p&gt;In September 2019, that personal library turned into a microframework to help build &lt;strong&gt;server-side&lt;/strong&gt; applications in &lt;strong&gt;NodeJS&lt;/strong&gt; and use modern &lt;strong&gt;Javascript&lt;/strong&gt; powered by &lt;strong&gt;Typescript&lt;/strong&gt;. Also, one of the main goals is to make this compatible with all the express middleware and plugins available when I wrote this article; the list continues to expand with ideas and recommendations from friends and users that already started using Expressive Tea.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the heck is Expressive Tea?
&lt;/h2&gt;

&lt;p&gt;I define Expressive Tea as a flexible framework that gives freedom to build their own architectures by providing descriptive decorators, a plugin engine, shareable modules, and modern Javascript.&lt;/p&gt;

&lt;p&gt;But what does that mean? As I mentioned in the introduction, it is just making developers lazier (for a good reason), helping them forget how to set up ExpressJS, and just working on the business logic. &lt;/p&gt;

&lt;p&gt;Take into consideration that Expressive Tea has not come with anything out of the box; this means no additional plugins or middlewares with exceptions of certain features like &lt;strong&gt;Websockets&lt;/strong&gt;, and you might need to provide the necessary plugins to achieve your unique server flavor.&lt;/p&gt;

&lt;p&gt;So, telling that, you might still wonder where the flexibility or freedom is in something with nothing more than essential matters. Pleasantly, Expressive Tea has no attachment on any naming convention, any special plugin, or any data source, so this allows you to freely change the application in the way you prefer and introduce two critical entities in the Expressive Tea framework.&lt;/p&gt;

&lt;h3&gt;
  
  
  Plugins
&lt;/h3&gt;

&lt;p&gt;A Plugin is an entity to implement features that we need for our applications, such as assigning a connection to MongoDB or changing the server behavior such as adding a view engine, adding authorization middleware, or just adding a body parser.&lt;/p&gt;

&lt;p&gt;But more importantly, this can be share between Expressive Tea projects; the method depends on you (like npm package, git submodule, or copy and paste). Still, essentially you will get a path to create many plugins and combine them to flavor on your own Expressive Tea.&lt;/p&gt;

&lt;h3&gt;
  
  
  Modules
&lt;/h3&gt;

&lt;p&gt;A Module is pretty similar to Plugin, but instead of change the behavior of the server is used to create a placeholder route to encapsulate controllers and services to respond to a user request through declared endpoints. &lt;/p&gt;

&lt;p&gt;All the endpoints are defined in each ** controller** that have a corresponding method to respond to the client; in simple terms, a Module helps to contain routers in one place, by example, we can have a signing module that contains all the endpoints to sing in or sign up a user into our application.&lt;/p&gt;

&lt;p&gt;As same as Plugins, Modules can be share between Expressive Tea applications if they have the following specifications:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;All code must be in the same place; anyone has their own structure and scaffolding methods, ideally set all in the same root directory; the other thing is depends on you ;).&lt;/li&gt;
&lt;li&gt;Should not contain external code, like external classes, services, or constants with certain exceptions like Expressive Tea settings and node packages.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;We have enough for boring theory; it might be better with a bit of demonstration of how this is working, shall we?..&lt;/p&gt;

&lt;h3&gt;
  
  
  Dependencies.
&lt;/h3&gt;

&lt;p&gt;We can start by installing &lt;code&gt;tea&lt;/code&gt; CLI with the following command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-g&lt;/span&gt; @expressive-tea/tea
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;blockquote&gt;
&lt;p&gt;Tea CLI is under development; take patience ;), send comments or open an issue.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h3&gt;
  
  
  Create a project.
&lt;/h3&gt;

&lt;p&gt;Once &lt;code&gt;tea&lt;/code&gt; finished to install, now is time to start to create a project with the following command:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tea brew to-do
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="ltag_asciinema"&gt;
  
&lt;/div&gt;



&lt;p&gt;Follow all the instructions and change them if you want them. Once create project finished and create our &lt;code&gt;to-do&lt;/code&gt; project, just go inside with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;cd &lt;/span&gt;to-do
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;and execute the project with:&lt;br&gt;
&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;tea serve
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="ltag_asciinema"&gt;
  
&lt;/div&gt;



&lt;p&gt;if you get the following message, congratulations; this is your first Expressive Tea application:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[INFO] 14:08:05 ts-node-dev ver. 1.1.6 (using ts-node ver. 9.1.1, typescript ver. 3.9.9)
Fri, 21 May 2021 19:08:06 GMT helmet deprecated helmet.noCache is deprecated and will be removed in helmet@4. You can use the `nocache` module instead. For more, see https://github.com/helmetjs/helmet/issues/215. at server/plugins/express.ts:59:31
Running HTTP Server on [8080]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;if you open your browser and type &lt;code&gt;http://localhost:8080/&lt;/code&gt;&lt;br&gt;
you will get the following response:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://res.cloudinary.com/practicaldev/image/fetch/s--mQ_ca5TK--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vz6vxaru0kfl9hfvyxzn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--mQ_ca5TK--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vz6vxaru0kfl9hfvyxzn.png" alt="Alt Text"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h3&gt;
  
  
  time to play
&lt;/h3&gt;

&lt;p&gt;Let the fun start :), it is time to move our a little bit our project; the main goal is to create a REST API for a To-Do application, and the first thing to do is remove the static HTML that shows as the index.&lt;/p&gt;

&lt;p&gt;It is time to open the &lt;code&gt;main.ts&lt;/code&gt; file from the application and remove the line &lt;code&gt;@Static('./public')&lt;/code&gt; (you can use the IDE that you like) as is the following screencast.&lt;/p&gt;


&lt;div class="ltag_asciinema"&gt;
  
&lt;/div&gt;



&lt;p&gt;Now the main page is gone is time to implement our first endpoint to this new REST API.&lt;/p&gt;


&lt;div class="ltag_asciinema"&gt;
  
&lt;/div&gt;


&lt;p&gt;The application now will return a To-Do list that is on memory, for now, just to not overcomplicate the example.&lt;/p&gt;


&lt;div class="ltag_asciinema"&gt;
  
&lt;/div&gt;


&lt;p&gt;For this example need to add one of the elements from Expressive Tea for decorating arguments in the endpoints methods; the &lt;code&gt;@body()&lt;/code&gt; annotation is part of the package on &lt;code&gt;@zerooneit/expressive-tea/decorators/annotations&lt;/code&gt; and helps to extract the whole body object as is showing in the screencast above.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;More details you will get in the documentation.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  The Extra
&lt;/h3&gt;

&lt;p&gt;The main goal is to present you a little bit of Expressive Tea, but how to leave without giving you an extra; in this case, I will show you how to modify the Express plugin in the demo to just show a console log.&lt;/p&gt;


&lt;div class="ltag_asciinema"&gt;
  
&lt;/div&gt;


&lt;h3&gt;
  
  
  Check our live playground.
&lt;/h3&gt;

&lt;p&gt;I know this is not extensive details in this article, and this is because I just want to inform you more than training you, but if you're going to sneak pick a little more content, there is a series of articles that you can follow, it might be a little old but contains the essential:&lt;/p&gt;


&lt;div class="ltag__link"&gt;
  &lt;a href="https://medium.com/@diego.resendez/a-simple-rest-api-with-expressive-tea-part-three-6b09f2ad73ad" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__pic"&gt;
      &lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--QplbVQgo--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://miro.medium.com/fit/c/56/56/2%2AtQQDZttulH9Uzvzs5M5-JA.jpeg" alt="Diego Resendez"&gt;
    &lt;/div&gt;
  &lt;/a&gt;
  &lt;a href="https://medium.com/@diego.resendez/a-simple-rest-api-with-expressive-tea-part-three-6b09f2ad73ad" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__content"&gt;
      &lt;h2&gt;A Simple REST API with Expressive Tea | Part Three | by Diego Resendez | Medium&lt;/h2&gt;
      &lt;h3&gt;Diego Resendez ・ &lt;time&gt;Oct 3, 2019&lt;/time&gt; ・ 
      &lt;div class="ltag__link__servicename"&gt;
        &lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--ze5yh_2q--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/medium_icon-90d5232a5da2369849f285fa499c8005e750a788fdbf34f5844d5f2201aae736.svg" alt="Medium Logo"&gt;
        Medium
      &lt;/div&gt;
    &lt;/h3&gt;
&lt;/div&gt;
  &lt;/a&gt;
&lt;/div&gt;



&lt;div class="ltag__link"&gt;
  &lt;a href="https://medium.com/@diego.resendez/a-simple-rest-api-with-expressive-tea-part-two-718b1e3ed265" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__pic"&gt;
      &lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--QplbVQgo--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://miro.medium.com/fit/c/56/56/2%2AtQQDZttulH9Uzvzs5M5-JA.jpeg" alt="Diego Resendez"&gt;
    &lt;/div&gt;
  &lt;/a&gt;
  &lt;a href="https://medium.com/@diego.resendez/a-simple-rest-api-with-expressive-tea-part-two-718b1e3ed265" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__content"&gt;
      &lt;h2&gt;A Simple REST API with Expressive Tea | Part Two | by Diego Resendez | Medium&lt;/h2&gt;
      &lt;h3&gt;Diego Resendez ・ &lt;time&gt;Oct 8, 2019&lt;/time&gt; ・ 
      &lt;div class="ltag__link__servicename"&gt;
        &lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--ze5yh_2q--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/medium_icon-90d5232a5da2369849f285fa499c8005e750a788fdbf34f5844d5f2201aae736.svg" alt="Medium Logo"&gt;
        Medium
      &lt;/div&gt;
    &lt;/h3&gt;
&lt;/div&gt;
  &lt;/a&gt;
&lt;/div&gt;



&lt;div class="ltag__link"&gt;
  &lt;a href="https://medium.com/@diego.resendez/a-simple-rest-api-with-expressive-tea-part-one-97ef6c08f9c4" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__pic"&gt;
      &lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--QplbVQgo--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://miro.medium.com/fit/c/56/56/2%2AtQQDZttulH9Uzvzs5M5-JA.jpeg" alt="Diego Resendez"&gt;
    &lt;/div&gt;
  &lt;/a&gt;
  &lt;a href="https://medium.com/@diego.resendez/a-simple-rest-api-with-expressive-tea-part-one-97ef6c08f9c4" class="ltag__link__link"&gt;
    &lt;div class="ltag__link__content"&gt;
      &lt;h2&gt;A Simple REST API with Expressive Tea | Part One. | by Diego Resendez | Medium&lt;/h2&gt;
      &lt;h3&gt;Diego Resendez ・ &lt;time&gt;Sep 23, 2019&lt;/time&gt; ・ 
      &lt;div class="ltag__link__servicename"&gt;
        &lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--ze5yh_2q--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/medium_icon-90d5232a5da2369849f285fa499c8005e750a788fdbf34f5844d5f2201aae736.svg" alt="Medium Logo"&gt;
        Medium
      &lt;/div&gt;
    &lt;/h3&gt;
&lt;/div&gt;
  &lt;/a&gt;
&lt;/div&gt;


&lt;p&gt;Or you can use our sandbox in CodeSandbox, where you will be able to play a little bit or understand more about Expressive Tea.&lt;/p&gt;

&lt;p&gt;&lt;iframe src="https://codesandbox.io/embed/expressive-tea-2kmg7?runonclick=1"&gt;
&lt;/iframe&gt;
&lt;/p&gt;

&lt;h2&gt;
  
  
  Conclusions
&lt;/h2&gt;

&lt;p&gt;As I continue improving this project, I learned and still learn many things technically. Still, the real value is discovering new things to do, challenges, and, more importantly, being active in the Open Source community. Even if this project does not grow much or maybe overextend it, the self-learning of many soft skills comes with this, for example, managing your time, executing a project, and actively checking new features.&lt;/p&gt;

&lt;p&gt;Hence, as this work I just take it as a personal challenge it will be good to have help, so if any of you want to help, improving code, adding new issues, do a review, write an article, maybe a YouTube video, or ask for new feature down below you will get the link to the Github Repository of this project.&lt;/p&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://res.cloudinary.com/practicaldev/image/fetch/s--i3JOwpme--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev.to/assets/github-logo-ba8488d21cd8ee1fee097b8410db9deaa41d0ca30b004c0c63de0a479114156f.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/Expressive-Tea"&gt;
        Expressive-Tea
      &lt;/a&gt; / &lt;a href="https://github.com/Expressive-Tea/expresive-tea"&gt;
        expresive-tea
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      A Express and Typescript REST Service Template.
    &lt;/h3&gt;
  &lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;Without more to aggregate, let me thank you for reading my article, which is the first on this platform.&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>typescript</category>
      <category>node</category>
      <category>express</category>
    </item>
  </channel>
</rss>
