DEV Community: HasanH47

Our 72-hour soak lost 81% of its throughput. We had to prove it wasn't the runtime.

HasanH47 — Mon, 28 Sep 2026 12:42:07 +0000

Evidence box: every Usai post states the limits of its claims.

Status: Usai v0.0.15, alpha. The soaks ran on 2026-09-19/22 and 2026-09-23/24, on the builds of those days.

Setup: one VM (16 vCPU Xeon) running a production-shaped deployment: Caddy in front of the published Usai runtime image, PostgreSQL 18, and the examples/invoicing app. Eight closed-loop clients; a status sample once a minute. Other work on the same host is named in the post.

Supports: over 73.6 M and 124.5 M requests, no 5xx and no 503 from the runtime, with memory, descriptors and ownership counters flat; the 72-hour decay came from the application's unpruned table; the runtime's cost per request stayed flat while throughput fell.

Does not support: behaviour under saturation, on other hardware, across several hosts, or for other applications; a general claim about PostgreSQL performance; that Usai is ready for every production workload.

Source data: 72-hour soak · 24-hour control run · idle decommit

Our three-day soak served 73,627,701 requests without a single 5xx, and still lost 81% of its throughput along the way. A second, 24-hour run with the application's data pruned showed no slowdown at all. The cause was one ever-growing table, not the runtime. This post is how we got from the first observation to the second, and why we didn't stop at the easy answer.

72-hour soak     hour 0–6: 794 requests/second    hour 66–72: 154 requests/second
24-hour control  hour 1: 1,416 requests/second    hour 24: 1,480 requests/second

The setup

A soak test runs steady load for a long time. It catches what short benchmarks miss: slow leaks, gradual drift, and things that only break on the millionth request.

Ours ran against a production-shaped deployment on one VM. Caddy sat in front of the published Usai runtime image, which talked to PostgreSQL 18 and ran the examples/invoicing application from the repository. Eight closed-loop clients repeated one cycle: list invoices, fetch one, create one. About 10% of requests wrote a new invoice. Once a minute, a sampler recorded memory, CPU, open file descriptors and the runtime's own status page.

Three days, one falling line

By every correctness criterion we wrote down beforehand, the 72-hour soak passed. There was no 5xx and no 503, and memory stayed flat. Throughput was another story:

Window	Requests/second	Median latency (p50)	Slow latency (p99)
0–6 h	794	4.6 ms	84 ms
12–18 h	349	5.1 ms	289 ms
24–30 h	244	5.8 ms	450 ms
48–54 h	181	5.4 ms	666 ms
66–72 h	154	5.3 ms	810 ms

The median barely moved. The slow tail grew tenfold, and throughput fell to a fifth.

Meanwhile, everything the runtime owns stayed flat:

CPU inside the application's world went from 1.56 to 1.59 ms per request.
CPU for the whole process went from 2.80 to 3.04 ms per request.
SQL operations went from 2.52 to 2.59 per request.
Memory charged to the container went from 51.5 to 53.5 MiB, and open descriptors from 29 to 30.
The runtime created 73,627,744 execution worlds, detected no detached work, quarantined no database connection, and logged no WARN or ERROR line in three days.

The easy explanation, and the other one

The workload wrote about 7.4 million invoices, with their line items, into one table that nobody pruned. More rows mean bigger indexes, more write-ahead log, and more checkpoint and autovacuum work. The application waits longer for PostgreSQL, so a closed-loop client completes fewer requests per second. Flat CPU per request fits that story.

It also fits a less comfortable one. A runtime leaking something outside the CPU would draw the same curve. Think of a structure that keeps growing and gets walked once per request, a connection pool that slowly degrades, or a table of handles nobody frees. CPU per request stays flat, memory can stay flat if the growth is small, and throughput still falls.

Both stories predicted exactly what we saw. Picking the convenient one would have been a guess, written up as a finding.

The control run

So we ran it again: the same deployment, the same eight clients and the same load, for 24 hours. The only change was that the harness deleted what the load created, the way an operator would. Every minute it removed invoices beyond the newest 20,000, and finished queue rows older than a minute.

If the data caused the decay, the decay would disappear. If the runtime caused it, the decay would still be there.

It disappeared. Throughput was 1,416 requests per second in the first hour and 1,480 in the last, roughly twice the rate of the 72-hour run's opening hours. Over 124,471,717 successful requests:


Server errors	0 × 5xx, 0 × 503 (and 4 × 4xx)
Mean latency	4.39 ms across the whole run
Lifecycle	0 detached work, 0 quarantined connections, 0 late or stale completions
Memory charged to the container	53.2 to 55.2 MiB
Open descriptors	29 to 31

The decay was the application's table, not the runtime. A day of machine time let us say that as a measurement rather than an opinion.

The bad seconds

The control run was not perfect. The load client recorded 344 timeouts, spread over 91 seconds. None came from the server, which answered no 5xx and no 503 in those seconds. All of them fell in two windows:

In hour 1, another job on the same host was building a Docker image. That explains 12 timeouts in 28 seconds.
In hours 16 to 18, other work on the host cut the container's CPU share from about 415% to 65% for a few minutes. A closed-loop client that gets fewer cycles sends fewer requests, and eventually hits its own 10-second timeout.

Outside those two windows, over 22 hours and 114 million requests, not one second produced an error. We report the timeouts anyway. A soak that hides its bad seconds makes readers wonder what else it hides.

Two memory numbers, 36 MiB apart

The control run also taught us something about our instrument. Midway through, the runtime's status page said the process used 89 MiB, while docker stats said 53 MiB. Both were correct.

The process figure, VmRSS, counts a shared page once for every place it is mapped. Usai maps one prepared application image into each of its eight pooled world slots, so about 13 MiB of real pages showed up as 36 MiB.

The container figure, the cgroup charge, is what decides whether a container gets killed. It under-counts the runtime's own binary. Linux charges a file's pages to whichever container read them first, and something else had read that image layer before.

The honest middle is PSS, which splits shared pages between the processes using them: about 66 MiB here. The soak summary now prints all three numbers. The result doesn't change, because each run used the same instrument from start to end. But a memory figure from one tool should never be compared with a figure from another.

What the same campaign caught earlier

A soak that passes is only half the story. The same campaign found and fixed real problems before the long runs.

The first was about replacing the running application. Doing it 1,000 times in a row killed the process after about 270 replacements, as memory grew about 2 MB each time until it hit the container's 512 MB limit. It wasn't a leak in the runtime, since the count of compiled images stayed at two. The cause was glibc's malloc, which keeps a separate memory arena per worker thread; each arena held on to a freed, module-sized chunk the others could not reuse. Capping the number of arenas fixed it. The final run did 1,000 replacements under load, with 2,039,500 requests and no errors.

The second is a trade-off rather than a bug: memory after a burst does not come back. A world slot that has been used keeps its pages, which buys about 16% more throughput on the next burst. A new release path added upstream in the engine reclaims only about 12 MiB and leaves the plateau where it was. The sizing guide says so, but it means your peak concurrency sets your memory, not your current load.

What this means if you run a backend

Most of this is not about Usai.

A long soak of your application is also a soak of your database. If one table takes millions of inserts, the soak will measure that table. Prune or partition it, or run a control with bounded data, before you blame the runtime.
Flat CPU per request does not prove there is no leak. It only rules out one kind. Change one variable and run again.
Know which memory number you are reading. Process RSS, PSS and the container's charge answer different questions, and only the last one decides whether the container gets killed.

Try it

npm create @sakaladev/usai@latest my-app
cd my-app && npm install && npm run dev

Everything above is on the site with the raw numbers: the benchmark page, the 72-hour soak report and the 24-hour control run. If you see a hole in the reasoning, open an issue. That is exactly what we want to hear.

Usai is an open-source, workload-native application runtime for TypeScript backends: usai.sakala.dev · github.com/gmedia/usai.

Drafted with AI assistance and reviewed by the Usai maintainers.

Why should request state survive after the request is finished?

HasanH47 — Thu, 24 Sep 2026 21:16:27 +0000

Evidence box — every Usai post states the limits of its claims.

Status: Usai v0.0.15, alpha. Production qualification is in progress; contracts may still change. The sweep was measured on 0.0.8.

Supports: a fresh execution world per unit of work (a module-level counter answers 1 on every request); on routes that touch PostgreSQL, throughput level with a single Node process and ahead of it on transactions; a measurable CPU cost for the fresh world, 6–14× Node's on a trivial route.

Does not support: that Usai is generally faster than Node, Bun, Deno, PHP or Rust; that a fresh world is a security boundary (it is semantic isolation); that Usai is ready for every production workload.

Source data: the concurrency sweep · lifecycle contracts · AWS Lambda best practices · Cloudflare Workers best practices

A request arrives.
It creates some mutable application state.
The request finishes.

Why is that state still alive?

The usual answer is simple: because the process is still alive. In most backends the process starts once, and every request after that runs inside the same long-lived application world.

process starts
├─ request A
├─ request B
├─ request C
└─ …and every mutable thing any of them left behind

That model is fast, familiar and correct for a great many programs. It also means state gets its lifetime from where it happens to be stored, not from what it is for. Usai is an attempt to separate those two things.

The bug nobody writes on purpose

Here is a handler with a variable at the top of the file:

let counter = 0; // module-level state

export const count = http.get("/count", async () => {
  counter += 1;
  return { count: counter };
});

Nobody ships a counter like this. People do ship its cousins: a currentUser set by middleware, a tenant id cached "for convenience", a memoised object that someone later mutates. They behave exactly like counter: they outlive the request that set them.

We ask every server in our benchmark suite for exactly this counter, ten times in a row. Here is what comes back:

Server	Ten requests
One Node process	`1 2 3 4 5 6 7 8 9 10`
Node cluster, 8 workers	`1 1 1 1 1 1 1 1 2 2`
Usai	`1 1 1 1 1 1 1 1 1 1`

The first row is the obvious leak. The second row is worse. Eight processes each remember something different, so the same request gets a different answer depending on which worker picked it up. A bug like that passes review, passes tests on a laptop, and then shows up in production as "cannot reproduce".

This is not a problem we invented

Platforms that reuse execution environments for speed document the same tension, and they leave the fix to you.

AWS Lambda asks you to reuse the environment for expensive things like SDK clients and database connections, and in the same breath warns:

To avoid potential data leaks across invocations, don't use the execution environment to store user data, events, or other information with security implications.

Cloudflare Workers is blunter:

Workers reuse isolates across requests. A variable set during one request is still present during the next.

Both platforms reach the same conclusion. Reusing infrastructure is essential for performance, and reusing mutable application state is dangerous. Both ask developers to hold that line by discipline.

Usai asks a different question: what if that line were part of the runtime's structure instead?

Persistent infrastructure is not persistent state

The inference we question is small:

the infrastructure is persistent
therefore
the application state is persistent

The first line is good engineering. Listeners, schedulers, connection pools and compiled code are expensive to build and safe to reuse. The second line does not follow from the first. It is simply what happens when both live in the same process.

PHP-FPM has kept these apart for a long time. Its worker processes persist, but application state is rebuilt for every request. That is why PHP-FPM answers 1 1 1 1 1 1 1 1 1 1 to the same probe, for the same reason Usai does.

Usai splits a backend into three layers, each with its own lifetime:

persistent runtime              lives for the process
  listeners · scheduler · pools · metrics

immutable application definition   built once per deploy
  compiled code · routes · schemas · config

execution world                 lives for one unit of work
  request state · auth context · globals · leases

Every request, task, cron tick and queue message gets a fresh execution world. When the work is done, the world ends. The runtime does not.

Not everything should be short-lived

A common misreading is that Usai throws everything away. It does not. Some things deserve to live long, and you say so:

Work	Declared with	The world lives for
HTTP request	`http.get` / `http.post` / …	one request
Task	`task`	one invocation
Cron	`cron`	one tick
Queue message	`queue.consume`	one message
WebSocket	`socket`	the connection
Service	`service`	while the revision is active

A PostgreSQL pool is a resource. It outlives worlds on purpose, and each world borrows a connection through a lease. A WebSocket world lives exactly as long as its connection. A service is persistent because you declared it persistent. The default is ephemeral, and persistence is always a deliberate choice.

Death is not cleanup

Throwing a world away does not make everything safe. A query can still be running on a database connection after the world that started it has ended. A timer can still be scheduled. So Usai adds a second rule: asynchronous work always has an owner.

export const order = http.post("/orders", async () => {
  setTimeout(sendEmail, 1000); // nobody owns this
  return { ok: true };
});

In a long-lived Node process this usually works, until the process restarts at the wrong moment. In Usai the runtime cancels the timer and tells you why:

WARN http work `POST /orders` ended with live asynchronous work (1 timer).

The http lifetime ended when its result was produced. Work that is
still pending cannot remain owned by this world, so it was cancelled.

Use:
  task()    for independent finite work
  cron()    for scheduled work
  service() for intentional long-running work
or await the work before returning.

The fix says what you mean: await ctx.tasks.dispatch(sendEmail, …) hands the work to a task that runs in its own world.

The same rule decides when a database connection can be reused. A connection goes back to the pool only when Usai knows how its last operation ended: it finished normally, it failed with a known SQL error, or its cancellation was confirmed. If the outcome is ambiguous, the connection is quarantined and replaced, never reused.

What it costs

A fresh world is not free, and we would rather tell you than let you find out.

On a route that does nothing, Usai spends 6–14× Node's CPU per request. That is the price of entering a fresh world, validating the boundary, and settling ownership.
On routes that touch PostgreSQL, which is what real applications are mostly made of, the difference mostly disappears. Usai runs level with a single Node process, ahead of it on transactions, and 1.8–7× ahead of Laravel. As the sweep puts it: the query is the bill; the runtime is rounding.
Memory grows with concurrency, not with request count. 64 concurrent requests cost about 0.5 GiB.

So Usai is not the choice for a hot, CPU-bound endpoint where every microsecond counts. It is for backends where you want persistent infrastructure without every piece of state quietly inheriting the lifetime of the process. That usually means mostly-idle services, multi-tenant APIs, webhooks, and apps that mix HTTP with tasks, cron, queues and sockets.

Try it, and try to break it

npm create @sakaladev/usai@latest my-app
cd my-app && npm install && npm run dev

With pnpm it is pnpm dlx @sakaladev/create-usai my-app, then pnpm install && pnpm dev. pnpm 12 installs a new release only once it is a day old, so on a release day npm is the way to get the latest.

Usai is alpha, at 0.0.15. It has survived a 72-hour soak and a campaign of deliberate failures, and it has not yet met many developers outside the project. That is the next experiment. If you try it, we want to hear where the model confused you, where it got in your way, and where it caught a bug you did not know you had.

The next posts in this series go deeper: how a world is created cheaply, what 73 million requests over three days taught us, and where Usai loses.

Usai is an open-source, workload-native application runtime for TypeScript backends. Docs, benchmarks and the research behind it: usai.sakala.dev. Source: github.com/gmedia/usai.

Drafted with AI assistance and reviewed by the Usai maintainers.

I Tried to Compress an LLM by 545x. Here's What Happened

HasanH47 — Mon, 04 May 2026 07:09:32 +0000

A solo dev's journey questioning a 40-year-old assumption in deep learning

The Question That Started It All

I was frustrated.

VS Code was getting heavier on my laptop. Cursor wanted $20/month. The best AI agents were owned by 5 mega-corporations. As a developer in Indonesia, I sometimes felt we were perpetual consumers, never creators.

So I asked Claude: "Can AI be smaller?"

That conversation led somewhere unexpected. We started questioning the most fundamental assumption in deep learning since 1986:

Do weights have to be stored as matrices of numbers?

Think about it. A human brain doesn't store information as numbers. A seed doesn't contain all the leaves of a tree inside it — a seed contains instructions to grow leaves.

What if AI weights could be grown from a small seed when needed, instead of stored as massive matrices? A 30B model could fit on a smartphone. No cloud needed. No subscription. No billion-dollar hardware.

I named the project WIJI — "seed" in Javanese. The Javanese script: ꦮꦶꦗꦶ

Slogan: "Memaksimalkan yang minimal" — maximize the minimal.

The Setup

I'm a solo developer. No PhD. No GPU cluster. Just a laptop, curiosity, and AI as a research collaborator.

The plan:

Take TinyLlama 1.1B (small enough to run on CPU)
Try to reconstruct its weights using a tiny generator network
Replace original weights with generated ones
See if the model still works

If single layer works, scale up. If not, learn why and pivot.

Experiment 1: Single Matrix

I started with one weight matrix: o_proj of layer 0. It has 4.2 million parameters.

I built a coordinate-based MLP generator: input is (row, col) coordinates, output is the weight value at that position. The generator has only 164K parameters — 25x smaller than the target.

Training was simple: sample random coordinates, predict their values, minimize MSE.

After 5000 steps, MSE settled at 0.000067. I reconstructed the full matrix and replaced the original in the model.

Test prompt: "What is the capital of Indonesia?"

Original output: "Indonesia's capital is Jakarta."

Reconstructed output: "The capital of Indonesia is Jakarta."

It worked. Different words, same meaning. The model still functioned with weights compressed 25x.

I was elated. Phase 0 looked promising.

Experiment 2: Going Big

If 1 layer works, why not all 22 layers?

I added a layer embedding to the generator so it could handle multiple layers. Same 164K params, but now needed to represent 22 different weight distributions — 92M parameters total.

That's 545x compression.

I trained for 3000 steps. MSE settled at 0.000234 — only 3x higher than experiment 1. Should be fine, right?

Output: "Ingatescripturecordialoisimoisequalifiesearchivedeastern Discogs, and"

Complete gibberish.

This was my first lesson: MSE Loss is not a reliable predictor for LLM output quality. The loss only got 3x worse, but the output collapsed entirely.

Experiment A: Diagnostic

Before scaling more, I needed to understand: was the failure because of multi-component (Q/K/V/O matrices in one layer) or multi-layer (across 22 layers)?

So I tested: single generator handling all 4 attention components in just layer 0. 4 matrices, but still 1 layer.

MSE settled at 0.000400 — 6x higher than experiment 1.

Output: "The capital of Indonesia is Jakarta."

Still functional. So the issue wasn't multi-component. The issue was multi-layer.

Now I had a hypothesis: error compounds across layers. Each layer's small error becomes the next layer's wrong input, which produces bigger errors, until the model collapses.

Experiment B2: Microservices for Layers

If one generator can't handle 22 layers, what if I built 22 separate generators? One per layer, each specializing.

22 generators × 164K params = 3.6M total. Compression: 25x. Same as experiment 1.

I trained each generator for 1000 steps. The training logs revealed something important:

Layer	MSE Loss
0	0.000063
5	0.000203
10	0.000225
15	0.000244
21	0.000373

Error increases monotonically from early layers to late layers. Same generator capacity, same training budget, but layer 21 was 6x harder to fit than layer 0.

This makes sense: late layers in transformers capture complex semantic patterns. Early layers capture simple syntax. A small generator can fit the latter but struggles with the former.

Output: "WHEREASPark. ."

Failure. But informative failure.

Experiment B3: Adaptive Capacity

If late layers need more capacity, give them more capacity:

Layers 0-7: 128 hidden dim (small)
Layers 8-15: 256 hidden dim (medium)
Layers 16-21: 512 hidden dim (large)

Total params: 5.9M. Compression: 15x (smaller compression but should help).

I also tripled training steps to 3000 per generator.

Result for layer 21: MSE = 0.000366 (vs B2's 0.000373).

Almost identical. 10x more capacity, 3x more training, virtually no improvement.

Output: (degenerate loop)

This was the most important finding of the entire project:

There's a fundamental limit. MSE plateau at ~0.0003-0.0004 is independent of capacity and training time.

This phenomenon has a name in research: spectral bias. Neural networks with ReLU/GELU activations have an inductive bias toward smooth functions. Transformer weights look like noise — high-frequency random distributions.

Throwing capacity at the problem doesn't help because the architecture itself is wrong for this task.

Experiment B4: The Cliff Edge

I had 22 trained generators. Before giving up, I wanted to know: where exactly does the model fail?

I ran a progressive swap test. Replace layers 0 to N-1 with generated weights. Test inference. Increment N. See what happens.

N (layers replaced)	Output	Status
1	"The capital of Indonesia is Jakarta."	✅ Perfect
3	"The capital of Ia is 10."	⚠️ Partial collapse
5	"" (empty)	❌ Collapse
8	"" (empty)	❌ Collapse
12	"" (empty)	❌ Collapse
16	"" (empty)	❌ Collapse
22	"Ingunsuretournalty. WHERE2..."	❌ Gibberish

Cliff edge between N=1 and N=3. Sharp, not gradual. Phase transition.

But the most counterintuitive finding: N=22 produces output, while N=5-16 produce empty strings.

When you replace some layers but keep others, the corrupt layers produce outputs that are "out of distribution" for the original layers. The mismatch causes probability collapse — the model produces nothing.

When you replace ALL layers, the corruption is internally consistent. The model still produces gibberish, but it produces something.

The lesson: internal consistency matters more than absolute correctness.

What I Actually Learned

After 5 experiments and many hours of failure, here's what I have:

Validated empirically:

✅ Weight matrices have significant redundancy (compress 25-56x for single layer)
✅ MSE Loss is a misleading metric for LLM compression quality
✅ Cliff edge phenomenon exists at N=2 layers
✅ Capacity scaling doesn't solve spectral bias
✅ Internal consistency > absolute correctness in deep networks

Open questions for next phase:

Can Fourier features (NeRF-style positional encoding) overcome spectral bias?
Are FFN layers easier to reconstruct than attention layers?
Can output-aware loss (KL divergence) replace MSE?
Does cliff edge shift with bigger models?

Honest probability assessment:

40% that the next phase finds something useful
25% that we get a working prototype
5-10% that this leads to a genuine breakthrough

But 10% × "fundamentally change AI deployment" = high expected value for a solo dev with AI as collaborator.

Why I'm Sharing This

I could have buried these failures and only shared the success. That's the temptation.

Instead, I'm publishing everything: code, failures, insights, and methodology. Why?

Negative results are valuable. Someone else attempting this will save weeks knowing where the cliff edge is.

Open source is legacy. Even if I stop maintaining this, the experiments stay accessible forever.

Solo dev + AI is a new research methodology. I want to demonstrate what's possible. Other developers in Indonesia, in developing countries, in their bedrooms — they can ask hard questions and explore them. They don't need to wait for FAANG employment to contribute to AI research.

Centralization is the enemy. AI is concentrating into the hands of 5 corporations. If we accept that, our future is dystopian. WIJI is a contrarian bet — that intelligence can be made minimal, affordable, and owned by everyone.

What's Next

Phase 1 plans:

Fourier Features experiment — likely solves spectral bias based on NeRF research
FFN layer test — different weight distribution, possibly easier to compress
Streaming inference system — pragmatic system that accepts N=1 limit but works
Rust port — for proper performance benchmarking

I'll publish results as I go. Failures and successes both.

If you're interested in this kind of research, the repo is fully open:

🔗 github.com/sangkan-dev/wiji-experimental

Critique welcome. Collaboration welcome. Even philosophical disagreement welcome.

"Mari kita lebih menggila di dunia yang udah gila ini."

Let's get crazier in this already-crazy world.

About the author: HasanH47, a DevOps Enginner based in Yogyakarta, Indonesia. Building products at the intersection of local context and frontier technology. Project under Sangkan organization.

If this article gave you something to think about, consider following for updates on Phase 1 results. And if you're working on anything related, please reach out.