DEV Community: Sarthak Agrawal

Study distributed systems through failure, not diagrams

Sarthak Agrawal — Thu, 24 Sep 2026 04:30:00 +0000

A clean architecture diagram is drawn after the network has behaved. Distributed systems become interesting when a request is retried, a replica lags, a partition appears, or a workflow resumes after a crash.

The 12-week Distributed Systems roadmap begins with the mechanisms that shape those failures. HTTP lifecycle, idempotency, load balancing, consistent hashing, retries, consensus, replication, sharding, and the CAP tradeoff form the first four weeks.

Each mechanism has a practical question attached. What makes a retry safe? Which key should choose a shard? What does a stale read look like to the caller? When does jitter prevent a recovery storm? These questions turn vocabulary into behavior.

The middle phase moves into queues, event streams, caching, and real-time messaging. The focus is on production boundaries: backpressure, ordering, cache stampedes, delivery guarantees, and the point where a synchronous request should become durable work.

The final four weeks cover distributed infrastructure, durable workflows, and failure recovery. The synthesis artifact asks the learner to test failures and explain recovery, not only present a happy-path design. A credible result names the state owner, retry policy, idempotency boundary, visibility into partial progress, and terminal failure state.

This is also a better interview habit. When a design includes a queue or replicated store, the next sentence should describe what happens when it is delayed, duplicated, stale, or unavailable. The failure path often reveals whether the component belongs in the design at all.

The complete roadmap is at https://learn.significanthobbies.com/curriculum/roadmaps/distributed-systems-12w.

Learn LLM serving as a memory and scheduling problem

Sarthak Agrawal — Wed, 23 Sep 2026 04:30:00 +0000

A language model server is not only a model behind an HTTP endpoint. Its behavior depends on memory layout, token scheduling, kernels, routing, hardware, and the economics of each request.

The 12-week Inference and Serving roadmap begins with the model mechanics required to reason about that system. Tokenization, attention, transformer blocks, sampling, checkpoints, LoRA, quantization, and structured output establish what the engine must execute and preserve.

The path then turns those operations into serving mechanisms. Inference engines organize execution. KV caches retain state between generated tokens. Paged attention changes how that state is managed. Continuous batching mixes work from different requests. Model routing chooses a path based on cost or capability rather than sending every query to the same model.

Weeks five through eight move closer to the hardware. Flash Attention, speculative decoding, GPU utilization, and inference hardware are treated as tradeoffs, not isolated tricks. A faster kernel can still lose at the system level if it increases memory pressure, complicates batching, or only helps a shape the service rarely sees.

The final phase connects cost and latency with local inference, browser runtimes, and WebGPU. The synthesis requires measurement across the path. Useful evidence includes the workload, model and precision, batch behavior, memory use, latency distribution, throughput, and the quality boundary that the optimization must keep.

This perspective changes the learning question. Instead of asking which serving framework is fastest, ask which resource is limiting the workload and which scheduling or representation change addresses that limit.

The complete roadmap is at https://learn.significanthobbies.com/curriculum/roadmaps/inference-serving-12w.

Build agent systems from bounded loops outward

Sarthak Agrawal — Tue, 22 Sep 2026 04:30:00 +0000

Many agent demos begin with a model, a tool list, and a loop. Production problems begin as soon as the loop has authority, state, and time.

The 12-week Agent Systems roadmap starts with the small mechanics: structured output, tool calling, bounded agent loops, memory, and MCP integrations. The step-limit exercise is deliberately basic. A loop that cannot explain when it stops is not ready for more tools.

The next phase introduces coordination, explicit communication interfaces, and browser or computer use. These topics are grouped because coordination is not just adding more agents. Each participant needs a contract for messages, ownership, failure, and handoff. Computer use raises the cost of ambiguity because the tool can act on a real interface with state the model may not fully observe.

The final four weeks focus on durability and control. Long-running work needs persisted state, resumable execution, scheduled triggers, permission boundaries, and safe recovery after partial failure. A sandbox limits what an agent can touch. Durable execution limits what a crash can erase. Both are part of correctness.

The synthesis artifact asks the learner to combine these pieces into one system and test its failure modes. A useful result should show what the agent can do, which actions require authority, where state lives, how retries behave, and how an operator can stop or inspect the run.

Starting with bounded loops keeps the design honest. Autonomy is not the number of iterations a model can run. It is the amount of useful work the system can complete while its scope, state, and failure behavior remain legible.

The roadmap is at https://learn.significanthobbies.com/curriculum/roadmaps/agent-systems-12w.

Build AI evaluations that survive a model change

Sarthak Agrawal — Mon, 21 Sep 2026 04:30:00 +0000

An AI evaluation is easy to trust when it confirms the release you already want. The harder test is whether it still helps after the prompt, model, tool, or traffic mix changes.

The 12-week Evaluation and AI Reliability roadmap begins below the evaluation layer. It covers enough model mechanics, probability, and descriptive statistics to explain what a metric measures and what it misses. It then moves into LLM evaluations, coding-agent benchmarks, and tool-use evaluations.

That foundation matters because different tasks fail differently. Exact match may fit a constrained output. A tool-using agent needs checks for action selection, arguments, side effects, recovery, and final evidence. A coding benchmark can reward a patch that passes a narrow test while missing the requested behavior.

The middle four weeks move from a local score to a production evaluation system. Prompt versions are explicit. Regression tests use frozen cases. Hallucination and failure detection become named behaviors. Quality, cost, and latency are measured together, because a release that improves one dimension can quietly damage another.

The final phase adds agent observability, tracing and replay, evidence-backed verification, and human review. The synthesis artifact is not a leaderboard. It is an evaluation and observability system that can reproduce a failure, compare a candidate with a baseline, and show where human judgment still enters.

A useful evaluation should make a bad release harder to justify. It should also make a good release easier to explain without stretching one number into a general claim.

The complete roadmap is at https://learn.significanthobbies.com/curriculum/roadmaps/ai-reliability-12w.

A coding agent needs a harness, not a longer prompt

Sarthak Agrawal — Sun, 20 Sep 2026 04:30:00 +0000

A coding agent can produce a plausible patch with a good prompt. Reliable progress across a real repository needs more than prompt quality.

The Harness Engineering roadmap treats the environment around the agent as a system. The first build turns implicit expectations into scoped repository instructions and executable checks. The second makes the workspace readable from a fresh checkout: structure, setup, readiness, and permissions should be discoverable without private history.

Continuity arrives in the third build. A later session needs the goal, current progress, evidence, blockers, and next action. Without that durable state, long-running work becomes a series of confident restarts.

The fourth build adds scope control and runtime feedback. The agent needs a bounded authority surface and evidence that tells it whether to continue, replan, complete, or escalate. This is where a harness starts to look less like a prompt wrapper and more like an operating system for work.

The fifth build separates maker and checker. The checker reconstructs the contract and gathers fresh evidence instead of accepting the maker's claim that the task is done. The final two builds connect initialization, execution, recovery, verification, and terminal states, then compare the automated loop with a baseline and remove one component to see whether it mattered.

This sequence makes one distinction clear: agent capability and agent reliability are different properties. A strong model can still fail in an unreadable workspace, lose state between sessions, exceed its scope, or verify its own assumptions.

The seven-build roadmap is at https://learn.significanthobbies.com/curriculum/roadmaps/harness-engineering.

Trace one tensor from model math to serving cost

Sarthak Agrawal — Sat, 19 Sep 2026 04:30:00 +0000

AI systems material often splits into separate subjects: model math, GPU kernels, runtimes, and production serving. That makes each topic easier to name and harder to connect.

Trace a Tensor uses one workload as the thread. The first week starts with data representation and backpropagation. The point is to explain what a tensor contains, how its shape affects a computation, and what work a forward pass creates for the backward pass.

The second week follows the same data through the compute, memory, and storage hierarchy. Before changing a kernel, the learner has to profile the runtime and identify where time or bandwidth is actually going. That prevents an interesting optimization from becoming a substitute for diagnosis.

The third week changes the work. Flash Attention and quantization are studied as different ways to reduce movement or numerical cost. Both come with a quality or correctness boundary, so the exercise is not complete when the faster path runs. It must preserve the behavior the system needs.

The final week moves into engines, continuous batching, and serving economics. Latency, throughput, utilization, quality, and cost now sit in the same decision. The capstone asks for a reproducible workload or performance model, a measured bottleneck, a before-and-after comparison, and a defended optimization.

That shape is more useful than memorizing a stack diagram. It gives every abstraction a place in one causal chain: values become tensors, tensors create work, the machine schedules that work, and the serving system pays for the result.

The complete roadmap is at https://learn.significanthobbies.com/curriculum/roadmaps/trace-a-tensor.

AI pair-programming became useful when every idea met a benchmark

Sarthak Agrawal — Fri, 18 Sep 2026 04:30:00 +0000

The most confident optimization suggestions were not always the useful ones.

An 8 by 8 register block sounded better than 4 by 4 because it offered more reuse. It ran slower. Packed f16 storage sounded additive with tiling. It was not. A vectorized kernel won its standalone benchmark and initially broke the real training loop.

The productive workflow was simple: take the suggestion, define the right baseline, implement the smallest test, measure it, and keep the negative result. The benchmark had authority over the explanation.

The negative results became more reusable than many wins. They recorded which limit an idea actually attacked and why the next attempt should differ. End-to-end parity tests caught errors that standalone kernel tests could not see.

AI was useful for generating hypotheses and tracing failures. It was unreliable as the judge of whether its own proposal worked. That role stayed with frozen inputs, reproducible commands, and measured output.

The complete pair-programming notes are at https://posttrainllm.com/devlog.

Two days of kernel debugging ended at one default value

Sarthak Agrawal — Thu, 17 Sep 2026 04:30:00 +0000

Browser training plateaued near a loss of 2.45. The curve was smooth enough to look like a model or kernel limit.

The WebGPU kernels were not the cause. The browser used a learning rate of 3e-3 while the Python reference used 3e-4. The default had drifted by a factor of ten.

Kernel parity tests covered the numerical operations, but nothing compared the configuration defaults. That gap made a correct implementation behave like a broken model. The reference path needed to be the authority for hyperparameters as well as math.

The same investigation found two more boundaries. The default corpus was only 863 bytes, which hid model capacity behind a tiny dataset. The Memory64 benchmark exercised an allocation path but not the browser bridge used by the product.

All three failures had the same shape: a narrow test passed while the real workflow remained wrong. The repair was to add end-to-end checks at the configuration, data, and integration boundaries.

The full debugging record is at https://posttrainllm.com/devlog.

A public roadmap should show which predictions aged badly

Sarthak Agrawal — Wed, 16 Sep 2026 04:30:00 +0000

An older PostTrainLLM devlog ended with two large next steps: a pre-trained model gallery and a native macOS application.

The project moved. The browser gallery and native Swift and MLX surfaces now exist, but neither became the permanent product center. The active center is the Mac-local specialist factory: target, data, post-training, evaluation, packaging, and report.

This is why keeping the old roadmap visible matters. A roadmap is a prediction made with partial information. It should not be rewritten later to make the current direction look inevitable.

The useful update is to mark what shipped, what became infrastructure, and what was parked. Browser polish and broad research remain available, but they do not compete with the current factory proof. The next milestone is a stronger validated specialist result, not another surface for its own sake.

The older prediction and current direction can both be inspected at https://posttrainllm.com/devlog.

Write the reference before debugging the shader

Sarthak Agrawal — Tue, 15 Sep 2026 04:30:00 +0000

The Flash Attention 2 path was easier to finish after the algorithm existed in plain JavaScript first.

The forward pass used one workgroup per batch, head, and query tile. It walked keys and values in blocks while keeping the online-softmax state in registers. The backward pass recomputed attention from q, k, and a saved log-sum-exp value instead of reading a cached attention matrix.

At the recorded Mega-class shape, removing that cached-matrix traffic avoided about 67 MB of global memory movement per layer per step. On the Medium preset, the end-to-end comparison moved from 6.8 seconds for WASM SIMD to 0.7 seconds for WebGPU with the new forward and backward paths. Loss drift was 2.5%.

Both halves were checked against a naive Node reference before the WGSL implementation was written. That separated algorithm errors from shader and integration errors. The shader became a translation of known math rather than the first place the math was tested.

The implementation record is at https://posttrainllm.com/devlog.

The browser-training speedup was a curve, not one headline number

Sarthak Agrawal — Mon, 14 Sep 2026 04:30:00 +0000

The WebGPU path did not have one honest speedup number.

Against multithreaded WASM SIMD, the measured improvement was 2.6 times on the Small preset, 6.8 times on Medium, 9.3 times on Large, and 12.1 times on XL. Loss drift ranged from 1.1% to 2.5% across the same presets.

The improvement grew with model size because larger matmuls made better use of workgroup tiling and register blocking. Dispatch overhead mattered more on the smaller preset and became less important as the matrix work grew.

This made the old single-number headline misleading even though that number had been measured. It described one point on a curve and invited readers to apply it to every model size.

The better reporting shape keeps the preset, baseline, step definition, and loss-parity result beside each speed measurement. That is enough context to reproduce the comparison and to see where the gain comes from.

The full table is at https://posttrainllm.com/devlog.

The more aggressive matmul kernel lost to the register budget

Sarthak Agrawal — Sun, 13 Sep 2026 04:30:00 +0000

The WebGPU matmul sweep started with a naive kernel, then added 16 by 16 workgroup tiling and a 4 by 4 output block per thread.

At a 2048 cubed matrix size, the measured time moved from 47.24 ms for the naive kernel to 17.23 ms for tiling and 9.12 ms for the 4 by 4 blocked kernel. The blocked version was 5.18 times faster than naive at that size.

The obvious next idea was an 8 by 8 block. It increased reuse on paper and lost at every measured size. At 2048 cubed, the 4 by 4 version took 10.15 ms while the 8 by 8 version took 11.52 ms. The likely cause was register pressure and lower workgroup occupancy.

Packed f16 storage was another apparent win. It beat the naive baseline but became slower when stacked on the already tiled kernel. Both optimizations were attacking the same bandwidth limit, so the speedups did not compound.

Keeping these rejected variants in the record prevents the next optimization pass from rediscovering them.

The complete sweep is at https://posttrainllm.com/devlog.