DEV Community: Dhanush Kumar Sivaji

The path to AI FDE #05: AI, ML and LLMs in plain words, and how an LLM writes

Dhanush Kumar Sivaji — Fri, 23 Oct 2026 17:16:40 +0000

Originally published on my newsletter, The Path to AI FDE. Subscribe free to get each issue by email.

This is a “From Zero” issue. If you’ve never written code or used an AI API, start here. If you’re a developer, the second half (how text actually gets generated) explains most of the weird behaviour you’ll meet later. No maths and no code: just the words people throw around, and what’s going on underneath.

Part A · The words, in plain English

1. AI is the broad umbrella. The OECD’s definition, widely cited in AI policy, boils down to this: an AI system is a machine that works out, from the input it gets, how to produce outputs such as predictions, content, recommendations or decisions. Systems differ in how independently they act and whether they keep adapting after launch.

2. Machine learning means learning from examples instead of hand-written rules. You train a piece of software called a model on data, and it learns to make predictions or generate content. Google’s crash course sorts ML into four families: supervised (learning from labelled examples), unsupervised (finding patterns without labels), reinforcement (learning from rewards) and generative AI.

3. Deep learning is ML with large neural networks. Many-layered networks learn their own features from raw input (pixels, characters) instead of humans designing them. The approach took off around 2010 and powers nearly everything that follows.

4. A large language model (LLM) is a deep network trained on a huge amount of text to predict what comes next. Do that across a large slice of the written internet, and the model picks up grammar, facts, style and a surprising amount of reasoning-like behaviour along the way.

5. A foundation model is one big model reused for many jobs. Researchers at Stanford coined the term in 2021 for models trained on broad data at scale and then adapted to a wide range of tasks. That’s why companies build products on top of a few such models instead of training their own.

6. Generative AI creates new content. Text, images, audio or code from your input, instead of just a label. Every LLM chatbot is generative AI, but not all AI is generative: a fraud detector that says “suspicious: yes/no” isn’t.

7. A hallucination is a confident answer that’s wrong. OpenAI defines it as a plausible but false statement. Part of the cause is that training and evaluation reward a good guess over “I don’t know”, and rare facts, like one person’s birthday, can’t be inferred from patterns in text. So an LLM is a brilliant pattern engine, not a database of verified facts. This is why later issues ground answers in real documents.

Part B · How an LLM writes text (no maths)

8. Text becomes tokens first. The model reads tokens: common words, word pieces and punctuation, not letters or whole words. Everything is counted in tokens, including limits and prices.

9. At each step it scores every possible next token. Given everything so far, each token in the vocabulary gets a likelihood. (The figure’s bars are illustrative, not from a real model.)

10. It writes one token at a time, in a loop. Each new token depends on all the ones before it, and the loop stops at a special end token, a stop sequence you set, or a length limit. That’s why answers stream in word by word, and why long answers take longer.

11. A little randomness is on purpose. The model doesn’t always take the top choice. Temperature is the dial (higher is more varied, lower more focused), and settings like top-p limit picks to the most likely candidates. So the same question can give different answers, and that’s normal.

12. Two training stages: read a lot, then learn to help. Pre-training is self-supervised: the model reads raw text and learns to predict the next token, no human labelling needed. Then it’s tuned on examples and feedback so it follows instructions instead of just continuing text. InstructGPT (2022) was an early public example.

13. Every call starts from zero. The model doesn’t remember your last message. Chat apps feel continuous because they resend the whole conversation with each request. (That’s the stateless API from #01: memory is the app’s job, not the model’s.)

Try it (10 minutes, no code)

In any chatbot, ask the same open question three times in fresh chats (”Give me a name for a bakery”) and compare the answers. Then ask something obscure and specific, such as a small-town council member’s birthday, and notice how confident the answer sounds. You’ve just seen points 11 and 7.

Common beginner mistakes

Treating “AI”, “ML” and “LLM” as synonyms. They’re nested, not the same.
Assuming the model looks facts up. By default it predicts; it doesn’t retrieve.
Thinking the chatbot remembers you. The app resends history.
Counting words instead of tokens when estimating limits or cost.

Takeaway

An LLM is a next-token predictor, trained on a lot of text and then tuned to be helpful. It writes one token at a time with a bit of chosen randomness, and it remembers nothing between calls.

Where this fits in my roadmap: this is on-ramp material, not part of my current study block. I already work with these ideas as a TypeScript engineer, so I’m writing it for readers starting from zero. It also gives later issues shared vocabulary. The deeper versions (tokenizers, context windows, prefill and decode, sampling, how training works) come in Section 6, once my roadmap gets there.

Over to you: which AI term did you nod along to for longest before you really knew what it meant?

Sources

OECD.AI: Updates to the OECD’s definition of an AI system explained (2023). https://oecd.ai/en/wonk/ai-system-definition-update
Google ML Crash Course: What is machine learning? (training a model on data; supervised, unsupervised, reinforcement, generative AI). https://developers.google.com/machine-learning/intro-to-ml/what-is-ml
Bommasani et al., On the Opportunities and Risks of Foundation Models (Stanford CRFM, 2021): defines “foundation model”; history of deep learning. https://arxiv.org/abs/2108.07258
Hugging Face LLM Course, Ch. 1: How do Transformers work? (self-supervised pre-training on raw text; causal language modelling; fine-tuning; InstructGPT, Jan 2022). https://huggingface.co/learn/llm-course/chapter1/4
Hugging Face LLM Course, Ch. 1: Deep dive into text generation inference (one token at a time; temperature, top-p, top-k; stop conditions). https://huggingface.co/learn/llm-course/chapter1/8
Hugging Face LLM Course, Ch. 2: Tokenizers (”token” + “ization”). https://huggingface.co/learn/llm-course/chapter2/4
OpenAI: Why language models hallucinate (Sep 5, 2025). https://openai.com/index/why-language-models-hallucinate/
Anthropic: Working with the Messages API (stateless; resend the conversation). https://platform.claude.com/docs/en/build-with-claude/working-with-messages

Sources checked Oct 18, 2026.

Free: the 5-page roadmap from Novice to AI FDE.

If this was useful, subscribe to The Path to AI FDE for the next issue.

The path to AI FDE #04: Many calls at once, and a FastAPI service you can lean on

Dhanush Kumar Sivaji — Thu, 22 Oct 2026 17:14:51 +0000

Originally published on my newsletter, The Path to AI FDE. Subscribe free to get each issue by email.

Issue #03 ended with tasks: the way async work actually overlaps. Real AI backends fan out, calling a model several times or fetching a few sources in parallel. So this issue starts with how to do that without losing errors or hanging forever. Then it turns to the FastAPI features that move an endpoint past hello-world. As always, every example is a toy one (a notes app), not my roadmap exercises.

Part A · Many calls at once

1. gather returns results in the order you asked. a, b = await asyncio.gather(fetch(1), fetch(2)) runs both concurrently, and the result list matches the order of the arguments, not the order the calls finished in. It’s the closest thing to Promise.all.

2. Know what gather does with an error. By default, the first exception is raised straight to you, and the other calls are not cancelled: they keep running in the background. With return_exceptions=True, errors come back as items in the result list instead, which suits best-effort work.

3. TaskGroup is all-or-nothing. Inside async with asyncio.TaskGroup() as tg:, you start work with tg.create_task(...). The first failure cancels the rest, and when the block exits the failures are raised together as an ExceptionGroup, which you handle with except* SomeError:. Use it when partial results are useless. It also keeps task references for you (#03, point 11).

4. Every network wait needs a deadline. async with asyncio.timeout(10): cancels whatever is inside after 10 seconds and raises TimeoutError, which you can only catch outside the block. For a single awaitable, await asyncio.wait_for(coro, timeout=10) does the same job. Without a deadline, one slow provider call can hold a request open indefinitely.

5. A semaphore caps how many run at once. sem = asyncio.Semaphore(5), then async with sem: around each call, lets at most five run concurrently while the rest wait their turn. That’s how you fan out over 200 items while keeping concurrency low enough to stay under a provider’s rate limit (the 429s from #01).

6. Don’t swallow cancellation. TaskGroup and timeout() work by cancelling tasks, and the asyncio docs warn that they can misbehave if your code catches and hides asyncio.CancelledError. A bare except:, an except BaseException: or a re-raise you forgot quietly breaks your deadlines. (except Exception: doesn’t catch it: since Python 3.8, CancelledError is a BaseException.) Catch specific errors (#02, point 4).

Part B · FastAPI properly

7. Request models give you 422s for free. Declare a body parameter as a Pydantic model (note: NoteIn), and FastAPI validates the request, answering 422 with a list of what’s wrong before your function even runs.

8. Response models validate and filter what goes out. Set a return type or response_model=NoteOut, and FastAPI checks your output and drops any field the model doesn’t declare. That’s the docs’ security point: an internal field (an owner ID, a password hash) can’t leak just because it was on the object you returned. If you set both, response_model takes priority.

9. HTTPException for errors you mean. raise HTTPException(status_code=404, detail="Note not found") stops the handler and returns that status with a JSON body. Raise it for expected failures: a missing item, a bad state, an upstream provider that’s down.

10. Dependencies hold the shared plumbing. A dependency is a function FastAPI calls for you before your endpoint runs, declared as a parameter with Depends(get_db), usually through an Annotated alias so it’s written once. Shared logic, database sessions and auth live there instead of in every handler. Dependencies can be async def or plain def.

11. def versus async def endpoints. An async def endpoint runs on the event loop, so anything blocking inside it stalls every request (#03, point 12). A plain def endpoint runs in a threadpool instead, so blocking calls there are survivable. Rule of thumb: async def with async clients, plain def when your libraries are synchronous.

12. Settings come from the environment, validated. A pydantic-settings BaseSettings class reads environment variables (case-insensitively) and validates their types when it’s created. A missing API key or TIMEOUT_S=abc fails right there, not halfway through a request that uses it: at startup if you create the settings at import time, or on first use if you serve them the way the FastAPI docs do, from an @lru_cache‘d get_settings() dependency. That dependency also lets tests swap settings with app.dependency_overrides.

Try it (20 minutes, toy example)

Make async def fetch(i) that sleeps for a random 0–2 seconds and sometimes raises. Run 20 of them three ways: gather, gather(return_exceptions=True) and a TaskGroup. Then add Semaphore(5) and a 1-second timeout, and compare what you get back each time. For the FastAPI half, give a toy notes app a NoteOut model without an internal owner_id field, return an object that has one, and check that it doesn’t appear in the response.

Common beginner mistakes

Assuming gather stops the other calls when one fails.
Writing try/except TimeoutError inside async with asyncio.timeout().
Fanning out with no concurrency limit and getting rate-limited.
Returning a database object without a response model, and leaking fields.
Doing blocking calls in async def endpoints.

Takeaway

Fan out with a limit and a deadline. Let FastAPI validate both directions, and keep shared plumbing in dependencies.

Where this fits in my roadmap: this is the second half of my foundations block (async in practice, plus FastAPI and Pydantic). In my own study, I’m building that block’s small LLM endpoint myself, with real timeouts and error handling. So this post sticks to the concepts and toy examples, not my solution. Next comes testing (pytest, plus mocking the provider), and then structured outputs, where the Pydantic schema from #03 becomes the contract the model must follow.

Over to you: gather(return_exceptions=True) or TaskGroup: which do you reach for by default, and why?

Sources

Python docs: Coroutines and Tasks (gather order, errors and return_exceptions; TaskGroup cancellation and ExceptionGroup; timeout() raising TimeoutError catchable only outside; wait_for; the warning against swallowing CancelledError). https://docs.python.org/3/library/asyncio-task.html
Python tutorial 8.9: Raising and handling multiple unrelated exceptions (ExceptionGroup, except*). https://docs.python.org/3/tutorial/errors.html
Python docs: Synchronization primitives (Semaphore). https://docs.python.org/3/library/asyncio-sync.html
FastAPI: Request Body (Pydantic models; invalid data returns an error). https://fastapi.tiangolo.com/tutorial/body/
FastAPI: Response Model (validates returned data; filters output to the declared type for security; response_model priority). https://fastapi.tiangolo.com/tutorial/response-model/
FastAPI: Handling Errors (HTTPException; RequestValidationError and 422). https://fastapi.tiangolo.com/tutorial/handling-errors/
FastAPI: Dependencies (Depends, shared logic, DB connections, auth; Annotated aliases; async def or def). https://fastapi.tiangolo.com/tutorial/dependencies/
FastAPI: Concurrency and async/await (plain def path operations run in an external threadpool). https://fastapi.tiangolo.com/async/
FastAPI: Settings and Environment Variables (BaseSettings, case-insensitive env vars, lru_cache, dependency_overrides, .env). https://fastapi.tiangolo.com/advanced/settings/
Python docs: asyncio Exceptions (CancelledError is a BaseException since 3.8; must almost always be re-raised). https://docs.python.org/3/library/asyncio-exceptions.html

Sources checked Oct 18, 2026.

Originally published on The Path to AI FDE on Substack. Subscribe for the next issue: https://dhanushkumarsivaji.substack.com/subscribe

If this was useful, subscribe to The Path to AI FDE for the next issue.

The path to AI FDE #03: Python types you can trust, and async waiting done well

Dhanush Kumar Sivaji — Wed, 21 Oct 2026 17:16:57 +0000

Originally published on my newsletter, The Path to AI FDE. Subscribe free to get each issue by email.

Issue #02 was the quick TypeScript-to-Python map. This issue goes one level deeper on the two ideas every AI backend leans on: types (what data is allowed in) and async (how one process waits on many slow network calls). Twelve short points, each with the trap I’d fall into first.

Part A · Types you can trust

int, collection hints like list[str], str | None as the Python 3.10 spelling of Optional[str], and count_words(42) running anyway unless mypy or Pyright flags it" width="800" height="510">

1. Hints describe intent. def count_words(text: str) -> int tells readers, editors and type checkers what goes in and what comes out. Collections take their item types in brackets: list[str], dict[str, int].

2. “Maybe missing” is spelled X | None. Since Python 3.10 (PEP 604) you write str | None instead of Optional[str]. Trap: in Pydantic v2, subtitle: "str | None with no default is still a required field. Add = None if callers may leave it out."

3. Checked before running, not while running. As covered in #02, Python ignores hints at runtime, so count_words(42) runs anyway and only mypy or Pyright flags it. Run one of them in CI.

4. Pydantic models check data when it arrives. A model is a class that inherits from BaseModel, with typed fields, and creating an instance runs validation. Field() adds constraints such as gt=0 or max_length=200. If no error is raised, the fields are guaranteed to match their types.

5. Lax by default: know what it lets through. Pydantic coerces when it can (pages="412" becomes the integer 412) and silently ignores extra keys. That’s handy for query strings and environment variables, but risky for data from a model or another service. For those, opt into strict=True and extra='forbid'.

6. One error lists every problem, and a model can describe itself. A ValidationError reports all failing fields with their locations (such as tags.2), not just the first. Book.model_validate_json(raw) validates a JSON string directly, and Book.model_json_schema() exports a JSON Schema, which is how structured outputs get their contract (coming later in the series).

Part B · Waiting done well

7. AI backends mostly wait. An LLM call spends most of its time waiting on someone else’s server, not computing on yours. Async lets one process overlap those waits instead of standing idle through each one.

8. The event loop runs one task at a time. asyncio uses cooperative scheduling: whenever the running task hits an await on something not yet ready, the loop switches to another. Nothing is truly simultaneous; the overlap comes only from the waiting.

9. async def makes a coroutine, and asyncio.run() starts the loop. asyncio.run(main()) is the usual entry point in a script. Calling main() on its own only creates a coroutine object, which does nothing until awaited (#02, point 6). In FastAPI, the framework runs the loop for you.

10. Back-to-back awaits are still sequential. This is the trap I’d hit first. In the asyncio docs’ example, await say_after(1, ...) then await say_after(2, ...) takes 3 seconds: async syntax alone doesn’t make anything concurrent.

11. Tasks are how work overlaps. asyncio.create_task(coro) schedules the coroutine to run soon, alongside the current one, which brings the same example down to about 2 seconds. Keep a reference to every task: the loop holds tasks only weakly, so an unreferenced task can be garbage-collected before it finishes. TaskGroup handles that for you (next issue).

12. One blocking call stalls everyone. time.sleep(1) or a synchronous HTTP client inside async def freezes the whole loop, while await asyncio.sleep(1) always suspends just the current task. For blocking I/O you can’t avoid, await asyncio.to_thread(fn) moves it off the loop.

Try it (15 minutes, toy example)

Write async def nap(n) that does await asyncio.sleep(n). Time await nap(1); await nap(2), then the same two calls wrapped in create_task. Then swap in time.sleep and watch the overlap vanish. Separately, define a Book model with title: str and pages: int = Field(gt=0), and feed it one good dict and one bad dict.

Common beginner mistakes

Assuming hints validate input. Only Pydantic (or your own checks) does that.
Writing x: str | None and expecting it to be optional in Pydantic v2. It needs = None.
Thinking async def plus await equals parallel. It doesn’t without tasks.
Creating tasks and dropping the reference.

Takeaway

Hints describe, Pydantic enforces. await waits; tasks overlap. That’s the whole issue in two lines.

Where this fits in my roadmap: these are the concepts behind the foundations block of my roadmap (Python typing, errors and async; FastAPI and Pydantic models). That block’s exercises (a typed document loader, a small validated LLM endpoint and its tests) are mine to build, so nothing above is that code; the examples are deliberately toy ones. Next issue: running many calls at once safely (gather, TaskGroup, timeouts) and FastAPI beyond hello-world.

Over to you: which bit you first: a missing await, or an await that should have been a task?

Sources

Python docs: typing (annotations aren’t enforced at runtime). https://docs.python.org/3/library/typing.html
PEP 604: Allow writing union types as X | Y (Python 3.10). https://peps.python.org/pep-0604/
mypy documentation. https://mypy.readthedocs.io/en/stable/
Pyright. https://github.com/microsoft/pyright
Pydantic: Models (validation on instantiation, guarantees, coercion, extra data ignored by default, ValidationError lists all errors, model_validate_json, model_json_schema). https://pydantic.dev/docs/validation/latest/concepts/models/
Pydantic: Fields (Field() constraints such as gt and max_length; v2 no longer gives Optional an implicit None default). https://pydantic.dev/docs/validation/latest/concepts/fields/
Pydantic: Strict Mode (default coercion of '123' to 123; enabling strict mode). https://pydantic.dev/docs/validation/latest/concepts/strict_mode/
Python docs: Coroutines and Tasks (calling a coroutine doesn’t schedule it; asyncio.run; the 3-second vs 2-second example; create_task; keep task references; sleep always suspends; to_thread; cooperative scheduling). https://docs.python.org/3/library/asyncio-task.html
Python docs: Developing with asyncio (”Running Blocking Code”). https://docs.python.org/3/library/asyncio-dev.html

Sources checked Oct 18, 2026.

Originally published on The Path to AI FDE on Substack. Subscribe for the next issue: https://dhanushkumarsivaji.substack.com/subscribe

If this was useful, subscribe to The Path to AI FDE for the next issue.

The path to AI FDE #02: Python for AI backends: a TypeScript engineer's refresher

Dhanush Kumar Sivaji — Wed, 21 Oct 2026 17:16:38 +0000

Originally published on my newsletter, The Path to AI FDE. Subscribe free to get each issue by email.

Most AI engineering tutorials assume Python. I’ve written TypeScript for years and have working, not deep, Python. My roadmap starts with a short Python + FastAPI refresher rather than a full course, and this post is the map I’m following through it: eight concepts AI backends lean on, each mapped to TypeScript I already know, plus the spot where my TS instincts are most likely to mislead me.

Each concept gets its own figure first, then the short version in words.

Setup and types: concepts 1–3

1. Isolated dependencies: node_modules ↔ virtual environments. A virtual environment is a self-contained directory with its own interpreter and packages, so two projects can pin different versions of the same library. Create it with python -m venv .venv, then activate it: activation changes your shell so python points into that environment. The TS instinct that misleads: there’s no automatic per-folder install. Forget to activate, and pip targets whichever Python comes first on your PATH. On Pythons marked “externally managed” (PEP 668), as some OS-provided ones are, pip refuses to install outside a virtual environment and exits with an error.

2. Type hints are for tools, just like TS types. The TypeScript handbook shows types being erased when tsc compiles to JavaScript. Python’s typing docs say the runtime does not enforce annotations; they exist for type checkers, IDEs and linters. So def f(x: float) -> str happily accepts a string at runtime. Run mypy or Pyright in CI, the way you’d run tsc --noEmit.

3. Runtime validation: Zod ↔ Pydantic. Since annotations aren’t enforced, untrusted data needs a validator. Zod describes itself as TypeScript-first schema validation with static type inference. Pydantic plays the same role in Python: validation is driven by type annotations, and once data passes, the fields are guaranteed to match the declared types. For LLM work, FastAPI uses it to validate requests and responses, and OpenAI’s Python SDK accepts a Pydantic class directly (client.responses.parse(..., text_format=MyModel)): it converts the class to a JSON Schema, sends it and parses the reply back into the model. Only the subset of JSON Schema that structured outputs supports will work.

Errors and cleanup: concepts 4–5

4. Errors: throw/catch ↔ raise/except. The shape maps closely, plus one extra: else, which runs only if the try block raised nothing. Idiomatic Python catches specific exception classes. In FastAPI, raise HTTPException(...) returns an HTTP error and ends the request, even from deep inside a helper. But FastAPI’s docs call HTTPException a normal Python exception, so a broad except Exception: between the raise and FastAPI swallows it. One more reason to catch narrowly.

5. Guaranteed cleanup: using ↔ with. TypeScript 5.2 added using declarations (explicit resource management). Python has had the with statement for a long time: the cleanup step runs whether the block finishes normally or raises, and async with is the asynchronous version. In AI backends, that’s the home for HTTP clients, open files and database sessions.

Async: concepts 6–8

6. Starting async work: the subtle one. The syntax looks the same; what happens at call time doesn’t. MDN: an async function runs synchronously up to its first await, so calling fetchDoc() starts the work immediately. Python’s asyncio docs: simply calling a coroutine will not schedule it. Nothing runs until you await it or wrap it in a task, so a forgotten await means the work never happens. asyncio emits a RuntimeWarning (”coroutine ... was never awaited”); treat it as a bug.

7. Blocking the event loop: same trap, both runtimes. Node’s docs warn against blocking the event loop. Python’s asyncio docs say a CPU-intensive calculation that takes 1 second delays every concurrent task by 1 second, and blocking I/O (time.sleep(), a synchronous HTTP client inside async def) does the same. The fix depends on the work: for blocking I/O, an awaitable client or asyncio.to_thread(), which the docs say is mainly for I/O-bound work because the GIL usually stops threads speeding up CPU-bound Python; for CPU-bound work, a process pool via loop.run_in_executor(ProcessPoolExecutor(), fn); in FastAPI, a plain def endpoint runs in a threadpool instead of on the loop.

8. Fan-out: Promise.all ↔ asyncio.gather / TaskGroup. You’ll want this for parallel calls, like embedding several chunks or querying two retrievers. Promise.all rejects with the first rejection but doesn’t stop the other operations: MDN notes promises have no built-in cancellation (you’d reach for AbortController). asyncio.gather behaves the same by default, and with return_exceptions=True it collects every result and error, much like Promise.allSettled. asyncio.TaskGroup (Python 3.11+) is the real difference: when one task fails, the rest of the group is cancelled, and the failures are raised together as an ExceptionGroup. That’s usually what you want when partial results are useless.

Takeaway

Coming from TypeScript, the biggest shift isn’t syntax. It’s where the guarantees live. In both languages, types vanish at runtime. Real safety comes from validating at the edges (Pydantic, the way Zod does on the front end), closing resources deterministically (with) and respecting the event loop. Points 6 and 8 are the two I expect to trip over most, so they go on my checklist.

Where this fits in my roadmap: the foundations block (Python typing, errors, environments and async, plus the FastAPI tutorial’s request/response models, errors, dependencies and testing), building toward a small typed LLM API. Next post: structured outputs, from “please return JSON” to contracts.

Over to you: if you moved from TypeScript to Python, which habit took longest to unlearn?

Sources

Python Tutorial: Virtual Environments and Packages. https://docs.python.org/3/tutorial/venv.html
PEP 668: Marking Python base environments as “externally managed”. https://peps.python.org/pep-0668/
TypeScript Handbook: The Basics, “Erased Types”. https://www.typescriptlang.org/docs/handbook/2/basic-types.html
Python docs: typing (”The Python runtime does not enforce function and variable type annotations”). https://docs.python.org/3/library/typing.html
mypy documentation (”a static type checker for Python”). https://mypy.readthedocs.io/en/stable/
Pyright (”a full-featured, standards-based static type checker for Python”). https://github.com/microsoft/pyright
Zod documentation (TypeScript-first schema validation).

https://zod.dev/

Pydantic: Why use Pydantic (type-hint-driven validation, JSON Schema). https://pydantic.dev/docs/validation/latest/get-started/why/
Pydantic: Models (post-validation guarantees). https://pydantic.dev/docs/validation/latest/concepts/models/
OpenAI: Structured model outputs (pydantic.BaseModel schemas, text_format, supported JSON Schema subset). https://developers.openai.com/api/docs/guides/structured-outputs
openai-python helpers.md (.parse() converts a Pydantic model to JSON Schema and parses the reply; output_parsed). https://github.com/openai/openai-python/blob/main/helpers.md
Python Tutorial: Errors and Exceptions (else, finally). https://docs.python.org/3/tutorial/errors.html
FastAPI: Handling Errors (HTTPException is a normal Python exception; raising it from a helper ends the request). https://fastapi.tiangolo.com/tutorial/handling-errors/
TypeScript 5.2 release notes: using declarations and explicit resource management. https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-2.html
Python Language Reference: the with statement. https://docs.python.org/3/reference/compound_stmts.html#the-with-statement
MDN: async function (runs synchronously up to the first await). https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function
Python docs: Coroutines and Tasks (calling a coroutine doesn’t schedule it; create_task; gather; TaskGroup and ExceptionGroup; to_thread and the GIL). https://docs.python.org/3/library/asyncio-task.html
Python docs: Developing with asyncio (”Running Blocking Code”; “Detect never-awaited coroutines”). https://docs.python.org/3/library/asyncio-dev.html
Python docs: Event loop, run_in_executor with ProcessPoolExecutor. https://docs.python.org/3/library/asyncio-eventloop.html
Node.js: Don’t Block the Event Loop (or the Worker Pool). https://nodejs.org/learn/asynchronous-work/dont-block-the-event-loop
FastAPI: Concurrency and async / await (plain def path operations run in a threadpool). https://fastapi.tiangolo.com/async/
MDN: Promise.all(). https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all
MDN: Promise (no first-class cancellation; use AbortController). https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
MDN: Promise.allSettled(). https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled

Sources checked Oct 18, 2026.

Originally published on The Path to AI FDE on Substack. Subscribe for the next issue: https://dhanushkumarsivaji.substack.com/subscribe

If this was useful, subscribe to The Path to AI FDE for the next issue.

The path to AI FDE #01: Anatomy of an LLM app request

Dhanush Kumar Sivaji — Mon, 19 Oct 2026 19:22:12 +0000

Originally published on my newsletter, The Path to AI FDE. Subscribe free to get each issue by email.

I've spent 8+ years building React/TypeScript front ends. Now I'm learning applied AI in public, working toward AI Engineer, then AI Solution Architect, and (I hope) AI Forward Deployed Engineer. Before tokens, RAG or agents, I wanted one map: what actually happens when a user clicks "Summarize"?

Each step below gets its own figure first, then the short version in words.

Before the call: steps 1–4

1. The client sends a request. From the browser's side, an LLM feature is just an HTTP call to your backend: POST /summarize with the user's text, plus a conversation id if it's a chat. OpenAI's docs say never to expose an API key in client-side code, so the key lives on your server and only the server talks to the provider.

2. Your API validates the input. FastAPI checks the body against a Pydantic model. Invalid data never reaches your handler: FastAPI raises a RequestValidationError, and its default handler returns HTTP 422. Once validation passes, Pydantic guarantees the fields match their declared types. It's the cheapest place to stop bad input.

3. You assemble the prompt. The model sees only what you send. Anthropic's Messages API is stateless, so you resend the full history every time. OpenAI also calls each request independent and stateless, and offers optional server-side conversation state, but notes that earlier input tokens in the chain are still billed on each turn. Later in this series, retrieved documents (RAG) join the prompt here.

4. Count tokens and check the budget. Models read tokens, not characters, and tokenizers are tied to the model, so counts differ between models. System prompt, tool definitions, history and the output all share the context window. If the input alone is over the limit, the request fails (Anthropic returns a 400, "prompt is too long"). Estimate tokens before sending and set a maximum output length.

Inside the model: steps 5–6

5. Prefill: the model reads all the input at once. Inference has two phases. In prefill, the model processes every input token in parallel. It's compute-heavy, and it's the main model-side driver of time to first token (TTFT). Network time and provider queueing add to that pause.

6. Decode: one token at a time. Then the model generates autoregressively: score every token in the vocabulary, pick one, append it, repeat. A KV cache keeps earlier work so each step doesn't start over, and the per-step pace is the time per output token. It stops at an end-of-sequence token, a stop sequence or the token limit.

How it picks is a sampling setting: greedy, or sampled with temperature, top-k or top-p. Don't assume those settings exist on every model: Anthropic's docs say Claude 4.7 and later return a 400 if you set them to non-default values.

After the call: steps 7–9

7. Stream the tokens back. OpenAI and Anthropic both stream tokens as server-sent events (SSE) while decoding is still running, so steps 6 and 7 overlap. Two catches from the docs: an error can arrive mid-stream, after a 200 status, and partial output is harder to moderate than a finished response.

8. Validate the output too. Structured outputs constrain the model to a JSON Schema you supply (OpenAI supports a subset), but OpenAI's guide says they can still contain mistakes, and documents two cases to handle: refusals and responses cut off at the token limit. If you stream, validate once the stream completes. FastAPI's response_model is a second check: a mismatch becomes a server error, not wrong data. A valid shape isn't a correct answer; evals cover correctness later in this series.

9. Retry safely, then record. The same status can mean different things: a 429 can be a rate limit, but Anthropic also uses it when you hit your monthly spend cap, so read the error body before retrying. Never retry quota or billing errors. For transient errors, honor Retry-After when present, otherwise use exponential backoff with jitter, and cap attempts and total time. Official SDKs already retry some errors (Anthropic's twice by default), so count that in your limits rather than nesting loops. Then log input/output token usage, latency and the provider's request id. Those records feed debugging, cost tracking and, eventually, evaluation.

Takeaway

As a front-end engineer, I found this oddly familiar: it's a request lifecycle with an unusual backend in the middle. Two of the nine steps are the model. The other seven are contracts, budgets, streams and failure handling, the parts an engineer owns. My working rule from here on: treat the model as an unreliable, metered, remote dependency, and put validation on both sides of it.

Where this fits in my roadmap: this post is the map I'm following into the foundations block: Python typing and async, FastAPI and Pydantic, and LLM mechanics (tokens, context, inference). Next post: Python for AI backends: a TypeScript engineer's refresher.

Over to you: which of these nine steps broke first in the first LLM feature you built?

Sources

OpenAI API reference: Authentication (don't expose API keys in client-side code). https://developers.openai.com/api/reference/overview
FastAPI: Handling Errors (RequestValidationError and its default handler). https://fastapi.tiangolo.com/tutorial/handling-errors/
FastAPI source: exception_handlers.py (default validation handler returns 422). https://github.com/fastapi/fastapi/blob/master/fastapi/exception_handlers.py
Pydantic: Models (post-validation type guarantees). https://pydantic.dev/docs/validation/latest/concepts/models/
Anthropic: Working with the Messages API (stateless, resend full history; sampling parameters on Claude 4.7+). https://platform.claude.com/docs/en/build-with-claude/working-with-messages
OpenAI: Conversation state (requests independent and stateless; optional state management; earlier input tokens still billed). https://developers.openai.com/api/docs/guides/conversation-state
Hugging Face LLM Course, Ch. 2: Tokenizers (subword tokenization, "token" + "ization"). https://huggingface.co/learn/llm-course/chapter2/4
Anthropic: Context windows (what counts toward the window; oversize input returns 400; token counting). https://platform.claude.com/docs/en/build-with-claude/context-windows
Hugging Face LLM Course, Ch. 1: Deep dive into text generation inference (prefill/decode, TTFT, TPOT, KV cache, sampling, stop conditions). https://huggingface.co/learn/llm-course/chapter1/8
Vaswani et al., "Attention Is All You Need" (2017), the transformer/attention paper. https://arxiv.org/abs/1706.03762
OpenAI: Streaming API responses (SSE, moderation risk for partial output). https://developers.openai.com/api/docs/guides/streaming-responses
Anthropic: Streaming Messages (SSE). https://platform.claude.com/docs/en/build-with-claude/streaming
OpenAI: Structured model outputs (schema adherence, supported subset, refusals, incomplete responses, mistakes still possible). https://developers.openai.com/api/docs/guides/structured-outputs
FastAPI: Response Model (server error instead of invalid returned data). https://fastapi.tiangolo.com/tutorial/response-model/
Anthropic: Errors (429 incl. spend caps, 402 billing, 400 spend limit, 529 overloaded, SDK retries, mid-stream errors after a 200, request ids). https://platform.claude.com/docs/en/api/errors
OpenAI: Rate limits (429/503, Retry-After, backoff with jitter, cap retries, nested SDK retries, don't retry quota/billing errors). https://developers.openai.com/api/docs/guides/rate-limits

Sources checked Oct 18, 2026. Provider APIs change quickly. Re-check model-specific details (status codes, sampling support) right before publishing.

Originally published on The Path to AI FDE on Substack. Subscribe for one diagram-led post every weekday: https://dhanushkumarsivaji.substack.com/p/the-path-to-ai-fde-01-anatomy-of

If this was useful, subscribe to The Path to AI FDE for the next issue.

I built kerf-cli because Claude Code told me not to worry about cost

Dhanush Kumar Sivaji — Wed, 08 Apr 2026 01:07:15 +0000

A few weeks ago I logged into Claude Code, typed /cost, and got back this:

With your Claude Max subscription, no need to monitor cost.

Two days later I had used 91% of my weekly limit in a single morning of work. I had no idea which session, which project, or which model was responsible. I tried ccusage (which is great) and it gave me totals — but I wanted to ask questions like "which of my projects is eating Opus tokens unnecessarily?" and "what's my actual cache hit rate over the last 30 days?" Those answers weren't there.

So I built kerf-cli — a local-first cost intelligence tool for Claude Code. This post is about why it exists, what it does, and what I learned about Claude Code billing along the way.

The actual problem

Anthropic gives you a lot of data and almost no analytics on top of it. Every Claude Code session is logged to ~/.claude/projects/<encoded-cwd>/<session-id>.jsonl with full token breakdowns: input, output, cache_read, cache_creation, model, timestamp, git branch. The data is rich. The tooling on top of it is thin.

Here's what existed when I started:

Claude Code's /cost command — current session only, and it actively discourages Max subscribers from looking
Anthropic's web console — org-level dashboards for Teams/Enterprise, nothing for solo developers on Pro/Max
ccusage — excellent for quick reports, but parses JSONL on every invocation with no persistence
A handful of other CLIs and menu-bar apps — most are read-only reporters

What I wanted was three things none of them offered:

A persistent analytical layer I could query with SQL
Real budget enforcement that blocks Claude Code when I exceed a cap, not just a warning
Concrete optimization recommendations — not "you spent $47" but "switch these 12 sessions from Opus to Sonnet and you'll save $140/month"

How kerf works

Kerf is a TypeScript CLI built on commander, ink, and better-sqlite3. The architecture is dead simple: it reads Claude Code's existing JSONL session files, ingests them into a local SQLite database, and then every command and the web dashboard query that database directly. You run kerf sync once and it ingests every Claude Code session you've ever had. Subsequent syncs are incremental — only changed files are re-parsed. Then everything else is fast queries against the local DB.

The commands that matter

kerf summary

The bread and butter — what did I spend?

$ kerf summary --period week --by-project

For week (Apr 1 → Apr 7):
  Total cost:     $178.04
  Sessions:       25
  Tokens:         454.0M
  Cache hit:      98%

By project:
  projects       $117.00  (66%)   5 sessions
  subagents       $42.50  (24%)  14 sessions
  kerf            $14.54   (8%)   4 sessions

kerf query

The SQL escape hatch I built mostly for myself:

$ kerf query "SELECT date(timestamp) as day, ROUND(SUM(cost_usd), 2) as cost
              FROM messages
              WHERE timestamp > date('now', '-7 days')
              GROUP BY day ORDER BY day DESC"

day         cost
2026-04-07  $46.62
2026-04-06  $11.84
2026-04-04  $33.51
2026-04-03  $28.20
2026-04-02  $14.33

The --examples flag prints a dozen useful queries to copy. The --schema flag prints the database schema. Writes are blocked — only SELECT statements are allowed.

kerf efficiency

The command that actually saves money. This is the one I use every Monday morning.

This was the moment kerf paid for itself. I had been blindly using Opus for everything because it was the default. The analyzer pointed out that most of my Opus sessions had patterns that would have been fine on Sonnet — file edits, small refactors, dependency bumps. I switched those workflows, cut a meaningful chunk of my monthly Claude bill, and noticed zero quality difference.

kerf budget + kerf init --enforce-budgets

The killer feature. kerf budget set 50 --period weekly sets a cap, then kerf init --enforce-budgets installs a Claude Code PreToolUse hook that runs kerf budget check before every tool call. If you're over budget, the hook returns exit code 2 and Claude Code blocks the action.

This is the difference between knowing you blew your budget and not being able to. Other tools warn. Kerf enforces.

kerf dashboard

The local web UI for visual people. Opens at http://localhost:3847 — SQLite-backed so queries are sub-100ms, three killer-features cards (budget, efficiency, cache) front and center, sortable session table with drill-down, stacked cost-over-time chart by model. Zero auth, zero cloud, zero data leaving your machine. That's the screenshot at the top of this post.

What I learned about Claude Code billing

A few non-obvious things from spending too much time staring at JSONL files:

1. Cache reads can be 60–80% of your total cost. This was the biggest surprise. Cache reads are billed at 10% of standard input rate, which sounds cheap until you realize you're caching 50K tokens per turn and reading them on every message. Optimizing your CLAUDE.md and reducing cache invalidation was the biggest single lever I found.

2. Opus is the default and it almost never needs to be. I ran kerf efficiency on a month of data: 90% of my Opus tokens were on sessions that had no complexity signal (no debugging, no architecture decisions, no large refactors — just file edits and small fixes). Switching them to Sonnet was a 4x cost reduction with no measurable quality drop.

3. Claude Code's JSONL streams partial usage updates. When you parse them, you have to keep the MAX value per field across duplicate message IDs, not the latest value. I learned this the hard way — my v2.1 parser was undercounting input tokens because it kept the "last" entry instead of the "max," which meant the final zero-input chunk overwrote the real numbers from earlier chunks. Fixed before launch, but it's a subtle trap anyone parsing these logs will hit.

4. The 5-hour billing window is real but Anthropic doesn't expose it clearly. Max subscribers are billed against a rolling 5-hour window, not a daily quota. If you don't track this, you can get surprised when the window rolls over mid-session.

Technical decisions I'd defend

SQLite over a JSON file. JSON is fine for ccusage's "read once, compute, discard" model. For an analytics layer you want sub-100ms queries, joins, aggregations, and a schema migration story. SQLite via better-sqlite3 is the right tool.
Local-only over cloud-first. The moment you add cloud sync, you need auth, storage, privacy controls, GDPR compliance, a business model. None of that serves the primary use case of "show me where my money went." Kerf is local-first on purpose. A hosted team tier is on the roadmap but will always be optional.
Ink for the terminal UI. React components in the terminal feel weird at first but the composability is worth it.
Hooks as the enforcement mechanism. Claude Code ships a native hook system. Using hooks means kerf doesn't have to intercept or proxy Claude Code's traffic — it just responds to events Claude Code already emits.

What's next

v1 is focused on doing one thing well: Claude Code observability for individual developers. The roadmap from here:

v2.x: Cursor and Codex support
v2.x: Slack/Discord alerts on budget thresholds
v2.x: GitHub Actions integration for cost gates on PRs
v3.x (paid team tier): cloud sync, team aggregation, SSO

But none of that ships before I'm sure v1 is rock solid. The CLI will always be free and MIT licensed.

Try it

npm install -g kerf-cli
kerf sync
kerf summary

GitHub: github.com/dhanushkumarsivaji/kerf-cli

Show HN discussion: news.ycombinator.com/item?id=47683060

If you've hit billing surprises with Claude Code, I'd love to hear about them in the comments. The more weird patterns I see, the better the analyzer gets.