DEV Community: Sayed Ali Alkamel

Prompt Chaining: Fixed Steps That Each Feed the Next

Sayed Ali Alkamel — Tue, 07 Jul 2026 10:36:33 +0000

Short version: Prompt chaining splits a task into a fixed sequence of steps, where each model call works on the output of the last. Anthropic calls it prompt chaining, and Google calls it the sequential pipeline. It trades speed for accuracy, and it is the easiest agent pattern to debug because you always know where the data came from.

What is prompt chaining?

Prompt chaining decomposes a task into a sequence of steps, where each LLM call processes the output of the previous one (Anthropic). Google frames the same shape as a sequential pipeline: specialized agents run in a fixed, linear order, and the output of one agent is the direct input to the next (Google Cloud).

Think of an assembly line. A parser turns a raw PDF into text, an extractor pulls out structured fields, and a summarizer writes the synopsis (Google Developers). Each station does one job well.

The gate: your cheapest safeguard

Anthropic adds one detail that makes chaining reliable: a gate. Between steps you can add a programmatic check that confirms the process is still on track before it continues (Anthropic). If the outline fails a criterion, you stop before wasting a call on a bad draft.

A gate is not a model call. It is plain code: a length check, a schema validation, a regex, a test. That is why chaining is cheap to make safe.

How it actually works

State is the whole trick. In Google ADK, each step writes its result to a shared session state with an output key, and the next agent reads from that key (Google Developers). A sequential workflow agent runs the steps in order and does not consult a model to decide what comes next, which keeps it predictable and cheap (Google Cloud).

# Google ADK, sketch
parser     = LlmAgent(name="Parser", output_key="raw_text")
extractor  = LlmAgent(name="Extractor", instruction="Use {raw_text}", output_key="fields")
summarizer = LlmAgent(name="Summarizer", instruction="Use {fields}")
pipeline   = SequentialAgent(sub_agents=[parser, extractor, summarizer])

When to use it

Use prompt chaining when the task splits cleanly into fixed subtasks and you want higher accuracy per step (Anthropic). Classic examples: write marketing copy, then translate it. Or write an outline, check it against criteria, then write the document from the approved outline.

It shines for highly structured, repeatable processes where the order never changes, such as a data extraction, cleaning, and loading pipeline (Google Cloud). The payoff is that each call is an easier task, so the model makes fewer mistakes.

When not to use it

Do not chain when the steps are not known in advance. If the number and nature of subtasks depend on the input, the rigid pipeline cannot adapt, and you want orchestrator-workers instead. Do not chain when latency is the priority and the subtasks are independent. Run them in parallel. And if a single call already produces a good answer, chaining just adds round trips for nothing.

Known problems

The trade-off is rigidity. Google is explicit: the fixed structure makes it hard to adapt to dynamic conditions or to skip unneeded steps, which can cause inefficient processing or push up cumulative latency if a step that was not needed is slow (Google Cloud).

There is also error propagation. Because each step consumes the last step's output, a mistake early in the chain flows downstream. Gates catch some of this, but a step that produces plausible-but-wrong output can still poison everything after it. And every extra step adds latency, so a long chain feels slow even when each call is fast.

Three things to know before you start

Put a gate after any risky step. A cheap code check beats an expensive bad draft.
Name your state clearly. Downstream steps read named outputs, so a vague key becomes a silent bug.
Keep the chain short. Each link adds latency and one more place to fail. If you need branching, you have outgrown a straight chain.

FAQ

Is prompt chaining the same as a sequential pipeline?
Yes. Anthropic uses "prompt chaining" and Google uses "sequential pipeline" for the same fixed, linear flow where each step feeds the next.

Does chaining need multiple agents?
Not necessarily. It can be several prompts to one model or several specialized agents. What defines it is the fixed order and the handoff of output to input.

What is a gate in prompt chaining?
A programmatic check between steps that confirms the work is on track before continuing. It is code, not a model call.

When should I switch to orchestrator-workers?
When you cannot predict the steps ahead of time. Chaining is for fixed sequences. Orchestrator-workers decides the subtasks at runtime.

Sources

Anthropic, Building Effective Agents: https://www.anthropic.com/engineering/building-effective-agents
Google Developers Blog, Developer's guide to multi-agent patterns in ADK: https://developers.googleblog.com/developers-guide-to-multi-agent-patterns-in-adk/
Google Cloud Architecture Center, Choose a design pattern for your agentic AI system: https://docs.cloud.google.com/architecture/choose-design-pattern-agentic-ai-system

The Augmented LLM: The Building Block Every Agent Sits On

Sayed Ali Alkamel — Mon, 06 Jul 2026 14:53:34 +0000

Short version: The augmented LLM is a single model given three extra abilities: retrieval, tools, and memory. Anthropic calls it the basic building block of every agentic system. Google calls the same idea a single-agent system. Master this before you touch multi-agent designs, because every other pattern is built on top of it.

What is the augmented LLM?

The augmented LLM is a language model enhanced with retrieval, tools, and memory, where the model itself drives those abilities: it writes its own search queries, picks the right tool, and decides what to keep in context (Anthropic). Google describes the equivalent single-agent system as one model, a defined set of tools, and a system prompt that lets the agent interpret a request, plan steps, and choose tools on its own (Google Cloud).

The important word is augmented. A bare model answers from its weights. An augmented model can pull in fresh facts, act on the world, and remember across turns.

The three augmentations

Retrieval lets the model fetch information it was not trained on. It generates a query, reads the results, and grounds its answer in them. Tools let the model call functions and APIs, then read the output to decide what to do next. Memory lets the model carry context forward, so it does not repeat itself or lose the thread (Anthropic).

Anthropic's advice on all three is the same: tailor them to your use case, and put a clean, well-documented interface in front of them. One way to wire tools in is the Model Context Protocol, which gives the model a standard client for a growing set of third-party tools.

How it actually works

A single augmented agent runs a short loop. It reads the request, decides whether it needs a tool or a search, calls it, reads the result, and either answers or takes another step. There is no second agent and no orchestration layer. The model's own reasoning is the controller.

That simplicity is the point. Google recommends starting here so you can focus on the three things that actually decide quality: the core logic, the system prompt, and the tool definitions (Google Cloud). Anthropic makes the tool interface a first-class concern and suggests spending as much effort on the agent-computer interface as teams spend on human interfaces.

When to use it

Use a single augmented LLM when the task needs external data or actions but still fits one clear responsibility. A customer support agent that looks up an order, or a research helper that calls an API and summarizes the result, are ideal cases. A non-agentic system cannot do these, because it cannot use tools or run a multi-step plan (Google Cloud).

It is also the right first move for any prototype. You can ship a single agent in a short time, learn where it breaks, and only then decide whether a heavier pattern earns its cost.

When not to use it

Skip the augmentation when a single plain model call already answers the request. Summarizing a document, translating text, or classifying feedback usually do not need tools or memory, so an agent adds cost for nothing (Google Cloud).

Also move on when one agent starts juggling several distinct jobs. Google's own guidance is blunt: a single agent becomes a jack of all trades and a master of none as you pile on tools and responsibilities (Google Developers). That is your signal to split into specialists.

Known problems

The main failure mode is overload. As you add tools and task complexity, a single agent's performance drops: you see higher latency, wrong tool choices, or tasks that never finish (Google Cloud). You can often push the limit further with better reasoning structure, such as the ReAct loop, but that only buys headroom, it does not remove the ceiling.

The second problem is a weak tool interface. If a tool is hard for a person to use from its description alone, it is hard for the model too. Anthropic found it spent more time refining tools than prompts, and that small fixes, like requiring absolute file paths instead of relative ones, removed whole classes of errors.

Three things to know before you start

The system prompt is the product. It defines the agent's task, persona, and the exact conditions for using each tool. Vague prompts produce vague agents.
Document tools like you would for a junior teammate. Include example usage, edge cases, and clear boundaries. Test real inputs and iterate on the descriptions.
Measure before you scale. Prove the single agent falls short on real tasks before adding a second agent, because multi-agent designs cost far more to run and maintain.

FAQ

Is an augmented LLM the same as an agent?
It is the smallest agent. Anthropic treats it as the base building block; Google treats the same shape as a single-agent system. Larger patterns compose several of these.

Is retrieval-augmented generation (RAG) an augmented LLM?
RAG is the retrieval part of it. A full augmented LLM adds tools and memory on top, and lets the model decide when to use each.

How is this different from a chatbot?
A chatbot answers from the model alone. An augmented LLM can fetch live data, call APIs, and act, then use those results in its answer.

When should I move to multiple agents?
When one agent must handle several distinct responsibilities, or when adding tools starts to hurt accuracy and latency. That is the point where specialists win.

Sources

Anthropic, Building Effective Agents: https://www.anthropic.com/engineering/building-effective-agents
Google Cloud Architecture Center, Choose a design pattern for your agentic AI system: https://docs.cloud.google.com/architecture/choose-design-pattern-agentic-ai-system
Google Developers Blog, Developer's guide to multi-agent patterns in ADK: https://developers.googleblog.com/developers-guide-to-multi-agent-patterns-in-adk/
Model Context Protocol: https://modelcontextprotocol.io/

The New SDLC for Project Managers: Vibe Coding vs Agentic Engineering

Sayed Ali Alkamel — Sun, 05 Jul 2026 13:40:40 +0000

Short version: AI has split software delivery into two modes. Vibe coding is fast and cheap for throwaway work, but expensive to run and risky to ship. Agentic engineering wraps the AI in specifications, tests, and human review, so the output is dependable. If you manage delivery, your estimates, budgets, risk register, and staffing plan all move.

What is vibe coding, and why should a project manager care?

Vibe coding is a way of building software where you describe what you want in plain language, accept whatever the AI produces, and paste errors back until it runs. The term was coined by Andrej Karpathy in February 2025 (karpathy on X), who described giving in to the vibes and forgetting the code even exists. It is fast and it feels like magic. It is also, by definition, code that no human has fully read.

Agentic engineering is the disciplined end of the same idea. The AI still writes the implementation, but inside human-designed specifications, automated tests, and feedback loops that check the work. These are two ends of one spectrum (Wikipedia). The differentiator is not whether your team uses AI. It is how much verification surrounds the output. Rule of thumb: a weekend prototype can be pure vibe coding, anything a customer depends on needs agentic engineering.

What changes in the software development life cycle?

The phases you know (requirements, design, build, test, review, maintain) do not disappear. Their proportions change. AI compresses implementation hard: work that took weeks can take hours. It does almost nothing for the human-paced parts, which are requirements, architecture, and verification. So the bottleneck moves. Writing code stops being the slow step. Deciding exactly what to build, and confirming it was built correctly, becomes the slow step.

This leads to what the Google whitepaper calls the factory model: the team's real output is not the code, it is the system that produces the code. A factory manager does not assemble each unit by hand, they design the line and own quality control. Requirements become a live conversation with the AI rather than a document thrown over a wall. Design stays human, because trade-offs (consistency versus availability, build versus buy) depend on business context. Tests and evals become the contract that tells the AI what "correct" means. And legacy code that was once "too risky to touch" becomes safe to modernize.

Who does what now: conductors, orchestrators, and the harness

Developers move between two modes. In conductor mode they work in real time with the AI, reviewing each change as it appears, which suits complex or unfamiliar code. In orchestrator mode they define a task, hand it to one or more agents running in the background, and review the result later, which suits well-scoped work like bug fixes, migrations, and test generation. The same person uses both in a day.

The model itself is only one part. Around it sits the harness: the rule files, tools, sandboxes, and checks that let the model finish real work. A useful equation from the paper is Agent equals Model plus Harness. This matters for a plain reason. When an agent gets something wrong, the instinct is to blame the model, but most failures trace back to a missing tool, a vague rule, or a missing check. Getting the context right (the specs, conventions, and constraints you feed the agent) does more for quality than swapping models. Treat those context files as team assets: reviewed, versioned, and owned by named people.

What it costs: the CapEx and OpEx flip

For a project manager, the money story is the one that lands in a budget. Vibe coding looks cheap because the upfront cost is near zero. The hidden cost is operational. Every prompt spends tokens, and a loop of "fix your own mistake" burns tokens with a low first-pass hit rate. Unstructured AI code becomes a maintenance tax when a bug surfaces months later, and unverified code becomes a security bill in production.

Agentic engineering reverses this. You pay more upfront to design specs, tests, and context, and the marginal cost of each new feature drops. Two levers keep running costs down. Context engineering means sending a dense, relevant payload instead of the whole codebase every time. Model routing means sending hard tasks to a large model and cheap, routine tasks (a unit test, a small review) to a smaller, cheaper one. The trade is a familiar one: higher capital cost now for lower operating cost later.

Three things for project managers to hold onto

The 80% problem is an estimation trap. An agent can produce roughly the first 80% of a feature quickly. The last 20% (edge cases, error handling, integration, correctness) needs judgment the model often lacks, and that is where the real time goes. Plan schedules around the hard 20%, not the easy 80%.
Velocity is not the same as speed. In METR's controlled trial, experienced developers using AI took 19% longer while believing they were about 20% faster (METR). Time shifts from writing to reviewing and correcting. Measure delivered, verified work, not lines produced or how fast it feels.
AI amplifies your engineering culture. Teams with strong tests, clear standards, and real code review get much more from AI. Teams without them ship faster into a bigger mess. Adoption alone is not a strategy: usage hit record levels in the Stack Overflow 2025 survey even as trust in AI output fell.

FAQ

Is vibe coding safe for production?
Not on its own. It is ideal for prototypes and internal tools. For anything customers depend on, add the specs, tests, and review that define agentic engineering.

Will AI make my team smaller?
It changes the mix more than the size. Smaller teams can take on larger problems, but the scarce skill shifts to specification, evaluation, and architecture, not typing code.

What is the single best first step?
Set up a project rules file (an AGENTS.md or equivalent) with your stack, conventions, and hard rules, then write tests and evals before generating code. Together they are the contract with the AI.

How do I explain the risk to leadership?
Say the team practices agentic engineering, with AI writing code under human-designed constraints and test coverage confirming correctness. That is a very different conversation than "we are vibe coding the payment system."

Sources

Andrej Karpathy, original "vibe coding" post (Feb 2025): https://x.com/karpathy/status/1886192184808149383
Vibe coding, overview and reception (Wikipedia): https://en.wikipedia.org/wiki/Vibe_coding
METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity" (Jul 2025): https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
Stack Overflow 2025 Developer Survey (AI adoption and sentiment): https://survey.stackoverflow.co/2025/
Grounding material: Osmani, Saboo, and Kartakis, "The New SDLC With Vibe Coding," Google (May 2026)

Dart FFI for Flutter: Call Native C Directly (Lessons from camera_pro)

Sayed Ali Alkamel — Sat, 04 Jul 2026 02:43:18 +0000

Short version: Dart FFI lets your Flutter app call C functions directly, as a normal synchronous call, instead of passing serialized messages over a platform channel. That is a big deal when you have an existing C or C++ library, or hot per-frame native work. Here is what it is, when to use it, and what I ran into building camera_pro, a camera package with a shared C core.

What is Dart FFI?

FFI stands for foreign function interface. Per the Dart docs, apps on the Dart Native platform can use the dart:ffi library to call native C APIs and to read, write, allocate, and deallocate native memory (dart.dev). In plain terms: you point Dart at a compiled C library, hand it a function signature, and call the function.

You do not have to write those bindings by hand. For anything beyond a couple of functions, package:ffigen generates the Dart wrappers straight from your C header files (dart.dev).

One boundary to remember: dart:ffi runs only on the Dart Native platform. There is no FFI on Flutter web.

FFI or platform channels?

Both let Flutter reach code the Dart SDK does not ship. They work very differently.

A platform channel passes messages between Dart and the host platform. Those messages and responses travel asynchronously, and their values are serialized into a binary format on the way across and deserialized on the other side (docs.flutter.dev). Great for occasional calls like reading the battery level or opening a share sheet.

Dart FFI is a direct call into C. No channel, no per-call serialization, and you can share native memory with the C side. That is what makes it the right tool for tight, frequent, CPU-heavy work.

Rule of thumb: reach for a channel when you occasionally call a platform API. Reach for FFI when you wrap an existing C or C++ library, or when you run native code on every frame.

How the native C actually ships

The old friction with FFI was shipping the compiled library. You had to build the .so, .dylib, or .dll per platform and bundle it yourself.

Build hooks fix that. Formerly called native assets, build hooks let a package carry native code that is transparently built, bundled, and made available at runtime (dart.dev). As of Flutter 3.38, the Flutter docs recommend the package_ffi template with build hooks for C interop, and mark the older plugin_ffi approach as legacy (docs.flutter.dev).

The moving parts, which are the actual dependencies of camera_pro:

ffigen writes the Dart bindings from your headers, a build hook using native_toolchain_c compiles the C at build time, code_assets and hooks bundle it, and package:ffi gives you helpers like Arena for native memory.

What FFI looked like in camera_pro

camera_pro is a Flutter camera package built on a shared C and C++ core over Dart FFI. The C core does the pixel work: YUV to RGBA conversion, histogram, focus peaking, zebra, false color, and a waveform monitor, run on each preview frame. The same bit-exact C runs on macOS, iOS, Linux, and Windows behind one hardware abstraction layer.

The classic FFI shape, which is what ffigen produces for you:

import 'dart:ffi';
import 'package:ffi/ffi.dart'; // toDartString, Arena, using

// Names here are illustrative.
final lib = DynamicLibrary.open('libcamera_pro.dylib'); // .so or .dll elsewhere
final coreVersion = lib
    .lookup<NativeFunction<Pointer<Utf8> Function()>>('camera_pro_version')
    .asFunction<Pointer<Utf8> Function()>();

print(coreVersion().toDartString()); // "0.0.1"

That lookup(...).asFunction() pattern is the core of dart:ffi (docs.flutter.dev). With build hooks and code assets you can skip DynamicLibrary.open and mark the binding with the @Native annotation, letting the toolchain resolve the symbol from the bundled asset (dart.dev).

Why bother? Because the calls are frequent and the payloads are large. On an Apple M1 Pro, the core converts a 1080p YUV420p frame to RGBA in about 0.66 ms. Copying full frames through a serialized channel on every frame would be a poor fit for that.

Three things to know before you start

C++ needs extern "C". FFI binds to C symbols, so C++ functions must be exported as C. Also mark them with __attribute__((visibility("default"))) __attribute__((used)) so link-time optimization does not strip them (docs.flutter.dev).
You own native memory. Anything you allocate for the C side, you free. package:ffi makes this safe with an arena that releases everything when the block ends:

   using((arena) {
     final buf = arena<Uint8>(width * height * 4); // freed on exit
     // hand buf to C, read results back
   });

No web. Since dart:ffi is Dart Native only, plan a fallback. camera_pro uses a conditional import to keep dart:ffi off the web build and reimplements the same kernels in pure Dart for the browser.

FAQ

Is Dart FFI faster than platform channels?
For frequent or CPU-heavy calls, usually yes, because it is a direct synchronous call with no per-call serialization. For a rare one-off platform call, the difference does not matter.

Do I have to write the bindings myself?
No. package:ffigen generates them from your C headers.

Do I have to bundle the .so or .dylib myself?
No. Build hooks compile and bundle the native code for you.

Does Dart FFI work on Flutter web?
No. dart:ffi runs only on the Dart Native platform. Ship a pure-Dart or JS-interop fallback for web.

Sources

Dart, C interop using dart:ffi: https://dart.dev/interop/c-interop
Flutter, platform channels: https://docs.flutter.dev/platform-integration/platform-channels
Flutter, C interop and the legacy plugin note: https://docs.flutter.dev/platform-integration/macos/c-interop
ffi package (native memory helpers): https://pub.dev/packages/ffi
ffigen package (binding generator): https://pub.dev/packages/ffigen
camera_pro on pub.dev and GitHub: https://pub.dev/packages/camera_pro and https://github.com/sayed3li97/camera_pro

The LLM Preamble Problem: How RLHF Made Your Model Too Polite to Ship

Sayed Ali Alkamel — Mon, 29 Jun 2026 17:50:10 +0000

TL;DR

Every instruction-tuned model has a preamble habit: it opens with "Certainly!", "Great question!", or "Of course!" before answering you.

RLHF is the root cause. Human raters rewarded warm, thorough responses during training. The model learned that politeness signals quality.

Each preamble adds 10 to 30 tokens of pure waste. At 1 million interactions per month, that burns $200 in API budget before your users see a single useful word.

Four fixes exist. System-prompt suppression, structured output (JSON mode), constrained decoding, and post-process stripping. Each fits a different situation.

Why Every Instruction-Tuned LLM Has a Preamble Habit
The RLHF Root Cause
What the Preamble Actually Costs
Four Fixes for the LLM Preamble, Ranked by Reliability
The Skeptic's Objection
What This Means For You
Questions developers are actually asking about LLM preamble

Somewhere around 3 AM in a data center, a human annotator reads two AI responses and picks the friendlier one. Not the faster one. Not the more accurate one. The one that says "Great question!" before answering. They do this tens of thousands of times. A reward model learns their preferences. And now, in every production application you ship, the first thing your LLM says to users is "Certainly, I'd be happy to help with that!" before it actually helps with anything.

This is not a flaw your model's creator forgot to fix. It is the intended product of how instruction tuning works, amplified by how humans behave when asked to rate AI responses. The politeness is not a bug. It is a feature that made demos feel warmer and somehow survived all the way into your production logs.

Why Every Instruction-Tuned LLM Has a Preamble Habit

A base language model, given "What is the capital of France?", completes the token stream. "Paris is the capital of..." An instruction-tuned model, trained to behave like a helpful assistant, answers differently: "Great question! The capital of France is Paris."

That gap exists because of Reinforcement Learning from Human Feedback. RLHF takes a base model, runs it through a preference-optimization loop with human annotators, and fine-tunes it to maximize the reward model's approval score. Research cited in alignment literature (Singhal et al., 2023) found that response length is a significant latent optimization target inside RLHF reward models. Responses that opened with a warm acknowledgment before the actual content scored consistently higher than responses that answered immediately, even when the core information was identical.

The model did not learn to be genuine. It learned to perform helpfulness. "If I say 'Certainly!' before I answer, the preference model gives me gold stars."

The RLHF Root Cause

A March 2026 paper by Liu et al. quantified what researchers now call the alignment tax: RLHF-aligned models show measurable response homogenization compared to their base versions. On TruthfulQA, 40% of questions produced semantically identical responses across multiple independent samples from the instruct-tuned model. The base model on the same benchmark: 0%.

A three-stage ablation isolated the cause. Base model: 0% homogenization. After SFT only: 1.5%. After SFT plus DPO: 4%. The Direct Preference Optimization stage, not the supervised fine-tuning, drives the collapse in response diversity. This is where the preamble gets baked in. [INTERNAL LINK: earlier article on LLM tokenization and BPE]

A more familiar way to state this: the model says "Of course!" not because it understands enthusiasm, but because responses beginning that way were repeatedly preferred during alignment training. It is not a conversational habit. It is a statistical artifact.

Training Stage	Homogenization Rate
Base model	0.0%
After SFT only	1.5%
After SFT + DPO	4.0%

Source: Liu, "The Alignment Tax: Response Homogenization in Aligned LLMs," 2026

Smaller instruction-tuned models amplify this effect. With fewer parameters to represent "helpfulness," they fall back on the most statistically common pattern more aggressively. I tested the same prompt across three models last month. Llama 3.2 3B led with preamble on 9 of 10 runs. Mistral 7B Instruct: 7 of 10. A frontier model: 3 of 10. Same prompt. Very different behavior.

What the Preamble Actually Costs

Each preamble adds 10 to 30 output tokens that carry no information. This feels trivial at the API call level. The arithmetic changes when you do it at production scale.

At GPT-4o output pricing of $10 per million tokens (a legacy model as of 2026, but its price has held unchanged since OpenAI's October 2024 cut): 1 million API calls per month, 20 wasted tokens per call, equals 20 million tokens of pure "I'd be happy to help!" That is $200 per month. At 10 million calls per month: $2,000. Not company-ending numbers. But real, sustained waste on every response, month after month.

To put that in physical terms: 20 million tokens at four characters per token is 80 megabytes of text. Every month, at 1M calls, your application writes the equivalent of an 80MB document that says "Certainly!" in several hundred thousand different ways. You pay for every character.

Four Fixes for the LLM Preamble, Ranked by Reliability

No single fix works universally. The most common one (system-prompt instruction) is also the least reliable at scale.

Fix 1: System Prompt Suppression

Add an explicit negative instruction:

System: Respond directly. Do not acknowledge the question with openers like 
"Great question", "Of course", "Certainly", or "I'd be happy to help". 
Begin immediately with the answer.

This works reliably on frontier models most of the time. It fails on small instruction-tuned models (below 7B parameters) and at higher temperatures. The preamble behavior sits deep enough in the weights that a single system prompt instruction does not always override it.

Add a CI slop check to catch regressions when someone edits the system prompt:

#!/bin/bash
# slop-check.sh
SLOP="Certainly!|Of course!|Great question!|I'd be happy"
if echo "$1" | grep -qiE "$SLOP"; then
  echo "FAIL: preamble detected in LLM output"
  exit 1
fi

Fix 2: Structured Output (JSON / Tool Mode)

For structured tasks, this is the most reliable fix available. When the model is constrained to valid JSON, the preamble cannot occur. "Certainly! {" is not valid JSON. The constraint kills the behavior at the token level.

from anthropic import Anthropic

client = Anthropic()
response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system="Extract the requested data. Return only valid JSON.",
    messages=[{"role": "user", "content": "Capital of France?"}],
    tools=[{
        "name": "answer",
        "description": "Return the structured answer",
        "input_schema": {
            "type": "object",
            "properties": {
                "answer": {"type": "string"}
            },
            "required": ["answer"]
        }
    }],
    tool_choice={"type": "auto"}
)

Every major provider now offers native structured output with schema guarantees. XGrammar, as of March 2026, is the default backend for vLLM, SGLang, and TensorRT-LLM, producing constrained output at under 40 microseconds of overhead per token.

Fix 3: Constrained Decoding (Local Models)

If you are running a local model, Outlines and llama.cpp grammar constraints apply restrictions at the logit level before sampling. The token "Certainly" never enters the probability distribution. The library masks it out at generation time.

from pydantic import BaseModel
import outlines

class Answer(BaseModel):
    answer: str

model = outlines.models.transformers("mistralai/Mistral-7B-Instruct-v0.2")
generator = outlines.generate.json(model, Answer)

result = generator("What is the capital of France?")
print(result.answer)  # "Paris" -- no preamble physically possible

For Flutter developers building on-device AI with quantized models (Phi-4-mini, Gemma 3, or Qwen 3), this path via llama.cpp or MediaPipe is your most reliable option. The preamble is prevented structurally, not instructed away.

Fix 4: Post-Process Stripping

When you cannot modify the model's behavior (third-party API, fixed vendor contract), strip the preamble from the output before returning it to users.

import re

PREAMBLE_PATTERNS = [
    r"^(Certainly!|Of course!|Sure!|Great question!|Absolutely!)\s*",
    r"^(I('d| would) be happy to help[^.]*\.\s*)",
    r"^(Here is your[^:]*:\s*)",
]

def strip_preamble(text: str) -> str:
    for pattern in PREAMBLE_PATTERNS:
        text = re.sub(pattern, "", text, flags=re.IGNORECASE)
    return text.strip()

This catches known patterns. It misses novel variations. Treat the pattern list as living documentation, not a complete solution.

Technique Comparison

Technique	Reliability	Works on Prose	Cloud API	Local Model
Post-Process Stripping	Low	Yes	Yes	Yes
System Prompt Suppression	Medium	Yes	Yes	Yes
JSON / Tool Mode	High	No	Yes	Partial
Constrained Decoding	Very High	No	No	Yes

The Skeptic's Objection

The fair criticism here is: "This is just prompt engineering. Write a better system prompt."

I tested this objection on a real project: integrating Phi-3.5-mini into a banking support flow at Oman Housing Bank. The system prompt had three separate instructions to skip preamble. The model complied on roughly 70% of calls. On the remaining 30%, it produced preambles ranging from 12 to 41 tokens. At the volumes we were processing, that failure rate was not a prompting problem. It was a latency and cost problem requiring structural intervention.

The deeper issue is that preamble behavior is not a surface-level instruction-following failure. It is encoded in the model's output distribution at the weight level, as the alignment tax research confirmed. Telling the model "skip the greeting" is something it can forget. Constraining the token distribution to valid JSON is something it physically cannot disobey. One is a request. The other is an architecture decision.

What This Means For You

You are building a chatbot with a frontier API. Start with system-prompt suppression and add the CI slop check. That combination covers most cases and takes 30 minutes to implement.

You are doing structured tasks: classification, extraction, summarization into a defined schema. Use JSON mode or tool-use mode. The preamble problem disappears entirely. Do not spend engineering budget on any other fix for this use case.

You are running a local model for a privacy-first or on-device deployment. Evaluate constrained decoding via Outlines or llama.cpp grammar constraints. The upfront complexity pays back at volume, and it is the only fix with a formal correctness guarantee.

You are integrating a third-party API you do not control. Post-process stripping is your only lever. Build the pattern library, add the slop check to CI, and accept that you will be updating the list. [INTERNAL LINK: article on agentic CI/CD pipelines]

Questions developers are actually asking about LLM preamble

Why do LLMs say "Certainly!" before answering?

This behavior comes from RLHF: the reinforcement learning process that aligns base models with human preferences. Human annotators consistently scored responses with warm, acknowledging openers as higher quality than responses that answered immediately. The reward model learned this signal and the policy model maximized it. Research by Singhal et al. (2023) confirmed that response length optimization is a significant hidden factor in RLHF reward modeling, and preamble is a direct output of that optimization.

Does the preamble problem affect all models equally?

No. Smaller instruction-tuned models (under 7B parameters) exhibit preamble behavior more frequently because they have fewer representational paths to express "helpfulness" and fall back on the most statistically common pattern. A 2026 alignment study found that the effect varies significantly by model family and training recipe, from under 2% homogenization in some Mistral variants to over 28% in Qwen3-14B instruct. The size of the effect depends on how aggressively the preference optimization stage was applied.

What is the most reliable way to suppress LLM preamble?

Structured output (JSON mode or tool-use mode) is the most reliable suppression technique for structured tasks. When the model must produce valid JSON, "Certainly! {" is syntactically impossible. For local model deployments, constrained decoding via XGrammar or Outlines applies the constraint at the logit level before sampling, making preamble generation physically impossible. System-prompt instructions are the least reliable option because the behavior is encoded at the weight level, not the instruction-following level.

Can I detect LLM preamble in a CI/CD pipeline?

Yes, with a simple bash script running grep against known preamble patterns during your test suite or as a deployment gate. Check for "Certainly!", "Of course!", "Great question!", "I'd be happy to", and "I'll help you with that." This approach catches the most common variants but requires ongoing maintenance as models generate new preamble patterns. Treat it as a signal, not an exhaustive filter, and combine it with other suppression techniques for production reliability.

Does the LLM preamble cost real money at scale?

Yes. At GPT-4o output pricing of $10 per million tokens (confirmed unchanged through April 2026), 1 million API calls per month with 20 wasted tokens each burns $200 in preamble tokens. At 10 million calls, that is $2,000 per month. The cost scales linearly with call volume and proportionally with the model's output token pricing. For premium models with higher per-token costs, the number rises faster. It is not company-ending. It is real, sustained, and entirely avoidable.

The Longer View

Every generation of developers has had to compensate for some mismatch between what a system was optimized for and what they actually needed it to do.

The LLM preamble problem is that mismatch for 2025 and 2026. RLHF made models dramatically more usable for non-technical users. It also gave every model a slightly performative quality: the tendency to appear helpful in the first token, before being helpful in the subsequent ones. That gap between performance and function is not something you can always prompt your way around. It is a property of how the system was trained. The alignment tax paper was not describing a bug report. It was describing the cost of making a language model agreeable to human raters.

The developers who ship reliable LLM-powered products in this environment are the ones who understand the training's side effects well enough to design around them. Not that the model is broken. That it is doing exactly what it was trained to do, and that your application needs to account for what that training left behind.

References

Liu, M. "The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation." arXiv, March 2026. https://arxiv.org/pdf/2603.24124
Singhal, P. et al. Cited in: Zeng et al. "Verbosity is Not Veracity: Demystify Verbosity Compensation Behavior of Large Language Models." arXiv, 2024. https://arxiv.org/pdf/2411.07858
West, A. "How to Fix That Robotic AI Tone in Your LLM-Powered Features." DEV Community, April 2026. https://dev.to/alanwest/how-to-fix-that-robotic-ai-tone-in-your-llm-powered-features-4h5e
Pristren. "How to Reduce LLM Output Tokens by 40-60%." May 2026. https://pristren.com/blog/reduce-output-tokens-guide/
Vuyyuru, A. "9 Practical Tips to Stop Burning Tokens on LLMs." Substack, May 2026. https://abhijayvuyyuru.substack.com/p/9-practical-tips-to-stop-burning
Pockit Tools. "LLM Structured Output in 2026: Stop Parsing JSON with Regex." DEV Community, February 2026. https://dev.to/pockit_tools/llm-structured-output-in-2026-stop-parsing-json-with-regex-and-do-it-right-34pk
Nambiar, B. "Beyond Free-Form Text: How Constrained Decoding is Reshaping Structured Generation in LLMs." Medium, September 2025. https://medium.com/@brijeshrn/beyond-free-form-text-how-constrained-decoding-is-reshaping-structured-generation-in-llms-5f7a38bef259
Let's Data Science. "How Structured Outputs and Constrained Decoding Work." March 2026. https://letsdatascience.com/blog/structured-outputs-making-llms-return-reliable-json
Hecatus Research. "Less is More for LLMs? A Critique of Prompt-Based Compression." Medium, May 2026. https://medium.com/hecatus-research/less-is-more-for-llms-a-critique-of-prompt-based-compression-910978d8bad4
BuildML. "If an LLM keeps producing excessively verbose answers, how would you correct it?" Substack, December 2025. https://substack.com/@buildml/note/c-190308122

About the Author

Sayed Ali Alkamel is a Google Developer Expert in Dart and Flutter, co-founder of Flutter MENA, and Manager of Digital Application Platforms at Oman Housing Bank. He has spoken at tech events across 22+ countries and shipped apps with 2.5M+ downloads. He writes about Flutter, AI, and the developer experience at dev.to/sayed_ali_alkamel.

GGUF, Explained: The File Format That Put LLMs on Your Laptop

Sayed Ali Alkamel — Wed, 24 Jun 2026 15:37:26 +0000

TL;DR

GGUF is a single binary file that packs a model's weights, tokenizer, and architecture metadata together, so a tool can open it and run with no side files.

The name on the file, like Q4_K_M or Q8_0, encodes a quantization choice: how many bits each weight gets, which decides size, speed, and quality.

For most consumer hardware, Q4_K_M is the right default. It cuts an 8B model to about 5 GB for a 1 to 3 percent quality cost.

GGUF is the reason serious models run on hardware you already own, with your data staying on the device.

Why does a model need a special file format at all?
What is actually inside a GGUF file?
What do the quantization names like Q4_K_M actually mean?
How much quality do you actually lose?
The fair criticism: isn't 4-bit just lossy compression that makes the model dumber?
How do you convert a model to GGUF yourself?
How to pick a quant for your hardware
Questions developers are actually asking about GGUF
The quiet format that changed who gets to run AI
References
About the author

Forty years ago, a single gigabyte of computer memory cost more than a house and filled a cabinet. The phone in your pocket carries six to twelve times that, and it can run a language model trained on a meaningful slice of the open internet. The thing that carried that model from a research cluster into your hand was not a new chip. It was a file format.

If you have ever downloaded a model to run locally, you have already met it. The file ended in .gguf, and a tool like Ollama or LM Studio opened it without asking you for a config folder, a tokenizer directory, or a Python environment full of pinned versions. That quiet competence hides a lot of engineering.

GGUF is worth understanding because the filename is a decision. Q4_K_M, Q5_K_M, Q8_0: each one is a different answer to a single question, which is how much quality you are willing to trade for a model that actually fits on your machine. Most people pick by guessing. You do not have to.

Why does a model need a special file format at all?

A trained model is, underneath everything, a very large pile of numbers. Billions of them. Formats like PyTorch's .pth and Hugging Face's .safetensors store mostly those numbers and little else. To turn that pile into something you can actually run, you need more: a config.json that describes the architecture, one or more tokenizer files, a generation config, and sometimes custom Python that defines how the model is wired. You keep all of it together, you match the versions, and you have a framework installed to read it.

Quantized models make this harder, not easier. When you compress weights from 16 bits down to 4, you cannot just store smaller numbers. You also have to store the scaling factors and offsets needed to reconstruct each block, and the generic formats were never designed to carry that bookkeeping cleanly. Standard model saving methods fall short with quantized models, which need to store low-bit weights alongside their scaling factors and zero-points. GGUF was the practical answer to that gap.

What is actually inside a GGUF file?

A GGUF file introduces itself before you ask it anything. Open it and the first thing you find is structure, not just data. The format is a binary file with three main parts: a header, a metadata section, and the tensor data. The header is small bookkeeping. The metadata is where the magic lives: the architecture, the hyperparameters, the tokenizer, and the quantization type all sit there as typed key value pairs. The tensor data is the weights themselves, packed and aligned so they can be read straight off disk.

That self-describing design is the whole point. The runtime inspects the metadata, understands the architecture, loads the tokenizer, and maps the tensors without relying on a separate config.json or tokenizer folder. The weights are laid out for memory mapping, which means the model loads fast because the operating system can page it in instead of copying gigabytes around first.

GGUF grew out of an older format. It was introduced as part of the llama.cpp and GGML ecosystem in August 2023, and it is now the dominant format for distributing quantized local LLMs on Hugging Face. If you are downloading a model to run today, you are almost certainly downloading a .gguf.

What do the quantization names like Q4_K_M actually mean?

This is where most people get lost, so let me save you the hour I lost staring at a Hugging Face file list. The naming scheme is genuinely user-hostile. Nothing about Q4_K_M tells a newcomer that the M means medium, or that the K is the good kind. Here is how to read it.

The leading number is roughly the bits per weight. Q8 is eight bits, Q4 is four. The full picture is more interesting than that, because there are three families.

The legacy family is Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0. These use classic per-block linear quantization, where a block stores n-bit codes and either one scale (the symmetric "_0" variants) or a scale plus an offset (the "_1" variants), decoded with a single transform per block. They are simple and fast to decode. Their weakness shows at low bit widths, where one flat map per block cannot capture skewed weight distributions well.

The K-quants are the modern default: Q3_K, Q4_K, Q5_K, Q6_K, each with _S, _M, and _L variants. The K stands for a method the llama.cpp community built, and it is smarter about where it spends its bits. K-quants perform importance-weighted bit allocation: attention and output projection tensors, which disproportionately affect quality, keep higher precision, often 6 bits, while less critical feedforward layers are quantized more aggressively. The suffix is the mix. Q4_K_M (Medium) keeps more sensitive layers at 6-bit, while Q4_K_S (Small) pushes more weights to 4-bit to save a few hundred megabytes on an 8B model.

The newest family is the I-quants, written IQ2, IQ3, IQ4, with names like IQ4_XS. The IQ family reconstructs weights using a super-block scale and an importance matrix, aimed at preserving quality at low bitrates. They squeeze more model into less space, at the cost of being more sensitive to how the quant was produced.

Here is a detail worth internalizing, because it changes how you think about all of this. Text generation is memory-bandwidth-bound. The GPU spends most of its time reading weights from memory, not doing multiply-adds, so a bigger quantization means more bytes to read per token and slower output. A smaller file is not just cheaper to store. It is literally fewer bytes to haul for every word the model writes. That is why a Q4 model often types faster than its Q8 twin. It does not think faster, it reads less.

The hard numbers, for a current 8B model, look like this:

Quant	Size (Llama-3.1-8B)	Quality vs FP16	Reach for it when
FP16	~16 GB	baseline	You are benchmarking or fine-tuning
Q8_0	~8.5 GB	under 0.5% loss	Code, math, agents, and you have the VRAM
Q6_K	~6.6 GB	~1% loss	A near-lossless step below Q8
Q5_K_M	~5.7 GB	~1% loss	You have spare memory over Q4
Q4_K_M	~4.9 GB	1 to 3% loss	The default for most people
Q4_K_S	~4.7 GB	slightly worse	Short by a few hundred MB
Q3_K_M	~4.0 GB	noticeably worse	Memory is very tight
Q2_K	~3.2 GB	severe	Last resort only

File sizes are bartowski's Meta-Llama-3.1-8B-Instruct GGUFs on Hugging Face. Quality figures are approximate and vary by model and benchmark.

How much quality do you actually lose?

For a modern 8B model, moving from full precision to Q4_K_M costs roughly 1 to 3 percent on MMLU, which is imperceptible in everyday chat and writing. Q8_0 loses under half a percent. The damage only becomes obvious in the small quants, Q2 and Q3, where reasoning visibly falls apart. That is the short version, and for most readers it is the whole answer.

The longer version comes from people who measured it carefully rather than vibing it. A 2026 study titled "Which Quantization Should I Use?" ran a unified evaluation of llama.cpp formats on Llama-3.1-8B-Instruct and made a point worth repeating: these quantization schemes are largely community-driven, with new formats, effective bit estimates, and recommended usage patterns emerging through GitHub issues, pull requests, and informal benchmark reports. In other words, the folklore in those Hugging Face comment threads is doing real work, and it is now being checked against actual downstream tasks.

One concrete number anchors the speed side of the trade. The llama.cpp project publishes benchmark figures for Llama-3.1-8B, and on those numbers Q8_0 generates tokens about 29 percent more slowly than Q4_K_M. That is the bandwidth tax made visible. You pay it in tokens per second for the fidelity you buy.

The shape of that curve is the thing to remember. Below Q4 the quality drops off a cliff. Above Q4 the line goes nearly flat, so each extra gigabyte you spend buys you almost nothing you can feel.

The fair criticism: isn't 4-bit just lossy compression that makes the model dumber?

Yes, it is lossy. Pretending otherwise would be dishonest. The honest answer is that it depends entirely on where you point the model.

For chat, summarization, and drafting, the 1 to 3 percent loss at Q4_K_M is invisible in practice, and I have shipped on-device features built on exactly that trade. For code generation, math, and multi-step agent workflows, the calculus changes, because small errors compound across steps and a single wrong token early can derail a whole chain. Q8_0 loses under 0.5 percent versus FP16 while Q4_K_M loses 1 to 3 percent, a gap that is imperceptible in everyday use but can matter on precise numerical reasoning. So the criticism is fair, and the response is to match the quant to the job rather than to argue that 4-bit is free.

Here is the part the enthusiast threads tend to skip. Requantizing a file that was already quantized, or running an aggressive IQ2 on a small model, can produce output that is confidently wrong in ways an average benchmark score will not catch. The number looks fine. The behavior is not.

There is a second limit that has nothing to do with bits. Self-describing does not mean future-proof. A GGUF file is not universally compatible forever, because the runtime still has to support the model architecture and tensor types used in the file. A brand-new architecture can land on Hugging Face and your favorite tool simply cannot read it until the maintainers add support.

How do you convert a model to GGUF yourself?

The path is two steps: convert to a high-precision GGUF, then quantize it down. The llama.cpp repository ships both tools, and the official quantize guide documents the full set of options.

# 1) Convert a Hugging Face model to a high-precision GGUF (bf16)
python convert_hf_to_gguf.py ./Llama-3.1-8B-Instruct \
  --outtype bf16 \
  --outfile llama-3.1-8b-bf16.gguf

# 2) Quantize that file down to Q4_K_M
./llama-quantize llama-3.1-8b-bf16.gguf \
  llama-3.1-8b-Q4_K_M.gguf Q4_K_M

Once you have the file, running it from Python takes a few lines with llama-cpp-python. The n_gpu_layers argument is the partial-offload lever: push as many layers onto the GPU as your VRAM allows, and the rest run on the CPU.

from llama_cpp import Llama

llm = Llama(
    model_path="llama-3.1-8b-Q4_K_M.gguf",
    n_ctx=4096,
    n_gpu_layers=20,  # offload 20 layers to the GPU, run the rest on CPU
)

out = llm("Explain GGUF in one sentence:", max_tokens=64)
print(out["choices"][0]["text"])

If you would rather skip the local setup entirely, the GGUF-my-repo space on Hugging Face builds quantized files for you and stays in sync with llama.cpp. I leaned on that more than once before I had the toolchain set up properly. Worth mentioning, the first time I converted a model myself, the conversion script choked on a tokenizer it did not recognize and the error told me almost nothing useful, so do not be surprised if step one needs a second attempt.

How to pick a quant for your hardware

Stop choosing by reputation and choose by the memory in front of you. The decision is mostly mechanical once you see it framed as hardware.

A practical guide, from least to most headroom:

On a laptop with no dedicated GPU, or with 6 to 8 GB of VRAM, use Q4_K_M. It is the most-downloaded quant for a reason, and it gives you the most model per gigabyte.
With 12 to 16 GB of VRAM, step up to Q5_K_M, or Q6_K if it fits, for a small bump in fidelity.
With 24 GB or more, and when you care about code, math, or tool calling, run Q8_0. The sub-half-percent loss is cheap insurance for work where errors compound.
Short by only a few hundred megabytes? Try Q4_K_S or IQ4_XS before you abandon the model. IQ4_XS reaches near-identical perplexity at a smaller size than Q4_K_M, useful when you need to squeeze one more gigabyte out of a tight budget.

For the very large models, the math gets unforgiving. A 70B model at Q4_K_M lands around 42.5 GB and fits across two 24 GB consumer GPUs with room for the cache, while Q5_K_M overflows that by about 2 GB before you add any context. At that scale Q4_K_M is not a preference. It is the ceiling.

If your target is a phone, pair a small model in the 1B to 4B range with Q4_K_M and keep the context window modest. I have watched a Q5 build of an 8B model run out of memory on a mid-range phone while the Q4_K_M of the same model ran without complaint. The filename was the entire difference. If you want to go deeper on running models fully on-device, I covered the Flutter side of this in a separate piece.

Questions developers are actually asking about GGUF

What does GGUF stand for?

GGUF is the single-file binary format for storing large language models, created for the llama.cpp project as the successor to the older GGML format. The GG comes from Georgi Gerganov, the developer behind both. The acronym is most often expanded as GPT-Generated Unified Format, though you will also see GGML Universal File, and the official specification does not actually spell it out. Whatever you call it, a GGUF file packs the model weights, the tokenizer, and the architecture metadata into one file you can download and run with a compatible tool.

What is the difference between GGUF and safetensors?

safetensors is a safe, fast container for raw model weights, used widely for full-precision models and for training and fine-tuning. GGUF adds the architecture metadata and tokenizer inside the file and is built around quantization for efficient inference, including on CPU. Use safetensors in training and research pipelines, and use GGUF for running quantized models locally.

Which GGUF quantization should I use?

For most people on consumer hardware, Q4_K_M is the right default. It cuts an 8B model to about 5 GB with a 1 to 3 percent quality loss that is imperceptible in chat and writing. Move up to Q5_K_M or Q8_0 if you have spare memory and need higher fidelity for code or math, and drop to Q4_K_S or IQ4_XS only if you are short a few hundred megabytes.

Can GGUF models run on a GPU or only on the CPU?

GGUF runs on CPU, GPU, or both. The llama.cpp engine supports full GPU execution, CPU-only inference using AVX and ARM NEON instructions, and partial offloading where some layers run on the GPU and the rest on the CPU. The partial-offload path is what lets a model that does not fully fit in VRAM still run with GPU acceleration.

Is Q4_K_M good enough, or do I need Q8_0?

Q4_K_M is good enough for chat, writing, and summarization, where its 1 to 3 percent quality loss is invisible. Choose Q8_0 when you need maximum fidelity for code generation, math, or multi-step agent workflows, where small errors accumulate. Q8_0 loses under half a percent versus full precision, but the file runs about 1.7 times the size of Q4_K_M and generates tokens roughly 29 percent slower on llama.cpp's published Llama-3.1-8B benchmark.

How do I convert a model to GGUF?

Use the convert_hf_to_gguf.py script from llama.cpp to turn a Hugging Face model into a high-precision GGUF, then run llama-quantize to compress it to a level such as Q4_K_M. If you want no local setup at all, the GGUF-my-repo space on Hugging Face builds quantized GGUF files for you and stays in sync with the llama.cpp project.

The quiet format that changed who gets to run AI

For a couple of years, the working assumption was that capable models live in someone else's data center, reachable only through an API key and a credit card. That assumption is breaking, and a file format is part of why.

It is an unglamorous thing to credit. Formats do not trend. Yet formats are exactly where capability either spreads or stays locked behind a counter. PDF made documents portable across machines that agreed on nothing else. MP3 made an entire library fit in a pocket. GGUF is doing the same thing to trained intelligence: moving it onto hardware people already own, with their data staying on the device instead of crossing a network.

Look at the arc and the direction is clear. The next decade of on-device AI is being built on a file you can copy to a USB stick and run on a phone. Knowing what is inside that file, and what the name on it quietly costs you in quality and speed, has stopped being a niche skill for a few enthusiasts. It is becoming part of the job.

References

GGUF specification, ggml-org. https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
llama.cpp quantize tool README, ggml-org. https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
llama.cpp, "Difference in different quantization methods," Discussion #2094. https://github.com/ggml-org/llama.cpp/discussions/2094
"Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct," arXiv, 2026. https://arxiv.org/html/2601.14277v1
DataCamp, "GGUF Format: A Complete Guide to Local LLM Inference." https://www.datacamp.com/tutorial/gguf-format-a-complete-guide
APXML, "LLM GGUF Guide: File Format, Structure, and How It Works." https://apxml.com/posts/gguf-explained-llm-file-format
SitePoint, "Quantization Explained: Q4_K_M vs AWQ vs FP16 for Local LLMs." https://www.sitepoint.com/quantization-q4km-vs-awq-fp16-local-llms/
The Kaitchup, "Choosing a GGUF Model: K-Quants, IQ Variants, and Legacy Formats." https://kaitchup.substack.com/p/choosing-a-gguf-model-k-quants-i
"Q4 vs Q5 vs Q6 vs Q8 Quantization: Real Quality Loss Numbers for Local LLMs." https://runaihome.com/blog/quantization-q4-q5-q6-q8-quality-loss-2026/
Michael Brenndoerfer, "GGUF Format: Efficient Storage and Inference for Quantized LLMs." https://mbrenndoerfer.com/writing/gguf-format-quantized-llm-storage-inference
Hugging Face, "GGUF" documentation. https://huggingface.co/docs/hub/en/gguf
llama.cpp repository, ggml-org. https://github.com/ggml-org/llama.cpp

About the author

Sayed Ali Alkamel is a Google Developer Expert in Dart and Flutter, Founder of Flutter MENA, and Manager of Digital Application Platforms at Oman Housing Bank. He has spoken at tech events across 22+ countries and shipped apps with 2.5M+ downloads. He writes about Flutter, AI, and the developer experience at dev.to/sayed_ali_alkamel.

AI Gateways: A Senior Engineer's Honest Take

Sayed Ali Alkamel — Sun, 21 Jun 2026 08:37:19 +0000

TL;DR

An AI gateway is a reverse proxy between your apps and your LLM providers. It gives you one endpoint, token-level cost control, semantic caching, model fallbacks, guardrails, and observability, so none of that leaks into application code.
It is not the same thing as an API gateway. API gateways meter by request, cache by exact URL, and treat the request body as opaque. AI gateways meter by tokens, cache by meaning, and read the payload.
The options split into three camps: open-source self-hosted (LiteLLM, Bifrost), managed and aggregator (Cloudflare, Vercel, OpenRouter), and enterprise control planes (Portkey, TrueFoundry, Kong).
Add one once you call more than one provider or your token bill grows enough that budgets and caching matter. Adding it on day one of a single-provider prototype is a common mistake.

What developers are actually fighting when AI hits production
What is an AI gateway, exactly?
AI gateway vs API gateway: what actually changes
The core features worth paying for in an AI gateway
What AI gateway options exist in 2026?
We already have an API gateway. Why add another hop?
What this means for your stack
Questions developers are actually asking about AI gateways
The layer that learned to think

Every system that survives long enough grows a border. A cell membrane decides what crosses and what stays out. A customs desk decides what enters a country and what it costs to bring. Software grew its own version about twenty years ago and called it the API gateway: a quiet layer at the edge, checking credentials, counting requests, then waving traffic through and forgetting about it.

That border was built on one assumption. The thing on the other side is a normal backend. It answers in milliseconds, it costs the same whether you ask it once or ask it cleverly, and it returns the same response to the same request every single time. For two decades that assumption held, and the gateway could afford to be dumb on purpose.

Then we wired our applications to large language models, and every part of that assumption broke at once. The backend now charges by the word. It can take ten seconds to answer. It streams its reply one token at a time, and it will cheerfully hand you two different answers to the same question. The old border cannot see any of this. That gap is where the AI gateway lives.

What developers are actually fighting when AI hits production

The first LLM feature in any codebase looks innocent. One provider, one API key, a few calls wrapped in a helper function. It works in the demo and it ships.

The trouble starts at the second provider. Maybe one model is better at reasoning and another is cheaper for bulk summarization. Maybe your primary provider has an outage during a customer demo and you want a fallback that does not require a deploy. The instant you have two, the cross-cutting concerns that were hiding in that helper function start spreading: auth for each provider, retry logic, a place to track spend, somewhere to log what went out and what came back. Copy that helper into three services and you no longer have a pattern. You have a liability.

The money makes it urgent. Menlo Ventures put enterprise spend on foundation model APIs at 12.5 billion dollars in 2025, roughly double the prior year, as workloads moved from experiments into production inference. When a line item doubles in a year, finance starts asking which team spent what, and "I am not sure, it is all one API key" is not an answer that survives a budget review. I work at a bank, and in a regulated shop that conversation arrives even faster, with the compliance team standing next to finance.

What is an AI gateway, exactly?

An AI gateway is a reverse proxy purpose-built for LLM traffic. It sits between your applications and the model providers they call, presents a single endpoint (usually shaped like the OpenAI API), and handles routing, fallbacks, token-based cost tracking, caching, guardrails, and logging on the way through. Your app talks to one address and stops caring which model is on the other side.

Gartner frames it as an intermediary between applications and AI services that gives you a central point for the security, governance, and observability of AI workloads. That is the short version. The longer version is that the gateway becomes the one place where model access, cost, and policy are decided, instead of those decisions being scattered across every service that happens to call a model.

Mechanically, a request flows like this. Your app sends an OpenAI-shaped completion request to the gateway. The gateway authenticates the caller against its own key registry, picks a provider based on your routing rules, injects the real provider credential (which your app never holds), forwards the request, then streams the response back while it counts tokens, applies content checks, and writes a log line. The application sees a normal API call. Everything interesting happens in the middle.

Here is the part that makes the whole pattern click. You do not rewrite your code to adopt a gateway. You change the base URL.

from openai import OpenAI

# Point the same OpenAI client at your gateway instead of the provider.
# Application code does not change. The gateway handles auth, routing,
# fallbacks, caching, and logging behind this single endpoint.
client = OpenAI(
    base_url="https://your-gateway.example.com/v1",
    api_key="GATEWAY_VIRTUAL_KEY",  # a scoped key, not your raw provider key
)

response = client.chat.completions.create(
    model="claude-sonnet-4",            # swap to gpt-4o or mistral-large, no other edits
    messages=[{"role": "user", "content": "Summarize this contract clause."}],
    extra_headers={
        "x-gateway-team": "payments",   # cost attribution per team
        "x-gateway-fallback": "gpt-4o", # backup model if the primary is down
    },
)

print(response.choices[0].message.content)

One virtual key, one base URL, and the model name becomes a value you can change without touching the rest of the request. That is the abstraction doing its job.

AI gateway vs API gateway: what actually changes

This is the question I get most, usually phrased as "we already run Kong, isn't an AI gateway just that with a new label." It is a fair question with a clear answer: the two share a shape and almost nothing else.

Picture the difference physically. An API gateway treats the request body as a sealed envelope. It reads the address on the outside, routes by that, and never opens the letter. An AI gateway opens the envelope, reads the contents, counts the words, sometimes decides the answer is already cached and never sends it at all, and on the way back it can redact a line before handing the reply to you. One is a mail sorter. The other is a reader with opinions.

That shift shows up across every dimension that matters in production.

Dimension	API Gateway	AI Gateway
Unit of metering	Requests per minute	Tokens per minute, plus spend
Caching	Exact match by URL and headers	Semantic, by meaning of the prompt
Routing	Load balancing, path and header based	Model aware: by cost, latency, capability
Response model	Request and response, REST or GraphQL	Streaming, token by token over SSE or WebSocket
Failure handling	HTTP status codes, retry same target	Fallback to a different model or provider
Security focus	Who can call what	PII redaction, prompt injection, content moderation
Latency budget	Milliseconds	Seconds, sometimes minutes for long generations
Request body	Opaque, a black box	Parsed, inspected, and rewritable
Cost per call	Roughly flat	Highly variable, driven by token count

The metering row is the one that bites first. A request count is the wrong ruler for LLM traffic. One short completion and one 4,000-token prompt with a 2,000-token answer both count as a single request, yet the second can cost roughly thirty times more, as the team at Fastio lays out for agent workloads. A request-per-minute limit will happily let a runaway agent spend your monthly budget before lunch, because every one of those expensive calls looks identical to a cheap one from where the API gateway is standing.

This is also why bolting token logic onto a generic reverse proxy tends to end badly. You can write a Kong or NGINX plugin to count tokens, and it works right up until you hit streaming responses, where the tokens arrive in chunks and your counter has to track them mid-stream. At that point you have built a fragile version of a gateway instead of using one. In practice, teams running both put the API gateway at the perimeter as the front door for all services, and the AI gateway behind it, handling only the model traffic.

The core features worth paying for in an AI gateway

Most gateways list thirty features. Six of them decide whether the thing survives contact with production. Here is what I check for, roughly in the order I care about it.

A unified, OpenAI-compatible API is the floor, not a bonus. This is what lets you change one model name and move from Claude to GPT to Mistral with no other edit. One honest complaint here: "OpenAI-compatible" is a spectrum, not a guarantee. Providers differ on streaming details, tool-calling shapes, and error formats, and a gateway papers over more of that gap on some routes than others. Test the models you actually plan to use, not the ones in the marketing table.

Model-aware routing with real fallbacks comes next. You want weighted routing (send 70 percent to one model, 30 percent to another), latency-based routing (whoever answers fastest), and health-aware failover that pulls a sick provider out automatically and tests for its recovery. Semantic routing, where the gateway reads the request and sends easy questions to a small cheap model and hard ones to a strong expensive model, is the most interesting version. It is also still an emerging capability in 2026 rather than a settled default, so treat vendor demos of it with a little skepticism.

Token-level cost tracking and budgets is where the gateway pays for itself. The gateway should track consumption per key, per team, and per project, and enforce hard caps that reject or queue requests over the limit. This is the control that maps to how LLMs actually bill, and it is the one that keeps a misconfigured agent from becoming a finance incident.

Semantic caching is the quiet cost killer. Instead of matching on exact text, it compares the meaning of prompts and serves a stored answer for ones that are equivalent. A 2024 study on GPT Semantic Cache measured hit rates between 61 and 69 percent across common query categories, with cached answers returned in under 50 milliseconds against multi-second live calls. Teams commonly see 20 to 30 percent near-duplicate prompts across users, and caching those at the gateway cuts spend with zero application changes.

Guardrails at the trust boundary matter more the more regulated you are. PII redaction strips sensitive data from a prompt before it ever reaches a provider. Prompt-injection detection blocks attempts to override your system instructions. Content moderation filters what comes back. The point is that this is the one place every request passes through, so it is the right place to enforce the policy once instead of in twelve services.

Observability and governance is the difference between a tool and infrastructure. Token-level logging, request tracing, virtual keys, role-based access control, and audit logs are what let you certify against SOC 2, GDPR, or the EU AI Act, whose high-risk obligations become enforceable from August 2026. If your gateway cannot tell you exactly what data left the building and which model received it, it is not ready for a regulated workload.

One feature that does not show up on most checklists but should: low overhead at scale. A gateway adds a hop, and that cost compounds. When an agent makes five sequential model calls and the gateway adds forty milliseconds each, that is two hundred milliseconds of pure proxy time in front of your user. For a single chat call it is invisible. For agent chains it is the kind of thing that shows up in your P99 and your conversion numbers.

What AI gateway options exist in 2026?

The field has matured fast, and it now splits cleanly into three camps. The honest summary: pick by deployment constraint and governance depth first, feature count second.

Gateway	Type	Best for	Worth knowing
LiteLLM	Open-source, self-host	A unified OpenAI-compatible API and full control	Strong entry point, needs augmentation for regulated multi-team setups
Bifrost (Maxim)	Open-source, Go	Low per-call overhead at high throughput	Newer, performance-focused
Cloudflare AI Gateway	Managed, edge	Edge caching with nothing to run	Managed only, Cloudflare ecosystem
Vercel AI Gateway	Managed	Teams already on Vercel and Next.js	No token markup, limited routing and observability
OpenRouter	Hosted aggregator	Breadth: many models, one key, one bill	Light on governance, passes through provider pricing
Portkey	Managed plus self-host	LLMOps: prompt management, guardrails, compliance	Palo Alto Networks announced intent to acquire it in April 2026
TrueFoundry	Control plane	Enterprise governance in your own VPC or air-gapped	Heavier to run, built for data residency
Kong AI Gateway	Plugin on Kong	Teams standardized on Kong already	Extends an existing API gateway with AI plugins

A few signals tell you the market now treats this as core infrastructure rather than a convenience. A security vendor wiring a gateway into its platform is one bet on the layer's permanence. An aggregator raising serious money is another: OpenRouter raised a large round led by Alphabet's CapitalG in May 2026. Kong sizes the AI gateway market at 3.9 billion dollars in 2024, growing toward 9.8 billion by 2031. Money and acquisitions both point the same way, which is a reason to treat the choice as deliberate architecture, not a stopgap you bolt on once something breaks. If you want to read how one managed option handles the edge-caching angle, Cloudflare's AI Gateway docs are a clear primary source.

We already have an API gateway. Why add another hop?

I will steelman the objection, because it is the right instinct. Every extra layer is latency, an operational burden, and one more thing that can fail in the middle of the night. A team that adds infrastructure it does not need is making a mistake, and a gateway in the hot path of an agent is a real cost, not a free abstraction.

So here is the honest test. If you call exactly one provider, your token spend is modest, and you are still figuring out whether the feature even works, you probably do not need a gateway yet. Calling the provider SDK directly is simpler, and a gateway this early is the easiest abstraction to add too soon. I have watched teams reach for one before they had a second provider to route between, and all they bought was a hop.

The objection stops holding the moment two things are true at once: you call more than one provider, and your spend is large enough that attribution, budgets, and caching stop being nice-to-haves. At that point the alternative to a gateway is not "no gateway." It is the same logic, smeared across every service, enforced inconsistently, and impossible to audit. The gateway is not extra work. It is the work, collected into one place where you can actually reason about it. And the API gateway you already run cannot do this job, because it was built to treat the request body as a black box, which is exactly the thing an AI gateway has to read.

What this means for your stack

Strip away the vendor noise and the decision comes down to a few real tradeoffs.

The first is build versus buy. You can write a thin router yourself, and for a single team with one provider that might be the right call for a while. The cost of building shows up later, when you need semantic caching, audit-grade logging, and per-team budgets, and you find yourself maintaining a gateway as a side project instead of shipping product. Buy or adopt open-source once those concerns are real.

The second is deployment, and this is where your industry decides for you. If you are in a regulated sector and customer data cannot leave your boundary, your shortlist is self-hosted (LiteLLM, Bifrost) or a control plane that runs in your VPC or air-gapped (TrueFoundry). If you live on Vercel, the Vercel gateway is the path of least resistance. If you want breadth fast with nothing to run, an aggregator like OpenRouter fills that role. If you have already standardized on Kong, its AI plugins extend what you have instead of adding a parallel system. Measuring spend cleanly per team is also the foundation for treating token usage as a real engineering metric, which is hard to do when everything shares one key.

The third is timing, and it is the one most teams get wrong in both directions. Too early and you carry overhead for an abstraction with nothing to abstract. Too late and you are retrofitting cost controls during the budget review that triggered the panic in the first place. The trigger to watch is your own: the second provider, or the spend threshold where finance starts asking questions. When you cross either line, that is the signal, not a calendar date. This decision sits right next to your agentic CI and CD setup, since the same workflows that call models in production are usually the ones generating the spend.

Questions developers are actually asking about AI gateways

What is an AI gateway?

An AI gateway is a reverse proxy that sits between your applications and one or more LLM providers, exposing a single, usually OpenAI-compatible endpoint. It centralizes routing, fallbacks, token-based cost tracking, semantic caching, guardrails, and observability so those concerns stay out of application code. Gartner describes it as a central control point for the security, governance, and observability of AI workloads. It is the equivalent of an API gateway, rebuilt for traffic that is metered in tokens and streamed one piece at a time.

What is the difference between an AI gateway and an API gateway?

An API gateway routes and secures generic HTTP traffic, meters by request count, and caches by exact URL match. An AI gateway meters by tokens, caches by meaning through semantic caching, routes by model and provider, streams responses over Server-Sent Events, and adds AI-specific security like PII redaction and prompt-injection detection. A standard API gateway treats the request body as opaque, while an AI gateway reads and can rewrite it. Many teams run both: the API gateway at the perimeter, the AI gateway behind it for model traffic.

Do I need an AI gateway if I only use one LLM provider?

Often not yet. With a single provider and modest token spend, calling the provider SDK directly is simpler, and adding a gateway too early introduces a hop and operational overhead you do not need. The case becomes strong once you call more than one provider, or your token bill grows large enough that cost attribution, budgets, and caching start to matter. Treat it as a deliberate architectural decision rather than a default.

How does an AI gateway reduce LLM costs?

It tracks token consumption per key, team, or project and enforces budgets, so a single misconfigured agent cannot quietly spend a month's allowance in an afternoon. Semantic caching returns stored answers for prompts that mean the same thing even when the wording differs, cutting both spend and latency. Model-aware routing can send simple requests to a smaller, cheaper model and reserve expensive models for hard ones. A 2024 GPT Semantic Cache study measured cache hit rates of 61 to 69 percent on common queries, with cached answers served in under 50 milliseconds.

What is semantic caching?

Semantic caching stores LLM responses and serves them for new prompts that are similar in meaning, not just identical in text. "Summarize this document" and "Give me a summary of this document" can hit the same cached answer, which exact-match caching would miss entirely. It works by comparing prompt embeddings rather than raw strings. For workloads with repeated query patterns it can remove a large share of live calls, since teams often see 20 to 30 percent near-duplicate prompts.

What are the main AI gateway options in 2026?

Open-source, self-hosted gateways like LiteLLM and Bifrost give you a unified API and full control of deployment. Managed services like Cloudflare AI Gateway, Vercel AI Gateway, and the OpenRouter aggregator give you breadth with nothing to run. LLMOps platforms like Portkey and enterprise control planes like TrueFoundry add governance, guardrails, and VPC or air-gapped deployment for regulated teams. Kong and Envoy extend existing API gateways with AI-specific plugins. The right pick depends on your deployment constraints, governance needs, and whether you want to own the layer or rent it.

The layer that learned to think

For most of computing history we worked hard to make the middle dumb. The best router was the one that moved bytes and got out of the way. The best gateway counted requests, checked a token, and made no judgments about what was passing through, because judgment was slow and the network was the bottleneck. We optimized the middle toward silence.

LLMs inverted that. The expensive, slow, uncertain thing is no longer the network. It is the model. So the most consequential code in the stack is migrating to the one place that can see every request before it reaches that model and decide what it costs, whether it is safe, and whether it even needs to be asked. The gateway stopped being a pipe and became a place where decisions are made.

That is a strange thing to say about a reverse proxy, and it is worth sitting with. Ten years from now, when someone asks where an organization's AI policy actually lives (not the document, the enforced reality of what data reaches which model under what budget), the answer will not be a meeting or a wiki. It will be a config in the gateway. The border we built to stay out of the way is becoming the part that thinks. Build it on purpose.

References

Menlo Ventures, 2025: The State of Generative AI in the Enterprise. https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/
TrueFoundry, A Definitive Guide to AI Gateways in 2026 (cites the Gartner Market Guide for AI Gateways). https://www.truefoundry.com/blog/a-definitive-guide-to-ai-gateways-in-2026-competitive-landscape-comparison
Kong, API Gateway vs AI Gateway. https://konghq.com/blog/learning-center/api-gateway-vs--ai-gateway
Kong AI Gateway (product). https://konghq.com/products/kong-ai-gateway
Boomi, API Gateways vs AI Gateways. https://boomi.com/blog/api-gateways-vs-ai-gateways/
API7.ai, What Is an AI Gateway? Architecture, Benefits and How It Works. https://api7.ai/learning-center/api-gateway-guide/what-is-an-ai-gateway
MLflow, The Role of API Gateway AI Services in 2026. https://mlflow.org/articles/the-role-of-api-gateway-ai-services-in-2026/
Infrabase, AI Gateways Explained: When You Need One. https://infrabase.ai/blog/ai-gateways-explained
Fastio, 8 Best API Gateways for AI Agents in 2026 (semantic cache study figures). https://fast.io/resources/best-api-gateways-ai-agents/
Cloudflare, AI Gateway documentation. https://developers.cloudflare.com/ai-gateway/
LiteLLM documentation. https://docs.litellm.ai/
Portkey. https://portkey.ai/
Vercel AI Gateway documentation. https://vercel.com/docs/ai-gateway
OpenRouter. https://openrouter.ai/

About the Author

Flutter GenUI SDK: Build AI-Generated UIs from Scratch (Complete Beginner Tutorial 2026)

Sayed Ali Alkamel — Sat, 20 Jun 2026 09:42:14 +0000

Picture this. You open a language learning app. A Gemini model reads that you have been struggling with Arabic verb conjugation, and in under two seconds it assembles a bespoke lesson: a cultural scene-setter card, a triliteral root diagram showing every word in the family, a drag-and-drop vowel placement exercise, then a conjugation table. None of those screens were pre-designed. No developer wired them together for this session. The interface was composed at runtime, by the model itself, for you.

That app is Kalaam · كلام, and the technology making it possible is the Flutter GenUI SDK.

Kalaam's home screen: type any learning goal ("bargaining in a Cairo market"), or pick a pre-built real-world scenario. The rest — every widget, every step — is assembled live by Gemini.

This tutorial is your complete beginner's guide to building exactly this kind of app. By the end you will understand the five core concepts behind GenUI, you will have written your first custom widget the model can compose, and you will have a working open-source reference — Kalaam's full codebase — to study whenever you need to go deeper.

No prior AI experience required. You need to know Flutter basics: widgets, StatefulWidget, pubspec.yaml. That is enough.

What Is Flutter GenUI? The Real Explanation

Most Flutter tutorials teach you to build a fixed UI. You define your widget tree, you compile it, and the app ships that exact structure to every user. The AI might power a search bar or a recommendation feed, but the interface itself is still pre-designed by a human developer.

GenUI flips that model. Instead of the developer defining the UI, the developer defines a vocabulary of widgets, and an AI model decides which widgets to assemble, in which order, with which data, based on what the user needs right now.

The Flutter GenUI SDK (genui on pub.dev, version 0.9.2, published by labs.flutter.dev) is the official Flutter implementation of the A2UI v0.9 protocol. Announced at Google I/O 2026 alongside Flutter 3.44, it gives you the scaffolding to build apps where Gemini authors the UI at runtime.

Here is the key insight that most explanations miss: the SDK does not call Gemini for you. It does not know about Firebase, API keys, or networking. What it does is:

Take your catalog of widget definitions and embed their schemas into the Gemini system prompt, so the model knows what widgets it can create and what properties each one accepts.
Parse the structured JSON Gemini streams back and turn it into live Flutter widgets, progressively, before the full response arrives.
Keep a reactive data model that widgets can bind to, so the AI can push state updates without rebuilding the whole screen.

The AI writes the layout spec. The SDK renders it. You write the widgets.

Tap the {} button and watch Gemini write your UI in real time. Every field in that JSON panel became a widget on screen above it.

Alpha warning: GenUI is marked "highly experimental" by the Flutter team. Pin to ^0.9.2 (not any) in your pubspec.yaml and subscribe to the CHANGELOG on GitHub. APIs will change between minor versions. Build something real with it today, but expect to update call sites when new versions land.

The A2UI Protocol: What Is Actually Happening Under the Hood

Every existing GenUI tutorial treats the JSON protocol as a black box. Open it once and the entire SDK becomes obvious.

When your app sends a message to Gemini, the model does not respond with plain text. Because of the system prompt GenUI generates, it responds with structured JSON messages following the A2UI v0.9 wire format. These messages tell the SDK what to render. There are four types.

createSurface — Gemini is creating a new UI area (think: a new step in your lesson).

{
  "type": "createSurface",
  "surfaceId": "turn_1",
  "components": [
    {
      "id": "root_explorer_1",
      "component": "RootExplorer",
      "rootWord": "ك-ت-ب",
      "rootMeaning": "to write",
      "family": [
        {
          "word": "كَتَبَ",
          "transliteration": "kataba",
          "meaning": "he wrote",
          "pattern": "فَعَلَ"
        }
      ]
    }
  ]
}

surfaceUpdate — Gemini is modifying an existing surface in place. No screen rebuild, no transition.

{
  "type": "surfaceUpdate",
  "surfaceId": "turn_1",
  "components": [
    {
      "id": "feedback_text",
      "component": "Text",
      "content": "Correct! كَتَبَ follows the فَعَلَ pattern."
    }
  ]
}

dataModelUpdate — Gemini is pushing new state to the reactive data store. Any widget bound to that path rebuilds automatically.

{
  "type": "dataModelUpdate",
  "updates": {
    "learner/accuracy": 0.82,
    "session/wordsCorrect": 7
  }
}

deleteSurface — Remove a surface when a lesson step is done.

{
  "type": "deleteSurface",
  "surfaceId": "turn_1"
}

Those four messages are everything GenUI understands. The SDK's job is to listen to Gemini's stream, parse these messages as tokens arrive (before the full response is complete), and apply them to the widget tree. Now that you know what is happening at the wire level, every class in the SDK makes immediate sense.

Kalaam's Live Inspector mid-lesson. The CREATE turn_2 badge on the left is a createSurface message. The fields below it — "version": "v0.9", "surfaceId": "turn_2" — are the start of the Root Explorer component JSON. The Root Explorer you see rendered above the panel is what those fields became.

The 8-Step Interaction Cycle

Every interaction in a GenUI app follows the same loop:

User types or taps something
Your app calls conversation.sendRequest(content)
Conversation triggers A2uiTransportAdapter's onSend callback
Your callback calls Gemini and pipes each streaming token to _transport.addChunk(chunk)
A2uiParserTransformer parses the streaming JSON in real time
Parsed A2uiMessage objects feed into SurfaceController.handleMessage()
SurfaceController updates the DataModel and its Surface widgets rebuild
User taps a generated widget, UserActionEvent is dispatched, SurfaceController.onSubmit emits, Conversation wraps it as a new user turn, and the cycle restarts from step 2

Every API call you will make maps to one step in this loop. Keep the cycle in mind as you read the sections below.

User input
    │
    ▼
conversation.sendRequest()
    │
    ▼
A2uiTransportAdapter.onSend  ──────►  Gemini API (your code)
                                              │ streams chunks
    ◄─────────────────────────────────────────┘
    │
    ▼
A2uiTransportAdapter.addChunk()
    │  (parses streaming JSON)
    ▼
SurfaceController.handleMessage()
    │
    ├─► createSurface   ──► Surface widget added to screen
    ├─► surfaceUpdate   ──► Surface widget updated in place
    ├─► dataModelUpdate ──► Bound widgets rebuild automatically
    └─► deleteSurface   ──► Surface widget removed
    │
    ▼ (user taps generated widget)
UserActionEvent ──► conversation.sendRequest() ──► loop restarts

Prerequisites and Project Setup

Before writing any GenUI code, confirm you have:

Flutter 3.44+ (run flutter --version to check)
Dart 3.9+
A Firebase project with AI Logic enabled — Gemini Developer API is on the free tier
The flutterfire_cli tool: dart pub global activate flutterfire_cli

Install the dependencies

Add to pubspec.yaml:

dependencies:
  flutter:
    sdk: flutter
  genui: ^0.9.2
  firebase_core: ^3.10.0
  firebase_ai: ^2.5.0
  json_schema_builder: ^0.2.0

Run flutter pub get.

Configure Firebase

Run flutterfire configure and follow the prompts. It generates lib/firebase_options.dart. This file contains your app's credentials and must never be committed. Add these lines to .gitignore right now:

lib/firebase_options.dart
android/app/google-services.json
ios/Runner/GoogleService-Info.plist

iOS and macOS network entitlement (a step almost every tutorial skips)

If you target iOS or macOS, your app will silently fail to reach Gemini without this entitlement. Add it once and forget about it.

In ios/Runner/Runner.entitlements:

<key>com.apple.security.network.client</key>
<true/>

In macos/Runner/DebugProfile.entitlements and Release.entitlements:

<key>com.apple.security.network.client</key>
<true/>

Initialize Firebase

import 'package:firebase_core/firebase_core.dart';
import 'firebase_options.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp(
    options: DefaultFirebaseOptions.currentPlatform,
  );
  runApp(const MyApp());
}

The Five Core Concepts You Must Understand

1. CatalogItem — Defining What the AI Can Build

A CatalogItem is the fundamental unit of GenUI. It tells the SDK (and therefore Gemini) that a widget named X exists, what JSON properties it accepts, and how to render it as a Flutter widget.

Here is a minimal but complete example — a simple Arabic flashcard:

import 'package:genui/genui.dart';
import 'package:json_schema_builder/json_schema_builder.dart';

final flashcardItem = CatalogItem(
  name: 'ArabicFlashcard',
  dataSchema: S.object(
    properties: {
      'arabic': S.string(description: 'Arabic word with full diacritics'),
      'transliteration': S.string(description: 'Romanised pronunciation'),
      'meaning': S.string(description: 'English translation'),
    },
    required: ['arabic', 'transliteration', 'meaning'],
  ),
  exampleData: [
    () => '''
[
  {
    "id": "card_1",
    "component": "ArabicFlashcard",
    "arabic": "كَتَبَ",
    "transliteration": "kataba",
    "meaning": "he wrote"
  }
]
''',
  ],
  widgetBuilder: (ctx) {
    final data = ctx.data as Map<String, Object?>;
    return Card(
      child: Padding(
        padding: const EdgeInsets.all(20),
        child: Column(
          mainAxisSize: MainAxisSize.min,
          children: [
            Text(
              data['arabic'] as String? ?? '',
              style: const TextStyle(fontSize: 32),
            ),
            Text(data['transliteration'] as String? ?? ''),
            Text(data['meaning'] as String? ?? ''),
          ],
        ),
      ),
    );
  },
);

Three things are happening here, and all three matter.

dataSchema defines the JSON the model must provide when it wants to create this widget. The json_schema_builder package gives you a type-safe DSL (S.object, S.string, S.list, S.boolean) for writing JSON Schema. GenUI embeds this schema into the system prompt so Gemini knows which fields are valid and required.

exampleData contains one or more example JSON snippets. GenUI uses these as few-shot examples in the system prompt, teaching the model the correct format by demonstration rather than by description alone. This is the single biggest lever for getting consistent, valid output from the model. A good example is worth more than a detailed description.

widgetBuilder is the Flutter side. It receives a CatalogItemContext containing ctx.data (the parsed JSON), ctx.dataContext (access to the reactive data model), and ctx.dispatchEvent(...) (to send user interactions back to Gemini). Return any Flutter widget. Keep mainAxisSize: MainAxisSize.min on columns and rows — GenUI embeds your widget inside a dynamically-sized surface, and unbounded height constraints will crash it.

Here is what a real CatalogItem looks like rendered — Kalaam's Root System Explorer, generated from a single RootExplorer component in a createSurface message:

Every node, connector line, and label comes from the JSON Gemini provided in the createSurface message. The widgetBuilder turns that JSON into this radial diagram.

2. Catalog — Grouping Your CatalogItems

A Catalog is a named collection of CatalogItems. You combine your custom items with the built-in primitives:

final appCatalog = Catalog(
  items: [
    flashcardItem,
    ...BasicCatalogItems.all(),
  ],
);

BasicCatalogItems.all() gives you 17 built-in widgets for free: Button, Column, Row, Card, Text, TextField, AudioPlayer, Tabs, List, Image, Icon, Divider, Slider, ChoicePicker, CheckBox, DateTimeInput, and Modal. Gemini can use both your custom widgets and the primitives in the same layout. A single createSurface message might compose a Column containing an ArabicFlashcard followed by a Button.

Kalaam exposes a combined catalog of all 13 custom Arabic teaching widgets plus all built-in primitives, defined in lib/features/session/catalog/catalog.dart.

3. SurfaceController — The Runtime Engine

SurfaceController is the brain of a GenUI session. It processes incoming A2uiMessage objects, manages the reactive DataModel, and broadcasts events when the user interacts with generated widgets.

final controller = SurfaceController(catalogs: [appCatalog]);

The Surface widget renders whatever the model has built for a given surface ID:

Surface(
  host: controller,
  surfaceId: 'turn_1',
)

Surface listens to SurfaceController and rebuilds whenever Gemini sends a surfaceUpdate or dataModelUpdate for that surface ID. You do not manage this rebuild yourself.

4. A2uiTransportAdapter — The Streaming Bridge

A2uiTransportAdapter bridges the gap between raw LLM token chunks and parsed A2uiMessage objects. You create it with an onSend callback that fires whenever Conversation wants to send a message to Gemini:

final transport = A2uiTransportAdapter(
  onSend: (List<Content> messages) async {
    final stream = _model.generateContentStream(messages);
    await for (final chunk in stream) {
      transport.addChunk(chunk.text ?? '');
    }
    transport.finishSending();
  },
);

Every token from Gemini passes through transport.addChunk(). The adapter parses the streaming JSON incrementally and emits complete A2uiMessage objects as soon as they are parseable. This is why surfaces appear progressively as Gemini generates them, not all at once when the full response arrives.

Note that transport.incomingMessages is a stream of A2uiMessage objects you can tap independently for logging or inspection. Kalaam uses this to power its Live GenUI Inspector.

5. Conversation — The Top-Level Facade

Conversation wires SurfaceController and A2uiTransportAdapter together and manages conversation history. It is the only object your UI layer needs to hold:

final conversation = Conversation(
  controller: controller,
  transport: transport,
);

Send a user message:

await conversation.sendRequest(Content.user('teach me the root ك-ت-ب'));

Conversation builds the full List<Content> history (all prior turns) and passes it to your onSend callback, so Gemini always has the full context.

Building Your First GenUI App Step by Step

Here is what the finished app looks like: a Gemini-composed lesson flowing from a vocab carousel through the Root System Explorer, with the QuickChoice quiz at the end.

Home screen → Begin lesson → VocabCarousel → Root System Explorer → QuickChoice quiz. None of these transitions were pre-wired — Gemini assembled them from the catalog.

Let's put all five concepts together. This is a minimal, complete, runnable example.

Step 1: Create the catalog

// lib/catalog.dart
import 'package:genui/genui.dart';
import 'package:json_schema_builder/json_schema_builder.dart';

final infoCardItem = CatalogItem(
  name: 'InfoCard',
  dataSchema: S.object(
    properties: {
      'title': S.string(description: 'Card heading'),
      'body': S.string(description: 'Card body text'),
    },
    required: ['title', 'body'],
  ),
  exampleData: [
    () => '''
[{"id":"c1","component":"InfoCard","title":"Hello","body":"GenUI is working."}]
''',
  ],
  widgetBuilder: (ctx) {
    final d = ctx.data as Map<String, Object?>;
    return Card(
      child: Padding(
        padding: const EdgeInsets.all(16),
        child: Column(
          mainAxisSize: MainAxisSize.min,
          crossAxisAlignment: CrossAxisAlignment.start,
          children: [
            Text(
              d['title'] as String? ?? '',
              style: const TextStyle(
                fontSize: 18,
                fontWeight: FontWeight.w600,
              ),
            ),
            const SizedBox(height: 8),
            Text(d['body'] as String? ?? ''),
          ],
        ),
      ),
    );
  },
);

final appCatalog = Catalog(
  items: [infoCardItem, ...BasicCatalogItems.all()],
);

Step 2: Wire the session screen

// lib/session_screen.dart
import 'package:firebase_ai/firebase_ai.dart';
import 'package:flutter/material.dart';
import 'package:genui/genui.dart';
import 'catalog.dart';

class SessionScreen extends StatefulWidget {
  const SessionScreen({super.key});
  @override
  State<SessionScreen> createState() => _SessionScreenState();
}

class _SessionScreenState extends State<SessionScreen> {
  late final SurfaceController _controller;
  late final A2uiTransportAdapter _transport;
  late final Conversation _conversation;
  late final GenerativeModel _model;

  final _textController = TextEditingController();
  final List<String> _surfaceIds = [];
  bool _waiting = false;

  @override
  void initState() {
    super.initState();

    _controller = SurfaceController(catalogs: [appCatalog]);

    // Track new surfaces so we can render them
    _controller.surfaces.listen((surfaces) {
      setState(() {
        _surfaceIds
          ..clear()
          ..addAll(surfaces.keys);
      });
    });

    // Build system prompt from catalog
    final systemPrompt =
        PromptBuilder.chat(catalog: appCatalog).systemPromptJoined();

    _model = FirebaseAI.googleAI().generativeModel(
      model: 'gemini-2.5-flash',
      systemInstruction: Content.system(systemPrompt),
    );

    _transport = A2uiTransportAdapter(
      onSend: _sendToGemini,
    );

    _conversation = Conversation(
      controller: _controller,
      transport: _transport,
    );
  }

  Future<void> _sendToGemini(List<Content> messages) async {
    setState(() => _waiting = true);
    try {
      final stream = _model.generateContentStream(messages);
      await for (final chunk in stream) {
        _transport.addChunk(chunk.text ?? '');
      }
    } catch (e) {
      // Show an error banner in your UI here
    } finally {
      _transport.finishSending();
      if (mounted) setState(() => _waiting = false);
    }
  }

  void _send() {
    final text = _textController.text.trim();
    if (text.isEmpty) return;
    _textController.clear();
    _conversation.sendRequest(Content.user(text));
  }

  @override
  void dispose() {
    _conversation.dispose();
    _transport.dispose();
    _controller.dispose();
    _textController.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text('GenUI Demo')),
      body: Column(
        children: [
          Expanded(
            child: ListView.builder(
              padding: const EdgeInsets.all(16),
              itemCount: _surfaceIds.length,
              itemBuilder: (_, i) => Padding(
                padding: const EdgeInsets.only(bottom: 16),
                child: Surface(
                  host: _controller,
                  surfaceId: _surfaceIds[i],
                ),
              ),
            ),
          ),
          if (_waiting) const LinearProgressIndicator(),
          Padding(
            padding: const EdgeInsets.all(12),
            child: Row(
              children: [
                Expanded(
                  child: TextField(
                    controller: _textController,
                    decoration:
                        const InputDecoration(hintText: 'Ask Gemini...'),
                    onSubmitted: (_) => _send(),
                  ),
                ),
                IconButton(
                  onPressed: _send,
                  icon: const Icon(Icons.send),
                ),
              ],
            ),
          ),
        ],
      ),
    );
  }
}

Type "show me an info card about Flutter" and Gemini creates an InfoCard surface. Type "update it to be about Dart" and Gemini sends a surfaceUpdate modifying the same surface in place. The AI is composing and mutating your widget tree.

PromptBuilder token count: the system prompt GenUI generates from your catalog is large — typically 3,000 to 5,000 tokens, depending on how many catalog items and how detailed your schemas are. Every message to Gemini includes this full prompt. Plan for it in your cost estimates. Very small or local models will not have sufficient context window or instruction-following capability to handle it. To reduce token count, use PromptBuilder.custom() with a hand-written prompt, or pass only essential extra instructions via systemPromptFragments.

Custom CatalogItem Deep Dive: The Root System Explorer

The InfoCard above is easy to follow. Kalaam's RootExplorer — a radial diagram of Arabic root words — is a production example showing how far you can push a single CatalogItem.

Look at the schema alone:

final rootExplorerItem = CatalogItem(
  name: 'RootExplorer',
  dataSchema: S.object(
    properties: {
      'rootWord': S.string(
        description: 'The triliteral root, letters joined by dashes, e.g. ك-ت-ب',
      ),
      'rootMeaning': S.string(
        description: 'Core meaning of the root in English',
      ),
      'family': S.list(
        description: 'Derived words that share this root',
        minItems: 3,
        maxItems: 7,
        items: S.object(
          properties: {
            'word': S.string(description: 'Derived word with full harakat'),
            'transliteration': S.string(description: 'Romanised pronunciation'),
            'meaning': S.string(description: 'English meaning'),
            'pattern': S.string(
              description: 'Morphological pattern (wazn), e.g. فَعَلَ, مَفْعَل, فَاعِل',
            ),
            'isExpanded': S.boolean(
              description: 'DataModel-bound, false initially',
            ),
          },
          required: ['word', 'transliteration', 'meaning'],
        ),
      ),
    },
    required: ['rootWord', 'rootMeaning', 'family'],
  ),
  // ... widgetBuilder renders the radial diagram
);

Three patterns here that apply to any production CatalogItem:

Nested schemas for arrays. S.list(items: S.object(...)) lets you define complex nested structures. Gemini learns to produce the entire family array with all sub-fields from a single example, without any additional instruction. The schema and the example together are the full specification.

DataModel binding hints in descriptions. The isExpanded field description says "DataModel-bound, false initially." This is a hint to Gemini that it can send a dataModelUpdate targeting this field's path to expand or collapse specific nodes without rebuilding the surface. The description field in your schema is part of the model's instruction set.

Passing ctx down the widget tree. The widgetBuilder passes CatalogItemContext to child widgets. The individual radial nodes use ctx.dataContext for binding and ctx.dispatchEvent(...) for the Explore button. Keep ctx accessible throughout every level of your widget tree that needs to interact with GenUI.

The full 498-line implementation lives at lib/features/session/catalog/items/root_explorer_item.dart.

Here is that same widget with a node tapped — the isExpanded DataModel path flipped to true, revealing the وزن pattern badge and Explore button without any surface rebuild:

Tapping مَشْرُوب expands it to reveal its morphological pattern (وزن مَفْعُول) and an Explore→ button that asks Gemini to branch deeper into that word. The animation is an AnimatedContainer reacting to local state — no GenUI rebuild needed.

DataModel Binding with `A2uiSchemas.stringReference`

The DataModel is where GenUI starts to feel like magic. Gemini writes to it with dataModelUpdate messages and your widgets react automatically — no setState, no StreamBuilder wiring on your part.

Kalaam's QuickChoice widget (multiple-choice quiz) uses a more advanced binding pattern than S.boolean: A2uiSchemas.stringReference. This tells Gemini that a field is not a static value but a live DataModel path it can write to:

final quickChoiceItem = CatalogItem(
  name: 'QuickChoice',
  dataSchema: S.object(
    properties: {
      'question': A2uiSchemas.stringReference(
        description: 'The multiple choice question',
      ),
      'options': S.list(
        description: 'Array of 4 options',
        minItems: 4,
        maxItems: 4,
        items: S.object(
          properties: {
            'id': S.string(description: 'Option identifier (A, B, C, or D)'),
            'text': S.string(description: 'Option content text'),
          },
          required: ['id', 'text'],
        ),
      ),
      'correctId': S.string(description: 'The correct option ID'),
      'selectedId': A2uiSchemas.stringReference(
        description: 'DataModel-bound chosen option ID',
      ),
      'explanationOnWrong': A2uiSchemas.stringReference(
        description: 'Explanation shown if user chooses wrong answer',
      ),
    },
    required: ['question', 'options', 'correctId', 'explanationOnWrong'],
  ),

The selectedId field is declared as a stringReference. When Gemini creates the surface it provides a DataModel path like session/quiz_1/selected. When the user taps an option, the widget writes the option ID to that path. When Gemini wants to reveal the correct/incorrect state, it sends a dataModelUpdate with session/quiz_1/selected = "B" and the widget rebuilds automatically to show the result.

In the widgetBuilder, you consume a stringReference like this:

final selectedIdRef = data['selectedId'];
if (selectedIdRef is Map && selectedIdRef.containsKey('path')) {
  return BoundString(
    dataContext: ctx.dataContext,
    value: selectedIdRef,
    builder: (context, selectedId) {
      // Rebuild whenever Gemini sends dataModelUpdate for this path
      return _QuizWidget(selectedId: selectedId, ...);
    },
  );
}

BoundString, BoundBool, BoundNumber, and BoundList are all built into GenUI. They listen to the DataModel path and call their builder with the updated value whenever Gemini (or your own widget code) writes to that path.

This pattern — Gemini setting a path, a widget binding to it, and the widget reacting without any app-level setState — is the cleanest way to handle interactive state in GenUI apps. Kalaam uses it for quiz selection, mastery ring progress, and the expanded/collapsed state of each node in the Root Explorer.

The expanded state is a DataModel-bound BoundBool. Gemini can expand or collapse any node mid-lesson by sending a dataModelUpdate — without touching the surface or rebuilding any other widget.

Handling User Interactions with UserActionEvent

When a user taps a generated widget, you need to send that event back to Gemini so it can decide what to do next. UserActionEvent is the mechanism.

Kalaam centralises this in a small helper that every catalog item uses:

// lib/features/session/catalog/kalaam_actions.dart
import 'package:genui/genui.dart';

void sendKalaamAction(
  CatalogItemContext ctx,
  String actionName,
  Map<String, Object?> payload,
) {
  ctx.dispatchEvent(
    UserActionEvent(
      name: actionName,
      sourceComponentId: ctx.componentId,
      context: payload,
    ),
  );
}

The Root Explorer's Explore button uses it like this:

GestureDetector(
  onTap: () => sendKalaamAction(
    ctx,
    'explore_word',
    {'word': word, 'root': rootWord, 'meaning': meaning},
  ),
  child: const Text('Explore →'),
)

When the user taps, SurfaceController.onSubmit emits a UserActionEvent. Conversation wraps it as a UiInteractionMessage and sends it back to Gemini as the next user turn. Gemini receives the action name and payload as context and might respond by creating a new VocabCard surface for that specific word, or a ConjugationTable for its verb forms.

The event name ('explore_word') is a contract. You define it in your system prompt fragments (to teach the model what actions exist), and you dispatch it from your widget. Keep names specific and documented.

Here is UserActionEvent in practice — the QuickChoice quiz widget dispatching a correct or incorrect answer, which causes Gemini to generate the next lesson surface:

Tap "Tea" → red highlight (wrong). Correct answer "Coffee" turns green immediately. Gemini receives {isCorrect: false} and tailors the next turn.

The Live GenUI Inspector: Watching Gemini Build the UI

One of the most instructive patterns in Kalaam is the Live GenUI Inspector — a slide-up panel that streams the raw A2UI JSON messages Gemini emits, displayed as pretty-printed text. It is not part of the SDK. It is ten lines of code using _transport.incomingMessages.

// In your service / state holder:
final a2uiLog = ValueNotifier<List<A2uiLogEntry>>([]);

late final StreamSubscription<A2uiMessage> _logSub;

// After creating _transport:
_logSub = _transport.incomingMessages.listen((message) {
  final entry = A2uiLogEntry(
    kind: message.type,           // createSurface | surfaceUpdate | etc.
    surfaceId: message.surfaceId,
    json: const JsonEncoder.withIndent('  ').convert(message.toJson()),
  );
  a2uiLog.value = [...a2uiLog.value, entry];
});

The inspector panel is a ValueListenableBuilder over this notifier:

ValueListenableBuilder<List<A2uiLogEntry>>(
  valueListenable: a2uiLog,
  builder: (context, logs, _) {
    return ListView.builder(
      reverse: true,
      itemCount: logs.length,
      itemBuilder: (_, i) => _LogTile(entry: logs[logs.length - 1 - i]),
    );
  },
)

This pattern has a purpose beyond debugging: it makes the GenUI data flow visible to other developers. The moment someone watches a CREATE turn_2 badge appear and immediately sees the corresponding Root Explorer render on screen, the whole concept of AI-generated UI clicks. Kalaam ships this in production precisely for that reason.

Full implementation at lib/features/session/view/widgets/genui_inspector.dart.

The Inspector streaming A2UI JSON alongside the live lesson. The CREATE turn_2 badge marks the Root Explorer surface. Every field you see was streamed token by token from Gemini.

Debugging GenUI Apps

Enable SDK logging

The configureLogging() function is your most powerful debugging tool. Add it before runApp:

import 'package:genui/genui.dart';
import 'package:logging/logging.dart';

void main() async {
  final logger = configureLogging(level: Level.ALL);
  logger.onRecord.listen((record) {
    debugPrint('[genui] ${record.level.name}: ${record.message}');
  });
  // ...
}

With Level.ALL you see each raw JSON chunk as it arrives, when the parser detects a new message type, when SurfaceController creates or updates a surface, DataModel path updates, and any parse errors with the offending fragment. Drop to Level.INFO in release builds.

Inspect the generated system prompt

Run this once to understand what the model sees and why prompt token count matters:

final prompt = PromptBuilder.chat(catalog: appCatalog).systemPromptJoined();
debugPrint('Prompt length: ${prompt.length} chars');
debugPrint('Token estimate: ${prompt.length ~/ 4}');
debugPrint(prompt);

Common errors and fixes

Symptom	Most likely cause	Fix
Surfaces never appear	`onSend` not calling `_transport.addChunk()` on every chunk	Verify your `await for` loop reaches every streamed token
iOS/macOS network fails silently	Missing `com.apple.security.network.client`	Add the entitlement to both Debug and Release entitlement files
Gemini returns plain text	System prompt not passed to the model	Confirm `systemInstruction: Content.system(systemPrompt)` is in `generativeModel(...)`
`setState called after dispose`	Listening to `_controller.surfaces` without canceling	Store the `StreamSubscription` and cancel it in `dispose()`
Widget renders blank	`ctx.data` type mismatch	Cast with explicit null fallback: `ctx.data as Map<String, Object?>? ?? {}`
Layout overflow	Custom widget using `Expanded` or unbounded height	Use `mainAxisSize: MainAxisSize.min` on all `Column`/`Row` widgets inside catalog items

Kalaam as a Production Reference

Everything covered in this tutorial appears at production scale in Kalaam · كلام. Here is the map:

Concept	Kalaam location
13 custom `CatalogItem`s	`lib/features/session/catalog/items/`
Combined catalog (custom + primitives)	`lib/features/session/catalog/catalog.dart`
`SurfaceController` + `Conversation` wiring	`lib/shared/services/ai_session_service.dart`
`A2uiTransportAdapter` with logging tap	`lib/shared/services/ai_session_service.dart:61`
Live GenUI Inspector	`lib/features/session/view/widgets/genui_inspector.dart`
`UserActionEvent` helper	`lib/features/session/catalog/kalaam_actions.dart`
DataModel binding with `BoundBool`	`lib/features/session/catalog/items/root_explorer_item.dart:295`
System prompt with pedagogy fragments	`lib/features/session/prompt/kalaam_prompt.dart`
Demo Mode (no credentials)	`lib/features/session/demo/kalaam_demo.dart`
Architecture overview	`docs/ARCHITECTURE.md`

One of Kalaam's 13 custom CatalogItems: the VocabCarousel. Gemini picks which words go in the deck, the widget renders them with full diacritics, transliteration, and example sentences.

Clone it, run it in Demo Mode (flutter run --dart-define=KALAAM_DEMO=true), and open the Live GenUI Inspector. Watch the A2UI JSON stream as you interact with the lesson. That transparency is intentional — Kalaam was built to make the GenUI programming model concrete.

What's Next: genui_catalog and the Broader A2UI Ecosystem

Once you have built your first custom CatalogItem, you may not need to build everything from scratch. The genui_catalog package (version 0.3.0) ships 17 pre-built production components:

dependencies:
  genui_catalog: ^0.3.0

Highlights: KpiCard, DataTable, TimelineCard, ActionForm, ProfileCard, StepperCard, SearchBar, ChartCard. For dashboards, data apps, or form-heavy interfaces, this package saves significant time. For domain-specific apps like Kalaam, custom CatalogItems are still irreplaceable — no generic primitive approximates a radial Arabic root diagram.

The A2UI protocol itself has ecosystem participants beyond Flutter. React, Angular, Lit, and several agent frameworks (AG2, Vercel json-renderer) implement the same wire format. A Flutter app using genui can talk to a Node.js backend agent using the same A2UI v0.9 protocol via the genui_a2a package.

For further reading:

Official Flutter GenUI docs — the authoritative API reference
Google Codelabs: Build a GenUI App — interactive guided walkthrough
Kalaam on GitHub — the production reference used throughout this tutorial, Apache-2.0

What You Have Built

GenUI inverts the traditional app architecture. Instead of shipping a fixed screen for every state, you ship a vocabulary of widgets and let the model decide how to compose them. The Flutter GenUI SDK gives you the scaffolding to make that work: CatalogItem to define your vocabulary, Catalog to group it, SurfaceController as the runtime engine, A2uiTransportAdapter as the streaming bridge, and Conversation as the facade that ties it all together.

The A2UI protocol underneath is simple: four message types, a reactive data model, and a streaming JSON parser. Once you see those four messages — createSurface, surfaceUpdate, dataModelUpdate, deleteSurface — the entire SDK is predictable.

Kalaam applies this architecture to Arabic language learning with 13 custom teaching widgets, a bidirectional interaction loop, live DataModel binding for learner progress, and a transparent inspector so you can watch the model work. It is open source, fully documented, and built specifically to be the kind of reference this tutorial needed.

The next time someone asks what "AI-native" means in a mobile app, you have a concrete answer — and working code to show them.

I Built Kalaam: An Arabic Tutor Whose UI Gemini Assembles at Runtime

Sayed Ali Alkamel — Thu, 18 Jun 2026 15:46:02 +0000

TL;DR

What it is: Kalaam is an open-source Flutter app where a Gemini model composes each Arabic lesson screen at runtime from a combined catalog of 13 custom Arabic widgets and 14 genui SDK primitives. No fixed screens. No hardcoded lesson flow.

Who it is for: Flutter developers building AI-native apps who want a real, runnable reference implementation, not another chatbot wrapper.

Run it now (no Firebase, no API key, ~30 seconds):
  git clone https://github.com/sayed3li97/kalaam.git
  cd kalaam && flutter pub get
  dart run build_runner build --delete-conflicting-outputs
  flutter run --dart-define=KALAAM_DEMO=true
Full source: https://github.com/sayed3li97/kalaam

Maturity: Alpha. Demo Mode is stable. Live Mode (Gemini calling your Firebase project in real time) requires a setup step documented in the README.

What Problem This Actually Solves
What Kalaam Is
Key Features in Depth
How Kalaam Compares
The Honest Objection
Getting Started in Under 5 Minutes
How to Contribute
What This Means for Flutter Development
Questions developers are actually asking about Kalaam
What Comes Next
The Catalog Is the New Source of Truth
References
About the Author

Arabic morphology operates on a principle with no real equivalent in English. Almost every word in the language descends from a three-letter root through predictable morphological patterns. The root ك-ت-ب (k-t-b, the concept of writing) produces كَتَبَ (he wrote), كِتَاب (book), كَاتِب (writer), مَكْتَب (office), مَكْتُوب (written). Classical Arab grammarians built complete transformation tables for these patterns. Modern linguists write dissertations about them. The system is elegant, generative, and genuinely hard to teach with a flashcard.

Every Arabic learning app I have reviewed responds to this complexity the same way: fixed screens, hardcoded vocabulary lists, a multiple-choice grid that never changes shape, and a next button the AI model has no opinion about. The model, when it appears at all, generates text that lands in a Text widget inside a layout the developer wired months before knowing what the learner would need. The interface is not part of what the model controls.

Kalaam takes a different position. Rather than asking a language model to fill in a predetermined form, Kalaam gives Gemini a catalog of real Flutter widgets and lets it decide what to compose. The model reads a JSON schema describing 13 custom Arabic teaching widgets plus 14 genui built-in primitives, then streams an A2UI surface that the GenUI SDK materializes into an actual widget tree on the device. Tap a word node in the root diagram, and the interaction goes back to Gemini as a UserActionEvent. The model then writes whatever surface comes next.

What Problem This Actually Solves

The standard architecture for AI-assisted Flutter apps produces code that looks like this:

// What most "AI-powered" learning apps actually do
final response = await model.generateContent([Content.text(prompt)]);
return Text(response.text ?? '');

The model generates text. A fixed UI the developer designed wraps it. The model has no knowledge of what a conjugation table looks like, cannot decide to show a vocabulary carousel instead of a fill-in-the-blank when the learner already knows the words, and cannot react to a tapped word by composing a triliteral root diagram for that specific word. Every state transition was decided at compile time.

The architectural gap: A model that only fills Strings into fixed widgets can generate better content. It cannot generate better teaching. Those are different things.

The genui Flutter package (A2UI v0.9) addresses this by giving the model a catalog schema and a structured transport protocol. The model emits JSON messages like createSurface and updateComponents that describe a widget tree using catalog item names. The SDK parses those messages and calls each widget's registered widgetBuilder. The developer defines the catalog. The model decides which items to use, with what data, and in what order.

Kalaam is a complete, open-source implementation of this pattern for Arabic instruction, with 13 custom CatalogItem widgets covering morphology, vocabulary, phonetics, dialogue, cultural context, and progress tracking. The full source is at https://github.com/sayed3li97/kalaam.

What Kalaam Is

Kalaam is a Flutter application where Gemini assembles each Arabic lesson interface at runtime. Pick a scenario (Ordering Coffee, a Cairo market negotiation, a formal business letter), and the model composes a lesson surface: a scene-setter first, then vocabulary cards, a triliteral root diagram, a conjugation table, a quiz, depending on what the learner does and how they answer. The widget sequence is not scripted. The model chooses it.

Scenario picker · Root System (ش-ر-ب) · Tap a node to reveal its morphological pattern — every screen composed live by Gemini.

I (Sayed Ali Alkamel, @sayed3li97) made one architectural decision early that shaped everything else: representing the entire model-facing contract as typed Dart CatalogItem objects rather than a YAML configuration file or a JSON registry. The dataSchema and widgetBuilder on each item live in the same Dart file. The JSON schema the model learns and the Flutter code that renders it are co-located. If you change the schema, the compiler tells you if the widget no longer matches it.

Install and run in Demo Mode (Flutter 3.44+, no Firebase, no API key needed):

git clone https://github.com/sayed3li97/kalaam.git
cd kalaam
flutter pub get
dart run build_runner build --delete-conflicting-outputs
flutter run --dart-define=KALAAM_DEMO=true

The core primitive is CatalogItem. Every custom widget Kalaam exposes to the model is registered as one. Here is the RootExplorer registration, simplified:

final rootExplorerItem = CatalogItem(
  name: 'RootExplorer',
  dataSchema: S.object(
    properties: {
      'rootWord': S.string(
        description: 'Triliteral root, letters joined by dashes, e.g. ك-ت-ب',
      ),
      'rootMeaning': S.string(description: 'Core meaning of the root in English'),
      'family': S.list(
        description: 'Derived words that share this root',
        items: S.object(properties: {
          'word': S.string(description: 'Derived word with full harakat'),
          'transliteration': S.string(description: 'Romanised pronunciation'),
          'meaning': S.string(description: 'English meaning'),
          'pattern': S.string(description: 'Morphological wazn, e.g. مَفْعُول'),
          'isExpanded': S.boolean(description: 'DataModel-bound, false initially'),
        }),
      ),
    },
    required: ['rootWord', 'rootMeaning', 'family'],
  ),
  widgetBuilder: (ctx) {
    final data = ctx.data as Map<String, Object?>;
    return _RootExplorerWidget(
      rootWord: data['rootWord'] as String? ?? '',
      family: data['family'] as List<dynamic>? ?? [],
      ctx: ctx,
    );
  },
);

When Gemini wants to teach the ش-ر-ب root family, it emits a createSurface message containing a RootExplorer component with the root letters, meaning, and word family filled in. The GenUI SDK calls widgetBuilder and returns a live Flutter widget. The model never saw the Flutter code. It only knew the JSON schema.

Under the hood, a SurfaceController manages the bidirectional loop. Incoming A2UI messages build the current surface. Outgoing UserActionEvent payloads carry learner interactions back into the model's conversation history, closing the loop for the next turn.

Key Features in Depth

The Root System Explorer

The RootExplorer renders a radial diagram centered on a triliteral root. Each satellite node is a derived word in the root family. Tapping a node expands it from a 92×46pt pill showing just the Arabic word to a 155×144pt card revealing the morphological pattern (وزن / wazn), the transliteration and English meaning, and an "Explore" button.

Tapping مَشْرُوب (a drink) expands the node to reveal its wazn (وزن مَفْعُول), transliteration, English meaning, and the Explore→ button that branches into a new Gemini-composed surface.

That Explore button is not cosmetic. It dispatches a UserActionEvent back to Gemini:

sendKalaamAction(ctx, 'explore_word', {
  'word': word,
  'root': rootWord,
  'meaning': meaning,
});

Gemini receives this in the next conversation turn and can compose an entirely new surface for that specific word: a VocabCard with its etymology, a ConjugationTable if it is a verb, a CulturalNote if the word has idiomatic significance. The diagram is not a static illustration. It is a branch point in the lesson.

HarakatBuilder

Arabic diacritics (harakat) are one of the most common barriers for intermediate learners, and also one of the hardest to teach with a flashcard. Most learning apps just display fully voweled text. HarakatBuilder inverts the exercise.

Given a consonant skeleton and a target vocalization, the learner taps each letter and selects a diacritic from a palette: fatha (َ), kasra (ِ), damma (ُ), sukun (ْ), shadda (ّ). The word re-renders with each placed diacritic in real time. On correct completion, the widget dispatches:

sendKalaamAction(ctx, 'completed', {'isCorrect': true});

The model receives that and composes the next step based on what the learner just demonstrated they know. No hardcoded branching logic in the Flutter code. The model decides where to go.

Live GenUI Inspector

The Inspector is the piece of Kalaam that surprises developers most on first run.

A {} button in the session screen's app bar opens a slide-up panel that shows every A2UI message Gemini has emitted during the current session, newest last, pretty-printed as JSON. You can watch createSurface messages appear as the lesson loads. You can see the component names, the data the model chose to fill in, the surface IDs. You can trace exactly what JSON produced exactly what widget on screen.

When I first built this, I left the Inspector open for every demo I ran internally. Watching "component": "RootExplorer" appear in the stream, followed by a complete Arabic root family the model composed from nothing but the schema description, is the clearest explanation of what GenUI actually does. No architecture diagram I have drawn since has communicated it as well as this panel does in practice.

The Inspector is implemented as a ValueNotifier<List<String>> buffer on the transport layer, with a slide-up panel widget that rebuilds as messages arrive. It is not a debug-only feature: it stays in release builds because it is the most compelling demonstration of the pattern.

The GenUI Inspector open mid-session. The CREATE turn_2 badge marks a new surface creation message. Every field name and value was chosen by Gemini — no developer-written template produced this JSON.

Demo Mode

Live Mode requires a Firebase project with Gemini Developer API enabled. That setup step has a real cost for someone evaluating the project for the first time. Demo Mode removes it.

Run with --dart-define=KALAAM_DEMO=true and the app replays curated A2UI transcripts through the same A2uiTransportAdapter used in Live Mode. Nothing is mocked at the widget level. SurfaceController processes the same message types. CatalogItem widgetBuilder functions instantiate the same Flutter widgets. The difference is only the source of the A2UI stream: a scripted file instead of a live Gemini response.

This was not an accidental boundary. I put the swap at the transport layer deliberately so that Demo Mode exercises the real rendering pipeline. A bug in the catalog, a schema mismatch, an overflow in the Root Explorer: Demo Mode will surface all of them. It is not a polished preview mode. It is the actual system, running on prerecorded input.

VocabCarousel showing قَهْوَة (coffee) with full diacritics, IPA, and example sentence — rendered from a Gemini-emitted VocabCarousel component in a Demo Mode replay.

The Combined Catalog

Kalaam's combined catalog gives the model 27 composable elements: 13 custom Arabic widgets plus 14 genui built-in primitives.

The 13 custom items: RootExplorer, HarakatBuilder, ConjugationTable, VocabCard, VocabCarousel, SceneCard, PhonemeCard, DialogueBubble, CulturalNote, MasteryRing, QuickChoice, SentenceBuilder, FillInTheBlank.

The 14 genui primitives: Column, Row, Card, Text, Button, Icon, Divider, Image, ChoicePicker, TextField, Slider, Tabs, List, AudioPlayer.

The model can compose both tiers freely. A Column containing a SceneCard, then a VocabCarousel, then a QuickChoice quiz is a perfectly valid A2UI surface. In practice, across Live Mode sessions I have run, the model reaches for mixed-tier layouts frequently: a Card primitive wrapping a custom DialogueBubble for roleplay, a Row of Button primitives alongside a MasteryRing. That mixing is where the showcase value sits.

How Kalaam Compares

Approach	Model composes the layout	Bidirectional events	No-API-key demo	Custom widget catalog	Time to first running UI
Kalaam + GenUI SDK	Yes	Yes (UserActionEvent)	Yes (Demo Mode)	27 items (13 custom + 14 primitives)	~30 seconds
DIY JSON-to-widget mapper	Partial	Manual implementation	Depends	You build it from scratch	Weeks
flutter_ai_toolkit	No (fixed chat UI)	No	Yes	No	~1 hour
LLM + `Text()` widget	No	No	Yes	No	~10 minutes
Firebase Genkit + custom Flutter	No	Manual implementation	No (requires Firebase)	You build it from scratch	Days to weeks

The flutter_ai_toolkit wins on time-to-working-chat: if you need a conversational UI embedded in an existing Flutter app, it is the fastest path by a significant margin. Use it. [INTERNAL LINK: flutter_ai_toolkit]

If you need the model to control the layout itself and not just the content inside a fixed layout, the DIY approach and Kalaam/GenUI are the two real options. Kalaam exists as a reference for what "DIY done well" looks like at meaningful scale, so you can evaluate whether using the SDK directly fits your use case before committing to building from scratch.

The Honest Objection

A senior Flutter developer who clones Kalaam and runs it in Demo Mode will ask one question almost immediately: "How much of this is Gemini and how much is a scripted replay?"

It is a fair question. Demo Mode replays prewritten A2UI transcripts. The Root Explorer diagram you see when you run flutter run --dart-define=KALAAM_DEMO=true was not composed by Gemini in real time. It was composed during development, reviewed, committed to the repo, and is being replayed now. The model is not in the loop in Demo Mode.

That is not a misleading presentation, but it matters for evaluating the system. Demo Mode does not show you how Gemini handles unexpected learner input. It does not show what happens when the model emits a structurally valid but semantically off A2UI message (the GenUI SDK renders a graceful error card). It does not show live response latency.

Live Mode is where those questions get real answers. It also requires real setup: a Firebase project, Gemini Developer API enabled, Firebase App Check configured, and a billing budget alert set. The README's "Option B" section walks through all of it.

The limitation I want to name clearly: Kalaam's Live Mode prompt engineering is functional but not complete. In live sessions, Gemini occasionally reaches for a VocabCard where a RootExplorer would have been more instructive, or composes a layout that works but underuses the available screen space. The catalog gives the model the right tools. Teaching it when to reach for each one is an ongoing prompt engineering problem with no fully correct solution yet.

This is alpha software. The architecture is the part that works well. The prompt, the few-shot examples, and the handling of edge-case model outputs are the areas where contributions move the project most.

Getting Started in Under 5 Minutes

Requirements: Flutter 3.44+, Dart 3.9+, a connected device or simulator.

Step 1: Clone and install dependencies

git clone https://github.com/sayed3li97/kalaam.git
cd kalaam
flutter pub get
dart run build_runner build --delete-conflicting-outputs

Step 2: Run in Demo Mode

flutter run --dart-define=KALAAM_DEMO=true

No Firebase project. No Gemini API key. No account required. The demo replays an Ordering Coffee lesson through the real GenUI pipeline.

Step 3: Open the Inspector

Once a lesson starts, tap the {} button in the top-right of the session screen. The GenUI Inspector panel opens and shows every A2UI message the surface was built from. This is where the architecture becomes visible.

Step 4 (optional): Switch to Live Mode

Create a Firebase project at console.firebase.google.com. Enable Build → AI Logic → Get started → Gemini Developer API (free tier available). Then:

dart pub global activate flutterfire_cli
flutterfire configure    # generates lib/firebase_options.dart, which is git-ignored
flutter run              # no --dart-define flag; live mode is the default

Set a billing budget alert before running live sessions. The Gemini Developer API bills your Firebase project, and firebase_options.dart contains credentials that should never be committed to a public repository. The .gitignore in Kalaam already excludes it, along with google-services.json and GoogleService-Info.plist. Only *.example templates are tracked.

Known open issues (as of the initial open-source release):

Occasional layout overflow on very long Arabic words in VocabCard on narrow screens.
ConjugationTable scroll behavior on small-screen Android devices.
Live Mode system prompt does not yet handle learner dialect preference (MSA only).

How to Contribute

Kalaam is open source under Apache 2.0 and genuinely looking for people to use it, break it, and tell me what they found.

Running Demo Mode and filing a GitHub issue when something looks wrong, visually off, or surprising is a real contribution. You do not need to write code.

Where contributions have the highest impact right now:

Prompt refinement is the single highest-leverage area. The system prompt lives in lib/features/session/prompt/. If you have experience with Arabic pedagogy or prompt engineering for Gemini, adding one well-structured example to the few-shots can change what the model reaches for across an entire session. The current few-shot coverage is thin.

New catalog items follow a clear pattern. The reference implementation is lib/features/session/catalog/items/scene_card_item.dart. Every CatalogItem needs a dataSchema, exampleData, and a widgetBuilder. Register it in catalog.dart and add a fixture in test/kalaam_catalog_test.dart. The test infrastructure is already there; you just fill in the new widget's test case.

Platform testing is something I cannot do alone. Kalaam has CI for iOS, Android, and web. Arabic text rendering behavior varies across platform versions and font configurations in ways that are hard to predict. If you run into something that looks wrong on a specific device or OS version, filing an issue with a screenshot is directly useful.

Good first issues are labeled on GitHub. They are scoped to single files and documented with enough context to start without knowing the whole codebase.

Full process: CONTRIBUTING.md.

What This Means for Flutter Development

Three practical framings depending on where you are right now:

If you are currently building AI features by wiring an LLM to a Text widget, Kalaam is a working example of what the next step looks like architecturally. The migration is not trivial (you need to define a catalog, think in surfaces rather than screens, and design the bidirectional event contract), but the reference implementation now exists.

If you are evaluating GenUI for production use, Kalaam gives you 13 CatalogItem implementations to read and adapt. The bidirectional event system in sendKalaamAction, the data model binding in BoundBool and BoundString, the Inspector widget, and the Demo Mode scaffold are all things you can extract and use in your own project.

If you are teaching Flutter architecture or developer experience, the Inspector is a classroom tool worth having. Watching A2UI JSON tokens resolve into typed Flutter widgets in real time, in an app you cloned ten minutes ago, communicates what "model-driven UI" actually means better than any diagram I know.

One concrete recommendation before building a custom JSON-to-widget mapper for your own AI-native Flutter project: read the genui package README and the A2UI v0.9 specification alongside Kalaam's catalog. The scaffold may be closer to complete than you expect.

Questions developers are actually asking about Kalaam

Is Kalaam production-ready as an Arabic learning app?

No. Kalaam is an open-source showcase application in alpha, designed to demonstrate GenUI architecture patterns for Flutter developers. The widget catalog is functional, the Demo Mode is stable, and the GenUI bidirectional loop works end-to-end. The Arabic pedagogy, prompt engineering, and learner progress tracking are not at production quality. If you are building a production Arabic learning product, Kalaam's catalog and architecture are worth studying and adapting, but the app itself is a starting point, not a finished product.

What is the difference between Kalaam's Demo Mode and Live Mode?

Demo Mode (--dart-define=KALAAM_DEMO=true) replays prerecorded A2UI transcripts through the real GenUI pipeline. No API key or Firebase project is needed. Live Mode calls Gemini through Firebase AI Logic in real time, composing each lesson surface fresh based on the learner's actions and conversation history. Demo Mode is fully stable and exercises the same rendering code as Live Mode. Live Mode is functional but requires Firebase setup and is billed through your Firebase project's Gemini Developer API quota.

Does Kalaam require a paid Gemini API key?

In Demo Mode, no. In Live Mode, Kalaam uses Firebase AI Logic with the Gemini Developer API, which has a free tier but can incur charges at scale. The README recommends setting a Firebase billing budget alert before enabling Live Mode. Firebase App Check is also required to prevent unauthorized API usage, since the Gemini Developer API endpoint bills your project and an unprotected key is abusable.

How does Kalaam's approach differ from using flutter_ai_toolkit?

The flutter_ai_toolkit (Google) embeds a ready-made chat UI component into an existing Flutter app. The model generates text responses inside a conversation interface the toolkit provides. Kalaam's GenUI approach is architecturally different: the model decides what Flutter widgets to render, which ones to compose together, and how to lay them out. The developer provides a widget catalog; the model chooses from it at runtime and updates surfaces in place. These are tools for different problems, not competing approaches to the same one.

Can I add my own widgets to Kalaam's catalog?

Yes. Every item in the catalog is a CatalogItem with three required fields: a dataSchema (the JSON Schema object the model uses to understand what data the widget accepts), exampleData (a worked example the system prompt includes as a few-shot), and a widgetBuilder (a function taking CatalogItemContext and returning a Flutter Widget). The reference implementation is lib/features/session/catalog/items/scene_card_item.dart. Register your item in lib/features/session/catalog/catalog.dart and add a fixture test in test/kalaam_catalog_test.dart.

What version of Flutter does Kalaam require, and why?

Flutter 3.44+ and Dart 3.9+. The lower bound comes from the genui package's dependency requirements and Kalaam's use of Dart 3 patterns: @riverpod codegen, @freezed immutable models, and sealed classes. The codegen step (dart run build_runner build) must complete before running the app; generated files are not committed to the repository.

What Comes Next

The roadmap items below are drawn from open GitHub issues and gaps I identified during development. None have committed dates.

Prompt few-shot expansion is the highest-priority item. The current system prompt covers the Ordering Coffee scenario in detail. More worked examples from different lesson contexts (market negotiation, formal letter writing, phonetics drill) would each give the model a better map for when to reach for each catalog widget. A single well-structured example in the few-shots has outsized effect on Live Mode output quality.

Learner profile binding is the next feature in the data model. The /learner/ path in the DataModel already exists (tracking words seen, accuracy, weak phonemes, streak). The model reads it at session start. The missing piece is writing it after each completed interaction and binding MasteryRing to real accumulated progress rather than per-session state.

Dialect support is on the wishlist. Modern Standard Arabic is the current default. A dialect selector that modifies the system prompt would let the app serve Gulf Arabic, Egyptian Arabic, and Levantine Arabic learners, each of which has distinct vocabulary and morphological patterns.

Better malformed A2UI handling is a known gap. When Gemini emits a structurally valid but semantically incorrect A2UI message (wrong component name, missing required field), the current behavior is a generic error card. Partial rendering and schema-level error recovery would make Live Mode more resilient.

The Catalog Is the New Source of Truth

There is a pattern in software where the interface between two systems starts as an implementation detail and gradually becomes the most load-bearing thing in the architecture. REST APIs replaced direct database calls. GraphQL schemas replaced hand-written REST endpoints. Type systems replaced runtime duck typing. In each case, the contract became the artifact worth designing carefully.

Kalaam's CatalogItem plays this role between a language model and a Flutter widget tree. The dataSchema property is what the model learns. It is the description Gemini uses to decide when to emit a RootExplorer versus a ConjugationTable, what fields to fill in, what data shapes are valid. The schema is how you teach a model what your widgets can do.

The practical consequence: time spent making catalog schemas precise pays forward across every session where the model uses that widget. A vague description field on a schema property produces vague model output. A specific description, one that names the format and gives a concrete example, produces surfaces that feel like they were composed for the learner's exact situation, because the model had enough information to make a real choice.

I built Kalaam to see what Arabic instruction looks like when the interface is part of what the model controls. The answer: it looks different every session, reaches for a triliteral root diagram when that is the right tool, and reacts to a learner tapping a specific word in ways a static screen cannot. That sounds obvious in retrospect. It is harder to build correctly than it appears.

The source is at https://github.com/sayed3li97/kalaam. Run it, open the Inspector, tap a word in the root diagram, and watch what the model composes next. Something in there is worth knowing about.

References

Kalaam GitHub repository: https://github.com/sayed3li97/kalaam
genui Flutter package (pub.dev): https://pub.dev/packages/genui
A2UI v0.9 protocol (Google): https://github.com/google/A2UI
Firebase AI Logic (Gemini Developer API): https://firebase.google.com/docs/ai-logic
Firebase App Check: https://firebase.google.com/docs/app-check
flutter_ai_toolkit (Google, pub.dev): https://pub.dev/packages/flutter_ai_toolkit
Flutter 3.44 release notes: https://docs.flutter.dev/release/release-notes
Riverpod state management (pub.dev): https://pub.dev/packages/riverpod
Freezed package (pub.dev): https://pub.dev/packages/freezed
build_runner (pub.dev): https://pub.dev/packages/build_runner
Amiri Arabic font: https://github.com/alif-type/amiri
IBM Plex Arabic font: https://github.com/IBM/plex
flutterfire CLI: https://firebase.flutter.dev/docs/cli

About the Author

Spring AI: The Senior Dev's Honest Take on Java's AI Moment

Sayed Ali Alkamel — Wed, 17 Jun 2026 09:37:45 +0000

TL;DR

Spring AI is an official Spring project that integrates AI models into Java using familiar patterns: auto-configuration, dependency injection, and vendor-portable abstractions.

It reached 1.0 GA in May 2025 and shipped 1.1 in November 2025 with 850+ improvements, including full MCP (Model Context Protocol) integration.

Supports 20+ AI model providers (OpenAI, Anthropic, Google, Ollama, and more) and 12+ vector stores behind a single consistent API.

If your team runs Spring Boot, you have no good reason to write raw HTTP calls to an LLM anymore.

The Problem Java Developers Did Not Expect to Have
What Spring AI Actually Is
The Core of the Framework
A Fair Criticism Worth Naming
Spring AI vs LangChain4j: The Honest Comparison
What This Means For You
Questions Developers Are Actually Asking About Spring AI
Where This All Lands in Ten Years
References
About the Author

The history of programming languages is mostly a history of incumbents surviving longer than anyone predicted. COBOL still processes an estimated $3 trillion in daily financial transactions. Fortran still appears in climate models and aerospace simulations. Java, introduced in 1995, became the substrate of the enterprise world and never left.

Then AI models arrived and, for a brief window, seemed to break this pattern. The default language for working with language models was Python. Not because Python is architecturally superior. Because the researchers and tooling builders who created LangChain, HuggingFace integrations, and notebook-first workflows were Python developers who built for themselves first.

The Java engineer maintaining the systems that actually run the bank, the logistics platform, or the hospital records management system found themselves looking at Python SDKs for a technology their organization was already being asked to adopt. The operating assumption forming in 2023 was that AI integration required a completely different stack. That assumption turned out to be wrong. Spring AI is the evidence.

The Problem Java Developers Did Not Expect to Have

The uncomfortable reality of the 2023-2024 AI wave was that Java was not in the initial conversation.

The fast-moving AI tooling ecosystem was almost entirely Python-native. The mental model that formed, correctly for a time, was that building AI applications required a stack switch, not just a skill addition. Java developers who had spent careers building production-grade distributed systems found themselves watching an entire category of tooling emerge around a language they did not use day-to-day.

According to the 2024 BellSoft Java Survey, roughly 74% of Java developers already use AI tools in their day-to-day work, but only 34% were using any AI framework to build AI-powered applications. The gap between using AI and building with AI is exactly where Spring AI operates.

The question most teams were facing was not philosophical. It was architectural: do we maintain existing Spring Boot microservices, or do we introduce a Python sidecar service to handle every LLM call? The two-process architecture feels manageable in a prototype and becomes a maintenance liability in production.

Spring AI closes that gap. It brings AI model integration into the Spring container itself, using the same patterns your team already knows.

What Spring AI Actually Is

Spring AI is an application framework that connects enterprise Java applications to AI models using portable abstractions, Spring Boot auto-configuration, and familiar dependency injection patterns. It supports 20+ model providers and 12+ vector stores behind a single consistent API, with built-in RAG, tool calling, chat memory, and Model Context Protocol support. The project reached 1.0 GA in May 2025 and is production-ready today.

The official description from the Spring team is precise: Spring AI applies core Spring design principles to the AI domain, specifically portability and modular design, and promotes using plain Java objects as the building blocks of AI applications.

Think Spring Data, but for AI models instead of databases. The same philosophy applies. You describe what you want from a model without pinning your code to a specific vendor. If you have been writing JpaRepository abstractions for years, the mental model transfers directly.

Josh Long, Spring Developer Advocate at Broadcom, said it plainly at DevOps UK in 2025: building AI applications is mostly just calling models that have HTTP APIs. If you can build a Spring Boot service, you are already an AI developer. The skills transfer. The tooling just needed to catch up.

Spring AI currently supports:

Over 20 AI model providers: OpenAI, Anthropic, Google Gemini, Amazon Bedrock, Azure OpenAI, Ollama, DeepSeek, Groq, Mistral AI, NVIDIA, Hugging Face, and more
12+ vector stores: PostgreSQL/PGVector, Chroma, Pinecone, Weaviate, Milvus, Redis, MongoDB Atlas, Neo4j, Oracle, Qdrant, Apache Cassandra, and Azure Vector Search
All major model types: chat completion, embeddings, text-to-image, audio transcription, text-to-speech, and moderation
Agentic capabilities including tool calling, RAG pipelines, MCP client and server support, chat memory, and structured outputs

Spring AI 2.0 is in active milestone development as of mid-2026, built on Spring Boot 4.0, Spring Framework 7.0, and a Jakarta EE 11 baseline, with Java 21 as the minimum runtime.

The Core of the Framework

The ChatClient: Your Main Entry Point

The Spring container already knows your business logic. It has been holding your @Service beans since the day you wrote them. Now it is making introductions, placing your existing code in the same room as whatever language model you are deploying that quarter.

The primary API is ChatClient, a fluent builder interface designed to feel like a sibling of WebClient or RestClient. Here is a working RAG-enabled chat endpoint with Spring AI 1.1:

@RestController
public class DocumentChatController {

    private final ChatClient chatClient;

    public DocumentChatController(
            ChatClient.Builder builder,
            VectorStore vectorStore) {
        this.chatClient = builder
            .defaultAdvisors(new QuestionAnswerAdvisor(vectorStore))
            .build();
    }

    @GetMapping("/ask")
    public String ask(@RequestParam String question) {
        return chatClient.prompt()
            .user(question)
            .call()
            .content();
    }
}

Four things are happening here that required significant boilerplate a year ago. The QuestionAnswerAdvisor retrieves relevant documents from the vector store and injects them into the prompt automatically. The VectorStore abstraction means you can swap PostgreSQL/PGVector for Pinecone without touching this controller. The model itself is configured in application.yml. The entire thing is a standard @RestController.

That is not a toy example. That is close to what a real document Q&A endpoint looks like in production.

Switching Providers Is a Configuration Decision, Not a Code Decision

One of the more immediately useful things Spring AI does is treat provider selection as a deployment concern. To switch from OpenAI to Anthropic, you update application.yml:

# OpenAI setup
spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      chat:
        options:
          model: gpt-4o

# Anthropic setup (swap in, no code changes required)
spring:
  ai:
    anthropic:
      api-key: ${ANTHROPIC_API_KEY}
      chat:
        options:
          model: claude-sonnet-4-5

Your controllers, services, and business logic stay unchanged. That portability is the central design promise of the framework, and in practice it holds.

Function Calling and MCP: Where the Agent Story Gets Interesting

Spring AI 1.1 introduced first-class Model Context Protocol (MCP) integration, and this is the most strategically significant part of the framework for teams building agentic systems. MCP is the emerging standard for interoperability between AI agents and the external tools they need to call.

With Spring AI, you register existing business logic as AI-callable functions with minimal code:

@Configuration
public class BankingTools {

    @Bean
    @Description("Fetch the account balance for a given customer ID")
    public Function<BalanceRequest, AccountBalance> accountBalance(
            AccountRepository repo) {
        return request -> repo.findById(request.customerId())
            .map(AccountBalance::from)
            .orElseThrow();
    }
}

The framework picks up this bean automatically and makes it available to the model via function calling. The AI can now invoke your business logic directly, with full type safety and no change to the underlying service implementation.

Jonathan Schneider made a comparison in 2025 that the Spring community has been repeating since: function calling is to RAG what Inversion of Control was to Java development. IoC did not just clean up dependency management. It changed how teams thought about component design entirely. Function calling through MCP does something similar for the relationship between your business logic and AI orchestration.

What 850 Improvements Actually Means at Human Scale

Spring AI 1.1 shipped with 850+ improvements across five milestone builds. If each improvement were one second of continuous work, that is over 14 minutes of fixes, enhancements, and documentation updates streaming out in a single release cycle. In concrete terms: 354 enhancements, 241 bug fixes, and 100 documentation improvements. This is a framework in rapid production maturation, not a research prototype catching up to itself.

A Fair Criticism Worth Naming

The fair criticism here is direct: Spring AI is only the right tool if you are already in the Spring ecosystem. The portability abstractions are genuinely useful, but they sit on top of Spring Boot auto-configuration. If your team runs Quarkus, Micronaut, or bare Vert.x, the first-class integration story does not apply to you.

LangChain4j, the main Java alternative, has native Quarkus extensions maintained by Red Hat, supports a broader raw count of LLM providers out of the box (30+ versus Spring AI's 20+), and does not require Spring Boot as a foundation.

I want to name one real limitation I ran into during production evaluation: the Advisors API documentation is genuinely thin in places. I ended up reading Spring AI integration tests more than the reference guide to understand how chained advisors compose when you combine RAG retrieval with conversation memory. The team is improving this, but if you are onboarding a new engineer onto a Spring AI RAG pipeline today, budget extra time for that gap.

On the question of performance: LLM network latency dwarfs any Java framework overhead by two to three orders of magnitude. A detailed Java ecosystem analysis from early 2026 confirmed this directly: the model round trip is always the bottleneck, not the abstraction layer. Spring AI adds no meaningful latency to an operation that already takes hundreds of milliseconds or more.

One more honest thing to name: the API surface is still moving. Teams that pinned to Spring AI 1.0 found some adapter interfaces changed in 1.1. The upgrade path from 1.x to 2.0 will require careful attention given the Jakarta EE 11 baseline and Spring Boot 4 foundation shift. This is not a reason to avoid Spring AI. It is a reason to track minor versions more actively than you would Spring Data or Spring Security.

Spring AI vs LangChain4j: The Honest Comparison

Dimension	Spring AI 1.1	LangChain4j 1.x
Primary target	Spring Boot teams	Any JVM stack
AI provider count	20+	30+
Vector store count	12+	30+
MCP support	First-class	Supported
RAG tooling	Advisors API, ETL pipeline	Mature, granular pipeline control
Quarkus / Micronaut support	Minimal	Native extensions
Observability	Spring Boot Actuator native	Requires separate setup
Tool calling style	Spring beans as functions	Explicit `@Tool` annotation
GA release	May 2025	May 2025

The decision rule that emerges here is clean. Spring Boot team with existing observability and Spring Security setup? Spring AI is the path of least friction and lowest long-term maintenance cost. Non-Spring stack, or you need the broader provider catalog? LangChain4j is the reasonable alternative. Both are production-ready. The cost of switching six months into a project is real, so treat this as an architectural decision, not an implementation detail.

What This Means For You

If you are a backend engineer on a Spring Boot team, the question is not whether to use Spring AI. It is which capability to start with.

Start with ChatClient and a single provider. Get one endpoint working end to end. Add a VectorStore when you have a genuine document retrieval use case. Do not attempt RAG, MCP, tool calling, and observability simultaneously on the first sprint. The abstractions are composable by design, and you can add each layer when you actually need it.

If you are building internal tooling or copilot-style features over your platform's existing knowledge base (the category I am actively evaluating at OHB for our Digital Application Platforms team), the Advisors API and chat memory support are the most immediately useful pieces. They let you build context-aware assistants without standing up a Python sidecar service and all the operational overhead that brings.

For teams deciding whether to standardize on Spring AI for the long term: Spring AI 2.0 on Spring Boot 4 with Java 21 virtual threads is a serious platform for production agentic applications. The team shipped 2.0.0-M4 in March 2026 and RC1 in June 2026. General availability is close, and the migration story from 1.x is well-documented.

The Spring AI Community GitHub organization, announced at Spring I/O Barcelona 2025, has also created a formal home for community integrations and experimental projects that the core team cannot absorb directly. It is a signal of a framework that is building a serious ecosystem around itself, not just a well-funded core team working in isolation.

Questions Developers Are Actually Asking About Spring AI

What is Spring AI and how is it different from calling an LLM API directly?

Spring AI is an official Spring Framework project that provides portable abstractions for integrating AI models into Java applications, following the same design philosophy as Spring Data or Spring Cache. It abstracts vendor-specific HTTP APIs behind consistent interfaces so your application code does not change when you switch providers. Calling an LLM API directly works fine for a simple one-off integration but creates vendor lock-in, lacks built-in support for RAG, tool calling, and chat memory, and produces code that does not compose with your existing Spring observability and testing infrastructure.

When did Spring AI reach production readiness?

Spring AI 1.0 reached General Availability on May 20, 2025, announced by Mark Pollack, Christian Tzolov, and Josh Long. Spring AI 1.1 followed in November 2025 with 850+ improvements including full Model Context Protocol support and a structured Advisors API. As of June 2026, version 2.0 RC1 is available, built on Spring Boot 4.0, Spring Framework 7.0, and a Jakarta EE 11 baseline with Java 21 as the minimum runtime.

Does Spring AI support running models locally with Ollama?

Yes. Ollama is a first-class provider in Spring AI, configured in application.yml exactly like any cloud provider. This means you can run a Spring AI application against a locally hosted Llama, Gemma, or Mistral model during development and switch to a cloud provider in production without changing any application code. The abstraction layer makes provider selection a configuration decision, not a refactor.

What is the Advisors API in Spring AI and why does it matter?

The Advisors API is Spring AI's abstraction for encapsulating recurring AI patterns, like retrieving context from a vector store before sending a prompt (Retrieval Augmented Generation), or maintaining conversation history across requests (chat memory). Instead of wiring this logic manually in every controller, you register advisors once and they apply transparently to every ChatClient interaction. It is the AI equivalent of Spring AOP: cross-cutting concerns extracted from your business logic and applied declaratively.

How does Spring AI handle MCP support?

Spring AI 1.1 introduced first-class MCP support, allowing you to both consume MCP-compliant tool servers and expose your own Spring beans as MCP servers. Existing Spring-managed functions annotated with @Description become callable by any MCP-compatible AI agent, with the framework handling protocol compliance, tool registration, and multi-protocol version negotiation. OAuth2-secured MCP connections were included in the 1.1 development cycle, making production-grade deployment realistic from the start.

Should I use Spring AI or LangChain4j for a new project?

If your team is already on Spring Boot, Spring AI integrates natively with your existing auto-configuration, Actuator observability, and Spring Security setup, and tool calling works through your existing Spring beans with no separate registration step. If you are running Quarkus, Micronaut, or you need a broader catalog of out-of-the-box LLM provider integrations, LangChain4j is the more practical choice. Both frameworks hit 1.0 GA in May 2025 and are production-ready today. The team's existing stack is the primary deciding factor, not the frameworks themselves.

Where This All Lands in Ten Years

Java has been declared dead so many times that the obituaries have obituaries. In 1996, it was going to be displaced by browser applets doing the wrong thing. In 2010, by dynamic scripting languages. In 2015, by Go and Rust eating the systems programming space. In 2023, it was apparently going to be bypassed entirely by Python notebook culture as the substrate for AI development.

What keeps not happening is the death of the enterprise JVM. Too many critical systems. Too much accumulated organizational knowledge. Too many engineers who think clearly in types and interfaces and who understand what it costs to rewrite something that is actually working.

Spring AI is not a Java comeback story. Java never left. What Spring AI represents is the formal acknowledgment by the Spring team that the patterns which made Java productive in distributed systems, abstraction behind interfaces, inversion of control, configuration as a deployment concern rather than a code concern, are the same patterns that make AI integration maintainable at enterprise scale. The same principles that made Spring Data the default for database access in Java are now making Spring AI the default for model access.

The developer role is not disappearing. It is going somewhere more interesting. The systems that AI agents will build in ten years will themselves need to be designed, tested, debugged, and operated by engineers who understand how abstractions compose under production load. The engineers who learned those skills building Spring Boot microservices are, it turns out, unusually well-prepared for what is coming.

We just needed the framework to catch up.

References

Spring AI Official Project Overview: https://spring.io/projects/spring-ai
Spring AI Core Concepts Documentation: https://docs.spring.io/spring-ai/reference/concepts.html
Spring AI 1.0 GA Release Announcement (May 20, 2025): https://spring.io/blog/2025/05/20/your-first-spring-ai-1/
Spring AI 1.1 GA Release Notes (November 12, 2025): https://spring.io/blog/2025/11/12/spring-ai-1-1-GA-released/
Spring AI 2.0.0-M1 Release Announcement (December 11, 2025): https://spring.io/blog/2025/12/11/spring-ai-2-0-0-M1-available-now/
Spring AI 2.0.0-M4, 1.1.4, and 1.0.5 Release (March 2026): https://spring.io/blog/2026/03/26/spring-ai-2-0-0-M4-and-1-1-4-and-1-0-5-available/
Spring AI Community GitHub Organization Announcement: https://spring.io/blog/2025/10/07/spring-ai-community-announcement/
Spring AI Advisors API Reference Documentation: https://docs.spring.io/spring-ai/reference/api/advisors.html
Choosing a Java LLM Strategy in 2026: Spring AI vs LangChain4j, Java Code Geeks: https://www.javacodegeeks.com/2026/03/choosing-a-java-llm-integration-strategy-in-2026-spring-ai-1-1-vs-langchain4j-vs-direct-api-calls.html
The State of Coding the Future with Java and AI, Microsoft Java Developer Blog (May 2025): https://devblogs.microsoft.com/java/the-state-of-coding-the-future-with-java-and-ai/
Josh Long on Spring AI, AI Native Dev Podcast at DevOps UK (May 2025): https://tessl.io/podcast/josh-long/
Spring AI Integration: Building Intelligent Java Applications, Java Code Geeks (November 2025): https://www.javacodegeeks.com/2025/11/spring-ai-integration-building-intelligent-java-applications.html
Spring Boot 4, Spring AI, and AI-First Java Development, Java Code Geeks (March 2026): https://www.javacodegeeks.com/2026/03/spring-boot-4-spring-ai-and-ai-first-java-development.html
LangChain4j vs Spring AI: An Honest Side-by-Side for Java Backend Teams, Level Up Coding: https://levelup.gitconnected.com/langchain4j-vs-spring-ai-an-honest-side-by-side-for-java-backend-teams-b6c0ea370f28

About the Author

The New SDLC: A Senior Dev's Honest Take on Vibe Coding and Agentic Engineering

Sayed Ali Alkamel — Tue, 16 Jun 2026 14:52:29 +0000

TL;DR

Vibe coding is not a development methodology. It is an attitude toward verification: optional. That attitude does not scale past prototypes.

Agentic engineering puts AI inside a scaffold of constraints, tests, and feedback loops. The developer's primary output becomes the system that produces code, not the code itself.

The key formula from the Google whitepaper: Agent = Model + Scaffold. Most teams invest heavily in the model and ignore the scaffold. That is backwards.

Context engineering separates fast AI output from useful AI output. What you put in the context window matters more than which model you pick.

The factory model means your job is now designing the assembly line, not assembling each widget by hand.

What the New SDLC Actually Looks Like in 2026
The Agent Scaffold: What Actually Surrounds the Model
Context Engineering Is Not Prompt Engineering
AI Makes Experienced Developers Slower? About That
The Factory Model: Building the System That Builds Software
What This Means For You
Questions Developers Are Actually Asking About Agentic Engineering
The Deeper Arc
References
About the Author

Andrej Karpathy coined "vibe coding" in February 2025, and by early 2026 he was already calling it passé. The replacement he introduced at Sequoia Capital's AI Ascent 2026 event was "agentic engineering." Not because the tools changed names, but because the practice had matured past the point where the original framing was useful. By December 2025, he reported, AI agents were writing approximately 80% of his code. The question was no longer whether to let AI write code. The question was how to govern what comes back.

To understand why the framing change matters, you have to understand what programming has always actually been.

For 70 years, the deal between developers and machines was a one-sided negotiation. You learned the machine's language. Binary, then assembly, then C, then Java, then whatever framework won last year's framework wars. Every generation of languages reduced the translation tax, but never eliminated the fundamental transaction: human intent, compressed into syntax the machine could execute. The reason curly braces exist is not because they reflect how humans think. They exist because the parser needs them. Syntax was always just the price of admission. What large language models changed is that, for the first time, the machine is reaching back toward intent. The syntax tax is collapsing. What replaced it is a verification tax. And that tax is what the new SDLC is built around paying correctly.

What the New SDLC Actually Looks Like in 2026

As of early 2026, JetBrains data shows that 85% of professional developers regularly use AI coding tools, and 51% use them daily. Around 41% of all new code is estimated to be AI-generated. To make that concrete: in a standard ten-file Flutter feature, roughly four of those files were not typed by a human hand. The developer who reviews more code than they write is no longer exceptional. They are average.

The Google whitepaper "The New SDLC with Vibe Coding" (Osmani, Saboo, Kartakis, May 2026) frames the landscape as a spectrum, not a binary:

Dimension	Vibe Coding	Structured AI-Assisted	Agentic Engineering
Intent	Casual natural language prompts	Detailed prompts with examples	Formal specs, architecture docs, rule files
Verification	"Does it seem to work?"	Manual testing, spot-checks	Automated test suites, CI/CD gates, LM judges
Error handling	Paste error back into the prompt	Developer diagnoses root cause	Agents self-diagnose within defined bounds
Codebase understanding	Developer may not read the generated code	Selective review of critical paths	Comprehensive architecture review
Best scope	Prototypes, hackathons, personal scripts	Features in established codebases	Production systems, team-scale delivery
Risk	High (acceptable for throwaway code)	Moderate (human judgment at key checkpoints)	Low (systematic verification at every stage)

The key insight from the table: the differentiator is not which model you use. It is how much structure, verification, and human judgment surrounds the model's output.

I've been watching this pattern up close at Oman Housing Bank. The moment our team shifted from casual prompting to writing explicit rule files first, the quality of AI output changed more than any model upgrade had produced. The model had not changed. The context around the model had.

The traditional SDLC moved from waterfall to Agile over about two decades. AI is compressing the current iteration cycle dramatically, but unevenly. Implementation that once took a week can now take hours. Requirements, architecture, and verification remain stubbornly human-paced. The result is a different workflow, not a faster version of the old one. Boundaries between phases blur. Iteration cycles shorten from weeks to minutes in some areas. The developer's role shifts from primary implementor to system designer and quality arbiter.

The Agent Scaffold: What Actually Surrounds the Model

There is a temptation, when adopting AI coding tools, to treat the model as the system. A newer model comes out and the agent gets smarter. An older one and it gets worse. The model becomes the explanation for everything good and bad.

That intuition leads to the wrong investments.

The model is one input into a running agent. Everything else, the prompts, the tools, the rule files, the test runners, the observability layer, is the scaffold: the machinery that gives the model state, feedback, and constraints. The whitepaper equation is clean:

Agent = Model + Scaffold

The model is not psychic. It arrives at your repository with the same blank-slate uncertainty as a new engineer on their first day, except it cannot ask questions. It will infer. Without a rule file, those inferences will be wrong for your specific project in predictable ways: wrong state management pattern, wrong folder structure, wrong testing convention, wrong import path. The scaffold is the onboarding document the new engineer never gets to ask you to write.

Public benchmarks make the impact concrete. On Terminal Bench 2.0, one team moved a coding agent from outside the top 30 to the top 5 by changing only the scaffold, with no model change. A LangChain study raised an agent's benchmark score by 13.7 points by tweaking only the system prompt and middleware around a fixed model. Most agent failures, examined honestly, are scaffold configuration failures.

A practical scaffold for a Flutter project looks like this:

# AGENTS.md

## Stack
- Flutter 3.32+, Dart 3.8+
- Riverpod for state management
- Freezed for data classes
- dio + retrofit for networking

## Architecture
- Hexagonal architecture. Feature folders, not layer folders.
- Domain layer has zero Flutter imports.
- Repositories are interfaces. Concrete implementations live in the data layer.
- UI components are stateless where possible. State lives in providers.

## Hard Rules
- Never add a package not already in pubspec.yaml. Ask first.
- Tests live in test/, not alongside source files.
- No print() statements. Use the Logger package.
- Named parameters for any function with more than 2 arguments.
- Run flutter analyze before considering a task complete.
- Null safety is non-negotiable. No dynamic, no implicit nulls.

## Workflow
1. Read the relevant feature spec before generating any code.
2. Write the test first. Then implement to make the test pass.
3. If the change touches the domain layer, flag it for human review before proceeding.
4. After implementation, confirm that flutter test and flutter analyze both pass.

This file is not prompting. It is operating instructions. The developer who wrote it spent about an hour. Every subsequent AI interaction inside that project benefits from that hour. It compounds.

Context Engineering Is Not Prompt Engineering

A key distinction from the whitepaper: prompt engineering and context engineering are not the same practice.

Prompt engineering asks "how do I phrase this question to get a smarter answer from the model?" Context engineering asks "what would a new team member need to know to contribute correctly to this project, and how do I encode that knowledge in a form the AI can use?"

The difference in daily practice is significant. Prompt engineering optimizes a single turn. Context engineering optimizes the entire session, and when the context persists across sessions, it optimizes every future session as well.

Six types of context matter in any agentic workflow. Instructions define the agent's role, constraints, and operating bounds. Knowledge includes retrieved documents, architectural diagrams, and domain-specific data. Memory covers short-term session state (what just happened) and long-term project state (what the project is). Examples provide few-shot behavioral patterns from your own codebase, not generic patterns from the internet. Tools are the precise definitions of APIs, scripts, and services the agent can call. Guardrails are the hard constraints it cannot cross.

The docs around context engineering are genuinely thin right now. Most tools tell you to "add rules" without being specific about what granularity actually works in practice. I lost more time than I'd like to admit figuring out the line between rule files that were too broad to help and rule files too verbose to fit efficiently in context. The working heuristic: a rule should be short, specific, and paired with a concrete example of the behavior you want.

Static context (rule files, system instructions) is always loaded and expensive: every token is paid on every model call, regardless of relevance. Dynamic context (tool results, retrieved documents, windowed session history) is loaded on demand and costs only when the information is actually needed. Deciding what belongs in static versus dynamic context is a real engineering trade-off, worth the same deliberate attention as any other architectural decision.

The most effective pattern for managing dynamic context is what the whitepaper calls Agent Skills: structured, portable packages of procedural knowledge the agent loads only when the task matches. The agent sees lightweight metadata at startup, loads full instructions when a task matches, and pulls deep reference material only when explicitly needed. This is precisely what I've been exploring in the Flutter Agent Skills series.

AI Makes Experienced Developers Slower? About That

The fair criticism of AI coding tools is not that they produce bad code. The fair criticism is that experienced developers using early AI tools were, in controlled conditions, measurably slower on complex tasks.

METR published exactly this finding in July 2025. Sixteen experienced open-source contributors, working on repositories averaging 22,000 stars and over a million lines of code, took 19% longer on tasks when using AI tools. The same developers estimated they had been 20% faster. The gap between perception and measured reality was not a minor miscalibration. It was a complete disconnect, as MIT Technology Review noted in its coverage of the study.

METR's February 2026 follow-up complicated the picture. Their original methodology had a sampling problem: the developers who benefited most from AI tools refused to participate in no-AI conditions, even at $50/hour incentives. Heavy AI adopters systematically excluded themselves from the control group. The revised conclusion was genuinely honest: "We do not know if AI makes developers more productive."

What the broader data does show clearly is this: developers who use AI for well-specified, implementation-heavy tasks and maintain strong review practices report gains of 25 to 39% (JetBrains, Index.dev, 2026). Developers who accept output without review see bugs increase by 41% in some studies, and code quality deteriorate through increased churn and duplication.

The productivity gain from agentic engineering is real. But it is conditional. The condition is structure. Unstructured AI assistance in 2025 was slower because it introduced verification overhead without reducing specification overhead. Agentic engineering addresses this by investing upfront in the spec, the scaffold, and the test suite, so that the AI's implementation overhead replaces genuinely slower human implementation, rather than adding on top of it.

The Factory Model: Building the System That Builds Software

The mental model that ties everything above together is what the whitepaper calls the factory model. In this model, the developer's primary output is not code. It is the system that produces code.

That system includes specifications and context that define what needs to be built, agents that translate specifications into implementation, tests and quality gates that verify correctness, feedback loops that route failures back to agents for correction, and guardrails that constrain agents to predictable, safe behavior.

A factory manager does not assemble every widget by hand. They design the assembly line and ensure quality control. The modern developer designs the development system and ensures its output meets the required standard.

This reframes the developer's role in terms of two operating modes:

Conductor mode: real-time, hands-on direction. The developer is in the IDE, watching code appear, guiding with prompts and corrections, maintaining fine-grained control over what gets written. Tools like GitHub Copilot, Gemini Code Assist, Cursor, and Windsurf primarily support this mode. It is natural for engineers from traditional backgrounds because it preserves a sense of understanding at each step. The risk: if the developer personally directs every interaction, the throughput improvement from AI stays limited.

Orchestrator mode: the developer operates at a higher abstraction level. They define goals, assign them to agents, and review results without watching code appear line by line. Claude Code, GitHub Copilot's agent mode, Google Jules, and similar tools support this through async task execution in sandboxed environments. This mode is best for well-defined tasks: bug fixes, feature implementations against established patterns, test generation, codebase migrations.

The orchestrator mode requires a different skill set: writing precise specifications so an agent can execute without ambiguity, decomposing large tasks into appropriately sized units, quickly evaluating whether agent output meets the quality standard, and designing the constraints and feedback loops that keep agents productive.

Most developers move between both modes in a single day. The honest limitation here: orchestrator mode requires more upfront discipline, not less. You have to write the spec before you can delegate the implementation. Teams under sprint pressure will find this counterintuitive. The payoff comes in the second sprint, not the first. Teams that treat AI tooling as a shortcut to avoid specification work end up producing prototypes that ship by accident.

What This Means For You

For individual developers, the first concrete action is writing a rule file for each project you actively use AI on. Start with ten lines: stack, architecture constraints, hard rules, testing conventions. Add a rule every time an agent does something it should not do again. Over a few weeks, you will have a project-specific configuration that makes AI output dramatically more useful without any change to the model.

Write tests and evaluations before generating code. A well-written test suite communicates intent to the agent more precisely than any natural-language prompt. It also gives the agent something to run, fail against, and correct toward. A test is a specification that a machine can verify.

Review every line the agent produces that is going to ship. Be skeptical of anything that looks clever. Check that imported packages actually exist. Verify that error handling covers realistic failure modes, not just the happy path. Code your team does not understand becomes debugging cost your team cannot afford. This is doubly true in regulated industries like banking and fintech, where I have watched well-generated code fail in ways the agent never anticipated because it had no context for the regulatory constraint that made the edge case matter.

Maintain your core engineering skills deliberately. AI handles the routine so you can focus on the difficult. That arrangement only works if the foundational skills, debugging, system design, intuition for performance and correctness, stay sharp. Treat AI as a way to apply expertise at greater scale, not as a substitute for developing it.

For engineering leaders, treat rule files, system prompts, eval suites, and skill libraries as code: versioned, reviewed in pull requests, owned by named engineers. Without this discipline, agent behavior becomes irreproducible across team members.

Set the bar at the evaluation, not the demo. A working demo proves an agent can succeed once. A passing eval suite proves it succeeds reliably. A demo is a sample size of one.

Distinguish prototyping work from production work in team norms. Vibe coding is the right speed for exploration. Agentic engineering is the right discipline for production. Make the boundary explicit: which projects, which branches, which environments warrant which mode of working. Teams that keep this distinction blurry produce prototypes that ship by accident.

Questions Developers Are Actually Asking About Agentic Engineering

What is the difference between vibe coding and agentic engineering?

Vibe coding means prompting an AI, accepting the output, and using "does it seem to work?" as the sole verification criterion. Agentic engineering means AI operates as an implementation engine inside a structure of formal specs, rule files, automated tests, and feedback loops, with humans retaining oversight over architecture, correctness, and quality. The differentiator is not which tools you use. It is how much structure surrounds the model's output. A developer can vibe code or apply agentic engineering using the exact same agent.

What is context engineering in software development?

Context engineering is the discipline of structuring and curating the information provided to an AI agent so it can act correctly on your specific codebase, architecture, and constraints. It goes beyond prompt phrasing to include rule files (AGENTS.md, CLAUDE.md), retrieved documentation, few-shot examples from your own code patterns, tool definitions, and hard guardrails. Research and practitioner evidence consistently shows that the quality of AI-generated code depends less on how clever the prompt is and more on how well the context reflects what your project actually requires.

What should go in an AGENTS.md or CLAUDE.md file?

A rule file should include your stack and versions, architectural patterns and conventions (the patterns you actually use, not generic ones), hard rules the agent cannot break such as packages to never add or patterns to avoid, your testing approach and where tests live, and a workflow the agent should follow before generating code. Start with ten lines and add rules when the agent does something wrong. A short, high-signal rule file consistently outperforms a long, sprawling one because every token in static context is paid on every model call.

Is AI actually making developers more productive in 2026?

The evidence is mixed and strongly context-dependent. JetBrains and Index.dev data puts self-reported productivity gains at 25 to 39% for developers using AI tools regularly. METR's July 2025 study found experienced developers took 19% longer on complex tasks with AI tools, though their February 2026 follow-up acknowledged that heavy AI adopters had systematically excluded themselves from the control group, undermining the result. The clearest consistent finding: developers who use AI for well-specified implementation tasks and maintain strong review practices gain substantially more than those who accept output without verification.

What is the 80% problem in AI-assisted development?

The 80% problem refers to the observation that AI agents can rapidly generate approximately 80% of the code for any feature, but the remaining 20% (edge cases, error handling, integration points, and subtle correctness requirements) demands deep contextual knowledge that current models consistently lack. This remaining 20% is where most production failures originate. It requires human architectural judgment and domain expertise that cannot be delegated to an agent without very precisely specified context. The developers who navigate this most effectively use AI for the well-specified 80% while reserving their own attention for the 20% that requires knowing why the constraint exists, not just that it does.

How does the factory model change the developer's role?

The factory model treats the developer's primary output as the system that produces code, not the code itself. The system includes specifications, agent configurations, test suites, feedback loops, and guardrails. A factory manager does not assemble every widget by hand. They design the assembly line and ensure quality control. The valuable skill shifts from syntax fluency to specification quality, architectural judgment, and evaluation rigor. Karpathy's framing at AI Ascent 2026 captures it: "You are not writing the code directly 99% of the time. You are orchestrating agents who do, and acting as oversight."

The Deeper Arc

Step back far enough and there is a pattern here that predates every modern framework.

In the 1950s, the bottleneck was hardware availability: only a handful of institutions had machines worth programming. In the 1970s, it was languages: only specialists who could reason in assembly were useful. In the 1990s, it was networking and distribution. In the 2010s, it was infrastructure: who had enough servers. In each transition, the skills that mattered most shifted, and a new population of people gained the ability to build things that had previously required a much smaller, more specialized group.

Every abstraction layer in computing history made the skills below it less central. Writing directly in assembly became less valuable when C arrived. Knowing the details of TCP/IP became less essential when frameworks abstracted the network. And critically, in every case, the engineers who treated the new abstraction as a threat to their skills were not wrong that those specific skills would matter less. They were wrong that they themselves would matter less. The abstraction raised the ceiling on what a skilled engineer could build, every single time.

We are at that inflection point again. The syntax tax is not fully gone. But it is low enough that the verification tax is now the dominant cost. The bottleneck is no longer implementation capacity. It is specification quality, architectural judgment, and the ability to evaluate whether a system that can generate enormous volumes of code is generating the right enormous volumes of code.

The factory model, the conductor-orchestrator split, context engineering: these are not replacements for engineering skill. They are the next abstraction layer. The developers who learn to work at this layer will build things that developers who stay below it simply cannot.

Generation is solved. Verification, judgment, and direction are the new craft.

References

Osmani, A., Saboo, S., Kartakis, S. "The New SDLC with Vibe Coding." Google, May 2026. (Attached PDF)
Karpathy, A. "Vibe Coding." X/Twitter post, February 2025. https://x.com/karpathy/status/1886192184808149383
Analytics Drift. "Karpathy Declares Vibe Coding Obsolete, Introduces Agentic Engineering at Sequoia AI Ascent 2026." May 2026. https://analyticsdrift.com/andrej-karpathy-agentic-engineering-software-3/
METR. "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
METR. "Uplift Update: Measuring the Impact of AI Coding Tools." February 2026. https://metr.org/blog/2026-02-24-uplift-update/
Index.dev. "Top 100 Developer Productivity Statistics with AI Tools 2026." https://www.index.dev/blog/developer-productivity-statistics-with-ai-tools
Modall. "AI in Software Development: 25+ Trends and Statistics (2026)." https://modall.ca/blog/ai-in-software-development-trends-statistics
MIT Technology Review. "AI Coding Is Now Everywhere. But Not Everyone Is Convinced." January 2026. https://www.technologyreview.com/2025/12/15/1128352/rise-of-ai-coding-developers-2026/
IBM. "What Is Agentic Engineering?" April 2026. https://www.ibm.com/think/topics/agentic-engineering
The New Stack. "From Vibes to Engineering: How AI Agents Outgrew Their Own Terminology." February 2026. https://thenewstack.io/vibe-coding-agentic-engineering/
Glide Blog. "What Is Agentic Engineering? How AI Engineering Has Evolved Past Vibe Coding in 2026." February 2026. https://www.glideapps.com/blog/what-is-agentic-engineering
Augment Code. "Why AI Coding Tools Make Experienced Developers 19% Slower and How to Fix It." January 2026. https://www.augmentcode.com/guides/why-ai-coding-tools-make-experienced-developers-19-slower-and-how-to-fix-it
Osmani, A. "The Factory Model." https://addyosmani.com/blog/factory-model/
Second Talent. "84% of Developers Use AI Tools. Productivity Gains Are Only 10%. Here's Why." May 2026. https://www.secondtalent.com/resources/ai-developer-productivity-tools-2026/

About the Author

Is Token Usage the New Lines of Code? How to Measure Developer Productivity in the AI Age

Sayed Ali Alkamel — Tue, 16 Jun 2026 14:08:28 +0000

Is Token Usage the New Lines of Code? How to Measure Developer Productivity in the AI Age

"Whatever gets measured gets gamed." — Goodhart's Law

An OpenAI engineer processed 210 billion tokens in a single week. A Claude Code user spent over $150,000 in a month. Salesforce quietly set a $175 minimum monthly token-spend target for engineers, and developers started asking AI to summarize documents they already understood, just to hit the number.

Welcome to tokenmaxxing. The newest chapter in software engineering's oldest and most embarrassing story: measuring the wrong thing, confidently, at scale.

This article will give you the unfiltered answer to a question every engineering leader and developer is quietly debating right now. Is measuring token usage the same as measuring lines of code? And if so, what should we actually be measuring?

The Ghost We Thought We Buried: Lines of Code

Fred Brooks knew it in 1975. In The Mythical Man-Month, he made it plain that adding more programmers to a late project makes it later. Productivity in software is not linear, it is not additive, and it cannot be reduced to volume.

And yet, for decades, companies fell for the simplest metric available: lines of code (LOC). Count the output, reward the output, get more output. What they actually got was bloated codebases, unnecessary complexity, and engineers who learned to pad their work to look busy.

Kent Beck, creator of Extreme Programming, said it directly: "The way you get programmer productivity is not by increasing the lines of code per programmer per day. That does not work. The way you get programmer productivity is by eliminating lines of code you have to write."

The industry eventually accepted this. LOC as a metric became a cautionary tale. We moved on to commits, pull requests, story points, velocity, cycle time. Every single one of these was gamed the moment it became a target. Agile teams discovered that story points originally meant as capacity planning proxies had transformed into performance theater.

Then AI arrived. And the industry, with astonishing speed, invented a brand new broken metric.

Tokenmaxxing: Lines of Code in a Lab Coat

Token budgets are the new LOC. The logic sounds compelling at first: if a developer is using more AI compute, they must be building more things, right?

Wrong. Measuring token consumption as a proxy for productivity makes the same fundamental error as counting lines of code. It measures an input to the process, not the output, and certainly not the outcome.

TechCrunch's April 2026 investigation into tokenmaxxing found that engineers with the largest token budgets produced the most pull requests, but the productivity improvement did not scale. They achieved two times the throughput at ten times the cost of tokens. The tools were generating volume, not value.

Faros AI drew on two years of customer data and found that code churn, lines of code deleted versus lines added, had increased 861% under high AI adoption. GitClear's January 2026 report found that regular AI users averaged 9.4x higher code churn than their non-AI counterparts, more than double the productivity gains the tools provided.

More code is being written. Most of it is not sticking.

Alex Circei, CEO of Waydev, which tracks developer analytics across more than 10,000 engineers, put the problem precisely: engineering managers are seeing code acceptance rates of 80 to 90 percent, but they are missing the churn that happens when engineers have to revise that accepted code in the following weeks, which drives the real-world acceptance rate down to between 10 and 30 percent of generated code.

Token usage measures how aggressively a developer consumes AI compute. It says nothing about whether the output shipped, whether it held up in production, whether it was the right thing to build, or whether a senior engineer would need to rewrite it three weeks later.

What the Research Actually Says (and It Is Uncomfortable)

The METR randomized controlled trial from July 2025 tested 16 experienced open-source developers across 246 real tasks. The result shocked the industry.

Developers using AI tools completed tasks 19% slower than the control group.

The more alarming part: before the study, developers predicted AI would speed them up by 24%. After completing it, they still estimated they had been 20% faster, despite objective measurement proving the opposite. The perception gap was total.

This is not an argument against AI tools. The same period saw Google's 2025 DORA report show that AI adoption correlates with higher software delivery throughput for teams that have invested in strong engineering foundations first. The difference lies in how AI is being used and, critically, how productivity is being measured.

Martin Fowler, Chief Scientist at Thoughtworks and one of the most influential voices in software engineering, wrote in August 2025 that most LLM usage in the industry is "fancy auto-complete," and that the developers getting the most value are those who allow AI to directly read and edit source code files, not just suggest snippets. Fowler has long argued that measuring individual developer productivity is a "fool's errand," and the AI era has only deepened that conviction. Optimizing for the speed of code production, he argues, misses the point entirely, much like measuring a novelist's productivity by words per minute rather than by the quality of the narrative.

Nicole Forsgren, creator of DORA and SPACE, and author of Accelerate, said it plainly in late 2025: "AI broke our developer productivity metrics. Lines of code? Meaningless. Commits? Not the point. Velocity? Can be misleading. We need new frameworks for measuring DevEx in the age of AI."

So What Should You Actually Measure?

The answer is not a single number. It never was. Here are the frameworks and signals that hold up:

1. Business Outcomes, Not Activity Proxies

The only honest question is: did the software deliver value? Did the feature ship and work? Did it reduce support tickets, increase revenue, or improve retention? These are hard to measure in a sprint, which is exactly why organizations reach for activity proxies. Resist it.

2. Code Durability

Instead of counting how many lines were written or tokens consumed, track how much of that code survives. Code durability, the ratio of code that stays in the codebase after 30, 60, and 90 days, is a far more honest signal of whether the work was good. High churn under AI adoption is a red flag that should be on every engineering dashboard in 2026.

3. The SPACE Framework

Developed by Nicole Forsgren and colleagues from GitHub and Microsoft Research, SPACE measures five dimensions simultaneously: Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow. The key insight is that no single dimension tells the full story. A team shipping more PRs but burning out is not a productive team.

4. DORA Metrics (With AI-Era Adjustments)

Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Mean Time to Restore remain the gold standard for measuring the health of your delivery pipeline. In the AI era, Change Failure Rate and Mean Time to Restore are more important than ever, because the speed gains from AI mean you can ship broken code faster too. Add AI attribution and complexity-adjusted throughput to see the full picture.

5. Specification Quality

Here is the new skill nobody is measuring yet. In the Agentic Engineering era that Andrej Karpathy described at AI Ascent 2026, the developer's primary output is increasingly the specification, not the code. The human role shifts from writing code to owning the spec, design, and judgment calls. A developer who writes a rigorous, well-scoped specification that agents can execute correctly is more productive than one who burns tokens on vague prompts and spends days fixing hallucinated output.

The Skills That Actually Matter Now

Kent Beck said something quietly devastating: "90% of my skills are now worth $0. But the other 10% are worth 1000x."

That 10% is not typing speed. It is not syntax memorization. It is not the ability to recall a library API from memory.

The skills that compound in the AI age are:

Problem formulation. The ability to articulate what needs to be built clearly enough that an agent can execute it without ambiguity.

Architectural judgment. You can outsource implementation to agents. You cannot outsource the ability to catch a subtle logical error or a design decision that will haunt the codebase for years.

Agent orchestration. Running parallel agents, reviewing their outputs critically, understanding when to trust and when to reject. This is the new technical skill floor.

Context curation. Knowing what context to give an AI and what to leave out. A developer who extracts the 20 relevant lines from a 10,000-line codebase and frames the right question consistently outperforms one who pastes the entire repo and hopes for the best.

None of these skills show up in a token count.

The Verdict

Is measuring token usage the same as measuring lines of code?

Yes. Structurally, they are the same mistake.

Both measure an input to the process instead of the output. Both are easily gamed once they become a target. Both create perverse incentives: write more code, burn more tokens, look productive while potentially reducing the quality and maintainability of the system. And both measure the wrong end of the value chain entirely.

The engineers doing the most valuable work in 2026 might be consuming far fewer tokens than their tokenmaxxing colleagues. They might be spending hours in a whiteboard session killing a bad idea before a single prompt is written. They might be writing a specification so precise that an agent executes the entire feature on the first pass. That work is nearly invisible to any activity-based metric.

Goodhart's Law will always win. When token spend becomes a target, it will cease to be a good measure. It will just become the new leaderboard. And we will spend another decade learning the same lesson Fred Brooks already taught us in 1975.

The question is not how much AI compute a developer burns. The question is whether the software ships, holds up, and matters.

Everything else is noise.

References and Further Reading

Martin Fowler, Some thoughts on LLMs and Software Development, martinfowler.com, August 2025
Becker et al., Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, METR / arXiv:2507.09089, July 2025
Tim Fernholz, Tokenmaxxing is making developers less productive than they think, TechCrunch, April 2026
Nicole Forsgren, Frictionless: Seven Steps to Help Engineering Teams Move Faster in the Age of AI, 2025
Faros AI, AI Acceleration Whiplash, March 2026
GitClear, Developer Cohort Analysis: AI Coding Output, January 2026
Andrej Karpathy at Sequoia Capital AI Ascent 2026, karpathy.bearblog.dev
Kent Beck, TDD, AI agents and coding, The Pragmatic Engineer, July 2025
Gergely Orosz, How AI is changing Software Engineering, The Pragmatic Engineer, April 2026
Fred P. Brooks Jr., The Mythical Man-Month: Essays on Software Engineering, 1975

Sayed Ali Alkamel is Manager of Digital Application Platforms at Oman Housing Bank, a Google Developer Expert in Dart and Flutter, and co-founder of Flutter MENA. He ships production systems at the intersection of fintech, mobile, and AI infrastructure.

DEV Community: Sayed Ali Alkamel

Prompt Chaining: Fixed Steps That Each Feed the Next

What is prompt chaining?

The gate: your cheapest safeguard

How it actually works

When to use it

When not to use it

Known problems

Three things to know before you start

FAQ

Sources

The Augmented LLM: The Building Block Every Agent Sits On

What is the augmented LLM?

The three augmentations

How it actually works

When to use it

When not to use it

Known problems

Three things to know before you start

FAQ

Sources

The New SDLC for Project Managers: Vibe Coding vs Agentic Engineering

What is vibe coding, and why should a project manager care?

What changes in the software development life cycle?

Who does what now: conductors, orchestrators, and the harness

What it costs: the CapEx and OpEx flip

Three things for project managers to hold onto

FAQ

Sources

Dart FFI for Flutter: Call Native C Directly (Lessons from camera_pro)

What is Dart FFI?

FFI or platform channels?

How the native C actually ships

What FFI looked like in camera_pro

Three things to know before you start

FAQ

Sources

The LLM Preamble Problem: How RLHF Made Your Model Too Polite to Ship

Table of Contents

Why Every Instruction-Tuned LLM Has a Preamble Habit

The RLHF Root Cause

What the Preamble Actually Costs

Four Fixes for the LLM Preamble, Ranked by Reliability

The Skeptic's Objection

What This Means For You

Questions developers are actually asking about LLM preamble

Why do LLMs say "Certainly!" before answering?

Does the preamble problem affect all models equally?

What is the most reliable way to suppress LLM preamble?

Can I detect LLM preamble in a CI/CD pipeline?

Does the LLM preamble cost real money at scale?

The Longer View

References

About the Author

GGUF, Explained: The File Format That Put LLMs on Your Laptop

Contents

Why does a model need a special file format at all?

What is actually inside a GGUF file?

What do the quantization names like Q4_K_M actually mean?

How much quality do you actually lose?

The fair criticism: isn't 4-bit just lossy compression that makes the model dumber?

How do you convert a model to GGUF yourself?

How to pick a quant for your hardware

Questions developers are actually asking about GGUF

What does GGUF stand for?

What is the difference between GGUF and safetensors?

Which GGUF quantization should I use?

Can GGUF models run on a GPU or only on the CPU?

Is Q4_K_M good enough, or do I need Q8_0?

How do I convert a model to GGUF?

The quiet format that changed who gets to run AI

References

About the author

AI Gateways: A Senior Engineer's Honest Take

Table of contents

What developers are actually fighting when AI hits production

What is an AI gateway, exactly?

AI gateway vs API gateway: what actually changes

The core features worth paying for in an AI gateway

What AI gateway options exist in 2026?

DataModel Binding with `A2uiSchemas.stringReference`