DEV Community: Piyush Singh

How a Large Language Model Actually Generates Text

Piyush Singh — Mon, 27 Jul 2026 12:14:05 +0000

Everything an AI agent does — writing code, answering support tickets, browsing the web — bottoms out in a single operation: a language model predicting the next token. Understand that one mechanism, and its limits, and the entire "AI agent" stack stops being magic and starts being engineering.

Let's build it up from nothing.

1. Text becomes tokens

A model never sees letters. The first thing that happens to your prompt is tokenization: the text is chopped into tokens — sub-word chunks produced by an algorithm called Byte-Pair Encoding (BPE). A token is roughly ¾ of an English word (about 4 characters). Common words like "the" are a single token; "unbelievable" might be three (un / believ / able). Each token maps to an integer ID in a fixed vocabulary of tens of thousands of entries.

Why should you care? Because everything downstream is counted and billed in tokens, not words. Context limits, latency, and cost are all token-denominated. When someone says a model has a "200K context window," they mean tokens.

2. The transformer reads everything at once — through attention

Token IDs become embedding vectors (lists of numbers), which flow through a stack of transformer layers. The engine of each layer is self-attention, and it's the idea that made modern AI work.

Self-attention lets every token look at every other token to build context — for example, linking the pronoun "it" back to the noun it refers to. Mechanically, each token is projected into three vectors:

a Query ("what am I looking for?"),
a Key ("what do I offer?"),
a Value ("what do I contribute if attended to?").

A token's attention to another is the dot product of its Query with the other's Key, squashed through a softmax so the weights are positive and sum to 1. The output for each token is a weighted blend of the Value vectors. Run several of these in parallel — the original Transformer used eight attention heads — and concatenate the results. Because attention has no built-in sense of order, a positional encoding is added so the model knows word order.

Here's the catch that shapes the whole industry: attention compares every token to every other token, so n tokens create n² relationships. That quadratic cost is the root reason context windows are finite, expensive, and degrade as they fill.

3. It generates one token at a time

An LLM is autoregressive. The final layer produces a logit vector — a raw score for every token in the vocabulary — which softmax turns into a probability distribution over "the next token." The model picks one, appends it to the input, and repeats. That loop is text generation.

So how does it "pick"? Through sampling knobs:

Greedy decoding always takes the single most probable token. Deterministic, but repetitive.
Temperature sharpens or flattens the distribution. Low temperature → focused and repetitive; high temperature → diverse but risks incoherence.
Top-p (nucleus) sampling draws only from the smallest set of tokens whose probabilities sum to p. It adapts to the model's confidence at each step, which is why it's usually preferred over top-k.
Repetition penalty (~1.2–1.5) down-weights tokens it has already produced, to break loops.

4. Why models hallucinate

This is the part people find unsettling: the model's objective is plausible next tokens, not truth. It has no built-in database and no grounding step. When it lacks a fact, it still produces the most likely-sounding continuation — confidently.

Hallucination isn't a bug waiting for a patch; it's a property of the mechanism. And it's precisely why the rest of the stack exists. Retrieval (RAG), tool use, and grounding all do the same job: feed the model real facts inside its context so that "most plausible" and "correct" line up.

5. The context window is the model's entire memory

The single most important thing to internalize: an LLM is a stateless function. Tokens in, next-token distribution out. It remembers nothing between calls. Its only working memory is the context window — the tokens you pass in this call (system prompt + conversation history + tools + retrieved documents).

Two consequences shape all of agent engineering:

Every token has an opportunity cost. Recall accuracy degrades as the window fills — more tokens means worse retrieval from within them. This is sometimes called "context rot," and the n² attention cost is why.

Lost in the middle. A well-known 2023 study ("Lost in the Middle") showed that models attend best to information at the start and end of the context and worst to the middle, producing a U-shaped accuracy curve — even in models explicitly built for long context. Simply moving a relevant passage to the middle can degrade the answer without changing a single word of content.

The one idea to carry forward

An LLM is a stateless, next-token predictor with a fixed attention budget. Memory, tools, agents, RAG, and caching are all machinery bolted around that stateless core — to give it state, facts, actions, and efficiency.

Once you see it this way, "agents" stop being a mysterious new species. They're a control loop wrapped around a very good autocomplete. In the next article, we'll build that loop.

From a Language Model to an Agent: The Loop That Changes Everything

Piyush Singh — Mon, 27 Jul 2026 12:10:23 +0000

A large language model, on its own, can only do one thing: emit text. It can't check your calendar, run a test suite, or refund a payment. So how did we get from "very good autocomplete" to systems that book travel and fix codebases?

The answer is almost embarrassingly simple: you wrap the model in a loop and give it tools. That's an agent. Everything else is detail — but the details are where production systems live or die.

What actually counts as an "agent"

There's a lot of hype-driven vagueness here, so let's use the definitions the labs themselves use.

OpenAI's working definition: an agent is a system that uses an LLM to control workflow execution. It decides the next step, recognizes when the task is complete, self-corrects, and can hand control back to a human. Crucially, things that embed an LLM but don't let it control the flow — a simple chatbot, a single-turn call, a sentiment classifier — are explicitly not agents.

Anthropic draws the same line differently: a workflow runs LLMs and tools through predefined code paths; an agent lets the model dynamically direct its own process and tool usage. The dividing line is who decides what happens next — your code, or the model.

Every agent is built from three components:

Model — the LLM doing the reasoning and deciding.
Tools — functions and APIs it can call to take action.
Instructions — the guidelines and guardrails that define behavior.

Tools come in three flavors: data tools (query a database, read a PDF, search the web), action tools (send an email, update a CRM record), and orchestration tools (other agents, exposed as callable tools).

The agent loop (the "run")

The core mechanism is a loop — often called a run — that repeatedly calls the model until an exit condition is met. Each pass accumulates state in the context, so a decision made late in the run still carries the residue of every earlier step.

Assemble context — system prompt + tool definitions + conversation history + any retrieved data.
Call the model — it returns either a final answer or a request to call a tool.
If it's a tool call, execute it — your harness (not the model) runs the actual function and captures the result.
Feed the result back — append it to the context and loop.
Exit — on a final answer, no tool calls, an error, or a max-turn cap.

Anthropic frames the same cycle as gather context → take action → verify work → repeat. That word verify matters: agents get "ground truth" from tool results and code execution at each step — real feedback from the environment, instead of reasoning in a vacuum. A coding agent that runs the tests after editing is doing exactly this.

Tool calling: the exact mechanics

This is the single most important mechanism to understand, because it's how a text model "does things." Walk through it carefully:

1. You declare tools as JSON Schema. Each tool is a name, a description, and a parameters schema, passed in the API's tools field. That description is not decoration — a vague one will misdirect the model entirely.

2. The model decides. With the default tool_choice: "auto", it may call zero, one, or several tools. When it decides a tool is needed, it interrupts its own prose and emits structured JSON arguments that match the schema — for example, a location argument with the value "Paris".

3. Your code executes. You parse the tool call (function name + JSON arguments), run the real function, and get a result. The model never executes anything itself. It only emits intentions; your harness does the work. This separation is also your security boundary.

4. You feed the result back. Append a message with role tool, the matching tool_call_id, and the result content — then call the model again so it can incorporate the output into its next step.

Structured outputs

To guarantee valid JSON (not just hope for it), strict mode constrains generation to the schema (additionalProperties: false, all fields required). Without it, JSON output is best-effort: you validate after generation and re-prompt on failure — a feedback loop, not a guarantee. For anything downstream-critical, use the constrained version.

Design the interface like a product

Anthropic makes a point worth stealing: the Agent–Computer Interface (ACI) deserves as much design care as a human UI. That includes error-proofing ("poka-yoke") — designing tool signatures so the model can't easily make a mistake. Clear names, tight schemas, and helpful error messages do more for reliability than any amount of clever prompting.

Think about it from the model's side: it's operating your system blind, through a keyhole of text. Good tools are self-describing, hard to misuse, and fail loudly with actionable messages.

Single agent first

One last, load-bearing piece of advice before the fancy stuff: start with a single agent. A single model in a loop with well-designed tools solves a surprising share of real problems. Reach for multiple agents only when specialization genuinely removes a bottleneck — because, as we'll see later in the series, more agents means distributed-systems complexity, not maturity.

Key takeaways

An agent = an LLM + tools + instructions, run in a loop the model controls.
The model only emits intentions; your harness executes them and feeds results back.
Tools are declared as JSON Schema; strict mode makes outputs reliable.
Treat the tool interface like a product surface — clarity beats cleverness.

When SSL Expiry Takes Down Giants: What Happened with Cricbuzz

Piyush Singh — Mon, 19 May 2025 11:51:26 +0000

⚠️ When SSL Expiry Takes Down Giants: What Happened with Cricbuzz — And How zop.dev Would Have Prevented It

On 19th May 2025, Cricbuzz packed with live cricket action, millions of users were shocked to find Cricbuzz, one of the world’s most popular cricket platforms, completely inaccessible.

The culprit?

An expired SSL certificate.

🔒 “This domain was pending verification and has since been verified. It may take 24–48 hours for the website to come back online.”

🌐 What Actually Happened?

Cricbuzz’s domain was temporarily suspended due to unverified WHOIS information, as mandated by ICANN (Internet Corporation for Assigned Names and Numbers).

Since January 1, 2014, ICANN requires that:

All new domain registrations and contact detail changes must be verified within 15 days
If not, the domain is automatically suspended by the registrar
The site remains down until verification is complete

So what users saw

🔒 _“This domain was pending verification and has since been verified. It may take 24–48 hours for the website to come back online.”

❌ What Happens When an SSL Certificate Expires?

Secure Sockets Layer (SSL) certificates are what allow your browser to create an encrypted connection with a website. When these certificates expire, users are immediately warned that the website is no longer secure — and most browsers block access altogether.

That’s exactly what happened with Cricbuzz.

Users were greeted with messages like:

"Your connection is not private."

"NET::ERR_CERT_DATE_INVALID"

For a high-traffic site like Cricbuzz, especially during ongoing tournaments, this kind of downtime is costly — both in lost ad revenue and in damage to user trust.

💥 Why SSL Expiry Still Happens — Even in 2025

Despite SSL certificates being foundational to web security, many organizations still rely on manual processes to renew and install them.

The reality is:

SSL certificates (especially from Let’s Encrypt) often expire every 90 days
Renewal requires attention, scripting, and integration
Any delay—even by minutes—can cause a global outage
Most teams don’t monitor certificates proactively

In short: SSL expiry is still a very real threat, even for large enterprises with DevOps and SRE teams.

🛡️ How zop.dev Prevents SSL Outages — Automatically

At zop.dev, we’ve seen this problem too many times. That’s why SSL automation is built in by design.

Here’s how we do it:

✅ 1. Automated Issuance & Renewal

zop.dev uses cert-manager, an open-source Kubernetes add-on that automates the entire lifecycle of TLS certificates. Whether you're using Let’s Encrypt or your internal CA, zop.dev ensures your SSL certs are:

Requested automatically
Issued and stored as Kubernetes Secrets
Renewed automatically before expiry

🔄 2. No Human Involvement Required

Once configured, you don’t need to touch your SSL setup again:

Renewal is scheduled before expiry
Failures are retried automatically
Ingress controllers or services are auto-reloaded after renewal

You won’t even get a Slack alert — because there’s no incident to respond to.

🔐 3. Seamless HTTPS for Every Service

Whether you’re running one microservice or hundreds, zop.dev ensures that every public endpoint is:

Secured with HTTPS
Always backed by a valid, trusted certificate
Reloaded with zero downtime

🚀 SSL Is Just One Piece of What zop.dev Offers

While we’re proud of our SSL automation, it’s just one part of zop.dev’s mission:

To streamline infrastructure and deployment so you can build and scale without firefighting.

With zop.dev, you get:

Automated Kubernetes deployments
Managed autoscaling and cron jobs
Secure Redis & MySQL integrations
Built-in monitoring & alerting
Zero YAML

⚡ Don’t Let a Certificate Be Your Single Point of Failure

Cricbuzz’s SSL expiry is a wake-up call:

Even the biggest, most visited platforms are vulnerable to the smallest oversights.

But they don’t have to be.

With zop.dev, SSL expiry is one less thing you ever have to think about again.

🔗 Learn more about how zop.dev automates SSL management in Kubernetes: https://zop.dev

📬 Got questions? Let’s talk about how we can help your team ship faster — without surprises.