DEV Community: Sri Balaji

Secure API Design: Building APIs That Resist Abuse

Sri Balaji — Tue, 22 Sep 2026 15:18:26 +0000

⚡ TL;DR: The server must trust nothing the client sends. Design APIs that authenticate every route, enforce ownership server-side, and validate input, so an attacker changing one ID in a URL cannot read someone else's data.

The bug that ships in every junior API
One principle: the server trusts nothing the client sends
The request gauntlet
The controls, and what each one stops
The insecure endpoint vs. the secure one
Common mistakes that cost hours (or headlines)
Takeaways
Where to go next

The bug that ships in every junior API

You build an invoices API. A logged-in user fetches their bill at GET /api/invoices/1043. It works, the demo is clean, you ship it. A week later someone is bored and types GET /api/invoices/1044. They are still logged in as themselves, but they are now reading another customer's invoice. Name, address, line items, total. Then they write a loop from 1 to 9999 and download everyone's.

Nothing crashed. No error appeared. The code did exactly what it was told: "find invoice 1044 and return it." The problem is what it was never told, "...but only if it belongs to the person asking." This is the single most common serious flaw in real APIs, and it is called IDOR (Insecure Direct Object Reference), or BOLA (Broken Object Level Authorization) in the OWASP API Top 10. It sits at number one for a reason.

📌 Who this is for: Developers who can already build a working REST or GraphQL endpoint and now want it to survive a hostile internet. You do not need a security background, if you can read a route handler, you can follow this. We use TypeScript and Python, but the principles are framework-agnostic.

One principle: the server trusts nothing the client sends

The server trusts nothing the client sends. Not the body, not the headers, not the URL, not even the user id the client claims to be.

The one rule under every control in this article

Every API security control is really one idea applied in a different place. The client is not your code running in a friendly browser, it is anything that can open a socket: a script, a proxy, curl, a malicious app. The user id, the resource id, the price field, the isAdmin flag, all of it is just attacker-controllable text until your server independently verifies it.

In the real world	In tech
A boarding pass proves who you are	Authentication, a verified token identifies the caller
Your seat assignment is yours, not whoever's number you shout	Authorization, ownership checks tie a resource to the caller
Security scans every bag, even frequent flyers'	Input validation, every request body is screened, no exceptions
Only so many people board per minute	Rate limiting, throughput is capped to stop abuse and scraping
The jet bridge is sealed end to end	TLS, the channel is encrypted from client to server

The same airport security model, mapped onto an HTTP request.

The request gauntlet

A secure request runs a gauntlet. Each stage can reject it, and the order matters: cheap checks (is the channel encrypted? is the caller known? are they over their limit?) run before expensive ones (parse the body, hit the database, check ownership). By the time your handler runs, every assumption has already been verified.

Every request runs the gauntlet left to right. Any stage can reject; the handler only runs once all of them pass.

TLS terminates: The request arrives over HTTPS. Plaintext on port 80 is redirected to 443. Nothing, tokens, bodies, cookies, travels in the clear.
Authenticate the caller: A signed token (session cookie or JWT) is verified. No valid token on a protected route means 401. The server now knows who is asking, not who the client claims to be.
Check the rate limit: Has this caller exceeded their quota for this window? If so, 429 before any real work happens. This caps brute-force, scraping, and accidental loops.
Validate the input: The body and params are parsed against a strict schema. Unknown fields, wrong types, missing required values, or out-of-range numbers are rejected with 400, before they reach business logic.
Authorize the action and the object: Two questions: may this role do this action, and does this specific resource belong to this caller? Failing either is 403 (or 404 to avoid confirming the resource exists).
Run the handler: Only now does the handler execute, and it queries the database scoped to the caller's id. Every step gets recorded to an audit log.

The controls, and what each one stops

There is no single "make it secure" switch. Each control closes a specific class of attack. Skip one and you leave that door open, no matter how solid the others are.

Control	What it stops
TLS everywhere	Eavesdropping and tampering, stolen tokens and modified requests on the wire
Authentication on every route	Anonymous access to protected data; relying on "nobody knows the URL"
Object-level authorization	IDOR / BOLA, reading or editing other users' objects by guessing ids
Input validation (allow-list)	Injection, type-confusion, and oversized payloads reaching your logic
Rate limiting	Brute-force, credential stuffing, scraping, and accidental DoS
No mass assignment	Privilege escalation by sending fields like `role` or `isAdmin` in the body
Strict CORS	Malicious sites making authenticated cross-origin calls on a user's behalf
Generic error responses	Information leaks, stack traces, SQL, and internal ids handed to attackers

Map each control to the concrete abuse it prevents.

The insecure endpoint vs. the secure one

Here is the invoices endpoint as it usually ships first. It authenticates (you must be logged in) and then... trusts the id from the URL completely. This is the IDOR.

routes/invoices_insecure.py

# INSECURE, do not ship this
@app.get("/api/invoices/{invoice_id}")
def get_invoice(invoice_id: int, user=Depends(current_user)):
    # We checked the user is logged in (authentication).
    # We NEVER checked the invoice belongs to them (authorization).
    invoice = db.query(
        "SELECT * FROM invoices WHERE id = :id", id=invoice_id
    )
    if not invoice:
        # Leaks the DB error and the SQL to the client
        raise Exception(f"no invoice {invoice_id}: query failed")
    return invoice  # returns ANY user's invoice

Three flaws in nine lines: no ownership check (IDOR), a verbose error that leaks internals, and a query scoped only by id. Now the secure version. The fix is small but the order is everything, validate, then authorize against the caller's id, then return a generic error.

routes/invoices.py

from fastapi import HTTPException
from pydantic import BaseModel, conint

class InvoicePath(BaseModel):
    invoice_id: conint(gt=0)  # input validation: positive int only

@app.get("/api/invoices/{invoice_id}")
def get_invoice(invoice_id: int, user=Depends(current_user)):
    InvoicePath(invoice_id=invoice_id)  # 400 on garbage input

    # Authorization happens IN the query: scope by the caller's id,
    # not just the resource id. The DB can only ever return
    # invoices this user owns.
    invoice = db.query_one(
        "SELECT * FROM invoices WHERE id = :id AND owner_id = :uid",
        id=invoice_id, uid=user.id,
    )
    if invoice is None:
        # 404, not 403, don't confirm the row exists to a stranger.
        # Generic message: no SQL, no stack trace, no internal ids.
        raise HTTPException(status_code=404, detail="Invoice not found")

    return invoice

The same shape in TypeScript, and this time guarding against mass assignment on an update. Notice we never spread the request body into the database, we pick exactly the fields a user is allowed to set, and we set owner_id from the verified token, never from the body.

routes/invoices.ts

import { z } from "zod";

// Allow-list: the ONLY fields a client may send. No role, no
// owner_id, no isPaid, those are server-controlled.
const UpdateInvoice = z.object({
  note: z.string().max(500),
  dueDate: z.string().datetime(),
}).strict(); // .strict() rejects unknown keys outright

app.patch("/api/invoices/:id", requireAuth, async (req, res) => {
  const id = Number(req.params.id);
  if (!Number.isInteger(id) || id <= 0)
    return res.status(400).json({ error: "Bad request" });

  const parsed = UpdateInvoice.safeParse(req.body);
  if (!parsed.success)
    return res.status(400).json({ error: "Bad request" });

  // Ownership + update in one scoped statement.
  const updated = await db.invoice.updateMany({
    where: { id, ownerId: req.user.id }, // authZ lives here
    data: parsed.data,                   // only allow-listed fields
  });

  if (updated.count === 0)
    return res.status(404).json({ error: "Invoice not found" });

  return res.json({ ok: true });
});

💡 Tip: Push authorization into the query whenever you can, WHERE owner_id = :me. A separate "fetch, then check in code" step is one forgotten if away from an IDOR. If the database can only return your rows, the bug is structurally impossible.

Common mistakes that cost hours (or headlines)

Authorization only in the UI. Hiding the "Delete" button does nothing, the endpoint is still one curl away. Every check must live on the server. The client is for convenience, never for security.
IDOR / BOLA. Authenticating but not verifying ownership. If GET /orders/123 works for anyone logged in, you are leaking every order. Scope every query by the caller's id.
Mass assignment. Spreading req.body straight into a model lets a user send "role": "admin" or "balance": 999999. Always allow-list the writable fields explicitly.
Verbose errors. Returning stack traces, SQL, or "user exists but wrong password" hands attackers a map. Log the detail server-side; return a generic message and a correct status code.
No rate limiting. Without it, login endpoints get credential-stuffed and list endpoints get scraped. Cap requests per caller per window, and return 429 when exceeded.
Wide-open CORS. Access-Control-Allow-Origin: * with credentials lets any site call your API as your logged-in user. Allow-list specific origins; never reflect the request origin blindly.

Takeaways

The whole article in seven lines

The server trusts nothing the client sends, body, headers, URL, or claimed identity.
Authenticate on every protected route; "unguessable URL" is not authentication.
Authorize the object, not just the action: scope every query by the caller's id to kill IDOR/BOLA.
Validate input with a strict allow-list schema; reject unknown fields and bad types with 400.
Never spread request bodies into your models, allow-list writable fields to stop mass assignment.
Rate-limit, lock down CORS to known origins, and return generic errors with correct status codes.
TLS everywhere. No plaintext, ever.

Where to go next

Secure API design sits at the intersection of identity, input handling, and transport. Each of those is its own deep topic, and each has a companion article on this site.

Authentication vs Authorization, the difference between who you are and what you may touch, which is exactly the IDOR fix.
Preventing Injection Attacks (SQLi & XSS), what rigorous input validation defends against, in depth.
HTTPS, TLS & Encryption Basics, how the encrypted channel at the front of the gauntlet actually works.
Build the muscle memory on the DevOps Engineer path, where these controls show up in real pipelines and deploys.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Preventing Injection Attacks (SQLi, XSS & Friends)

Sri Balaji — Mon, 21 Sep 2026 16:43:46 +0000

⚡ TL;DR: Injection is one bug class wearing many masks: untrusted input treated as code. One rule, never let data become code, kills SQLi, XSS, and command injection at once. Use parameterized queries and allowlists, not blocklists.

A search box that ran DROP TABLE
The principle: data is not code
The picture: input crossing the boundary
The injection family at a glance
Code: the vulnerable query and the fix
Why allowlists beat blocklists
Common mistakes that get people breached
Takeaways
Where to go next

A search box that ran DROP TABLE

A user types into your site's search box. Most people search for "running shoes". One person types '; DROP TABLE products;--. A few seconds later, your products table is gone, your store is throwing 500s, and your on-call phone is buzzing. Nobody hacked your server, broke your firewall, or stole a password. They just typed text into a box, and your code ran that text as a database command.

That is injection, and it is the single most common root cause of serious breaches. The mechanism is always the same: your program takes input it does not control and feeds it somewhere that input can become executable, a SQL query, an HTML page, a shell command. The attacker writes the code; your app obediently runs it. The good news is that one mental model defends against the entire family at once.

📌 Who this is for: Junior and mid-level developers who write code that touches a database, renders user content, or shells out to the OS, which is nearly everyone. No security background needed. If you have ever built an WHERE clause by gluing strings together, this article is for you. It pairs with The OWASP Top 10, Explained and Secure API Design.

The principle: data is not code

Treat all input as hostile. Your job is to make sure data stays data, never let it cross the line and become code.

The one rule behind every injection fix

Every injection vulnerability is the same bug wearing different clothes. Somewhere, your code builds a string that is part structure (the query, the markup, the command) and part untrusted value (what the user typed). When those two are concatenated into one blob and handed to an interpreter, the interpreter has no way to tell where your intent ends and the attacker's begins. A stray quote, angle bracket, or semicolon flips the value into structure, and now the user is writing your code.

The fix is not to scrub away every dangerous character. The fix is to keep data and code in separate channels so the interpreter never has to guess. That is what parameterized queries, output encoding, and argument arrays all do, different mechanisms, one idea.

In the real world	In tech
A mail-merge template: "Dear {name}, your order ships today."	A query template: SELECT * FROM users WHERE name = ?
The {name} field is filled in from a spreadsheet of addresses.	The ? placeholder is filled in with the user's input as a value.
Someone writes a macro into the name column, and the merge runs it.	Concatenated input becomes executable SQL/HTML/shell instead of a value.
Fix: the field is always text, never instructions, no matter what's in it.	Fix: the input is always bound as a value, never parsed as code.

Injection is a mail-merge gone rogue.

The picture: input crossing the boundary

Picture untrusted input arriving at a trust boundary, the edge of your system. Everything past that boundary is your code and your data store. The boundary's job is to decide how the input is allowed through. Take the safe path and the input is bound, escaped, or rejected; take the unsafe path, string concatenation, and the input flows straight into the interpreter as code.

Untrusted input hits a boundary. The safe path keeps data as data; the unsafe path lets it become code.

Input arrives: A request carries values you did not write, query params, JSON body, headers, cookies. Assume every one is attacker-controlled.
Hit the boundary: Before the value touches any interpreter, route it through validation and a safe-construction mechanism. This is the only place the decision is made.
Validate the shape: Check the value is what you expect, an integer ID, an email, one of a fixed set of sort columns. Reject anything that fails (allowlist), don't try to clean it.
Bind, don't build: Pass the value as a parameter to a prepared statement, or encode it for the exact output context. The interpreter receives structure and data on separate channels.
Execute safely: The query runs with the input as a literal value; the page renders it as visible text; the command treats it as a single argument. Data never became code.

The injection family at a glance

Three interpreters, three contexts, one bug. Notice that the fix column is the same idea each time, separate the data channel from the code channel, in the language of whatever interpreter you are talking to.

Type	Where it hits	The fix
SQL injection (SQLi)	Database queries built by concatenating input	Parameterized queries / prepared statements, bind input as values
Cross-site scripting (XSS)	HTML/JS rendered with un-encoded input in the browser	Context-aware output encoding; use textContent, not innerHTML; CSP as defense in depth
Command injection	OS commands run via a shell with input in the string	Pass args as an array to the program directly; never invoke a shell with concatenated input

Same root cause, different interpreter.

Code: the vulnerable query and the fix

Here is the classic SQLi bug. The input is glued into the query string, so a crafted value rewrites the query. The infamous ' OR '1'='1 turns a login check into "return every row," and '; DROP TABLE ...;-- runs a second statement entirely.

vulnerable.py

# DANGER: input is concatenated straight into SQL
def find_user(username):
    query = "SELECT * FROM users WHERE username = '" + username + "'"
    return db.execute(query)

# Attacker sends:  username = "' OR '1'='1"
# Resulting query: SELECT * FROM users WHERE username = '' OR '1'='1'
#   -> matches every row
#
# Attacker sends:  username = "'; DROP TABLE users;--"
# Resulting query: SELECT * FROM users WHERE username = ''; DROP TABLE users;--'
#   -> runs a second, destructive statement

The fix is a parameterized query. The ? (or %s, or $1, driver-dependent) is a placeholder. You hand the driver the query and the values as two separate arguments. The driver sends the SQL structure and the data over different channels, so the value is always treated as a literal, even if it contains quotes, semicolons, or whole SQL statements.

safe.py

# SAFE: query structure and values travel separately
def find_user(username):
    query = "SELECT * FROM users WHERE username = ?"
    return db.execute(query, (username,))   # username is bound as a VALUE

# Attacker sends:  username = "' OR '1'='1"
# The driver looks for a user literally named  ' OR '1'='1
#   -> zero rows. The input never becomes SQL.
#
# Rule of thumb: if you are using string concatenation, +, f-strings,
# or .format() to build SQL, you have a bug. Always parameterize.

XSS is the same mistake in the browser. If you drop user input into the page as markup, a <script> payload executes in your visitors' sessions. The fix is context-aware output encoding: encode the value for the exact place it lands. The simplest version, assign to textContent (or let a framework escape by default) so the value renders as visible text, never as HTML.

output-encoding.ts

// DANGER: input becomes live HTML
function renderComment(comment: string) {
  el.innerHTML = "<p>" + comment + "</p>";
  // comment = '<img src=x onerror="steal(document.cookie)">'
  //   -> the onerror handler runs in your user's session
}

// SAFE: input is rendered as text, not parsed as markup
function renderComment(comment: string) {
  const p = document.createElement("p");
  p.textContent = comment;   // < > & are shown literally, never executed
  el.replaceChildren(p);
}

// In React/Vue/Angular, {value} / {{ value }} encode by default.
// The danger is only when you opt out: dangerouslySetInnerHTML, v-html.
// If you MUST render rich HTML, sanitize first (e.g. DOMPurify),
// and add a Content-Security-Policy as defense in depth.

Why allowlists beat blocklists

When people first meet injection, the instinct is to block the bad stuff, strip out ', ban the word DROP, filter <script>. This is a blocklist, and it loses every time. You are trying to enumerate infinity. Attackers have endless encodings: SeLeCt, URL-encoding, Unicode homoglyphs, nested tags like <scr<script>ipt> that survive a single naive strip, comment tricks, and case games. Miss one and you are owned.

An allowlist flips the logic: define what is valid and reject everything else. A user ID must match ^[0-9]+$. A sort column must be one of {"name", "created_at", "price"}. A country must be a known ISO code. You no longer need to imagine every attack, anything that is not explicitly allowed is gone. Validation is about shape and meaning; it complements (never replaces) parameterizing and encoding.

allowlist.py

# Allowlist: only known-good values get through.
ALLOWED_SORT = {"name", "created_at", "price"}

def list_products(sort_by):
    if sort_by not in ALLOWED_SORT:        # reject, don't sanitize
        sort_by = "name"
    # Identifiers (column names) can't be parameterized, so an
    # allowlist is exactly the right tool here.
    query = f"SELECT * FROM products ORDER BY {sort_by}"
    return db.execute(query)

💡 Use both layers: Parameterize/encode is your primary defense, it makes injection structurally impossible at the interpreter. Allowlist validation is the second layer, and the only option when you can't parameterize (e.g. column or table names). Defense in depth means an attacker has to beat both.

Common mistakes that get people breached

String-concatenated queries. Any +, f-string, or .format() building SQL is a latent SQLi. Parameterize every value, no exceptions, even "internal" or "admin-only" endpoints.
Blocklist filtering. Stripping ', banning keywords, or regex-matching "bad" patterns. Attackers route around blocklists with encoding and casing tricks. Allowlist the valid shape instead.
innerHTML with user data. Assigning untrusted input to innerHTML (or dangerouslySetInnerHTML / v-html) parses it as live markup. Use textContent, or sanitize with a vetted library if rich HTML is truly required.
Trusting client-side validation. The browser check is UX, not security, attackers call your API directly and skip it entirely. Always re-validate and parameterize on the server.
Shelling out with concatenated strings. os.system("ping " + host) is command injection. Pass arguments as an array (subprocess.run(["ping", host])) so no shell parses the input.

Takeaways

The whole article in seven lines

Injection = untrusted input getting treated as code. It is the #1 root-cause class.
One rule fixes the whole family: never let data become code, keep them in separate channels.
SQLi → parameterized queries. XSS → context-aware output encoding. Command injection → argument arrays, no shell.
Validate input as an allowlist (known-good shape); reject the rest. Never blocklist.
Allowlists beat blocklists because you can't enumerate every attack, only every valid value.
Client-side validation is UX; the server must re-validate and parameterize.
Layer it: parameterize/encode as the primary defense, allowlist validation as the backstop, CSP as defense in depth.

Where to go next

Injection is one slice of a bigger picture. See how it ranks and connects in The OWASP Top 10, Explained, then learn where these defenses sit at the edge of a service in Secure API Design.

Bake validation and parameterization into your service templates from day one, follow the DevOps Engineer path to see where security checks fit in the pipeline.
Practice spotting unsafe string-building in code review, treat every concatenated query, command, or markup string as a finding.
Add a Content-Security-Policy header to your apps as an XSS backstop, and wire dependency/secret scanning into CI so injection isn't your only line of defense.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Secrets Management: Keeping Keys, Passwords, and Tokens Out of the Wrong Hands

Sri Balaji — Sun, 20 Sep 2026 14:23:47 +0000

⚡ TL;DR: Keys, passwords, and tokens never belong in Git, logs, or screenshots. Learn to inject secrets at runtime, store them in a vault, rotate on a schedule with least privilege, and scan to catch leaks before attackers do.

The 3 a.m. cloud bill
What counts as a secret
How secrets should actually flow
Where NOT to put secrets (and where to)
Reading a secret the right way
Rotation and least-privilege
Detecting leaked secrets
Common mistakes that cost hours (or accounts)
Takeaways
Where to go next

The 3 a.m. cloud bill

A developer pushes a quick fix on a Friday evening. Buried in the diff is a config file with an AWS access key, committed "just to get it working." The repo is public. Within minutes, automated bots scraping GitHub find the key. By Saturday morning the account has spun up hundreds of GPU instances mining cryptocurrency, and the bill reads five figures. The developer did not get hacked through some clever exploit. They handed over the key themselves and pushed it to the internet.

This is not a rare horror story. Scanners watch public Git pushes in real time, and a leaked credential is often abused before the developer has even noticed it is gone. Secrets are the single highest-leverage thing to protect, because one leaked key can bypass every other control you built.

📌 Who this is for: Developers and ops folks who have ever pasted an API key into a config file, dropped a password into a .env, or wondered where credentials are supposed to live. No prior security background needed, we start from "what is a secret" and end with rotation and leak detection.

What counts as a secret

A secret is any value that grants access or proves identity and would cause harm if a stranger held it: API keys, database passwords, OAuth tokens, TLS private keys, signing keys, SSH keys, webhook signing secrets. The defining trait is simple, if it leaks, someone can act as you. Treat anything matching that test as a secret, not as ordinary config.

A secret is only secret while exactly the systems that need it can read it, and nothing else can.

That sentence is the whole discipline. Everything below, where you store secrets, how you hand them to apps, how often you change them, who can read them, is just making that one sentence true and keeping it true.

In the real world	In tech
You don't tape your house key to the front door	Don't hardcode credentials in source or commit them to Git, that's leaving the key on the lock
You don't make 100 copies and hand them out	Grant least-privilege access, each service gets only the secrets it needs, nothing more
If a key is lost, you change the locks	Rotate the secret immediately on suspected leak; the old value stops working
A locksmith keeps the master copies in a safe	A secret manager (Vault, AWS Secrets Manager, KMS) is the safe, one guarded source of truth

Secrets behave exactly like the keys to your house.

How secrets should actually flow

The goal is that a secret never lives in your codebase, your container image, or your Git history. Instead the app fetches it at startup or runtime from a dedicated secret manager, holds it in memory, and never writes it down. Behind the scenes a rotation loop keeps swapping the stored value for a fresh one.

An app authenticates to the secret manager at startup, the secret is injected into memory at runtime, and a rotation loop quietly replaces it on a schedule.

App boots with no secret: The image and the repo contain zero credentials. There is nothing to leak even if the image is pulled or the repo is cloned.
App proves who it is: It authenticates to the secret manager using its workload identity, an IAM role, a Kubernetes service account, or a short-lived token, not a stored password.
Manager checks the policy: The secret manager confirms this identity is allowed to read this specific secret, then returns it over an encrypted channel.
Secret is injected at runtime: The value lands in the process's memory (or a tmpfs-backed env var). It is never written to disk, the image, or version control.
Rotation runs in the background: On a schedule, the manager issues a fresh value and expires the old one. Apps re-fetch and keep working, a leaked old copy becomes useless.

Where NOT to put secrets (and where to)

Most leaks come from picking the wrong storage location, not from sophisticated attacks. Here is how the common options actually compare on the three things that matter: how exposed the secret is, whether you can rotate it cleanly, and whether you can see who used it.

Approach	Leak risk	Rotation	Audit trail
Hardcoded in source	Severe, lives forever in Git history, images, and every clone	Painful, code change + redeploy everywhere	None
.env committed to Git	Severe, same as hardcoding, just in a different file	Painful, and old commits still hold it	None
Env var (set at deploy, not in Git)	Moderate, can leak via logs, crash dumps, `/proc`, child processes	Manual, redeploy to change it	Weak, no record of reads
Secret manager (Vault / AWS SM / KMS)	Low, encrypted at rest, scoped by policy, short-lived	Automated, rotate on a schedule, no redeploy	Full, every read is logged

The further down this table you go, the smaller the blast radius when something goes wrong.

⚠️ Warning: A .env file is fine for local development as long as it is in .gitignore and never committed. The danger is the moment it lands in version control, at that point it is identical to hardcoding the secret, and deleting it later does not remove it from Git history.

Reading a secret the right way

The code that consumes a secret should fetch it from the environment or a manager client, never embed the literal. The example below reads a database password from a secret manager with an env-var fallback for local dev, and fails loudly if it is missing rather than silently using a blank value.

db.py

import os
import boto3
from functools import lru_cache


@lru_cache(maxsize=1)
def get_db_password() -> str:
    # Local dev: read from an env var that is NEVER committed.
    local = os.environ.get("DB_PASSWORD")
    if local:
        return local

    # Production: fetch from AWS Secrets Manager at runtime.
    secret_id = os.environ["DB_SECRET_ID"]  # just a name, not the value
    client = boto3.client("secretsmanager")
    resp = client.get_secret_value(SecretId=secret_id)
    return resp["SecretString"]


def connect():
    password = get_db_password()
    # ... open the connection; never log `password`
    return password  # used in-memory only


# NEVER do this:
# DB_PASSWORD = "hunter2-prod-9f3a"   # hardcoded, leaks on first commit

Notice three things: the value of the secret never appears in the file (only the secret's name, DB_SECRET_ID), the result is cached so you do not hammer the manager on every call, and there is no print(password) anywhere, secrets in logs are one of the most common silent leaks.

Rotation and least-privilege

Two practices turn "we have a secret manager" into "we are actually safe." Rotation means every secret has an expiry; the manager issues a new value and retires the old one on a schedule, so a copy stolen six months ago is already dead. The gold standard is dynamic secrets, credentials minted on demand that live for minutes, not months.

Least-privilege access means each identity can read only the exact secrets it needs. The billing service has no business reading the email provider's API key. Scope policies per-service so a single compromised workload exposes one secret, not the whole vault. Pair this with the audit log: if you cannot answer "who read this secret and when," you cannot tell whether a leak has been used.

Detecting leaked secrets

Assume a secret will eventually slip through. Your job is to catch it fast. Run a secret scanner in pre-commit hooks and in CI so a credential is blocked before it ever reaches the remote, and enable your Git host's push protection (GitHub and GitLab both scan pushes for known token formats).

scan.sh

# Block secrets before they are committed (run as a pre-commit hook)
gitleaks protect --staged --verbose

# Scan full history for anything already leaked
gitleaks detect --source . --report-path leaks.json

# If a real secret is found in history, ROTATE it first,
# then purge from history (e.g. git filter-repo), deleting the
# file in a new commit does NOT remove the old value.

💡 Tip: When a secret leaks, rotate first, clean up second. Revoking the credential makes the leaked copy worthless immediately; scrubbing Git history is housekeeping that can wait minutes. Doing it in the other order leaves a live key exposed while you wrestle with history rewrites.

Common mistakes that cost hours (or accounts)

Committing secrets to Git. Even a deleted .env lives forever in history. Once pushed, treat the secret as compromised and rotate it, do not just remove the file.
Never rotating. A secret that has not changed in two years has likely been copied into someone's notes, a Slack message, or a screenshot. Set an expiry on everything.
Granting broad access. One "admin" policy that can read every secret turns any single compromised service into a full breach. Scope per-service.
Leaking secrets in logs. print(token), verbose error traces, and crash dumps quietly ship credentials to your log aggregator, which often has far weaker access controls than your vault.
Putting secrets in container image layers or CI variables in plaintext. Images get pushed to registries; CI logs get shared. Inject at runtime instead.

Takeaways

Secrets management in seven lines

A secret is anything that lets someone act as you, keys, passwords, tokens, certs.
Never hardcode secrets or commit .env files; Git history is forever.
Store secrets in a dedicated manager (Vault, AWS Secrets Manager, cloud KMS).
Inject at runtime into memory using workload identity, not baked into images.
Rotate on a schedule so a stolen copy expires; prefer short-lived dynamic secrets.
Grant least-privilege: each service reads only the secrets it needs, and every read is logged.
Scan for leaks in pre-commit and CI; if one leaks, rotate first, then clean history.

Where to go next

Secrets management is one pillar of a secure delivery pipeline. Build the surrounding picture next:

Start with the fundamentals: What Is Application Security? frames where secrets fit among the broader threats.
Then extend the discipline to your build inputs: Securing the Software Supply Chain covers dependencies, provenance, and signing.
Practice the delivery mechanics in the CI/CD lab, where injected secrets and pipeline variables come to life.
See how this fits a full role on the DevOps Engineer path.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

The OWASP Top 10, Explained

Sri Balaji — Sat, 19 Sep 2026 14:04:36 +0000

⚡ TL;DR: A beginner-followable tour of the OWASP Top 10: what each web risk really is, a concrete example, and the specific fix. You will leave able to use the list as a working checklist, not just recite it.

The list every attacker has already read
One sentence, then a picture
The OWASP Top 10 at a glance
A few of these, made concrete
How to use the Top 10 in practice
Fixing a real one: a server-side ownership check
Common mistakes that cost hours
Takeaways
Where to go next

The list every attacker has already read

You ship a feature. It works in the demo, the tests are green, the PR is approved. Three weeks later someone changes a number in a URL and reads another customer's invoices. Nothing crashed. No alarm fired. The code did exactly what you wrote, you just never told it who was allowed to ask.

Almost every breach you read about traces back to a small, named, well-understood mistake. The OWASP Top 10 is the industry's shared list of those mistakes, the ten categories of web application risk that show up most often and hurt the most. It is maintained by the Open Worldwide Application Security Project, a nonprofit, and it is the single best starting map for anyone who builds or operates web software.

📌 Who this is for: Developers, DevOps and platform engineers, and anyone shipping a web app or API who wants a working mental model of how things get broken, and a checklist they can act on this week. No security background assumed. If you can read a function, you can follow this.

We will walk all ten: what each one is, a concrete example, and how to prevent it. Then we will turn the list into a practical routine you can actually run.

One sentence, then a picture

The OWASP Top 10 is not a list of bugs, it is a list of the ten ways trust gets misplaced in a web application.

The mental model worth keeping

Here is the analogy. Think of your application as a building. Most of the Top 10 are not exotic break-ins, they are the unlocked side door, the master key that opens every room, the contractor you let in without checking ID, and the security camera that was never plugged in.

In the real world	In tech
An unlocked side door anyone can walk through	Broken Access Control, the server never checks who you are
Valuables left on a desk by the window	Cryptographic Failures, sensitive data stored or sent in the clear
A contractor you let in without checking ID	Injection, untrusted input treated as trusted instructions
A camera installed but never plugged in	Logging & Monitoring Failures, the attack happened, nobody saw it

Each abstract risk maps to something physical you already understand.

Before the list, look at where these risks actually land. A web app is not one thing, it is a chain of trust boundaries, and each risk hits a specific link.

The attack surface of a typical web app, and which Top 10 risks strike at each layer.

The happy path is the straight line: browser to API to database. The two dashed branches, your dependencies and your config, are the ones teams forget, and they are where two whole categories of the Top 10 live.

The OWASP Top 10 at a glance

This is the centerpiece. Read it once top to bottom for the shape of the landscape, then keep it open as a reference. The order mirrors the 2021 list, which is ranked by real-world prevalence and impact.

Risk	What it is	How to prevent it
A01 Broken Access Control	Users can act outside their permissions, read, edit, or delete data that isn't theirs by changing an ID, URL, or method.	Deny by default. Enforce ownership and role checks server-side on every request, never in the UI alone.
A02 Cryptographic Failures	Sensitive data (passwords, tokens, PII) is stored or transmitted weakly, or not encrypted at all.	TLS everywhere; encrypt data at rest; hash passwords with bcrypt/argon2; never invent your own crypto.
A03 Injection	Untrusted input is interpreted as code or commands, SQL, NoSQL, OS, LDAP, letting attackers rewrite your query.	Use parameterized queries and ORMs; validate and escape input; never string-concatenate user data into commands.
A04 Insecure Design	The flaw is in the plan, not the code, a missing security control the design never accounted for.	Threat-model before building; add abuse cases to stories; use secure design patterns and rate limits by default.
A05 Security Misconfiguration	Default passwords, verbose errors, open cloud buckets, unnecessary features left enabled.	Harden every environment; disable defaults; automate config so dev, staging and prod match; minimize attack surface.
A06 Vulnerable & Outdated Components	A dependency (library, framework, OS package) has a known CVE and you are still running it.	Inventory dependencies; scan continuously; patch on a schedule; remove what you don't use.
A07 Identification & Auth Failures	Weak login: brute-forceable passwords, broken session handling, missing MFA.	Enforce strong passwords + MFA; rate-limit and lock out; use a vetted auth library; rotate and expire sessions.
A08 Software & Data Integrity Failures	Code or data is trusted without verifying it wasn't tampered with, unsigned updates, poisoned CI/CD pipelines.	Verify signatures and checksums; lock dependency versions; secure your build pipeline; never auto-load untrusted code.
A09 Logging & Monitoring Failures	Attacks happen and nobody notices, no logs, no alerts, no way to investigate after the fact.	Log security events with context; centralize and alert; test that alerts actually fire; keep an incident plan.
A10 Server-Side Request Forgery (SSRF)	The server is tricked into making requests to internal systems the attacker can't reach directly.	Validate and allowlist outbound URLs; block internal IP ranges; isolate egress; never fetch raw user-supplied URLs.

The OWASP Top 10, each risk, what it actually is, and the one move that prevents most of it.

💡 Tip: Notice that A01 (Broken Access Control) sits at the top. In OWASP's 2021 data it was the most common serious issue by a wide margin, more applications had it than any other category. If you fix one thing first, fix access control.

A few of these, made concrete

Tables are great for scanning, but risks click when you see them happen. Three quick ones.

Broken Access Control (A01)

Your invoice page is at /invoices/1043. A curious user changes it to /invoices/1042 and sees someone else's bill. The server happily returned it because the only check was "are you logged in?", never "is this yours?". This is called an insecure direct object reference (IDOR), and it is the most common breach you will ever cause by accident.

Injection (A03)

A login form builds a query like SELECT * FROM users WHERE name = '" + input + "'. The attacker types ' OR '1'='1 as the username, and suddenly the WHERE clause is always true. They are in. The fix is mechanical: stop building queries by gluing strings together. We cover this in depth in Preventing Injection Attacks (SQLi & XSS).

Server-Side Request Forgery (A10)

Your app has an "import image from URL" feature. An attacker submits http://169.254.169.254/latest/meta-data/, the cloud metadata endpoint, and your server, which can reach it, fetches and returns your cloud credentials. The user could never reach that address; your server could, and you let it become a proxy.

⚠️ Warning: The pattern across all three: the server trusted input it should have questioned. Almost the entire Top 10 is variations on that one sentence.

How to use the Top 10 in practice

A list you read once and forget changes nothing. Here is how to turn it into a routine that actually catches issues, on a feature, a service, or a whole codebase.

Map your attack surface: Sketch the layers from the diagram above for your own app: client, API, database, dependencies, config. You can't defend trust boundaries you haven't drawn.
Walk the data flow: Follow one real request end to end. At every hop ask: where does untrusted input enter, and where do I trust it without checking? Those points are your candidates.
Score each risk against your app: Go down the Top 10 row by row and ask "could this happen here?" Mark each Yes / No / Not sure. "Not sure" is the answer that needs follow-up.
Fix highest-impact first: Start with A01 access control and A03 injection, they are common and devastating. Don't polish logging while the side door is unlocked.
Automate the repeatable parts: Add a dependency scanner (A06) and secret/config checks (A05) to CI so regressions get caught without a human remembering to look.
Re-run it on every meaningful change: New endpoint, new dependency, new integration? Re-walk the relevant rows. Security is a habit, not a one-time audit.

Fixing a real one: a server-side ownership check

Let's fix the A01 example from earlier. The vulnerable version checks that you are logged in, then trusts the ID in the URL. The fixed version checks that the record actually belongs to you, on the server, where the user can't tamper with it.

invoices.py

# VULNERABLE: any logged-in user can read any invoice
@app.get("/invoices/<invoice_id>")
@login_required
def get_invoice(invoice_id):
    invoice = db.invoices.find_by_id(invoice_id)
    return jsonify(invoice)  # never checked who owns it


# FIXED: deny by default, enforce ownership server-side
@app.get("/invoices/<invoice_id>")
@login_required
def get_invoice(invoice_id):
    invoice = db.invoices.find_by_id(invoice_id)

    # 1. Does it exist at all?
    if invoice is None:
        abort(404)

    # 2. Does it belong to the caller? (the check that matters)
    if invoice.owner_id != current_user.id:
        # 404, not 403, don't even confirm the record exists
        abort(404)

    return jsonify(invoice)

Two things to notice. First, the ownership check uses current_user.id from the verified session, never an ID sent by the client. Second, when the record isn't yours we return 404, not 403: a 403 quietly confirms the invoice exists, which leaks information. Deny by default, and reveal nothing.

💡 Tip: Best practice is to centralize this. Instead of repeating the ownership check in every handler, wrap it in a policy/authorization layer (e.g. a decorator or middleware) so a new endpoint is secure by default and a forgotten check is impossible rather than easy.

Common mistakes that cost hours

Checking permissions in the UI only. Hiding a button is not security, the API behind it must enforce the rule. Attackers call the API directly.
Trusting an ID from the request to decide ownership. If the client sends user_id=42, an attacker sends user_id=7. Derive identity from the verified session, never the payload.
Treating the Top 10 as a one-time audit. It is a recurring lens. A new endpoint added next week reintroduces A01 if nobody re-checks.
Rolling your own crypto or auth. Hashing passwords with SHA-256, hand-writing JWT validation, inventing a token scheme, use vetted libraries (bcrypt/argon2, a maintained auth framework) instead.
Forgetting dependencies and config (A05, A06). Teams obsess over their own code and ship a 3-year-old library with a known CVE or a publicly readable storage bucket.
Logging the wrong things. Either nothing useful (so you're blind during an incident) or too much, dumping passwords and tokens into logs, which becomes a breach of its own.

Takeaways

The whole article in seven lines

The OWASP Top 10 is the shared map of the ten ways web apps most often get broken.
Almost every entry reduces to one idea: the server trusted input it should have questioned.
Broken Access Control (A01) is the most common serious risk, fix it first.
Enforce authorization server-side, on every request, deny-by-default, using verified identity.
Stop concatenating user input into queries or commands, parameterize everything (A03).
Don't forget the dashed branches: dependencies (A06) and configuration (A05).
If you can't see an attack, you can't stop it, logging and monitoring (A09) is not optional.

Where to go next

The Top 10 is the map; the next step is going deep on the territory. Start with the categories you marked "Yes" or "Not sure" in the practice walkthrough.

Preventing Injection Attacks (SQLi & XSS), A03 in depth, with the exact query patterns that are safe and the ones that aren't.
Secure API Design, how to build access control, input validation, and rate limiting in from the start rather than bolting them on.
Authentication vs Authorization, the distinction underneath A01 and A07: who you are versus what you're allowed to do.
Practice the surrounding skills on the DevOps Engineer path, where security threads through pipelines, config, and infrastructure.

Read the official OWASP Top 10 once for the canonical detail, but you now have the mental model to make it land. Pick one risk, run the six-step routine against something you own this week, and fix what you find.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Authentication vs Authorization: The Two Most-Confused Words in Security

Sri Balaji — Fri, 18 Sep 2026 14:43:04 +0000

⚡ TL;DR: Authentication proves who you are; authorization decides what you can do. Run them in that order, carry identity with sessions or tokens, and put each check in the right place. Confuse the two and you ship security holes.

Same prefix, completely different jobs
Two questions, in order
Where each check lives in a request
Sessions vs tokens: how identity is carried
OAuth and OIDC, without the jargon
In code: verify identity, then check permission
Common mistakes that ship security holes
Takeaways
Where to go next

Same prefix, completely different jobs

You add a login page, users sign in, and you breathe out. The app feels secure. Then a junior engineer signs in with their own valid account, changes the URL from /orders/42 to /orders/43, and sees someone else's invoice. Nothing was "hacked", every request had a valid session. The login worked perfectly. The problem was never login at all.

That gap is the difference between authentication and authorization. They share a prefix, they sit next to each other in the request, and they get used interchangeably in standups, which is exactly why people ship the bug above. Once you can keep them apart in your head, a huge class of security mistakes simply stops happening.

📌 Who this is for: Developers who can build a login form but get fuzzy on what happens after sign-in. If you have ever wondered "is this a 401 or a 403?", written an if (user.isAdmin) check in a React component, or copy-pasted JWT verification without quite knowing what it proves, this is for you. No prior security background needed.

Two questions, in order

Authentication (authn) answers "who are you?", it proves identity. Authorization (authz) answers "what are you allowed to do?", it grants or denies access to a specific action or resource. Authn always runs first; authz depends on its answer.

Notice the ordering. You cannot decide what someone is allowed to do until you know who they are. But knowing who they are tells you nothing, on its own, about what they may do. Those are two separate decisions, made at two separate moments, often by two separate parts of your system.

In the real world	In tech
Showing your passport at the security checkpoint	Authentication, the system confirms you are who you claim to be
Your boarding pass says seat 14C on flight BA42 today	Authorization, you may board THIS flight, in THIS seat, right now
A valid passport does not let you board any plane you want	Being authenticated does not mean you are authorized for an action
The lounge agent re-checks your pass at the gate	Authorization is re-checked at each protected resource, not just once

The airport makes the split obvious, two checkpoints, two different questions.

Your passport is real whether you are flying to Tokyo or standing in a parking lot, identity is stable. The boarding pass is scoped: one flight, one seat, one day. That scoping is the whole point of authorization, and it is why "they logged in" is never a complete answer to "should they be able to do this?".

Where each check lives in a request

Every protected request runs the same gauntlet. The client presents a credential, the server verifies it to establish identity, then a separate step asks whether that identity may perform this exact action on this exact resource. Only if both pass does your business logic ever run.

One request, two gates: authenticate (who), then authorize (allowed for this action?), then the handler runs.

Client sends a credential: A request arrives carrying proof of identity, a session cookie, a bearer JWT in the Authorization header, or an API key. No credential at all means the request is anonymous.
Authenticate: who are you?: The server validates the credential, verifies the JWT signature and expiry, or looks up the session ID in its store. Success yields a trusted identity (a user id, maybe roles/scopes). Failure returns 401 Unauthorized and stops here.
Authorize: allowed for THIS?: Now that identity is known, a policy check asks whether this user may perform this specific action on this specific resource, not "are they logged in" but "may user 42 delete order 43?". Failure returns 403 Forbidden.
Handler runs: Only after both gates pass does the actual business logic execute. By the time your handler runs, identity is proven and the action is permitted, the handler should not have to re-litigate either.

💡 Tip: The two failure codes map cleanly: 401 Unauthorized means "I don't know who you are" (authn failed, go log in). 403 Forbidden means "I know exactly who you are, and you still can't do this" (authz failed, logging in again won't help). Despite its name, 401 is the authentication error.

Sessions vs tokens: how identity is carried

Authentication does not stop after login, proof of identity has to ride along on every subsequent request, because HTTP is stateless. There are two dominant ways to carry it, and they make opposite trade-offs.

Sessions are server-side. After login the server creates a session record (in memory, Redis, or a database) and hands the client an opaque session ID in a cookie. Every request sends the cookie; the server looks the ID up to recover the identity. The token itself carries no information, it is just a key into server state. Revoking access is trivial: delete the row.

Tokens (typically JWTs) are self-contained. After login the server signs a token that embeds the claims, user id, roles, expiry, and the client sends it on each request. The server verifies the signature and trusts the contents without a lookup, which scales beautifully across many services. The catch: a signed token is valid until it expires, so you cannot easily revoke one mid-flight. That is why access tokens should be short-lived and paired with a refresh token.

Dimension	Server sessions	JWT / tokens
Where state lives	Server (store / DB)	Inside the token itself
Lookup per request	Yes, fetch the session	No, verify the signature
Revocation	Instant (delete the record)	Hard until expiry; needs a denylist
Scales across services	Needs shared session store	Naturally, any service can verify
Best lifetime	Long-ish, revocable	Short access + refresh token
Carried in	Cookie (HttpOnly)	Authorization: Bearer header

Sessions vs tokens, the trade-off is statefulness vs revocability.

OAuth and OIDC, without the jargon

Two more terms get tangled into the authn/authz knot, so let us untangle them in one breath. OAuth 2.0 is an authorization framework, it lets a user grant one app limited access to their data in another app without sharing a password. When you click "Connect your Google Calendar", OAuth is what issues an access token scoped to just your calendar.

OIDC (OpenID Connect) is an authentication layer built on top of OAuth. It adds an id_token, a JWT that proves who the user is, so apps can use "Sign in with Google" for login, not just data access. Rule of thumb: OAuth = delegated authorization (access to resources); OIDC = federated authentication (proof of identity). Most "Sign in with X" buttons are OIDC; most "Allow X to access your Y" consent screens are OAuth scopes.

The scopes you see on those consent screens (read:calendar, email, profile) are authorization data riding inside an authentication flow. Your app should treat scopes as one input to its own policy decisions, not as the final word on what a user may do inside your system.

In code: verify identity, then check permission

Here is the whole idea in one Python handler. The first block proves identity from a JWT (authentication). The second block, a completely separate decision, asks whether that proven identity may act on this particular resource (authorization). Keep them visibly distinct.

orders_api.py

import jwt  # PyJWT
from fastapi import HTTPException, Request

JWT_SECRET = "...load-from-env..."

# --- AUTHENTICATION: who are you? ---
def authenticate(request: Request) -> dict:
    header = request.headers.get("Authorization", "")
    if not header.startswith("Bearer "):
        # No credential at all -> we don't know who you are.
        raise HTTPException(status_code=401, detail="Missing token")
    token = header.removeprefix("Bearer ")
    try:
        # Verifies signature AND expiry. Returns the claims if valid.
        claims = jwt.decode(token, JWT_SECRET, algorithms=["HS256"])
    except jwt.InvalidTokenError:
        raise HTTPException(status_code=401, detail="Invalid token")
    return claims  # e.g. {"sub": "42", "roles": ["customer"]}


# --- AUTHORIZATION: are you allowed to do THIS? ---
def authorize_order_access(user: dict, order) -> None:
    is_owner = order.customer_id == user["sub"]
    is_admin = "admin" in user.get("roles", [])
    if not (is_owner or is_admin):
        # We know exactly who you are -- you still can't have this.
        raise HTTPException(status_code=403, detail="Forbidden")


async def get_order(request: Request, order_id: str):
    user = authenticate(request)              # 1. prove identity
    order = await db.orders.find(order_id)
    if order is None:
        raise HTTPException(status_code=404)
    authorize_order_access(user, order)        # 2. check permission
    return order                               # 3. only now: business logic

The authorization check is resource-scoped: it compares the resource's owner against the caller. That is what stops the /orders/43 attack from the intro, a valid token authenticates the attacker, but they are not the owner, so authorize_order_access returns 403. The bug at the top of this article is exactly an app that ran step 1 and skipped step 2.

This pattern of "is the caller the owner, or do they have a role that overrides it?" is RBAC (role-based access control). When rules get richer, "managers can refund orders under €500 in their own region", you graduate to ABAC (attribute-based access control), where the decision is a function of attributes of the user, the resource, and the context. Same place in the request; just a more expressive policy.

Common mistakes that ship security holes

Assuming authentication implies authorization. "They're logged in" is not "they may do this." Every protected action needs its own permission check against the specific resource, the single most common real-world breach (broken object-level authorization, the /orders/43 bug).
Enforcing authorization only in the UI. Hiding the Delete button for non-admins is UX, not security. The endpoint is still reachable with curl. Every check that hides a button in the frontend must be re-enforced on the server, where it cannot be bypassed.
Long-lived or non-revocable tokens. A JWT with a 30-day expiry and no denylist means a leaked token is a 30-day master key. Keep access tokens short (minutes), use refresh tokens, and have a revocation path for the cases that matter.
Trusting claims you didn't verify. Decoding a JWT is not verifying it. Always check the signature and expiry (and issuer/audience) before trusting a single claim, an unverified "roles": ["admin"] is just attacker-supplied JSON.
Checking permissions far from the resource. Authorization decided at the gateway with stale role data can drift from reality. Decide as close to the protected resource as you can, with current data.

Takeaways

The whole article in seven lines

Authentication = who are you. Authorization = what may you do. Authn runs first; authz depends on it.
401 = I don't know you (authn failed). 403 = I know you and you still can't (authz failed).
Sessions store state on the server (easy revoke); tokens carry state in the JWT (easy scale, hard revoke).
OAuth = delegated authorization (resource access). OIDC = authentication on top of OAuth (proof of identity).
Authorization must be resource-scoped: not "are you logged in" but "may YOU touch THIS".
Always enforce authorization on the server, UI hiding is not security.
Short-lived tokens, verify signatures, decide permissions close to the resource.

Where to go next

Authn and authz are the spine of every secure system, but they live inside a bigger picture. Two natural next steps: see how these checks fit into a hardened API surface, and how identity stops being a one-time gate and becomes a continuous decision.

Secure API Design, where authn/authz sit among input validation, rate limiting, and safe defaults.
Zero Trust Networking for Beginners, "never trust, always verify": authorization re-checked on every hop, not just at the edge.
DevOps Engineer path, the role track where securing services and pipelines is part of the day job.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

What Is Application Security?

Sri Balaji — Thu, 17 Sep 2026 15:22:38 +0000

⚡ TL;DR: AppSec in one read: protect confidentiality, integrity, and availability by mapping trust boundaries and attack surface, then layering defense in depth and least privilege. Security is a phase of the whole SDLC, not a final gate.

Your app works. Now someone is trying to break it.
A one-sentence definition
Where security lives in the SDLC (shift-left)
Trust boundaries and attack surface
The principles that hold it all together
The main domains of AppSec
Common misconceptions that get people breached
Takeaways
Where to go next

Your app works. Now someone is trying to break it.

You ship a feature. It works in the demo, the tests are green, and users are happy. Then one quiet afternoon a stranger pastes a weird string into your login box, and suddenly they are reading other people's data, or your database is gone. Nothing "broke" in the way a bug breaks. The code did exactly what it was told. It was just told the wrong thing by the wrong person.

That gap, between what your app does and what an attacker can convince it to do, is the whole subject of application security (AppSec). It is not a product you buy at the end. It is a way of thinking that runs through every line you write, every dependency you pull, and every server you deploy to.

📌 Who this is for: Developers, students, and new engineers who can build a working app but have never been taught to think about who is attacking it. No prior security background needed. By the end you will have a mental model and the core vocabulary the rest of this track builds on.

A one-sentence definition

Application security is the practice of building, deploying, and running software so that it keeps doing the right thing even when someone is actively trying to make it do the wrong thing.

The working definition we'll use all track

Read that twice. The key phrase is even when someone is actively trying. Ordinary engineering asks "does it work for a cooperative user?" Security asks "what happens when the user is hostile, the input is malicious, and the network is untrusted?" Same code, much harder question.

In the real world	In tech
A fence around the property	A firewall / network perimeter, keeps out casual passers-by, not a determined intruder
Locks on every door, not just the front	Auth checks on every endpoint, not only the login page
Different keys for different rooms	Least privilege, each component gets only the access it needs
Security guards who verify ID	Authentication and authorization at each trust boundary
Cameras and an alarm log	Logging, monitoring, and alerting, you detect the break-in
A safe inside the locked office	Encryption, even past the doors, the valuables stay sealed

Securing software is like securing a building, no single measure is enough.

Notice what the analogy rules out: a fence alone is not security. Beginners often equate "we have a firewall" or "we use HTTPS" with "we are secure." Real security is layers, many independent measures, so that one failure does not hand over the building. That idea has a name, and we will get to it.

Where security lives in the SDLC (shift-left)

The most expensive myth in software is that security is a step near the end, a scan you run, or a pen-test you book, the week before launch. By then the architecture is set, the risky shortcuts are buried, and every fix is a rewrite. The modern approach is shift-left: push security earlier, into every phase of the software development lifecycle, so each phase has its own gate.

Security is a gate at every stage of the SDLC, not a single checkpoint at the end.

The solid line is the path your code travels. The dashed lines are the security reality: attackers probe the running system constantly, what you detect becomes findings, and findings flow back into design so the next loop is safer. Security is a cycle, not a finish line.

Design, threat modeling: Before writing code, ask "what could go wrong?" Who are the attackers, what are they after, and where are the trust boundaries? A 30-minute conversation here prevents whole categories of bugs.
Code, secure coding + SAST: Developers validate input, avoid dangerous patterns, and let static analysis (SAST) flag risky code automatically in the editor and on every pull request.
Build, dependency scanning (SCA): Most code in a modern app is not yours, it's open-source dependencies. Software composition analysis (SCA) checks those libraries for known vulnerabilities before they ship.
Deploy, config and secrets checks: Catch the boring-but-fatal mistakes: a public storage bucket, a hardcoded API key, an over-permissioned role, debug mode left on in production.
Run, monitor and patch: Production is where attacks actually happen. Log security events, alert on anomalies, and patch fast when a new vulnerability lands in a dependency you use.

Trust boundaries and attack surface

Two ideas unlock most of security thinking. The first is the trust boundary: the line where data or control crosses from something you trust into something you do not (or vice versa). The browser-to-server line is a trust boundary. So is the line between your service and a third-party API, between your app and the database, and even between two of your own microservices. The rule is simple and absolute: validate everything that crosses a trust boundary. Never trust input just because the last component was yours.

The second idea is attack surface: the sum of all the places an attacker can poke at your system, every endpoint, form field, file upload, query parameter, header, open port, and dependency. Every feature you add grows the attack surface. The most reliable way to be more secure is, paradoxically, to have less: fewer endpoints, fewer dependencies, fewer permissions, less data retained. You cannot be breached through a door you never built.

💡 A useful instinct: Whenever you draw an architecture diagram, draw the trust boundaries on it as dashed lines. Every arrow that crosses a dashed line is a place that needs authentication, authorization, and input validation. This single habit catches an enormous share of real bugs.

The principles that hold it all together

AppSec rests on a small set of timeless principles. Learn these four and you have the load-bearing 80%. Everything else, specific attacks, specific tools, is a consequence of them.

Principle	What it means	In practice
CIA, Confidentiality	Only authorized parties can read the data.	Encryption in transit (TLS) and at rest; access controls; not logging secrets.
CIA, Integrity	Data and code cannot be altered by unauthorized parties without detection.	Checksums, signatures, immutable audit logs, validating input before it's stored.
CIA, Availability	The system stays up and usable for legitimate users.	Rate limiting, redundancy, DDoS protection, sane timeouts and resource limits.
Defense in depth	Multiple independent layers, so one failure isn't fatal.	WAF + input validation + parameterized queries + least-privilege DB user, all at once.
Least privilege	Every user, service, and token gets the minimum access it needs.	Scoped API keys, read-only DB roles, short-lived credentials, no shared admin accounts.
Fail secure	When something breaks, default to denied, not allowed.	If the auth check errors, reject the request. A crashed gate should be a locked gate.

The core security principles every engineer should internalize.

The CIA triad, in one breath

Confidentiality, Integrity, Availability are the three things security protects. Confidentiality is keeping secrets secret. Integrity is making sure data is correct and untampered. Availability is keeping the lights on. Every attack you will ever read about violates at least one of these: a data leak breaks confidentiality, a tampered transaction breaks integrity, a ransomware lockout breaks availability. When you assess any risk, ask: which letter of CIA does this threaten?

Defense in depth and least privilege

Defense in depth is the building analogy made concrete: never rely on one control. If your input validation has a bug, a parameterized query still saves you; if that fails, a least-privileged database user limits the blast radius. Least privilege is the discipline of shrinking that blast radius everywhere, the service account that only needs to read one table should not be able to drop the whole schema. Together they assume something will go wrong and arrange for it not to be catastrophic.

The main domains of AppSec

"Application security" is an umbrella over several connected areas. You do not need to master all of them today, but knowing the map tells you what to learn next.

Authentication & authorization, proving who a user is, then controlling what they're allowed to do. The two are different, and confusing them causes real breaches.
Input validation & injection defense, treating all input as hostile so SQL injection, XSS, and command injection have nowhere to land.
Secrets management, keeping API keys, passwords, and tokens out of source code and configuration, and rotating them safely.
Dependency & supply-chain security, knowing what open-source you ship and patching vulnerable libraries before attackers exploit them.
Data protection & cryptography, encryption in transit and at rest, hashing passwords correctly, and not rolling your own crypto.
Secure configuration, hardening servers, containers, and cloud resources so defaults don't leave the door open.
Logging, monitoring & incident response, detecting attacks, and having a plan for when (not if) one succeeds.

Common misconceptions that get people breached

"We use HTTPS, so we're secure." HTTPS protects data in transit between browser and server. It does nothing about injection, weak passwords, broken access control, or a leaked API key.
"Security is the security team's job." The team sets standards and tools, but the vulnerabilities live in your code. Security that isn't owned by developers always loses to the deadline.
"We're too small to be a target." Most attacks are automated and indiscriminate, bots scan the entire internet for the same known holes. Being small just means no one is watching when it happens.
"We'll add security before launch." Bolted-on security is expensive, incomplete, and brittle. Shift-left exists precisely because retrofitting is the worst time to do it.
"Obscurity is protection." Hiding an admin page at a secret URL or rolling a homemade encryption scheme is not security. Assume the attacker can read your code, because often they can.

Takeaways

The whole article in seven lines

AppSec = building software that does the right thing even when someone is attacking it.
The CIA triad, Confidentiality, Integrity, Availability, is what security protects. Every attack breaks at least one.
Validate everything that crosses a trust boundary; shrink your attack surface by having less.
Defense in depth: layers, so one failure isn't fatal. Least privilege: minimum access everywhere. Fail secure: break to denied.
Security is a gate at every SDLC phase, design, code, build, deploy, run, not a final checkpoint. That's shift-left.
Security is a developer's job, not just the security team's. The bugs are in your code.
Most attacks are automated; "too small to be a target" is a myth.

Where to go next

You now have the mental model. The next articles turn each principle into something you can actually build. Start with the one beginners get wrong most often, then learn the named attacks and how to keep secrets out of your code.

Authentication vs Authorization, the difference between who you are and what you're allowed to do, and why mixing them up is dangerous.
The OWASP Top 10, Explained, the canonical list of the most common, most dangerous web vulnerabilities, in plain language.
Secrets Management, how to keep API keys, tokens, and passwords out of your source code and config.
Building toward a security-aware engineering role? The DevOps Engineer path weaves these practices into real pipelines and infrastructure.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Fine-Tuning LLMs: When and How

Sri Balaji — Wed, 16 Sep 2026 15:14:02 +0000

⚡ TL;DR: Fine-tuning changes how a model behaves, not what it knows, so reach for it on style and format, not facts. Learn when prompting or RAG wins instead, and how to build a dataset that actually moves the needle.

The trap everyone falls into
One principle, one analogy
The fine-tuning workflow
Prompting vs RAG vs fine-tuning
What the data actually looks like
Evaluating the result
Common mistakes that cost weeks
Takeaways
Where to go next

The trap everyone falls into

Your model keeps answering in the wrong tone. It ignores your JSON schema. It writes like a chatbot when you need a terse internal tool. So someone says the magic words: "let's just fine-tune it." Three weeks later you have a labeled dataset, a training bill, a checkpoint that is already stale, and the model still hallucinates yesterday's pricing, because fine-tuning was never the tool for that problem.

Fine-tuning is powerful and badly misunderstood. The single most useful thing you can learn about it is what it cannot do. Get that right and the rest is mechanical.

📌 Who this is for: Engineers who can already call an LLM API and have shipped a prompt or two, maybe even a RAG pipeline, and are now wondering whether fine-tuning is the next step. No ML PhD required. We assume you know what a prompt is and can read JSON and YAML.

One principle, one analogy

Fine-tuning teaches a model how to behave. It does not teach it what is true today.

The rule that prevents 90% of wasted fine-tuning projects

Think of a base model as a brilliant new graduate. They reason well, write well, and know an enormous amount about the world up to the day they stopped reading. Fine-tuning is sending them to a finishing school for tone and craft, teaching them your house style, your output format, your domain's voice. What it is not is handing them this morning's newspaper. For fresh facts you put the newspaper on their desk at question time. That desk is your prompt, and the newspaper is retrieval.

In the real world	In tech
Writing a clear brief before each task	Prompt engineering, cheap, instant, changeable
Giving them the relevant files for this task	RAG, fresh facts injected at query time
Sending them to finishing school for your house style	Fine-tuning, bakes behavior into the weights
Expecting school to teach today's stock price	The mistake, weights are frozen at training time

Same employee, three different interventions

The fine-tuning workflow

When fine-tuning is the right call, the path is the same every time. Curate a clean dataset, format it the way the trainer expects, train a small adapter (not the whole model), evaluate honestly, deploy, and then watch it in production. The diagram below is the loop; the steps after it walk one pass through it.

The fine-tuning loop, most of the value is in the first two boxes and the eval gate

Curate the dataset: Collect a few hundred to a few thousand high-quality input/output pairs that demonstrate exactly the behavior you want. Quality beats quantity by a mile, 500 clean examples outperform 5,000 noisy ones.
Format to the chat schema: Convert each pair into the JSONL message format your trainer expects (system / user / assistant turns). Consistency here is non-negotiable; one malformed field can skew the whole run.
Train a LoRA adapter: Freeze the base model and train a small set of low-rank adapter weights. Minutes to a couple of hours on a single GPU, not a data-center job.
Evaluate against a held-out set: Score on examples the model never saw during training, plus an LLM-judge or human pass for quality. If it does not beat your prompt-only baseline, stop, do not deploy.
Deploy the adapter: Serve the small adapter on top of the base model. You can keep several adapters and swap them per use case without re-hosting the base.
Monitor and feed back: Watch for regressions and distribution drift in production, harvest fresh examples, and feed them into the next curation round.

Prompting vs RAG vs fine-tuning

These three are not competitors, they solve different problems and stack happily. The mistake is reaching for the most expensive one first. Try prompting. Then try RAG. Only then consider fine-tuning. Most teams never need the third box.

	Prompting	RAG	Fine-tuning
Best for	Instructions, tone, simple format	Fresh / private facts	Consistent style, strict format, narrow domain
Freshness	Whatever you paste in	Live, updates with your data	Frozen at training time
Cost	Near zero	Low, vector store + retrieval	High, data prep + GPU + upkeep
Effort to change	Edit a string	Update an index	Re-curate + re-train + re-eval
Time to first result	Minutes	Hours to days	Days to weeks

Reach for the cheapest tool that solves your actual problem

💡 They compose: The strongest systems use all three: a fine-tuned model for reliable house-style output, RAG for the facts, and a tight prompt to orchestrate both. Fine-tuning is the last layer you add, not the first.

What the data actually looks like

The dataset is the project. Each line is one self-contained example in the chat schema, a system message setting the role, the user input, and the assistant output you wish the model had produced. Below is a tiny slice of a support-triage dataset teaching the model to always reply with a fixed JSON structure.

train.jsonl

{"messages": [{"role": "system", "content": "You are a support triage bot. Reply only with JSON."}, {"role": "user", "content": "My invoice is wrong and I want a refund."}, {"role": "assistant", "content": "{\"category\": \"billing\", \"priority\": \"high\", \"needs_human\": true}"}]}
{"messages": [{"role": "system", "content": "You are a support triage bot. Reply only with JSON."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "{\"category\": \"account\", \"priority\": \"low\", \"needs_human\": false}"}]}
{"messages": [{"role": "system", "content": "You are a support triage bot. Reply only with JSON."}, {"role": "user", "content": "The app crashes every time I open a report."}, {"role": "assistant", "content": "{\"category\": \"bug\", \"priority\": \"high\", \"needs_human\": true}"}]}

Notice what these examples teach: format and routing behavior, not facts. None of them depend on information that changes day to day. That is the signature of a good fine-tuning dataset.

Training itself is mostly configuration. A LoRA run reads like this, a base model, a handful of hyperparameters, and the rank that controls how big your adapter is.

lora.yaml

base_model: meta-llama/Llama-3.1-8B-Instruct
dataset: ./train.jsonl

lora:
  r: 16              # adapter rank, higher = more capacity, more params
  alpha: 32          # scaling factor, usually 2x r
  dropout: 0.05
  target_modules:    # which layers get adapters
    - q_proj
    - v_proj

training:
  epochs: 3          # too many = memorization / overfit
  learning_rate: 2.0e-4
  batch_size: 8
  eval_split: 0.1    # hold out 10%, never train on your eval set

📌 Why LoRA / PEFT instead of full fine-tuning: Full fine-tuning updates every weight in the model, huge memory, huge cost, and a fresh multi-gigabyte checkpoint each time. Parameter-efficient fine-tuning (PEFT) methods like LoRA freeze the base model and train a tiny set of low-rank adapter matrices, often under 1% of the parameters. You get most of the benefit for a fraction of the cost, the adapters are megabytes not gigabytes, and you can host one base model with many swappable adapters.

Evaluating the result

A fine-tune that you did not evaluate is a liability, not an asset. The non-negotiable rule: always compare against your prompt-only baseline on data the model never trained on. If the fine-tune does not clearly beat the baseline, you just spent weeks to ship a regression risk.

Hold out a test set early. Split before you train and never let those examples leak into training.
Beat the baseline or stop. Run the same eval against plain prompting and against RAG. The fine-tune has to win to justify itself.
Measure what you actually fine-tuned for. Format compliance, tone match, schema validity, task-specific metrics, not a generic benchmark score.
Watch for regressions. A model tuned hard on one task often gets worse at everything else. Keep a small general-capability check in the loop.
Use an LLM judge for the fuzzy stuff, but spot-check the judge against human ratings so you trust it.

For the full discipline of measuring LLM systems, judges, regression suites, and offline-vs-online evals, see Evaluating LLM Applications.

Common mistakes that cost weeks

Fine-tuning for fresh facts. Weights freeze at training time. If the answer changes daily, you want RAG, not a fine-tune. This is the number-one wasted project.
Too little or dirty data. A few dozen inconsistent examples teach the model noise. Curate for quality; 500 clean pairs beat 5,000 messy ones.
No evaluation. Shipping a checkpoint because the loss curve looked nice, with no held-out comparison to the baseline. You have no idea if it is better.
Skipping prompting and RAG first. Reaching straight for the most expensive, slowest, hardest-to-change tool when a one-line prompt edit or a retrieval step would have solved it.
Overfitting on epochs. Training too long memorizes your examples and destroys generalization. Watch eval loss, not just training loss.
One model to rule them all. Cramming many unrelated tasks into one fine-tune. Prefer small, focused adapters you can swap per use case.

Takeaways

The whole article in seven lines

Fine-tuning changes behavior (style, tone, format), never knowledge of fresh facts.
Try prompting first, then RAG, and only then fine-tuning. Most teams never need the third.
Use RAG for anything that changes; use fine-tuning for consistent house style and strict formats.
The dataset is the project, a few hundred clean, on-task examples beat thousands of noisy ones.
Use LoRA / PEFT, train tiny adapters, not the whole model. Cheaper, faster, swappable.
Always evaluate on a held-out set against your prompt-only baseline. No win, no deploy.
Then monitor in production and feed new examples into the next round.

Where to go next

Fine-tuning sits near the top of the AI engineering stack, reach for it after you have squeezed prompting and retrieval. Build the layers underneath it first.

Solve the freshness problem the right way: RAG Architecture Explained.
Build the eval discipline before you train anything: Evaluating LLM Applications.
See where fine-tuning fits in the broader role: the AI Engineer career path.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

LLMOps: Productionizing LLM Apps

Sri Balaji — Tue, 15 Sep 2026 15:21:35 +0000

⚡ TL;DR: LLMOps is DevOps for non-deterministic software. The five ops concerns that keep a model from going off-script at 2am: versioned prompts, tracing, guardrails, fallbacks, and drift monitoring.

It is 2am and the model is off-script
LLMOps is DevOps for non-deterministic software
The picture: an LLMOps pipeline
Five ops concerns, translated
Wrapping a call with tracing and a guardrail
Common mistakes that page you at 2am
Takeaways
Where to go next

It is 2am and the model is off-script

The demo went perfectly. You typed a question, the model answered like a senior colleague, the room nodded, and the feature shipped. Three weeks later it is 2am and your phone is buzzing. A user pasted a screenshot into your support chat: your friendly assistant has cheerfully recommended a competitor, leaked a chunk of someone else's order history, and signed off with a joke that will not read well on social media. Nothing crashed. No exception was thrown. The system did exactly what it was told, it just should not have been told that.

This is the gap nobody mentions in the demo. Shipping an LLM call is easy. Running an LLM feature, one that stays safe, stays good, and stays debuggable as prompts change and the model drifts under you, is a different discipline. That discipline is LLMOps: the operational practices that turn a clever prompt into software you can sleep through the night with.

📌 Who this is for: Engineers who have a working LLM prototype and now have to make it production-grade. You know how to call an API and write a prompt. You have not yet been paged because the model said something it should not have. This article is the playbook for the layer between "it works on my machine" and "it is safe in front of users."

LLMOps is DevOps for non-deterministic software

LLMOps is the practice of treating prompts, models, and their outputs as production artifacts, versioned, observed, guarded, and continuously evaluated, exactly because the software underneath is non-deterministic.

The working definition

Here is the mental model. Traditional software is a machine: same input, same output, every time. You test it once and trust it forever. An LLM feature is more like a very capable, very literal new hire who occasionally has a bad day. You cannot unit-test a personality. What you can do is give them a written playbook (a versioned prompt), put a reviewer on the door for anything that goes in and out (guardrails), record every conversation so you can review what happened (tracing), and quietly score a sample of their work each week to catch the day their quality slips (drift monitoring).

In the real world	In tech
Source code in Git, tagged releases	Prompts versioned and pinned per deploy
CI test suite gates the merge	Eval suite gates the prompt change
APM traces every request through services	Tracing captures every LLM call: prompt, tokens, latency, cost
A WAF inspects traffic at the edge	Guardrails validate model input and output
Alert when error rate creeps up	Alert when answer quality drifts down

You already have the muscles for this, LLMOps just points them at non-deterministic software.

If you have done DevOps and observability, you have the instincts. The twist is that your unit of work is probabilistic, so "correct" is a distribution rather than a checkmark, and that changes how you version, test, monitor, and roll back. Everything below is one of those five concerns, translated.

The picture: an LLMOps pipeline

Every production LLM request runs a gauntlet. It is not "user text in, model text out", there is a checkpoint before the model and a checkpoint after it, and everything that happens in between is recorded for later review and scoring.

A request passes an input guardrail, hits a versioned prompt and the model, then an output guardrail before the response. Every hop is traced; a sampled copy feeds the eval loop.

Request arrives: Raw user input enters the pipeline. Treat it as untrusted, it may contain PII, a prompt injection payload, or something that violates your policy.
Input guardrail: Before a single token reaches the model: strip or block PII, scan for jailbreak and injection patterns, and reject inputs that fail policy. Fail fast and cheap, before you pay for inference.
Versioned prompt: The cleaned input is rendered into a prompt template pinned to a specific version (e.g. support-agent@v7). The version is part of the deploy, not a string buried in a function.
Model call: The composed prompt hits the LLM. This hop is wrapped in a trace span that records model id, prompt version, token counts, latency, and cost.
Output guardrail: The raw completion is validated before anyone sees it: enforce schema, redact leaked PII, check for policy violations, and refuse or repair if it fails.
Response + feedback: The safe output returns to the user. A sampled, traced copy flows to the eval loop, where it is scored over time so you can see quality drift before your users do.

Five ops concerns, translated

The fastest way to internalize LLMOps is to map it onto the operations you already run. Same five concerns, versioning, testing, monitoring, rollback, security, with an LLM-shaped answer for each.

Concern	Traditional ops	LLMOps equivalent
Versioning	Tagged code releases in Git	Prompts and model ids pinned and versioned per deploy
Testing	Deterministic unit and integration tests	Eval suites scoring outputs against a graded dataset
Monitoring	Latency, error rate, throughput	Quality scores, refusal rate, token cost, drift over time
Rollback	Redeploy the previous build	Re-pin the previous prompt version or model snapshot
Security	AuthN/Z, input validation, WAF	Guardrails: PII handling, jailbreak and injection defense, output validation

Traditional ops concern mapped to its LLMOps equivalent.

⚠️ Non-determinism breaks the green checkmark: Your eval suite will not return a clean pass/fail like a unit test does. It returns scores on a sample, and those scores have variance. Gate on thresholds and trends ("faithfulness must stay above 0.85, and not drop more than 5 points week over week"), not on exact equality. Read Evaluating LLM Applications for how to build the graded dataset and judges this depends on.

Wrapping a call with tracing and a guardrail

Here is the smallest version of the pipeline that is still real: one LLM call, wrapped so that it is traced end to end and so that nothing leaves without passing an output guardrail. Notice the prompt is pinned to a version, the whole call is inside a trace span, and the output is validated before it is returned, a failed check raises rather than silently shipping a bad answer.

llm_call.py

import re
import time
import uuid

from prompts import get_prompt          # returns (text, version)
from client import call_model            # thin wrapper over your provider SDK
from tracing import tracer               # OpenTelemetry-style tracer

PII_PATTERN = re.compile(r"\b\d{3}-\d{2}-\d{4}\b")  # naive SSN example


class GuardrailError(Exception):
    """Raised when an output fails validation so it never reaches the user."""


def output_guardrail(text: str) -> str:
    # 1. Refuse if the model leaked something it should not have.
    if PII_PATTERN.search(text):
        raise GuardrailError("output contained PII")
    # 2. Enforce a minimum-quality contract (e.g. non-empty, bounded length).
    if not text.strip() or len(text) > 4000:
        raise GuardrailError("output failed length/format policy")
    return text


def answer(user_input: str) -> str:
    prompt_text, prompt_version = get_prompt("support-agent")
    request_id = str(uuid.uuid4())

    with tracer.start_as_current_span("llm.answer") as span:
        span.set_attribute("request.id", request_id)
        span.set_attribute("prompt.version", prompt_version)

        composed = prompt_text.format(question=user_input)
        start = time.perf_counter()
        result = call_model(composed)            # -> {text, model, tokens}
        latency_ms = (time.perf_counter() - start) * 1000

        # Observability: the data you will wish you had at 2am.
        span.set_attribute("model.id", result["model"])
        span.set_attribute("model.tokens", result["tokens"])
        span.set_attribute("latency.ms", round(latency_ms, 1))

        try:
            safe = output_guardrail(result["text"])
            span.set_attribute("guardrail.passed", True)
            return safe
        except GuardrailError as exc:
            # Do not ship the unsafe output. Fall back, and record why.
            span.set_attribute("guardrail.passed", False)
            span.set_attribute("guardrail.reason", str(exc))
            return "Sorry, I cannot help with that right now."

That fallback at the end is the unglamorous hero of production LLM work. When the guardrail trips, or when the model times out, or the provider returns a 500, you return a safe, boring response and log the reason, instead of leaking the failure to the user. A real system layers fallbacks: retry, then a cheaper backup model, then a canned safe answer. The point is that the user never sees the raw failure, and you always see the trace.

💡 Do not build guardrails entirely from regex: The PII regex above is a teaching stub. In production, combine deterministic checks (schema validation, allowlists, format contracts) with a dedicated classifier or a small judge model for fuzzy concerns like toxicity, jailbreak intent, and topic policy. Cheap deterministic checks first, expensive model-based checks only when needed.

Common mistakes that page you at 2am

Unversioned prompts. The prompt lives as a string literal in a function, someone tweaks it to fix one edge case, and now you cannot tell which deploy produced last week's bad answers, or roll back to the version that was fine. Pin every prompt to a version and ship the version with the deploy.
No tracing. When a user reports a bad answer and you have no record of the exact prompt, model, tokens, and output for that request, you are debugging blind. You cannot reproduce what you did not capture. Trace every call.
No guardrails. Trusting the model to police its own input and output is how PII leaks, prompt injections succeed, and off-policy text reaches users. Validate what goes in and what comes out, assume both are hostile.
No drift monitoring. Quality does not fail loudly. The model provider silently updates a snapshot, your data distribution shifts, an upstream prompt changes, and your answers get a little worse every week until someone notices on Twitter. Score a sample continuously and alert on the trend.
Treating eval as a one-time gate. Passing evals before launch tells you nothing about month three. The eval loop has to run continuously on production traffic, not once in CI.

Takeaways

LLMOps in nine lines

Shipping an LLM call is easy; running an LLM feature safely is a discipline, LLMOps.
It is DevOps and observability for non-deterministic software: version, test, monitor, roll back, secure.
Every request runs a gauntlet: input guardrail → versioned prompt → model → output guardrail → response.
Version prompts and model ids per deploy so you can pin, diff, and roll back.
Trace every call, prompt version, model id, tokens, latency, cost, or you will debug blind.
Guardrails validate input and output: PII, jailbreak and injection defense, schema and policy.
Always have a fallback: retry, backup model, then a safe canned answer. Never leak the failure.
Gate on thresholds and trends, not exact equality, quality is a distribution, not a checkmark.
Monitor drift continuously on sampled production traffic; quality fails quietly.

Where to go next

LLMOps stands on two skills you can deepen right now. The guardrails and prompts you version are only as good as the prompts themselves, and the drift you monitor is only measurable if you can score quality at all.

Build the scoring foundation that drift monitoring and eval gates depend on: Evaluating LLM Applications.
Write prompts worth versioning in the first place: Prompt Engineering Fundamentals.
See where this fits in the broader role and what to learn next: the AI Engineer career path.

Start small: pick your one live LLM feature, pin its prompt to a version, wrap the call in a trace, and add a single output guardrail. That is the minimum that lets you debug the next 2am page instead of guessing through it.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

LLM Cost & Latency Optimization

Sri Balaji — Mon, 14 Sep 2026 16:43:19 +0000

⚡ TL;DR: The cheapest token is the one you never send. A practical playbook for cutting the bill and the p95: token budgeting, caching, model routing to smaller models, and streaming so apps stay cheap and fast enough to ship.

Your demo worked. The bill and the p95 did not.
The principle: don't send the token in the first place
The picture: a request's journey to the cheapest answer
The levers, and what each one actually buys you
A minimal router and token-budget trimmer
Common mistakes that quietly drain the budget
Takeaways
Where to go next

Your demo worked. The bill and the p95 did not.

The demo was magic. You wired up a model, pasted in a fat system prompt, stuffed the whole knowledge base into the context, and it answered beautifully. Then you shipped it to real traffic. Two things broke at once: the monthly bill climbed past what the feature is worth, and the p95 latency, the slow-tail experience your impatient users actually feel, crept toward six, eight, ten seconds. Suddenly the magic feels like a liability.

Here is the good news: almost none of this requires a smarter model. It requires engineering. The same disciplines you already apply to databases and APIs, caching, routing, trimming, batching, map cleanly onto LLM calls. This article is the playbook: the levers that make an LLM app cheap and fast enough to keep in production, in the order you should reach for them.

📌 Who this is for: Engineers who have an LLM feature working and now need it to be affordable and responsive at scale. You should be comfortable reading Python and thinking about percentiles and per-request cost. No ML background required, this is systems work, not model training. New to how these models tick? Start with How LLMs Actually Work.

The principle: don't send the token in the first place

The cheapest, fastest token is the one you never send.

The whole article in one line

Every optimization here is a variation on that idea. You pay, in dollars and in milliseconds, roughly in proportion to the tokens that flow in and out of the model. Input tokens cost money to send and add to the time-to-first-token. Output tokens cost more per unit and are generated one at a time, so they dominate latency. So the entire game is: send fewer tokens, generate fewer tokens, and avoid the call entirely when you can.

Crucially, optimizing cost and optimizing latency are not always the same move. Some levers buy you both (a cache hit is free and instant). Some buy one at the expense of the other (batching is cheaper but slower per request). Knowing which is which is what lets you tune deliberately instead of flailing.

In the real world	In tech
A pricey consultant who bills by the word	The LLM, every input and output token costs money and time
Checking your notes before booking the meeting	A cache lookup before calling the model at all
Asking the junior analyst the easy questions first	Routing simple requests to a small, cheap model
Sending a one-page brief instead of the whole filing cabinet	Trimming the prompt and context to what's relevant
The consultant talking while they think, not after	Streaming tokens so the user sees output immediately

Treat the model like an expensive specialist consultant, not a search box you spam.

The picture: a request's journey to the cheapest answer

Before reaching for any single model, a well-built LLM service runs each request through a funnel. Every stage is a chance to answer cheaper or not at all. Only the requests that survive every filter reach the expensive large model.

A request flows through a semantic cache and a router before ever touching a model; streaming carries the answer back token-by-token.

Check the semantic cache: Embed the incoming request and look for a near-identical question answered before. On a hit, return the stored answer instantly, zero model tokens, single-digit-millisecond latency.
Route on a cache miss: A lightweight classifier (rules, a tiny model, or a heuristic) decides whether this is an easy request or a hard one. Most production traffic is easy.
Try the small model first: Send easy requests to a small, cheap, fast model. It handles the long tail of simple lookups, classifications, and short rewrites for a fraction of the cost.
Escalate only when needed: Hard requests, or small-model answers that fail a confidence/validation check, escalate to the large model. You pay the premium only for the requests that truly need it.
Stream the answer back: Whichever model answers, stream tokens to the client as they generate. Total time is unchanged, but perceived latency collapses because the user sees words immediately.
Write back to the cache: Store the answer keyed by the request embedding so the next similar question skips the whole funnel.

The levers, and what each one actually buys you

These are the six moves worth knowing, ranked roughly by impact-per-effort. The honest framing matters: each lever has a trade-off, and stacking them blindly can hurt. Read the table as a menu, not a checklist.

Lever	Saves cost?	Saves latency?	Trade-off
Exact / prompt caching	Yes, cached input is heavily discounted	Yes, skips reprocessing	Only helps on repeated prefixes; needs stable prompt ordering
Semantic caching	Yes, full call avoided on hit	Yes, instant on hit	Risk of serving a stale or subtly-wrong match; tune the threshold
Model routing (small first)	Yes, most traffic on cheap model	Yes, small models respond faster	Routing mistakes send hard queries to a weak model; needs a fallback
Prompt / context trimming	Yes, fewer input tokens	Yes, less to process	Trim too aggressively and you cut the context the answer needed
Batching requests	Yes, better throughput per dollar	No, adds queueing delay	Wrong for interactive UX; great for offline / bulk jobs
Smaller model outright	Yes, lower per-token price	Yes, faster generation	Quality drop on complex tasks; verify on your eval set
Streaming	No, same tokens billed	Perceived only, same total time	More complex client code; partial output can mislead if it errors mid-stream

Pick levers by what you're optimizing, cost, latency, or both, and budget for the trade-off.

💡 Tip: Reach for the levers that buy both axes first, caching, routing, trimming, a smaller model. Streaming and batching are special-purpose: streaming fixes perceived latency for interactive apps; batching trades latency for cost in offline pipelines.

A minimal router and token-budget trimmer

Here is a stripped-down version of the funnel as code: a token-budget trimmer that keeps context under a hard cap, and a router that tries a small model first and escalates only when the answer looks weak. The point is the shape, swap in your own client, models, and validation.

router.py

import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

def token_len(text: str) -> int:
    return len(enc.encode(text))

def trim_to_budget(chunks: list[str], budget: int) -> list[str]:
    """Keep the most relevant chunks first; drop the rest once the budget is spent."""
    kept, used = [], 0
    for chunk in chunks:  # chunks arrive pre-sorted by relevance
        cost = token_len(chunk)
        if used + cost > budget:
            break  # the cheapest token is the one you never send
        kept.append(chunk)
        used += cost
    return kept

SMALL, LARGE = "small-fast-model", "large-strong-model"

def looks_weak(answer: str) -> bool:
    """Cheap heuristics to decide if we must escalate."""
    flags = ("i'm not sure", "i cannot", "as an ai")
    return len(answer) < 20 or any(f in answer.lower() for f in flags)

def answer(question: str, context_chunks: list[str], client) -> str:
    context = "\n\n".join(trim_to_budget(context_chunks, budget=1500))
    prompt = f"Context:\n{context}\n\nQuestion: {question}"

    # 1) try the small model first
    draft = client.complete(model=SMALL, prompt=prompt, max_tokens=300)

    # 2) escalate only if the cheap answer is weak
    if looks_weak(draft):
        return client.complete(model=LARGE, prompt=prompt, max_tokens=300)
    return draft

Two details earn their keep. First, trim_to_budget enforces a hard input ceiling, without it, a few oversized retrieved chunks silently double your bill on every call. Second, looks_weak is deliberately crude; in production you'd validate against the task (did it return valid JSON? did it cite a source?) rather than sniffing for phrases. The architecture is identical to what you'd build for an AI agent: a cheap default path with a deliberate escalation rule.

Common mistakes that quietly drain the budget

Always using the biggest model. The flagship is the most expensive and the slowest. Most production traffic is easy and a small model handles it fine, you just never measured the split. Route first; reserve the big model for the requests that earn it.
No caching of any kind. Real traffic is repetitive: the same questions, the same system prompt, the same retrieved chunks. Without exact, prompt, or semantic caching you re-pay for identical work all day. Caching is usually the single highest-leverage change you can ship.
Ignoring output tokens. Teams obsess over shrinking the prompt and forget that output tokens cost more and dominate latency. A max_tokens cap and a prompt that says "answer in two sentences" often saves more than any input trimming.
Letting context grow unbounded. Stuffing the whole knowledge base "just in case" inflates every single call. Retrieve and trim to a token budget; relevance beats volume.
Optimizing latency you can't feel. Chasing total generation time when the real fix is perceived latency. Stream the response and the same model feels twice as fast.
Tuning blind. Shipping levers without per-request cost and p50/p95 latency dashboards. If you can't see the bill and the tail, you can't tell whether a change helped or hurt.

Takeaways

The whole article in seven lines

The cheapest, fastest token is the one you never send, that single idea drives every lever.
You pay in dollars and milliseconds roughly in proportion to input and output tokens.
Run requests through a funnel: semantic cache, then router, then small model, then large model only on escalation.
Caching, routing, trimming, and a smaller model save both cost and latency, reach for these first.
Batching trades latency for cost (offline only); streaming trades nothing but transforms perceived latency.
Output tokens cost more and dominate latency, cap them and ask for shorter answers.
You cannot optimize what you cannot see: instrument per-request cost and p50/p95 before tuning.

Where to go next

Cost and latency optimization is the final discipline that turns an LLM demo into a shippable product. The strongest next step is to wire a real dashboard, per-request cost, cache hit rate, and the small-vs-large routing split, so every lever you add is measured, not guessed.

Solidify the fundamentals these levers rest on with How LLMs Actually Work, why tokens cost what they cost.
See the routing-and-escalation pattern in a fuller system in Building AI Agents.
Put it all together along the AI Engineer career path, where cost and latency sit alongside evaluation and reliability.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Building AI Agents: From One LLM Call to a Reasoning Loop

Sri Balaji — Sun, 13 Sep 2026 14:44:43 +0000

One call answers. An agent finishes the job.
What an agent actually is
The agent loop, drawn out
One call vs. chain vs. agent, pick the smallest thing that works
A minimal agent loop you can read
Planning, memory, and context
Guardrails: tools are the attack surface
Failure modes that bite in production
Takeaways
Where to go next

One call answers. An agent finishes the job.

You wire up an LLM, send a prompt, get a paragraph back. It's impressive, until the task needs more than one shot. "What's our current AWS bill, and which service jumped this month?" A single call can't answer that. It has no live data, it can't run a query, and it can't check its own math. So it guesses, confidently, and you ship a wrong number.

The fix isn't a smarter prompt. It's a different shape. Instead of one call that must know everything, you give the model tools and let it loop: think about what it needs, call a tool to get it, look at the result, and decide whether it's done. That loop is what people mean by an agent.

📌 Who this is for: Engineers who've made a few LLM calls and now want the model to do things, query a database, hit an API, read a file, and self-correct along the way. You should be comfortable with structured output and tool calling and have a rough feel for LLM cost and latency. No agent framework required; we build the loop by hand so you can see every moving part.

What an agent actually is

An agent is an LLM running in a loop: it reasons about a goal, picks a tool, executes it, observes the result, and repeats, until it decides the goal is met or it hits a limit you set.

The pattern has a name: ReAct, reason → act → observe → repeat. The model reasons in plain language about what to do next, acts by emitting a tool call, then observes the tool's output, which gets fed back into the next turn. Nothing magical: it's a while loop where the LLM is the decision-maker and tools are the hands.

In the real world	In tech
You hand an intern a goal, not step-by-step instructions	You give the agent a task and a system prompt, not a fixed script
The intern looks things up, runs the report, calls the API	The agent calls tools, search, SQL, HTTP, a calculator
They read the result and decide what to do next	Tool output is fed back; the model reasons about the next step
They stop when the task is done, or ask when stuck	The loop ends on a 'final answer' or a step / cost limit
A bad intern loops forever re-checking the same thing	An unguarded agent loops, re-calls tools, and burns tokens

An agent is an intern who can use tools and check their own work.

The agent loop, drawn out

Every agent, no matter the framework, is this cycle. The model never touches the outside world directly; it asks for a tool, your code runs the tool, and the result comes back as the next observation.

The ReAct loop: reason, act, observe, repeat, with a guard that forces an exit.

Start with a goal: The user's task plus a system prompt describing the role and the tools available. This is the only fixed input.
Reason: The model thinks about what it needs next, in plain language. "To find the cost spike I need this month's bill broken down by service."
Choose a tool: It emits a structured tool call: a name (query_billing) and arguments ({ month: "2026-06" }). This is just structured output.
Execute: Your code, not the model, runs the tool, hits the API or DB, and captures the result. The model is sandboxed; it only asks.
Observe: The tool's output is appended to the conversation as an observation and handed back to the model for the next turn.
Done?: The model decides: is the goal met? If yes, it returns a final answer. If no, the loop repeats, bounded by a max-step guard so it can never run forever.

One call vs. chain vs. agent, pick the smallest thing that works

An agent is the most powerful shape and the most expensive, slowest, and least predictable. Reach for it only when the number of steps isn't known up front. If you can hardcode the steps, don't make the model decide them.

Shape	How it works	Use when	Cost & risk
Single LLM call	One prompt, one response. No tools, no loop.	The model already knows the answer or it's pure text-in / text-out (summarize, classify, rewrite).	Cheapest, fastest, most predictable. Bounded by one call.
Chain	A fixed sequence of calls / steps you wire yourself (e.g. retrieve → answer).	You know the steps in advance and the order never changes. RAG is a chain.	Predictable cost (N steps). You control the flow, not the model.
Agent	The model decides which tools to call and how many times, in a loop.	The path depends on intermediate results, unknown number of steps, branching, self-correction.	Variable cost, can loop, hardest to debug. Needs guardrails.

Match the shape to the task. Default to the simplest one that fits.

💡 The honest default: Most "agent" projects are really a chain in disguise. If you can draw the steps on a whiteboard and they never branch, build a chain, it's cheaper, faster, and you can actually test it. Save the agent for the genuinely open-ended cases.

A minimal agent loop you can read

Here's the whole idea in ~50 lines: one tool, the loop, and a hard step cap. No framework. The model gets a tool definition, and on each turn it either calls the tool or returns a final answer. The MAX_STEPS guard is the single most important line, it's the difference between a bounded agent and a runaway bill.

agent.py

import anthropic

client = anthropic.Anthropic()
MAX_STEPS = 6  # hard guard: the loop can NEVER run longer than this

# One tool the model is allowed to use.
TOOLS = [{
    "name": "query_billing",
    "description": "Return this month's cloud spend, broken down by service.",
    "input_schema": {
        "type": "object",
        "properties": {"month": {"type": "string", "description": "YYYY-MM"}},
        "required": ["month"],
    },
}]

def run_tool(name, args):
    # YOUR code runs the tool, the model only asked for it.
    if name == "query_billing":
        return {"EC2": 4200, "S3": 180, "RDS": 2100, "NAT": 950}
    raise ValueError(f"unknown tool: {name}")

def agent(task: str) -> str:
    messages = [{"role": "user", "content": task}]

    for step in range(MAX_STEPS):
        resp = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            system="You are a FinOps assistant. Use tools to get real numbers; never guess.",
            tools=TOOLS,
            messages=messages,
        )

        # No tool call -> the model is done. Return its answer.
        if resp.stop_reason != "tool_use":
            return "".join(b.text for b in resp.content if b.type == "text")

        # Otherwise: execute every requested tool, feed results back, loop.
        messages.append({"role": "assistant", "content": resp.content})
        results = []
        for block in resp.content:
            if block.type == "tool_use":
                out = run_tool(block.name, block.input)
                results.append({
                    "type": "tool_result",
                    "tool_use_id": block.id,
                    "content": str(out),
                })
        messages.append({"role": "user", "content": results})

    # Hit the cap without finishing, fail loud, don't loop forever.
    return "Stopped: hit MAX_STEPS without a final answer."

print(agent("Which service drove our cost up this month, and by how much?"))

Read the control flow, not the API. The model reasons, asks for query_billing, your run_tool executes it, the result goes back, and the model reasons again, this time with real numbers, and returns the answer. The for step in range(MAX_STEPS) is the loop and the guard in one line. Everything an agent framework adds is built on exactly this skeleton.

Planning, memory, and context

Three things separate a toy loop from something that survives real tasks.

Planning

For multi-step tasks, ask the model to lay out a plan first, then execute step by step. A plan in the context window keeps the agent on-rails and gives it something to check progress against, instead of re-deciding the whole approach on every turn (which is how agents drift and loop).

Memory and context

The model has no memory between turns except what's in the message list. Every tool result you append is context the model now "remembers", and context isn't free. Each turn re-sends the entire growing history, so a 10-step agent pays for the conversation 10 times over. The practical levers: summarize old tool results once they're no longer needed verbatim, trim giant payloads before appending them, and keep the system prompt tight. This is the same cost math from LLM cost and latency optimization, just amplified by the loop.

Stopping conditions

Natural stop, the model returns a final answer with no tool call (stop_reason != "tool_use"). The good case.
Step cap, MAX_STEPS hit. Non-negotiable; it's your circuit breaker.
Budget cap, track cumulative tokens / dollars and bail when you cross a threshold, independent of step count.
Wall-clock timeout, for user-facing agents, cap total latency so a slow tool can't hang the request forever.

Guardrails: tools are the attack surface

The model decides what to call; your code decides whether to allow it. Never trust tool arguments blindly, the model can hallucinate a DROP TABLE just as easily as a SELECT. Treat every tool call as untrusted input.

Validate arguments against the schema and your own rules before executing, reject anything out of bounds.
Scope tool permissions, give a support agent read-only DB access, not write. The blast radius of a bad call is whatever the tool can do.
Make destructive tools confirm, require a human approval step for anything that deletes, pays, or emails.
Return errors as observations, not exceptions, when a tool fails, feed the error back so the model can recover instead of crashing the loop.
Log every reason → tool → result triple, when an agent does something weird, this trace is the only way you'll understand why.

Failure modes that bite in production

Agents fail in ways single calls never do, because the loop compounds mistakes. The four below cause most of the pain.

Infinite / stuck loops. The model keeps re-calling the same tool with the same args, never converging, often because the tool result didn't actually answer its question. Without a step cap, this runs until you notice the bill.
No step cap. The single most common omission. One off-by-one in the model's reasoning and you've got an unbounded while True. Always set MAX_STEPS and fail loud when you hit it.
Cost blowups. Every turn re-sends the full history, so cost grows roughly quadratically with step count. A chatty agent that takes 12 turns can cost 20× a single call. Track cumulative tokens, summarize old context, and cap the budget.
Bad tool use. The model calls the wrong tool, passes malformed args, or invents a tool that doesn't exist. Clear tool descriptions, strict schemas, and good error-as-observation handling fix most of it, but expect it and test for it.
Over-agenting. Using an agent where a chain or a single call would do. You inherit all the failure modes above for a task whose steps you already knew. The cure is upstream: pick the right shape.

Takeaways

The whole article in seven lines

An agent is an LLM in a loop: reason → act → observe → repeat (ReAct) until done.
The model only asks for tools; your code executes them and feeds results back as observations.
Pick the smallest shape: single call if it knows the answer, chain if the steps are fixed, agent only when the path is unknown.
A max-step guard is mandatory, it's the line between a bounded agent and a runaway bill.
Context isn't free: every turn re-sends the history, so summarize and trim as the loop grows.
Tools are the attack surface, validate args, scope permissions, confirm destructive actions.
The classic failures are infinite loops, no step cap, cost blowups, bad tool use, and over-agenting, design against all five.

Where to go next

Build the minimal loop above with one real tool against an API you control, then add a step cap and a token counter before you add a second tool. You'll learn more from one hand-built agent than from any framework tutorial.

Tighten the foundation: Structured Output & Tool Calling, tool calls are just structured output, so reliability there is reliability here.
Control the loop's economics: LLM Cost & Latency Optimization, the techniques matter most once every turn re-sends the context.
See where agents fit in the broader role: the AI Engineer career path walks the path from prompting to retrieval to agentic systems.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Evaluating LLM Applications

Sri Balaji — Sat, 12 Sep 2026 13:48:42 +0000

⚡ TL;DR: Treat evals as unit tests for non-deterministic output. Build golden datasets, deterministic checks, and LLM-as-judge scoring, then add regression gates so you ship on evidence instead of vibes. Run both offline and online.

You changed a prompt and have no idea if it got better
The principle: evals are unit tests for non-deterministic output
The shape of an eval pipeline
Choosing a scoring method
A tiny eval harness
Offline vs online: you need both
Common mistakes that cost hours
Takeaways
Where to go next

You changed a prompt and have no idea if it got better

Here is the most common moment in LLM development. You tweak a system prompt, add a line, reword an instruction, swap a model. You run it on the one example you happen to have open, the output looks nicer, and you ship it. Then a week later a user complains the bot got worse at something you never thought to test, and you have no way to prove whether your change helped, hurt, or did nothing at all.

This is vibes-based shipping, and almost everyone does it at first. The reason is simple: with normal code you write a test, it passes or fails, done. With an LLM the output is different every run, there is no single "correct" answer, and "better" is a judgment call. So people skip evaluation entirely, which makes it the hardest and most-skipped part of building real AI products.

This article is the antidote. You will build a tiny eval harness that scores your prompt's outputs against a fixed set of examples, so the next time you change something you get a number, not a feeling.

📌 Who this is for: Anyone who has wired up an LLM call and now wants to change it safely, app developers, prompt engineers, and AI engineers. You should be comfortable reading Python and have made at least one API call to a model. No ML background needed. If you have not written a prompt yet, start with Prompt Engineering Fundamentals.

The principle: evals are unit tests for non-deterministic output

If you cannot measure whether a change made your LLM app better, you are not engineering, you are gambling with extra steps.

The eval mindset

The mental model that unlocks everything: an eval is a unit test for a system whose output you cannot predict exactly. A normal unit test asserts add(2, 2) === 4. You can't do that with an LLM, the same input yields different wording every time. So instead of asserting exact equality, an eval asserts a property holds: the answer contains the right account ID, the JSON parses, the tone is professional, the summary doesn't invent facts.

In the real world	In tech
Unit test asserting exact output	Exact-match scorer (when there IS one right answer)
Test asserting a property (sorted, non-empty)	Heuristic scorer (regex, JSON-parses, length, contains-keyword)
Code reviewer judging readability	LLM-as-judge scoring tone, relevance, faithfulness
Test suite you run before every merge	Eval suite you run before every prompt change
CI failing the build on a regression	Regression gate failing the deploy on a score drop

Same instinct as classic testing, just adapted for output you can't pin down exactly.

The shape of an eval pipeline

Before writing code, picture the flow. You start with a golden set, fixed inputs paired with what a good answer looks like. Each input runs through your prompt, the output goes to one or more scorers, and the scores roll up into an aggregate that a regression gate compares against your last known-good run.

An eval pipeline: golden set → run prompt → layered scorers → aggregate score → regression gate.

Collect a golden set: Gather 20-50 real inputs your app will actually see. Pull them from logs, support tickets, or your own head. For each, write down what a good output must contain, not the exact wording, just the must-haves.
Run the prompt over every case: Loop through the golden set, call the model with your current prompt for each input, and capture the raw output. Pin temperature low (or 0) so runs are comparable.
Score each output: Send every output through your scorers. Start with cheap deterministic checks (does it contain the order ID? does the JSON parse?), then add an LLM-judge for the fuzzy stuff (is the tone right? is it faithful to the source?).
Aggregate and gate: Average the scores into a pass rate per scorer. Compare against your last saved baseline. If the new run drops below it, the gate fails and you investigate before shipping.

Choosing a scoring method

There is no single "the" eval method, you layer several, cheapest first. Run deterministic checks on every case because they cost nothing, and reserve expensive methods (model judges, humans) for the questions cheap checks can't answer.

Method	What it measures	Cost	Signal quality
Exact match	Output equals expected string (classification labels, structured IDs)	Near zero	Perfect, but only where one right answer exists
Heuristic	Properties: regex, JSON-parses, contains keyword, length, no banned words	Near zero	Good for structure/format; blind to meaning
LLM-as-judge	Fuzzy qualities: relevance, tone, faithfulness, helpfulness	$ per case (extra model call)	Strong if the judge prompt is calibrated; biased if not
Human review	Anything subtle a model still gets wrong; ground truth	$$$ + slow	Highest, but doesn't scale; use to calibrate the others

Eval methods, cheapest to most expensive. Layer them, don't pick just one.

💡 Always start deterministic: A surprising amount of "quality" is just structure: valid JSON, the right field present, no apology when none is needed. Heuristic scorers catch these for free and run in milliseconds. Only reach for an LLM-judge once the deterministic checks pass and you still need to grade meaning.

A tiny eval harness

Here is the whole idea in about 60 lines of Python: a golden set, two scorers (one heuristic, one LLM-judge), and an aggregate with a regression gate. The model call is stubbed behind run_prompt and judge so you can drop in whatever provider you use.

eval_harness.py

from dataclasses import dataclass
from statistics import mean

# 1. The golden set: real inputs + what a good answer must contain.
GOLDEN = [
    {"input": "What's the refund window?", "must_include": "30 days"},
    {"input": "Do you ship to Germany?", "must_include": "yes"},
    {"input": "Is my data sold to third parties?", "must_include": "no"},
]

# Your app's prompt under test. Swap in a real API call.
def run_prompt(user_input: str) -> str:
    system = "You are a concise support agent. Answer in one sentence."
    return call_model(system=system, user=user_input, temperature=0)

# --- Scorers: each returns a float in [0, 1] ---

def heuristic_scorer(output: str, case: dict) -> float:
    # Deterministic: does the answer contain the required fact?
    return 1.0 if case["must_include"].lower() in output.lower() else 0.0

JUDGE_PROMPT = (
    "Score the ANSWER from 0-1 for how clearly and politely it "
    "resolves the QUESTION. Reply with ONLY the number.\n"
    "QUESTION: {q}\nANSWER: {a}"
)

def llm_judge_scorer(output: str, case: dict) -> float:
    raw = judge(JUDGE_PROMPT.format(q=case["input"], a=output))
    try:
        return max(0.0, min(1.0, float(raw.strip())))
    except ValueError:
        return 0.0  # un-parseable judge output = fail loudly

SCORERS = {"heuristic": heuristic_scorer, "judge": llm_judge_scorer}

@dataclass
class Result:
    scores: dict          # scorer name -> mean score
    overall: float

def evaluate() -> Result:
    per_scorer = {name: [] for name in SCORERS}
    for case in GOLDEN:
        output = run_prompt(case["input"])
        for name, fn in SCORERS.items():
            per_scorer[name].append(fn(output, case))
    scores = {name: mean(vals) for name, vals in per_scorer.items()}
    return Result(scores=scores, overall=mean(scores.values()))

# --- Regression gate: fail if we dropped below the saved baseline ---
BASELINE = 0.85  # load this from your last green run, don't hardcode forever

if __name__ == "__main__":
    result = evaluate()
    print("per-scorer:", result.scores, "overall:", round(result.overall, 3))
    assert result.overall >= BASELINE, (
        f"REGRESSION: {result.overall:.3f} < baseline {BASELINE}"
    )
    print("PASS, safe to ship.")

Run this before and after a prompt change. The same three cases, the same scorers, two numbers you can compare. That single assert is your regression gate, wire it into CI and a prompt edit that quietly breaks an answer can no longer reach production.

Offline vs online: you need both

Everything above is offline eval, a fixed golden set you run on demand. It is fast, repeatable, and perfect for catching regressions before you ship. But a golden set only contains the inputs you thought of. Real users will phrase things you never imagined.

That's where online eval comes in: scoring a sample of live production traffic. You log real inputs and outputs, run the same scorers (plus lightweight signals like thumbs-up/down, did-the-user-retry, did-they-escalate-to-a-human), and watch the trend. Online eval tells you what offline can't, how the app behaves on the long tail of real questions. The two feed each other: failures you spot online become new cases in your golden set, which makes your offline gate smarter over time. Wiring this into production logging and dashboards is the subject of LLMOps: Productionizing LLM Apps.

Common mistakes that cost hours

No dataset at all. Eyeballing one example is not evaluation. Without a fixed golden set every "improvement" is anecdote. Twenty real cases beat zero, start there, grow it as you find failures.
Judge bias. An LLM-judge inherits the quirks of its model: it favors longer answers, rewards confident tone over correctness, and rates its own outputs higher than a rival model's. Calibrate the judge against human labels on a handful of cases before you trust it.
Only offline. A green offline suite feels safe, but it only tests inputs you imagined. Skipping online eval means real-world failures stay invisible until a user reports them.
Scoring with the model you're testing. Using the same model as both the app and the judge bakes in shared blind spots. Use a different (often stronger) model to judge.
A vague golden answer. "A good response" is unscorable. Write the must-haves, the fact, the format, the forbidden phrases, so a scorer (or a human) can decide objectively.
Letting the baseline rot. A regression gate is only as honest as its baseline. Update it from your last green run, and never lower it to make a failing change pass.

Takeaways

The whole article in seven lines

Evals are unit tests for non-deterministic output, assert properties, not exact equality.
Build a golden set first: 20-50 real inputs + the must-haves of a good answer.
Layer scorers cheapest-first: exact match → heuristic → LLM-as-judge → human review.
Deterministic checks are free and catch most structure bugs, always run them.
A regression gate (new score ≥ baseline) is what kills vibes-based shipping.
Offline eval catches regressions; online eval catches the long tail. You need both.
Watch for judge bias, calibrate against humans and never let the model grade itself.

Where to go next

Evaluation is the discipline that turns prompt-tweaking into engineering. The next step is putting the harness somewhere it runs automatically and watching real traffic.

Sharpen the thing you're evaluating: Prompt Engineering Fundamentals.
Run evals in CI and score live traffic: LLMOps: Productionizing LLM Apps.
See where eval fits in the broader role: the AI Engineer career path.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

Structured Output & Tool Calling

Sri Balaji — Fri, 11 Sep 2026 14:45:10 +0000

⚡ TL;DR: Stop regex-parsing prose. You will learn to force clean JSON against a schema you define and run the request, execute, return loop so the model can safely call your functions, with four levels of reliability to dial in.

The problem with prose
Two mental models
The tool-calling loop
Four levels of reliability
Code: a tool definition and the loop
Code: structured output with a schema
Common mistakes that cost hours
Takeaways
Where to go next

The problem with prose

You ask the model to "extract the customer's name, email, and order total" and it replies: "Sure! The customer is Jane Doe (jane@acme.io) and her order came to $429." Lovely for a human. Useless for your code. Now you are writing a regex to pull the email out, a second one for the dollar amount, and a third to handle the time it decided to answer in Spanish. Every prompt tweak breaks your parser.

The fix is to stop treating the model as a chatbot and start treating it as a function that returns data. Modern LLM APIs give you two tools for this: structured output (the model must answer in a JSON shape you define) and tool calling (the model can ask your code to run a function and use the result). Get these right and the LLM becomes a reliable component you can wire into real software.

📌 Who this is for: You have made a basic LLM API call (see Working with the LLM API) and now you need machine-readable answers or you want the model to actually do things, query a database, call an API, send an email. No ML background required. We use Python, but the concepts map to any SDK.

Two mental models

Structured output is a form the model must fill in. Tool calling is a set of buttons the model is allowed to press.

The whole article in one line

Both are about constraining the model. Left unconstrained, it generates the most plausible-sounding text. That is great for an essay and terrible for an integration. A form has labelled boxes and the model has to put something valid in each one. A button does nothing until pressed, and when pressed, your code runs, not the model's imagination.

In the real world	In tech
A paper form with labelled fields and checkboxes	JSON schema / structured output, the model must return the exact shape
Required fields you cannot leave blank	Schema 'required' array, the model cannot omit them
A row of buttons on a dashboard	Tool definitions, named functions the model may invoke
Pressing a button triggers a machine, not the operator	Your backend executes the tool; the model never runs code itself

Two ways to constrain a free-text model into something your code can trust.

The tool-calling loop

The single most misunderstood thing about tool calling: the model does not run your code. It only emits a request that says "please call get_weather with city=Amsterdam." Your program receives that request, runs the real function, and hands the result back. The model then continues with that fact in hand. It is a conversation that loops until the model has everything it needs to answer.

One full turn of the tool-calling loop. The model and your code take turns until a final answer falls out.

You send the prompt plus tool definitions: Each API call includes the user message AND a list of tools the model is allowed to use, name, description, and a JSON schema for the arguments.
The model decides: It either answers directly, or it returns a 'tool_use' block: the tool name and the arguments it wants, as structured JSON. No prose answer yet.
Your code executes the tool: You match the tool name to a real Python function, validate the arguments, and run it. The model is paused, waiting.
You return the result: Append the tool result to the conversation (linked to the call's id) and send the whole thing back to the model.
The model continues, or loops again: With the result in context it may answer, or request another tool. You repeat until it stops asking and produces a final answer.

Four levels of reliability

There is a ladder from "hope it parses" to "guaranteed valid." Climb only as high as you need, strict schemas and tools cost more tokens and more setup. Match the technique to the job.

Technique	What you get	Reliability	Use it when
Free text	Natural-language prose	Low, you parse strings yourself	Human reads the output directly
JSON mode	Valid JSON, but any shape	Medium, parses, fields not guaranteed	Quick prototypes, loose shapes
Strict JSON schema	JSON matching your exact schema	High, fields, types, enums enforced	Extraction, classification, data into a DB
Tool calling	Validated args for named functions	High, plus the model can act, not just answer	The model must DO something (query, send, fetch)

From least to most constrained. Higher rows are cheaper and looser; lower rows are reliable and machine-ready.

💡 JSON mode is not schema enforcement: Plain JSON mode guarantees the output parses, not that it has the keys you expect. If you need email to always be present and a string, use a strict schema, not just JSON mode. Many a 2am bug is a missing field that "usually" appears.

Code: a tool definition and the loop

Here is a complete, minimal tool-calling loop. The tool is described with a JSON schema for its arguments. The loop keeps running the model until it stops asking for tools, with a hard cap so a confused model can never spin forever.

tool_loop.py

import json
import anthropic

client = anthropic.Anthropic()

# 1. Describe the tool: name, when to use it, and an argument schema.
TOOLS = [
    {
        "name": "get_weather",
        "description": "Get the current temperature for a city, in Celsius.",
        "input_schema": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name, e.g. 'Amsterdam'"},
            },
            "required": ["city"],
        },
    }
]

# 2. The real function the model is allowed to trigger.
def get_weather(city: str) -> dict:
    # In real life: call a weather API. Here we fake it.
    return {"city": city, "tempC": 14, "summary": "cloudy"}

TOOL_FNS = {"get_weather": get_weather}

def run(prompt: str, max_turns: int = 5) -> str:
    messages = [{"role": "user", "content": prompt}]

    for _ in range(max_turns):  # bounded loop, never unbounded!
        resp = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            tools=TOOLS,
            messages=messages,
        )

        if resp.stop_reason != "tool_use":
            # No tool requested -> this is the final answer.
            return "".join(b.text for b in resp.content if b.type == "text")

        # 3. Run every tool the model asked for, collect the results.
        messages.append({"role": "assistant", "content": resp.content})
        results = []
        for block in resp.content:
            if block.type != "tool_use":
                continue
            fn = TOOL_FNS.get(block.name)
            if fn is None:
                output = {"error": f"unknown tool {block.name}"}
            else:
                try:
                    output = fn(**block.input)  # validate in real code!
                except Exception as e:
                    output = {"error": str(e)}
            results.append({
                "type": "tool_result",
                "tool_use_id": block.id,
                "content": json.dumps(output),
            })

        # 4. Feed the results back and loop.
        messages.append({"role": "user", "content": results})

    return "Stopped: hit max_turns without a final answer."

print(run("What's the weather in Amsterdam? Should I take an umbrella?"))

Notice three things: the assistant's tool-call message is appended before the result, each result is keyed to its tool_use_id, and the loop has a max_turns ceiling. Drop any one of those and the loop breaks subtly.

Code: structured output with a schema

When you do not need the model to act, you just want clean data, use a structured-output schema. Define the shape, ask the model to fill it, then validate before you trust it. Pydantic makes the schema and the validation the same source of truth.

extract.py

import json
import anthropic
from pydantic import BaseModel, EmailStr, ValidationError

client = anthropic.Anthropic()

# The form the model must fill in.
class Order(BaseModel):
    customer_name: str
    email: EmailStr
    total_usd: float
    priority: str  # one of: low | normal | high

SCHEMA = {
    "type": "object",
    "properties": {
        "customer_name": {"type": "string"},
        "email": {"type": "string", "format": "email"},
        "total_usd": {"type": "number"},
        "priority": {"type": "string", "enum": ["low", "normal", "high"]},
    },
    "required": ["customer_name", "email", "total_usd", "priority"],
}

# Trick: expose the schema AS a tool and force the model to call it.
# The 'arguments' it produces are your structured output.
def extract(text: str) -> Order:
    resp = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=512,
        tools=[{
            "name": "record_order",
            "description": "Record the extracted order details.",
            "input_schema": SCHEMA,
        }],
        tool_choice={"type": "tool", "name": "record_order"},  # force it
        messages=[{"role": "user", "content": f"Extract the order:\n{text}"}],
    )

    raw = next(b.input for b in resp.content if b.type == "tool_use")

    # NEVER skip this step. The schema guides the model; it does not
    # guarantee semantics. Validate before the data touches your DB.
    try:
        return Order(**raw)
    except ValidationError as e:
        raise ValueError(f"Model returned invalid order: {e}")

order = extract("Jane Doe (jane@acme.io) placed a rush order totalling $429.")
print(order.model_dump())

Common mistakes that cost hours

No validation. A schema guides the model; it does not certify the data is sane. A total_usd of -1 or an email of "n/a" can still come back. Always re-validate (Pydantic, zod, JSON-Schema) before the value reaches your database or an irreversible action.
Trusting hallucinated arguments. The model can invent a plausible-looking user_id that does not exist, or a city you never support. Treat tool arguments like untrusted user input: check ranges, enums, and existence before you act on them.
Unbounded tool loops. Without a max_turns cap, a confused model can call the same tool forever, burning tokens and money. Always bound the loop and log when you hit the ceiling.
Vague tool descriptions. "Gets data" tells the model nothing about when to use it. Write the description like a docstring for a junior dev: what it does, what each argument means, and when NOT to call it.
Forgetting the result id. Each tool_result must reference the tool_use_id of the call it answers. Mismatch them and the model loses the thread of which result belongs to which request.
Side effects on retry. If the model calls send_email twice (it happens), you double-send. Make irreversible tools idempotent or require a human confirmation step.

Takeaways

The whole article in seven lines

Treat the LLM as a function that returns data, not a chatbot that returns prose.
Structured output = a form the model must fill in. Tool calling = buttons it may press.
JSON mode guarantees it parses; a strict schema guarantees the fields and types.
The model never runs your code, it emits a request, your code executes, you feed the result back.
The loop is: prompt+tools → tool call → execute → return result → continue → final answer.
Always validate model output before it touches a database or an irreversible action.
Always bound the tool loop with a max-turns ceiling.

Where to go next

Structured output and tool calling are the two primitives that turn an LLM from a text generator into a software component. The natural next step is to chain many tool calls together with memory and a goal, that is what an agent is.

Came in cold? Start with Working with the LLM API for the request/response basics this article builds on.
Ready to go further? Building AI Agents wires this loop into a goal-driven agent with planning and memory.
Want the full track? Follow the AI Engineer path from foundations to production.

Originally published on TheSimplifiedTech, where this guide is interactive, with in-browser terminal labs and diagrams. Learn cloud and DevOps by doing, no videos.

DEV Community: Sri Balaji

Secure API Design: Building APIs That Resist Abuse

Contents

The bug that ships in every junior API

One principle: the server trusts nothing the client sends

The request gauntlet

The controls, and what each one stops

The insecure endpoint vs. the secure one

Common mistakes that cost hours (or headlines)

Takeaways

Where to go next

Preventing Injection Attacks (SQLi, XSS & Friends)

Contents

A search box that ran DROP TABLE

The principle: data is not code

The picture: input crossing the boundary

The injection family at a glance

Code: the vulnerable query and the fix

Why allowlists beat blocklists

Common mistakes that get people breached

Takeaways

Where to go next

Secrets Management: Keeping Keys, Passwords, and Tokens Out of the Wrong Hands

Contents

The 3 a.m. cloud bill

What counts as a secret

How secrets should actually flow

Where NOT to put secrets (and where to)

Reading a secret the right way

Rotation and least-privilege

Detecting leaked secrets

Common mistakes that cost hours (or accounts)

Takeaways

Where to go next

The OWASP Top 10, Explained

Contents

The list every attacker has already read

One sentence, then a picture

The OWASP Top 10 at a glance

A few of these, made concrete

Broken Access Control (A01)

Injection (A03)

Server-Side Request Forgery (A10)

How to use the Top 10 in practice

Fixing a real one: a server-side ownership check

Common mistakes that cost hours

Takeaways

Where to go next

Authentication vs Authorization: The Two Most-Confused Words in Security

Contents

Same prefix, completely different jobs

Two questions, in order

Where each check lives in a request

Sessions vs tokens: how identity is carried

OAuth and OIDC, without the jargon

In code: verify identity, then check permission

Common mistakes that ship security holes

Takeaways

Where to go next

What Is Application Security?

Contents

Your app works. Now someone is trying to break it.

A one-sentence definition

Where security lives in the SDLC (shift-left)

Trust boundaries and attack surface

The principles that hold it all together

The CIA triad, in one breath

Defense in depth and least privilege

The main domains of AppSec

Common misconceptions that get people breached

Takeaways

Where to go next

Fine-Tuning LLMs: When and How

Contents

The trap everyone falls into

One principle, one analogy

The fine-tuning workflow

Prompting vs RAG vs fine-tuning

What the data actually looks like

Evaluating the result