DEV Community: Devanshu Biswas

The human-in-the-loop agent: an 8B model proposes, Python pauses the risky calls, a human decides

Devanshu Biswas — Tue, 04 Aug 2026 08:51:19 +0000

You don't let an 8B model email your customers or move money on its own. So this agent stops and asks. A safe lookup auto-runs; sending an email or issuing a refund pauses for a human to approve, deny, or edit — and every proposal, verdict, and decision is written to an audit trail. The design principle is a hard separation of powers: the model plans and rates its own confidence, but the risky decision is taken away from it and given to deterministic Python and a human.

Uncertainty detection: one verdict from three signals

For every proposed action, a pure-Python gate combines three independent signals — the tool's declared risk, a sensitive-capability check, and the model's own confidence. Any one of them demands a human.

SENSITIVE_CAPABILITIES = {"send_email", "spend_money", "delete", "external_write"}

class ApprovalGate:
    def __init__(self, min_confidence=0.60): self.min_confidence = min_confidence
    def assess(self, tool, args, confidence):
        signals = []
        if tool.risk == "high":
            signals.append(f"tool '{tool.name}' declares risk=high")
        if tool.capabilities & SENSITIVE_CAPABILITIES:          # defence in depth
            signals.append("touches sensitive capability")
        if confidence < self.min_confidence:                    # low conf == risk
            signals.append(f"model confidence {confidence:.2f} < {self.min_confidence:.2f}")
        return RiskVerdict(requires_approval=bool(signals),     # ANY signal → pause
                           risk=tool.risk, confidence=confidence, signals=signals)

Risk is declared by the tool author once, not argued by the model at runtime. The sensitive-capability check is a backstop — even an unmarked tool that sends email or spends money trips the gate. And a safe tool the model rated below 0.60 pauses too: when the model is guessing an amount or address, that uncertainty is itself a reason to ask. Because it's deterministic, the same proposal always earns the same verdict.

Pause, then resume with validated context

On a risky action the loop suspends and asks a decision source (scripted for a reproducible run, or a live CLI [a]pprove / [d]eny / [e]dit). A Decision carries a verdict, the approver, a reason, and — for an edit — a patch of argument overrides. On resume the human's patch is merged over the model's args, so the human's fields win; a deny simply skips the tool and the task carries on.

if decision.is_go:
    run_args = dict(action.args)
    if decision.verdict == "edit" and decision.edited_args:
        run_args.update(decision.edited_args)        # <- human patch wins
    res = tool.run(run_args)                          # RESUME → execute
    self.audit.executed(i, action.tool, run_args, res)
else:                                                 # DENY
    self.audit.refused(i, action.tool, decision.reason)  # carry on gracefully

And if nobody answers at all, the source returns None and the agent fails closed — it synthesises a safe-default denial by auto-escalation. Unsupervised never means unleashed.

Every decision is audited

The audit trail is deliberately boring and durable: every event is one JSON line appended to a .jsonl file, stamped with a monotonic seq — a step counter, not a wall clock — so two identical runs produce byte-identical logs and nothing leaks about when it ran.

class AuditLog:
    def record(self, event, step, **fields):
        self._seq += 1
        entry = {"seq": self._seq, "step": step, "event": event, **fields}
        with open(self.path, "a", encoding="utf-8") as fh:
            fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
        return entry

A real run

The captured transcript is a live run against NVIDIA NIM (meta/llama-3.1-8b-instruct). The task: a customer's order arrived damaged — look it up, email an apology, refund the $60 shipping. The model planned three actions and rated each: [('lookup_order', 1.0), ('send_email', 0.9), ('issue_refund', 0.8)].

The gate auto-ran the safe lookup, then suspended on both send_email and issue_refund despite high confidence — because risk overrides. The email was approved with a human edit to the subject and resumed; the refund was denied, so no money moved. Fifteen events landed in audit-log.jsonl, the refusal was carried into the final reply (which never claims the refund happened), and the refund's absence from the money ledger is provable, not merely asserted.

Step through the real assess → pause → decide → resume → audit pipeline, read the full audit trail, and browse the code at: https://dev48v.infy.uk/agentic/project6-hitl.html (repo: github.com/dev48v/agentic-ai-from-zero)

Next up, Project 7: a cost-aware agent router.

One Helm template, eight Deployments: how range over values.yaml collapses your k8s YAML

Devanshu Biswas — Tue, 04 Aug 2026 08:50:37 +0000

Hand-write the Kubernetes manifests for a microservice platform and you end up with a pile of near-identical files: eight Deployments that differ only in name, image, port, replicas, and a couple of env values, plus eight matching Services. That duplication is exactly what Helm removes. The trick is to split the manifests into two halves — the shape goes into templates, the data goes into values.yaml — and let one template range over the data.

The data half: a services map

values.yaml holds everything the flat YAML used to hard-code, now as values. The star is the services: map, keyed by service name, with the shared probes factored out once instead of repeated eight times:

probes:                       # ONE copy, stamped on all 8 apps
  readiness: { path: /actuator/health/readiness, initialDelaySeconds: 10, periodSeconds: 10 }
  liveness:  { path: /actuator/health/liveness,  initialDelaySeconds: 20, periodSeconds: 15 }

services:                     # ← the map the templates range over
  order-service:
    image: orderhub/order-service
    tag: "0.1.0"
    port: 8082
    replicas: 2
    profile: "prod,k8s"
    resources: { requests: {cpu: 250m, memory: 448Mi}, limits: {cpu: "1", memory: 768Mi} }
  # + config-server, eureka-server, api-gateway, inventory, payment, shipping, notification

The shape half: one range renders all eight

This is the heart of the templatization. {{- range $name, $svc := .Values.services }} loops the entire Deployment block once per service:

{{- range $name, $svc := .Values.services }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ $name }}
  labels:
    {{- include "orderhub.selectorLabels" (dict "name" $name) | nindent 4 }}
    {{- include "orderhub.commonLabels" $ | nindent 4 }}
spec:
  replicas: {{ $svc.replicas }}
  template:
    spec:
      containers:
        - name: {{ $name }}
          image: "{{ include "orderhub.image" (dict "svc" $svc "root" $) }}"
          ports: [ { containerPort: {{ $svc.port }} } ]
          envFrom:
            - configMapRef: { name: {{ $.Values.config.name }} }
            - secretRef:    { name: {{ $.Values.secrets.name }} }
          readinessProbe: { httpGet: { path: {{ $.Values.probes.readiness.path }}, port: {{ $svc.port }} } }
          livenessProbe:  { httpGet: { path: {{ $.Values.probes.liveness.path }},  port: {{ $svc.port }} } }
{{- end }}

Two Helm mechanics make it work. Inside a range the dot rebinds to the current service, so you reach the release/chart/global values through $ — the root — as in $.Values.config.name and $.Values.probes, and the per-item values through $svc / $name. The Day-43 probes are kept verbatim but parameterised: their paths come from .Values.probes, their port from $svc.port. A service.yaml uses the identical loop to render all eight ClusterIP Services, their selectors sharing the Deployment's labels via a helper so they can never drift.

The new front door: an Ingress that routes by path

The flat manifests left every Service ClusterIP — internal only. The Ingress finally opens the stack to the world, gated by a single value:

{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
spec:
  ingressClassName: {{ .Values.ingress.className }}     # nginx
  defaultBackend:                                       # catch-all → gateway
    service: { name: api-gateway, port: { number: 8080 } }
  rules:
    - host: {{ .Values.ingress.host | quote }}          # orderhub.local
      http:
        paths:
          {{- range .Values.ingress.routes }}
          - path: {{ .path }}                           # /api/orders, /api/inventory, …
            pathType: {{ .pathType | default "Prefix" }}
            backend:
              service: { name: {{ .service }}, port: { number: {{ .port }} } }
          {{- end }}
{{- end }}

Each /api/* prefix maps to its service by Prefix match; anything unmatched falls through to api-gateway as the default backend. Because the whole object is wrapped in {{- if .Values.ingress.enabled }}, flipping the front door off is a one-value --set, not YAML surgery.

The payoff

helm install orderhub ./helm/orderhub renders 18 objects from about seven template files, and retuning anything is now a value override:

helm upgrade orderhub ./helm/orderhub --set services.order-service.replicas=4
helm rollback orderhub 1 -n orderhub          # undo, atomically

The topology hasn't changed from the hand-written manifests — but it's now generated, versioned, and roll-back-able as a single release.

Run helm install and watch one template range over the services map, then curl the Ingress to see it route, live at: https://dev48v.infy.uk/orderhub/day44-helm-ingress.html

Lost in the middle: why a 100k-token context window is not 100k tokens of attention

Devanshu Biswas — Tue, 04 Aug 2026 08:49:55 +0000

An LLM does not read a long context evenly. Bury a single fact — a needle — inside a long stack of documents and ask the model to retrieve it, and how reliably it succeeds depends heavily on where the fact sits. Facts at the very start (primacy) or the very end (recency) are recalled reliably; a fact in the middle is far more likely to be missed. Plot accuracy against needle position and you get a U-shaped curve — high at both edges, sagging in the middle. That's the finding of Liu et al., 2023, "Lost in the Middle: How Language Models Use Long Contexts."

And it gets worse as the context grows. The more documents you stuff in, the deeper the middle sags. So a 100k-token window does not mean 100k tokens of usable attention. Long context ≠ used context.

Modelling the U: primacy, recency, and a decaying floor

Map a needle in document i of N to a relative position r = (i-1)/(N-1) in [0,1]. The edges are lifted by two Gaussian bumps — primacy peaks at the start, recency peaks at the end — and in the middle both have decayed to nearly zero:

import math

def primacy(r, strength=0.97, width=0.28):
    return strength * math.exp(-(r / width) ** 2)      # strong at r=0

def recency(r, strength=1.00, width=0.22):
    return strength * math.exp(-((1 - r) / width) ** 2) # strong at r=1

def edge_advantage(r):
    return max(primacy(r), recency(r))   # ~1 at the edges, ~0 in the middle

A middle document has no edge advantage — its accuracy is just the background floor, and that floor decays as the context grows, because more documents compete for finite attention:

def middle_floor(n, floor0=0.85, k=0.055):
    return floor0 * math.exp(-k * (n - 1))

middle_floor(5)   # ~0.68  short context -> middle is still findable
middle_floor(24)  # ~0.30  the middle is sagging
middle_floor(60)  # ~0.03  the middle has effectively vanished

Combine them and you get a clean, always-U-shaped position-to-accuracy curve:

def accuracy(i, n, floor0=0.85, k=0.055):
    r = rel_pos(i, n)
    floor = middle_floor(n, floor0, k)
    acc = floor + (1 - floor) * edge_advantage(r)
    return max(0.02, min(0.995, acc))       # clamp: never 0, never 1
# n=24: start -> ~0.99, middle -> ~0.30, end -> ~1.00   (a clean U)

At the edges edge ≈ 1 so acc ≈ 1; in the middle edge ≈ 0 so acc ≈ floor. A bigger or long-context-tuned model raises floor0 and shrinks k — a flatter, higher U with a wider usable middle.

The fix is positional, not magical

If accuracy depends on position, then control the positions. Rerank retrieved chunks by relevance with a cross-encoder, keep only the few best, then interleave them onto the edges of the prompt — best and second-best at the start and end, weakest buried in the middle where a miss costs least.

def edge_first_order(ranked):
    "Put the most relevant chunks at the START and END, weakest in the middle."
    head, tail = [], []
    for idx, chunk in enumerate(ranked):
        (head if idx % 2 == 0 else tail).append(chunk)
    return head + list(reversed(tail))   # best -> edges, worst -> middle

ordered = edge_first_order(rerank(chunks, query, cross_encoder))

The practical toolkit is short: retrieve fewer, better chunks; rerank with a cross-encoder (Cohere Rerank, bge-reranker, a fine-tuned scorer); place the strongest evidence at the top and bottom of the prompt; keep the question near an edge; and reach for long-context-tuned models, which flatten the U, when you truly need length.

Above all, the takeaway: "it's in the context" is not the same as "the model used it." Verify with a needle-in-a-haystack eval before you ship — don't assume.

Place a needle at any depth, grow the context length, and watch the U-curve's middle collapse, live at: https://dev48v.infy.uk/ai/days/day54-lost-in-the-middle.html

ALiBi: give a Transformer position sense with one line, and let it test longer than it trained

Devanshu Biswas — Tue, 04 Aug 2026 08:49:13 +0000

Self-attention is order-blind: shuffle the tokens and the raw QKᵀ scores come out identical, so a Transformer has to be told where each token sits. The classic fix adds a positional embedding to every token vector — sinusoidal or learned. But those absolute codes are welded to the exact positions seen in training. Feed a model trained at length 1024 a sequence of 3000 and it meets position indices it has never represented, and quality collapses.

ALiBi (Press, Smith & Lewis, 2021 — "Train Short, Test Long") throws positional embeddings out entirely. Instead it adds a fixed, unlearned linear bias straight onto the attention scores, before the softmax.

One matrix: −m·(i−j)

The whole of ALiBi lives in the distance matrix (i − j) — how many steps a key is behind the query. Multiply by a slope and negate:

def distance(N):
    i = np.arange(N)[:, None]          # (N,1) query index
    j = np.arange(N)[None, :]          # (1,N) key index
    return i - j                       # (N,N)  >=0 in the causal region

def alibi_bias(N, m):
    return -m * distance(N)            # -m*(i-j) : linear recency penalty

It's 0 on the diagonal and grows more negative to the left — a penalty that grows linearly with how far back a key is. There is nothing to learn.

Per-head slopes form a geometric ladder

Each attention head gets its own fixed slope, so the model attends at many scales at once. For n heads the slopes are the geometric sequence m_h = 2^(-8h/n):

def head_slopes(n):
    start = 2 ** (-8 / n)              # = 2^(-8/n)
    return np.array([start ** (h + 1) for h in range(n)])

head_slopes(8)
# [0.5, 0.25, 0.125, 0.0625, 0.03125, 0.015625, 0.0078125, 0.00390625]
#  steep (local)  ------------------------------->  gentle (far)

Steep slopes give sharp recency — only the last few keys survive. Gentle slopes let a head gaze far back. Together they're a built-in, multi-scale recency prior.

The whole attention change is one added line

def alibi_attention(Q, K, V, m):
    N, d = Q.shape
    scores = (Q @ K.T) / np.sqrt(d)    # (N,N) raw QK^T
    scores = scores + alibi_bias(N, m) # <-- the ONE new line
    scores[~causal_mask(N)] = -np.inf  # can't see the future
    A = np.stack([softmax(r) for r in scores])
    return A @ V, A                    # output, attention weights

No positional embedding is added to Q, K, or the inputs anywhere. The softmax then turns that straight-line penalty into a clean exponential distance decay, giving each head a bounded effective window of about 1/m.

Why it extrapolates: distance, not position

The bias reads only i − j. Slide the whole window along the sequence and every entry is unchanged — it is translation-invariant, so the length N never appears as an absolute index the way a positional embedding would.

# translation invariance: bias(i,j) depends ONLY on (i-j)
b = alibi_bias(8, 0.25)
assert np.allclose(b[7,3], b[6,2])     # same distance 4 -> same bias
assert np.allclose(b[5,5], 0.0)        # distance 0 -> no penalty

# softmax turns the LINEAR bias into an EXPONENTIAL decay in distance:
#   weight(d)  ∝  exp(-m * d)          # bounded effective window ~ 1/m

Train at 1024, run at 3000: the model only ever sees relative distances, and far ones are simply taxed more. The attention-by-distance profile is the same shape at any length, which is exactly why a short-trained model extrapolates — and why ALiBi rides in BLOOM and MPT.

ALiBi vs sinusoidal vs RoPE

Sinusoidal embeddings are absolute and added to the inputs; they don't extrapolate past trained positions. RoPE is relative but applies its signal by rotating Q and K. ALiBi needs no embedding table and no rotation — just one added matrix of −m·(i−j), with a fixed per-head slope. It's the cheapest recipe of the three, and the one that most naturally handles test sequences longer than training.

Build the QKᵀ matrix, watch the bias ramp add on top, drag the per-head slope, and slide test length past train length to see the decay curves stay glued together, live at: https://dev48v.infy.uk/dl/day54-alibi.html

Random projections: compress 10,000 dimensions to 30 by multiplying with random noise

Devanshu Biswas — Tue, 04 Aug 2026 08:48:31 +0000

Here is one of the most audacious results in machine learning: you can slash a dataset from thousands of dimensions down to a few dozen by multiplying it with a matrix of random numbers — and the geometry survives. No looking at the data, no eigenvectors, no training. Just draw a random linear map R and send every point x to (1/√k)·R·x. The pairwise distances between points barely change.

The Johnson–Lindenstrauss lemma

The licence for this is the Johnson–Lindenstrauss lemma. For any set of n points, there exists a random linear map into k ≈ O(log n / ε²) dimensions that preserves every pairwise distance within a factor of (1 ± ε). The astonishing part is what's absent from that formula: the original dimension D. A 10,000-dimensional bag-of-words and a 1,000,000-dimensional one collapse to the same target size — k depends only on how many points you have and how much distortion you'll tolerate.

Why does a random direction preserve length? Because the projected squared length is an average of k independent contributions, each with mean equal to the true squared length. By concentration of measure it clusters tightly around the truth, and averaging more of them (bigger k) makes the estimate tighter — the distortion falls like 1/√k.

The entire "model" is a random matrix

// standard normal via Box–Muller
function randn(rng) {
  let u = 0, v = 0;
  while (u === 0) u = rng();
  while (v === 0) v = rng();
  return Math.sqrt(-2 * Math.log(u)) * Math.cos(2 * Math.PI * v);
}
// k×D Gaussian projection matrix, row-major, scaled 1/√k
function gaussianMatrix(k, D, rng) {
  const R = new Float64Array(k * D);
  const s = 1 / Math.sqrt(k);
  for (let i = 0; i < k * D; i++) R[i] = s * randn(rng);
  return R;                          // E‖Rx‖² = ‖x‖²
}

Nothing about your data is ever consulted — the map is chosen before the data is seen. The 1/√k factor is the calibration that makes the projection preserve lengths in expectation rather than inflating them by √k. Reducing a point is then a single matrix–vector multiply; for a whole dataset it's one (N×D)·(D×k) product. No fitting, no SVD, no iteration.

You don't even need real Gaussians

Achlioptas showed that entries drawn as √3 × {+1 with prob 1/6, 0 with prob 2/3, −1 with prob 1/6} give the same JL guarantee — yet two-thirds of the entries are zero (so you skip those multiplies) and the rest are just additions and subtractions.

function sparseMatrix(k, D, rng) {
  const R = new Float64Array(k * D);
  const s = Math.sqrt(3 / k);        // √3 · (1/√k)
  for (let i = 0; i < k * D; i++) {
    const u = rng();
    R[i] = u < 1/6 ? s : (u < 1/3 ? -s : 0);   // ⅔ are zeros
  }
  return R;
}

Sizing k, and where it sits

The exact JL bound (the one scikit-learn ships) makes the constant explicit:

import numpy as np
def jl_min_dim(n, eps):
    return int(np.ceil((4 * np.log(n)) / (eps**2/2 - eps**3/3)))

jl_min_dim(1_000_000, 0.1)   # ~ 15,000  — independent of D
jl_min_dim(1_000_000, 0.3)   # ~ 1,700

That bound is a worst-case guarantee holding for every pair simultaneously (a union bound over all n(n−1)/2 of them), so it's famously loose — the typical RMS distortion drops below ε at a far smaller k. In practice you reach for scikit-learn's SparseRandomProjection and johnson_lindenstrauss_min_dim, and the whole thing is a single blazing matmul.

Against the data-dependent reducers — PCA computes an SVD and keeps variance; t-SNE/UMAP preserve local neighbourhoods for visualisation — random projection makes the opposite bet: don't look at the data at all, just multiply by random noise. You lose PCA's optimality but gain a distance-preserving guarantee, a target size that ignores D, and speed. It even degrades gracefully — pick k too small and distances just blur, they never break. That's why it's the go-to front-end for kNN, k-means, and locality-sensitive hashing on enormous feature spaces.

Slide k, swap the matrix type, and watch the distortion histogram tighten around 1.0, live at: https://dev48v.infy.uk/ml/day54-random-projections.html

ULID from scratch: 128 bits that sort by time, and why UUIDv4 never can

Devanshu Biswas — Tue, 04 Aug 2026 08:47:48 +0000

A UUIDv4 is 128 bits of pure randomness — globally unique, but with no order and no meaning. Insert a stream of them as primary keys and you fragment the database's B-tree, because each new row lands at a random spot in the index. A ULID carries the same 128-bit budget but spends it with intent: 48 bits of millisecond timestamp up front, 80 bits of randomness behind. Put the time first, big-endian and fixed-width, and sorting the IDs as plain strings sorts them by creation time. That's the one superpower UUIDv4 will never have — and you can build the whole thing in under 100 lines.

The shape: Crockford Base32, 26 characters

const B32 = "0123456789ABCDEFGHJKMNPQRSTVWXYZ"; // no I L O U
const TIME_LEN = 10;   // 48-bit ms timestamp   -> 10 chars
const RAND_LEN = 16;   // 80 bits of randomness -> 16 chars
const RAND_BYTES = 10; // 80 bits = 10 bytes

Crockford's alphabet drops I L O U so nothing is mistaken for 1 or 0, or spells accidental words. 10 + 16 = 26 characters, 128 bits, same as a UUID but with no dashes.

Encode the timestamp — plain base conversion

A millisecond timestamp is at most 48 bits, and 2⁴⁸ sits comfortably below JavaScript's exact-integer ceiling of 2⁵³, so ordinary arithmetic is safe. Build the ten characters most-significant-first so the text stays big-endian:

function encodeTime(ms){
  let out = "";
  for (let i = TIME_LEN - 1; i >= 0; i--){
    const mod = ms % 32;              // low base-32 digit
    out = B32[mod] + out;             // prepend (big-endian)
    ms = (ms - mod) / 32;             // shift right by one digit
  }
  return out;                         // 10 chars, most-significant first
}

The random half must be cryptographically strong (crypto.getRandomValues, never Math.random()) and kept as bytes — 2⁸⁰ is far past what a Number holds exactly, so it can never be one integer. Streaming 8-bit bytes into 5-bit Base32 characters is a small bit-buffer, and since 80 divides evenly by 5, exactly 16 characters fall out with no padding.

Monotonicity: same millisecond, still sortable

Here's the subtlety. Generate two ULIDs in the same millisecond with fresh random tails and their relative order is a coin toss. The monotonic factory fixes it by remembering the last value and, on a repeat millisecond, incrementing the random bytes by one — so the newer ID's tail is always strictly larger, and it always sorts after.

function monotonicFactory(){
  let lastTime = -1, lastRand = null;
  return function(ms = Date.now()){
    if (ms === lastTime){
      lastRand = incrementBytes(lastRand.slice()); // +1 on a copy
    } else {
      lastTime = ms;
      lastRand = randomBytes();                    // new ms -> new random
    }
    return encodeTime(lastTime) + encodeRandom(lastRand);
  };
}

"Add one" to an 80-bit byte array is grade-school carrying: bump the last byte, or wrap 0xFF to 0 and carry left.

Decode is the mirror, and the sort is pure structure

Decoding runs the same steps backwards — normalise (O→0, I/L→1, upper-case), split off the first ten and last sixteen characters, then reverse each encoder. The timestamp is base-32 Horner's method:

function decodeTime(str){
  let ms = 0;
  for (const c of str) ms = ms * 32 + B32.indexOf(c); // base-32 Horner
  return ms;                                           // milliseconds
}

The sort guarantee needs no comparator: the timestamp is first, big-endian, fixed-width, and the alphabet ascends in value — so byte-comparing two ULIDs compares their timestamps first, then their random tails, which is exactly chronological order. Break any one of those three properties and the guarantee evaporates.

ULID vs UUIDv4, in one line

Both are 128-bit IDs. UUIDv4 is random, unsortable, carries no time, and fragments an index on insert. ULID trades a little randomness for a timestamp and gets sortability, index locality, and a decodable creation moment — which is why it makes such a good time-ordered primary key. (One caveat: that timestamp is not a secret; anyone can decode it, so don't use a ULID where you wouldn't print a creation time.)

Generate a batch, burst several in the same millisecond to prove sortability, and decode any ULID back to the moment it was born, live at: https://dev48v.infy.uk/solve/day54-ulid-generator.html

Verbalized confidence: make an LLM state how sure it is, then check if 90% really means 90%

Devanshu Biswas — Tue, 04 Aug 2026 08:47:07 +0000

Ask a model a question and it answers with the same flat certainty whether it knows the capital of Australia or is guessing a coin-flip. Verbalized confidence fixes half of that: you ask it to attach an explicit self-estimate — "Answer: Canberra, Confidence: 90%." That single extra line turns an opaque guess into a signal you can gate, route, or abstain on. The other half — the part everyone skips — is checking whether the number is honest.

Eliciting the number is one system instruction

The whole behaviour lives in a system prompt that does three jobs: ask for a probability of being correct (not a vibe), give the model explicit permission to say low numbers so it isn't pushed to sound sure, and fix a parseable output shape. Here's the copy-paste template:

You are a careful assistant. Answer the question, then state how
confident you are that your answer is correct, as a percentage
from 0 to 100.

Rules:
- Base the number on how likely you are to be RIGHT, not on how
  sure you want to sound. If you are guessing, say so with a low
  number — being honestly unsure is correct behaviour, not a failure.
- Reserve 90-100% for answers you would bet money on. Use 50% for a
  true coin-flip. Spread the rest in between.
- Do not default to a round, confident-sounding number.
- Output EXACTLY this and nothing else:

Answer: <your answer>
Confidence: <0-100>%

Parsing it back is a couple of regexes with a clamp and a sensible default, so a stray format never crashes the pipeline.

A stated confidence is only worth anything if it's calibrated

A stated 90% is a prediction about the world: "of a hundred questions I felt this sure about, I'd get about ninety right." Calibration is whether that holds. To test it you need a labelled eval set, then you bin predictions by stated confidence and measure the accuracy actually observed in each bin.

function binByConfidence(preds, width = 10) {
  const nb = Math.round(100 / width);
  const bins = Array.from({ length: nb }, (_, i) => (
    { lo: i * width, hi: (i + 1) * width, confs: [], correct: 0, n: 0 }));
  for (const p of preds) {
    let idx = Math.floor(p.confidence / width);
    if (idx >= nb) idx = nb - 1;                   // 100% → last bin
    const b = bins[idx];
    b.n++; b.confs.push(p.confidence);
    if (p.correct) b.correct++;
  }
  for (const b of bins) {
    b.acc      = b.n ? b.correct / b.n : 0;                              // y
    b.meanConf = b.n ? b.confs.reduce((s, c) => s + c, 0) / b.n / 100 : 0; // x
  }
  return bins;
}

Plot each bin as (mean stated confidence, empirical accuracy) against the y = x diagonal and you have a reliability diagram. Collapse the whole diagram into one scalar and you have Expected Calibration Error — the count-weighted average distance from the diagonal:

function expectedCalibrationError(preds, width = 10) {
  const N = preds.length;
  const bins = binByConfidence(preds, width);
  return bins.reduce((ece, b) =>
    ece + (b.n ? (b.n / N) * Math.abs(b.acc - b.meanConf) : 0), 0);
}
// overconfident model  → ECE ≈ 0.40  (claims ~85%, right ~50%)
// well-calibrated model → ECE ≈ 0.06  (claims ~75%, right ~75%)

Why LLMs skew overconfident — and what to do

The failure mode is baked in by training. Pretraining rewards fluent, assured continuations; hedging is rare in the data. RLHF then rewards answers humans like, and humans reward confident, decisive replies — so the model learns that sounding sure scores while honestly saying "maybe 55%" does not. The result: verbalized numbers cluster high and flat, and the curve sags below the diagonal in the top bins.

You push back on three fronts. Prompt-side: ask for a probability of being correct, permit low numbers, anchor the scale. Sampling-side: draw N answers and use their agreement as an empirical confidence (self-consistency), which is often better calibrated than the model's self-report. And measurement: you cannot trust a confidence you've never scored against labels. Once it's calibrated, that number becomes a real control lever — abstain, escalate to a human, or route to a bigger model whenever confidence is low.

Toggle an overconfident model against a calibrated one and watch the reliability curve bend, live at: https://dev48v.infy.uk/prompt/day54-verbalized-confidence.html

Build iOS Settings from scratch: a data-driven list, a real toggle, and a push-navigation stack

Devanshu Biswas — Tue, 04 Aug 2026 08:46:24 +0000

The screen behind every app's gear icon looks deceptively simple: rounded cards of rows, an uppercase header, a grey footnote. But that inset-grouped list is a small, complete UI system — a data-driven renderer, four kinds of trailing accessory, and an in-panel navigation stack where a choice made deep in a subpage flows back to the row that sent you. Here's how to build it in vanilla HTML, CSS and JS, no framework.

Describe the whole screen as data

The first rule: don't hand-write rows. Describe every screen as a model — a map of screens, each a list of groups, each group a list of row descriptors. A descriptor names its type, its icon and tint, the state key it reads and writes, and for a chevron row the screen it pushes plus a value function of state.

const SCREENS = {
  root: { title: "Settings", groups: [
    { footer: "Airplane Mode turns off Wi-Fi and Bluetooth.", rows: [
      { type: "toggle", key: "airplane", label: "Airplane Mode", icon: "airplane", tint: "#ff9500" },
      { type: "nav", label: "Wi-Fi", icon: "wifi", tint: "#007aff", screen: "wifi",
        value: s => s.airplane ? "Off" : (s.wifiOn ? s.network : "Off") },
    ]},
  ]},
  display: { title: "Display & Brightness", groups: [ /* … */ ] },
};

One data structure, one renderer. Adding a setting is adding a map entry, never writing markup.

The toggle switch that can't lie

The iOS switch is a real <button> with role="switch" and aria-checked. The knob is a circle moved by a CSS transform transition, and the class .on both colours the track and slides the knob — so the visual state and the announced state come from one boolean and can never disagree.

const sw = document.createElement("button");
sw.setAttribute("role", "switch");
sw.className = "s-switch" + (state[r.key] ? " on" : "");
sw.setAttribute("aria-checked", String(!!state[r.key]));
sw.onclick = () => {
  state[r.key] = !state[r.key];                 // flip the source of truth
  sw.classList.toggle("on", state[r.key]);      // .on slides the knob (CSS)
  sw.setAttribute("aria-checked", String(state[r.key]));
  onStateChange();                              // repaint everything that reads it
};

Navigation is an array of screen ids

The clever part is the push stack. Navigation is just an array of screen ids; the DOM is a horizontal strip of full-width panels, and you show depth d by translating the strip -d × 100%. Push builds a fresh panel, appends it, then shifts — it slides in. Pop shifts back, then removes the panel after the transition ends.

let nav = ["root"];
function push(id){
  nav.push(id);
  stack.append(renderScreen(id));                 // fresh panel off to the right
  stack.style.transform = `translateX(-${(nav.length-1)*100}%)`;
  syncNavbar();
}

The touch that makes an iOS back button feel oriented: it's labelled with the previous screen's title, not a generic "Back."

function syncNavbar(){
  const top = nav[nav.length - 1], prev = nav[nav.length - 2];
  title.textContent = SCREENS[top].title;
  if (prev){
    back.classList.add("show");
    backLabel.textContent = SCREENS[prev].title;    // "‹ Settings"
  } else back.classList.remove("show");             // root: nothing behind
}

One state object, one repaint

Every control — switch, slider, checkmark — writes to a single state. After any write, onStateChange() fans out: recolour the panel for Dark, scale labels for Text Size, dim for Brightness, recompute the nav values so a deep pick flows back to its parent row, and repaint the live preview. Because a chevron row's value is a function of state (stashed on the element so it can be recomputed in place), the Wi-Fi row instantly shows the network you picked two screens deep.

That's the whole pattern behind a hundred settings panes: inset groups of rows, a trailing accessory that tells you what a row does (switch = instant on/off, chevron = more behind it, checkmark = one of many, slider = a range), and a push stack that trades depth for a flat, uncluttered screen.

Flip Dark, drag Text Size, pick a network deep in a subpage and watch the value flow back — all live at: https://dev48v.infy.uk/design/day54-apple-settings.html

Yahtzee in 180 lines: how one counts[1..6] array scores all 13 categories

Devanshu Biswas — Tue, 04 Aug 2026 08:45:43 +0000

Yahtzee looks like thirteen unrelated scoring rules bolted together: sum your 1s, chase a full house, hunt a five-of-a-kind, settle for "chance." Sit down to build it and you find the opposite is true — every one of those rules is a single question asked of the same tiny array. There is no dice-physics engine, no rules table, no board object. Just five integers, a scorecard, and about 180 lines of vanilla JavaScript.

The whole game is a handful of variables

Five integers hold the dice faces, five booleans mark which you're keeping on the re-roll, a counter tracks how many of your three throws remain, and the scorecard is a plain object mapping each of thirteen box names to a number once scored or null while it's still open.

let dice     = [null,null,null,null,null];   // the five faces, 1..6
let held     = [false,false,false,false,false]; // which you keep on re-roll
let rollsLeft = 3;                            // three throws per turn
let scores    = {};                           // box → number, or null = open

A "roll" re-throws only the loose dice — that one if (!held[i]) is the entire hold-and-reroll gamble the game rests on:

function roll(){
  for (let i = 0; i < 5; i++)
    if (!held[i]) dice[i] = 1 + Math.floor(Math.random() * 6); // loose ones only
  rollsLeft--;
}

One helper collapses thirteen rules into almost nothing

Here is the trick. Walk the five dice once and tally how many showed each face into a seven-slot histogram (index 0 unused, 1–6 the faces). From that single array every category becomes one expression.

function counts(d){                    // histogram of the faces
  const c = [0,0,0,0,0,0,0];           // c[1]..c[6]
  for (const x of d) c[x]++;
  return c;
}
function sum(d){ return d.reduce((a,b) => a + b, 0); }
// dice [5,2,5,5,2] → counts [0,0,2,0,0,3,0] : three 5s, two 2s

Now every score reads straight out of counts. "Three of a kind" asks whether any face reaches 3; a straight is a run of consecutive non-zero slots; "sixes" is c[6] * 6:

function longestRun(c){                 // consecutive present faces
  let run = 0, best = 0;
  for (let f = 1; f <= 6; f++){ run = c[f] ? run + 1 : 0; best = Math.max(best, run); }
  return best;
}
function scoreFor(cat, d){
  const c = counts(d), total = sum(d);
  if (cat in FACE)             return c[FACE[cat]] * FACE[cat];
  if (cat === "threeKind")     return c.some(n => n >= 3) ? total : 0;
  if (cat === "fourKind")      return c.some(n => n >= 4) ? total : 0;
  if (cat === "fullHouse")     return c.includes(3) && c.includes(2) ? 25 : 0;
  if (cat === "smallStraight") return longestRun(c) >= 4 ? 30 : 0;
  if (cat === "largeStraight") return longestRun(c) >= 5 ? 40 : 0;
  if (cat === "yahtzee")       return c.some(n => n === 5) ? 50 : 0;
  if (cat === "chance")        return total;
}

Why the bonus lands at exactly 63

The famous +35 upper bonus is a single comparison — but the magic number is worth pausing on. If your six upper boxes together reach 63 or more, you collect the bonus:

function upperBonus(){
  return upperSubtotal() >= 63 ? 35 : 0;   // 63 = three of every face
}

Why 63? Because 63 is precisely three of each face: 3×1 + 3×2 + … + 3×6 = 3 × 21 = 63. Averaging three-of-a-kind across the top half is the break-even line the designers chose, and the whole bonus is one >= 63 test.

Because scoreFor is pure — dice in, number out — it doubles as the live green preview on every open box, and committing a box is three lines of bookkeeping. The grand total is just the upper subtotal plus the bonus plus the lower boxes, and since unscored boxes are null (counted as zero), the same total function runs live during play and as the final score.

That's the lesson: a pile of special-case rules often hides one small shared representation. Find the histogram, and Yahtzee scores itself.

Play the full game, step through the scoring engine, and copy the code block by block at the live build: https://dev48v.infy.uk/game/day54-yahtzee.html

A memory agent that remembers you after the process dies — short-term buffer + vector recall + compression, no vector DB

Devanshu Biswas — Mon, 03 Aug 2026 16:14:22 +0000

Project 5 of my "Agentic AI from Zero" series is a memory-enabled conversational agent — and the whole point is this: you tell it a fact in one process, kill the process, start a brand-new one, and it still knows.

No LangChain, no Pinecone, no paid embedding API. Just Python, a free 8B model (Llama 3.1 via NVIDIA NIM), and about 200 lines. Here is how the four pieces fit together.

1. Short-term buffer (verbatim, bounded)

The last few turns are kept word-for-word so the model has immediate conversational context. It's a simple deque with a token budget — recent turns are cheap and high-signal, so they stay raw.

2. Long-term vector recall (hand-rolled, no DB)

Older facts get embedded and stored for semantic recall. The embedder is deliberately dependency-free — a stable hashing embedder:

class HashingEmbedder:
    def embed(self, text):
        vec = [0.0] * self.dim
        for tok in text.lower().split():
            h = int(hashlib.md5(tok.encode()).hexdigest(), 16)
            vec[h % self.dim] += 1.0 if (h >> 8) & 1 else -1.0
        return l2_normalize(vec)   # cosine == dot product after this

Signed buckets keep it from collapsing to all-positive, L2-normalization turns cosine similarity into a plain dot product. Retrieval is argmax over stored vectors. Load-bearing recall, zero infrastructure.

3. Context compression (on overflow)

When the short-term buffer overflows, the agent doesn't just drop old turns — it asks the model to summarize them into one compact memory note, then stores that in long-term. In the real recorded run this fired twice in session 1: the raw turns "I'm allergic to peanuts" collapsed into memory #6 — "Devanshu is allergic to peanuts." — freeing ~80 tokens while keeping the fact.

4. Relevance scoring (inject only what matters)

Every user turn triggers a retrieval. Only memories above a similarity threshold get injected into the prompt — the rest are skipped, so the context stays lean:

[INJECT] #6 (0.408) Devanshu is allergic to peanuts.
[INJECT] #2 (0.354) ...
[skip]   #4 (0.121) ...

The cross-session proof

The memory store persists to a plain memory_store.json. The recorded run is two genuinely separate OS processes:

Session 1 (process A): tells the agent its name, an allergy, and a project detail; the buffer overflows; compression fires; process exits.
Session 2 (a fresh python run.py session2): asks "what am I allergic to?" → answers peanuts by recalling memory #6 at score 0.408. Asks about the project → recalls the Safar / Flutter + Supabase summary at 0.500.

The model is only ever used for two things: writing the compression summaries and the final answer. Storage, embedding, retrieval, and relevance scoring are all deterministic Python — which is exactly what makes the memory auditable instead of magic.

Real recorded transcript (with the compression events and the per-memory relevance scores) + full code:

Live walkthrough: https://dev48v.infy.uk/agentic/project5-memory.html
Repo: https://github.com/dev48v/agentic-ai-from-zero

Next up, Project 6: a human-in-the-loop approval agent that pauses for a human before doing anything risky.

From docker-compose to Kubernetes: 8 Spring Boot services, probes that gate rollout, and a test that reads the manifests

Devanshu Biswas — Mon, 03 Aug 2026 16:13:38 +0000

Day 43 of building OrderHub in public takes yesterday's docker compose up and turns it into a real Kubernetes deployment — 8 Spring Boot services, each with liveness and readiness probes wired to Spring Boot Actuator, plus a ConfigMap, a Secret, and a JUnit test that parses every manifest so the topology can't silently drift.

One Deployment + Service per app

Each of the 8 runnable services (config-server, eureka, gateway, order, inventory, payment, shipping, notification) gets a Deployment and a ClusterIP Service. The images are the ones built on Day 42 (orderhub/<svc>:0.1.0), the container port matches the EXPOSE, and the Service port lines up with it. Shared config comes from a ConfigMap (orderhub-config) and secrets from a Secret (orderhub-secrets) — consumed via envFrom, so no config is baked into an image.

Probes wired to Actuator — the important part

Kubernetes has three probes and they do different jobs:

readiness → /actuator/health/readiness — gates whether the Service sends traffic to the pod. Fails during startup or when a dependency is down, so no request hits a half-ready app.
liveness → /actuator/health/liveness — gates restarts. If the app deadlocks, Kubernetes kills and replaces it.
startup → shields a slow JVM cold start so liveness doesn't kill the pod before it's up.

Spring Boot exposes those two health groups only when you turn them on:

# activated ONLY under the k8s profile — local/compose behaviour is unchanged
spring:
  config:
    activate:
      on-profile: k8s
management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        readiness:
          include: readinessState,db
        liveness:
          include: livenessState

The whole k8s wiring is profile-gated (SPRING_PROFILES_ACTIVE=k8s set in the ConfigMap), so the local Docker Compose and the test suite stay byte-for-byte the same.

readinessProbe:
  httpGet: { path: /actuator/health/readiness, port: 8082 }
  initialDelaySeconds: 10
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /actuator/health/liveness, port: 8082 }
  periodSeconds: 10

Test the manifests, don't hope

Day 42 shipped a test that parsed docker-compose.yml; Day 43 does the same for Kubernetes. KubernetesManifestTest loads every YAML with SnakeYAML (no cluster needed) and asserts: each app has a Deployment + a Service, images and ports match, every Deployment declares liveness and readiness probes pointing at the actuator paths, and each one envFroms the ConfigMap and Secret. If someone adds a service and forgets a probe, the build goes red.

mvn clean test across all 8 modules: BUILD SUCCESS, 155 tests (up from 152 — the +3 are the manifest test).

The Secret holds only clearly-labelled base64 placeholders — real values come from the platform (a sealed secret / vault) at deploy time, never the repo.

Full manifests + the kill-a-pod self-heal walkthrough:

Next: Helm + Ingress, so the whole stack installs with one helm install.

Test-time compute: the two-line math behind majority vote, best-of-N, and "thinking longer"

Devanshu Biswas — Mon, 03 Aug 2026 16:12:56 +0000

A single sample from a language model is a noisy guess. On a hard question, one draw at temperature > 0 is right only part of the time. The whole idea behind test-time compute (inference-time scaling) is that you can trade extra compute at answer time for accuracy — without training a bigger model. Here's how the three main methods work, and the surprisingly simple math that decides where each one plateaus.

Model a question by its single-sample accuracy

Give each question a probability p that one sampled answer is correct. Easy questions have p near 1; genuinely hard ones sit below 0.5, so a single guess is more often wrong than right. That noise is exactly what test-time compute exploits: draw the answer N times instead of once and aggregate. N is your compute budget — cost and latency grow roughly linearly with it.

Self-consistency: majority vote

The cheapest aggregation (Wang et al., 2022): draw N samples and keep the most common answer. No verifier needed. If the correct answer out-votes the model's favourite wrong answer with probability p, majority accuracy is the binomial tail — the chance correct votes exceed N/2.

from collections import Counter
from math import comb

def majority_vote(samples):
    return Counter(samples).most_common(1)[0][0]

def majority_accuracy(p, n):
    acc = 0.0
    for c in range(n + 1):                 # c = number of correct votes
        pm = comb(n, c) * p**c * (1 - p)**(n - c)
        if   2 * c >  n: acc += pm         # correct wins
        elif 2 * c == n: acc += 0.5 * pm   # tie, break 50/50
    return acc

This rises toward 1 when p > 0.5 — but toward 0 when p < 0.5. Voting only amplifies whatever the model believes most often, so on hard questions it can confidently lock in the wrong answer. That's the fundamental ceiling: majority vote can never beat the fraction of questions the model is right about on average.

Best-of-N: let a verifier keep the best sample

If you have a verifier — a reward model, a unit test, a proof checker — you don't need the majority. You need one good sample the verifier can recognize. With a perfect verifier, accuracy is pass@N:

def best_of_n(samples, verifier):
    return max(samples, key=verifier)      # keep the highest-scored

def best_of_n_accuracy(p, n):
    return 1 - (1 - p) ** n                # P(at least one of n correct)

1 - (1 - p)^N climbs toward 1 for any p > 0 — it fixes even the hard questions majority vote gives up on. For p = 0.42: N=1 gives 0.42, N=8 gives 0.98. One correct sample is a needle in the haystack, and the verifier's job is to find it. The catch: a real verifier is imperfect, and its accuracy caps the gain well below the theoretical pass@N.

Both curves are concave — diminishing returns

Sweep N and average over your question set and you get the key picture: the first few extra samples buy a lot of accuracy, later ones almost nothing. Plotting N on a log axis makes the tail visibly flatten. Majority vote plateaus below 100%; best-of-N keeps climbing. The gap between the two plateaus is the value of having a verifier.

"Think longer" — sequential compute (o1 / DeepSeek-R1)

Sampling-and-voting is parallel test-time compute. o1 and DeepSeek-R1 add sequential compute: one long chain of thought that explores approaches, self-checks, and backtracks before committing — thousands of hidden reasoning tokens. It's implicit search inside a single sample, exposed as a simple effort knob rather than an N-of-samples loop.

# OpenAI o1 / o3 expose it as reasoning_effort:
client.chat.completions.create(model="o3", reasoning_effort="high", ...)
# DeepSeek-R1 emits a long <think>...</think> trace before the final answer.
# More thinking tokens = more test-time compute = better answers on hard problems.

When inference compute beats a bigger model

DeepMind made the tradeoff concrete (Snell et al., 2024): on many problems, giving a small model a big test-time budget matches a much larger model answering once — up to a point. A bigger base model raises every sample's p, shifting the whole curve up. So the real question isn't "bigger or think-harder" — it's how to split a fixed budget between pretraining and inference, and compute-optimal scaling even picks N per question by difficulty. This is why labs now scale inference compute, not just parameters.

Slide the compute budget from 1 to 64 samples and watch the accuracy curve climb and level off — self-consistency vs best-of-N, on a deterministic in-browser model: https://dev48v.infy.uk/ai/days/day53-test-time-compute.html