DEV Community: Mohanad Abu-Nayla

Spec-driven development with AI agents: constitutions, checkpoints, and handoffs

Mohanad Abu-Nayla — Sat, 01 Aug 2026 13:09:19 +0000

AI coding agents are strangely lopsided. Hand one a well-scoped task and it writes better code, faster, than I do. Hand it a large feature, the kind that takes days and touches a dozen files across several layers, and the failure mode changes. The context window fills up and quality quietly degrades (AKA context rot). A fresh session, missing the history, confidently invents an API that never existed. And even when the code is good, you're left staring at a 3,000-line diff you can't meaningfully review.

Those failures are not solved by the model alone. They are largely a process problem.

The backbone I work inside isn't mine and isn't novel: a constitution, specs, and deliberately stabilized plans. It's spec-driven development, which has been around in various shapes for years. What I added from running it day to day were two operational mechanisms that made it survive long features: explicit checkpoints and structured handoffs.

The mental model I use: treat the agent like a brilliant but forgetful senior engineer. Sharp in the moment, no long-term memory, occasionally overconfident. You wouldn't hand that person a two-week feature and walk away. You'd give them the ground rules, a spec, and a plan, then review their work in slices. That's the whole idea.

The loop: constitution → spec → plan → implement

Standard stuff, so I'll keep it short.

Constitution. A short, binding document of principles the project may never violate: dependency direction, the testing bar, security rules, "spec before code." It's the highest authority; when a request conflicts with it, the conflict gets surfaced, not silently coded around.
Spec. WHAT and WHY only. User value, functional requirements, acceptance criteria. No tech, no code.
Plan. HOW. Architecture, contracts, and an ordered list of tiny, test-first tasks, each justified against the constitution. Crucially, the plan is frozen once approved: it's the source of truth, and the agent doesn't get to quietly rewrite it mid-flight.

That gives the agent rails. But rails aren't enough for long work.

Checkpoints: break the feature into slices you can actually review

A plan might have 25 tasks. Executing all 25 in one shot lands you right back at the unreviewable-diff problem. So I slice the plan into checkpoints, review-sized groups of tasks, and the agent works one checkpoint at a time, then stops.

At each stop it has to: get the gate green (lint, types, architecture checks, tests), tick off the tasks, write a short note recording every deviation from the plan, and hand off for review.

The review itself is two-stage. The coding agent doesn't send its work straight to me. It writes a brief for a separate reviewer agent, stating what changed and why. I read that brief first, because what the coder chooses to explain (and what it leaves out) tells me where to look. Then the reviewer agent goes to work on the diff, and I read its findings against my own. The slice is a few hundred lines, one coherent unit, so this is actually tractable. Approval is mine either way; nothing continues on an agent's say-so.

Two things fall out of this:

Complexity stays bounded. I'm never reviewing "a feature"; I'm reviewing a checkpoint. Drift and bad assumptions get caught at checkpoint 2, not discovered at checkpoint 7.
Verification is real. Because each checkpoint ends on a green gate with evidence, "done" means something. I run the commands and read the output before I believe it.

The idea of pausing between steps isn't new. What I've made repeatable is the artifact: a shared checkpoint tracker that both the human and the agent write to, so stopping-for-review is the default, not an afterthought.

Handoffs: kill context rot and hallucination

Long features outlive a single agent session. The context window fills, or I come back the next day, or I switch models. A fresh session starts with zero memory of the last three days of decisions. Left alone, it will re-litigate settled choices or hallucinate the current state.

I learned this the tedious way. Fresh sessions reintroduced an API we'd deleted two days earlier, reopened architectural decisions that were already settled, quietly dropped an invariant because the reasoning behind it had scrolled out of the window. After the third time reviewing the same decision, I stopped waiting for it to happen.

So I don't wait for a session to degrade. I retire it on purpose and have it hand the work to a fresh agent on my terms.

Every feature carries a HANDOFF document, a living page that tells the next session exactly what it needs and nothing it doesn't:

Orientation: which docs to read, in order (constitution, spec, the frozen plan, the checkpoint tracker).
The hard rules it must not break.
Current state: branch, commit, which checkpoints are done.
Decisions already made, with the *why*, so they don't get reopened.
The next task, and the items consciously deferred.

A fresh agent reads the handoff and the frozen plan, and it's grounded again. No rot, no invention, because the ground truth lives in the documents rather than in a context window that's about to overflow.

Six sessions, one document

On a recent AI-system backend feature, six different agent sessions touched the code across its seven checkpoints. Each started from a fresh context, some were different models, and none of them needed me to re-explain a decision that had already been made. I read every handoff as it was written, and tightened the template whenever a fresh session asked a question the document should have already answered.

Tiering models by leverage

One more lever: not every step deserves the same horsepower. Planning and code review are the highest-leverage, hardest-to-reverse steps, and a bad plan poisons everything downstream. Implementation against a frozen, detailed plan is comparatively mechanical.

So I tier:

Top reasoning tier for plans and the reviewer agent (currently Fable 5), where judgment compounds.
A tier down for the actual coding (Opus 5, or Sonnet 5 on the mechanical stretches), where the plan and the checkpoint gate are already the guardrails.

The human review at each checkpoint is the safety net that makes the cheaper coding pass safe. It's a nice cost/quality trade: spend where thinking matters, save where the rails are already in place.

The whole thing, on five lines

Request → Constitution → Spec → Frozen Plan
    │
    └─> ( Checkpoint → Gate → Review ) × N
              ↑                    │
              └───── HANDOFF ◀─────┘

What it buys me

Reviewable diffs. Always a checkpoint, never a monolith.
Grounded agents. The handoff plus the frozen plan beat context rot and hallucination.
Real verification. A green gate and evidence at every stop.
Cost control. Heavy models where they earn it, a tier down where the rails already hold.
Continuity. The feature survives any single session ending.

The surprising lesson: as agents get better at writing code, the bottleneck moves to keeping them grounded and keeping yourself in the loop. A constitution, checkpoints, and handoffs are how I do both, and honestly, they've made me a more disciplined engineer even on the parts I write by hand.

I've templated the whole thing (constitution, spec/plan templates, the checkpoint tracker, and the handoff doc) as a starter kit I drop into new projects: hannody/spec-kit. It's language- and domain-agnostic; fork it and tune the constitution to your stack.

On-Demand Namespace Error Log Watcher in a Locked-Down Kubernetes Environment

Mohanad Abu-Nayla — Sat, 01 Aug 2026 13:03:37 +0000

During a large-scale stress test, the last thing you want is to be hunting errors across 150 pods manually.

So I built an on-demand error log watcher for a complex Kubernetes namespace under real-world constraints, without touching anything outside the namespace, and without installing a single binary on the server.

What we are building: a Kubernetes Deployment, scaled to zero by default, that runs Stern as a container inside the cluster. It watches all pods in a target namespace, filters output to errors only, and can be started or stopped with a single kubectl scale command. The entire setup RBAC, ServiceAccount, and Deployment lives within the targeted namespace it monitors and leaves no trace when removed.

The Environment

The namespace in question that I was stress-testing, ironlight-ai, runs on a private GPU cluster. The infrastructure is completely private cloud environment, Node-level access is restricted, and installing tools server-wide is not permitted — and frankly, not something I would do even if I could. Any solution I introduce must be contained, reversible, and have zero footprint beyond its own namespace.

The namespace itself is not small. It runs:

On-Prem LLM inference services
Vision-Language Model (VLM) pipelines
Classical ML classification workers
Deep learning document processors
OCR pipelines with pre- and post-processing stages
Backend orchestration services
File conversion, image processing, and rule evaluation workers

At peak stress test load, this namespace runs around 150 pod replicas. The goal of the stress tests is to burst large numbers of concurrent users through the full pipeline to expose weak points in the software stack and infrastructure before they become production incidents.

The Observability Gap

I had already set up a full Grafana observability stack for this environment: Alloy for log collection, Prometheus for metrics, and Loki for log aggregation, all configured and functional. However, during high volume stress tests, log visibility for this namespace became inconsistent, but functional for other namespaces and services. Rather than interrupt the test to debug the collection pipeline, I needed a solution I could reach for immediately; something that would give me error visibility in minutes, not hours.

The observability stack will be revisited and fixed properly. Since the stress-test uncovered this gap in it. However, this stern tool exists for the time between now and then and as a backup for future incidents.

Why Not Just Install Stern the Normal Way

Stern is the obvious tool for this. It tails logs across multiple pods and containers simultaneously, supports regex filtering, and produces clean, readable output.

The challenge is getting it running in this environment.

I tried the straightforward approaches first:

Local install with SSH tunnel: Installed Stern on my local machine and attempted to tunnel access through SSH and VPN to reach the cluster API. It did not work. The network topology between my machine and the cluster API server under VPN made this unreliable.

Python helper script: I wrote a helper to establish and maintain the connection programmatically. It did not work reliably either. At this point I had spent more time on the tooling problem than on the actual debugging I needed to do.

Following the official Kubernetes and GitHub documentation: I checked the standard installation paths. They assumed a level of access to the nodes and the cluster that I simply do not have them readily available in this environment.

I also considered the Kubernetes native alternatives before settling on this approach:

kubectl logs with label selectors: Works well for a handful of pods. At 150 replicas it becomes unwieldy, and it offers no log filtering — every line from every container comes through regardless of severity.

Parallel kubectl logs with xargs: A common pattern for watching many pods at once looks like this:

kubectl get pods -n ironlight-ai -o name \
  | xargs -n1 -P 10 kubectl logs -n ironlight-ai --all-containers --tail=100 -f

This is clever, but it has the same problems as the label selector approach: no filtering, pure log volume from 150 replicas. It also batches with -P 10, so at 150 pods you are waiting for processes to cycle rather than watching the full namespace simultaneously.

Ephemeral debug pod: A kubectl debug or temporary utility pod could run tools interactively, I have tried this approach, for another scenarios, e.g. use 'curl' commands for connectivity checks as most of my microservices images are minimal images optimized for production with no extra tooling, but it still requires either installing Stern inside the pod at runtime or building a custom image — both of which reintroduce the installation problem in a different form.

Fix the Loki ingestion issue first: The correct long-term answer. Not the right call during an active stress test.

I stopped pursuing these paths when I recognised a pattern: every approach either required access I did not have, depended on local connectivity I could not trust, or introduced more complexity than the problem warranted.

That is the wrong direction. The solution should make my day easier, not more complicated.

The Design Criteria

Before writing a single line of YAML, I defined what an acceptable solution looked like:

Namespace-contained. No resources, no side effects, nothing outside ironlight-ai.
No server-wide installation. Nothing touches the nodes or the cluster outside my namespace.
On-demand only. It should run when I need it and consume zero cluster resources when I do not.
Minimal RBAC surface. Only the permissions strictly necessary — nothing broader.
Readable output. During a stress test I need to see errors fast. The output must be clean.
Reversible. I can remove the entire thing with one command and leave no trace.

The solution that met all of these criteria: run Stern as a container inside the cluster itself, deployed as a Kubernetes Deployment scaled to zero by default.

Why a Deployment Scaled to Zero

A standalone Pod would work, but it has no scaling controller. Once created it runs until deleted.

A Deployment with replicas: 0 solves this cleanly:

The configuration lives in the cluster permanently
No pod runs, no log streams are open, no API calls are made
When I need it, I scale to 1 with a single command
When I am done, I scale back to 0
The Deployment itself stays, ready for next time

This is the operational model I wanted: always available, never running unless explicitly started.

RBAC Design

Because Stern runs as a pod inside the cluster, it needs permission to read pod logs. I created a ServiceAccount, Role, and RoleBinding scoped exclusively to the ironlight-ai namespace.

Note the use of Role, not ClusterRole. A ClusterRole would grant permissions across the entire cluster. A namespace-scoped Role limits access to only the namespace it is defined in which is exactly the boundary I wanted.

This also means the entire setup must be deployed into the same namespace you want to monitor. The ServiceAccount, Role, RoleBinding, and Deployment all carry namespace: ironlight-ai. If you are adapting this for your own use, replace every occurrence of ironlight-ai with your target namespace. Stern will then watch only that namespace, with permissions that cannot reach anything outside it.

The Role grants only what Stern needs:

resources:
- pods
- pods/log

verbs:
- get
- watch
- list

Nothing else. No cluster-wide permissions. No write access.

Handling logging for 150 Replicas

Stern has a default limit of 50 concurrent log streams. With 150 pod replicas in ironlight-ai, it exits immediately:

Error: stern reached the maximum number of log requests (50), use --max-log-requests to increase the limit

The fix is straightforward:

--max-log-requests 200 # HALT! Higher numbers could risk throttling or stressing the Kubernetes API server.

This gives enough headroom for the full namespace at peak replica count.

Filtering for What Matters

During a stress test, I do not want to see every log line from 150 pods. I want errors, and only errors.

Different services in this namespace log at different severity levels and with different casing conventions:

ERROR / Error / error
CRITICAL / Critical / critical
FATAL / Fatal / fatal

A case-insensitive regex covers all of them:

(?i)error|critical|fatal

I also exclude the Stern pod itself from its own log watch to avoid a feedback loop:

--exclude-pod stern.*

And I use --tail 0 so Stern does not replay historical logs on startup — only new lines from the moment it starts.

Output Template

The default Stern output includes full pod names and container metadata. That is useful in some contexts, but during a stress test it is noise due to replicas.

I used a minimal Go template:

{{printf "%s %s\n" .ContainerName .Message}}

This produces:

backend 2026-05-22 13:23:25,094 ERROR streams.document.views 1619 ...
ocr 2026-05-22 13:23:31,201 CRITICAL pipeline.processor 88 ...

Service name, timestamp, severity, location, message. Everything I need, nothing I do not.

The Full YAML 🫡

apiVersion: v1
kind: ServiceAccount
metadata:
  name: stern
  namespace: ironlight-ai
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: stern
  namespace: ironlight-ai
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: stern
  namespace: ironlight-ai
subjects:
- kind: ServiceAccount
  name: stern
roleRef:
  kind: Role
  name: stern
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stern
  namespace: ironlight-ai
spec:
  replicas: 0
  selector:
    matchLabels:
      app: stern
  template:
    metadata:
      labels:
        app: stern
    spec:
      serviceAccountName: stern
      containers:
      - name: stern
        image: ghcr.io/stern/stern:v1.34.0
        args:
        - "."
        - "-n"
        - "ironlight-ai"
        - "--include"
        - "(?i)error|critical|fatal"
        - "--exclude-pod"
        - "stern.*"
        - "--tail"
        - "0"
        - "--max-log-requests"
        - "200"
        - "--only-log-lines"
        - "--template"
        - '{{printf "%s %s\n" .ContainerName .Message}}'

Applying and Using It

Before applying: make sure every namespace: field in the YAML matches the namespace you want to monitor. All resources, the ServiceAccount, Role, RoleBinding, and Deployment must live in the same namespace as the pods you are watching. The image ghcr.io/stern/stern:v1.34.0 is publicly available; this solution was tested against this version at time of writing. In a restricted deployment environment, substitute your internal registry path.

Apply once:

kubectl apply -f stern-error-watcher.yaml

Start watching:

kubectl scale deployment -n ironlight-ai stern --replicas=1
kubectl logs -n ironlight-ai -f deployment/stern

Stop:

kubectl scale deployment -n ironlight-ai stern --replicas=0

Check status:

kubectl get deployment -n ironlight-ai stern
kubectl get pods -n ironlight-ai -l app=stern

What This Is and What It Is Not

This is not a replacement for a properly functioning Grafana/Loki observability stack. That stack exists, it covers the full environment, and it will be fixed properly.

This is a deliberate tool built for a specific scenario: active stress testing, high replica counts, and a need for immediate error visibility scoped to one namespace.

It is:

Entirely self-contained within the namespace
Zero-cost when not running
Removable without side effects
Scoped to the minimum necessary RBAC permissions
Fast to start and fast to stop

In a constrained environment, the right solution is not always the one the documentation recommends. Sometimes it is the one that fits cleanly inside the boundaries you are given 🙄 and leaves no mess behind.