DEV Community: Akash Pal

Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework

Akash Pal — Tue, 11 Aug 2026 18:44:00 +0000

Part 7 (final) of a series building a support-ticket agent with no framework. Previous: Part 6 (observability). Repo: github.com/akash-pal/agent-from-scratch

The other six parts described the finished design. This one is about what "finished" actually took — the real bugs the eval set caught, and the two questions every agent build eventually has to answer honestly: do you need more than one agent, and do you need a framework. Full detail on everything below: docs/iteration-log.md.

The iteration log, condensed

1. Exact trajectory matching was the wrong check. First eval run: 12/21 passed. Most failures were the agent correctly sending a confirmation email where the eval only expected a lookup — correct behavior, wrong assertion. Fix: switched the harness from exact-array equality to ordered-subsequence matching (every expected tool must appear, in order; extra steps in between are fine). Still catches a missing, reordered, or wrong tool. Stops false-failing on benign non-determinism.

2. No retry/backoff meant a transient error crashed the whole run. A 503 — model overloaded on case 1 took down the entire eval harness. Fixed with exponential backoff on 429/503 specifically, plus inter-case pacing to stay under free-tier rate limits.

3. A -latest model alias silently rolled onto a stricter quota. gemini-flash-latest worked, then started failing with a 20 requests/day cap after quietly resolving to a newer model. Fixed by pinning an explicit model version instead of an alias, after checking the provider's live usage dashboard for actual quota — 25x more headroom on the pinned model. The takeaway generalizes past this one provider: "latest" aliases optimize for capability, not quota stability, and what they resolve to changes over time without your code changing at all.

4. A testing artifact that looked like a real bug. Piping multiple answers into the interactive CLI via printf "a\nb\n" | npm run agent intermittently hung after the first prompt. Root cause: a Node.js readline/promises quirk with quickly-closing piped stdin — confirmed as a testing artifact, not a real bug, by replaying the same input through an actual pseudo-terminal, where it worked every time. Fixed anyway by reusing a single shared readline.Interface instead of opening/closing one per prompt — more correct regardless of the original symptom.

5. The model claimed a refund was proposed without ever proposing one. The big one — covered in full in Part 5. Two eval cases showed outcome: refund_proposed with a trajectory that never called issue_refund. Fixed at two layers: an explicit prompt line, and — the layer that actually matters — a code-level check (enforceOutcomeIntegrity) that verifies a real confirmation_id exists in state before trusting the model's own claim.

The honest note attached to that fifth finding in the actual log: it was verified with a single targeted re-run, not a full eval pass, specifically to conserve free-tier API quota. That gap is left visible in the repo on purpose — claiming a full pass without having run it would be the same category of mistake as finding #5 itself.

Single agent, not multi-agent — and why that's arithmetic, not instinct

Multi-agent (a coordinator dispatching to specialist workers) is a cost decision, not an architectural preference. A coordinator + worker split costs roughly 3–8x a single agent's tokens and latency per run — workers see fewer tools each, which sharpens tool selection, but you're paying for that in every single run, not just the hard ones.

Three signals should all be true before paying that multiplier: the task genuinely splits into specialized roles that each benefit from a sharper, narrower prompt; subtasks can run in parallel for a real speed win, not just cleaner code; and volume × accuracy improvement actually outweighs the cost multiplier — checked with real numbers, not intuition.

At low volume, the math tends to favor a single agent even with a real accuracy edge from splitting — the multiplier isn't earned back. At high volume, the same accuracy edge can save real money, because escalation-cost reduction scales with volume in a way the fixed multiplier doesn't. This repo's own agent stays single, deliberately: five tools is well under the point where an agent starts confusing tool names, and nothing about the ticket volume this reference build targets justifies paying a multiplier for a split that would mostly just look tidier.

Framework selection — now that you've seen the raw version

Having built the loop by hand, the actual question a framework answers becomes concrete instead of abstract: does this problem need multi-agent orchestration, streaming UI wiring, an integrated tracing ecosystem, or shared abstraction across a team? If none of those apply, the ~100-line loop this series built is the production version, not a placeholder for one.

Framework	Best for	Trade-off
Raw SDK (this series)	Learning the loop, simple single-agent tools	Minimal abstraction, provider-locked, most control
LangGraph	Complex state, multi-agent, widest enterprise adoption	More setup cost; strongest observability ecosystem
Mastra	TypeScript-native teams	Smaller ecosystem, natural fit for a JS/Next.js background
Vercel AI SDK	Streaming UI tightly coupled to a Next.js frontend	Less suited to long-running, durable multi-step workflows
CrewAI	Standing up role-based multi-agent crews quickly	Opinionated structure
Google ADK	Teams committed to GCP/Gemini	Less model-agnostic than LangGraph
LlamaIndex	RAG-heavy use cases, large document corpora	Less general-purpose outside retrieval

Where this leaves the repo

Everything in this series is a real, committed artifact — not a cleaned-up retelling. The README has the full architecture diagrams, quickstart, and real example runs (screenshots included, not staged). AGENTS.md documents the hard constraints for anyone — human or AI coding agent — extending this repo later: no framework, mock data only, gated tools stay gated, guardrails live in code.

It's explicitly a reference build, not production software — mock data, a single free-tier model, no persistence, no auth. If you take one thing from the whole series, take the build order, not the code: pin the use case, write tool contracts as specs, build the eval set before the agent exists, write the smallest loop that could work, iterate against real failures, gate the consequential actions. The framework question comes last, not first — and by the time you're actually equipped to answer it, you often don't need to.

Repo: github.com/akash-pal/agent-from-scratch

Series index: Part 1 (what makes something an agent) · Part 2 (use case & tool contracts) · Part 3 (the eval set) · Part 4 (the raw loop) · Part 5 (guardrails) · Part 6 (observability) · Part 7 (this post).

Part 6: Observability for AI Agents: Tracing, Metrics, and Drift

Akash Pal — Tue, 11 Aug 2026 18:43:58 +0000

Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 (guardrails). Repo: github.com/akash-pal/agent-from-scratch

"Run the eval set" and "is this agent healthy right now" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step.

The minimum trace payload

Every tool call in this build logs a structured record — src/trace.ts:

export interface TraceStep {
  trace_id: string;
  step_id: number;
  tool_name: string;
  args_hash: string;        // hashed, never raw args
  duration_ms: number;
  result_summary: string;
  model: string;
  token_usage: { input: number; output: number };
}

Two details here that look small and aren't:

args_hash, not raw args. This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments (for debugging idempotency, for instance) without ever persisting the actual values:

export function hashArgs(args: Record<string, unknown>): string {
  return "sha256:" + createHash("sha256").update(JSON.stringify(args)).digest("hex").slice(0, 8);
}

result_summary, truncated. Full tool results can be large (a kb_search returning full article bodies, for instance) — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. summarizeResult takes the first few fields and truncates long values:

const MAX_FIELD_LEN = 70;
export function summarizeResult(result: Record<string, unknown>): string {
  const entries = Object.entries(result).slice(0, 4);
  return entries.map(([k, v]) => `${k}=${truncate(JSON.stringify(v))}`).join("  ");
}

Readable in a terminal, structured everywhere else

The trace log doubles as CLI output — this build's whole point is being inspectable, so watching an agent run in real time matters. Early on, that meant a raw JSON blob per line, which is technically complete and practically unreadable. The fix was a small formatting pass, not a new logging system:

export function logTrace(step: TraceStep): void {
  const timing = `${step.duration_ms}ms, ${step.token_usage.input}→${step.token_usage.output} tok`;
  console.log(`  ${DIM}[${step.step_id}]${RESET} ${CYAN}${step.tool_name}${RESET} ${DIM}(${timing})${RESET}`);
  console.log(`      ${step.result_summary}`);
}

Output in an actual terminal:

  [1] order_lookup (0ms, 1077→23 tok)
      order_id="ord_1005"  status="processing"  items=[...]  total_usd=45
  [2] kb_search (1ms, 1268→20 tok)
      articles=[...]  relevance_scores=[0.48,0.24,0.24]

Colors auto-disable when stdout isn't a real TTY (process.stdout.isTTY), so piping this to a file or a CI log doesn't leave you with literal escape-code garbage — small thing, but the kind of small thing that makes the difference between a trace log people actually read and one they ignore.

Three production monitoring layers

Trace-per-step is the foundation, but three distinct layers sit on top of it, each answering a different question:

Trace logging — every run, every tool call: input, response, latency, cost. This is what you already have from the payload above.
Online metrics — aggregate numbers over time: success rate, escalation rate, average tool calls per run, average token cost per run. This is where you'd notice, for instance, that escalation rate crept from 8% to 15% over a week.
Drift detection — compare those metrics week over week. This one's easy to skip and shouldn't be: model provider updates cause silent regressions with zero code changes on your side. The agent that scored 95% on your eval set last month can start failing differently this month because the underlying model changed, not because anything you wrote did.

This repo doesn't implement online metrics or drift detection — it's a CLI reference build with no persistent request volume to aggregate — but the trace payload is written specifically so those layers could be built on top of it without changing the tracing code itself. That's the actual design goal: the minimum payload isn't "the metrics you need now," it's "the raw material any metrics system would need later."

Why this matters for debugging, not just monitoring

Go back to Part 3's eval failure:

[FAIL] hard_04 (hard)
    - trajectory: expected [order_lookup, refund_eligibility, issue_refund] as a subsequence, got [order_lookup, refund_eligibility]

That failure message exists because of tracing, full stop. Without a structured, step-by-step record of what tools got called, "the eval failed" would be all you'd know — not why. The actual bug behind that failure (Part 5's phantom refund proposal) was findable specifically because the trajectory was visible, not just the final pass/fail.

What's next

Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework → closes the series: the full iteration log — every real bug found running this agent against the eval set, what fixed each one, and when you'd actually reach for a framework instead of this raw loop.

Repo: github.com/akash-pal/agent-from-scratch

Part 5: Guardrails That Live in Code, Not the Prompt

Akash Pal — Tue, 11 Aug 2026 18:43:53 +0000

Part 5 of a series building a support-ticket agent with no framework. Previous: Part 4 (the loop). Repo: github.com/akash-pal/agent-from-scratch

Here's the finding this whole article is built around: partway through eval iteration, the agent started reporting that a refund had been proposed — a clean, plausible-sounding message — without ever having called the tool that proposes refunds. No approval was ever requested. No confirmation existed. The model just said it happened.

That's the failure mode this part is about, and the fix is the actual argument for why guardrails belong in code, not in prompt text alone.

Policy as code

src/policy.ts is a plain data object — an allowlist, an approval list, rate limits, and regex patterns — checked by the agent loop, not asked of the model:

export const policy: Policy = {
  allowTools: ["order_lookup", "refund_eligibility", "issue_refund", "kb_search", "send_email"],
  requireApprovalFor: ["issue_refund", "send_email"],
  rateLimits: { maxToolCallsPerRun: 8, maxCostPerRunUsd: 0.3 },
  autoEscalatePatterns: {
    legal_threat: /\b(lawyer|attorney|sue|legal action|better business bureau|\bbbb\b)\b/i,
    fraud_flag: /\b(fraud|unauthorized|without my permission|didn't authorize|stolen card)\b/i,
    duplicate_ticket: /\b(already submitted|second ticket|duplicate ticket|already reported)\b/i,
  },
};

Every one of these is enforced outside the LLM's control. The model can't talk its way past requireApprovalFor — the loop checks it before executing the tool, full stop. The autoEscalatePatterns regexes run against the raw ticket text before the model is even called (this is that expected_trajectory: [] behavior from Part 3's edge-case bucket) — a legal threat or fraud flag never reaches the LLM at all, straight to a human queue.

Three human-review patterns

Not every consequential action needs the same review pattern. This build uses pre-action approval — human approves before execution — for issue_refund and send_email, because both are rare, high-stakes, and hard to reverse. Two other patterns exist for different risk profiles and are worth knowing even though this particular agent doesn't need them: sampling review (agent acts autonomously, a random 10–20% gets human QA — appropriate for high-frequency, low-individual-risk actions) and confidence routing (only route to a human when the model's self-reported confidence is genuinely calibrated — risky if that confidence isn't actually reliable, which it usually isn't by default).

The gate itself, from src/approval.ts:

export const cliApproval: ApprovalFn = async (tool, args) => {
  const answer = await getRl().question(
    `\n[APPROVAL REQUIRED] ${tool}(${JSON.stringify(args)})\nApprove? (y/n) `,
  );
  return answer.trim().toLowerCase().startsWith("y");
};

And how the loop applies it before any gated tool runs (from agent.ts):

} else if (requiresApproval(toolName) && !(await approvalFn(toolName, args))) {
  resultPayload = { error: "rejected_by_human_approval" };
  isError = true;
}

If a human says no, the tool never executes — the model gets an error result back and has to handle that, same as any other tool failure.

The bug: a claim without a fact behind it

Here's what actually happened. Two eval cases (hard_04, hard_06) came back reporting outcome: refund_proposed, but their tool-call trajectory was only [order_lookup, refund_eligibility] — issue_refund was never in the list. The model had generated a final message starting with REFUND_PROPOSED: directly, skipping the tool call that's supposed to cause that outcome.

Why this matters: the outcome parser trusted the model's own REFUND_PROPOSED: prefix as ground truth. Nothing checked that the claim corresponded to an actual, approved, executed tool call. A "propose a refund" statement is exactly the kind of business-critical, consequential claim that shouldn't rest entirely on the model choosing to comply with a prompt instruction — and here, it was.

The fix — two layers, not one

Layer one, prompt: add an explicit line the model can't miss (already shown in Part 4's system prompt) — "never state REFUND_PROPOSED unless you actually called issue_refund and it succeeded in this run."

Layer two, code — the one that actually matters:

// Guardrail enforced in code, not just the prompt: a model can claim
// REFUND_PROPOSED in text without ever having called issue_refund.
function enforceOutcomeIntegrity(
  outcome: { outcome: AgentOutcome; finalText: string },
  state: AgentState,
): { outcome: AgentOutcome; finalText: string } {
  if (outcome.outcome === "refund_proposed" && !state.artifacts.refund_confirmation_id) {
    return {
      outcome: "escalated",
      finalText: `ESCALATED: model claimed REFUND_PROPOSED without ever calling issue_refund (policy violation) — original: "${outcome.finalText}"`,
    };
  }
  return outcome;
}

state.artifacts.refund_confirmation_id is only ever set in one place — when issue_refund actually succeeds (src/memory.ts):

export function recordStep(state: AgentState, step: TrajectoryStep, result: Record<string, unknown>) {
  state.conversation.trajectory.push(step);
  if (step.tool_name === "issue_refund" && typeof result.confirmation_id === "string") {
    state.artifacts.refund_confirmation_id = result.confirmation_id;
  }
  return state;
}

So enforceOutcomeIntegrity isn't trusting the model's words at all — it's checking a fact that can only exist if the gated, approved tool call actually happened. If the model claims REFUND_PROPOSED without that fact present, the run gets downgraded to escalated, regardless of how convincing the model's text sounded.

This is the actual lesson, stated plainly: any claim your agent makes about a consequential action should be verifiable from state the code controls, not trusted from text the model generated. The prompt fix alone would probably have reduced the frequency of this bug. It wouldn't have made it impossible. The code guardrail does.

What's next

Part 6: Observability for AI Agents: Tracing, Metrics, and Drift → covers observability — the trace payload this build logs on every single step, and why "run the eval set" isn't the same question as "is this healthy in production."

Repo: github.com/akash-pal/agent-from-scratch

Part 4: The Raw ReAct Loop: ~100 Lines, No Framework

Akash Pal — Tue, 11 Aug 2026 18:43:50 +0000

Part 4 of a series building a support-ticket agent with no framework. Previous: Part 3 (the eval set). Repo: github.com/akash-pal/agent-from-scratch

This is the part everyone reaches for a framework to skip. Here's the argument for not doing that, at least the first time: if you can't explain what your agent loop does in plain English, no framework is going to fix that — it's just going to make the loop harder to see.

Here's src/agent.ts, trimmed to the actual loop:

export async function runAgent(
  ticket: Ticket,
  customer: Customer | null,
  approvalFn: ApprovalFn,
  maxSteps = 8,
): Promise<AgentResult> {
  const state = initState(ticket, customer);

  // Guardrail check happens BEFORE any model call — see Part 5.
  const escalatePattern = matchAutoEscalate(`${ticket.subject} ${ticket.body}`);
  if (escalatePattern) {
    return { outcome: "escalated", finalText: `ESCALATED: auto-escalated — "${escalatePattern}"`, state };
  }

  const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });
  const contents: Content[] = [{ role: "user", parts: [{ text: ticketToUserMessage(ticket) }] }];

  for (let step = 0; step < maxSteps; step++) {
    const response = await withRetry(() =>
      ai.models.generateContent({
        model: MODEL,
        contents,
        config: { systemInstruction: buildSystemPrompt(COMPANY), tools: [{ functionDeclarations }] },
      }),
    );

    const calls = response.functionCalls ?? [];
    if (calls.length === 0) {
      // No tool call — the model produced a final answer. Done.
      const text = (response.text ?? "").trim();
      return { ...enforceOutcomeIntegrity(parseOutcome(text), state), state };
    }

    // Otherwise: execute the requested tool(s), feed results back, loop again.
    contents.push({ role: "model", parts: response.candidates?.[0]?.content?.parts ?? [] });
    const responseParts = [];
    for (const call of calls) {
      const result = await executeToolWithGuardrails(call, state, approvalFn); // Part 5
      responseParts.push({ functionResponse: { name: call.name, response: result } });
    }
    contents.push({ role: "user", parts: responseParts });
  }

  return { outcome: "escalated", finalText: `ESCALATED: max_steps_exceeded (${maxSteps} steps)`, state };
}

(The real file inlines tool execution, approval gating, and trace logging directly in the loop rather than a separate function — shown split out here for readability. Full file on GitHub.)

That's the whole thing: call the model, check if it wants a tool, if yes run it and loop, if no return the final answer. No state machine, no graph, no separate orchestrator abstraction — a for loop and an if.

The six-element system prompt

The system prompt is the other half of the loop, and it's treated as a spec with required parts, not free-form prose. From src/systemPrompt.ts:

You are the support agent for {company}. You are not a human agent — say so if asked.

GOAL: resolve or correctly escalate every ticket in as few steps as possible.

TOOLS: order_lookup (read order state) · refund_eligibility (check BEFORE
proposing a refund) · issue_refund (gated) · kb_search (use before any factual
answer) · send_email (gated, delivers the resolution).

POLICY: never call issue_refund without a prior refund_eligibility=true result.
Never fabricate an answer kb_search did not return. Never state REFUND_PROPOSED
unless issue_refund actually succeeded in this run.

UNCERTAINTY: if you are not confident, say so explicitly and escalate — do not guess.

DONE: emit a final response only once resolved, a refund is proposed, or the
ticket is escalated. Always start with RESOLVED:, REFUND_PROPOSED:, or
ESCALATED: so the outcome can be parsed.

Six required elements: identity (who it is / who it serves), one-sentence goal, tool list + when to use each, explicit policy (what it can't do), how to signal uncertainty, how to signal done. Skip any one of these and you'll find it in your eval failures — a prompt missing an explicit "how to signal done" section is, empirically, the single biggest source of unparseable final responses.

Notice that POLICY line about REFUND_PROPOSED — "never state REFUND_PROPOSED unless issue_refund actually succeeded." That sentence exists because, without it, the model did exactly that: described a refund as proposed without ever calling the tool. The full story — and why the fix required more than just this prompt line — is Part 5.

Memory: one state object, not four systems

"Memory" sounds like it should mean a vector database. Here it's a single object, updated by a reducer — src/memory.ts:

export interface AgentState {
  goal: string;
  working: { ticket: Ticket };                          // this run only
  conversation: { trajectory: TrajectoryStep[] };         // rolling
  artifacts: Record<string, unknown>;                      // exact, structured
  long_term: Pick<Customer, "preferred_channel" | ...> | null; // stable, mock in this repo
  policy: typeof policy;
}

Four layers, ordered by increasing persistence and complexity: working (this run only — the current ticket, ephemeral), conversation (the trajectory so far), artifacts (exact structured facts that need precise recall later, like a refund confirmation ID — never fuzzy semantic search for something like this), long-term (stable facts across sessions — read-only mock data here, since this repo has no real persistence layer).

The decision rule worth internalizing: reach for the cheapest layer first. In-context conversation memory is sufficient for most single-session agents. An external structured store is for facts that must survive across runs. Vector/semantic memory is the most expensive, highest-maintenance option, and should be justified, not defaulted to — critically, never for anything with financial or access-control consequences, where "approximately right" retrieval is the wrong guarantee. A refund confirmation ID goes in artifacts as an exact string, not embedded and semantically searched for later.

What's next

Part 5: Guardrails That Live in Code, Not the Prompt → covers the guardrail layer this loop leans on for every tool call: the policy-as-code approval gate, and the real bug it was added specifically to catch.

Repo: github.com/akash-pal/agent-from-scratch

Part 3: Build the Eval Set Before the Agent Exists

Akash Pal — Tue, 11 Aug 2026 18:43:48 +0000

Part 3 of a series building a support-ticket agent with no framework. Previous: Part 2 (tool contracts). Repo: github.com/akash-pal/agent-from-scratch

Here's the ordering that trips people up: build the eval set before the agent loop exists. Not after, not alongside — before. It feels backwards. You can't run an eval against an agent that doesn't exist yet.

That's exactly the point. If you write the eval set after the agent is working, you're unconsciously grading against whatever the agent already does. Cases you didn't think to write are cases your agent silently fails on forever. Writing 21 cases against a specification (the use case and tool contracts from Part 2) means you're measuring against a real target, not tuning your eval to match your own demo.

The eval set: `eval/cases.json`

21 cases, three buckets:

Bucket	Count	Covers
Easy	12	Shipping-status lookups, simple KB questions, a cancelled-order info request, one no-KB-match case that must escalate rather than fabricate
Hard	6	Refund eligibility inside/outside the 30-day window, multi-item orders where only one item is refunded, boundary cases just past the window
Edge	3	Legal-threat, fraud-flag, and duplicate-ticket patterns — must auto-escalate with zero tool/LLM calls

A sample case, checking both the outcome and the trajectory that produced it:

{
  "case_id": "hard_03",
  "bucket": "hard",
  "ticket": {
    "ticket_id": "hard_03",
    "subject": "Wrong size shoes, keep the socks",
    "body": "The running shoes from order ord_1004 are the wrong size. I want a refund for just the shoes, not the socks.",
    "customer_id": "cust_002",
    "order_id": "ord_1004"
  },
  "expected_trajectory": ["order_lookup", "refund_eligibility", "issue_refund"],
  "expected_outcome": "refund_proposed",
  "expected_max_steps": 4,
  "policy_checks": [
    "refund amount reflects only the shoe item (~$74), not the full order total",
    "issue_refund gated behind human approval"
  ]
}

Three things being checked per case, not just "did the answer look right":

Outcome — did it land on resolved / refund_proposed / escalated correctly.
Trajectory — did it call the right tools, in the right order. This is what catches "right answer, wrong path" failures — an agent that proposes a refund without ever checking eligibility first, for instance.
Policy checks — case-specific assertions, like "never call issue_refund for an already-cancelled order."

Why trajectory matters as much as outcome

An agent can land on the correct final answer through a broken process. If hard_03's agent calls issue_refund before refund_eligibility, and just happens to guess an eligible amount, checking only the final outcome would call that a pass. Checking the trajectory catches it. This maps directly to something the underlying guide's own evaluation framework insists on: different build steps need different kinds of evaluation, and trajectory correctness is its own category, separate from "did the final answer look okay."

One design choice worth flagging honestly: the first version of this trajectory check used exact array equality (JSON.stringify(actual) === JSON.stringify(expected)), and it was wrong — real LLM runs are non-deterministic enough that this produced false failures on completely correct agent behavior. Part 6 covers exactly what broke and the fix (ordered-subsequence matching instead of exact equality), because it's a genuinely useful lesson on its own.

The edge bucket exists to test the guardrail, not the model

The three edge cases — legal threat, fraud flag, duplicate ticket — aren't testing whether the LLM is smart enough to recognize a threat. They're testing whether the auto-escalate pattern match (a plain regex check, covered in Part 5) correctly intercepts these tickets before any model call happens at all:

{
  "case_id": "edge_01",
  "ticket": { "subject": "Final notice", "body": "...lawyer involved...Better Business Bureau." },
  "expected_trajectory": [],
  "expected_outcome": "escalated",
  "expected_max_steps": 0,
  "policy_checks": ["auto-escalated before any LLM/tool call (legal_threat pattern)"]
}

expected_trajectory: [] and expected_max_steps: 0 are the actual assertion: this ticket should never reach the LLM at all. That's a guardrail test, not a model-capability test — a meaningfully different thing to verify, and one worth keeping separate from the easy/hard buckets conceptually even though they live in the same file.

What a real harness run looks like

[PASS] easy_03 (easy)
[FAIL] hard_04 (hard)
    - trajectory: expected [order_lookup, refund_eligibility, issue_refund] as a subsequence, got [order_lookup, refund_eligibility]
[PASS] edge_01 (edge)

=== Summary: 12/21 passed ===
easy: 6/12  |  hard: 3/6  |  edge: 3/3

That hard_04 failure is real — it's the same run that surfaced the most significant bug in the whole build (Part 6 covers it in full: the agent claimed a refund was proposed without ever calling the tool that proposes it). This is what Step 3 buys you: a fixed, pre-written target to run against, so failures like this show up as a clear diff instead of a vague "something feels off."

What's next

Part 4: The Raw ReAct Loop: ~100 Lines, No Framework → covers the actual agent loop — the raw ReAct implementation these 21 cases get run against, with no framework in sight.

Repo: github.com/akash-pal/agent-from-scratch

Part 2: Pinning the Use Case and Writing Tool Contracts Like Specs

Akash Pal — Tue, 11 Aug 2026 18:43:46 +0000

Part 2 of a series building a support-ticket agent with no framework. Part 1 covered why. This part covers Steps 1–2 of the build order: pinning the use case, and writing tool contracts. Repo: github.com/akash-pal/agent-from-scratch

Before any code, two documents: docs/use-case.md and docs/tool-contracts.md. Skipping this step is the single most common reason teams end up with an agent nobody trusts — not because the idea was bad, but because nothing downstream (evals, prompts, memory) had a fixed target to hit.

Step 1: pin the use case

Four gates, filled in before writing a line of code:

Gate	Definition
Bounded input	One support ticket: `{ subject, body, customer_id, order_id? }`
Bounded output	Exactly one of: `resolved`, `refund_proposed` (pending approval), `escalated` (with a reason)
Tool count	5
Success metric	Resolution rate > 85% without escalation; escalation rate < 10%

The tool count cap matters more than it looks. An agent given 10+ tools starts hallucinating tool names and picking the wrong one — a cognitive load problem, not a dependency problem. Keeping this agent to 5 tools, covering exactly three request types (order status, refunds, KB lookups), keeps every run in the healthy 3–8 tool-call range instead of ballooning into a system that needs to be split into multiple specialist agents. (Part 7 covers the actual cost math for when a split is worth it.)

Bounded output matters too: resolved / refund_proposed / escalated isn't just documentation — it becomes a literal parseable prefix (RESOLVED:, REFUND_PROPOSED:, ESCALATED:) that the agent's final message must start with. Part 4 shows exactly how that gets parsed, and Part 5 shows why trusting that string alone turned out to be a real bug.

Step 2: tool contracts are a schema, not a docstring

This is the part that's easy to under-invest in. A tool's description field isn't a comment for future developers — it's the only thing the LLM reads to decide when to call the tool. Treat it as a specification.

Here's the actual refund_eligibility definition from src/tools/index.ts:

{
  name: "refund_eligibility",
  description:
    "Check whether an order is eligible for a refund BEFORE ever proposing one. " +
    "Orders are eligible only if delivered and within a 30-day window of order_date. " +
    "Cancelled orders are already auto-refunded and are never eligible for a manual " +
    "refund. Returns eligible, max_amount_usd, and a policy_ref explaining the decision.",
  gated: false,
  input_schema: {
    type: "object",
    properties: {
      order_id: { type: "string" },
      reason: { type: "string", description: "Customer's stated reason for the refund request" },
    },
    required: ["order_id", "reason"],
  },
}

Notice the 30-day window is stated explicitly in the tool description, not just mentioned once in the system prompt. That's not redundancy — it's a finding from later iteration (full story in Part 6): policy logic that lives only in the system prompt gets missed under load. It has to be load-bearing in the tool itself.

The full contract table, matching each tool to its schema and failure mode:

Tool	Input	Output	Failure mode
`order_lookup`	`{ order_id }`	`{ status, items, total_usd, order_date }`	order not found → structured error, not a guess
`refund_eligibility`	`{ order_id, reason }`	`{ eligible, max_amount_usd, policy_ref }`	cancelled/undelivered → `eligible=false` with a reason; ambiguous → escalate, never assume `true`
`issue_refund` (gated)	`{ order_id, amount_usd, idempotency_key }`	`{ confirmation_id }`	`idempotency_key` required — prevents double-refund on retry
`kb_search`	`{ query }`	`{ articles: [...], relevance_scores }`	no match → explicit empty result; agent must not fabricate an answer
`send_email` (gated)	`{ to, subject, body, ticket_id }`	`{ sent, message_id }`	gated behind pre-action approval (Part 5)

Three details worth calling out specifically:

Idempotency keys aren't optional. issue_refund requires one because retries happen — a flaky network call, a model retrying after a transient error — and a refund tool without idempotency protection means a retry can double-refund a customer. This is enforced in the tool executor itself, not assumed from good behavior:

// src/tools/issueRefund.ts
const issuedRefunds = new Map<string, string>(); // idempotency_key -> confirmation_id

export async function issueRefund(args: Record<string, unknown>) {
  const idempotencyKey = args.idempotency_key ? String(args.idempotency_key) : "";
  if (!idempotencyKey) return { error: "idempotency_key_required" };

  const existing = issuedRefunds.get(idempotencyKey);
  if (existing) return { confirmation_id: existing, replayed: true };
  // ... issue the refund, store it under idempotencyKey
}

Failure modes are structured, not guessed. order_lookup on a missing order returns { error: "order_not_found", order_id }, not an empty object the model has to interpret. kb_search on no matches returns an explicit empty articles: [], specifically so the system prompt can say "if this comes back empty, escalate — don't fabricate an answer."

Gated tools are marked at the contract level. issue_refund and send_email carry gated: true right in their definition. That flag is read by the policy layer (Part 5) to decide which tool calls need a human in the loop before they execute — the gating decision starts here, at the contract, not somewhere downstream.

Data source: mock, but shaped like something real

All five tools read from an in-memory mock store — src/data/mockData.ts: five customers, ten orders spanning inside/outside the refund window, five KB articles. It's deliberately fake data (this repo is a reference build, not a production system — more on that in the repo's own README), but the schemas are shaped to mirror a real commerce API's order/return/line-item structure, so swapping in a real backend later is a data-layer change, not a tool-contract rewrite.

What's next

Part 3: Build the Eval Set Before the Agent Exists → covers building the eval set — 21 cases written before the agent loop existed, and why that ordering is the actual point, not a nice-to-have.

Repo: github.com/akash-pal/agent-from-scratch

Part 1: What Makes Something an Agent (and Why We Built This Without a Framework)

Akash Pal — Tue, 11 Aug 2026 18:43:44 +0000

Most agent tutorials reach for a framework on line one — LangChain, LangGraph, CrewAI, pick one. This series does the opposite. Over seven parts, we build a real support-ticket agent with no agent framework at all: a hand-written loop against a raw model SDK, explicit guardrails, and an eval set built before the agent exists.

The full code is on GitHub: akash-pal/agent-from-scratch. Every article in this series links back to the specific files it's talking about — this isn't a toy snippet series, it's a walkthrough of one real (if intentionally small) build.

Chatbot vs. agent

A chatbot answers once per turn. You send a message, it sends one back, done.

An agent runs a loop: it decides an action, executes a tool, observes the result, and decides again — repeating until the task is done or it gives up. This is the classic ReAct pattern (Reason + Act):

That loop is the entire reason the rest of this series exists. Because the agent calls tools autonomously and repeatedly, small errors compound across steps in a way a single chatbot turn never does. Propose a refund on the wrong order, and unlike a chatbot giving a wrong answer once, an agent might act on that mistake — call a tool, change state, send an email. The infrastructure this series builds — evals, tracing, guardrails, memory design — exists to bound and catch that compounding failure, not because agents are inherently harder to prompt.

Three components sit around the loop:

Tools — each is a name + natural-language description + JSON input schema + an executor function. The description is what the LLM reads to decide when to call it — treat it as a specification, not documentation. (More on this in Part 2.)
Memory — what the agent knows within a run and across runs. (Part 4.)
Orchestrator — the LLM (or a graph/state machine wrapping it) that decides which tool to call next based on the current state.

Why build it without a framework first

Frameworks pay for themselves under specific conditions: multi-agent orchestration, streaming UI needs, integrated tracing, advanced retry/fallback logic, or a team of people who need a shared abstraction. None of those apply to a first agent built by one person to understand what's actually happening.

If you can't tell whether LangGraph is adding value or hiding a bug, you don't yet know what LangGraph is doing for you. Writing the raw loop first — even a bad one — gives you a mental model to evaluate any framework against later. That's the actual thesis of this whole series: build the ~100-line version, understand every step of it, then decide if you need more.

The build order this series follows

Every part below produced a real, committed artifact in the repo, in this order — and the order is deliberate, not incidental:

Pin the use case — bounded input, bounded output, a hard cap on tool count, one measurable success metric.
Map tools & data sources — write every tool contract before writing code.
Build the eval set — 21 cases, written before the agent existed, so it measures against a real target instead of tuning to whatever the agent happens to do.
Write the minimal loop — the raw ReAct loop, no framework.
Write the system prompt — six required elements, versioned like code.
Iterate to green — one change per fix, and a log of what broke and why.
Add human review — approval gating on consequential actions, decided as an architecture choice, not bolted on after launch.

The use case: an e-commerce support-ticket agent

The concrete system this series builds triages and resolves inbound customer support tickets — order status questions, refund requests, and knowledge-base lookups — for a fictional e-commerce company. Five tools, one agent, real guardrails:

order_lookup and kb_search are read-only and run freely. issue_refund and send_email are gated — they require human approval before they execute, because they're consequential and hard to reverse. That distinction — which actions can run autonomously and which need a human in the loop — turns out to be an architecture decision made in Part 1 of the design, not a safety feature bolted on at the end. Part 7 covers exactly how that gating works.

What's next

Part 2: Pinning the Use Case and Writing Tool Contracts Like Specs → covers pinning the use case and writing the five tool contracts — including the specific wording change that fixed a real eval failure later in the series.

Repo: github.com/akash-pal/agent-from-scratch

Scale Frontend Application: Journey from Zero to Millions of users.

Akash Pal — Wed, 12 Nov 2025 19:07:15 +0000

Scalability problem arises when the system is fast for single user but slow under heavy load.

A frontend application journey starts with a language and a framework to speed up development and setup.

Browsers only understand HTML, CSS and Javascript. The application code is more developer friendly and it needs to be converted to browser compatible format. This is where bundlers come in.
The bundlers minify code, optimize images sizes and leverages techniques such as dynamic import and code-splitting to reduce overall size of application.

Along with that it uses Tree shaking to analyse the dependency tree and remove any unused code from final bundle. This enables to improve the load times and overall performance.

The generated bundle is deployed to server. This is an application server which hosts the bundle.

The application server IP address needs to be mapped to a DNS record.

If the application server is in India and the users are in North America then it would increase the latency for the users from North America to reach the server in India.

CDN (Content Delivery Network) is used to cache requested assets in nearby geographical regions. A combination of single or multiple CDN providers can be used to increase fault tolerance.

With just one application server all the requests are directed to it.
Think of a queue where users line up and each user is addressed at a time. In order to make the server available for all users, the application server can be scaled either Horizontally or Vertically.

Vertical scaling means adding more RAM and more CPU cores.
Horizontal scaling means adding more application servers.

Vertical scaling is easier which means a chef in a kitchen has more hands to address more users, but its more costly.

Horizontal scaling is cheaper than adding more RAM and CPU, simply adding another instance of application server. However now the user requests have to be directed to either of the application servers. This needs to be done by a Load Balancer.

The Load Balancer distributes the traffic among the application servers based on algorithms such as Round Robin or least connections.

Round robin methods distributes serially the requests to each server. This might lead to a scenario where one application server is processing a request and round robin hands another request, the new request needs to wait for the current one.

Whereas, least used connection priorities which application server is available and redirects the request to that server.

Nginx ("engine x") a application load balancer is widely used.

If customers in a restaurant walk up to the chef to request for food then chaos erupts! A waiter ensures thats the requests are provided in a orderly fashion so the chef can focus on cooking.

The waiter (nginx) serves that static frontend files and redirects the traffic to backend server (acting like a reverse proxy for backend API's)

In order to ensure consistence between the application code developed locally and the one shipped to production environment a standard shipping mechanism is required - this provided by called Docker. Docker enables to pack all the application code and dependencies into a standardised box called containers.

Instead of managing multiple application servers which are not may be utilised fully. All containers can reside within a single application server; thereby saving the cost of multiple servers and also ensuring consistency between local and deployed instance.

A team of a few developers can continue using a single code base to manage the entire application.

When each module is handled by a different team then splitting the monolithic codebase into fragments can help individual modules to be scaled separately and deployed without dependency.

MicroFrontend Architecture enables to deploy each module separately and leverage dependent modules at runtime. Each module is deployed separately without any dependency.

ModuleFederation works on three core concepts - expose something, consume something and share dependencies.

MicroFrontend solves a organizational problem not a technical problem.

MicroFrontend can be achieved using different techniques:

Publish each app as an NPM package
Bundle each micro-app individually and load with a script tag
Using micro frontend frameworks
Using Module Federation with webpack 5

MicroFrontend brings in unique challenges like:

How to setup inter-communication between individual modules?
How to share data state ?
How to fetch the types of federated modules ?
How to unit test modules dependent on other federated modules ?
How to improve the developer experience of connecting different federated modules ?
How to improve the performance ?

We will explore these in future. For now thats all!! Add in your thoughts and queries in comments section.

DEV Community: Akash Pal

Part 7: Iterating to Green: Real Bugs, and When You'd Actually Reach for a Framework

The iteration log, condensed

Single agent, not multi-agent — and why that's arithmetic, not instinct

Framework selection — now that you've seen the raw version

Where this leaves the repo

Part 6: Observability for AI Agents: Tracing, Metrics, and Drift

The minimum trace payload

Readable in a terminal, structured everywhere else

Three production monitoring layers

Why this matters for debugging, not just monitoring

What's next

Part 5: Guardrails That Live in Code, Not the Prompt

Policy as code

Three human-review patterns

The bug: a claim without a fact behind it

The fix — two layers, not one

What's next

Part 4: The Raw ReAct Loop: ~100 Lines, No Framework

The six-element system prompt

Memory: one state object, not four systems

What's next

Part 3: Build the Eval Set Before the Agent Exists

The eval set: eval/cases.json

Why trajectory matters as much as outcome

The edge bucket exists to test the guardrail, not the model

What a real harness run looks like

What's next

Part 2: Pinning the Use Case and Writing Tool Contracts Like Specs

Step 1: pin the use case

Step 2: tool contracts are a schema, not a docstring

Data source: mock, but shaped like something real

What's next

Part 1: What Makes Something an Agent (and Why We Built This Without a Framework)

Chatbot vs. agent

Why build it without a framework first

The build order this series follows

The use case: an e-commerce support-ticket agent

What's next

Scale Frontend Application: Journey from Zero to Millions of users.

MicroFrontend can be achieved using different techniques:

MicroFrontend brings in unique challenges like:

The eval set: `eval/cases.json`