Part 7 (final) of a series building a support-ticket agent with no framework. Previous: Part 6 (observability). Repo: github.com/akash-pal/agent-from-scratch
The other six parts described the finished design. This one is about what "finished" actually took — the real bugs the eval set caught, and the two questions every agent build eventually has to answer honestly: do you need more than one agent, and do you need a framework. Full detail on everything below: docs/iteration-log.md.
The iteration log, condensed
1. Exact trajectory matching was the wrong check. First eval run: 12/21 passed. Most failures were the agent correctly sending a confirmation email where the eval only expected a lookup — correct behavior, wrong assertion. Fix: switched the harness from exact-array equality to ordered-subsequence matching (every expected tool must appear, in order; extra steps in between are fine). Still catches a missing, reordered, or wrong tool. Stops false-failing on benign non-determinism.
2. No retry/backoff meant a transient error crashed the whole run. A 503 — model overloaded on case 1 took down the entire eval harness. Fixed with exponential backoff on 429/503 specifically, plus inter-case pacing to stay under free-tier rate limits.
3. A -latest model alias silently rolled onto a stricter quota. gemini-flash-latest worked, then started failing with a 20 requests/day cap after quietly resolving to a newer model. Fixed by pinning an explicit model version instead of an alias, after checking the provider's live usage dashboard for actual quota — 25x more headroom on the pinned model. The takeaway generalizes past this one provider: "latest" aliases optimize for capability, not quota stability, and what they resolve to changes over time without your code changing at all.
4. A testing artifact that looked like a real bug. Piping multiple answers into the interactive CLI via printf "a\nb\n" | npm run agent intermittently hung after the first prompt. Root cause: a Node.js readline/promises quirk with quickly-closing piped stdin — confirmed as a testing artifact, not a real bug, by replaying the same input through an actual pseudo-terminal, where it worked every time. Fixed anyway by reusing a single shared readline.Interface instead of opening/closing one per prompt — more correct regardless of the original symptom.
5. The model claimed a refund was proposed without ever proposing one. The big one — covered in full in Part 5. Two eval cases showed outcome: refund_proposed with a trajectory that never called issue_refund. Fixed at two layers: an explicit prompt line, and — the layer that actually matters — a code-level check (enforceOutcomeIntegrity) that verifies a real confirmation_id exists in state before trusting the model's own claim.
The honest note attached to that fifth finding in the actual log: it was verified with a single targeted re-run, not a full eval pass, specifically to conserve free-tier API quota. That gap is left visible in the repo on purpose — claiming a full pass without having run it would be the same category of mistake as finding #5 itself.
Single agent, not multi-agent — and why that's arithmetic, not instinct
Multi-agent (a coordinator dispatching to specialist workers) is a cost decision, not an architectural preference. A coordinator + worker split costs roughly 3–8x a single agent's tokens and latency per run — workers see fewer tools each, which sharpens tool selection, but you're paying for that in every single run, not just the hard ones.
Three signals should all be true before paying that multiplier: the task genuinely splits into specialized roles that each benefit from a sharper, narrower prompt; subtasks can run in parallel for a real speed win, not just cleaner code; and volume × accuracy improvement actually outweighs the cost multiplier — checked with real numbers, not intuition.
At low volume, the math tends to favor a single agent even with a real accuracy edge from splitting — the multiplier isn't earned back. At high volume, the same accuracy edge can save real money, because escalation-cost reduction scales with volume in a way the fixed multiplier doesn't. This repo's own agent stays single, deliberately: five tools is well under the point where an agent starts confusing tool names, and nothing about the ticket volume this reference build targets justifies paying a multiplier for a split that would mostly just look tidier.
Framework selection — now that you've seen the raw version
Having built the loop by hand, the actual question a framework answers becomes concrete instead of abstract: does this problem need multi-agent orchestration, streaming UI wiring, an integrated tracing ecosystem, or shared abstraction across a team? If none of those apply, the ~100-line loop this series built is the production version, not a placeholder for one.
| Framework | Best for | Trade-off |
|---|---|---|
| Raw SDK (this series) | Learning the loop, simple single-agent tools | Minimal abstraction, provider-locked, most control |
| LangGraph | Complex state, multi-agent, widest enterprise adoption | More setup cost; strongest observability ecosystem |
| Mastra | TypeScript-native teams | Smaller ecosystem, natural fit for a JS/Next.js background |
| Vercel AI SDK | Streaming UI tightly coupled to a Next.js frontend | Less suited to long-running, durable multi-step workflows |
| CrewAI | Standing up role-based multi-agent crews quickly | Opinionated structure |
| Google ADK | Teams committed to GCP/Gemini | Less model-agnostic than LangGraph |
| LlamaIndex | RAG-heavy use cases, large document corpora | Less general-purpose outside retrieval |
Where this leaves the repo
Everything in this series is a real, committed artifact — not a cleaned-up retelling. The README has the full architecture diagrams, quickstart, and real example runs (screenshots included, not staged). AGENTS.md documents the hard constraints for anyone — human or AI coding agent — extending this repo later: no framework, mock data only, gated tools stay gated, guardrails live in code.
It's explicitly a reference build, not production software — mock data, a single free-tier model, no persistence, no auth. If you take one thing from the whole series, take the build order, not the code: pin the use case, write tool contracts as specs, build the eval set before the agent exists, write the smallest loop that could work, iterate against real failures, gate the consequential actions. The framework question comes last, not first — and by the time you're actually equipped to answer it, you often don't need to.
Repo: github.com/akash-pal/agent-from-scratch
Series index: Part 1 (what makes something an agent) · Part 2 (use case & tool contracts) · Part 3 (the eval set) · Part 4 (the raw loop) · Part 5 (guardrails) · Part 6 (observability) · Part 7 (this post).
Top comments (0)