DEV Community: Sayam Jain

Not every AI feature needs an agent

Sayam Jain — Wed, 11 Nov 2026 12:58:57 +0000

These days almost every LLM integration gets called an agent. Add a model, give it two tools, and suddenly it's "agentic".

I build a WhatsApp assistant for myself, and it is deliberately not an agent. The design is a workflow: plain code decides what the model sees, the model gets one narrow job, and code checks its answer. That choice makes the behaviour easier for me to inspect.

Here's how I think about the difference, and the same task built both ways so you can see what changes.

Workflow or agent, in one line each

Anthropic's foundational guide to building effective agents, published in December 2024, draws a useful distinction. In a workflow, the model and tools run through code paths you define. In an agent, the model directs its own next steps and decides which tools to use.

The guide's practical advice is to start with the simplest approach and add complexity only when the problem needs it. Sometimes that means no agent at all.

Five shapes an AI feature can take

Shape	Who controls the steps	Good for
Single model call	Your code	Classifying, summarising, rewriting or extracting from text you already have
Retrieval, then generation	Your code	Finding relevant document passages before asking the model to answer
Workflow	Your code	A known sequence: rank, call the model, validate and apply business rules
Agent	Mostly the model, within code defined tools and limits	Problems where the next step depends on what the model discovers
Several agents	Models work across separate roles, within an orchestration layer	Work that genuinely splits into independent jobs with clear boundaries

These aren't levels you have to climb. They're different ways of organising an AI feature. Handing more decisions to the model can mean more requests, more accumulated context and a less predictable run. It depends on the design; a model can also return more than one tool call in a single response.

The workflow behind my WhatsApp assistant

Message Yourself OS reads my WhatsApp and tells me who's waiting on me and what I promised. Every chat is indexed on my machine. In the ranked mode I'm testing, the flow looks like this:

every chat, indexed on my machine
        |
        v
plain code scores each chat on simple signals
(unread messages, they wrote last, a message ending in "?",
 my last message using words like "I'll" or "I will",
 activity today; muted chats and quiet groups lose points)
        |
        v
12 points to qualify, at most 50 chats, 40,000 token budget
        |
        v
one model call: what's waiting on me, what I promised,
each with the exact quote
        |
        v
plain code checks every quote against its message
        |
        v
!today: checked findings go into Focus, the rest are marked to verify

One detail matters: the ranked mode is behind a switch. Until it passes a gate of seven reviewed days of real use, the assistant uses unread chats instead. The diagram and comparison below show the ranked workflow I'm testing, not a claim that the new ranked brief is already the default.

The model never decides which chats to open. Code does, with rules I can read and explain. Those rules don't understand meaning; they look for signals like a question mark or the words "I'll" and "I will". Understanding whether a message is really a question for me or a promise is the model's job. Code then checks the evidence and sender.

One thing to be clear about, because this is WhatsApp: indexing and scoring happen on my machine, but the messages in the chats that make the shortlist do go to the model provider. The bot now uses Groq, with OpenAI as fallback. Local indexing means the model sees a shortlist, not that it sees nothing.

I should also be honest about the plan the bot used. Until November 2026 it ran on Gemini's unpaid quota. Google's terms say content sent through unpaid services can be used to improve Google products and may be reviewed by people, and they ask you not to submit sensitive, confidential or personal information. My shortlist contains other people's messages, so I've switched the bot's default provider to Groq. Groq's documentation says it doesn't keep inference inputs or outputs by default, may keep logs for up to 30 days when it investigates reliability or abuse, and lets every customer turn on Zero Data Retention to stop even that, which is how the bot's account is set. Read the terms for every provider in your own fallback chain before sending real messages; mine still has OpenAI as a fallback. The public demo below uses only made up chats. Google's Gemini API terms and Groq's data controls explain the details.

Could an agent do this? Sure. Give it list_chats and read_chat and let it figure things out. But I already know the steps. For this task, asking the model to rediscover them each morning adds calls and more places for the run to stop or go wrong.

The same task, built both ways

To make the difference concrete I built a small demo: twelve made up chats, and the same triage task done twice. The code, prompts, settings and raw traces are on GitHub.

The workflow ranks chats with the same point rules, sends only the ones that qualify, makes one model call with a JSON schema, and checks the quotes in code:

const ranked = pickCandidates(chats, today);
const reply = await model.generate({
  system,
  turns: [{ role: "user", text: prompt }],
  jsonSchema: findingSchema,
});
const findings = verifyFindings(parseFindings(reply.text), chats);

The agent gets three tools (list_chats, read_chat, finish) and a loop. It decides what to read, one step at a time, until it calls finish or hits 15 steps:

for (let step = 1; step <= maxSteps; step++) {
  const reply = await model.generate({ system, turns, tools: agentTools });
  const finish = reply.toolCalls.find((call) => call.name === "finish");
  if (finish) return verifyFindings(finish.args.findings, chats);
  for (const call of reply.toolCalls) {
    turns.push({
      role: "tool",
      name: call.name,
      result: runTool(call.name, call.args, chats),
    });
  }
}

Both end with the same check, so the comparison is fair in one important sense: same chats, same model and same rules for checking evidence. For each finding, the check confirms that the quote appears in the named message and that the sender matches the claim: a promise has to be a message I sent; "waiting on me" has to be a message someone else sent. That verifies evidence and attribution. It doesn't prove the model understood the meaning correctly.

What the traces showed

I ran the comparison three times on 10 November 2026, on openai/gpt-oss-120b through Groq. I started with gemini-2.5-flash, but my AI Studio project showed a daily limit of 20 requests for that model. I kept the Gemini run separate from the three Groq runs rather than mixing the numbers. It is in the repo too and showed the same pattern.

These are the three Groq runs. Ranges go from the lowest to the highest run; findings show each run's result.

Measure	Workflow	Agent
Model calls	1	5 to 6
Tools run (`finish` not counted)	0	4 to 5
Input tokens, all calls added up	878	3,633 to 4,695
Visible output tokens, reasoning excluded	299 to 503	377 to 464
Thinking tokens	435 to 723	613 to 1,029
Model time, not counting waits	2.7 to 3.5 s	5.9 to 8.2 s
Waits after an HTTP 429 response	0 to 10 s	0 to 29 s
Findings that passed the evidence check	8, 7, 7	4, 5, 5
Promises found, out of 2	2 every run	0 every run

Visible output tokens and thinking tokens are separate rows in these traces. Groq reports thinking as part of completion tokens; the demo subtracts it to show visible output and reasoning separately. Add the two rows to get the completion token total reported by the provider.

This is three runs of two implementations on one synthetic dataset. It isn't a benchmark. A better agent prompt, different stopping conditions, another model or another task could change the result. Treat these as a few honest data points, not a law.

In this implementation, each agent step makes another model request and sends the conversation so far again. That is why its input token count grows as it opens chats, while the workflow sends its shortlist once.

Added up across its calls, the agent's input came to about 4.1 to 5.3 times the workflow's for the same twelve chats. That number is cumulative across requests, not one large context window. Every request also counts against the provider's rate limits. The demo waits only when a provider returns HTTP 429, uses the delay the provider supplies and records it. In these back to back Groq runs, the agent waited up to 29 seconds; the workflow waited once, for 10 seconds. In the Gemini run, my project showed a five requests per minute limit for that model, and the agent's sixth request waited 94 seconds. Those are the limits I saw on my accounts that day, not fixed limits for everyone. Groq's documentation explains that its limits vary by model and account: Groq rate limits. Google's quota documentation works the same way: Gemini API rate limits.

The bigger surprise was what it found. In every run the agent listed the chats, opened Riya, Mom, the team standup and usually the recruiter, all chats where someone asked me something, and then decided it was done. It wasn't forced to read every chat. It could call finish when it thought it had enough, within the 15 step limit. That's the decision I had handed to the model, and it stopped before reaching the two chats that mattered. It never opened Arjun's or Neha's chats, where I had made promises. So it missed both promises in all three Groq runs and in the Gemini run. On Gemini it also labelled the standup message "can you share the deploy notes before 4?" as a promise I made. The quote check caught that because the message wasn't sent by me.

The workflow found both promises in every run because my ranking gives my last message containing "I'll" or "I will" the same 14 points as a message ending in a question mark. Both chats qualified, so the model saw them. The rule is in code; it doesn't depend on the model remembering to look.

The check has limits too. In one run the workflow listed Arjun's "Bro the API docs link is broken" as waiting on me. The quote is real and Arjun sent it, so it passed the evidence check even though it isn't really a question. Code can check evidence; it can't check judgement.

Aur yahi interesting part hai. The agent isn't wrong for every problem. It was the wrong fit for this one: it spent model calls deciding which chats to inspect even though my application already had a rule for that.

What this experiment doesn't prove

A workflow can fail too. If my scoring rules are wrong, a chat that matters never reaches the model, and nothing downstream can bring it back. Someone who writes "kal tak bhej dunga" instead of "I'll send it tomorrow" scores no promise points in my rules. An agent can adapt to things my rules never anticipated.

That's the real tradeoff:

Workflow: you encode more of your assumptions in code.
Agent: you hand more decisions to the model.

Neither removes failure. They move where the failure happens. For my inbox I'd rather have failures I can find by reading my own rules than failures I have to dig out of a trace.

Where the difference really shows up

	Workflow	Agent
Model calls	Fixed for this design: one	One per step in this loop; the count depends on the run
Tokens	Predictable within the input budget	Grow as the conversation is sent again on each step
Control flow	Defined by code	Directed by the model within the tools and limits you expose
Next action	Defined by code	Chosen at run time
Model output	Can vary	Can vary
Debugging	Read the code path and model result	Read the trace and work out why it chose that step
Tool permissions	Code calls only what it needs	The model can call any tool you expose, so expose less
When it fails	Easier to locate within a known sequence	Can stop early or keep going; set step limits
Human checks	Easy to place at fixed points	Put gates around the actions that matter

When I would use an agent

When I can't write the steps down. Investigating an unfamiliar codebase, research where each answer decides the next question, or debugging when I don't know what I'll find. That's where letting the model choose its next move can earn its cost.

Even then, I'd keep the parts I do know in code: permissions, validation and the final check. The model proposes; my application decides what actually happens.

What I'd tell someone building an AI feature

Write down the steps first. If you can, you probably want a workflow.
Give the model the smallest job only it can do, and let code do the rest.
Check its output in code, not with another prompt.
Add agency only where the steps are truly unknown, and put a step limit on it.

An agent isn't a better kind of software. Sometimes giving the model fewer decisions is exactly what makes the product good.

Sources

Evals before vibes

Sayam Jain — Wed, 11 Nov 2026 12:05:22 +0000

Here's how a lot of prompt changes ship. You tweak the prompt, try it on three examples you remember, the answers look better, and you ship it. That's vibes.

I've done it too. Then I spent a few months building checks for my WhatsApp assistant, and the most useful thing they taught me was where my own checks were weaker than I thought. This post is what I built, what it caught, and a small demo that shows the difference between trying a prompt on a few examples and running it against a fixed set.

There's also a practical reason this is timely. OpenAI's hosted Evals platform became read only for existing users on 31 October 2026, and its dashboard and API are scheduled to shut down on 30 November. If your test cases and graders live there, this is a good time to move them somewhere you can version and run yourself. OpenAI's deprecation notes have the current dates.

What I built for my WhatsApp assistant

Message Yourself OS reads my WhatsApp and tells me who's waiting on me and what I promised. Being wrong is expensive there: a bot that says "you promised this" had better be right. So most of the work went into checks.

Every finding carries its evidence. The model has to return the exact quote, the chat and the message it came from. Plain code then fetches that message and checks that the quote is really in it and the time matches. A finding ends up in one of a few states: verified, missing, not found, quote mismatch or time mismatch. Only verified findings reach the Focus list at the top of !today. Anything else appears under "Needs verification", never as fact.

58 test files pin the parsers, the evidence check and the rule that unverified items stay out of Focus.

A local evaluation runner has a fast tier that needs no WhatsApp connection or network, and a full tier that runs commands through a local server and tracks latency and regressions.

An AI judge scores replies. If the judgement itself fails, the result is reported as "not scored" and is never shown as a number.

A gate on real use. The newer, ranked !today stays behind a switch until it passes seven reviewed days of real use, with no errors and nothing missed.

!correct records why an extraction was wrong, in my own words.

What those checks caught, and what they couldn't

The evidence check earned its keep straight away. It's the reason a made up quote can't turn into a reminder. It's also the part I trust most, because it's plain code checking plain facts.

The rest taught me more by showing me their limits:

The gate did its job by saying no. When I last checked in August 2026, the ranked brief had nine runs over three days and almost none had been reviewed. So it stayed off. Without that gate, I might have switched it on because the replies looked good.
My fast checks had a blind spot. They confirmed that all 44 commands were wired up, but the heading checks compared against text I had typed into the test, not the actual output. Useful as a wiring check, not an evaluation of what the command returned.
My first AI judge only saw the reply. It never saw the messages the reply was about, so it could judge tone and format, not whether the answer was true. I wrote in my own docs that its scores were advisory for exactly that reason.
I wrote six labelled scenarios and never wired them in. An explicit promise, an implied one, a resolved one, group noise, several people and a dead conversation. They sat in a file that no code read.
!correct records reasons, but nothing reads them back. Every correction should become a test case. Mine became a line in a file, but the runner did not use those lines.

Pattern dikh raha hai? Everything that worked compared the output against something fixed: a real message, a test or a calendar of reviewed days. Everything that didn't was either checking itself or never run.

The same lesson, in one small demo

To show the idea without using my private chats, I built a small eval with 24 synthetic cases. Each case contains one to three chats and the expected findings written down by hand. The set has 17 expected findings in total, and nine cases should return nothing. It covers plain questions, requests without question marks, Hinglish questions and promises, group messages addressed to someone else, requests already answered, promises already kept, rhetorical questions and messages that should not count.

The task is the same as in my last post: find what's waiting on me and what I promised, with exact quotes. The code, cases, prompts and results are in the companion demo on GitHub.

I compared two prompts:

v1 is the original prompt from the last post. It asks for what is waiting on me and what I promised, but leaves much of the meaning implicit.
v2 spells out the rules: a question or request to me that I haven't answered, a promise in any language including Hinglish, and no statements, rhetorical questions, questions addressed to other people or things already done.

First, the vibes check. I ran both prompts on three easy cases, the kind you'd try by hand. Both got all three right in every run. If that's all you look at, v1 is fine.

Then I ran both prompts on all 24 cases using openai/gpt-oss-120b through Groq. I repeated the run three times. Each sequence in the table is in run order: one, two, then three.

Measure	v1, runs 1 / 2 / 3	v2, runs 1 / 2 / 3
Three easy cases passed	3/3, 3/3, 3/3	3/3, 3/3, 3/3
Cases passed, out of 24	18, 17, 16	22, 23, 23
False positives	7, 6, 8	1, 1, 1
Missed findings, out of 17	0, 1, 1	1, 0, 0

A case only passes when the expected findings are present, nothing extra is returned, the finding has the right kind and its quote passes the evidence check. One bad finding can make the whole case fail.

The difference is in the cases you wouldn't think to try. v1 said a client's "Can you send the invoice?" was still waiting on me, two messages after I'd sent it. It flagged "Why is Bengaluru traffic like this every single day?" as a question for me. And it marked Arjun's "Bro the API docs link is broken" as waiting on me, the same mistake I found by eye in my last post. This time a test case catches it every run.

v2 isn't perfect either. The rhetorical question fooled it in every run, and in one run it missed a Hinglish promise that v1 got right. That's the point of a fixed set: you see exactly which cases moved, in which direction, instead of relying on a feeling that it's better.

The judge agreed with the mistakes

I also ran an AI judge over the findings the prompts returned. It saw each finding together with the chat it came from, and had to give a pass or fail: was this really something waiting on Sayam, or really a promise he made? I then compared those judgements with the labels I had written by hand.

In every v1 run, the judge passed six findings that my labels said were wrong. It never failed one of the correct findings it judged. The examples included the broken link, the rhetorical question and the invoice I had already sent. Its explanations sounded reasonable: a broken link "requires Sayam to respond or fix"; a rhetorical question "legitimately expects a reply".

There is an important limit here: the judge only sees findings that the prompt returned. It cannot catch a missed promise, because there is no finding for it to judge. Its agreement is not a second score for the prompt and it is not a measure of recall.

This also reminded me of the limitation in my first production judge, which only saw the generated reply and not the messages behind it. The setups are different, but the lesson is similar: a plausible explanation is not evidence.

This is why I wouldn't trust an LLM judge just because it returns a confident explanation. Anthropic recommends calibrating judges against human experts, and the 2023 LLM as a judge study documented position, verbosity and self enhancement biases. A judge can be useful at scale, but its disagreements need to be checked against labels for your own task.

How I'd start evals for an AI feature tomorrow

Start with 20 to 50 cases from real failures. That's Anthropic's recommendation for a useful first set. My public demo uses synthetic messages for privacy, so it demonstrates the method rather than measuring the assistant on production conversations. When building your own set, turn real failures into cases after removing or replacing private details.
Write the expected answer down by hand. Your labels define what success means. If the right answer is unclear to you, the model cannot reliably infer it.
Prefer pass or fail for concrete checks. Hamel Husain and Shreya Shankar argue that binary judgements make the criteria clearer and more consistent than a score out of ten. A seven out of ten doesn't tell you what to fix.
Grade with code first. Is the quote in the message? Is the sender right? Is the JSON valid? Code is cheap, fast and consistent for facts that can be checked deterministically.
Add a judge only when code cannot judge the quality. Give it a clear rubric and compare its decisions with your labels. Check the disagreements instead of trusting its score.
Keep a holdout set. Use some cases while changing the prompt, but keep others out of that loop so you can see whether the improvement generalises.
Run the set after every prompt or model change. Keep old failures in it, so a fix never quietly brings an old bug back.

You don't need an evaluation platform to start. My demo is a JSONL file, two graders and a loop. The TypeScript runner has no package dependencies; a real model run still needs a provider API key and has to respect that provider's limits.

If you do want a tool, OpenAI's migration guide points to Promptfoo. In March 2026 Promptfoo announced it had agreed to be acquired by OpenAI and said it would remain open source. Inspect AI is another open source option, developed by the UK AI Security Institute and Meridian Labs.

And this isn't just my concern. In LangChain's State of Agent Engineering survey, 52.4% of organisations surveyed reported running offline evaluations on test sets. It's a useful signal about the teams in that survey, not a measurement of every company building AI.

What this demo doesn't prove

These are 24 synthetic cases, labelled by me, run three times on one model. They show how two prompts behaved on this particular set. They don't prove that v2 is better on real conversations, on another model or on a different task. My labels are judgement calls too: you could argue that a broken link really is waiting on me, and the judge did.

The judge results have a narrower meaning still. It only judges returned findings, so it cannot say anything about missed items. This is a small check of where the judge agrees with my labels, not a measure of recall or a production quality score.

The point isn't that v2 is the best prompt. It's that with a fixed set I can see which cases it fixed, which one it still gets wrong and which one it broke once. Then I can argue about specific cases instead of a feeling that a prompt is better.

Vibes are how you find what to test. Evals are how you know what changed.