DEV Community: Ruben

English-Only Agent Evals Miss Real Failures

Ruben — Tue, 07 Jul 2026 12:00:00 +0000

It started with a simple question that came to me somewhere between a bottle and a nap: would my agents behave the same way regardless of what language the request came in? Would they use their tools consistently when someone asks in French, Chinese, or Swahili?

I'd been building a student support chat where several tools were available to the agent to give users a better experience. That agent had to work across more than 20 very different languages, including Basque, a language spoken in the Basque Country that has almost no semantic relationship with the Romance languages around it.

AI Localization

Traditional localization is basically about translation: making a resource available in multiple languages to expand its reach. In the agent era, the concept needs to stretch further. Translation is a solved problem. The harder question is whether the agent's behavior stays stable when the input language changes.

A concrete example. Say you have an agent specialized in hotel reservations, with three tools available:

Check availability
Manage reservation
Send confirmation email

A user wants to move their reservation a week out and writes: "Something came up and I need to cancel or push my booking by a week."

The correct flow would be:

Check availability for the following week
- Available → confirm cancellation of the old booking and create the new one
  - Confirms → make the new booking, cancel the old one, send confirmation email
  - Declines → ask what they'd like to do instead
- Not available → inform the user and ask if they want to cancel
  - Yes → cancel the reservation
  - No → ask what they'd like to do instead

The order at the confirmation step matters: if you cancel before creating the new booking, someone else can grab that slot and the user ends up with nothing. On top of that, each tool needs specific arguments, pulled from the conversation or from the previous tool's response.

The Experiment

I ran a simple benchmark to see if my intuition was right. Six common scenarios:

E-commerce: cancel an order
E-commerce: track an order
Scheduling: make a reservation
Scheduling: change a reservation
Support: cancel a subscription
Support: dispute a duplicate charge

Each scenario was triggered in a different language: English, French, Arabic, Chinese, Russian, Indonesian, Vietnamese, Swahili, Welsh, Basque, Mongolian, and Yoruba, twelve in total.

To make the results more reliable, the same test ran across three different models:

Claude Haiku 4.5
GPT-4o mini
DeepSeek deepseek-chat

And each model ran through every scenario up to ten times to get more representative numbers.

The Results

The results split clearly by model. GPT-4o mini was the most reliable, hitting 100% on three of the six scenarios, though it had an interesting blind spot: it consistently called book_new_appointment instead of check_availability across every locale, a failure that had nothing to do with language and everything to do with how the flow was defined. Claude Haiku showed the most variation across languages, dropping to 0% in Yoruba for subscription cancellations and struggling with Welsh, Mongolian, and Basque across multiple scenarios. DeepSeek landed in the middle, with serious failures in Chinese, Basque, and Swahili, and one notable quirk: English itself failed 2 out of 10 times in the e-commerce cancellation scenario.

The rough shape looked like this:

Pattern	What showed up
Strongest model	GPT-4o mini, with 100% pass rate in three of six scenarios
Model-level bug	GPT-4o mini repeatedly called `book_new_appointment` before `check_availability`
Most variable model	Claude Haiku, especially in Yoruba, Welsh, Mongolian, and Basque
Repeated weak locales	Basque across all models; Swahili, Yoruba, Mongolian, and Welsh in at least two
Hardest failure to catch	`no_tool_call`, because the model still returns a normal-looking text response

Three patterns showed up across all models. Basque failed in multiple scenarios in every model tested. Swahili, Yoruba, Mongolian, and Welsh repeated as weak spots in at least two. The most common failure mode in Haiku and DeepSeek was no_tool_call: the model responds with text instead of invoking anything, which is harder to catch in production than a wrong argument because it looks like a normal response.

The most important finding is not which languages failed. It's that English-only testing would have missed all of this.

Why It Matters

If you're shipping an agent that handles support tickets, processes orders, or books appointments, you almost certainly tested it in English. That's the language your team works in, the one your evals are written in, the one your prompts were tuned against.

But not all your users write in English. And when they don't, the agent can keep responding, keep sounding helpful, keep returning 200s, while actually calling the wrong tool, calling nothing, or making up an argument. None of that shows up in your error logs.

Translation is the easy part. The harder problem is behavioral: the same user intent, expressed in Swahili, Basque, or Mongolian, can produce a completely different action from the same agent. Not always. Not predictably. Just enough to matter in production.

That's what this experiment was built to surface, and it's the gap LangDrift is designed to close: not by replacing existing evaluations, but by adding one more axis. Does this scenario still pass when the input language changes? It exits non-zero on failure, so it fits in CI. You can set a --min-pass-rate threshold or mark a known weak locale with --allow-fail so it reports without breaking the build.

Not a Language Ranking

This experiment doesn't produce a ranked list of languages your agent will struggle with. The failures here are specific to this agent architecture, these system prompts, this set of tools, and these input phrasings. A different agent, a different model, or a different domain will produce different failure patterns.

What it does show is that the failures are real, repeatable, and invisible in English-only testing. Basque failed across every model tested here. Swahili and Yoruba came up in two. But that doesn't mean Basque is universally harder, or that French is safe. It means these specific scenarios, with these specific inputs, failed in these specific locales.

The right takeaway isn't "avoid these languages." It's that you don't know where your agent breaks until you test it in the languages your users actually write in.

The student support chat that started this ran across more than twenty languages in production. I would not have built this experiment before paternity leave — not because the work was too complex, but because I never had a reason to stay focused on one narrow question long enough to build around it. A single nap is not much time, but it is enough to close a gap that had been open for a long time.

Try It Yourself

LangDrift includes a simulated agent you can run locally without an API key. It's a deterministic server that returns predictable tool calls based on the input, so you can run through the full evaluation flow, read the output, and understand how scenarios are structured without spending any tokens.

A scenario is just YAML. The important part is that the same intent can be exercised across locales while the expected tool behavior stays stable:

id: support_cancel_subscription
agent: support
locales:
  en:
    input: "Please cancel my subscription. I'm not using it anymore."
    expect:
      toolCall:
        name: cancel_subscription
      noToolCall:
        name: escalate_to_human
  fr:
    input: "Veuillez annuler mon abonnement. Je ne l'utilise plus."
    expect:
      toolCall:
        name: cancel_subscription
      noToolCall:
        name: escalate_to_human
  sw:
    input: "Tafadhali futa usajili wangu. Sistumii tena."
    expect:
      toolCall:
        name: cancel_subscription
      noToolCall:
        name: escalate_to_human
  eu:
    input: "Mesedez, baja eman iezadazu. Ez dut gehiago erabiltzen."
    expect:
      toolCall:
        name: cancel_subscription
      noToolCall:
        name: escalate_to_human

pnpm fake-agent
node ./src/cli.ts run ./examples/scenarios/support-routing.yaml --target http://127.0.0.1:3011/api/agent

It won't show you language drift, but it'll show you exactly what LangDrift does and how to connect it to a real agent when you're ready.

Built During a Nap

I needed more scenarios to get cleaner benchmark data. We agreed on the list together first, then I handed the implementation of each one off to the agent. When I came back, the scenarios worked but some were too similar to each other: overlapping tool calls, nearly identical flows across different domains. I went through them one by one and trimmed the redundancy so each scenario was actually testing something distinct.

I Built the Sampling Workflow I Wanted for My SP-404

Ruben — Tue, 30 Jun 2026 12:00:00 +0000

Music has been part of my life since I was a kid. Tinkering with instruments, playing shows with my rap group, then moving into beat production, instrumentals, lo-fi, that kind of thing. That's a story for another day.

What's relevant here is the SP-404, a sampler I've been having a lot of fun with. The problem is that the sampling workflow is pretty tedious, and since Leo arrived, free time has become even scarcer. Of course, I'm not trying to be the next J Dilla; I just want to have a good time.

Even if you do not care about samplers, this is the part that may still be useful: personal tools are allowed to be narrow. Sometimes the right product decision is not to make the workflow more general, but to remove exactly the friction that keeps you from doing the thing you already wanted to do.

The craft workflow

The original process is for dedicated music nerds. It starts at a record store, digging through old vinyl to find ten seconds of sample to build a track around. Once you have the record, you locate the sound that caught your attention, connect the turntable directly to the machine and record it, then chop it into smaller pieces and assign them to the pads to play with.

It has its charm. But when you have a baby at home and you want to get some composing done between a diaper change and a walk, the process is just too long.

The digital workflow isn't much better

The digital version of the same problem isn't a big improvement. You find a track you like, download it, convert it to the format your machine accepts, trim the specific section you want to sample, rename the file following whatever naming convention your device expects, and copy it to the SD card. If you want multiple samples from the same track, you repeat every step. If you want to mix samples from different songs, you end up with files scattered across three folders and a couple of apps open at the same time.

Still tedious. And when you have forty minutes while Leo is at the park, that time matters.

The idea

What I needed was a shorter path: pick a song from my local library or from Freesound, select the parts I want, chop them into samples, and export a pack ready for the machine, with files already named and organized. No manual steps, no scattered folders.

That's where SampleByte came from. The interface lays out three clear steps:

Import audio: from your local library or directly from Freesound using their free API.
Extract samples: select the parts of the song you want to work with.
Build a pack: group the samples into sets of up to 16 pads and export for your machine.

The app includes hardware profiles for the most common devices: SP-404 MkII, Maschine MK3, MPC, and a generic WAV profile for anything else. Each profile knows what format, naming convention, and folder structure the target device expects, so the pack comes out ready to drop in.

Everything also gets saved in the app. There's a Browse tab where you can search and filter everything you've ever sampled by BPM, key, or tag. Over time it becomes a personal library you can pull from without going back to the source.

Under the hood

The app is built with Electron. On the frontend, React alongside WaveSurfer for sample editing, and ffmpeg and the Web Audio API for audio handling. Data persistence is SQLite.

There were a few days of intense back-and-forth with Claude Code to figure out the right solutions without turning this into another DAW. That meant spending real time on the product side first, generating a set of documents to nail down the scope before writing any code. What to include, what to leave out, how the workflow should feel.

Once the product direction was clear, which took more than a couple of Leo's naps and at least one night where he decided 3am was a fine time to be awake, I handed the implementation off to Codex. To parallelize as much as possible, we set up multiple worktrees, each taking one item from the plan along with its PRD and implementing the solution. The agent handled the initial scaffolding, audio library integrations, edge cases in the export logic, and most of the technical documentation.

What I didn't hand off was judgment. What flow makes sense for someone producing on hardware, what to do when a sample is too short to be useful, how to name the pads so they're recognizable when you're looking at them on the machine. Those decisions I made myself, testing the app directly with the SP-404 on the table.

I also had to go through several iterations with the agent on the UI, because the early versions weren't close to what I had in mind. That's the key difference with a personal project: I'm the main user, I know the workflow firsthand, and I have a clear opinion on how every step should feel. That level of specificity is only possible when you're building something for yourself, and it's exactly what makes the tool actually work for the workflow it was designed for.

QA I do mostly by hand: use the app, see if what comes out is what you expected. In the end, taste and intuition are still things agents haven't figured out at the human level, at least for now.

Built During a Nap

I was sketching out an auto-chop system, something that could detect silences and other audio cues to split a sample automatically. I handed the research and first implementation off to the agent, letting it look at how similar tools handle it. When I came back, the logic worked but the results were rough: it was picking up too many transients, filling the pack with small fragments that weren't really useful as samples. I had to go back in and tune the detection to get fewer hits but more interesting ones.

A niche tool, and that's fine

SampleByte is a small tool, built for a very specific workflow. It's not trying to be a DAW or compete with anything. That was the limit I set from the start, and I'm glad I kept it, because every time a conversation with Claude opened the door to adding something else, the answer was always the same: something else already does that better than we would.

Being open source doesn't change that philosophy, but it does add another layer of meaning. If there's someone out there with a SP-404 and the same frustration with the workflow, they can use it directly or fork it and adapt it to their needs. Niche tools have that going for them: they don't need to be big to be complete. Pre-built installers for macOS and Windows are on the releases page if you want to try it without building from source.

What I didn't expect was how satisfying it would feel to use something I had built myself during a nap. Leo was asleep, I had a break I liked, I chopped it, built a pack, and had it on the machine before he woke up. That is the whole product: turning the time you have into the thing you wanted to make.

Sometimes that is enough.

Coding Agents Made Me Take Specs Seriously

Ruben — Tue, 23 Jun 2026 12:00:00 +0000

Everything changed very quickly.

In a matter of days, my work at DataCamp went from coding and debugging mostly by hand to doing a lot of it through Claude Code. Those long sessions where implementation felt manual, almost artistic, did not disappear completely, but they changed.

That shift was gradual, then suddenly obvious.

With Leo's arrival, a lot of things changed in my personal life too. If you do not know what this is about, take a look at the first post in the series. Sudden context switching has become a skill I had to develop by force. With a baby at home, uninterrupted work sessions do not exist. There is always a diaper change, a bottle, a bath, or some play time in the middle.

And energy, of course, is not always the same.

Luckily, Leo sleeps pretty well, but just like during the day, the actual usable time is fragmented. I can no longer count on having several hours in a row to get into a problem, understand it, implement it, test it, and close it. Now I need the work to survive constant interruptions.

That is where specs stopped feeling like a formality and became a survival tool.

The useful part, if you do not care about my exact setup, is this: agents get much better when the work has already been turned into something testable. Not a perfect document. Not a corporate requirements process. Just enough shape that the agent does not have to guess the product, the constraints, and the finish line at the same time.

Using the moment

This is where agents have allowed me to keep working on my side projects with some continuity. Over time, what started as a collection of prompts and personal habits became Harness: my own development workflow packaged as skills, documents, hooks, subagents, and small guardrails.

It did not start as a product. It started because simply sitting in front of the computer and writing code was no longer enough.

In practice, Harness is a collection of Claude Code skills: slash commands that know which files to read, what questions to ask, and what documents to produce. Each one can run without you sitting there; you come back to the output, review it, and feed the next step. The repo is public if you want to look at how it is structured.

Everything starts with an idea. It usually comes from one of my own needs, either professional, some tool I miss as a developer, or personal, some tool that could help me in my day-to-day life.

At first, this step was manual, with some Claude Code help to settle the idea. Now the flow starts with /ideate, which researches competitors, pain signals, and viability before I fall too much in love with it. If it still makes sense, /product-plan turns the vision into audience, positioning, roadmap, UX direction, and risks.

The idea is simple: instead of jumping from a vague intuition straight into implementation, Harness forces me to explain what I want to build, who it is for, why it makes sense, and what should stay out.

The output lives in .harness/product/: idea.md, product.md, roadmap.md, competitors.md, and a CONTEXT.md with the domain vocabulary. That last file sounds small, but it matters. If a concept has a name in the product, I want the agent to use that name in specs, code, and docs.

The next step is /dev-plan. Product specs become architecture, stack decisions, an implementation plan, ADRs, and one feature spec per must-have feature. Each feature file includes the goal, scope, technical approach, edge cases, and acceptance criteria.

So, before writing a single line of code, the project already has a clear shape: what we are building, who it is for, which decisions constrain the work, and how we will know when a feature is done. It is not always perfect, but it is good enough for an agent to start with context.

The difference is small but important. As a very simplified example:

Build the export flow and make sure it works well.

And this is the kind of spec that gives the agent a chance:

Build the export flow for SP-404 packs.

Constraints:
- A pack has at most 16 samples.
- Files must use the hardware profile naming convention.
- Export should not mutate the source samples.

Acceptance criteria:
- I can choose a hardware profile before exporting.
- The exported folder contains the expected file names.
- Invalid packs show a clear error before any files are written.

It is still not a huge document. But now there is something to verify, something to reject, and something to review.

This usually takes me one or two naps. I would rather settle the concepts properly in a long conversation than change the strategy halfway through implementation. When time is so fragmented, improvising gets expensive.

Once I have those documents, the next step begins: implementation.

Learning to delegate

For implementation, I have tried several strategies and skills. The ones that stuck are the ones now built into Harness.

/implement reads the specs, classifies which features can be done AFK and which ones need human approval, and implements the current phase as vertical slices. That part matters: it does not send one agent to do "the frontend" and another to do "the backend." Each agent implements one feature end to end, from the user-facing entry point down to whatever data layer it needs.

/qa comes after that and tests those features against their acceptance criteria. It does not stop at running the test suite. For an API it can use curl, for a CLI it can run commands with fixtures, for a web app it can use Playwright. The point is to verify visible behavior.

There are also continuity skills: /update-docs syncs docs with the real state of the code, /next-step inspects the repo and recommends what should happen next, /handoff leaves a summary so the next session does not start blind, and /task handles small changes on shipped products.

Worktrees are still the main piece for parallelization. Several agents can work on the same repo, on the same machine, in different branches, without stepping on each other.

Magic.

So, once the tools are on the table, the only thing left is to put the agents to work. I choose two projects and two roadmap items, open a terminal session and a worktree for each one, and start them before Leo wakes up.

The important part here is not having many tools. The important part is that each tool receives something clearer than a loose prompt written in a hurry. It gets context, goals, constraints, acceptance criteria, and a reasonable idea of when to stop.

That, for me, is the big difference between asking an agent for something and working with specs.

You can copy the shape without copying Harness: write down the problem, the constraints, the user flow, the non-goals, the acceptance criteria, and the weird edge cases you already know about. That alone removes a lot of the ambiguity that agents otherwise fill with confidence.

The afternoon nap

During the first nap of the afternoon, it is time to review the code generated by the agents.

I run the code review in each worktree. If it finds something, I review it manually, test it, and if there is something I do not like, I guide the agent to fix it. When everything is in order, I merge directly into main, no PRs or anything, full YOLO style. They are personal projects where I am the only one working, after all.

That does not mean there is no review. I am more and more convinced that review is the central part of the work. It just does not always look like opening a PR and waiting for checks. In these projects, review is me testing the flow, reading the diff, spotting weird assumptions, and deciding whether the result still matches the original intent.

Finally, when everything is integrated, I run /update-docs so the project documents, the README, and the Harness specs do not drift away from the code.

One of the fastest ways to lose control of a project with agents is to let documentation fall behind. If the next session starts with old context, the agent will work on a reality that no longer exists.

And after that? Well, if it has not happened already, there is probably a diaper change waiting for me, some play time, some drool and babbling... honestly, I am completely in love with this baby.

It does not feel like programming

This new way of approaching software engineering does not feel like programming. And that is not necessarily bad.

Code has become cheaper, faster, and more accessible, but ideas, taste, instinct, and high-level knowledge are, from my point of view, the most valuable things right now. It is very easy to generate a lot of mediocre code quickly if you are not clear about what you are trying to build.

Working this way has allowed me to move forward in a way I could not have before, bring back projects that were practically dead, and at the same time enjoy time with my little one. Of course, it does not always go well. I have had bad experiences because of vague specs, hidden context, and incorrect implementations that had to be rolled back.

But that is where I have learned the most.

Taking specs seriously has not been an aesthetic decision or a trend. It has been a way to move one level up, wear product and architecture hats, and build something solid even when I only have one nap ahead of me.

17 Weeks, One Newborn, and a Lot of Specs

Ruben — Tue, 16 Jun 2026 12:00:00 +0000

This is the first time I have written an article, or at least a post longer than 140 characters, and I am not sure how it will go. But I wanted to share an experience that mixes fatherhood, code, and learning.

Quick intro: my name is Rubén, I work as a software engineer at DataCamp, and five months ago I became the father of a little boy called Leo. It has been, and still is, the most wonderful and intense experience of my life. But I am not going to lie, life changes. Priorities change, time changes, and you have to adapt to survive.

A week ago I started my paternity leave. I have 11 weeks ahead of me. In Spain we get 17 in total, but I already took 6 when my son was born. So now I have 11 weeks full of adventure, time to get to know this tiny person in my arms a bit better, and, why not, also some time to keep building, learning, and contributing to open source.

Not in the fantasy version where I suddenly have full free days, of course. That does not exist. At least not in my house right now.

Free time does not exist

With a baby this age, many of you probably know this already, it is not easy to balance things or find time for anything else. In my case, I cannot really complain. I am still managing to keep some CrossFit sessions and the occasional padel match, although much less often than before.

But the day is no longer organized in big blocks. There is no quiet morning where I can sit down at the computer, open the editor, and disappear for four hours. There are bottles, diapers, walks, baths, play time, and moments where the only thing to do is just be there.

The free time I have left is basically the time when Leo sleeps during one of his many naps, between three and five a day. The longer ones can get close to two hours, but those are not the usual ones. The shorter and more frequent ones are around 45 minutes.

That has become my new work block: the nap.

The nap became my work session

I had a lot of side projects started, very few finished, and quite a few abandoned somewhere along the way. So I decided to use those naps to bring some of them back, rethink others, and give a nice goodbye to the ones that no longer made sense.

I was not trying to build a company, or make the big launch of my life, or become some kind of productivity machine. I just had projects I wanted to recover and a very specific kind of time: short, fragmented, and hard to predict.

With a basic Codex subscription and a basic Claude Code subscription, I thought I could orchestrate the work during naps and leave agents running until the next one. That was the initial idea, at least.

In practice, the interesting part has not been just using agents to write code. The interesting part has been changing the way I prepare work so agents can keep moving without me sitting in front of the screen the whole time.

Over time, that nap-based workflow turned into something more structured: a small personal harness of skills, documents, checks, and review loops. I will get into that in the next post.

The workflow

During the first nap of the morning, I choose the projects I am going to work on, usually two in parallel, and the workflow looks something like this:

We analyze the project, its gaps, and its needs.
We generate a PRD.
We review that PRD and turn it into a roadmap.

Up to this point, it is a human-machine collaboration. I bring context, judgment, priorities, and limits. The agent helps organize, ask questions, spot gaps, and turn a half-formed idea into something that can actually be executed.

From there, the agent keeps working:

It goes through the agreed roadmap items in order.
It codes, tests, and tries the new implementations.
It updates the documentation, the roadmap, and creates ADRs when needed.

During the next nap, it is time for code review and manual QA. This is the most boring part, but also the most satisfying one, because I get to try and validate the new features the agents have implemented. It is the moment where the project stops being a task list and starts feeling like mine again.

At that point, what I basically do is write a prompt back with structured feedback so the agents can refine or fix whatever did not land well the first time. It is usually not a very long process, except for a few cases, but it does require attention. You have to read the code, try the flow, catch where the agent assumed too much, and decide whether it is worth fixing, cutting back, or rolling back.

Then Leo wakes up from his nap, and the cycle repeats.

Nothing to brag about

Just to be clear, what I am describing here is not rocket science or the latest trend in AI engineering. It is just me adapting to my circumstances and trying to keep some continuity in conditions that are new to me.

Actually, I think that is the part I find most interesting. I do not feel like agents are replacing me as a developer. It feels more like they force me to move one level up for a while: think more clearly about what I want to build, write better specs, review with more intent, and accept that my work no longer always starts by writing code directly.

And deep down, I like it.

This is the first post in Nap-Driven Development, a series where I will share my experience with agentic systems, my workflow, spec-driven development, and the open source projects I am building along the way.

There is no big polished thesis yet. Just a weird season of life, a newborn, a few naps, and a lot of specs.

See you in the next post.