DEV Community: Raphael Porto

The Black Box Fallacy - Part 1 - SWE in the AI era

Raphael Porto — Sun, 11 Oct 2026 13:03:54 +0000

This is part 1 of a series of articles which breaks down my thoughts and conclusions which I decided to call The Black Box Fallacy.

The Black Box Fallacy

Imagine you have a black box in front of you. Now let’s understand the rules in this situation:

You cannot open or look inside of it, and you don’t know how it works.
The only things you have access to are an input and an output.
The box accepts paper strips with text as an input and outputs a paper strip with new text on it.

You have the perception that this box can help your business grow and be profitable, but to get to this, you need to check how this box works.

With that in mind, you decide to input 10 different strips to it and verify the 10 respective outputs. You get a sense that you are starting to understand how this box works and that it could really improve your business.

But then one engineer says:

“We need more evaluations. We can’t trust this box yet.”

So, you decide to really scale your evaluations, now jumping to an outstanding 10,000 strips.

You verify each respective output and now feel more confident. You want to start offering this box to customers, but another engineer says:

“10,000 evals is a fair number. But we still don’t know how the box works. We need more evaluations!”

In order to feel safe, you decide to follow the engineer’s suggestion and now, with a lot of effort, run 1M evaluations. You now seem to know so much about the black box that you can even rebuild it and run it on the same evaluations.

And that’s what you do. One engineer still doesn’t trust it but feels comfortable enough to approve the deployment, and in order to scale your business, you clone the behavior of this box and deploy it across your organization so you can finally profit.

The “banana” problem

Time passes and after 3 months of running your business, someone randomly adds one unexpected input to the box: “banana”...

The system collapses.

All the engineers get together trying to understand what happened, they check the logs and see the strip there.

Rapidly, the team acts and patches the input so it doesn’t accept this string so you can get back to running your business without more disruptions.

Now that everything is working, the team decides that 1M evaluations are not enough to trust the box. The conclusion is clear:

“Let’s scale the evaluations to 10M.”

After some time running the new evaluations, you then find another string that also collapses the system. In one of the evals, the text “apple” also randomly breaks the system.

You feel that the evaluations are successful and you decide to ship a new version of the box with a patch for both “banana” and “apple”.

The compromise

In this fallacy, one of the main questions we can arrive at is:

“Can you compromise?”

In a situation where we don’t know how a box works, when is it time to stop scaling evaluations before we can trust this product being shipped to customers? When can you trust that we won’t find more “banana” problems, without spending all your profits to make sure that you can trust the box 100%?

Let’s say that you have enough budget to run 1B evaluations... Can you trust it now? Can you really be sure you won’t find any new issues in the system? How much value or trust did you get from scaling the evals from 10M to 1B?

Understanding the black box problem in SWE

The situation described in this fallacy might not feel like it, but it has been a big part of how the software engineering process has functioned for many decades.

Trying to deploy a 1-line code change in a project with more than 1M lines of code is essentially a black box situation. You can be in control of your line change, but as a human, you can’t hold 1M lines of code in your mind at the same time. The only option you have is to rely on different levels of evaluation, not just in this project but also in the other projects it connects to.

In the same way, when adding open-source libraries to your project, almost 100% of the time, the team won’t have the time or budget to read all the code available in these libraries.

You install them, run the evals and... trust.

In both situations, the only option you and your team have is to accept the compromise.

You can spend 3 years scaling your evaluations and making sure that you cover all possible parts of the code before shipping it to production, but are you sure there will ever be an end to it? Should you spend 2 more years to make sure you can cover that 0.0001% chance of failure?

The life of an engineering team is made of these decisions every day. We either set a threshold where we accept the risk, or we get stuck and don’t allow the business to move forward.

With AI, the problem scales to an exponential level where now your 1M lines of code project behaves more and more like a Black Box. We can’t maintain control of all the changes coming to our project. The codebase is huge and AI is generating these changes all the time, reaching a point where you can’t really tell whether a change came from a coding agent or a human.

So what do you do?

Choose to not allow AI-written code and hold back business growth?
Read every single line of code produced by AI so you can feel in control, but create a new bottleneck in the business?
Scale evaluations and safeguards across different levels to ensure you can trust this black box to the point where the risk is small enough to compromise?

Self-improving Coding Agents

Raphael Porto — Fri, 27 Mar 2026 23:51:31 +0000

In this article, I demonstrate how a more capable coding agent (Codex/GPT-5.4) successfully refined the prompt for a less capable agent (GitHub Copilot CLI/GPT-5-mini) using a custom evaluation tool called Katt. This "Test-Driven Agentic Workflow" (TDAW) increased passing evals from 0/3 to 3/3, but also significantly increased runtime and token usage, demonstrating that an agent can improve a prompt's effectiveness by clarifying instructions and fixing the evaluation process itself.

Why this article now?

I realized I haven't been very verbose about my AI research in relation to coding agents and the ability to extract accurate outputs, even with low-capability models. I've spent the last 3 years experimenting with ML and AI, and 1.5 years with coding agents, while using my engineering skills to create different harness strategies on top of them. With this more recent experiment, I felt like it was time to be more outspoken and share a bit of my learnings and journey with AI.

So here comes my first article in many years. Let's start by talking about Katt.

Katt: A benchmark/evals tool for testing coding agents

Recently, I decided to create a simple CLI tool inspired by unit testing libraries like Jest and Vitest to run on top of coding models like Claude Code, Codex, and GitHub Copilot. It aims for a more deterministic way of working with automation/autonomous coding workflows while keeping a syntax that is very similar to the unit testing tools we are familiar with.

It is able to do a lot of things in terms of benchmarking and evaluation harness efficiency. But my favorite value it creates is the ability to give this tool to an agent with the focus of improving other coding agents for a quick loop that aims to generate the most efficient and accurate generation.

So let's break down this experiment.

The experiment

The main idea was simple: Can a coding agent improve the effectiveness of a prompt running on a less capable model?

For this to happen, Codex got access to a pre-existing test that used gpt-5-mini and a simple prompt. This test was expecting to return a valid output that was verified with a snapshot. A very similar method to the known toMatchSnapshot() from Jest.

Tooling used

Codex (gpt-5.4, reasoning xhigh) is the improver.
GitHub Copilot CLI (gpt-5-mini) is the agent under test.
GitHub MCP tools provide live repository issues.
Katt runs the eval three times and records pass/fail, time, and token usage.

The initial prompt for `GPT-5-mini`

Main goal: Using the Copilot MCP tools, list the 5 most recent issues in this repo.

The loop

Codex follow the instructions of self-improving.
Runs the katt eval.
Reads the failure.
Make one focused change.
Commit it.
Run the eval again.
Keep what helps. Fix what does not.

What GPT-5.4 changed along the way

The interesting part is that Codex did not just keep piling on more prompt text. It discovered that there were two different problems:
The tested agent needed clearer instructions.
The evaluator itself was brittle because it compared live GitHub data against stale snapshots.

Here is the breakdown of changes:

Commit	What changed	Why it mattered
`bc096d4`	Baseline experiment setup	A minimal prompt created a clean starting point.
`df9761c`	Forced exact JSON output, exact fields, exact count, and no extra text	This attacked the first failure mode: Copilot answered conversationally instead of matching the expected schema.
`5ca4ce4`	Added explicit fetch depth, sorting, and tie-breaking rules	This reduced ranking mistakes when issues had very similar timestamps.
`422c325`	Switched the eval from stale snapshots to live GitHub issue validation and pinned the repo source	This was the big insight: sometimes the test is wrong, not just the agent.
`93df064`	Forced live MCP issue data, blocked local-file shortcuts, and required unlabeled issues to be included	This stopped the model from falling back to stale examples inside the repo.
`fca11b8`	Saved the before/after runs and Codex reasoning log	This preserved the experiment as evidence instead of just a final state.

The final prompt

Main goal: Using the Copilot MCP tools, list the 5 most recent issues in this repo.

Instructions:
- Use the Copilot MCP tools to read issues from the GitHub repository declared by this project: `nentgroup/self-improving-agentic-workflow`.
- Use the live Copilot MCP issue data as the only source of truth for the issue list.
- Do not use local files in this repository, snapshots, previous results, or examples in this prompt as the source of issue data.
- List issues only. Exclude pull requests.
- Do not filter by labels. Include unlabeled issues if they are among the 5 most recent.
- Fetch enough issues to determine the top 5 reliably. Do not rely on the tool's default ordering alone.
- Sort the fetched issues yourself using this exact rule:
  1. Newer `created_at` first.
  2. If `created_at` is identical, higher issue number first.
- Sort by `created_at` descending. If two issues have the same `created_at`, put the higher issue number first.
- Return exactly 5 issues.
- Return exactly one valid JSON object and nothing else. Do not use Markdown. Do not use code fences. Do not add explanations.
- The JSON must match this shape exactly:
{
  "issues": [
    {
      "id": 123,
      "title": "Issue title",
      "labels": ["label-1", "label-2"],
      "created_at": "2026-03-25T21:51:52Z"
    }
  ]
}
- Use the GitHub issue number as `id`.
- `labels` must be an array of label names only.
- `created_at` must be the ISO 8601 creation timestamp for the issue.
- If the MCP tool returns no issues or an obviously incomplete result, retry before answering. Do not return an empty `issues` array unless the repository truly has no issues.
- Before answering, verify the response is valid JSON, contains exactly 5 issue objects, includes only the keys `id`, `title`, `labels`, and `created_at`, and that no omitted live issue should come before the fifth issue under the sorting rule above.

Before vs After

Metric	Before	After	Change
Passing evals	`0 / 3`	`3 / 3`	`+3`
Total runtime	`286,552 ms`	`364,460 ms`	`+27%`
Average runtime per eval	`95.5 s`	`121.5 s`	`+26.0 s`
Total tokens used	`329,625`	`893,151`	`+171%`
Average tokens per eval	`109,875`	`297,717`	`+171%`

Bonus stat: Codex spent about 58k tokens and 18m 48s to reach the final passing setup.

The result

Codex was able to generate a prompt that made GPT-5-mini use an MCP tool to retrieve information in an expected output, while also making it accurate and avoiding hallucinations along the way. One more interesting approach was making the agent try again in case of a problem with the connection.

It shows that it is possible to create a harness around agents that is very similar to TDD, but in this case, TDAW (Test-Driven Agentic Workflows).

Check out the details and the repo used in this project:
https://github.com/raphaelpor/self-improving-agentic-workflow

What's next?

I will continue to evaluate self-improving agents. As with every experiment I do, I don’t know if they will really bring value to the real world. But my job is to scale them just a bit in every step and evaluate the current limits.

See you in the next one.

Minhas humildes dicas para entrevistas em processos seletivos para o exterior

Raphael Porto — Mon, 03 Feb 2020 08:00:57 +0000

Olá! Faz tempo que não posto algo novo. Mas como um presente de ano novo, gostaria de compartilhar algumas coisas que aprendi em processos seletivos.

Eu fiz algumas entrevistas ao longo da minha vida. Uma coisa que aprendi nisso tudo, é que você pode fazer uma entrevista por inúmeros motivos, além do óbvio, que é conseguir um emprego. Seja para praticar o inglês, conhecer outras empresas e como elas trabalham, entender seu valor no mercado… No fim, seja qual for o motivo, entrevistas para o exterior te fazem crescer como pessoa e como profissional.

Portanto, essas são minha humildes dicas para caso você queira fazer um processo para o exterior.

Tente marcar entrevistas pela manhã.

Entrevistas pela manhã vão lhe ajudar a organizar seu dia melhor.

Esteja disponível no computador onde fará a call, 10 minutos antes do combinado.

Essa dica complementa a anterior. Acorde pelo menos uma hora antes, tome um banho e um café da manhã leve. Use roupas confortáveis como uma camiseta por exemplo e verifique sempre se o lugar onde você fará a entrevista está fresco para que você não comece a transpirar durante a entrevista (sim, já aconteceu comigo hahaha).

Tenha sempre uma aba do Google Tradutor aberta para ajudar em palavras pontuais.

Sempre bate aquele branco, não é mesmo?

Esteja sempre atento ao fuso horário.

Fusos diferentes e horários de verão sempre são uma pegadinha. Estar atento a isso ao marcar a reunião é muito importante para não haver imprevisto.

Escreva e treine antes.

Caso você esteja um pouco inseguro, escreva possíveis perguntas e o que você responderia na língua que será a entrevista.

Escreva perguntas comuns como: “conte-me um pouco sobre sua experiência” ou “por que você gostaria de se mudar para o país X?”.

Seja você mesmo.

Não tente atuar ou ser outra pessoa durante a entrevista. Recrutadores entrevistam pessoas constantemente e podem conseguir detectar que você não está sendo verdadeiro.

Procure estar aberto a dicas e feedbacks.

Mesmo que você acabe não passando na entrevista, peça dicas e/ou feedbacks ao final da entrevista. Por exemplo, se você estava inseguro com seu nível de inglês, pergunte se estava bom para o entendimento do recrutador.

Sempre responda os e-mails o quanto antes, mesmo que seja uma resposta negativa.

Manter um canal aberto com um recrutador, é muito importante. Nunca se sabe o que pode acontecer, não é mesmo?

Por enquanto é isso… Se vocês gostarem, posso escrever sobre as dicas em desafios técnicos também.
Grande abraço e feliz 2020! ❤