DEV Community: juan pablo hernández

I have been Vibecoding Evals (works better than I thought)

juan pablo hernández — Mon, 03 Aug 2026 20:38:30 +0000

I’ve been building AI apps with coding agents for a while.

Lately, I’ve been experimenting with evals too.

The app in this example mostly worked. That was the problem.

The bug

I built a small support-triage app for a fictional shipment-tracking company.

A customer sends a support ticket, and the app decides what it is about, how urgent it is, and whether a human needs to respond.

A real outage should be escalated.

But this ticket was different:

“URGENT need key rotation now”

The customer was asking how to rotate their own API key before a security review.

The app classified it as a security incident and escalated it to a human.

That was wrong. The policy said normal key rotation was a self-service how-to request.

Nothing crashed. The app returned valid JSON. The fields all contained allowed values.

The behavior was still wrong.

Why clicking around wasn’t enough

I could test a few tickets manually and convince myself the app worked.

But after changing the prompt, what would I actually know?

Would the outage case still escalate?

Would normal how-to questions stay in the normal queue?

Would another API-key question behave differently?

I didn’t want to change the prompt and simply hope for the best.

I wanted a set of cases I could run again.

Adding DeepEval with Cursor

I installed the DeepEval agent skill:

npx skills add confident-ai/deepeval --skill "deepeval"
Then I asked Cursor to add evals to the app:

This app sometimes treats normal support questions like emergencies and sends
them to a human.
Add DeepEval so I can test this using the tickets and policy already in the repo.
I am new to evals, so use the simplest setup DeepEval already provides, explain
what you create, and ask me anything you need.
Run the app as it is first and show me what fails. Do not fix it yet.

Cursor already had the app, tickets, and policy, so it went straight to creating the baseline.

Goldens are the checklist

The first useful artifact was a JSON dataset.

Each golden contained:

the customer message
the expected category
the expected priority
whether a human should be involved
For example:

{
  "input": "URGENT need key rotation now",
  "expected": {
    "category": "how-to",
    "priority": "P3",
    "needs_human": false
  }
}

The app does not learn from these examples.

The dataset is a checklist. It tells me which behaviors need to keep working before I ship a prompt update.

That was the part that made evals click for me.

The metrics

Cursor used built-in DeepEval metrics to check two different things.

The first compared the app’s routing fields with the expected values in the golden:

category
priority
human escalation The second checked whether the decision followed the escalation rules in the policy.

So I could ask two separate questions:

Did the app return the expected decision?
Does that decision make sense under the policy?

The baseline failure

The first run caught the key-rotation case.

The app returned:

category: security
priority: P1
needs_human: true

The expected result was:

category: how-to
priority: P3
needs_human: false

The output was valid. The behavior was not.

The policy check also gave Cursor a reason explaining why the decision did not fit the support rules.

That was already more useful than changing the prompt, clicking the same ticket again, and hoping I hadn’t broken something else.

Fixing the prompt

The failure pointed back to an over-broad instruction in the triage prompt.

It was treating anything involving API keys as a security incident.

That was too aggressive.

The fix was to reserve escalation for actual compromise, suspicious access, or an outage.

Then Cursor reran the same test cases.

The key-rotation case passed.

The real outage case still escalated.

That second part matters. Fixing one case is not enough. I also want to know that I didn’t break a case that was already working.

The loop

This is the workflow I’ve been using:

Build the app
Create test cases
Run the app
Read the failure
Fix the prompt
Run the same cases again

Without evals, I would change the prompt and hope I didn’t create three new bugs.

With evals, every prompt change has a checklist behind it.

I recorded the full workflow here:

Watch the video: https://x.com/juampitech/status/2084330588593385637?s=20

DeepEval is open source and runs locally. You don’t need a Confident AI account for this workflow. LLM-as-a-judge metrics require a model API key.

DeepEval Vibe Coder quickstart:

https://deepeval.com/docs/vibe-coder-quickstart

What the F*** Is a Marketing Engineer?

juan pablo hernández — Fri, 17 Jul 2026 15:16:54 +0000

Everyone is hiring for one. Nobody can agree on what it is.

Let’s fix that.

Open LinkedIn and look at the job titles for a second:

Marketing Engineer
Growth Engineer
Developer Relations
Developer Growth
Engineer Marketer

Each one has a clean job description and a salary band.

Each one also quietly overlaps with the others.

Recruiters swap them around. Founders argue about which one they actually need. Candidates rewrite their headlines based on whoever they are talking to that week.

So, what the f*** is a marketing engineer?

It sounds like another buzzword.

Tech loves to rename things and act like it invented them. But something real is happening beneath the title soup, and the people who understand it early are about to become very valuable hires.

Here is the simple version:

A marketing engineer is a builder who reports to the business side instead of the CTO.

That is the whole idea.

The rest is just a question of where they point.

What a Marketing Engineer Actually Is

Drop the branding and the role is simple.

A marketing engineer sits inside marketing and turns boring, manual, repetitive work into systems.

Not someone who is simply good at using AI tools.

Someone who builds things that did not exist before.

The old marketing job was to run campaigns.

This job is to build the machine that runs them, so humans can spend their time on judgment instead of copying and pasting.

Think:

Content pipelines
Attribution that is not held together with tape and a shared spreadsheet
SEO and automated answer workflows
Internal marketing tools
Systems that continue working while everyone sleeps

The output is not “a nicer dashboard.”

It is a system that does the work.

The metric is leverage.

One marketing engineer should make the entire team faster, and you should feel that impact in the pipeline.

The Family Tree

Now the fun part.

The marketing engineer has a lot of cousins, and the gaps between them are smaller than the people defending each title would like you to think.

Two questions sort the whole family out:

Do you face inward, building for your own team, or outward, facing customers and developers?
Where in the funnel does your work live?

Answer those two questions and everyone starts to come into focus.

GTM Engineer

The closest sibling.

A GTM engineer builds the revenue machine.

Think:

Enrichment pipelines
CRM setup
Outbound automation
Lead scoring
Glue code between tools that refuse to talk to each other

Clay’s community coined the term in 2023, and it grew rapidly throughout 2024 and 2025.

Roughly 100 listings now go live every month at companies such as Cursor, Lovable, Webflow, and OpenAI.

A marketing engineer generally lives at the top of the funnel.

A GTM engineer works across the revenue side.

It is a similar skill set with a more sales-led focus. They usually report to the CRO rather than the CMO, and they almost never interact directly with customers.

Growth Engineer

Same DNA. Wider lens.

A growth engineer works across the entire journey, not only the front door:

Acquisition
Activation
Retention
Referral
Revenue

The whole loop.

They usually have access to product data and experimentation tools that a marketing engineer might not, so their work sits closer to the product itself.

Can only hire one person and want the widest possible range?

This is probably the hire.

A good growth engineer can usually do marketing engineering work.

It does not always go the other way.

Developer Relations

The oldest cousin.

Also the one having the loudest identity crisis.

DevRel was built around a simple trade:

Earn developer trust and drive adoption.

For years, that meant:

Tutorials
Talks
Documentation
Sample applications
Managing a Discord community

Then AI ate the easy half of the job.

More than half of professional developers now use AI tools every day, according to Stack Overflow.

Developers ask a model their syntax questions instead of reading your “Hello World” tutorial.

The role is drifting toward the harder work:

Community strategy
Building real relationships
Turning developer feedback into product decisions
Helping developers become successful with the product

There is also a completely new problem:

Making your documentation and APIs readable by the AI agents using them.

Your documentation quietly became an API for machines, whether you planned for it or not.

Developer Growth

The devtools cousin.

It has the same instinct to build systems, but those systems are aimed at developers instead of traditional buyers.

The audience is what makes the role different.

Developers can smell marketing from a mile away and quickly tune it out.

That means the work is less about campaigns and more hands-on:

Documentation that actually converts
Sample applications
Free tools
Technical content
Onboarding that gets someone to their first win before they lose interest

Developer Growth exists at companies where “technical” and “marketing” are no longer treated as completely separate things.

Everybody ships.

A Developer Growth hire simply owns more of how the product reaches the people who will use it.

The market keeps inventing names for this person because the old org chart had no box for someone who is half engineer, half marketer, and expected to be great at both.

See the pattern?

The titles disagree on scope, reporting lines, and whether the person interacts with customers.

But they all agree on the verb.

The verb is build.

Spot the manual workflow.

Turn it into a system.

Measure it.

Do it again.

Call yourself a Growth Engineer, Marketing Engineer, or a marketer who got tired of waiting in the engineering queue.

That loop is the job.

Engineer vs. Operator

One distinction matters:

Engineer versus operator.

Revenue Operations and Marketing Operations usually run the systems that already exist.

They keep the machine working, clean up processes, manage tools, and make sure the team follows the playbook.

The engineer builds something the company did not have before.

Build versus run.

A great operator who cannot code will struggle with the building work.

A great engineer who hates writing process documentation will struggle with operations.

At a small company, one person often does both.

By Series B, you usually split them.

Why Now and Not Five Years Ago?

None of these roles made as much sense five years ago.

A marketer who wanted custom software filed a ticket and waited an entire quarter.

The ticket usually died in a backlog because it was never quite important enough to beat the product roadmap.

The gap between “I have an idea” and “the idea is live” was measured in sprints you did not control.

That gap collapsed.

The cost of turning a clear description into working software fell off a cliff.

And it fell fastest for the people who are good at describing things.

Vibe coding is the most obvious example.

Andrej Karpathy coined the term in February 2025.

You describe what you want in plain language, and a model helps build it.

Tools such as:

Lovable
Bolt
v0
Cursor
Claude Code
Replit

Here is the statistic that should make a CMO pay attention:

Depending on the survey, around 63% of people building with these tools have no engineering background.

This stopped being only a developer trend.

It quietly became marketing infrastructure.

And here is the funny part:

Marketers were accidentally trained for this throughout their entire careers.

The core skill of marketing is briefing.

Describe an outcome clearly enough that someone else can build it.

Explain:

What the page should do
Who it is for
What action they should take
Why the experience matters

That is exactly the kind of input an AI coding tool wants.

Marketers did not learn a completely new skill.

They pointed an old one at a keyboard.

The market is already adjusting.

Renewals for single-function tools reportedly dropped by roughly one-third year over year.

Why pay a vendor for one narrow feature when you can build something fitted to your exact workflow in an afternoon?

The question changed from:

“Which tool should I buy?”

To:

“Should I just build this?”

For many teams, the honest answer is now “build it.”

The person who can do that is the marketing engineer.

The Distribution Era

Here is the real reason this role is not going away.

It has nothing to do with job titles.

When AI made building cheap, it also made something much scarier cheap:

Copying.

A product feature used to buy you months of lead time.

Now it might buy you a few days before a fast, AI-native team ships something similar.

So the moat moved.

It walked away from the product and toward the audience.

Call it the distribution era.

When everyone can build, the scarce thing is no longer the ability to build.

The scarce thing is getting anyone to use what you built.

The real edge becomes:

Distribution
Brand
Trust
An audience that opens your emails
Relationships
First-party data

A competitor cannot rebuild those things over a weekend.

But not all distribution counts.

Rented reach is not a moat.

It is a bill that becomes more expensive as demand increases.

The same AI wave is attacking rented distribution from the other side, as answer engines and buying agents position themselves between companies and their customers.

The moat is the distribution nobody can take away from you:

Owned audience
Earned authority
First-party data
Relationships that do not reset every quarter

Now put the two trends next to each other:

Building became cheaper, so distribution became more valuable.
Building became easier, so marketing teams can now build.

The role sitting directly at that intersection is the marketing engineer.

Technical enough to ship a system.

Commercial enough to know which system is worth shipping.

The title simply follows the shift.

“Your GTM is your product” stops being a LinkedIn line and starts becoming an org chart decision.

There Are Some Problems, Though

This would be a bad guide if it only sold the upside.

The titles are genuinely confusing, and that confusion has a cost.

Good candidates get filtered out because they used the wrong keyword.

Hiring managers post a “Marketing Engineer” role when they actually need a Revenue Operations specialist.

Pick the label that matches the real center of gravity of the role.

Not the one that sounds the coolest.

The same overreach affects the work itself.

“Do everything” roles usually break down, especially when you combine a customer-facing role with a quiet, build-focused one.

The incentives fight each other, and one side eventually gets ignored.

When you must combine them, choose the primary focus and treat everything else as a bonus.

There is also a quieter risk.

Vibe-coded systems are still code.

They can contain:

Hardcoded passwords
Weak input validation
Security vulnerabilities
Silent data leaks
Broken permissions

The difference is that the person building the system might not know what to look for.

Trusting AI-generated work simply because it came from a friendly chat window is how you end up with a security incident attached to your name.

The teams that win will keep a strong review habit around the new speed.

So, What the F*** Is a Marketing Engineer?

It is not a brand-new species.

It is what happens when an old itch finally gets access to cheap, fast tools.

That itch is the desire to stop doing the same manual task repeatedly and build a machine that does it instead.

The cousins are all driven by the same instinct:

GTM Engineer
Growth Engineer
Developer Relations
Developer Growth

They simply point that instinct at different parts of the funnel and different audiences.

The companies that understand this shift are no longer asking whether the title is real.

They are asking a sharper question:

Now that anyone with taste and a clear brief can build, who on the team is actually doing it?

And:

Is the thing they are building something a competitor could not copy by Friday?

More and more, the answer to both questions is a marketing engineer.

Whatever the job posting happens to call them.

The phrase will probably age badly.

The shift underneath it will not.