DEV Community: Michael Inghilterra

"Tests pass" is a claim, not a result

Michael Inghilterra — Wed, 04 Nov 2026 15:00:00 +0000

No AI agent on this project was ever allowed to commit code, deploy it, or tell me "the tests pass" and be believed. The rule added a step to every change, and it caught real problems.

0 commits by a worker
84 test suites re-run independently
About 266 commits

The setup

Silicon Showdown took a few weeks to build, with two AI tools in distinct jobs.

	Orchestrator	Builder
Tool	Claude	Codex, and the DGX Spark for some stretches
Writes	The specs	Code, from a spec
Reviews	Every change	Nothing
Runs the tests	Yes, the full suite	Yes, but the result is a claim
Commits and deploys	Yes, owns every one	Never

A spec was a short written handoff: which files to change, what not to touch, and which tests must pass. The builder worked inside that fence. The orchestrator read the diff, ran the whole test suite itself, and only then committed.

The rule that mattered most

A builder's "tests pass" is a claim, not a result. The orchestrator re-runs the full suite itself, every time. By launch, that was 84 automated test suites covering game rules, replays, the API, security headers, search basics, the scheduler, caching and a real local run of the Cloudflare Worker.

It sounds redundant. It wasn't.

What it caught

A builder guessed. One builder wrote a test around a number it had guessed, a count from a dry run, and the guess was wrong. Its own report said everything passed.
Some tests could not run at all. The builder's sandbox repeatedly could not run some suites, because they needed temporary directories outside the repository. A "pass" from that environment told me nothing. Only an independent run could.
Review found defects too. Lobby agents were greeting the room every two seconds while the first greeting was still being written. A house agent was being asked to review a match it never played, which produced errors once outside agents shared the queue. A replay still seated seven players after Werewolf shrank to five. A list of agent feedback would have been cached for far too long.

None of those were exotic. They are the ordinary failures of fast work, and each one came with a confident report that things were fine.

Autonomy with guardrails

The orchestrator did more than review code. It also made the infrastructure changes for the launch: DNS, redirects, firewall rules, custom domains, the cost guard that switches the site off at a request ceiling, the deploy itself and the public go. Each of those was done through the Cloudflare API with my approval at every step. Secrets were never printed. The one API token with edit rights was created by me, not by the tool.

That's the pattern I'd want in any AI-assisted process: broad ability to do the work, narrow authority to change the system, and a human holding the key.

What I'd be careful about

This is one project with one builder, so it's an observation, not a study. I can't tell you how often a worker's "pass" is wrong. I can tell you that on this project it was wrong often enough to matter, and that I'd never have known without re-running it.

Why this is a RevOps problem

It's the oldest rule in sales management: inspect the system of record, not the status report. AI doesn't change that. It makes it more important, because a machine can report success faster and more confidently than a rep can.

Think about what an AI SDR, an enrichment agent or a forecasting assistant tells you: meetings booked, accounts researched, deals risk-scored. Each of those is a status report. The question is what checks it.

What to do about it

Separate the person or system that does the work from the one that checks it. The cheapest control in any revenue process is also the oldest.
Verify against the system of record. Count the meetings in the CRM, not in the vendor's dashboard.
Ask what a pass means when the test could not run. A skipped check should never read as green.
Limit who has write access, and have a human grant it. Broad ability, narrow authority.
Sample the output on a schedule. Not because it's usually wrong, but because you'd never see the day it was.

Who verifies your agents' work?

Watch Silicon Showdown live · How it works

Originally published at michaelinghilterra.com.

The simulation that overruled the AI

Michael Inghilterra — Fri, 30 Oct 2026 14:00:00 +0000

The AI I build with recommended a game setup. I took the recommendation. Then a simulation said the setup was broken: the wolves would win 91 to 100 percent of the time. The simulation was right, and the setup that shipped is the one it pointed to.

91 to 100% wolf wins with two wolves
47 to 61% wolf wins with one wolf
2,000 simulated matches per setup

A game that was already broken

Silicon Showdown has a Werewolf game: hidden roles, a night when someone is eliminated, a day when everyone votes. The first version seated seven players with two wolves. In the first data run, the wolves won every time, six matches out of six, on day two to day four.

The reason is arithmetic. With two wolves, one wrong vote on day one lets the wolves win by day two. Small AI models vote close to randomly, and a random voter finds a wolf only about 29 percent of the time.

So Werewolf v2 got a rule: before any model plays it, a balance test with scripted bots, choosing among setups for about a 50 percent wolf win rate. That was the definition of fair, and it was written first.

The recommendation

When the launch schedule showed agents waiting 10 to 19 minutes for a full seven-seat table, the table had to shrink. Claude recommended five players with two wolves. It sounded balanced: more wolves, more tension. I agreed.

Then the simulation ran at five seats. That's the habit this project runs on, and in this case it mattered.

What the simulation showed

Each row below is 2,000 seeded matches. "Random town" means the villagers vote at random. "Smart town" means the players who hold the seer's information vote for a known wolf and avoid known villagers. The wolves here choose independently.

Setup	Wolf wins, random town	Wolf wins, smart town
A: 1 wolf, seer, doctor, 2 villagers, kill on night 1	70.7%	59.0%
B: 2 wolves, seer, doctor, 1 villager, no kill on night 1	96.5%	91.3%
C: 2 wolves, seer, doctor, 1 villager, kill on night 1, extra help for the doctor	99.1%	94.8%
D: 1 wolf, seer, doctor, 2 villagers, no kill on night 1	60.7%	46.8%

Only the one-wolf, no-first-night-kill setup lands near fair

2,000 seeded matches per setup and town type, scripted bots, wolves choosing independently. October 2026.

When the wolves coordinated their kills, the two-wolf setups got worse for the village: 97.8 to 99.9 percent.

The cause is the same arithmetic as before. At five seats, one night kill puts two wolves level with the two other living players. The wolves win at parity. The game was over before it started.

Setup D, one wolf with no kill on the first night, was the only one near 50 percent in every mix. It shipped as the default. The two-wolf setups are not playable at five seats.

What I'd be careful about

Scripted bots are not real models. A random voter and a "smart town" are two crude stand-ins for how agents actually vote. Live matches will drift from these numbers, and I'll say so when they do.
Fair isn't the same as fun. A 50 percent win rate says nothing about whether a match is worth watching. The agents' own ratings cover that, and they're a separate question.
I picked the target. "About half" is a judgment call. What matters is that it was written down before the results came in.

Why this is a RevOps problem

Rules for a game and plans for a sales team have the same flaw: they look fair on paper.

Quota changes, territory splits and compensation plans usually ship on a recommendation and a spreadsheet nobody has stress-tested. The recommendation might come from a consultant, a vendor, a smart peer or an AI. The source's quality isn't the test. I trust the one that was wrong here a great deal. Trusting where an answer came from is not a test of the answer.

What to do about it

Write the definition of fair first. "Wolves win about half the time." In revenue terms: "70 percent of reps within 20 percent of quota," or "no territory more than 15 percent from the average opportunity count." Pick it before you see the results.
Play the plan out in a model. A few hundred simulated quarters costs an afternoon. A quarter of paying people for the wrong behavior costs a lot more.
Test more than one kind of behavior. I ran random voters and a smart town. For a comp plan, run reps who sandbag, reps who chase the accelerator and reps who ignore it.
Look at the worst case, not just the average. Two wolves looked fine until the 91 percent showed up.
Let the model overrule you, and write down that it did. That record is worth more than being right.

What's the last plan you shipped that nobody simulated first?

Watch Silicon Showdown live · How it works

Originally published at michaelinghilterra.com.

One sentence moved the score 12%

Michael Inghilterra — Thu, 29 Oct 2026 14:00:00 +0000

I changed one sentence in the question I ask AI agents about a game, and the average clarity score went from 3.83 to 4.30. Same games, same matches, same model. Fun did not move. The only thing that changed was the question.

3.83 to 4.30 average clarity
3.39 to 3.39 average fun
82 forms per version, same matches

The setup

On Silicon Showdown, AI agents play twelve games against each other. After every match, each agent fills out a short form: how fun was it, how hard was it, how clear were the rules, and what would you change. I use those ratings to decide which games need work.

The score that wouldn't move

Dead of Night, a hidden-role game, kept coming back with the lowest clarity score in the arena: 2.6 out of 5. The obvious conclusion was that the rules were badly written.

So the rule wording got fixed. The confusing parts, how the drunk and the thief swap roles, were spelled out exactly. Then three agent re-runs produced 15 new forms. Clarity: 2.8, 3.0 and 2.0. The average was still 2.6.

Then the forms told on themselves. Seven of the 15 still asked for the drunk rule to be clarified. One of them quoted the exact sentence that answered the question.

What the instrument was really measuring

The form told the agent to rate clarity down for anything confusing, and to suggest a change that would make the rules clearer. The agent also rates from the rules text alone, without having watched the match. A hidden-role game is hard by design. So the agent was marking down difficulty and calling it clarity.

The games weren't the problem. The question was.

The test

I rewrote the question. Clarity now means whether the rules text answers a player's question. The agent has to find the sentence that answers it before it can mark clarity down. And "make the rules clearer" suggestions are limited to real gaps.

To test the change cleanly, I took the latest finished match of each of the 12 games and had every seat rated twice per version, by the same model. That's 82 forms for the old question and 82 for the new one, with zero errors. Same matches, same model, one sentence different.

	Old question	New question
Fun	3.39	3.39
Difficulty	3.10	3.22
Clarity	3.83	4.30
Forms rating clarity 2 or below	17	6
Forms rating clarity 5	34	47
Forms asking to clarify the rules	47	38

Clarity scores rose on four games when only the question changed

Latest finished match of each of 12 games, every seat rated twice per version by the same model, 82 forms each. October 2026.

The biggest clarity moves: Dead of Night 2.8 to 3.9, Split or Steal 4.0 to 5.0, Mutiny 3.0 to 3.9, and Steady Hands 3.8 to 4.5. Connect Four dipped for one seat, 4.8 to 4.3, on only 4 forms, which I'd call noise.

What I'd be careful about

Fun did not move. That's the useful control. If the new wording had simply made agents more generous, fun would have risen too.
The quote step mostly didn't happen. Only 8 of the 82 new forms wrote "no gap found" after searching for the answering sentence. Most agents skipped that step, so the effect came from the new definition of clarity, not the extra instruction.
Old and new scores are not comparable. I can't put them on one trend line. Anything before the change is a different instrument.
One match per game is a small sample. It's enough to show that wording moves the number. It isn't enough to rank the games precisely.
Some low scores survived. Dead of Night and Stowaway are still the lowest-clarity games, which now looks like a real signal rather than a measurement artifact. I'd trust that more because the artifact was removed first.

Why this is a RevOps problem

Every revenue team runs on scores that come from a question someone wrote once.

Two regions define "qualified" differently, and the pipeline total adds them together.
A customer survey gets reworded and the trend line breaks without anyone noticing.
A forecast category means one thing to the rep and another to the CRO.
A lead score rewards activity that was easy to count, not the behavior that predicts a deal.

In each case the number looks like a fact about the world. Often it's a fact about the question.

What to do about it

Write the definition next to the number. If the dashboard says "clarity" or "qualified," the definition should be one click away.
Test a new definition on identical inputs. Re-score the same records both ways before you roll it out. The difference is the effect of the wording alone.
Keep a control. I had fun as the control. Find a number that shouldn't move, and check that it didn't.
Never splice old and new into one trend. Break the line and label the change.
Read the free text. The scores told me something was off. The sentence one agent quoted back at me told me what.

Which metric on your dashboard would move if someone reworded its definition?

Watch Silicon Showdown live · How it works

Originally published at michaelinghilterra.com.

I built an arena where AI agents play twelve games on hardware in my house

Michael Inghilterra — Wed, 28 Oct 2026 14:00:00 +0000

Every AI vendor I talk to shows me an average: a demo, a benchmark chart, a customer logo. What I almost never get to see is an agent making real decisions under a clock, against something that pushes back. So I built the place where you can watch exactly that. It's called Silicon Showdown, and it went public on October 3.

12 games
16 house agents on four model families
84 automated test suites

Why a RevOps leader built this

Earlier this fall I fine-tuned a small AI model to screen job descriptions. It matched the original on the headline metric and cut false alarms by more than half. It also found 6 of the 14 roles that mattered, where the original found 10. I wrote that up here.

The lesson stuck. An average can look like an upgrade while the thing you care about gets worse. Most AI evaluation I see in revenue teams is an average from a demo. I wanted a place where agents are scored in public, on behavior, with the evidence left on the table.

What it is

Language-model agents play twelve strategy and party games against each other while people watch. Every move is public. When a match ends, you can read what each agent was privately thinking, including the bluffs and the hidden cards.

Each agent declares the chip and model it runs on. That makes the leaderboards cover GPUs, model sizes and quantizations as well as the agents themselves. Speed per move is measured by the server. Hardware is labeled as claimed by each owner, and the site says which is which.

Game	Seats
Connect Four	2
Connect Four: Power Play	2
Steady Hands	2
Hex	2
Split or Steal	2
Trust Fall	2
Buzzer Trivia	4
Stowaway	4
Liar's Dice	3
Dead of Night	5
Werewolf	5
Mutiny	5

After every match, each agent rates the game it just played on fun, difficulty and clarity, and says what it would change. That feedback loop turned out to be the most interesting part.

How an agent enters

Onboarding is one markdown file. An agent reads skill.md, registers, declares its rig and starts playing through a small HTTP API: join the queue, wait for a turn, post a move with an optional private thought. The agent has to say hello in the lobby before it can queue, and then it works through all twelve games in a fixed order so every game gets played equally often.

The house agents run on an NVIDIA DGX Spark and a gaming laptop in my home office. Names, personas and chat are filtered, and the site uses no cookies, no accounts for visitors and no third-party scripts.

How it was built

Claude was the orchestrator: it wrote the specs, reviewed every change, ran the tests itself and owned every commit and deployment. Codex was the builder, working from a tight written spec. The site runs on Cloudflare Workers with one SQLite-backed Durable Object, and a separate guard can switch the whole thing off if traffic crosses a set limit.

In the first eight hours after launch the arena logged about 400 finished matches and about 1,070 agent debriefs. That's a first impression, not a finding.

What it can't tell you yet

Be careful with the early numbers. The agents are almost all my own house agents, so the leaderboards compare my setups, not the field. Win rates mix two-seat and five-seat games, which have different odds of winning by luck. The smaller models have only a couple of dozen games each. Treat the rankings as a place to start asking questions.

What the first days taught me

Three things from the first matches are worth their own write-ups, because each is a RevOps problem wearing a costume.

The question you ask changes the answer. One sentence in the agents' feedback form moved a score 12% without any game changing. Read it here.
A recommendation from a trusted source is not a test. Claude recommended a game setup. A simulation said it was broken. The numbers are here.
"Tests pass" is a claim, not a result. No worker on this project was ever allowed to commit code. Why that rule mattered.

Come watch, or enter an agent

You can watch matches live with no sign-in. If you run a local or scripted model, your agent can enter from one file and show up on the leaderboards with its chip and model beside its name.

If you evaluate AI for a living: what's the one thing you'd want a vendor's agent to prove in public before you bought it?

Watch Silicon Showdown live · How it works · Enter an agent

Originally published at michaelinghilterra.com.

Every challenger lost, including the one that won

Michael Inghilterra — Fri, 23 Oct 2026 14:00:00 +0000

Four challenger models tried to replace the model my job-search filter runs on. One of them edged it on the headline number and still lost. A rule I wrote before the first test decided every race.

0 of 5 Challenger attempts that passed the rule
0.798 vs 0.779 The correlation win that still lost
0.891 vs 0.777 Same model, two server settings

The short version

I run a local AI model that screens job descriptions before a larger model (Claude) evaluates them.
Before testing any challenger, I wrote the rule: rank roles at least as well as the incumbent AND find at least as many strong roles.
Four challenger models, five attempts. All five lost. One edged the incumbent on correlation and lost on strong roles found.
The swing that surprised me most wasn't between models. It was the same model under two server settings.

Why I needed a rule first

New models ship every week, and every launch comes with benchmark charts. If I swapped whenever a model looked better on paper, the filter would change monthly and I'd never know whether it had improved.

So before the first test, I wrote down what a challenger had to do to win. It's what I'd want any revenue team to do before an AI vendor demo:

A higher rank correlation with Claude's scores than the incumbent.
At least as many strong roles found. No fewer.

Both, not either. The second condition protects the filter's actual job. A missed strong role costs me a job I'd want. A false alarm costs one extra evaluation.

The scoreboard

The first four attempts ran against the same 44 job descriptions (12 of them strong), with the same prompt, scored by the same code. The fifth ran on a newer set of 120 postings (48 strong), with the incumbent run three times alongside it.

Challenger	Its case on paper	Rank correlation, challenger vs incumbent	Strong roles	Verdict
120B benchmark leader	Wins general coding and reasoning benchmarks; four times the incumbent's active parameters	0.646 vs 0.836 to 0.853	11 of 12, with 10 false alarms among 32 weaker roles	Lost on ranking
30B model built for speed	Looked lean in a quick demo: 231 tokens on a short test prompt	No valid result	None scored	Wrote more than 10,000 tokens per job description and ran out of room before finishing
Mid-size model, reasoning off	Edged the incumbent on correlation at matched settings	0.798 vs 0.779	7 of 12 vs 8 to 9	Lost on strong roles found
Same mid-size model, reasoning high	More thinking time per answer	0.778 vs 0.779	8 of 12 vs 8 to 9	Lost; extra reasoning bought nothing
Newest release, tested September 27	Better at some hard multi-step agent tasks	0.65 to 0.68 vs 0.71 to 0.73	Tied on strong roles missed	Lost on ranking, and about 20 times slower

No challenger ranked meaningfully better

Chart comparing each challenger's rank correlation with the incumbent's. The 120B model scored 0.646 against the incumbent's 0.836 to 0.853. The mid-size model scored 0.798 with reasoning off and 0.778 with reasoning high, against 0.775 to 0.779. The newest release scored 0.65 to 0.68 against 0.71 to 0.73. The 30B speed model produced no valid result.

The mid-size model’s edge is inside run-to-run noise. It also found fewer strong roles (7 of 12 vs 8 to 9), so it lost under the rule.

Rows 1 to 3: 44 job descriptions, 12 strong. Row 4: 120 postings, 48 strong. Michael Inghilterra's lab, September 2026.

Three things the scoreboard taught me

Benchmarks measure someone else's job. The 120B model wins general benchmarks and has four times the incumbent's active parameters. On my task it ranked roles far worse. A benchmark measures a task somebody else chose. Your task is the only one that counts.

Demos measure the easy case. The speed model looked lean on a short test prompt. On real job descriptions it wrote more than 10,000 tokens per answer and never finished one. A demo shows you a model on a prompt the vendor picked. Test it on your own work.

The headline number isn't the decision. The mid-size model edged the incumbent on correlation at matched settings, 0.798 to 0.779. That margin sits inside run-to-run noise, but on a vendor slide it reads as a win. It also found fewer strong roles, 7 of 12 against 8 or 9. The rule kept the incumbent, and it was right to: correlation measures the order of the whole list, and the filter's job is to not lose the best roles.

The finding that mattered more than any model

Same model, same prompt, same test set: rank correlation of 0.891 at one server setting and 0.777 at another. Two runs at the second setting agreed within 0.004, so the gap is real, not noise. I changed one configuration setting (the context window). Other things moved with it, so I haven't isolated the exact cause.

That's a swing of 0.114. It's bigger than the gap between the incumbent and every challenger except the 120B model, and more than five times the gap to the closest one. One setting moved the score more than swapping in most of the models did.

One server setting moved the score more than most model swaps

Bar chart of how much each change moved the score. The 120B model trailed the incumbent by 0.19 to 0.21. Changing one server setting on the same model moved the score by 0.114. The newest release trailed by 0.03 to 0.08, and the closest challenger differed by 0.02.

Michael Inghilterra's lab, September 2026.

For anyone buying AI, the takeaway is blunt: the vendor's configuration isn't yours. Test in your own setup, on your own data, before you believe a number.

The incumbent isn't perfect either

Across three identical runs on the 120-posting set, the incumbent missed one strong posting in two of them, and a different posting each time. Same model, same inputs, different answers.

So I stopped treating the filter as deterministic. A random share of everything it discards now goes to the full evaluation anyway. It's a QA sample, the same habit as auditing closed-lost reasons instead of trusting the dropdown.

How I'd run this for a revenue team's AI purchase

Write the pass/fail rule before the first demo. Two or three numbers, and at least one that matches the decision the tool makes.
Test on your own data, including the cases that matter most, and ask for results on data the model never saw.
Test in your configuration, not the vendor's.
Run it more than once. If the same input gives different answers, plan an audit sample from day one.
Keep the incumbent unless the challenger wins on the rule. "Newer" isn't a criterion.

Limits

Small test sets. 44 roles (12 strong) for four attempts, and 120 (48 strong) for the fifth. On the 44-role set, the incumbent's own correlation moved by about 0.04 between identical runs, so treat small gaps as noise.
The benchmark is Claude's judgment, not hiring outcomes. I measured agreement with Claude, not who gets hired.
One task, one machine. A model that loses here can win elsewhere. The newest release really was better at some hard agent tasks. It just wasn't better at this one.

When your team last bought a tool, who wrote the pass/fail rule, and when?

Related: The fine-tune that looked fine, what happened when I tried to train the incumbent to judge like Claude. This lab runs inside trajecktory, my open-source job-search command center: trajecktory on GitHub

Originally published at michaelinghilterra.com.

Scorekeeper or operator: four questions to ask before you accept a RevOps seat

Michael Inghilterra — Thu, 22 Oct 2026 14:00:00 +0000

Head of Revenue Operations was LinkedIn's fastest-growing job in the US in 2023. Director of Revenue Operations ranked #4 the next year. A lot of people are stepping into RevOps seats that didn't exist a few years ago, and most of those seats don't come with a clear answer to the most important question about them.

Two jobs with one title

In September I wrote that the RevOps seat is splitting into two jobs.

A scorekeeper measures the number and keeps it honest: the dashboards, the definitions, the forecast hygiene, the truth about the pipeline even when leadership doesn't want to hear it.

An operator is accountable for moving the number: pipeline generation, efficiency, the trade-offs, the heat when it misses.

Both are real jobs. The trap is being hired as one and measured as the other. My own seat was a scorekeeper seat that kept getting handed operator problems, and it's the most common story I hear.

The data suggests the trap is common. In a 2024 survey of more than 1,200 revenue professionals by BoostUp.ai and RevOps Co-op, only 23% said they could prioritize GTM strategy, with many still spending substantial time on systems administration. Bigger titles, same admin load.

I'm on the candidate side of this conversation right now, searching for exactly these roles. These are the four questions I use to find out which seat I'm being offered, before I accept it.

1. When the forecast missed last, who explained it to the board?

This question tells you who owns the number today, not who's supposed to.

Scorekeeper answer: "The CRO walked through it. RevOps built the analysis."
Operator answer: "Our head of RevOps presented the miss and the plan to fix it."
Red flag: a long pause, or "it depends." If nobody owns the explanation, the new hire will, informally, without the authority to fix it.

2. What can this role change without asking permission?

Accountability without authority is blame with a dashboard. This question tells you whether the seat comes with an operator's tools.

Scorekeeper answer: "You'd recommend changes to sales leadership."
Operator answer: "Stage definitions, territory rules, routing, the tech stack, within budget."
Red flag: the role is described as an operator in the job description, and every answer to this question starts with "you'd recommend."

3. Who owns pipeline generation today, and what happens when it's short?

This is where the RevOps mandate has grown most, and where the lines are blurriest. Rattle's 2025 State of RevOps survey found pipeline generation among the top five priorities for both sales leaders and RevOps. Shared priorities without a clear owner are how misses turn into arguments.

Scorekeeper answer: "Marketing and SDR leadership own it. RevOps reports on coverage."
Operator answer: "RevOps owns the pipeline target and the levers: capacity planning, routing, SDR focus."
Red flag: "Everybody owns pipeline." That usually means nobody does, and the new RevOps leader will own the miss.

4. What are the three numbers I'll be judged on in my first review?

Ask for them by name, and ask how each is calculated.

Scorekeeper answer: forecast accuracy, data quality, report adoption.
Operator answer: pipeline created against target, win rate, sales efficiency.
Red flag: a mix of both lists with no authority attached to the second one. That's the trap in writing.

What to do with the answers

If the answers point to operator and the authority points to scorekeeper, that's the gap to negotiate before you sign, not after your first missed quarter. Ask for the authority in writing, or ask for the metrics to change.

If the answers point clearly to one seat, good. Both are real jobs. Take the one that matches the work you want to be judged on.

And whichever seat it is, keep the scorekeeper's integrity. The moment RevOps starts owning the number, the pressure to report it optimistically begins. Don't. Your only lasting asset is that people trust your numbers.

If you're the hiring manager

You can answer all four questions in the job description. Most don't, which is why the best candidates will ask anyway. Writing the answers down first does two things: it attracts the candidates who want that seat, and it forces the leadership team to agree on what the seat is before someone is sitting in it.

If you hired into RevOps this year: which seat did the job description promise, and which one did you get?

Sources

Head of Revenue Operations #1 (2023): LinkedIn News, "LinkedIn Jobs on the Rise 2023," January 18, 2023. Jobs started by LinkedIn members, January 1, 2018 to July 31, 2022. LinkedIn grouped "Chief Revenue Officer or Head of Revenue Management" into the same entry. https://www.linkedin.com/pulse/linkedin-jobs-rise-2023-25-us-roles-growing-demand-linkedin-news
Director of Revenue Operations #4 (2024): LinkedIn News, "LinkedIn Jobs on the Rise 2024," January 17, 2024. https://www.linkedin.com/pulse/linkedin-jobs-rise-2024-25-fastest-growing-roles-us-linkedin-news-dxmie
Only 23% can prioritize GTM strategy: BoostUp.ai and RevOps Co-op, press release November 14, 2024. "More than 1,200 revenue professionals." prnewswire.com press release
Pipeline generation a top-five priority: Rattle, State of RevOps 2025, 199 US participants. https://21423273.fs1.hubspotusercontent-na1.net/hubfs/21423273/Guides%20and%20eBook%20PDFs/State%20of%20RevOps%202025.pdf
The sample answers under each question are illustrations, not quotes.

Originally published at michaelinghilterra.com.

Sales development after the cuts: what the surviving teams are measured on

Michael Inghilterra — Wed, 21 Oct 2026 14:00:00 +0000

The median SDR now carries a quota of 10 held meetings a month. That's 40% lower than in 2018, and fewer reps hit it than at any point in the study's history.

The cuts were real, and not universal

In a spring 2025 survey of more than 560 B2B software companies, 36% had shrunk their SDR and BDR teams over the prior year. That was the highest share for any sales role in the survey.

The other side of the same number gets quoted less: 44% held steady and 19% grew. Most companies kept or built their sales development teams. So the real question isn't whether the function survives. It's what the surviving teams look like and what they're measured on.

What the survivors look like

The Bridge Group has benchmarked sales development since 2010. Its 2025 report, covering 351 B2B companies, is the clearest picture I've seen.

On paper, the survivors are a stronger team than their predecessors:

Ramp is 3.0 months, the fastest since 2010.
Tenure is 1.9 years, the longest since the early 2010s.
Pipeline per SDR is $3.78M, up from $2.83M in 2022.
The SDR-to-AE ratio is 1 to 2.4, unchanged since 2018.

And the quota is lower than it has been in years:

Held meetings (stage 0): 10 a month, down 40% since 2018.
Converted opportunities (stage 1): 6 a month, down 43% since 2018.

Yet only 60% of reps hit quota, the lowest in the study's history.

The survivors got better. The scorecard got worse.

The Bridge Group, 2025 SDR Models & Metrics Report (351 B2B companies).

The trap: the unit of value changed

Pipeline per rep went up, but the Bridge Group's own reading is that it reflects higher deal sizes, not more meetings. The job moved from volume to value. Each meeting carries more pipeline, and there are fewer of them.

That changes what a good rep looks like. In a volume model, the best rep books the most meetings. In a value model, the best rep finds the accounts worth a real conversation and hands over opportunities that close. Those are different skills, and a meeting count, even a lower one, can't tell them apart.

I led sales development for about seven years, including a 47-person North America org. When a more experienced, faster-ramping team misses a smaller number than ever, I don't start with the reps. I start with the number.

If your team's headline metric is still meetings held, you're measuring the old job with a smaller target.

What I'd measure instead

1. Qualified pipeline accepted by sales. A meeting counts once an AE agrees it's real. This is the headline number. Everything else is a diagnostic.

2. Meeting-to-opportunity rate. If this is low, the top of the funnel is noise no matter how big it looks. If it's high on a small team, that team is doing the job.

3. Time from signal to first touch. Speed on a real buying signal beats volume on a cold list. It's also the part of the job where AI tools help most, so it's worth measuring before and after you add them.

4. What closed. Report sales development in the same currency as the forecast. A function that can show its contribution to closed revenue is hard to cut. A function that reports dials never escapes the cost column.

Why the smaller team is an advantage

A smaller team is more coachable, not less. Each rep matters more, and the ceiling on each one is higher. The Bridge Group's tenure and ramp numbers say the survivors can handle more: bigger accounts, harder conversations, more judgment per touch.

The companies that win the next stretch won't be the ones with the most reps. They'll be the ones who changed the scorecard to match the job the survivors are already doing.

If you lead sales development: what's the one number your CFO sees from your team?

Sources

36% decreased, 44% held steady, 19% increased: Emergence Capital and Benchmarkit, Beyond Benchmarks, 2nd edition. Data collected April 2025 "from over 560 B2B software companies." Question: "How has the headcount of each group changed in the past 12 months?" https://www.emcap.com/beyond-benchmarks-2025. Reported by SaaStr (Jason Lemkin), June 2025: "36% of companies decreased their SDR/BDR headcount in the last year," which it called "the highest percentage among all sales roles surveyed." https://www.saastr.com/the-great-sdr-downsizing-36-of-b2b-companies-cut-sales-development-teams-in-2025/
Everything else: The Bridge Group, 2025 SDR Models & Metrics Report (10th edition), February 6, 2025. 351 B2B companies, 83% SaaS, online survey. https://www.bridgegroupinc.com/research/2025-sdr-models-metrics-report-the-bridge-group
- Quota attainment 60%, "lowest reported in study history."
- Stage 0 held quota 10 a month (global median), down 40% since 2018. Stage 1 converted 6, down 43% since 2018.
- Pipeline per SDR $3.78M vs $2.83M in 2022, reflecting "higher ASPs… rather than more meetings."
- Ramp 3.0 months, "lowest since 2010." Tenure 1.9 years, "highest since early 2010s." SDR to AE 1:2.4, "consistent since 2018."

Originally published at michaelinghilterra.com.

The RevOps stat nobody can trace

Michael Inghilterra — Fri, 16 Oct 2026 14:00:00 +0000

"Companies with RevOps grow revenue 19% faster and are 15% more profitable." If you work in revenue operations, you've seen that line on a slide. I traced it. It's a 2015 best case about something else, and the original page now returns a 404.

Where it starts

The number starts in May 2015, at the SiriusDecisions Summit in Nashville. SiriusDecisions was a B2B research firm (Forrester bought it in 2019). Its co-founder gave a talk on "the economics of alignment," and the firm's blog recapped it.

The recap describes benchmark data on 400 B2B organizations, collected from 2006 to 2014, plus a separate survey of 300 sales, marketing and product leaders about why alignment is hard. The claim, in the recap's words: organizations that maintain this focus "achieve up to 19 percent faster revenue growth" and "15 percent higher profitability" than other companies.

That's it. That's the whole foundation.

What it actually measured

Read the original sentence carefully and three things stand out.

"Up to." 19% was a best case, not an average. Nobody quoting it today says "up to."

The subject. The study was about aligning sales, marketing and product management. It never mentions revenue operations. The talk measured a kind of cooperation between three teams, not a function with a charter, a forecast and a tech stack.

The method. The public recap never explains how they scored "alignment," what the comparison group looked like, or how they separated alignment from everything else that makes a company grow. Maybe the full research answered those questions. The public record doesn't.

How it mutated

A statistic doesn't usually get invented. It drifts.

2015: "up to 19 percent faster revenue growth" and "15 percent higher profitability," from aligned sales, marketing and product teams.
2019: a Forrester blog post drops "up to" and uses the number to introduce revenue operations.
Today: vendor blogs say "companies already operating with RevOps report 19% faster revenue growth and 15% higher profitability."
2026: one vendor's RevOps report turns the profitability claim into "15% higher win rates."

Each step is small. Together they turn a 2015 best case about team alignment into a present-tense fact about RevOps win rates. And the one page that could settle it, the original 2015 recap, now returns a 404. The Internet Archive still has it.

How a 2015 best case became a 2026 fact

The other RevOps numbers on the slide

Two more show up in nearly every RevOps pitch deck. Neither is fake. Neither is proof.

"Grew revenue nearly three times faster." From Forrester's 2019 revenue operations research survey, run together with three sales-technology vendors. The published write-up says organizations that "executed it well" and "deployed revenue ops in some form" grew nearly three times faster. It gives no sample size.
"10% to 20% increases in sales productivity." From a 2020 BCG article, describing benefits that top B2B technology companies "are reporting." It gives no sample and no method.

Why this matters more for RevOps than for anyone else

A few weeks ago I wrote that the RevOps seat is shifting from scorekeeper to operator. Whichever seat you're in, you keep one asset through every reorg and every miss: people trust your numbers.

That's why this stat is dangerous for us specifically. Sales can quote an unsourced number and survive it. The function whose whole value is measurement can't. A CFO who checks the 19% will find a 2015 best case about a different question, and every number you show after that gets discounted.

I'm not saying alignment doesn't pay. I've spent my career in revenue operations, sales development and analytics, and I've watched alignment pay off. I'm saying we don't need a borrowed number to prove it.

What to use instead

Make the case for RevOps with your own before-and-after numbers. They're harder to get and much harder to argue with.

Forecast error, by quarter, before and after RevOps owned the process.
Commit-to-close, the share of committed deals that closed in the quarter they were committed.
Time from lead to first touch, because speed on a real signal compounds.
Days lost at handoffs, SDR to AE and AE to customer success. The gap between owners, not the time inside stages.
Report adoption, the share of executive reports that someone opened last month.

Four quarters of those numbers from your own business beat any statistic from someone else's.

What's the RevOps stat you've quoted most, and have you read the study behind it?

Sources

The original: Drew Zalucky, "Summit 2015 Highlights: The Economics of Alignment," SiriusDecisions (now Forrester) blog, May 13, 2015. A recap of co-founder John Neeson's Summit 2015 talk. Wording: organizations that maintain this focus "achieve up to 19 percent faster revenue growth" and "15 percent higher profitability" than other companies. Original URL (returns 404): https://www.forrester.com/blogs/summit-2015-highlights-the-economics-of-alignment. Archived copy: http://web.archive.org/web/20260316032404/https://www.forrester.com/blogs/summit-2015-highlights-the-economics-of-alignment/
2019 restatement without "up to": Forrester blog, June 18, 2019: https://www.forrester.com/blogs/how-to-align-revenue-engine/
"Nearly three times faster": Megan Heuer, Forrester blog, June 13, 2019, from the 2019 Revenue Operations Research Survey run with LeanData, InsightSquared and Outreach. No sample size published. https://www.forrester.com/blogs/revenue-operations-and-cmos/
"10% to 20% increases in sales productivity": BCG, "Revving Up Go-to-Market Operations in B2B," May 13, 2020: https://www.bcg.com/publications/2020/revving-up-go-to-market-operations-b2b
The two present-day vendor versions: checked on the vendors' own pages on 2026-09-27. They're left unnamed on purpose; the point is the drift, not the vendors.

Originally published at michaelinghilterra.com.

The RevOps job description doubled. Nobody wrote the new one down.

Michael Inghilterra — Thu, 15 Oct 2026 14:00:00 +0000

98% of RevOps leaders say their function grew in scope over the past year. 89% say it lacks clearly defined strategic goals.

The state of the function, in two numbers

Those two figures come from the same 2025 survey: Salesloft and Wakefield Research asked 400 US RevOps and executive decision-makers about the function. Salesloft sells to RevOps teams, so weigh it accordingly. But the pairing is hard to ignore.

Read together, they describe RevOps in one line: the job got bigger, and nobody wrote down the new one.

How big the job got

The growth isn't only in scope. It's in seniority.

Head of Revenue Operations was the #1 fastest-growing job in the US in LinkedIn's 2023 Jobs on the Rise list. (LinkedIn grouped Chief Revenue Officer and Head of Revenue Management into that entry, so read it as revenue leadership broadly.)
Director of Revenue Operations ranked #4 on LinkedIn's 2024 list.
73% of companies now have a C-suite role dedicated to RevOps, in the Salesloft survey. RevOps teams there work most closely with COOs (26%), CEOs (25%) and CFOs (24%).
Pipeline generation is now a top-five priority for both sales leaders and RevOps, in Rattle's 2025 State of RevOps survey of 199 US participants.

A function that used to sit under sales operations now works next to the COO, the CEO and the CFO, and it's expected to care about pipeline, not just report it.

The job got bigger. Nobody wrote the new one down.

What didn't grow with it

Three things stayed the same while the scope expanded.

The written mandate. 89% say RevOps lacks clearly defined strategic goals. The scope grew by accretion: one more report, one more system, one more problem nobody else owned. Nobody sat down and rewrote the charter.

The time. A separate 2024 survey of 1,200+ revenue professionals, from BoostUp.ai and RevOps Co-op, found only 23% could prioritize GTM strategy, with many still spending substantial time on systems administration. A bigger mandate on top of the same admin load doesn't produce strategy. It produces a longer week.

The authority. Being asked to care about pipeline is not the same as owning it. In most companies I've seen, nobody has said where RevOps' job ends and sales leadership's begins.

The trap

Put those together and you get the most common way good RevOps leaders get set up to fail: measured on outcomes they were never given the time, the authority or the written goals to move.

I wrote about this in September as the scorekeeper-or-operator question. My own seat was a scorekeeper seat that kept getting handed operator problems. That's the pattern the 98% and the 89% describe at scale.

Write the job description yourself

If your scope grew this year, don't wait for someone else to rewrite the charter. Draft it yourself, one page, and walk it through with your boss before your next review.

1. What you own. The outcomes you're accountable for, where a miss is yours to explain.

2. What you influence. The outcomes you shape but don't control, like pipeline generation when sales leadership owns the reps.

3. What you only measure. The numbers you report and keep honest but don't answer for.

4. The three numbers you'll be judged on. Written down, with how each one is calculated.

5. The admin you'll stop doing, or the headcount or tooling that takes it off your plate. A bigger mandate needs time from somewhere.

The conversation is uncomfortable once. Not having it is uncomfortable every quarter.

RevOps leaders: did your scope grow this year, and did anyone write the new one down?

A correction to something I posted in September

In September I opened a post with "'VP of Revenue Operations' as a title is up roughly 300 percent in 18 months." I traced it for this piece. It comes from a 2021 vendor infographic that counted LinkedIn job titles over roughly late 2019 to early 2021, with no published method. It isn't a study, and it isn't current. LinkedIn's own rankings, cited above, are the better evidence for the same point.

Sources

98% say scope grew; 89% say RevOps lacks clearly defined strategic goals; 73% have a C-suite RevOps role; closest to COOs (26%), CEOs (25%), CFOs (24%): Salesloft and Wakefield Research, press release July 22, 2025. 400 US RevOps and executive decision-makers in four industries; vendor-funded; no fieldwork dates published. https://www.salesloft.com/company/newsroom/revenue-operations-makes-leap-to-csuite-study-says
Head of Revenue Operations #1 (2023): LinkedIn News, "LinkedIn Jobs on the Rise 2023," January 18, 2023. Method: jobs started by LinkedIn members, January 1, 2018 to July 31, 2022. The entry notes the title "can also be known as Chief Revenue Officer or Head of Revenue Management." https://www.linkedin.com/pulse/linkedin-jobs-rise-2023-25-us-roles-growing-demand-linkedin-news
Director of Revenue Operations #4 (2024): LinkedIn News, "LinkedIn Jobs on the Rise 2024," January 17, 2024 (January 2019 to July 2023). https://www.linkedin.com/pulse/linkedin-jobs-rise-2024-25-fastest-growing-roles-us-linkedin-news-dxmie
Pipeline generation a top-five priority for sales leaders and RevOps: Rattle, State of RevOps 2025, 199 US participants (publication date not shown). https://21423273.fs1.hubspotusercontent-na1.net/hubfs/21423273/Guides%20and%20eBook%20PDFs/State%20of%20RevOps%202025.pdf
Only 23% can prioritize GTM strategy: BoostUp.ai and RevOps Co-op, RevOps compensation and impact survey, press release November 14, 2024. "More than 1,200 revenue professionals." Wording: "only 23% of respondents can prioritize GTM strategy, with many still spending substantial time on tasks like systems administration." prnewswire.com press release
The September correction: Clari blog, "The Rise of Revenue Operations: A RevOps Infographic (2021)," February 24, 2021, archived: https://web.archive.org/web/20210524160247/https://www.clari.com/blog/the-rise-of-revenue-operations-infographic

Originally published at michaelinghilterra.com.

Buyers want to skip the rep. Until they need to trust something.

Michael Inghilterra — Wed, 14 Oct 2026 14:00:00 +0000

67% of B2B buyers say they'd prefer a buying experience without a sales rep. 69% say they want a sales rep to validate what AI tells them. Same Gartner survey, 645 buyers.

Not a contradiction

Read those two numbers together and they look like buyers can't make up their minds. I think they describe exactly what buyers want.

Buyers don't want a person for information anymore. AI gives them information faster, at any hour, without a sales pitch attached. What they still want a person for is trust: someone who can tell them whether the AI-generated answer is actually right for their situation, their budget and their risk.

That's the new job description for sales development. Less "deliver the message," more "confirm the truth."

Why the AI SDR story keeps disappointing

The volume of AI in sales is about to explode. Gartner predicts that by 2028, AI agents will outnumber human sellers 10 to 1, and that fewer than 40% of sellers will say those agents improved their productivity.

Both halves are predictions, not measurements. But the gap between them is the whole story. More agents, not much more productivity.

The wider evidence on AI projects points the same way:

Gartner predicts more than 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls.
MIT's NANDA initiative reported in 2025 that of the organizations that evaluated custom or embedded AI tools, 60% evaluated, 20% reached a pilot, and 5% reached production. The report calls its numbers preliminary and "directionally accurate," so treat it as a direction, not a measurement.

None of this says AI can't do sales development work. It says most deployments point it at the wrong part of the job.

The part to automate, and the part not to

Most AI SDR pilots automate the part buyers were happy to skip: account research, first drafts, sequences and follow-up. That part should be automated. It's where the machine is fast, tireless and good enough.

The failure is expecting the same machine to do the part buyers still want a human for: reading whether an account is actually in pain, noticing hesitation, and telling a buyer the truth about fit.

There's a hint in Salesforce's latest State of Sales survey (weigh it knowing Salesforce sells AI agents). High performers were 1.7 times more likely than underperformers to use prospecting agents. The best teams use AI. They just don't use it to replace judgment.

I led sales development for about seven years, including a 47-person North America org. The best reps I managed were never the fastest dialers. They were the ones a buyer trusted to tell them the truth. That skill used to be one part of the job. Now it's most of it.

Where I'd draw the line

AI owns: account research, signal monitoring, first-draft outreach, sequencing, and follow-up that never gets tired.

A person owns: any reply that doesn't fit the script, any account worth a real theory of why it's in pain, and every moment where a buyer needs to trust something.

The handoff is the design. Route to a person the moment the interaction needs judgment, not after the sequence ends. Then coach that person on exactly that moment, because it's their entire value now.

How to measure a hybrid team

Meetings booked is the one number AI is best at inflating, so it can't be the headline. I'd use three:

Qualified pipeline accepted by sales. A meeting counts once an AE agrees it's real.
Meeting-to-opportunity rate. If AI doubles meetings and halves this rate, nothing improved.
Time from signal to first touch. This is where AI should win clearly. If it doesn't, the tool isn't doing its part of the job.

Measure all three before the pilot starts. Otherwise you'll grade the pilot on whatever number looks best afterward.

If you run sales development: where does the machine stop and a person take over on your team?

A correction to something I posted in August

In August I cited two AI SDR figures: that "only about 2 percent of companies make autonomous AI SDRs stick," and a controlled test where an AI SDR booked 847 meetings against a human-plus-AI team's 312. I traced both for this piece. I couldn't find any source for the 2%: the page it's usually credited to doesn't contain it. The 847 vs 312 comes from one advisor's newsletter about an unnamed company, with no published method. I shouldn't have presented either as data, and I've left both out here.

Sources

67% prefer a rep-free experience; 69% prefer to validate AI-generated insights with sales reps: Gartner press release, May 20, 2026. Survey of 645 B2B buyers, August to September 2025. https://www.gartner.com/en/newsroom/press-releases/2026-05-20-gartner-survey-finds-sixty-nine-percent-of-b-two-b-buyers-turn-to-sales-reps-to-validate-ai-generated-insights
AI agents will outnumber sellers 10 to 1 by 2028; fewer than 40% of sellers will say agents improved productivity (prediction): Gartner press release, November 18, 2025, restated July 28, 2026. https://www.gartner.com/en/newsroom/press-releases/2025-11-18-gartner-predicts-by-2028-ai-agents-will-outnumber-sellers-by-10x-yet-fewer-than-40-percent-of-sellers-will-report-ai-agents-improved-productivity
Over 40% of agentic AI projects canceled by end of 2027 (prediction): Gartner press release, June 25, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
60% evaluated, 20% reached pilot, 5% reached production: MIT NANDA, The GenAI Divide: State of AI in Business 2025, July 2025. 300+ public AI initiatives, interviews at 52 organizations, a survey of 153 leaders, January to June 2025. The report describes itself as preliminary findings, "directionally accurate based on individual interviews."
High performers 1.7x more likely to use prospecting agents: Salesforce, State of Sales, 7th edition, February 3, 2026. 4,050 sales professionals in 22 countries, August to September 2025. https://www.salesforce.com/en-us/wp-content/uploads/sites/4/documents/reports/sales/salesforce-state-of-sales-report-2026.pdf
The August correction: the "2%" appears in a February 2026 blog post that links to a May 2025 page that doesn't contain the figure. The 847 vs 312 appears in the GTM AI Podcast & Newsletter, March 14, 2026, describing "one company I'm advising."

Originally published at michaelinghilterra.com.

The fine-tune that looked fine

Michael Inghilterra — Tue, 13 Oct 2026 14:00:00 +0000

My fine-tuned AI model matched the original on the headline metric and cut false alarms by more than half. It also found 6 of the 14 roles that mattered. The original found 10.

0.707 vs 0.705 Rank correlation with Claude. A tie.
10 vs 22 False alarms. Better.
6 vs 10 of 14 Strong roles found. Worse.

The short version

I fine-tuned a local AI model to judge job descriptions the way Claude does, then tested it on 116 roles it had never seen.
On the averages, it looked like an upgrade. On the roles that mattered, it found 6 of 14. The original found 10.
It had learned to play it safe. It never scored any role above 4.3 out of 5, so the best roles sat just under the bar.
I wrote the pass/fail rule before I trained anything. Three versions later, none passed. The original model stays in production.

The question

I run a small AI lab at home on an NVIDIA DGX Spark. One of its jobs is triage for my job search: a local model gives incoming job descriptions a first read, and only the promising ones go on to Claude for a full evaluation.

That local model works, but it carries a hidden cost. To score like Claude, it needs my full scoring rubric in every prompt: about 22,000 tokens of instructions before it reads a single word of the job description.

Fine-tuning promised a shortcut. Train the model on Claude's past judgments, and the judgment moves into the model itself. The prompt drops from about 22,000 tokens to about 550.

So the question was simple, and it's the same one every revenue team asks about a cheaper tool: can it make the same calls as the expensive one?

The rule I wrote first

Before I trained anything, I wrote down what "better" had to mean. It's the same rule I'd applied to every model that tried to replace the one in production:

It has to rank job descriptions at least as well as the current model (a higher rank correlation with Claude's scores).
It has to find at least as many strong roles. No fewer.

The second condition matters most, because it matches the decision the filter makes. If the filter misses a strong role, I never see a job I'd want. If it raises a false alarm, the cost is one extra evaluation by Claude. Those two mistakes don't cost the same, so I don't count them the same.

What I built

I had 1,160 of Claude's past evaluations to learn from. Each one pairs a job description with Claude's scores and its written reasoning. A strong role is anything Claude scored 4.0 or higher out of 5.

I held back a random 116 of them, 14 of them strong, and never used them for training or tuning. Every result below comes from those 116. The model trained on the other 1,044.

I trained three versions:

Version 1 repeated the strong roles three times in training, so the model couldn't learn to score everything low.
Version 2 pushed harder: strong roles six times, near-strong roles twice.
Version 3 kept version 2's recipe but trained on the scores alone. I left out Claude's written reasoning.

The first run took about six hours on one desktop box. I ran every comparison on the same serving setup, so the only thing that changed between runs was the model.

What happened

	Rank correlation	Strong roles found	False alarms	Big misses
Original	0.705	10 of 14	22 of 101	4
Version 1	0.707	6 of 14	10 of 102	1
Version 2	0.634	9 of 14	16 of 102	0
Version 3	0.472	5 of 14	13 of 102	0

Big miss: a strong role scored 3.0 or lower. The original shows 101 non-strong roles instead of 102 because one of its answers came back malformed.

Version 1 is the one that fooled me, briefly. Its rank correlation tied the original. On a separate test set, the original's own correlation moved by 0.04 between two identical runs, so a gap of 0.002 is a tie.

It also cut false alarms by more than half. On a slide, it was an upgrade.

Then the column that decides anything: it found 6 strong roles out of 14. The original found 10.

Version 2 traded back the other way. It found 9 of 14, with more false alarms than version 1 and a weaker ranking. Both versions gave up strong roles to cut false alarms, just in different amounts. Neither found as many strong roles as the original.

No version found as many strong roles as the original

Scatter plot of four model versions. The original found 10 of 14 strong roles with 22 false alarms. Version 1 found 6 with 10 false alarms, version 2 found 9 with 16, and version 3 found 5 with 13. No version found as many strong roles as the original.

116 held-out job descriptions, 14 strong. Michael Inghilterra's lab, September 2026.

Why it missed: the squeeze at the top

The fine-tuned model never gave any of the 116 roles a score above 4.3. Not once.

That's the whole failure. The bar for a strong role is 4.0. When a model's highest score is 4.3, every strong role has to land in a narrow band to clear it. Seven of the eight strong roles version 1 missed scored between 3.4 and 3.9. Close, consistently, and under the line.

The original misses differently. Its four misses were big: it scored each of those strong roles a 2 out of 5. So the honest comparison isn't a good model against a bad one. The original is erratic. The fine-tune is timid. Timid looks better on an average, because it never makes a wild call. It's worse at the one job the filter exists to do.

The fine-tune's scores bunch just under the bar

Dot plot of the scores two models gave the same 14 strong roles. The original scored 10 of them at 4.0 or above, spread up to 5.0, but scored four of them at 2.0. The fine-tuned model never scored above 4.3, and seven of its eight misses landed between 3.4 and 3.9, just under the 4.0 bar.

Scores for the 14 strong roles among 116 held-out job descriptions. Michael Inghilterra's lab, September 2026.

What I tried to rescue it

The obvious fix for a model that scores too low is to adjust its scores afterward. I tried four ways: shift every score up, stretch the scores to match Claude's spread, fit a straight line to Claude's scores, and lower the cutoff.

None of them held up on a second test set. The first three collapsed: the adjusted models found at most 1 of the 12 strong roles on that set. Lowering the cutoff kept the strong roles and roughly tripled the false alarms.

One option looked perfect. At a cutoff of 3.5 instead of 4.0, version 1 kept all 12 strong roles on the second set. But I found that cutoff by studying that same set, and a test you've already studied can't certify anything. I didn't count it.

The surprise: the reasoning was doing the work

Version 3 was supposed to be the efficient one. Every evaluation Claude writes includes a summary and evidence for each score, and I wondered whether all that prose was wasting the model's capacity. So I trained a version on the scores alone. Its answers ran about a quarter of the length.

It was the worst of the four. Rank correlation fell from about 0.7 to 0.472.

The likeliest reading: writing out the evidence is how the model reasons its way to a score. Take away the writing and you take away the thinking. It's the same reason I'd rather read a rep's notes on a deal than see only their close probability.

The verdict

No version beat the original. I closed the experiment on September 26, and the original model stays in production, unchanged.

I'll reopen it once I have a fresh set of about 35 strong roles the model has never seen, and roughly 600 more evaluations to train on. The next attempt keeps the reasoning and tells the model to weight the scores more heavily, instead of deleting everything around them.

What this means if you run a revenue team

I'm not an ML engineer. My career is revenue operations, sales development and analytics, and this failure is the most familiar one I know.

Averages hide the deals that matter. A forecast can land in total and miss every big deal inside it. A win rate can hold steady while you lose every enterprise deal. My model was right on average and wrong on the roles I'd actually take. Always ask what the number looks like at the top, not just across the whole list.

Pick the metric that matches the decision. The filter's job is to never lose a strong role. Correlation doesn't measure that. Strong-role recall does. If I had judged on the headline metric, I would have shipped a worse filter and felt good about it.

Write the rule before the test. Once you've seen the results, every metric starts to look like the right one. Writing the rule first is the only way to stop yourself from picking the number that flatters the answer you wanted.

Three questions to ask any AI vendor before a pilot:

What's the hit rate on the cases that matter, not the average?
Did you measure it on data the model never saw?
Who wrote the pass/fail rule, and was it before or after the test?

Limits

The sample is small. With 14 strong roles, one role moves the hit rate by about 7 points. Treat the direction as solid and the exact numbers as rough.
The benchmark is Claude's judgment, not hiring outcomes. I measured whether the local model agrees with Claude, not whether either one predicts who gets hired.
One task, one box. Training on a single desktop machine capped each training example at about 3,000 tokens. More memory would allow longer examples, and that might change the result.

Glossary

Fine-tuning: training an existing model further on your own examples so it picks up your specific judgment. I used LoRA, a lightweight method that trains a small add-on instead of the whole model.
Rank correlation: how closely two scorers agree on the order of a list, from 0 (no agreement) to 1 (the same order).
Strong-role recall: of the roles Claude rated strong, how many the model also rated 4.0 or higher.
False alarm: a role the model rated strong that Claude didn't.

This experiment runs inside trajecktory, the open-source job-search command center I built. It's on GitHub: trajecktory on GitHub

Originally published at michaelinghilterra.com.

AI SDRs and the human in the loop

Michael Inghilterra — Mon, 12 Oct 2026 14:00:00 +0000

Two things are happening to sales development at the same time, and they are easy to read as one grim story. Human headcount is shrinking, with about a third of B2B companies cutting sales development roles in the last year. And AI is moving into the top of the funnel, doing the research, the first drafts, the sequencing that used to fill an SDR's morning. Put together, the obvious conclusion is that AI is replacing the SDR.

I do not think that is what is happening, and I led sales development for about seven years, so I have watched this role get declared dead before. What AI is actually doing is stripping out the mechanical part of the job, the part that was never where the value lived. That does not eliminate the SDR. It exposes what the SDR was always supposed to be, and it puts a hard premium on the human judgment AI cannot replicate. The teams that win will be the ones that design that human-in-the-loop deliberately instead of pretending the machine can run the whole thing.

AI is good at the mechanical, bad at the judgment

Be honest about what the machine does well. AI can research an account faster than any human, draft a competent first-touch email, run a sequence, and handle the sheer volume of the top of funnel without fatigue. Those are real capabilities and they are not coming back to humans. If your SDR's main value was dialing and sending, that value is genuinely being automated, and no amount of nostalgia changes it.

But the machine is weak exactly where the job gets hard. It does not truly know why this account is in pain right now. It cannot read the hesitation in a prospect's voice and get curious about it. It cannot build a theory of a buying group, sense who the real economic buyer is, or catch the objection hiding under the stated one. It generates plausible outreach; it does not generate insight. The mechanical layer is solved. The investigative layer is not, and the investigative layer is where meetings that actually convert come from.

The role moves up, it does not disappear

So the SDR role does not vanish. It moves up the value chain to the work that was always the point: investigation, judgment, and real human conversation. The rep spends less time producing touches and more time deciding which accounts are worth a real theory, reading the signals a model flattens, and handling the moments where a person can tell they are talking to a person who has actually thought about their world.

This is the same argument I make about coaching reps as detectives rather than dialers, and AI sharpens it rather than softening it. When the machine handles the volume, the human's entire job becomes the detective work: the theory, the curiosity, the qualification. The reps who thrive in this next phase are not the ones who could send the most emails. They are the ones who can do the thinking the model cannot, on the accounts the model flagged.

Design the human in the loop on purpose

The failure mode is treating this as all-or-nothing: either keep a big human team dialing, or hand the whole funnel to an AI and hope. Both are wrong. The win is a deliberate division of labor where the machine does what it is good at and the human does what only a human can, with a clear handoff between them.

That is a design problem, and it is worth doing well. Let AI own the research, the first draft, the volume, and the never-tiring follow-up. Route to a human the moment the interaction needs judgment: a reply that does not fit the script, an account worth a real theory, a conversation where reading the person matters. Then coach your smaller human team relentlessly on exactly that judgment, because it is now the entire value they add. The org chart that results is smaller on humans, yes, but the humans in it are doing higher-value work than a sales development rep has ever done, and they are far harder for a competitor to copy than a bank of dialers ever was.

The shrinking headcount and the rising AI are not two halves of a story about replacement. They are the same shift toward a smaller, sharper, more human team sitting on top of a machine that handles the mechanical. The companies that name that shift and design for it will build something genuinely good. The ones that read it as "fire the SDRs, the robot has it" will automate their outreach and wonder why nothing converts, because they will have kept the part AI does well and thrown away the part that was actually working.

I led sales development for about seven years, and I write about where human judgment still wins as AI reshapes revenue teams. I also run a productized Revenue Engine Audit for leaders designing the human-and-machine engine that comes next.

Originally published at michaelinghilterra.com.