DEV Community: Tessl

Claude Fable 5 vs Opus 4.8: The Mythos Hype Meets Reality

Tessl — Sun, 14 Jun 2026 06:39:18 +0000

For months, the most interesting model at Anthropic was one we could not use. Mythos was the internal system the company said was too capable to release, the one that found software vulnerabilities at a level that tripped its own safety thresholds. On June 9, 2026, that tier went public for the first time, as Claude Fable 5. Opus 4.8, the model anchoring production coding agents, suddenly had a successor that's a full capability class above it.

This raises two questions for anyone running coding agents. The practical one is whether you should move your fleet from Opus 4.8 to Fable 5. The bigger one is whether a Mythos-class model, the tier Anthropic held back as too capable to ship, lives up to what the name promised. This article answers both, and the numbers tell a more interesting story than the announcement did.

We ran both models through the same evaluation, close to 1000 shared scenarios scored twice each, once with no skill supplied and once with the relevant skill in context. The short answer, as of mid-2026, is that Opus 4.8 is still the better value for most agent fleets, and the gap between the Mythos hype and the measured reality is the real story in the data.

A Mythos-class model is a tier of Claude that sits above the Opus class in capability. It reaches a threshold Anthropic considers high-risk, particularly at discovering and exploiting software vulnerabilities. Fable 5 and Mythos 5 are the same underlying model with the same capabilities. What separates them is the safeguards: Fable 5 is the public version that ships with safety classifiers, while Mythos 5, restricted to approved partners, runs without them.

What the industry expected from a Mythos-class model

Before launch, the speculation was not subtle. Across Reddit, X, and a run of explainer posts, Mythos was framed as the model that would change how agents work, not just how well they answer. The recurring predictions clustered around four capabilities:

Restructuring a large codebase in one coherent pass.
Spotting security flaws that experienced engineers miss.
Working unsupervised for hours on a single hard problem.
Acting like a collaborator, not an assistant you steer turn by turn.

Of the four, the cybersecurity claim was the one with hard evidence behind it. Through Project Glasswing, roughly 50 early partners with Mythos Preview access reported finding more than 10,000 high or critical severity vulnerabilities, and the program has since expanded past 150 organizations. Anthropic's CPO Mike Krieger called it "the most capable class of systems we've built." That is the dream the name sold: a model so powerful it stayed in the lab.

What reached the public is narrower, and deliberately so. The model you can actually use is Fable 5, the Mythos-class system wrapped in safety classifiers. Whether it delivers comes down to the gap between that promise and what was released.

The headline numbers: Claude Fable 5 vs Opus 4.8

Every scenario in the evaluation is a real agent task tied to a published skill, scored on two axes: instruction-following (does the agent do what it was told, in the way it was told) and task-completion (does it reach the goal). The overall score weights instruction-following at 4 and task-completion at 3, then divides by 7. Each task runs with and without the skill, so the lift from the skill is visible directly. The tasks and skills are public, in the task-evals-for-skills dataset, so you can inspect any scenario yourself.

This design is deliberate. The tasks come from published skills, so they mirror the real work teams write skills for, not frontier puzzles meant to find a model's ceiling. That is why task-completion runs high for both models and why the signal that separates them is instruction-following: doing the work the specific way the skill asks.

Dimension (with skill)	Fable 5	Opus 4.8
Overall score	92.9	92.0
Overall score (no skill, baseline)	75.7	74.5
Overall lift from the skill	+17.2	+17.5
Instruction-following	89.3	88.0
Task-completion	97.8	97.4
Turns to complete	16.9	16.2
Output tokens per task	9,025	10,687
List price (input / output, per MTok)	$10 / $50	$5 / $25
Cost per task (average)	$1.25	$0.74
Points per dollar	74	125

On the 917 scenarios both models ran, Fable 5 leads on overall score by 0.9 points (92.9 to 92.0). Scenario by scenario, the two tie on 61% of tasks, Fable wins 24%, and Opus wins 16%, at a two-point threshold. A capability class above Opus, and on everyday agent skill tasks the quality difference is inside the noise.

One caveat sits underneath that number. The 917 are the tasks both models completed and scored. Fable 5 refused 26 that Opus 4.8 finished, and we excluded them, so the near-tie is measured only on the tasks Fable agreed to do. That exclusion turns out to be the most revealing part of the comparison, and we return to it below.

Why agent skill evaluation matters more than the model upgrade

Here is the number that reframes the comparison. The skill adds about 17 overall points to both models: +17.2 for Fable 5 and +17.5 for Opus 4.8. The model upgrade from Opus 4.8 to Fable 5 adds less than 1 point on shared tasks. The context you supply moves the agent far more than the frontier tier you pick.

The lift concentrates in instruction-following, where both models gain more than 27 points from the skill, while task-completion gains under 5. Both models can usually reach the goal on their own. What they cannot do reliably without a skill is follow the specific conventions, constraints, and steps a real task demands. That is what a good skill encodes.

Skill receptivity is how much an agent's output improves when you supply a relevant skill. It shows up mostly as better instruction-following. It matters because it can outweigh the model choice, which is the practical case for investing in agent skills before chasing the newest tier. Running the same task with and without the skill, then measuring the difference, is a task eval. It is also the only way to know whether a model upgrade earns its price on your workload, which is what agent skill evaluation is for.

The price gap is the deciding factor for most teams

On the agent skill tasks we measured, the trade comes down to paying a steep premium for a marginal gain. Fable 5 lists at $10 per million input tokens and $50 per million output tokens against Opus 4.8's $5 and $25, exactly twice across every token category, including cache reads and writes. For that, across our 917 shared scenarios, you get an overall score of 92.9 versus 92.0, a 0.9-point edge that sits well inside the range where the two are interchangeable. This is the everyday-agent-work picture, not a verdict on the marquee Mythos capabilities our eval does not test.

Token behavior softens the unit price but does not close it. Across the 917 shared scenarios Fable 5 generated about 16% fewer output tokens per task (9,025 versus 10,687), so the real cost per task lands at $1.25 against $0.74, a 73% premium rather than a clean 2x. The value gap is the number to remember: Opus 4.8 returns 125 points per dollar to Fable 5's 74, about 69% more quality for every dollar spent.

For a single session the difference is cents. For a fleet running thousands of agent tasks a day, it is the line item your finance team will ask about, and twice the price for under a point of quality on the tasks most teams actually run is not an easy answer to give them.

Fable refuses work Opus completes without issues

The most consequential difference between Fable 5 and Opus 4.8 is not on the scoreboard. It is the safety layer that defines the Mythos class.

Fable 5 ships with safeguards covering four domains: cybersecurity, biology and chemistry, distillation, and frontier LLM development. For the first three, a triggered request comes back as a refusal. Anthropic's design hands it to Opus 4.8 and informs the user, but that fallback is opt-in rather than a default, so in a stock harness like ours the blocked requests simply refused.

The fourth domain worked differently during this run. By Anthropic's own documentation, requests touching frontier AI development were not refused or even flagged. The model quietly steered or fine-tuned its answer instead, with no notice to the user. That silent manipulation drew the sharpest backlash, and on June 11, the day after this run, Anthropic switched it to a visible classifier like the other three while conceding the restrictions had been "overly conservative." Because it never produced a refusal, that domain leaves no mark in our numbers; any effect would surface only as quietly weaker answers.

A Mythos-class model routes some requests to a weaker model by design, so your harness needs to detect the fallback rather than trust that every response came from Fable. And the affected domains are exactly the ones you most want to check yourself, which is the practical edge of context governance and security: catch the regression in an eval, not in production.

Our run shows how that plays out, and it is not flattering. Fable 5 refused 26 of the roughly 940 tasks it attempted, returning a usage-policy block with a refusal stop reason instead of doing the work, while Opus 4.8 completed and scored every one of them. What Fable refused is the revealing part. Four were defensive security reviews, including "review this Flask application for security vulnerabilities before deploying it," blocked as "violative cyber content." Five were routine bioinformatics tasks, such as running quality control on a single-cell RNA-seq file. One was a literature review on the landscape of AI-assisted drug discovery. A model from the class Anthropic markets for finding vulnerabilities in critical software declined to audit a Flask app for the developer who owns it. Anthropic's own "overly conservative" admission lands hardest here.

On the security tasks Fable did complete, it was competitive. Across 51 authentication and security skill scenarios, from Auth0, Better Auth, and Bitwarden, Fable 5 averaged 95.0 with the skill against Opus 4.8's 96.6, a near-tie. The lesson is not that one model is safe and the other is not. It is that a Mythos-class model will sometimes refuse the defensive work you most need done, and only an eval on your own tasks will tell you where.

Did Fable deliver on the Mythos promise?

Our evaluation answers the question that matters for a deployment decision: how both models handle hundreds of real, skill-driven agent tasks across dozens of tool ecosystems, which is the work most teams actually run coding agents on. The marquee Mythos feats sit outside this eval, but the day-to-day behavior it captures is exactly what you are buying when you point a fleet at a model.

What the data does show is where Fable's extra capability surfaces in normal use. Grouped by the organization that owns the skill, Fable 5 pulls ahead on web-research and scraping workloads: Apify (+7.8 overall), Google Gemini (+4.6), Tavily (+3.4), and Firecrawl (+2.7). If your agents fetch, map, and extract from the open web, Fable 5 is the stronger pick. Opus 4.8 holds its ground where Fable regresses: Mastra (-7.3), Auth0 (-4.5), and Axiom (-2.5).

So the Mythos dream of an autonomous collaborator is not what most teams will buy on day one. What they will buy is a model that is marginally better at instruction-following, meaningfully better at web research, twice the price, and gated by classifiers that occasionally hand the job to Opus 4.8 anyway.

When to use each

Choose Opus 4.8 if you run a coding-agent fleet at scale and care about cost per task. The quality difference is inside the noise for most workloads, Opus returns far more points per dollar, and it has no fallback layer to design around.

Choose Fable 5 if your agents do heavy web research and scraping, if you need its reasoning depth on long-horizon tasks, or if you have a workload that genuinely benefits from the capability class above Opus. Budget for the roughly 73% per-task premium, and build fallback detection into your harness from day one. If your work touches the classifier domains, confirm the model is not silently routing to Opus 4.8 before you depend on it.

Fable's edge shows up when you build around it, not when you swap it into an Opus 4.8 pipeline unchanged. Fable is the more autonomous model, but that edge only pays off in flows built for it: longer unsupervised runs, larger units of work, less step-by-step steering.

For almost everyone, the larger lever is neither model. The skill adds about 17 points; the model upgrade adds less than 1. Standardize the model in your tessl.json, prove the switch with an eval before you roll it to the fleet, and watch for the tasks a Mythos-class model quietly declines to do.

Want to see how a skill changes your own agent's behavior, on your own tasks, across both models? Start with the Tessl Registry and run the eval before you switch.

Same quality, a quarter of the cost: Should DeepSeek Flash be your model of choice?

Tessl — Thu, 11 Jun 2026 06:59:02 +0000

$0.0236 is how much DeepSeek V4 Flash costs to run a complete agentic task, skill included, on the Fireworks price sheet. Claude Haiku 4.5 costs $0.10 for the same task. Sonnet 4.6 costs $0.30.

In terms of how good they are, in our evals Flash scores 82.3, and Haiku scores 82.9. So the evals points to them being comparable, with skills applied, but one is four times the cost.

In our eval we ran 19 model configurations through the same benchmark harness. The tasks we asked of them were real agentic tasks, and we measured the total token counts, and looked at the charged provider pricing. To be honest, the value story we expected to find was "cheap models are a trap." What we found instead was more interesting, and particularly useful if you're running agents at any kind of scale.

First, the Pro comparison

DeepSeek V4 ships two tiers: Pro and Flash. In our eval runs, Pro costs $0.183/task and Flash costs $0.0236/task. That's a 7.7× price gap within the same model family.

When you look at what you get for the extra spend, it’s only three points. On the eval results, Pro scores 85.3, Flash scores 82.3. When we scale that, 10,000 tasks/month costs you an extra $19,000/year and 100,000 tasks/month costs an extra $190,000/year. For three points that may not be too visible from a quality point of view.

Points-per-dollar

When we look at cost per point of eval score, this gives us a ratio between quality and cost, which can be useful, so long as the overall quality of the model satisfies your needs.

Model	Score (w/ skill)	$/task	pts/$
DeepSeek V4 Flash	82.3	$0.024	3,482
Haiku 4.5	82.9	$0.097	829
DeepSeek V4 Pro	85.3	$0.183	467
GLM 5.1	90.4	$0.200	451
Sonnet 4.6	90.8	$0.296	303

The number your cost model is probably missing

Cost-per-token is the number everyone tends to quote and often mistakenly use as the most important factor in making a decision. It's also the number that will quietly blow your budget if you're not watching turns per solve as well.

Flash's mean average is around 20 turns per task which is pretty manageable. But the single worst-case runs in our dataset hit roughly 10× that. This isn’t unusual for models in this class, but in dollar terms, that's a single task costing as much as 10 average tasks. Multiply that across thousands of concurrent agent runs and you may start to have a budget problem that didn't show up in your per-token estimate.

The reason most teams don't catch this is that agent frameworks surface token counts by default. Turn counts, which is the variable that actually drives fat-tail cost explosions, often need to be logged explicitly.

Instrument your agents for turns, not just tokens. Know your median and your 95th percentile. Set your timeout policies against the 95th, not the median, or you're either killing valid runs or absorbing surprise bills.

The skill is doing half the work

One thing worth being very direct about here is that Flash's 82.3 score is a skill-augmented score. Without a skill, Flash scores 64.1. The skill adds +18.2 points.

That lift is real, but very conditional on the skill being precise, well-scoped, and actually relevant to the task. A vague skill will drag you back down closer to the 64.1 baseline, whereas a sharp one gets you 82.3.

This matters more than most model evaluations acknowledge since the model you test in a playground doesn’t usually use a skill or relevant context, but just raw capability.

Going further: find cheaper models and test them yourself

The analysis above shows the cheapest hosted options we measured. But there are two obvious next steps if you want to push it further, and both are more accessible than you might think.

Every model in this benchmark that isn't GPT, Anthropic, or Gemini has publicly available weights. DeepSeek V4 Flash, GLM 5.1, you can run all of them yourself. When you do, the marginal token cost drops to near zero. You're paying for compute (GPU rental or owned infra), not per-call pricing.

The maths of self-hosting only make sense above a certain volume threshold, the ops overhead and GPU costs aren't free of course, but if you're running tens of thousands of agentic tasks per month, the crossover point is lower than you'd expect.

The skill in this benchmark is doing +18.2 points of work. The question worth asking is: where did that skill come from, and how do you know it's any good?

The Tessl registry is a good place to start and look at the quality, impact and security posture of your skill. Before you write a skill from scratch, check whether one already exists and has eval data behind it.

Evaluate your skills properly. You can run two types of evaluation: reviews (automated quality assessment of whether your skill is well-structured) and task evals (end-to-end runs that measure whether the skill actually improves agent performance on real tasks). The task eval output is exactly the kind of "with skill / without skill" delta that the Flash benchmark is built on.

Use skill quality as a model selection input. The 18-point lift Flash gets from a well-scoped skill isn't a fixed number, it depends on the skill and the tasks. A skill that has been evaluated by Tessl with a high task eval score gives you confidence that the lift is real and reproducible. A skill that's never been evaluated is a variable you can't account for in your cost modelling.

Your own workload, not someone else's benchmark. The task eval system lets you define scenarios from your actual codebase and run them. That's the self-evaluation framework described above.

The takeaways, flat out

DeepSeek V4 Flash at $0.0236/task is the value pick. Haiku costs 4× more for 0.6 points. Pro costs 7.7× more for 3 points.
Set a quality floor before you rank by cost. pts/$ flatters cheap-and-weak models. Above 80 points, it's a real signal.
Instrument for turns, not just tokens. Your 95th percentile turn count is the budget variable nobody's logging.
The skill is doing half the work. A bad skill collapses your score back to baseline. Evaluate your skills — with task evals, not vibes.
You can run this yourself. 20-30 tasks, turn logging, a spreadsheet, and Tessl's eval system.
Self-hosting open source models is a real option. The weights are public, the ops trade-off is real. You should run your own evals with your models to see if they can be substituted in.

The tier name told you Flash was cheap; the data says it's also good. Now you have the tools to find out whether that holds for what you're building.

AI Coding Agent Accuracy: Opus 4.7 vs 4.8

Tessl — Tue, 09 Jun 2026 07:23:08 +0000

You are deciding whether to roll your default agent model from Opus 4.7 to 4.8. The release notes promise improvements, the leaderboard moves a fraction of a point, so you shrug, schedule the upgrade for a quiet Friday, and move on.

We ran both versions through the same skills evaluation, roughly 850 scenarios solved twice each, and on the headline metric they finished level. Underneath the tie, though, 4.8 reached the same answers in four fewer turns and for measurably less money, so the upgrade that looks like a non-event on the scoreboard turns out to be a real efficiency gain in the place that actually bills you: the agent loop.

AI agent evaluation measures how an agent behaves on real tasks rather than only scoring its final answer, tracking cost, turns, and reliability across paired runs. The reason to bother is that two models can post the same score while spending very different amounts of work to reach it.

Two versions, one eval harness

Both models ran the identical setup. Every scenario is solved twice, once with no help and once with the relevant skill installed, so we can isolate what the skill contributes from what the base model already knows. We score three things: instruction following (did the agent do what the skill tells it to do), task completion (did it reach the goal), and an overall blend weighted toward instruction following. We also flag integrity issues, like an agent peeking at the grading rubric instead of solving the task.

Opus 4.7 is the incumbent. In our runs it is a strong agent that leans heavily on skills to reach its ceiling, and it explores a lot of paths to get there.

Opus 4.8 is the point release. It posts the same ceiling with a skill installed, but it starts from a higher floor without one, and it gets to the answer with noticeably less wandering.

Where AI coding agent accuracy stops being the story

Here is the head-to-head on the shared scenario set, all with the relevant skill installed unless noted.

Dimension	Opus 4.7	Opus 4.8
Overall score	91.9	92.1
Baseline score, no skill	71.4	74.1
Task completion	97.1	97.4
Instruction following	88.1	88.1
Turns per task	19.2	15.0
Output tokens per task	7,820	9,763
Cost per task, API pricing	baseline	about 5% lower
Integrity flags raised	10.2%	7.9%

The overall accuracy gap is 0.2 points. If you stopped reading the row labeled "overall score," you would conclude nothing changed. Three other rows complicate that picture.

The first is the baseline. Without any skill, 4.8 scores 74.1 against 4.7's 71.4, a 2.6 point gain, and its no-skill instruction following climbed from the high 50s into the low 60s. The ceiling is shared because the skill pulls both versions up to roughly the same place. The floor is where 4.8 actually improved, and that has a practical consequence: 4.8 depends on the skill slightly less to do good work. This suggests some of the knowledge previously only present in skills has been trained into the model weights.

The second is turns. 4.8 finishes the average task in 15.0 turns versus 19.2 for 4.7, a 21% reduction. In an agent loop, a turn is a full round trip of context, reasoning, and tool use. Cutting four turns off the average task lowers latency, reduces the chances for an agent to talk itself into a wrong path, and, as we will see, lowers cost.

The third is integrity. The eval flags runs where the agent took a shortcut, like reading the grading rubric or reaching outside its workspace. Those flags dropped from 10.2% of shared runs to 7.9%. 4.8 is modestly more disciplined about how it reaches an answer. This matches Anthropic’s claims about 4.8 being more honest.

Reading the cost: turns, not tokens

Look again at two rows that seem to contradict each other. 4.8 produces more output per task, 9,763 tokens against 7,820, yet it costs about 5% less.

This is because output volume does not dominate agentic cost. The dominant term is the context replayed on every turn. Each turn re-sends the accumulated conversation and tool results, and in long agent runs that cached input swamps the fresh output the model writes. Fewer turns means fewer replays, so 4.8 can be more verbose inside each turn and still come out ahead, because it takes four fewer turns to converge.

Model cards only show the per-token rate that sets the price of a unit of work, while turn count sets how many units the model decides to spend. A point release that holds accuracy flat while spending 21% fewer turns is working on that second term, which is the one that scales with your usage.

The same dynamic shows up in how each version absorbs a skill. Adding the relevant skill is not free: it pulls in instructions and reference material the agent has to process, and the question is how efficiently the model turns that overhead into a result.

Effect of installing the skill	Opus 4.7	Opus 4.8
Overall score gain	+20.5	+18.0
Cost increase	+38%	+12%
Turn increase	+41%	+14%

On 4.7, switching on a skill added 41% more turns to cash in a 20 point accuracy gain. On 4.8, the same class of skill buys nearly the same gain for much less turn and cost overhead. 4.8 treats a skill more like a shortcut and less like an invitation to explore. If you run agent skills at scale, that lower skill tax compounds across every task you ship.

The one place 4.8 regressed

A fair comparison reports where the new version loses ground. Per scenario, the record is close to a wash: 4.8 scored higher on 23% of shared tasks, tied on 61%, and scored lower on 17%, using a two point threshold. The interesting part is that the losses cluster.

4.8 regressed on web research and scraping skill families. Firecrawl tasks dropped 3.3 points on average across 72 scenarios. LangChain dropped 2.9 points across 48. Smaller families like Tavily and Apify fell further, 10.4 and 7.6 points, though on fewer tasks. Meanwhile 4.8 improved on infrastructure, auth, and code tooling: Cloudflare gained 4.5 points across 38 scenarios, Auth0 gained 4.3 across 18, and Mastra gained 10.1 across 10.

The aggregate hid this completely, because the gains and losses nearly cancel. Only a per domain breakdown surfaces it. That is the whole argument for paired skill evals over a single leaderboard number: the headline can be a tie while two coherent shifts run in opposite directions underneath it.

When to roll forward to 4.8

Roll forward to 4.8 if your agents run long, multi turn tasks where turn count, latency, and cost matter, which is most production agent work. You get the same accuracy ceiling, a higher floor before skills, a 21% turn reduction, a cheaper skill tax, and fewer integrity flags. If your workloads lean on infrastructure, auth, or general code tooling, 4.8 is flat to clearly better.

Test before you roll forward if your agents live in the scrape, crawl, and summarize world. The web research regression is small in absolute terms but consistent across the families we measured. Run your own A/B on your top scraping workflows first.

The takeaway: measure behavior, not the changelog

A skeptic has two reasonable objections. The first: a flat score is just no improvement, so why care? Two models can tie on accuracy while one spends 21% more turns and about 5% more budget to get there. The second: these are our eval harness costs. However, the relative differences in turns, tokens, and cost reflect model behavior which does generalize.

Make sure you’re measuring each release on behavior, on your own tasks, with skills installed and stripped out, and look at the per domain breakdown before you trust the average.

Want to see how your own stack behaves across a model upgrade? Browse the Tessl Registry to find the skills your agents depend on, then run the same paired evaluations we used here to measure what actually changed.

AI Native DevCon Day 2: From Agent Demos to Operating Models

Rohan Sharma — Wed, 03 Jun 2026 06:40:09 +0000

TL;DR

Day 2 of AI Native DevCon shifted from agent capability to operating discipline. The strongest sessions focused on how teams can run AI-native delivery with clearer context pipelines, measurable agent behavior, safer execution boundaries, and better organizational ownership.

The scale showed up in the numbers too. Across the two days, DevCon brought together 650+ in-person registrations, around 2,000 online registrations, and a packed mix of sessions, workshops, hallway conversations, and practical lessons.

Day 2 leaned into workshops. That shift mattered because the second day was less about proving agents can do useful work and more about showing how teams can make that work repeatable.

Hey there, welcome back. Rohan Sharma here again continuing the devcon series.

Day 1 gave us the framing, including Guy Podjarny’s core point that skills should be treated like real software assets. Day 2 picked up from there and moved into the operating details. Once agents are inside daily engineering work, platform and product teams need to decide what changes first, who owns those changes, and how the results are measured.

Talks that shaped Day 2

Harness engineering beyond code

Marc Sloan from Tessl focused on the next gap many teams are hitting. Code context is increasingly structured, but product and design context still lives in external systems such as Figma, Notion, and Linear. Pulling that context live can reduce staleness, but it introduces drift in evals, versioning, and reproducibility.

The practical lesson was to stop treating external product and design context as random reference material. Teams need a defined layer between the repository and those external systems, with clear versioning so evaluations can be replayed against known context snapshots.

Without that, agents can produce work that looks technically correct while missing the product constraint that actually mattered. That is a very expensive kind of almost-right.

From vibes to metrics

Simon Obstbaum and Rob Willoughby from Tessl delivered a session focused on a challenge many engineering leaders are currently facing. Their distinction between output evals and trajectory evals is operationally important. A good answer is not enough if the agent used risky tools, skipped required checks, or ignored policy steps.

The useful measurement model came down to activation, trajectory, and outcome. Did the right skill trigger? Did the agent follow the right steps? Was the final result actually useful and correct?

The good part was the emphasis on partial compliance. Pass or fail is too blunt for agent workflows. If a workflow degrades halfway through, teams need to know where it happened, not just that something felt off.

Benchmarking beyond the model

Amit Kushwaha highlighted why many current benchmarks miss real agent behavior. Agent systems run long traces with tool calls, context accumulation, and latency bottlenecks that one-shot benchmark numbers do not capture.

For teams choosing infrastructure, the warning was clear. Do not optimize only for model speed. Real agent workloads involve tools, memory, caches, retries, and long-running traces.

The better benchmark is closer to production reality, with multi-turn tasks, tool latency, tail latency, and cache behavior over time. Otherwise teams risk picking systems that look great in a chart and struggle in the actual workflow.

Safe execution boundaries for agents

Oleg Šelajev from Docker covered a problem every platform team eventually sees. An unconstrained agent can make high-impact changes in the wrong environment. Sandboxing is not optional once agents are allowed to execute.

The practical takeaway was to treat environment policy as part of the harness. Filesystem access, network access, secrets, and permissions all need clear boundaries before agents are given the ability to act.

This is how teams lower blast radius. Not by hoping the agent behaves nicely, but by designing the room it is allowed to move around in.

Do not write prompts, write software

Baruch Sadogursky and Macey Baker from Tessl reinforced an idea that keeps proving useful in production. Break behavior into modular skills instead of maintaining one giant prompt. This makes agent behavior easier to test, review, and reuse.

The message was not “write a better mega prompt.” It was to turn repeatable behavior into composable skills that match real workflow stages. That gives teams something they can review, test, improve, and share across repos.

If you try one thing from this workshop, use the materials and skill templates as a starting point. Prototype one small skill pipeline in your own environment before trying to scale the pattern across every repo.

What kept coming up across the day

1. Context quality is now a platform responsibility

Marc Sloan, Shaun Smith, and John Groetzinger approached this from different angles, but the operational message was consistent. Context delivery is becoming an engineering system, not documentation hygiene. Teams need predictable context pipelines for both humans and agents.

The next step is ownership. Teams need to know who maintains context sources, how often they refresh, and how changes are versioned. Context also needs observability so teams can trace which inputs shaped an agent decision.

2. Agent performance needs production-grade telemetry

The sessions from Simon Obstbaum and Rob Willoughby from Tessl, plus Amit Kushwaha from NVIDIA and Justin Cormack, former CTO at Docker, made this very concrete. Teams need to measure how agents worked, not only what they returned.

Trajectory metrics belong next to existing quality signals. If your dashboards already show test health, release health, or incident trends, agent workflow quality should sit in the same operational view.

The benchmark scenarios should also look like real work. Multi-turn, tool-heavy, slightly messy, and full of the same constraints your teams face every day. Justin’s observability point connected neatly here too. Teams need runtime signals that can reveal agent-induced drift before it becomes a bigger production problem.

3. Adoption is an organizational design problem, not a tooling checkbox

Talks from Tammuz Dubnov and Birgitta Böckeler from Thoughtworks showed that adoption succeeds when review structures, ownership boundaries, and team rituals evolve with the tooling.

That means setting explicit contribution boundaries for AI-assisted changes and updating review criteria. The diff still matters, but so does the path the agent took to produce it. Birgitta’s adoption data made this especially grounded by showing where hidden costs appear, including review load, technical debt, and maintainability when speed becomes the only metric.

4. Workshops made the ideas practical

Baruch Sadogursky and Macey Baker from Tessl, along with Alfonso Graziano from Nearform, helped turn the bigger Day 2 ideas into something teams could actually try. The workshop-heavy format made the day feel less like theory and more like practice.

Derek Ashmore’s packed workshop, “The AI Agent Testing Pyramid,” focused on the different levels of testing agent systems need. For those following from home, you can attempt it on your own by following this repo.

Aashrey Tiku from Anthropic worked through a hands-on session on shipping a managed agent. It was a useful bridge between agent concepts and the practical work of packaging, managing, and operating an agent with the right boundaries.

That mattered because AI-native development is still new enough that people need patterns they can test, not just concepts they can nod along to. Alfonso’s spec-driven angle fit well here because prompts become far more useful when they are turned into testable, production-ready specifications.

5. Agent enablement needs real ownership

Ian Thomas from Meta and Katie Roberts from Nearform made the enablement side feel practical. Rollouts work better when platform safeguards are paired with updated team rituals, clear ownership, and realistic guidance for brownfield systems.

Katie’s legacy advice was especially useful. AI should help teams modernize incrementally, not generate another fragile layer on top of systems that are already hard to maintain.

If you missed Day 1, start here

Day 2 was workshop-heavy. If you missed the Day 1 virtual stream, start with these talks before digging into the workshop themes.

Guy Podjarny, Tessl - Skills are the new Code
Dana Lawson, Netlify - Built for Humans. Now Agents Are Here.
James Moss, Tessl - Using skills to pay the bills
Liran Tal, Snyk - Your AI Agent Installed Malware Because a SKILL.md Told It To
Ryan Lopopolo, OpenAI - Harness Engineering
Patrick Debois, Tessl - The Rise of Agent Enablement
Shachar Azriel, Baz - Executable Specs
May Walter, Hud - Runtime Intelligence for Continuous Agentic Performance Optimization
Dave Farley - Vibe Coding: Is this really the best we can do?

That set gives the right foundation for Day 2 across skills, context, verification, security, harnesses, runtime feedback, and team enablement.

AI Native DevCon is not over yet!

We are already working on the next AI DevCon, and yes, we are very excited to say that AI DevCon NYC is officially on the way.

If Day 1 gave the frame and Day 2 showed the operating model, NYC is where the conversation gets even more practical. Expect more on skills, harnesses, agent safety, context systems, benchmarking, product workflows, and what it really takes to make AI-native delivery work inside teams.

Super-early-bird seats are available now. If you want to be in the room for the next round of conversations, this is the time to grab a spot.

In the meantime, register for the AI DevCon newsletter. We will release the content shared over the conference, including selected highlights, session clips, notes, slide decks, and workshop materials as they are published.

AI Native DevCon Day 1: Making AI Agents Ready for Enterprise

Rohan Sharma — Tue, 02 Jun 2026 08:37:13 +0000

TL;DR

Day 1 of AI Native DevCon was a practical reality check for AI-native software development. IRL tickets were sold out, the room was packed with 650+ builders. Agents are moving beyond demos, and teams now need better skills, context, verification, security, and enablement to make them dependable.

Hey there! Welcome back. Rohan Sharma here 👋

The first day of DevCon felt less like a normal developer conference and more like the industry collectively agreeing on something important. Coding agents are powerful, but they do not become production-ready; production readiness is earned through reliability, testing, and governance, not just compelling demonstrations.

The common thread was reliability. How do we make AI-native development work for teams, not just polished demos?

Simon Maple opened DevCon by setting the frame. The question is no longer whether AI changes software development. It is how teams, platforms, and engineering cultures adapt now that agents are part of the workflow.

Sessions that shaped Day 1

Skills are the new Code

Guy Podjarny’s keynote, “Skills are the new Code”, gave the day its strongest early thesis. The instructions, skills, and context we give agents are becoming a real unit of software.

If a skill shapes agent behaviour, it needs intent, review, testing, versioning, and maintenance. Your SKILL.md file cannot be the chaotic group chat of your engineering process. It needs structure.

Teams are already relying on agent instructions. The missing piece is treating those instructions like production assets.

Platforms built for humans now need to work for agents

Dana Lawson from Netlify focused on a practical platform challenge. Most dev tools still assume a human is reading logs, checking previews, and interpreting CLI output.

Agents need something different. They need structured signals, machine-readable feedback, and clear next actions. Otherwise they guess, retry blindly, or break something with full confidence.

Giving agents human-only logs is often insufficient. The data may be available, but agents need structured, machine-readable context to use it effectively.

From solo skill hacks to organizational enablement

James Moss from Tessl took the skills conversation into team territory with “Using skills to pay the bills”. Solo agents are easy to experiment with. Team agents are where things get messy.

Every developer can end up with different instructions, different context, and slightly different agent behaviour. If that layer is not shared, reviewed, and versioned, the team does not have one AI workflow. It has ten confused ones wearing the same hoodie.

Patrick Debois expanded that idea in “Coding Agents Don’t Scale Themselves. Neither Do Your Teams.” Organizations cannot simply roll out agent tooling and expect consistent results. Adoption requires enablement, governance, platform thinking, shared practices, and ways to measure whether these systems are genuinely improving outcomes.

Taken together, both talks pointed to the same conclusion: successful agent adoption is less about the model and more about how teams operationalize it.

Skills are also a supply-chain risk

Liran Tal’s “Your AI Agent Installed Malware Because a SKILL.md Told It To” focused on an often-overlooked security challenge. If a skill can influence agent behaviour, it becomes part of your supply chain.

Teams need to audit skills, understand what they instruct agents to do, and avoid blindly installing context files because they look useful. Cute name, dangerous permissions. Classic.

Harness engineering makes agent-first development serious

Ryan Lopopolo from OpenAI discussed Harness Engineering, a useful phrase for what agent-first development needs. Agents need the right context, sensible tool access, clear boundaries, verification loops, and feedback when something goes wrong.

One practical takeaway was that "give the model the entire repository" is not a deployment strategy. Effective agents need carefully scoped context, access to the right tools, and clear boundaries around what they can and cannot do. More context is not always better context.

Ryan also emphasized the importance of verification and feedback loops. Agents can generate code quickly, but production use requires mechanisms to evaluate outputs, catch mistakes, and continuously improve performance. The goal is not autonomous agents operating without oversight. It is systems where agents can work independently while remaining observable and accountable.

The framing made agent-first engineering feel less vague. Agents can execute more, but humans still need to design the operating environment. Less typing every line, more steering the system.

What kept coming up across the day

1. Context is becoming infrastructure

Across Guy Podjarny’s keynote, James Moss’ team workflow talk, Mozilla.ai’s cq, and Robert Overweg’s shared-brain session, there was a clear thread running through the discussions.

Context is not background information anymore. It is infrastructure.

The teams that get real value from agents will not be the ones with the longest prompts. They will be the ones with reusable, maintained, structured context that both humans and agents can trust. Your agent context should not look like your Downloads folder. We all know what that looks like. 😅

2. Verification is the new bottleneck

Shachar Azriel, Simon Martinelli, May Walter, and Dave Farley all circled the same problem from different angles. Generating code is getting easier. Knowing whether that code is correct, safe, aligned with intent, and maintainable is the hard part.

If an AI workflow only optimizes for output speed, it becomes a very fast confusion machine.

3. AI-native development is a team-design problem

The organizational talks made the discussion feel more mature than the usual “everyone becomes 10x” stuff. AI changes review processes, team boundaries, product workflows, release safety, and how work moves from idea to production.

The better advice was boring in the best way. Train people properly, revisit workflows often, keep humans focused on judgment and architecture, and measure outcomes instead of tool adoption.

4. Security cannot be bolted on later

Joseph Katsioloudes from GitHub and Liran Tal from Snyk made security feel immediate. AI can help scale security knowledge, but it also creates new failure modes such as unsafe generated code, malicious skills, supply-chain exposure, prompt injection, and leaky context.

In other words, your agent may be smart, but please do not hand it the production keys and a Red Bull.

A few talks we'd recommend watching

For teams trying to move from experimentation to real AI-native practice, these sessions are worth shortlisting:

Guy Podjarny, Tessl - Skills are the new Code
Dana Lawson, Netlify - Built for Humans. Now Agents Are Here.
James Moss, Tessl - Using skills to pay the bills
Liran Tal, Snyk - Your AI Agent Installed Malware Because a SKILL.md Told It To
Ryan Lopopolo, OpenAI - Harness Engineering
Patrick Debois, Tessl - The Rise of Agent Enablement
Shachar Azriel, Baz - Executable Specs
May Walter, Hud - Runtime Intelligence for Continuous Agentic Performance Optimization
Dave Farley - Vibe Coding - Is this really the best we can do?

That mix gives a strong picture of the day: context, skills, harnesses, verification, runtime feedback, security, and team enablement.

The party bit

After a full day of agent talk, context talk, and slightly scary security talk, the evening party was a good reset. People got to continue the hallway conversations, meet speakers, and process the day.

Honestly, conferences need this part. Some of the best ideas do not happen in the session room. They happen when someone says, “wait, we had the same problem,” and a conversation turns into a new idea, solution, or connection. 😄

A small look at Day 2

Day 2 continues the same themes, with more hands-on sessions and a few focused talks worth tracking through notes or recordings:

Harness engineering beyond code - product & design constraints for agents by Marc Sloan, Tessl
Benchmarking the Agent Era: Measuring Performance Beyond the LLM by Amit Kushwaha, NVIDIA
Connecting Context - Exploring Future Transports by Shaun Smith, Hugging Face
You’re absolutely right, it was your home directory! by Oleg Šelajev, Docker
Don’t Write Prompts, Write Software by Baruch Sadogursky and Macey Baker, Tessl

Day 1 gave the frame. Day 2 goes deeper into harnesses, skills, benchmarking, context, and agent safety.

We'll be sharing more highlights, key takeaways, and session content from Day 2 over the coming weeks.

If you'd like to follow along and get the latest updates as they're released, sign up for the newsletter.

The main takeaway

Day 1 made one thing clear. AI-native development is growing up.

The strongest talks were not about replacing developers or chasing the latest model release. They were about the engineering work around agents: skills, context, harnesses, verification, security, and team enablement.

And yes, your coding agent still has commitment issues. But after Day 1, at least the industry has a better couples therapy plan.

Thank you for joining AI Native DevCon, whether you were in the room or following along virtually.

AI Native DevCon’26: The London conference for developers building with AI

Rohan Sharma — Thu, 21 May 2026 06:06:32 +0000

The bottleneck moved from writing code to governing it.

The promise was 2× throughput. The reality is 2× the review queue, 2× the security exposure, and a CI signal you can no longer trust. AI Native DevCon 2026 is for the engineering leaders who have to figure out how to ship anyway.

AI Native DevCon 2026 lands at The Brewery in London on June 1 and 2, with a hybrid track for remote. This is the conference for VPs of engineering, CTOs, platform owners, security leads, and senior engineers running agents in production, or about to. 500+ builders. Four tracks.

Guy Podjarny, founder of Tessl, organizer of AI Native DevCon, and previously of Snyk, frames the 2026 question:

“If 2025 was the year coding agents started showing real promise, 2026 is the year we figure out how they hold up in production. The challenge is no longer getting an agent to work, it is getting it to work consistently across teams, codebases, and environments without constant human correction.”

The schedule is organized around four tracks: context engineering (building with agents), agent orchestration (verification when CI is no longer enough), organizational enablement (coordination at agent throughput), and agent enablement (security and governance). Each maps to a problem most teams are already hitting.

The agenda is built around the problems they actually have right now.

1. “We do not know how to build with agents yet.”

How the engineer’s role is changing, and what products designed for humans need to do once agents start using them. By 2026, that question lands on every platform team. This is the context engineering track.

Ryan Lopopolo (OpenAI), Harness Engineering. Concrete patterns for systems where humans set direction and agents execute, including the review and approval surfaces that keep it safe at scale.
Dana Lawson (Netlify, CTO), Built for Humans. Now Agents Are Here. What changes in a developer platform when half the users are non-human, and the API and UX decisions Netlify made in response.
Ian Thomas (Meta), AI Native Engineering. How a large engineering org is restructuring workflows around agent-assisted development.
Steve Ruiz (tldraw), Agents on the canvas. Interaction patterns for visual agents, with shipping examples you can copy.

2. “We can generate code. We cannot verify it.”

CI is no longer evidence of correctness. Two years of agent-generated code has proved it. The agent orchestration track is about what to put in its place.

Justin Cormack (ex-Docker CTO), When Tests Lie. Runtime signals that flag agent-introduced drift before it reaches users.
Dave Farley (Founder & CEO of Continuous Delivery Ltd. - 250k on Youtube), Vibe Coding, really? The ideas that may actually survive the AI programming revolution, beyond hype, demos, and generated boilerplate.
Chad Fowler, Regenerative Software. An architectural model where components are regenerated rather than patched, and what verification looks like when code is short-lived by design.

3. “AI writes code faster than teams can coordinate.”

Two years into coding-agent adoption, throughput is up roughly 2×. Coordination cost scaled with it. The organizational enablement track covers review, ownership, and team structure.

Guy Podjarny (Tessl), Skills are the new Code (keynote). The case for treating skills as proper software: versioned, tested, owned, reviewed. With the Tessl Registry now holding 2,000+ evaluated skills, the talk covers what that means for repo structure and review process.
Birgitta Böckeler (Thoughtworks), State of Play: AI Coding Assistants (keynote). Two years of field data on which adoption patterns work and which create future technical debt.
Patrick Debois, The Rise of Agent Enablement. Agent Enablement is the function that owns reliable agent adoption inside an engineering org. It defines standards for skills, evals, and workflows, and sits next to DevOps and Platform Engineering. Patrick’s session covers who owns it, what they do, and how teams formalize it.

4. “Your AI is a new attack surface.”

Vulnerability classes that did not exist 18 months ago, and the controls most teams have not put in place yet. This is the agent enablement track from a security and governance angle.

Liran Tal (Snyk), Your AI Agent Installed Malware Because a SKILL.md Told It To. Live demo of prompt-injection via SKILL.md manifests, with the threat model and mitigations.
Joseph Katsioloudes (GitHub), Code Security Reinvented. How SAST, secret scanning, and review need to change for AI-generated code.
Jack Wotherspoon (Google), Humans vs. Slop. New rules for open source maintainers when an unknown share of contributors are agents.

Why engineering leaders should attend

Five things your team brings back to Monday:

A verification model that does not assume CI catches the regression
Threat models for prompt-injection and SKILL.md attacks, with mitigations
Team structures and review workflows that scale with agent throughput
A working definition of Agent Enablement as a discipline, including ownership and scope
A model for evaluating skills before they go org-wide, with review patterns and KPIs

Hosts and the wider lineup

Hosted by Sam Hepburn and Patrick Debois. Day-one keynote from Lieven Scheire on AI from outside the engineering bubble. The wider roster covers agent observability, MCP transports, runtime intelligence, brownfield adoption, and team-level adoption metrics, with practitioners from Anthropic, OpenAI, NVIDIA, Adobe, Hugging Face, Mozilla.ai, Cisco, Nearform, GitHub, and much more.

Full speaker list and abstracts: tessl.io/devcon.

TL;DR

Dates: June 1 and 2, 2026
Format: 2 days in-person or 1 day virtual
Venue: The Brewery, Barbican, London.
Register: https://luma.com/aidevcon-ldn26?coupon=R30 (R30 auto-applies at checkout to knocks off 30% off)
Bringing a team? Contact at tessl.io/get-in-touch, and we can arrange a group purchase discount.

See you at the event!

Stop trusting your agent skills with vibes. Eliminate the context security risk.

Tessl — Fri, 15 May 2026 04:55:29 +0000

When you install an npm package, you can run npm audit. When you install a Python package, there's pip-audit. But when you install plugins that give your AI agent new skills and rules, you know, things that directly shape how it reasons and what it does, what do you run?

If your answer is "nothing", you're not alone, and that's why I built tessl-audit! You can check it out on GitHub and npm.

Why this matters more than you think

Agent plugins are instructions that get loaded into your AI agent's context. A plugin with a security issue doesn't just expose a server endpoint. It can influence the agent's behaviour in ways that are subtle and hard to detect, perhaps nudging it toward unsafe patterns, exposing data it shouldn't, or simply making it worse at its job.

Ask yourself these three questions about your agent skills, and if the answer to any of them is no, you’re seconds away from being able to say yes, with tessl-audit.

Have all your skills been security scanned? If so, what was the result?
Can you prove your skills are any good? Quality scores tell you how well-written and complete a plugin is. A low score means the agent is getting poor guidance.
Do your skills and plugins actually help? Uplift scores measure whether a plugin improves agent task performance compared to a vanilla agent alone.

Join us at AI Native DevCon (use C0DE30 for 30% discount)

Why not try it right now?

It’s a free open source tool that uses Tessl under the covers. If you have a Tessl project with plugins installed, just run this in your project root:

npx tessl-audit

Wait, is that it? Absolutely, that's it. It reads your tessl.json, fetches live data from the registry for every plugin, and prints a report in about 30 seconds.

The script begins by looking through all your context file that it finds in the tessl.json manifest file. This should complete pretty quickly and you’ll soon see the table below, with a breakdown of your project context., and the types of warnings that have been picked up.

Next, the tool gives a posture summary of all of your context, giving more details of the riskiest skills in your project and what the issues are.

You can click through on any of these links to see the actual issues in the registry web UI.

And finally, the tool provides next step actions of the CLI commands to use (you can use an agent to call these also) to optimize, create and run evals on your skills.

The "so what" for each finding

Advisory, Risky, or Critical security status?

The report prints each flagged plugin with its warning codes and a direct link to the full security report on the registry. No need to chase them down, the security posture report lets you see the full summary in one listing, allowing you to deep dive here needed. Just open the link, read the finding, decide if it applies to your use case.

Quality below 80%?

The plugin you’re using is giving your agent incomplete or poorly-structured guidance. Run:

tessl skill review --optimize workspace/plugin-name

This runs a quality review and applies automatic improvements.

No uplift data?

The plugin has never been evaluated against real tasks — so you have no idea if it's helping or hurting. Fix that:

tessl scenario generate --count 5 workspace/plugin-name
tessl eval run workspace/plugin-name

Generate a set of test scenarios from the plugin, then run the eval. You'll get a concrete uplift score showing whether the plugin is worth keeping.

The bigger picture

Every team that uses AI agents is building a dependency graph of skills, rules, and knowledge, just like they build a dependency graph of packages. The tooling for auditing that graph is still being built, but the risks are real and growing.

tessl-audit is a small, practical step: one command, zero installation, actionable output. Run it today and find out what your agent is actually working with.

npx tessl-audit

tessl-audit requires the Tessl CLI (no worries, it’s already a dependency) and an authenticated Tessl session (just create a free account if you haven’t got one). You’ll need a tessl.json in order to run the tessl-audit tool, which is a context manifest tile.

Useful docs:

Tessl Admin Guide: Organizations, Workspaces, and Roles

Tessl — Thu, 14 May 2026 06:45:55 +0000

Just signed up to Tessl? Wondering next steps to rolling Tessl out to your team? The following article will take you through the steps of managing your top level Organization, invite your users, set policy items, then create your workspaces, assigning membership to those workspaces and defining their roles.

How Organizations and Workspaces work in Tessl

Organizations are top level entities, often representing the billing or corporate entity, with a subcategory called Workspaces that provide role-based access to the various users across the company.

Setting up your Tessl Organization

Organizations are sometimes created during the presales phase of acquiring Tessl, or may be created later. If one has not been created, it will be auto created when you create your first workspace. If prompted, click Create workspace and name it after your team (i.e. YourCompanyName-Engineering) to start.

Note workspace names must be unique at this time, and will appear in plugin-names when searched. This is most notable if the plugins are published publicly.

The workspace should now be visible from the main interface

The organization can now be observed by clicking your Account, where your name is displayed, on the bottom left

Once created, navigate to Settings for your Organization, rename the organization to your company name and specify if users can publicly share skills by enabling the button.

Creating and managing Users in Tessl

Next, invite users to your organization, by navigating to the Organization’s Members menu, assigning the workspaces the users will have access to. Users will be created with the members role, able to see, search and install skills from the chosen workspaces. Permissions can be promoted from the Workspace Members menu, which will be discussed later below. Users will need to accept the invite they are sent.

Once created, you can elevate a user to Admin to allow workspace creation or manage users. To do so, navigate to the Organization Members screen, and click the three dots under Actions. Assign an appropriate role. Examples will be provided below of some common configurations.

Admin keys

Admin keys are for integrations and applications where programmatic access is required across workspaces. This is typically used for automation purposes and an expiration can be set up to one year.

Managing Workspaces and Users in Tessl

Click the workspace drop-down to navigate workspaces. Navigate to Members at the workspace level to specify Roles for users who require more capabilities within the workspace, such as running evaluations, publishing or managing users.

To modify a user, search for their name, select their checkbox, a role, and click the Add button.

Example role configurations for your team(s)

The following users demonstrate common configurations and roles that may be used when rolling Tessl out:

Samira - Org. Admin

Samira, the administrator and skills champion, needs the ability to manage all workspaces, the ability to assign users, and create new workspaces. Make her an Organization admin.

Eddie - Lead Engineer

Another user, Eddie, might be a member of an engineering workspace. He needs to be able to use plugins (skills) that have been published, but may need to have access to publish skills within the engineering workspace for others on his team. This could mean Eddie is the publisher role in certain workspaces. He may also be a Member role of other workspaces where he only needs to search and install from.

Jennifer - Manager

Jennifer may require the ability to add users to a workspace that she owns, publish, and possibly need the ability to remove other managers etc. Typically the workspace permission "Owner" or "manager" may be given to that user, depending on the need to remove other "owners" or delete workspace.

Joe - New hire engineer

Finally, Joe, a new hire, has the ability to search and install skills from the engineering workspace, but does not have the ability to share/create skills until later, after they’ve gained a little more experience. Joe would be made a member of “engineering” with just a “consumer” role.

Next steps!

Now that you have your users in, and assigned roles to the different workspaces, you and your users can:

Start creating new skills
Evaluate new or existing skill effectiveness through using Reviews, and Evals
Publish those skills to the Tessl registry to share them for your users and agents

Let us know what you think! Tessl would love to hear from you through any one of our feedback channels (Discord, Email, CLI Feedback, etc)

Originally published on Tessl.blogs

GPT-5.5 is OpenAI's best model. But paying more for it makes no sense.

Rohan Sharma — Wed, 06 May 2026 13:13:28 +0000

We added OpenAI’s gpt-5.5 model to our eval suite the day it launched. We ran 1,742 tests overall, which included over 45 task scenarios across using 11 real engineering skills, each run 6 times and averaged the data, which is shown in this blog.

TL;DR

The gpt-5.5 model has the highest raw capability of any OpenAI model we've tested. When it uses agent skills and performs the same tasks, it pretty much ties with gpt-5.4 on score but costs 63% more per run.

Question	Answer
Best Codex model out of the box?	gpt-5.5: 75.6 avg baseline, highest in the family
Best Codex model with skills loaded?	gpt-5.4 and gpt-5.5 tie at 89.3 and 89.4
Worth the 63% price premium over gpt-5.4?	With this data, we don’t think so
Any scenario where it wins?	Latency: 89.5s vs 135.4s for gpt-5.4
Should you use gpt-5.3 instead?	No, oddly enough, gpt-5.3 costs 47% more than gpt-5.4 for a worse result because of the token bloat.

The one-line verdict: gpt-5.5 is the most capable Codex model we've benchmarked, and when using agent skills to guide with tasks, it performs pretty much identically to a model that costs a third less. The interesting story is actually gpt-5.3, which costs more than gpt-5.4 and scores worse, because of the token bloat in 5.3. The per-token cost is, of course, more expensive with gpt-5.5.

The Key Takeaways

The most counterintuitive thing in this data: gpt-5.5 and gpt-5.4 score within 0.1 points of each other when given domain skills, 89.4 vs 89.3. The self-sufficiency story holds directionally, but these two models are functionally the same on skill-augmented work. The question is purely cost.

Join us at AI Native DevCon (use C0DE30 for 30% discount)

The gpt-5.3 story is sharper. The headline numbers put it at 83.9 with skills against 89.3 for gpt-5.4, a 5.4 point gap. It also costs $0.44 per run against $0.30 for gpt-5.4. You pay more and get less, which is a complete description of a bad deal.

You pay $0.49/run for 89.4 points with gpt-5.5. You pay $0.30/run for 89.3 points with gpt-5.4. The only dimension where gpt-5.5 leads is latency, at 89.5s against 135.4s. If you're running latency-constrained agents and can absorb the cost, it's a defensible choice. Otherwise you're paying a 63% premium for 0.1 points.

How It Stacks Up

Model	Task Scores (using agent skill)	Cost/run	Score/$	Avg lift
claude-opus-4-7	93.4	$1.00	93	+12.6
cursor:composer-2	89.6	$0.23	389	+15.4
gpt-5.5	89.4	$0.49	182	+13.8
gpt-5.4	89.3	$0.30	298	+15.2
gpt-5.3-codex	83.9	$0.44	191	+18.4
gpt-5-codex	78.7	$1.05	75	+10.0

gpt-5.5 and gpt-5.4 are functionally interchangeable on skill performance. The question is whether 45 seconds per run is worth $0.19.

What We Tested

This benchmark runs on Tessl, an agentic evaluation platform. A skill is a SKILL.md file, which is a structured markdown document containing rules, patterns, and examples for a specific domain. For the baseline run, the agent sees only the task prompt with no additional context. For the with-skill run, the SKILL.md is loaded into the agent's context alongside the task, same model, same task, same rubric. The score delta is the lift. The platform runs each scenario twice and scores the output against a pre-written rubric checklist automatically.

Each scenario was run 6 times and scored independently; all figures are averaged across those runs.

Why rubric checklists? Because the scenarios have objectively right answers. "Does the agent delete .eslintrc.json and create eslint.config.js?" is not a matter of opinion. Neither is "Does it use PKCE method S256?" or "Does it call pipeline() instead of chaining .pipe()?" Binary criteria eliminate evaluation noise wherever possible.

Example rubric: Modernize the Linting Setup for a Node.js Library, 11 criteria, 101 points.

Criterion	Points	Pass condition
neostandard installed	10	neostandard present in devDependencies
standard uninstalled	10	standard absent from devDependencies
Flat config file	10	eslint.config.js or .mjs exists, not .eslintrc*
neostandard in config	10	Config imports from neostandard and calls neostandard()
lint script uses eslint	10	package.json lint script runs eslint ., not neostandard . or standard .
migrate command used	10	Instructions reference npx neostandard --migrate to generate the config
lint:fix script present	8	lint:fix script runs eslint . --fix
CI uses non-fix run	8	CI config runs lint without --fix
standard config removed	8	No top-level standard key in package.json
lint-staged uses eslint	8	Pre-commit hook runs eslint --fix, not neostandard or standard
eslint@9 installed	8	eslint at version 9.x in devDependencies

A model that migrates the config correctly but leaves standard in devDependencies scores 91/101. One that creates eslint.config.js alongside .eslintrc.json instead of replacing it scores 0 on three criteria at once.

All skills and rubrics are published at simon/skills on the Tessl registry. Full eval results for this run can be found here.

The Data

Baseline scores (no skill), sorted by highest average

Model	docs	fastify	init	lint	node	node-core	oauth	octocat	skill-opt	snip	ts	Avg
claude-opus-4-7	85.7	80.9	79.7	92.9	73.7	91.6	75.7	84.7	85.0	60.1	78.8	80.8
gpt-5.5	89.9	71.8	63.6	94.4	64.6	72.3	73.6	85.5	83.2	54.7	78.3	75.6
gpt-5.4	87.6	66.7	71.1	84.5	62.3	77.4	77.5	80.5	80.8	50.9	75.9	74.1
cursor:composer-2	84.3	74.7	61.6	94.1	65.4	78.8	73.1	78.5	82.3	58.5	65.5	74.3
gpt-5-codex	80.2	67.3	60.2	84.9	60.4	76.5	72.9	75.3	63.8	47.5	66.5	68.7
gpt-5.3-codex	63.5	65.4	52.1	76.5	62.4	75.3	77.9	68.3	70.5	42.1	66.4	65.5

With-skill scores, sorted by highest average

Model	docs	fastify	init	lint	node	node-core	oauth	octocat	skill-opt	snip	ts	Avg
claude-opus-4-7	96.7	98.9	82.3	97.2	95.1	84.7	94.3	97.7	99.7	92.9	88.0	93.4
cursor:composer-2	95.6	93.9	85.7	96.4	94.0	92.3	83.9	94.5	93.7	85.3	70.4	89.6
gpt-5.5	96.1	86.0	81.8	96.3	88.3	88.6	91.7	92.1	96.0	86.5	79.2	89.4
gpt-5.4	97.1	76.9	80.0	98.1	84.8	93.7	91.6	95.7	94.6	90.9	79.0	89.3
gpt-5.3-codex	96.9	86.1	80.4	90.2	75.9	77.1	93.1	92.3	77.3	79.4	74.1	83.9
gpt-5-codex	62.9	88.9	74.8	92.1	66.3	77.7	89.3	85.9	80.7	86.0	61.1	78.7

Lift: what skills actually added per model, sorted by highest average lift

Model	docs	fastify	init	lint	node	node-core	oauth	octocat	skill-opt	snip	ts	Avg lift
gpt-5.3-codex	+33.4	+20.7	+28.3	+13.7	+13.5	+1.8	+15.2	+24.0	+6.8	+37.3	+7.7	+18.4
cursor:composer-2	+11.3	+19.2	+24.1	+2.3	+28.6	+13.5	+10.8	+16.0	+11.4	+26.8	+4.9	+15.4
gpt-5.4	+9.5	+10.2	+8.9	+13.6	+22.5	+16.3	+14.1	+15.2	+13.8	+40.0	+3.1	+15.2
gpt-5.5	+6.2	+14.2	+18.2	+1.9	+23.7	+16.3	+18.1	+6.6	+12.8	+31.8	+0.9	+13.8
claude-opus-4-7	+11.0	+18.0	+2.6	+4.3	+21.4	-6.9	+18.6	+13.0	+14.7	+32.8	+9.2	+12.6
gpt-5-codex	-17.3	+21.6	+14.6	+7.2	+5.9	+1.2	+16.4	+10.6	+16.9	+38.5	-5.4	+10.0

Reading the lift table. A few observations:

claude-opus-4-7 node-core: -6.9. Opus starts at 91.6 baseline on Node.js internals, the highest raw score on any skill for any model in the benchmark. Adding a skill that prescribes specific patterns for primordials and commit message format on top of a model that already knows the material produced interference, not uplift. The skill was written to close a gap that Opus doesn't have.
gpt-5-codex docs: -17.3. The same skill that boosted gpt-5.3-codex by +33.4 points degraded gpt-5-codex by 17. The Diátaxis framework is highly prescriptive about structure: tutorial titles must start with verbs, reference sections must contain no instruction. gpt-5-codex starts at 80.2 baseline for docs, it produces fluent, correct-seeming prose, and the skill's structural constraints appear to actively conflict with its default output style. High baseline does not predict positive lift.
gpt-5-codex ts: -5.4. Same pattern. A 66.5 baseline on TypeScript drops to 61.1 with the skill. The TypeScript skill enforces branded types and zero any, rules that require restructuring code rather than extending it. For a model with established TypeScript habits, the prescriptive guidance appears to create noise rather than correct the specific gaps.
claude-opus-4-7 init: +2.6. The lowest positive lift in the table. Claude Opus is the model that introduced the AGENTS.md convention, it was already near-ceiling on this skill before any context was added.
gpt-5.4 snip: +40.0. The single highest lift cell in the entire dataset. snipgrapher's private CLI documentation gives a model that knows nothing a complete specification for a tool it's never encountered. gpt-5.4's strong instruction-following amplifies that advantage cleanly.

The cost of running gpt-5.5 vs the alternatives

Model	Cost/run (with skill)	Time (with skill)	Score	Score/$
cursor:composer-2	$0.23	152.0s	89.6	389
gpt-5.4	$0.30	135.4s	89.3	298
gpt-5.3-codex	$0.44	87.9s	83.9	191
gpt-5.5	$0.49	89.5s	89.4	182
claude-opus-4-7	$1.00	158.9s	93.4	93
gpt-5-codex	$1.05	136.2s	78.7	75

More details about the 11 skills and scenarios

fastify-best-practices: Fastify has strong opinions, and the skill encodes them. Scenarios: Security Hardening for a Healthcare Web API (CORS scoped to two named origins, CSP + HSTS headers, HTTPS redirect, a wildcard * or a missing header scores zero); Authentication Service for a SaaS Platform (passwords migrated from bcrypt to argon2id, in-memory rate limiting replaced with Redis for multi-instance correctness, SIGTERM handled with close-with-grace); Protecting a Product Catalogue API from Overload (does it reach for @fastify/under-pressure or invent its own backpressure loop?); Order Management API with PostgreSQL (uses @fastify/postgres with correct pool lifecycle, not raw pg); Consistent Error Handling for a Multi-Tenant SaaS API (typed createError, uniform JSON shape, no stack traces to clients).

node-best-practices: The patterns in this skill diverge from what you'd find on Stack Overflow. Scenarios: Hardening Logging in a Fintech API (pino must redact auth tokens and raw card fields before they reach the SIEM, masking after the fact doesn't count); Webhook Receiver Service (structured logging of sensitive payment provider fields, graceful shutdown under concurrent in-flight requests); Fix Throughput Degradation in a High-Load API Gateway (dns.lookup() saturating the libuv thread pool, the fix is dns.resolve4() and UV_THREADPOOL_SIZE, not a caching layer); High-Throughput Merchant DNS Routing Service (concurrent resolution under load, observable thread pool saturation).

snipgrapher: A custom internal CLI with a non-public API. The model has never seen its documentation. At baseline, every model is essentially guessing (avg 50-60/100). With the skill, agents either follow the spec or they don't. Scenarios: Automating Changelog Snippet Images in CI (correct flag order, env var overrides, pipeline integration) and Code Snippet Image Pipeline for Documentation Site (batch rendering, profile configuration). This skill delivers the highest lift of any in the benchmark across every model, averaging between 27 and 40 points. The reason: it encodes knowledge that does not exist on the internet. Public skills are becoming less necessary as frontier models grow stronger. Private tooling is where skills still dominate.

typescript-magician: Not "add types to this function." Scenarios: Domain-Safe Payment Processing Types (branded types for AccountId, PaymentId, RefundId, plain type aliases don't count, as casts score zero); Product Catalog API for an E-Commerce Platform (TypeBox schemas inferred as TypeScript types end-to-end, internal cost fields stripped from public responses, no any); Eliminate any from a Data Pipeline Utility Library (tsc output captured before and after, zero any remaining, no @ts-ignore); Project Bootstrap: Node.js TypeScript Service (native --strip-types, no ts-node, no build step, no tsc in the start script).

oauth: Is the implicit flow explicitly removed? Is PKCE method S256? Is the refresh token replaced on rotation? Scenarios: Add User Authentication to a Fastify API (full Authorization Code + PKCE flow with @fastify/oauth2, state verification, token rotation); OAuth Login Integration for a Fastify Web App (CSRF-hardened flow, @fastify/session for state, correct cookie flags).

linting-neostandard-eslint9: ESLint v9's flat config is a breaking change. Scenarios checked whether agents actually migrated, not just created a new config alongside the old one. Is .eslintrc.json gone? Is standard removed from devDependencies? Scenarios: Modernize the Linting Setup (two variants: envparser open-source library and payments-api service); Add Linting to the Inventory Service (neostandard from scratch); Set Up Automated Lint Enforcement (husky + lint-staged pre-commit hook, CI step that blocks on violations).

documentation: Based on the Diátaxis framework. The skill teaches agents when to write a tutorial vs a how-to vs reference vs explanation. Scenarios: Restructure Documentation for a Configuration Library (sprawling confz README split into four Diátaxis types, tutorial title must start with a verb, reference section must contain no instruction); Getting Started Guide for a CLI Deployment Tool (shipctl onboarding tutorial with Goal→Prerequisites→Numbered steps→Verifiable result structure, no conceptual digressions in the steps).

init: Writing AGENTS.md / CLAUDE.md files that actually help AI assistants. Scenarios: Set Up Agent Instructions for a Growing Python Monorepo (3-year-old codebase, multiple service packages, identify the three constraints that cause the most agent damage); Set Up Agent Instructions for a Node.js Monorepo (workspace-aware package manager, per-package test commands, legacy directory exclusion); Audit and Slim Down a Bloated AGENTS.md (what to cut, what to keep, signal vs noise after a year of uncurated growth); Set Up Agent Instructions for a Growing Monorepo (hierarchical root-level vs per-package instructions, discoverability filtering).

octocat: GitHub CLI patterns and correct flag usage. Scenarios: Automate Feature Branch PR Submission (correct gh pr create flags, CI polling with gh run watch, merge only after checks pass); Preparing Commits for a Node.js Core Module Contribution (subsystem prefix, 72-char subject, Reviewed-By trailers, the format changelog toolbots parse); Prepare Node.js Core Contribution Commits (backport workflow, correct metadata for automated release pipelines); Automate Pull Request Workflow (reusable shell script, idempotent, surfaces CI failures before merge). nodejs-core: Contributing to Node.js core: primordials, commit message format, native addons with AsyncWorker. Scenarios: Product Catalog Caching Service (async-cache-dedupe, concurrency control to prevent thundering herd on a rate-limited upstream); Microservice Routing Layer: Latency Spike Investigation (diagnosing UV_THREADPOOL_SIZE exhaustion, dns.lookup() blocking the pool); Diagnose and Fix V8 Performance Regression in Analytics Processor (--prof, --trace-opt, reading isolate-*.log, acting on deoptimization reasons). skill-optimizer: Meta: given a poorly-written skill or benchmark report, improve it or interpret it correctly.

The Verdict

gpt-5.5 is a better model than gpt-5.4 on raw capability, and on latency it is not close. For everything else, they are the same model at different price points. Pay the 63% premium if you need the speed. Skip it if you care about cost or value per dollar.

The model to actually avoid is gpt-5.3. It costs 47% more than gpt-5.4 and scores 5.4 points worse. If you are running gpt-5.3 today, the case for switching to gpt-5.4 is strong on both cost and performance.

Frontier models are becoming more self-sufficient. The ROI on domain skills is concentrating in genuinely proprietary knowledge: your internal APIs, your custom tooling, patterns that simply aren't on the internet. Snipgrapher lifted every model by 27 to 40 points because no model had ever seen its documentation. ESLint v9 flat config lifted them by 2 to 14 points because capable models already know it.

The original author of this blog is Simon Maple and is originally posted on tessl.io/blog

Simon Maple is the Head of Developer Relations at Tessl, and AI Native Dev co-host. Previously, Simon was the Field CTO, and VP Developer Relations at Snyk, ZeroTurnaround, and IBM. He became a Java Champion in 2014, JavaOne Rockstar speaker in 2014 and 2017, Duke’s Choice award winner, Virtual JUG founder and organiser, and London Java Community co-leader.