DEV Community: Tessl

Your benchmarks are lying to you, and your judge is to blame!

Tessl — Tue, 19 May 2026 09:18:00 +0000

Last week I published a benchmark comparing six models across eleven agent skills. The numbers in that post are averages, and we did not explain why.

When I shared the data internally, Maria from our AI Research team pointed out something that we should take very seriously: an LLM judge is likely to favour outputs from its own model family.

So I ran the full benchmark again with a second judge, then a third, to see if this hypothesis held any water. The scores shifted, the rankings moved, and one model swung 47 percentage points on a single skill depending on the judge who graded it. If you are publishing or trusting eval numbers from a single LLM judge, you are partly benchmarking judge preference rather than model capability.

TL;DR for this post

We’re all busy people, here’s the tl;dr.

The same benchmark, graded by three different LLM judges, produced different scores and different rankings. One model swung 47 points on a single skill.
Sonnet is the most generous judge, GPT-5.5 the strictest. The gap between them averages 6.9 points across all models and skills.
LLM judges favour their own model family. Opus gave itself a 4.6 point boost over what the other two judges awarded it.
Rankings only stay stable at the top of the agents we tested. opus-4-7 held first place under all three judges. Everything below it moved position.
Eval criteria built around binary, verifiable things (file deleted or not, flag enabled or not) produce stable scores regardless of the judge. Skills that require qualitative judgment can easily swing 25 percentage points.
Fix: run multiple judges and average. Favour the same judge as the model you’ll tend to use in development, if you know. Design rubrics with yes/no criteria wherever the task allows it.

Join us at AI Native DevCon (use C0DE30 for 30% discount)

The Setup

Six models, eleven skills, five scenarios per agent skill, one rubric. The only variable was the scoring model, i.e. the judge: Sonnet, GPT-5.5, and Opus-4-7 each graded every run independently. The figures in our main benchmark are averaged across all three. This post is about what happens before the averaging.

The raw results

The Tessl UI shows eval results as an average of all the runs, but you can still see all the information about the scenarios/tasks that were set and how they fared with and without the skills, on the Tessl registry here.

Criteria are pretty easy to create with Tessl. Once you’ve installed and authenticated with a free account using Tessl command line tool:

`$ curl -fsSL https://get.tessl.io | sh`

Simply ask your agent to create them using Tessl, or just run the following at the command line:

`$ tessl scenario generate <path/to/tile> --count=5`

You can then download them to disk once to validate, and make any changes you want:

`tessl scenario download --last`

You can now run them, choosing the using the model you want to run the scenarios with the --agent flag, as well as the model you wish to judge the output using the --scorer-agent flag.

`tessl eval run <path/to/tile> --agent=claude:claude-opus-4-6 --scorer-agent codex:gpt-5.5`

Judge Strictness: Sonnet Grades Easiest, GPT-5.5 Grades Hardest

Averaged across all six models and all eleven skills, here is what each judge returned:

Judge	Avg without-skill	Avg with-skill	Avg lift
Sonnet	76.1	90.3	14.2
Opus-4-7	72.6	88.3	15.7
GPT-5.5	70.7	83.4	12.7

Sonnet grades most generously. GPT-5.5 is 6.9 points stricter on average. If your pipeline scores agents with Sonnet as the default judge, your numbers are probably 5 to 7 points higher than a stricter grader would return. That gap is real and it is not uniformly distributed across models.

The Rankings Shift

Here are the per-judge leaderboards for with-skill performance, using the same models and rubrics with only the judge swapped:

Rank	Sonnet	GPT-5.5	Opus-4-7
1	opus-4-7 (94.5)	opus-4-7 (89.2)	opus-4-7 (96.5)
2	gpt-5.4 (92.7)	gpt-5.5 (88.4)	gpt-5.5 (92.3)
3	gpt-5.3 (91.9)	composer (88.0)	composer (90.3)
4	composer (90.5)	gpt-5.4 (86.5)	gpt-5.4 (88.8)
5	gpt-5.5 (87.4)	gpt-5.3 (75.7)	gpt-5.3 (84.0)
6	gpt-5-codex (85.1)	gpt-5-codex (72.9)	gpt-5-codex (78.1)

The leaderboard format shows position but hides distances. Here are the raw scores:

Model	Sonnet	GPT-5.5	Opus-4-7	Avg	Swing
opus-4-7	94.5	89.2	96.5	93.4	7.3
composer	90.5	88.0	90.3	89.6	2.5
gpt-5.5	87.4	88.4	92.3	89.4	4.9
gpt-5.4	92.7	86.5	88.8	89.3	6.2
gpt-5.3	91.9	75.7	84.0	83.9	16.2
gpt-5-codex	85.1	72.9	78.1	78.7	12.2

The swing column is the gap between the highest and lowest score any judge gave a model. composer swings 2.5 points, meaning all three judges broadly agree on it. gpt-5.3 swings 16.2 points, which is more than the gap between first and last place in the averaged rankings.

opus-4-7 holds first place under every judge, and that is the one stable finding in the table. Everything else shifts. gpt-5.3 sits third under Sonnet and falls to fifth under both GPT-5.5 and Opus. gpt-5.5 sits fifth under Sonnet and climbs to second under the other two. The Sonnet-only leaderboard, which is what most default Tessl runs would have produced, gives a flattering picture of gpt-5.3 and an unflattering one of gpt-5.5.

Judge choice also affects how much credit each model gets for using skill context. Lift scores, meaning the gap between baseline and with-skill performance, vary considerably by judge:

Model	Sonnet lift	GPT-5.5 lift	Opus lift	Avg lift
gpt-5.3	16.1	16.2	22.9	18.4
composer	16.9	13.8	15.4	15.4
gpt-5.4	16.8	13.5	15.3	15.2
gpt-5.5	10.2	15.1	16.2	13.8
opus-4-7	14.0	9.7	14.1	12.6
gpt-5-codex	11.3	8.3	10.4	10.0

The Opus judge gives gpt-5.3 a lift of 22.9 points. Sonnet and GPT-5.5 give it 16. The rubric was identical. The disagreement is purely about whether gpt-5.3's output counted as genuine compliance or a close approximation, and a single judge cannot tell you which reading is correct.

Self-Judge Bias Is Measurable

The results split along model family lines, though not symmetrically.

Model	Own judge score	Other judges avg	Boost
opus-4-7 (Opus judge)	96.5	91.9	+4.6
gpt-5.5 (GPT-5.5 judge)	88.4	89.9	-1.5

The Opus case is unambiguous. Opus gives itself 96.5; Sonnet gives it 94.5; GPT-5.5 gives it 89.2. The 7.3 point gap between Opus-as-judge and GPT-5.5-as-judge for the same model on the same runs is entirely a grading artefact. The gpt-5.5 case does not follow the same pattern: GPT-5.5 actually scores its own model lower than the other two judges do, and Opus gives gpt-5.5 its highest score at 92.3. Self-favour exists but is not symmetric, and its size and direction vary by model and judge pairing.

The practical consequence for the Opus case specifically: if you are using Claude models to grade Claude outputs, expect a systematic upward bias of 4 to 5 points. It does not show up as a bias in your data; it just looks like good scores.

The Averaged Picture

Given all this variance, here is what you get when you average across all three judges:

Model	Avg baseline	Avg with-skill	Avg lift
opus-4-7	80.8	93.4	12.6
composer	74.2	89.6	15.4
gpt-5.5	75.5	89.4	13.8
gpt-5.4	74.1	89.3	15.2
gpt-5.3	65.5	83.9	18.4
gpt-5-codex	68.7	78.7	10.0

The lift column deserves a second look. gpt-5.3 shows the largest average lift at 18.4 points, which sounds like a strength, but its baseline is also the weakest of any non-codex model by nearly ten points. It benefits most from skill context and starts furthest behind without it.

What the gpt-5.3 Drop Actually Tells Us

gpt-5.3 scored 91.9 under Sonnet, 75.7 under GPT-5.5, and 84.0 under Opus. Two judges independently came back with substantially lower scores, which means the Sonnet-only number was inflated, and that inflation was specific to gpt-5.3 in a way it was not specific to gpt-5.4 or composer.

The pattern in the per-skill data points to one cause: gpt-5.3 produces outputs that are in the right direction but not precisely correct. Sonnet, the most generous judge, gives partial credit. GPT-5.5, the strictest, does not. If you care about whether your agent follows a spec exactly rather than approximately, GPT-5.5's score is the more informative one. If you care about general capability, the average of three judges is probably right.

What to Do About It

Single-judge evals are benchmarking judge preference as much as model capability. Running multiple judges and averaging the results fixes this: three independent judges will smooth out individual preferences and give you a number that is harder to game and more stable across reruns.

Beyond averaging, design your rubric for binary criteria wherever the task allows it. "Is the file deleted?" is a better eval item than "how well did the agent explain the migration?" The first gives every judge the same answer. The second gives every judge a different one.

The models that perform consistently across all three judges in this benchmark, gpt-5.4 and composer, share one characteristic: their outputs are correct rather than approximately correct. A strict grader and a generous one disagree less when the answer is unambiguously right.

Stop trusting your agent skills with vibes. Eliminate the context security risk.

Tessl — Fri, 15 May 2026 04:55:29 +0000

When you install an npm package, you can run npm audit. When you install a Python package, there's pip-audit. But when you install plugins that give your AI agent new skills and rules, you know, things that directly shape how it reasons and what it does, what do you run?

If your answer is "nothing", you're not alone, and that's why I built tessl-audit! You can check it out on GitHub and npm.

Why this matters more than you think

Agent plugins are instructions that get loaded into your AI agent's context. A plugin with a security issue doesn't just expose a server endpoint. It can influence the agent's behaviour in ways that are subtle and hard to detect, perhaps nudging it toward unsafe patterns, exposing data it shouldn't, or simply making it worse at its job.

Ask yourself these three questions about your agent skills, and if the answer to any of them is no, you’re seconds away from being able to say yes, with tessl-audit.

Have all your skills been security scanned? If so, what was the result?
Can you prove your skills are any good? Quality scores tell you how well-written and complete a plugin is. A low score means the agent is getting poor guidance.
Do your skills and plugins actually help? Uplift scores measure whether a plugin improves agent task performance compared to a vanilla agent alone.

Join us at AI Native DevCon (use C0DE30 for 30% discount)

Why not try it right now?

It’s a free open source tool that uses Tessl under the covers. If you have a Tessl project with plugins installed, just run this in your project root:

npx tessl-audit

Wait, is that it? Absolutely, that's it. It reads your tessl.json, fetches live data from the registry for every plugin, and prints a report in about 30 seconds.

The script begins by looking through all your context file that it finds in the tessl.json manifest file. This should complete pretty quickly and you’ll soon see the table below, with a breakdown of your project context., and the types of warnings that have been picked up.

Next, the tool gives a posture summary of all of your context, giving more details of the riskiest skills in your project and what the issues are.

You can click through on any of these links to see the actual issues in the registry web UI.

And finally, the tool provides next step actions of the CLI commands to use (you can use an agent to call these also) to optimize, create and run evals on your skills.

The "so what" for each finding

Advisory, Risky, or Critical security status?

The report prints each flagged plugin with its warning codes and a direct link to the full security report on the registry. No need to chase them down, the security posture report lets you see the full summary in one listing, allowing you to deep dive here needed. Just open the link, read the finding, decide if it applies to your use case.

Quality below 80%?

The plugin you’re using is giving your agent incomplete or poorly-structured guidance. Run:

tessl skill review --optimize workspace/plugin-name

This runs a quality review and applies automatic improvements.

No uplift data?

The plugin has never been evaluated against real tasks — so you have no idea if it's helping or hurting. Fix that:

tessl scenario generate --count 5 workspace/plugin-name
tessl eval run workspace/plugin-name

Generate a set of test scenarios from the plugin, then run the eval. You'll get a concrete uplift score showing whether the plugin is worth keeping.

The bigger picture

Every team that uses AI agents is building a dependency graph of skills, rules, and knowledge, just like they build a dependency graph of packages. The tooling for auditing that graph is still being built, but the risks are real and growing.

tessl-audit is a small, practical step: one command, zero installation, actionable output. Run it today and find out what your agent is actually working with.

npx tessl-audit

tessl-audit requires the Tessl CLI (no worries, it’s already a dependency) and an authenticated Tessl session (just create a free account if you haven’t got one). You’ll need a tessl.json in order to run the tessl-audit tool, which is a context manifest tile.

Useful docs:

Tessl Admin Guide: Organizations, Workspaces, and Roles

Tessl — Thu, 14 May 2026 06:45:55 +0000

Just signed up to Tessl? Wondering next steps to rolling Tessl out to your team? The following article will take you through the steps of managing your top level Organization, invite your users, set policy items, then create your workspaces, assigning membership to those workspaces and defining their roles.

How Organizations and Workspaces work in Tessl

Organizations are top level entities, often representing the billing or corporate entity, with a subcategory called Workspaces that provide role-based access to the various users across the company.

Setting up your Tessl Organization

Organizations are sometimes created during the presales phase of acquiring Tessl, or may be created later. If one has not been created, it will be auto created when you create your first workspace. If prompted, click Create workspace and name it after your team (i.e. YourCompanyName-Engineering) to start.

Note workspace names must be unique at this time, and will appear in plugin-names when searched. This is most notable if the plugins are published publicly.

The workspace should now be visible from the main interface

The organization can now be observed by clicking your Account, where your name is displayed, on the bottom left

Once created, navigate to Settings for your Organization, rename the organization to your company name and specify if users can publicly share skills by enabling the button.

Creating and managing Users in Tessl

Next, invite users to your organization, by navigating to the Organization’s Members menu, assigning the workspaces the users will have access to. Users will be created with the members role, able to see, search and install skills from the chosen workspaces. Permissions can be promoted from the Workspace Members menu, which will be discussed later below. Users will need to accept the invite they are sent.

Once created, you can elevate a user to Admin to allow workspace creation or manage users. To do so, navigate to the Organization Members screen, and click the three dots under Actions. Assign an appropriate role. Examples will be provided below of some common configurations.

Admin keys

Admin keys are for integrations and applications where programmatic access is required across workspaces. This is typically used for automation purposes and an expiration can be set up to one year.

Managing Workspaces and Users in Tessl

Click the workspace drop-down to navigate workspaces. Navigate to Members at the workspace level to specify Roles for users who require more capabilities within the workspace, such as running evaluations, publishing or managing users.

To modify a user, search for their name, select their checkbox, a role, and click the Add button.

Example role configurations for your team(s)

The following users demonstrate common configurations and roles that may be used when rolling Tessl out:

Samira - Org. Admin

Samira, the administrator and skills champion, needs the ability to manage all workspaces, the ability to assign users, and create new workspaces. Make her an Organization admin.

Eddie - Lead Engineer

Another user, Eddie, might be a member of an engineering workspace. He needs to be able to use plugins (skills) that have been published, but may need to have access to publish skills within the engineering workspace for others on his team. This could mean Eddie is the publisher role in certain workspaces. He may also be a Member role of other workspaces where he only needs to search and install from.

Jennifer - Manager

Jennifer may require the ability to add users to a workspace that she owns, publish, and possibly need the ability to remove other managers etc. Typically the workspace permission "Owner" or "manager" may be given to that user, depending on the need to remove other "owners" or delete workspace.

Joe - New hire engineer

Finally, Joe, a new hire, has the ability to search and install skills from the engineering workspace, but does not have the ability to share/create skills until later, after they’ve gained a little more experience. Joe would be made a member of “engineering” with just a “consumer” role.

Next steps!

Now that you have your users in, and assigned roles to the different workspaces, you and your users can:

Start creating new skills
Evaluate new or existing skill effectiveness through using Reviews, and Evals
Publish those skills to the Tessl registry to share them for your users and agents

Let us know what you think! Tessl would love to hear from you through any one of our feedback channels (Discord, Email, CLI Feedback, etc)

Originally published on Tessl.blogs