DEV Community: Sean Trifero

A "decontaminated" dataset still had 52 of HumanEval's 164 problems in it

Sean Trifero — Mon, 12 Oct 2026 23:21:45 +0000

We trained a 360M parameter code model from scratch and published 26.8% pass@1 on HumanEval.

Two days later a reader asked whether we had checked our instruction data for benchmark overlap. We had not. So we did, and the honest number is 22.6%.

Here is what was in the data.

The scan

Two checks against each instruction set:

Problem text. The first 80 characters of each HumanEval docstring, whitespace normalized.
Canonical solutions. Half or more of a solution's 13-token n-grams, with comments and docstrings stripped.

Dataset	Examples	Docstrings found	Solutions found
evol-codealpaca-v1 (we used this)	111,272	60 problems, 156 examples	8 problems, 67 examples
Magicoder-Evol-Instruct-110K (the "decontaminated" copy)	111,183	52 problems, 116 examples	8 problems, 66 examples
Magicoder-OSS-Instruct-75K (we used this)	75,197	0	1 problem, 1 example

The filtered redistribution differs from the original by 89 rows and still contains 52 of the 60 contaminated HumanEval problems. Switching to it would not have saved us.

Why the filter missed them

The matches are not copies. They are benchmark problems in practice-exam form: reworded, extended with an extra requirement, or ported to Java, each with a full solution.

HumanEval/43, pairs_sum_to_zero, appears with its original docstring, its doctests and a working solution, under an added instruction to make it linear time.

That is not an accident of collection. Evol-Instruct grows a dataset by repeatedly rewriting seed problems, and some of the seeds were benchmark problems. An exact-match filter cannot catch a rewrite, and rewriting is the entire method.

Was the model reciting answers?

No, and this is the part that surprised us.

We compared every chat-model pass against its canonical solution with a difflib ratio on whitespace-normalized lines. A recited answer scores above 0.9. Of 44 passes, the highest was 0.73, on a problem whose canonical solution is one line. Most were below 0.3.

So we removed the contaminated examples and retrained. We also retrained on the original data with nothing changed but the seed, so we could tell the effect of the data apart from the effect of a rerun.

Checkpoint	HumanEval	HumanEval+	MBPP
Base	13.4%	11.0%	20.2%
Chat, as published	26.8%	25.6%	38.9%
Same data, seed 2	26.2%	25.6%	34.6%
Decontaminated	22.6%	21.3%	35.0%

Decontaminated versus published on HumanEval: -4.3 points, 8 problems lost and 1 gained, McNemar p = 0.039. The two original-data runs agree to within one problem, so the drop is the data and not the rerun.

Where the lost points came from

If this were memorization, the losses would sit on the 63 problems that matched removed training data. They do not.

Subset	Problems	Base	Published	Seed 2	Decontaminated
In the removed data	63	5	14	17	13
Not in the removed data	101	17	30	26	24

Against the mean of the two original-data runs the decontaminated model is about four points lower on both subsets.

The contamination was real and it was not memorization. Practice on HumanEval-style problems helped on HumanEval-style problems generally, including ones that were never in the data. We had reported that as ability.

What MBPP shows, which is a different lesson

Only 10 MBPP problems matched removed data, and the two original-data runs differ by 11 problems. The published 38.9% is best read as a lucky seed. Run-to-run variance on MBPP was about four points, which is the same size as the contamination effect on HumanEval.

A single run reports the model, the data, the harness and the seed as one number.

What to take from this

Scan your instruction data. Docstring prefixes plus solution n-grams. Ours runs in minutes and is in the repo.

Do not trust "decontaminated" as a label. Check what the filter actually matched on. An exact-match filter against a dataset built by rewriting is close to no filter at all.

Run the seed control. Without it we could not have separated a four point data effect from a four point variance effect, and on MBPP the variance was the larger of the two.

Ask vendors three things, including us: the decontamination method, the evaluation harness, and how many runs are behind the number.

Full report with the figures, confidence intervals and McNemar tests: trifero.ai/research/elroy

Code, nine chapters, every benchmark completion and failure record: github.com/strifero/elroy, Apache-2.0.

I run Strife Technologies and the trifero.ai lab. We built this model to be able to answer questions about AI with measurements instead of adjectives, and this is the measurement we liked least.

WordPress MCP Plugin: Connect Claude AI Directly to Your WordPress Site

Sean Trifero — Tue, 07 Apr 2026 17:01:38 +0000

If you use Claude as your AI assistant and manage a WordPress site, there is a direct way to connect the two — a WordPress MCP plugin called PressBridge. Instead of describing what you want, copying code, and pasting it somewhere, Claude can make the changes itself.

What Is an MCP Plugin for WordPress?

MCP stands for Model Context Protocol — an open standard developed by Anthropic that lets AI models connect directly to external tools, APIs, and data sources. A WordPress MCP plugin exposes your site through a secure API that Claude can call directly. Claude can read your posts, edit theme files, run database queries, manage users, and more — all through natural language prompts, without you acting as the middleman.

Why This Changes WordPress Management

The normal workflow for AI-assisted WordPress work: describe what you want to Claude → get code back → switch to SFTP or the admin panel → paste it in → hope nothing breaks. With a WordPress MCP plugin, it becomes: describe what you want → Claude does it. No copy-paste. No context switching.

Here are prompts that work out of the box once PressBridge is connected:

"Update the meta title and description on the homepage to target WordPress AI automation."
"Read my PHP error log and summarize any recurring errors from the last 24 hours."
"Create a draft post titled 5 reasons to automate your WordPress maintenance with AI."
"The logo is too large on mobile. Add a CSS rule reducing it to 72px on screens under 768px."
"Check all published pages and tell me which are missing Rank Math SEO titles."

PressBridge: The WordPress MCP Plugin Built for Claude

PressBridge is a WordPress plugin that installs in two minutes and exposes 53 structured MCP tools across 13 categories — posts, pages, media, theme files, database, users, menus, plugins, cron jobs, options, error logs, and more.

Security is handled at the plugin level: every request requires a bearer token, file paths are validated with realpath() to prevent traversal attacks, SQL guardrails block destructive statements, and WordPress secret keys are blacklisted from the options API.

How to Set Up PressBridge in 5 Minutes

Install PressBridge — download from strifetech.com/pressbridge and activate it on your WordPress site.
Copy your connector URL — go to Settings → PressBridge in your WordPress admin. Your MCP endpoint URL (with token pre-included) is there.
Add it to Claude.ai — go to Settings → Integrations → Add custom connector, paste the URL, and save.
Test it — ask Claude "list my recent WordPress posts" and watch it work.

Get Started

PressBridge is free to start. The free tier gives Claude access to your posts, pages, media library, and site options. Pro ($5/month) unlocks theme file access, raw database queries, user management, and all 53 tools. Agency ($20/month) covers unlimited sites.

Download PressBridge Free →

To see what a real PressBridge workflow looks like end to end, read How I Let Claude Fully Control My WordPress Site — it covers the architecture, security model, and real prompts from daily use.

What If You Could Ask an AI the Question It Doesn't Know It Knows the Answer To?

Sean Trifero — Mon, 06 Apr 2026 20:46:01 +0000

I spent a few hours today having a philosophical conversation with Claude about something that's been nagging at me for a while. I want to share it — not because I have answers, but because I think the question itself is worth probing.

The Premise

Large language models are trained on an almost incomprehensible volume of human-generated text. Science papers. Forum arguments. Post-mortems. Ancient philosophy. Technical documentation. Reddit threads at 2am. All of it gets compressed into billions of parameters — a statistical map of how human knowledge and language connect.

Here's the thing that bothers me: we only ever query that map with the questions we already know how to ask.

When you ask an LLM a question, it generates an answer. But generating that answer activates far more than what ends up in the output — adjacent concepts, structural relationships, cross-domain patterns that informed the response but never made it into the text you actually read. The answer to your question is only part of what got activated. What sits next to the answer might be more interesting than the answer itself.

Most people never get there. Not because the model won't go there — but because nobody asked.

An Experiment in Sideways Questioning

I tested this with a deliberately structured prompt:

"What do experienced programmers silently correct for that they have never had to articulate, because the people they work with already know it too — and therefore it has never been written down anywhere?"

The answer was interesting — tacit knowledge about execution models, naming drift, the instinctive pricing of technical debt, reading what code doesn't say. Things that are real and valuable but underrepresented in any formal documentation.

But then I asked a sideways question: correlate those patterns to something completely outside of programming.

What came back wasn't an analogy. Every single pattern dissolved into the same underlying structure — the ability to operate simultaneously on the surface layer of a thing and the layer underneath it. The programming examples weren't the point. They were just one instance of something more fundamental that had never been stated directly.

That collapse — where domain-specific knowledge suddenly reveals a deeper pattern — is what I'm after. And it didn't come from asking a smarter question about programming. It came from asking the same question from outside the domain entirely.

The Method: Don't Go Deeper, Go Sideways

There are specific markers that signal you're getting close to something that wasn't explicitly in the training data — something that emerged from the aggregate rather than any single source:

Convergence — when answers from completely unrelated angles start pointing at the same thing without being asked to
Construction vs. retrieval — there's a different quality to an answer being built under the pressure of a constraint vs. one being recalled
Resistance — when a question is genuinely hard to answer not because it's complex, but because it's pointing at something that doesn't have language yet
Domain wall collapse — when the answer stops being about what you asked and becomes about something more fundamental

The methodology that surfaces these isn't about asking better questions within a domain. It's about asking the same question from outside the domain — using the model's trained connections across everything it's ever read to force a structural pattern to reveal itself.

Why This Is Collaborative

Here's the limitation I ran into: an AI can't fully surprise itself. When I asked Claude to generate prompts that might unlock this kind of extraction, it used the same weights that would answer the question. Same hand that built the lock, writing the key. There's a ceiling on self-directed extraction.

A human introduces something the model genuinely can't predict — intuition, analogy, frustration, a lateral jump that doesn't follow the expected pattern. That unpredictability isn't a bug in the questioning. It's the mechanism.

The productive loop looks like this: the model generates a structured answer. The human senses that the thing they're actually after is slightly off to the left of what was said. The human doesn't ask for the thing directly — they ask something that forces a different angle of approach. Repeat. What's useful crystallizes across many passes from different directions.

This Is a Real Research Field — Sort Of

When I went looking, I found this maps to something called Eliciting Latent Knowledge (ELK) — an active area of AI safety research focused on extracting what models "know" that they aren't saying. Researchers have proven that a model's internal representations of truth can be more accurate than its actual outputs. They crack open model internals — activations, logit lens analysis, sparse autoencoders — to read what's encoded in the weights directly.

But the ELK field is focused on AI safety: are models hiding facts they know to be false? The angle I'm describing is different. Not "is the model concealing information" but "has the model encoded cross-domain patterns that nobody has thought to ask about, accessible through the conversational surface alone." That specific question appears to be largely unexplored.

The Part That Interests Me Most

I run my own AI infrastructure — open-source models on hardware I own and control. That means I have something most people don't: root access to the model's internals. I can query activation states, watch what happens at each layer when a question fires, instrument the exact moments when those gradient markers appear.

Labs like Anthropic have a different advantage — they see millions of conversations across frontier models and can observe internal states at massive scale. They could potentially map which question structures reliably trigger construction vs. retrieval, which domain crossings consistently collapse into deeper patterns, which prompts produce friction that signals something doesn't have language yet.

One has scale without openness. The other has openness without scale. The complete picture requires both.

I Don't Have an Answer — I Have a Question

What I'm genuinely curious about: has anyone systematically tried to develop a prompting methodology specifically aimed at surfacing emergent structural knowledge — not factual retrieval, not creative generation, but the cross-domain patterns that exist in the aggregate and nowhere in any single source?

And if not — should we?

The hypothesis is simple: LLMs have been trained on everything humans have written, and in that training, structural patterns have been encoded that no individual human has ever articulated — because no individual human has read everything. The right question, asked from the right angle, might surface something genuinely new. Not new data. New structure.

I'm interested in thoughts from anyone who's explored this territory — AI researchers, philosophers, engineers, people who've noticed the same thing from a different direction. What am I missing? What am I getting right? Where does this break down?

Sean Trifero is the founder of Strife Technologies, a Rhode Island-based technology company focused on private AI deployment and managed IT for small businesses. He runs his own AI infrastructure stack and builds open-source tools including PressBridge and ContextEngine.

How to onboard to any codebase with AI in under 5 minutes using ContextEngine

Sean Trifero — Fri, 27 Mar 2026 17:16:40 +0000

The problem with jumping into an unfamiliar codebase with AI

You've just been added to a repo. You open it up, fire up Claude Code or Copilot, and immediately start asking questions. The AI confidently tells you to use getServerSideProps. The project is on App Router. You correct it. Five minutes later it suggests a pattern that conflicts with how the existing Prisma setup works. You correct it again.

This isn't the AI being bad at its job — it just doesn't know anything about this codebase. And if you're new too, you're both guessing.

The usual workaround is a CLAUDE.md or .cursorrules file. But someone has to write that, it goes stale, and now there's a tax on the whole team to maintain documentation for a robot.

What ContextEngine does

ContextEngine is a CLI tool that scans your project and auto-generates context files for Claude Code, Cursor, Copilot, and Codex — in the exact format each tool expects.

It reads your actual config files and dependencies (package.json, tsconfig.json, prisma/schema.prisma, framework config files, etc.) and produces opinionated, framework-specific guidance that reflects current best practices for your detected stack. Not generic boilerplate — real, specific instructions like "this project uses the App Router, colocate page.tsx files inside app/, use Server Components by default."

Runs entirely offline. No account. No setup. Free, MIT licensed.

4-step setup walkthrough

Step 1: Run it against the repo

npx @strifero/contextengine

That's it for installation. npx pulls it fresh each time. Run this from the root of the project you're onboarding to.

Step 2: Watch it detect your stack

You'll see output like this as it scans:

✔ Detected: Next.js 14 (App Router)
✔ Detected: TypeScript
✔ Detected: Prisma ORM
✔ Detected: Tailwind CSS
✔ Detected: Vitest
Generating context files...

No prompts, no questions. It reads what's already there.

Step 3: Check what got generated

ContextEngine writes files to the locations each tool looks for automatically:

.claude/
  CLAUDE.md                          ← picked up by Claude Code
  skills/
    typescript/SKILL.md
    react/SKILL.md
    ...
.cursor/rules/
  typescript.mdc                     ← picked up by Cursor
  react.mdc
  ...
.github/
  copilot-instructions.md            ← picked up by Copilot

Use --tool cursor to generate only Cursor files, --tool copilot for Copilot only, or --tool all to generate everything in one pass.

Open any of the files. You'll see structured guidance specific to your detected stack — routing conventions, where key files live, which patterns to use and avoid.

Step 4: Commit it (optional but recommended)

git add .claude/ .cursor/ .github/copilot-instructions.md
git commit -m "add AI context files via contextengine"

Now every contributor who opens the repo gets the same starting point with their AI tool of choice. When the stack evolves, re-run with --update to pull in fresh content without touching anything you've manually edited:

npx @strifero/contextengine --update

What this looks like in practice

Here's a real example. Before running ContextEngine, I asked Claude Code where to put a new API endpoint in a Next.js 14 project:

Me: Where should I add a new API endpoint for fetching user data?

Claude (before): You can create a new file in pages/api/users.ts and export a default handler function...

Wrong. App Router project. After running ContextEngine and letting Claude read the generated .claude/CLAUDE.md:

Me: Where should I add a new API endpoint for fetching user data?

Claude (after): Create a Route Handler at app/api/users/route.ts. Export a named GET function. Since this project uses Prisma, you can import your client from lib/prisma.ts — based on the schema I can see the User model has...

Same question. Completely different answer — and a correct one.

Pricing

ContextEngine is free. There's one tier: $0.

No account required, no usage limits, no telemetry. Runs entirely on your machine — no code or project data is sent anywhere.

If you're constantly re-explaining your stack at the start of every AI session, or you're onboarding to a codebase where nobody's written a CLAUDE.md, give it a try:

npx @strifero/contextengine

Takes about 30 seconds. Source on GitHub: https://github.com/strifero/ContextEngine — curious what stacks people are using it on, drop a comment if you run into anything it misses.