DEV Community: Agentic Drifter

𝗣𝗼𝘀𝘁 𝟯 — 𝗧𝗵𝗲 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗟𝗼𝗼𝗽: 𝗪𝗵𝗲𝗿𝗲 𝘁𝗵𝗲 𝟭.𝟵𝟲% 𝗦𝘁𝗼𝗽𝘀 𝗕𝗲𝗶𝗻𝗴 𝗧𝗵𝗲𝗼𝗿𝘆

Agentic Drifter — Mon, 27 Jul 2026 14:33:00 +0000

LLMs don’t fail at hard problems. They fail at the (medium) ones – the ones that require 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴, not pattern‑matching. That’s the 𝟭.𝟵𝟲% Gap.

This week, I saw it directly.

────────────────────────────

The Medium Issues I Actually Found.

A multi‑file trace probe across a #GitHub repository: models, sessions, and utils, surfaced five medium issues – not because the code was broken, but because the system had to 𝗿𝗲𝗮𝘀𝗼𝗻.

𝗛𝗲𝗮𝗱𝗲𝗿 𝗺𝗲𝗿𝗴𝗲 𝗱𝗿𝗶𝗳𝘁

X-Test and x-test stored separately → 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗱𝗿𝗶𝗳𝘁.
𝗨𝗥𝗟 𝗻𝗼𝗿𝗺𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗱𝗿𝗶𝗳𝘁

Unicode left unencoded → 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁 𝗱𝗿𝗶𝗳𝘁.
𝗖𝗼𝗼𝗸𝗶𝗲 𝗽𝗿𝗼𝗽𝗮𝗴𝗮𝘁𝗶𝗼𝗻 𝗱𝗿𝗶𝗳𝘁

Cookies missing after prepare → 𝘀𝘁𝗮𝘁𝗲 𝗱𝗿𝗶𝗳𝘁.
𝗥𝗲𝗱𝗶𝗿𝗲𝗰𝘁 𝗖𝗼𝗻𝘁𝗲𝗻𝘁‑𝗧𝘆𝗽𝗲 𝗱𝗿𝗶𝗳𝘁

POST → GET kept Content-Type → 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 + 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁 𝗱𝗿𝗶𝗳𝘁.

-𝗥𝗲𝗱𝗶𝗿𝗲𝗰𝘁 𝗵𝗲𝗮𝗱𝗲𝗿 𝗿𝗲𝘂𝘀𝗲 𝗱𝗿𝗶𝗳𝘁

POST headers leaked into GET → 𝗽𝗿𝗲𝗺𝗶𝘀𝗲 𝗱𝗿𝗶𝗳𝘁.

None crashed the system. All fractured behavior. That’s what 𝗺𝗲𝗱𝗶𝘂𝗺 𝗶𝘀𝘀𝘂𝗲𝘀 are.

────────────────────────────

Why They Only Appear Under Reasoning Pressure

Medium issues emerge when you force a model through a 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗹𝗼𝗼𝗽 – not just “answering,” but:

Interpret → Extract premises → Build chain → Critique → Revise.

Pattern models break early. Reasoning models break later.

Medium issues live between the steps – where the 𝟭.𝟵𝟲% becomes visible.

────────────────────────────

The Breakthrough: A 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴-𝗳𝗶𝗿𝘀𝘁 𝗰𝗿𝗶𝘁𝗶𝗰

I didn’t need a critic to find medium issues.

I needed it to 𝗲𝘅𝗽𝗹𝗮𝗶𝗻 them.

It caught:

• 𝗽𝗿𝗲𝗺𝗶𝘀𝗲 𝗱𝗿𝗶𝗳𝘁

• 𝘀𝗲𝗺𝗮𝗻𝘁𝗶𝗰 𝗱𝗿𝗶𝗳𝘁

• 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝘁 𝗱𝗿𝗶𝗳𝘁

Example: http, port 80, length 12.

Pattern model: 12 chars → https → encrypted.
Reasoning critic: HTTP + port 80 contradict encryption; length irrelevant; HTTPS never introduced.

That’s the 𝟭.𝟵𝟲% Gap.

────────────────────────────

The Takeaway

To see the 𝟭.𝟵𝟲% gap, stop testing patterns.

Force the model to reason.

Then critique the reasoning.

That’s where fractures appear.

That’s where drift becomes visible.

That’s where medium issues live.

Next, soon: I’m breaking down the 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴‑𝗳𝗶𝗿𝘀𝘁 𝗰𝗿𝗶𝘁𝗶𝗾𝘂𝗲 – the same style of internal analysis that shows up inside big tech evaluation stacks when they need to expose drift, catch inference fractures, and pressure‑test reasoning chains.

𝗣𝗼𝘀𝘁 𝟮 — 𝗪𝗵𝘆 𝗠𝗲𝗱𝗶𝘂𝗺 𝗜𝘀𝘀𝘂𝗲𝘀 𝗠𝗮𝘁𝘁𝗲𝗿 (finding medium software engineering issues, within a #codebase)

Agentic Drifter — Fri, 24 Jul 2026 14:33:00 +0000

𝗦𝗲𝗿𝗶𝗲𝘀: 𝗧𝗵𝗲 𝟭.𝟵𝟲% 𝗚𝗮𝗽 - Medium Issues

A model cannot learn medium-tier reasoning from one prompt, one shot. Here is where #HumanintheLoop comes in.

This is the part most people miss.

Medium issues force the model to confront things it cannot shortcut: the underlying intent of the code, the expected behavior, the conditions that must remain true, and the consequences of a change. These are not “harder bugs” – they are 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗽𝗿𝗼𝗯𝗹𝗲𝗺𝘀.

A model cannot learn medium-tier reasoning from one prompt.

It needs a structured loop:
•Identify relevant files
•Describe current vs. expected behavior
•List invariants
•Propose minimal patch
•Apply patch
•Run tests
•Diagnose failures
•Revise patch

This loop is how you train the model to reason – and it’s why medium issues require Human-in-the-loop oversight.

Next, putting reasoning under real pressure for the SWE‑bench.

Post 1 — SWE‑Bench Reliability Series

Agentic Drifter — Wed, 22 Jul 2026 14:33:00 +0000

𝗧𝗵𝗲 𝟭.𝟵𝟲% 𝗚𝗮𝗽 - 𝗧𝗵𝗲 𝗟𝗶𝗻𝗲 𝗧𝗵𝗮𝘁 𝗗𝗲𝗳𝗶𝗻𝗲𝘀 𝘁𝗵𝗲 𝗙𝗿𝗼𝗻𝘁𝗶𝗲𝗿

LLMs look impressive until you ask them to solve something real – the moment a problem requires reasoning instead of pattern‑matching, the 1.96% ceiling shows up.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
See more: (https://arxiv.org/abs/2310.06770v3). Not my article.

ArXiv finds: “State-of-the-art proprietary models — and even the fine-tuned SWE-Llama — can resolve only the simplest issues. Claude 2 tops out at 1.96%.”

But it is the line that defines the frontier.

𝟭.𝟵𝟲%.

Not 20.

Not 10.

Not 5.

𝗢𝗻𝗲 𝗽𝗼𝗶𝗻𝘁 𝗻𝗶𝗻𝗲 𝘀𝗶𝘅.

That number tells you exactly where the capability gap is: models handle simple issues, and then collapse the moment the problem requires multi-file reasoning, invariant awareness, or any real systems-layer understanding.

That gap – the space between “simple” and “medium” – is where I’m building.

I’m approaching SWE bench with a hybrid persona model designed for that tier, finding medium issues in a #GitHub or similar repository:

•𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗼𝗿 𝗽𝗲𝗿𝘀𝗼𝗻𝗮 — reproducibility, deterministic patches, invariant preservation, regression avoidance.

•𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗽𝗲𝗿𝘀𝗼𝗻𝗮 — dependency awareness, side-effect mapping, multi-file reasoning.

•𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗽𝗲𝗿𝘀𝗼𝗻𝗮 — multiple patch strategies, comparative reasoning, failure-mode tracing.

𝗧𝗵𝗲 𝗴𝗼𝗮𝗹 𝗶𝘀 𝘁𝗼 𝗼𝗽𝗲𝗿𝗮𝘁𝗲 𝗶𝗻 𝘁𝗵𝗲 𝘁𝗶𝗲𝗿 𝘄𝗵𝗲𝗿𝗲 𝘁𝗵𝗲𝘆 𝗰𝘂𝗿𝗿𝗲𝗻𝘁𝗹𝘆 𝗳𝗮𝗶𝗹 – 𝗺𝗲𝗱𝗶𝘂𝗺 𝗰𝗼𝗺𝗽𝗹𝗲𝘅𝗶𝘁𝘆 𝗦𝗪𝗘.

This is Part 1.
𝗡𝗲𝘅𝘁: 𝗪𝗵𝘆 𝗠𝗲𝗱𝗶𝘂𝗺 𝗜𝘀𝘀𝘂𝗲𝘀 𝗠𝗮𝘁𝘁𝗲𝗿.

Small Model SWE‑bench: What Happens When You Push Tiny Models Into Full Task Pipelines

Agentic Drifter — Wed, 22 Jul 2026 02:52:13 +0000

I ran SWE‑bench on a small LLM to map failure modes and understand how tiny models behave under full task‑grounded pressure. This experiment tested whether a small model could sustain a multi‑stage evaluator pipeline under frontier‑level task conditions — and what breaks first when it can’t.

𝐓𝐡𝐞 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞 𝐟𝐨𝐥𝐥𝐨𝐰𝐞𝐝 𝐚 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞: 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠 → 𝐜𝐫𝐢𝐭𝐢𝐪𝐮𝐞 → 𝐩𝐚𝐭𝐜𝐡 → 𝐜𝐨𝐦𝐩𝐚𝐫𝐢𝐬𝐨𝐧. The objective was not correctness but signal: determining whether the model could produce meaningful evaluator grade behavior and reveal its own capability limits.

𝐒𝐮𝐜𝐜𝐞𝐬𝐬𝐞𝐬
•The pipeline architecture executed end‑to‑end.
•The model produced coherent reasoning about the SQLFluff quiet‑mode issue.
•The critique mechanism worked; several critiques were semantically valid even when mixed with drift.
•The prompt scaffolding held long enough for the model to respond in the requested structure.
•The model demonstrated an ability to reason SWE‑bench‑style tasks despite its size.

𝐋𝐢𝐦𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬
•Small models cannot sustain long‑context, multi‑step reasoning loops. They enter a frontier envelope, begin to fail in predictable ways.
•Token‑budget tradeoffs were clear: shorter prompts improved stability but reduced semantic richness; longer prompts increased drift and boundary loss.
•Strict output formats caused placeholder fallbacks; looser formats caused instruction boundary loss.
•The environment introduced variability in GPU availability and model initialization, which affected stability.

𝐅𝐚𝐢𝐥𝐮𝐫𝐞𝐬
•Patch generation was unreliable. The model could not consistently produce unified diffs or structured patch output.
•Critique stability degraded as the prompt length increased. Critiques were mixed with prompt echoing, repetition, and partial collapse.
•Instruction boundaries were repeatedly lost, causing the model to reprint the prompt or earlier reasoning segments.
•Placeholder fallbacks appeared when the model exceeded its reasoning envelope.

𝐖𝐡𝐚𝐭 𝐭𝐡𝐞 𝐄𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐞𝐝
•The evaluator pipeline design is sound.
•The prompt boundaries are functional even under stress.
•The critique mechanism is viable and produces meaningful signal.
•The model can reason about real tasks but cannot sustain multi‑step evaluator behavior at this scale.
•The failure modes are model capacity related, not pipeline related.

𝐒𝐮𝐦𝐦𝐚𝐫𝐲
As a self‑taught LLM Reliability Researcher, this experiment shows you can build and validate a multi-stage evaluator pipeline without SWE‑bench tooling or a software engineering background.

The pipeline produced real reasoning, multi‑pass critiques, and clear failure traces. Collapse patterns matched known small model frontier task limits. At this stage, the pipeline is validated and the model – not the method – is the limiting factor.