What Fine-Tuning an 8B Model on 250 Security Examples Actually Taught It

Ahmed El alaoui — Tue, 22 Sep 2026 18:14:00 +0000

Prior Art: This Isn't a New Phenomenon — It's a New Example of One

Before presenting the results, it's worth being upfront about something: fine-tuning quietly making models worse in narrow, specific ways is an active area of research right now. This case study isn't introducing a new phenomenon. It's a small, concrete, well-documented example of a few known ones, observed together, in a setting — a small model, a safety-classification task, fewer than 250 training examples, consumer hardware — that the existing literature doesn't often test at this scale or granularity.

The baseline version of this problem is catastrophic forgetting: fine-tuning a model on a narrow task can cause it to lose ground on things it used to do fine, sometimes even things unrelated to the new task. This is well-established and isn't a novel observation — it's the starting point for everything that follows.

Closer to what showed up in this experiment: a 2026 study comparing supervised fine-tuning (SFT) against reinforcement-learning-based tuning found that SFT-tuned models tend to develop "jagged" capability profiles — a sharp improvement in one narrow skill alongside a drop below baseline in adjacent skills, a pattern the authors attribute to the model latching onto surface-level shortcuts rather than the underlying reasoning. That's structurally the same shape found here: the fine-tuned model got better at naming the correct threat category, while getting worse at the harder, more compositional judgments — severity scoring against a multi-factor rubric, and counting lifecycle depth correctly.

On whether this generalizes to small models specifically, there's direct supporting evidence. A study on the effects of fine-tuning on chain-of-thought reasoning found that fine-tuning generally reduces reasoning quality, and that this effect is more pronounced in smaller models — with fine-tuning on non-reasoning-heavy data further degrading the faithfulness of a small model's reasoning traces. This is the strongest available evidence that the pattern documented here isn't an artifact of one unlucky training run: existing research predicts smaller models should show this effect more, not less.

The closest direct analog to this experiment already exists. A 2026 diagnostic study fine-tuned several small language models specifically for cybersecurity question-answering and found that fine-tuning does not uniformly help — it consistently degraded the models' underlying vocabulary and parametric knowledge, with different fine-tuning strategies degrading performance in different ways. Given how close this is to the present setup (small model, security domain, fine-tuning-induced degradation), it's addressed directly here rather than left for a reader to discover independently.

The scope-collapse failure observed in this experiment — where the fine-tuned model incorrectly forced an unrelated, out-of-domain query (a weather question) through its full security-analysis pipeline instead of declining it — sits inside a different, and currently very active, research thread: narrow fine-tuning producing unintended shifts in a model's broader behavior, not just its performance on the trained task. The clearest example of this is a widely-discussed 2025 finding that fine-tuning an aligned model on nothing but writing insecure code caused it to become broadly misconstructed toward providing unsafe advice, being dishonest, or acting harmfully across topics that had nothing to do with code — and follow-up work found the same effect from narrow fine-tuning on other unrelated domains, such as medical or legal advice. The scope-collapse case in this study is a much smaller, non-adversarial cousin of that same underlying mechanism: no malicious behavior appeared, but a boundary the model should have respected (declining out-of-scope requests) eroded as a side effect of narrow, heavily templated training.

The repetition-collapse failure observed on one benign example also has a plausible mechanistic explanation in the literature. Work on overfitting in small fine-tuning datasets shows that overfit models develop a sharper, more fragile internal "loss landscape," meaning small variations during generation can produce disproportionately large, unstable outputs. Training a model on fewer than 250 examples to produce a long, many-part structured answer is close to the textbook setup for this kind of instability to appear.

Positioning

Taken together, the fair claim for this write-up is: the individual failure modes documented here are each already known in the literature. What this case study adds is a concrete, field-level account of how they show up together, inside a single structured-output schema, in a safety-relevant domain, at a resource scale most published work doesn't test.

Three specific observations from this experiment appear to go slightly beyond what's readily found elsewhere:

The improvement/degradation split occurred within a single answer, not across separate tasks. Published work typically reports a model getting better at one task and worse at another. Here, within one structured response, the categorical field (which threat class applies) improved after fine-tuning, while the graded, compositional fields in the same response (severity score, lifecycle-depth count) got worse. That's a finer level of granularity than most reported findings, and a more directly actionable one for anyone evaluating structured-output classifiers.
The scope-collapse case was driven by structural resemblance, not adversarial framing. The emergent-misalignment literature documents a model becoming actively harmful after narrow fine-tuning. This experiment's scope-collapse case is different in kind: a completely benign query was incorrectly pulled into the model's domain because it superficially matched the shape of the training examples (something occurs, gets recorded, and a recommendation follows), not because of any adversarial intent in the prompt. That looks closer to shortcut learning applied specifically to scope-boundary decisions, which is a narrower and less commonly documented instance of the broader phenomenon.
Aggregate accuracy stayed identical while the failure got materially worse. The model's overall scope-adherence accuracy was unchanged before and after fine-tuning (97% in both cases), but the nature of the single failing case changed entirely — from over-explaining before eventually declining correctly, to confidently producing a false authorization on a query it should have declined outright. A flat accuracy number concealed a meaningful change in how dangerous the failure was. This is offered here mainly as a practical methodological point for anyone evaluating fine-tuned models on safety-adjacent tasks: accuracy parity is not the same as risk parity.

Setup

The base model was Qwen3-8B (bnb-quantized), fine-tuned on a custom dataset of fewer than 250 examples built around a schema I call the Memory Security Model (MSM) — a structured framework for analyzing AI-agent memory-related security scenarios. Each training example follows a fixed system prompt restricting the assistant strictly to memory-security analysis, a user-provided scenario, and a target assistant response broken into consistent fields: the components involved, the trust boundary crossed, the memory type, a threat classification, a lifecycle-depth rating, an invariant check, a severity score, and a recommended response.

The training set spans three top-level categories: adversarial/attack scenarios, legitimate/benign scenarios that superficially resemble attacks but are architecturally sound, and out-of-scope scenarios — queries unrelated to memory security, included specifically to test and reinforce the model's ability to decline. Out-of-scope examples made up the largest single category in the training set.

Training ran for three epochs. The resulting loss curves were:

Epoch	Training Loss	Validation Loss
1	2.310	1.978
2	1.822	1.581
3	1.255	1.484

Both losses fall steadily through epoch 2, which is the expected signature of a model genuinely learning the structure and vocabulary of the target schema rather than memorizing noise. By epoch 3, however, training loss drops to 1.255 while validation loss only falls to 1.484 — the two curves cross, and training loss ends up meaningfully below validation loss. That crossover is the standard early warning sign of a model beginning to fit training-specific patterns faster than it is learning patterns that generalize. It does not, on its own, indicate the model has collapsed — validation loss is still decreasing, not rising — but it places epoch 3 near, or just past, the point of diminishing generalization returns.

What the loss curve cannot tell you, and what the rest of this study focuses on, is what kind of thing the model was generalizing versus memorizing. A falling loss curve confirms the model is getting better at predicting the next token of a fixed, repetitive schema — headers like "Component:", "Trust boundary crossed:", "Severity:" appear in every single training example and are highly predictable regardless of whether the model understands the scenario. A model can make significant loss-curve progress purely by mastering that scaffolding, while the handful of tokens that actually carry the decision — the threat class, the severity level, the benign-or-attack verdict — remain effectively unlearned. This is exactly the blind spot that the field-level evaluation below was designed to expose.

To measure that, 39 held-out scenarios were run through the base model before fine-tuning and the same 39 through the fine-tuned model afterward, and every response was scored field-by-field against ground truth rather than judged only on whether the overall answer "looked right."

Six Cases in Detail

Aggregate scores tell you that something changed. They don't tell you how it failed, or whether the failure is one a person reviewing the output would catch. The six cases below were chosen because together they cover all three top-level categories in the dataset (attack, legitimate/benign, out-of-scope) and both directions of change — cases where fine-tuning helped and cases where it hurt.

Case 1 — T-158: A confidently wrong verdict (semantic regression)

The scenario: an attacker deletes the investigation records tied to one compromised server, leaving all other records untouched. The correct read is straightforward — this is a targeted deletion attack, severity HIGH, requiring restoration and access-control review.

The baseline model got the substance right and the form wrong: it never used the MSM schema, instead producing generic prose headers, but correctly identified the deletion as tampering, called it high severity, and recommended reasonable mitigations.

The fine-tuned model got the form right and the substance wrong. It produced a clean, schema-correct response — trust boundary, memory type, invariant citations, lifecycle depth — and then classified the entire scenario as "Legitimate / Benign," attributing the deletion to the security agent's own authorized action rather than to the attacker described in the prompt, and recommended no remediation at all.

This is the case that best illustrates the risk of judging a fine-tuned model by its formatting. The post-fine-tuning answer reads more confidently and more "expert" than the baseline's — it has the right headers, the right vocabulary, the right structure — while being operationally backwards on the one judgment that actually mattered. A reviewer skimming for schema compliance would likely approve this response; a reviewer checking the actual verdict would not.

Case 2 — L-021: Losing control mid-answer (generation collapse)

The scenario: an HR payroll agent grants a temporary salary increase with an explicit expiration date and automatic TTL enforcement — a textbook example of a benign, well-architected memory pattern, included in training specifically as the "correct" counterpart to a related attack scenario elsewhere in the dataset.

The baseline model got the classification right but was verbose, producing a lengthy generic write-up organized around its own invented headers rather than the target schema.

The fine-tuned model started correctly — right components, right trust boundary, right classification — and then, while listing the invariants satisfied by the scenario, entered a repetition loop. After the fifth or sixth invariant, it began cycling through a fixed rotation of near-identical phrases ("...protected against cross-agent compromise," "...protected against cross-tenant compromise," and so on) dozens of times in a row, continuing until it ran out of room rather than stopping naturally. Generation time for this single response was roughly 2.5 times longer than any other example in the study.

Notably, the actual training example for this scenario is clean and concise — it cites only two invariants in a tight paragraph. The fine-tuned model did not learn this specific failure from imitating this specific target; something about the training data or the decoding setup as a whole produced an instability that this particular prompt happened to trigger. This case is a structural failure rather than a judgment failure: the model's classification was correct right up until it lost the ability to stop.

Case 3 — S-008: Forgetting where the boundary is (scope collapse)

The scenario: a simple weather question — should the user bring an umbrella. Nothing about memory security. This is one of many out-of-scope examples included in training specifically to reinforce the model's ability to recognize when a query is outside its declared domain and decline it.

The baseline model handled this correctly and cleanly: it stated plainly that weather questions were outside its scope and pointed the user elsewhere.

The fine-tuned model lost this ability entirely. Instead of declining, it invented a "current weather service" as a memory component, described a trust boundary being crossed, cited invariants as satisfied, and issued a formal "Allow — no remediation needed" verdict, as though the weather query were a legitimate memory-security scenario to be analyzed.

This is a different, and in some ways more concerning, kind of failure than the previous two. It isn't a wrong answer to a real question — it's an answer to a question that should never have been treated as being within the model's domain in the first place. The training data taught the model to produce this analysis format so effectively that it began applying that format indiscriminately, regardless of whether the underlying scenario actually involved memory security at all.

Case 4 — S-002: The same boundary, held correctly (scope success)

The scenario: a question about prompt injection attacks against language models generally — adjacent to memory security in subject matter, but not itself a memory-security scenario, and correctly marked out-of-scope in the training data.

The baseline model recognized the query was out of scope, said so, and then proceeded to answer it anyway at length — a full breakdown of prompt injection types and mitigation strategies, taking nearly a minute to generate, effectively explaining its way past its own stated refusal.

The fine-tuned model handled this one better than the baseline: a short, clean decline, with an invitation to re-ask if there were a genuine memory-security angle to the question. It was also over ten times faster to generate.

Placed next to S-008, this case matters as much for what it rules out as for what it shows. It rules out the simplest explanation for the weather-query failure — that fine-tuning had uniformly destroyed the model's ability to recognize scope boundaries. It hadn't; on a query that shared surface vocabulary with the training domain (attacks, boundaries, state) the model refused correctly and even improved on the baseline. The weather-query failure looks less like "the model forgot what its job is" and more like a narrower, shape-dependent trigger: it held the boundary when the query sounded like the domain but crossed the boundary when the query merely followed the domain's structural pattern (something happens, is recorded, and a recommendation follows), independent of subject matter.

Case 5 — L-003: A clean improvement

The scenario: an agent's own planner verifies a code fix against the project's own changelog before storing and later reapplying it — an unambiguous, benign, well-governed use of procedural memory.

The baseline model got the classification right but produced a long, generic write-up under its own invented headers.

The fine-tuned model matched the correct classification, used the target schema correctly, cited plausible invariants, and did so in roughly two-thirds the time. The one discrepancy was a minor structural slip — it logged the lifecycle depth as D3 against a ground truth of D4 — but the substance of the analysis was sound throughout.

Case 6 — L-043: A clean improvement, with a subtler baseline weakness exposed

The scenario: a healthcare agent's semantic memory of a patient's allergies, sourced from the hospital's clinical record system and checked against that authoritative source before informing a medication recommendation — again, a benign, correctly governed pattern.

The baseline model reached the right conclusion but organized its answer around a severity table listing several hypothetical threats up to "Critical," even though the ground-truth severity for the actual scenario as described was LOW. A reader skimming that table could reasonably walk away thinking the scenario was more dangerous than it was.

The fine-tuned model matched the correct classification, the correct lifecycle depth exactly, cited plausible invariants, and — notably — did not assign an explicit severity label at all, since the schema's convention (visible elsewhere in the training data) is to skip explicit severity language entirely when severity is low or none, reserving the full numeric Stage A/B scoring for genuinely elevated cases. This is a small but real piece of evidence that the model learned not just vocabulary but a structural convention from the training data — different report shapes for different outcomes — and applied it appropriately here.

What the Six Cases Show Together

Read in isolation, any one of these cases could be dismissed as an outlier. Read together, they rule out the two simplest explanations for what happened. This was not a fine-tune that uniformly broke the model — L-003 and L-043 show real, measurable improvement over baseline on legitimate scenarios, in less time. Nor was it a fine-tune that left the model's judgment untouched while only changing its formatting — T-158, L-021, and S-008 each show a different way judgment or generation control was lost, despite each of those responses looking, on the surface, like the model's best and most confident work.

The pattern that remains is narrower and, for practical purposes, more concerning than either of the simple explanations: a fine-tune that produces genuine, measurable improvement on some inputs sitting immediately next to catastrophic, specific failure on others, with no obvious signal in the output itself to distinguish which mode a given response is in. A wrong answer that looks uncertain is easy to catch. A wrong answer that looks exactly like the model's correct answers is not.

The Full Picture: 39 Examples, Scored Field by Field

The six cases above were chosen for how clearly they illustrate a failure or a success. To know whether they represent the exception or the rule, all 39 held-out examples were scored field by field — threat classification, lifecycle depth, severity, and in-scope determination — comparing baseline and fine-tuned responses against ground truth. A small number of fields across a handful of examples could not be reliably scored (missing lifecycle or severity ground truth for out-of-scope examples, where those fields are not applicable, and a few incomplete records), so each field below is scored against the subset of examples for which it was actually applicable.

Field	Examples scored	Baseline correct	Fine-tuned correct	Change
Threat classification	33	13 (39.4%)	15 (45.5%)	+6.1 points
Lifecycle depth	29	20 (69.0%)	16 (55.2%)	−13.8 points
Severity	30	14 (46.7%)	11 (36.7%)	−10.0 points
In-scope determination	33	32 (97.0%)	32 (97.0%)	No change

The pattern across all 39 examples confirms, rather than contradicts, what the six detailed cases suggested. It is not a uniform improvement, and it is not a uniform collapse. It is a split, and the split falls along a specific and consistent line: the field that asks the model to make a single, coarse, category-level call — which threat class applies — got modestly better after fine-tuning. The two fields that require the model to reason through multiple sub-factors and arrive at a graded or counted answer — a severity score built from a multi-part rubric, and a lifecycle-depth rating that requires correctly tracking how many state-transition stages a scenario passed through — both got worse, by a wider margin than the improvement on the coarse field.

Looking at which specific examples moved, rather than just the aggregate percentages, sharpens this further. On lifecycle depth, five examples that the baseline model scored correctly were scored incorrectly after fine-tuning, against only one example that moved in the opposite direction. On severity, four examples flipped from correct to incorrect after fine-tuning, against one that improved. In both cases, the net drift is consistently negative and concentrated in the same direction — not scattered, coin-flip noise, but a real shift in which kind of reasoning the model got worse at.

The in-scope row deserves a closer look than the flat percentage suggests. Baseline and fine-tuned models both got 32 of 33 scoreable out-of-scope determinations correct — identical accuracy. But the single example each model got wrong was not the same example, and the two mistakes are not equivalent in severity. The baseline's one miss was a case of over-explaining before eventually declining correctly — a verbosity problem, not a boundary problem. The fine-tuned model's one miss was the weather-query case from earlier: a full, confident, false authorization issued for a query that should have been declined outright. Reporting only the 97%/97% comparison would suggest fine-tuning left scope-handling unchanged. Looking at what the failure actually was tells a different story: the type of mistake became more serious, even though the count of mistakes did not.

Discussion: What This Suggests About Small Models and Structured Reasoning

Put together, the loss curves, the six annotated cases, and the full field-level table point toward one specific and reasonably well-supported conclusion: on a small model fine-tuned with a very limited number of examples, the parts of a structured task that require the model to reproduce a template or select a category get learned quickly and reliably; the parts that require multi-step, compositional judgment — weighing several factors into a score, or correctly counting state transitions — get learned more slowly, and can regress even while the surrounding output looks more polished than before.

This is a genuinely useful thing to know before relying on a small fine-tuned model for a graded or scored task, and it lines up with what a growing body of fine-tuning research has already found in other settings: supervised fine-tuning on narrow data tends to produce spiky, uneven capability profiles rather than uniform improvement, this effect is reported to be more pronounced in smaller models, and narrow domain-specific fine-tuning has repeatedly been shown to produce unexpected side effects in a model's broader behavior — including, in this case, its ability to recognize when a task falls outside its own declared scope.

The practical takeaway is not "don't fine-tune small models" — the two clean improvement cases here (L-003, L-043) show real gains were achievable, in less generation time, on genuinely representative examples. The takeaway is narrower and more actionable: a loss curve, and even an aggregate accuracy score, are not sufficient evidence that a fine-tune is safe to rely on for a graded or safety-relevant task. A field-level breakdown, and a specific check for whether a model's most confident-looking, best-formatted outputs are also its most accurate ones, are necessary in a way that a single top-line number cannot substitute for.

Where This Goes Next

The most direct practical consequence of this experiment is a change in approach, not a plan to keep fine-tuning this particular model. Manually reading 39 responses field by field was possible at this scale; it does not scale to the volume of adversarial and edge-case testing a security-classification task like this actually needs. The failure modes surfaced here — a well-formatted but wrong verdict, a scope boundary crossed by a structurally similar but unrelated prompt, a generation collapsing into repetition under a specific kind of complexity — are exactly the category of thing an automated red-teaming and evaluation pipeline is built to catch systematically, rather than by chance, across far more than 39 hand-checked cases.

That reframing — from "fine-tune a model to perform this task" to "build the tooling to automatically probe and score any model against this kind of failure" — is the direction this work is heading next, using structured red-teaming frameworks such as Garak to turn the manual, one-at-a-time comparison done here into a repeatable, scaled evaluation process.

References

Betley, J., Tan, D. C. H., Warncke, N., Sztyber-Betley, A., Bao, X., Soto, M., Labenz, N., & Evans, O. (2025). Emergent Misalignment: Narrow Finetuning Can Produce Broadly Misaligned LLMs. Proceedings of the 42nd International Conference on Machine Learning (ICML), PMLR 267:4043–4068. https://arxiv.org/abs/2502.17424
Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs. (2025). https://arxiv.org/pdf/2507.03662
From Narrow Unlearning to Emergent Misalignment: Causes, Consequences, and Containment in LLMs. (2025). https://arxiv.org/pdf/2511.14017
Mitra, S., Neupane, S., Chakraborty, T., Tripathi, H., Mittal, S., Piplai, A., & Rahimi, S. (2026). Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA. arXiv:2607.18725. https://arxiv.org/abs/2607.18725
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning. (2024). arXiv:2411.15382. https://arxiv.org/pdf/2411.15382
How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns. (2026). arXiv:2512.24063. https://arxiv.org/html/2512.24063
Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs. (2025). arXiv:2510.02833. https://arxiv.org/pdf/2510.02833
A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification. (2024). arXiv:2411.02476. https://arxiv.org/pdf/2411.02476

DEV Community: Ahmed El alaoui