DEV Community

Cover image for I built an epistemic gate to stop LLM data poisoning during fine-tuning. Tested across 5 architectures, orchestrated on a 2006 Toshiba laptop for $0.
Eduardo
Eduardo

Posted on

I built an epistemic gate to stop LLM data poisoning during fine-tuning. Tested across 5 architectures, orchestrated on a 2006 Toshiba laptop for $0.

Hi everyone,
For a long time, the AI industry has pushed the narrative that advanced safety research, data poisoning auditing, and model alignment require massive clusters and millions of dollars.
I wanted to test if that's true. Over the last few months, using a 2006 Toshiba Satellite (2GB RAM) for orchestration and free Kaggle T4 GPUs ($0 cost), I built and ran a 16-experiment series (EXP08–EXP16) exploring epistemic collapse and data poisoning in LLMs.
Today, I'm open-sourcing the first phase of this work: Beatriz Epistemic Gate.
What problem does it solve?
Malicious data poisoning during fine-tuning is often surgical and invisible to aggregate metrics (like standard perplexity). An attacker can destroy specific facts down to an exact indifference point while global metrics actually improve because of generic fine-tuning on fluent prose. This makes silent poisoning a stealthy threat for small teams and startups fine-tuning local models.
How does Beatriz work?
Instead of invading or rewriting complex training loops, Beatriz acts as a lightweight, non-invasive defensive proxy (~0.1ms decision latency) that sits between a generative source and a training student model. It verifies generated text against an immutable anchor corpus using cosine similarity in embedding spaces, applying a composite loss function:
• LceLce: Preserves linguistic fluency.
• LcontrastivaLcontrastiva (Softplus): Keeps active pressure on the model to anchor to truth even after cross-entropy saturates.
Key Results (EXP08–EXP16):
• Multi-architecture validation: Tested across GPT-2 (124M), Qwen-2.5-0.5B, TinyLlama-1.1B, Pythia-1.4B, and Phi-3-mini-4k-instruct (3.8B).
• Ablation (EXP15): Pure filtering (GATE_ONLY) provides 65% of the defensive benefit out of the box. The contrastive term adds the remaining 35%.
• Scaled Held-Out (EXP16): Validated on a rigorous 30 multi-domain unattacked facts benchmark, consolidating a robust truth margin of +4.19 ± 0.08 with 0.93 Precision and 0.80 Recall on the gate.
What's in the repo right now?
To avoid overwhelming the community, I'm releasing the project in phases. Today you'll find:

  1. EXP08 & EXP09 code & notebooks (fully reproducible on Kaggle).
  2. Technical Whitepaper (WHITEPAPER.md / WHITEPAPER.es.md) detailing all 16 experiments.
  3. Corrective Manual (docs/MANUAL_CORRECTIVO.md) with formal mathematical loss specifications. If you're an indie dev, researcher, or startup exploring safe local fine-tuning, check it out, run the notebooks, and let me know your thoughts! GitHub Repo: [https://github.com/EduardoAyalaT/beatriz-epistemic-gate-]

Top comments (0)