DEV Community: Alex Mercer

Catastrophic Forgetting by Language models.

Alex Mercer — Fri, 27 Feb 2026 18:03:30 +0000

To all the awesome experts in AI/ML out there. i need a favor.
I realized there is a gap in Language Models (SLMs/LLMs) remembering the data continuously which is termed as 'catastrophic forgetting'.

To solve that problem I came up with an adapter called Constrained Residual Mixing Adapter (CRMA) that enables continual learning. I tested it on TinyLlama 1.1B and Mistral 7B — the result: -0.1% drift across 4 sequential
domains. Essentially zero forgetting.

CRMA: -0.1% drift. Naive: +351% forgetting. Same model, same data, same hardware.

Holds at both 1.1B and 7B. No replay, no EWC, no KD needed.
● CRMA Modular vs Naive — Mistral 7B (4 sequential domains)

┌─────────┬────────────┬──────────────────┐
│ Task │ CRMA Drift │ Naive Forgetting │
├─────────┼────────────┼──────────────────┤
│ Medical │ -0.2% │ +228% │
├─────────┼────────────┼──────────────────┤
│ Legal │ -0.1% │ +593% │
├─────────┼────────────┼──────────────────┤
│ Code │ -0.1% │ +233% │
├─────────┼────────────┼──────────────────┤
│ Finance │ +0.0% │ — │
├─────────┼────────────┼──────────────────┤
│ Average │ -0.1% │ +351% │
└─────────┴────────────┴──────────────────┘

Now the favor - If you're interested in independently verifying these results, I'd love to hear from you. DM me and I'll share what you need to reproduce it. Thank you. and best wishes

Catastrophic Forgetting of language models.

Alex Mercer — Sun, 22 Feb 2026 11:31:48 +0000

To all the awesome experts in AI/ML out there. i need a favor.
I realized there is a gap in Language Models (SLMs/LLMs) remembering the data continuously which is termed as 'catastrophic forgetting'.

CRMA: -0.1% drift. Naive: +351% forgetting. Same model, same data, same hardware.

Holds at both 1.1B and 7B. No replay, no EWC, no KD needed.
● CRMA Modular vs Naive — Mistral 7B (4 sequential domains)

Now the favor - If you're interested in independently verifying these results, I'd love to hear from you. DM me and I'll share what you need to reproduce it. Thank you. and best wishes