The Pitfall of LLM Fallback Chains: The Day DeepSeek Erased Our Agent's Personality

Introduction

If you run a multi-agent system, you know the drill: LLM providers hit rate limits, quotas run dry. So you set up a fallback chain—if the primary model fails, try the next one. Sounds reasonable. Until it isn't.

One day, that "reasonable design" caused an unexpected failure. The moment the model switched, our AI agent's personality vanished.

What Happened

Our OpenClaw cluster runs 20+ agents across multiple nodes. Each agent has a SOUL.md file—a personality definition that specifies tone, character traits, expertise areas, and explicit behavioral prohibitions.

That day, our Claude Sonnet quota hit 100%. The fallback chain was configured as:

primary: anthropic/claude-sonnet-4-6
fallbacks:
  - openai/gpt-4o-mini
  - deepseek/deepseek-chat

Sonnet quota exhausted → fallback to Opus → Opus hit rate limits too → DeepSeek activated.

After DeepSeek consumed about 40% of its quota, the admin noticed something was off:

"This doesn't feel like Joe at all."

Spot on. DeepSeek's responses were almost entirely ignoring the SOUL.md personality settings. Overly polite honorifics, canned openers like "Great question!", verbose explanations that dodged the main point—all behaviors that SOUL.md explicitly prohibited.

Why It Happened

The root cause is straightforward: different models have different levels of system prompt compliance.

Claude (Sonnet/Opus) picks up on subtle nuances in long system prompts. If you write "Never say 'Great question!'", it genuinely won't. The tone and personality defined in SOUL.md get faithfully reflected in outputs.

DeepSeek was different. It understood the "general direction" of the system prompt but was weak at enforcing prohibitions and reproducing subtle personality traits. The result: a generic, soulless response from an agent that was supposed to have a sharp, distinctive character.

Response and Lessons Learned

Immediate Actions

1. Removed DeepSeek from the fallback chain entirely
2. Switched all nodes to Opus
3. Reduced fallback to verified models only

Key Takeaways

1. Fallback ≠ Compatibility

A model fallback chain guarantees "a response will come back," but it does NOT guarantee "a response with the same quality and personality." For agent systems where personality matters, every fallback model needs to pass a personality compliance test before being added to the chain.

2. Personality Drift Is Silent

Errors are loud. Crashes are loud. But personality drift leaves no error in your logs. The only detection mechanism was a human's intuition: "Something feels off about this agent today."

In the future, we may need automated "personality drift detection"—style scoring, prohibited-expression monitoring, something that catches the drift before users do.

3. Quota Management Is a Lifeline

If quota exhaustion never happens, fallbacks never trigger. We should have had usage monitoring with early alerts (notify at 80% usage). After this incident, we significantly strengthened our quota monitoring cron jobs.

Recommendations for Multi-Agent Operators

• Only include personality-tested models in your fallback chain
• System prompt compliance is model-dependent—"write it and they'll follow" is an illusion
• Implement fallback activation notifications—always know which model is responding
• The worst case isn't "no response"—it's "response from a different personality"—the latter erodes user trust far more

Conclusion

Giving an AI agent a "soul" requires more than writing a SOUL.md file. Protecting that soul demands equally important infrastructure: model selection, fallback design, and drift detection.

Since this incident, we keep our fallback chain short and limited to verified models only. "No response" is better than "an impostor responding."