What's your rule for when an LLM call should NOT be retried?

Everyone has a retry story. Mine: an agent that retried a malformed request eleven times before anyone noticed, each attempt replaying the full conversation. The request was never going to succeed — a 400 doesn't heal — but the retry policy didn't care, because it only looked at "did it throw?"

Our working rules now:

• Retry timeouts, 429s and 5xx — with backoff, jitter, and the server's Retry-After winning over our schedule.
• Never retry 400/401/403. Fail fast, surface it, fix the request.
• A retry budget, not just a count: retries get a slice of the task's total budget, then the task fails honestly.
• If it dies mid-stream, restart the message cleanly — never append to a half-generated answer.

What I haven't settled: provider 529s / overloaded errors during peak. Retry in place, or fail over to a different model immediately and accept the quality change?

What's your rule set — and which error taught it to you?