Same fine-tune recipe, different seeds — one narrow behaviour swings wildly. What actually stabilises it?

I'm fine-tuning a small adapter for a talking-face model — a short recipe, same data throughout. The only thing I change between runs is the seed. The overall score barely moves, but one narrow behaviour swings: on one seed it collapses, on another it lands fine, on a third it overshoots.

Has anyone dealt with this pattern? What actually helped in your experience — seed averaging, EMA / checkpoint soup, a longer schedule, or judging the behaviour with its own per-behaviour metric instead of the aggregate score?