I started watching this series: https://www.youtube.com/watch?v=5LOdKDqdWYU&list=PLlmVY7qtgT_lkbrk9iZNizp978mVzpBKl&index=1
The second part of the series has a great message.
You want your tests to be insensitive to the structure of your code. I picked this up somewhere along the way but strongly with Java and interfaces. You write your tests to your interfaces not the internals of the concrete implementations.
If you are not in Java you still have an interface. Even if you have not explicitly defined it is there.
To quote Rush
If you choose not to decide You still have made a choice
Another point here is that refactoring should not break tests in this way. You are changing how things work not the interfaces.
A good series of videos if you have time to watch it.