Should I test absolutely every public method?

No, of course not. Tests are also code, on which we spend time and money. Don't be paranoid: don't strive for 100% test coverage.

Don't test trivial methods, third-party libraries code and things you have already tested indirectly: presence of classes, methods, number of arguments.

Here's what Kent Beck writes on the subject:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence (I suspect this level of confidence is high compared to industry standards, but that could just be hubris). If I don't typically make a kind of mistake (like setting the wrong variables in a constructor), I don't test for it. I do tend to make sense of test errors, so I'm extra careful when I have logic with complicated conditionals. When coding on a team, I modify my strategy to carefully test code that we, collectively, tend to get wrong.

And here's what DHH says:

1. Don't aim for 100% coverage.
2. Code-to-test ratios above 1:2 is a smell, above 1:3 is a stink.