Over the last year I have been getting random ActiveRecord::ConnectionTimeoutError errors, where the application could not obtain a connection from the pool within 5 seconds. But I'm stumped as to what the actual issue is...
- we're using an AWS RDS instance that can handle hundreds of connections, which we never even come close to by the RDS connection log (max use is ~30).
- I have been over forums that talk about using Thread.new... but we don't spin any of our own threads.
- I've seen forums that talk about workers using more pool connections than necessary, but I haven't set up any puma workers, we just use web instances!
- There is no consistency in which controller#action throws the error, and no consistency on time of day or week.
- Sometimes the error will not show up for weeks, and then I'll get 15 instances of it in a few days
I'm at a loss as to where to even start looking...