Because people are shite at security.
Ok I should probably elaborate:
Many people tend to pick bad passwords if left to their own devices: God, love, 123456, qwertyuio, etc. This is why we have obnoxious password requirements.
The same holds for security questions. E.g. somebody might think "what is your favourite band/singer?" is a good secure question, without realizing that their profile picture show a teenage girl and that "Justin Bieber" might then be a pretty good bet (I don't actually know what teenage girls listen to nowadays, but let's pretend it's JB). Basically you could hack by statistic. In contrast you are far less likely to reconstruct the "name of your first pet" from information users post freely on social media.
However, security questions should not be a thing at all considering the ridiculous wealth of information you can find on people. IIRC this is how "the fappening" came to be: just googling the answers to celebrities' security questions. And unlike passwords, answers to security questions cannot be changed between sites to protect yourself from leaky, poorly encrypted (if at all) databases.
You raise some good points there. Another thing these sites don't consider is that someone may not have a pet, they may not have a spouse or sibling to name or even know what city they were born in. Not allowing people to put in their own questions is not inclusive at all. Some of the questions in this limited set may also be things people rather not reveal to god knows who.
I think you may have missed my point: users are dumb and cannot be trusted to pick their own security questions.
It really depends on the audience you are expecting. If you are expecting information security experts or privacy enthusiasts it's probably safe to say they know enough to not do the "What's my favorite band?" kinda thing.
But if you are building services for the average user then you do not give them any choices that could make them less secure if they don't do it right.
So always encourage passphrases and a good password manager, U2F and TOTP based authentication apps.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.