Seems like you're talking about FaaS and not serverless in general.
I know Lambda is seen as the very incarnation of serverless, but actually it's just a tiny part of it and many services have serverless properties (S3, API-Gateway, DynamoDB, AppSync, Cognito, etc.) and those usually don't have the latency problems you mention.
What I am talking about is the serverless hype that everything should be serverless in the app stack. That is clearly highly problematic. For a fuller read on this I recommend theregister.co.uk/2018/12/19/serve...
which points to a good study from UCB at www2.eecs.berkeley.edu/Pubs/TechRp....
I am not at all against these various services. What I am against is yet another set of baseless solve everything silver bullet claims. I have seen organizations go so overboard on "no servers of our own" that they produce much more complicated and inflexible stacks requiring understanding of a dozen different technologies or so just for the plumbing bits and not having to do much in house. This on a project that would have been fairly trivial to do most of in a more conventional manner at lower money, time, aggravation, lost opportunity cost. They are not alone.
It's just a quesion of risk distribution.
If you think it's harder to learn these technologies and easier and more cost efficient to manage your own infra, could be that you're right. I know many people who think that way.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.