Yep, tried running on localhost.
Interesting observation - gunicorn 9 sync workers - the query time to db is around ~500ms, RPS is 5.
gunicorn + gevent - query time gets from 1 sec to 100 sec and it has the same RPS result as sync. I think psycorp2 monkey patching is doing a poor job and with gevent it blocks the main thread. sigh.
I honestly don't understand how the app is setup, can you describe it better?
You have a flask web app which runs on production with gunicorn but why did you choose to use gevent as well? Have you tried without it?
With psycorp2 you mean psycopg2? Why is it monkey patched? Because of gevent? Which version of PostgreSQL are you using?
What happens if you just start the app without gunicorn and gevent locally and you benchmark it? 5 requests per seconds is pretty low, what is your app doing?
Thank you rhymes for feedback! Basically, I found out that the bottleneck was the db, the query performance is 5 transactions per second. After query optimization to 10 rps I decided to cache some info in redis and update the cache in background jobs. But another problem came out (I think it's good karma 😇). Redis benchmark shows 50k rps, but in my endpoint it makes only 50rps 😫 Soo, I'm currently profiling the sh*t of the code.
The app setup is next ->
postgresql 11 with 47 connections with pgbouncer
redis
flask
gunicorn with gevent worker (why gevent because it's good for I/O bound work and for traffic spikes)
Let's recap because you're giving out information in small pieces and I'm having difficulties following what's happening:
you have a Flask/Flask-Alchemy/PostgreSQL app
this app runs on production with gunicorn and gevent
PostgreSQL is on a digital ocean
using pgbench from your localhost you're getting 5 RPS
There a few aspects to consider:
pgbench tests PostgreSQL, not your app. It also tells you transactions per second, not requests per second.
I'm not sure testing a remote PostgreSQL from your local machine is useful at all. Your app if deployed in the same network as PostgreSQL is not going to behave the same way anway. You should probably test it from the server that hosts the app
Are you running pgbench correctly? Benchmarking can introduce methodology errors. How many connections are you opening? Can you provide the command line you run the test with?
What happens to said PostgreSQL's memory when benchmarked?
Finally, have you tried to actually load test the production app? Or at least a copy of it. With tools like wrk for example.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Yep, tried running on localhost.
Interesting observation - gunicorn 9 sync workers - the query time to db is around ~500ms, RPS is 5.
gunicorn + gevent - query time gets from 1 sec to 100 sec and it has the same RPS result as sync. I think psycorp2 monkey patching is doing a poor job and with gevent it blocks the main thread. sigh.
I honestly don't understand how the app is setup, can you describe it better?
You have a flask web app which runs on production with gunicorn but why did you choose to use gevent as well? Have you tried without it?
With psycorp2 you mean psycopg2? Why is it monkey patched? Because of gevent? Which version of PostgreSQL are you using?
What happens if you just start the app without gunicorn and gevent locally and you benchmark it? 5 requests per seconds is pretty low, what is your app doing?
Sorry but it's hard to help without information
Thank you rhymes for feedback! Basically, I found out that the bottleneck was the db, the query performance is 5 transactions per second. After query optimization to 10 rps I decided to cache some info in redis and update the cache in background jobs. But another problem came out (I think it's good karma 😇). Redis benchmark shows 50k rps, but in my endpoint it makes only 50rps 😫 Soo, I'm currently profiling the sh*t of the code.
The app setup is next ->
Sorry if I re-iterate: have you tried without pgbouncer? Because 10 rps is ridicously low. How many RPS do you measure without it?
You shouldn't need redis either, at least not as a patch for the problem.
The easiest way you can get out of this is to isolate each component.
What happens if you remove pgbouncer AND gevent?
I measure on localhost without any pgbouncer, using pgbench, so no gevent and pgbouncer.
Let's recap because you're giving out information in small pieces and I'm having difficulties following what's happening:
pgbenchfrom your localhost you're getting 5 RPSThere a few aspects to consider:
pgbenchtests PostgreSQL, not your app. It also tells you transactions per second, not requests per second.pgbenchcorrectly? Benchmarking can introduce methodology errors. How many connections are you opening? Can you provide the command line you run the test with?Finally, have you tried to actually load test the production app? Or at least a copy of it. With tools like wrk for example.