Steps
- Refactoring existing code base, to split db logic and business logic
- Integration test
- Implementation
- Load test and Performance Result
Load test
Load test is used here to measure the performance.
- same Machine (my personal macbook laptop)
- same spec for DB
- 10GB Volume
- Same ARM64 Instance
Load Test result and Performance Compare
Original Postgres Result
Switch to use Redis through Redis-Py
Switch to use Redis through Redis OM
The 2 key results here are iterations (higher is better) and iteration_duration(avg)(lower is better)
| Fact | Postgres | Redis-Py | Redis OM |
|---|---|---|---|
| Iterations | 25.26/s | 5.52/s | 2.30/s |
| Iteration Duration | 560.15ms | 5.53s | 8.07s |
This performance benchmark is surprising to myself as well, Redis as a well performanced DB,
in most of time, it give a very good result of performance.
Why Performance get worse after switching, 3 Potential Reasons
This performance compare may not be objective, the reasons behind it requires more researching. These are some potential reasons I assume,
1. Min Specs for different DB are difference
Even the spec for Postgres and Redis Stack here are the same, but their min spec requirements may be different.
- Postgres: 1 GHz processor. 2 GB of RAM. 512 MB of HDD.
- Redis: 2.6GHz(2 CPUs), 4 GB of Ram, 10 GD of HDD (Redis Stack supposed to be higher)
2. Data Type Convert May Slow the API
Decimal in Python as a class could not be saved directly as a JSON attribute in RedisJSOM,
so it was stores as string, and it will be converted to Decimal when read. This may slow the APP down.
3. Query was not used (RedisSearch) was not used,
So when query history data, a loop of GET are sent to DB, which is not efficient
Conclusion 1
This comparing result doesn't tell which is better between Postgres and RedisJSON, it only says that at current stage, at current DB Instance spec, for this API,
postgres gives better performance, and the way of Decimal Convert and Loop of Get in Application level slows the app too.
(If anyone knows better about Redis to solve these problems, please let me know). So this comparing seems not very fair.
2022 October 25, Get Rid of Assuming Data Conversion in Codes
The result is in the 3rd Table Column Above.
Try solve the issue No 2, stop using Python data convert in application level, and No 3, tried to use
findfrom Redis Search
Tried use Redis-OM for the querying, in this branch,
The result is still not good enough, (actually even worse), issues found:
Redis-OM-Python function was limited,
sort byfunction is not supported,findfunction does not support 2 + conditions (not sure way, this is different with official document)
Conclusion 2
- Redis-OM-Py is very new, it requires more community contribution
- The root cause of the performance issue after switch is not from application level, it is still something either from db level, or db SDK/driver performance
Repos
tim-hub
/
sanic-currency-exchange-rates-api
This is a self hosted, free, open source Python Currency Exchange Rate API fork.
Sanic Currency Exchange Rates Api
This is a self hosted, free, currency exchange rate api, free demo at exchange-rate.bai.uno.
The current and historical foreign exchange rates data are from European Central Bank .
- More about Redis Stack
Usage
Latest date will be returned, if the date was not in Euro Central Bank history records
Latest & specific date rates
Get the latest foreign exchange rates.
GET /api/latest
Get historical rates for any day since 1999.
GET /api/2018-03-26
Rates are quoted against the Euro by default. Quote against a different currency by setting the base parameter in your request.
GET /api/latest?base=USD
Request specific exchange rates by setting the symbols parameter.
GET /api/latest?symbols=USD,GBP
Rates history
Get historical rates for a time period.
GET /api/history?start_at=2018-01-01&end_at=2018-09-01
Limit results to specific exchange rates to save bandwidth with the symbols parameter.
GET /api/history?start_at=2018-01-01&end_at=2018-09-01&symbols=ILS,JPY
Quote the historical rates against a different currency.
GET /api/history?start_at=2018-01-01&end_at=2018-09-01&base=USD
Top comments (0)