Help me Devs! I work on a web app which deals with a lot of data that moves really fast. Our primary database (MySQL) simply cannot keep up, so we store a lot of “current state” data in Redis.
Today we (de)serialize the objects with JSON and store the whole object. That leads to several issues, so we’re going to switch to storing each field individually.
The code for managing all these fields is going to get icky fast, so I’m thinking about building a general purpose ORM for Redis. Some of these already exist but don’t have all the features I need (e.g. transactions with pipelines), locks.
Is this a horrible idea? Am I just building a product that already exists elsewhere that I just don’t know about?
Thanks for your response!
EDIT: There have been a few responses suggesting that the problem has to do with JSON. To clarify, when we were storing this sort of data in MySQL it was normalized, not JSON. Using JSON is only a means to persist objects to Redis.
Latest comments (31)
Hi Dylan,
I'm new in DEV and I'me fallen on you're post, just few days after implementing a redis connector to small-orm.
For now, it is implemented only on swoft (The connector was specific because of async specificities of swoole) but it can inspire your will : github.com/sebk69/small-orm-swoft and github.com/sebk69/small-orm-core
You can store your data in Redis, no problem. I don't know what language are you using (I guess Python), but in Ruby there are several Redis ORMs. Maybe you can take a look and get some ideas:
I don't think Redis is only intended for caching, it can be used to calculate distances between coordinates, do unions/intersections between groups, create scoreboards, etc. There are background job processors like Sidekiq that also use it to store and manage their jobs in a persistent way.
You might want to give CassandraDB a try, since it sounds like you need a non relational database instead of MySQL.
Thanks! I’ve never looked at that one before
If you decide to go with Redis, and the built-in data structures (sets, hashes, etc') don't cover your requirements (and I do suggest you first try to make it fit - you should get into the "Redis mindset" and not always expect Redis to fit your code as it is), you may want to look into RedisJSON - it's an official Redis module for storing and editing JSON natively without (de)serialization. There are client libraries for several platforms.
Boy would I love to. Unfortunately we’re using Elasticache and it isn’t supported :(. Good suggestion though!
An interesting resource is redisson: github.com/redisson/redisson
If You work with Java it make your day... if not, in the source You'll find a lot of interesting idea...
Cool, I’ll take a look for sure! I’m working with Python which has a few options but none are quite all the way there. If I do end up going this route though I’ll check out this project for inspiration. Thanks!
This keeps coming up, I'll definitely try MongoDB! Also considering DynamoDB because I'm on AWS.
If you're on AWS I strongly suggest DynamoDB. It has a fairly low barrier to entry and is covered by their forever-free tier if you want to ptototype a bit with it. (PS. Use on-demand tables, not the auto-scaling throughput ones. On-demand is magic.)
Im definitely thinking about it. Unsure how pricing will scale, napkin calculations look a little scary 👻. It’s on the short list though, thank you!
Thanks a bunch! Several people have suggested trying out a NoSQL database so that's probably the direction I'm going.
It seems like you're in need of a document database. Have you considered the option?
Oh hey rhymes! I'm considering the option now due to all these excellent replies!
There are three ways:
I recommend the 3.
Thank you! I clarified a bit in an edit but JSON was not the goal, just a means to store data in Redis. In MySQL we had normal tables with columns. I hadn't been considering a NoSQL database before, but I'm certainly considering it now!
Oooh now I understand. Writes on normalized tables (with index rewrites and such) can be slow. I'm not 100% up to date on MySQL (I main use PostgreSQL) but I remember there's the possibility of having a JSON column for destructured data. Maybe you can test that before moving to a separate NoSQL DB.
I would test a JSON column, then if it's still too slow I would consider a separate DB. Maintaining one DB is still better than multiple DBs :D
Let us know about your findings, I'm very curious!
"...Our primary database (MySQL) simply cannot keep up..." MySQL runs some of the biggest data sets in the world. Find a good DBA. Clustering, federation, caching.
Using Redis as a caching layer is a great idea, but as a persistent storage media; I would not recommend that. Sure, it can save to disk, has replication, complex object types, etc. But Redis is, by its nature, a cache, not persistent storage.
Right tool for the right job.
Thanks for the suggestion! I'll clarify the use case just a bit. The problem is not with read performance from MySQL, but with write performance to it. The data we're storing here is transient by its very nature, with each record changing several times per minute. It's also not data that we'd care much about losing (we persist as much as we need to in a down-sampled fashion to other storage mediums). So even if we could tune MySQL to accept writes fast enough (which would be expensive if we need to hire a new full time employee to figure it out!) we don't need most of the features it gives us.