Ruby has so many awesome, easily accessible gems, but their ease of use can lull you into a sense of complacency. This is what happened to us at Ke...
For further actions, you may consider blocking this person and/or reporting abuse
Curious: Have you monkeypatched any gem functionality to alter its behavior while problem-solving along these lines?
We have, probably more than I care to admit.
Luckily, most of the monkey patches are not permanent. Usually we will monkey patch a gem in our code for a quick fix and then often will issue a PR against the gem's open source repo with our update. Couple recent times we have done this are:
1) With the redis-rack gem when dealing with blank session keys
2) Getting Honeybadger to respect sidekiq defined thresholds
Other times we get a little lazy and forget to open a PR against the gem's repo. Your question actually just reminded me of one monkey patch that we use to optimize some Resque code that others would probably appreciate. Excuse me while I go write myself a TODO for that 😊
I tried to ask the question in the most neutral way in terms of judgment knowing there's bound to be some rationalization abound.
I think it's a tool at the disposal of Rubyists that gets fairly polarized, but everybody does it at some point for practical reasons.
Do you know of any static code analyzers that can detect all monkeypatches against vendor gems in a codebase?
find config/initializers -type f -name '*active*':-) :troll:
I do not know of any off the top of my head and a quick Google search didn't turn up much. Given handy Ruby methods like
source_locationI would bet it would not be hard to create one.Rather than monkey patching, have you considered forking the gem, making your update and installing the gem from that fork?
That way you avoid monkey patching, fix the issue and have a ready made pull request to the original repo.
This is an interesting journey of discovery. I bet there are many developers that install a gem (or other library for some other language) for a specific use but never find out how much it could do for them.
Glad you solved your issue and stopped pulling all that data from Redis every request!
"...Soon we were reading 7.8 MB/second from Redis which was not going to be sustainable as we continued to grow...."
Was the option of a multi-node Redis cluster on dedicated hardware considered? 7.8MB/s read from RAM memory does not seem all that terrible.
Could clarify the solution a bit please. Instead of hitting Redis before each MySQL request you will referance the local AR conneciton object; does the connection object keep the Redi data in memory? I am confused (not familar with the Octopus gem).
"...Another great way to learn more about how your gems are working is through logging!..."
Great article Molly, I look forward to the next one!
Thanks David!
First point:
We honestly could have let this ride for a bit. 7.8 MB/sec was high but was not causing any issues for our beefy ElastiCache(In that logging post I mention that due to other requests we actually got to a level where it was causing other errors). However, we knew it was only going to get worse as we grew and while we could have beefed up ElastiCache we really wanted to save that as a last resort. We felt there had to be a way to be smarter about how we were getting the sharding data we needed which is why we went poking around ActiveRecord.
A big theme for our SRE team over the past year has been "working smarter" not harder. Every time we come across high loads we first look for ways to "fix" the loads. We use beefing up hardware as last resort.
Second point:
The ActiveRecord/Octopus Proxy object reads the data from MySQL actually when the process starts up and then holds that hash in memory to be used throughout the lifetime of the process. Prior to this, when the process would start up we would write the data to Redis from MySQL then every time we made a MySQL request we would read it from Redis.
Let me know if that makes sense or if you have any other questions!
Next post coming up will involve processing data in bulk, stay tuned 😁