In an effort to better protect our production data, last year we choose to setup a readonly production console. This allows developers to poke around production data without having to worry that they might accidentally change something they shouldn't. In this post, I will break down exactly what we did in order to accomplish this for our Ruby on Rails application.
Before I dive into the specifics for each database, I want to first mention that we use a completely separate server for console access. Using a separate server allows us to tweak application settings in order to achieve readonly access. In order to deploy these changes we use Ansible. When Ansible runs a deploy it looks for the console box tag to know what settings and configs need to be deployed to that particular box.
MySQL
The first thing we did was setup a user with readonly access in MySQL. Then, in order to make our application readonly for MySQL, all we simply had to do was put the readonly user credentials in our database.yml file on our console server box.
production:
adapter: mysql2
encoding: utf8
reconnect: true
pool: 16
database: "prod_db"
username: "readonly"
password: "you_wish"
host: "127.0.0.1"
strict: false
Now any time someone opens up a Rails console on our console server it is automatically using the readonly credentials from the config.
However, there are still times when we want to be able to allow devs to edit data via the console. In order to accomplish this, we setup a bash script which is used to open a Rails console. In this bash script we choose to use a lesser known feature that Rails offers, DATABASE_URL. If you set the DATABASE_URL variable in your environment, Rails will use it to connect to your database rather than reading from your database.yml file. This allows us to override our database configs when we need to. We set it up in our bash script like so:
#!/bin/bash
cd /application_path
if [ "$1" = 'write' ]; then
export DATABASE_URL="mysql2://write_username:write_password@host/db_name"
fi
RAILS_ENV=production /usr/local/bin/bundler exec rails console
Now if a developer needs to edit data they can simply open up a write console using the command console write.
Redis
To handle setting up Redis as readonly we choose to override our Redis client to explicitly block any write commands. Since we have a Ruby on Rails application we use the redis-rb gem in order to talk to Redis. To block write commands we first collected all the commands that were write based by calling the command method on our Redis client. For reference, Rails.cache.data will simply give you your Redis client.
dev> Rails.cache.data
=> #<Redis client v4.0.3 for redis://127.0.0.1:6379/15>
The command method will return an array of all the commands your Redis instance will respond to along with some additional information about each command.
dev> Rails.cache.data.command.first(3)
=> [["expireat", 3, ["write", "fast"], 1, 1, 1],
["setnx", 3, ["write", "denyoom", "fast"], 1, 1, 1],
["getrange", 4, ["readonly"], 1, 1, 1]]
To filter out only the write commands we simply checked for the "write" value in the list of command attributes.
WRITE_COMMANDS = Rails.cache.data.command.map { |a| a[0] if a[2].include?('write') }.compact.to_set
Once we had a list of write commands, we overrode the process method in our gem to raise an error if any of those methods were called.
def process(commands)
if commands.flatten.any? { |c| WRITE_COMMANDS.include?(c.to_s) }
raise NotImplementedError, "REDIS_ACCESS_MODE is set to 'readonly', disallowing writes"
end
# additional method logic
end
Those two pieces allow us to block Redis write commands. But, the question still remains, how do we block those write commands ONLY on our console box? Once again, we turned to our environment variables and our bash console script. In our console script we set our environment variables based on if the console was the default readonly or if it was a write console.
#!/bin/bash
cd /application_path
if [ "$1" = 'write' ]; then
export DATABASE_URL="mysql2://write_username:write_password@host/db_name"
export REDIS_ACCESS_MODE=""
else
export REDIS_ACCESS_MODE="readonly"
fi
RAILS_ENV=production /usr/local/bin/bundler exec rails console
Then, in our redis.rb initializer file in our application, we monkey patched the process method to return an error if a write command was called in readonly access mode.
if ENV['REDIS_ACCESS_MODE'] == 'readonly'
class Redis
class Client
WRITE_COMMANDS = ::Rails.cache.data.command.map { |a| a[0] if a[2].include?('write') }.compact.to_set.freeze
def process(commands)
if commands.flatten.any? { |c| WRITE_COMMANDS.include?(c.to_s) }
raise NotImplementedError, "REDIS_ACCESS_MODE is set to 'readonly', disallowing writes"
end
# additional method logic
end
end
end
end
BOOM! The console box was now readonly for MySQL and for Redis by default. Only one piece of the puzzle was left, Elasticsearch.
Elasticsearch
Elasticsearch is at the cornerstone of our application so we needed that to be readonly as well. To talk to Elasticsearch we use the elasticsearch-ruby gem. Much the same way we did Redis, we found the core method used to make external requests to Elasticsearch, perform_resquest and patched it so that it would raise an error whenever a write method was executed. Since we talk to Elasticsearch using HTTP requests, the methods we wanted to block were PUT, POST, and DELETE.
module Elasticsearch
module Transport
class Client
if ENV['ELASTICSEARCH_ACCESS_MODE'] == 'readonly'
def perform_request(method, path, params={}, body=nil, headers=nil)
raise 'Elasticsearch is in readonly mode.' if method.to_s.match?(/PUT|POST|DELETE/)
method = @send_get_body_as if 'GET' == method && body
transport.perform_request(method, path, params, body, headers)
end
end
end
end
end
Once again, we also choose to use an environment variable to determine whether or not we should be patching the perform_request method. We then took that environment variable and added it to our bash script. Our completed bash script looks like this:
#!/bin/bash
cd /application_path
if [ "$1" = 'write' ]; then
export DATABASE_URL="mysql2://write_username:write_password@host/db_name"
export REDIS_ACCESS_MODE=""
export ELASTICSEARCH_ACCESS_MODE=""
else
export REDIS_ACCESS_MODE="readonly"
export ELASTICSEARCH_ACCESS_MODE="readonly"
fi
RAILS_ENV=production /usr/local/bin/bundler exec rails console
This script ensures that when a dev or support person is opening a console using the console command, by default it will be readonly. When necessary, they can call console write if they need to update any data. Even though it is very easy to open a write console, the vast majority of the time people are working in readonly consoles. The readonly consoles have proven themselves many times over by saving people from making silly mistakes while browsing production data.
Other Options
There are many other ways to approach data safety when it comes to working with production data. This is just one approach and the one we have chosen to use at Kenna. One other very popular option is to make a replica, or clone, of your data and then allow people to run whatever queries they want against that replica or clone. The downside to this is that getting an up-to-date replica every time you need it can be time consuming depending on the size of your dataset. In addition, cloning a single database such as MySQL is pretty straightforward. However, when you are working with 3 different data stores, such as we do at Kenna, it is a lot more effort to replicate all of that data together.
At Kenna, devs have the ability to clone production data, but it is only available in MySQL at the moment. Normally, a MySQL clone is only used to check high risk migrations or scripts that are going to change a lot of production data at once. Beyond that, we have found our readonly console has worked great for our use case because the majority of the time devs and support simply want to look at up-to-date production data.
Hope you found this post useful! As always, please let me know if you have any questions! 🤗
Top comments (7)
This article is gold for any team running Rails and needing to limit access! I love that it is running on a separate server as well. I could see having two console servers and only allowing write access on one as well. This way, you could only provide credentials to specific people for the write server.
While it can be useful to have access to real-time production data, I think this lowers the bar too much to actually 'poke around'.
In most cases, I think there are 2 options. Either a database copy is OK (which can be incremental and pretty close to the production instance), but anonimysed. Or, in case your application is on fire, you actually need write access to the database. Then a read only console will not do.
For our use case, a lot of times support will get an inquiry about some data and this allows them to go and dig through all the details to figure out what is happening and why the data looks the way it does. In our case, "poking around" for our support team is a daily normal
Molly done it again!
Just wondering - does anyone have access to a read-write Rails console? This would be a great idea for junior devs or incoming hires that aren't familiar with the DB yet but want to poke around. I'm gonna float this to my company.
Yep! You can open a write console anytime you want by issuing the command
console writeEven though its easy, people only open write consoles if they absolutely have to change something otherwise everyone loves the read-only consoles bc they feel "safe" in themIs the console an interactive, Ruby-specific thing? Like interactive Python? Or did you create a custom CLI to your poke at your infra?
It is a Rails specific thing. You can read more about it in the rails guides if you are interested.