DEV Community: Kolide

A Rails Multi-Tenant Strategy That's ~30 Lines and "Just Works"

Jason Meller — Tue, 09 Nov 2021 15:33:14 +0000

When engineering a new SaaS app, how you plan to handle customer data tenancy is usually one of the first decisions you and your team will need to make. If you are writing a Rails app and decide on a multi-tenant strategy (one instance of the app serves many customers), then this article is for you.

I contend that modern Rails has everything you need to build a multi-tenant strategy that scales, is easy for others to use, and can be written in just a handful of lines of code. Kolide (btw we're hiring) has been using this simple strategy since the inception of its product, and it's been one of the most elegant and easiest to understand parts of our code-base.

So before reaching for a gem, consider if the following meets your needs.

The Code

The entire implementation is contained in just two files and requires no additional dependencies.

# app/models/concerns/account_ownable.rb

module AccountOwnable
  extend ActiveSupport::Concern

  included do
    # Account is actually not optional, but we not do want
    # to generate a SELECT query to verify the account is
    # there every time. We get this protection for free
    # because of the `Current.account_or_raise!`     
    # and also through FK constraints.
    belongs_to :account, optional: true
    default_scope { where(account: Current.account_or_raise!) }
  end
end

# app/models/current.rb
class Current < ActiveSupport::CurrentAttributes
  attribute :account, :user

  resets { Time.zone = nil }

  class MissingCurrentAccount < StandardError; end

  def account_or_raise!
    raise Current::MissingCurrentAccount, "You must set an account with Current.account=" unless account

    account
  end

  def user=(user)
    super
    self.account      = user.account
    Time.zone         = user.time_zone
  end
end

That's it.

Using the Code In Practice

To use this code, simply mix-in the concern into any standard ActiveRecord model like so...

class ApiKey < ApplicationRecord
  # assumes table has a column named `account_id`
  include AccountOwnable
end

When a user of ours signs in, all we need to do is simply set Current.user in our authentication controller concern which is mixed into our ApplicationController

# app/controllers/concerns/require_authentication.rb
module RequireAuthentication
  extend ActiveSupport::Concern

  included do
    before_action :ensure_authenticated_user
  end

  def ensure_authenticated_user
    if (user = User.find_by_valid_session(session))
      Current.user = user
    else
      redirect_to signin_path
    end
  end
end

For this small amount of effort we now get the following benefits:

Because of the default_scope, once a user is signed in, data from sensitive models is automatically scoped to their account. We just don't need to think about it, no matter how complicated our query chaining gets.
Again, because of the default_scope creating new records for these AccountOwnable models will automatically set the account_id for us. One less thing to think about.
In situations where we are outside of the standard Rails request/response paradigm (ex: in an ActiveJob) any AccountOwnable models will raise if Current.account is not set. This forces us to constantly think about how we are scoping data for customer needs.
The situations where we need to enumerate through more than one tenant's data at a time are still possible but now require a Model.unscoped which can be easily scanned for in linters requiring engineers to justify their rationale on a per use-case basis.

One thing that became slightly annoying was constantly setting Current.account = in the Rails console. To make that much easier we wrote a simple console command.

# lib/kolide/console.rb
module App
  module Console
    def t(id)
      Current.account = Account.find(id)
      puts "Current account switched to #{Current.account.name} (#{Current.account.id})"
    end
  end
end

# in config/application.rb
console do
  require 'kolide/console'
  Rails::ConsoleMethods.send :include, Kolide::Console
  TOPLEVEL_BINDING.eval('self').extend Kolide::Console # PRY
end

Now we simply run t 1 when we want to switch the tenant with an id of 1. Much better.

In the test suite, you should also reset Current before each spec/test as it's not done for you automatically. For us that was simply a matter of adding...

# spec/spec_helper.rb
config.before(:each) do
  Current.reset
end

Now we don't have to worry about our global state being polluted when running our specs serially in the same process.

Concerns we had that didn't end up being an issue

Kolide has been successfully using this strategy since the inception of our Ruby on Rails SaaS app. While we arrived at this strategy in the first few days of our app's formation, we definitely were less confident in the approach. Here is a list of concerns we held and how they ended up panning out.

Will this approach be acceptable to our customers?

Kolide is a device security company, and since our buyers are likely to be either security engineers or security minded IT staff, the bar we need to meet is much higher than the normal SaaS company. We were nervous that an app-enforced constraint would feel flimsy, despite how well it works in practice.

In reality, we found the opposite. Customers were mostly ambivalent about our app-enforced constraint approach. Why? It's because it's an approach that's common among their other vendors and matches their pre-conceived expectations about how most SaaS software works. Matched expectations = less concern.

In prior iterations of our app where we did extreme things like spin up separate Kubernetes namespaces and DBs for each customer, we found our efforts were paradoxically met with more concern, not less. This concern manifested as additional process, review, and ultimately unnecessary friction as our buyers grappled to bring more and more technical folks into the procurement process to simply understand the unfamiliar architecture.

With our current approach, our development and deployment story is simpler, and simplicity has significant security advantages.

Current.rb is too magical, will multi-threading in production cause someone's `default_scope` to leak to another request?

There is a lot of consternation in the Rails community when DHH introduced the CurrentAttributes paradigm in Rails 5.2. DHH talks about his rationale for adding this in his Youtube video entitled, "Using globals when the price is right".

Ryan Bigg on the other-hand felt this addition to Rails would cause developers to write a lot of code with unpredictable behavior expressed these views in his blog post entitled, "Rails' CurrentAttributes considered harmful"

After reading more into the original PR...

ActiveSupport::CurrentAttributes provides a thread-isolated attributes singleton #29180

dhh posted on May 22, 2017

Abstract super class that provides a thread-isolated attributes singleton. Primary use case is keeping all the per-request attributes easily available to the whole system.

The following full app-like example demonstrates how to use a Current class to facilitate easy access to the global, per-request attributes without passing them deeply around everywhere:

# app/models/current.rb
class Current < ActiveSupport::CurrentAttributes
  attribute :account, :user
  attribute :request_id, :user_agent, :ip_address 
  
  resets { Time.zone = nil }
  
  def user=(user)
    super
    self.account = user.account
    Time.zone = user.time_zone
  end
end

# app/controllers/concerns/authentication.rb
module Authentication
  extend ActiveSupport::Concern

  included do
    before_action :set_current_authenticated_user
  end

  private
    def set_current_authenticated_user
      Current.user = User.find(cookies.signed[:user_id])
    end
end

# app/controllers/concerns/set_current_request_details.rb
module SetCurrentRequestDetails
  extend ActiveSupport::Concern

  included do
    before_action do
      Current.request_id = request.uuid
      Current.user_agent = request.user_agent
      Current.ip_address = request.ip
    end
  end
end  

class ApplicationController < ActionController::Base
  include Authentication
  include SetCurrentRequestDetails
end

class MessagesController < ApplicationController
  def create
    Current.account.messages.create(message_params)
  end
end

class Message < ApplicationRecord
  belongs_to :creator, default: -> { Current.user }
  after_create { |message| Event.create(record: message) }
end

class Event < ApplicationRecord
  before_create do
    self.request_id = Current.request_id
    self.user_agent = Current.user_agent
    self.ip_address = Current.ip_address
  end
end

A word of caution: It's easy to overdo a global singleton like Current and tangle your model as a result. Current should only be used for a few, top-level globals, like account, user, and request details. The attributes stuck in Current should be used by more or less all actions on all requests. If you start sticking controller-specific attributes in there, you're going to create a mess.

View on GitHub

I found a lot of thoughtful consideration for how to make this code truly thread-safe which convinced me to bet big on this approach.

Now over two years later, I can say with confidence that in our app, which serves nearly 1,800 HTTP requests per second across hundreds of different tenants, this code works as described in a real production setting.

Will setting `Current.account` in asynchronous jobs or in the test suite become tiresome or problematic?

No, the ceremony here is worth it because it forces us to think carefully about how we are acting on our customer's data. Situations where we need to iterate through more than one customer's data are also trivial to achieve through an #each block.

ApiKey.unscoped.find_each do |api_key|
  Current.account = api_key.account
  api_key.convert_to_new_format
end

Closing Thoughts

We are now sharing this simplistic approach because it's worked so well for us at Kolide. I imagine other burgeoning Rails apps considering a multi-tenant strategy will appreciate being able to do this in their own codebase vs offload something so important to an external gem.

We will continue to share our learnings as Kolide grows. In fact prepping for future scale is what I like best about this approach. By marking all of our records with a tenant identifier like account_id we gain the future option of leveraging more sophisticated solutions at the PostgreSQL level like multi-DB sharding or even products like Citus.

I hope you found this post useful. If you found any errors in this guide or suggestions to improve it, please reach out in the comments or hit me up on twitter @jmeller.

Oh, and we're hiring!

How to Migrate a Rails 6 App From sass-rails to cssbundling-rails

Jason Meller — Thu, 21 Oct 2021 02:59:07 +0000

At Kolide (btw we're hiring), we move swiftly to adopt to new versions of Ruby, Rails, and other major dependencies within a few months of them becoming available. We are at our happiest when we get to use the latest language and framework features. Additionally, forging ahead to uncharted waters allows us to contribute back bug reports, PRs, and guides for other rubyists also interested on being on the bleeding edge.

To that end, I wanted to share our recent experience with upgrading our production Rails 6.1 app from sprockets/sass-rails to the brand-new cssbundling-rails. If you are considering an eventual transition to Rails 7, this is a great first step in that direction.

Why `cssbundling-rails`?

Or, more precisely, why move aways from sass-rails?

When the webpacker gem was released as part of Rails 5.1 release in 2017, DHH made it clear that while webpacker could be used to bundle CSS, he highly recommended to keep it simple and continue to use the Rails asset pipeline.

// Detect dark theme var iframe = document.getElementById('tweet-808349072734027776-654'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=808349072734027776&theme=dark" }

Heeding this advice, many Rails projects (like ours) continue to dutifully serve their SCSS files via sprockets and the sass-rails gem, the same way it's been doing circa Rails 3.1 in 2011.

This sprockets setup has always worked great, but lately some serious bit-rot has set in. Over the last few years, the Sass Team has deprecated both its original ruby-based version of sass, and more recently, the libsass/sassc library in favor of dart-sass. As of this writing, I could not find any sprockets compatible versions of dart-sass. Further, as time marches on, the sassc gem is beginning to accumulate some pretty nasty bugs and inefficiencies. With no fixes on the horizon, it's time to move on.

This is where the newly minted cssbundling-rails comes in. Inspired by the also new jsbundling-rails library, it allows folks to leverage yarn/npm to build a much simpler and more canonical CSS processing pipeline with a setup that will be familiar to JS developers.

rails / cssbundling-rails

Bundle and process CSS in Rails with Tailwind, PostCSS, and Sass via Node.js.

CSS Bundling for Rails

Use Tailwind CSS, Bootstrap, Bulma, PostCSS, or Dart Sass to bundle and process your CSS, then deliver it via the asset pipeline in Rails. This gem provides installers to get you going with the bundler of your choice in a new Rails application, and a convention to use app/assets/builds to hold your bundled output as artifacts that are not checked into source control (the installer adds this directory to .gitignore by default).

You develop using this approach by running the bundler in watch mode in a terminal with yarn build:css --watch (and your Rails server in another, if you're not using something like puma-dev). You can also use ./bin/dev, which will start both the Rails server and the CSS build watcher (along with a JS build watcher, if you're also using jsbundling-rails).

Whenever the bundler detects changes to any…

View on GitHub

With our mission set, let's roll up our sleeves and get started.

Upgrading a Rails 6.x App

According to its gemspec, cssbundling-rails is not just for new Rails 7 apps, it's also compatible with 6.0.

Step 1 - Prepare Your `Gemfile`

Our goal is to not just transition to cssbundling-rails, but to also remove sass-rails gem. To get started, remove sass-rails and any other potential references to sass like sass-ruby and sassc (if defined).

Next, add gem cssbundling-rails, '>= 0.2.4' (the version at the time of this writing) and run bundle install.

Step 2 - Prepare Your SCSS Files

First, let's take some inventory. Open up config/initializers/assets.rb and at the bottom of that file you will see something like the following:



# Precompile additional assets.
# application.js, application.css, and all non-JS/CSS in the app/assets
# folder are already added.
Rails.application.config.assets.precompile += %w( sessions.css staff.css marketing.css )

If the Rails.application.config.assets.precompile line is uncommented, take note of the .css files referenced in the array. In addition to application.css these are all separate top-level CSS files that we will need to convert to the new format.

For each file, (or just application.[s]css) you should do the following:

If you haven't already, convert the files to use SCSS's @import syntax instead of the sprockets magic comments like *= require_self or *= require_tree.
Rename each file to match this format <name>.sass.scss (so application.scss would become application.sass.scss).

Step 3 - Run the Installation (And Then Fix What It Broke)

Run ./bin/rails css:install:sass.

If you receive an overwrite warning for app/assets/stylesheets/application.sass.scss, you can respond with N when prompted.

After the installation completes, we need to clean up a few things.

First, in our case, the installation inserted an extra stylesheet_link_tag at the bottom of app/views/layout/application.html.erb. You should delete this extra line.

Second, while the installation command updates the app/assets/config/manifest.js file with a few new lines, it often doesn't remove explicit references to any sass files. After the upgrade, ours looks like this:



//= link_tree ../images
//= link_tree ../fonts
//= link_tree ../builds

Note: We added the line for fonts as we use the font-url helper in our SCSS files. These fonts didn't need to be explicitly included in the manifest before because sprockets would include them dynamically as they were referenced in the source CSS file. After this upgrade sprockets isn't processing the file so it's important that we ensure it's in the manifest.

Finally, the build:css script the installation creates in package.json is only sufficient if you only have one main application.scss, if you have other files you need to output, you are going to need to modify the script's contents. If this is the case, my suggestion is to create a new file called bin/build-css and do something like the following:



#!/usr/bin/env bash

./node_modules/sass/sass.js \
  ./app/assets/stylesheets/application.sass.scss:./app/assets/builds/application.css \
  ./app/assets/stylesheets/sessions.sass.scss:./app/assets/builds/sessions.css \
  ./app/assets/stylesheets/staff.sass.scss:./app/assets/builds/staff.css \
  ./app/assets/stylesheets/marketing.sass.scss:./app/assets/builds/marketing.css \
  --no-source-map \
  --load-path=node_modules \
  $@

The $@ at the bottom ensures we pass along any additional arguments like --watch when this is invoked via bin/dev (more on that later).

Now in the package.json file do the following:



  "scripts": {
    "build:css": "./bin/build-css"
  }

Don't forget to also run chmod 755 ./bin/build-css in your terminal before moving on to the next step.

Step 5 - Handle `asset-url` And Friends

Often in Rails, you need to reference files from the app/assets/images or app/asset/fonts folders directly in CSS. Since sprockets computes hashes for each asset, you can't just hard-code the name of the asset in there. To work around this, sprockets introduced helper functions like asset-url, font-url, and image-url that resolve the relative path to the asset correctly.

dart-sass has no knowledge of sprockets, so before we finalize the build we need sprockets to run through each file quickly and add these assets paths in. While official support for this is pending, we arrived at a workaround that seems to do the trick:



# config/initializers/asset_url_processor.rb

class AssetUrlProcessor
  def self.call(input)
    context = input[:environment].context_class.new(input)
    data = input[:data].gsub(/(\w*)-url\(\s*["']?(?!(?:\#|data|http))([^"'\s)]+)\s*["']?\)/) do |_match|
      "url(#{context.asset_path($2, type: $1)})"
    end
    {data: data}
  end
end

Sprockets.register_postprocessor "text/css", AssetUrlProcessor

This regex will match these url functions and convert their contents to the appropriate location of the asset on disk in both development and production.

Step 6 - Test It Out With `bin/dev`

Before trying this out, you'll likely want to clear out any sprockets cache in tmp/. To do that, you can simply run bin/rake tmp:clear.

As part of the earlier ./bin/rails css:install:sass command, a new file called bin/dev was created. Additionally, a new gem dependency called foreman and its associated config file Procfile.dev was installed.

By running bin/dev you are now invoking foreman which will read the Profile and simultaneously run the rails server and the yarn build:css --watch commands. This should give you a very similar development experience to the original setup where you can make changes to CSS files and, after a refresh, those changes will be immediately reflected in the browser.

If all went well, bin/dev should start right up and a visit to your app locally should "just work."

Credits & Closing Thoughts

A big thank you to Alex Jarvis for leading the charge on this upgrade at Kolide and collaborating with me.

I hope you found this guide useful. If you found any errors in this guide or suggestions to improve it, please reach out in the comments or hit me up on twitter @jmeller.

Oh, and we're hiring!

Three Myths about Honest Security

Antigoni Sinanis — Tue, 08 Dec 2020 22:20:16 +0000

Today at Kolide, we published our guide to Honest Security. It’s our North Star for Kolide and represents our vision of the future for the endpoint security and device management.

Honest Security focuses on the following five tenets:

The values your organization stands behind should be well-represented in your security program.
A positive working relationship between the end-user and the security team is incredibly valuable and worth fostering.
This relationship is built on a foundation of trust that is demonstrated through informed consent and transparency.
The security team should anticipate and expect that end-users use their company owned devices for personal activities and design their detection capabilities with this in mind.
End-users are capable of making rational and informed decisions about security risks when educated and honestly motivated.

Listen to the podcast about Honest Security at Hacker Valley Studio.

While much of the free guide we posted focuses on what Honest Security is and how it should work mechanically, I imagine there are many IT and security practitioners out there that might be automatically shut off to something like this due to previous bad experiences.

After chatting with a few folks who have read the tenets, I’ve noticed there are already some common misconceptions forming about the Honest Security approach. As we launch Honest Security, I thought I would author a supplementary post that dispels some of the myths behind the methodology.

Myth #1: Honest Security is incompatible with Device Management

Many folks believe Honest Security advocates for an approach that is fundamentally incompatible with device management like MDM. You could hardly be blamed if you do; I have been advocating strongly against blanket device management since the inception of Kolide. Since 2019, our thinking has evolved.

In the chapter six Achieving Compliance Objectives, we acknowledge that education isn’t enough to move the needle towards acceptable levels of adherence to the company’s objectives. To that end, we suggest generating predictable and proportionate consequences that can be applied to end-users who are not heeding the important recommendations of the security team. We go on to describe the concept of Opt-in Management.

While this process is effective, there are just some people who will continually find themselves always on the brink of the consequence activating (or worse, serial offenders). In some situations, these users may do much better with the recommendations they regularly fail to implement on time if the security team could just do it for them. This is where Honest Security can allow the users to opt-in to traditional device management solutions (where applicable) and not have to worry about getting locked out of critical services or accounts.

We feel so strongly about this, we are working on an Honest MDM solution for our customers. The MVP is almost complete!

Myth #2: Honest Security blinds security teams in the name of privacy

Critics of Honest Security approach often get defensive about its strong push away from blanket collection of data that might be useful later. Many folks I deeply respect remind me that if it were not for a certain piece of data they weren’t sure they should gather, they would have never been able to detect <insert really bad thing>.

While Honest Security definitely advocates for being mindful and intentional about the data you collect, it primarily champions for an informed-consent and transparency based approach. If the security team really does think a piece of data is important, then we should be up-front about it. If that data could be dangerous or extremely personal (like GPS coordinates), then informed-consent is the best option.

The best summary of his position is in Chapter 4. Collecting Data Honestly.

Do you know if your organization looks at your web browser history?

I want to clarify that I am not asking them whether or not the company collects web browsing history. I am asking them whether they know, with 100% certainty, whether the security team is or is not. It is one thing to know whether your organization can view your browsing history, and another to know if they do. Also notice I didn’t ask them if the organization usually looks. One person, looking once because they were curious, is looking.

After making these clarifications, it is my experience that there are three camps of people who can still emphatically answer “yes.”

[…]

The third camp are the folks who can answer “yes”, because they know exactly what tools are installed on their devices and what the tools are capable of collecting. More importantly, they know they can independently verify how the security and IT team is using these tools in practice. They know if the security team is looking at their web browser history because the tools the security team uses require them to know. These are people who work for companies that practice Honest Security.

Myth #3: Honest Security hurts Insider Threat detection and deterrence

Many folks think the quality of their insider threat defense strategy is inextricably tied to the act of obfuscating exactly how the endpoint security team performs its detection mission. We address this myth directly in the fourth chapter, Collecting Data Honestly. In there you will find a section aptly titled, “The Insider Threat”. (reproduced below)

The most common argument I see against transparency is that it gives bad-actors within your organization an advantage. The rationale is that an insider threat might be able to identify gaps in the security team’s detection capabilities and systematically abuse them to complete their mission. I disagree. As we all know security through obscurity rarely works. Also, it’s much more likely that this transparency and regular contact will instead serve as a deterrent. Unlike end-users who are making unforced errors, malicious insiders are afraid of being caught. The more interactions they have with a team practicing Honest Security, the more uncomfortable they will get.

Part of Honest Security is trusting end-users because they are our colleagues. If you build a dystopian and cynical security program born out of fear, mistrust, and suspicion, then you will inevitably make your fellow-employees your enemies. The positive working relationship we are advocating for in this guide cannot exist under such a program. Only you can judge if that trade-off is ultimately worthwhile.

I hope this short post cleared up some of the myths we’ve seen in the early hours of promoting this honest approach to security. While we’ve thought very carefully about what we want to include in our guide, it’s a document we expect to evolve based on the feedback, opinions, and most importantly, the experiences of security/IT professionals all over the world.

The Honest Security Guide can be found at https://honest.security.

-Jason Meller, Co-Founder and CEO of Kolide, Inc.