DEV Community: Adrien.S

Custom translations per tenant in a Rails app thanks to I18n backends

Adrien.S — Mon, 28 Apr 2025 14:02:22 +0000

When building a SaaS or any multi-tenant application, it is common to want to adapt certain translation keys for certain customers, in order to provide a truly customized experience. In this article, we'll see a very powerful yet easy way to achieve it thanks to I18n backends in any Rails app.

The i18n lib in ruby works with "backends". Given a specific key and locale, they are tasked with retrieving the correct translation key. The default backend is I18n::Backend::Simple, which looks at the translations in YML files stored in config/locales, but it is possible to create custom backends and even chain them to find translations in different places with a sense of priority.

Identifying the current customer / tenant

Rails provide us with the great Current class that allows us to store variables scoped to the current HTTP request. If your application is already multi-tenant capable, you probably already have something similar :

# app/models/current.rb
class Current < ActiveSupport::CurrentAttributes
  attribute :tenant
  attribute :user
end

# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
  before_action :set_current_tenant

  # ...
  def set_current_tenant
    Current.tenant = current_user.tenant
  end
end

Solution 1: store custom keys in the YML files directly

The simplest approach is to store the custom translations along the others in the yaml, but with a prefix. For example, we can imagine using tenant_%{id} as a prefix, so when looking for the key en.dashboard.index.title we will first try to look into en.tenant_%{id}.dashboard.index.title

We can achieve this by extending the default I18n backend so that it will first look for the translation in the custom scope, and return it if found. This can be done by overloading the translate method :

# config/initializers/i18n.rb
Rails.application.to_prepare do
  class I18nTenantPrefix < I18n::Backend::Simple
    def translate(locale, key, options = {})
      return super unless Current.tenant.present?

      tenant_key = "#{Current.tenant.id}_#{key}"
      return super unless exists?(locale, tenant_key, options)

      super(locale, tenant_key, options)
    end
  end
  I18n.backend = I18nTenantPrefix.new
end

This is a quick and efficient way to customize keys for our different tenants. To manage the translations in the YML file, we can use a tool like YAMLFish to allow anyone in the team to work on those translations.

Solution 2: store custom keys in the database

If we want the custom keys to be instantly available after update, without needing to redeploy the app, we can store them in the database. We can achieve this thanks to a special I18n backend that we can chain with the default one.

Writing an i18n backend means implementing a class that inherits I18n::Backend::Base and implements missing methods to store translations from arbitrary data (usually in memory) and translate a given key. This can be tedious but thankfully the i18n lib provide us with a backend named I18n::Backend::KeyValue which accepts a store class for which we only have to define 2 methods :

Store#[key] that should return the translation for the given key
Store#keys that should return all possible keys that the backend can handle

Then we can chain the default simple backend with our custom one, to provide a fallback mechanism :

# config/initializers/i18n.rb
Rails.application.config.to_prepare do
  class CurrentTenantTranslationsStore
    def [](key)
      Current.tenant&.custom_keys&.dig(key)&.to_json
    end

    def keys
      Current.tenant&.custom_keys&.keys || []
    end
  end

  I18n.backend = I18n::Backend::Chain.new(
    I18n::Backend::KeyValue.new(CurrentTenantTranslationsStore.new),
    I18n::Backend::Simple.new
  )
end

This provides a very easy way to customize translations for each tenant.

Conclusion

I18n backends are a powerful way to "hack" into the way translations are fetched, allowing, in our case, to provide custom translations per tenant.

Launching YAMLFish, a simple translations management tool

Adrien.S — Mon, 16 Dec 2024 19:51:58 +0000

As a software engineer, I've always dreaded working with translations. It can get increasingly tricky with the number of locales and parallel features being developed.

As a CTO of a small company, when updating the YML files manually didn't cut it anymore, I looked for a simple tool that would help me manage the translations, that would not cost a fortune.

I did not find any that suited my needs, so, you guessed it, I decided to build one (with Rails and Turbo, obviously!).

It's been in use for a bit in this small company, even after I left.

I figured, let's share it with the world and see if others find it useful.

Introducing YAMLFish !

Features

Push and pull YML translations with a CLI tool (framework agnostic)
Manage translations in a simple web interface
Branching support
Deepl integration for automatic translations (BYO api key)

That's it ? ... 😅

Less is more

YAMLFish is a simple tool, written by a single developer, designed to be as simple and agnostic as possible.
No github integration, no massive AI features. Just a simple web interface and CLI tool to manage your translations.

Pricing

One of the initial motivation for building YAMLFish was the absence of inexpensive solutions.
It's still a very early tool, and I'm not charging anything for it until interest is clearly manifested (if any!).

If and when I end up charging, the idea is to have a generous free tier and a cheap paid tier that scales with the number of translations.

How to use it

I've made a series of posts on how to use YAMLFish, so I encourage you to check out the series for tips and best practices.

Conclusion

I'm very happy to share YAMLFish with the world, and I hope some of you will find it useful.
If you have any feedback, please feel free to reach out to me on adrien@yamlfish.dev.

Avoid constantize in Rails

Adrien.S — Tue, 26 Nov 2024 20:10:23 +0000

What do you do when you need to instanciate a class in Rails, but its name is dynamic?

Let's say we want to serialize an object for use in an API payload and we have to handle different versions.

We could have a MySerializer::V1 and MySerializer::V2 classes.

And to instanciate the correct one, we could do something like this:

serializer_class = "MySerializer::V#{version}".constantize
serializer = serializer_class.new(object)

It works, but we should really avoid it.

It could be unsafe

If the string that is constantized ends up containing user-controlled data, we could be in trouble.

Let's say you are allowing the user to create different record types based on a type attribute.

params[:record_type].constantize.create!(...)

Your front-end could only send types like Article or Category, but a malicious user could send something like AdminUser and create themself an admin account.

This is highly unlikely, but it's still a risk.

It has bad developer experience

It's hard to read and hard to follow
You can't use go to definition, forcing you to open the file manually
When searching for a class name, you won't find the lines with constantize, giving you a false sense of security

How to do instead ?

You could have a whitelist of classes you allow to be instanciated, but this doesn't solve all problems.

What I like to do is use a hash to map some value with the class you want to call, for instance :

SERIALIZER_CLASSES = {
  "v1" => Myserializer::V1,
  "v2" => Myserializer::V2
}

def serializer_for(version)
  SERIALIZER_CLASSES[version] || raise "Unknown serializer for version #{version}"
end

serializer_for("v2").new(...)

I like it because it's explicit, easy to read, and allows you to use go to definition or find the line when you search for a class name.

Automation and branches- Using YAMLFish to easily manage I18n translations in your project

Adrien.S — Fri, 22 Nov 2024 18:53:32 +0000

In the last article, we've discovered the basics of YAMLFish, the simple and easy translation management tool.

In this article, we'll learn how to integrate YAMLFish into our development processes and how to use it to have keys translated by a non-developer person, and get the keys back into our codebase thanks to branches !

Main locale ?

Usually, in software projects you have a "main" locale, this is the one features are developed in, the keys for this locale are usually handled by the tech/product people.
Then, you have secondary locales, which are handled by translators.

For this article we'll assume our main locale is en🇬🇧 and our secondary locales are fr🇫🇷 and es🇪🇸

Automatically push keys to YAMLFish

For our en🇬🇧 locale, our source of truth is the codebase.
So we want our en keys to always be in sync.

We can easily accomplish that by automatically pushing en keys to YAMLFish when building our app. This can be added to any CI/CD platform in a breeze.

Here's the example for Github Actions :

 - name: Push translations to yamlfish
   if: github.ref == 'refs/heads/master'
   run: gem install yamlfish --pre && yamlfish push en

Now, we are sure that each time we push to master, our keys are sent to YAMLFish.

Working on a new feature that needs translations

Let's start working on a new feature. We'll open a new branch and push our code to it.
This feature introduces new translation keys and change a few existing ones, we need to ask for the fr and es translations.

This is where we can use YAMLFish branches. After creating a branch on the yamlfish dashboard, we can push our updated translations to it using :

yamlfish push en --branch my-awesome-feature

Sending the branch for translation

We can then ask our translators to fill in the missing translations.

From the YAMLFish dashboard, they will be able to see translation keys specific to the branch, and update the value for their locale.

Fetching the new translations

Once the translators have finished filling the keys, we can now move forward.

The first thing we need to do is merge our branch, so that our CI/CD pipeline will run and push the new translations for our en🇬🇧 locale.

Once they are pushed, we can go on the YAMLFish dashboard and merge our branch :

We only want to merge our secondary locales as our main locale has been updated already.

Once this is done, we can simply pull our new translations :

yamlfish pull fr
yamlfish pull es

This will update our fr.yml and es.yml files (in config/locales by default) with the new translations.

We can now commit the changes, and deploy our new feature !

Conclusion

Thanks to YAMLFish we were easily able to have our new feature translated by non-dev people in a frictionless way, from the comfort of our CLI.

In the next article, we'll see how to manage cases when non-dev people want to update translations in the main locale.

The basics - Using YAMLFish to easily manage I18n translations in your project

Adrien.S — Sat, 16 Nov 2024 14:03:21 +0000

Translations management is never easy, without the proper tools, it can become tedious to work on translation keys, especially when involving non technical people in the process. Existing tools are very pricey and often very complex, making developers dread working on translations.

YAMLFish aims to help software teams managing their translations in a simple and cheap way.

In this series we'll learn how to use YAMLfish and how to implement basic use-cases. In this first article, we'll learn about the basics of the push and pull commands.

Setting-up

First, let's create a free account and create our first project and locales. For this demonstration, we'll assume our main locale is en and we want to translate our app in fr.

Next, let's install the YAMLFish CLI and add the configuration file.

The CLI is a ruby gem, so you'll need ruby installed first.

gem install --pre yamlfish

We need to save a YAMLFish config file .yamlfish.yml at the root of our project. The project token and API key can be found in the footer of the project page on the YAMLFish dashboard :

project_token: some-token
api_key: some-api-key

By default, YAMLFish will look for translations in config/locales, you can override this with the locales_path config key (see the docs).

Pushing existing translations

Pushing our existing translations is as simple as

yamlfish push en
yamlfish push fr

And our keys are now available on YAMLFish

Retrieving updated translations

Let's assume the fr keys have been updated on YAMLFish by a translator, it's now time to retrieve them into our codebase.

yamlfish pull fr

This will download all fr keys and store them into config/locales/fr.yml

Conclusion

We've learned how to push and pull keys from YAMLFish, allowing non developers to update our translation keys.

In the next articles, we'll learn about how to integrate YAMLFish into the day-to-day of our project, and avoid any conflicts, thanks to branches.

Generate magic tokens in Rails with generates_token_for

Adrien.S — Sun, 28 Apr 2024 16:43:56 +0000

For a long time, and probably still today, the reference for authentication in Rails is using a gem like Devise.

Thing is, you'll probably end up customizing it a lot: views, emails, onboarding flow, etc.
Since Rails 7.1, we have access to several new features that make it easier to implement authentication with minimal extra code, making it a viable option for many projects.

One of these features is generates_token_for, which allows you to generate non-persisted tokens for your models, allowing you to implement features such as passwordless auth, password reset, email confirmation, and more.

When I stumbled upon this feature, my first thought was: That's magic.

In this post, We'll see how to use generates_token_for to generate magic tokens in Rails, then we'll dive into the code to understand how it works.

How to use `generates_token_for`

Here's a basic example of how to use generates_token_for in your Rails models:

class User < ApplicationRecord
  generates_token_for :account_activation
end

user = User.find(42)
token = user.generate_token_for(:account_activation) # => "sometoken===--somesignature"

User.find_by_token_for(:account_activation, token) # => #<User id: 42, ...>

Once we declare that we want to generate a token for :account_activation, we can call generate_token_for to generate a token and find_by_token_for to find a user from a given token.

This token is not persisted anywhere, it just contains the user id and a signature to verify its authenticity, making it a very convenient way to implement features that require a token.

Expiration

Token expiration is also supported, you can pass a expires_in option to generates_token_for to set the expiration time :

class User < ApplicationRecord
  generates_token_for :account_activation, expires_in: 1.day
end

If you try to find a user by an expired token, it will return nil.

Invalidating the token when something changes

generates_token_for supports making the token dependant on an arbitrary block of code, allowing to implement features like password reset tokens that are invalidated when the password changes:

class User < ApplicationRecord
  generates_token_for :password_reset, expires_in: 1.day do
    password_salt&.last(10)
  end
end

In this example, the token is dependent on the last 10 characters of the password salt.
The generated token will contain the content of the block, this is why it should be deterministic and not contain any sensitive information, and why we use the last 10 characters of the password salt in this example instead of the password hash directly.

When trying to find a user by a token, the block will be called again and compared to the value in the token, if they don't match, the token is considered invalid.

This is very powerful and allows to implement complex token invalidation logic with minimal code, you can make tokens dependent on values, state, timestamps, etc.

How it works

Let's have a look at the code to understand how generates_token_for works. Here's a portion of the ActiveRecord::TokenFor module that is included in our active record classes :

# activerecord/lib/active_record/token_for.rb

included do
  class_attribute :token_definitions, instance_accessor: false, instance_predicate: false, default: {}
end

# ...

def generates_token_for(purpose, expires_in: nil, &block)
  self.token_definitions = token_definitions.merge(purpose => TokenDefinition.new(self, purpose, expires_in, block))
end

We see that a token_definitions hash is defined, and that generates_token_for is just a method that adds a TokenDefinition to it.

The TokenDefinition is a class defined in the same file, through a Struct :

# activerecord/lib/active_record/token_for.rb

TokenDefinition = Struct.new(:defining_class, :purpose, :expires_in, :block) do # :nodoc:
  # Some methods we'll see right after
end

It's basically a class that accepts params such as defining_class purpose, expires_in and block, with some methods used to do the token generation and verification.

Before diving in, let's have a quick look at the generate_token_for method used on an instance of a model :

# activerecord/lib/active_record/token_for.rb

def generate_token_for(purpose)
  self.class.token_definitions.fetch(purpose).generate_token(self)
end

Pretty straight forward, we're looking for the token definition for the good purpose, and calling generate_token on it.

# activerecord/lib/active_record/token_for.rb

def full_purpose
  @full_purpose ||= [defining_class.name, purpose, expires_in].join("\n")
end

def payload_for(model)
  block ? [model.id, model.instance_eval(&block).as_json] : [model.id]
end

def generate_token(model)
  message_verifier.generate(payload_for(model), expires_in: expires_in, purpose: full_purpose)
end

We use the rails MessageVerifier, that can generate and verify signed messages, based on a secret key. It's also used in other features such as CSRF token validation.

We'll use it to generate our token, based on a payload consisting either only of the model id if no block was passed, or the id along the content of the block if one was passed.

The generated string will look like this :

eyJfcmFpbHMiOnsiZGF0YSI6WzQyXSwicHVyIjoiVXNlclxuc2Vzc2lvblxuIn19--7db5fb8690104cec00ec6443353c2362760e7078

The first part is the actual data, in Base64, and the second is the signature.

If we decode the first part, we obtain this :

{"_rails":{"data":[42],"pur":"User\nsession\n"}}

And here is another example for a token using expiration and a block :

{"_rails":{"data":[42,"93LHs7.oVu"],"exp":"2024-04-28T16:44:03.463Z","pur":"User\npassword_reset\n3600"}}

Basically, a token is just a big JSON object containing our purpose, our model id, and optionally the expiration time and arbitrary block value.

Now, the last part to inspect is the find_by_token_for method used on a model class to retrieve a record from a token :

def find_by_token_for(purpose, token)
  raise UnknownPrimaryKey.new(self) unless primary_key
  token_definitions.fetch(purpose).resolve_token(token) { |id| find_by(primary_key => id) }
end

and the resolve_token method on the TokenDefinition:

 def resolve_token(token)
  payload = message_verifier.verified(token, purpose: full_purpose)
  model = yield(payload[0]) if payload
  model if model && payload_for(model) == payload
end

We use the message verifier to decode the token and ensure it was generated by our rails app, this is all done inside the MessageVerifier, this will return nil if the data is invalid, not verified, or expired.

We find the model by its primary key (through yielding its id to the calling method)

Then, we recompute the payload, and compare it with the one we got from the token, if it matches, the model is returned, otherwise nil. Not so magic after all.

Conclusion

As always, reading through the code can help us broader our understanding of a feature, and understand any possible gotchas.

Features such as generates_token_for are great tools to implement strong authentication features in a Rails app, making it possible to drop big dependencies and stay pretty vanilla, with minimal extra code.

The Rails asset pipeline, old and new

Adrien.S — Sun, 28 Apr 2024 16:17:11 +0000

Sprockets, Webpack(er), Importmaps, esbuild, propshaft, rollup, sass, postCSS, tailwind?
It's easy to get lost between all the tools offered by Rails to manage your assets, especially if you're a beginner in web software development or new to the Rails ecosystem.

In this article, we'll take a deep dive into the ways Rails has helped us managing assets, starting with sprockets back in 2012, and ending with the most recent available tools released in 2022. We'll try to understand the reasons between the different changes, and deeply comprehend the ways they work, reading some code along the way.

The Dark Age

Before Rails 3.1 in 2011, there was no official way to help with the management of assets in a Rails app.

You would have to simply put your assets in the public directory and require them directly.

Importing external libraries meant copying and pasting the source code in your public folder.

Back then JavaScript modules did not exist, so all JS code would be global and living in the window namespace, SPAs were not a thing, nor was React or TypeScript. What a blissful time...

Enters Sprockets

Sprockets was first released in 2009 but was integrated into rails in 2011 with Rails 3.1.

Surprisingly, it hasn't changed much since.

Let's look at the release notes from Rails 3.1 :

The asset pipeline provides a framework to concatenate and minify or compress JavaScript and CSS assets. It also adds the ability to write these assets in other languages such as CoffeeScript, Sass and ERB.

Prior to Rails 3.1 these features were added through third-party Ruby libraries such as Jammit and Sprockets. Rails 3.1 is integrated with Sprockets through Action Pack which depends on the sprockets gem, by default.

Making the asset pipeline a core feature of Rails means that all developers can benefit from the power of having their assets pre-processed, compressed and minified by one central library, Sprockets. This is part of Rails’ “fast by default” strategy as outlined by DHH in his keynote at RailsConf 2011.

Sprockets allowed us to integrate third party dependencies much faster (through vendored assets in gems, remember bootstrap-rails ?) and offered us out of the box tools such as minification or the ability to use variables in our front-end code.

Let's have a look at how it works.

All code and behaviour described here comes from the latest available version at the time I wrote this article, which is v4.2.0

General behaviour

Assets go into app/assets/javascripts and app/assets/stylesheets respectively.

(Sprockets also supports app/assets/images and app/assets/fonts but we won't look into them in this article)

To include them on our web pages, we need to use the javascript_include_tag and stylesheet_link_tag helpers.

It is possible to include 3rd party libs or our own files by using special comments in JS files, and classic @import statement in CSS files.

//= require jquery

$().ready({
  // my custom code here
})

If config.assets.compile is set to true, assets will be compiled live when requested. This is perfect for development environments.

On production environment, assets are generally precompiled, meaning they are all generated at one point, copied to the public/assets directory, then forwarded directly when requested. This is done by running the rails assets:precompile command.

To avoid issues with caching, the assets are fingerprinted, meaning a hash of the contents of the file is added to the filename, so if a change happens in the file, a new fingerprint will be generated, ensuring the browser has to request a fresh file it does not have in cache. This happens in all environments, and is handled through a manifest file, which is a simple JSON file mapping the original filename to the fingerprinted one.

This way, when calling javascript_include_tag('application'), the helper will look into the manifest file to find the correct filename, and generate the correct script tag, such as <script src="/assets/application-1234567890.js"></script>.

In the code

`Sprockets::Server`

The most interesting part too look at to me is the Sprockets web server.
It is run when the browser tries to fetch an asset through a HTTP Query and compile mode is enabled. Otherwise the assets will be rendered through the classic static files middleware if they are present in public/assets.

As most web-related ruby things, it is indeed a simple rack middleware.

Its call method is actually defined inside the sprockets "environment" class, which is itself included in the middleware stack through the sprockets-rails gem.

Let's have a look at a (really) stripped-down version of the call method.

def call(env)
  # ...

  # Extract the path from everything after the leading slash
  full_path = Rack::Utils.unescape(env['PATH_INFO'].to_s.sub(/^\//, ''))
  path = full_path

  # ...

  # Strip fingerprint
  if fingerprint = path_fingerprint(path)
    path = path.sub("-#{fingerprint}", '')
  end

  # ...

  # Look up the asset.
  asset = find_asset(path)

  if asset.nil?
    status = :not_found
  # elsif ....
  else
    status = :ok
  end

  case status
  when :ok
    logger.info "#{msg} 200 OK (#{time_elapsed.call}ms)"
    ok_response(asset, env)
  end
  # ...
end

First, we get the path we want through the rack env and some cleaning.
Then, the fingerprint is removed, we don't need it in dev environment as we just want to find the correct file and compile it.

Then, we try to find_asset(path), that will return an Asset object.

Digging into it could be a whole article, so I'll try to describe what is done in my own words :

Possibly return a cached version of the file
Find the correct processor(s) given the file type (sass, babel, etc)
Run the processor(s), if any
Run the compressor, to minify the file
Build an Asset object with the result

One of the processors is DirectiveProcessor, it is used to resolve the magic //= require comments to replace them with the correct file contents.

Note that the same code will be run when using assets:precompile, in order to write the assets to the public/assets directory.

Once the asset is returned, and if everything is ok, ok_response will be called, it's as simple as that :

      # Returns a 200 OK response tuple
      def ok_response(asset, env)
        if head_request?(env)
          [ 200, headers(env, asset, 0), [] ]
        else
          [ 200, headers(env, asset, asset.length), asset ]
        end
      end

It will just pass the asset object to rack.

Rack is expecting an array of strings, let's have a look into the Asset class.

    # Public: Add enumerator to allow `Asset` instances to be used as Rack
    # compatible body objects.
    #
    # block
    #   part - String body chunk
    #
    # Returns nothing.
    def each
      yield to_s
    end

    # Public: Alias for #source.
    #
    # Returns String.
    def to_s
      source
    end

These methods make the asset object compatible with the rack API, rack will call each on it and the block will be yielded once with the full source of the asset, nice !

Webpacker

Around 2015, JavaScript bundlers were getting a lot of traction with the promise of easily using npm packages in the browser, transpiling modern JS, etc.

Webpack quickly became the reference and became inescapable for frontend heavy applications.

If a Rails developer wanted to include a NPM package into their app, they would have to either rely on a gem being built and maintained to deliver the assets through sprockets, or implement a fully-fledged JS bundler stack such as webpack in their app, which was quite tedious for our simple ruby brains.

Webpacker was designed as a way to benefit from webpack inside a rails app without having to know and understand how webpack worked.

It was added as default in rails 6.0, for JS only, sprockets still being the default for CSS; although it is possible to use webpacker for CSS too.

General behaviour

Let's consider the default behaviour, with webpacker only used for JS.

Webpacker will look for JS files in app/javascript, and especially webpack entry files in app/javascript/packs.

To include them on our web pages, we need to use the javascript_pack_tag helper.

This helper will generate a script tag with the correct path to the compiled JS file.

In development,the file will either be compiled directly by the javascript_pack_tag method, straight into the public/packs folder, and then served as any static asset, or proxified on-the fly to a tool named webpack-dev-server which allows for code reloading and other goodies.

In production, the file is served statically from the public/packs directory. It will be generated when we run rails assets:precompile, which will run webpack in production mode under the hood.

Some configuration is possible, through a YML config file and some JS files, the @rails/webpacker package is used to generate the webpack config. I remember it being quite tedious as it was not possible to override the webpack config directly. The web was full of articles, and stackoverflow answers that worked with webpack directly, but adapting them to webpacker was not always easy.

Similarly to sprockets, webpack (or its dev server) will fingerprint the assets, and store the mapping in a manifest file stored in public/packs/manifest.json.

In the code

In production, most of the work is done by webpack directly, webpacker acting as a wrapper and a configuration generator. It's not super interesting to look at.

In development though, since assets are compiled on-demand, or proxied to the webpack dev server, it's a bit more interesting.

Let's first look at the javascript_pack_tag method and follow the rabbit hole.

  def javascript_pack_tag(*names, **options)
    javascript_include_tag(*sources_from_manifest_entries(names, type: :javascript), **options)
  end

  # ...

  def sources_from_manifest_entries(names, type:)
    names.map { |name| current_webpacker_instance.manifest.lookup!(name, type: type) }.flatten
  end
end

Basically, this helper will try to find the correct files from the webpacker manifest, and then call the classic javascript_include_tag method.

Let's look at the lookup! method and its friends.

  def lookup!(name, pack_type = {})
    lookup(name, pack_type) || handle_missing_entry(name, pack_type)
  end

  def lookup(name, pack_type = {})
    compile if compiling?

    find(full_pack_name(name, pack_type[:type]))
  end

  def compiling?
    config.compile? && !dev_server.running?
  end

If the dev server is not running, and the config is set to compile on demand (which is true in development by default), it will run the compilation synchronously.

Then, it tries to find the asset in the manifest, and if it can't find it, it will call handle_missing_entry which will raise an error.

If the dev server is running though, it will skip the compilation step and just try to find the asset in the manifest.

The asset is going to be found because the webpack-dev-server adds it to the manifest when it starts.

We don't have the file compiled yet, but the dev server will compile it on the fly when the browser requests it.

This is done through the Webpacker::DevServerProxy which is a rack middleware that is added by Webpacker.

Let's see its complete code :

require "rack/proxy"

class Webpacker::DevServerProxy < Rack::Proxy
  delegate :config, :dev_server, to: :@webpacker

  def initialize(app = nil, opts = {})
    @webpacker = opts.delete(:webpacker) || Webpacker.instance
    opts[:streaming] = false if Rails.env.test? && !opts.key?(:streaming)
    super
  end

  def perform_request(env)
    if env["PATH_INFO"].start_with?("/#{public_output_uri_path}") && dev_server.running?
      env["HTTP_HOST"] = env["HTTP_X_FORWARDED_HOST"] = dev_server.host
      env["HTTP_X_FORWARDED_SERVER"] = dev_server.host_with_port
      env["HTTP_PORT"] = env["HTTP_X_FORWARDED_PORT"] = dev_server.port.to_s
      env["HTTP_X_FORWARDED_PROTO"] = env["HTTP_X_FORWARDED_SCHEME"] = dev_server.protocol
      unless dev_server.https?
        env["HTTPS"] = env["HTTP_X_FORWARDED_SSL"] = "off"
      end
      env["SCRIPT_NAME"] = ""

      super(env)
    else
      @app.call(env)
    end
  end

  private
    def public_output_uri_path
      config.public_output_path.relative_path_from(config.public_path).to_s + "/"
    end
end

We can see it inherits from Rack::Proxy, which is a rack middleware that allows us to proxy requests to another server.

It will only proxy requests that start with the public_output_uri_path which is by default packs/. And only if the dev server is running.

When this is the case, it will change the request headers to point to the dev server instead of the rails server, and then call super(env) which will call the perform_request method of the parent class, which will proxy the request to the dev server.

Propshaft & friends

Propshaft is a new asset pipeline that was introduced in Rails 7.0, although it is not expected to be the default until Rails 8.0.

Here are the first lines of its README :

Propshaft is an asset pipeline library for Rails. It's built for an era where bundling assets to save on HTTP connections is no longer urgent, where JavaScript and CSS are either compiled by dedicated Node.js bundlers or served directly to the browsers, and where increases in bandwidth have made the need for minification less pressing. These factors allow for a dramatically simpler and faster asset pipeline compared to previous options, like Sprockets.

Propshaft is kind of sprockets stripped down to the bare minimum.
Its philosophy is based on the idea that assets will be bundled by external tools, such as bun, esbuild, rollup, webpack for JS, and postCSS, sass, tailwind standalone for CSS.

Those tools are integrated through the jsbundling-rails and cssbundling-rails gems, which provide simple config and generators, and hooks into the commands we already know, such as rails assets:precompile. In development, they are configured in "watch" mode, so they will recompile the assets when they change.

Propshaft is expected to be the default asset pipeline in Rails 8.0. Note that you can also use jsbundling-rails and cssbundling-rails with Sprockets.

General behaviour

Propshaft will look for files in the app/assets/builds directory and handle them in a similar way to sprockets. However, besides the fingerprinting, no extra processing is done, and the files are served as-is. All the heavy lifting is expected to be done by the external tools (esbuild, postCSS, etc).

In the code

Propshaft is much simpler than sprockets, and it's easier to understand what is going on.

It has a rack middleware as well, which is used to compile and serve the assets in development mode.

Here's its call method :

  def call(env)
    path, digest = extract_path_and_digest(env)

    if (asset = @assembly.load_path.find(path)) && asset.fresh?(digest)
      compiled_content = @assembly.compilers.compile(asset)

      [
        200,
        {
          Rack::CONTENT_LENGTH  => compiled_content.length.to_s,
          Rack::CONTENT_TYPE    => asset.content_type.to_s,
          VARY                  => "Accept-Encoding",
          Rack::ETAG            => asset.digest,
          Rack::CACHE_CONTROL   => "public, max-age=31536000, immutable"
        },
        [ compiled_content ]
      ]
    else
      [ 404, { Rack::CONTENT_TYPE => "text/plain", Rack::CONTENT_LENGTH => "9" }, [ "Not found" ] ]
    end
  end

Nothing too fancy, if the asset is found, it will be compiled and served.

The @assembly object is an instance of Propshaft::Assembly, which is the main object of the gem.

Here's its compilers method :

  def compilers
    @compilers ||=
      Propshaft::Compilers.new(self).tap do |compilers|
        Array(config.compilers).each do |(mime_type, klass)|
          compilers.register mime_type, klass
        end
      end
  end

Propshaft::Compilers is a class that handles a list of registered compilers and allow to compile an asset to be compiled by them when relevant (based on the mime type).

At the time of writing, there are only 2 compilers :

css_asset_urls.rb , to resolve asset urls in CSS files (background images, etc) and replace them with the fingerprinted version.
source_mapping_urls.rb, to do the exact same with source maps. (sourceMappingURL=xxx)

That's the only processing that is done by propshaft, although it is built to support custom compilers.

The compile method of the Propshaft::Compilers class is quite simple :

   def compile(asset)
    if relevant_registrations = registrations[asset.content_type.to_s]
      asset.content.dup.tap do |input|
        relevant_registrations.each do |compiler|
          input.replace compiler.new(assembly).compile(asset.logical_path, input)
        end
      end
    else
      asset.content
    end
  end

It tries to find the correct compilers for the asset, and if it finds one, it will call its compile method, which will return the compiled asset.

The same code path is used in production when running assets:precompile. The task is implemented in lib/railties/assets.rake.

It calls Rails.application.assets.processor.process, which writes the manifest and compiles the assets.

Importmaps

Importmaps are a relatively recent tool to control the way we import JS modules in the browser.
By specifying a mapping between a module name and a URL, we can import modules without having to specify the full path to the file.

For example, if we have an import map that looks like this :

 <script type="importmap">
{
  "imports": {
    "react": "https://ga.jspm.io/npm:react@17.0.1/index.js"
  }
}
</script>

and a JS file that looks like this :

import React from "react"

The browser will know to fetch the module from the URL specified in the import map, allowing us to write code without having to specify the full path to the module.

Combined with some tooling such as importmap-rails, it allows us to use npm packages in the browser without having to use tools like yarn, npm, or even a bundler such as webpack, rollup, or esbuild.

The developer experience is quite nice as managing versions of packages is done through the import map, and the browser will fetch the correct version of the package automatically, without the need to change any code.

Usage

We can add libraries by using the CLI provided by the importmap-rails gem.

bin/importmap pin react
# or with a specific version
bin/importmap pin react@17.0.1

This will :

add the react package (and its dependencies) to the importmap, through the config/importmap.rb file
download the files to the vendor/javascript directory, so they can be served by the asset pipeline

Then, in the layout, we can use the javascript_importmap_tags helper to generate the importmap script tag as well as import the application.js entry point.

<%= javascript_importmap_tags %>

In the code

The importmap-rails gem is quite simple, most of the heavy lifting is done by the browser anyway !

It just adds a few rake tasks to manage the importmap and JS files, and a few helpers to generate the necessary tags in the layout.

First, let's have a look at what happens when you run bin/importmap pin react.

It is implemented as a thor task in lib/importmap/cmmands.rb

  option :env, type: :string, aliases: :e, default: "production"
  option :from, type: :string, aliases: :f, default: "jspm"
  def pin(*packages)
    if imports = packager.import(*packages, env: options[:env], from: options[:from])
      imports.each do |package, url|
        puts %(Pinning "#{package}" to #{packager.vendor_path}/#{package}.js via download from #{url})
        packager.download(package, url)
        pin = packager.vendored_pin_for(package, url)

        if packager.packaged?(package)
          gsub_file("config/importmap.rb", /^pin "#{package}".*$/, pin, verbose: false)
        else
          append_to_file("config/importmap.rb", "#{pin}\n", verbose: false)
        end
      end
    else
      puts "Couldn't find any packages in #{packages.inspect} on #{options[:from]}"
    end
  end

  # ...

  def packager
    @packager ||= Importmap::Packager.new
  end

Looks simple enough, it will call the import method on the packager object, which is an instance of Importmap::Packager.

This returns an hash mapping the package names with their urls, which are then used to generate the importmap entry and download the files.

Let's dig into it.

We'll start with the import method for Importmap::Packager.

  class Importmap::Packager
    # ...
    self.endpoint = URI("https://api.jspm.io/generate")
    # ...

    def import(*packages, env: "production", from: "jspm")
      response = post_json({
        "install"      => Array(packages),
        "flattenScope" => true,
        "env"          => [ "browser", "module", env ],
        "provider"     => from.to_s,
      })

      case response.code
      when "200"        then extract_parsed_imports(response)
      when "404", "401" then nil
      else                   handle_failure_response(response)
      end
    end

    # ...

    def post_json(body)
      Net::HTTP.post(self.class.endpoint, body.to_json, "Content-Type" => "application/json")
    rescue => error
      raise HTTPError, "Unexpected transport error (#{error.class}: #{error.message})"
    end

    def extract_parsed_imports(response)
      JSON.parse(response.body).dig("map", "imports")
    end

end

importmap-rails uses the jspm.io Generator API to get the list of packages to be imported. Interestingly, it can return an url on jspm (by default), but also on jsdelivr or unpkg.

It will return something like this :

{
    "dynamicDeps": [],
    "map": {
        "imports": {
            "react": "https://ga.jspm.io/npm:react@18.2.0/index.js"
        }
    },
    "staticDeps": [
        "https://ga.jspm.io/npm:react@18.2.0/index.js"
    ]
}

And importmap-rails will fetch the imports key from the map object in extract_parsed_imports.

Going back to the pin method, we can see that it will then call download on the packager object.

  def download(package, url)
    ensure_vendor_directory_exists
    remove_existing_package_file(package)
    download_package_file(package, url)
  end

  # ...

  def download_package_file(package, url)
    response = Net::HTTP.get_response(URI(url))

    if response.code == "200"
      save_vendored_package(package, response.body)
    else
      handle_failure_response(response)
    end
  end

Gotta love those explicit method names !
This is pretty self explanatory, it will download the file from the url and save it in the vendor/javascript directory.

Finally, the pin method will try to update the config/importmap.rb file with the new entry, either update the existing line if it exists (packager.packaged?(package)), or append it to the end of the file.

javascript_importmap_tags handles most of the rest, it will generate the importmap script tag, which is generated from the config/importmap.rb file which is a sort of glorified ruby hash.

This method will also generate tags to preload module files (this is the default now), and to import the application.js entry point.

  def javascript_importmap_tags(entry_point = "application", importmap: Rails.application.importmap)
    safe_join [
      javascript_inline_importmap_tag(importmap.to_json(resolver: self)),
      javascript_importmap_module_preload_tags(importmap),
      javascript_import_module_tag(entry_point)
    ], "\n"
  end

Conclusion

Along the years, Rails has evolved to adapt to the ever-changing landscape of web development, and the asset pipeline is no exception.

It has gone through many hoops, and it's interesting to see how it has evolved, and how it has been simplified over time. I really like the direction it's taking with propshaft and the bundling gems, that I use professionally in production with great success.

Importmaps look very promising, and are actually the default JS experience if you run rails new, but I feel the developer experience is not quite there yet, especially when it comes to managing versions of packages (no dependabot support, no easy way to update everything).

I remember fighting many times with the asset pipeline along the years. Taking time to understand how it works and reading some code helped me solve many issues, and I hope this article will help you too !

Using minio to mock S3 in rails test and development

Adrien.S — Tue, 03 Nov 2020 08:54:02 +0000

I recently stumbled upon minio when looking at the shrine doc while setting up tests related to file uploads.

One could say minio is like a self-hosted S3 object storage. It can be used on production systems as an amazon S3 (or other) alternative to store objects.

One other interesting aspect is to use it development and test environments when you already use a cloud provider for production. This allows you to test end-to-end file operations extensively without the need to mock some operations or network queries.

In my case, I use Scaleway object storage (S3 compatible), with the shrine gem, it looks like this :

s3_options = {
  bucket: ENV['S3_BUCKET'],
  access_key_id: ENV['S3_KEY_ID'],
  secret_access_key: ENV['S3_SECRET_KEY'],
  region: ENV['S3_REGION'],
  endpoint: ENV['S3_ENDPOINT'],
  force_path_style: true # This will be important for minio to work
}

Shrine.storages = {
  cache: Shrine::Storage::S3.new(prefix: "cache", **s3_options),
  store: Shrine::Storage::S3.new(prefix: "invoices", **s3_options)
}

Installing Minio

I use docker-compose for my database and redis, adding minio was a breeze :

Adding a dedicated volume :

volumes:
  pg:
  redis:
  minio:

Then adding the service :

  minio:
    restart: always
    image: minio/minio:latest
    ports:
      - "9000:9000"
    volumes:
      - minio:/data
    entrypoint: minio server /data

If you're not using docker, you can find instructions for installing minio locally in the docs.

There we go, we can browse to http://localhost:9000/ to make sure our minio instance is running, and create our bucket.

Integration in dev

In the development environment, all I had to do was to update my env variables for minio. If you keep the default credentials, it will look like this :

S3_KEY_ID=minioadmin
S3_SECRET_KEY=minioadmin
S3_BUCKET=bucket-name
S3_ENDPOINT=http://localhost:9000

If you use direct upload, you may need to tweak the javascript code that catches the path of the uploaded object, to make sure it works with both minio and whatever cloud provider you use, as the addresses may be formatted differently.

Integration in test

Same goes about the environment variables in test. You will also need to create a bucket for your test env.

This works but every time a new developer will need to set-up their system they will have to do this task, and if you use a CI it will be tedious. Moreover, each time you will perform an upload in your tests, your storage will grow.

To avoid any issues, we can programatically create and delete the storages before and after the test suite.

Here it is with shrine, but you can adapt this code to use your favorite adapter instead :

if ENV['S3_ENDPOINT'].include?('localhost')
  RSpec.configure do |config|
    config.before(:all) do
      Shrine.storages[:store].bucket.create unless Shrine.storages[:store].bucket.exists?
    end
    config.after(:all) do
      Shrine.storages[:cache].clear!
      Shrine.storages[:store].clear!
    end
  end
else
  puts("It doesn't seem like S3 is mocked in test, skipping auto clearing of bucket")
end

As you can see, I'm a bit paranoid of clearing the wrong bucket so I added a guard to make sure we are only cleaning something containing localhost.

Thanks for reading !

Your javascript can reveal your secrets

Adrien.S — Wed, 28 Oct 2020 13:51:00 +0000

Security is hard. It’s often very easy to overlook things, and one small mistake can have a very big impact.

When writing JavaScript, it’s easy to forget that you’re writing code that will be sent in plain text to your users.

Recently I have been doing a bit of offensive security, with a special interest on JavaScript files, to see what kind of information could be retrieved from them.

Here’s what I’ve learned.

Business logic and other business leaks

It’s not uncommon to see some business logic in JavaScript files, especially for frontend-heavy websites.

While this is not a direct security problem, it can tell a great deal about your internals.

It could be a secret pricing function, a list of states that reveal an upcoming feature, or an array of translation strings that uncover some internal tools.

You wouldn’t want your secret algorithms exposed to the face of the world, would you?

Internal API paths

Another interesting find in JavaScript files is API paths.

Frontend-heavy applications need to make calls to an internal API, and often the list of API endpoints is conveniently stored in an Object in one of the JavaScript files.

This makes the work of security searchers very easy as they have access to all endpoints at once. Some endpoints are maybe deprecated but are still showing in the list: this is more attack surface for a security searcher.

Access tokens

This one is really bad, but is really not that uncommon.

In JavaScript files, I’ve found the following:

AWS S3 id and secret key giving anyone full control over a S3 bucket
Cloudinary credentials giving anyone full control over the bucket
A CircleCI token, allowing me to launch builds, view commit history, and more
Various other third party API keys

Those are often found in the admin / internal JS files. Developers may think these files won’t be served to regular users so it’s fine to put sensitive information inside, but more often that not, it’s easy to get access to those files.

Getting to the interesting files

The interesting files are often the ones not intended for regular users: it can be an admin part, some internal tools, etc.

Every website has a different JS architecture. Some will load all the JS in every page, some more modern will have different entry points depending on the page you are visiting.

Let’s consider the following:

<script src="/assets/js/front.js"></script>

It’s very trivial, but in this case, one could try to load back.js, or admin.js.

Let’s consider another example:

<script src="/static/compiled/homepage.d1239afab9972f0dbeef.js"></script>

Now this is a bit more complicated, the file has a hash in its name so it’s impossible to do some basic enumeration.

What if we try to access this url: https://website/static/compiled/manifest.json?

{
  "assets": {
    "admin.js": "admin.a8240714830bbf66efb4.js",
    "homepage.js": "homepage.d1239afab9972f0dbeef.js"
  },
  "publicPath": "/static/compiled/"
}

Ooops! In this case this website is using webpack, a famous assets bundler. It is often used with a plugin that generates a manifest.json file containing the link to all assets, which is often served by the web server.

If you manage to find which tools a website is using, it’s easier to find this kind of vulnerabilities.

How to protect yourself

Here are a few tips to avoid being vulnerable to this kind of attacks:

Consider your JavaScript code public, all of it
If you really need access tokens in the front-end, get them via (secure & authenticated) API
Know your front-end toolbelt well to avoid basic attacks (manifest.json example)
Regularly audit your front-end code and look for specific keywords:
- secret
- token, accessToken, access_token, etc
- your domain name, for possible API urls
- your company name, for possible 3rd party credentials

Conclusion

Security issues can come from a lot of unexpected spots. When writing any kind of code, when pasting sensible data, it’s always good to ask yourself who will have access to this code, to avoid leaking all your secrets!

DEV Community: Adrien.S

Custom translations per tenant in a Rails app thanks to I18n backends

Identifying the current customer / tenant

Solution 1: store custom keys in the YML files directly

Solution 2: store custom keys in the database

Conclusion

Launching YAMLFish, a simple translations management tool

Features

Less is more

Pricing

How to use it

Conclusion

Avoid constantize in Rails

It could be unsafe

It has bad developer experience

How to do instead ?

Automation and branches- Using YAMLFish to easily manage I18n translations in your project

Main locale ?

Automatically push keys to YAMLFish

Working on a new feature that needs translations

Sending the branch for translation

Fetching the new translations

Conclusion

The basics - Using YAMLFish to easily manage I18n translations in your project

Setting-up

Pushing existing translations

Retrieving updated translations

Conclusion

Generate magic tokens in Rails with generates_token_for

How to use generates_token_for

Expiration

Invalidating the token when something changes

How it works

Conclusion

The Rails asset pipeline, old and new

The Dark Age

Enters Sprockets

General behaviour

In the code

Sprockets::Server

Webpacker

General behaviour

In the code

Propshaft & friends

General behaviour

In the code

Importmaps

Usage

In the code

Conclusion

Using minio to mock S3 in rails test and development

Installing Minio

Integration in dev

Integration in test

Your javascript can reveal your secrets

Business logic and other business leaks

Internal API paths

Access tokens

Getting to the interesting files

How to protect yourself

Conclusion

How to use `generates_token_for`

`Sprockets::Server`