DEV Community: Stefan Wienert

Migrating from (Rails) Webpack(er) to Vite

Stefan Wienert — Mon, 11 Jul 2022 09:00:00 +0000

After introducing the Rails wrapper webpacker in 2018/2019 into our projects, it has been a great addition and helped to propel our Javascript frontend development. But since a couple of months, there are other alternatives to consider. Rails in general likes to move to more simpler Import-Maps by default, or using esbuild directly. But in our case, having a separate build server that enables productivity enhancing features such as Hot Module Reload, or playing around with SSR, is a much more useful alternative for us. That’s why, after evaluating a couple of other options, we are going to Vite.

Why move?

Compared to Webpack, it’s much faster , as it does less and relies more on the Module loading (and thus, delayed Runtime errors) in the browser. Especially the initial start took like 1-2 minute with Webpack and after a while the server took several (~1-4) gigabyte of RAM (multiplied by Number of Projects Times Number of Developers can grow to unhealthy limits) and needed regular restarts.

Configuration fatigue: Starting with Webpack 2, and going up to 4, migrating between the Rails wrapper Webpacker 3 - 5, it found it is a major PITA to manage all the details, such as Babel-config, loaders, plugins and many other configuration snippets. Vite is a more opinionated approach and works OOTB without many plugins - We only use Vue plugin, the rest, like Sass, Typescript, even Pug, just work if you have the related packages installed). For our migration, the vite.config.js is much smaller than the whole of Webpack config:

git diff --shortstat origin/master -- package.json vite.config.js config/webpack config/webpacker.yml
 11 files changed, 91 insertions(+), 255 deletions(-)

Comparing only the dep-size of the yarn.lock is even more shocking:

git diff --shortstat origin/master -- yarn.lock
 1 file changed, 1358 insertions(+), 7642 deletions(-)

Vite (as does Webpack) supports configuration options for using a public development proxy that is handled by a Reverse Proxy. In our case, we mostly work on a central server, so everybody got a couple of ports assigned to map to frontend https. Vite can be configured via server.hmr.clientPort + server.hmr.host and still support HMR live updates via websockets, which was not the case for some other bundlers.

The biggest drawback against Vite would be, that we could not use wkhtmltopdf with styled documents anymore. For this particular project, that is not necessary, but in the future other projects, that generate styled PDFs must be migrated to puppeteer or similar (See the discussion on Github).

Migration

At the beginning, I’ve tried 2 automatic migration tool, that did almost nothing. I throw them away and started with a fresh vite-config generated by vite-ruby. Then I replaced all javascript_pack_tag with vite_javascript_tag and removed the stylesheet_pack_tag (Vite only needs the entrypoint javascript, and then it will include all dependencies and generated stylesheets). Then, I removed all of webpack, webpacker and babel: babelrc, package.json, Gemfile, config/webpack*, bin/webpack*. You may keep postcss.yml.

The various problems that occurred afterwards were solved as followed below:

Fix import aliases

To reference our own Javascript modules, with used plain path without @/ or ~/ at the beginning.

Like:

import Stuff from 'admin/Stuff.vue'

while admin/Stuff.vue would be located in app/javascript/admin/Stuff.vue

With Vite, the simplest approach seemed to be to define an alias for each folder in app/javascript:

// vite.config.js

import path from 'path';

export default defineConfig({
  resolve: {
    alias: {
      admin: path.resolve(__dirname, './app/javascript/admin'),
      candi: path.resolve(__dirname, './app/javascript/candi'),
      channels: path.resolve(__dirname, './app/javascript/channels'),
      hrfilter_check: path.resolve(__dirname, './app/javascript/hrfilter_check'),
      "job-booking": path.resolve(__dirname, './app/javascript/job-booking'),
      next: path.resolve(__dirname, './app/javascript/next'),
      public: path.resolve(__dirname, './app/javascript/public'),
      registration: path.resolve(__dirname, './app/javascript/registration'),
      utils: path.resolve(__dirname, './app/javascript/utils'),
      images: path.resolve(__dirname, './app/assets/images'),
    },
  },
})

Or automatic:

import fs from 'fs'

const items = fs.readdirSync("app/javascript")
const directories = items.filter(item => fs.lstatSync(path.join("app/javascript", item)).isDirectory())

const aliasesFromJavascriptRoot = {}
directories.forEach(directory => {
  aliasesFromJavascriptRoot[directory] = path.resolve(__dirname, "app/javascript", directory)
})
export default defineConfig({
  resolve: {
    alias: {
      ...aliasesFromJavascriptRoot
    }
  }

In any case, you can also rename all the imports and prefix them with “@/”, but it’s harder to automate, and the method above worked great!

require.context - load a whole folder into Javascript

In some cases, we were using require.context to load a whole folder of javascript into a module.

In one case, it was simplest to just inline all the necessary imports under each other. That makes it also easier for Typescript to analyze imports.

If you want to keep the behavior of loading a whole folder, Vite has a similiar thing, called import.meta.globEager.

So, if you want to load your Stimulus controllers like:

// webpack:
const context = require.context("controllers", true, /_controller\.[jt]s$/)
application.load(definitionsFromContext(context))

…then, have a look at ElMassimo’s repo: https://github.com/ElMassimo/stimulus-vite-helpers

Import needs file suffix for non-js/ts files

Vue-files need to be fully specified with suffix:

- import FooComponent from './FooComponent'
+ import FooComponent from './FooComponent.vue'

Same with lazy loading:

   components: {
- "backlink-modal": () => import("../modals/BacklinkModal"),
+ "backlink-modal": () => import("../modals/BacklinkModal.vue"),

The same was true when loading scss files from Javascript.

Import GraphQL queries

In some cases, we are importing GraphQL-queries, like:

import FETCH_USER from '../queries/FetchUser.graphql'

There are a myriad of not-functioning plugins out there. In the end @rollup/plugin-graphql worked:

// yarn add @rollup/plugin-graphql
// vite.config.js
import graphql from '@rollup/plugin-graphql';
...
plugins: [
    ...,
    graphql()
  ]

require not found / not a function

Vite does not support commonjs import anymore! Most dependencies already support the ESM (EcmaScript Modules import/export syntax), some older not just needed a upgrade. So usually it was enough to change it like this:

# find all usages of require("foo")
rg -F 'require('


- const qs = require("qs")
+ import qs from 'qs'

Sentry error tracking also changed the import, but * as was needed to added here:

-const Sentry = require('@sentry/browser')
+import * as Sentry from '@sentry/browser';

We dropped moment.js and replaced it with dayjs or straight Intl. Dayjs is for the most part compatible with momentjs and in many cases a simple search & replace is all you need (After configuring the necessary plugins for Dayjs)

Scss Postcss import

When removing @rails/webpacker from the package.json, the deps that it brought, like sass, postcss, were missing now, so we added it manually:

yarn add postcss-import postcss-preset-env postcss sass

When importing deps, reference it without the ~ at the beginning:

- @import "~bootstrap-vue/src/index";
+ @import "bootstrap-vue/src/index";

We had one Stylesheet theme (Bootswatch) which did not work, as it always wanted to include a Google Font. We copied the Scss file from the module into our repo and deleted the font import.

Also, we are using an older SASS version, so we received a lot of Scss warnings every time. The best way seems to be to downgrade sass for now.

yarn add sass@1.32.0

vuelidate in test/production

One page failed ONLY in test mode when running the full suite, using development server worked always! It was a frustrating and time consuming thing to debug, First I thought vue-router is the culprit, but after forcing sourcemaps and using Chrome to get onto the test-server, I found that vuelidate was the thing responsible.

The error I received, was:

> TypeError: can't access property "components", e is undefined

Which, according to Stackoverflow was related to mixins, which are using with Vuelidate’s ValidationMixin. It seemed, that in production builds, Vite (Rollup) dropped the content, and just imported “undefined”.

So I tried to just change the import syntax slightly:

import { validationMixin } from 'vuelidate/src/index'
console.log(validationMixin) // works!

import { maxLength, required } from "vuelidate/lib/validators"

And also disabling optimization for this particular dependency:

// vite.config.js
defineConfig({
  optimizeDeps: {
    exclude: ['vuelidate']
  },

Then it worked! If this helped you, make sure to upvote my Stackoverflow answer ;)

Legacy / IE11

I checked our access logs, and it seems that the customers using that particular service are finally free from using Internet explorer 11.

But some customers are using older Firefox versions (ESR), so adding a legacy polyfill etc. was necessary.

Have a look at the final vite.config.js (Without Sentry):

import { defineConfig } from 'vite';
import legacy from "@vitejs/plugin-legacy"
import { createVuePlugin } from 'vite-plugin-vue2';
import RubyPlugin from "vite-plugin-ruby"
import FullReload from "vite-plugin-full-reload"
import path from 'path';
import fs from 'fs'
import graphql from '@rollup/plugin-graphql';

const isProd = process.env.NODE_ENV === "production" || process.env.RAILS_ENV === "production"

const items = fs.readdirSync("app/javascript")
const directories = items.filter(item => fs.lstatSync(path.join("app/javascript", item)).isDirectory())
const aliasesFromJavascriptRoot = {}
directories.forEach(directory => {
  aliasesFromJavascriptRoot[directory] = path.resolve(__dirname, "app/javascript", directory)
})

export default defineConfig({
  server: {
    hmr: {
      // in our case the environment variable VITE_WSS_HOST is set by our system
      // so VITE knows were it should direct the wss://HOST requests
      host: process.env.VITE_WSS_HOST,
      clientPort: 443,
      protocol: "wss",
    },
  },
  resolve: {
    alias: {
      ...aliasesFromJavascriptRoot,
      images: path.resolve(__dirname, './app/assets/images'),
    },
  },
  plugins: [
    RubyPlugin(),
    graphql(),
    createVuePlugin({ jsx: true }),
    FullReload(["config/routes.rb", "app/views/ **/*", "app/controllers/** /*"], {
      delay: 200,
    }),
    isProd
    // if you want to support older browsers.
    // make sure, you have core-js 3 installed
      ? legacy({
          targets: ["defaults"],
          polyfills: ["es.promise.finally", "es/map", "es/set"],
          additionalLegacyPolyfills: ["regenerator-runtime/runtime"],
        })
      : undefined,
  ],
  optimizeDeps: {
    exclude: ['vuelidate', 'vuelidate/lib']
  },
  build: {
    // keep files between deployments - we are using capistrano-like deployment,
    // if you are using Docker or Heroku, then you can leave it
    emptyOutDir: !isProd,
    // force sourcemaps in test, too!
    sourcemap: true,
  }
})

Improving Typescript experience in Rails by generating a Typescript schema from Rails models (with enums and associations)

Stefan Wienert — Wed, 15 Sep 2021 14:01:17 +0000

For a side project (which I won’t reveal at the moment), I develop most of the frontend in Vue/TSX+Typescript. At the moment, I don’t have a specific “API”, but just use Inertia.js with a couple of main models that are simply serialized by model.as_json. Having then to type out a deep model schema with dozens of connected models (using delegated types in my case) by hand is a PITA. Fortunately, I came across schema2type, which is a Rails plugin that generates a Typescript schema for all of your models by only using schema.rb, BUT because of this constraint, it cannot know about Rails enums and associations.

This is why I quickly cobbled together a script that improves on that idea:

Gist

To “install” and run:

wget https://gist.github.com/zealot128/419949f1c426330493c84bb8eadc4533/raw/1c4629c05506630d3fe8543320cd6d2026405d8e/rails-models-to-typescript-schema.rb

rails runner rails-models-to-typescript-schema.rb > app/javascript/types/schema.d.ts

Afterwards you can easily use schema.MyModel.<TAB> to autocomplete all the things! In my usecase, I then built upon that schema to define more specific types that I serialize and pass to my frontend (via Inertia.js). It also works for the built-in ActiveStorage models and spit out this:

declare namespace schema {

  interface ActiveStorageBlob {
    id: number;
    key: string;
    filename: string;
    content_type: string | null;
    metadata: string | null;
    service_name: string;
    byte_size: number;
    checksum: string;
    created_at: string;
    variant_records?: ActiveStorageVariantRecord[];
    attachments?: ActiveStorageAttachment[];
    preview_image_attachment?: ActiveStorageAttachment;
    preview_image_blob?: ActiveStorageBlob;
  }

  interface ActiveStorageVariantRecord {
    id: number;
    blob_id: number;
    variation_digest: string;
    image_attachment?: ActiveStorageAttachment;
    image_blob?: ActiveStorageBlob;
    blob?: ActiveStorageBlob;
  }

  interface ActiveStorageAttachment {
    id: number;
    name: string;
    record_type: string;
    record_id: number;
    blob_id: number;
    created_at: string;
    blob?: ActiveStorageBlob;
  }
}

The 80 line script already supports:

belongs_to, has_one and has_many associations (un-polymorphic)
enums
null and not null
type mapping, borrowed from the original schema2type script

Security implications: The script generates a type with all attributes, that includes password-digests, tokens etc. You probably shouldn’t send those informations to frontend clients in general. For the typings that should be fine though, because, typings are only used in development and discarded in a production build.

To further improve your Typescript experience, have a look at https://github.com/bitjourney/ts_routes-rails - IMO here you really should make sure to not leak Routes to clients that you don’t want them to access. Depending on your production build unused route helpers may or may not be removed due to treeshaking.

Previewing Rails 7 upcoming changes

Stefan Wienert — Tue, 24 Aug 2021 12:00:00 +0000

Rails 7 is taking up speed. There is no beta out yet, but a lot of features, especially in ActiveRecord are available, if one want’s to wade through the Changelogs.

To view all the changes per framework gem, just access the main branch’s CHANGELOG.md:

Too many? … Most of those changes are very specific and maybe not interesting for all audiences, and ActionText, ActionMailbox is not used by us. I collected the most interesting changes for me below.

Active Record

Attribute level database encryption The most relevant feature in the bunch, this will make it so much easier to be more secure “by default”. See the new guide
Post.where(...).in_order_of(1, 2, 3) why would you need that? For example if you get the ordering from another source, like Redis inline calculations, Elasticsearch Results. You want to load the same models from the DB and keep the order from Elasticsearch.
Tagging of SQL queries to make it easier to find the culprit in the slowlog: config.active_record.query_log_tags_enabled = true. Controller & action name are already used as the default tags.
Nulls First/Nulls Last can be specified, unfortunately only in the internal Arel, not in Relation AFAICS: User.arel_table[:first_name].desc.nulls_last and nulls_first
MyModel.enumerate_columns_in_select_statements = true per Model
- Normally if you make Post.first you will generate a select * from. With this flag, AR will use the column names: select title, description from posts
Enum: removing the underscore from prefix, suffix, default: enum state: [], _prefix: :state, _suffix: :foo -> prefix: suffix
Load Async: Another maybe major feature: You are now able to parallelize the SQL loading, best on controller level, by appending .load_aync the query:@posts = Post.load_asyncI need to fiddle around with this feature and try to find limitations/thread limits etc.
relation.excluding(post, another_post) replaces the common pattern of finding sibling objects without one self:

  class Post
    def simliar_posts
      user.posts.excluding(self)
    end

timstamptz now supported as a column type - I have one app, where I only use structure.sql because of this. Happy to be able to go back to schema.rb
relation.where().invert_where nice convenience
relation.where.associated(:user) all records that have an association, shorthand for where.not(user_id: nil)
relation.where.missing(:user) opposite of above

Active Storage

Predefined variants

  has_one_attached :avatar do |attachable|
    attachable.variant :thumb, resize: "100x100"
    attachable.variant :medium, resize: "300x300", monochrome: true
  end
  ...
  user.avatar.variant(:thumb)

nice! I used model methods that made the commonly used variants, e.g. company logo. Can remove that now.
- vips is the new default, but mini_magick can still be used. To opt-in with old apps:
config.active_storage.variant_processor = :vips.
- Analyzer improved: will determine if the file has audio and/or video data and prefills: metadata[:audio] and metadata[:video]
- Previewing video files: FFMpeg args configurable under config.active_storage.video_preview_arguments.
- Audio analyzer added: duration + bitrate extracted
- blob expiration can be set individually per call::
rails_blob_path(user.avatar, disposition: "attachment", expires_in: 30.minutes)

Active Support

Added a faster and more compact ActiveSupport::Cache serialization format.
- Opt-in: config.active_support.cache_format_version = 7.0, it is backwards compatible
Improved test parallelization - only in effect if testing more than one file etc.
- as RSpec user I look with envy to the easier parallelization features over at Test::Unit…
rails app:update doesn’t overwrite en.yml, routes.rb, puma.rb, storage.yml
- Nice! Always a nuisance when using that very very helpful task
Quick benchmark: Rails.benchmark("some info") { ... }
- I guess that’s just an alias for Benchmark.measure { ... }, but good to have it available

How long until Rails 7 is released/RC is published?

Of course, no one except the core team knows this. To make an educated guess, these are the days between each releases (from Rubygems):

Version jump	days
4.0.0 -> 4.1.0.beta1	176
4.1.0 -> 4.2.0.beta1	134
4.2.0 -> 5.0.0.beta1	363
5.0.0 -> 5.1.0.beta1	301
5.1.0 -> 5.2.0.beta1	214
5.2.0 -> 6.0.0.beta1	284
6.0.0 -> 6.1.0.rc1	444

The “minor” jumps between 4.0 -> 4.1 -> 4.2 etc. took an average of 273 days from release of the previous minor upgrade to the first beta/rc or release.

6.1.0 was released on 9th December 2020, adding 258 days would put the release on September 2021! Hopefully not taking 444 days as Rails 6.1 took… that would put the release on February 2021. 😉

Endless Scroll / Infinite Loading with Turbo Streams & Stimulus

Stefan Wienert — Mon, 03 May 2021 19:00:00 +0000

Hotwire Turbo by the Ruby on Rails developers is the new solution to enhance server side rendered apps with interactive behavior without much Javascript at all.

In this post, I want to show you, how I built an infinite scrolling feature, meaning: When reaching the bottom of a list, load the next page and add it to the DOM. There are many many ways in handling this kind of solution, like, using a 3rd party library like “lazyload” and more. To get more familiar with the Hotwire Stack of Stimulus + Turbo, I decided to use Turbo Streams for handling the DOM-part, and Stimulus to connect with an Interaction Observer.

For this post, I will make the following assumptions:

We hava a controller PostsController with an index action
We already handling pagination via the great pagy Gem
Turbo + Stimulus are all set up
I use SLIM as the template language, because I like it’s brevity and clearness

Disclaimer: When this PR get’s released, this solution might be simplified much more, but just using <turbo-frame action="append"> with a little glue code.

Add stimulus to our posts

// index.html.slim

.list-group(data-controller="infinite-scroll")
  // If you need to enable Live Updates, you could connect to a
  // = turbo_stream_from current_user, :posts
  #posts
    = render @posts
  div(data-infinite-scroll-target='scrollArea')

  #pagination.list-group-item.pt-3(data-infinite-scroll-target="pagination")
    == pagy_bootstrap_nav(@pagy)

In this index we,

wrap our posts with a Stimulus Controller and
mark the posts into a div with id=posts (to later append to)
add a scrollArea empty element div just below our posts list - This area will be used for our Intersection Observer later on
add the pagy_nav or pagy_bootstrap_nav pagination tags on the bottom, also wrapped in a Stimulus Target to later on pick the next page’s link from it

Now, before we modify the controller to respond to Turbo events, we implement the Stimulus Controller

Stimulus controller

// app/javascript/controllers/infinite_scoll_controller.js

import { Controller } from "stimulus"

export default class extends Controller {
  static targets = ["scrollArea", "pagination"]

  connect() {
    this.createObserver()
  }
  createObserver() {
    const observer = new IntersectionObserver(
      entries => this.handleIntersect(entries),
      {
        // https://github.com/w3c/IntersectionObserver/issues/124#issuecomment-476026505
        threshold: [0, 1.0],
      }
    )
    observer.observe(this.scrollAreaTarget)
  }
  handleIntersect(entries) {
    entries.forEach(entry => {
      if (entry.isIntersecting) {
        this.loadMore()
      }
    })
  }
  loadMore() {
    const next = this.paginationTarget.querySelector("[rel=next]")
    if (!next) {
      return
    }
    const href = next.href
    fetch(href, {
      headers: {
        Accept: "text/vnd.turbo-stream.html",
      },
    })
      .then(r => r.text())
      .then(html => Turbo.renderStreamMessage(html))
      .then(_ => history.replaceState(history.state, "", href))
  }
}

We define the areas ScrollArea + Pagination
When loaded, the controller puts an Interaction Observer onto the scroll area
When the scroll area get’s into the viewport, we loadMore posts, by picking the next page’s url from the pagination
Import: We can’t use Turbo.visit but we are using fetch instead, because the Turbo-Stream Magic only works on POST/PATCH/… requests. But we want our controller to respond to this GET request with a specific Turbo Stream that handles the DOM manipulation. That’s why we use fetch manually with the special Accept: text/vnd.turbo-stream.html header here.
When the fetch returns, we pipe the result manually to Turbo.renderStreamMessage which evaluates the html content for Turbo Stream actions.

PostsController

class PostsController < ApplicationController
  include Pagy::Backend
  helper Pagy::Frontend

  def index
    @pagy, @posts = pagy Post.order(:created_at)
    respond_to do |f|
      f.turbo_stream
      f.html
    end
  end

Straight forward Turbo: we respond with “turbo_stream” format, or with html (template at the top in this post). Let’s show the turbo_stream template:

index.turbo_stream.slim - Turbo Stream response

turbo-stream action="append" target="posts"
  template
    = render partial: 'posts/post', collection: @posts, formats: [:html]

turbo-stream action="update" target="pagination"
  template
    == pagy_bootstrap_nav(@pagy)

We append this page’s posts to the div with id=posts
We replace the pagination completely to reflect the new page
Important to switch the format to html to get our ‘post’ partial, don’t know if intended or bug.

That’s it! Because the scroll area will be always on the bottom, it will trigger over and over, Stimulus will pick out the current next page’s url from the pagination part, and Rails will respond with a Turbo stream that updates both the new posts and replace the pagination.

Growing Rails - Utilizing Form Models for complex validations or side effects

Stefan Wienert — Sat, 01 May 2021 22:00:00 +0000

Rails by default has a couple of “buckets” to put your code into. Some people claim that this is not sufficient when building larger apps, other people hold against, that you could upgrade your “model” directory and put all kinds of Ruby POROs (Plain Ruby Objects) into it. I am not totally convinced to mix database-backed models and all kinds of different objects, but rather like to identify common patterns and create subdirectories directly under app/, like: app/queries, app/api, app/api_clients or the bespoken Form Models under app/forms.

Advantages of Form Models

Validations

The longer I work with Rails, and the longer or larger the project horizon is planned, the less I like to use complex validations on the model. Why? Because in my experience, the validations only make sense, when the model is created by a user through a UI. But not all objects are created by a UI. Think of:

Ad-hoc creation of small objects in development, test
Cronjobs that create or update a model and fail because one item has gone invalid because a validation / column has been introduced after the creation of the item itself - this happens quite often IME
different validations depending on the state of the user (Form wizards, registration vs. updating profile, etc.)

Side Effects

The second big win of form models are side effects , like sending a email/notification, enqueuing jobs, creating log/audit records, update elasticsearch, etc.pp. Doing those in the controller is maybe feasible but it can go out of hand very fast. Doing those in callbacks is IMO a very bad practice: Thinking about backfilling some attributes, but accidentally sending a notification to all. You always have to know, which side effects are present, even when updating in the background. So, I think the save method of a Form Object is a perfect place to kick off various actions.

Database-less actions

Also, you sometimes have actions that not necessarily have a database table attached, think of: CSV export (with config), Providing a test action for an integration (Webhook test, IMAP integration, SAML integration, …). Those are perfect candidates for Form Models!

Controller does not need to know the model’s attributes

Another advantage, which I later found out about, is that I can get rid of the permitted_params / params.require stuff from the controller (which is there rightly so to prevent Mass Assignment Injections). Because our form model can only reveal the attributes which the user can update anyways, we can build a very simple wrapper, that automatically permits all attributes of the form model. I really like that, because now the controller does not have to know about the model’s fields – How often did you forgot to add a missing attribute to the permit(..) method?

Our Form base class

Over the years, our base class changed. One thing I want of a Form Model, is Parameter Coercion (e.g. casting “1” to true for a boolean). In the past, we used the virtus Gem to handle the definition of the attributes and coercion. But recently, after Rails released the Attributes API, we can just use ActiveModel::Attributes.

# app/forms/application_form
class ApplicationForm
  # ActiveModel: We get validations, model_name stuff, etc.
  # now our object quaks almost like an ActiveRecord::Base model
  include ActiveModel::Model

  # Gives us the `attribute `` method to define attributes with data types:
  # attribute :email, :string,
  # attribute :active, :boolean, default: true
  include ActiveModel::Attributes

  # Helper Method to call from the controller:
  #
  # MyForm.new_with_permitted_params(params)
  #
  # It fetches the correct key, e.g. params.require(:my_form).permit(:a, :b, c: [], d: {})
  def self.new_with_permitted_params(params)
    permitted_params = params.
      require(model_name.param_key).
      permit(*permitted_params_arguments)
    new(permitted_params)
  end

  # Maps the defined `attributes` to a argument list for params.permit()
  # Array and Hash attribues must be written in hash form.
  def self.permitted_params_arguments
    structures, primitives = attribute_types.
      map { |name, type|
        if type == :array
          { name => [] }
        elsif type == :hash
          { name => {} }
        else
          name
        end
      }.partition { |i| i.is_a?(Hash) } # rubocop:disable Style/MultilineBlockChain
    params = [*primitives, structures.reduce(&:merge)].reject(&:blank?)
    if params.length == 1
      params.first
    else
      params
    end
  end

  # placeholder to implement by the inherited form instances
  def save
    return false unless valid?

    raise NotImplementedError
  end
end

Example Usage

Imagine you are for once not using Devise and implementing Password Reset yourself.

# app/forms/password_reset_form.rb
class PasswordResetForm < ApplicationForm
  attribute :email, :string
  validates :email, presence: true, format: { with: URI::MailTo::EMAIL_REGEXP }

  def save
    return unless valid?

    account = User.where(email: email).first
    unless account
      sleep 1
      return true
    end

    account.regenerate_reset_password_token_if_not_active
    Mailer.password_reset(account).deliver_now
    true
  end
end

As the model is 100% compatible with a ActiveRecord Model, we can use it the same in the controller:

class PasswordResetController < ApplicationController
  def new
    @form = PasswordResetForm.new
  end

  def create
    @form = PasswordResetForm.new_with_permitted_params(params)
    if @form.save
      redirect_to root_path, notice: "E-Mail instructions have been sent"
    else
      render :new
    end
  end
end

Now, if we get the requirement to log account activities (audit trail), the save method is a perfect place to continue. For this purpose, I usually define “normal” attribute accessors that the controllers fill. Those fields will not be available through the permitted params sieve and are safe for this purpose.

# ...
  def create
    @form = PasswordResetForm.new_with_permitted_params(params)
    @form.request = request
    ...
  end

###

class PasswordResetForm < ApplicationForm
  # ...
  attr_accessor :request

  def save
    #...
    account.activities.create(ip: request.ip, user_agent: request.user_agent)
    #.
  end

Hope that helps you in your organisation of form models! For us, those are a frequently used pattern.

PostgreSQL (Rails) - Reverse search (tagging, subscription)

Stefan Wienert — Tue, 03 Nov 2020 09:11:03 +0000

Using PostgresQL’s search capabilities (tsvector etc.) is a well-known low-key alternative to using a full blown search engine, like Elasticsearch/Solr/Algolia. But it turns out, it is also quite easily usable as a “reverse search engine”, what Elasticsearch calls “Percolate”. One use case could be user supplied “Subscriptions”, think of “search subscriptions” of classified that you want to trigger whenever a new item reaches the database.

After fiddling around one day with Elasticsearch percolators I was not so satisfied with the results, that I decided to trying PG’s tsvector/tsquery.

Recently, I’ve wanted to build a keyword tagger , that given a blob of text uses a predefined vocabulary database to give you the “most relevant” tags for that tags. The process should be absolutely the same as with search subscriptions by reversing Query (queries are stored in the DB) and Documents (only one document: the document to match with the queries). For this example I will use keyword matching as we implemented it in Empfehlungsbund in various places:

Having a table keywords with a string column keyword, e.g. a managed table of most important keywords for your domain. In our cases that includes all the technolgy terms of developers/administrators (like “Javascript” or “Ruby on Rails”)
there is also a little hierarchie of keywords in “competence groups” and we also add the “search relevance” which expresses how often this term is used by Job Seekers on our job platforms of Empfehlungsbund.

1. Adding TS-columns

We add a tsquery column keyword_query and a normalized token keyword_search_token

CREATE TABLE public.keywords (
    id bigint NOT NULL,
    keyword character varying,
    keyword_search_token character varying,
    keyword_query tsquery,
);

keyword_query if you follow several guides about PG’s fulltext search, I never came across one that proposed to just use a tsquery instead of a tsvector. But in our case (percalote) this makes total sense to add the string as tsquery.

keyword_search_token is a normalized version of our query, a processing pipeline that we ran before saving a record AND before tagging a document on that document later on. In our case the Pipeline consists of:

Remove All HTML,
normalize collocations (“java-developer” and “java developer”)
Remove common (German) stop words,
REPLACE important special chars that our search engine will need, like “.” (‘.NET’, “Vue.js”) or prefix/suffix ‘#’, ‘+’ (C++, C#). Many search engines does get this not right and strip out all special chars by default.

Here a Ruby snippet from our codebase:

STOPWORDS = YAML.load_file('config/stopwords.yml')                                                                                                 
WORD_BOUNDARY = '(^|\s|[,\.\-\!\?])'
REPLACE_REGEX = /#{WORD_BOUNDARY}(#{STOPWORDS.join('|')})#{WORD_BOUNDARY}/i

REMOVE_STRING = [
  REPLACE_REGEX,
  %r{\([mwd/ ]+\)}
]

def preprocess_query(text)
   ActionController::Base.helpers.strip_tags(text).
      yield_self { |s| replace_special_chars(s) }.
      yield_self { |s| REMOVE_STRING.reduce(s) { |agg, elem| agg.gsub(elem, ' ') } }.
      # "Foo/Bar"
      gsub('/', ' / ').
      # EcmaScript5 -> Ecmascript 5
      gsub(/(\w+)(\d+)#{WORD_BOUNDARY}/, '\\1 \\2 \\3').
      tr('-', ' ').
      gsub(/\s+/, ' ').
      strip
end

def self.replace_special_chars(string)
  # C# C++, F#, ...
  string.
    gsub(/[#\+]/, '#' => "RAUTE", '+' => 'PLUS').
    # vue.js node.js, .net but not sentence dot
    gsub(/(\w+)?\.(\w+)/, '\\1DOT\\2')
  end

2. Adding a tsquery column and trigger

Now, we want to build a trigger that manages the keyword_query conversion automatically. We can to this inline by splitting the words by whitespace and joining them back together with the infix “<->” operator, which means “the words on either side of the operator should be near together”. (Introduced in PG 9.6). This is extremely useful for searching for collocations like “Ruby on Rails”

CREATE FUNCTION public.keyword_results_before_insert_update_row_tr() RETURNS trigger
  LANGUAGE plpgsql
  AS $$
BEGIN
  new.keyword_query := to_tsquery(array_to_string(string_to_array(new.keyword_search_token, ' '), ' <-> '));
  RETURN NEW;
END;
$$;
CREATE TRIGGER keyword_results_before_insert_update_row_tr 
BEFORE INSERT OR UPDATE ON public.keyword_results 
FOR EACH ROW EXECUTE PROCEDURE  public.keyword_results_before_insert_update_row_tr();

2a. Rails tsquery migration

If you are using Rails you can use hairtrigger Gem to more easily create the trigger in a migration:

add_column :keyword_results, :keyword_search_token, :string
add_column :keyword_results, :keyword_query, :tsquery
add_index :keyword_results, :keyword_query, using: 'gist'

create_trigger(compatibility: 1).on(:keyword_results).before(:insert, :update) do
  "new.keyword_query := to_tsquery(array_to_string(string_to_array(new.keyword_search_token, ' '), ' <-> '))"
end

BUT: ActiveRecord’s PG-Adapter at this moment does not know at the moment to handle tsquery columns and will fail with:

unknown OID 3615: failed to recognize type of 'keyword_query'. It will be treated as String.

And the table will be missing from db/schema.rb, and thus not loaded in test, with this comment:

# Could not dump table "keywords" because of following StandardError
# Unknown type 'tsquery' for column 'keyword_query'

The normal solution which you find for this class of problem is generally to switch to “SQL”-mode of Schema dumper. I don’t like that, because it makes the diffs gigantic and merges/rebases fail more often. I found, I could just tell Rails via patching how to handle that problem:

# config/initializers/active_record_pg_types.rb
require "active_record/connection_adapters/postgresql_adapter"

module PGAddTypes
  # 1st place: how to handle this column = it's a string!
  def initialize_type_map(m = type_map)
    super
    m.register_type "tsquery", ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::OID::SpecializedString.new(:tsquery)
    m.register_type(3615, ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::OID::SpecializedString.new(:tsquery))
  end
end
# 2nd place:
ActiveRecord::ConnectionAdapters::PostgreSQLAdapter::NATIVE_DATABASE_TYPES[:tsquery] = { name: "tsquery" }

ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.prepend PGAddTypes

# 3rd place: mapping a schema / migration line to a database mapping
module ActiveRecord::ConnectionAdapters::PostgreSQL::ColumnMethods
  def tsquery(*args, **options)
    args.each { |name| column(name, :tsquery, options) }
  end
end

With this code in place, schema-dumping and loading will work again!

3. Reverse searching against table

preprocessed_query = preprocess_query(params[:query])
# create intermediate from table with our keywords and the preprocessed full_text:
from = KeywordResult.sanitize_sql_array ['keywords, to_tsvector(?) full_text', preprocessed_query]
KeywordResult.
  # where: having a match
  where(%{full_text @@ tsv_body}).
  from(from).
  order('score desc').
  # select: all normal columns + a "score"
  select("keywords.*, ts_rank(full_text, tsv_body) as score")

In SQL:

SELECT keywords.*, ts_rank(full_text, tsv_body) as score
FROM keyword_results, to_tsvector('Tag this string with keywords like Ruby on Rails') full_text
WHERE (full_text @@ tsv_body)
ORDER BY score desc

(Of course, you should NOT pass the string in raw like this but use prepared statements with a placeholder variable)

Tips

Improving the accuracy by scoping

If you have a large spectrum of keywords there is the problem that some words can mean different things for different audiences. For example, when parsing a job ad or a applicant CV, imagine you find the word “Amazon” - For an IT person that probably is related to “AWS”, for a Retail staff maybe the Amazon Ad or Shops. Or think of generic greek/latin names for all kind of technologies (“Atlas”, “Prometheus” etc.) which could have all kind of meanings in different contexts, or roles like “Architect” or “Designer”. In our case we add the general sector to a keyword, like “IT” (programmer, architects, admins, tech designers), “Engineering”, “medicine”, “white-collar office” (Marketing, Sales, HR, Accounting) and similar. In a first step we first find, which of the Tags would have the most direct keyword matches and then only search for those keywords in this sector.

Suggest new keywords

As mentioned in the beginning, we have a 2 tier hierarchy of our keywords. Each keyword belongs to a “competence” (E.g. “Ruby on Rails”, “RSpec” belongs to “Ruby”). To suggest keywords we just take the keywords found, go up to the competence and then take the most relevant N keywords ordered by the “search popularity”.

Internationalize a medium Rails App with Vue frontend effectively with tooling

Stefan Wienert — Sat, 10 Oct 2020 12:00:00 +0000

TLDR: I released a Ruby CLI tool that can semi-automatically extract raw Strings from Ruby-files, SLIM, ERB-Views and Vue-Pug templates. Read on about the background.

Recently, we started to internationalize our ATS (Applicant tracking system) recruiter backend, to also show support English speaking recruiters, a step that most growing Rails apps sooner or later will face. That’s why I want to share some tooling that I use (or build) to help you make this very first step, extracting ALL the strings in the app into the I18n space.

Additionally, we also use bunch of Vue-components (about ~175) to add interactivity in many parts of the app. Having started lean with YAGNI in mind, Internationalization for the internal users (the Recruiters) was not a major concern in the first 2 years of running our ATS, so most Strings where plain German around the app. In our case, I used some Regex searches and found:

160 SLIM-Views with at least one string
175 Vue-Components
bunch of Controller flash messages, page titles, breadcrumbs
Models/Forms (custom error messages, enum-translations etc.)
…and more business related classes with custom descriptions/titles (in our case, E-mail-template definitions with placeholders, states, events, actions etc.)

1. Slim - SlimKeyfy

Manually extract keys from templates is tedious as well as error prone, as one will mix up keys quiet easily, forgot to save files/overwrite changes and so forth. This is why I tried searching for some tooling that helps with Extraction. Luckily I found Slimkeyfy first, which did an OK job extraction most SLIM-keys. But first, there where some problems with the tool, namely it could not parse parts of our Slim-syntax, and some of the extraction regexp could be improved. This is why I started forking the Gem to add missing functionality. Anyhow, this still leaves us with a bunch of Vue-Components and Ruby-classes… But the slimkeyfy gave me the idea.

2. Ruby Classes - Using Ruby-parser

I was reading a couple of posts about using the awesome Parser((-Gem, which allows to rewrite Ruby code on the fly using hooks. This is what tools like **Rubocop using under the hood. Using another great Gem, tty-prompt , I cobbled together a small CLI that interactively asks the user for every “interesting” String in given Ruby files and automatically extract those. The result was extremely satisfying, as all the code was still valid and the transformations very precise. Even extraction of Heredocs and interpolating arguments worked perfectly!

3. Vue Extraction

This one is a little trickier… Most of the components are written with the Pug -templating language, which is very similar to Slim. So I hacked the Slimkeyfy Gem to also handle Vue-Pug files, which worked good, too. That means I could extract those in the same manner.

What about the Vue-components that are still using the default HTML-like templating? After fiddling (without success) with HTML-parsers like Nokogiri, that had major showstoppers (Nokogiri will gulp the “@click” and similiar attributes of Vue), I just decided to convert the remaining html templates to Pug. Luckily, there are several online API’s that can convert that easily and are about 98% correct, so I just build a small Ruby script to walk all the unconverted files, send them to the “API” and write it into a file. Afterwards, extraction worked the same as above

The missing 2%: Just watch out if you are using a bunch of self-closing custom tags next to each other, those were nested after converting with the tool in 1 case instead of next to each other which a test picked up.

4. Manual effort required for the finishing

In the end, most apps have some kind of exceptional usage, that will not be covered by the tooling above, but the Tooling helped to reach the Goal to 90%. To find obvious omissions, I’ve created a ToDo-List by using Rip-grep/Ack/Ag regex search, e.g. Find all Strings that begin with a uppercase letter or Heredocs is a very good heuristic to find untranslated Strings:

rg '"[A-Z][a-z]' app/ >> todos.txt
rg "'[A-Z][a-z]" app/ >> todos.txt

Putting it together: Introducing ExtractI18n Gem

Afterwards, after having used different tools, I decided to put it all together, optimize and streamline the interaction. This is why I want to show you a Gem that does all the work above, but also can convert ERB.

ExtractI18n

ExtractI18n is Ruby Command-Line-Tool, that given a path to Ruby/Slim/Vue-Pug/ERB-Files will walk those and ask you for every string, if you want to extract those in a given yml file (e.g. config/locales/unsorted.de.yml). Before every file system change you will be shown a Diff at the end, so you can decide if the tool messed up.

extract-i18n --locale de --yaml config/locales/unsorted.de.yml app/views/user

It uses SlimKeyfy code as a basis for the SLIM/Vue-extraction, also HTML-Extractor for ERB and the mentioned Ruby-Parser into a unified interface.

Sorting it out: I18n-tasks

I18n-tasksn

EVERY Rails app, that has a non-trivial amount of I18n-strings, should make use of the awesome i18n-tasks. I ran i18n-tasks normalize -p repeatedly during the internationalization process to sort the just extracted keys into the right files. In the end, every major view/part of the app has it’s own I18n-file, like: config/locales/email_templates.LOCALE.yml, config/locales/admin.LOCALE.yml etc.

We also use the Rails I18n backend for storing the Vue’s frontend translations under the js. namespace. I’ve wrote an article about that integration some time ago. To make I18n-tasks find usages of those Vue-strings, you can also download and require my Vue-Scanner for I18n-tasks.

Also checking the “health” with i18n-tasks health -l de (just focus on the extracted language, as the other language is still missing) and see, if some keys are missed, overwritten etc. Granted, that will produce a huge Git-diff once, but afterwards the extraction and I18n is totally managed by I18n-tasks.

Then, I exported all missing keys for the target language, used the Google-Translate feature to make a basic translation and generated a CSV with the suggested translation by filtering for the missing keys. After getting those fixed by a non-technical person, I can reimport it, too! Great tool!

Stats

In the end, I extracted about 2000 keys of about 120.000 characters, which is not too much. Google Translate API via I18n-tasks took about 3 seconds to translate those. During this process, I touched almost every file of the app and produced a giant diff, here only the app/ folder:

git diff origin/master --shortstat -- app
512 files changed, 4641 insertions(+), 5096 deletions(-)

But … with the tooling and producing a Gem along the way, learning about the Ruby-Parser Transformer, it was much more fun than anticipated! :) In the future, when building new views, I will still write those out in German first, and run this tool in the end, as the result is more correct than me copy pasting stuff and messing up interpolations.

Self-hosting Sentry Error Tracking starting at 5 EUR/m on Hetzner Cloud with Docker + SAML/Mattermost integration

Stefan Wienert — Tue, 14 Jan 2020 10:00:00 +0000

Motivation

Tracking application errors should be a check list item for all production apps. In simple cases, a automatic "email the stacktrace to admin" is suffice to start but reaches the limit very fast. Items like

Javascript Error Tracking, (in 2020 with source maps),
Environment information, like logged in user,
combining, ignoring, merging errors,

are requirements most growing projects will have sooner or later. Having an error tracking system in place is the best solution.

For our case, we prefer to self-hosted solutions like this, which has significant advantages:

no additional data protection issues, or signing additional data processing agreements, as all the data remains in the company's sphere of influence.
integration with company systems, like VPN/SAML usually more easy or is even only possible by self hosting,
sometimes being the only user of your systems provides performance benefits , e.g. comparing with using Sentry's hosted version, our current installation seems to be orders of magnitude faster (experienced response times like 5-10 seconds when using Sentry free tier), also because the error tracking and app servers are not only in the same country but maybe even the same DC like your app which drastically reduces latency (Sentry) or transport cost when using AWS.
saving some money per month instead of paying "per app"

On the downside, running things themselves can be more risky, especially when a Error tracking system is hard to setup.

From the very beginning, we used Errbit error tracking, which provides a Airbrake compatible API. But having tried out Sentry.io for private projects before, I was blown away by the features, like tons of plugins, beautiful clean UI, top notch Javascript integration with breadcrumbs and source maps, so I've decided to set it up in a self-hosted manner for our company. Fortunately, Sentry provides a Docker installation script, which I will use during this guide.

Installation of self hosted Sentry

First, we set up a Cloud VPC. Size of the cloud instance depends on your prospected error volume. We started with a Hetzner CX21 which costs about 5 EUR per month and can be upgraded easily via button click and rescaled later on, if necessary.

After starting the instance, install Docker and Docker compose on the host, or do it like us, and use a cloud-config yaml during installing (Pasting into field user config on Hetzner Cloud), e.g.:



# cloud-config
apt:
  sources:
    docker.list:
      source: 'deb [arch=amd64] https://download.docker.com/linux/ubuntu $RELEASE stable'
      keyid: 0EBFCD88 # GPG key ID published on a key server
packages:
- apt-transport-https
- bash-completion
- ca-certificates
- command-not-found
- curl
- debian-archive-keyring
- dnsutils
- fail2ban
- docker-ce
- docker-compose
- golang-go
- git-core
- htop
- lshw
- lsof
- ltrace
- make
- software-properties-common
- sysstat
- tar
- unattended-upgrades
- vim

After powering up, clone this repository to /opt/sentry



$ git clone https://github.com/getsentry/onpremise /opt/sentry

# Or, use this fork that has Caddy/Letsencrypt support
$ git clone https://github.com/merantix/sentry /opt/sentry

This setup will include everything you need BUT a HTTPS proxy. To have Caddy run as an HTTPS Proxy with Auto-Letsencrypt, check out this fork merantix/sentry.

Adjustments and configuration

If you are wanting to run Sentry in a organisation setting, you might want to install some plugins. In our case, these are a Mattermost plugin and a SAML2 plugin for user authentication.

Add those requirements to the Dockerfile in the getsentry folder:



# /opt/sentry/Dockerfile
ARG SENTRY_IMAGE
FROM ${SENTRY_IMAGE}-onbuild

+ RUN pip install https://github.com/getsentry/sentry-auth-saml2/archive/master.zip
+ RUN pip install -e git+https://github.com/NDrive/sentry-mattermost@master#egg=sentry-mattermost
+ # Or instead, a forked version with Mattermost multi channel support
+ RUN pip install -e git+https://github.com/zealot128/sentry-mattermost.git@merged#egg=sentry-mattermost

(If you are using Mattermost, and like to reuse webhooks between multiple channels/projects, I've recommend my fork until this PR is merged)

Now, check out configuration in docker-compose.yml and env. In general, we had very little configuration:

SENTRY_SECRET_KEY in env (or will generate during installation)
changed Caddy hostname in Caddy/Caddyfile, make sure, that hostname resolves to the server's ip for letsencrypt
set, or remove email settings in docker-compose.yml or env

If you are finished, run:



# will ask for a start password for the admin user
$ ./install.sh

# start all the services
$ docker-compose up

Some quick docker-compose commands you might need later on:



# show stdout of a host
$ docker-compose logs web

# enter a host
$ docker-compose exec web bash

# start/stop a host
$ docker-compose restart web
$ docker-compose up web
$ docker-compose down web

SAML

If you are using a SAML2 provider for your organisation, you can try to add the SAML authentication next. The specific settings are not standardized, and almost all supplier use different kind of names for the person's attributes (Claims). For example, we are using a (of course self-hosted) custom SAML2 server based on Ruby-SAML by Onelogin and also added several extra fields via a OID extra.

Those settings are based on these attributes (claims) by Ruby SAML, but like I said, that depends on your provider:

Debugging attribute mapping

To find out the specific names of the mapping, that you can use, you can add a print statement into the right Django file. I've gone this way:



$ docker-compose exec web bash
$ apt install vim
$ vim /usr/local/lib/python2.7/site-packages/sentry/auth/helper.py

Add import pprint somewhere in the top of the file, and add a printf statement in the finish_pipeline function:



+ import pprint

# ...

    def finish_pipeline(self):
        data = self.fetch_state()

        # The state data may have expried, in which case the state data will
        # simply be None.
        if not data:
            return self.error(ERR_INVALID_IDENTITY)

        try:
+           pprint.pprint(data)
            identity = self.provider.build_identity(data)

Exit the docker container and restart web docker-compose restart web. Now try set up of auth again and watch docker-compose logs web for debugging output.

Mattermost

Mattermost integration was straight forward:

Create a webhook as a Mattermost admin, copy the API url
Mattermost must be enabled as a Legacy Integration per project, after enabling add Webhook url.
My fork of the plugin also supports changing the channel per project and reusing the Webhook.

Open Source used:

getsentry / self-hosted

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept

Self-Hosted Sentry

Sentry, feature-complete and packaged up for low-volume deployments and proofs-of-concept.

Documentation here.

View on GitHub

vara-ai / sentry

Sentry On-Premise setup

Self-Hosted Sentry Nightly

Official bootstrap for running your own Sentry with Docker.

Requirements

Docker 19.03.8+
Compose 1.24.1+

Minimum Hardware Requirements:

You need at least 2400MB RAM

Setup

To get started with all the defaults, simply clone the repo and run ./install.sh in your local check-out.

During the install, a prompt will ask if you want to create a user account. If you require that the install not be blocked by the prompt, run ./install.sh --no-user-prompt.

There may need to be modifications to the included example config files (sentry/config.example.yml and sentry/sentry.conf.example.py) to accommodate your needs or your environment (such as adding GitHub credentials). If you want to perform these, do them before you run the install script and copy them without the .example extensions in the name (such as sentry/sentry.conf.py) before running the install.sh script.

The recommended way to customize your configuration is using the files…

View on GitHub

getsentry / sentry-auth-saml2

SAML2 SSO provider for Sentry

SAML2 Auth for Sentry

DEPRECATED: This project now lives in sentry

Note: SAML2 Authenttication is still currently an experimental feature.

An SSO provider for Sentry which enables SAML SSO and SLO support, including various identity provider support.

The following identity providers are supported

A generic SAML2 module is also provided, which may be configured with any Identity Provider that conforms to the SAML2 specification.

Install

$ pip install https://github.com/getsentry/sentry-auth-saml2/archive/master.zip

Configuration

Refer to the Sentry Single Sign-On documentation for individual SAML Identity Provider configurations.

Refer to the Enabling SSO documentation for what feature flags to enable for this plugin.

View on GitHub

NDrive / sentry-mattermost

Sends Sentry notifications to Mattermost Open Source Chat

Sentry Mattermost

A plugin for Sentry to enable notifications to Mattermost Open Source Chat. This is based in the sentry-slack plugin: https://github.com/getsentry/sentry-slack

Usage

Install with pip and enable the plugin in a Sentry Project:

pip install sentry_mattermost

Configure Mattermost:

Create an Incoming Webhook
Enable override usernames and profile picture icons in System Console Integrations

Contributing

We use Docker to setup a development stack. Make sure you have the latest Docker Toolbox installed first.

First time setup

Setups Docker containers and Sentry admin:

make bootstrap restart

Development

Each time you update the code, restart the containers:

make restart

And access the sentry admin at

http://<DOCKER IP>:8081

View on GitHub

Upgrading Rails from 4.2 to 5.2, 6.0 - collected notes

Stefan Wienert — Tue, 14 Jan 2020 08:00:00 +0000

Recently, I've upgraded a bunch of apps to Rails 5.2 and 6.0, I want to share some common issues and the process I follow:

General upgrade process

Have a quick overview of the Rails upgrade guide, to get a feeling what kind of things changed between versions:
- https://edgeguides.rubyonrails.org/upgrading_ruby_on_rails.html
I change the Rails requirement in Gemfile to the next minor version, like gem "rails", "~> 5.0.0", Maybe also bump the Ruby version to the required one, if your current one is too old. But I try to usually do it separately, because newer Ruby versions can also break your app.
Now comes the bundle update rails, This will probably fail because some other Gems collide. If those Gems have version constraints in your Gemfile, relax that. Try adding more and more Gems to bundle update rails rack sass-rails etc., Use a bundle update without Gems only as last resort, as this will update ALL Gems at once and usually break a lot of things.
In my experience, common Gems that have to be upgraded together with Rails, are:
- paginators (will_paginate)
- form gems (simple-form)
- ActiveRecord Gems (paper_trail, acts-as-taggable-on)
- Admin-interfaces (Active Admin)
- Asset-Pipeline Gems (sass, sass-rails, font-awesome-rails)
- bootsnap, rspec-rails
Try to boot app, like bundle exec rails c, Fix any errors that happen, then
Run rails app:update. The will launch a command line wizard, that will interactively compare all the config files of your app to that of a newly generated Rails app. My tip: Try to adjust your files in a editor to match the Rails standard as closest as possible. E.g. our config/environments/development.rb looks almost the same like the generated. All custom config by us is then on the bottom of the file and can be easily moved between upgrades later. Alternatively, use Rails-Diff to compare config changes between versions.
Try to run tests, fix deprecations
Run development server and click around
Push to CI. Hint: Try to make deprecations more visible, make them an error, and eager load on test (to remove bugs that happen by invalid files that the development system didn't catch):

# config/environments/test.rb

    config.eager_load = ENV['CI'].present?
    config.active_support.deprecation = ENV['CI'].present? ? :raise : :stderr
    # Show the full stacktrace of deprecations, 
    #   e.g. middleware. Maybe put that line in application.rb after loading rails
    ActiveSupport::Deprecation.debug = true
  end

Rails 5.0

Protected Attributes is gone!

If you previously have used attr_accessible and similar, that stuff is gone!

Finding possible occurences:

ag new.\*params app | grep -v permit
ag update.\*params app | grep -v permit

Sometimes, e.g. for query models on a get request, this kind of pattern is useful:

# passing whole params
MyForm(params.permit!.to_h)

# passing only params, that might not be there on the first page load
MyForm(params[:my_form]&.permit!&.to_h)

Important: Belongs_to required by default!

This will probably brake old apps. To reduce friction for future upgrades, adjust config/application.rb:

module MyApp
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 5.0
    ...

    config.active_record.belongs_to_required_by_default = false
  end
end

Some later Rails generator brings this line into a config/initializers, but that seems to be having no effect, only adding to application.rb.

before_filter -> before_action

(Same with after_action)

sed -i 's/before_filter/before_action/g' `find app -type f`
sed -i 's/after_filter/after_action/g' `find app -type f`

Controller tests/specs - get/post must be keyword arguments

In case you are using RSpec, just use this awesome Gems to convert everything:

gem install rails5-spec-converter
rails5-spec-converter

This converts 95% of the scenarios, only very custom session/cookies stuff must be checked manually.

redirect_to :back deprecated

- redirect_to :back, alert: "whatever"
+ redirect_back fallback_location: '/', alert: "whatever"

Rails 5.1

image_tag does not allow nil! nil is not a valid asset source, wrap all image_tag in an if my_model.attachment.present?
database_cleaner is not required for browser tests anymore (Rspec: System Tests)
add 'listen' gem to development/test group gem "listen"

response.success? -> response.sucessful?

sed -i 's/be_success$/be_successful/g' `ag be_success$ spec -l`

Foreign Key mismatch

ActiveRecord::MismatchedForeignKey: Column `cooperation_id` on table `cooperation_data_points` does not match column `id` on `cooperations`, which has type `bigint(20)`. To resolve this issue, change the type of
the `cooperation_id` column on `cooperation_data_points` to be :bigint. (For example `t.bigint :cooperation_id`).
Original message: Mysql2::Error: Cannot add foreign key constraint: ALTER TABLE `cooperation_data_points` ADD CONSTRAINT `fk_rails_3979ee89c8`
FOREIGN KEY (`cooperation_id`)
  REFERENCES `cooperations` (`id`

Problem: db/schema.rb does not specify correct primary key types (integer vs. bigint)
Solution: Run rails db:schema:dump

Rails 5.2

Arel.sql

All order and pluck columns should be checked and any non-trivial statement must be wrapped in Arel.sql:

- .order('length(name) asc').first
+ .order(Arel.sql('length(name) asc')).first

Find occurences of order/pluck with a string, check if there is function call or even an SQL injection possibility :) (like order(params[:sort]))

ag "order\('"
ag 'order\("'
ag "pluck\('"
ag 'pluck\("'

Rails 6.0

One very obvious error was the introduction of host checking (against DNS rebinding attacks). We don't need that, as all our apps are proxied in production. In addition, we have dynamic hostnames in development for every developer, so we disable that:

# config/application.rb
config.hosts.clear

Other things we noticed:

scope - cannot contain the class name itself, just plain calls to where/order etc. (no scope :active { User.where(active: true) })
update_attributes -> update simple sed
Tests/specs: content_type -> media_type
Gems:
- in one project, that update line succeeded first: bundle update rails draper devise sass-rails font-awesome-rails annotate premailer-rails
- other gems that produced errors while booting: bundle update coffee-rails slim-rails bullet pry-rails pry-rescue bootsnap airbrake rspec-rails
- gem 'rspec-rails', '~> 4.0.0.beta2'
- gem "cancancan", "~> 3.0"
- will-paginate must be upgraded
If you are needing helpers in non-controller/view files, there was an older snippet on Stackoverflow, to use ActionView::Base.new, replace with: ActionView::Base.new(ActionView::LookupContext.new('.'), {})

Ruby 2.4+

Meanwhile, if you also upgrading Ruby, one error we ran into:

webmock > 2, vcr > 3

undefined method `<3, :continue_timeout=>nil, :debug_output=>nil}:Hash

If you had running VCR version less than 3 and used HTTP Basic auth, you need to convert your cassettes: https://gist.github.com/glaszig/9170b1cf2186674faeead74a68606c5d

Deprecations Fixnum/Bignum -> Integer

Substitute your own usages of "Fixnum" with Integer.
Upgrade all Gems with the error

So far, I will update that post if I find more common stuff.

Integrating Javascript (Vue) I18n into Rails pipeline with missing/auto translate features

Stefan Wienert — Sat, 02 Nov 2019 13:48:00 +0000

Adding I18n features to a backend / "vanilla" Rails app is a common and well documented use case. Awesome tools, like i18n-tasks helping manage to grow the translations by providing tasks like:

Auto translate (via Google Translate),
check for unused/missing translations,
auto sorting translations,
keeping all locales in sync and same structure/order.

Most of our apps nowadays are hybrid apps, though, that are using some amount of Javascript (Vue.js/Angular/React) on the frontside. A full I18n will also have to take that into account. When starting an app, I tend to keep the translations together with the components (like Vue i18n which adds a new block into the single file components). But using this approach, one loses the consistent tooling and translations are starting to sprinkle around the application, which makes adding another locale harder and harder.

For this purpose, I've build the following pipeline, which fulfills the following requirements:

i. fast development cycle - reloading a page reloads the translations also for I18n
ii. separate shipping of locale texts for each locale - e.g. a German user only has to download German locales and not download all
iii. fingerprinting of said separate locales - so that those files can be cached efficiently and at the same time don't get stale
iv. unified storage for all translations and support for the Rails tooling (i18n-tasks)

The high-level overview is the following:

Using i18n-js gem, we can generate a Javascript file for each locale easily. That gem also helps us during development by providing a middleware that reloads those files
We use a namespace in our I18n locale tree, e.g. js. to ship only that tree to the client and not all of that keys
In every layout that requires Javascript translations, we put a little partial, that loads those files and put them into a global Javascript field onto window
The Javascript initializer, in our case Vue-i18n can take that data from the window and load it as locale data
During production asset building, we need to add the generation of those files to our build pipeline
We can add a custom i18n-tasks scanner that helps collecting ALL usages of our Javascript keys to sync which the actual translations (e.g. integrating that into "i18n-tasks unused", "i18n-tasks missing")

The whole process is described in the following flow chart:

i. Adding dependencies and configure I18n-js Gem

Add Gem, bundle, add configuration

# Gemfile
gem "i18n-js"

# config/i18n-js.yml
translations:
  - file: 'vendor/assets/javascripts/locales/%{locale}.js'
    namespace: "LocaleData"
    only: '*.js'

export_i18n_js: "vendor/assets/javascripts/locales/"
js_extend: false # this will disable Javascript I18n.extend globally

Yes, that's right! We are using the old asset pipeline to handle the auto generated translations files, here under vendor/assets/javascript/locales, because that provides a extremely simple fingerprinting and no-fuzz integration into our existing precompile step. To make that work, we need to add all those files to the asset-precompile list, e.g. when using sprockets 4+ in app/assets/config/manifest.js:

//= link locales/en.js
//= link locales/de.js

Also, manually add the I18n-JS Middleware, to enable a seamless development reloading experience.

Rails.application.configure do
  ...
  # config/environment/development.rb
  config.middleware.use I18n::JS::Middleware

Also, add the export path into gitignore, because that files change often and are auto generated anyways:

echo "vendor/assets/javascripts/locales" >> ~/.gitignore

Don't forget, to run the i18n-js Rake export task before deployment, otherwise those files might be missing in your deployment server/ci whatever:

echo "task 'assets:precompile' => 'i18n:js:export'" >> Rakefile

ii. Loading the correct locale data into your Javascript

First, add some example locale. As stated in the introduction, we are using a namespace js.* where all Javascript keys are kept inside. This has the advantage that we can very simply export only the necessary locale data to the client. There is an option in i18n-js, which can hide that "js." namespace in the client, e.g. a Rails i18n key of js.component.button would be accessible as component.button in the frontend. But I've decided against, because that namespace interferes when using I18n-tasks to check for missing/unused in several cases^².

# config/locales/js.en.yml'
en:
  js:
    button_text: "Save"

Now, make sure, every relevant layout, that has translated Javascript on it, loads the locale before everything else, e.g. using a layout partial which you require before loading any other Javascript.

<!-- app/views/layouts/_i18n_loader.html.erb -->
<script>
window.Locale = <%= raw I18n.locale.to_json %>
window.LocaleData = {}
</script>

<%= javascript_include_tag "locales/#{I18n.locale}" %>

Then, our pack / framework initializer can pick that up. In our case, we are using Vue-i18n:

// app/javascripts/utils/i18n.js
import Vue from 'vue'
import VueI18n from 'vue-i18n'

Vue.use(VueI18n)

const i18n = new VueI18n({
  locale: window.Locale,
  fallbackLocale: 'de',
})

const rawData = window.LocaleData.translations[window.Locale]
i18n.setLocaleMessage(window.Locale, rawData)

export default i18n

Now we can include that i18n key when initializing every Vue root "app" on our site:

// app/javascripts/packs/app.js
import i18n from 'utils/i18n'

new Vue({
  el,
  i18n,
})

That's it! Every component can now access locale data, like that:

<template>
  <!-- passing as efficient v-t handler -->
  <button v-t="'js.button_text'"></button>

  <!-- using i18n arguments -->
  <button>{{ $t('js.text_with_arguments', { name: name }) }}
</template

Make sure to reference the documentation^³ of Vue-i18n, as there are some differences to Rails built-in message style:

Passing arguments : Instead of "somekey %{argument}" use somekey {argument}
Pluralization : instead of:

somekey:
  one: "1 car"
  other: "%{count} cars"

use: somekey: 1 car | {n} cars or with "nothing": somekey: no car | {n} car | {n} cars

iii. Adding I18n-Tasks Scanner Adapter

Translations should work find now. To make I18n-Tasks aware of our locales, use this scanner:

# lib/vue_i18n_scanner.rb
require 'i18n/tasks/scanners/file_scanner'
require 'pry'

# finds v-t="" and $t(c) usages
class VueI18nScanner < I18n::Tasks::Scanners::FileScanner
  include I18n::Tasks::Scanners::OccurrenceFromPosition

  KEY_IN_QUOTES = /(?["'])(?[\w\.]+)(?["'])/.freeze
  WRAPPED_KEY_IN_QUOTES = /(?["'])#{KEY_IN_QUOTES}(?["'])/.freeze

  # @return [Array]
  def scan_file(path)
    text = read_file(path)
    # single file component translation used
    return [] if text.include?('<i18n')

    out = []
    # v-t="'key'" v-t='"key"'
    text.to_enum(:scan, /v-t=#{WRAPPED_KEY_IN_QUOTES}/).each do |_|
      key = Regexp.last_match[:key]
      occurrence = occurrence_from_position(
        path, text, Regexp.last_match.offset(:key).first
      )
      out << [key, occurrence]
    end
    text.to_enum(:scan, /\$tc?\(#{KEY_IN_QUOTES}/).each do |_|
      key = Regexp.last_match[:key]
      occurrence = occurrence_from_position(
        path, text, Regexp.last_match.offset(:key).first
      )
      out << [key, occurrence]
    end
    out
  end
end
I18n::Tasks.add_scanner 'VueI18nScanner', only: %w(*.vue)

And load that scanner in your config/i18n-tasks.js

echo "<% require './lib/vue_i18n_scanner' %>" >> config/i18n-tasks.js

Now, i18n-tasks health/unused/missing will scan our javascripts too.

Used Open-Source:

glebm / i18n-tasks

Manage translation and localization with static analysis, for Ruby i18n

i18n-tasks

i18n-tasks helps you find and manage missing and unused translations.

This gem analyses code statically for key usages, such as I18n.t('some.key'), in order to:

Report keys that are missing or unused.
Pre-fill missing keys, optionally from Google Translate or DeepL Pro.
Remove unused keys.

Thus addressing the two main problems of i18n gem design:

Missing keys only blow up at runtime.
Keys no longer in use may accumulate and introduce overhead, without you knowing it.

Installation

i18n-tasks can be used with any project using the ruby i18n gem (default in Rails).

Add i18n-tasks to the Gemfile:

gem 'i18n-tasks', '~> 0.9.33'

Copy the default configuration file:

$ cp $(i18n-tasks gem-path)/templates/config/i18n-tasks.yml config/

Copy rspec test to test for missing and unused translations as part of the suite (optional):

$ cp $(i18n-tasks gem-path)/templates/rspec/i18n_spec.rb spec/

Or for minitest:

$ cp $(i18n-tasks gem-path)

…

View on GitHub

fnando / i18n-js

It's a small library to provide the I18n translations on the Javascript. It comes with Rails support.

It's a small library to provide the Rails I18n translations on the JavaScript

Features:

Pluralization
Date/Time localization
Number localization
Locale fallback
Asset pipeline support
Lots more! :)

Version Notice

The main branch (including this README) is for latest 3.0.0 instead of 2.x.

Usage

Installation

Rails app

Add the gem to your Gemfile.

gem "i18n-js"

Rails with webpacker

If you're using webpacker, you may need to add the dependencies to your client with:

yarn add i18n-js
# or, if you're using npm
npm install i18n-js

For more details, see this gist.

Rails app with Asset Pipeline

If you're using the asset pipeline, then you must add the following line to your app/assets/javascripts/application.js.

//
// This is optional (in case you have `I18n is not defined` error)
// If you want to put this line, you must put it BEFORE `i18n/translations`
//= require i18n
// Some people

…

View on GitHub

kazupon / vue-i18n

🌐 Internationalization plugin for Vue.js

vue-i18n

Internationalization plugin for Vue.js

🥇 Gold Sponsors

🥈 Silver Sponsors

🥉 Bronze Sponsors

📢 Notice

vue-i18n will soon be transferred to intlify organization. After that, it will be developed and maintained on intlify.

The vue-i18n that has been released on npm will be released as @intlify/vue-i18n in near future.

vue-i18n next major version repo is here

Intlify is a new i18n project kickoff by @kazupon . 😉

📖 Documentation

See here

📜 Changelog

Detailed changes for each release are documented in the CHANGELOG.md.

❗ Issues

Please make sure to read the Issue Reporting Checklist before opening an issue. Issues not conforming to the guidelines may be closed immediately.

💪 Contribution

Please make sure to read the Contributing Guide before making a pull request.

©️ License

MIT

View on GitHub

2. Adding the prefix in our custom scanner while scanning the files was a way that worked, BUT the default I18n-scanner already picked up some of the keys by it's own regex scanner ($t(..)-usages) which then would then be marked as missed. There seemed to be no way to ignore the app/javascript/ for only the default scanner but only for all.

3. https://kazupon.github.io/vue-i18n/guide/formatting.html#named-formatting

Simple but realistic Elasticsearch load test before deploying

Stefan Wienert — Mon, 14 Jan 2019 09:00:00 +0000

Load testing an Elasticsearch cluster before a migration, upgrade or similar is always a good strategy to reduce bad surprises afterwards. With pludoni GmbH, we use Elasticsearch since 2014 for our Job search backend for all of our (German) community websites (Empfehlungsbund), as well as blog article search and for keyword optimizations.

Recently, I've prepared such a migration and needed a way to verify that the cluster holds after switching, of better yet, improving the performance in regards to the costs. After checking Github et. al. for other specific code, I was not satisfied enough and build some small script around the awesome siege tool.

Preparation - Gather good queries for test

To get a realistic performance test, I suggest to grab original payloads of queries that your production ES runs. Even more, I only took a couple of queries that are the slowest to boost my confidence in the end.

To to that, first enable Slow Log in ES settings in your UI (cerebro/kopf whatever frontend) or via curl:

curl 'http://localhost:9200/index_settings/update' \
  -H 'Accept: application/json' \
  -H 'Content-Type: application/json;charset=utf-8' --data \
  '{"index":"CLUSTERNAME","settings":{"index.search.slowlog.threshold.query.warn":"1s","index.search.slowlog.threshold.query.info":"0.5s"},"host":"http://localhost:9200"}'

Then, wait a while or produce slow logs via querying. Afterwards the file /var/log/elasticsearch/*_index_search_slowlog.log will fill up.

Extract the query payloads by copying all json between brackets:

[2019-12-11 04:30:29,725][INFO][index.search.slowlog.query] [es01.localhost] [ebsearch_production][2] took[1.7s],
  took_millis[1774], types[], stats[], search_type[QUERY_THEN_FETCH], total_shards[5], source[<>], extra_source[],

Put each payload each in individual file in a common folder, e.g. payloads/1, payloads/2 etc.

Build new test cluster

One hint if not yet used: Use Repository + Snapshots (S3) to quickly seed a new cluster with production grade data.

Test the (old/new) cluster

The test is run by the battle-tested tool Siege , which should be easy to install from all your OS repo (Apt, Brew, etc.). Siege supports a input parameter with a file with urls to test. Later, we will utilize Siege like that:

# Concurrency: 3, for 1 minute
siege -b --log=./siege.log -H 'Content-Type: application/json' \ 
  --internet --delay=15 -c 3 -t 1M --file=urls.txt

The urls.txt has the format:

http://server/index/_search POST {".....search payload"}

To generate urls.txt easily with all of the payloads, I've created a Rakefile, because Ruby is awesome. Also, we are living in 2019(+), so the Ruby that shipped with your distro should be just fine, no rvm/rbenv needed.

SERVER = '10.10.10.100:9200'.freeze
DURATION = '1M'.freeze # 1 minute each test
CONCURRENCY_TESTS = (1..10) # or [1, 5, 10, 20, 100] etc.
INDEX_NAME = 'ebsearch_production'

desc 'create urls.text file with all payloads in payloads/*'
task :urls do
  out = Dir['payloads/*'].map do |pl|
    "http://#{SERVER}/#{INDEX_NAME}/_search POST #{File.read(pl)}"
  end
  puts "recreating urls.txt for #{SERVER} with #{out.count} requests"
  File.write('urls.txt', out.join("\n"))
end

desc 'run series!'
task :run do
  File.unlink('siege.log')
  (1..MAX_CONCURRENCY).each do |c|
    puts "==== #{c} Concurrent ==== "

    sh %[siege -b -m "#{SERVER}-C#{c}" --log=./siege.log -H 'Content-Type: application/json' --internet --delay=15 -c #{c} -t #{DURATION} --file=urls.txt]
  end
end

desc 'show csv as tsv for copy paste into google spreadsheets'
task :csv do
  lines = File.read('siege.log').lines
  csv = lines.reject { |i| i.include?(" ****") }.map { |line| line.gsub(',', '').gsub('.', ',') }.join
  puts csv
end

task default: [:urls, :run]

Modify the params in the header of the file
Run it! rake
After it is finished (CONCURRENCY_TESTS * DURATION), you can output the data: rake csv and copy the output in e.g. Google Spreadsheets to easily generate charts

Bonuspoints: quick chart with ascii-charts

Install bundler, if not done yet gem install bundler

Append to Rakefile:

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'ascii_charts'
end
desc 'chart'
task :chart do
  require 'ascii_charts'
  lines = File.read('siege.log').lines
  csv = lines.reject { |i| i.include?(" ****") || i.include?('Elap Time') }
  data = csv.map { |i| i.split(',').map(&:strip) }.map { |a| %w[date transactions duration transfer response_time requests_s mbs conc success failed].zip(a).to_h }
  require 'pry'
  puts "======= Response Time / Concurrency ========"
  items = data.each_with_index.map { |d| [d['conc'].to_f.round, d['response_time'].to_f] }
  puts AsciiCharts::Cartesian.new(items).draw

  puts "======= Requests/s / Concurrency ========"
  items = data.each_with_index.map { |d| [d['conc'].to_f.round, d['requests_s'].to_f] }
  puts AsciiCharts::Cartesian.new(items).draw
end

Anecdote: results for our smallish cluster

Our search used custom search plugins that are quite CPU intensive, especially with long queries. Overall our concurrent users are not that many, so a 3-4 node cluster is generally enough.

Deployment target is the very cost efficient Hetzner Cloud (HCloud). Here a quick overview over the different cloud instance types Hetzner offers at this point (2019):

CX11 (1 Core, 2GB, 3 EUR)
CX21 (2 Core, 4GB, 6 EUR)
CX31 (2 Core, 8GB, 11 EUR) (not included, because no CPU improvement and RAM is not utilized)
CX41 (4 Core, 16GB, 19 EUR)
CX51 (8 Core, 32GB, 36 EUR)

I've tried the following combinations in the Hetzner cloud. Please note, that if the rq/s looks ridiculous low, but please keep in mind that those 10 concurrent users are only searching with the worst queries that I could found.

nodes	EUR/month	rq/s @ 10ccu	response time @ 10 ccu	requests/s/EUR
1x Coordinator (CX11) + 2x Data CX21	15 EUR	5.90	1.67	0.39
1x Coordinator (CX11) + 3x Data CX21	21 EUR	4.84	2.03	0.23
1x Coordinator (CX11) + 2x Data CX41	41 EUR	11.24	0.80	0.27
1x Coordinator (CX11) + 3x Data CX41	60 EUR	17.19	0.58	0.28
1x Coordinator (CX11) + 4x Data CX41	79 EUR	16.56	0.54	0.20

My findings:

Even the smallest instance seemed to be fine for the Coordinator node, the CPU usage never reached any kind of utilization number
Going from 2 CX21 to 3 CX21 did not improve the core metrics (reqs/s, response time), but worsening it. My conclusion is, that the CX21 has too low CPU power to
Same, going to 4 CX41 seems to be worse than 3 CX41
2 or 3 CX41 are best performance for the price
CX51 untested
PLEASE NOTE: That findings could be totally related to our type of querying which includes custom search algorithm written in Groovy, Also: I am not a Elasticsearch expert, so there might be tuning params, sharding settings that could be adjusted.