DEV Community: JetRockets

Scaling Rails: Docker & AWS Beanstalk

Igor Alexandrov — Wed, 20 May 2020 11:44:00 +0000

Amazon AWS has always been our preferred infrastructure provider. We love to use AWS EC2 for our applications, AWS RDS for databases, and AWS ElastiсCache as the ‘key-value’ storage.

Terminology
Due to NDA and security-related restrictions we are unable to use actual project and client names, so here is some terminology we will use in this article:

X-Project – this will be the name of our project;
x-project.com – the main domain;
git://x-project.com/x-project-server – the repository with the Rails application;
git://x-project.com/x-project-ci-image – the repository with the Docker images;

For a better experience of exploring the code, we pulled all the config files out to a separate repository: https://github.com/jetrockets/rails-elastic-beanstalk. This repository contains two directories:

x-project-ci-image contains files that were used in git://x-project.com/x-project-ci-image;
x-project-server contains the files that were added to our Rails application from git://x-project.com/x-project-server.

Inside x-project-server/eb/production we have configs of all three of our EBS projects. An EBS project usually consists of two main components:

.elasticbeanstalk directory with the global scenario files for the project configuration and the Elastic Beanstalk environment: name, instance types, variables of the environment and the environment settings;
.ebextensions directory which contains a collection of scenario files for instance configuration. These files represent a set of instructions that are loaded automatically upon creation of a new instance or update of an existing one.

Use Case
Usually we deploy our servers on EC2 using Ansible scripts and Capistrano, which is a common approach. However, this approach may lead to a state that Martin Fowler referred to as SnowflakeServer. Over time, the configuration of servers starts to have too many dependencies and it becomes very difficult to replicate quickly. That's why even if you're not using Docker on EC2 machines, it's vital to make all changes to the server by modifying the Ansible scripts.
This ensures that if necessary, you can deploy a new environment relatively quickly.

One of our recent projects was originally focused on a high load that was difficult to predict. We were building a chat room for online stores with enhanced retail sales capabilities. It was a great opportunity to try Elastic Beanstalk. To understand why, let's first look at what it is.

AWS Elastic Beanstalk is an AWS manipulation service. A preset configuration can quickly set up several application instances, link them to a database or a cache, and set up a balancer between them. With EBS you can run the application code on several platforms: Puma or Passenger for Ruby, Python, Java, and of course you can run Docker containers. We would be satisfied with the Docker container option, so let's explore this further.

Architecture
Our application was built using the following tech stack:

A Rails application (web and Sidekiq);
AnyCable for a websocket chat;
Embeddable Widgets in HTML and JS which are integrated into client e-commerce platforms;

The project required at least two environments: Production (for the live system) and Integration (for demo and testing purposes).

Let’s start with the Rails app.

Configuration
By default, Rails 6 stores credentials in separate files specific to each environment. To learn more about this, go to Rails Guides. We opted to use environment variables for managing settings, because this approach could provide certain advantages when working with AWS. For this, we used a gem dotenv-rails.

# git://x-project.com/x-project-server/Gemfile

gem 'dotenv-rails'

Environment variables are added from the .env file in the application root, which is convenient for local development. Moreover, we always store the .env.sample file inside the project, which contains a template for all project settings.

In our experience, using .env is a more convenient, clear, and consistent way to configure Rails, rather than using rails credentials. You can define the settings in a file, copy the finished file, or run Rails and transfer the settings directly using the command line.

Container Registry
Amazon offers a container management tool – Amazon Elastic Container Registry. Amazon ECR is a fully automated Docker container registry that makes it easy for developers to store, deploy, and manage Docker container images.

Getting Ready

To work with git, we use a self-hosted version of GitLab, so we wanted to integrate GitLab & GitLab CI into the AWS infrastructure.
We decided to store all the application settings in AWS. This means that GitLab-runner should not have access to these settings and so we had to make a second docker image, which receives the keys to work with AWS while the app is building. To do this, the following Dockerfile should be added to a separate repository.

# git://x-project.com/x-project-ci-image/aws-utils/Dockerfile

FROM docker:latest     

ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_REGION

RUN apk add --no-cache python py-pip python-dev build-base libressl-dev musl-dev libffi-dev git
RUN pip install awsebcli awscli s3cmd

RUN  aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
RUN  aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY 
RUN  aws configure set region $AWS_REGION

The settings for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY & AWS_REGION will be based on the GitLab environment variables while the app is building. To accomplish this we created a user in the Amazon IAM with the required permissions. In our case, we selected AWSElasticBeanstalkFullAccess (to work with EC2, S3, RDS, and other services), AWSElasticBeanstalkMulticontainerDocker (to work with Docker images) & ECRFullAccess (a custom policy with full ECR access). As a result of the app build, this container will be able to work with AWS API according to the set permissions, so the main application will be built on its basis.

To make it easier to run tests and release the app, we decided to build an intermediate Docker image in order to launch rspec and collect images to run Rails & Sidekiq. This image already has everything you need: nodejs & yarn for compiling JS & CSS, and client PostgreSQL for connecting to RDS. It is also necessary to remember about static distribution, therefore we’ll add Nginx. We use ruby 2.6.5, as Rails (at the time of writing this article) still generates a large number of warnings with ruby 2.7.0.


# git://x-project.com/x-project-ci-image/app-image/Dockerfile

ARG RUBY_VERSION_IMAGE=2.6.5
FROM ruby:${RUBY_VERSION_IMAGE}-slim-buster

ARG APP_PATH="/app"
ARG APP_USER=app
ARG NODE_VERSION=12
ARG POSTGRES_VERSION=12

RUN adduser --disabled-password --home ${APP_PATH} --gecos '' ${APP_USER}

RUN mkdir -p ${APP_PATH} && \
    chown ${APP_USER}. ${APP_PATH}

# Install packages
RUN apt-get update -qq && \
    apt-get install -yq --no-install-recommends \
    curl \
    gnupg2 \
    lsb-release \
    gcc \
    g++ \
    git \
    openssh-client \
    libxml2-dev \
    libxslt-dev \
    libjemalloc2 \
    make \
    nginx \
    pkg-config \
    file \
    imagemagick

# Nginx settings
RUN touch /run/nginx.pid && \
    chown -R ${APP_USER}. /run/nginx.pid && \
    chown -R ${APP_USER}. /var/log/nginx && \
    chown -R ${APP_USER}. /var/lib/nginx && \
    chown -R ${APP_USER}. /etc/nginx/conf.d && \
    chown -R ${APP_USER}. /usr/share/nginx && \
    unlink /etc/nginx/sites-enabled/default

# Forward request logs to Docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
 && ln -sf /dev/stderr /var/log/nginx/error.log

# Add Node.js repo
RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \
    echo "deb https://deb.nodesource.com/node_${NODE_VERSION}.x $(lsb_release -s -c) main" | tee /etc/apt/sources.list.d/nodesource.list && \
    echo "deb-src https://deb.nodesource.com/node_${NODE_VERSION}.x $(lsb_release -s -c) main" | tee -a /etc/apt/sources.list.d/nodesource.list

# Add yarn repo
RUN curl -fsSL https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
    echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list

# Install packages
RUN apt-get update -qq && \
    apt-get install -qy --no-install-recommends \
    nodejs \
    yarn

RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - &&\
    echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -sc)-pgdg main" | tee /etc/apt/sources.list.d/postgresql.list

# Install dependency for gem pg
RUN apt-get update -qq && \
    apt-get install -qy --no-install-recommends \
    libpq-dev \
    postgresql-client-${POSTGRES_VERSION}

COPY --chown=${APP_USER}:${APP_USER} . ${APP_PATH}

USER ${APP_USER}

ENV GEM_HOME=${APP_PATH}/bundle \
    BUNDLE_PATH=${APP_PATH}/bundle \
    BUNDLE_BIN=${APP_PATH}/bundle/bin \
    PATH=${APP_PATH}/bin:${APP_PATH}/bundle/bin:/usr/local/bin:$PATH

WORKDIR ${APP_PATH}

# Install dependences
# Ruby-2.6.5 supplied with bundler:1.17.2
RUN gem install bundler -v '2.1.4'

RUN bundle install --jobs=$(nproc)
RUN yarn install --check-files

The build of both images is performed in the following scenario.

# git://x-project.com/x-project-ci-image/.gitlab-ci.yml

variables:
   RUBY_VERSION_IMAGE: "2.6.5"
   NODE_VERSION: "12"
   APP_PATH: "/app"
   APP_USER: "app"

stages: 
  - build

build_image:
  # Docker image with pre-installed docker package 
  image: docker:latest
  stage: build
  before_script:
    # login to git://x-project.com/x-project-ci-image and pull images from regestry
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # Pull all images for use caching. 
    # Statement "|| true" used if images not found and pipeline not failing
    - docker pull $CI_REGISTRY/x-project/x-project-ci-image/app-image:latest || true
    - docker pull $CI_REGISTRY/x-project/x-project-ci-image/aws-utils:latest || true

  script:
    # Build docker image with aws utils and and aws secrets
    - docker build --cache-from $CI_REGISTRY/x-project/x-project-ci-image/aws-utils:latest
      -t $CI_REGISTRY/x-project/x-project-ci-image/aws-utils
        --build-arg AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} 
        --build-arg AWS_REGION=${AWS_REGION}
        --build-arg AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
        aws-utils

    # Build docker image with pre-installed linux packages, gems and frontend packages    
    - docker build --cache-from $CI_REGISTRY/x-project/x-project-ci-image/app-image:latest 
      -t $CI_REGISTRY/x-project/x-project-ci-image/ruby-image
        --build-arg APP_PATH=${APP_PATH}
        --build-arg APP_USER=${APP_USER}
        --build-arg NODE_VERSION=${NODE_VERSION}
        --build-arg RUBY_VERSION_IMAGE=${RUBY_VERSION_IMAGE}
        app-image

    # login to git://x-project.com/x-project_ci_image for push images
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

    # Push images to git://x-project.com/x-project_ci_image in regestry
    # See git://x-project.com/x-project_ci_image/container_registry
    - docker push $CI_REGISTRY/x-project/x-project-ci-image/aws-utils:latest    
    - docker push $CI_REGISTRY/x-project/x-project-ci-image/app-image:latest

Everything described above can be illustrated in the following diagram.

Building an App

We use GitLab-runner for building as well as for the intermediate image.
Building and deployment of the project begin after the code gets into the master or integration branch. Currently, our pipeline consists of five stages.

Let’s skip notify & rspec to take a closer look at how build & deploy are going.

#.gitlab-ci.yml

docker-build:
  image: $BASE_IMAGES_URL:$CI_PROJECT_NAMESPACE-aws-utils
  stage: build
  environment:
    name: $CI_COMMIT_BRANCH
  only:
    - master
    - integration
  script:
    - $(aws ecr get-login --no-include-email --profile default)
    - aws s3 cp s3://s3.environments.x-project.com/.env.$RAILS_ENV .env
    - aws s3 cp s3://s3.environments.x-project.com/$RAILS_ENV.key config/credentials/$RAILS_ENV.key
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $BASE_IMAGES_URL:$CI_PROJECT_NAMESPACE-app-image
    - docker build -t $REPOSITORY_URL:$REPOSITORY_TAG
      -t $REPOSITORY_URL:$RAILS_ENV-latest
      --build-arg APP_PATH=$APP_PATH
      --build-arg DATABASE_URL=$DATABASE_URL
      --build-arg RAILS_ENV=$RAILS_ENV
      --build-arg REDIS_URL=$REDIS_URL
      --build-arg S3_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID
      --build-arg S3_BUCKET=$S3_BUCKET
      --build-arg S3_REGION=$S3_REGION
      --build-arg S3_SECRET_KEY=$S3_SECRET_KEY
      --build-arg SECRET_KEY_BASE=$SECRET_KEY_BASE
      --build-arg WEBHOOKS_API_URL=$WEBHOOKS_API_URL
      --build-arg WIDGET_SRC=$WIDGET_SRC .
    - docker push $REPOSITORY_URL:$REPOSITORY_TAG

    # "Hypervisor" settings
    - docker tag $REPOSITORY_URL:$REPOSITORY_TAG $REPOSITORY_URL:$RAILS_ENV-latest
    - docker push $REPOSITORY_URL:$RAILS_ENV-latest

To build the image we use the previously prepared aws-utils image with ready-to-go settings for aws. Then we copy the file with the environment variables from S3 and run the build based on app-image, which we also built earlier. Thus, we get a universal container that can run rails, sidekiq, & anycable-rails.
To be able to quickly roll back to the previous version of the application without doing an additional build, we use tags with a hash of the commit and the current environment (for example production-2a4521a1). Each docker image has a tag with the current version marked as :latest. The same tag is used to run in hypervisors (we’ll talk about them below). Accordingly, a rollback to any version of the system that is stored in the registry takes no more than a minute.

Pushing configuration files to AWS EB

So, we have collected the images and now should only deliver them to the Elastic Beanstalk.

docker-deploy:
  image: $BASE_IMAGES_URL:$CI_PROJECT_NAMESPACE-aws-utils
  stage: deploy
  environment:
    name: $CI_COMMIT_BRANCH
  only:
    - master
    - integration
  script:
    - ./bin/ci-deploy.sh

For convenience, we wrote a small script that runs in the aws-utils image and therefore can work with the AWS API. So, all that is left to do is run eb deploy command with the corresponding parameters in the directory, where the .elasticbeanstalk configuration file is located.
At this stage we have already received an image with a tag in the format RAILS_ENV-SHORT_SHA_COMMIT; it remains to specify ElasticBeanstalk in the configuration file, so that it downloads this particular version. To do that, let’s use sed and replace the tag with RAILS_ENV-SHORT_SHA_COMMIT.

# bin/ci-deploy.sh

#!/bin/sh
xargs -P 3 -I {} sh -c 'eval "$1"' - {} << "EOF"
cd eb/$RAILS_ENV/rails; sed -i 's/tag/'$RAILS_ENV-$CI_COMMIT_SHORT_SHA'/' Dockerrun.aws.json; eb deploy --profile default --label $LABEL-rails --timeout 30

cd eb/$RAILS_ENV/sidekiq; sed -i 's/tag/'$RAILS_ENV-$CI_COMMIT_SHORT_SHA'/' Dockerrun.aws.json; eb deploy --profile default --label $LABEL-sidekiq --timeout 30

cd eb/$RAILS_ENV/anycable-rails;  sed -i 's/tag/'$RAILS_ENV-$CI_COMMIT_SHORT_SHA'/' Dockerrun.aws.json; eb deploy --profile default --label $LABEL-anycable-rails --timeout 30
EOF

After successfully uploading the image to Elastic Beanstalk each container starts according to the script from Dockerrun.aws.json.

Balancers and Autoscaling

Now that we have prepared the environments and configured the deploy process, we are fully ready for production. Running a final performance check, we are seeing WebSocket connection errors.

A couple of hours of checking EB components and we discover that the AWS Application Load Balancer doesn’t want to route traffic to the anycable-rails port. This is due to a health check failure: incorrect response from the port (204 or 200). Dealing with anycable-rails, it’s clear that it runs with 50051 port and responds via HTTP2/gRPC by default, while ALB is waiting for a reply via HTTP. HTTP health check can be configured in anycable-rails with the --http-health-port option. However, ALB is not able to do a health check for one port and proxy traffic to another. At this point we realize that we could try to proxy traffic to anycable-rails through Nginx as well as parse the requests for health checks.

After reconfiguring Nginx, and re-checking, the ALB accepts our response for a health check, and traffic successfully goes to the container with anycable-rails. However, right away, we are running into a new error at anycable-go.

context=ws Websocket session initialization failed: rpc error: code = Unavailable
desc = all SubConns are in TransientFailure, latest connection error: connection
closed

So we go back to anycable-rails to carefully check the logs.

"PRI * HTTP/2.0" 400 173 "-" "-"

An idea occurs that there might be a problem with ALB, which disassembles HTTPS on its side but sends requests to the host via HTTP. After creating a self-signed certificate and configuring Nginx to listen to the port in order to forward gRPC to HTTP/2 with the certificate, we get another error.

"PRI * HTTP/2.0" 400 173 "-" "-”

We continued EB configuration, made full-fledged end-2-end encryption, but unfortunately this didn’t help. Finally, we tried to work with anycable-rails directly without ALB, and it didn’t work out either. It became clear that the problem was in ALB, though according to the documentation ALB supports HTTP/2. While ALB parses HTTP/2 traffic on its side, it sends requests to the host via HTTP/1.1, and gRPC uses HTTP/2.
Then we remembered that apart from ALB, Elastic Beanstalk has its own second balancer – NLB, which does nothing with traffic, unlike ALB. We decided to try it out. Besides dividing application and anycable we can be more accurate with autoscaling. As a result, we arrived at the following scheme for our service.

Hypervisors

Elastic Beanstalk does not imply the use of an EC2 instance as we are accustomed to: you can, of course, connect via ssh with a key, but setting up a system of rights through IAM will be problematic. Only administrators should have access to such instances in order to resolve extraordinary cases.
And the first question is how to give developers access to the console rails.

We decided to make two micro-instances (t2.micro) on EC2, one for each environment, and called them hypervisors. Developers can use this environment to perform various tasks: launching rails console, connecting to RDS or ElasticCache, viewing logs, and so on.

We deployed Docker to both instances, so AWS IAM created the user (registry-user) with two policies: AmazonEC2ContainerRegistryReadOnly & CloudWatchReadOnlyAccess. The selected policies allow us to download the Docker image from the registry and view CloudWatch logs. To make these operations easier we wrote two tooling bash scripts.

Tooling

The first script accesses the rails console. The integration and production options differ only in ECR addresses. It is important in CI to make integration-latest & production-latest tags, to download the current image in the script.

#!/bin/bash
COMMAND=$@
if test -z $COMMAND; then
  COMMAND="rails c"
fi

$(aws ecr get-login --no-include-email --profile default)
docker pull some-address.dkr.ecr.us-east-1.amazonaws.com/x-project-server:integration-latest
docker run -it --rm some-address.dkr.ecr.us-east-1.amazonaws.com/x-project-server:integration-latest $COMMAND

The second script allows viewing logs from various instances in the console. Docker sends all logs to stdout by default, no one can read them there, so we add them to the file, and stream the file to CloudWatch. The problem is Docker doesn’t send multi-container logs to CloudWatch. To solve this you need to add .ebextensions settings for the CW agent and configure Dockerrun.aws.json for logging.

Now when all the logs are in CloudWatch, tooling can be built. To get the logs we use awslogs. Below is a sample script for integration.

#!/bin/bash

service=$1
minutes=$2
if test -z $minutes ;then
  minutes=20
fi

case $service in
  rails)
    awslogs get /aws/elasticbeanstalk/Application-integration/var/log/containers/app/integration.log --start=${minutes}'m ago'
  ;;
  anycable)
    awslogs get /aws/elasticbeanstalk/Anycable-rails-integration/var/log/containers/RPC/integration.log --start=${minutes}'m ago'
  ;;
  nginx)
    awslogs get /aws/elasticbeanstalk/Application-integration/var/log/containers/nginx/access.log --start=${minutes}'m ago'
    echo "↓--- NGNIX ERROR LOGS ---↓"
    awslogs get /aws/elasticbeanstalk/Application-integration/var/log/containers/nginx/error.log --start=${minutes}'m ago'
  ;;
  sidekiq)
    awslogs get /aws/elasticbeanstalk/Sidekiq-integration/var/log/eb-docker/containers/eb-current-app/stdouterr.log --start=${minutes}'m ago'
  ;;
  *)
    echo "Supported serices rails, anycable, nginx, sidekiq."
    echo "Usage ./logs ralis 30"
    echo "Output will be logs from rails env from the last 30 minutes"
    echo "If output is emply, it seems last N (default 20) minuters no logs from app"
    exit 1
  ;;
esac

Key Takeaways

AWS Elastic Beanstalk seems to be the best solution for scalable Rails hosting. After complex first time configuration every new project can be deployed to it within one hour.

The most valuable thing is that developers don’t need any additional help from dev-ops team to scale their apps. All needed information about current application performance can be found in AWS dashboards and scaling configuration can be edited directly in web-interface. Another good point is that if your project has peak load only a few days within a month (e.g. report generation by the end of month) you will not pay for not needed instances during the whole month.

Migrating user passwords from Django to Ruby

Igor Alexandrov — Wed, 19 Feb 2020 06:40:34 +0000

One of our clients asked us to migrate his existing Django application to Ruby. A part of this process was a migration of an existing users database.
Of course we had to find a way to use existing crypto passwords from Django and not asking our users to reset them

Passwords in Django are stored as a string that consists of four parts splitted with a dollar sign.

<algorithm>$<iterations>$<salt>$<hash>

By default, Django uses the PBKDF2 algorithm with a SHA256 hash. I found a rather outdated Ruby gem that implements a Password-Based Key-Derivation Function and gave it a try.

def migrate_django_password_seamlessly(user, password)
  alg, iteration, salt, django_password = user.crypted_django_password.split('$')
  attempt = Base64.encode64(
    PBKDF2.new(password: password, salt: salt, iterations: 36000).bin_string
  ).strip

  # check if hash of user provided password equals to the password in a database
  if attempt == django_password
    user.update(password: password, password_confirmation: password)
  end
end

The method above is a part of user sign in service and called only if crypted_django_passwordcolumn from users table is not null.

Two edge cases in PostgreSQL full-text search

Igor Alexandrov — Thu, 09 Jan 2020 17:30:38 +0000

We widely use PostgreSQL full-text search in our projects. It is fast, reliable, and doesn't add any additional technical complexity. But sometimes it may not work as you expect it too.

Usually we add a ts_vector column right in our data tables. Let's see how it has it is done.

title	tsv
Handyman	‘handyman’:1
Heating & Cooling	‘cool’:2 ‘heat’:1
Painting	‘paint’:1
Roofing	‘roof’:1

Column tsv is generated with to_tsvector function from a title column with a trigger function.

CREATE FUNCTION skills_tsv_update_trigger() RETURNS trigger AS $$
begin
  new.tsv :=
    to_tsvector('pg_catalog.english', COALESCE(new.title, ''))
  return new;
end
$$ LANGUAGE plpgsql;

Then we can query this table using to_tsquery fuction.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'roof')

-- 1 record found

A record with Roofing has been found, and this makes sense. But what should be found for handy query? You can imagine that it will be a Handyman skill and you are wrong. Before explaining why this happens, let's try another query.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'roofing')

-- no records found

No records were found, even if we have a record with Roofing title in our table.

The to_tsvector function internally calls a parser which breaks the document text into tokens and assigns a type to each token. For each token, a list of dictionaries is consulted, where the list can vary depending on the token type. The first dictionary that recognizes the token emits one or more normalized lexemes to represent the token. For example, roofing became roof. Some words are recognized as stop words, like & in our example. It is possible to have many different configurations in the same database, and predefined configurations are available for various languages. In our example we used the default configuration english for the English language.

to_tsquery function parses user input and converts it to tsquery data type. It uses the same dictionaries as to_tsvector function. Let's try to understand what happens in two queries that were listed above.

The result of to_tsquery('pg_catalog.english', 'handy') will be handi because of pluralization rules for words ending on y. The query listed below doesn't return any records because 'handyman':1 vector does not overlap with handi query.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'handy')

-- no records found

How we can cover this situation? As you remember you can change a configuration that is used to parse user input. Let's try a simple configuration.

select to_tsquery('pg_catalog.simple', 'handy')

-- 'handy'

Looks like this is what we need, let's add this to our query.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'handy')
    OR
    tsv @@ to_tsquery('pg_catalog.simple', 'handy')    

-- 1 record found

A problem with Roofing and roof can be solved in the same way. Let's change the function that is used to populate tsv column.

CREATE FUNCTION skills_tsv_update_trigger() RETURNS trigger AS $$
begin
  new.tsv :=
    to_tsvector('pg_catalog.english', COALESCE(new.title, '')) ||
    to_tsvector('pg_catalog.simple', COALESCE(new.title, ''))
  return new;
end
$$ LANGUAGE plpgsql;

title	tsv
Handyman	‘handyman’:1,2
Heating & Cooling	‘cool’:2 ‘cooling’:4 ‘heat’:1 ‘heating’:3
Painting	‘paint’:1 ‘painting’:2
Roofing	‘roof’:1 ‘roofing’:2

Now vector has more data and query below works as expected.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'roofing')

-- 1 record found

How to add HTTP Basic auth to Amber application

Igor Alexandrov — Mon, 25 Nov 2019 21:44:17 +0000

As you already may know, one of our projects have a Crystal application in production. It is created with Amber framework and works just perfect.

The only thing that I don't personally like in Amber is not clear and sometimes outdated documentation, after about 11 years with Rails I still think that Rails Guides are the number one developer documentation in the world.

I had a task to add HTTP Basic auth to a couple of URLs in Amber application, and after studying documentation found that Amber doesn't provide necessary Pipe out of the box. Ok, next place to search for the answer was Gitter and after about an hour Dru Jensen helped me with a code example.

Amber uses internally HTTP::Handler for Pipes as well as Kemal does for Middlewares, so we can easily use code from Basic Auth for Kemal.

# src/pipes/http_basic_auth_pipe.cr

require "crypto/subtle"

class HTTPBasicAuthPipe
  include HTTP::Handler
  BASIC = "Basic"
  AUTH = "Authorization"
  AUTH_MESSAGE = "Could not verify your access level for that URL.\nYou have to login with proper credentials"
  HEADER_LOGIN_REQUIRED = "Basic realm=\"Login Required\""

  property credentials : Credentials?

  def initialize(@credentials : Credentials)
  end

  def initialize(username : String, password : String)
    initialize({ username => password })
  end

  def initialize(hash : Hash(String, String))
    initialize(Credentials.new(hash))
  end

  def initialize
    if ENV["HTTP_BASIC_USERNAME"]? && ENV["HTTP_BASIC_PASSWORD"]?
      initialize(ENV["HTTP_BASIC_USERNAME"], ENV["HTTP_BASIC_PASSWORD"])
    end
  end

  def call(context)
    if credentials
      if context.request.headers[AUTH]?
        if value = context.request.headers[AUTH]
          if value.size > 0 && value.starts_with?(BASIC)
            return call_next(context) if authorized?(value)
          end
        end
      end
      headers = HTTP::Headers.new
      context.response.status_code = 401
      context.response.headers["WWW-Authenticate"] = HEADER_LOGIN_REQUIRED
      context.response.print AUTH_MESSAGE
    else
      call_next(context)
    end
  end

  private def authorized?(value)
    username, password = Base64.decode_string(value[BASIC.size + 1..-1]).split(":")
    credentials.not_nil!.authorize?(username, password)
  end

  class Credentials
    def initialize(@entries : Hash(String, String) = Hash(String, String).new)
    end

    def authorize?(username : String, given_password : String) : String?
      test_password = find_password(username, given_password)
      if Crypto::Subtle.constant_time_compare(test_password, given_password)
        username
      else
        nil
      end
    end

    private def find_password(username, given_password)
      # return a password that cannot possibly be correct if the username is wrong
      pw = "not #{given_password}"

      # iterate through each possibility to not leak info about valid usernames
      @entries.each do |(user, password)|
        if Crypto::Subtle.constant_time_compare(user, username)
          pw = password
        end
      end

      pw
    end
  end
end

And in routes.cr:

  # ...
  pipeline :api do
    # ...
    plug HTTPBasicAuthPipe.new
  end
  # ...

How quickly and easily run a local server with fake api data (mocks)?

Maxim Romanov — Fri, 11 Oct 2019 07:48:30 +0000

Sometimes you need to develop a frontend part of a project without a ready-made api, knowing only its structure. In this case, using json-schema-faker, you can generate fake data (mocks) and deploy it to your local server.

Firstly u will need to install json-schema-faker package

yarn add json-schema-faker

Then open the package.json file and add scripts with the following

// ...
"scripts": {
  // ...
  "start-mockapi": "json-server --watch ./mocks/api/db.json --port 3001",
  "generate-mock-data": "node ./generateMockData",
}

After installation, you will need to describe the structure in ./mocks/dataSchema.js of future mocks. You can find more information here.

const schema = {
  reports: {
    type: 'array',
    minItems: 5,
    maxItems: 10,
    items: {
      id: {
        type: 'integer',
        unique: true,
        minimum: 1,
        maximum: 1000,
      },
      title: {
        enum: ['production', 'azure data', 'azure data 2'],
      },
      logo: 'https://picsum.photos/200'
    },
  },
}

module.exports = schema;

Copy paste script for generating mock data from here in ./generateMockData.js and run the following

yarn generate-mock-data && yarn start-mockapi

Double splat arguments in Crystal

Igor Alexandrov — Thu, 10 Oct 2019 08:48:27 +0000

In Crystal, as well as in Ruby you can use double splat arguments. Unfortunately they behave a bit different.

def foo(**options)
  baz(**options, a: 1)
end

def baz(**options)
  puts options
end

foo(b: 2, a: 3) # {:b=>2, :a=>1}

This code in Ruby works as it should. If we try the same in Crystal (https://play.crystal-lang.org/#/r/7r0l), we got an error:

error in line 2
Error: duplicate key: a

This happens because **options is a NamedTuple and it cannot have duplicate keys. I found that using NamedTuple#merge can be a workaround (https://play.crystal-lang.org/#/r/7s1c):

def foo(**options)
  baz(**options.merge(a: 1))
end

def baz(**options)
  puts options
end


foo(b: 2, a: 3) # {:b=>2, :a=>1}

Hack!

Rails 5.2 changes in callbacks

Alexander Budchanov — Tue, 17 Sep 2019 14:16:02 +0000

In version 5.1 you may see deprecation warnings in after_save callbacks (related to changes in ActiveRecord::Dirty module).
But since 5.2 these changes were applied.

For examples, I will use Rails 4.2.11 and Rails 5.2.3 and model User with email attribute. Let's do:

u = User.new(email: 'old@domain.com')
u.save
u.email = 'new@domain.com'
u.save

and look at after_save callback in time of last save.

1. attribute_changed?

Rails 4

> email_changed?
=> true

Rails 5.2

> email_changed?
=> false

but you can use saved_changes?

> saved_change_to_email?
=> true

2. changed?

Rails 4

> changed?
=> true

Rails 5.2

> changed?
=> false

but you can use saved_changes?

> saved_changes?
=> true

3. changes

Rails 4

> changes
=> {"email"=>["old@domain.com", "new@domain.com"]}

Rails 5.2

> changes
=> {}

but you can use saved_changes

> saved_changes
=> {"email"=>["old@domain.com", "new@domain.com"]}

4. previous_changes

Rails 4

> previous_changes
=> {"email"=>[nil, "old@domain.com"]}

Rails 5.2
Now, this method returns the changes that were just saved (like saved_changes).

> previous_changes
=> {"email"=>["old@domain.com", "new@domain.com"]}

this method has no replacement.

Simple way to get all values from hash

Andrey — Tue, 17 Sep 2019 05:28:51 +0000

In the ruby from the box, we can't find all the hash values if it's nested.
I suggest an easy way to find all the values using recursion.

Example hash:

hash = {
  a: 2,
  b: { c: 3, d: 4, e: {f: 5}}
}

> hash.values
=> [2, {:c=>3, :d=>4, :e=>{:f=>5}}]

That's not an option, we need all the values.

def deep_values(array = [], object:)
  object.each do |_key, value|
    if value.is_a?(Hash)
      deep_values(array, object: value)
    else
      array << value
    end
  end
  array
end

> deep_values(object: hash)
=> [2, 3, 4, 5]

If you run the benchmark with this data, we get the following data:

>  puts Benchmark.measure { 100_000.times { hash.values } }
=> 0.028920   0.002643   0.031563 (  0.032759)

>  puts Benchmark.measure { 100_000.times { deep_values(object: hash ) } }
=> 0.140439   0.003318   0.143757 (  0.146637)

How to use Rails translations for Reform attributes

Dmitry Voronov — Mon, 16 Sep 2019 19:19:00 +0000

If you use Reform for the form objects and want the translations in your Rails application to work as with ActiveRecord objects, then you can add ActiveModel::Translation module to the form class or base class and specify the method with the key to translations.

class BaseForm < Reform::Form
  extend ActiveModel::Translation

  def self.i18n_scope
    :forms
  end
end

class UserForm < BaseForm
  feature Coercion

  property :login
  property :change_password, virtual: true, type: Types::Form::Bool
  property :password
  property :password_confirmation

  validation do
    required(:login).filled
  end

  validation if: ->(_) { change_password } do
    required(:password).confirmation
  end
end

Set translations for form fields.
You can move them to a new file config/locales/forms.yml.

en:
  forms:
    attributes:
      user_form:
        login: Enter login
        change_password: Change password?
        password: Enter password
        password_confirmation: Confirm password

Now, translations will be used by the simple form or you can call them manually by UserForm.human_attribute_name.

How to parse CSV with double quote (") character in Crystal

Igor Alexandrov — Mon, 16 Sep 2019 18:19:17 +0000

We use microservice written in Crystal to parse large CSV files (about 1.5Gb). Some rows in these files may contain no closed " characters:

,Y,FEDERAL NATIONAL MORTGAGE ASSOCIATION "F,,

With Crystal default CSV parse settings this row and everything after it won't be parsed correctly because DEFAULT_QUOTE_CHAR constant is equal to ". Of course you can override quote_char param in CSV constructor with something that cannot be found in your document.

From my point of view the best is to use zero byte which is '\u0000' in Crystal.

csv = CSV.new(file, headers: true, strip: true, quote_char: '\u0000')
while csv.next
  # ... 
end

Hack!

Metka Gem

Igor Alexandrov — Wed, 04 Sep 2019 19:21:30 +0000

Based on work that I've done in this post https://jetrockets.pro/blog/migrate-tags-in-rails-to-postgresql-array-from-actsastaggableon I decided to create a gem.

Today I made the first release and prepared a lot of work for future.

Here is a link to repository: https://github.com/jetrockets/metka

Git switch command

Andrey — Tue, 03 Sep 2019 19:20:02 +0000

A switch command has been added in the new version of git
Let's look at examples:

# switched to <branch>
git switch <branch>

# creates a new <branch>
git switch -c <branch>

# switched to commit 
git switch -d <commit> 

# creates and switches to branch from remote. 
# need to use if branch exists in multiple remotes 
git switch -c <branch> --track <remote>/<branch> 

# switch to a branch even if the index or working tree is different from HEAD
# this is used to throw away local changes
git switch --discard-changes <branch>

# alias for  --discard-changes
git switch -f <branch> 

# switch back to the previous branch before we switched
git switch -

Command available on version Git 2.23.0 or higher