DEV Community: Igor Alexandrov

How To Use Basecamp’s MRSK With AWS and GitHub

Igor Alexandrov — Fri, 23 Jun 2023 13:57:12 +0000

Hey fellow devs! 👋 Just published a new article on deploying Rails apps like a breeze! 🚀 Check it out here: How to Use Basecamp's MRSK with AWS and GitHub.

I spill the beans on how you can leverage MRSK, AWS, and GitHub to streamline your Rails deployments. No more headaches or complicated setups! 😎

Read the article to discover practical tips, tricks, and insights to make your deployment process smooth and efficient. Let's level up our dev game together!

Join the conversation in the comments. Can't wait to hear your thoughts and experiences. Happy deploying! 💻💪

Scaling Rails: Docker & AWS Beanstalk

Igor Alexandrov — Wed, 20 May 2020 11:44:00 +0000

Amazon AWS has always been our preferred infrastructure provider. We love to use AWS EC2 for our applications, AWS RDS for databases, and AWS ElastiсCache as the ‘key-value’ storage.

Terminology
Due to NDA and security-related restrictions we are unable to use actual project and client names, so here is some terminology we will use in this article:

X-Project – this will be the name of our project;
x-project.com – the main domain;
git://x-project.com/x-project-server – the repository with the Rails application;
git://x-project.com/x-project-ci-image – the repository with the Docker images;

For a better experience of exploring the code, we pulled all the config files out to a separate repository: https://github.com/jetrockets/rails-elastic-beanstalk. This repository contains two directories:

x-project-ci-image contains files that were used in git://x-project.com/x-project-ci-image;
x-project-server contains the files that were added to our Rails application from git://x-project.com/x-project-server.

Inside x-project-server/eb/production we have configs of all three of our EBS projects. An EBS project usually consists of two main components:

.elasticbeanstalk directory with the global scenario files for the project configuration and the Elastic Beanstalk environment: name, instance types, variables of the environment and the environment settings;
.ebextensions directory which contains a collection of scenario files for instance configuration. These files represent a set of instructions that are loaded automatically upon creation of a new instance or update of an existing one.

Use Case
Usually we deploy our servers on EC2 using Ansible scripts and Capistrano, which is a common approach. However, this approach may lead to a state that Martin Fowler referred to as SnowflakeServer. Over time, the configuration of servers starts to have too many dependencies and it becomes very difficult to replicate quickly. That's why even if you're not using Docker on EC2 machines, it's vital to make all changes to the server by modifying the Ansible scripts.
This ensures that if necessary, you can deploy a new environment relatively quickly.

One of our recent projects was originally focused on a high load that was difficult to predict. We were building a chat room for online stores with enhanced retail sales capabilities. It was a great opportunity to try Elastic Beanstalk. To understand why, let's first look at what it is.

AWS Elastic Beanstalk is an AWS manipulation service. A preset configuration can quickly set up several application instances, link them to a database or a cache, and set up a balancer between them. With EBS you can run the application code on several platforms: Puma or Passenger for Ruby, Python, Java, and of course you can run Docker containers. We would be satisfied with the Docker container option, so let's explore this further.

Architecture
Our application was built using the following tech stack:

A Rails application (web and Sidekiq);
AnyCable for a websocket chat;
Embeddable Widgets in HTML and JS which are integrated into client e-commerce platforms;

The project required at least two environments: Production (for the live system) and Integration (for demo and testing purposes).

Let’s start with the Rails app.

Configuration
By default, Rails 6 stores credentials in separate files specific to each environment. To learn more about this, go to Rails Guides. We opted to use environment variables for managing settings, because this approach could provide certain advantages when working with AWS. For this, we used a gem dotenv-rails.

# git://x-project.com/x-project-server/Gemfile

gem 'dotenv-rails'

Environment variables are added from the .env file in the application root, which is convenient for local development. Moreover, we always store the .env.sample file inside the project, which contains a template for all project settings.

In our experience, using .env is a more convenient, clear, and consistent way to configure Rails, rather than using rails credentials. You can define the settings in a file, copy the finished file, or run Rails and transfer the settings directly using the command line.

Container Registry
Amazon offers a container management tool – Amazon Elastic Container Registry. Amazon ECR is a fully automated Docker container registry that makes it easy for developers to store, deploy, and manage Docker container images.

Getting Ready

To work with git, we use a self-hosted version of GitLab, so we wanted to integrate GitLab & GitLab CI into the AWS infrastructure.
We decided to store all the application settings in AWS. This means that GitLab-runner should not have access to these settings and so we had to make a second docker image, which receives the keys to work with AWS while the app is building. To do this, the following Dockerfile should be added to a separate repository.

# git://x-project.com/x-project-ci-image/aws-utils/Dockerfile

FROM docker:latest     

ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_REGION

RUN apk add --no-cache python py-pip python-dev build-base libressl-dev musl-dev libffi-dev git
RUN pip install awsebcli awscli s3cmd

RUN  aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID
RUN  aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY 
RUN  aws configure set region $AWS_REGION

The settings for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY & AWS_REGION will be based on the GitLab environment variables while the app is building. To accomplish this we created a user in the Amazon IAM with the required permissions. In our case, we selected AWSElasticBeanstalkFullAccess (to work with EC2, S3, RDS, and other services), AWSElasticBeanstalkMulticontainerDocker (to work with Docker images) & ECRFullAccess (a custom policy with full ECR access). As a result of the app build, this container will be able to work with AWS API according to the set permissions, so the main application will be built on its basis.

To make it easier to run tests and release the app, we decided to build an intermediate Docker image in order to launch rspec and collect images to run Rails & Sidekiq. This image already has everything you need: nodejs & yarn for compiling JS & CSS, and client PostgreSQL for connecting to RDS. It is also necessary to remember about static distribution, therefore we’ll add Nginx. We use ruby 2.6.5, as Rails (at the time of writing this article) still generates a large number of warnings with ruby 2.7.0.


# git://x-project.com/x-project-ci-image/app-image/Dockerfile

ARG RUBY_VERSION_IMAGE=2.6.5
FROM ruby:${RUBY_VERSION_IMAGE}-slim-buster

ARG APP_PATH="/app"
ARG APP_USER=app
ARG NODE_VERSION=12
ARG POSTGRES_VERSION=12

RUN adduser --disabled-password --home ${APP_PATH} --gecos '' ${APP_USER}

RUN mkdir -p ${APP_PATH} && \
    chown ${APP_USER}. ${APP_PATH}

# Install packages
RUN apt-get update -qq && \
    apt-get install -yq --no-install-recommends \
    curl \
    gnupg2 \
    lsb-release \
    gcc \
    g++ \
    git \
    openssh-client \
    libxml2-dev \
    libxslt-dev \
    libjemalloc2 \
    make \
    nginx \
    pkg-config \
    file \
    imagemagick

# Nginx settings
RUN touch /run/nginx.pid && \
    chown -R ${APP_USER}. /run/nginx.pid && \
    chown -R ${APP_USER}. /var/log/nginx && \
    chown -R ${APP_USER}. /var/lib/nginx && \
    chown -R ${APP_USER}. /etc/nginx/conf.d && \
    chown -R ${APP_USER}. /usr/share/nginx && \
    unlink /etc/nginx/sites-enabled/default

# Forward request logs to Docker log collector
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
 && ln -sf /dev/stderr /var/log/nginx/error.log

# Add Node.js repo
RUN curl -fsSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add - && \
    echo "deb https://deb.nodesource.com/node_${NODE_VERSION}.x $(lsb_release -s -c) main" | tee /etc/apt/sources.list.d/nodesource.list && \
    echo "deb-src https://deb.nodesource.com/node_${NODE_VERSION}.x $(lsb_release -s -c) main" | tee -a /etc/apt/sources.list.d/nodesource.list

# Add yarn repo
RUN curl -fsSL https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - && \
    echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list

# Install packages
RUN apt-get update -qq && \
    apt-get install -qy --no-install-recommends \
    nodejs \
    yarn

RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add - &&\
    echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -sc)-pgdg main" | tee /etc/apt/sources.list.d/postgresql.list

# Install dependency for gem pg
RUN apt-get update -qq && \
    apt-get install -qy --no-install-recommends \
    libpq-dev \
    postgresql-client-${POSTGRES_VERSION}

COPY --chown=${APP_USER}:${APP_USER} . ${APP_PATH}

USER ${APP_USER}

ENV GEM_HOME=${APP_PATH}/bundle \
    BUNDLE_PATH=${APP_PATH}/bundle \
    BUNDLE_BIN=${APP_PATH}/bundle/bin \
    PATH=${APP_PATH}/bin:${APP_PATH}/bundle/bin:/usr/local/bin:$PATH

WORKDIR ${APP_PATH}

# Install dependences
# Ruby-2.6.5 supplied with bundler:1.17.2
RUN gem install bundler -v '2.1.4'

RUN bundle install --jobs=$(nproc)
RUN yarn install --check-files

The build of both images is performed in the following scenario.

# git://x-project.com/x-project-ci-image/.gitlab-ci.yml

variables:
   RUBY_VERSION_IMAGE: "2.6.5"
   NODE_VERSION: "12"
   APP_PATH: "/app"
   APP_USER: "app"

stages: 
  - build

build_image:
  # Docker image with pre-installed docker package 
  image: docker:latest
  stage: build
  before_script:
    # login to git://x-project.com/x-project-ci-image and pull images from regestry
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # Pull all images for use caching. 
    # Statement "|| true" used if images not found and pipeline not failing
    - docker pull $CI_REGISTRY/x-project/x-project-ci-image/app-image:latest || true
    - docker pull $CI_REGISTRY/x-project/x-project-ci-image/aws-utils:latest || true

  script:
    # Build docker image with aws utils and and aws secrets
    - docker build --cache-from $CI_REGISTRY/x-project/x-project-ci-image/aws-utils:latest
      -t $CI_REGISTRY/x-project/x-project-ci-image/aws-utils
        --build-arg AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} 
        --build-arg AWS_REGION=${AWS_REGION}
        --build-arg AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
        aws-utils

    # Build docker image with pre-installed linux packages, gems and frontend packages    
    - docker build --cache-from $CI_REGISTRY/x-project/x-project-ci-image/app-image:latest 
      -t $CI_REGISTRY/x-project/x-project-ci-image/ruby-image
        --build-arg APP_PATH=${APP_PATH}
        --build-arg APP_USER=${APP_USER}
        --build-arg NODE_VERSION=${NODE_VERSION}
        --build-arg RUBY_VERSION_IMAGE=${RUBY_VERSION_IMAGE}
        app-image

    # login to git://x-project.com/x-project_ci_image for push images
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

    # Push images to git://x-project.com/x-project_ci_image in regestry
    # See git://x-project.com/x-project_ci_image/container_registry
    - docker push $CI_REGISTRY/x-project/x-project-ci-image/aws-utils:latest    
    - docker push $CI_REGISTRY/x-project/x-project-ci-image/app-image:latest

Everything described above can be illustrated in the following diagram.

Building an App

We use GitLab-runner for building as well as for the intermediate image.
Building and deployment of the project begin after the code gets into the master or integration branch. Currently, our pipeline consists of five stages.

Let’s skip notify & rspec to take a closer look at how build & deploy are going.

#.gitlab-ci.yml

docker-build:
  image: $BASE_IMAGES_URL:$CI_PROJECT_NAMESPACE-aws-utils
  stage: build
  environment:
    name: $CI_COMMIT_BRANCH
  only:
    - master
    - integration
  script:
    - $(aws ecr get-login --no-include-email --profile default)
    - aws s3 cp s3://s3.environments.x-project.com/.env.$RAILS_ENV .env
    - aws s3 cp s3://s3.environments.x-project.com/$RAILS_ENV.key config/credentials/$RAILS_ENV.key
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $BASE_IMAGES_URL:$CI_PROJECT_NAMESPACE-app-image
    - docker build -t $REPOSITORY_URL:$REPOSITORY_TAG
      -t $REPOSITORY_URL:$RAILS_ENV-latest
      --build-arg APP_PATH=$APP_PATH
      --build-arg DATABASE_URL=$DATABASE_URL
      --build-arg RAILS_ENV=$RAILS_ENV
      --build-arg REDIS_URL=$REDIS_URL
      --build-arg S3_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID
      --build-arg S3_BUCKET=$S3_BUCKET
      --build-arg S3_REGION=$S3_REGION
      --build-arg S3_SECRET_KEY=$S3_SECRET_KEY
      --build-arg SECRET_KEY_BASE=$SECRET_KEY_BASE
      --build-arg WEBHOOKS_API_URL=$WEBHOOKS_API_URL
      --build-arg WIDGET_SRC=$WIDGET_SRC .
    - docker push $REPOSITORY_URL:$REPOSITORY_TAG

    # "Hypervisor" settings
    - docker tag $REPOSITORY_URL:$REPOSITORY_TAG $REPOSITORY_URL:$RAILS_ENV-latest
    - docker push $REPOSITORY_URL:$RAILS_ENV-latest

To build the image we use the previously prepared aws-utils image with ready-to-go settings for aws. Then we copy the file with the environment variables from S3 and run the build based on app-image, which we also built earlier. Thus, we get a universal container that can run rails, sidekiq, & anycable-rails.
To be able to quickly roll back to the previous version of the application without doing an additional build, we use tags with a hash of the commit and the current environment (for example production-2a4521a1). Each docker image has a tag with the current version marked as :latest. The same tag is used to run in hypervisors (we’ll talk about them below). Accordingly, a rollback to any version of the system that is stored in the registry takes no more than a minute.

Pushing configuration files to AWS EB

So, we have collected the images and now should only deliver them to the Elastic Beanstalk.

docker-deploy:
  image: $BASE_IMAGES_URL:$CI_PROJECT_NAMESPACE-aws-utils
  stage: deploy
  environment:
    name: $CI_COMMIT_BRANCH
  only:
    - master
    - integration
  script:
    - ./bin/ci-deploy.sh

For convenience, we wrote a small script that runs in the aws-utils image and therefore can work with the AWS API. So, all that is left to do is run eb deploy command with the corresponding parameters in the directory, where the .elasticbeanstalk configuration file is located.
At this stage we have already received an image with a tag in the format RAILS_ENV-SHORT_SHA_COMMIT; it remains to specify ElasticBeanstalk in the configuration file, so that it downloads this particular version. To do that, let’s use sed and replace the tag with RAILS_ENV-SHORT_SHA_COMMIT.

# bin/ci-deploy.sh

#!/bin/sh
xargs -P 3 -I {} sh -c 'eval "$1"' - {} << "EOF"
cd eb/$RAILS_ENV/rails; sed -i 's/tag/'$RAILS_ENV-$CI_COMMIT_SHORT_SHA'/' Dockerrun.aws.json; eb deploy --profile default --label $LABEL-rails --timeout 30

cd eb/$RAILS_ENV/sidekiq; sed -i 's/tag/'$RAILS_ENV-$CI_COMMIT_SHORT_SHA'/' Dockerrun.aws.json; eb deploy --profile default --label $LABEL-sidekiq --timeout 30

cd eb/$RAILS_ENV/anycable-rails;  sed -i 's/tag/'$RAILS_ENV-$CI_COMMIT_SHORT_SHA'/' Dockerrun.aws.json; eb deploy --profile default --label $LABEL-anycable-rails --timeout 30
EOF

After successfully uploading the image to Elastic Beanstalk each container starts according to the script from Dockerrun.aws.json.

Balancers and Autoscaling

Now that we have prepared the environments and configured the deploy process, we are fully ready for production. Running a final performance check, we are seeing WebSocket connection errors.

A couple of hours of checking EB components and we discover that the AWS Application Load Balancer doesn’t want to route traffic to the anycable-rails port. This is due to a health check failure: incorrect response from the port (204 or 200). Dealing with anycable-rails, it’s clear that it runs with 50051 port and responds via HTTP2/gRPC by default, while ALB is waiting for a reply via HTTP. HTTP health check can be configured in anycable-rails with the --http-health-port option. However, ALB is not able to do a health check for one port and proxy traffic to another. At this point we realize that we could try to proxy traffic to anycable-rails through Nginx as well as parse the requests for health checks.

After reconfiguring Nginx, and re-checking, the ALB accepts our response for a health check, and traffic successfully goes to the container with anycable-rails. However, right away, we are running into a new error at anycable-go.

context=ws Websocket session initialization failed: rpc error: code = Unavailable
desc = all SubConns are in TransientFailure, latest connection error: connection
closed

So we go back to anycable-rails to carefully check the logs.

"PRI * HTTP/2.0" 400 173 "-" "-"

An idea occurs that there might be a problem with ALB, which disassembles HTTPS on its side but sends requests to the host via HTTP. After creating a self-signed certificate and configuring Nginx to listen to the port in order to forward gRPC to HTTP/2 with the certificate, we get another error.

"PRI * HTTP/2.0" 400 173 "-" "-”

We continued EB configuration, made full-fledged end-2-end encryption, but unfortunately this didn’t help. Finally, we tried to work with anycable-rails directly without ALB, and it didn’t work out either. It became clear that the problem was in ALB, though according to the documentation ALB supports HTTP/2. While ALB parses HTTP/2 traffic on its side, it sends requests to the host via HTTP/1.1, and gRPC uses HTTP/2.
Then we remembered that apart from ALB, Elastic Beanstalk has its own second balancer – NLB, which does nothing with traffic, unlike ALB. We decided to try it out. Besides dividing application and anycable we can be more accurate with autoscaling. As a result, we arrived at the following scheme for our service.

Hypervisors

Elastic Beanstalk does not imply the use of an EC2 instance as we are accustomed to: you can, of course, connect via ssh with a key, but setting up a system of rights through IAM will be problematic. Only administrators should have access to such instances in order to resolve extraordinary cases.
And the first question is how to give developers access to the console rails.

We decided to make two micro-instances (t2.micro) on EC2, one for each environment, and called them hypervisors. Developers can use this environment to perform various tasks: launching rails console, connecting to RDS or ElasticCache, viewing logs, and so on.

We deployed Docker to both instances, so AWS IAM created the user (registry-user) with two policies: AmazonEC2ContainerRegistryReadOnly & CloudWatchReadOnlyAccess. The selected policies allow us to download the Docker image from the registry and view CloudWatch logs. To make these operations easier we wrote two tooling bash scripts.

Tooling

The first script accesses the rails console. The integration and production options differ only in ECR addresses. It is important in CI to make integration-latest & production-latest tags, to download the current image in the script.

#!/bin/bash
COMMAND=$@
if test -z $COMMAND; then
  COMMAND="rails c"
fi

$(aws ecr get-login --no-include-email --profile default)
docker pull some-address.dkr.ecr.us-east-1.amazonaws.com/x-project-server:integration-latest
docker run -it --rm some-address.dkr.ecr.us-east-1.amazonaws.com/x-project-server:integration-latest $COMMAND

The second script allows viewing logs from various instances in the console. Docker sends all logs to stdout by default, no one can read them there, so we add them to the file, and stream the file to CloudWatch. The problem is Docker doesn’t send multi-container logs to CloudWatch. To solve this you need to add .ebextensions settings for the CW agent and configure Dockerrun.aws.json for logging.

Now when all the logs are in CloudWatch, tooling can be built. To get the logs we use awslogs. Below is a sample script for integration.

#!/bin/bash

service=$1
minutes=$2
if test -z $minutes ;then
  minutes=20
fi

case $service in
  rails)
    awslogs get /aws/elasticbeanstalk/Application-integration/var/log/containers/app/integration.log --start=${minutes}'m ago'
  ;;
  anycable)
    awslogs get /aws/elasticbeanstalk/Anycable-rails-integration/var/log/containers/RPC/integration.log --start=${minutes}'m ago'
  ;;
  nginx)
    awslogs get /aws/elasticbeanstalk/Application-integration/var/log/containers/nginx/access.log --start=${minutes}'m ago'
    echo "↓--- NGNIX ERROR LOGS ---↓"
    awslogs get /aws/elasticbeanstalk/Application-integration/var/log/containers/nginx/error.log --start=${minutes}'m ago'
  ;;
  sidekiq)
    awslogs get /aws/elasticbeanstalk/Sidekiq-integration/var/log/eb-docker/containers/eb-current-app/stdouterr.log --start=${minutes}'m ago'
  ;;
  *)
    echo "Supported serices rails, anycable, nginx, sidekiq."
    echo "Usage ./logs ralis 30"
    echo "Output will be logs from rails env from the last 30 minutes"
    echo "If output is emply, it seems last N (default 20) minuters no logs from app"
    exit 1
  ;;
esac

Key Takeaways

AWS Elastic Beanstalk seems to be the best solution for scalable Rails hosting. After complex first time configuration every new project can be deployed to it within one hour.

The most valuable thing is that developers don’t need any additional help from dev-ops team to scale their apps. All needed information about current application performance can be found in AWS dashboards and scaling configuration can be edited directly in web-interface. Another good point is that if your project has peak load only a few days within a month (e.g. report generation by the end of month) you will not pay for not needed instances during the whole month.

Migrating user passwords from Django to Ruby

Igor Alexandrov — Wed, 19 Feb 2020 06:40:34 +0000

One of our clients asked us to migrate his existing Django application to Ruby. A part of this process was a migration of an existing users database.
Of course we had to find a way to use existing crypto passwords from Django and not asking our users to reset them

Passwords in Django are stored as a string that consists of four parts splitted with a dollar sign.

<algorithm>$<iterations>$<salt>$<hash>

By default, Django uses the PBKDF2 algorithm with a SHA256 hash. I found a rather outdated Ruby gem that implements a Password-Based Key-Derivation Function and gave it a try.

def migrate_django_password_seamlessly(user, password)
  alg, iteration, salt, django_password = user.crypted_django_password.split('$')
  attempt = Base64.encode64(
    PBKDF2.new(password: password, salt: salt, iterations: 36000).bin_string
  ).strip

  # check if hash of user provided password equals to the password in a database
  if attempt == django_password
    user.update(password: password, password_confirmation: password)
  end
end

The method above is a part of user sign in service and called only if crypted_django_passwordcolumn from users table is not null.

Two edge cases in PostgreSQL full-text search

Igor Alexandrov — Thu, 09 Jan 2020 17:30:38 +0000

We widely use PostgreSQL full-text search in our projects. It is fast, reliable, and doesn't add any additional technical complexity. But sometimes it may not work as you expect it too.

Usually we add a ts_vector column right in our data tables. Let's see how it has it is done.

title	tsv
Handyman	‘handyman’:1
Heating & Cooling	‘cool’:2 ‘heat’:1
Painting	‘paint’:1
Roofing	‘roof’:1

Column tsv is generated with to_tsvector function from a title column with a trigger function.

CREATE FUNCTION skills_tsv_update_trigger() RETURNS trigger AS $$
begin
  new.tsv :=
    to_tsvector('pg_catalog.english', COALESCE(new.title, ''))
  return new;
end
$$ LANGUAGE plpgsql;

Then we can query this table using to_tsquery fuction.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'roof')

-- 1 record found

A record with Roofing has been found, and this makes sense. But what should be found for handy query? You can imagine that it will be a Handyman skill and you are wrong. Before explaining why this happens, let's try another query.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'roofing')

-- no records found

No records were found, even if we have a record with Roofing title in our table.

The to_tsvector function internally calls a parser which breaks the document text into tokens and assigns a type to each token. For each token, a list of dictionaries is consulted, where the list can vary depending on the token type. The first dictionary that recognizes the token emits one or more normalized lexemes to represent the token. For example, roofing became roof. Some words are recognized as stop words, like & in our example. It is possible to have many different configurations in the same database, and predefined configurations are available for various languages. In our example we used the default configuration english for the English language.

to_tsquery function parses user input and converts it to tsquery data type. It uses the same dictionaries as to_tsvector function. Let's try to understand what happens in two queries that were listed above.

The result of to_tsquery('pg_catalog.english', 'handy') will be handi because of pluralization rules for words ending on y. The query listed below doesn't return any records because 'handyman':1 vector does not overlap with handi query.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'handy')

-- no records found

How we can cover this situation? As you remember you can change a configuration that is used to parse user input. Let's try a simple configuration.

select to_tsquery('pg_catalog.simple', 'handy')

-- 'handy'

Looks like this is what we need, let's add this to our query.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'handy')
    OR
    tsv @@ to_tsquery('pg_catalog.simple', 'handy')    

-- 1 record found

A problem with Roofing and roof can be solved in the same way. Let's change the function that is used to populate tsv column.

CREATE FUNCTION skills_tsv_update_trigger() RETURNS trigger AS $$
begin
  new.tsv :=
    to_tsvector('pg_catalog.english', COALESCE(new.title, '')) ||
    to_tsvector('pg_catalog.simple', COALESCE(new.title, ''))
  return new;
end
$$ LANGUAGE plpgsql;

title	tsv
Handyman	‘handyman’:1,2
Heating & Cooling	‘cool’:2 ‘cooling’:4 ‘heat’:1 ‘heating’:3
Painting	‘paint’:1 ‘painting’:2
Roofing	‘roof’:1 ‘roofing’:2

Now vector has more data and query below works as expected.

SELECT * FROM skills
WHERE
    tsv @@ to_tsquery('pg_catalog.english', 'roofing')

-- 1 record found

How to add HTTP Basic auth to Amber application

Igor Alexandrov — Mon, 25 Nov 2019 21:44:17 +0000

As you already may know, one of our projects have a Crystal application in production. It is created with Amber framework and works just perfect.

The only thing that I don't personally like in Amber is not clear and sometimes outdated documentation, after about 11 years with Rails I still think that Rails Guides are the number one developer documentation in the world.

I had a task to add HTTP Basic auth to a couple of URLs in Amber application, and after studying documentation found that Amber doesn't provide necessary Pipe out of the box. Ok, next place to search for the answer was Gitter and after about an hour Dru Jensen helped me with a code example.

Amber uses internally HTTP::Handler for Pipes as well as Kemal does for Middlewares, so we can easily use code from Basic Auth for Kemal.

# src/pipes/http_basic_auth_pipe.cr

require "crypto/subtle"

class HTTPBasicAuthPipe
  include HTTP::Handler
  BASIC = "Basic"
  AUTH = "Authorization"
  AUTH_MESSAGE = "Could not verify your access level for that URL.\nYou have to login with proper credentials"
  HEADER_LOGIN_REQUIRED = "Basic realm=\"Login Required\""

  property credentials : Credentials?

  def initialize(@credentials : Credentials)
  end

  def initialize(username : String, password : String)
    initialize({ username => password })
  end

  def initialize(hash : Hash(String, String))
    initialize(Credentials.new(hash))
  end

  def initialize
    if ENV["HTTP_BASIC_USERNAME"]? && ENV["HTTP_BASIC_PASSWORD"]?
      initialize(ENV["HTTP_BASIC_USERNAME"], ENV["HTTP_BASIC_PASSWORD"])
    end
  end

  def call(context)
    if credentials
      if context.request.headers[AUTH]?
        if value = context.request.headers[AUTH]
          if value.size > 0 && value.starts_with?(BASIC)
            return call_next(context) if authorized?(value)
          end
        end
      end
      headers = HTTP::Headers.new
      context.response.status_code = 401
      context.response.headers["WWW-Authenticate"] = HEADER_LOGIN_REQUIRED
      context.response.print AUTH_MESSAGE
    else
      call_next(context)
    end
  end

  private def authorized?(value)
    username, password = Base64.decode_string(value[BASIC.size + 1..-1]).split(":")
    credentials.not_nil!.authorize?(username, password)
  end

  class Credentials
    def initialize(@entries : Hash(String, String) = Hash(String, String).new)
    end

    def authorize?(username : String, given_password : String) : String?
      test_password = find_password(username, given_password)
      if Crypto::Subtle.constant_time_compare(test_password, given_password)
        username
      else
        nil
      end
    end

    private def find_password(username, given_password)
      # return a password that cannot possibly be correct if the username is wrong
      pw = "not #{given_password}"

      # iterate through each possibility to not leak info about valid usernames
      @entries.each do |(user, password)|
        if Crypto::Subtle.constant_time_compare(user, username)
          pw = password
        end
      end

      pw
    end
  end
end

And in routes.cr:

  # ...
  pipeline :api do
    # ...
    plug HTTPBasicAuthPipe.new
  end
  # ...

Double splat arguments in Crystal

Igor Alexandrov — Thu, 10 Oct 2019 08:48:27 +0000

In Crystal, as well as in Ruby you can use double splat arguments. Unfortunately they behave a bit different.

def foo(**options)
  baz(**options, a: 1)
end

def baz(**options)
  puts options
end

foo(b: 2, a: 3) # {:b=>2, :a=>1}

This code in Ruby works as it should. If we try the same in Crystal (https://play.crystal-lang.org/#/r/7r0l), we got an error:

error in line 2
Error: duplicate key: a

This happens because **options is a NamedTuple and it cannot have duplicate keys. I found that using NamedTuple#merge can be a workaround (https://play.crystal-lang.org/#/r/7s1c):

def foo(**options)
  baz(**options.merge(a: 1))
end

def baz(**options)
  puts options
end


foo(b: 2, a: 3) # {:b=>2, :a=>1}

Hack!

How to parse CSV with double quote (") character in Crystal

Igor Alexandrov — Mon, 16 Sep 2019 18:19:17 +0000

We use microservice written in Crystal to parse large CSV files (about 1.5Gb). Some rows in these files may contain no closed " characters:

,Y,FEDERAL NATIONAL MORTGAGE ASSOCIATION "F,,

With Crystal default CSV parse settings this row and everything after it won't be parsed correctly because DEFAULT_QUOTE_CHAR constant is equal to ". Of course you can override quote_char param in CSV constructor with something that cannot be found in your document.

From my point of view the best is to use zero byte which is '\u0000' in Crystal.

csv = CSV.new(file, headers: true, strip: true, quote_char: '\u0000')
while csv.next
  # ... 
end

Hack!

Metka Gem

Igor Alexandrov — Wed, 04 Sep 2019 19:21:30 +0000

Based on work that I've done in this post https://jetrockets.pro/blog/migrate-tags-in-rails-to-postgresql-array-from-actsastaggableon I decided to create a gem.

Today I made the first release and prepared a lot of work for future.

Here is a link to repository: https://github.com/jetrockets/metka

Request Api Adapter

Igor Alexandrov — Fri, 09 Aug 2019 18:31:03 +0000

When developing client applications, it is often necessary to send requests to the server.

// ...
client({ url: "/users.json", method: "GET" }).then(...)

We can make our lives a little easier. A convenient abstraction is apiAdapter

// apiAdapter.js
function getUsers() {
  return { url: "/users.json", method: "GET" };
}

const apiAdapter = createAdapter(client, {}, {
  getUsers,  
})

By defining a request in one place, you can now simply call the adapter method you want.

import apiAdapter from './apiAdapter'

apiAdapter.getUsers().then(...)

It is also a useful option to specify basic settings for all requests, as well as handling errors and successful requests.

const apiAdapter = createAdapter(
  client,
  { withCredentials: true },
  {
    getUsers,  
  },
  successHandler,
  errorHandler, 
)

Github
Live example

Migrate tags in Rails to PostgreSQL array from ActsAsTaggableOn

Igor Alexandrov — Wed, 31 Jul 2019 17:24:30 +0000

ActsAsTaggableOn is a swiss army knife solution if you need to add tags to your ActiveRecord model.
Just by adding one gem to your Gemfile and acts_as_taggable to the model you get everything you need: adding tags, searching for a model by tag, getting top tags, etc. However, sometimes you don't need all these.

In our project, we used acts_as_taggable to store tags for Note model. Then we displayed a list of notes on several pages with assigned tags and had autocompleted input for tags on Note form. Everything worked well, but since we use PostgreSQL, I decided to store tags as an array in Note model.

First of all, I added tags Array<String> column to Note, after this migrated acts_as_taggable tags to notes table with migration.

class MigrateNoteTags < ActiveRecord::Migration[5.2]
  def change
    execute <<-SQL
    UPDATE notes 
    SET tags = grouped_taggings.tags_array 
    FROM
      (
      SELECT
        taggings.taggable_id,
        ARRAY_AGG ( tags.NAME ) tags_array 
      FROM
        taggings
        LEFT JOIN tags ON taggings.tag_id = tags.ID 
      WHERE
        taggable_type = 'Note' 
      GROUP BY
        taggings.taggable_id 
      ) AS grouped_taggings 
    WHERE
      notes.ID = grouped_taggings.taggable_id
    SQL
  end
end

To have backward compatibility, I added Note#tag_list method:

def tag_list
  tags.join(', ')
end

The last thing is to add the ability to search for tags. Since there about 500k records in the Notes table, I decided to create an SQL view:

CREATE OR REPLACE VIEW note_tags AS

SELECT UNNEST
  ( tags ) AS name,
  COUNT ( * ) AS taggings_count 
FROM
  notes 
GROUP BY
  name

That's it! It takes from 100ms to 150ms to search for tags in this view, which is fine for me.

If you have more significant data sets, then the best would be to create tags table and add triggers to notes table that will update tags on INSERT/UPDATE/DELETE.

Mistakes to Avoid When Outsourcing Web and Mobile Development

Igor Alexandrov — Wed, 31 Jul 2019 07:15:21 +0000

The pursuit of a low price. Developing and launching a successful web or mobile application requires the participation of many professionals: business analysts, designers, programmers, QA testers and managers, among others. Software development is a complex and knowledge-intensive process, which makes the work of dependable programmers extremely valuable. Because of this, hiring a company that offers you a low hourly rate means that their employees are not being well compensated in turn and thus are less likely to be highly skilled.

The best plan is to look for a company headquartered in the United States that has an offshore development center. That way, you will be able to take advantage of lower prices, at least compared to U.S.-based development teams, while still enjoying superior quality and the legal protection offered by in-country corporations.

Ignoring the language barrier. Choosing a team with poor English abilities may seem like a mere inconvenience, but this supposedly minor obstacle can actually endanger a project's success. Misunderstanding certain nuances of a desired product can cause architectural mistakes that could lead to significant issues later on, resulting in rework, delays and further additional expenses. When interviewing your potential product team, make sure to speak with the actual developers that will be assigned to your project, not just the sales person or project manager. Use a video conference to conduct an in-person interview with the team and confirm that you can understand each other clearly.

Lack of technical oversight. If you do not have a technical leader on your internal team, you may run into significant problems. This is a familiar scenario: during the sales process, the team you are thinking about hiring demonstrates great technical knowledge and sounds very convincing, but as you begin the work process, you realize that the team may not have the necessary technical skills after all. To avoid a situation like this, you should hire a freelance technical advisor, who will be able to interview the team beforehand and ensure that they have the necessary technical skills to deliver a successful solution. Online services such as Codementor, Upwork, or Toptal are a few options that help you hire such an individual.

Hiring developers vs hiring consultants. Unless you have a strong internal technical leadership, it is advisable to outsource your product development to a company that provides a full range of technical consulting services as opposed to a company that offers developers for-hire. When working with a consulting firm, you get access to a dedicated team of professionals who can help you design, architect and implement your product while taking into account both your short-term and long-term business goals. Developers for-hire simply execute the tasks you present to them, often ignoring best practices and your long-term vision at once.

As an example, let’s say that you want to build a mobile application with a minimal budget and you read online that it can be done quickly as a container app. You then hire a team of programmers to execute this specific task. In the beginning everything looks great—the project has been developed to your specification and it functions just as planned. After a month or two, you decide to add animations, improve the design or add a little more complexity to the logic of the application.Unfortunately, though, your hired development team is unable to accomplish any of these tasks because the technology they used for development does not support these requirements. If the changes you want to make are critical for your business, there is only one way out: rewriting your application from scratch.

This situation can easily be avoided by working with a team of development consultants who know how to ask the right questions and can help you solve immediate problems while planning for the future.

Treating estimates as definitive project budgets. Consider the following scenario: you spend six months developing a detailed technical specification document and you send an RFP to a development company. After a week or so, the company responds with a proposal that includes a budget of 3,000 man-hours. I can guarantee with 100% certainty that your expenses will run over from this proposal. Why? Because it is impossible to accurately assess such a large and abstract number of work hours.

Instead, ask for a rough assessment of the entire project to ensure that the overall estimated budget fits your goals. Also try to pinpoint the composition of the development team and request a clear short-term plan that covers the initial development phase, usually a window of the first four to six weeks. After this you can choose to either proceed with the team on a retainer basis, which will help you manage your costs, or ask about a quote for the next four to six weeks of work based on priorities. Make sure to attend weekly meetings with the team and ask for regular budget updates.

Finally, before making a hiring decision, ask for three or four references and be sure to contact them over the phone, not via email. Inquire about the team's management skills, their responsiveness, and their ability to handle emergency situations.

Finding a reliable technology partner is a daunting task, but when approached with patience and due diligence it can quickly pay off, helping your business grow and achieve its goals.

At JetRockets we specialize in designing, developing and supporting custom software solutions that help clients achieve their unique business goals. If you have any questions or are looking for a complimentary consultation, don’t hesitate to reach out.

Networking and thermal conditions debugging in Xcode 11

Igor Alexandrov — Wed, 24 Jul 2019 06:42:37 +0000

Xcode 11 introduces tool for networking and thermal conditions debugging. This feature requires device running iOS 13. Previously network conditioning feature was available within Additional Tools for Xcode.