DEV Community: Ivan Iliukhin

Overcoming complexity of service objects with dry libraries

Ivan Iliukhin — Thu, 02 Sep 2021 19:35:47 +0000

Introduction

Almost everyone in the rails community(and not only) talks about the slim controllers: "Let's move everything to the service objects and functions, keep controllers as simple as possible". It's a great concept, and I fully support it. Service objects and functions for controllers' methods(not only but mainly for them) usually have two additional parts besides business logic:

input validation;
errors passing.

They aren't a big problem if your function does not call other functions and does not have nested objects as the input arguments. The problem that they are love to grow and mysteriously interact with each other.

In this post, I want to share the way how you can overcome these problems. Grab yourself a cup of your favourite beverage ☕ and join this journey.

Time for experiments

Imagine that you are designing internal software for a laboratory. It should run experiments from the passed arguments. Arguments should be validated to get a sensible response instead of meaningless 500 errors.

Without dry libraries

➡️ Let's start from the controller for the experiment. The controller should only run the function and render results or errors:

class ExperimentsController
  def create
    result = ConductExperiment.call(experiment_params)

    if result[:success]
      render json: result[:result]
    else
      render json: result[:errors], status: :bad_request
    end
  end

  private

  def experiment_params
    params.require(:experiment).permit(
      :name,
      :duration,
      :power_consumption
    )
  end
end

➡️ Next, add the service object. It is a good practice to move the repeated part to the base object:

class ApplicationService
  def initialize(*params, &block)
    @params = params
    @block = block
  end

  def call
    validation_result = validate_params
    if validation_result[:success]
      execute
    else
      validation_result
    end
  end

  def validate_params
    { success: true }
  end

  def self.call(*params, &block)
    new(*params, &block).call
  end
end

class ConductExperiment < ApplicationService
  def initialize(params)
    super
    @params = params
  end

  def validate_params
    errors = []
    if @params[:duration].blank?
      errors << 'Duration should be present'
      return { success: false, errors: errors }
    end

    if @params[:duration] < 1
      errors << 'Duration should be positive number'
      return { success: false, errors: errors }
    end

    { success: true }
  end

  def execute
    if Random.rand > 0.3
      { success: true, result: 'Success!' }
    else
      { success: false, errors: ['Definitely not success.'] }
    end
  end
end

It already looks cumbersome, don't ya?😕 But all of a sudden, users of our API requested to add the type validation to all fields and a new entity - research, that will accept an array of experiments and checks that summary power consumption is less than a defined limit. Sounds scary 😲. Doing this will be really hard with the current configuration, so let's add some dry libraries and look at how easy it will be.

The full version is here.

Update validations

➡️ First, we need to upgrade the validation process. For this will be used the gem dry-validation. It adds validations that are expressed through contract objects.

A contract specifies a schema with basic type checks and any additional rules that should be applied. Contract rules are applied only once the values they rely on have been successfully verified by the schema.

➡️ Begin by updating the ApplicationService. We replace validate_params method with validator, where can be defined either class name of the contract or contract itself. In the call, add the call of the validator and change the validation_result check, respectively.

class ApplicationService
  def initialize(*params, &block)
    @params = params
    @block = block
  end

  def call
    validation_result = validator.new.call(@params) unless validator.nil?
    if validator.nil? || validation_result.success?
      execute
    else
      { success: false, errors: validation_result.errors.to_h }
    end
  end

  def validator
    nil
  end

  def self.call(*params, &block)
    new(*params, &block).call
  end
end

➡️ Next, add the class where will be defined validations. Contracts have two parts:

schema(or params) where are defined basic type checks
rules for complex checks which are applied after schema validations.

⚠️ Be careful with optional values. You should check that value is passed before the checking.

class ConductExperimentContract < Dry::Validation::Contract
  schema do
    required(:duration).value(:integer) # in seconds
    optional(:title).value(:string)
    optional(:power_consumption).value(:integer) # in MW
  end

  rule(:duration) do
    key.failure('should be positive') unless value.positive?
  end

  rule(:power_consumption) do
    key.failure('should be positive') if value && !value.positive?
  end
end

➡️ In the ConductExperiment class, just replace the validation with the previously defined validator class.

class ConductExperiment < ApplicationService
  def initialize(params)
    super
    @params = params
  end

  def validator
    ConductExperimentContract
  end

  def execute
    if Random.rand > 0.3
      { success: true, result: 'Success!' }
    else
      { success: false, errors: ['Definitely not success.'] }
    end
  end
end

Looks great! Time to add functionality for the research process.

This class will be finished successfully if all experiments are finished successfully or will return an error message from the first failed experiment.

class ConductResearch < ApplicationService
  def initialize(params)
    super
    @params = params
  end

  def validator
    ConductResearchContract
  end

  def execute
    result = []
    @params[:experiments].each do |experiment_params|
      experiment_result = ConductExperiment.call(experiment_params)

      return experiment_result unless experiment_result[:success]

      result << experiment_result[:result]
    end
    { success: true, result: result }
  end
end

➡️ The most interesting part is the ConductResearchContract. It validates each element of the experiments array for compliance with the schema defined in the ConductExperimentContract. Sadly, now, to run rules for each experiment, you must run them manually like in the rule(:experiments).

class ConductResearchContract < Dry::Validation::Contract
  MAX_POWER_CONSUMPTION = 30

  schema do
    required(:title).value(:string)
    required(:experiments).array(ConductExperimentContract.schema).value(min_size?: 1)
  end

  rule(:experiments).each do
    result = ConductExperimentContract.new.call(value)
    unless result.success?
      meta_hash = { text: 'contain bad example' }.merge(result.errors.to_h)
      key.failure(meta_hash)
    end
  end

  rule do
    total_power_consumption = values[:experiments].reduce(0) do |sum, experiment|
      experiment[:power_consumption].nil? ? sum : sum + experiment[:power_consumption]
    end

    if total_power_consumption > MAX_POWER_CONSUMPTION
      key(:experiments).failure("total energy consumption #{total_power_consumption} MW exceeded limit #{MAX_POWER_CONSUMPTION} MW")
    end
  end
end

✨ We did it! It validates everything and works great. But we can make it even better.

The result version of this example is here.

Make it simpler with monads

In this part I won't explain what are monads and how this library works, thats very good explained in the documentation and in thesecool posts: Functional Ruby with dry-monads, Improve Your Services Using Dry-rb Stack and Five common issues with services and dry-monads. Here I will just demonstrate how it can improve the existing code.

➡️ Firstly add dry-monads gem to the Gemset.

# ...
gem 'dry-monads'
# ...

➡️ After this add the add the behaviour of monads to contracts. For this, you should run the command Dry::Validation.load_extensions(:monads). For rails applications I usually create file config/initializers/dry.rb where store all global configs for dry gems.

➡️ Next update service objects. Replace these ifs with the do notation syntax.

class ApplicationService
  include Dry::Monads[:result, :do]

  def initialize(*params, &block)
    @params = params
    @block = block
  end

  def call
    yield validator.new.call(@params) unless validator.nil?
    execute
  end

  def validator
    nil
  end

  def self.call(*params, &block)
    new(*params, &block).call
  end
end

➡️ In the ConductExperiment, wrap results to the Dry::Monads::Result monads

class ConductExperiment < ApplicationService
  def initialize(params)
    super
    @params = params
  end

  def validator
    ConductExperimentContract
  end

  def execute
    if Random.rand > 0.3
      Success('Success!')
    else
      Failure('Definitely not success.')
    end
  end
end

and replace that ugly construction with temporary variables with this neat form in the ConductResearch service.

class ConductResearch < ApplicationService
  def initialize(params)
    super
    @params = params
  end

  def validator
    ConductResearchContract
  end

  def execute
    result =
      @params[:experiments].map do |experiment_params|
        yield ConductExperiment.call(experiment_params)
      end
    Success(result)
  end
end

➡️ Because failure objects are different, result messages should also be handled differently.

class ExperimentsController < ApplicationController
  def create
    result = ConductExperiment.call(experiment_params.to_h)
    if result.success?
      render json: result.value!
    else
      error_object =
        if result.failure.is_a?(Dry::Validation::Result)
          result.failure.errors(full: true).to_h
        else
          result.failure
        end

      render(json: error_object, status: :bad_request)
    end
  end

  private

  def experiment_params
    params.require(:experiment).permit(
      :name,
      :duration,
      :power_consumption
    )
  end
end

That's all!🎆 Compare it with the previous version. I think you will agree - it looks more readable.

Result version is available here.

I hope this post was helpful for you. Have a nice day!

Small tips that will help you to use ActiveRecord callbacks more efficiently

Ivan Iliukhin — Fri, 28 May 2021 20:06:09 +0000

Everything there is related the Rails 6 but also actual for 5 and 4.

Do not use callbacks if you can - callbacks are incredibly helpful when you need to add a function that should run before/after/around specific lifecycle events. If they do not depend from the context it is possible the best choice. But if the model is used almost everywhere and there are a lot of create/update/destroy events adding new callbacks can drastically reduce the application performance. So I recommend to add them only if you absolutely confident and aware about consequences.

Despite the upper warning, you are decided to add a callback to your model. There are a few things that I recommend to keep in mind.

Use saved_change_to_attribute? in after_save/after_update to except unnecessary calls. - If your callback should be called only after changes in a limited set of attributes, this method can help with it. For example, you have the model User with fields: email, password, first_name and last_name. The goal is to update related certificates only when the first_name or last_name changed.

class User < ApplicationRecord
  after_update :update_certifcates,
             if: Proc.new{ saved_change_to_first_name? || saved_change_to_last_name? }

  private

  def update_certificates
    User::UpdateCertificates.call(self)
  end
end

Use more suitable methods whenever possible - instead of using common methods like: *_save, *_commit, *_validate is better to use more specific: *_update or *_create. It will also reduce the count of calls.
Use after_commit instead of after_save when you want to complete the action - If something went wrong in the after_commit callback there will be an error but the action will be finished because it is called when changes are already flushed to the database. Other callbacks will rollback changes, sometimes it can be useful.
Do not run expensive operations in callbacks - better to move them into ActiveJobs

If you know other problems with callbacks or know how to make live easier with them, please share in comments.

Some useful links

Rails guide about callbacks — Active Record Callbacks
Rails 6 Callbacks API - ActiveRecord::Callbacks
Kelly Sutton — 5 Rails Callbacks Best Practices Used at Gusto

CircleCI + AWS How to create CI/CD pipeline from scratch Part 2 - Setup and update servers

Ivan Iliukhin — Sat, 25 Jul 2020 09:35:48 +0000

After all previous steps, you've got the images that are stored inside the ECR and the script that automatically build
them. In this part of the tutorial we will:

setup ECS clusters for development and production environments;
add deployment commands to the CircleCI script.

Let's do it!

Initialize ECS cluster

Creating of cluster consists of three steps:

Create an empty cluster with a VPC
Define the task that will launch the selected container
Add service that will launch and maintain the desired count of ec2 instances with the previously defined task

Create cluster

Let's start with the definition of cluster:

An Amazon ECS cluster is a logical grouping of tasks or services.

Roughly saying, clusters define the scope and set of rules for the launched tasks. To create one, follow the steps below:

Select EC2 Linux + Networking you need two clusters one for development and one for master branches
Select 1 On Demand t2.micro instance(or other types of ec2 instances), other configurations by default
Create cluster
For the networking section, I recommend to use the default parameters too. It will create a new VPC with
security group allowing income traffic
to the 80 PORT.

Configure network during creation

Yes, that's all, you should create one for the development and one for the production.

Define task

Tasks are used to run Docker containers in Amazon ECS.

Select EC2 launch type compatibility
Choose the name for the task definition(or family, like sometimes it's called)
For the as the task role choose the role with the AmazonECSTaskExecutionRolePolicy policy that we previously created
Select the memory limit, for example, 128 MB if your server is not going to handle a lot of complex requests
And finally, add a container. On this tab we are interested in the following fields:
- Standard -> Image - initial image for the task should be copied from the ECR looks like 845992245040.dkr.ecr.us-west-2.amazonaws.com/simple_plug_server:master_latest
- Standard -> Port mappings - associate host 80 with the port which is using our application and will be defined next Add container to a task and configure the port mapping

Advanced container configuration -> ENVIRONMENT -> Environment variables - define the variable with the name PORT and desired value for example - 4100. This value must be used in the port mapping as the container port Set up environment variables

Great! You've created the first revision of the task definition. Of course, these tasks can be launched right inside the cluster, but we will use services to simplify updating tasks revisions. Let's add them.

Add service

An Amazon ECS service enables you to run and maintain a specified number of instances of a task definition simultaneously in an Amazon ECS cluster.

To create a service just click on the Create button on the Services and fill the form:

Select EC2 in the Launch type selector, because we are deploying our tasks on EC2 instances
In the Task Definition select the task and revision that you are created earlier
In the Cluster select the cluster where you want to define a service. When you are creating service from a cluster this field will be already selected
Service type - REPLICA
Number of tasks - 1 because we do not care about scaling for now.
Other setting set by default

Service creation

After repeating all steps for the development and production cluster, you've got two EC2 instances with running applications. They are available by Public DNS or IP.

Service creation

Now let's go to the final part of this tutorial. 🚀

Add deployment scripts

With an official orb aws-ecs you can make this very easy. We already added all necessary environment variables in the second part of this tutorial so you should only modify the circleci config.

Result version of the .circleci/config.yml

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@6.9.1
  aws-ecs: circleci/aws-ecs@1.2.0
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test-build-deploy:
    jobs:
      - test
      - aws-ecr/build-and-push-image:
          repo: "simple_plug_server"
          tag: "${CIRCLE_BRANCH}_${CIRCLE_SHA1},${CIRCLE_BRANCH}_latest"
          requires:
            - test
          filters:
            branches:
              only:
                - master
                - development
      - aws-ecs/deploy-service-update:
          name: deploy-development
          requires:
            - aws-ecr/build-and-push-image 
          family: "simple-plug-server-development"
          cluster-name: "SimplePlugServer-development"
          service-name: "sps-dev-serv"
          container-image-name-updates: "container=simple-plug-server-development,tag=${CIRCLE_BRANCH}_${CIRCLE_SHA1}"
          filters:
            branches:
              only:
                - development
      - approve-deploy:
          type: approval
          requires:
            - aws-ecr/build-and-push-image
          filters:
            branches:
              only:
                - master
      - aws-ecs/deploy-service-update:
          name: deploy-production
          requires:
            - approve-deploy
          family: "simple-plug-server-production"
          cluster-name: "SimplePlugServer-production"
          service-name: "simple-plug-server-production"
          container-image-name-updates: "container=simple-plug-server-production,tag=${CIRCLE_BRANCH}_${CIRCLE_SHA1}"
          filters:
            branches:
              only:
                - master

In this file was added three new jobs. Two aws-ecs/deploy-service-update respond for the updating respective services in the clusters and approve-deploy that's waiting for confirmation before the last step for the master branch. For different branches, flows will be a little different. It can be achieved by using parameter filters in job definitions, where you can specify for which branches or git tags launch jobs.

aws-ecs/deploy-service-update job configuration

I would like to tell you about the parameters for the job aws-ecs/deploy-service-update:

name - the name is used to make jobs in a workflow more human-readable. I am sure you would agree that's deploy-production looks much more clearer than aws-ecs/deploy-service-update.
requires - used to define the order of jobs execution, namely the previous job that must be finished successfully.
family - there you should write the name of the task definition(Define task) that you used when you created the task
cluster-name - it's pretty obvious - the name of the desired cluster where all magic happens
service-name - the name of the service that's managing tasks inside the previously mentioned cluster
container-image-name-updates - updates the Docker image names and/or tag names of existing containers that had been defined in the previous task definition
- container - the name of the container that you used when you added the container to the task(circled in blue on the screenshot Add container to a task and configure the port mapping)
- tag - one of the tags that you are defined in the aws-ecr/build-and-push-image job, in this example it's a ${CIRCLE_BRANCH}_${CIRCLE_SHA1}

And that's all 🎉. When you push your branch with the new circleci config and start to work you will see something like that.

Service creation

I hope that this tutorial was helpful and was not wasted your time. If you have any question and problems feel free to ask me about it in the comments. 👋

CircleCI + AWS How to create CI/CD pipeline from scratch Part 1 - Build and push images

Ivan Iliukhin — Sat, 25 Jul 2020 09:35:34 +0000

Prepare environment for containers

Create an IAM user

Right after the creation of an AWS account, you are logging in under the root user. You have full access to every service and the billing management console. To secure interaction with AWS it is a good practice to create a new user inside the group that has only required permissions.

A few words about managing permissions. There are two main ways to add them to users through groups and roles. The main difference is in that groups are a collection of users with the same policies. Roles, in turn, can be used to delegate access not only to users but also to other services and applications, we will use both. Let's create them.

Create group window

On the second step select the next policies

AmazonEC2ContainerRegistryFullAccess
AWSCodeDeployRoleForECS
AmazonEC2ContainerServiceFullAccess
AmazonECSTaskExecutionRolePolicy

Group after creation

Then create the role that we will give to ECS to deploy our containers to EC2 instances.

Create role

and on the second step select in policies AmazonEC2ContainerServiceforEC2Role

Result

More about it is showed there

And finally, let's add a new user and add to the previously created group:

New user

This user should have only programmatic access because you will use it only from the circleci. After it, generate access keys and save them, they will need you later.

Create ECR

ECR a place where you will store containers from they will be deployed. Just go to the ECR and click on the "Create repository" button. You will see the window where you should select the name for the repository. Other
settings use by default

ECR Create

Great! You have a repository and all required credentials to build and push images. Time to automatize it.

ECR After creation

Configuring Circle CI

The main idea is to run tests after each commit for all branches and deploy after changes in the development and master.

Before you start to configure the pipeline, you will need to prepare the application
following this fantastic getting started page.

Tests

The most popular use case of the Circle CI that I've seen is running tests (not all developers trust to external
services to deploy applications). To run them you should define a job
and add it as a step to a workflow. There is an example of test workflow
for the simple_plug_server application:

version: 2.1
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test:
    jobs:
      - test

It has only the one workflow test with the one job test. This job has three parts:

docker - where is defined as a container inside which you will deploy test environment and run tests
working_directory - the name of the folder where everything is happening
steps - the set of commands where you download code, setup dependencies and finally run tests. You can also cache dependencies on this step.

We can also improve the representing of the failed tests, for it you should add a JUnit formatter for test results
(for elixir it is the hex package JUnitFormatter) and specify
the directory containing subdirectories of JUnit XML or Cucumber JSON test metadata files.
More information about it and how to add support for other languages and test frameworks look here.

Build and push containers

On the previous steps we created the ECR repository and the user that can push images, time to setup CircleCI config.

For work with images we will use the official orb for ECR circleci/aws-ecr@6.9.1
It significantly simplifies building and pushing images, let's add the new step to our config file:

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@6.9.1
  aws-ecs: circleci/aws-ecs@1.2.0
jobs:
  test:
    docker:
      - image: elixir:1.10
        environment:
          MIX_ENV: test
    working_directory: ~/repo
    steps:
      - checkout
      - run: mix local.hex --force
      - run: mix local.rebar --force
      - run: mix deps.get
      - run: mix deps.compile
      - run: mix test
      - store_test_results:
          path: _build/test/lib/simple_plug_server
workflows:
  version: 2
  test-and-build:
    jobs:
      - test
      - aws-ecr/build-and-push-image:
          repo: "simple_plug_server"
          tag: "${CIRCLE_BRANCH}_${CIRCLE_SHA1},${CIRCLE_BRANCH}_latest"
          requires:
            - test
          filters:
            branches:
              only:
                - master
                - development

Briefly about the steps of this job:

repo - the name of the repository (last part of the 815991645042.dkr.ecr.us-west-2.amazonaws.com/simple_plug_server)
tag - tags that we apply to the built container, for the master branch it will add two tags: master_02dacfb07f7c09107e2d8da9955461f025f7f443 and master_latest
requires - there you should describe the previous necessary steps, in this example we build an image only if all tests pass
filters - describe for which branches this job should execute. There are a lot of other filters that you can use to customize a workflow

But before you start to run this workflow you should add the next environment variables:

AWS_ACCESS_KEY_ID - access key for circleci that you obtained on this step
AWS_SECRET_ACCESS_KEY - secret key for circleci that you obtained on this step
AWS_REGION - region where placed your ECR instance
AWS_ECR_ACCOUNT_URL - url of the ECR(looks like 815991645042.dkr.ecr.us-west-2.amazonaws.com)

CircleCI ENV Settings example

After successful build of the development and master branches you will see something like there:

ECR after the successful push

Great! You automatized process of running tests and building images in the next chapter you will see how to
setup servers on the AWS infrastructure and redeploy them after successfully passed tests.

CircleCI + AWS How to create CI/CD pipeline from scratch Part 0 - Preparation

Ivan Iliukhin — Sat, 25 Jul 2020 09:35:20 +0000

Introduction

In this series of posts, I'm going to show how to set up the CI/CD environment using AWS and CircleCI.
As a final result, you will get the pipeline that:

runs tests after each commit;
builds containers from development and master branches;
pushes them into the ECR;
redeploys development and production EC2 instances from the development and master containers respectively.

This guide consists of three parts:

➡️ Preparation - where I'll explain this workflow, and show how to prepare an application for it
AWS - this chart is about how to set up the AWS environment from scratch;
CircleCI - here I'll demonstrate how to automatize deployment process using circleci.com

Workflow

When you developing applications in "real life" you usually(but not always), sooner or later, found that you need to:

🤖 run automatic tests;
🔍 run different checks(code coverage, security audit, code style, etc.);
🧪 test how a feature works before you deploy it to the production;
💸 deliver results as fast as possible.

There is an example of a workflow that was used in many projects where I've participated. It works very well in small(3-8 person) teams:

1) Create a branch for a feature from master
2) Work
3) Push this feature
4) Optionally Run tests, checks, etc. for this branch
5) In case of success merge this branch to the development branch
6) Run everything again and redeploy the development server
7) Test it manually
8) Merge feature to the production branch
9) Redeploy production

Workflow example

I'm going to show how it can work on the workflow with two main branches:

master - has tested, production-ready code(deploys on a production server after approving)
development - based on the master and also has changes that being tested now(deploys on a development server right after changes on the github)

Prepare the application

As an example let's take a simple API server with one route written on the Elixir. I will not explain here how to create it, there is a fantastic post about. Or you can use my application, it is already configured and prepared.
There I'll focus only on the specific moments that are needed to prepare the server for work in this environment.

Prepare release application

This Elixir application I am going to deploy using mechanism of releases.
Briefly, it generates an artefact that contains the Erlang VM and its runtime, compiled source code and launch scripts.

Let me make a little digress to tell about the methodology I'm trying to follow for designing microservices.
I'm talking about 12 factor app manifest. It's a set of recommendations for building software-as-a-service apps that:

Use declarative formats for setup automation, to minimize time and cost for new developers joining the project.
Have a clean contract with the underlying operating system, offering maximum portability between execution environments.
Are suitable for deployment on modern cloud platforms, obviating the need for servers and systems administration.
Minimize divergence between development and production, enabling continuous deployment for maximum agility.
And can scale up without significant changes to tooling, architecture, or development practices.

One of these principles recommends us to store configurable parameters(ports, API keys, services addresses, etc.) in system environment variables. To configure our release application using them you should create the file config/releases.exs and describe these variables:

import Config

config :simple_plug_server, port: System.get_env("PORT")

More about different config files in Elixir applications you can find here
and here

Launch script

Next thing I would like to cover is starting an application. The most common way is to use a special shell script
for it that contains different preparation steps like waiting for a database, initializing system variables, etc. Also,
it makes your Docker file more expressive. I think you will agree that CMD ["bash", "./simple_plug_server/entrypoint.sh"] looks better than CMD ["bash", "cmd1", "arg1", "arg2", ";" "cmd2", "arg1", "arg2", "arg3"]. The entrypoint script for this server is very simple:

#!/bin/sh

bin/simple_plug_server start

This application works in the docker container so the last command bin/simple_plug_server start starts
app without daemonizing it and writes logs right into the stdout. That allows us to gather logs simpler(AWS Cloudwatch gathers these data without additional configs).

Dockerization

And on the last step, you should create the Dockerfile that builds result container. I prefer to use two steps builds for Elixir applications because result containers are very thin(approx. 50-70MB).

FROM elixir:1.10.0-alpine as build

# install build dependencies
RUN apk add --update git build-base

# prepare build dir
RUN mkdir /app
WORKDIR /app

# install hex + rebar
RUN mix local.hex --force && \
    mix local.rebar --force

# set build ENV
ENV MIX_ENV=prod

# install mix dependencies
COPY mix.exs mix.lock ./
COPY config config
RUN mix deps.get
RUN mix deps.compile

# build project
COPY lib lib
RUN mix compile

# build release
RUN mix release

# prepare release image
FROM alpine:3.12 AS app
RUN apk add --update bash openssl

RUN mkdir /app
WORKDIR /app

COPY --from=build /app/_build/prod/rel/simple_plug_server ./
COPY --from=build /app/lib/simple_plug_server/entrypoint.sh ./simple_plug_server/entrypoint.sh
RUN chown -R nobody: /app
USER nobody

ENV HOME=/app

CMD ["bash", "./simple_plug_server/entrypoint.sh"]

Finally you can build it using docker build ., and run docker run -it -p 4000:4000 -e PORT=4000 {IMAGE_ID}.
The server will be available on the localhost:4000 and will write logs to the stdout. 🎉

On the next part of this tutorial, I'll show how to automatically build images using CircleCI and push them to the Amazon ECR.

Rails API + React SPA authentication problem — Authentication by cookies

Ivan Iliukhin — Tue, 24 Dec 2019 17:28:34 +0000

Introduction

In this series of articles, I’ll cover different ways for a user authentification in systems with separated frontend and backend. As an example, I took my lovely programming language Ruby with RoR, with which I’ve been working already four years, for API and React application, based on CRA template, for separated frontend.

Source code for SPA you can find here. For API — here.

Problem

Imagine that some people request to develop a system for storing the most valuable thing for them — their names. Besides, users love to admire their treasure only personally. For it, they wish that the system must show name only after logging in and must not ask it for one week. Moreover, they are planning to develop GUI and API by different teams, so these parts must be independent applications.

Design — API

A core entity of the API has a model User that contains only three fields:

login — string which users do not scare to show;
password — stored as a password digest;
name — sacred for every user information that we only show when they are authorized.

Design — SPA

The page has only one block, that is show login form if user not authorized and not blank field “Name” above in case of successful authentication.

Let’s go further and consider how to authenticate our users by cookies.

Authentication by cookies

The most common and obvious approach is to use HTTP cookies for storing auth information. Ruby on Rails has two similar mechanisms for working with cookies, it’s cookies themselves and sessions. For cookies, we can set an httponly flag, that protects from xss attacks, domain, and expiration date. Sessions are stored in cookies inside an encrypted string, where an httponly flag is set by default. For this example, I took sessions because the SPA does not read from cookies.

How it works:

SPA sends a POST request with login and password
API write user.id in the session cookie
Component try to get the name of the user sending a request with the session
API find a user by user id and if all right return name of this user
Component is updated

Let’s dive deeper.

Usually, SPA and API are deployed on different hosts, hence there appears the next problem — how to pass and modify cookies. By default browser does not set cookies from another origin by javascript. But we can easily enable it.

SPA side.

For communicating with a server SPA uses the Fetch API that is provided in a global window scope. For enabling a possibility to send and receive cookies from resources with a different origin. We must set the next option:

credentials: ‘include’ — it enables sending cookies for cross-origin requests by default it is set for the same origin;
mode: ‘cors’ — allows to work with all headers related to CORS. By default, it allows only for same-origin requests.

Examples you will find further.

Server side.

To enable the supporting of cross-origin requests in RoR, you must add gem rack-cors that provides support for CORS for a Rack middleware. When you create rails application from a generator with API you need only uncomment string “gem ‘rack-cors’” in Gemfile and content of the config file config/initializers/cors.rb. For setting cookies you must set parameter credentials as true. Important notice, it works only if the origin is not a wildcard. For security reason and flexibility I usually set it from environment variables like there:

Rails.application.config.middleware.insert_before 0, Rack::Cors do
  allow do
    origins ENV['SPA_ORIGIN']

    resource '*',
      headers: :any,
      methods: [:get, :post, :put, :patch, :delete, :options, :head],
      credentials: true
  end
end

Sending and handling requests

After the setting our projects for work with cookies let’s look at how are requests handled.

Post request contains data and cors friendly settings, about I mentioned above.


    const authUrl = apiUrl + 'login'
    let payload = {
      'data': {
        'login': this.state.login,
        'password': this.state.password
      }
    }

    let headers = {
      'Content-Type': 'application/json'
    };

    fetch(authUrl, {
      method: 'POST',
      mode: 'cors',
      cache: 'no-cache',
      headers: headers,
      redirect: 'follow',
      referrer: 'no-referrer',
      body: JSON.stringify(payload),
      credentials: 'include'
    });

Request handled by standard Rails controller. API finds a user and if all right writes user’s id in a session.

class AuthController < ApplicationController
  include ::ActionController::Cookies

  def login
    if params['data']['login'] && params['data']['password']
      user = User.find_by(login: params['data']['login'])
      if user && user.authenticate(params['data']['password'])
        session[:user_id] = user.id
      else
        render json: {message: 'Wrong login or password'}, status: 403
      end
    else
      render json: {}, status: 401
    end
  end
end

Next requests for getting the name send this session and controller just read it and sends name.

let username_url = apiUrl + "name";

let headers = new Headers({
  'Content-Type': 'application/json'
});

if(this.state.name === null) {
  fetch(username_url, {
    method: 'GET',
    mode: 'cors',
    headers: headers,
    cache: 'no-cache',
    redirect: 'follow',
    referrer: 'no-referrer',
    credentials: 'include'
  })
  .then(function (response) {
    return response.json();
  })
  .then(myJson => {
    this.setState({name: myJson['name']});
  });
};

..and related controller:

class UsersController < ApplicationController
      include ::ActionController::Cookies
      before_action :find_user

      def name
        if @current_user.present? && @current_user.is_a?(User)
          render json: {name: @current_user.name}
        else
          render json: {message: 'Bad user'}, status: 401
        end
      end

      private

      def find_user
        user_id = session[:user_id]
        @current_user = User.find_by(id: user_id)
      end
end

Pretty simple!

Pros

Security — httponly flag prevents cookies from stealing your auth data by XSS attacks. (I hope that you use https by default).

Simplicity — mechanisms for working with cookies and sessions are proven and exist almost in all frameworks.

Cons

Works only inside with web-browsers.

Ruby Interview Tasks - Simple word calculator

Ivan Iliukhin — Mon, 23 Dec 2019 16:17:46 +0000

Recently during an interview live coding session, I got a pretty interesting task:

# Make this code working
one.plus.two.equal # =>> 3
one.minus.three.equal # => -2

I was tired and tried to solve it by adding some method to a String instance methods like there:

one = '1'

def one.plus
  temp_plus = "#{self}+"

  def temp_plus.two
    temp_plus_two = "#{self}2"

    def temp_plus_two.equal
      eval(self)
    end

    temp_plus_two
  end

  temp_plus
end

def one.minus
  temp_minus = "#{self}-"

  def temp_minus.three
    temp_minus_three = "#{self}3"

    def temp_minus_three.equal
      eval(self)
    end

    temp_minus_three
  end

  temp_minus
end

# Yes, it works
pp one.plus.two.equal # => 3
pp one.minus.three.equal => -2

Looks bulky and terrible. Yes, Ruby allows to do this, but it is natural monkey patching, and I would not be used similar construction in real projects.
Six hours later, when I woke up, I realized that there is exist a significantly more straightforward and elegant solution. It’s just a realization of the method chaining pattern:

class Evaluator
  class << self
    def operation(name, expression_part)
      define_method(name) do
        return_new_instance expression_part
      end
    end
  end

  operation :plus,  '+'
  operation :minus, '-'
  operation :two,   '2'
  operation :three, '3'

  def initialize(expression = '')
    @expression = expression
  end

  def equal
    eval(@expression)
  end

  private

  def return_new_instance(expression_part)
    @expression = "#{@expression}#{expression_part}"
    self.class.new(@expression)
  end
end

one = Evaluator.new('1')
pp one.plus.two.equal # => 3
pp one.minus.three.equal # => -2

Pretty neat!
This task is well for estimating the level of a developer. Specifically, the knowledge of method chaining pattern(interviewee mentioned it or just used it), how to use metaprogramming and awareness about monkey patching and bad solutions.

Guide to install PyTorch with CUDA on Ubuntu 18.04

Ivan Iliukhin — Fri, 20 Dec 2019 18:40:28 +0000

Guide to install PyTorch with CUDA on Ubuntu 18.04

Yesterday I was installing PyTorch and encountered with different difficulties during the installation process. Let me share the resulting path, that brought me to the successful installation. I hope this little instruction will save you time and show further direction.

Install Python3.8

PyTorch requires Python version 3.7 or above, so I decided to install the latest stable version on this moment Python 3.8. Standard Ubuntu repository has maximum 3.6, so I added repository from the “deadsnakes” team link.

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt install python3.8 python3.8-dev python3.8-venv

python3.8-dev - requires for building C extensions
python3.8-venv - provides support for creating lightweight “virtual environments” with their own site directories

Install CUDA

My graphic card supports CUDA, so why not take advantage of this.
Just choose one of the type of installation there. I selected deb(network):

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pinsudo 
mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo add-apt-repository "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/ /"
sudo apt-get update
sudo apt-get -y install cuda

Install pip

Before pip installation, you should create a new virtual environment for Python. On the first step, we installed python3.8-venv exactly for this purpose.

python3.8 -m venv ~/python_env/my_env

This command creates a new local environment in your local folder. After the directory has been created "activate" this environment by the next command. (Works only for bash/zsh, for other shells look there)

source ~/python_env/my_env/bin/activate

Then install pip following by the (instruction)[https://pip.pypa.io/en/stable/installing/]

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py

Perfect! Let's go to the finish. 🏁

And the final step - Get, compile and install PyTorch

Firstly install cmake and clone the necessary version of the PyTorch. Don't be afraid, repository really big, so it takes a lot of time:

sudo apt install cmake
git clone --recursive https://github.com/pytorch/pytorch.git
git submodule sync
git submodule update --init --recursive

Than install all required Python packages using pip:

Ensure that you use right Python virtual environment

python3.8 -m pip install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi typing

Ensure that you are in the cloned directory and run the next commands:

export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
python setup.py install

🎉 Congratulations, you are installed the PyTorch in your local virtual environment. Time to check it!

Running

Create the file torch_program.py and add the next lines:

from __future__ import print_function
import torch

x = torch.ones(5, 5)

print(x)

if torch.cuda.is_available():
    device = torch.device("cuda") # a CUDA device object
    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU
    x = x.to(device)                       # or just use strings ``.to("cuda")``
    z = x + y
    print(z)
    print(z.to("cpu", torch.double))

If all installed correctly there will be the followed resuls:

tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])
tensor([[2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.]], device='cuda:0')
tensor([[2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.],
        [2., 2., 2., 2., 2.]], dtype=torch.float64)

If you get stuck following this manual feel free to write in comments about it, I'll be sure to expand this post with the new information.

Happy Machine Learning! 🤖📖

Rails API + SPA authorization — Authorization by JWT

Ivan Iliukhin — Tue, 29 Oct 2019 08:55:45 +0000

One of the ways to provide authorization is to use JSON Web Tokens or JWT. It is a popular way to implement authorization between different services by logging in only once. In this blog post, I’ll briefly explain what is it, tell about some pitfalls and show how to add in an application.

What is it, and why is it so popular?

Briefly, it’s a signed string that contains some information in a not ciphered form and a verification signature, that proves that information was not changed. JWTs doesn’t protect data from stealing. They are only protected from changing. The closest analogy from the real world is an RFID ID card with the printed on it information. Everyone can see what is written on a card, but only who know secret can change it. Furthermore, someone can steal your card and freely use one. I will not explain in detail about the structure of JWTs, it very clear and short described on the official site.

And why is this approach so popular today?

Most of the modern applications consist of a bunch of microservices developing by different teams. First returns frontend, second provides an API, third gathers statistics of usage, etc., and they communicate with each other. We want to know some information about a request (who made it, user type and etc.) and that this information has not tampered.

There are listed the most popular cases of usage JWTs:

authorization between independent SPA and API;
providing authorization for different web-applications using only one for an authentification;
secured information exchange between services, tokens guarantees that information was not changed during transmission.

JWT is a good solution for authorization but not without some controversies.

Let’s go to nuances of usage and caveats.

Confidential information exchange

This mentioned in almost all posts about JWTs, and I will remind too.

DO NOT USE JSON WEB TOKENS FOR STORING AND PASSING CONFIDENTIAL DATA!
Like passwords, information from official documents and etc. Everyone who has a token sees the content.

Stateless requests

In many articles are spoked of the next advantage of JWTs— stateless. It means that we store all necessary information inside the token, though we can reduce the amount of connection to a central database. It makes sense in the environment with dozens of microservices. But, what if a token was compromised. Yes usually it has an expiration date, but what if we must cancel it right now.

There is a simple solution, unfortunately, it violates the mentioned above advantage. We can store blacklisted tokens in a database (Redis for example) to deny access by using an invalid token. Another good idea is to save user sessions(user id + valid token + token expiration date). It will be a little bit slow, but we can always get the information about how much users are logged in. Furthermore, the application must go to a database every time processing a request, as a consequence recommended to use for a blacklist/session storage an in-memory database with O(1) access time.

In many cases, it is not necessary, however, keep it in mind when you decide to use JWTs for users authentication/authorization.

Time to practice

For this service, I used the most popular authentification solution for the Ruby on Rails — Knock. This gem leverages efforts for creating and decoding tokens. Let’s look at how it easy.

add gem to the project link;
add finding user by the login to the User model(by default Knock uses email field);

def self.from_token_request(request)
  # Returns a valid user, `nil` or raise `Knock.not_found_exception_class_name`
  login = request.params['auth'] && request.params['auth']['login']
  user = find_by(login: login)
  if user.present?
    return user
  else
    raise Knock.not_found_exception_class_name
  end
end

Create UserTokenController to generate token and UsersController, that returns user’s name, when token is correct and user still exists

module Api
  module Jwt
    class UsersController < ApplicationController
      include Knock::Authenticable

      before_action :authenticate_user

      def name
        render json: {name: current_user.name}
      end
    end
  end
end

and controller, that generates tokens for the User.

module Api
  module Jwt
    class UserTokenController < Knock::AuthTokenController
      skip_before_action :verify_authenticity_token
    end
  end
end

Pretty simple!

On this scheme is depicted how it works.

We make POST request with the form-data(login and password in our case) and get token in response. This token can be attached as the bearer authentification token in headers, or just transferred in a request body.

Yes, it looks like a simple and working solution, but what can happen if you decide to use it in a web application(SPA, for example) that works from a browser. A gotten token can be stored by javascript to next usage in headers to local storage or normal cookies(not HTTP only), it means that anyone(third-party javascript libraries, browser extensions, etc.) has access to this token. What is wrong with it? Correct! The token is vulnerable to XSS attacks.

But there is a way to avoid it — cookies with HttpOnly flag. In Rails you can use or sessions (special encrypted cookies, httponly by default) or just cookie with this flag. The main difference between them is next. A session cookie is encrypted by default, a usual cookie you must encrypt manually. I’ll show how to realize it on sessions.

JWT inside session

By default sessions are destroyed right after closing a browser, though if you want to increase their expiration date add this parameter to the session initializer config/initializers/session_store.rb:

Rails.application.config.session_store(:cookie_store, expire_after: 7.days)

or to the middleware in config/application.rb when you use api-only application:

config.middleware.use ActionDispatch::Session::CookieStore, expire_after: 7.days

In case, when you want to store and configure it separately from session use encrypted, httponly cookie.

Next, you should redefine current knock behaviour for creating token:

class UserTokenController < Knock::AuthTokenController
  include ::ActionController::Cookies
  skip_before_action :verify_authenticity_token

  def create
    session[:jwt] = auth_token.to_json
    render body: nil, status: :created
  end
end

As you can see, I skipped CSRF verification, because this API works with SPA with another origin and Knock::AuthToken depends from AuthController::Base, where it enabled by default. In the redefined method create I add token to the session and render nothing with the status code 201.

UsersController will be a little bit more complicated:

class UsersController < ApplicationController
  include ::ActionController::Cookies
  before_action :find_user

  def name
    if @current_user&.is_a?(User)
      render json: { name: @current_user.name }
    else
      render json: { message: 'Bad user' }, status: 401
    end
  end

  private

  def find_user
    jwt = session[:jwt]
    if jwt.present?
      token = JSON.parse(jwt)['jwt']
      user_id = Knock::AuthToken.new(token: token).payload['sub']
      @current_user = User.find_by(id: user_id)
    else
      render json: { message: 'Token not found' }, status: 401
    end
  end
end

In find_user I try to find the token in the session and decipher it. How it looks on scheme:

That’s all. We saw how to authenticate a user by using JWT itself and with combination with cookies using Knock.

P.S. Some useful articles that helped me to understand this topic

Flavio Copes - JWT authentication: When and how to use it
Jan Brennenstuhl— Stateless JWT authentification
Sophie DeBenedetto — Great short article about adding JWT to Rails