DEV Community: Pramod Shinde

Rails: Searchable logs with Filebeat and Elastic Stack

Pramod Shinde — Sat, 17 Apr 2021 15:17:16 +0000

In a data sensitive application debugging is the most critical and time consuming especially if requests traffic is tremendously high, when it comes to debugging every developer’s last resort is application logs and debugging in large logs file is not an easy task.

In this post I will brief you about how we can make rails logs searchable, Using Filebeat and Elastic Stack for making debugging fast and easy via UI enabled log search. Wait…, Filebeat is not a separate entity it falls under an elastic stack only which is a lightweight log shipper which will not hoard your resources.

This post will cover following

Brief intro to Elastic Stack
Filebeat configuration and setup
Custom Index Template with rollover for application logs
Lograge setup to output logs into JSON format
Demo Rails application refer this repository

Brief about Elastic Stack

Elastic stack comprises of Elasticsearch, Kibana, Beats, and Logstash formerly known as the ELK Stack which is used to monitor, visualize, search and analyse the applications data in realtime and Filebeat is a lightweight data shipper belonging to the Beats family which is mainly used to ship data from files.

Filebeat setup and configuration

Installation

Installation of filebeat is dead simple. You can follow simple steps described in official documentation.

Sample filebeat.yml – /usr/local/etc/filebeat/filebeat.yml

#---------- Filebeat inputs ----------
filebeat.inputs:
- type: log
  enabled: true
  json:
    message_key: message
    keys_under_root: true
    overwrite_keys: true
  paths:
    - /Users/pramod/projects/efk-rails/log/sync_development.log
#ElasticSearch template setting 
setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "efk-rails-sync-dev"
setup.template.pattern: "efk-rails-sync-dev-*"
#---------- Kibana ----------
setup.kibana:
  host: "0.0.0.0:5601"
#---------- Elasticsearch output ----------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["0.0.0.0:9200"]
  index: "sync-log-dev"

In above configuration file we have configured following options

inputs – Used for specifying input for filebeat, here we have configured input as rails log file /Users/pramod/projects/efk-rails/log/sync_development.log with JSON option which will decode logs structured as JSON messages
output – Used to set the output for filebeat, here we will write to elasticsearch indexes.
setup.kibana – the Kibana endpoint which will load the kibana dashboard

Custom Index Template with rollover

We off course can use the default template and rollover options but you cannot use the custom index template names, this is specially needed if you want the setup environment wise kibana dashboards on a single kibana and elasticsearch instances and this is what we needed in most of the Rails applications.

Here we will use the custom template and index lifecycle management (ilm) for rotating our logs indexes as per different rails environments.

To enable a custom index template name and rollover policy you need to follow the following steps

Setup a ilm enabled to false, add custom template name and pattern with following options

setup.ilm.enabled: false
setup.template.enabled: false
setup.template.name: "efk-rails-sync-dev"
setup.template.pattern: "efk-rails-sync-dev-*"

Above config will tell filebeat to use efk-rails-sync-dev and efk-rails-sync-dev-* name and index pattern respectively, It’s a common practice to specify the index pattern rather than using index name.

Create a life cycle policy using Kibana dev tool

PUT _ilm/policy/sync-log
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "3d",
            "max_size": "100mb"
          }
        }
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Above will create index lifecycle policy which will rollover indexes if its size reaches to 100mb or index age is 3 days whichever turns first true.

Create template pattern using Kibana dev tool

PUT _template/efk-rails-sync-dev
{
  "index_patterns": ["efk-rails-sync-dev-*"], 
  "settings": {
    "number_of_shards": 1,
    "index.lifecycle.name": "sync-log", 
    "index.lifecycle.rollover_alias": "sync-log-dev"
  }
}

This will create a template efk-rails-sync-dev with index alias sync-log-dev and attach lifecycle policy sync-log that created in step 2.

In last step we will create the first index managed by a policy that uses the rollover action and designate it as the write index

PUT efk-rails-sync-dev-000001 
{
  "aliases": {
    "sync-log-dev":{
      "is_write_index": true
    }
  }
}

This will create initial index efk-rails-sync-dev-000001 which will be used by filebeat to write output of application.

Application logs in JSON format

Once you create index life cycle policy, template and first rollover index which will capture logs from filebeat harvester. You need make your application logs to output in JSON format.

To output rails logs into JSON format, we are using lograge gem once you add it in Gemfile and bundle install it will be available to use in you application.

For demo purpose, We will be logging only single controller actions into JSON and search via Kibana.

Following lograge configuration will output logs in JSON for SyncContoller#index and SyncController#Create action

$config/initializers/lograge.rb

Rails.application.configure do
  config.lograge.enabled = true
  config.lograge.keep_original_rails_log = true
  config.lograge.base_controller_class = 'ActionController::API'
  config.lograge.logger = ActiveSupport::Logger.new "#{Rails.root}/log/sync_#{Rails.env}.log"
  config.lograge.formatter = Lograge::Formatters::Json.new
  config.lograge.ignore_custom = lambda do |event|
    # return true here if you want to ignore based on the event
    return false if event.payload[:controller].include?("SyncController") &&
     ['index', 'create'].include?(event.payload[:action])
    return true
  end
  config.lograge.custom_payload do |controller|
    if controller.class.name.include?('SyncController')
      SyncLogger.new(controller).log_data
    end
  end
end

SyncLogger service to add some meta data to logger file
$app/services/sync_logger.rb

class SyncLogger
  def initialize(controller)
    @controller = controller
  end
  def log_data
    request_type = @controller.request.method
    username = @controller.current_user.try(:username) || 'guest'
    r_data = request_type == 'GET' ? JSON.parse(@controller.response.body) : @controller.request.parameters
    log_data = {
      u_agent: @controller.request.user_agent,
      request_time: Time.now.to_s(:number),
      search_key: "#{request_type}_#{username}_#{Time.zone.now.to_s(:number)}",
      ip: @controller.request.ip,
      user_id: @controller.current_user.try(:id),
      username: username,
      data: r_data
    }
  end
end

Above service will create JSON output for SyncContoller#index and SyncController#Create action in following format

$log/sync_development.log

{"method":"GET","path":"/sync","format":"*/*","controller":"SyncController","action":"index","status":200,"duration":16.64,"view":1.18,"db":0.2,"u_agent":"PostmanRuntime/7.26.1","request_time":"20200818210529","search_key":"GET_pramod_20200818153529","ip":"127.0.0.1","user_id":1,"username":"pramod","data":[{"username":"pramod"},{"username":"pramdo1"}]}
{"method":"GET","path":"/sync","format":"*/*","controller":"SyncController","action":"index","status":200,"duration":15.82,"view":0.48,"db":0.93,"u_agent":"PostmanRuntime/7.26.1","request_time":"20200818210647","search_key":"GET_pramod_20200818153647","ip":"127.0.0.1","user_id":1,"username":"pramod","data":[{"username":"pramod"},{"username":"pramdo1"}]}

Starting filebeat to ship above logs to elasticsearch by executing following command from the root of the filebeat if you have downloaded the tar and extracted it or if you have installed filebeat as a service you can start filebeat as a service

./filebeat -e
or 
sudo service filebeat start

You should see following filebeat logs on successful
logs output

INFO    registrar/registrar.go:152  States Loaded from registrar: 2
2020-08-18T21:46:30.722+0530    INFO    crawler/crawler.go:72   Loading Inputs: 1
2020-08-18T21:46:30.723+0530    INFO    log/input.go:152    Configured paths: [/Users/pramod/projects/efk-rails/log/sync_development.log]
2020-08-18T21:46:30.723+0530    INFO    input/input.go:114  Starting input of type: log; ID: 15918551478222648142
2020-08-18T21:46:30.723+0530    INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 1

Kibana in Action

If above all is set correctly, We can search logs via Kibana by creating index pattern. You should see efk-rails-sync-dev-000001 index in kibana index management, Using same index you can create index pattern for search.

Searching via Kibana Discover

Yes, We are done! if you follow above mentioned steps you should be able to do similar setup in your application. If you have any doubts or suggestions let me know in comments section.

Creating {legacy} static build via docker and deploying with mina-scp

Pramod Shinde — Thu, 15 Apr 2021 16:03:16 +0000

I am sure! Many of our applications have turned into a legacy codebase, such applications might have some outdated scripts or build process that might need frequent maintenance and updates.

In this post, I will walk you through the steps with which we can move our frontend build creation script into the docker container and deploy through mina and mian-scp gems seamlessly.

Even if your code is not a legacy codebase it is always better to build your application into a docker container because

You need to create build in a production like environments
If you want to do a production system upgrades, it's lot more easier to test build creation in docker containers in advance
Avoid any package missing or mis-match issues at runtime, specially on older node versions < 6.x.x.

Brief background

I had a angular 1.x application running with node 6.x.x, things were all good until we noticed following

First issue was previously on servers node-modules and bower packages were moved on servers manually
We were having two(production/staging) servers with different node 5.x.x and 6.x.x setup receptively.
In older node <= 5.x.x, npm install does not support package.json
Package addition and updates were a nightmare
We were frequently facing package related issues because of lack of proper package.json

We resolved all above issues one by one after deciding to fix the node version and upgraded to version 6.x.x and moved the build creation process into a docker container to have a standard build creation and deployment process.

Creating a build in docker container via docker compose

If you are new to docker and docker-compose, I would advise you to get your hands dirty in docker concepts by referring it here and here

Dockerfile

Dockerfile: In this file we are only specifying required ubuntu, node, npm and yarn versions that are required for application and before that we are installing few dependencies.

FROM ubuntu:18.04
SHELL ["/bin/bash", "-l", "-c"]

RUN mkdir /usr/local/nvm
ENV NVM_DIR /usr/local/nvm
ENV NODE_VERSION 6.13.0
ENV NVM_INSTALL_PATH $NVM_DIR/versions/node/v$NODE_VERSION

# install ubuntu related dependencies
RUN apt-get update -q && \
    apt-get install -qy curl ca-certificates gnupg2 build-essential --no-install-recommends && apt-get clean
RUN apt-get install -y git zip

# install nvm
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash

# install node version 6.13.0
RUN source $NVM_DIR/nvm.sh \
    && nvm install $NODE_VERSION \
    && nvm alias default $NODE_VERSION \
    && nvm use default

WORKDIR /usr/bullet

# copy dependencies json file to docker container
COPY package.json $WORKDIR
COPY yarn.lock $WORKDIR
COPY bower.json $WORKDIR

# install package manager
RUN source $NVM_DIR/nvm.sh && npm install -g yarn@1.17.0
RUN source $NVM_DIR/nvm.sh && npm install -g bower@1.8.0

RUN source $NVM_DIR/nvm.sh && npm install -g gulp@3.9.1

RUN source $NVM_DIR/nvm.sh && yarn install --quiet
RUN source $NVM_DIR/nvm.sh && bower install --quiet --allow-root

COPY . $WORKDIR

Once the dockerfile is ready we can use this in docker-compose with following docker-compose.yml file to create multiple services

docker-compose.yml

version: '3.4'

services:  
  frontend-app: &frontend-app
    build:
      context: .
      dockerfile: ./Dockerfile
    image: my-frontend-app:0.0.1
    volumes:
      - ./:/usr/frontend-app/
      - ./static_build:/usr/frontend-app/static_build
      - nodemodules:/usr/frontend-app/node_modules  
      - bowercomponents:/usr/frontend-app/bower_components
  start:
    <<: *frontend-app   
    ports:
      - '8000:8000'
    command: bash -c "source /usr/local/nvm/nvm.sh && gulp dev"

  create_build:
    <<: *frontend-app 
    command: bash -c "source /usr/local/nvm/nvm.sh && gulp prod && rm -f static_build/build.zip && zip -rq static_build/build.zip build/"

volumes: 
  nodemodules: {}
  bowercomponents: {}

In docker-compose file we are creating start and create_build services, start is to start the application inside the docker container with following command

#Running build on port 8080
source /usr/local/nvm/nvm.sh && gulp dev

create_build is to create and compress a build with following command

#Creating prod build, removing old build and creating new build
source /usr/local/nvm/nvm.sh && gulp prod && rm -f static_build/build.zip && zip -rq static_build/build.zip build/

Important to note that we have created shared volumes and folders between host and container, So that we can have build.zip to be created on host and node modules can be shared across the builds if no changes to package.json

Once above setup is done we can easily create or start the build using following commands

#Starting a build
docker-compose up start
#Creating a build
docker-compose up create_build

After above setup is verified we can deploy this build via mina and mina-scp gem as following

Note: Hope you have done basic setup which is needed for mina deployment.(Let me know if you need any help!)

Deploying static build to servers

Following is the sample mina deployment script which is copying and zipping build to the server.

config/deploy.rb

#Mina, version v0.3.8
require 'mina/scp'

# Basic settings like branch, server repo settings goes here
# set :branch, staging
# gem install mina -v 0.3.8 (you can use latest mina version)
# gem install mina-scp -v 0.1.2

# Put any custom mkdir's in here for when `mina setup` is ran.
# all releases.
task :setup => :environment do
end

desc "create a build on the local"
task :create_build => :environment do
  to :before_hook do
    # Put things to run locally before ssh
    queue! %[echo "-----> creating static build..."]
    queue! %[sudo docker-compose up create_build]
  end
end

desc "Deploys the current version to the server."
task :deploy => :environment do

  deploy do
    # Put things that will set up an empty directory into a fully set-up
    # instance of your project.
    invoke :'deploy:link_shared_paths'
    queue! %[echo "-----> uploading build domain: #{domain} branch: #{branch}, deploy_to: #{deploy_to}"]
    scp_upload('./static_build/build.zip', "#{deploy_to}/build.zip", verbose: true) 
    invoke :'deploy:cleanup'

    to :launch do
      queue "echo '-----> Unziping build...'"
      queue "cp #{deploy_to}/build.zip #{deploy_to}/current/build.zip" 
      queue "unzip -q #{deploy_to}/current/build.zip"
      queue "echo '-----> Unzip completed'"
    end
  end
end

Here we are creating and deploying build via

mina create_build
mina deploy

If you have noticed we are copying build ./static_build/build.zip created in docker via mina-scp's scp_upload command and unzipping in deploy task.

scp_upload('./static_build/build.zip', "#{deploy_to}/build.zip", verbose: true)

That's it, We have successfully created and deployed static build with the help of docker and mina. If you have reached here and have any suggestions or thoughts let me know in comments section.

10 Signs of a good Ruby on Rails Developer

Pramod Shinde — Thu, 15 Apr 2021 08:02:38 +0000

I have been working as Ruby on Rails developer since last five years with Josh Software, I felt that I should write down my learnings about the best practices followed by the RoR developer. How I learned …? of course to learn something you need to commit the mistakes, thats how we learn right?

Let’s see, What all you should follow to be a ‘Good’ Ruby on Rails developer.

1. Your migrations are “thoughtful” …

Whenever you come across database table schema design do you think through all the aspects like

The table which is being designed where its going to be used? How much it might grow in terms of the data size? (Imagine the worst future of your design)
Have I kept correct data types, defaults, constraints if any? Most of the times we really don’t need integer columns, We can use smallint instead for smaller set of integers, similarly varchar(10) vs varchar(255) vs text.
Have I added indexes wherever necessary? Thinking through What kind of the queries this table is going to handle?

A special point…Do you write multiple migrations for same table? if yes, it's a bad habit.

Often we don’t think through all the points mentioned above and end up creating multiple migrations for same table which causes codebase to look scary.

Instead you should use up and down on the migration to fix or alter the table, change in the requirement is an exception to this.

2. You always follow the single responsibility principle

We all know a convention of “skinny controller and fat model”, some of us already follow this but do we follow it wisely.

We are living in the Rails 5 era, so Why overload models?

Why not to follow “keep everything skinny move extra fat to concerns or service objects from the models”, the classes in the codebase should be designed to handle single responsibility.

I came across the following posts about how to organise controllers and using service objects in Rails.

3. You write test cases to test the “code”

I have seen many applications whose CI builds takes ages to finish, What exactly they are testing?

Your test cases should be testing the “code” not the machine performance, better test suits

Share objects between different examples.
Uses method stubs and avoid the repetitive calls to the methods.
Don’t test same code twice, if you have sharable piece of code and used at multiple places then don’t write test cases in multiple places.
Does not creates unnecessary test records, unknowingly many developers end up creating unnecessary test records.

If your are using gems like faker , factory_bot_rails and database_cleaner to create and clean test records then creating unnecessary records can cost you time and speed.

Simple example,

  create_list(:user, 10)

Much better will be reduce the list size, if you are not doing anything special with 10 users.

  create_list(:user, 2)

4. You keep production environment healthy

If you are an engineer and reduce efforts of others, then you use the utilities of other engineers to reduce your efforts.

A healthy Rails production environment always have

Monit – Is everything up and running? if not get notified.
logrotate – rotates, compresses, and mails system logs.
crontabs with whenever, schedules work for you.
Database backup scripts running in maintenance window.
Exception notifiers like Sentry or Rollbar or ‘anything that suits you’.

5. You follow basic git etiquettes

If you are working in a team and using git then you follow the git etiquettes like

Don’t commit untracked files – we often keep git untracked files like something.swp, backup.sql, ‘schema.rborstructure.sql backups,some.test.script`, you should not commit such files.
Branch naming – naming something is always difficult but you have to do it, the feature branches should have sensible names, don’t use names like something-wip, somthing-test.
Delete the feature branches after merge – no explanation required.
Commit messages – your commit messages must have Github issue number or any project management story number/link, brief description about feature/task

6. You don’t ignore README.md

Remember you are not the only one who is going to work on particular application for your lifetime. Someone will takeover you and he should not waste his time in figuring out that how to setup things.

Your application repository must have updated README.md with detail steps about setting up an application for the first time.

7. Secrets are “really” secrets for you

We often use credentials for database configs, secrets.yml, third party api’s like AWS, payment gateway, sentry etc.

You should not commit such credentials/secrets/environment variables to the Github instead you keep them secure with gems like dotenv-rails, figaro or simple dot files that are not committed to the repository.

A sample file of such credentials should be committed and updated regularly.

8. You do code reviews and discuss feature with team

While working in a team you should get your feature reviewed from another team mate or before starting on any feature discuss it with the team thoroughly, advantages of the code reviews or feature discussion are you will come across many scenarios that are not thought of.

If you are the only one who is working on a application then you must criticise your own code and cover all the scenarios in test cases.

9. You are up-to-date and keep updating

In open source community we get frequent updates or releases for ruby, rails and gems, you must keep yourself aware and informed by subscribing to the repositories or mailing lists and update your application libraries.

Also you should stay alert on security fixes about the production operating system, database so you can take necessary action on time.

10. Need not to say…

You write clean and maintainable code and your codebase is

Properly indented
Only 80 columns wide
Maintainable with smaller methods, less complexity – To know more on this make a habit of using code analyzer like rubocop, Code Climate
Codebase follows Ruby best practices and style guide.

Well there are many more points that can be included in this list but I feel these are the most important to fill in first into this list, If you find that I have missed anything more important then you can comment on this post.

Thanks for the reading upto here, hope this will help you to become a ‘Good’ developer.

PS: I am moving out of wordpress this is a cross post.