DEV Community: Prathamesh Sonpatki

Starting o11y.wiki

Prathamesh Sonpatki — Tue, 07 Mar 2023 15:22:20 +0000

I have started a project for maintaining glossary of all terms and definitions related to Observability.

It is called o11y.wiki.

The GitHub repo can be found here https://github.com/prathamesh-sonpatki/o11y-wiki

The contents are available at https://o11y.wiki/.

This is an open source project aimed at beginners like me who don’t know a lot of observability and reliability. The idea is to learn about these terms and note them down at one place, so others can benefit from it.

Here are few next set of terms that can be added --

MTTR
MTTD
MTBI
Incident
Alert
Metric
Samples
Cardinality
Log
Span
Event
Exceptions
Serverless
SRE
Platform engineering
PromQL
Service Discovery

I am looking for help while making this project better and solid for everyone! 🏗️ ❤️

Prometheus Alternatives

Prathamesh Sonpatki — Tue, 07 Feb 2023 11:29:16 +0000

Prometheus is a popular open-source platform for metrics and alerting created by SoundCloud in 2012 and officially released as open-source in 2015. Designed for both dynamic service-oriented architectures and system monitoring, Prometheus focuses on reliability, multidimensional data collection, and data visualization.

While Prometheus is an excellent option for tracking metrics, other open-source and SAAS alternatives in the ecosystem might better suit your needs.

This article compares Prometheus with InfluxDB, Zabbix, Datadog, and Graphite, Grafana based on their data model and storage, architecture, APIs and access methods, partitioning, compatible operating systems, pricing, visualization, alerting, and supported programming languages, use cases and supported workloads.

Prometheus Alternatives

The following is an overview of each tool compared in this article.

What is Prometheus?

As mentioned above, Prometheus is a monitoring and alerting system that helps developers manage applications, tools, databases, and even network monitoring. It has a comprehensive set of built-in features for collecting metric data and acts as a full-stack observability and monitoring system for microservices and cloud-native applications. It has merged with Cloud Native Computing Foundation(CNCF) since 2016 as the second most popular project after Kubernetes . While Prometheus is an excellent tool for DevOps and SRE teams, it can run into scalability issues where tools such as Thanos, Cortex, and Levitate can help.

InfluxDB

InfluxDB is a leading time series database that comes in three editions: an open-source version called InfluxDB and two commercial versions called InfluxDB Cloud and InfluxDB Enterprise. It provides a complete set of data tools for ingesting, processing, and manipulating multiple data points. It includes the InfluxDB user interface (InfluxDB UI) and Flux, a functional scripting and query language.

Zabbix

Zabbix is a scalable, accessible, open-source monitoring solution used for both small environments and enterprise-level distributed systems with millions of metrics.

Datadog

Datadog is a monitoring and analytics platform used for event monitoring and measuring the performance of cloud applications and infrastructure. It combines real-time metrics from disparate sources such as applications, servers, databases, and containers with end-to-end tracing to deliver alerts and visualizations. It can collect data from various data sources with its built-in integrations.

Graphite

Created by Chris Davis at Orbitz in 2006 and released as open source in 2008, Graphite is a monitoring solution that collects time series data from applications, servers, infrastructure, and networks. It focuses on storing passive time series data and analyzing it through the Graphite web UI.

Grafana

Grafana is a data visualization tool developed by Grafana Labs. It is available as open source, managed (Grafana Cloud), or enterprise edition. Grafana can combine data from many data sources into a single dashboard. It solves the problem of visualization of time series data.

Is Grafana the same as Prometheus?

We keep seeing this common question; while Prometheus is a time series database, Grafana is a data visualization tool. It supports Prometheus, Graphite, and InfluxDB as data sources. So they are not the same, but they work better together. Grafana is a standard for the visualization of Prometheus data.

Prometheus Alternatives in action

This section compares Prometheus to InfluxDB, Zabbix, Datadog, and Graphite using the following criteria:

Data model and storage
Architecture
APIs and access methods
Partitioning
Compatible operating systems
Supported programming languages
Open Source vs. Proprietary

Data Model and Storage

Prometheus captures and accumulates metric data as time series data and stores it in a local database. A metric name and optional key-value pairs are unique identifiers or labels for each time series.

Data can be queried in real-time using the Prometheus Query Language (PromQL) and presented in tabular or graphical form.

Prometheus supports the float64 data type with limited support for strings and millisecond resolution timestamps. Prometheus also supports long-term storage to different layers via Prometheus remote write protocol and can be run in an agent mode.

InfluxDB: Data Model and Storage

InfluxDB maintains a time series database optimized for time-stamped data, much like Prometheus. Data elements also comprise a unique combination of timestamps, tags, fields, and measurements. Tags are indexed key-value pairs used as labels, while fields are sequenced key-value pairs, which function as secondary labels with limited use.

InfluxDB uses a proprietary query language similar to SQL called InfluxQL and supports timestamp, float64, int64, string, and bool data types.

Zabbix: Data Model and Storage

Zabbix uses an external database to store the collected data and configuration information. It integrates with leading relational database management system (RDBMS) database engines such as MySQL, MariaDB, Oracle, PostgreSQL, IBM Db2, and SQLite, which allows Zabbix to store more complex data types such as system logs. Zabbix stores raw data collected from hosts in history tables, while trends tables store consolidated hourly data.

Datadog: Data Model and Storage

Datadog uses Kafka to process incoming data points and a mix of Redis, Cassandra, and S3 to store and query time series. It also uses Elasticsearch to store and query events (such as alerts and deployments) that are not represented as a time series and uses PostgreSQL for metadata.

Graphite: Data Model and Storage

Like Prometheus, Graphite stores time series data using its specialized database, but data collection is passive. Data is collected from collection daemons or other monitoring tools (including Prometheus) and sent to Graphite's Carbon component.

Summary

InfluxDB and Graphite both use time series databases similar to Prometheus. Graphite, however, doesn't store raw data as Prometheus does. InfluxDB offers full support for strings and timestamps as well as int64 and bool data types, while Prometheus only provides full support for float64. Zabbix integrates with more familiar RDBMS database engines and is suitable for storing historical data. At the same time, Datadog uses several data models and storage types to store both time-series and non-time-series data.

Architecture

Prometheus servers are standalone and run independently of each other. They rely on local on-disk storage rather than network or remote storage services for the core functionality of scraping, rule processing, and alerting. Data is stored for fourteen days, but Prometheus can be integrated with remote solutions such as Levitate for long-term storage.

InfluxDB: Architecture

Like Prometheus, open-source InfluxDB servers are standalone and use local storage for scraping, alerting, and rule processing. Commercial InfluxDB versions come with distributed storage by default that allows queries and storage to be managed by many nodes simultaneously, making it easier to perform horizontal scaling.

Zabbix: Architecture

Zabbix architecture comprises servers that store statistical, operational, and configuration data and agents installed on the machines that collect the data. Agents monitor and report data collected from local resources and applications to Zabbix servers.

Agents and servers support passive checks, where the server requests a value from the agent, and active checks, where the agent periodically sends results to the server.

Datadog: Architecture

Datadog uses Kafka for independent storage systems. It acts as a persistent storage and query layer. Kafka is an open-source, distributed, partitioned, replicated log service developed by LinkedIn as a unified platform for handling large-scale, real-time data feeds.

Graphite: Architecture

Graphite architecture is made up of three components:

Carbon, the primary backend daemon that listens for time series data sent to Graphite and stores it in Whisper, the backend database
Whisper, a fast, file-based local time series database that creates one file per stored metric
The Graphite web UI, the frontend UI for the backend storage system that renders graphs on demand

Summary

While InfluxDB and Prometheus both use standalone servers, commercial versions of InfluxDB offer distributed storage to support horizontal scaling. The Zabbix architectural model uses servers with agents, which allows for both passive and active data checks. Datadog's use of Kafka for its persistent data storage layer will enable it to store large amounts of real-time data. Graphite's architecture includes a web app, which is a good choice if you want to render graphics on demand.

APIs and Access Methods

Prometheus uses RESTful HTTP endpoints with responses in JSON.

InfluxDB: APIs and Access Methods

The InfluxDB API provides a set of HTTP endpoints for accessing and managing system information, security and access control, resource access, data I/O, and other resources and returns JSON-formatted responses. The Enterprise version also provides support for TCP and UDP ports.

Zabbix: APIs and Access Methods

Zabbix uses the JSON-RPC 2.0 protocol. Requests and responses between clients and the API are encoded using JSON.

Datadog: APIs and Access Methods

Datadog uses the HTTP REST API. Resource-oriented URLs are used to call the API, with JSON being returned from all requests.

Graphite: APIs and Access Methods

Graphite data is queried over HTTP via its Metrics API or the Render URL API. The Graphite API is an alternative to the Graphite web UI that retrieves metrics from a time series database and renders graphs or generates JSON data based on these time series.

Summary

All tools provide support for HTTP requests and JSON-formatted responses.

Partitioning

Prometheus supports sharding. You can scale horizontally by splitting target metrics into shards on multiple Prometheus servers to create more minor instances.

InfluxDB: Partitioning

InfluxDB organizes data into shards to create a highly scalable approach that increases throughput and maintains performance as the data grows. Shards are placed into shard groups containing encoded and compressed time series data for a specific time range. The shard group duration defines the period for each shard group, and each group has a corresponding retention policy that applies to all the shards within the group.

Zabbix: Partitioning

Partitioning with Zabbix depends on the database being used. MySQL, PostgreSQL, IBM Db2, and MariaDB (with the Spider storage engine) offer sharding capabilities.

Datadog: Partitioning

Datadog uses Kafka partitions to scale by customer, metric, and tag set. You can isolate by the customer or scale concurrently by metric. Sharding is implemented as a group of Kafka partitions.

Graphite: Partitioning

Graphite does not support partitioning.

Summary

All tools except for Graphite offer some form of support for portioning. Prometheus, InfluxDB, and Datadog provide sharding and horizontal scaling features, while Zabbix support depends on your chosen external database.

Compatible Operating Systems

Prometheus supports the Linux and Windows operating systems.

InfluxDB: Compatible Operating Systems

InfluxDB supports Linux, Windows, and macOS.

Zabbix: Compatible Operating Systems

Zabbix supports Linux, Windows, macOS, IBM AIX, Solaris, and HP-UX operating systems.

Datadog: Compatible Operating Systems

Datadog supports Windows, Linux, and macOS operating systems and cloud service providers, including Google Cloud, AWS, Red Hat OpenShift, and Microsoft Azure.

Graphite: Compatible Operating Systems

Graphite supports Linux and Unix operating systems.

Summary

All tools except Graphite supports Windows and Linux operating systems; Graphite only supports Linux and Unix. InfluxDB, Zabbix, and Datadog also support macOS, with Datadog providing additional support for cloud service providers.

Supported Programming Languages

Prometheus provides several official and unofficial client libraries for .NET, C++, Go, Haskell, Java, JavaScript (Node.js), Python, and Ruby. It also supports Prometheus Exporters to collect data from systems that do not directly have client libraries.

InfluxDB: Supported Programming Languages

InfluxDB supports client libraries for C++, Java, JavaScript, .NET, Perl, PHP, and Python. It can be directly used with the REST API.

Zabbix: Supported Programming Languages

Zabbix supports Java, JavaScript, .NET, Perl, PHP, Python, R, Ruby, Elixir, Go, and Rust.

Datadog: Supported Programming Languages

Client libraries are available in C#/.NET, Java, Python, PHP, Go, Node.js, Ruby, and Swift, along with many integrations.

Graphite: Supported Programming Languages

Graphite has client libraries in Python and JavaScript (Node.js) programming languages.

Summary

Prometheus, InfluxDB, Zabbix, and Datadog all support the major programming languages. Graphite, however, only provides support for Python and JavaScript.

Comparison summary

	Prometheus	InfluxDB	Zabbix	Datadog	Graphite	Levitate
Data Model and Storage	Multi-dimensional data model with Time series data	Time series data	External database stores including RDBMS	Both time series and non time series data	Time series data	PromQL compatible time series data
API and Access methods	HTTP API	HTTP API	HTTP API	HTTP API	HTTP API	HTTP API
Partitioning	Supported	Supported	Supported, depends on RDBMS of choice	Supported	Supported	Managed TSDB
Open Source	Yes	Yes. Proprietary also available.	Yes	No. Proprietary	Yes	No. Proprietary
Programming languages	Tons of client libraries and exporters	C++, Java, JavaScript, .NET, Perl, PHP, and Python.	Java, JavaScript, .NET, Perl, PHP, Python, R, Ruby, Elixir, Go, and Rust	Tons of integrations	Python and JavaScript (Node.js)	It can be directly used with the REST API

Prometheus's strengths lie in its support for multidimensional data collection. It has a powerful query language that can be used for both dynamic service-oriented architectures and machine-centric monitoring. It's a good choice when you primarily want to record numeric time series.

InfluxDB and Prometheus use similar data compression techniques and support multidimensional data using key-value data stores; InfluxDB is better for event logging. A commercial version provides the best option if you need to process large amounts of data, as its default configuration scales horizontally.

Zabbix focuses on hardware and device management and monitoring. It's a better option than Prometheus if you are more familiar with RDBMS database engines and need to store many historical and varied data types. However, the use of an external database can slow down performance.

Prometheus's internal time series database provides faster connectivity to data but is not suitable for storing data types like text or event logs. Since Prometheus only keeps data for fourteen days, it's also not a good option if you need to store historical data (unless configured for remote storage).

Datadog and Prometheus can be used for application performance monitoring(APM). However, Datadog has more application monitoring capabilities than Prometheus and is geared toward monitoring infrastructure at scale. Datadog is best for monitoring infrastructure and apps and visualizing data from disparate sources in mid to large-scale environments.

Graphite runs well on all hardware and cloud infrastructure, making it suitable for small businesses with limited resources and large-scale production environments. Choose Graphite when you need a solution focused on storing and analyzing historical data and fast retrieval.

Conclusion

Prometheus is a popular option for tracking metrics and alerting, but one of the four alternatives mentioned above might suit your needs depending on your requirements.

For processing large amounts of data, choose a commercial version of InfluxDB, but if you want the familiarity of an RDBMS engine, then go with Zabbix. Datadog's wide range of monitoring features makes it the go-to choice for monitoring infrastructure in larger environments. Still, if you operate on a smaller scale, Graphite can get the job done with whatever hardware and resources you have.

Last9, a site reliability engineering (SRE) platform. We remove the guesswork in improving the reliability of your distributed systems. Last9's Levitate, a managed time series database(TSDB), helps you understand, track, and improve your organization's system dependencies to reduce the challenges of time series database management.

Access the intelligence you need to deliver reliable software with Last9's reliability platform.

This post was originally published on Last9 Blog.

A practical guide for implementing SLO

Prathamesh Sonpatki — Thu, 12 Jan 2023 05:30:00 +0000

This is a mini guide to the SLO process that SREs and DevOps teams can use as a rule of thumb. This guide not necessarily automates the SLO process but gives a direction in which one can go using SLOs effectively.

The process essentially involves 3 steps

Identify the level of the Service
Identify the right type of the SLO
Set the SLO Targets

Before diving deep into it, let’s understand a few terminologies in the Site Reliability Engineering and Observability world.

SLO Terminologies

Service Level Indicator(SLI)

A Service level indicator ( SLI ) is a measure of the service level provided by a service provider to a customer. It is a quantitative measure that captures key metrics, like the percentage of successful requests or completed requests within 200 milliseconds, for example.

Service Level Objective(SLO)

A Service Level objective is a codified way to define a goal for service behaviour using a Service Level indicator within a compliance target.

Service Level Agreement(SLA)

A service level agreement defines the level of service expected by users in terms of customer experience. They also include penalties in case of agreement violation.

Let’s go through the SLO process now.

Identify the level of Service

Customer-Facing Services

A Service running HTTP API / apps/ GRPC workloads where the caller expects an immediate response to the request they submit.

Stateful Services

Services like a database. It is common to confuse a database as not being a service in a microservices environment where multiple services call the same database. Try answering this straightforward question next time you are unable to decide.

My service HAS a database OR my Service CALLS a database.

Asynchronous Services

Any service that does not respond with the request result instead queues it to be processed later. The only response is to acknowledge whether the service successfully accepted the task or not; the service will process the actual result/available later.

Operational Services

Operational Services are usually internal to an organization and deal with jobs like Reconciliation, Infrastructure bring-up, tear-down, etc. These jobs are typically asynchronous. But with a greater focus on accuracy vs. throughput. The Job may run late, but it must be correct as much as possible

Identify the right type of the SLO

Request Based SLO

Request-based SLOs perform some aggregation of Good requests vs. The total number of requests.

First, there is a notion of a Request. A request is a single operation on a component that succeeds or fails in generic terms.
Secondly, the SLIs have to be not pre-aggregated because Request SLOs perform an aggregation over a period of time. One can’t use pre-aggregated metrics(eg. Cloudwatch / Stackdriver which directly returns P99 latency rather than total requests and latency per request) for Request SLOs.
Additionally, for low-traffic services, Request SLOs can be noisy because they can keep flapping even when a very small % of requests fail. Eg. if your throughput is 10 rpm in a day, setting a 99% compliance target does not make sense because 1 request will bring down the compliance to 90% depleting the error budget.

Window Based SLO

Window-based SLO is a ratio of Good time intervals vs. total time intervals. For some sources, the requests are not available.

For example, In the case of a Kubernetes Cluster, the availability of a Cluster is the percentage of pods allocated vs. pods requested. Sometimes, you may not want to calculate the SLO as the overall performance of the service over a period of time.

Eg. in the case of a payment service, even if only 2% of requests fail in a window of 5 minutes, it is unacceptable because it is a critical service for my business. Even though overall performance has not degraded but that 2% of requests none of the payments was successful. Window-based SLOs are useful in such cases.

Using the above guidelines, we can create a rough flowchart to decide which type of SLO to choose depending on certain decision points.

SLO Process

Set the SLO Targets

When you start thinking about setting objectives, some questions will arise:

Should I set 99.999% from the start or be conservative?

Start conservatively. Look at historical numbers and calculate your 9s or dive right in with the lowest 9 such as 90%.
The baseline of the service or historical data of the customer experience can be helpful in this case.
Keep your systems running against this objective for a period of time and see if there is no depletion of the error budget.
If there are, improve your system’s stability. If there aren’t, move up to the next ladder of service reliability. From 90% go to 95 % then to 99% and so on.
Keep in mind Service Level agreements or SLAs that you may have with customers or third-party upstream services that you are dependent on. You can’t have a higher compliance target than a third-party service giving you a lower SLA.

What should be the compliance window?

Generally, this is 2x of your sprint window so that you can measure the performance of the service in a large enough duration to make an informed decision in the next sprint cycle on whether to focus on new features or maintenance.
If you are not sure start with a day and expand to a week. Remember that the longer your window, the longer the effects of a broken / recovered SLO.

How many ms should I set for latency?

It depends. What kind of user experience are you aiming for? Is your application a payment gateway? Is it a batch processing system where real-time feedback isn’t important?
To start out, measure your P50, and P99 latencies and initially give yourself some headroom and set your SLOs against P99 latency. Depending on the stability of your systems, use the same ladder-based approach as shown above and iterate.

Service Level Objectives are not a silver bullet

Let us take a simple scenario:

A user makes a request to a web application hosted on Kubernetes served via a load balancer. The request flow is as follows:

Request Flow

Instead of setting a blind SLO on the load balancer and calling it a day, ask yourself the following questions:

Where should I set the SLO - ALB or Ambassador or K8s or all of them? Typically SLOs are best set closest to the user or something that represents the end user’s experience e.g. if in the above example, one might want to set an SLO on the ALB but if the same ALB is serving multiple backends it might be a good idea to set the SLO on the next hop - Ambassador.
If I set a latency SLO, what should be the right latency value? Look at baseline percentile numbers. Do you want to catch degradations of the P50 customer experience, the P95 customer experience, or a static number?
Do I have enough metrics I need to construct an SLI expression? AWS Cloudwatch reports latency numbers as pre-calculated P99 values i.e. if you want to set a request-based SLO with the expression, you can’t do that because the data is pre-aggregated. So you cannot set request-based SLOs, you can only use window-based SLOs.
Suppose you set an availability SLO on Ambassador with the expression availability = 1 - (5xx / throughput).
What happens if the Ambassador pod crashes on K8s and does not emit 5xx / throughput signal?
Does the expression become availability = 1 - 0 / 0 or availability = undefined?
For a payment processing application, there might be a lag between the time at which the transaction was initiated v/s the time at which it was completed.
How does availability = 1 - (5xx / throughput) work now?
How do I know 5xx that I got was for a request present in the current throughput or was it a previous retry that failed?

This is not an exhaustive list of questions. Real-world scenarios will be complicated and that makes the task of setting achievable reliability targets involving multiple stakeholders and critical user journeys tricky.

So does this mean all hope is SLOst?

Of course not! SLOs are a way to gauge your system’s health and customer experience over a time period. But they are not the only way. In the above scenario, one could:

Set a request-based SLO on the Ambassador.
Set an uptime window SLO or an alert that checks for no-data situations for signals that are always ≥ 0 e.g. Ambassador throughput.
Set relevant alerts to catch pod crashes of the application.
Set alerts on load balancer 5xx to catch scenarios where ALB had an issue and the request was not forwarded to the Ambassador backend.

Want to know more about Last9 and how we make using SLOs dead simple? Check out last9.io; we're building SRE tools to make running systems at scale, fun, and _ embarrassingly easy. _ 🟢

Changing default git branch to main locally

Prathamesh Sonpatki — Mon, 24 May 2021 11:55:19 +0000

GitHub has shifted to main branch for new projects but when creating a new project locally with git init still creates master branch. In this post, we will see how to create main branch locally by default.

The git init command accepts value for name of the branch in the -b flag.

$ git init -b main
Initialized empty Git repository in /private/tmp/food/.git/
$ touch tmp.txt
$ git add .
$ git commit -m "main"
[main (root-commit) 0b3550b] main
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 tmp.txt

This is possible only with Git 2.24 onwards.

We can also configure the default branch globally.

git config --global init.defaultBranch main

For older Git versions, we can switch to the main branch before committing anything because the branch doesn't exist for real until something is committed.

cd sport
git init
git branch -m main

My git version was 2.31.1 while writing this post.

git --version
git version 2.31.1

Installing GNU grep and find on OS X

Prathamesh Sonpatki — Mon, 24 May 2021 10:07:47 +0000

brew install grep
brew install findutils

After that update your .bashrc or .zshrc with following to setup the paths correctly.

# .bashrc/.zshrc
export PATH="/usr/local/opt/findutils/libexec/gnubin:$PATH"
export PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"

That's all!

Adding jemalloc to Rails apps on Heroku

Prathamesh Sonpatki — Sun, 23 May 2021 01:46:23 +0000

jemalloc is a malloc implementation developed by Jason Evans which is known to improve memory consumption of Rails apps, especially on Heroku. By default Ruby uses malloc from C to manage memory but it can run into memory fragmentation issues.

Whereas jemalloc describes itself as a malloc implementation which tries to avoid memory fragmentation. A lot of people have successfully tested jemalloc in production apps deployed on Heroku to verify that it reduces memory usage compared to the default memory management that comes with Ruby.

Enabling it in a Rails app on Heroku consists of following steps.

Add jemalloc buildpack

heroku buildpacks:add --index 1 https://github.com/gaffneyc/heroku-buildpack-jemalloc.git

Enable jemalloc

This can be done in two ways. Either we can set the environment variable, JEMALLOC_ENABLED to true.

heroku config:set JEMALLOC_ENABLED=true

Alternatively, we can add jemalloc.sh prefix to the processes listed in the Procfile.

# Procfile

web: jemalloc.sh bin/puma -C config/puma.rb
worker: jemalloc.sh bundle exec sidekiq -q default -q mailers -c ${SIDEKIQ_CONCURRENCY:-5}

Note that setting the JEMALLOC_ENABLED environment variable will enable jemalloc for all processes of your app, where as adding jemalloc.sh prefix in Procfile can give you control over for which processes you want to enable it.

After this is done, deploying the app on Heroku will enable jemalloc and we can monitor the memory consumption.

jemalloc version

We can choose the jemalloc version by setting JEMALLOC_VERSION to a version number from this list. By default, the buildpack chooses the most recent version.

Conclusion

With just enabling jemalloc, we can see significant drop in memory usage on Heroku without doing any code change. So I highly recommend enabling it in production apps deployed on Heroku.

Bundler 2.2.3+ and deployment of Ruby apps

Prathamesh Sonpatki — Sun, 18 Apr 2021 14:06:49 +0000

While deploying a new Rails 6.1.3.1 application built with Bundler 2.2.16 on Heroku, I ran into this error in Heroku build log.

-----> Ruby app detected
-----> Installing bundler 2.2.15
-----> Removing BUNDLED WITH version in the Gemfile.lock
-----> Compiling Ruby/Rails
-----> Using Ruby version: ruby-3.0.1
-----> Installing dependencies using bundler 2.2.15
       Running: BUNDLE_WITHOUT='development:test' BUNDLE_PATH=vendor/bundle BUNDLE_BIN=vendor/bundle/bin BUNDLE_DEPLOYMENT=1 bundle install -j4
       Your bundle only supports platforms ["x86_64-darwin-20"] but your local platform
       is x86_64-linux. Add the current platform to the lockfile with `bundle lock
       --add-platform x86_64-linux` and try again.
       Bundler Output: Your bundle only supports platforms ["x86_64-darwin-20"] but your local platform
       is x86_64-linux. Add the current platform to the lockfile with `bundle lock
       --add-platform x86_64-linux` and try again.
 !
 ! Failed to install gems via Bundler.
 !
 ! Push rejected, failed to compile Ruby app.
 ! Push failed

If you are using Bundler 1.x you will not run into this error. It only happens with Bundler 2.2.3 and above. My bundler version is 2.2.16.

The app was built on Mac OS X. The Gemfile.lock contained following line.

PLATFORMS
  x86_64-darwin-20

As per the warning it seems that my Gemfile was generated for Mac OS X but I am deploying to Linux and it was raising a flag. Kudos to Heroku buildpack to raise a user friendly error message!

A lot of gems such as nokogiri ship platform specific releases. In previous versions of Bundler, the approach for detecting the platform specific version was error-prone as per this comment. To mitigate such errors, bundler now sets up Gemfile.lock for the platform on which it was generated.

When deploying to Heroku, the Heroku Ruby buildpack runs bundle install in deployment mode. It expects that Gemfile.lock to be frozen and already compatible with the platform on which it is being run. In our case, the Gemfile.lock is not compatible with the platform on which Heroku is deploying which is Linux and the platform on which it was generated which is Mac.

If we do not use the deployment mode, then bundler will resolve the gems in real time with current platform and this problem will not happen.

Though you should not do this as dev-prod parity breaks down and in production, bundler may resolve to the gems which you have not tested in development or test mode.

To fix this, as the warning recommended, I added linux platform using following command.

bundle lock --add-platform x86_64-linux

This generated following diff in Gemfile.lock.

diff --git a/Gemfile.lock b/Gemfile.lock
index 9f4c3ad..ab20d37 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -102,6 +102,8 @@ GEM
     nio4r (2.5.7)
     nokogiri (1.11.3-x86_64-darwin)
       racc (~> 1.4)
+ nokogiri (1.11.3-x86_64-linux)
+ racc (~> 1.4)
     pg (1.2.3)
     public_suffix (4.0.6)
     puma (5.2.2)
@@ -198,6 +200,7 @@ GEM

 PLATFORMS
   x86_64-darwin-20
+ x86_64-linux

After pushing this change to Heroku, the deployment went through.

Non-Heroku deployments

This is not specific to deployments on Heroku only. It can be seen in any other deployment platform as well where we are deploying to Linux and developing on Mac. The fix is same, add the platform to Gemfile.lock and redeploy.

Puma installation issue due to missing ctype.h on Mac OS X

Prathamesh Sonpatki — Sun, 04 Oct 2020 10:01:39 +0000

puma_http11.c:203:22: note: include the header <ctype.h> or explicitly provide a declaration for 'isspace'

Are you facing this error recently while trying to install puma gem 4.3.5 on Mac OS X?

This issue is reported on Puma issue tracker here and fixed in version 4.3.6 and in the latest 5.0.0 release.

But if you want to fix it without updating the Puma gem version from 4.3.5, you can use run following command to update your Bundler configuration.

bundle config build.puma --with-cflags="-Wno-error=implicit-function-declaration"

bundle install will run successfully after this.

Even better is to just update Puma to latest version which also fixes this issue.

OR query with multiple conditions on same column using Sequel

Prathamesh Sonpatki — Sun, 16 Aug 2020 13:40:50 +0000

I recently started using Sequel to manipulate the PostgreSQL database in our Rails application at Last9.

I had to write an SQL query as follows.

SELECT * FROM users where status = 'inactive' OR status = 'deleted'

Sequel provides or function that can be used to construct OR expressions.

exp = Sequel.or(x: 1, y: 2)
DB[:users].where(exp)

This results into following SQL.

"SELECT * FROM \"users\" WHERE ((\"x\" = 1) OR (\"y\" = 2))"

Now let's try this same technique to write SQL for our use case. We want to select all users whose status is either

inactive or deleted.

> DB[:users].where(Sequel.or(status: 'deleted', status: 'inactive')).sql
(pry):8: warning: key :status is duplicated and overwritten on line 8
=> "SELECT * FROM \"users\" WHERE (\"status\" = 'inactive')"

But as you can see if we use same key which is status in this case, Ruby ignores it and the query that gets generated only has last value which is inactive in our case.

Ruby ignores multiple keys with same name in a hash so beware if you are using multiple values with same keys in a hash.

So how do we generate the OR query on same column using Sequel?

We can pass the arguments to Sequel.or as Array instead of Hash.

> DB[:users].where(Sequel.or([
                              ["status", "inactive"], 
                              ["status", "deleted"]
                             ])
                   )
=> "SELECT * FROM \"users\" WHERE (('status' = 'inactive') 
    OR ('status' = 'deleted'))">

This change makes sure that the OR query on same status column gets generated correctly.

Subscribe to my newsletteror follow me on Twitterto know more about how to use Sequel with Rails apps.

Managing infra code ⚙️🛠🧰

Prathamesh Sonpatki — Wed, 12 Aug 2020 16:52:58 +0000

Do you care about the quality of your infra code?

A. As much as product code
B. Somewhat but mostly no
C. We create infra via UI

Let's discuss how do you manage Infra code! Feel free to share your thoughts in the comments section.

Creating unlogged (PostgreSQL) tables in Rails

Prathamesh Sonpatki — Mon, 10 Aug 2020 16:52:30 +0000

One of the most important aspects of a relational database is durability. The database has to make certain guarantees which add overhead to the database system. But what if you want to give up on the durability aspect and increase the speed instead?

This can be especially be done in test environment where one may not care about the durability aspect and want to run tests faster. PostgreSQL supports multiple settings for non durability which forgo data integrity and can increase the performance. One such thing is unlogged tables.

Data written to unlogged tables is not written to the write-ahead log which makes them considerably faster than ordinary tables. But it comes with a rider. These tables are not crash proof. Whenever there is a crash or unintended shutdown, such tables are truncated.

But they can be used in test environment where we don't really care about the durability aspect. They can also be used for temporary tables which are recreated even if they are wiped out. We can create unlogged tables as follows.

prathamesh@/tmp:prathamesh> create unlogged table users (name varchar, email varchar);
CREATE TABLE
Time: 0.031s

Unlogged tables and Rails

Rails allows creating unlogged tables with PostgreSQL adapter from Rails 6 onwards. We can either create unlogged tables in a migration or we can set a global setting that all tables are created as unlogged.

Creating unlogged table in a migration

Rails provides create_unlogged_table similar to create_table which creates an unlogged table.

class CreateUsers < ActiveRecord::Migration[6.0]
  def change
    create_unlogged_table :users, id: :uuid do |t|
      t.text :name
      t.text :email

      t.timestamps
    end
  end
end

Creating all tables as unlogged tables

We can set ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.create_unlogged_tables to true to create all tables as unlogged. This can be set for test environment as follows.

# config/environments/test.rb

# Create unlogged tables in test environment to speed up build
  config.to_prepare do
    ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.create_unlogged_tables = true
  end

That's it. All tables created in test environment are by default unlogged.

This setting is set to false by default in all environments.

Caution!

Unlogged tables are not crash proof and should not be used in production environment unless durability is not a concern. Don't use them blindly.

Interested in knowing more about Rails and PostgreSQL, subscribe to my newsletter.

Effective enqueuing of background jobs

Prathamesh Sonpatki — Wed, 05 Aug 2020 18:39:50 +0000

TLDR;

Lot of times we can avoid enqueuing the jobs to the background queue if they are going to be discarded immediately upon execution.

Let's say we have a background job to send a webhook notification to Slack.

# app/services/events_service.rb

class EventsService
  def process
    @event = create_event
    SlackJob.perform_later(@org, @event)
  end
end

# app/jobs/slack_job.rb

class SlackJob < ApplicationJob
  def perform(org, payload)
    if org.slack_enabled?
      SendSlackNotification.new(org, payload).process
    end
  end
end

The job is enqueued for sending the notification and then inside the job we are checking if the organization has enabled slack or not. Only if slack is enabled then the job will be executed. But it still gets enqueued in the queue and the worker process has to pick it up and start executing. Only then it realizes that the job does not satisfy required conditions and discards it.

This means that each time the SlackJob gets enqueued and gets picked up for execution by the worker process.

A better approach will be to enqueue only the jobs that are going to be actually executed. This will avoid unnecessarily enqueuing jobs which are going to be discarded immediately.

# app/services/events_service.rb

class EventsService
  def process
    @event = create_event
    if @org.slack_enabled?
      SlackJob.perform_later(@org, @event)
    end
  end
end

# app/jobs/slack_job.rb

class SlackJob < ApplicationJob
  def perform(org, payload)  
    SendSlackNotification.new(org, payload).process
  end
end

This pattern should be used in cases where the decision of whether to execute the job or not too complex and does not depend on too many entities. If the job has to take decision based on complex logic then it can he handled in the job itself instead of doing it before enqueuing the job. But for simple conditionals, we can check them beforehand to avoid enqueuing of the job and immediately discarding.

Interested in knowing more about my thoughts on web programming using Ruby on Rails? Subscribe here or follow me on Twitter.