DEV Community: Dmitry Daw

A way to cache responses in Grape API

Dmitry Daw — Fri, 28 Jun 2024 19:42:34 +0000

Caching is a great way to speed up slow pages and to make your API faster in general.

Let's say we want to cache the response of our API.

Key to cache invalidation

First thing to think when adding a cache is the cache invalidation.

Rails have handy methods for that: ActiveRelation and ActiveRecord methods cache_key and cache_key_with_version.
While cache_key returns only the id of a model, e.g.

Product.where("name like ?", "%Game%").last.cache_key
=> "products/124"

cache_key_with_version additionally returns the timestamp of the last change, e.g.

Product.find(124).cache_key_with_version
=> "products/124-20240624103815954181"

By default it works by the updated_at column, but you can customize it like this:

Product.find(124).cache_key(:last_reviewed_at)

For relation it works a bit differently, returning the key based on the SQL query hash. It takes into consideration params too.

Product.where("name like ?", "%Game%").cache_key
# => "products/query-1850ab3d302391b85b8693e941286659"

And cache_key_with_version additionally returns the id and timestamp of the last entity

Product.where("name like ?", "%Game%").cache_key_with_version
# => "products/query-e0db51fbb1a07ab9545d84d80aac3d16-124-20240628093023387346"

Two caching strategies

Adding caching is easy, but the problem is the cache invalidation. There are two stategies for doing that

Push method - by some triggers (e.g., once an hour, or when a product changes), we prepare data and store it in the cache.
The positives are that data always will be in the cache when a user calls an API.
The negatives - we need to prepare data in all places where there are triggers, and be careful with cache invalidation (it's very easy to miss a place that occasionally changes your model).
Pull method - the moment we need data from the cache, we check it, and if there is no data - we get the data in the usual way and store it in the cache for the future.
It's easier in implementation, as we're worrying less about invalidation (though it still needs some work),
But it works worse with cache misses - as some users will wait for a response full time.
So to use that, your API should not be too slow!

For the push method, you are good to use cache_key, but for the pull method we need to check if the cache is still correct, so cache_key_with_version is our friend

I'd say generally it's better to start with pull method, and use push method in cases when it's possible to update the cache at some time, e.g. once a day.

But you should always take into consideration your requirements.

Going to the project

We'll go with pull method as an easier one, and use cache_key_with_version. Let's add a helper method in Grape API:

  helpers do
    def present(resource, namespace='MyAPI', caching: false)
      return resource.to_json unless caching

      Rails.cache.fetch([resource.cache_key_with_version, namespace].join('-'), expires_in: 1.day) do
        resourse.to_json
      end
    rescue Redis::TimeoutError
      resource.to_json
    end
  end

Besides resource.cache_key_with_version, we also make separate caches for different API namespaces (we don't want users to see admin's output).
In more complex examples, you could add more params, e.g., options for serializers if they change the output when SQL is the same.

Then we could use it in the API like this

class Edu::API::V2::Products < Grape::API
  resources :products do
    get do
      products = Product.some_query(params)
      present products, "API::V2::Products", caching: true
    end
  end
end

The next thing is to make sure we invalidate products if the output depends on related models. As our cache depends on the updated_at column of Products, and other models will not affect it by default.

class Option < ApplicationRecord
  # invalidate product cache if option changes 
  belongs_to :product, touch: true
end

class Product < ApplicationRecord
  # invalidate product cache if promos change
  has_and_belongs_to_many :promos,
                          after_add: :touch_updated_at,
                          after_remove: :touch_updated_at

  def touch_updated_at(_promo)
    touch if persisted?
  end
end

Things to take into consideration

It's better to have a separate Redis database number for the cache. There could be a case that Redis will be full of cache entries and remove e.g., Sidekiq-related data. (it depends on the Redis eviction config)
Beware of untrusted input. There could be a case when an attacker could fill your cache up by sending different versions of parameters.

E.g., if you have the parameter search, better not cache it as every cache will be different.

You could easily disable cache for specific params

    params do
      optional :search
    end
    get do
      products = Product.some_query(params)
      caching = params[:search].blank?
      present products, "API::V2::Products", caching: caching
    end

How to fix a segfault in Ruby

Dmitry Daw — Sat, 22 Jun 2024 19:16:22 +0000

Let's say you got a "Segmentation fault" error in Ruby

[BUG] Segmentation fault at 0x0000000000000028
ruby 3.4.0preview1 (2024-05-16 master 9d69619623) [x86_64-linux-musl]

-- Machine register context ------------------------------------------------
 RIP: 0x00007fefe4cd4886 RBP: 0x0000000000000001 RSP: 0x00007fefc95d3a10
 RAX: 0x0000000000000001 RBX: 0x00007fefc94212e0 RCX: 0x00007fefc95d0b70
 RDX: 0x0000000000000010 RDI: 0x0000000000000000 RSI: 0x00007fefc95d08f0
  R8: 0x0000000000000000  R9: 0x0000000000000000 R10: 0x0000000000000000
 R11: 0x0000000000000217 R12: 0x00007fefc9421340 R13: 0x00007fff5a0ec750
 R14: 0x00007fefe4649b10 R15: 0x00007fefc95d3b38 EFL: 0x0000000000010202

-- Other runtime information -----------------------------------------------

...

0x0000000000000028 near zero points out that something is NULL, but that's not much. To get more info, you could run your program under gdb

/app # gdb -q --args ruby test.rb

(gdb)

Here we are in a debugger. To run your program write run

(gdb) run
Starting program: /usr/local/bin/ruby test.rb
warning: Error disabling address space randomization: Operation not permitted
[New LWP 36]
[New LWP 37]
[New LWP 38]
execution expired

Thread 4 "ruby" received signal SIGSEGV, Segmentation fault.
[Switching to LWP 38]
0x00007f0a2c33b886 in freeaddrinfo (p=0x0) at src/network/freeaddrinfo.c:10
warning: 10     src/network/freeaddrinfo.c: No such file or directory

Okay, we see there is something with system's src/network/freeaddrinfo.c.
Let's get backtrace and check variables

(gdb) bt
#0  0x00007f0a2c33b886 in freeaddrinfo (p=0x0) at src/network/freeaddrinfo.c:10
#1  0x00007f0a10c1e940 in do_getaddrinfo (ptr=0x7f0a10f61200) at raddrinfo.c:426
#2  0x00007f0a2c35c349 in start (p=0x7f0a10afaa88) at src/thread/pthread_create.c:207
#3  0x00007f0a2c35e95f in __clone () at src/thread/x86_64/clone.s:22
Backtrace stopped: frame did not save the PC

(gdb) info args
p = 0x0

Okay, now we see that the problem comes from ruby's raddrinfo.c, and there is argument p that is NULL.

We could also go up in the stack, and check variables

(gdb) info locals
cnt = 1
b = <optimized out>

(gdb) frame 1
#1  0x00007f1068ec6940 in do_getaddrinfo (ptr=0x7f1068cf0c40) at raddrinfo.c:426
warning: 426    raddrinfo.c: No such file or directory

(gdb) info args
ptr = 0x7f1068cf0c40

(gdb) info locals
arg = 0x7f1068cf0c40
err = <optimized out>
gai_errno = <optimized out>
need_free = 0

Now we're prepared to look what is happening in the code. Lets check raddrinfo.c, line 426

// ext/socket/raddrinfo.c
...
        if (arg->cancelled) {
            freeaddrinfo(arg->ai);
        }
...

Indeed freeaddrinfo is called.
Now could make a bug in https://bugs.ruby-lang.org/, or try to debug it by yourself.

I've tried :)

We're on Alpine, on ruby:3.3.3-alpine. And on different system, e.g. ruby:3.3.3 is all okay, so it should be something with Alpine.

Some search tells us that indeed: freeaddrinfo in Alpine's musl library does not accept NULL pointer(link), in difference with glibc(which is used e.g. in Ubuntu)(link)

So let's fix it.

Firstly we need to build ruby. For convenience lets create a small Dockerfile

FROM alpine:3.20

WORKDIR /usr/src/app

RUN apk update && apk add autoconf gcc build-base ruby ruby-dev openssl openssl-dev yaml-dev zlib-dev yaml gdb

CMD sh

Clone ruby

$ git clone --depth 1 git@github.com:ruby/ruby.git
$ cd ruby

Go inside our alpine docker container

$ docker build -t my-ruby-develop .
$ docker run -it --rm -v $(pwd):/usr/src/app -w /usr/src/app  my-ruby-develop sh

And build the latest ruby version - to check the bug is still present.

$ ./autogen.sh
$ mkdir build && cd build
$ mkdir rubies
$ ../configure --prefix="/usr/src/myapp/build/rubies/ruby-master" && make && make install

Now we have our latest ruby in /usr/src/myapp/build/rubies/ruby-master folder.

Let's check the problem is still exist

/app # /usr/src/app/build/rubies/ruby-master/bin/ruby test.rb
Operation timed out - user specified timeout
[BUG] Segmentation fault at 0x0000000000000028

-- Machine register context ------------------------------------------------
 RIP: 0x00007f561acf6886 RBP: 0x0000000000000001 RSP: 0x00007f55ff5d2a10
 RAX: 0x0000000000000001 RBX: 0x00007f55ff43ff30 RCX: 0x00007f55ff5cfb70
 RDX: 0x0000000000000010 RDI: 0x0000000000000000 RSI: 0x00007f55ff5cf8f0
  R8: 0x0000000000000000  R9: 0x0000000000000000 R10: 0x0000000000000000
 R11: 0x0000000000000217 R12: 0x00007f55ff43ff90 R13: 0x00007f55ff236040
 R14: 0x00007f55ff236b38 R15: 0x00007f55ff5d2b38 EFL: 0x0000000000010202

It is for sure.

Then we need to

check in which case the problem is happening
find if it should be changed inside Alpine or Ruby
are there other places that could be related to the same problem
make a reproducible example

etc etc - typical engineering work.

In our case, it should be changed inside ruby. Let's make the change

// ext/socket/raddrinfo.c
...
        if (arg->cancelled) {
            if (arg->ai) freeaddrinfo(arg->ai);
        }
...

Make the ruby(with make clean first) and try again. It works!

/app # make clean && ../configure --prefix="/usr/src/myapp/build/rubies/ruby-master" && make && make install
/app # /usr/src/app/build/rubies/ruby-master/bin/ruby test.rb
Good

Great! Now we can run the tests, for related file, and the whole set

$ make test-all TESTS=../test/socket/test_addrinfo.rb
$ make test-all
$ make test-spec

And if possible - write a test for your change(in this case it is hard to write a reliable test because of getaddrinfo internals).

Don't forget to describe a bug in ruby's bugtracker, and attach all useful info(e.g. https://bugs.ruby-lang.org/issues/20592)

And voila! You made the world a bit better https://github.com/ruby/ruby/commit/fba8aff7af450e476e97b62385427dfa51850955

How to use rbtrace from outside of docker container

Dmitry Daw — Sat, 15 Jun 2024 09:02:45 +0000

Rbtrace is a great tool to see which functions your app is calling at the time.

In most cases you want it to be installed inside a docker container, but sometimes you can't.

In this cases, you could run it from outside, but you need to share several things with the host:

IPC(Inter-process communication) namespace. Rbtrace is using IPC to connect his inside app process with the outside tracer.
PID namespace. Rbtrace is sending signals to PID, so you should be able to send these.
The /tmp folder. Rbtrace creates sockets to connect with process, like /tmp/rbtrace-1688246.sock.

So, you can setup your app like this:

Inside the app

docker-compose.yml

  app:
    # ...
    ipc: host
    pid: host
    volumes:
      - /tmp/:/tmp
      # ...

Gemfile

gem 'rbtrace'

Outside the app

# run as root
$ rbtrace -p 1688246 --firehose

where 1688246 - is PID of the process you want to trace.
And voila! You could get your nice and shiny traces, e.g.

Kernel#public_send                                      
  Puma::ThreadPool#reap                                    
    Thread::Mutex#synchronize                                  
      Array#reject                                      
        Thread#alive? <0.000002>                                  
        Thread#alive? <0.000001>                              
        Thread#alive? <0.000001>                                 
        Thread#alive? <0.000001>                                  
        Thread#alive? <0.000002>                               
      Array#reject <0.000020>                                  
      Array#each <0.000002>
      Array#delete_if
        Array#include? <0.000002>
        Array#include? <0.000001>
        Array#include? <0.000001>
        Array#include? <0.000001>
        Array#include? <0.000002>
      Array#delete_if <0.000019>
    Thread::Mutex#synchronize <0.000053> 
  Puma::ThreadPool#reap <0.000058>
Kernel#public_send <0.000067>

Another great parameter for rbtrace is --slow, which shows only methods that took more than N ms.

rbtrace -p 1746688 --slow=500 > trace.slow.log

NOTE: rbtrace uses PID in three ways - as PID to send signals, as a key for IPC message queue, and as a prefix for the socket file.

Base test profiling inside docker with test-prof

Dmitry Daw — Thu, 30 May 2024 18:32:27 +0000

Install test-prof and stackprof gems.

group :test do
  gem "test-prof", "~> 1.0"
  gem "stackprof", '>= 0.2.9', require: false
end

Run test with flag TEST_STACK_PROF=1. Better to run only part of tests or use SAMPLE=10 flag, otherwise you could get too big of a stack dump.

TEST_STACK_PROF=1 bundle exec rspec spec/my_test.rb

Then you could visualize it.

Visualize with speedscope

Another way is to use speedscope, as it provides better interface. For this, generate json output

stackprof --flamegraph tmp/test_prof/stack-prof-report-wall-raw-total.dump > tmp/test_prof/stack-prof-report-wall-raw-total.json

And upload the file to speedscope, e.g. at https://www.speedscope.app/

Visualize with stackprof's viewer

Then convert flamegraph dump to html

stackprof --flamegraph tmp/test_prof/stack-prof-report-wall-raw-total.dump > tmp/test_prof/stack-prof-report-wall-raw-total.html

Then you could see the flamegraph with gem's viewer. From inside docker you could copy flamegraph.js and viewer.html to same shared folder, e.g. tmp folder in project.

cp /usr/local/bundle/gems/stackprof-0.2.26/lib/stackprof/flamegraph/viewer.html ./tmp/test_prof/
cp /usr/local/bundle/gems/stackprof-0.2.26/lib/stackprof/flamegraph/flamegraph.js ./tmp/test_prof/

And then open in the browser url like

file:///home/my/path_to_project_folder/tmp/test_prof/viewer.html?data=/home/my/path_to_project_folder/tmp/test_prof/stack-prof-report-wall-raw-total.html

Is ActiveRecord right in omitting parentheses in queries? (and how ChatGPT lies again)

Dmitry Daw — Tue, 16 Apr 2024 11:45:19 +0000

I need to get a selection of users with an SQL query like this:



sql =<<~SQL
    token IS NULL OR
    (
      token = 'some_token'
      AND (
        state = 'cancelled' AND created_at > ?
        OR state = 'submitted'
      )
    )
  SQL
User.where(sql, Time.now)

I don't like to have SQL in my queries — all hidden code in scopes is now pops up, and it is hard to compose.

But could I rewrite this query with ActiveRecord? Let's try:



User
  .where(token: nil)
  .or(
    User.where(token: 'some_token').and(
      User.where(state: 'cancelled')
        .or(User.where('created_at > ?', Time.now))
    )
  )

Looks better, but does it generate the same SQL?



SELECT "users".*
FROM "users"
WHERE ("users"."token" IS NULL
       OR "users"."token" = 'some_token'
       AND ("users"."state" = 'cancelled'
            OR created_at > '2024-04-16 10:46:43.129109'))

Hm, where do the parentheses after IS NULL OR go? Doesn't it look like a different condition?

Tests are passing, but if ActiveRecord omits parentheses - wouldn't it be problematic in some cases?

For example, if we have an expression like this a || (b && (c || d)) - would it be the same if there were no parentheses, like a || b && c || d?

We are programmers in the modern era, so let's ask ChatGPT.

ChatGPT answer:

So it seems all is okay. But let's (just for the sake of the experiment) check manually too:



booleans = [true, false]
combinations = booleans.product(booleans, booleans, booleans)
combinations.each do |a, b, c, d|
  expression1 = a && (b || (c && d))
  expression2 = a && b || c && d
  if expression1 != expression2
    puts "a: #{a}, b: #{b}, c: #{c}, Expression1: #{expression1}, Expression2: #{expression2}"
  end
end

=> a: false, b: true, c: true, Expression1: false, Expression2: true
=> a: false, b: false, c: true, Expression1: false, Expression2: true

So, it's actually not. This robotic liar!

Okay, a && (b || (c && d)) and a && b || c && d are not equivalent.
Let's check another one:



combinations = booleans.product(booleans, booleans, booleans, booleans,
                                booleans, booleans, booleans, booleans, booleans)
combinations.each do |a, b, c, d, e, f, g, i, k|
  expression1 = a && b || (c && d || (e && f || (g && i || k)))
  expression2 = a && b || c && d || e && f || g && i || k
  if expression1 != expression2
    puts "a: #{a}, b: #{b}, c: #{c}, Expression1: #{expression1}, Expression2: #{expression2}"
  end
end

These are the same! But how? So many groups we have!

How's that? It's because of logical operators' precedence: logical AND evaluates before logical OR.

So in a || b && c, it will first evaluate b && c, and then ||. We could see it like this: a || (b && c).

And it's the same in most languages - in Ruby and PostgreSQL for sure.

To make it easier to understand, we could rewrite expressions as multiplication and addition: because logical AND with binary values works exactly like normal arithmetic multiplication, and logical OR works in many senses as arithmetic addition.

E.g. a + (b * c) is the same as a + b * c.

And in our expressions:



  # these are not the same
  expression1 = a * (b + (c * d))
  expression2 = a * b + c * d

  # these are the same
  expression3 = a * b + (c * d + (e * f + (g * i + k)))
  expression4 = a * b + c * d + e * f + g * i + k

Now it's starting to make sense - ActiveRecord is in its right to omit parentheses in our SQL example, as it is similar to expression3. And it's actually an optimization to send less data over the network.

But let's check, does it work correctly with expressions like a && (b || (c && d))?



User
  .where(token: nil)
  .and(
    User.where(token: 'some_token')
      .or(
        User.where(state: 'cancelled')
            .where('created_at > ?', Time.now)
      )
    )



SELECT "users".*
FROM "users"
WHERE "users"."token" IS NULL
  AND ("users"."token" = 'some_token'
       OR "users"."state" = 'cancelled'
       AND (created_at > '2024-04-16 11:15:36.584382'))

Good - parentheses after AND are in their place!

And if we rewrite OR to AND?



User
  .where(token: nil)
  .and(
    User.where(token: 'some_token')
      .and(
        User.where(state: 'cancelled')
            .where('created_at > ?', Time.now)
      )
    )



SELECT "users".*
FROM "users"
WHERE "users"."token" IS NULL
  AND "users"."token" = 'some_token'
  AND "users"."state" = 'cancelled'
  AND (created_at > '2024-04-16 11:20:36.296399')

It omits all parentheses—and rightfully so.

Okay, but we could use not only the .or method but also OR directly, like .where("... OR ..."). Does ActiveRecord handle this?
This is easy — .where(sql) always wraps the expression inside it in parentheses:



User.where(token: 'some_token')
    .where("state = 'cancelled' OR created_at > ?", Time.now)



SELECT "users".*
FROM "users"
WHERE "users"."token" = 'some_token'
  AND (state = 'cancelled'
       OR created_at > '2024-04-16 11:23:00.877269')

Even if it's only one condition



User.where(token: 'some_token')
    .where("state = 'cancelled'").to_sql



SELECT "users".*
FROM "users"
WHERE "users"."token" = 'some_token'
  AND (state = 'cancelled')

So, it seems all is good, and we could use our nice and clean ActiveRecord! Nice!

References:

Deploy Anycable with Kamal(formely MRSK)

Dmitry Daw — Thu, 03 Aug 2023 13:43:17 +0000

Here we'll deploy Anycable wih Kamal(MRSK before).

We need our Rails code for anycable-rpc, so anycable-rpc will clearly be a role.
The Anycable websocket server (in our case, anycable-go) could be installed as an accessory, or as a role.

To simplify the setup, we'll utilize --server-command option of anycable.
This will run the specified command after anycable is launched.
In our case - it'll initiate anycable-go.

bundle exec anycable --server-command "anycable-go"

To run anycable-go, we need to install its binary directly in our Rails app's Docker container:

# Dockerfile
# ...
RUN curl -fsSL https://github.com/anycable/anycable-go/releases/download/v1.4.1/anycable-go-linux-amd64 -o anycable-go \
    && chmod +x anycable-go
    && cp anycable-go /usr/local/bin/anycable-go
# ...

MRSK config

We have websockets on the same host as our main web app. Therefore, we need to set up Traefik so that the /cable path will be handled by the anycable-go server, not the web.

To do this, we can utilize this config:

    # role's options in config/deploy.yml
    labels:
      traefik.enable: true
      traefik.http.routers.my_service-anycable.rule: 'PathPrefix(`/cable`)'
      traefik.http.routers.my_service-anycable.entrypoints: 'web'
      traefik.http.services.my_service-anycable.loadbalancer.server.port: 8080

In my_service-anycable, my_service is the service name in MRSK, and anycable is the role name in MRSK.
This could also contain a destination, if you're using it.
This name should be the same as the role's traefik_service name.

traefik.http.routers.my_service-anycable.rule: 'PathPrefix(/cable)' adds a rule that we only want requests from the /cable path.
traefik.http.routers.my_service-anycable.entrypoints: 'web' sets the entrypoint to the default web entrypoint at port 80
traefik.http.services.my_service-anycable.loadbalancer.server.port: 8080 sets the port to which Traefik should send requests to our container.

Then, we can add the necessary Anycable environment variables and set up the MRSK deploy config:

# config/deploy.yml

service: my_service
servers:
  web:
    - 10.0.0.155
  # anycable-rpc together with anycable-go
  anycable:
    hosts:
      - 10.0.0.155
    cmd: bundle exec anycable --server-command "anycable-go"
    env:
      clear:
        ANYCABLE_RPC_HOST: localhost:50051
        ANYCABLE_HOST: "0.0.0.0"
        ANYCABLE_PORT: 8080
        ANYCABLE_REDIS_URL: "redis://:<%= ENV['REDIS_PASSWORD'] %>@<%= ENV['REDIS_HOST'] %>:6379/0"
        ANYCABLE_RPC_HOST: "localhost:50051"
    labels:
      traefik.enable: true
      traefik.http.routers.my_service-anycable.rule: 'PathPrefix(`/cable`)'
      traefik.http.routers.my_service-anycable.entrypoints: 'web'
      traefik.http.services.my_service-anycable.loadbalancer.server.port: 8080

env:
  clear:
    ANYCABLE_REDIS_URL: "redis://:<%= ENV['REDIS_PASSWORD'] %>@<%= ENV['REDIS_HOST'] %>:6379/0"
    REDIS_URL: "redis://:<%= ENV['REDIS_PASSWORD'] %>@<%= ENV['REDIS_HOST'] %>:6379/1"
    ANYCABLE_URL: "wss://<%= ENV['ACTIONCABLE_HOST'] %>/cable"

That's all! This sets up anycable-rpc with anycable-go on your server.

Other deploy variants

If you want to separate anycable-rpc and anycable-go, you have several options to do so.

Separate hosts

You could deploy anycable-go as an accessory on another host, then you could use a different Docker image and not install it in the Rails app's Docker image.

But in this setup, you need to publish ports for both anycable-rpc and anycable-go.

service: my_service
servers:
  web:
    - 10.0.0.155
  anycable-rpc:
    hosts:
      - 10.0.0.155
    cmd: bundle exec anycable
    port: 50051
    env:
      clear:
        ANYCABLE_RPC_HOST: 0.0.0.0:50051
    labels:
      traefik.tcp.routers.my_service-anycable-rpc.rule: 'HostSNI(`*`)'
      traefik.tcp.routers.my_service-anycable-rpc.entrypoints: rpc
      traefik.tcp.services.my_service-anycable-rpc.loadbalancer.server.port: 50051

traefik:
  options:
    publish:
      - 50051:50051
  args:
    entrypoints.web.address: ":80"
    entrypoints.rpc.address: ":50051"
    accesslog: true

accessories:
  ws:
    image: anycable/anycable-go:1.4
    host: 10.0.0.166
    port: "80:8080"
    env:
      clear:
        ANYCABLE_HOST: "0.0.0.0"
        ANYCABLE_PORT: 8080
        ANYCABLE_REDIS_URL: "redis://:<%= ENV['REDIS_PASSWORD'] %>@<%= ENV['REDIS_HOST'] %>:6379/0"
        ANYCABLE_RPC_HOST: "10.0.0.155:50051"
        ANYCABLE_DEBUG: true

Setup anycable-go as accessory on the same host

You (most likely) could use anycable-go as an accessory, and write Traefik configs in the labels of the accessory.
But after doing this, you'll need to use the host's IP address or the Docker internal network (as mentioned below in other options).

Use the host's IP address

Just publish the ports to the outside and use the host's IP address from inside the container.

Use the host's network

Add the internal host IP as the Docker internal host.

servers:
  web:
    hosts:
      - 10.0.0.155
  anycable-rpc:
    hosts:
      - 10.0.0.155
    options:
      "add-host": host.docker.internal:host-gateway
    # ... env, cmd, etc
  anycable-go:
    hosts:
      - 10.0.0.155
    options:
      "add-host": host.docker.internal:host-gateway
    # ... env, cmd, etc

(source)

Then you could use host.docker.internal as host in anycable-rpc and anycable-go containers

Create docker internal network by hands and then use it

# Create the network first by ssh'ing into machine or using something like Ansible

servers:
  web:
    hosts:
      - 10.0.1.183
    options:
      "network": "my-network"

traefik:
  options:
    network: "my-network"

accessories:
  db:
    image: postgres:14
    host: 10.0.1.183
    ... etc
    options:
      network: "my-network"

(source)

And then use hosts from this network

How to port VS Code plugins to your favorite editor(e.g. Emacs), with the example of Github Copilot Labs.

Dmitry Daw — Sat, 01 Jul 2023 22:05:31 +0000

TLDR: Proof of concept code is here https://github.com/haukot/copilot_labs_plugin_base

VS Code plugins are essentially JS files that VS Code runs with its own callbacks. So we can write our own wrapper, which will define the functions needed for the plugin, and stub all other functions.

A simple example is the Github Copilot Labs extension, because it only needs selection and command.

VS Code extension is essentially a js module, which exports several functions. We are interested in init() and activate().

Now, let's download it, unzip it, and move it to the extension folder.

Then we can initialize it

var extension = require('./extension/extension/dist/extension.js');

await extension.init();
await extension.activate();

It fails, because it needs vscode to run(obviously!). So let's create our own VS Code.

// vscode/index.js
let vscode = {}
vscode.version = 'MINE'
module.exports = vscode;

// package.json
...
  "dependencies": {
    "vscode": "file:vscode"
  }
...

Now our extension will use our own VS Code! It still fails, though.
We need to stub the methods the extension waits for, but most of them will be simple dummy methods like this

vscode.Uri = {
  parse: (e) => {
    return {}
  }
}

You can see all of the methods here.

Now let's initialize extension(we have also added several dummy settings in activate)

await extension.init();
await extension.activate({
  extension: {
    packageJSON: {
      name: 'vscode-copilot',
    }
  },
  subscriptions: [],
  globalState: {
    setKeysForSync: () => {},
  },
});

Good, no errors! But we want output.
Github Copilot Labs main feature is "Use brush", which is works as an executeCommand in VS Code.

Let's look at our stub for commands

// NOTE: this is our variable, not vscode's
vscode._registeredCommands = {}

vscode.commands = {
  registerCommand: (e, t) => {
    let command = { name: e, callback: t }
    vscode._registeredCommands[e] = t
    return command
  },
  // NOTE: you need to replace this method to implement your functionality
  executeCommand: async (e, ...z) => {
    let command = { name: e, args: z }
    if (vscode._registeredCommands[e]) {
      vscode._registeredCommands[e](...z)
    } else {
      return { status: 'NotRegistered' }
    }
  }
}

It doesn't do anything complex, just registers the command from extension.js, and runs it by executeCommand.

Let's change it for Copilot Labs. Its command accepts a brush name and changes current selection.
So, let's redefine executeCommand so it'll accept text from an external source!

vscode.commands.executeCommand = async (name, ourParams, ...args) => {
  let command = { name, args }
  if (vscode._registeredCommands[name]) {
    return new Promise((resolve, reject) => {
      vscode.window.activeTextEditor = {
        ...vscode.window.activeTextEditor,
        ...{
          document: {
            getText: (currentSelection) => {
              // we already return content, so don't need to use currentSelection
              return ourParams.fileContent;
            },
            languageId: ourParams.languageId,
          },
          async edit(callback) {
            let toChange = {
              replace: (selection, newContent) => {
                // we'll return whole content, so don't need to use selection
                resolve(newContent)
              }
            }
            callback(toChange)
          }
        }
      }
      vscode._registeredCommands[name](...args)
    });
  } else {
    return { status: 'NotRegistered' }
  }
}

What's happening here:

We redefine the window.activeTextEditor.document.getText function so it'll return our code.
We redefine the window.activeTextEditor.edit function, which will be called after the extension completes the command and tries to change current selection.
We run vscode._registeredCommands[name](...args), which will run command inside extension.js and start the whole process.
We wrap all this in a Promise, so we can get the result of the execution.

And now we could execute the command like this

let fileContent = 'def hello():\n  print("Hello, world!")\n\nhello()'
let languageId = 'python';

vscode.commands.executeCommand('copilot-labs.use-brush', { fileContent, languageId }, 'debug')
    .then((result) => {
      console.log("RESULT", result);
    })
    .catch((error) => {
      console.log("ERROR", error);
    });

Good! To integrate it with an IDE, we need some interface to run it, e.g. a jsonrpc server. IDE plugin will start it and send commands to it.
It's not so interesting, so I'll just link the implementations: IDE plugin's side and jsonrpc-server. (Sorry for many comments, for now I don't have to clean all this up :')

This is a simple example with a simple extension, but it may pave the way for integrating more complex extensions too.

An example of a similar approach I found in in the LSP wrapper for typescript extension of vscode.
It goes much farther and uses vscode itself(see files 1, 2, 3).

It seems like overkill for an extension like Copilot Labs, but maybe it could be more useful for wrapping some complex extensions.

Goodbye, and happy hacking!

Trick to cleaner use Sidekiq::Testing.inline! in tests

Dmitry Daw — Mon, 12 Jun 2023 16:31:26 +0000

By default, Sidekiq jobs don't run in tests. But sometimes, e.g. in e2e tests, you'd want them to run. Sidekiq provides a method Sidekiq::Testing.inline! that enables job processing.

If you run this method once, it enables job processing in all tests; if you run it with a block, it processes jobs only within that block. But it's not very clean to have blocks everywhere in your tests. So there's a trick to isolate Sidekiq::Testing.inline! within the scope of your full test or specific contexts.

You can utilize RSpec's configuration options to control when Sidekiq::Testing.inline! is invoked. You can set it up to run based on the presence of specific metadata in your tests, like so:

# spec/spec_helper.rb
RSpec.configure do |config|
  ...
  config.around do |example|
    if example.metadata[:sidekiq_inline] == true
      Sidekiq::Testing.inline! { example.run }
    else
      example.run
    end
  end
  ...
end

With this configuration, Sidekiq::Testing.inline! will only run when the :sidekiq_inline metadata is present and set to true in a test. This can be done as follows:

context 'my context', :sidekiq_inline do
  # your tests go here
end

describe 'my describe', :sidekiq_inline do
  # your tests go here
end

it 'my test', :sidekiq_inline do
  # your test goes here
end

With this approach, you can better control when Sidekiq jobs are processed during testing, leading to cleaner tests.

How to cache Kamal(MRSK) deployments in CI

Dmitry Daw — Thu, 01 Jun 2023 17:04:39 +0000

Caching is a key strategy in optimizing Docker deployments. But because CI don't saves previous docker layers, and, unfortunalety, MRSK doesn't have options like --cache-to, MRSK(now Kamal) rebuilds each Docker container from scratch, leading to a slower deploys.

There are two methods to implement caching with MRSK in GitHub Actions: pulling the Docker image before deployment and using the --skip_push option with the build command.

Method 1: Pull the Docker Image before Deployment

This approach is suitable for simple Dockerfiles.
First, the Docker image is pulled from the registry, then the MRSK deploy command is run. By pulling the Docker image first, MRSK can use the layers from the previously built image, reducing the need for a complete rebuild.

- name: Login to DockerHub
  uses: docker/login-action@v2
  with:
    username: ${{ secrets.DOCKERHUB_USERNAME }}
    password: ${{ secrets.DOCKERHUB_PASSWORD }}

- name: Pull Docker Image
  run: docker pull ${{ secrets.DOCKERHUB_USERNAME }}/your-image-name || true

- name: Deploy with MRSK
  run: mrsk deploy

In this code snippet, the docker pull command pulls the Docker image from the registry. The || true part ensures that the workflow doesn't fail if the image doesn't exist yet (for instance, during the first run).

But the problem with this method is that it's not working for workflows where only final Docker image is pushed.

Method 2: Use the --skip_push Option and Build Command before Deployment

This method is for complex workflows, for example for Dockerfiles which have separated final and builder images. Because in such Dockerfiles we push only final image, we can't get builder cache from pulling latest image.

The --skip_push option in MRSK not only skips the push, but also the Docker build process. Therefore, we can perform the build manually and then use MRSK for deployment only.

- name: MRSK envify
  env:
    VERSION: ${{ github.sha }} # set version to ensure we'll build the same version
  run: 'mrsk envify -v'

- name: Set up Docker Buildx.
  uses: docker/setup-buildx-action@v2

- name: Build and push
  uses: docker/build-push-action@v4
  with:
    context: .
    push: true
    tags: ${{ secrets.DOCKERHUB_USERNAME }}/your-image-name:${{ github.sha }},${{ secrets.DOCKERHUB_USERNAME }}/your-image-name:latest
    cache-from: type=gha # https://docs.docker.com/build/cache/backends/gha/
    cache-to: type=gha,mode=max

- name: MRSK deploy
  run: 'mrsk deploy -vvvv --skip_push'

In that way we could fully embrace CI cache and speedup deployments.

What to do if Sidekiq job is stuck

Dmitry Daw — Fri, 27 Jan 2023 09:55:37 +0000

If your Sidekiq job is stuck, your first intention may be to try to stop this job. But Sidekiq doesn't have such mechanism, and it may make your worker stuck even futher, and to clean current processing job.

So the first thing that should be - to get logs about what is happening. To do that, send the TTIN signal to dump backtraces to the log, with command kill -TTIN [Sidekiq pid here. In docker container it most probably 1]

$ ps ax | grep sidekiq
PID TTY      STAT   TIME COMMAND                                                                                                                                            
  1 ?        Ssl   94:02 sidekiq 5.2.9 app [1 of 20 busy] 
$ kill -TTIN 1

It will dump the logs to stdout. Depending on your application settings, you may want to check logs on your logs collector service, e.g. in Kibana dashboard.

If the Sidekiq process is not responding, you can use GDB to dump backtraces for all threads with command gdb [ruby exec] [process number], like this:

$ which ruby                                                                                                                                                               
#=> /usr/local/bin/ruby
$ gdb /usr/local/bin/ruby 1

(gdb) info threads
  Id   Target Id         Frame 
  37   Thread 0x7f8b289d8700 (LWP 7994) "ruby-timer-thr" 0x00007f8b27a20d13 in *__GI___poll (fds=<optimized out>, fds@entry=0x7f8b289d7ec0, nfds=<optimized out>, 
    nfds@entry=1, timeout=timeout@entry=100) at ../sysdeps/unix/sysv/linux/poll.c:87
  36   Thread 0x7f8b23eb0700 (LWP 7995) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  35   Thread 0x7f8b23c2e700 (LWP 7996) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  34   Thread 0x7f8b239ac700 (LWP 7997) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  33   Thread 0x7f8b237aa700 (LWP 7998) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  32   Thread 0x7f8b28844700 (LWP 8002) "SignalSender" sem_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_wait.S:86
  31   Thread 0x7f8b1e1bf700 (LWP 8003) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  30   Thread 0x7f8b1e0be700 (LWP 8006) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  29   Thread 0x7f8b1dd81700 (LWP 8009) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  28   Thread 0x7f8b1dc80700 (LWP 8010) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
  27   Thread 0x7f8b1db7f700 (LWP 8011) "ruby" pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
...

(gdb) set logging file gdb_output.txt
(gdb) set logging on
(gdb) set height 10000
(gdb) t a a bt
(gdb) quit

It will save logs to gdb_output.txt.

You also can puts ruby logs to stdout, like so

(gdb) call (void)rb_backtrace()

After that, you can try to stop job in Sidekiq Admin with 'Stop' button, or with signals(TSTP to ask it to not pick up any new jobs, and TERM next).

Links

1 Sidekiq FAQ about frozen processes
2 Another guide about frozen processes
3 How to use GDB with ruby processes
4 How to stop Sidekiq jobs(as your application mechanic)

Troubleshooting weird DB connection errors in Rails with PostgreSQL

Dmitry Daw — Wed, 18 Jan 2023 13:28:08 +0000

There are a number of errors that occur when attempting to use a database connection that has suddenly become dead.

Errors like PG::ConnectionBad: PQsocket() can't get socket descriptor:, PG::UnableToSend: no connection to the server, PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly, PG::UnableToSend: SSL connection has been closed unexpectedly, and so on.

They may be caused by a few different factors, e.g. long command run, or long SSH connection, or closed connection in separate thread(like in 'parallel' gem), or long code processing before save.

The cause

If you see errors like that, and do have some slow processing, there are several things to check:

1) Rails's database idle_timeout setting, which is 300 seconds by default. This means that if a connection is idle for more than 300 seconds, the server will close it.
2) PgBouncer client_idle_timeout setting. This setting controls how long a connection can be idle before PgBouncer closes it. If this setting is set too low, it can cause the same errors as Rails's idle_timeout.
3) PostgreSQL idle_session_timeout settings (from PostgreSQL 14) and idle_in_transaction_session_timeout(for sessions that were idle in transaction).
4) TCP or other network or firewall timeouts(e.g. there is a case of the timeout in Azure Cloud)

The solution

To solve this problem, there are a few possible solutions:

0) Speed up slow processing
1) Increase the idle_timeout(or another guilty timeout) setting to a higher value to prevent a server from closing idle connections too soon.
2) Add a database setting called reaping_frequency: 10, which will clear and restore connections more frequently to prevent them from becoming idle.
3) Add manual reconnection, like the code snippet below, which releases the connection and re-establishes it before saving a record to the database.

    ActiveRecord::Base.connection_pool.release_connection
    ActiveRecord::Base.connection_pool.with_connection do
      # save record in database
    end

Resources

Links:
(1)(2)(3)(4) Info about reaping_frequency
(5)(6) Info about Rails's idle_timeout
(7)(8)(9) Solutions with manual reconnection

How to query Ancestry fast

Dmitry Daw — Thu, 29 Apr 2021 20:26:12 +0000

Ancestry is a great library to organize models in a tree structure.

For example, if you have Location model, which may be City, State, and Country, and you want to easily organize and query between all those entities.

By default Ancestry makes requests with LIKE query, e.g.

SELECT
  locations.*
FROM 
  locations 
WHERE
  "locations"."ancestry" LIKE '4a5b7a5d-1147-4d09-8a5d-a1a476c5be50/88fd88b7-eecd-4fc0-b18d-e2b6a077ab66/550f9809-e875-4fab-8cac-db8e6d59e0e5/%' 
OR 
  "locations"."ancestry" = '4a5b7a5d-1147-4d09-8a5d-a1a476c5be50/88fd88b7-eecd-4fc0-b18d-e2b6a077ab66/550f9809-e875-4fab-8cac-db8e6d59e0e5'

Sometimes you need to write a query to check all descendants, like

SELECT
  locations.*, count(distinct cities.id)
FROM 
  locations 
LEFT JOIN locations as cities ON 
  cities.ancestry LIKE '%' || locations.id || '%'
WHERE
  cities.location_type = 'CITY'
GROUP BY
  locations.id

(in this example locations.id is uuid, so we can use simplified LIKE query with '%' || locations.id || '%'. With int id it requires a different request)

But this request is running too long.
PostgreSQL does have indexes that may speed up LIKE queries, for example GIN and GiST.

But before using index, we can change that query to this

SELECT
  locations.*, count(distinct cities.id)
FROM 
  locations 
LEFT JOIN locations as cities ON 
  locations.id = ANY((string_to_array(cities.ancestry, '/')::uuid[]))
WHERE
  cities.location_type = 'CITY'
GROUP BY
  locations.id

And it runs MUCH faster. On our data, we get an improvement from 5 seconds to 6 ms(!).

And it's possible to use it with int ids, just change uuid[] to int[].

After we can also add GIN index to it, if we want to.

add_index :locations, "(string_to_array(ancestry, '/')::uuid[])", using: :gin

But it already runs much faster.

Another way to speed up Ancestry queries is to use ltree index, but it requires a diffent syntax.

Sources:

string_to_array: https://github.com/stefankroes/ancestry/issues/466
GIN and GiST index documentation: https://www.postgresql.org/docs/9.1/textsearch-indexes.html

ltree: https://github.com/stefankroes/ancestry/issues/102
ltree index documentation: https://www.postgresql.org/docs/9.1/ltree.html