DEV Community: PikachuEXE

How to: Use SourceTree, with git hook via lefthook, running bundle exec with asdf managing Ruby versions

PikachuEXE — Thu, 21 Oct 2021 05:40:35 +0000

That's a long title huh?

Anyway I got a combination of software I use with an issue that I cannot find the answer on the Internet.

Background

At the time of writing, I use

SourceTree 3.2.1
Git (latest, version doesn't matter)
lefthook 0.7.7 (as git hook manager)
asdf with ruby plugin

Lefthook Setup

I have hook scripts inside folder ./.githooks under project root (all version controlled)

git config core.hooksPath .githooks
brew install lefthook

Workaround for source tree for path issue

sudo launchctl config user path `echo $PATH`

Ref:

Issue

I have recently switched from RVM to asdf.
After a restart of my computer I realize that one of my git hooks running bundle exec rubocop no longer works.
The workaround above works for RVM.

Investigation

lefthook binary is found (with workaround above)
bundle is using the binary from MacOS built-in Ruby

Solution

In my hook script pre-commit
Add the following before the code running lefthook

# ASDF compatibility
if test -f /usr/local/opt/asdf/libexec/asdf.sh ; then
  . /usr/local/opt/asdf/libexec/asdf.sh
fi
if test -f /opt/homebrew/opt/asdf/libexec/asdf.sh ; then
  . /opt/homebrew/opt/asdf/libexec/asdf.sh
fi

Zulip Datadog Integration

PikachuEXE — Mon, 02 Aug 2021 02:59:22 +0000

Zulip Datadog Integration

This is to record my findings during the research for Zulip Datadog integration.

TL;DR is at the end

Slack Compatible Incoming Webhook

Might be used for custom format message on Slack too though I have not tested that

Also the formatting on Zulip is still limited and not to mention the translation of Slack message formatting to Zulip one

After testing this I go for native one instead

To integrate Datadog with Zulip, the webhook integration on Datadog should be used

And the payload can be sent to one of your Zulip organization's endpoint which is either:

Slack Compatible Incoming Webhook
Zulip Native API - Send Message Endpoint

References

URL
https://your_org.zulipchat.com/api/v1/external/slack_incoming?api_key=your_bot_api_key&stream=your_stream_name

Monitor Notification

The following one only tested for Monitor Notification

Not sure what else a webhook can be used for

Example Payload
Example JSON payload with similar format to DataDog Slack Integration one

Can be verified via https://app.slack.com/block-kit-builder

{
    "blocks": [
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "<https://duck.com|Triggered: CPU load is very high on your.host.name -  - [non China]>"
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "Check CPU usage just in case  @slack-datadog"
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "system.cpu.user over host:your.host.name was > 70.0 on average during the last 5m."
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "Metric value: 84.513"
            },
            "fields": [
                {
                    "type": "mrkdwn",
                    "text": "*Tags*\nhost:your.host.name"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Notified*\n@slack-datadog"
                }
            ]
        },
        {
            "type": "image",
            "image_url": "http://placekitten.com/700/500",
            "alt_text": "Multiple cute kittens"
        }
    ]
}

Actual Payload

{
    "blocks": [
        {
            "type": "header",
            "text": {
                "type": "mrkdwn",
                "text": "TEST MESSAGE ONLY"
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "<$LINK|$EVENT_TITLE>"
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "$EVENT_MSG"
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "$ALERT_STATUS"
            }
        },
        {
            "type": "section",
            "text": {
                "type": "mrkdwn",
                "text": "Metric value: {{value}}"
            },
            "fields": [
                {
                    "type": "mrkdwn",
                    "text": "*Tags*\nhost:$HOSTNAME"
                },
                {
                    "type": "mrkdwn",
                    "text": "*Name of the metric*\n$ALERT_METRIC"
                }
            ]
        },
        {
            "type": "image",
            "image_url": "$SNAPSHOT",
            "alt_text": "$ALERT_METRIC: {{value}}"
        }
    ]
}

Zulip Native API - Send Message Endpoint

Notice: due to an issue in Markdown image syntax implementation

it's impossible to put inline image in one message with presence of other content

(only shows and image in link format when one message contains only the link for an image)

URL
https://your_bot_email_address:your_bot_api_key@your_org.zulipchat.com/api/v1/messages

Monitor Notification

Example Payload

{
  "type": "stream",
  "to": "your_stream_name",
  "topic": "Castle",
  "content": "I come not, friends, to steal away your hearts."
}

Message Topic

$ALERT_TITLE

Message Content

[$EVENT_TITLE]($LINK)

$EVENT_MSG

**Alert Status**
$ALERT_STATUS

[$ALERT_METRIC Snapshot]($SNAPSHOT)"

Actual Payload
This is webhook payload content be sent in encoded form

(Check Encode as form when creating Datadog Webhook, Zulip API requires it)

JSON disallows multiline strings so \n is used instead

{
  "type": "stream",
  "to": "your_stream_name",
  "topic": "$ALERT_TITLE",
  "content": "[$EVENT_TITLE]($LINK)\n\n$EVENT_MSG\n\n**Alert Status**\n$ALERT_STATUS\n\n$ALERT_METRIC Snapshot Image:\n$SNAPSHOT"
}

References

TL;DR

It seems using Zulip Native API - Send Message Endpoint is better for Datadog integration

Though the current status of Zulip image handling is frustrating

The general formatting of messages also needs significant improvement to be "nice/comfortable to use"

I will keep on finding a better messaging solution with channel/stream first design

And maybe find another solution or fallbacks to Slack for service related notifications

Cells - Introduction

PikachuEXE — Fri, 12 Mar 2021 07:59:55 +0000

Background

GitHub has recently posted an article about view_component:
https://github.blog/2020-12-15-encapsulating-ruby-on-rails-views/
Before it gets too popular I think I should share my experience with cells

So that developers can have another chance to re-think and pick what to use for "encapsulated view components".

Scope of this post

(1) Only "concept cells", not view_component style cells.
The "default"/view_component naming style makes it difficult to just copy & rename a folder to bootstrap a new view component

(2) File Structure, Inputs, Rendering, View Inheritance & Caching
Testing - I am too lazy to do it (rarely beneficial for me)
Layout - Never used it.

File Structure

(Modified from cells old README)
(Remember this is for concept cells)

app
├── concepts
│   ├── copyable_url_box
│   │   ├── cell.rb
│   │   ├── views
│   │   │   ├── show.haml

File content

class CopyableUrlBox::Cell < Cell::Concept
  # ..
end

If you want to put these files into a non-default folder like app/resources
You need to update the view_paths

class CopyableUrlBox::Cell < Cell::Concept
  self.view_paths = Array("app/resources") + view_paths
  # ..
end

Inputs

The official doc suggests using this:

concept("comment/cell", @comment) #=> return a cell

This gives you access to an attribute model (which is @comment) inside the cell object by default

But I prefer passing a hash so I do this:

class ApplicationConceptCell < Cell::Concept
  module Cells3HashInputBehaviourReplication
    # Since in cells 3.x
    # When a hash is passed in
    # Everything in hash is injected as ivars
    # So we want to replicate the behaviour here
    #
    # This override works on cells 4.1.5
    def initialize(model, options={})
      if model.is_a?(::Hash)
        model.each_pair do |key, value|
          instance_variable_set("@#{key}", value)
        end

        # To avoid ivar `@model` being set in super with the Hash
        # We need to fetch value from hash and put it into `super`
        # Even usually it's `nil`
        super(model.fetch(:model) { nil }, options)
      else
        super
      end
    end
  end
  prepend Cells3HashInputBehaviourReplication
end

# Remember you have to inherit from the right cell!
class CopyableUrlBox::Cell < ApplicationConceptCell
  private

  # region Inputs

  # Contract Something
  # attr_reader :input_name

  Contract And[::String, Send[:present?]]
  attr_reader :url

  # endregion Inputs
end

This gives me access to input values as long as I defined them via attr_reader.
Oh what's the Contract XXX above attr_reader?
They are from contracts.ruby and completely optional and won't be explained in this post.
You can safely ignore those and maybe study that gem later.

I am also using another way to handle inputs which is to put input values into inputs attribute via a value object.

require "contracted_value"

class ApplicationConceptCell2222 < Cell::Concept
  module HashInputsAsImmutableInputs
    def initialize(model, options={})
      if model.is_a?(::Hash)
        @inputs = self.class.inputs_class.new(model)

        # To avoid ivar `@model` being set in super with the Hash
        # We need to fetch value from hash and put it into `super`
        # Even usually it's `nil`
        super(model.fetch(:model) { nil }, options)
      else
        super
      end
    end

    private

    attr_reader :inputs
  end
  prepend HashInputsAsImmutableInputs
  class_attribute(
    :inputs_class,
    instance_reader: false,
    instance_writer: false,
  )
  class << self
    def define_inputs(&block)
      new_inputs_class = Class.new(::ContractedValue::Value)
      new_inputs_class.include(::Contracts::Core)
      new_inputs_class.include(::Contracts::Builtin)
      new_inputs_class.class_eval(&block)

      self.inputs_class = new_inputs_class
    end
  end
end

# Remember you have to inherit from the right cell!
class CopyableUrlBox2222::Cell < ApplicationConceptCell2222
  private

  # region Inputs

  define_inputs do
    attribute(
      :url,
      contract: And[::String, Send[:present?]],
    )
  end

  # endregion Inputs
end

Above implemented via my own gem contracted_value and feel free to swap it with any other value object implementation which can be gems or your own code.

Rendering

To use a cell, by default just invoke #call

And you can do it by:

Get result string directly
Create a cell, do something in the middle, get result string later (no need to invoke #call right after cell creation, and feel free to add new public methods as your own API)

# controller
render(
  html: concept("copyable_url_box/cell, url: "...").call
)

Since I use cells-rails, #call returns HTML safe string.
If you don't use cells-rails you might need to do extra work or include extra module to make that happen (check doc yourself).

In cell class, it looks like this:

class CopyableUrlBox::Cell < ApplicationConceptCell
  # This is actually optional if you use >= 4.1
  # BUT you can make this return something else
  def show
    render
  end

  # inputs and other stuff...
end

Why #show? If you read the doc you will know that
concept_cell.call == concept_cell.call(:show)
You are free to use other names but not recommended.

The templates can access anything inside the cell.
Yes this includes

the inputs (yes even they are private attributes)
any public/protected/private methods
Some helper methods are delegated to controller if cells-rails is used
Extra helper methods from helper modules included into the cell

No more "locals" when rendering. What you see (in cell objects) is what you get (in templates).

Having template file(s) is optional.
Returning a tag directly in #show also works.

class WebsiteLogo::Cell < ApplicationConceptCell
  def show
    image_tag(
      website_logo_image_path,
      alt: "Logo",
      class: "...",
    )
  end

  private

  def website_logo_image_path
    case inputs.variant
    when :small
      "..."
    else
      "..."
    end
  end

  # inputs and other stuff...
end

View Inheritance

I generally prefer composition over inheritance (using a group of other cells instead a cell) but sometimes being able to use templates from parent cell class is more convenient.
I use it when I have several cells for page content of several page types but the layout is similar (same number of sections but different content).
Just read the view inheritance document yourself.

Caching

Best part comes last.
Caching is easy, cache invalidation is not.
But let's talk about how to enable caching first.

Enabling caching is quite easy according to caching doc:

class CopyableUrlBox::Cell < ApplicationConceptCell
  cache :show
end

Yup. The end.
Except when you pass different inputs into it, the cell will give you the same (cached) output. Which is why cache invalidation is the hard part.

First get some basic concept by looking at the caching doc

cache :show, expires_in: 10.minutes

This is just making the cached content expires after some time, nope not what we want.

cache :show { |model, options| "comment/#{model.id}/#{model.updated_at}" }
cache :show, :if => lambda { |*| has_changed? }
cache :show, :tags: lambda { |model, options| "comment-#{model.id}" }

Looks messy? Let me give you my version

cache(
  :show,
  :cache_key,
  expires_in: :cache_valid_time_period_length,
  if: :can_be_cached?,
)

def can_be_cached?
  # Use following code when child cell(s) is/are used
  # [
  #   child_cell_1,
  # ].all?(&:can_be_cached?)

  true
end

def cache_valid_time_period_length
  # Use following code when child cell(s) is/are used
  # [
  #   child_cell_1,
  # ].map(&:cache_valid_time_period_length).min

  # Long time
  100.years
end

def cache_key
  {
    current_locale: ::I18n.config.locale,

    url: Digest::MD5.hexdigest(url),
  }
end

If language changed (::I18n.config.locale) or input value different (url), a different cache key would be used and the render result (cached or not) would be different

But hold on, what if I updated the logic/template and now I want it to return a result rendered with latest logic?

This is what I did:

class ApplicationConceptCell < Cell::Concept
  DEFAULT_CLASS_LEVEL_CACHE_KEY = {
    # ONLY change this when there are too many cells caching needs updated
    # Also only affects cells that has `#cache_key` binding to `super`

    # Yes I use cells since 2015
    design: :v2015_12_22_1001,
  }.freeze
  private_constant :DEFAULT_CLASS_LEVEL_CACHE_KEY

  def self.cache_key
    DEFAULT_CLASS_LEVEL_CACHE_KEY
  end

  def cache_key
    self.class.cache_key
  end
end

class CopyableUrlBox::Cell < ApplicationConceptCell
  cache(
    :show,
    :cache_key,
    expires_in: :cache_valid_time_period_length,
    if: :can_be_cached?,
  )

  def can_be_cached?
    # Use following code when child cell(s) is/are used
    # [
    #   child_cell_1,
    # ].all?(&:can_be_cached?)

    true
  end

  def cache_valid_time_period_length
    # Use following code when child cell(s) is/are used
    # [
    #   child_cell_1,
    # ].map(&:cache_valid_time_period_length).min

    # Long time
    100.years
  end

  def self.cache_key
    super.merge(
      logic: :v2020_12_18_1727,
    )
  end

  def cache_key
    super.merge(
      current_locale: ::I18n.config.locale,

      url: Digest::MD5.hexdigest(url),
    )
  end
end

Notice that the above code all use cache_key.

But Rails 5.2 introduced cache versioning and we should use that too.

Caching - With Cache Versioning

The following code enable cache versioning

class ApplicationConceptCell < Cell::Concept
  DEFAULT_CLASS_LEVEL_CACHE_KEY = {
    # Empty
  }.freeze
  private_constant :DEFAULT_CLASS_LEVEL_CACHE_KEY

  DEFAULT_CLASS_LEVEL_CACHE_VERSION = {
    # ONLY change this when there are too many cells caching needs updated
    # Also only affects cells that has `#cache_key` binding to `super`

    # Yes I use cells since 2015
    design: :v2015_12_22_1001,
  }.freeze
  private_constant :DEFAULT_CLASS_LEVEL_CACHE_VERSION

  def self.cache_key
    DEFAULT_CLASS_LEVEL_CACHE_KEY
  end

  def cache_key
    self.class.cache_key
  end

  def self.cache_version
    DEFAULT_CLASS_LEVEL_CACHE_VERSION
  end

  def cache_version
    self.class.cache_key
  end
end

class ProductCardWide::Cell < ApplicationConceptCell
  cache(
    :show,
    :cache_key,
    expires_in: :cache_valid_time_period_length,
    if: :can_be_cached?,
    version: :cache_version,
  )

  def can_be_cached?
    # Use following code when child cell(s) is/are used
    # [
    #   child_cell_1,
    # ].all?(&:can_be_cached?)

    true
  end

  def cache_valid_time_period_length
    # Use following code when child cell(s) is/are used
    # [
    #   child_cell_1,
    # ].map(&:cache_valid_time_period_length).min

    # Long time
    100.years
  end

  # def self.cache_key
  #   super
  # end

  def cache_key
    super.merge(
      current_locale: ::I18n.config.locale,

      # Assume product is an active record object 
      product: product.id,
    )
  end

  def self.cache_version
    super.merge(
      logic:    :v2017_12_22_1124,
    )
  end

  def cache_version
    super.merge(
      # Assume product is an active record object 
      product: product.updated_at,
    )
  end
end

Point is when calling .cache in class, add option version.

Logic/Template/Translation/Images updated?

You can of course manually update the class level cache key/version.

But let's see how I make them auto update themselves.

Caching - Class Level Cache Key/Version Auto Update

1. Template Updates
This assumes every template update should cause cache invalidation (so even a refactor would invalidate the cache).

class Some::Cell < ApplicationConceptCell
  def self.cache_version
    super.merge(
      template: TEMPLATE_FILES_CONTENT_CACHE_KEY,
    )
  end

  # This generates the cache key/version from the content of all direct templates for this cell (templates from parent cells or custom paths NOT included)
  # Any change (even a refactor) would cause the value to change
  TEMPLATE_FILES_CONTENT_CACHE_KEY = begin
    view_folder_path = File.expand_path("views", __dir__)
    file_paths = Dir.glob(File.join(view_folder_path, "**", "*"))

    file_digests = file_paths.map do |file_path|
      ::Digest::MD5.hexdigest(File.read(file_path))
    end

    ::Digest::MD5.hexdigest(file_digests.join(""))
  end
  private_constant :TEMPLATE_FILES_CONTENT_CACHE_KEY
end

2. Translation Updates
This requires you to have separate folders of translation files for different components.

class Some::Cell < ApplicationConceptCell
  def self.cache_version
    super.merge(
      translations: TRANSLATION_FILES_CONTENT_CACHE_KEY,
    )
  end

  # This generates the cache key/version from the content of all specified folder's files
  # Any change (even a refactor) would cause the value to change
  TRANSLATION_FILES_CONTENT_CACHE_KEY = begin
    folder_path = ::Rails.root.join(
      "config/locales/your/cell/translation/folder/path",
    )
    file_paths = Dir.glob(File.join(folder_path, "**", "*"))

    file_digests = file_paths.map do |file_path|
      next nil unless File.file?(file_path)

      ::Digest::MD5.hexdigest(File.read(file_path))
    end

    ::Digest::MD5.hexdigest(file_digests.join(""))
  end
  private_constant :TRANSLATION_FILES_CONTENT_CACHE_KEY
end

3. Image Updates

class Some::Cell < ApplicationConceptCell
  def self.cache_version
    super.merge(
      images: IMAGE_FILES_CONTENT_CACHE_KEY,
    )
  end

  # This generates the cache key/version from the content of all specified folder's files
  # Any change (even a refactor) would cause the value to change
  IMAGE_FILES_CONTENT_CACHE_KEY = begin
    folder_paths = [
      # "app/assets/images/path/to/images",
    ]
    asset_logical_paths = [
      # "app/assets/images/path/to/images.jpg",
    ].map do |asset_logical_path|
      ::Rails.root.join(asset_logical_path)
    end
    file_paths = folder_paths.inject([]) do |paths, folder_path|
      paths + ::Dir.glob(::File.join(::Rails.root.join(folder_path), "**", "*"))
    end + asset_logical_paths

    file_digests = file_paths.map do |file_path|
      next nil unless File.file?(file_path)

      ::Digest::MD5.hexdigest(File.read(file_path))
    end

    ::Digest::MD5.hexdigest(file_digests.join(""))
  end
  private_constant :ASSET_FILES_CONTENT_CACHE_KEY
end

4. Cell Logic Updates
Nope. You have to update cache key/version manually for this kind of updates.

The End

I still got some more to share about caching but let's save it for another post. (This post is for "beginners")

The code above is extracted from an existing project so let me know if there is any mistake.

Took me almost 3 months to finish this article (o.0)

Ruby Gems CI using GitHub Action

PikachuEXE — Fri, 20 Nov 2020 06:07:40 +0000

Travis CI is probably "gone" for Open Source Projects.

And for small ruby gem projects I think GitHub Action is enough.

Some other articles if you want to read more about Travis CI:

https://www.reddit.com/r/ruby/comments/jt2uub/deep_dive_moving_ruby_projects_from_travis_to/

GitHub Action Official Doc:
https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions

TL;DR

Wait shouldn't that be put at the end?
Anyway here is my version of workflow.

For most basic testing like rspec,

Create .github/workflows/tests.yaml
Taken from One of my own gem's GitHub Action config

name: Tests

on:
  pull_request:
    branches:
      - master
    paths-ignore:
      - 'README.md'
  push:
    branches:
      - master
    paths-ignore:
      - 'README.md'


jobs:
  unit_tests:
    name: Unit Tests
    if: "contains(github.event.commits[0].message, '[ci skip]') == false"
    strategy:
      fail-fast: false
      matrix:
        os:
          - ubuntu
        ruby:
          - 2.4
          - 2.5
          - 2.6
          - 2.7
        gemfile:
          - gemfiles/contracts_15_0.gemfile
          - gemfiles/contracts_16_0.gemfile
        allow_failures:
          - false
        include:
          - os: ubuntu
            ruby: ruby-head
            gemfile: gemfiles/contracts_16_0.gemfile
            allow_failures: true
    env:
      BUNDLE_GEMFILE: "${{ matrix.gemfile }}"
      ALLOW_FAILURES: "${{ matrix.allow_failures }}"
    runs-on: ${{ matrix.os }}-latest
    continue-on-error: ${{ endsWith(matrix.ruby, 'head') || matrix.ruby == 'debug' }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: ${{ matrix.ruby }}
          bundler-cache: true
      - name: Test
        run: bundle exec rake spec || $ALLOW_FAILURES

GitHub Action Config with comment

OK You got some more time to read
Let me post one with comments inside

# Just the workflow name, you will see this on project's action tab
# Do give it a good name and stick to it
name: Tests

# When do you want this workflow to run
# https://docs.github.com/en/free-pro-team@latest/actions/reference/workflow-syntax-for-github-actions#on
on:
  # On pull request
  pull_request:
    # For some branches only (base branch)
    branches:
      - master
    # Don't run if changes only include these files
    paths-ignore:
      - 'README.md'
  # On push
  push:
    # For some branches only (base branch)
    branches:
      - master
    # Don't run if changes only include these files
    paths-ignore:
      - 'README.md'

jobs:
  # `unit_tests` is the job id here & can be renamed
  unit_tests:
    name: Unit Tests
    # Emulates Travis skip build on `[ci skip]` in commit message
    # But you can customize this or just remove it
    if: "contains(github.event.commits[0].message, '[ci skip]') == false"
    strategy:
      # Depends on whether you want to see result of all spawned jobs when one failed
      fail-fast: false
      # This is like Travis CI
      # But the keys can be renamed except a few keywords
      matrix:
        os:
          - ubuntu
        ruby:
          - 2.4
          - 2.5
          - 2.6
          - 2.7
        gemfile:
          - gemfiles/contracts_15_0.gemfile
          - gemfiles/contracts_16_0.gemfile
        allow_failures:
          - false
        # Part of a technique to allow some jobs to fail
        include:
          - os: ubuntu
            ruby: ruby-head
            gemfile: gemfiles/contracts_16_0.gemfile
            allow_failures: true
    env:
      # Required for action `ruby/setup-ruby` & other steps with bundler
      BUNDLE_GEMFILE: "${{ matrix.gemfile }}"
      # Part of a technique to allow some jobs to fail
      ALLOW_FAILURES: "${{ matrix.allow_failures }}"
    runs-on: ${{ matrix.os }}-latest
    # Actually unnecessary if we use `allow_failures` technique
    continue-on-error: ${{ endsWith(matrix.ruby, 'head') || matrix.ruby == 'debug' }}
    steps:
      - name: Checkout
        uses: actions/checkout@v2
      - name: Setup Ruby
        # NOT `action/setup-ruby` which is deprecated
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: ${{ matrix.ruby }}
          # With this option enabled
          # Running `bundle install` is not needed since it's included
          bundler-cache: true
      - name: Test
        # `$ALLOW_FAILURES` is part of a technique to allow some jobs to fail
        run: bundle exec rake spec || $ALLOW_FAILURES

Replacement for `allow-failure` on Travis CI

This one is tricky.

Currently (2020-11-20) I am not aware of any complete replacement allow-failure on GitHub Action.

Related:

https://github.com/actions/toolkit/issues/399

A close one would be like the allow_failures I put above.

Any job with allow_failures: true would look passed due to $ALLOW_FAILURES.

However there is no place that shows the test or whatever command run failed unless you look into the log.

But using continue-on-error would make the Check shown as failed (even we allow it to fail). This is not good looking in Pull Request UI & some project even require all Checks to be passed before merging a PR (sometimes auto merging).

This is a design issue so we have to wait.

Replacement of `Cron Jobs` on Travis CI

Nothing special just read the doc

https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#scheduled-events

Fixing coverage report on Coverall

I don't use Coverall on every gem I have (maybe I should)
Use these as examples:

Basically generate the coverage reports using simplecov-lcov and upload via action coverallsapp/github-action@master.

Gotcha on $stdout

GitHub Action's runner (the one that runs all the action/commands) has $stdout.tty? == false

This is the one that took me some time to figure out when migrating gem-release to GitHub Action

[Redis] How To: Remove Redis Keys without TTL (in Ruby)

PikachuEXE — Fri, 30 Oct 2020 03:55:14 +0000

Background

Going to update our caching Redis Server config
to have maxmemory-policy become volatile-ttl
Doc: https://redis.io/topics/lru-cache

Since we use allkeys-lru before and have keys without TTL set (TTL key would return -1)

Those keys are required to be removed or they will never be evicted after config update

Code

::Rails.application.config.cache_store.redis.with do |c|
  cursor = "0"
  count = 0
  # Fetch keys in batches using SCAN to avoid blocking the Redis server.
  begin
    cursor, keys = c.scan(cursor, count: 1000)
    all_key_count = c.dbsize

    puts("cursor: #{cursor}")
    puts("#{count}/#{all_key_count} keys scanned")

    keys_del = keys.each_with_object([]) do |key, keys_to_be_del|
      # Won't expire
      if !key.match?("sprockets|sidekiq|sessions") && c.ttl(key) == -1
        keys_to_be_del.push(key)
        puts("to be deleted: <#{key}>")
      end
    end
    c.del(*keys_del) unless keys_del.empty?

    count += keys.size
    puts("#{count} keys scanned")
  end until cursor == "0"
end

cursor, keys = c.scan(cursor, count: 1000)
SCAN is good, KEYS is bad

!key.match?("sprockets|sidekiq|sessions")
This is due to I am testing my code in development and the Redis server is used for sprockets caching, (rack?) sessions, sidekiq and caching
Feel free to remove it if you want

puts("to be deleted: <#{key}>")
Just another debugging code
I actually commented this code when running on production

c.del(*keys_del) unless keys_del.empty?
Time complexity: O(N)
You can choose to collect all keys then remove the keys in batches

PostgreSQL Logical Replication - For Upgrade

PikachuEXE — Fri, 23 Oct 2020 08:45:39 +0000

2022-05-25 Update

Add statement to drop replication slot to avoid making folder pg_wal becoming too large This happens when the old DB server is still being used after publication is removed

2021-01-07 Update

Add Prevent Replication Error
Update Setup New Server (Sequences) to append actual commands to run

2020-11-24 Update
Moved sequence fixing to the end (after subscription removed)

2020-11-19 Update
Wow I forgot to fix the sequences
Added back a section for it
I realize this after trying a few inserts on my staging environment

Main Article
Just notes for my own experience mostly
Upgrade from 12 to 13

If you don't know what's logical replication read some articles about it first
Official doc: https://www.postgresql.org/docs/12/logical-replication.html

I used to just upgrade PSQL like this:
pg_dump -> pg_restore to another instance -> Point processes (web, worker, etc.) to new DB
Data loss doesn't matter that much in my project but I rather have no data loss

Failed Attempts

If you are too busy or need the solution urgently you can skip this and come back later maybe.

Failed Attempts #1 - Replication after data restored

I used pg_restore to restore data
After subscription started the server will complaint about duplicate data
Although it's possible use copy_data = false but that means some data will be lost and make this replication meaningless

Failed Attempts #2 - Replication after table structure AND indexes restored

Used pg_restore to restore with --schema
After subscription started the server will take forever to copy data from table with many indexes (waited 2 days still not done)

Solution

General Steps

This probably lack details
See Actual Notes on Test Run for actual commands / queries

0.1. Setup new server

Setup new host
Deploy empty PG Server (without cron job that does backup)
Backup schema (only if no latest db structure backup available)
Restore schema & PK without indexes & constraints (section pre-data)

Config might have to be updated

0.2. Setup old server

Config might have to be updated

1. Add Publication

In existing DB server

CREATE PUBLICATION {publication_name}
FOR ALL TABLES

2. Add Subscription

In new DB server

CREATE SUBSCRIPTION {subcription_name}
CONNECTION 'hostaddr={old_db_server_ip} port={old_db_server_port} user={old_db_server_username} password={old_db_server_password} dbname={old_db_name}'
PUBLICATION {publication_name}

3. Wait until synced

https://severalnines.com/database-blog/how-upgrade-postgresql-11-postgresql-12-zero-downtime

4.1. Setup New Server (Indexes)

Restore indexes (section post-data)

4.2. Verify all indexes & constraints are restored

5. Update Users (web/worker process) to use new DB

Check things are working

6.1. Remove Subscription

In new DB server

DROP SUBSCRIPTION IF EXISTS {subcription_name}

6.2. Remove Publication

In old DB server

DROP PUBLICATION IF EXISTS {publication_name}
-- Prevents pg_wal folder getting very large in case old DB is still being used
SELECT pg_drop_replication_slot('{publication_name}');

Actual Notes on Test Run

Setup New Server (Data Structure)
You will see some docker commands since I use Docker container to run PSQL


sudo docker exec -it db.master.1 bash

curl -o backup.dump \
https://bucket.s3.ap-east-1.amazonaws.com/backup/postgresql/pg_dump/manual/2020/10/2020_xx_xx_xxxxxx.dump

PGPASSWORD=pa55w0rd pg_restore --verbose --clean --no-acl --no-owner --if-exists --section=pre-data --no-publications -h localhost -U useruser -d dbdbdbdb --jobs=$(nproc) ./backup.dump


PGPASSWORD=pa55w0rd psql -h localhost -U useruser -d dbdbdbdb

Prevent Replication Error
Errors like
logical replication target relation xxx has neither REPLICA IDENTITY index nor PRIMARY KEY and published relation does not have REPLICA IDENTITY FULL
is caused by not having PK since those are in section post-data to be restored in Setup New Server (Indexes)
Thus we need to restore PK only
Which require custom SQL file created
References:

https://www.2ndquadrant.com/en/blog/upgrading-to-postgresql-11-with-logical-replication/

PGPASSWORD=pa55w0rd \
pg_dump \
-h localhost -U useruser -d dbdbdbdb \
--schema-only --section=post-data > dbdbdbdb_postdata.sql

grep -B 1 'PRIMARY KEY' dbdbdbdb_postdata.sql > dbdbdbdb_pkeys.sql

# Just text, so show content and copy/paste to new DB console
cat dbdbdbdb_pkeys.sql

Add Publication/Subscription

CREATE PUBLICATION publication_test_2020_10_21_1146
FOR ALL TABLES;


CREATE SUBSCRIPTION subcription_test_2020_10_21_1146
CONNECTION 'hostaddr=10.170.0.6 port=5432 user=useruser password=pa55w0rd dbname=dbdbdbdb'
PUBLICATION publication_test_2020_10_21_1146;

Monitor


-- Publisher
SELECT pid, usename, application_name, state
, pg_current_wal_lsn() AS current_lsn
, state
, sync_state
, sent_lsn
, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)) AS sent_diff
, write_lsn
, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), write_lsn)) AS write_diff
, replay_lsn
, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)) AS replay_diff
--, write_lag
--, flush_lag
--, replay_lag
--, backend_start
--, reply_time
FROM pg_stat_replication
ORDER BY application_name, pid;



-- Subcriber
-- # Subscription Workers
SELECT pss.*, c.relname FROM pg_stat_subscription AS pss 
LEFT OUTER JOIN pg_class AS c
ON pss.relid = c.oid;

-- # Subscription Workers on Tables (not ready, srsubstate <> 'r')
SELECT psr.*, c.relname FROM pg_subscription_rel AS psr 
LEFT OUTER JOIN pg_class AS c
ON psr.srrelid = c.oid 
WHERE srsubstate <> 'r';

-- # Subscription Workers on Tables (not ready or synced)
SELECT psr.*, c.relname FROM pg_subscription_rel AS psr 
LEFT OUTER JOIN pg_class AS c
ON psr.srrelid = c.oid 
WHERE srsubstate NOT IN ('r', 's');





-- Monitor Create Index (in case "Broken pipe")
\x on

SELECT pid
--, backend_start
, query_start
, state_change
, wait_event_type
, wait_event
, state
, query
--, backend_type 
FROM pg_stat_activity
WHERE state = 'active';

Monitor network speed / errors

sudo nethogs

sudo docker logs --tail=100 --follow db.master.1

Setup New Server (Indexes)


sudo docker exec -it db.master.1 bash

PGPASSWORD=pa55w0rd pg_restore --verbose --clean --no-acl --no-owner --if-exists --section=post-data --no-publications -h localhost -U useruser -d dbdbdbdb --jobs=$(nproc) ./backup.dump

Verify

-- Index Count
SELECT COUNT(*)
FROM pg_class
WHERE relkind = 'i'
  AND relname NOT LIKE 'pg_%'
  AND relname NOT LIKE 'sql_%';

-- Table Count
SELECT COUNT(*)
FROM pg_class
WHERE relkind = 'r'
  AND relname NOT LIKE 'pg_%'
  AND relname NOT LIKE 'sql_%';

-- Constraint Count
SELECT COUNT(*), contype
FROM pg_constraint
GROUP BY contype;

-- Sequence Count
SELECT COUNT(*), seqtypid
FROM pg_sequence
GROUP BY seqtypid;

Remove Publication/Subscription


-- Safer to drop subscription first
DROP SUBSCRIPTION IF EXISTS subcription_test_2020_10_21_1146;

DROP PUBLICATION IF EXISTS publication_test_2020_10_21_1146;
-- Prevents pg_wal folder getting very large in case old DB is still being used
SELECT pg_drop_replication_slot('publication_test_2020_10_21_1146');

Setup New Server (Sequences)
This will not be restored since we cannot restore data via pg_restore --section=data
This can be done after subscription removed (no more inserts without calling nextval on sequences) and before you start inserting records into any table with a sequence (In Ruby on Rails that's almost every table)

The method I tested:
https://wiki.postgresql.org/wiki/Fixing_Sequences

You might be interested in other answers:

https://stackoverflow.com/questions/244243/how-to-reset-postgres-primary-key-sequence-when-it-falls-out-of-sync

touch reset.sql
tee -a reset.sql <<EOF
SELECT 'SELECT SETVAL(' ||
       quote_literal(quote_ident(PGT.schemaname) || '.' || quote_ident(S.relname)) ||
       ', COALESCE(MAX(' ||quote_ident(C.attname)|| '), 1) ) FROM ' ||
       quote_ident(PGT.schemaname)|| '.'||quote_ident(T.relname)|| ';'
FROM pg_class AS S,
     pg_depend AS D,
     pg_class AS T,
     pg_attribute AS C,
     pg_tables AS PGT
WHERE S.relkind = 'S'
    AND S.oid = D.objid
    AND D.refobjid = T.oid
    AND D.refobjid = C.attrelid
    AND D.refobjsubid = C.attnum
    AND T.relname = PGT.tablename
ORDER BY S.relname;
EOF
# Maybe check the content
# cat reset.sql
PGPASSWORD=pa55w0rd psql -h localhost -U useruser -d dbdbdbdb -Atq -f reset.sql -o temp.sql
# Maybe check the content
# cat temp.sql
PGPASSWORD=pa55w0rd psql -h localhost -U useruser -d dbdbdbdb -f temp.sql
rm temp.sql

The End

If you feel like this looks like a mess
I am sorry but it is mainly for myself
Feel free to write another one that looks more organized ;)

Similar/Related Articles

https://www.cybertec-postgresql.com/en/upgrading-postgres-major-versions-using-logical-replication/

[Sidekiq] How To: Interrupt a running job when Sidekiq worker process is shutting down

PikachuEXE — Tue, 29 Sep 2020 05:25:31 +0000

I have a Sidekiq worker class that run for a duration which can be quite long sometimes.
Jobs and job states are persisted in database records.

class Worker
  include Sidekiq::Worker

  def perform
    prepare_job_record

    do_work_that_might_be_long

    mark_job_completed
  end
end

Sometimes worker processes are restarted due to code release.
The job record has an attribute for job state and it's "stuck" on "started" when the the job is interrupted due to worker process terminated.

1st attempt

def perform
  # work...
rescue Sidekiq::Shutdown
  mark_job_interrupted
end

Failed~

2nd attempt
I read the source code of Sidekiq to see if I can patch it.
But after reading it I think that would be too complex and dangerous, so I didn't even try.
But if you are interested, you can take a look on how it shutdowns the running threads:

Sidekiq::Manager#stop - https://github.com/mperham/sidekiq/blob/v6.1.2/lib/sidekiq/manager.rb#L61
Do some work to stop processors processing new jobs, run callbacks...
Pausing to allow workers to finish... (The waiting time can be controlled and it is described in https://github.com/mperham/sidekiq/wiki/Deployment#overview)
Sidekiq::Processor#kill (https://github.com/mperham/sidekiq/blob/v6.1.2/lib/sidekiq/processor.rb#L49)

The Solution
Let's look at events
https://github.com/mperham/sidekiq/wiki/Deployment#events

On free version this also works but the fetch is delayed (something like 5s? Not sure).
Sidekiq Pro users have super_fetch and the callback will be triggered much quicker it seems.

So I wrote a singleton class in initializer:


module Sidekiq
  # Too lazy to think of better name...
  class LifecycleCenter
    include Singleton

    def initialize
      super
      @quiet = false
    end

    def quiet!
      @quiet = true
    end

    def quiet?
      @quiet
    end
  end
end

Sidekiq.configure_server do |config|
  # Other config

  config.on(:quiet) do
    Sidekiq::LifecycleCenter.instance.quiet!
  end
end

Nothing special. A singleton class instead of a global variable.

What takes time to figure out is the next part: Raise specific error in worker after Sidekiq fired quiet event.

Since I read the Sidekiq source code earlier, I think I should use a thread.
But I think I am a noob in threading and should try to find a gem that does the threading for me, so I found concurrent-ruby.

In JS there is setInterval, in concurrent-ruby there is Concurrent::TimerTask.
Note that any error raised inside the block passed into the task object will not raise any error (handled by the gem).
So this will NOT raise error:

sidekiq_state_checking_task = ::Concurrent::TimerTask.new(
  execution_interval: 1,
) do |task|
  task.shutdown if true
  raise CustomError if true
end
sidekiq_state_checking_task.execute

Feel free to try it in ruby console.

I studied the doc for Concurrent::TimerTask and tried several solutions and finally got this:

class Worker
  include Sidekiq::Worker

  # Not sure if inheriting from `Interrupt` is necessary
  # That's what Sidekiq does on `Sidekiq::Shutdown` though
  class CustomShutdown < Interrupt; end

  def perform
    prepare_job_record

    main_thread = ::Thread.current
    sidekiq_state_checking_task = ::Concurrent::TimerTask.new(
      execution_interval: 1,
    ) do |task|
      ::Sidekiq::LifecycleCenter.instance.quiet?.tap do |quiet|
        # If you don't shutdown the task
        # it might keep running
        task.shutdown if quiet
      end
    end
    sidekiq_state_checking_task.with_observer do |_time, quiet|
      next unless quiet

      main_thread.raise CustomShutdown
    end.execute

    do_work_that_might_be_long

    mark_job_completed

  rescue CustomShutdown => e
    mark_job_aborted
    reenqueue_the_job_sometimes
  end
end

The point is to use observer to handle the result from the block (the task). Remember to shutdown the task or it might keep running (even error raised in main thread).

TL;DR

Put the following code in initializer:


module Sidekiq
  # Too lazy to think of better name...
  class LifecycleCenter
    include Singleton

    def initialize
      super
      @quiet = false
    end

    def quiet!
      @quiet = true
    end

    def quiet?
      @quiet
    end
  end
end

Sidekiq.configure_server do |config|
  # Other config

  config.on(:quiet) do
    Sidekiq::LifecycleCenter.instance.quiet!
  end
end

Put the following code in your worker:

class Worker
  include Sidekiq::Worker

  # Not sure if inheriting from `Interrupt` is necessary
  # That's what Sidekiq does on `Sidekiq::Shutdown` though
  class CustomShutdown < Interrupt; end

  def perform
    prepare_job_record

    main_thread = ::Thread.current
    sidekiq_state_checking_task = ::Concurrent::TimerTask.new(
      execution_interval: 1,
    ) do |task|
      ::Sidekiq::LifecycleCenter.instance.quiet?.tap do |quiet|
        # If you don't shutdown the task
        # it might keep running
        task.shutdown if quiet
      end
    end
    sidekiq_state_checking_task.with_observer do |_time, quiet|
      next unless quiet

      main_thread.raise CustomShutdown
    end.execute

    do_work_that_might_be_long

    mark_job_completed

  rescue CustomShutdown => e
    mark_job_aborted
    reenqueue_the_job_sometimes
  end
end

Feel free to let me know what you think about this method.
Or write an article about your method and I will be happy to add a link to your article!

DEV Community: PikachuEXE

How to: Use SourceTree, with git hook via lefthook, running bundle exec with asdf managing Ruby versions

Background

Lefthook Setup

Issue

Investigation

Solution

Zulip Datadog Integration

Zulip Datadog Integration

Slack Compatible Incoming Webhook

Monitor Notification

Zulip Native API - Send Message Endpoint

Monitor Notification

Message Topic

Message Content

TL;DR

Cells - Introduction

Background

Scope of this post

File Structure

Inputs

Rendering

View Inheritance

Caching

Caching - With Cache Versioning

Caching - Class Level Cache Key/Version Auto Update

The End

Ruby Gems CI using GitHub Action

TL;DR

GitHub Action Config with comment

Replacement for allow-failure on Travis CI

Replacement of Cron Jobs on Travis CI

Fixing coverage report on Coverall

Gotcha on $stdout

[Redis] How To: Remove Redis Keys without TTL (in Ruby)

Background

Code

PostgreSQL Logical Replication - For Upgrade

Failed Attempts

Failed Attempts #1 - Replication after data restored

Failed Attempts #2 - Replication after table structure AND indexes restored

Solution

General Steps

0.1. Setup new server

0.2. Setup old server

1. Add Publication

2. Add Subscription

3. Wait until synced

4.1. Setup New Server (Indexes)

4.2. Verify all indexes & constraints are restored

5. Update Users (web/worker process) to use new DB

6.1. Remove Subscription

6.2. Remove Publication

Actual Notes on Test Run

The End

Similar/Related Articles

[Sidekiq] How To: Interrupt a running job when Sidekiq worker process is shutting down

TL;DR

Replacement for `allow-failure` on Travis CI

Replacement of `Cron Jobs` on Travis CI