I've created a tiny library connecting Rails and Digdag

Kent Fujii — Sun, 08 May 2022 09:34:18 +0000

What's this?

I've created a tiny library connecting Rails and Digdag.
The library "mogura" is now available.

What is this used for?

Rake tasks are a way to manage batches in Rails.
However, batch processing tends to bloat the codebase and increase operational costs.

In some cases, introducing a professional Python ETL tool such as Airflow may be a hurdle with learning/building/operating/expenses or recruiting human resources.
In addition, it is often not a realistic option to rewrite the Rails-related domain logic contained in Rake tasks as microservices in Python.
There may be a demand to include a workflow engine within a Rails application.

I have been looking for a way to use the domain logic of a Rails application while still satisfying the requirements of a workflow engine.
And I have come up with the option of creating a tiny library connecting Rails and Digdag.

Usage

Add the following to your Rails project Gemfile and bundle install

gem 'mogura'

Once the library is installed, try the following command in the root directory of your Rails project

$ bundle exec mogura help
Commands:
  mogura help [COMMAND] # Describe available commands or one specific command
  mogura init # Initialize Digdag files
  mogura push # Push Digdag workflows
  mogura version # Prints version

bundle exec mogura init will create the following files in your Rails project

RAILS_ROOT/config/digdag/sample.dig
RAILS_ROOT/app/dags/sample_dag.rb

_export:
  rb:
    require: {{ RAILS_ROOT }}/config/environment
+run:
  rb>: SampleDag.run

class SampleDag
  def run
    puts "Hello Rails #{Rails.env}"
  end
end

Commanding bundle exec mogura push will register sample.dig with Digdag.
By loading the Rails environment from Digdag inside sample.dig, you can call your domain logic of the Rails from your workflow.

Thanks for reading!

Currently, only a single Digdag project can be registered as workflows, but I think it would be better if multiple Digdag projects could be registered.
Also, this library currently allows Digdag operations only from the CLI, but I would like to be able to register and edit workflows from your domain logic of the Rails application.
And I'm actively looking for feedback or contributions to this library, feel free to have new functionality ideas to add! :)

Release Checklist

Kent Fujii — Tue, 21 Apr 2020 15:57:20 +0000

A checklist example to check before releasing a web service

Application

Test coverage

[ ] Is the test coverage 60% or higher?
- In general, 60% is “acceptable”

Log

[ ] Is there a log in CloudWatch Logs?

External monitoring

[ ] Is the stability of top page monitored by StatusCake?
- Polling at least every 15 minutes

Vulnerability

[ ] Vulnerability diagnosis for externally released services.
- Diagnostics by AWS inspector, IBM APP SCAN, etc. have been performed and the vulnerability has been fixed by the day of release

Load test

[ ] Has a load test been performed?

Infrastructure

Log (ex. CloudWatch)

[ ] Have you set how many days to retain logs?

Load Balancer (ex. ALB)

[ ] Is the response time monitored in NewRelic?
[ ] Is the load balancer 5xx monitored in NewRelic?

App/Batch Server (ex. EC2)

[ ] NewRelic's APMs are installed in your project.
[ ] CPU/memory/disk is monitored in NewRelic

Container Orchestrator (ex. ECS)

[ ] Check the hard limit of memory set in the task definition of ECS not to exceed the upper limit of the ECS instance memory.

Database (ex. RDS, ElastiCache)

[ ] RDS CPU/memory/disk is monitored in NewRelic
[ ] ElastiCache CPU/memory is monitored in NewRelic
[ ] Are you sure it's not provided to the public?

Network (ex. VPC, SG, WAF, S3)

[ ] VPC's IP range is not the same as your other AWS VPCs
[ ] Is there a clear separation between the Public and Private segments?
[ ] If it is an internal service, is SG's ingress limited to your organization?
[ ] Set WAF for services that are released to the public.
[ ] Are all S3 buckets isolated from the outside world?

DevOps

[ ] Are you able to run build/test/deploy with CircleCI?
[ ] Are you able to manage your cloud resources with Terraform?
[ ] Is there an on-call system like Opsgenie?
- Be able to receive the following error notifications by phone or other means and be ready to write a playbook/post-mortem after recovery
  - external monitoring
  - 5xx of load balancers
  - Disk space remaining in EC2
  - Disk Space in RDS
  - Amount of memory remaining in ElastiCache

DEV Community: Kent Fujii