DEV Community: mmyoji

How to use different MySQL database per pull-request with Heroku Review Apps (Rails)

mmyoji — Thu, 02 May 2019 05:46:36 +0000

Prerequisites

Rails v5.2.x
Ruby v2.6.x
Heroku
MySQL (ClearDB)

Summary

Use CLEARDB_<colorname>_URL for each Heroku Review Apps

The sample code is here:

mmyoji / hreviewapp

sample Rails app running on Heroku with ClearDB(MySQL)

Sample Rails app running on Heroku

Rails v5.2.3
Ruby v2.6.3
MySQL (ClearDB)
Heroku

Heroku setup

Create an app on Heroku
Connect GitHub and repository
Add ClearDB Addons
Add DATABASE_URL environment variable

See config/database.yml
Copy CLEARDB_DATABASE_URL, but replace the first mysql:// with mysql2://
Reference: https://devcenter.heroku.com/articles/cleardb

Deploy manually
Run database migration and etc. w/ heroku CLI
- e.g.) $ heroku run rails db:create -a <app_name>

Heroku Review Apps

Create a pipiline (if you don't have it)
Follow the instruction: https://devcenter.heroku.com/articles/github-integration-review-apps
If you wanna use different database per pull-request, see this commit
- ClearDB issues unique database URL per review apps like CLEARDB_PURPLE_URL or CLEARDB_NAVY_URL
- If it exists, use the URL for the review apps.

In PostgreSQL, I don't know whether this kind of database URL is issued or not.

View on GitHub

Background

Some of my developer teams want to have a different app environment through their development process. Some use their own staging environment, others don't have that and just use local environments.

Heroku Review Apps is a very good choice for that situation, but I didn't know whether it can use different database per pull-request. You might know Rails app development often has database migrations and separating database per pull-request makes developers happier :)

If you use MySQL (sorry I didn't research PostgreSQL case), YOU CAN DO IT and I explain this in this post.

Setup

I don't talk about setup the heroku app, creating a pipeline and setup review apps here.

They're not difficult, just talk about ClearDB useful feature.

CLEARDB_XXX_URL

I found that ClearDB issues unique database URL per review app.

It is like CLEARDB_NAVY_URL, CLEARDB_PURPLE_URL, etc.

And I thought I could use this if it is set in the environment and the dirty script is here:

https://github.com/mmyoji/hreviewapp/blob/master/config/database.yml#L54

production:
  # ...
  url: <%= ((key = ENV.keys.find { |k| k =~ /^CLEARDB_([A-Z]+)_URL$/ && k != "CLEARDB_DATABASE_URL" }) && ENV[key].sub("mysql://", "mysql2://")) || ENV.fetch('DATABASE_URL') %>

This code sucks, but you know what is done here.

If CLEARDB_XXX_URL exists, replace string in proper format, or use DATABASE_URL like as a normal app.

Don't forget to run rails db:schema:load or db:migrate after review app deployment.

see: https://github.com/mmyoji/hreviewapp/blob/master/app.json#L44

That's it 👋

Video Processing with AWS Lambda + Elastic Transcoder in Golang

mmyoji — Wed, 07 Mar 2018 10:11:46 +0000

I'm posting this because I could not find any blog posts with sample code for video processing with AWS Lambda + AWS Elastic Transcoder in Golang.

Dependencies

Golang v1.10
aws/aws-lambda-go v1.1.0
aws/aws-sdk-go v1.13.8

Premises

You know AWS Lambda and Elastic Transcoder roughly
You've already have a Elastic Transcoder Pipeline
You know Golang a little and like it :)

Flow of the operations

Upload a video file in a S3 bucket
Lambda is trigger by 1. and create a Elastic Transcoder Job

That's it.

Implementation

Create a Lambda function

Runtime: Go 1.x
Set an appropriate Role (Elastic Transcoder access is required)
Set a handler name as the same with the Go binary after go build
Set a trigger as S3 and choose the Pipeline's input bucket name and event type ObjectCreated
Set your PIPELINE_ID environment variable

Go code

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    transcoder "github.com/aws/aws-sdk-go/service/elastictranscoder"
)

var (
    // Generic 720p
    presetID = "1351620000001-000010"
)

func main() {
    lambda.Start(handler)
}

func handler(ctx context.Context, s3Event events.S3Event) {
    for _, record := range s3Event.Records {
        s3 := record.S3
        bucket := s3.Bucket.Name
        objKey := s3.Object.Key
        log.Printf("[%s - %s] Bucket = %s, Key = %s \n", record.EventSource, record.EventTime, bucket, objKey)

        sess := session.Must(
            session.NewSession(
                &aws.Config{
                    // Set supported region
                    Region: aws.String("ap-northeast-1"),
                },
            ),
        )

        svc := transcoder.New(sess)

        resp, err := svc.CreateJob(
            &transcoder.CreateJobInput{
                Input: &transcoder.JobInput{
                    Key: aws.String(objKey),
                },
                Outputs: []*transcoder.CreateJobOutput{
                    &transcoder.CreateJobOutput{
                        PresetId:         aws.String(presetID),
                        Key:              aws.String(objKey),
                        // If the original filename is "test-file.mp4",
                        // this generates "test-file.mp4-00001.png"
                        ThumbnailPattern: aws.String(fmt.Sprintf("%s-{count}", objKey)),
                    },
                },
                PipelineId: aws.String(os.Getenv("PIPELINE_ID")),
            },
        )
        if err != nil {
            log.Printf("Failed: Create Job, %v\n", err)
            return
        }

        log.Printf("Job Response: %v\n", resp.Job)
    }
}

The structs options are at minimum, but you can set CreateJobOutput.ThumbnailPattern as "" if you don't want to generate thumbnails.

ThumbnailPattern must contain {count}.

aws.String(str) is the same with &str, you can use this as a shorthand.

strPtr := aws.String("somestring")

// the same with above, but you need to set a variable first.
str := "somestring"
strPtr := &str

References

touch triggers other touches

mmyoji — Mon, 05 Mar 2018 02:39:56 +0000

ActiveRecord::Persistence#touch is a useful method and I like it.

But I didn't really understand its spec well and it caused a deadlock in our Rails project. You have to care about its impact.

class Post < ApplicationRecord
  belongs_to :post_category, touch: true
  belongs_to :writer, touch: true
end

# This class is a pseudo class
class SomeClass
  def initialize(post, args)
    @post = post
    # ...
  end

  def call
    # ...
    @post&.touch(:published_at)
    # triggers following:
    # @post.post_category.touch
    # @post.writer.touch
  end
end

In SomeClass#call, Post#touch is called and I thought this only update the post.published_at, but also update post_category.updated_at and writer.updated_at.

The above code is just an example and it might be a trouble when you use touch casually in the real application.

You just care about logs before deployment :)

Is it possible to use Windows Subsystem for Linux in web development?

mmyoji — Sat, 13 Jan 2018 06:54:12 +0000

I got a Windows 10 Home PC for gaming(PUBG) and tried Windows Subsystem for Linux (a.k.a. Bash on Ubuntu on Windows) in web development.

Conclusion

You can use WSL for light web development.

For example,

web frontend development (using webpack)
small web app (use Sinatra)
etc.

It's NOT suitable for heavy or modern development environment.

At least, Windows10 Home edition can't use Docker in easy way.

I've never tried Windows10 Pro + Docker for Windows and I read some articles about it, it didn't seem to be easy.

Languages

I installed and tried following languages:

Node.js
Ruby
Golang

Setup Windows Subsystem for Linux

It's not difficult at all! I skip the instruction.

Japanese

Windows Subsystem for Linuxをインストールしてみよう！ - Qiita

English

Install the Linux Subsystem on Windows 10 | Microsoft Docs

Editor - VSCode

I'm using NeoVim in my current main dev environment on Xubuntu.

But I discard NeoVim at this time and chose VSCode.

It's easy to install and configure, and I wanted to use it a little.

You just need to care about the terminal and git path setting.

Visual Studio CodeでWSLのgitを利用してソース管理を行う - Qiita

Install the latest version of wslgit.exe from here and put it wherever you like. (I put it under /mnt/c/Users/mmyoji/)

The VSCode setting file are like following:

{
  "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\bash.exe",
  "git.path": "C:\\Users\\<username>\\wslgit.exe"
}

Replace the <username> of your name like mmyoji.

Install plugins you like.

However, Golang integration with VSCode not worked when you install Gplang under WSL. If you want to use VSCode with Golang, it might be better to install to Windows itself.

Setup node

The versions of node and npm via apt is too old and the installation process was a bit bothering.

Ubuntuに最新のNode.jsを難なくインストールする - Qiita

$ sudo apt install -y nodejs-legacy npm
$ sudo npm i -g n

# Install latest node
$ sudo n stable
$ sudo ln -sf /usr/local/bin/node /usr/bin/node

$ node -v
v9.2.1
$ npm -v
5.5.1

$ sudo apt purge -y nodejs-legacy npm

Setup Terminal - Hyper

I cannot stay calm with Command Prompt nor PowerShell and took Hyper.

Get Hyper from the official site.
Install plugins if you want.

The config file was located under /mnt/c/Users/mmyoji in my environment.

This is an example:

// .hyper.js
{
  // ....
  plugins: [
    'hyper-dracula' // theme plugin
  ]
}

And select View -> Full Reload from hyper options.

It's easy to customize and I like it.

Setup Ruby environment

Worked normal rbenv / ruby-build setup instructions with git
Install $HOME/.rbenv and add some script in $HOME/.bashrc (not $HOME/.bash_profile)
One thing:
- before running rbenv install 2.5.0, you have to install readline
- $ sudo apt install -y libreadline-dev

Setup Golang

You can do the same setup operation like Linux.

But WSL+VSCode integration is not supported well.

Setup Docker

If you have to handle dockerized applications, you have to use:

Windows 10 Pro & Docker for Windows (and some customize)
Docker Toolbox(VirtualBox)
macOS
Linux

There might be a good solution, I couldn't find it for now.

Conclusion, again

It's a bit difficult to use Windows10 Home for modern web development.

Linux is better for it.

But you can use Windows10 Home as light dev environment as I said above.

Thanks.