DEV Community: Maful

Build Load More Pagination with Pagy and Rails Hotwire

Maful — Sun, 17 Sep 2023 14:19:19 +0000

Introduction

Hi folks! Welcome to another Hotwire tutorial. A few days ago, I shared a short video on Twitter from WrappedBy Dashboard and I thought it could be really cool to write a blog post to guide you through building an application with a similar concept. If you haven't seen the tweet yet, take a moment to check out what it looks like.

// Detect dark theme var iframe = document.getElementById('tweet-1702723310344347816-836'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1702723310344347816&theme=dark" }

Exciting, isn't it? In this post, we're going to explore how to implement a 'Load More' pagination feature using Pagy and Hotwire. What makes this even more fascinating is that we'll accomplish it without writing a single line of JavaScript. Instead, we'll harness the power of Turbo Streams to seamlessly update our page.

According to the official Hotwire documentation, Turbo Streams deliver page changes over WebSocket, SSE or in response to form submissions using just HTML and a set of CRUD-like actions. You can dive deeper into Turbo Streams in the Come Alive with Turbo Streams handbook.

Now, let's get started by creating our Rails application.

Create Application

As usual, we won't start everything from scratch. We'll focus on the core concept of 'Load More' pagination. To do that, we'll begin by creating a Rails application with the help of the UpperBracket template. If you're already using UpperBracket, you should be all set to follow along with this tutorial.

To kick things off, open your terminal and run the following command:

rails new hotwire-load-more-pagination \
  -d postgresql \
  -m https://raw.githubusercontent.com/maful/upperbracket/main/template.rb

Make sure you have your PostgreSQL database up and running.

Our goal for this tutorial is not to build a fancy, feature-rich application. Instead, we'll focus on displaying a list of comments and adding a pagination feature to it.

Create Comment model including views, controller, migrations etc. Specify the message and the author name in the Comment model

rails g scaffold Comment message author_name

Open the routes and change the content like this config/routes.rb

# frozen_string_literal: true

Rails.application.routes.draw do
  resources :comments
  root to: "comments#index"
end

Run the database migration to apply the changes

rails db:migrate

Now, let's generate some data for our Comment model by using the database seed feature. Open up the db/seeds.rb file in your Rails application directory and replace its contents with the following code:

names = [ "Maija", "Aniyah", "Artūras", "Leocadia", "Aikorkem", "Maxime", "Eemeli", "Rahmatullah", "Indrek", "Alfredo", "Villads", "Aelius", "Sofia", "Maor"]
comments = [
  "You've got this! Keep up the great work!",
  "Your smile can brighten anyone's day.",
  "You make the world a better place just by being you.",
  "Believe in yourself, and you can achieve anything!",
  "Sending you a virtual hug!",
  "Every day is a new opportunity to shine!",
  "Your kindness is like a ripple that spreads positivity.",
  "Keep your face always toward the sunshine, and the shadows will fall behind you.",
  "You're a true inspiration to those around you.",
  "Life is full of beautiful moments, and you're one of them.",
  "Your perseverance and hard work are paying off!",
  "Stay positive, and amazing things will happen.",
  "Your energy is contagious—in the best way!",
  "The world is a better place with you in it.",
  "You're a ray of sunshine on a cloudy day.",
  "Success is yours because you work for it!",
  "Dream big, and don't be afraid to chase those dreams.",
  "You're making progress, one step at a time.",
  "Don't forget to take care of yourself; you deserve it!",
  "Surround yourself with positivity, and watch your life change for the better."
]

100.times do
  Comment.create(message: comments.sample, author_name: names.sample)
end

In this code snippet:

We create two arrays, names and comments, to generate random data for our comments. This data is purely for testing purposes.
Using a loop (100.times), we create a total of 100 comment records by selecting random entries from the comments and names arrays.

To execute the database seed and populate your Comment model, simply run rails db:seed. After running the seed, you can verify that there are now 100 comments in your database. This data will serve as the foundation for our 'Load More' pagination demonstration.

Next we will update the UI of the comment index page, open app/views/comments/index.html.erb and replace with the following code

<div class="max-w-sm mx-auto py-6">
  <div id="comments" class="space-y-2">
    <%= render @comments %>
  </div>
</div>

Update the comment partial too by open app/views/comments/_comment.html.erb and replace with the following code

<div id="<%= dom_id comment %>" class="px-4 py-2 border rounded">
  <div class="text-sm">
    <span class="font-semibold"><%= comment.author_name %></span> says
  </div>
  <div class="mt-1 text-sm"><%= comment.message %></div>
</div>

Open the app/controllers/comments_controller.rb and update the index method

def index
  @comments = Comment.order(id: :desc)
end

Now run the application with bin/dev and access http://localhost:3000/ and you should see the list of comments.

Pagy Pagination

Now, let's dive into the pagination part of this post: setting up Pagy for handling pagination in our Rails application. If you haven't included the Pagy gem in your project, you'll need to add it manually. Here's how you can do it:

Tired of the endless server configuration headaches? WrappedBy takes care of it all for you. Say goodbye to manual setup and hello to hassle-free deployment. Discover the ease of deploying Ruby apps with WrappedBy and revolutionize your development process. Start deploying with WrappedBy

Manual Installation

Head over to the Pagy Installation Documentation for detailed instructions.

UpperBracket Users

If you've created your app using the UpperBracket template, adding Pagy is a breeze. Just follow these steps:

Open your ApplicationController located at app/controllers/application_controller.rb.

Include the Pagy::Backend module in your controller like this:

class ApplicationController < ActionController::Base
  include Pagy::Backend
end

Now we are gonna use pagy to handle the records from the database, open app/controllers/comments_controller.rb and replace the index method with the following code

def index
  query = Comment.order(id: :desc)
  @pagy, @comments = pagy(query, items: 10)
end

In the code above:

We create a query to retrieve comments from the Comment model, sorting them by ID in descending order (from the oldest to the newest).
Next, we utilize Pagy to paginate the query results. We assign the Pagy instance to @pagy and the paginated comments to @comments, specifying that there should be a maximum of 10 records per page.

After making these changes, restart your application, and you'll notice that the index page now displays the newest 10 comments.

Let’s add the ‘Load More’ button to our page that function to load the next group of data. Create a new file app/views/comments/_load_more_button.html.erb and add the following code

<% if @pagy.next.present? %>
  <%= button_to "Load more", comments_path,
    params: { page: @pagy.next },
    method: :get,
    class: "mt-4 inline-flex items-center justify-center text-sm font-medium ring-offset-background bg-slate-900 text-white hover:bg-slate-900/90 h-9 rounded-md px-3 transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50",
    "data-turbo-stream": true %>
<% end %>

Here's an overview of what this code does:

We first check if there's a next group of data available. If not, the 'Load More' button won't be displayed.
Next, we construct a GET request form using the button_to helper. This form sends data to the comments_path with the page parameter set to @pagy.next.
Take note of the "data-turbo-stream" attribute. We're configuring the form to be sent as a TURBO_STREAM instead of HTML. This feature was introduced in July 2022, allowing Turbo Streams even with GET requests. You can find more details in this pull request.

With the 'Load More' button ready, let's include it in our list page. Open app/views/comments/index.html.erb and insert the following code after the div#comments element:

<div class="flex justify-center" id="load_more_button">
  <%= render "comments/load_more_button" %>
</div>

Now, when you check your application, you should see the 'Load More' button in action, fetching the next group of data. Let's take a closer look at what we've accomplished so far.

However, please note that, as of now, the new data replaces the current data on the page, resembling conventional pagination. Additionally, you might notice changes in the URL based on the current page, similar to what's shown in the video.

The Magic of Turbo Streams

Now, it's time to introduce Turbo Streams to our application. Turbo Streams will enable us to load the next set of data and seamlessly append it to the existing content on the page, all without altering the URL. Here's how we'll integrate Turbo Streams:

Open app/controllers/comments_controller.rb and modify the index action to handle Turbo Streams requests triggered by the 'Load More' button:

def index
  query = Comment.order(id: :desc)
  @pagy, @comments = pagy(query, items: 10)

  respond_to do |format|
    format.html
    format.turbo_stream
  end
end

In this code, we use the respond_to method to handle different response formats, including HTML and Turbo Stream. Depending on the format of the incoming request, Rails will render the corresponding view. For HTML requests, it renders index.html.erb, while for Turbo Stream requests, it looks for index.turbo_stream.erb. However, we haven't created the index.turbo_stream.erb file yet, which may result in an error when you test the application.

To fix the errors, let’s create template for turbo_stream format. Create a new file app/views/comments/index.turbo_stream.erb and add the following code

<%= turbo_stream.append "comments" do %>
  <%= render partial: "comments/comment", collection: @comments %>
<% end %>

<%= turbo_stream.update "load_more_button" do %>
  <%= render "comments/load_more_button" %>
<% end %>

Here's a breakdown of what this code accomplishes:

In the first group, turbo_stream.append, we render partial data from app/views/comments/_comment.html.erb while passing the @comments collection. These comments are then appended to the element with the comments ID. This code ensures that the new group of data is added to the bottom of the existing content, rather than replacing it.
In the second group, we update the load_more_button element using the same partial file we used in index.html.erb. It checks for the presence of a next page; if one exists, it renders the button along with the new data. Otherwise, the button is not displayed.

And that's it! We've successfully implemented seamless 'Load More' pagination in Rails without writing a single line of JavaScript. All of this happens within a single page, providing a user-friendly experience.

Now, to see it in action, go ahead and restart the application and test it out.

There you have it – a 'Load More' pagination in Rails that enhances user experience, all without the need for JavaScript. How awesome is that?

Bonus

If your application experiences slower load times and you want to provide users with real-time feedback that the application is actively fetching data, you can implement a loading indicator. To achieve this, open app/views/comments/_load_more_button.html.erb and include the data-turbo-submits-with attribute in the button_to element. Here's the complete code:

<% if @pagy.next.present? %>
  <%= button_to "Load more", comments_path,
    params: { page: @pagy.next },
    method: :get,
    class: "mt-4 inline-flex items-center justify-center text-sm font-medium ring-offset-background bg-slate-900 text-white hover:bg-slate-900/90 h-9 rounded-md px-3 transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50",
    "data-turbo-stream": true,
    "data-turbo-submits-with": "Loading..." %>
<% end %>

For demonstration purposes, you can insert sleep 3 within the index method of CommentsController to simulate the delay. With this change, you'll see the "Loading..." text displayed on the button while the application processes the next page.

Summary

Turbo Streams offer numerous advantages for building interactive applications without the need for JavaScript. However, it's important to consider when and where to use Turbo Streams. I recommended if the form using Turbo Streams, you should consider to use respond_to in the controller to handle both HTML and TURBO_STREAM formats. This approach ensures that your application remains functional even if JavaScript is disabled in the user's browser. By accommodating both formats, your application gracefully falls back to HTML when JavaScript is unavailable, providing a seamless experience for all users.

Download the source code hotwire-load-more-pagination

Ever wished deploying Ruby on Rails apps could be as easy as a few clicks? 🚀 Curious about a platform that promises to simplify the entire process? Well, your curiosity is about to pay off! Introducing WrappedBy, your ultimate destination for lightning-fast Ruby application deployment right onto your very own server. Check out our YouTube video here:

Build Drag and Drop with Rails Hotwire

Maful — Sun, 10 Sep 2023 14:18:41 +0000

Introduction

Hey there, fellow web adventurers! 😄 Building web applications can be a wild ride, right? Well, today, we're diving into something super cool: adding drag-and-drop functionality to your Rails app, all powered by the magic of Hotwire! No worries, we won't start from scratch – we're keeping it hush-hush and diving straight into the fun part. 💫

Shhh... Our Little Secret Tool

We've got a secret weapon up our sleeves: UpperBracket. It's like having a magic wand for generating full-stack Rails applications. It comes with all the goodies – Vite, Tailwind CSS, Rodauth, Rubocop, and more – so we can focus on the fun stuff. Let's keep this between us, though! 😉

Curious about the final result, here it is

// Detect dark theme var iframe = document.getElementById('tweet-1700544976961384775-609'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1700544976961384775&theme=dark" }

Let’s build

So, what's the plan? We're creating a simple course list app where users can rearrange items with a flick of their mouse. Super cool, right? Let's kick things off by generating our Rails app using the UpperBracket template.

# create app
rails new hotwire-dragndrop \
  -d postgresql \
  -m https://raw.githubusercontent.com/maful/upperbracket/main/template.rb

# move to the app directory
cd hotwire-dragndrop

Make sure you've got your PostgreSQL database up and running before we dive in.

Create Course model with only title attribute, no fancy here.

rails g scaffold Course title

Open the generated migration for the courses table and define that title should not be null.

class CreateCourses < ActiveRecord::Migration[7.0]
  def change
    create_table :courses do |t|
      t.string :title, null: false

      t.timestamps
    end
  end
end

Add presence validation for the title in the Course model app/models/course.rb

# frozen_string_literal: true

class Course < ApplicationRecord
  validates :title, presence: true
end

Update the routes to set the root page to the courses list page config/routes.rb

# frozen_string_literal: true

Rails.application.routes.draw do
  root "courses#index"
  resources :courses
end

I’m going to add a small styling for the page with Tailwind CSS, this is optional. Open the course index page app/views/courses/index.html.erb and update with the following code

<div class="container">
  <div class="max-w-screen-md mx-auto py-10">
    <p style="color: green"><%= notice %></p>

    <div class="mb-6">
      <%= link_to "New course", new_course_path, class: "rounded border border-slate-500 px-2 py-3" %>
    </div>

    <h1 class="text-2xl font-medium mb-4">Courses</h1>

    <div id="courses" class="flex flex-col gap-4">
      <%= render @courses %>
    </div>
  </div>
</div>

Open the single course page in app/views/courses/_course.html.erb

<div
  id="<%= dom_id course %>"
  class="bg-gray-50 shadow-sm space-y-6 py-6 px-4"
>
  <div class="flex gap-4 items-center">
    <div class="text-gray-700 text-base">
      <%= course.title %>
    </div>
  </div>
</div>

Run rails db:migrate to run the database migration. You can now run the application using bin/dev

Add some courses to the application and you should now have something like this

You have a list of courses now and quite looks good for simple application. However, this isn’t the end and notice that we are still unable to drag and drop the course.

Adding Sortable Library

It is time to adding the main functionality drag and drop, add ranked-model gem to handle record ordering in the backend

bundle add ranked-model

And add node packages for reorderable library and http request

yarn add sortablejs @rails/request.js

Create migration to add row_order column to courses table

rails generate migration AddRowOrderToCourses row_order:integer

row_order column is nullable, and we already added some courses in the previous section. Let’s fill the row_order for the existing records in the courses table.

rails generate migration BackfillRowOrderToCourses

Update the generated migration file with this code

class BackfillRowOrderToCourses < ActiveRecord::Migration[7.0]
  disable_ddl_transaction!

  def up
    Course.unscoped.in_batches do |relation|
      relation.update_all('row_order = EXTRACT(EPOCH FROM created_at)')
      sleep(0.01)
    end
  end
end

This migration basically disable DDL (Data Definition Language) during migration of this file, then iterate over course records in batches and update the row_order from the created_at data. Execute the new migration by running rails db:migrate

Open the Course model and include the ranked-model gem and configure it.

# frozen_string_literal: true

class Course < ApplicationRecord
  include RankedModel

  ranks :row_order
end

Now, open the app/controllers/courses_controller.rb and update the @courses variable in index method to use ranked-model for the ordering

def index
-  @courses = Course.all
+  @courses = Course.rank(:row_order).all
end

Run the application and and check the courses page. Nothing difference in the context of UI, but if you check the development log, you will notice that the row_order has been use for for ordering record.

Course Load (1ms)  SELECT "courses".* FROM "courses" ORDER BY "courses"."row_order" ASC

Create Sortable Stimulus Controller

In previous section, we have install sortable node package, in this section we’re going to use it. Create stimulus controller named sortable in app/javascript/controllers/sortable_controller.js and add the following code

import { Controller } from "@hotwired/stimulus"
import Sortable from "sortablejs"

export default class extends Controller {
  connect() {
    const options = {
      onEnd: this.onEnd.bind(this)
    }
    Sortable.create(this.element, options)
  }

  onEnd(evt) {
    const body = { row_order_position: evt.newIndex }
    console.log(body)
  }
}

Explanation

Import the sortablejs to the controller
Override the connect function to create the Sortable instance for the current element. Meaning that it will register the element where you connect with the sortable controller.
Imagine you've just dragged an item to its new spot. What's next? We're adding a function called onEnd. It's like the grand finale of a show! But here's the twist – we're not sending data to the backend just yet. Instead, we're logging the new index to keep an eye on the changes.

Let’s try to connect the stimulus controller to our element. Open app/views/courses/index.html.erb file and add data-controller="sortable" to the div element with id courses

<div id="courses" class="flex flex-col gap-4" data-controller="sortable">

Now, let's put our work to the test! Fire up your app and give that course list a spin. You'll notice that you can drag and drop items, and the item you drag will change its position. Check your browser console to see the new position in action!

But here's the catch – when you refresh the app, it's like hitting the rewind button. Why? Because we haven't saved the changes to the database yet.

The position is based on the index, and index in JavaScript is start from 0. If you check again in sortable_controller.js, the body variable is what we need to send to the backend and backend save it to the database. Let’s do it, in the onEnd function, remove the console.log line because we don’t need it anymore. So here is the final sortable_controller.js for now.

The position we're talking about is based on the index. Quick heads-up – in JavaScript, the index starts at 0. Peek at sortable_controller.js, and you'll find the body variable. That's the golden nugget we need to send to the backend for database saving.

In the onEnd function, we're saying goodbye to the console.log line – we don't need it anymore. Here's our polished sortable_controller.js for now.

import { Controller } from "@hotwired/stimulus"
import Sortable from "sortablejs"
import { patch } from "@rails/request.js"

export default class extends Controller {
  connect() {
    const options = {
      onEnd: this.onEnd.bind(this)
    }
    Sortable.create(this.element, options)
  }

  onEnd(evt) {
    const body = { row_order_position: evt.newIndex }
    patch(evt.item.dataset.sortableUrl, {
      body: JSON.stringify(body),
      responseKind: "turbo-stream",
    })
  }
}

Oh, and one last thing – you'll spot evt.item.dataset.sortableUrl in the code. It's like a map to where the request should be sent. Let's create a route for it by opening up the routes and adding the rank with the patch method to the courses resources.

resources :courses do
  patch "rank", on: :member
end

Open the courses_controller.rb and add new method called rank, this will update the row_order in the course record

def rank
  @course.update(row_order_position: params[:row_order_position])
end

Don’t forget to include the rank method in the before_action callback at the top

- before_action :set_course, only: [:show, :edit, :update, :destroy]
+ before_action :set_course, only: [:show, :edit, :update, :destroy, :rank]

Open app/views/courses/_course.html.erb and add data-sortable-url attribute which contains the route that we created just now

<div
  id="<%= dom_id course %>"
  class="bg-gray-50 shadow-sm space-y-6 py-6 px-4"
  data-sortable-url="<%= rank_course_path(course) %>"
>

If you inspect the element, you should see something like this

So, the code evt.item.dataset.sortableUrl is to find the value of data-sortable-url on the item being dragged.

Tired of the endless server configuration headaches? WrappedBy takes care of it all for you. Say goodbye to manual setup and hello to hassle-free deployment. Discover the ease of deploying Ruby apps with WrappedBy and revolutionize your development process. Start deploying with WrappedBy

Let’s try again the application, the order of the position should be persisted in the database now.

Bonus 1

So, you've nailed the drag-and-drop with Hotwire in Rails, but here's a cool UX upgrade we can sprinkle in. How about adding a little marker to highlight where your item will land in the list? Let me show you what I mean. Open up your trusty sortable_controller.js and introduce the ghostClass option within the connect function.

const options = {
  onEnd: this.onEnd.bind(this),
  ghostClass: "bg-red-300"
}

If you try the app again, you'll see a red background at the new position of the dragged item.

Bonus 2

But wait, there's more! Here's another tip for you – what if you want to be super specific about which elements can be dragged and which ones stay put? Well, it's all about using the handle selector. Each element will look a little something like this:

First, add move icon in the left side of the course title. Here is the complete of app/views/courses/_course.html.erb file

<div
  id="<%= dom_id course %>"
  class="bg-gray-50 shadow-sm space-y-6 py-6 px-4"
  data-sortable-url="<%= rank_course_path(course) %>"
>
  <div class="flex gap-4 items-center">
    <div class="sortable-handle cursor-grab">
      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-move h-4 w-4 text-current"><path d="m5 9-3 3 3 3M9 5l3-3 3 3M15 19l-3 3-3-3M19 9l3 3-3 3M2 12h20M12 2v20"/></svg>
    </div>
    <div class="text-gray-700 text-base">
      <%= course.title %>
    </div>
  </div>
</div>

Open app/views/courses/index.html.erb and add data-sortable-handle-selector-value attribute after the data-controller

<div id="courses" class="flex flex-col gap-4" data-controller="sortable" data-sortable-handle-selector-value=".sortable-handle">

And also add handle selector in sortable_controller.js and define the Stimulus values to store the value of data-sortable-handle-selector-value attribute.

import { Controller } from "@hotwired/stimulus"
import Sortable from "sortablejs"
import { patch } from "@rails/request.js"

export default class extends Controller {
  static values = {
    handleSelector: String,
  }

  connect() {
    const options = {
      onEnd: this.onEnd.bind(this),
      ghostClass: "bg-red-300"
    }
    if (this.hasHandleSelectorValue) {
      options.handle = this.handleSelectorValue
    }
    Sortable.create(this.element, options)
  }

  onEnd(evt) {
    const body = { row_order_position: evt.newIndex }
    patch(evt.item.dataset.sortableUrl, {
      body: JSON.stringify(body),
      responseKind: "turbo-stream",
    })
  }
}

Let’s check again the final result of our application

Notice the difference here, only elements with class sortable-handle that can be dragged, the rest of the element back to the normal element.

Conclusion

With Stimulus, you've got a powerful tool to take your app's interactivity up a notch. For even more awesome features and Stimulus wizardry, check out the Stimulus Documentation.

And that's a wrap, folks! 🎉 We've successfully built a slick list that you can drag and drop like a pro, and guess what? It's all snugly persisted in the database. But hey, don't stop here – you can take it further by adding topics to each course and letting users drag and drop topics across different courses. Sounds intriguing, right? You can make it happen using the group options in Sortablejs.

Full source code can be download on hotwire-sortable repository.

Are you tired of the complexities of deploying Ruby on Rails applications? Discover WrappedBy – the ultimate deployment solution for Ruby enthusiasts. Curious to see how it works? Check out our YouTube video here:

How to use Cloudflare R2 with Ruby on Rails Active Storage

Maful — Mon, 26 Jun 2023 07:44:25 +0000

Introduction

File upload is one of the most common features in applications, not only in web applications but also in other platforms. For instance, a user is able to upload an avatar profile and a team owner can upload a logo. In web applications, there are three common approaches:

Local file system: Store the uploaded files directly on the file system of the host. This is commonly used in the development stage, storing in file system is simple to implement but it may not scale well and does not recommend in production.
Database: Did you know that you can store files in a database as binary? This approach can simplify the management files as they are stored with the rest of the applications’ data. While this approach is simple, it comes with the downside that it can increase database size very quickly and will impact the performance of the database.
Object Storage: In the production environment, cloud object services such as Amazon S3, Azure Blob Storage, and Cloudflare R2 is very common. This approach provides scalability, durability, and flexibility as it offers access from everywhere. It can be more cost effective and offers features like data redundancy and CDN integration.

The choice of storage strategy depends on several factors such as volume of files, performance requirements, cost consideration, and more. As a developer, it’s important to evaluate the trade-off of each strategy and select the best suit for your application.

In this post, we will go through object storage and use Cloudflare R2 for the service.

What is Cloudflare R2?

R2 is an object storage service from Cloudflare that offers Zero egress fee and the freedom to create multi-cloud architectures.

What does Zero egress fee mean? Simply put, it means you don't have to pay when you access your data.

What makes Cloudflare R2 more exciting is its S3-compatible API. Amazon S3 is the most popular object storage service, as it offers Amazon's architecture and effective pricing. If you plan to migrate from Amazon S3 to Cloudflare R2, you can simply change the configuration. Here are the key features of Cloudflare R2:

Globally distributed
S3-compatible API
Zero egress fee
Integration with Cloudflare’s CDN
In-workers API
Integration with Workers

R2 is charged based on the total volume of data stored. There are two classes of operation on the data:

Class A operations: more expensive and tend to mutate the data
Class B operations: cheaper and tend to read existing data

R2 also offers a FREE plan with 10 GB / month, here are the details per this article posted. With only a Free plan you can give it a try in your next project.

More Cloudflare R2 Pricing

In this post, we will be focusing on how to use R2 with Ruby on Rails Active Storage.

Build application

We will build a simple application where the users can crete post and upload the image. Note that we will not set up the authentication to simplify this post.

Open your terminal and let's create a Rails app without a test

rails new rails-cloudflare -T

Now we will set up Active Storage, install image_processing gem for image analysis and transformations by open Gemfile and uncomment the line image_processing and run bundle install

- # gem "image_processing", "~> 1.2"
+ gem "image_processing", "~> 1.2"

Now it’s time to install Active Storage, run the following command to copy database migration into application

bin/rails active_storage:install

then migrate the database

bin/rails db:migrate

Now, Active Storage is active and ready to use. You can check the service options in config/storage.yml. The default service is local as you can see in config/environments/development.rb

# Store uploaded files on the local file system (see config/storage.yml for options).
config.active_storage.service = :local

It’s time to setup R2, first register to Cloudflare if you don’t have an account. Then, in the dashboard click the R2 menu. Copy the Account ID and save it, we will use it later

Click Create bucket, enter the bucket name and leave the Location is automatic. Then create Create bucket to save it.

And now you have successfully create R2 bucket. Let’s create R2 API Tokens. Back to R2 Overview and click Manage R2 API Tokens. This tokens will give you an access to the bucket and authenticate your service.

Enter the token name that represents the bucket, this will help you to identify.
Choose Edit for the permissions as we will upload the file, not only read it
Select TTL, for this case I select the token will be valid for 1 year

Click Create API Token , you will be redirected to the page that shows the Access Key ID and Secret Access Key. Don’t forget to copy and save them in secure place as this will not be shown again.

Now you have Bucket and API Tokens, it’s time to use it in the code. Open config/storage.yml and add new service named cloudflare

cloudflare:
  service: S3
  endpoint: https://<%= Rails.application.credentials.dig(:cloudflare, :account_id) %>.r2.cloudflarestorage.com
  access_key_id: <%= Rails.application.credentials.dig(:cloudflare, :access_key_id) %>
  secret_access_key: <%= Rails.application.credentials.dig(:cloudflare, :secret_access_key) %>
  region: auto
  bucket: <%= Rails.application.credentials.dig(:cloudflare, :bucket) %>

We are using Rails Credentials to save the sensitive information under cloudflare key. Note that the service is S3 because we will using S3 SDK for Ruby, remember that Cloudflare R2 is S3-compatible API.

Tired of the endless server configuration headaches? WrappedBy takes care of it all for you. Say goodbye to manual setup and hello to hassle-free deployment. Discover the ease of deploying Ruby apps with WrappedBy and revolutionize your development process. Start deploying with WrappedBy

Because we will use R2 in development, edit the Rails Credentials for development by running

rails credentials:edit --environment development

it will open your editor, normally is vim and then add this code. Replace the value with your own information that you setup earlier.

cloudflare:
  account_id: your_account_id
  access_key_id: your_access_key_id
  secret_access_key: your_secret_access_key
  bucket: your_bucket

Open Gemfile and add aws-sdk-s3 gem and then run bundle install

gem "aws-sdk-s3", "~> 1.122", require: false

You have successfully create cloudflare service, now it’s time to use it in development environment. Open config/environments/development.rb and change this code

- config.active_storage.service = :local
+ config.active_storage.service = :cloudflare

By doing this, every file uploaded in development environment will go through in your R2 account.

Now create a simple resource called Post with image on it.

bin/rails generate scaffold Post title:string image:attachment

If you open app/models/post.rb you will see that the Post has one image relation.

class Post < ApplicationRecord
  has_one_attached :image
end

Open your terminal and migrate the database by running rails db:migrate. We can try it now, start the server rails s and open http://localhost:3000/posts. Click New post and enter Title and Image and then save it.

Now, go back to Cloudflare dashboard and check in the bucket, you will see new object has been added. You can also check in your app by open the latest post, which will redirected to post details. Then, click the image link, you will see that the app will redirect to cloudflare domain which is https://<bucket_name>.<account_id>.r2.cloudflarestorage.com/

Congratulations! 🎉 Now your application is using Cloudflare R2 now for the object storage which offers zero egress fee.

Bonus

In some case, you may need to allow the user to upload the file in the text editor like Trix editor. However, you current configuration not allowed it, you need to configure the CORS. Here the configuration

Open the Cloudflare dashboard and click the bucket, in the bucket details click the Settings. Scroll down util you found the CORS Policy section and then click Add CORS policy

Add this code and replace the domain origins with your domain.

[
  {
    "AllowedHeaders": [
      "*"
    ],
    "AllowedMethods": [
      "PUT"
    ],
    "AllowedOrigins": [
      "https://www.example.com"
    ],
    "ExposeHeaders": [
      "Origin",
      "Content-Type",
      "Content-MD5",
      "Content-Disposition"
    ],
    "MaxAgeSeconds": 3600
  }
]

Now your users can upload the file from the client side or normally called direct uploads.

Conclusion

Using object storage can help you to not deal with infrastructure problems and durability issues as it is already handled by the service provider. While R2 may offer flexible pricing, as a developer, you may need to consider using caching strategies to further reduce costs.

I have been using Cloudflare R2 in my recent projects, and overall my experience has been good. I appreciate the simplicity, performance, and pricing.

Download the source code https://github.com/daily-newer/rails-cloudflare-r2

How I use Nano ID in Rails

Maful — Fri, 16 Jun 2023 12:10:07 +0000

Introduction

When building applications normally we will use database to store the data. Defining the primary key in our database is straightforward, the hard part is what data type for the primary key. Basically, the default primary key is bigint and auto-increment, which means when there is a new record created the primary key will increment 1,2,3, and so on. This is the easiest and simplest way to ensure the primary key is unique, there is another option that you may hear which is using UUIDv4 for the primary key. UUIDv4 (Universally Unique Identifier version 4) is a type of UUID that is randomly generated and 128-bit numbers represented as strings, typically consisting of five groups of hexadecimal digits separated by hyphens. UUIDv4 is generated using random or pseudo-random values and each UUIDv4 is considered unique across systems and time and it is designated to have a low probability of collision. Format UUIDv4 looks like this:

xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx

x represents a hexadecimal digit (0-9, a-f) that is randomly generated
4 indicates that is version 4 of UUID (UUIDv4)
y specifies certain bits that define the UUID variant UUIDv4 has become popular in web development and making it suitable for generating unique identifiers in distributed systems.

I have personally used UUIDv4 in several applications and the experience has been great. It may be a good option to use UUIDv4 when you want to expose the primary keys to the end user (e.g URL, API) without providing information about the total amount of data in your table. For example:

{
  "id": "21ba44b5-8ce3-4d07-a303-925c01cd83c3",
  "name": "John Doe",
  "country": "Indonesia"
}

And here is when using bigint as a primary key

{
  "id": 201,
  "name": "John Doe",
  "country": "Indonesia"
}

As you can see, when using bigint we can guess that John Doe is the 201st user. However, if using UUIDv4 we can't guess it. But if you don't have concerns to expose the id of the users regardless the user can guess it or not, bigint could be your best option.

The problems

There is no problem when using UUIDv4 in terms of the performance of the application. But as a developer, I always try to find other options to improve my knowledge and I may implement it in the future. UUIDv4 is a great choice if you don't want to use bigint for the primary key, but as you already see that the result is pretty long and it is hard to copy the UUIDv4 with just a double click because it is separated by hyphen (-) so you need to select manually if you want to copy the whole id. This might be not a good experience at least from my point of view. Another problem is UUIDv4 is not sortable, it means you need to use different columns to sort your data. You can use a timestamp in your table if you have it, you will have this if using migration tools.

So, how we solve these problems?
What are we looking for for the solution?
What if I still want to use bigint but I need data that can safely show to the end user?

To answer these questions let's go to the next section where I share what I use for my recent applications.

Nano ID

Introducing Nano ID. Nano ID is a tiny, URL-friendly, and unique string ID generator. Nano ID is a library for generating random IDs that still have probability of duplicate IDs. However, this probability is extremely small and it is based on the rules you defined. Here are the features of Nano ID:

Small - No dependencies, Size Limit controls the size.
Safe - It uses hardware random generator and it can safely used in clusters.
Short IDs - Uses larger alphabet than UUID (A-Za-z0-9_-). We can control it.
Portable - Nano ID has been ported to many programming languages.

Source: Nano ID

To answer previous questions let's break them down one by one

So, how we solve these problems?

There are always advantages and disadvantages of the solution, however looking for other alternative that close to our requirements is important.

What are we looking for for the solution?

We are looking for a safe, small, and easy to control when generating random IDs where we safely can show to the end user and improve the user experience.

What if I still want to use bigint but I need data that can safely show to the end user?

You come to the right place, Nano ID is another good option where you want to control what the user can see while still maintaining using bigint for the primary key and of course you don't have to worry because bigint is sortable. Just like regular int but has more capacity.

Nano ID in Rails

In this post, we are focusing on the Rails ecosystem which try to implement Nano ID in the Rails application. Nano ID has been ported to many programming languages including Ruby, you can check the Nano ID for Ruby here.

We are going to build simple Rails application for the Customer data, we will look Stripe Customers page for basic inspiration. If you check out the Customer details page in Stripe, you will see the URL is using a random string instead of an integer. Here is an example

https://dashboard.stripe.com/test/customers/cus_MYkMUCVOIudoZg

The cus_MYgMUCVOIuqoZg is a random unique string generated for the customer and you will see this string across all systems in Stripe. So, we will implement this kind of strategy in the Rails application.

Note that what Stripe implements might be different, so take this post as a reference.

Building application

Let's start by creating empty rails application by run rails new rails-nanoid -T. The -T means we will skip the tests file as we don't need for now. If you want to can add it later.

Your Ruby applications have immense potential, and WrappedBy is here to help you unlock it. From server management to automatic HTTPS, WrappedBy ensures your projects reach their full potential. Ready to experience a new level of deployment? Start deploying with WrappedBy

Then, move to the application directory cd rails-nanoid and if you use Git, don't forget to commit your changes.

git add .

git commit -m "Initial commit"

Open the app in your text editor, open Gemfile and add nanoid gem or you can add it from your terminal by running

bundle add nanoid

This command will add nanoid to your Gemfile and download it. If you check Gemfile , now you will see the nanoid has been installed.

gem "nanoid", "~> 2.0"

We we build a customer data which includes name and country. We will still using bigint for the primary key and adding a new column called public_id to store the random IDs generated by Nano ID. We will not expose the id, instead we will use public_id for the URL routing. For example, to access the customer details the URL will be customers/cus_MYgMUCVOIuqoZg.

Create Customer Model

We will use the rails scaffold generator to create customers page, this command will create model, views, and controller for us.

rails g scaffold Customer name:string country:string

Add the public_id field with the type string and the length is 18. Why 18? Here is the illustration:

cus is the prefix for the public_id that has length 3 characters
_ is the separator that has 1 character
randomId followed by random ID that has length 14 characters

So the total is 18 characters in length for our public_id. We need to specify the length of the field so we don't waste resources in the database.

Then open up the customers migration, in my case will be db/migrate/20230531051206_create_customers.rb and add public_id field, you can named it whatever you want but for suggestion named it something meaningful and easy to understand. So the complete migration for the customers table looks like this:

class CreateCustomers < ActiveRecord::Migration[7.0]
  def change
    create_table :customers do |t|
      t.string :public_id, limit: 18, null: false
      t.string :name
      t.string :country

      t.timestamps
    end
    add_index :customers, :public_id, unique: true
  end
end

In this migration, we define public_id with string type with a limit of 18 and can't be null. Also, we add a new unique index to ensure there is no duplicate public_id.

Run the database migration by running rails db:migrate.

If you try to create a customer, you will get an error because public_id is empty and we have defined that public_id can't be null in the database.

Generate Random ID with Nano ID

It's time to add Nano ID to our code, and create a new concern file called public_id_generator.rb

touch app/models/concerns/public_id_generator.rb

And then add this code

# frozen_string_literal: true

module PublicIdGenerator
  extend ActiveSupport::Concern

  included do
    class_attribute :public_id_prefix
    self.public_id_prefix = nil

    before_create :set_public_id
  end

  PUBLIC_ID_ALPHABET = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
  PUBLIC_ID_LENGTH = 14
  MAX_RETRY = 1000

  class_methods do
    def generate_nanoid(alphabet: PUBLIC_ID_ALPHABET, size: PUBLIC_ID_LENGTH, prefix: nil)
      random_id = Nanoid.generate(size:, alphabet:)
      prefix.present? ? "#{prefix}_#{random_id}" : random_id
    end
  end

  def set_public_id
    return if public_id.present?

    MAX_RETRY.times do
      self.public_id = generate_public_id
      return unless self.class.exists?(public_id:)
    end
    raise "Failed to generate a unique public id after #{MAX_RETRY} attempts"
  end

  def generate_public_id
    self.class.generate_nanoid(prefix: self.class.public_id_prefix)
  end
end

Explanation:

Line 7-8: Define the class attribute called public_id_prefix where each model that includes the PublicIdGenerator can specify the prefix based on the requirements. Example cus for Customer model
Line 10: Add Active Record Callbacks before_create to assign public_id field to the random id generated by Nano ID. So this will only be called before creating a resource.
Line 13: Define constants about what characters that will be used in the generator, the Nano ID will take this constant to generate random IDs.
Line 14: Define the maximum length of the random ID.
Line 15: Define the maximum number the application try when there is a collision. In this case, the app will try maximum 1000 times until found the unique random id.
Line 18-21: Define class method for generating random id with Nano ID which is the actual code of generating random IDs with Nano ID.
Line 24-32: In line 10, we add callback to call this method. This method doing a job to check whether the unique id has been used or not and assign it to the public_id field.
Line 34-36: Call the generate_nanoid method and pass the prefix defined in the model.

Using PublicIdGenerator

After we create a module for generate the public id, now we include the module in Customer model. Also, define the public_id_prefix for the Customer in this case is cus.

class Customer < ApplicationRecord
+  include PublicIdGenerator
+  self.public_id_prefix = "cus"
end

So, if you want to use public_id in more models, what you have to do is add public_id field and include PublicIdGenerator module and you're ready to go.

Now let's try this out, run the application by rails server, open http://localhost:3000/customers in your browser and try to add a new customer. If success, you will be redirected to the details page, notice that the url is still using default id http://localhost:3000/customers/1. However, if you look at the logs you will see the public_id is generated

you can also check in the rails console to the first customer

As you can see, we have successfully generate random IDs with Nano ID. We have one left todo which is expose the public_id instead of id in the URL.

Expose public_id

Currently, in the URL is using id and we want it to use public_id instead. In Rails, this is very common to change the params id in the URL and it is very straightforward.

Open routes file config/routes.rb and change the customers route.

- resources :customers
+ resources :customers, param: :public_id

Open the customer model and add override to_param method which Action Pack uses for constructing a URL to this object. In this case, we will use public_id instead of id.

class Customer < ApplicationRecord
    # code before...

  def to_param
    public_id
  end
end

Lastly, open the customers controller app/controllers/customers_controller.rb and update to find the customer by public_id

def set_customer
-  @customer = Customer.find(params[:id])
+  @customer = Customer.find_by!(public_id: params[:public_id])
end

Let's try again, re-start the server and access the customers page. Click one of the customers you've created and see the URL is changed and now using the public_id. In my case, the first customer will be http://localhost:3000/customers/cus_B0VmfGwUidEvV4.

Congratulations!🎉 You have successfully implementing Nano ID in Rails application, with this approach we still maintain regular id for the primary key which supports unique data and sortable.

Conclusion

Although using the default primary key using bigint is good, we must have to look out for requirements first. Define what the application needs, and consider the pros and cons of using a different approach.

Using randomly generated IDs like Nano ID could be a good alternative, however, as a developer, we must understand what Nano ID really does in our application. Defining the number of characters in the generated IDs is also important, to help with that Nano ID has a Collision Calculator to give us how many years in order to have a 1% probability of collision.

In our case, we have 14 characters and it needs at least 57 thousand years to have 1% probability of collision if we generate 1000 IDs per hour.

Download source code: https://github.com/daily-newer/rails-nanoid

References

Getting Started Upstash Redis with Rails

Maful — Sat, 31 Dec 2022 03:44:53 +0000

Introduction

Have you ever wondered if there is an open source database that has a fast response time and key-value storage base?

The answer is yes. Introducing Redis, an open source database, in-memory, key-value data store and it is most-loved database by developers for five years running. Redis is commonly used as a primary database, cache, queue, and message broker. Redis is an open source (BSD licensed) that is used primarily as an application cache and quick response database and because the data is stored in memory rather than on a disk, Redis delivered unparalleled speed, reliability, and performance.

Redis provides data structures such as strings, hashes, lists, sets, sorted sets, bitmaps, streams, and much more. More about Redis data types. To achieve top performance, Redis can persist the data either by periodically dumping the dataset to disk or by appending each command to a disk-based log. Redis is written in ANSI C and works on most POSIX systems like Linux, and Mac OS X without external dependencies. Redis can be used in almost every programming language. You can find the Redis Clients page.

Upstash

Upstash is a cloud database provider with over 35 thousand users (per date) and with total over 48 thousand databases active. Upstash products including Redis, Kafka, and QStash and they offer a free plan which includes Max 10,000 commands daily, TLS Encryption, and Persistence storage which is perfect for the developer who wants to try their products without being charged. The best thing in my opinion is the pricing. Pay as you go, meaning that we will pay only for what we use with per-request pricing. Even if you upgrade to a paid plan, you will only pay $0.2 (regional) per 100K commands (More about Upstash Redis Pricing).

In this post, I’m going to give you a starter guide on how to use Upstash Redis, and later we will integrate it with Ruby on Rails application.

Upstash Console

Before starting to use Upstash Redis, the first thing you have to do is create an account in Upstash. Go over to https://console.upstash.com/login and create an account. Then you can log in to the Upstash Console. You will see something like this page

Start creating a database by clicking the Create database button, you will see a modal and fill up the form. For the Type, select Regional and select the Region that is closest to your application so you can access the database with low latency.

You will be redirected to the database details

There are some features you can use on this page:

Details: In this section, you can see the details of your database like the Region, Endpoint, Password, Port, and TLS/SSL status.
Usage: You can see the total number of requests, today’s bandwidth, daily requests, and the total cost.
CLI: This is just exactly the Redis CLI but in the browser, try to type ping. If successful, you will get the response PONG.
Data Browser: You can view the data you stored in the database on this page.
Backup: You can configure the backup configuration on this page. Upstash will only keep the latest backup and overwrites the previous one. This feature can be enabled only for paid users.
Quickstarts: If you don’t know how to start using Upstash Redis, this is one of the good places to start. You can see how to get started in some frameworks or edge functions.

Redis API Compatibility

Before start install the Redis client in the application, first we need to understand the Redis API compatibility in Upstash. Currently, Upstash supports Redis client protocol up to version 6.2. The following table shows the list of supported Redis commands:

Feature	Supported?	Supported Commands
String	✅	APPEND - DECR - DECRBY - GET - GETDEL - GETEX - GETRANGE - GETSET - INCR - INCRBY - INCRBYFLOAT - MGET - MSET - MSETNX - PSETEX - SET - SETEX - SETNX - SETRANGE - STRLEN
Bitmap	✅	BITCOUNT - BITFIELD - BITFIELD_RO - BITOP - BITPOS - GETBIT - SETBIT
Hash	✅	HDEL - HEXISTS - HGET - HGETALL - HINCRBY - HINCRBYFLOAT - HKEYS - HLEN - HMGET - HMSET - HSCAN - HSET - HSETNX - HSTRLEN - HRANDFIELD - HVALS
List	✅	BLMOVE - BLPOP - BRPOP - BRPOPLPUSH - LINDEX - LINSERT - LLEN - LMOVE - LPOP - LPOS - LPUSH - LPUSHX - LRANGE - LREM - LSET - LTRIM - RPOP - RPOPLPUSH - RPUSH - RPUSHX
Set	✅	SADD - SCARD - SDIFF - SDIFFSTORE - SINTER - SINTERSTORE - SISMEMBER - SMEMBERS - SMISMEMBER - SMOVE - SPOP - SRANDMEMBER - SREM - SSCAN - SUNION - SUNIONSTORE
SortedSet	✅	BZPOPMAX - BZPOPMIN - ZADD - ZCARD - ZCOUNT - ZDIFF - ZDIFFSTORE - ZINCRBY - ZINTER - ZINTERSTORE - ZLEXCOUNT - ZMSCORE - ZPOPMAX - ZPOPMIN - ZRANDMEMBER - ZRANGE - ZRANGESTORE - ZRANGEBYLEX - ZRANGEBYSCORE - ZRANK - ZREM - ZREMRANGEBYLEX - ZREMRANGEBYRANK - ZREMRANGEBYSCORE - ZREVRANGE - ZREVRANGEBYLEX - ZREVRANGEBYSCORE - ZREVRANK - ZSCAN - ZSCORE - ZUNION - ZUNIONSTORE
Transactions	✅	DISCARD - EXEC - MULTI - UNWATCH - WATCH
Generic	✅	COPY - DEL - EXISTS - EXPIRE - EXPIREAT - KEYS - PERSIST - PEXPIRE - PEXPIREAT - PTTL - RANDOMKEY - RENAME - RENAMENX - SCAN - TOUCH - TTL - TYPE - UNLINK
Connection	✅	AUTH - HELLO - ECHO - PING - QUIT - RESET - SELECT
Server	✅	ACL - DBSIZE - FLUSHALL - FLUSHDB - TIME
Scripting	✅	EVAL - EVALSHA - SCRIPT EXISTS - SCRIPT LOAD - SCRIPT FLUSH
Pub/Sub	✅	SUBSCRIBE - PSUBSCRIBE - UNSUBSCRIBE - PUNSUBSCRIBE - PUBLISH - PUBSUB
Cluster	❌
Geo	❌
HyperLogLog	❌
Stream	❌

Go to Upstash Redis® API Compatibility to check the latest version.

Time to Play

Creating Application

Let’s start creating a Rails application and integrating Upstash Redis. Because we are going to create a Rails application, we need to make sure Ruby is installed in your machine.

$ ruby -v
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [arm64-darwin21]

$ rails -v
Rails 7.0.4

I’m using ruby 3.1.2 which is the latest stable version and Rails 7.0 version. Let’s start by creating the application

$ rails new try-upstash

This command will create a rails application with the default setting.

Now install the Redis Ruby in the application, open Gemfile, and add the redis gem

gem "redis", "~> 5.0"

Open your terminal and run bundle install to install the gems.

Now it’s time to create some models and creating some data.

# create product model
$ rails g model Product name description:text price:float
      invoke  active_record
      create    db/migrate/20221229082154_create_products.rb
      create    app/models/product.rb
      invoke    test_unit
      create      test/models/product_test.rb
      create      test/fixtures/products.yml

# create user model
$ rails g model User email:string:uniq name
      invoke  active_record
      create    db/migrate/20221229084021_create_users.rb
      create    app/models/user.rb
      invoke    test_unit
      create      test/models/user_test.rb
      create      test/fixtures/users.yml

Open db/seeds.rb and create some data for product and user model

# Create 10 products
(1..10).each do |i|
  Product.create(name: "Product #{i}", price: 100, description: "Description")
end

# Create 3 users
(1..3).each do |i|
  User.create(email: "email#{i}@example.com", name: "User #{i}")
end

Then run the seed by running rails db:seed in your terminal. To make sure the data is created we can use GUI Database or using rails console. For now, I will use the rails console, open your terminal, and type rails c, you will see the rails console and you can run ruby script or rails specific command in this console. Let’s try to get the total number of Product and User.

The database seed is success, now we have 10 products and 3 users.

#1 Use Case - Saving model attribute in Upstash Redis

Imagine that your application needs to store a boolean value to indicate whether the user has been completing the onboarding process or not. You could add a new column in your table named onboarded with the type boolean, and every time you want to verify the onboarding, you will access the table very often. This is fine if you don’t have a lot of data and the traffic is not high. But, your application is growing and the application starting slow. One of the solutions here is to save the onboarded value in the Redis database and because Redis stores the data in memory, rather than on a disk, you will get top speed and performance. You may have a question like this

“If the server crashes, will the Redis data be lost?”

The answer depends on the Redis cloud you are using, but in this case, if you are using Upstash Redis the data will be still there in the Durable Storage such as EBS in AWS. In Upstash, the data is reloaded to memory from block storage in case of a server crash. Upstash also offers Multi Zone Replication to provide you with high availability and better scalability and this can be enabled for all paid users.

Let’s back to the application, open the Gemfile and add kredis and dotenv

gem "kredis", "~> 1.3"
gem "dotenv-rails"

Run bundle install and then initiate the kredis with ./bin/rails kredis:install. This will create a file in config/redis/shared.yml . You can configure the redis connection in this file. By default the configuration will read the env variable REDIS_URL in the development and production environment and fallback to default connection redis://127.0.0.1:6379/0 if the env variable doesn’t exist.

development: &development
  url: <%= ENV.fetch("REDIS_URL", "redis://127.0.0.1:6379/0") %>
  timeout: 1

Now for the env file, create .env in your root project and add REDIS_URL variable. You need to grab the value from your database details.

and then change the redis:// to rediss:// in order to use TLS for the connection

REDIS_URL=<valuehere>

Now the connection is complete, back to our use case where we want to save the onboarded status for each user. We will use kredis in this case to make our life easier by connecting redis database to the application model. Open the user model app/models/user.rb and then add the kredis_boolean attribute because the type we need is a boolean.

class User < ApplicationRecord
  kredis_boolean :onboarded
end

If you need other than boolean type, in kredis we can use list, unique_list, enum, counter, etc.

Now you have onboarded method in your instance model. Let’s try it out in the rails console.

Open your terminal and run rails c.
Grab the first user with u = User.first

Now you have the instance of User model for the first user with id 1. You can get the onboarded value from the redis database with u.onboarded.value

As you can see, kredis will automatically map the key for the user with the format model_name_plural:id:attribute_name. You can find it in the kredis repo.

The default value is nil and we want to change the value to true or false based on our needs.

Set the onboarded value to true because the user has been completed the onboarding

To make sure the value has been changed, you can use the previous command to get the value of onboarded

Now the onboarded is true which means the user has been completed the onboarding.

In Upstash, you can also see all the data you have inserted into redis database with Data Browser feature. Go back to Upstash Console and then click your redis database. You will see the Data Browser. You can view the value based on the key, delete and set the expiration of your data key.

Or if you want to try to get the value with Redis CLI, switch to CLI page and you can type the command there. For example, I want to get the value of the first user and then use GET command.

So in your application if you want to check whether the user is onboarded, simply check the value of onboarded attribute and it’s available for the instance of User model.

u = User.first
if u.onboarded.value
  # run some code if the user is onboarded
else
  # run some code if the user has not been onboarded
end

#2 Use Case - Caching with Upstash Redis

In this second use case, we will use Upstash Redis for caching. Let’s consider that we need to cache a particular value or some query result from an external application (API). It is common that the response of the API is JSON object and we want it to cache the result for some time and save it to the redis database. In this case, we can use Low-Level Caching in Rails by using the Rails.cache.fetch method. This method does both writing and reading to the cache. As long as the information is serializable like JSON then the Rails cache works just fine.

Here is an example, consider that we have product and each product instance has method to looks up the competitor price from other website or API and then we want to save it for only 8 hours after that the value should no longer exist in the database.

First, configure the cache stores to use redis as a database. Open the development environment file config/environments/development.rb and change the store to redis

- config.cache_store = :memory_store
+ config.cache_store = :redis_cache_store, { url: ENV["REDIS_URL"] }

and because caching is disabled by default in development environment, for testing purpose we need to enabled it with command rails dev:cache

Let’s add competitor_price in the Product model

class Product < ApplicationRecord
  def competitor_price
    Rails.cache.fetch("#{cache_key_with_version}/competitor_price", expires_in: 8.hours) do
      # call the API here
    end
  end
end

That is the basic usage for our use case. So here is how it works, when you call competitor_price, the block will be executed in the event that the cache key is not available in the database or the cache key has expired and the return value of the block will be written to the cache under the given cache key.

So in this case when first time calls competitor_price it will call the API and the result of the API will be written to the cache, the second time you call competitor_price it will get the value from the cache and after 8 hours the cache key will be deleted automatically.

cache_key_with_version is a Rails method that generates cache key based on model’s class name, id, and updated_at attributes. So the resulting cache key will be something like products/1-20221229085409206814/competitor_price

For the testing, let’s update the competitor_price method and change the expires to 30 seconds and write the static value from the block

class Product < ApplicationRecord
  def competitor_price
    Rails.cache.fetch("#{cache_key_with_version}/competitor_price", expires_in: 30.seconds) do
      sleep 3
      {
        "prices": [
          { "name": "Amazon", "price": 12.00 },
          { "name": "Ebay", "price": 11.00 },
        ]
      }
    end
  end
end

You may notice that I added sleep 3 to illustrate the process when calling the external API.

Let try out in the rails console, open your terminal and run rails c

First grab the first product and assign to variable product

Now call the competitor_price with product.competitor_price, you will notice that it will take some time around 3 seconds to complete the process because this is the first time which means the block will be executed. Remember we added sleep 3 before.

Now let’s call the method again and the block will not be executed, instead, the resulting value will be from the cache. You don’t need to wait for 3 seconds to get the value.
And after 30 seconds, the cache key will be deleted automatically. You can check in the Data Browser in Upstash Console

This method is useful when you need to cache a particular query or value and you need to get the value quite often in your application without sacrificing the response time.

Summary

Upstash Redis is compatible with almost all Redis API and can be used for popular use cases such as:

Query Caching
Session Caching
Message Broker
Queues, etc

Upstash Redis also can be used as a database without worrying the data will be lost if the server crashes, thanks to Durable Storage. Upstash Global Database replicates the database and is distributed across multiple regions and the user is routed to the nearest region to minimize the latency and optimize the performance.

References

#1 Made easy form in Rails with Simple Form - Gem Weekly

Maful — Sun, 20 Nov 2022 03:00:00 +0000

Introduction

There are many ways to create a form in Rails, whether you want to create a form with Form builder from Rails itself or maybe using third party gem like simple_form. Of course, simple_form is not the only option we have to build form in rails, there are alternatives out there such as formtastic, cocoon, nested_form, etc. However, in this post I'm going to give you a basic use case to use simple_form in Rails application.

Simple Form aims to be as flexible as possible while helping you with powerful components to create your forms. Most of the DSL was inherited from Formtastic, which we are thankful for and should make you feel right at home.

Installation

Add simple_form in your Gemfile application, this will install the latest stable version of simple_form

gem 'simple_form'

However, if you want to install a specific version of the gem, you could include the version number after the gem name. In the example below, you will install the simple_form gem with 5.0.2 version.

gem 'simple_form', '~> 5.0', '>= 5.0.2'

To run the installation process, run the following command

bundle install

Simple Form Generator

Simple Form has own generator after you installed it, it helps the user to start using the gem and telling the user where the configuration is. The easiest way to start is run the following command to install or copy the template code to your project.

rails generate simple_form:install

After you run the command, the gem will copy the starting template to your project and you will use the default configuration. Normally, the configuration will be on config/initializers/simple_form.rb. You also can check the configuration template on github file (simple_form). You may have to update the configuration based on your needs and your design system. The configuration pretty much readable and I believe you can learn it from the code.

You may have a question something like this. "I'm using CSS Framework Bootstrap, it would be great if there is something like configuration for Bootstrap out of the box, do they have it?"

YES! There is configuration for Bootstrap out of the box, simply run the following command and it will generate the configuration based on Bootstrap classes

rails generate simple_form:install --bootstrap

Hold on, not only Bootstrap, you can also generate the configuration for Zurb Foundation 5

rails generate simple_form:install --foundation

Build Application

Scaffold Application

Let's continue to build a real word application and we will be using simple_form as a Form Builder.

Create a new rails application, in this example we will just using Bootstrap for the CSS Framework. We don't focus on the database for now so using sqlite is enough. Take a note that I'm using Rails 7 in this example.

rails new simple-form-example --css bootstrap

Once success, navigate to the application directory

cd simple-form-example

In this example, we will create posts page. There will be no authentication in this example. Let's scaffold the posts first which consisting of title and the body of the post

rails g scaffold Post title body:text

Then migrate the database by running rails db:migrate
Add a validation in Post model, open app/models/post.rb . We will add validation for title and body is required

class Post < ApplicationRecord
+  validates :title, presence: true
+  validates :body, presence: true
end

To run the application, we don't use rails s because it will only run the Puma server. Instead, run with the following command that will run the Puma server, esbuild, and SASS.

bin/dev

Open your browser and access http://localhost:3000/posts and then click New post link
Try to submit the form by clicking Create button, as you can see that there is error in the form

We will replace the built-in form with simple_form to take an advantage of simplicity and flexibility.

Install Simple Form

Add simple_form gem in your Gemfile. The run bundle install to install the gem

gem "simple_form", "~> 5.1"

Run the simple form generator for Bootstrap

rails generate simple_form:install --bootstrap

Now check the configuration on config/initializers/simple_form_bootstrap.rb and you'll see the configuration for Bootstrap Forms. Don't forget to stop or restart the server.
Open up the form file app/views/posts/_form.html.erb. Delete all the existing code and replace with the following code

<%= simple_form_for post do |form| %>
  <%= form.input :title %>
  <%= form.input :body %>
  <%= form.submit %>
<% end %>

As you can see, we replace the previous form to simple_form to just only 5 lines of code. It is very flexible isn't it
Run the server bin/dev and go to http://localhost:3000/posts/new. When you click the Create Post button you'll see the error validation little bit different than the previous form where the form message placed in under the text field. The advantage of this UI is the user will quickly knows that the error is for the specific field.

Understanding Simple Form

Under the hood, Simple Form building your form component based on the configuration. Now, let’s take a look back on config/initializers/simple_form_bootstrap.rb and find the :vertical_form wrapper, usually in line 52

config.wrappers :vertical_form, tag: 'div', class: 'form-group', error_class: 'form-group-invalid', valid_class: 'form-group-valid' do |b|

Look at the tag and class options above, this is where the field wrapped by the div element with the class is form-group. To prove that, go back to your form in the browser and inspect the *******Title******* field. That’s where the configuration is converted into the form. You could change the class or tag if you want, but remember to restart your server to apply the changes because you’re changing the file under the initializer directory.

<div class="form-group string required post_title">

vertical_form wrapper is the default input field component in Simple Form for Bootstrap

You may wonder how Simple Form generate the text field for the Title and textarea for the Body. The answer is pretty simple, behind the scene Simple Form maps the input type retrieved from the column definition in the database and then generates to a specific helper method. Check out the available input types on README.

Advance Usage

Change the default input type

In order to change the default input type, you need to check whether the type is supported. Take an example, you want to change the input type of Body to text field instead of textarea. Open up the _form.html.erb under the posts directory in the views, simply add as: :<type> after the field name. Check out List of available input types.

- <%= form.input :body %>
+ <%= form.input :body, as: :string %>

Go back to your browser, reload and see the result. Instead of textarea, it will rendered as text field.

Add custom class or style to form field

Yes, Simple Form comes with the flexibility by default. But, if you have a case where you need to add a style or classes in place you read in the right place. For instance, adding a css class to Body field wrapper to add a margin top. Open your form file and change as follow

- <%= form.input :body %>
+ <%= form.input :body, wrapper_html: { class: "mt-4" } %>

And here is the result, now see there is a margin between Title and Body field

If you check the generated HTML, the mt-4 class will added it along with the default class

And not only can change the wrapper attribute, you can also change the label and the input attribute with label_html and input_html respectively.

Can I hide the label text while still take an advantage of the other attributes?

Absolutely! Here is the example case where you don’t want to show the label text but still want to show the error message or the other while still using simple form. Let’s hide the label text for Title field.

- <%= form.input :title %>
+ <%= form.input :title, label: false %>

Check the result, as you can see the label has been omitted from the form

Conclusion

Not only build a really simple form, but you can also build a really complex form with Simple Form. Just think Simple form is just a mechanism to build a form components and you can reused it in anywhere in your application. You can download the example code on GitHub.

Containerizing a Rails Application

Maful — Thu, 17 Nov 2022 02:43:24 +0000

Introduction

If you actively working in web development, you may ever hear about docker and how docker makes the development are fun and the environment consistent, and isolated.

In this post, we will go through containerizing a Rails API application. Consider you already have a Rails application and you want the same environment for all your team while developing the app and here is the Docker coming in.

Prerequisites

Docker installed on your machine
Rails application

Create a Dockerfile

Dockerfile is a file that contains all the commands that users can call to build an image for the application. Since we will build an image for rails, we need to add ruby as the base image.

Create file Dockerfile (without extension) inside your root project and add ruby for the base image. You can choose another version that is suitable for your app (Ruby image tags).

FROM ruby:2.7.3

Create a user and install the required packages

# argument 
ARG WORK_DIR=/my_app
ARG USER=my_user
ARG HOME=/home/${USER}
ARG UID=999

# create user and user group
RUN groupadd -g ${UID} ${USER} && useradd -r -u ${UID} -g ${USER} ${USER}

# update os and install required packages
RUN apt-get update -qq && apt-get upgrade -y && apt-get install -y postgresql-client

then, copy all necessary files from the host machine and then install the app dependencies

# set WORK_DIR 
RUN mkdir ${WORK_DIR}
COPY Gemfile ${WORK_DIR}/Gemfile
COPY Gemfile.lock ${WORK_DIR}/Gemfile.lock
COPY . ${WORK_DIR}
RUN chown -R ${USER}:${USER} ${WORK_DIR}

# set HOME
RUN mkdir -p ${HOME}
RUN chown -R ${USER}:${USER} ${HOME}
WORKDIR ${WORK_DIR}

# set user before bundle install for dev permission issue
USER ${USER}
RUN bundle install

Add script and main process, and here we expose port 3000 from the image

# Add a script to be executed every time the container starts. 
USER ${USER}
COPY entrypoint.sh /usr/bin/
ENTRYPOINT ["./usr/bin/entrypoint.sh/"]
EXPOSE 3000

# Start the main process.
CMD ["rails", "server", "-b", "0.0.0.0"]

Here is the full our Dockerfile

FROM ruby:2.7.3

# argument
ARG WORK_DIR=/my_app
ARG USER=my_user
ARG HOME=/home/${USER}
ARG UID=999

# create user and user group
RUN groupadd -g ${UID} ${USER} && useradd -r -u ${UID} -g ${USER} ${USER}

# update os and install required packages
RUN apt-get update -qq && apt-get upgrade -y && apt-get install -y postgresql-client

# set WORK_DIR
RUN mkdir ${WORK_DIR}
COPY Gemfile ${WORK_DIR}/Gemfile
COPY Gemfile.lock ${WORK_DIR}/Gemfile.lock
COPY . ${WORK_DIR}
RUN chown -R ${USER}:${USER} ${WORK_DIR}

# set HOME
RUN mkdir -p ${HOME}
RUN chown -R ${USER}:${USER} ${HOME}
WORKDIR ${WORK_DIR}

# set user before bundle install for dev permission issue
USER ${USER}
RUN bundle install

# Add a script to be executed every time the container starts.
USER ${USER}
COPY entrypoint.sh /usr/bin/
ENTRYPOINT ["./usr/bin/entrypoint.sh/"]
EXPOSE 3000

# Start the main process.
CMD ["rails", "server", "-b", "0.0.0.0"]

The last thing, create entrypoint.sh in your root project and add this script.

#!/bin/bash
set -e

# Remove a potentially pre-existing server.pid for Rails.
rm -f /my_app/tmp/pids/server.pid

# Then exec the container's main process (what's set as CMD in the Dockerfile).
"${@}"

Docker Compose

When you want to run multiple docker containers, you should consider using Docker compose. If your app database is PostgreSQL, you can add a postgres image to compose file.

Create a file called docker-compose.yml in your project directory, and add our image above (Dockerfile) and postgres image for the database.

version: '3'
services:
  db:
    image: postgres:11
    ports:
      - 5432:5432
    environment:
      - POSTGRES_USER=docker
      - POSTGRES_PASSWORD=docker
    volumes:
      - db:/var/lib/postgresql/data

  app:
    build: .
    command: bash -c "bundle exec rails s -p 3000 -b '0.0.0.0'"
    entrypoint: ./entrypoint.sh
    volumes:
      - ./:/my_app
    ports:
      - 3000:3000
    environment:
      - DATABASE_USERNAME=docker
      - DATABASE_PASSWORD=docker
      - DATABASE_HOST=db
    depends_on:
      - db

volumes:
  db:
    external: true

Setting up configuration

If you notice in the compose file above, we set the environment variable for DATABASE_USERNAME, DATABASE_PASSWORD, and DATABASE_HOST. Then, update the config/database.yml for the default part.

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  database: <%= ENV.fetch("DATABASE_NAME") { "myapp" } %>
  timeout: <%= ENV.fetch("DATABASE_TIMEOUT") { 5000 } %>
  username: <%= ENV.fetch("DATABASE_USERNAME") { "docker" } %>
  password: <%= ENV.fetch("DATABASE_PASSWORD") { "docker" }%>
  host: <%= ENV.fetch("DATABASE_HOST") { "localhost" } %>
  port: <%= ENV.fetch("DATABASE_PORT") { 5432 } %>

Build and Run Application

After setting up file configuration, the next thing we have to do is Build services. Open your terminal and navigate to your project directory, run compose build to build or rebuild the services.

docker compose build

It will take some time to download the image from Docker Hub if you haven't ruby image installed. And then run the application.

docker compose up

if you got an error message around database setup things, run the database migration just like usual, the difference is you have to add docker command first

Setup database

docker compose run --rm app bundle exec rake db:setup

Migrate database

docker compose run --rm app bundle exec rake db:migrate

To stop the application, you can use Ctrl+C or docker compose down when you run the application in the background.

Conclusion

If this is your first time using docker, maybe you think you should spend more time configuring all the above things. But, the other thing is you just got new knowledge and when you work with your team, you will have the same environment for your entire team. Sounds great right?

In the next tutorial, you will learn how to create a brand new rails application with Docker. See you in the next one.

Looks up the IP, Nameservers, and CNAME using Go

Maful — Tue, 15 Nov 2022 03:13:32 +0000

Introduction

One of the advantages of using Go is Standard Library, Go has built in library that is already included when installing Go on your machine. There are many libraries you can use to build such as web application, rest API, command line interface (CLI), and many more just using Standard Library.

Today, I'm going to show you how to Lookup the IP, Nameservers, and CNAME for a particular host.

I assume you know a basic understanding of Go and setup GOPATH on your machine. Check this link about GOPATH https://github.com/golang/go/wiki/SettingGOPATH

Build

We are gonna using Go modules to create the app.

Create directory go-lookup.
Move to the go-lookup directory and run go mod init go-lookup to initialize the modules.

Then create a new file main.go inside the root directory. We're gonna use main.go to create the function and act as the main application also.

Here is the initial code inside main.go

package main

import "fmt"

func main() {
    fmt.Println("Hello from APP")
}

Build and run the program with go run main.go, make sure you are already in the directory.

$ go run main.go
Hello from APP

As you can see, the program will print Hello from APP using Println from the fmt package.

Lookup the DNS Records from the given domain name

Open main.go and remove the Println line as before, and now we're going to use the net package to lookup the DNS name. First, create a variable to store the domain name inside the main function.

- import "fmt"
+ import ( 
+     "fmt"
+     "net"
+ )

- fmt.Println("Hello from APP") 
+ // Define domain name 
+ host := "maful.web.id"
+ fmt.Printf("Domain: %s\n\n", host)

Then, use LookupNS function from the net package

// Lookup the DNS Records from the domain name 
ns, err := net.LookupNS(host)
if err != nil {
    panic(err)
}

LookupNS will return the pointer of the NS array, so we have to loop the ns variable to print the Nameserver. If an error occurs, stop the program immediately by using panic.

// Looping the Nameserver and ignore the index 
for _, v := range ns {
    fmt.Println(v.Host)
}
fmt.Println()

Finally, build and run again. You should see the list of Nameserver for the given domain name, in this case, is my domain.

$ go run main.go
ns1.vercel-dns.com.
ns2.vercel-dns.com.

Lookup the IP addresses

Secondly, we still use the same package to lookup the IP addresses with the LookupIP function.

fmt.Println("Looks up the IP Addresses") 
ips, err := net.LookupIP(host)
if err != nil {
    panic(err)
}
for i := range ips {
    fmt.Println(ips[i])
}
fmt.Println()

LookupIP will return a slice of bytes, so we just need to use index to get the IP. Here is the result so far when you build and run the program.

$ go run main.go

76.76.21.21

76.76.21.21 is the IP of the domain.

Lookup the CNAME

Lastly, we will use the LookupCNAME function from the net library to get CNAME (wikipedia).

fmt.Println("Looks up the CNAME") 
cname, err := net.LookupCNAME(host)
if err != nil {
    panic(err)
}

fmt.Println(cname)

We don't need looping like the previous one, we just print the cname directly because LookupCNAME returns string. Here is the result:

$ go run main.go

maful.web.id.

Conclusion

Finally, we have created a simple application using Go. Based on what we code before, we can refactor the code into several functions, but we're not doing in this tutorial.

package main

import (
    "fmt"
    "net"
)

func main() {
    // Define domain name
    host := "maful.web.id"
    fmt.Printf("Domain: %s\n\n", host)

    // Lookup the DNS Records from the domain name
    fmt.Println("Looks up the Nameservers")
    ns, err := net.LookupNS(host)
    if err != nil {
        panic(err)
    }
    // Looping the Nameserver and ignore the index
    for _, v := range ns {
        fmt.Println(v.Host)
    }
    fmt.Println()

    fmt.Println("Looks up the IP Addresses")
    ips, err := net.LookupIP(host)
    if err != nil {
        panic(err)
    }
    for i := range ips {
        fmt.Println(ips[i])
    }
    fmt.Println()

    fmt.Println("Looks up the CNAME")
    cname, err := net.LookupCNAME(host)
    if err != nil {
        panic(err)
    }

    fmt.Println(cname)
}

Full result

$ go run main.go

Domain: maful.web.id

Looks up the Nameservers
ns2.vercel-dns.com.
ns1.vercel-dns.com.

Looks up the IP Addresses
76.76.21.21

Looks up the CNAME
maful.web.id.

Build REST API with Go Fiber and PlanetScale - Part 4

Maful — Wed, 19 Oct 2022 22:40:46 +0000

Update a user

Again, add a new function called UpdateUser in the users handler.

func UpdateUser(c *fiber.Ctx) error {
    // first, check if the user is exist
    user := models.User{}
    if err := models.DB.First(&user, "id = ?", c.Params("id")).Error; err != nil {
        return c.Status(http.StatusNotFound).JSON(&fiber.Map{
            "message": "Record not found!",
        })
    }

    // second, parse the request body
    request := &updateUserRequest{}
    if err := c.BodyParser(request); err != nil {
        return c.Status(http.StatusBadRequest).JSON(&fiber.Map{
            "message": err.Error(),
        })
    }

    // third, update the user
    updateUser := models.User{
        Name:    request.Name,
        Email:   request.Email,
        Website: request.Website,
    }
    models.DB.Model(&user).Updates(&updateUser)

    return c.Status(http.StatusOK).JSON(&fiber.Map{
        "user": user,
    })
}

app.Put("/users/:id", handlers.UpdateUser)

now, re-run the application. Update the user that we created before.

$ curl --location --request PUT 'http://localhost:3000/users/1' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "Machine name"
}'

response

{
  "user": {
    "ID": 1,
    "CreatedAt": "2021-09-08T08:07:25.042+07:00",
    "UpdatedAt": "2021-09-08T08:15:52.249+07:00",
    "DeletedAt": null,
    "name": "Machine name",
    "email": "joh@example.com",
    "website": "google.com"
  }
}

when the user doesn't exist

$ curl --location --request PUT 'http://localhost:3000/users/100' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "Machine name"
}'

response

{
  "message": "Record not found!"
}

Delete a user

Add the delete user function at the bottom of the users handler.

func DeleteUser(c *fiber.Ctx) error {
    // first, check if the user is exist
    user := models.User{}
    if err := models.DB.First(&user, "id = ?", c.Params("id")).Error; err != nil {
        return c.Status(http.StatusNotFound).JSON(&fiber.Map{
            "message": "Record not found!",
        })
    }

    // second, delete the user
    models.DB.Delete(&user)

    return c.Status(http.StatusOK).JSON(&fiber.Map{
        "message": "Success",
    })
}

app.Delete("/users/:id", handlers.DeleteUser)

so, create a new user again

$ curl --location --request POST 'http://localhost:3000/users' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "Badu",
    "email": "joh@example.com",
    "website": "google.com"
}'

see the id from the response, we will delete that user

$ curl --location --request DELETE 'http://localhost:3000/users/2'

response

{
  "message": "Success"
}

Summary

PlanetScale offers Developer plan pricing that you can use for the development lifecycle and it's completely FREE. You can create up to 3 databases and 3 branches for each database. Basically, this is going to be new knowledge for the developer who never use a serverless database and with a new workflow how to make a schema.

Fiber is a great web framework to build application in Go, they are fast, have rich features and the documentation is good.

This post, it's just a simple web API application to give a basic understanding of how to use Fiber and PlanetScale database. In the next one, we are going to build a more complex web API with the same tech stacks.

Download the full source code on this repository.

Thank you.

Build REST API with Go Fiber and PlanetScale - Part 3

Maful — Wed, 19 Oct 2022 11:36:38 +0000

Deploy the schema

Now, it is time to deploy the add-users-table branch into the main branch

$ pscale deploy-request create fiber-pscale add-users-table
Deploy request #1 successfully created.

Now check your dashboard, you'll see here that you requested schema changes from the development branch to the main branch. If you are familiar with Git, it is similar to Pull Request.

Now, approve the request and PlanetScale automatically apply the new schema into the main branch without downtime.

$ pscale deploy-request deploy fiber-pscale 1
Successfully deployed dhacze78ukhv from add-users-table to main.

Now, back and check the dashboard, it's deployed!🎊

if you no longer need to make schema changes from the branch, now you can safely delete it.

$ pscale branch delete fiber-pscale add-users-table

Handlers

Create a file called users.go inside the handlers directory. And now, create a connection to the main branch so the application can communicate with the database.

$ pscale connect fiber-pscale main --port 3309

Note: It's recommended to use a secure connection while connecting to the main branch once you deployed your application. More https://docs.planetscale.com/reference/secure-connections

Create a user

Create a function CreateUser and initialize createUserRequest struct in users.go.

package handlers

import (
    "net/http"

    "github.com/gofiber/fiber/v2"
    "github.com/maful/fiber-pscale/models"
)

type createUserRequest struct {
    Name    string `json:"name" binding:"required"`
    Email   string `json:"email" binding:"required"`
    Website string `json:"website" binding:"required"`
}

func CreateUser(c *fiber.Ctx) error {
    req := &createUserRequest{}
    if err := c.BodyParser(req); err != nil {
        return c.Status(http.StatusBadRequest).JSON(&fiber.Map{
            "message": err.Error(),
        })
    }
    user := models.User{
        Name:    req.Name,
        Email:   req.Email,
        Website: req.Website,
    }
    models.DB.Create(&user)
    return c.Status(fiber.StatusCreated).JSON(&fiber.Map{
        "user": user,
    })
}

open main.go, add a new handler to call that function

import (
    // ...
    "github.com/maful/fiber-pscale/handlers"
)

// app.Get("/" ...
app.Post("/users", handlers.CreateUser)

run the app, and try to create the user

$ curl --location --request POST 'http://localhost:3000/users' \
--header 'Content-Type: application/json' \
--data-raw '{
    "name": "John",
    "email": "john@example.com",
    "website": "https://example.com"
}'

response

{
  "user": {
    "ID": 1,
    "CreatedAt": "2021-09-06T20:04:31.022+07:00",
    "UpdatedAt": "2021-09-06T20:04:31.022+07:00",
    "DeletedAt": null,
    "name": "John",
    "email": "john@example.com",
    "website": "https://example.com"
  }
}

List all users

Create a new function called GetUsers below the create function.

func GetUsers(c *fiber.Ctx) error {
    var users []models.User
    models.DB.Find(&users)

    return c.Status(http.StatusOK).JSON(&fiber.Map{
        "users": users,
    })
}

app.Get("/users", handlers.GetUsers)
// app.Post("/users...

stop the existing app, and run again. You should always do this because the app doesn't refresh automatically when we made changes.

curl --location --request GET 'http://localhost:3000/users'

response

{
  "users": [
    {
      "ID": 1,
      "CreatedAt": "2021-09-06T20:04:31.022+07:00",
      "UpdatedAt": "2021-09-06T20:04:31.022+07:00",
      "DeletedAt": null,
      "name": "John",
      "email": "john@example.com",
      "website": "https://example.com"
    }
  ]
}

View user

Add a new function called GetUser in the bottom.

func GetUser(c *fiber.Ctx) error {
    var user models.User
    if err := models.DB.First(&user, "id = ?", c.Params("id")).Error; err != nil {
        return c.Status(http.StatusNotFound).JSON(&fiber.Map{
            "message": "Record not found!",
        })
    }

    return c.Status(http.StatusOK).JSON(&fiber.Map{
        "user": user,
    })
}

and add a new handler in main.go

// ...
app.Get("/users", handlers.GetUsers)
app.Get("/users/:id", handlers.GetUser) // new

now, we try to get the user details with id 1

curl --location --request GET 'http://localhost:3000/users/1'

response

{
  "user": {
    "ID": 1,
    "CreatedAt": "2021-09-06T20:04:31.022+07:00",
    "UpdatedAt": "2021-09-06T20:04:31.022+07:00",
    "DeletedAt": null,
    "name": "John",
    "email": "john@example.com",
    "website": "https://example.com"
  }
}

if try to get the id that doesn't exist

curl --location --request GET 'http://localhost:3000/users/100'

{
  "message": "Record not found!"
}

Build REST API with Go Fiber and PlanetScale - Part 2

Maful — Wed, 19 Oct 2022 11:24:39 +0000

Models

Create a User model and create a file called user.go inside models directory, and define a struct with gorm.

package models

import "gorm.io/gorm"

// User struct
type User struct {
    gorm.Model
    Name    string `json:"name"`
    Email   string `json:"email"`
    Website string `json:"website"`
}

Connect to Database

Create a development database branch called add-users-table.

$ pscale branch create fiber-pscale add-users-table

Open a new terminal tab, we're going to connect to the database inside add-users-table branch and listen to 3309 PORT. See more Connect using client certificates.

$ pscale connect fiber-pscale add-users-table --port 3309

Create a file called database.go inside the models directory and add a function to connect to the database.

package models

import (
    "gorm.io/driver/mysql"
    "gorm.io/gorm"
)

var DB *gorm.DB

func ConnectDatabase() {
    // refer https://github.com/go-sql-driver/mysql#dsn-data-source-name for details
    dsn := "root:@tcp(127.0.0.1:3309)/fiber-pscale?charset=utf8mb4&parseTime=True&loc=Local"
    database, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        panic("failed to connect database")
    }
    // Migrate the users table
    database.AutoMigrate(&User{})
    DB = database
}

open your main.go, and call the ConnectDatabase function to migrate the tables and connect to the database.

import (
    // ...
    "github.com/maful/fiber-pscale/models"
)

func main() {
    // ...
    models.ConnectDatabase() // New

    //
    app.Use(logger.New())
}

then run the app go run cmd/main.go, Gorm will automatically migrate the table into add-users-table branch. How to know if the migration is success? You can check in the PlanetScale dashboard for the following branch, or use CLI to see the schema for add-users-table branch.

$ pscale branch schema fiber-pscale add-users-table
-- users --
CREATE TABLE `users` (
  `id` bigint unsigned NOT NULL AUTO_INCREMENT,
  `created_at` datetime(3) DEFAULT NULL,
  `updated_at` datetime(3) DEFAULT NULL,
  `deleted_at` datetime(3) DEFAULT NULL,
  `name` longtext,
  `email` longtext,
  `website` longtext,
  PRIMARY KEY (`id`),
  KEY `idx_users_deleted_at` (`deleted_at`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci;

The schema won't apply to the main branch until you create a deploy request. Lastly, stop the connection from add-users-table branch or Ctrl+C.

Build REST API with Go Fiber and PlanetScale - Part 1

Maful — Wed, 19 Oct 2022 09:25:58 +0000

Introduction

Fiber (gofiber.io) is web framework written in Go that much like Express in Nodejs. Fiber have many built in features to build rich web application such as Middleware, API Ready, Template Engine, Websocket support, Rate limiter, etc. They used fasthttp for the engine that claims is the fastest HTTP Engine in Go.

Today, we are going to build a simple REST API with Fiber and integrated it with PlanetScale for the database.

PlanetScale is serverless database platform for developers that built on top Vitess.

Set up PlanetScale

Go to https://planetscale.com and sign up, then setup your account such as organization, etc. Then, we'll create a database via PlanetScale CLI, you can also create the database via web.

I don't want to explain how to setup the CLI, PlanetScale's team has managed to explain it very clearly. Check out the documentation PlanetScale CLI.

Once success to setup the CLI, then create a database called fiber-pscale and specify the region of your database. I recommended to using the region that close that where you are. In my case I would like to use Asia Pacific Singapore. You can check the list of region, there are 6 regions at the moment.

$ pscale region list
  NAME (6)                 SLUG           ENABLED
 ------------------------ -------------- ---------
  US East                  us-east        Yes
  US West                  us-west        Yes
  EU West                  eu-west        Yes
  Asia Pacific Mumbai      ap-south       Yes
  Asia Pacific Singapore   ap-southeast   Yes
  Asia Pacific Tokyo       ap-northeast   Yes

Create database

$ pscale database create fiber-pscale --region ap-southeast
Database fiber-pscale was successfully created.

Set up Application

We are going to use go modules, first we create empty directory and then initialize the modules. You can use your repo url or application name. In this case, I'm using my repo url.

mkdir fiber-pscale && cd fiber-pscale
go mod init github.com/maful/fiber-pscale

Then, there are 3 packages we're going to build. Package means the directory. Create thress directories.

cmd: this is the root application for initialize the Fiber and connect to PlanetScale database
handler: contains all handler for the routing such as list data, create, etc
models: contains data structure for the application

mkdir cmd
mkdir handlers
mkdir models

Since we are going to use Fiber and PlanetScale, install the modules

To access PlanetScale, we can use ORM from Go called Gorm, and since the PlanetScale database is built on top Vitess (MySQL), install the MySQL driver as well.

# install fiber
$ go get -u github.com/gofiber/fiber/v2

# install gorm to connect to planetscale database
$ go get -u gorm.io/gorm

# mysql driver
$ go get -u gorm.io/driver/mysql

Basic Handler

write basic handler with Fiber, create file called main.go inside cmd directory and add this script.

package main
import (
    "net/http"
    "github.com/gofiber/fiber/v2"
    "github.com/gofiber/fiber/v2/middleware/logger"
)
func main() {
    app := fiber.New(fiber.Config{
        AppName:      "Fiber with Planetscale",
        ServerHeader: "Fiber",
    })
    app.Use(logger.New())
    app.Get("/", func(c *fiber.Ctx) error {
        return c.Status(http.StatusOK).JSON(&fiber.Map{
            "message": "Hello world",
        })
    })
    app.Listen(":3000")
}

it will create a new instance of fiber server, and listen to PORT 3000, back to terminal and run the main.go.

$ go run cmd/main.go
 ┌───────────────────────────────────────────────────┐
 │              Fiber with Planetscale               │
 │                   Fiber v2.18.0                   │
 │               http://127.0.0.1:3000               │
 │       (bound on host 0.0.0.0 and port 3000)       │
 │                                                   │
 │ Handlers ............. 3  Processes ........... 1 │
 │ Prefork ....... Disabled  PID ............. 22443 │
 └───────────────────────────────────────────────────┘

and then you can access it at http://localhost:3000, below is an example with the curl command.

$ curl --location --request GET 'http://localhost:3000'
{"message":"Hello world"}

you could use Postman or other similar software if prefer to use GUI.