DEV Community: Camillo Visini

Phlex for Rails Emails: Action Mailer without ERB

Camillo Visini — Thu, 27 Feb 2025 09:00:00 +0000

Writing Ruby all day is fun, but writing Ruby in HTML is not. In the past months, I found switching from Ruby to ERB views and partials to be interrupting my flow and therefore increasingly annoying. Enter Phlex, which is a Ruby library that allows you to write HTML (components, views, layouts) in pure Ruby. After already completely ditching ERB for Phlex in Rails views for multiple applications, I wanted to see if I could do the same for Action Mailer. ERB be gone!

Why might this be interesting, besides just trying it out for the sake of it, and to see if it's possible? I see these benefits:

No more ERB: Use Phlex all the way, even for mailers
Manage the HTML and text version of the email in one place
Use Phlex components for layouts and views

As for my part, I could gladly live without ERB in my Rails applications. After all, if it does not spark joy when you touch it, you should get rid of it.

Check out the repository for the full code. I hope you find this as exciting as I do!

https://github.com/visini/rails-phlex-action-mailer

Let's begin by setting up a mailer in our Rails app so we can demonstrate how to move everything to Phlex.

The ERB Way

The UserMailer class is our entry point into Action Mailer, which handles email delivery in Rails. Nothing special to see here, this is Rails 101. It's basically what we get when we generate a mailer (e.g., rails g mailer UserMailer).

# app/mailers/user_mailer.rb
class UserMailer < ApplicationMailer
  def welcome
    @user = params[:user]
    mail(to: @user.email, subject: "Welcome")
  end
end

We also need some corresponding view files or "email templates". Not all email clients render HTML, so we need to provide both a HTML and a text version. We can use ERB to use Ruby code in the templates (for example, to access instance variables or to use helper methods). In terms of files, this means that we need one template for the HTML version:

<div>We are excited to have you join us! Thank you for signing up.</div>

<%= link_to 'Click here to sign in', 'https://example.com' %>

And one for the text version:

We are excited to have you join us! Thank you for signing up.

Click here to sign in: https://example.com

We also need to add some basic layouts. In the layout, you might add a logo at the top of the email, place links in a footer, add contact information, and so on.

Because these parts are in the layout, they become easy to manage: When you want to change the layout, you can change everything in one place. For now, we keep it simple, and just add the application name and a "signature" at the end, to demonstrate how the layout wraps around the email content.

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body>
    <div><b>Application Name</b></div>

    <%= yield %>

    <div>Best Regards,</div>
    <div>Company Name</div>
  </body>
</html>

For the text version, we mirror the of the HTML version. Since we can't use HTML, we need to keep it simple and work with plain text. We may use line breaks or adding horizontal rules to separate sections, but for now we keep it simple:

Application Name

<%= yield %>

Best Regards,
Company Name

The above template plus their layout produces the following email messages. First, the HTML version:

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body>
    <div><b>Application Name</b></div>

    <div>We are excited to have you join us! Thank you for signing up.</div>

    <a href="https://example.com">Click here to sign in</a>

    <div>Best Regards,</div>
    <div>Company Name</div>
  </body>
</html>

And the text version:

Application Name

We are excited to have you join us! Thank you for signing up.

Click here to sign in: https://example.com

Best Regards,
Company Name

As you can see, the email content is rendered with the layout wrapped around it.

By creating a test, we can assert the email content for the HTML and text version via fixtures. We can leverage the read_fixture helper (see the Rails Guide on Testing) to load the fixture files and compare them to the email content. We create a simple helper method assert_mailer_fixtures to conveniently assert both versions.

# test/mailers/user_mailer_test.rb
require "test_helper"

class UserMailerTest < ActionMailer::TestCase
  test "welcome" do
    user = User.new(email: "to@example.com")
    email = UserMailer.with(user: user).welcome

    assert_equal ["from@example.com"], email.from
    assert_equal ["to@example.com"], email.to
    assert_equal "Welcome", email.subject

    assert_mailer_fixtures("welcome", email)
  end

  private

  # This checks the email content against the fixtures.
  # For example, for UserMailer#welcome, it checks against:
  # test/fixtures/user_mailer/welcome.txt
  # test/fixtures/user_mailer/welcome.html
  def assert_mailer_fixtures(name, email)
    email_body_html = email.html_part.body.to_s
    email_body_text = email.text_part.body.to_s.strip
    expected_html = read_fixture("#{name}.html").join.strip
    expected_text = read_fixture("#{name}.txt").join.strip
    fixture_html = Nokogiri::HTML.parse(expected_html).to_html.strip
    email_html = Nokogiri::HTML.parse(email_body_html).to_html.strip
    assert_dom_equal fixture_html, email_html
    assert_equal expected_text, email_body_text
  end
end

We now have a basic setup for sending emails with Action Mailer – the ERB way. The next step is to refactor the views and layouts to use Phlex. This test ensures that while we refactor, we produce exactly the same email content (HTML and text) while we move from ERB to Phlex.

Using Phlex Views for Email Templates

To install Phlex in our Rails app, we follow the installation instructions. In particular, we run the install generator to create base classes and directories for our views (the generator provides an opinionated starting point with a folder structure and naming convention for views and components).

Then, we can create the Phlex views for the email templates. We might place all mailer views under app/views/mailers and create a subfolder for each mailer. Similar to with the ERB templates, we need to create a view for the HTML version and one for the text version. First, we create the HTML view:

# app/views/mailers/users/welcome/html.rb
class Views::Mailers::Users::Welcome::Html < Views::Base
  def initialize(user)
    @user = user
  end

  def view_template
    div do
      plain "We are excited to have you join us! Thank you for signing up."
    end

    a href: "https://example.com" do
      plain "Click here to sign in"
    end
  end
end

And then the text view:

# app/views/mailers/users/welcome/text.rb
class Views::Mailers::Users::Welcome::Text < Views::Base
  def initialize(user)
    @user = user
  end

  def view_template
    plain "We are excited to have you join us! Thank you for signing up."
    plain "\n\n"
    plain "Click here to sign in: https://example.com"
  end
end

Inside the ERB templates, we render the Phlex views. For the HTML version:

<!-- app/views/user_mailer/welcome.html.erb -->
<%= render Views::Mailers::Users::Welcome::Html.new(user: @user) %>

And for the text version:

<!-- app/views/user_mailer/welcome.text.erb -->
<%= render Views::Mailers::Users::Welcome::Text.new(user: @user) %>

Now what? This seems like a tiny step forward. We have the same content, only now it's rendered via Phlex views instead of ERB templates, but we still need the ERB files as an "entry point". Even worse, we have gone from two files to four files for the templates, which adds overhead – not ideal! But we are not done yet. Let's have a look at layouts next, Phlex style.

Setting up Phlex Layouts for Mailers

Similar to the ERB layouts, we can create Phlex layouts for our email templates. But first, let's tell Action Mailer to not use any layouts, because we will compose the templates on our own from layouts and views with Phlex.

# app/mailers/application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  default from: "from@example.com"
  layout false
end

We can delete app/views/layouts/mailer.html.erb and app/views/layouts/mailer.text.erb as they are no longer needed. Instead, we create Phlex layouts for the email templates – you could use a tool like phlexing.fun to convert HTML to Phlex. First, we create a layout for the HTML version:

# app/components/layouts/mailers/html.rb
class Components::Layouts::Mailers::Html < Phlex::HTML
  def view_template(&block)
    doctype
    html do
      head do
        meta charset: "utf-8"
      end
      body do
        div do
          b { "Application Name" }
        end
        yield block
        div { "Best Regards," }
        div { "Company Name" }
      end
    end
  end
end

And one for the text version:

# app/components/layouts/mailers/text.rb
class Components::Layouts::Mailers::Text < Phlex::HTML
  def view_template(&block)
    plain "Application Name"
    plain "\n\n"
    yield block
    plain "\n\n\n"
    plain "Best Regards,"
    plain "\n"
    plain "Company Name"
  end
end

In the views, we use the around_template hook to tell the view to render with the particular layout (see Phlex layout docs).

For the HTML variant, this means we need something like this:

# app/views/mailers/users/welcome/html.rb
class Views::Mailers::Users::Welcome::Html < Views::Base
  include Components

  def initialize(user)
    @user = user
  end

  def around_template
    render Components::Layouts::Mailers::Html.new do
      super
    end
  end

  def view_template
    div do
      plain "We are excited to have you join us! Thank you for signing up."
    end

    a href: "https://example.com" do
      plain "Click here to sign in"
    end
  end
end

And for the text variant:

# app/views/mailers/users/welcome/text.rb
class Views::Mailers::Users::Welcome::Text < Views::Base
  include Components

  def initialize(user)
    @user = user
  end

  def around_template
    render Components::Layouts::Mailers::Text.new do
      super
    end
  end

  def view_template
    plain "We are excited to have you join us! Thank you for signing up."
    plain "\n\n"
    plain "Click here to sign in: https://example.com"
  end
end

Now we have the email content rendered with the layout wrapped around it. But, for each email template, we need to repeat a lot of the same code (e.g., explicitly defining around_template hook for every single templates, duplicated initialize for HTML and text variant). We're not there yet, though. In order to DRY up the code, we need to refactor the views and how we're rendering the emails.

Refactoring ApplicationMailer and Views

Here comes the cool part. We can refactor the ApplicationMailer class to render the email content with the layout by way of a custom method render_email (we use layouts through composition). This way, we can remove the around_template hook which is cluttering up the views and thereby make them more compact. Also, we go from two separate views to one view class with two subclasses for the HTML and text version. No need to always remember to also change the text version when updating the HTML version – it's all in one place and almost impossible to miss. Nice!

Let's first have a look at the refactored ApplicationMailer class:

# app/mailers/application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  default from: "from@example.com"
  layout false

  def render_email(view_class, subject:, to:, view_params: {})
    mail({subject:, to:, content_type: "multipart/alternative"}) do |format|
      format.html {
        render Components::Layouts::Mailers::Html.new {
          render view_class.const_get(:Html).new(**view_params)
        }
      }
      format.text {
        render Components::Layouts::Mailers::Text.new {
          render view_class.const_get(:Text).new(**view_params)
        }
      }
    end
  end
end

The render_email method takes the view class and additional arguments, forwarding them to the Action Mailer mail method. In the block, it then renders the email content with the layout for the HTML and text versions.

Here's how we render the email from the mailer class:

# app/mailers/user_mailer.rb
class UserMailer < ApplicationMailer
  def welcome
    @user = params[:user]
    render_email(
      Views::Mailers::Users::Welcome,
      subject: "Welcome",
      to: @user.email,
      view_params: {user: @user}
    )
  end
end

In the views, we now centralize the HTML and text content in one class. It's easy to manage and change the content in one place, no need to jump between different files.

# app/views/user_mailer/welcome.rb
class Views::Mailers::Users::Welcome < Views::Base
  def initialize(user:)
    @user = user
  end

  class Html < self
    def view_template
      div do
        plain "We are excited to have you join us! Thank you for signing up."
      end

      a href: "https://example.com" do
        plain "Click here to sign in"
      end
    end
  end

  class Text < self
    def view_template
      plain "We are excited to have you join us! Thank you for signing up."
      plain "\n\n"
      plain "Click here to sign in: https://example.com"
    end
  end
end

We can now delete these four files:

app/views/mailers/users/welcome/html.rb
app/views/mailers/users/welcome/text.rb
app/views/user_mailer/welcome.html.erb
app/views/user_mailer/welcome.text.erb

By deleting the email templates (and before that, the layouts), we have removed all ERB and now can write our emails in plain Ruby with Phlex.

Inlining Styles

Let's add some finishing touches – after all, we want our emails to look good. But, email clients have limited support for CSS stylesheets, so we need to use inline styles. We can add a style tag to our HTML layout to get started. Something like this:

# app/components/layouts/mailers/html.rb
class Components::Layouts::Mailers::Html < Phlex::HTML
  def view_template(&block)
    doctype
    html do
      head do
        meta charset: "utf-8"
        style do
          plain <<~CSS
            body {
              font-family: Arial, sans-serif;
            }
            .highlight {
              background-color: yellow;
            }
            .signature {
              font-style: italic;
            }
          CSS
        end
      end
      body do
        div do
          b { "Application Name" }
        end
        yield block

        div { "Best Regards," }
        div(class: "signature") { "Company Name" }
      end
    end
  end
end

However, for emails, it's generally recommended to inline the styles of all classes we use by adding the styles to the respective elements. Let's add the roadie-rails gem to the mix, so that classes like highlight and signature are inlined with their actual style values (this works for both the layout, as shown above with signature, as well as within the views, e.g., when you use highlight for a particular element).

# app/mailers/application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  include Roadie::Rails::Automatic

  default from: "from@example.com"
  layout false

  #...
end

With roadie-rails in place, we can now use classes in our views and layouts, and the styles will be inlined automatically. Note that roadie-rails inlines the styles only when delivering – use UserMailer.with(user: user).welcome.deliver in Rails console to check the inlined styles. Here's an example of the resulting HTML email content:

<html>
  <!-- ... -->
   <body style="font-family:Arial, sans-serif">
      <div><b>Application Name</b></div>
      <div class="highlight" style="background-color:yellow">We are excited to have you join us! Thank you for signing up.</div>
      <a href="https://example.com">Click here to sign in</a>
      <div>Best Regards,</div>
      <div class="signature" style="font-style:italic">Company Name</div>
   </body>
</html>

Conclusion

By moving from ERB to Phlex for Action Mailer, we can write our email templates in pure Ruby. We can manage the HTML and text version in one place, and use Phlex components for layouts and views. Not a single ERB file left in our entire Rails app! Check out the repository for the full code:

https://github.com/visini/rails-phlex-action-mailer

I hope this is helpful for those looking into using Phlex for mailers as the last piece of the puzzle to get rid of ERB completely. If you have any questions or feedback, feel free to reach out.

Keep Phlexing!

Mocking MQTT with Playwright: Simplify Your System Tests

Camillo Visini — Fri, 17 Jan 2025 23:00:00 +0000

Leverage WebSocket interception to eliminate the need for a MQTT broker in your Playwright test environment

Testing real-time web applications often involves complex setups, for example, when your app relies on a protocol like MQTT for publish/subscribe messaging. Managing a MQTT broker "just for tests" adds unnecessary overhead to your development and continuous integration environments. With Playwright, it's possible to intercept WebSocket traffic when your test suite runs. This allows us to mock the MQTT interactions over WebSockets, thus giving us a way to run system tests without the need of spinning up an actual MQTT broker.

In this post, I explain how to mock MQTT traffic during Playwright system tests for a tiny demo application built for this purpose. The demo application uses Vue and MQTT.js.

Repository: https://github.com/visini/mqtt-playwright-example

Setting Up a Local Mosquitto Broker

To build the demo application where we will later demonstrate how to mock MQTT-over-WebSockets, we need a real MQTT broker. Mosquitto is easy to set up via Docker. Here's a minimal Docker Compose configuration to get started. Add the following to your docker-compose.yml file:

services:
  mosquitto:
    image: eclipse-mosquitto:latest
    container_name: mosquitto
    ports:
      - "1883:1883"
      - "8080:8080"
    volumes:
      - ./mosquitto.conf:/mosquitto/config/mosquitto.conf

Next, create a mosquitto.conf file to enable MQTT-over-WebSockets (served on port 8080):

listener 8080
protocol websockets
allow_anonymous true

Start it up with:

docker compose up

This will serve as a real MQTT broker for our demo application. The demo application will connect to this MQTT broker to send and receive messages. Eventually, we will mock these MQTT communication in our Playwright system tests, but before that, we have to build a tiny app that we can then write tests for.

A Minimal Vue App with MQTT

Here's a very simple Vue component which subscribes to an MQTT topic via MQTT.js. It's a message broadcaster, meaning whatever you type in the input field will be broadcasted to all subscribers (including itself). The component subscribes to a topic, in this case demo-broadcaster/messages, via MQTT-over-WebSockets, and displays all messages received in a list. Good enough for our purposes – that is, demonstrating how to mock MQTT in system tests run via Playwright!

<script setup lang="ts">
import { ref } from "vue"
import mqtt from "mqtt"

const brokerUrl = "ws://localhost:8080"
const topic = "demo-broadcaster/messages"
const messages = ref<string[]>([])
const inputMessage = ref("")

const client = mqtt.connect(brokerUrl)

client.on("connect", () => {
  client.subscribe(topic, (err) => {
    if (err) console.error("Subscription failed:", err)
  })
})

client.on("message", (_topic, message) => {
  messages.value.push(message.toString())
})

const sendMessage = () => {
  if (!inputMessage.value.trim()) return
  client.publish(topic, inputMessage.value)
  inputMessage.value = ""
}
</script>

<template>
  <h1>MQTT Message Broadcaster</h1>
  <input v-model="inputMessage" placeholder="Type your message" />
  <button @click="sendMessage">Send</button>
  <ul>
    <li v-for="(message, index) in messages" :key="index">
      {{ message }}
    </li>
  </ul>
</template>

This component (together with the real MQTT broker Mosquitto we started via Docker) simulates a realistic application for which we would want to run system tests with mocked MQTT communication, thereby eliminating the need for a running broker during tests.

Understanding how MQTT-over-WebSockets works

To reliably mock MQTT interactions in Playwright, we need to replicate the exact packet flow the client (in our case, MQTT.js) expects during an MQTT session. This goes beyond simply opening a WebSocket connection; you must provide valid MQTT responses for every major step – CONNECT, SUBSCRIBE, and PUBLISH – just as a real broker would. Here's the basic flow:

CONNECT and CONNACK → When an MQTT client first establishes a connection, it sends a CONNECT packet containing details like the client ID and protocol version. The broker must respond with a CONNACK packet to confirm the connection. Failing to send a CONNACK (or sending an invalid one) means the client library will assume the broker is offline or misconfigured, and the connection will fail.
SUBSCRIBE and SUBACK → After connecting, the client typically subscribes to one or more topics. That subscription is initiated with a SUBSCRIBE packet indicating which topics the client wants to receive messages for. A valid broker replies with a SUBACK packet to confirm that the subscription is acknowledged. Without this step, the client library won't treat the subscription as active, and no messages will be delivered. We need to replicate this behavior in our mock broker.
PUBLISH → To emulate messages being published to topics, we should handle all PUBLISH packets that the client sends. The simplest approach is to echo these messages right back via a PUBLISH packet to simulate new inbound messages. If your test scenario involves different MQTT QoS levels or retained messages, you can extend this logic to return additional properties that replicate real-world behavior more closely.

The MQTT protocol has a very specific byte-level format for each packet type. Instead of crafting all these packets by hand, libraries like mqtt-packet parse incoming frames into JavaScript objects and generate MQTT-compliant buffers for responses. This means you don't have to manually figure out each packet's binary layout – mqtt-packet does this job, ensuring you're always sending and receiving data that conforms to the MQTT spec.

By recreating these MQTT flows within a WebSocket route, you're effectively standing up a minimal broker implementation in your test environment. The result is a more reliable, self-contained system test suite that doesn't rely on an external MQTT server – yet still offers the same publish/subscribe behavior your application code expects.

Mocking MQTT with Playwright

To bring all of this together in Playwright, we rely on page.routeWebSocket to intercept the WebSocket connection the MQTT client establishes. See more details in Playwright's docs about WebSocket routes. Once intercepted, we feed any incoming data through mqtt-packet's parser, which turns raw bytes into JavaScript objects. From there, we can match the MQTT command (connect, subscribe, publish, etc.) and generate the corresponding response packet with mqtt-packet.generate. This ensures the client sees exactly what it would expect from a real broker – CONNACK to confirm a connection, SUBACK to acknowledge subscriptions, and PUBLISH messages when the client sends or should receive them.

In the following code example, the mockMQTT function sets up this interception and MQTT packet handling. We then perform a basic scenario in our demo application by filling in an input field, sending the message, and asserting that our messages make it into the DOM. Under the hood, there's no actual broker running – Playwright and mqtt-packet orchestrate the entire flow, everything is mocked. This approach keeps our tests completely self-contained, while still delivering a realistic MQTT-over-WebSockets experience for the client code.

import { test, expect } from "@playwright/test"
import mqttPacket from "mqtt-packet"

const brokerUrl = "ws://localhost:8080"

async function mockMQTT(page) {
  await page.routeWebSocket(brokerUrl, (ws) => {
    // Create MQTT parser and generator
    const parser = mqttPacket.parser({ protocolVersion: 5 })
    const generate = mqttPacket.generate

    parser.on("packet", (packet) => {
      if (packet.cmd === "connect") {
        // Confirm connection
        ws.send(
          generate({
            cmd: "connack",
            returnCode: 0,
            sessionPresent: false,
          }),
        )
      } else if (packet.cmd === "subscribe") {
        // Confirm subscription
        ws.send(
          generate({
            cmd: "suback",
            messageId: packet.messageId,
            granted: [0],
          }),
        )
      } else if (packet.cmd === "publish") {
        // Echo the message back
        ws.send(
          generate({
            cmd: "publish",
            topic: packet.topic,
            payload: packet.payload,
            qos: 0,
          }),
        )
      }
    })

    // Parse incoming messages
    ws.onMessage((raw) => {
      parser.parse(raw)
    })
  })
}

test("MQTT messaging works with a mocked broker", async ({ page }) => {
  await mockMQTT(page) // Intercept all MQTT traffic
  await page.goto("/")

  // Send first message
  await page.fill("input", "Hello from mocked broker")
  await page.click("button")

  // Send second message
  await page.fill("input", "Another test message")
  await page.click("button")

  // Check that two messages are listed
  const messages = page.locator("li")
  await expect(messages).toHaveCount(2)
  await expect(messages.nth(0)).toHaveText("Hello from mocked broker")
  await expect(messages.nth(1)).toHaveText("Another test message")
})

Beyond simple echo functionality, you can easily extend this code to reflect more sophisticated MQTT features. QoS levels, retained messages, wildcard subscriptions – these could all be added by extending the packet parsing and generation logic shown above, in order to match real-world broker expectations. The result is that your test setup remains both lightweight and realistic.

Conclusion

Mocking MQTT at the WebSocket level in Playwright allows you to maintain a fully self-contained test environment. By intercepting and parsing MQTT packets, we can replicate a minimal, yet realistic broker flow.

Repository: https://github.com/visini/mqtt-playwright-example

I hope this post is helpful for those looking to simplify their Playwright system tests that involve MQTT. If you have any questions or feedback, feel free to reach out. Happy testing!

Streamlining Your Rails 8 App: Migrating from Postgres to SQLite

Camillo Visini — Sat, 19 Oct 2024 18:53:29 +0000

Originally published on my blog: Streamlining Your Rails 8 App: Migrating from Postgres to SQLite

With Rails 8's shift to SQLite as the new default database, developers now have the opportunity to simplify their application stack while retaining power and flexibility.

SQLite's recent improvements for Rails make it a great option for production use. After all SQLite, which was initially released in 2004, has come a long way and is now a powerful, reliable, and performant database engine – perfect for many applications. With the Rails 8 release, SQLite has become the default database for new Rails applications, and comes with a host of performance tuning by default.

In this guide, I describe how to migrate an existing Rails 8 app from Postgres to SQLite to supercharge your application, while at the same time compressing your stack. Keep in mind that the process I describe below requires some downtime, so plan accordingly.

In the application I'm migrating, I'm using Rails 8 and Kamal for deployment. If you're using a different deployment tool, the steps might be different.

Create a Backup of the Postgres Database

Before migrating your Rails application from Postgres to SQLite, it's essential to create a backup of your Postgres database. This backup will ensure that you have a copy of your data in case anything goes wrong during the migration process.

In my case, Postgres is running in a Docker container. First, let's access the Postgres container using the following command:

export POSTGRES_CONTAINER_ID=$(docker ps | grep postgres | awk '{print $1}' | head -n 1)
docker exec -it $POSTGRES_CONTAINER_ID bash

Next, create a backup of the Postgres database using the pg_dump command:

pg_dump -U postgres -d myapp_production -f database.dump

This command will create a dump of the Postgres database in the file database.dump. Now, exit the Docker container. You can copy this file to the host machine using the following command:

docker cp $POSTGRES_CONTAINER_ID:/database.dump ./database.dump

To also download the dump file from the server to your local machine, you can use the scp command:

scp user@server:/path/to/database.dump ~/Downloads

Now, you have a backup of your Postgres database. You can use this backup to restore the data if the migration process fails.

Update the Rails Configuration

Ensure your application has been upgraded to Rails 8 before continuing. In order to point your Rails application to the SQLite database, you need to update the database configuration in the config/database.yml file. This is how the configuration used to look like with Postgres:

default: &default
  adapter: postgresql
  encoding: unicode
  port: 5432
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  timeout: 5000

development:
  <<: *default
  host: localhost
  database: myapp_development
  user: postgres
  password: password

test:
  <<: *default
  host: localhost
  database: myapp_test
  user: postgres
  password: password

production:
  <<: *default
  host: <%= ENV.fetch("POSTGRES_HOST") { "host" } %>
  database: myapp_production
  user: <%= ENV.fetch("POSTGRES_USER") { "user" } %>
  password: <%= ENV.fetch("POSTGRES_PASSWORD") { "password" } %>

And with SQLite, the configuration will look like this:

default: &default
  adapter: sqlite3
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  timeout: 5000

development:
  <<: *default
  database: storage/development.sqlite3

test:
  <<: *default
  database: storage/test.sqlite3

production:
  primary:
    <<: *default
    database: storage/production.sqlite3

Deploy with the New Configuration

Now, we can deploy the updated Rails application. In the app I'm migrating, I'm using Kamal. If you're using something else, be sure to re-deploy your application so the new configuration is loaded in your application. When starting for the first time, the Rails Docker entrypoint will create the production SQLite database and prepare it for use (via bin/rails db:prepare, which will set up and migrate the database).

kamal deploy -d production

Verify on the server that the SQLite database has been created and the application is running as expected.

docker exec -it $RAILS_CONTAINER_ID ls storage

This should list the SQLite database file named production.sqlite3 – if it doesn't, there might be an issue with the deployment. Check the logs to identify the problem.

kamal app logs -d production

As soon as you can see the SQLite database file, you can proceed with the next steps.

Import the Postgres Data into SQLite

Now that the Rails application is running with the (empty) SQLite database, we can import the data from Postgres. Unfortunately, there is no direct way to import a Postgres dump into SQLite.

In my research I discovered tools like pg2sqlite, but they fail to correctly prepare the SQLite database with the required column data types and constraints. It's best to start from a "clean slate" and import the data separately after Rails has set up the SQLite database with the proper schema.

One option is to create a one-off migration script to connect to both the Postgres and SQLite databases and transfer the data over to the newly created SQLite database.

Below is an example script to transfer the data. You might place this file in script/migrate_data.rb. For simple applications, this might be enough. For more complex scenarios, you might need to add additional logic to handle data transformations or relationships. Run this script while inside the container via bundle exec ruby script/migrate_data.rb

require "active_record"
require "pg"
require "sqlite3"

# PostgreSQL database configuration
postgres_config = {
  adapter: "postgresql",
  host: ENV["POSTGRES_HOST"],
  username: ENV["POSTGRES_USER"],
  password: ENV["POSTGRES_PASSWORD"],
  database: "myapp_production"
}

# SQLite database configuration
sqlite_config = {
  adapter: "sqlite3",
  database: "storage/production.sqlite3"
}

# Establish connections
ActiveRecord::Base.establish_connection(postgres_config)
postgres_connection = ActiveRecord::Base.connection
ActiveRecord::Base.establish_connection(sqlite_config)
sqlite_conn = ActiveRecord::Base.connection

# Define the order of tables to migrate
tables_to_migrate = [
  "posts",
  "comments",
  # Add all tables in the correct order (due to foreign key constraints)
]

tables_to_migrate.each do |table_name|
  puts "Migrating table: #{table_name}"

  # Fetch data from PostgreSQL
  data = postgres_connection.select_all("SELECT * FROM #{table_name}")

  # Insert data into SQLite
  data.rows.each do |row|
    attributes = data.columns.zip(row).to_h
    begin
      sqlite_conn.insert_fixture(attributes, table_name)
    rescue => e
      puts "Error inserting into #{table_name}: #{e.message}"
    end
  end

  puts "Table #{table_name} migrated successfully."
end

puts "Migration completed successfully!"

To run the import script again (e.g., after making changes), you can nuke the SQLite database and run the script again:

docker exec -it $RAILS_CONTAINER_ID bin/rails db:reset
docker exec -it $RAILS_CONTAINER_ID bin/rails runner script/migrate_data.rb

After importing the data, your Rails application should now be running as before with the original data from the Postgres database, but now from within SQLite! You can verify this by checking the data in the application or running queries against the SQLite database. Alternatively, download the SQLite database file to your local machine and inspect it using a SQLite database viewer.

To ensure your app is now talking to the SQLite database, you can stop the Postgres container and see if the Rails application is still running as expected.

docker stop $POSTGRES_CONTAINER_ID

Check the logs to confirm everything is working as expected. To tail the logs, you can using the following command:

kamal app logs -d production -f

Congrats! Your Rails application is now running on SQLite.

Adding Backups with Litestream

To ensure that your SQLite database is backed up regularly, you might rely on Litestream. Litestream is a tool that continuously backs up SQLite databases to cloud storage providers like AWS S3. For this, we can use the handy litestream-ruby gem. Read about how to install and configure it in the GitHub repository.

Stopping accessory and cleaning up

After you've confirmed that your Rails application is running smoothly with the SQLite database, you can stop the Postgres container and clean up the volumes. Here are the steps to do so:

docker stop $POSTGRES_CONTAINER_ID
docker rm $POSTGRES_CONTAINER_ID

To identify the volume name, you can use docker volume ls. Then, remove the volume using the following command:

docker volume rm <volume_name>

Alternatively, use Kamal to remove the Postgres accessory:

kamal accessory remove postgres -d production

Also, don't forget to...

Remove the pg gem from your Gemfile
Remove libpq-dev and other no longer needed Postgres-related dependencies from the Dockerfile
Remove the Postgres acessory from the Kamal deploy.yml file, if applicable
Remove any Postgres-related secrets from Kamal secrets, if applicable
Remove the migration script and the Postgres dump file

Conclusion

By migrating your Rails 8 application from Postgres to SQLite, you’ve simplified your setup while maintaining performance and reliability. SQLite’s integration with Rails 8 offers a straightforward, efficient solution for many use cases.

With proper backup strategies in place, you can confidently rely on SQLite in production. Following the steps outlined in this guide, your Rails app is now running smoothly on SQLite.

Migrating Plausible Analytics to a New Server

Camillo Visini — Sun, 26 Feb 2023 09:00:00 +0000

See original post: https://camillovisini.com/coding/migrating-plausible-analytics

Plausible Analytics is a simple, open-source web analytics tool designed to provide essential website metrics without compromising on user privacy. Unlike traditional analytics platforms, Plausible focuses on simplicity and ease of use, offering clear insights without tracking personal data or using cookies.

At Visini AG, we self-host Plausible Analytics for us and our clients. This guide explains how we migrated Plausible from a multi-service server with virtual machines (VMs) to a dedicated machine. The new machine will be hosting Plausible exclusively via Docker, allowing for easier maintenance and version upgrades. In the old as well as the new setup, Plausible and the associated services (Postgres, ClickHouse) are run with Docker Compose.

Create a Backup of Plausible Data

Backing up Postgres and ClickHouse data ensures we preserve customer site configurations and historical analytics, which can be restored on the new machine. Data stored in Postgres includes settings and metadata, such as the sites and their configurations. ClickHouse stores the actual analytics data, such as page views and unique visitors.

Postgres Backup

First, let's create a dump of the Postgres database. We need to connect to the Postgres container to create the dump. Let's check which databases exist in the Postgres container, via the following command (replace the example container ID a52ab8083b6b with the one of your Postgres container):

sudo docker exec -t a52ab8083b6b psql -U postgres  -l

This should provide an output similar to the following:

Name         | Owner    | Encoding |  Collate   |   Ctype    |
-------------+----------+----------+------------+------------+
plausible_db | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
postgres     | postgres | UTF8     | en_US.utf8 | en_US.utf8 |

Looks like we need to create a dump of plausible_db in order to save the configuration data. Let's create a dump of the entire database, like so:

sudo docker exec -t a52ab8083b6b pg_dump -U postgres plausible_db > /home/ubuntu/plausible_db.sql

Now let's copy the dump from the virtual machine to the VM host, and then to the local machine:

# First we copy the dump from the VM to the VM host
scp ubuntu@192.168.122.101:/home/ubuntu/plausible_db.sql /home/ubuntu

# Now from the VM host to the local machine
scp ubuntu@vm-host.com:/home/ubuntu/plausible_db.sql ~/Downloads

After completing the Postgres backup, the next step is to back up ClickHouse, which stores the analytics data.

ClickHouse Backup

Next, we need to create a dump of the ClickHouse data. ClickHouse stores the actual analytics data, such as page views and unique visitors. The relevant tables are ingest_counters, sessions_v2, and events_v2.

Note: In case you've imported data from Universal Analytics (UA), please also migrate the imported_* tables accordingly (not covered in the steps below). See a list of all tables via the following command (replace the example container ID aed6425a6303 with the one of your ClickHouse container):

sudo docker exec -it aed6425a6303 clickhouse client --database plausible_events_db

Running the command show tables should provide an output similar to the following:

-- ┌─name───────────────────────┐
-- │ events                     │
-- │ events_v2                  │
-- │ imported_browsers          │
-- │ imported_devices           │
-- │ imported_entry_pages       │
-- │ imported_exit_pages        │
-- │ imported_locations         │
-- │ imported_operating_systems │
-- │ imported_pages             │
-- │ imported_sources           │
-- │ imported_visitors          │
-- │ ingest_counters            │
-- │ schema_migrations          │
-- │ sessions                   │
-- │ sessions_v2                │
-- └────────────────────────────┘

Let's focus on the ingest_counters, sessions_v2, and events_v2 tables. We can create dumps of these tables by running the following commands (replace the example container ID aed6425a6303 with the one of your ClickHouse container):

# Clean up any old dumps
sudo docker exec -it aed6425a6303 bash -c "rm -f /var/lib/clickhouse/user_files/*.native"

# Create dumps of the relevant tables
sudo docker exec -it aed6425a6303 clickhouse-client --query "SELECT * FROM plausible_events_db.ingest_counters INTO OUTFILE '/var/lib/clickhouse/user_files/ingest_counters.native' FORMAT Native"
sudo docker exec -it aed6425a6303 clickhouse-client --query "SELECT * FROM plausible_events_db.sessions_v2 INTO OUTFILE '/var/lib/clickhouse/user_files/sessions_v2.native' FORMAT Native"
sudo docker exec -it aed6425a6303 clickhouse-client --query "SELECT * FROM plausible_events_db.events_v2 INTO OUTFILE '/var/lib/clickhouse/user_files/events_v2.native' FORMAT Native"

# Copy the dumps to the host
sudo docker cp aed6425a6303:/var/lib/clickhouse/user_files/ingest_counters.native ./ingest_counters.native
sudo docker cp aed6425a6303:/var/lib/clickhouse/user_files/sessions_v2.native ./sessions_v2.native
sudo docker cp aed6425a6303:/var/lib/clickhouse/user_files/events_v2.native ./events_v2.native

Now let's copy these files from the virtual machine to the VM host, and then to the local machine:

# First we copy the dumps from the VM to the VM host
scp ubuntu@192.168.122.101:/home/ubuntu/ingest_counters.native /home/ubuntu
scp ubuntu@192.168.122.101:/home/ubuntu/sessions_v2.native /home/ubuntu
scp ubuntu@192.168.122.101:/home/ubuntu/events_v2.native /home/ubuntu

# Now from the VM host to the local machine
scp ubuntu@vm-host.com:/home/ubuntu/ingest_counters.native ~/Downloads
scp ubuntu@vm-host.com:/home/ubuntu/sessions_v2.native ~/Downloads
scp ubuntu@vm-host.com:/home/ubuntu/events_v2.native ~/Downloads

With the backups of the Postgres and ClickHouse data complete, we can now proceed to import them into the new server.

Import Backup Data into New Server

Now that we have the dumps of the Postgres and ClickHouse data, we can import them into the new machine, which should be running Plausible Analytics inside Docker. We first import the Postgres data, followed by the ClickHouse data.

Import Postgres Backup

First, let's copy the dump of the Postgres database to the new server:

scp ~/Downloads/plausible_db.sql root@new-server-for-plausible.com:/root

Then we can import the dump into the Postgres container. First, connect to the Postgres container to import the dump. Stop the Plausible container to avoid conflicts. Drop the existing database, then create a new one for a fresh start. Use the following commands (replace the example container ID 5ab0dabcbaa4 with the one of your Postgres container, and 1455b8caae1c with the one of your Plausible container):

docker cp plausible_db.sql 5ab0dabcbaa4:/plausible_db.sql
docker stop 1455b8caae1c # Stop plausible container
sudo docker exec -t 5ab0dabcbaa4 psql -U postgres -c "DROP DATABASE plausible_db"
sudo docker exec -t 5ab0dabcbaa4 psql -U postgres -c "CREATE DATABASE plausible_db"
sudo docker exec -t 5ab0dabcbaa4 psql -U postgres -d plausible_db -f /plausible_db.sql
docker start 1455b8caae1c # Start plausible container

Now the Postgres data has been imported into the new server. We can proceed to import the ClickHouse data.

Import ClickHouse Backup

Next, we need to import the ClickHouse data. We can copy the ClickHouse dumps to the new server:

scp ~/Downloads/ingest_counters.native root@new-server-for-plausible.com:/root
scp ~/Downloads/sessions_v2.native root@new-server-for-plausible.com:/root
scp ~/Downloads/events_v2.native root@new-server-for-plausible.com:/root

Then we can import the dumps into the ClickHouse container. We need to connect to the ClickHouse container to import the dumps. We truncate the existing tables and then import the data from the dumps. Use the following commands (replace the example container ID e92e926fb935 with the one of your ClickHouse container):

# Copy the dumps to the ClickHouse container
docker cp ingest_counters.native e92e926fb935:/root/ingest_counters.native
docker cp sessions_v2.native e92e926fb935:/root/sessions_v2.native
docker cp events_v2.native e92e926fb935:/root/events_v2.native

# Connect to the container and truncate the tables
docker exec -it e92e926fb935 clickhouse-client --query "TRUNCATE TABLE plausible_events_db.ingest_counters"
docker exec -it e92e926fb935 clickhouse-client --query "TRUNCATE TABLE plausible_events_db.sessions_v2"
docker exec -it e92e926fb935 clickhouse-client --query "TRUNCATE TABLE plausible_events_db.events_v2"

# Import the data from the dumps
docker exec -it e92e926fb935 bash -c "clickhouse-client --query 'INSERT INTO plausible_events_db.ingest_counters FORMAT Native' < /root/ingest_counters.native"
docker exec -it e92e926fb935 bash -c "clickhouse-client --query 'INSERT INTO plausible_events_db.sessions_v2 FORMAT Native' < /root/sessions_v2.native"
docker exec -it e92e926fb935 bash -c "clickhouse-client --query 'INSERT INTO plausible_events_db.events_v2 FORMAT Native' < /root/events_v2.native"

Now the ClickHouse data has been imported into the new server. We have successfully migrated Plausible Analytics to the new server.

Disable ClickHouse-internal Logging

In order to keep the ClickHouse-internal data (such as the trace_log and metric_log tables) from growing indefinitely, we can disable unnecessary logging.

Mount the following configuration file (clickhouse-config.xml) to the ClickHouse container:

<clickhouse>
    <logger>
        <level>warning</level>
        <console>true</console>
    </logger>

    <!-- Stop all the unnecessary logging -->
    <query_thread_log remove="remove"/>
    <query_log remove="remove"/>
    <text_log remove="remove"/>
    <trace_log remove="remove"/>
    <metric_log remove="remove"/>
    <asynchronous_metric_log remove="remove"/>
    <session_log remove="remove"/>
    <part_log remove="remove"/>
</clickhouse>

We can additionally disable logging of queries and query threads via clickhouse-user-config.xml, which too should be mounted to the ClickHouse container:

<clickhouse>
    <profiles>
        <default>
            <log_queries>0</log_queries>
            <log_query_threads>0</log_query_threads>
        </default>
    </profiles>
</clickhouse>

Clean up ClickHouse Data (Optional)

ClickHouse-internal data may also be cleaned up manually. This might be required periodically, especially in case logging is not disabled as described above, and the data grows to an unmanageable size (which happened in our case on the old server). We can connect to the container and run clickhouse-client to check the size of all tables, via the following command:

sudo docker exec -it aed6425a6303 clickhouse-client

Then we can run this query to get the size of all tables:

SELECT
    database,
    table,
    formatReadableSize(sum(bytes)) AS size
FROM system.parts
GROUP BY
    database,
    table
ORDER BY
    size DESC;

This should provide an output similar to the following:

┌─database────────────┬─table───────────────────┬─size───────┐
│ system              │ asynchronous_insert_log │ 76.57 KiB  │
│ system              │ query_log               │ 44.91 MiB  │
│ system              │ trace_log               │ 4.21 GiB   │
│ system              │ part_log                │ 27.36 MiB  │
│ system              │ asynchronous_metric_log │ 2.89 GiB   │
│ system              │ metric_log              │ 2.45 GiB   │
│ plausible_events_db │ ingest_counters         │ 195.83 KiB │
│ plausible_events_db │ sessions_v2             │ 1.18 MiB   │
│ plausible_events_db │ events_v2               │ 1.17 MiB   │
└─────────────────────┴─────────────────────────┴────────────┘

Wow! Almost 10GB of logs accumulated over the past 12 months. We can delete data from these tables by running the following queries:

aed6425a6303 :) TRUNCATE TABLE system.trace_log;
aed6425a6303 :) TRUNCATE TABLE system.asynchronous_metric_log;
aed6425a6303 :) TRUNCATE TABLE system.metric_log;

Conclusion

In this guide, we've covered the process of migrating Plausible Analytics to a new server. We created dumps of the Postgres and ClickHouse data, and then imported them into the new server. We also disabled ClickHouse-internal logging and cleaned up old data to ensure the system runs smoothly and doesn't accumulate unnecessary data.

These steps will help you migrate Plausible Analytics seamlessly to a new server, ensuring no loss of configurations or historical data.

Setting up Dark Mode for Nuxt and Storybook via Tailwind CSS

Camillo Visini — Tue, 29 Dec 2020 13:30:02 +0000

This article reviews configuring dark mode via Tailwind CSS for Nuxt and Storybook. The following resources are used:

Nuxt and Vue
Color Mode module for Nuxt
Storybook via Storybook module for Nuxt
Dark Mode addon for Storybook
Tailwind CSS via Tailwind CSS module for Nuxt
Dark Mode Plugin for Tailwind CSS

The companion repository includes the complete configuration of all moving parts, demonstrated via the example of a simple Button component:

https://github.com/visini/nuxt-tailwind-storybook-dark-mode

Nuxt Color Mode – Enabling Dark Mode

Color Mode module adds boilerplate-free and effortless color mode switching – including auto detection based on the system's color-mode preferences – to any Nuxt app. See this excellent walkthrough for a more thorough introduction.

Different color modes require different styling. Thus, every component needs to be configured with conditional styling for all supported color modes, which will be applied based on the user's selection.

In order to define global styles for dark mode, add the respective CSS declarations. For instance, specify white text on dark background for when dark mode is activated by the user:

@import "tailwindcss/base";
@import "tailwindcss/components";
@import "tailwindcss/utilities";

.dark {
  @apply bg-dark;
  @apply text-white;
}

Tailwind CSS utility classes based on props such as the component's variant can be dynamically computed within buttonClass() – for both light and dark mode. A base class provides fundamental styling scoped to the button component via @apply.

<template>
  <button class="base" :class="buttonClass">
    <slot name="text">{{ text }}</slot>
  </button>
</template>

<script>
export default {
  props: {
    variant: {
      type: String,
      default: "primary",
    },
    square: {
      type: Boolean,
      default: false,
    },
    text: {
      type: String,
      default: "Button",
    },
  },
  computed: {
    buttonClass() {
      const c = []
      // Square
      if (!this.square) {
        c.push("rounded-md")
      }
      // Variant
      if (this.variant === "primary") {
        c.push("bg-primary-500 hover:bg-primary-600")
        c.push("dark:bg-primary-900 dark-hover:bg-primary-800")
      } else if (this.variant === "secondary") {
        c.push("bg-gray-500 hover:bg-gray-600")
        c.push("dark:bg-gray-700 dark-hover:bg-gray-600")
      }
      return c.join(" ")
    },
  },
}
</script>

<style lang="postcss" scoped>
.base {
  @apply font-bold py-2 px-4 text-white;
}
</style>

Storybook Dark Mode – Component Library

A third-party Dark Mode addon for Storybook allows switching both UI and component view between light and dark mode. Optional: Add argTypes to interact with a component's arguments dynamically via a graphical UI. See Storybook's control addon docs for more information. For instance, switch between rectangular and default (round) buttons by modifying component props via Storybook's UI.

import Button from "./Button"

export default {
  title: "Button",
  component: Button,
  argTypes: {
    variant: {
      control: {
        type: "select",
        options: ["primary", "secondary"],
      },
      defaultValue: "primary",
    },
    square: { control: "boolean" },
    text: {
      control: "text",
    },
  },
}

export const Template = (arg, { argTypes }) => ({
  components: { Button },
  props: Object.keys(argTypes),
  template: '<Button v-bind="$props" />',
})

export const Primary = Template.bind({})
Primary.args = {
  variant: "primary",
}

export const Secondary = Template.bind({})
Secondary.args = {
  variant: "secondary",
}

export const Square = Template.bind({})
Square.args = {
  square: true,
}

Adding dark mode support directly within Storybook enables effortless display of stories and components in both color modes – in the same way users would switch between modes – i.e., based on CSS class inheritance: A dark class is added to Dark Mode Plugin for Tailwind CSS to the <head> tag, while Color Mode module for Nuxt adds the dark class to the <html> tag.

The top right corner of the center toolbar enables switching between light and dark mode. This will switch the UI between color modes, and, as configured in the companion repository, at the same time modify the component view with a specified dark class, effectively simulating an user's color mode preference.

Optional – Tailwind Config Viewer

A useful tool for previewing your Tailwind CSS configuration is Tailwind Config Viewer, which is integrated into Nuxt's Tailwind CSS module since v3.4.0. Simply launch Nuxt in dev mode and access /_tailwind/ in your browser.

Conclusion

I hope this article proves insightful to anyone wanting to set up Nuxt with Tailwind CSS and Storybook. As shown here, supporting dark mode for component prototyping and design, and subsequent integration in your frontend app requires some additional tooling. See the companion repository for specific configuration of all moving parts. Let me know if you have suggestions – I'm curious to learn of alternative approaches!

Implementing FastAPI Services – Abstraction and Separation of Concerns

Camillo Visini — Tue, 29 Dec 2020 13:29:36 +0000

This article introduces an approach to structure FastAPI applications with multiple services in mind. The proposed structure decomposes the individual services into packages and modules, following principles of abstraction and separation of concerns.

The code discussed below can also be studied in its entirety in a dedicated companion GitHub repository: https://github.com/visini/abstracting-fastapi-services

FastAPI – Building High-performing Python APIs

FastAPI is a fast, highly intuitive and well-documented API framework based on Starlette. Despite being relatively new, it's gaining strong adoption by the developer community – it is even already being adopted in production by corporates like Netflix and Microsoft.

Following the UNIX philosophy of "doing one thing, and doing it well", separating parts of the application according to their task improves code readability and maintainability; ultimately reducing complexity. The main benefits of structuring applications in this way:

Separation of concerns – Decomposing the application into modules performing a single job. This allows accepting requests (top down: controller > service > data access) and returning responses (bottom up: data access > service > controller) with a clear separation of what particular functionality should be implemented in which particular module, reducing cognitive load for development.

Abstraction — Components of the application are designed in a reusable way. For instance, ServiceResult is implemented as a generic outcome of a service operation (which may be successful and return a response, or unsuccessful and raise an exception) able to be used by all services of the app, keeping code DRY.

The granular nature of namespacing allow to distinguish parts of the application, e.g., routes versus business logic belonging to a particular service – grouping similar tasks together, while keeping distinct parts separated. Four principal packages are needed. For each service, one module is added to these four packages. For instance, a service called "Foo" requires the following modules (discussed in detail below):

./routers/  foo.py # Router instance and routes
./services/ foo.py # Business logic (including CRUD helpers)
./schemas/  foo.py # Data "schemas" (e.g., Pydantic models)
./models/   foo.py # Database models (e.g., SQLAlchemy models)

The four principal packages are complemented by two generic packages which contain application-specific (and not service-specific) functionality, such as configuration or utility functions.

Controller Layer – Routes

Within main, the application is instantiated and all routers are included. Additionally, middlewares and/or exception handlers are implemented. The example application discussed below is based on a service called Foo, which requires a number of routes. To handle custom exceptions occurring at the service layer, as instances of class AppExceptionCase, a respective exception handler is added to the application.

# ...
from fastapi import FastAPI
from routers import foo

app = FastAPI()

@app.exception_handler(AppExceptionCase)
async def custom_app_exception_handler(request, e):
    return await app_exception_handler(request, e)

app.include_router(foo.router)

Routers and their routes are defined in modules within the routers package. Each route instantiates the respective service and passes on the database session from the request dependency. Handled by handle_result(), the service result (either the requested data as the result of a successful operation, or an exception) is returned. In case of an exception, instead of (and before) returning any response, the app exception handler in main picks up handling the exception and returns a response.

router = APIRouter(prefix="/foo")

@router.post("/item/", response_model=FooItem)
async def create_item(item: FooItemCreate, db: get_db = Depends()):
    result = FooService(db).create_item(item)
    return handle_result(result)

@router.get("/item/{item_id}", response_model=FooItem)
async def get_item(item_id: int, db: get_db = Depends()):
    result = FooService(db).get_item(item_id)
    return handle_result(result)

Service Result Class – Success & Exceptions

The ServiceResult class defines a generic outcome of a service operation. In case the operation is successful, the outcome (or "value") is returned contained within the value attribute of the instance. In case of a custom app exception, the service result instance contains information about the raised exception (e.g., which status code should be returned to the client).

class ServiceResult:
    def __init__(self, arg):
        if isinstance(arg, AppExceptionCase):
            self.success = False
            self.exception_case = arg.exception_case
            self.status_code = arg.status_code
        else:
            self.success = True
            self.exception_case = None
            self.status_code = None
        self.value = arg

    def __str__(self):
        if self.success:
            return "[Success]"
        return f'[Exception] "{self.exception_case}"'

    def __repr__(self):
        if self.success:
            return "<ServiceResult Success>"
        return f"<ServiceResult AppException {self.exception_case}>"

    def __enter__(self):
        return self.value

    def __exit__(self, *kwargs):
        pass


def handle_result(result: ServiceResult) -> :
    if not result.success:
        with result as exception:
            raise exception
    with result as result:
        return result

Service Layer – Business Logic

Services are defined in the services package. Each service is a subclass of AppService. The database session is passed down from the request dependency via an "interface-like" mixin utility class (other mixing classes may be added via multiple inheritance in order to extend available attributes).

class DBSessionMixin:
    def __init__(self, db: Session):
        self.db = db

class AppService(DBSessionMixin):
    pass

class AppCRUD(DBSessionMixin):
    pass

Routes belonging to service Foo are connected to methods of FooService, which encapsulates all business logic of the service. The return value are of type ServiceResult: Containing a value attribute with returnable data or, in case of an exception, an AppException. In both cases, the respective result is returned back "upwards" to the controller layer.

class FooService(AppService):
    def create_item(self, item: FooItemCreate) -> ServiceResult:
        foo_item = FooCRUD(self.db).create_item(item)
        if not foo_item:
            return ServiceResult(AppException.FooCreateItem())
        return ServiceResult(foo_item)

    def get_item(self, item_id: int) -> ServiceResult:
        foo_item = FooCRUD(self.db).get_item(item_id)
        if not foo_item:
            return ServiceResult(AppException.FooGetItem({"item_id": item_id}))
        if not foo_item.public:
            return ServiceResult(AppException.FooItemRequiresAuth())
        return ServiceResult(foo_item)

Data Access Layer – Database Operations

CRUD helper methods perform operations on the database and are subclassing AppCRUD. The database session is passed down from the AppService instance. These methods are atomic and only concerned with operating on the database. They do not contain any business logic.

class FooCRUD(AppCRUD):
    def create_item(self, item: FooItemCreate) -> FooItem:
        foo_item = FooItem(description=item.description, public=item.public)
        self.db.add(foo_item)
        self.db.commit()
        self.db.refresh(foo_item)
        return foo_item

    def get_item(self, item_id: int) -> FooItem:
        foo_item = self.db.query(FooItem).filter(FooItem.id == item_id).first()
        if foo_item:
            return foo_item
        return None

Pydantic "schemas" or models are defined in the schemas package. They contain mainly two different kinds of data models. First, those expected from clients as request data (route parameters in route method definitions). Second, those expected to be returned to clients as response data (defined in response_model parameter of route definitions).

Additionally, it's possible to model any kind of data being passed inbetween the layers of the app (controller, service and data-access).

class FooItemBase(BaseModel):
    description: str

class FooItemCreate(FooItemBase):
    public: bool

class FooItem(FooItemBase):
    id: int

    class Config:
        orm_mode = True

SQLAlchemy models are defined in the models package. They define how data is stored within the relational database. They are referenced from AppCRUD. If needed, make sure to differentiate between FooItem models (SQLAlchemy) and FooItem schemas (Pydantic) by appropriate import namespacing.

class FooItem(Base):
    __tablename__ = "foo_items"

    id = Column(Integer, primary_key=True, index=True)
    description = Column(String)
    public = Column(Boolean, default=False)

Exception Handling

App exceptions are implemented in the utils package. First, AppExceptionCase is subclassed from base Exception and includes various attributes for defining custom app exception scenarios. The exception handler with the task of handling custom app exceptions (added to main, see above) is defined with a response containing information about the app exception.

class AppExceptionCase(Exception):
    def __init__(self, status_code: int, context: dict):
        self.exception_case = self.__class__.__name__
        self.status_code = status_code
        self.context = context

    def __str__(self):
        return (
            f"<AppException {self.exception_case} - "
            + f"status_code={self.status_code} - context={self.context}>"
        )


async def app_exception_handler(request: Request, exc: AppExceptionCase):
    return JSONResponse(
        status_code=exc.status_code,
        content={
            "app_exception": exc.exception_case,
            "context": exc.context,
        },
    )

Second, defining and documenting custom exception scenarios happens in the same module and requires subclassing from AppExceptionCase. Each app exception includes a description within the docstring and defines the status code to be returned to the client.

The class names are reported back to inform clients of the particular exception scenario - see AppExceptionCase initalization, and JSONResponse within the app exception handler.

class AppException:
    class FooCreateItem(AppExceptionCase):
        def __init__(self, context: dict = None):
            """
            Item creation failed
            """
            status_code = 500
            AppExceptionCase.__init__(self, status_code, context)

    class FooGetItem(AppExceptionCase):
        def __init__(self, context: dict = None):
            """
            Item not found
            """
            status_code = 404
            AppExceptionCase.__init__(self, status_code, context)

    class FooItemRequiresAuth(AppExceptionCase):
        def __init__(self, context: dict = None):
            """
            Item is not public and requires auth
            """
            status_code = 401
            AppExceptionCase.__init__(self, status_code, context)

It's possible to add context to an exception AppException.FooCreateItem({"id": 123}) in order to inform the client of an exception's context, if needed.

To compile a list of all exceptions (e.g., in order to localize app exception messages displayed to the client), access all attributes within the scope of AppException:

from utils.app_exceptions import AppException

print([e for e in dir(AppException) if "__" not in e])
# ['FooCreateItem', 'FooGetItem', 'FooItemRequiresAuth']

Conclusion

The proposed approach allows designing FastAPI applications which are highly structured to accomodate multiple services, allowing the implementation of any number of services in a unified pattern.

I hope you enjoyed this article! I'm curious to hear your thoughts regarding this proposed structure and look forward to learn from alternative approaches - feel free to start a discussion by opening an issue on GitHub in the companion repository to this article, or reach out to me directly.

CI/CD Workflow for AWS ECS via Terragrunt and GitHub Actions

Camillo Visini — Tue, 29 Dec 2020 13:28:42 +0000

This project leverages Terragrunt, Terraform, and GitHub Actions to deploy a basic web app (dockerized JS frontend and dockerized Python API) to AWS ECS.

Check out the full source code on GitHub: https://github.com/visini/terragrunt-github-actions-aws-ecs

Initial Setup

Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules, remote state, and locking. It also provides a powerful and flexible way to hierarchically provide configuration to Terraform, without duplicating code across environments, AWS regions, and AWS accounts – keeping your Terraform config DRY.

The following hierarchy is proposed (aligned with directory structure):

terragrunt.hcl with configuration for remote_state and AWS provider
common.terragrunt.hcl defining common, project-specific variables
- account.terragrunt.hcl for each account
- region.terragrunt.hcl for each region within an account
  - environment.terragrunt.hcl for each environment within a region

This allows flexible configuration, just add additional folders and adjust the configuration files, for instance configuring...

Accounts main and secondary
Regions eu-west-1 and us-east-1 in main vs. us-east-1 in secondary
Environments prod in main regions vs. stage and dev in secondary regions

Workflow via GitHub Flow

This project leverages GitHub Flow for gradually merging changes existing on experimental branches and deployed to experimental environments, towards more mature branches and environments.

The companion repository contains functionality to deploy code to AWS ECS simply by adopting GitHub Flow principles. All integration and deployment steps are managed by GitHub Actions workflows, including: Unit testing, building and pushing Docker images, and releasing new images to the correct ECS cluster via Terraform and Terragrunt. Create a branch, push, create a pull-request, and, after verifying checks, merge all changes - these are the only steps needed to deploy new features by adopting this approach.

Assuming a running staging and production environment, here's how to deploy changes made for a recent feature "foo" to staging and production environments:

Step 1 - Deployment to staging environment stage via branch dev

Create a new branch feature/foo and check it out
- git checkout -b feature/foo
Push to remote and set up to track remote branch feature/foo from origin
- git push --set-upstream origin feature/foo
Open pull request from branch feature/foo to branch dev to plan deployment to stage environment
Wait for checks to complete: Workflow terragrunt will post terraform plan as a comment to pull request
Additional checks may include unit tests: See pytest-api.yml
After verifying terraform plan, merge pull request into branch dev
Workflow terragrunt will run again and apply deployment for stage environment
Code (and infrastructure) changes to branch feature/foo are now released to stage environment

Step 2 - Deployment to production environment prod via branch main

After verifying deployment to stage environment (e.g., e2e-testing), open pull request from branch dev to branch main to plan deployment to prod environment
Wait for checks to complete: Workflow terragrunt will post terraform plan as a comment to pull request
After verifying terraform plan, merge pull request into branch main
Workflow terragrunt will run again and apply deployment for prod environment
Code (and infrastructure) changes to branch dev originating from branch feature/foo are now released to prod environment

Configure Infrastructure and Deployment Targets

The hierarchical configuration via Terragrunt is enabled by a main configuration file in which all other more granular configuration files are imported. In terragrunt.hcl, both remote state and AWS provider are defined according to values in more specific configuration files.

Both remote state and provider are dynamically defined for each deployment target (e.g., prod vs. stage environment) and AWS account ID (e.g., main vs. secondary account). This means, prod and stage environments (which may even be residing on two separate AWS accounts) adopt separate remote state backend configurations, depending on which environment subfolder terragrunt commands are executed from. Review this file and the nested Terragrunt configuration files in the companion repository for the detailed implementation.

locals {
    common = read_terragrunt_config(find_in_parent_folders("common.terragrunt.hcl"))
    account = read_terragrunt_config(find_in_parent_folders("account.terragrunt.hcl"))
    region = read_terragrunt_config(find_in_parent_folders("region.terragrunt.hcl"))
    environment = read_terragrunt_config(find_in_parent_folders("environment.terragrunt.hcl"))
}

remote_state {
    backend = "s3"
    # remote_state dynamically configured based on:
    # local.region.locals.aws_region
    # local.account.locals.aws_account_id
    # local.common.locals.app_name

    # ...
}

generate "provider" {
    # AWS provider dynamically configured based on:
    # local.region.locals.aws_region
    # local.account.locals.aws_account_id

    # ...
}

# The following variables apply to all configurations in this subfolder
# and are automatically merged into the child `terragrunt.hcl` config
# via the include block.

inputs = merge(
    local.common.locals,
    local.account.locals,
    local.region.locals,
    local.environment.locals
)

Common variables, such as the app name, base domain name and hosted zone name, which apply to the project are configured in a separate file:

# The following define common variables for the project.
# They are automatically pulled in in the root terragrunt.hcl
# configuration to feed forward to the child modules.

locals {
    app_name = "example-app"
    app_domain_name = "app.example.com"
    route53_hosted_zone_name = "example.com"
    use_existing_route53_hosted_zone = true
    github_sha = "will_be_automatically_set_by_github_actions_or_manual_script"
}

One or any number of AWS accounts may be configured, each related to any number of regions and environments to be deployed via this AWS account:

# Set account-specific variables. They are automatically
# pulled in to configure the remote state bucket in the root
# terragrunt.hcl configuration.

locals {
  account_name   = "main"
  aws_account_id = "123456789000"
  aws_profile    = "default"
}

Similarly, the region configuration is provided in the nested level:

# Set region-specific variables. They are automatically
# pulled in to the root terragrunt.hcl configuration to
# feed forward to the child modules.

locals {
    aws_region = "eu-west-1"
}

Environment configuration regarding both infrastructure and containers are provided in the nested level. To illustrate a common use case: The stage environment may override variables previously defined in the parent-hierarchy, in tf/common.terragrunt.hcl, and for instance add a prefix stage.* to app_domain_name.

# Set environment-specific variables. They are automatically
# pulled in to the root terragrunt.hcl configuration to
# feed forward to the child modules.

locals {
    common = read_terragrunt_config(find_in_parent_folders("common.terragrunt.hcl"))
    account = read_terragrunt_config(find_in_parent_folders("account.terragrunt.hcl"))
    region = read_terragrunt_config(find_in_parent_folders("region.terragrunt.hcl"))

    # Configure environment
    environment = "stage"
    app_domain_name = "stage.${local.common.locals.app_domain_name}"

    app_name = local.common.locals.app_name
    aws_account_id = local.account.locals.aws_account_id
    aws_region = local.region.locals.aws_region

    parameter_group = "${local.app_name}/${local.environment}"

    service_configuration = {
        # ...
    }
}

Configure Container Environment and Secrets

Environment variables for the respective deployment target (e.g., for stage environment) are provided alongside terragrunt configuration in JSON files, following the naming .service.environment.json, and by specifying both keys and values. These files are committed to source control, since they do not contain any sensitive data. Adding a description key-value pair will inform development and ensure consistency of variable assignment.

{
  "DEBUG": {
    "value": "true",
    "description": "API debug environment"
  }
  // ...
}

Similarly, JSON files following the naming .service.secrets.json provide keys (not values) for all container secrets, which are injected by AWS Systems Manager Parameter Store into the service's ECS tasks (containers) as environment variables. Adding a description key-value pair will inform development and ensure consistency of variable assignment.

No secrets are present in code or source control — secrets such as database passwords or secret keys are generated as terraform resources, and stored within Systems Manager Parameter Store. Following the design of terraform state, they are additionally stored within remote state backend. While S3 backend supports encryption at rest, remote state is to be considered sensitive data.

{
  "SECRET_KEY": {
    "description": "Secret key required for API JWT authentication"
  }
  // ...
}

Integration via GitHub Actions – Pytest

Easily add continuous integration workflows for unit, integration, and e2e tests by adding a GitHub Action workflow. The companion repository includes an example for testing the API service via Pytest.

name: Pytest API

on:
  pull_request:

jobs:
  # ----------------------------------------------------------------
  # Checkout code, install dependencies, run pytest for API
  # ----------------------------------------------------------------
  pytest: #...

Deployment via GitHub Actions – Terragrunt

To support branch-aware continuous deployment of code to the respective environment, a GitHub Actions workflow is provided.

The workflow is based on Terraform's guide for automating Terraform with GitHub Actions, and adds support for nested configuration via Terragrunt. With this workflow, pull-requests will trigger terraform plan, and merging these pull requests will trigger terraform apply to deploy to the correct environment: Branch dev will deploy to environment stage and branch main will deploy to environment prod – all following the specified hierarchical configuration defined in *.terragrunt.hcl files. Review this file in the companion repository for the detailed implementation.

name: Terragrunt

on:
  push:
    branches:
      - main
      - dev
  pull_request:

jobs:
  # ----------------------------------------------------------------
  # Checkout code, setup target_push and tf_env
  # Configure AWS credentials via secrets in repo settings
  # ----------------------------------------------------------------
  setup: #...
  # ----------------------------------------------------------------
  # Push to ECR if target_push
  # Tag image with github.sha of commit
  # ----------------------------------------------------------------
  ecr: #...
  # ----------------------------------------------------------------
  # Terraform for deployment targets
  # Based on tf_env (pull request -> plan; push to branch -> apply)
  # ----------------------------------------------------------------
  terraform: #...

Conclusion

This article describes a flexible CI/CD workflow for AWS ECS based projects. Using GitHub Flow, changes to the infrastructure and/or codebase are deployed to the intended deployment targets.

Thanks for reading. I'm curious to hear your thoughts on this topic – don't hesitate to reach out to me on Twitter or start a discussion in the companion repository on GitHub!

Python i18n - Localizing Transactional Email Templates

Camillo Visini — Tue, 29 Dec 2020 13:28:29 +0000

Leverage Python, Jinja and i18n Ally to define, localize, and preview transactional emails

Applications which support frontend localization usually also require some localization strategies for backend services. A prominent example is the generation of localized transactional emails. This article discusses i18n strategies and workflows for transactional emails processed via Python backends. It relies mainly upon the following resources:

Jinja, a versatile templating language for Python
Postmark's excellent resource for transactional email templates
i18n Ally, a VS Code extension for localization

If you are interested in reviewing the complete implementation:

https://github.com/visini/email-templates-i18n

Defining Email Messages

For each "type" or kind of email message, inherit from a generic parent class EmailMessage, which in turn implements the required attributes and methods (e.g., rendering templates). See the companion repository for a possible implementation.

class EmailVerification(EmailMessage):
    def __init__(self, config, locale, variables):
        email_type = "email_verification"
        required = ["cta_url"]
        super().__init__(config, email_type, required, locale, variables)

class PasswordReset(EmailMessage):
    def __init__(self, config, locale, variables):
        email_type = "password_reset"
        required = ["cta_url", "operating_system", "browser_name"]
        super().__init__(config, email_type, required, locale, variables)

Instantiate a new message with locales and variables, and retrieve all required attributes for sending localized emails:

message = EmailVerification(config, "en-US", {"foo": "bar"})
# print(message.subject)  # localized subject
# print(message.html)     # localized HTML with inlined CSS
# print(message.txt)      # localized plaintext email
# ...
send_email(recepient, message.subject, message.html, message.txt)

Locales and Templates

All locale strings are stored in JSON format. This ensures flexibility with both localization workflow and should be relatively resilient against changing requirements. Global variables are defined across locales, since they usually do not depend on locale context (e.g., company or product names).

{
  "company_name": "Company Name",
  "product_name": "Product Name",
  "product_website": "https://www.example.com/"
}

Locale strings files contain a global key and a key for each email message type, for instance email_verification. The former contain "localized globals", i.e., strings reusable across various email message types. The latter define all strings of a particular email message type.

{
  "global": {
    "greetings": "Viele Grüsse,",
    "all_rights_reserved": "Alle Rechte vorbehalten."
  },
  "email_verification": {
    "subject": "E-Mail-Adresse bestätigen",
    "thank_you": "Vielen Dank für deine Registrierung!"
  }
}

Use blocks to inherit the layout from higher order, more generic templates. In the companion repository, a three-layer hierarchical inheritance is proposed (barebone layout with basic styling &rightarrow; reusable email layout &rightarrow; specific email message template). Template files interpolate both locale strings and variables with double curly brace notation.

{% extends "layouts/call_to_action.html" %}
{% block body_top %}
{{localized.thank_you}}
{% endblock %}
{% block body_bottom %}
{{localized.contact_us_for_help}}
{% endblock %}

Define a separate template for plain text emails - add dividers to structure the template without any markup. See Postmark's best practices for more details about how to format plain text emails.

{% extends "layouts/call_to_action.txt" %}
{% block body_top %}
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
{{localized.thank_you}}
{% endblock %}
{% block body_bottom %}
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
{{localized.contact_us_for_help}}
{% endblock %}

Besides in template files, variables can be used for interpolating values in localization files with the same notation. Interpolate...

global strings &rightarrow; {{product_name}}
locale-specific strings ("localized globals") &rightarrow; {{localized.support_phone_us}}
variables &rightarrow; {{variables.operating_system}}

{
  "global": {
    "greetings": "Regards,",
    "team_signature": "Your {{product_name}} Team",
    "support_phone_us": "1-800-000-0000"
  },
  "password_reset": {
    "subject": "Password Reset Request",
    "support_message": "Need help? Call {{localized.support_phone_us}}.",
    "security_information": "This request was received from a {{variables.operating_system}} device."
  }
}

It's even possible to use more complex Jinja functionality, such as conditional statements, directly within locale strings.

{
  "discounted_items_cart_abandonment_notification": {
    "subject": "There are {{variables.no_items_in_cart}} items in your shopping cart!",
    "promo_message": "{{'One item has a discount!' if variables.no_items_in_cart_discounted < 2 else variables.no_items_in_cart_discounted + ' items have a discount!'}}"
  }
}

A more maintainable approach however is to offload all logic to the respective templates and only use simple variable interpolation in locale string files for convenience.

Localization Workflow

i18n Ally is a VS Code extension for localization. It integrates with a variety of frameworks (e.g., Vue.js), but can also be used to speed up localization workflows of a static directory of locale JSON files. It even includes features to collaboratively review and discuss localizations.

Previewing Generated Emails

In the companion repository, a sample implementation for a thin utility to generate and serve rendered templates is provided.

Auto-reload upon detected file changes (templates and locale strings)
Interactively switch between locale, email type, and format (HTML and plain text)
Responsive view (e.g., Chrome Devtools) allows viewing rendered templates in various scenarios
Based on Vue.js and FastAPI, extensible and with minimal overhead
Work in progress

Sending Email (AWS SES)

Access the class attributes of the instantiated email message for implementing email sending functionality. For illustration purposes, an example for AWS SES is provided below:

import boto3
from botocore.exceptions import ClientError

from src.messages import EmailVerification

config_path = "./src/data"
lang_path = "./src/data/lang"
templates_path = "./src/templates"

# Message config
locale = "en-US"
variables = {"cta_url": "https://www.example.com/"}
message = EmailVerification(config, locale, variables)

#  Application config
SENDER = "Sender Name <sender@example.com>"
RECIPIENT = "recipient@example.com"
CONFIGURATION_SET = "ConfigSet"
AWS_REGION = "us-west-2"

client = boto3.client("ses", region_name=AWS_REGION)

try:
    response = client.send_email(
        Destination={"ToAddresses": [RECIPIENT] },
        Message={
            "Body": {
                "Html": {"Charset": "UTF-8", "Data": message.html},
                "Text": {"Charset": "UTF-8", "Data": message.txt},
            },
            "Subject": {"Charset": "UTF-8", "Data": message.subject},
        },
        Source=SENDER,
        ConfigurationSetName=CONFIGURATION_SET,
    )

except ClientError as e:
    print(e.response["Error"]["Message"])
else:
    print("Email sent! Message ID:"),
    print(response["MessageId"])

Conclusion

Supporting multiple locales for transactional emails requires some additional considerations for templates and locale strings definition. The proposed approach includes classes and additional tooling to implement i18n transactional emails in Python applications.

I hope you found this article informative for how to approach i18n in Python backends!

Create a macOS Menu Bar App with Python (Pomodoro Timer)

Camillo Visini — Sun, 09 Feb 2020 16:55:38 +0000

On my Mac, I use the menu bar countless times per day. In this post we will go through the process of creating a custom macOS menu bar app using Python. As an example, we will create a 🍅 pomodoro app, which you can use to boost your productivity and manage your time from the convenience of your menu bar. It serves as a starting point for customization to suit your individual needs – you can even use this code as a boilerplate to create a radically different application!

📌 Check out the full source code on GitHub

We will be using the following software:

Python 3 and PyCharm as an IDE
Rumps → Ridiculously Uncomplicated macOS Python Statusbar apps
py2app → For creating standalone macOS apps from Python code (how cool is that?)

Step 1: Set up project directory and install dependencies

In this step we will install all requirements on our local machine. Run the following commands in your terminal to get started and set up the project directory:

export PROJECT_NAME=pomodoro
mkdir ~/PycharmProjects/$PROJECT_NAME
cd ~/PycharmProjects/$PROJECT_NAME
touch pomodoro.py setup.py

Be sure to install both rumps and py2app via pip (on my macOS). Enter the following in your terminal:

pip3 install -U py2app # this will install py2app using pip, or to upgrade to the latest released version of py2app
pip3 install -U rumps # this will install rumps using pip, or to upgrade to the latest released version of rumps

Step 2: Basic example

Open the project directory in your favorite editor or IDE – in my case PyCharm – by typing:

charm .

In pomodoro.py we need to set up the following boilerplate code in order to get started:

import rumps

class PomodoroApp(object):
    def __init__(self):
        self.app = rumps.App("Pomodoro", "🍅")

    def run(self):
        self.app.run()

if __name__ == '__main__':
    app = PomodoroApp()
    app.run()

If you execute the python program using python3 pomodoro.py, you will notice a new addition to your menu bar – albeit with limited functionality, as you can see…

Step 3: Full implementation

Now we will implement the actual functionality of our pomodoro menu bar app – see the complete code for pomodoro.py below:

import rumps

class PomodoroApp(object):
    def __init__(self):
        self.config = {
            "app_name": "Pomodoro",
            "start": "Start Timer",
            "pause": "Pause Timer",
            "continue": "Continue Timer",
            "stop": "Stop Timer",
            "break_message": "Time is up! Take a break :)",
            "interval": 1500
        }
        self.app = rumps.App(self.config["app_name"])
        self.timer = rumps.Timer(self.on_tick, 1)
        self.interval = self.config["interval"]
        self.set_up_menu()
        self.start_pause_button = rumps.MenuItem(title=self.config["start"], callback=self.start_timer)
        self.stop_button = rumps.MenuItem(title=self.config["stop"], callback=None)
        self.app.menu = [self.start_pause_button, self.stop_button]

    def set_up_menu(self):
        self.timer.stop()
        self.timer.count = 0
        self.app.title = "🍅"

    def on_tick(self, sender):
        time_left = sender.end - sender.count
        mins = time_left // 60 if time_left >= 0 else time_left // 60 + 1
        secs = time_left % 60 if time_left >= 0 else (-1 * time_left) % 60
        if mins == 0 and time_left < 0:
            rumps.notification(title=self.config["app_name"], subtitle=self.config["break_message"], message='')
            self.stop_timer()
            self.stop_button.set_callback(None)
        else:
            self.stop_button.set_callback(self.stop_timer)
            self.app.title = '{:2d}:{:02d}'.format(mins, secs)
        sender.count += 1

    def start_timer(self, sender):
        if sender.title.lower().startswith(("start", "continue")):
            if sender.title == self.config["start"]:
                self.timer.count = 0
                self.timer.end = self.interval
            sender.title = self.config["pause"]
            self.timer.start()
        else:
            sender.title = self.config["continue"]
            self.timer.stop()

    def stop_timer(self):
        self.set_up_menu()
        self.stop_button.set_callback(None)
        self.start_pause_button.title = self.config["start"]

    def run(self):
        self.app.run()

if __name__ == '__main__':
    app = PomodoroApp()
    app.run()

By executing the python program again using python3 pomodoro.py, you can see our menu bar app in its full glory:

In the next step we will go ahead and create a macOS application using py2app so you don’t have to execute it via Python every single time. It needs to be ready for every day use, and that means adding it to the login items of your Mac!

Step 4: Create macOS app from our python code

In setup.py we need to add the following code, which provides all the necessary instructions to create the application bundle (app name, app version, app icon, etc.) to py2app in form of a setup script:

from setuptools import setup

APP = ['pomodoro.py']
DATA_FILES = []
OPTIONS = {
    'argv_emulation': True,
    'iconfile': 'icon.icns',
    'plist': {
        'CFBundleShortVersionString': '0.2.0',
        'LSUIElement': True,
    },
    'packages': ['rumps'],
}

setup(
    app=APP,
    name='Pomodoro',
    data_files=DATA_FILES,
    options={'py2app': OPTIONS},
    setup_requires=['py2app'], install_requires=['rumps']
)

Are you missing the icon file? You can download the file I used from the GitHub repository.

Now you can go ahead and create your Pomodoro.app bundle using py2app. Type the following in your terminal:

python3 setup.py py2app

The application bundle will be created in your project directory under /dist/Pomodoro.app.

Conclusion

I’ve implemented a similar menu bar app with some additional functionality. It’s called Timebox. Check it out and let me know what you think!

I hope you enjoyed this article – please let me know if you have any questions or if you run into any issues.

Deploy Flask Apps to AWS Elastic Beanstalk using Travis CI

Camillo Visini — Sun, 09 Feb 2020 16:01:46 +0000

In this article, we are going to deploy a Flask application to AWS Elastic Beanstalk via a GitHub and Travis CI deployment pipeline. Our goal: After pushing changes to your code to GitHub, Travis CI should pull our code, perform all tests, and if they pass, deploy the application to AWS Elastic Beanstalk. Let's go!

Step 1: Set up project directory and install dependencies

First, we need to install all dependencies on our local machine. Because AWS Elastic Beanstalk requires context about which environment our application needs to run in, we will create a Python virtual environment and install all dependencies within the virtual environment. Run the following commands in your terminal to get started and set up the project directory:

# create and go to project folder
export PROJECT_NAME=deploy-eb-via-travis-ci
mkdir ~/VSCodeProjects/$PROJECT_NAME
cd ~/VSCodeProjects/$PROJECT_NAME
# create virtual environment and activate it
python3 -m venv my_venv
source my_venv/bin/activate
# within virtual environment, install the following:
python3 -m pip install flask
python3 -m pip freeze > requirements.txt
deactivate
# set up file structure
touch app.py wsgi.py .travis.yml 
mkdir tests
touch tests/test_42.py

Then, open your project directory with your favorite code editor. You can use your command line: For example for PyCharm you can just type charm . or for VSCode you can type code . Learn more about how to set up command line interfaces for PyCharm here or VSCode here.

Step 2: "Hello World" Flask Application

We need to create a basic "Hello World" Flask application, which we can then use for the purposes of setting up our continuous deployment pipeline scenario. Assume that this is a placeholder before you are able to deploy your (real) Flask application. The deployment of a real-world application likely follows very similar steps as described here. As seen above, we created four empty files.

In app.py paste the following code:

from flask import Flask, jsonify
app = Flask(__name__)
@app.route("/")
def index():
    return jsonify({"hello": "world", "from": "index"})
@app.route("/foo")
def foo():
    return jsonify({"hello": "world", "from": "foo"})
if __name__ == '__main__':
    app.run(debug=True)

Your wsgi.py should look like this:

from app import app as application
if __name__ == "__main__":
    application.run()

In tests/test_42.py add a sample test (which will always pass):

def test_42():
    assert 42 == 42

We will revisit the fourth required file, .travis.yml, when we are setting up the Travis CI integration. As for now, we are ready to (manually) deploy our first version to AWS Elastic Beanstalk.

Step 3: Deploy to AWS Elastic Beanstalk

Before we can deploy our application, make sure that the AWS Elastic Beanstalk CLI eb is installed. If it's not installed on your local machine, you can follow the instructions here. You can check whether your local machine has eb installed by running the following command:

eb --version

The following commands will set up AWS Elastic Beanstalk with an empty application. Make sure to provide your AWS region (in my case eu-west-1) and required Python version (in my case python-3.6 is the latest Python version supported by EB at the time of writing) accordingly:

export REGION=eu-west-1
export PYTHON_VERSION=python-3.6 # latest version supported as of January 2020
eb init -p $PYTHON_VERSION -r $REGION $PROJECT_NAME

Next, we need to configure the default WSGI path. This will tell AWS Elastic Beanstalk where the entry point of our Flask application is located at. In our case, it's the file wsgi.py. Run the following commands in your terminal:

# default is application.py
# alternative way to configure this -> eb config and then set manually
mkdir .ebextensions
echo "option_settings:
  - namespace: aws:elasticbeanstalk:container:python
    option_name: WSGIPath
    value: wsgi.py" > .ebextensions/wsgi.config

Our continuous deployment pipeline will be git-based, since we push our changes to GitHub. Initialize git with the following commands:

git init
git add .
git commit -m "First commit"

Note: AWS Elastic Beanstalk will create .zip archives for application deployment based on your changes that are committed to git. Therefore, remember to commit your changes before manually deploying via eb deploy, otherwise your changes will not be deployed!

In a real-world context, you would have more than one environment for your application, the most basic scenario being a live deployment (where users are interacting with your app) and a test deployment (where you will deploy new releases to test them). You can create an environment on AWS Elastic Beanstalk with the following commands:

export ENVIRONMENT_NAME=test # for example: "test" or "live"
export INSTANCE_TYPE=t2.nano # specify the instance type
eb create $ENVIRONMENT_NAME-$PROJECT_NAME --single -i $INSTANCE_TYPE
eb use $ENVIRONMENT_NAME-$PROJECT_NAME

Now, you can trigger a manual deployment to AWS Elastic Beanstalk:

eb deploy

If you can confirm your Flask application is working as expected, we can move on and integrate Travis CI to implement our continuous deployment pipeline.

Step 4: Setting up Travis CI pipeline

Before we can proceed, you need to push your changes to GitHub:

git remote add origin https://github.com/owner/repo.git
git push -u origin master

Make sure to enable Travis CI in the repository settings, either in your GitHub account or your Travis CI account. This will make Travis CI listen to changes of your repository automatically trigger a build for this repository whenever a push is registered. Your Travis CI account needs to have the required permissions to access your GitHub account and your repositories.

Make sure the Travis command line interface travis is installed:

travis --version

If it's not installed, you can run the following command in order to install it:

gem install travis

There are two versions of Travis CI: A free version at travis-ci.org and a paid version travis-ci.com. If you are using the paid version instead of the open source version, you need to login first, and use the argument --pro with every command. In this article, we will use travis-ci.com. Ensure you are logged in to travis-ci.com by running:

travis whoami --pro

If your username shows up, you're all set. Otherwise, sign in from your terminal using:

travis login --pro

The repository is automatically inferred from your git remote (which is GitHub). To tell Travis CI more about our application, what needs to be tested, and where our application should be deployed to, add the following configuration to .travis.yml :

language: python
python:
  - 3.6
before_install:
  - python --version
  - pip install -U pip
  - pip install -U pytest pytest-cov
  - pip install codecov
script: pytest
after_success:
  - pytest --cov=./
  - codecov
deploy:
  provider: elasticbeanstalk
  access_key_id:
    secure: $AWS_ACCESS_KEY_ID
  secret_access_key:
    secure: $AWS_ACCESS_KEY_SECRET
  region: $AWS_REGION
  app: $AWS_EB_APPLICATION
  env: $AWS_EB_ENVIRONMENT
  bucket_name: $AWS_EB_S3_BUCKET

Then add the necessary encrypted environment variables automatically to your .travis.yml file (see references above):

travis encrypt --pro AWS_ACCESS_KEY_ID="YOUR_KEY_ID_HERE" --add
travis encrypt --pro AWS_ACCESS_KEY_SECRET="YOUR_KEY_SECRET_HERE" --add
travis encrypt --pro AWS_REGION=$REGION --add
travis encrypt --pro AWS_EB_APPLICATION=$PROJECT_NAME --add
travis encrypt --pro AWS_EB_ENVIRONMENT=$ENVIRONMENT_NAME-$PROJECT_NAME --add
travis encrypt --pro AWS_EB_S3_BUCKET="YOUR_BUCKET_HERE" --add # change this as needed – a new bucket will be automatically created if it does not yet exist

It's strongly recommended to encrypt environment variables with sensititve information, such as AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY_SECRET. Sensitive information should never end up in source control. Check out this resource for more information on how to encrypt environment variables for Travis CI. An alternative to the above step (travis encrypt ...) is to configure the environment variables using the Travis CI web application and adding them to the repository settings. An alternative to setting encrypted environment variables is to use the following command: travis env set VARIABLE_NAME "value"

And... we're done! From now on, you can make changes to your application and simply push them to GitHub. Everything else is set up and done automatically for you: Travis CI will pull your code, perform the tests, and if they pass, deploy the application to AWS Elastic Beanstalk. Should the tests not pass, the application version will not be deployed, and you will be able to investigate further as to why.

Conclusion

You now have gone through the required steps to create a continuous deployment pipeline for your Flask application. Newly pushed changes are automatically analyzed and tested by Travis CI. If all tests pass, new versions of your Flask application will be continuously deployed to the specified environment(s) within AWS Elastic Beanstalk.

I hope you enjoyed this article – please let me know what you think, or if you have any questions.