DEV Community: Mark Ericksen

Better Business Intelligence in Elixir with Livebook

Mark Ericksen — Fri, 30 Jul 2021 16:57:02 +0000

As a developer, has your manager ever come and asked a question like, "How much money are we making?" If you were a line-of-business developer at a global insurance company, you'd reach for your handy, nosebleed-expensive Business Intelligence (BI) suite to answer this question. But you're not, so how did you answer it for them?

Obviously, you'd do what we all do. You'd SSH into your server, start an Elixir iex session or a Rails console, then run some scripts to query data, sum numbers, and come up with answers.

Well, give yourself a raise! Because you just built a BI suite.

It may not seem super sophisticated, but it solves the business need. And for problems like this, Livebook can be a better BI tool for Elixir developers.

BI What?

What is a BI tool?

Business intelligence (BI) comprises the strategies and technologies used ... for the data analysis of business information.

Translated from Gartner-speak, that means any tools you run to get a picture of how the business is doing are BI tools.

In the last bootstrapped startup I worked at, management routinely asked backend developers for business numbers. It was simple stuff, like:

How many new clients did we add this week?
What was our customer's spend?
Who are our top 10 customers this week and what were their numbers?

As simple as this stuff seems, it's really important for those business focused leaders to understand and make better decisions. That's why global insurance companies with applications that are too complicated to bring up a Rails console on spend six figures on BI suites.

How did we get those answers? Using our Elixir iex, or interactive shell. We ran some scripts and gave them CSV friendly rows they could add to their spreadsheets. In that early stage startup, we were using iex as our BI tool. At a startup before that, we did the same thing but using the Rails console.

If you're using the Rails console, Elixir's iex, or another REPL to examine your data, then that's your BI tool for now. But with Elixir, we can do better. Livebook gives you data, charts and graphs too, but because it's executing your Elixir code, it can also call out to your other integrations and pull in even more.

To understand why Livebook can be a better tool, let's go further and talk about BI tools in general, not just your REPL.

Old School Business Intelligence

Our premise in this post is that we can give "serious" BI tools a run for their money with Elixir and Livebook. Let's see what we're up against.

Companies spend lots of money every year on their BI tools. You hear some of the numbers and it seems bananas. But it's because they add a lot of value. Spotting trends in your data and customer behavior can make the difference of success and failure for a company.

Most BIS tools are commercial. But there's a handful of credible open source projects. Metabase, for example, is an open source BI tool that works quite well. It connects directly to an application's database and helps you do some spelunking, aggregation, and shiny graphing. You can even create and share custom dashboards. Think of it as Grafana, but for MBAs – it's a great tool.

Deploying Metabase alongside your app might look like this:

Metabase is an application that probably shouldn't be exposed publicly and it needs direct access to your database. It's also a bit of a mammoth – one doesn't just walk into Metabase and expect to get anything done, there's a real learning curve even when you know how to write SQL. It can be a heavy tool when you just want to do some quick poking around.

It's also an app you need to keep running. We sell hosting, so we're generally OK with that. In fact, Metabase ships a Docker image and Fly lets you quickly deploy apps using Docker. Money.

This is fine when you want a dedicated data dashboard or you want to let non-developers see reports and graphs and be business-intelligent. However, when a project is young and you're a developer, digging with code is powerful. This is where Livebook can help!

Why is Livebook Better?

Let's start with what Livebook is.

Livebook started out as Elixir's version of Jupyter Notebooks. Jupyter is pretty great. But it turns out that code notebooks on Elixir are something special; they do something you usually can't pull off in Python. That's because Elixir has powerful built-in clustering, built on Erlang's BEAM/OTP runtime. Livebook notebooks can talk directly to running Elixir apps. And so we can do analysis and visualization directly off the models in our applications.

Livebook really sings on Fly.io. We make it easy to deploy clusters of Elixir applications that can talk privately between themselves. More importantly: it's easy to bring up a secure WireGuard tunnel to those applications. So I can run Livebook locally on my machine, and attach it to any of my apps running on Fly.io!

For lean-and-mean startups, this is a win. You only need your app and your database running. Then, on an "as-needed" basis, you connect to your app with Livebook for analysis. Inside Livebook, analysis is done using your app's Elixir code. Livebook therefore doesn't need to connect directly to the database to run queries, and, even better, we get to re-use the business logic, associations, and schemas our apps already have.

As a BI tool, Livebook notebooks have these benefits:

They use your application's code, like a REPL lets you.
They have the ability to generate charts, graphics, and tables.
Notebooks are markdown files and can be checked-in with your project. They can be shared with the team! No more, "I can't run the numbers today because Bill is out and he has the scripts."
They are self-documenting because it's just markdown.
They are designed to be highly reproducible. They are easy to re-run when you want updated numbers.

Some of this you can pull off using just your REPL and some raw SQL queries. But why would you? It's easier to use your project's code, database models (Ecto schemas for Elixir) and associations.

Further, your apps probably rely on external services like Stripe. Because Livebook talks directly to your Elixir code, you can query those external services with that code, and then combine the results with your data. This is a combination that even dedicated tools like Metabase have a hard time beating.

Connecting to Your App on Fly

Great! You have a notebook that loads and visualizes some data! To get the benefits, you need to connect it to your app running on Fly.io.

Follow the Fly.io Elixir Guide for Connecting Livebook to your App in Production to connect Livebook to your app.

With Livebook connected to your app, you can run your notebook and start gaining insight to your data!

Gaining Intelligence

How you use Livebook depends on your application and your industry.

Here are some ideas to get the brain juices flowing. Each of these example notebooks would be a "module" in a serious commercial BI suite, and you'd pay $45k for a license for it.

User Account Setup

How many of your users have fully set up their accounts?

Build and share a notebook that tracks where users are in your onboarding process.

Users Bouncing From The App

Where are accounts stalling out in onboarding?

Graph the stages accounts are at and let analysts drill into the onboarding funnel.

Sales Analysis

How were sales for your products or services last month?

Graph a multi-series chart comparing sales across products.

What about total sales per week?

Build a notebook with a date input making it easy to switch the week being charted.

Integrations

What external financial systems are you integrated with?

Notebooks can execute your code to query those services and visualize refund rates, processing fees, and more.

Wrapping Up

What's great about the Livebook approach is you are writing working Elixir code. When you are ready to build an Admin Dashboard page in your app, you've already done the hard work of figuring out what data is valuable and even the code for how to get it!

Monitoring Elixir Apps on Fly.io With Prometheus and PromEx

Mark Ericksen — Tue, 29 Jun 2021 20:06:15 +0000

Fly.io takes Docker containers and converts them into fleets of Firecracker micro-vms running in racks around the world. If you have a working Docker container, you can run it close to your users, whether they're in Singapore or Amsterdam, with just a couple of commands. Fly.io is particularly nice for Elixir applications, because Elixir's first-class support for distributed computing meshes perfectly with Fly.io's first-class support for clusters of applications.

This post is about another cool Fly.io feature --- built-in Prometheus metrics --- and how easy it is to take advantage of them in an Elixir application. I wrote and maintain an Elixir library, PromEx, that makes it a snap to export all sorts of metrics from your Elixir applications and get them on dashboards in Grafana. Let's explore some of the concepts surrounding Prometheus and see how we can leverage the Fly.io monitoring
tools in an Elixir application to get slick looking dashboards like this one:

Why Application Monitoring is Important

When customers are paying for your application or service, they expect it to work every time they reach for it. When things break or errors occur, your customers will not be happy. If you are lucky, your customers send you an email letting you know that things are not working as expected. Unfortunately, many of these occurrences go unreported.

Knowing exactly when things are going wrong is key to keeping your customers happy. This is the problem that monitoring tools solve. They keep an eye on your application, and let you know exactly when things are behaving suboptimally.

Imagine for example that you have an HTTP JSON API. You deploy a new version that changes a bunch of endpoints. Assume it's infeasible to go through every single route of your application every time you deploy, or to test each endpoint individually with every permutation of input data. That would take far too much time, and it doesn't scale from an organizational perspective: it would keep engineers constantly context switching between feature work and testing new deployments.

A more scalable solution: briefly smoke test the application after a deployment (as a sanity check), and then use monitoring tooling to pick up on and report on any errors. If your monitoring solution reports that your HTTP JSON API is now responding with 400 or 500 errors, you know you have a problem and you can either rollback the application, or stop it from propagating to across the cluster. The key point is that you can proactively address issues as opposed to being blind to them, and at the same time you can avoid sinking precious engineer time into testing all the things.

While ensuring that production users are not experiencing issues is a huge benefit of application monitoring, there are lots of other benefits. They include:

Quantifying stress testing results
Business priority planning based on real usage data
System performance and capacity planning

Let's dig into how Prometheus achieves these goals at the technical level.

How Does Prometheus Work?

At its core, Prometheus is a time-series database that enables you to persist metrics in an efficient and performant manner. Once your metrics are in the Prometheus time-series database, you can create alerting rules in Grafana. Those alerts can then be triggered once certain thresholds and criteria are met, letting you know that something has gone wrong.

"But how exactly do my application metrics end up in Prometheus?" Well, your Prometheus instance is configured to scrape all of your configured applications. At a regular interval, each of their instances is queried for metrics data, which is stored in a database. Specifically, it makes a GET HTTP call to /metrics (or wherever your metrics are exposed) and that endpoint will contain a snapshot in time of the state of your application. Once your metrics are in Prometheus, you can query the time-series database with Grafana to plot the data over time; Grafana uses PromQL to refresh data and update its panels.

Given that Prometheus scrapes your applications at a regular interval, the resolution of your time-series data is bound to that interval. In other words, if you get 1,000 requests in
the span of 10 seconds, you don't know exactly at what timestamps those 1,000 requests came in, you just know that you got 1,000 requests in a 10 second time window. While this may seem limiting, it is actually a benefit in disguise. Since Prometheus doesn't need to keep track of every single timestamp, it is able to store all the time-series data very efficiently.

Luckily with Fly.io, the administration and management of Prometheus can be taken care of for you!

Turning On Prometheus On Fly

Managing, configuring and administering your own Prometheus instance can be a bit of a tall order if you have never worked with Prometheus before. Fortunately, all you need to do to enable Prometheus metrics for your application is add a couple of lines to your fly.toml manifest file. All Fly.io needs to know is what port and path your metrics will be available at. For the TODO List Elixir application for example, the following configuration was all that was needed:

[metrics]
port = 4000
path = "/metrics"

In order to visualize your Prometheus metrics, you'll need to have an instance of Grafana running somewhere. You could deploy your own Grafana instance on Fly.io by following this guide, but you can also use Grafana Cloud (it has a free plan) --- Grafana Cloud works fine with Fly. Which ever route you take, all you then need to do is configure Grafana to communicate with the Fly.io managed Prometheus instance and you are good to go!

Now that we've got Prometheus hooked up, we need to get our Elixir application to start providing metrics.

Monitoring Elixir with PromEx

Whenever I write a production-grade Elixir application that needs monitoring, I reach for PromEx.

I wrote PromEx and maintain it because I wanted something that made it easy to manage both the collection of metrics and the lifecycle of a a bunch of Grafana dashboards. That's to say: PromEx doesn't just export Prometheus metrics; it also provides you with dashboards you can import into Grafana to immediately get value out of those metrics. I think this is a pretty ambitious goal and I'm happy with how it turned out. Let's dig in.

At a library design level, PromEx is a plugin style library, where you enable a plugin for whatever library you want to monitor. For example, PromEx has plugins to capture metrics for Phoenix, Ecto, the Erlang VM itself, Phoenix LiveView and several more. Each of these plugins also has a dashboard to present all the captured metrics for you. In addition, PromEx can communicate with Grafana using the Grafana HTTP API, so it will upload the dashboards automatically for you on application start (if you configure it that is). What this means is that you can go from zero to complete application metrics and dashboards in less that 10 minutes!

In the Elixir example application, you can see that the PromEx module definition specifies what plugins PromEx should initialize, and what dashboards should be uploaded to Grafana:

defmodule TodoList.PromEx do
  use PromEx, otp_app: :todo_list

  alias PromEx.Plugins

  @impl true
  def plugins do
    [
      # PromEx built in plugins
      Plugins.Application,
      Plugins.Beam,
      {Plugins.Phoenix, router: TodoListWeb.Router},
      Plugins.PhoenixLiveView
    ]
  end

  @impl true
  def dashboard_assigns do
    [
      datasource_id: "prometheus"
    ]
  end

  @impl true
  def dashboards do
    [
      # PromEx built in Grafana dashboards
      {:prom_ex, "application.json"},
      {:prom_ex, "beam.json"},
      {:prom_ex, "phoenix.json"},
      {:prom_ex, "phoenix_live_view.json"}
    ]
  end
end

With a little bit of configuration in runtime.exs PromEx can communicate with Grafana to take care of the graph annotations and dashboard uploads:

config :todo_list, TodoList.PromEx,
  manual_metrics_start_delay: :no_delay,
  grafana: [
    host: System.get_env("GRAFANA_HOST") || raise("GRAFANA_HOST is required"),
    auth_token: System.get_env("GRAFANA_TOKEN") || raise("GRAFANA_TOKEN is required"),
    upload_dashboards_on_start: true,
    folder_name: "Todo App Dashboards",
    annotate_app_lifecycle: true
  ]

With the managed Prometheus instance from Fly.io, and the metrics collection from PromEx, you have an instrumented application in record time! Here are some snapshots from the auto generated dashboards for the Todo List application:

And That's It!

Elixir makes it easy to run ambitious, modern applications that take advantage of distributed computing. It should be just easy easy to see what those applications are actually doing, and to have alerts go off when they misbehave. Between Fly.io's built-in Prometheus and the PromEx library, it's easy to get this kind of visibility. Your application can be instrumented with dashboards and annotations in a coffee break's worth of time.

Be sure to check out the Todo List application Repo for more technical details and all the code necessary to do this yourself. What used to take a few days to set up and run, now only takes a few hours, so be sure to give it a test drive!

Observing Elixir in Production

Mark Ericksen — Wed, 16 Jun 2021 21:25:05 +0000

Fly networking lets you VPN in and run Observer directly in production. Deploy your Elixir application and try it out!

Elixir, Erlang, and really just the BEAM has a feature called "Observer". It's fun showing it to people new to Elixir because it's just so cool! It's a WxWidgets graphical interface that connects in realtime to a running Erlang node and lets you "observe" what's going on. It has some limited ability to modify things as well, most notably you can kill running processes. This can help when something is misbehaving or you just want to play "chaos monkey" and kill parts of the system to see how it recovers.

This picture shows a process tree for the application. Using this I can inspect individual processes or even kill them!

One very cool way to run Observer is to run it on your local machine (which has the ability to display the UI) and connect to a production server (with no windowing UI available) and "observe" it from a distance. So yeah... have a problem in production? Not sure what's going on? You can tunnel in, crack the lid and poke, prod, and peek around to see what's going on.

The Fly platform makes it easy to do this for your own applications!

What We Will Do

Fly.io natively supports WireGuard, Jason Donenfeld’s amazing VPN protocol. If you’ve ever lost hours of your life trying to set up an IPSec VPN, you’ll be blown away by how easy WireGuard is. It’s so flexible and performant that Fly uses it as our network fabric. And it’s supported on every major platform, including macOS, iOS, Windows, and Linux. What that means for you is that if your app runs on Fly, you can open a secure, private, direct connection from your dev machine to your production network, in less time than it took me to write this paragraph. Cool, right?

This is what we're going to do.

We will bring up a secure WireGuard tunnel that links to your servers on Fly. In this graphic, there are two my_app Elixir nodes clustered together running on Fly.

From the local machine, we can open an IEx terminal configured to join that cluster of remote Elixir nodes. Our local machine supports running Observer and drawing the UI. We use our local observer to talk to the remote nodes in the cluster!

Making It Happen

To test this out, I follow this guide and apply the changes to the multi-region Tic-Tac-Toe game created here. The github repo for the project is here.

Here's what we do:

Configure an Elixir release to use a cookie value we provide.
Setup WireGuard for Fly. This is a VPN technology that let's us directly connect to the production private network.
Create a simple script to launch Observer for us.
Launch Observer and explore!

Again, follow the guide here for a step-by-step breakdown of how to do it for your project.

Multi-Region Support?

When Elixir nodes are clustered together and running in different regions, Observer can connect to any node in the cluster.

After making the changes to the TicTac project and deploying it to multiple regions, let's see what it looks like.

fly status

...

Instances
ID       VERSION REGION DESIRED STATUS  HEALTH CHECKS      RESTARTS CREATED
79510f86 17      fra    run     running 1 total, 1 passing 0        23m39s ago
df93ea35 17      lax(B) run     running 1 total, 1 passing 0        24m3s ago

I have the game scaled out to two regions. One is running in fra (Frankfurt, Germany) and the other is running in lax (Los Angeles, California (US)).

When I open Observer locally, I see two remote instances of tictactoe!

Exploring My App

I start playing a game on the web and use Observer to browse around and find the game. It's highlighted in blue and linked from GameRegistry.

Using Observer, I can double-click the selected game process and even view the GenServer's state. This gives a snapshot of the state at the time I double-clicked it. I highlighted in yellow some interesting parts of the game state.

If you're wondering why the state data looks strange (at least different from Elixir), it's because that's the Erlang representation of those data types.

Playing Chaos Monkey

Something fun you can do with Observer is identify processes, inspect them, and even kill them. This can help when something is misbehaving and you want to see more about what's going on. This can also help you test how your system recovers from unexpected failures.

I've already identified the running game process. By right-clicking it I see I have the option to "kill" it. What will happen when a game server dies?

After killing the game process, I see that a new process was immediately started. How can I tell? The PID (Process ID) value is different.

So, it looks like my system isolates the damage in that no other running games are impacted when one crashes. Yay! My system stays up and running!

From the user's perspective playing the game, it's not so graceful. The player is able to recover but it requires them to reload their page or restart the game joining processing. I see that my UX can be improved to make crash recovery better for the user.

That was a productive experiment!

Now You Try

Deploy a Phoenix application to Fly, setup your WireGuard VPN, and start observing your app in production!

With Elixir you can build resilient systems! Fly makes observing them easy. What will you find digging around in the state pile?

Deploy your Elixir App!

ThinkingElixir 049: Pushing for Modularity with Maciej Kaszubowski

Mark Ericksen — Tue, 25 May 2021 12:35:40 +0000

In episode 49 of Thinking Elixir, Maciej Kaszubowski returns to talk about how read models can help modularize our systems. There is a focus on background job systems and how they fit in our modular designs. We cover push vs pull based systems and which is more maintainable. Maciej also shares his approach to learning and understanding a new Elixir library and more!

Podcast Episode

ThinkingElixir 048: Ecto Associations with Sheharyar Naseer

Mark Ericksen — Tue, 18 May 2021 12:24:50 +0000

In episode 48 of Thinking Elixir, we talk with Sheharyar Naseer about Ecto’s feature “association defaults” and how they work. They support defining a function that gets called during your changeset operations and can be used to ensure that multi-tenancy IDs get populated down correctly. We learn more about Slab, talk about Single Table Inheritance designs and how we feel about them. We also talk about wrapping our Repo module for fun and profit!

Podcast Episode