DEV Community: Rosa Richter

Quick candlestick summaries with Elixir's Explorer

Rosa Richter — Mon, 22 Aug 2022 20:03:09 +0000

One of the main visualizations of any kind of financial market is the candlestick chart. In a candlestick chart, trades are summarized into time periods with that period's high price, low price, price at the beginning of the time period, and the price at the end of the time period. Here's an example, taken from TradingView, a popular website that provides a lot of charts and tools.

Each of the boxes represent the asset price at the beginning and ending of the time period, and the lines above and below indicate the maximum and minimum price over that period. It's a very succinct way of summarizing a lot of data.

I have a toy project where I'm making a cryptocurrency exchange website, and I've been playing with the Elixir Nx project's Explorer library to manipulate this data. I found that you can summarize trade data into candlestick data extremely easily. This transformation is also very fast, because Explorer uses a Rust-based backend called Polars for the actual data manipulation.

elixir-nx / explorer

Series (one-dimensional) and dataframes (two-dimensional) for fast data exploration in Elixir

Explorer brings series (one-dimensional) and dataframes (two-dimensional) for fast data exploration to Elixir. Its high-level features are:

Simply typed series: :float, :integer, :boolean, :string, :date, and :datetime.
A powerful but constrained and opinionated API, so you spend less time looking for the right function and more time doing data manipulation.
Pluggable backends, providing a uniform API whether you're working in-memory or (forthcoming) on remote databases or even Spark dataframes.
The first (and default) backend is based on NIF bindings to the blazing-fast polars library.

The API is heavily influenced by Tidy Data and borrows much of its design from dplyr. The philosophy is heavily influenced by this passage from dplyr's documentation:

By constraining your options, it helps you think about your data manipulation challenges.

It provides simple “verbs”, functions that correspond to the most common data manipulation tasks, to help you translate…

View on GitHub

The raw data for my candlestick chart comes from the trades table in my database, which looks like this (with some extraneous fields omitted):

`price`	`quantity`	`executed_at`
100	100	2022-08-22T12:47:17.123Z
99	1000	2022-08-22T12:47:17.456Z
101	5	2022-08-22T12:47:17.789Z

For simplicity, price and quantity are just integers. You can insert your decimal places as appropriate for a given currency.

The main challenge in summarizing this data into candlesticks is to group it by time period. This is a place where you'll need to iterate through all the query results, which you'd need to do anyway in order to transform the DateTimes from your Ecto query into NaiveDateTimes, which are the only time type that Explorer understands. To group this data by time period, you have to find the beginning of your time period when given the trade's timestamp. If you are creating one-second-long candlesticks, you'll use the NaiveDateTime.truncate/2 function for this. So you'll prepare your data for an Explorer.DataFrame like this:

iex> trades =
...>   MyApp.Repo.all(MyApp.Trade)
...>   |> Enum.map(fn t ->
...>     naive_time = DateTime.to_naive(t)
...>     open_time = NaiveDateTime.truncate(naive_time, :second)
...>
...>     t
...>     |> Map.put(:executed_at, naive_time)
...>     |> Map.put(:open_time, open_time)
...>   end)
[
  %{
    open_time: ~N[2022-08-22 12:47:17.000],
    executed_at: ~N[2022-08-22 12:47:17.123],
    price: 100,
    quantity: 100
  },
  ...
]

Now we can give this to Explorer.DataFrame.new().

iex> df = Explorer.DataFrame.new(trades)
#Explorer.DataFrame<
  Polars[3 x 4]
  executed_at datetime [2022-08-22 12:47:17.123000, ...]
  open_time datetime [2022-08-22 12:47:17.000000, ...]
  price integer [100, ...]
  quantity integer [100, ...]
>

Once we've got our data in a DataFrame, it's trivial to do the actual candlestick summary. That's because Explorer data frames offer the Explorer.DataFrame.summarise/2 function, which performs the exact operations we want. You'll just need to make sure the data is in the right order, and is grouped by the open_time value we calculated earlier.

iex> alias Explorer.DataFrame
...> candles =
...>   df
...>   |> DataFrame.arrange(:executed_at)
...>   |> DataFrame.group_by(:open_time)
...>   |> DataFrame.summarise(price: [:first, :last, :max, :min])
#Explorer.DataFrame<
  Polars[1 x 5]
  open_time datetime [2022-08-22 12:47:17.000]
  price_first integer [100]
  price_last integer [101]
  price_max integer [101]
  price_min integer [99]
>

Now you have data for a candlestick chart! There is plenty of room for optimization here, like storing your trading timestamps as NaiveDateTime structs so we don't need to do the conversion. You might be able to do this all within a database query, in fact, but I will argue that you can't beat the simplicity of Explorer.DataFrame.summarise/2. Turning this data into an actual image is an exercise left for the reader.

If you want to see an example of how an event-sourced cryptocurrency exchange might work, check out my exchange project.

Cantido / exchange

A market application with a focus on cryptocurrencies

Exchange

To start your Phoenix server:

Install dependencies with mix deps.get
Create and migrate your database with mix ecto.setup
Install Node.js dependencies with npm install inside the assets directory
Start Phoenix endpoint with mix phx.server

Now you can visit localhost:4000 from your browser.

Ready to run in production? Please check our deployment guides.

Learn more

Official website: https://www.phoenixframework.org/
Guides: https://hexdocs.pm/phoenix/overview.html
Docs: https://hexdocs.pm/phoenix
Forum: https://elixirforum.com/c/phoenix-forum
Source: https://github.com/phoenixframework/phoenix

View on GitHub

Phoenix LiveView, but event-sourced

Rosa Richter — Sun, 16 Jan 2022 21:15:05 +0000

Event sourcing is a powerful way to structure an application, and I've discovered that it pairs beautifully with Phoenix LiveView.

The context: I'm building a cryptocurrency exchange application. I don't have the business chops to run an actual exchange, so this is just for fun. The application is built in Elixir, using the Commanded framework for CQRS/ES goodness, and Phoenix LiveView because it's the hot new thing that I wanted to learn.

My goal is to use LiveView to update a price chart as trades are executed by the system. A LiveView process is a lot like a GenServer, with a bespoke process for each client, executing handle_* functions as the client does things. The first step to real-time chart updates is to trigger one of these handler functions in my LiveView controller when a trade is executed. I'm using Commanded's own EventStore library to dispatch and store my events, so their documentation is the place to start.

In the EventStore documentation, I found that subscribing to the event stream is really simple: it's a single function call. Here's the example from EventStore's documentation on transient subscriptions that we care about:

alias EventStore.RecordedEvent
alias MyApp.EventStore

EventStore.subscribe(stream_uuid, selector: fn
  %RecordedEvent{data: data} -> data != nil
end)

# receive first batch of mapped event data
receive do
  {:events, %RecordedEvent{} = event_data} ->
    IO.puts("Received non nil event data: " <> inspect(event_data))
end

All we need to do is call EventStore.subscribe/2, optionally with a selector function, and then the current process will start receiving events. We're going to call this in our LiveView's mount/3 callback. I'm also going to load the initial set of data here.

defmodule ExchangeWeb.DashboardLive do
  use ExchangeWeb, :live_view
  alias EventStore.RecordedEvent
  alias Exchange.EventStore

  def mount(_, _, socket) do
    :ok = EventStore.subscribe("$all",
      selector: fn %RecordedEvent{event_type: type, data: data} ->
          type == "Elixir.Exchange.Orderbook.TradeExecuted" and
          data.symbol == "BTCUSDT"
      end,
      mapper: fn %RecordedEvent{data: data} -> data end)

    trades = Exchange.Orderbooks.trades(symbol)

    {:ok, assign(socket, trades: trades)}
  end


  def handle_info({:events, events}, socket) do
    trades =
      Enum.map(events, &Map.take(&1, [:timestamp, :price]))
    {:noreply, push_event(socket, "trades", %{trades: trades})}
  end
end

Now, if you were using a server-side charting library like ContEx, then you would just append the new events to what you've already got assigned to the socket, and your normal rendering function would rebuild the chart. You're done! But I wanted to make it more complicated.

I'm using Chart.js, which is a popular graphing JavaScript library. It lives entirely on the client-side, which isn't very agreeable with Phoenix LiveView's server-side focus. Fortunately, LiveView allows you to set up JavaScript hooks, and then push events to them. We can make the client event-sourced, too! That's why I'm using push_event/3 instead of assign/3 in the example above. I'm using LiveView's JavaScript hooks to respond to events that I push from the LiveView process. Read more about LiveView JavaScript interoperability, it's really interesting.

A LiveView client hook is an object containing a couple of functions.
We're going to create a mounted() function to initialize the chart with the data we already have. After that, we're going to set up an event handler. The mounted() function receives a this object with a few utilities, with the handleEvent function being the most important to us. We call this.handleEvent to set up the function that will handle the event that we pushed in our LiveView module's handle_info callback.

import Chart from 'chart.js/auto';

let Hooks = {};

Hooks.TradesChart = {
    mounted() {
        let chart = new Chart(this.el, {
           # Configuration for your chart, nothing interesting here
        });

        this.handleEvent("trades", (event) => {
            event.trades.forEach((trade) => {
                chart.data.datasets.forEach((dataset) => {
                    trade.executed_at = new Date(Date.parse(trade.executed_at))
                    dataset.data.push(trade)
                })
            })
            chart.update();
        });
    },
};

let liveSocket = new LiveSocket("/live", Socket, {hooks: Hooks})

This will push new data into the chart without asking it to completely re-render.

Lastly, we need to attach the hook to the element containing the chart. That's accomplished by adding a phx-hook attribute to your markup element:

<canvas id="trades-chart" phx-hook="TradesChart" width="400" height="200"></canvas>

By adding that attribute, you've told Phoenix LiveView to call the mounted() hook when the <canvas> element is mounted, which will then subscribe the update function to "trade" events sent by the backend.

All together, EventStore pushes events to our LiveView process, which pushes an event to our client hook, which puts the new data in the chart. Event sourcing is so cool!

See the whole project here: