DEV Community: aziz abdullaev

WYSIWYG editor in LiveView with embedded images that are uploaded to S3 bucket

aziz abdullaev — Sat, 19 Apr 2025 18:03:41 +0000

In this tutorial, you will learn how to create WYSIWYG editor with image insertion (+ auto upload to S3) in LiveView, store + show the content preserving styling.

My setup:

Elixir 1.18.1
Erlang OTP 27
Phoenix 1.18.0-rc
LiveView 1.0.9

TLDR:
We are going to create a JS hook that uses Quill.js as WYSIWYG editor; convert quill.js HTML to Markdown, decode base64 images from editor, upload the binary to S3, then replace base64 image with S3 link, then save it to the DB. To show markdown on a page, we are going to convert markdown to HTML and inject it into our page with Mdex and Phoenix.HTML.raw(), then style it all with tailwind typography plugin

First, let’s create a Blogs context. By running:

 mix phx.gen.context Blogs Blog blogs title:string content:text meta:map

Now, let’s create a LiveView page for creating a blog post.

Here is what our HTML will look like. It is just a basic .form with an input field for title only and also a div with “WYSIWYG_editor” hook.

Let’s set up WYSIWYG editor on a hook.

Go to /assets folder add quill.js (wysiwyg editor) and turndown (html to markdown converter).

npm I quill turndown

Now, let’s set up WYSIWYG editor:

Then, add snow theme css to root.html.heex

    <link href="https://cdn.jsdelivr.net/npm/quill@2.0.3/dist/quill.snow.css" rel="stylesheet" />

Now, our /blogs/new page looks like this:

To validate content inside of editor, we will have to send changes to the LiveView and validate in the LiveView with Ecto.Changeset. To do that, we are going to add

# in the hook
        editor.on("text-change", () => {
          view.pushEvent("content-text-change", { content: editor.getText() });
        });

Event handler in LV:

  def handle_event("content-text-change", %{"content" => content}, socket) do
    params =
      socket.assigns.form.source.params
      |> Map.put("content", content)

    changeset = Blog.changeset(%Blog{}, params)

    socket =
      socket
      |> assign(form: to_form(changeset, action: :validate))

    {:noreply, socket}
  end

Handling errors happens with this hack function that is a clone of translate_error/1 from core_components (I tried to do directly use translate_error() but it did not work, idk why)

Now, let’s talk about how we are going to link LV form with the hook. When user submits a form, we are handling it in LV and sending a signal for the hook, hook prepares the content and sends it back to LV, then, we are going to extract + upload + replace any submitted images, then try to save the submitted blog.

LV (on form submission) — ‘form_submitted’ —> HOOK

HOOK (convert HTML to markdown) —‘editor_content_markdown’—> LV

LV (handle the submission, extract images if any, create a blog post, then redirect to blogs show page)

Below is the sample of submitting to a blog post with an image. As you can see, the image comes base64 encoded from quill.

What does extract_base64_images do?

It takes in the submitted content from the editor, and regex matches any images that are embedded, decodes the image into binary, uploads the binary to S3, gets the S3 link, and substitutes base64 encoded image with a link for the image in S3 bucket. It also returns some logs on files.

Here is what create blog post looks like after we extract + upload + update the submitted post

Now that we have content stored as markdown in our db, how are we going to present it to users with styling?

First, we have to convert markdown into html. We can do that with mdex package.

Then, we have to inject the html into the page.

To handle styling, we can use tailwind and its typography plugin. (Of course, you can do it yourself, create ‘md’ class to a container of the blog post, then manually write css rules for each element that comes of markdown, like `md > h1 {text….}; md > h2 {…}, but tailwind got us covered)

I discovered that Phoenix 1.8-rc does not come with typography plugin pre-installed, so we are going to install it on our own (can devs do that without cursor these days? Elixir devs surely can)

Go to /assets folder and run the command below:

npm install -D @tailwindcss/typography

Then, go to your css/app.css file and add this line: 
@plugin "@tailwindcss/typography”;

Then, we add prose as a class to a container containing the decoded markdown and finally arrive at:

Which beautifully shows the user submitted posts with the styling from WYSIWYG editor:

Repo with working demo is available here: https://github.com/azyzz228/wysiwyg-editor-liveview. Note that I am using DigitalOcean S3 bucket with can which is why in my dev.exs and runtime.exs, ex_aws config has custom host

Note:
Unordered lists are handled weirdly by Quill. Instead of having <ul> and <li>s inside, they have <ol><li class="unordered-kind-of"><ol>, which is why unordered lists are shown as ordered ones. It is an easy fix on JS side before converting to markdown with turndown, we gotta find ol nodes that have li-s with certain class name, then change ol to ul. but I was lazy to fix it here, sorry

AI powered app (with open-source LLMs like Llama) with Elixir, Phoenix, LiveView, and TogetherAI

aziz abdullaev — Thu, 25 Jul 2024 18:15:35 +0000

Ever wanted to raise millions of $ by adding "AI" to your company name; and you happen to want to code it using Elixir, Phoenix, LiveView? This is the tutorial on making API calls to model providers like Together AI; Wait, not just calls, but HTTP streams to receive the data in chunks.

Big picture for any stack:

Get prompt from the user
Send the prompt to the TogetherAI through API call
Handle incoming chunks of LLM output

TLDR:

In Elixir world, we are going to have two processes, one for liveview and another process that will handle HTTP call with streams. LiveView will send the prompt and its pid (process id) to the handler, that in turn will spawn a separate process that will make HTTP call and send the chunks of LLM output to the LiveView as the chunks arrive. When the last chunk arrives, we then notify the LiveView that the text generation has finished.

Setup

Generate new Phoenix project by running

mix phx.new phoenix_playground

cd ./phoenix_playground
# you will need to create running PostgreSQL db. Alternatively, just use --sqlite flag when generating phoenix project 
mix ecto.create
iex -S mix phx.server

Go to mix.exs and add Req to dependencies:

{:req, "~> 0.5.0"}

Implementation

Let's start with the LiveView. Here is the scaffold of the LiveView with a form that will handle input and on submission, set loading state to true and send the prompt to the TogetherAI.

defmodule PhoenixPlaygroundWeb.HomeLive do
  alias PhoenixPlayground.TogetherAi
  use PhoenixPlaygroundWeb, :live_view

  def mount(_params, _session, socket) do
    socket =
      socket |> assign(:loading, false) |> assign(:text, "")

    {:ok, socket}
  end

  def render(assigns) do
    ~H"""
    <div class="mx-auto max-w-3xl my-20">
      <div class="flex items-start space-x-4">
        <div class="min-w-0 flex-1">
          <form phx-change="validate" phx-submit="submit" id="prompt-form" class="relative">
            <div class="">
              <label for="prompt" class="sr-only">Add your prompt</label>
              <input
                type="text"
                name="prompt"
                id="prompt"
                class="block w-full "
                placeholder="Let me know what you want to achieve"
              />
            </div>

            <div class="flex-shrink-0">
              <button
                :if={not @loading}
                type="submit"
                class="tailwind-goes-here"
              >
                Send
              </button>
              <button
                :if={@loading}
                type="submit"
                class="tailwind-goes-here"
              >
                Loading...
              </button>
            </div>
          </form>
        </div>
      </div>

      <p :if={not is_nil(@text)}><%= @text %></p>
    </div>
    """
    
    def handle_event("validate", _, socket) do
        {:noreply, socket}
    end

    def handle_event("submit", %{"prompt" => prompt}, socket) do
        # TODO: submit our prompt to the Together AI (or OpenAI or Mistral)
        TogetherAI.stream_completion(prompt, self())

        socket =
          socket |> update(:loading, &toggle_loading/1)

        {:noreply, socket}
    end

  end

Now, add this LiveView to router.ex:

live "/home", HomeLive

You will need to register at TogetherAI and get your API key to access the platform. Upon registration, you will receive $5 in credits which is more than enough for this tutorial. Assuming you have your API key, go to the paltform, then pick a LLM model you would like to use.

Let's start writing TogetherAI module.

defmodule PhoenixPlayground.TogetherAi do
  def stream_completion(prompt, pid) when is_binary(prompt) do
    url = "https://api.together.xyz/v1/chat/completions"

    body = %{
      model: "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo", # your model id
      messages: [
        # some system prompts to help fine tune the output. Write whatever you want here or just ignore it
        %{
          role: "system",
          content:
            "Your response must be in a format of bullet list with actionable items, like a todo-list. Your response must not start with 'Here is...'. Give only 2 actionable items, no more. Do not add any additional comments after actionable items. Give brief response."
        },
        # here, we are including the prompt by user
        %{role: "user", content: prompt}
      ],
      stream: true,
      max_tokens: 512,
      temperature: 0.7,
      top_p: 0.7,
      top_k: 50,
      stop: "[DONE]"
    }

    Task.async(fn ->
      Req.post(
        url,
        json: body,
        auth: {:bearer, "<your-api-key>"},
        into: fn {:data, _data} = data, {_req, _resp} = req_resp ->
          handle_stream(data, req_resp, pid)
        end
      )
    end)
  end
end

Here, we are constructing the request configs as needed, then creating a separate process with Task.async() where we make a post request to the platform and specify the function that will be handling the streams.

Req does a lot of heavy lifing for us when it comes to HTTP request and streams. We will just need to write handle_stream(). More on it here https://hexdocs.pm/req/Req.Request.html.

Here is the implementation for handle_stream() given that succes is successful with 200 status code.

  defp handle_stream(
         {:data, data},
         {req = %Req.Request{}, resp = %Req.Response{status: 200}},
         pid
       ) do

    # string manipulations
    decoded =
      data
      |> String.split("data: ")
      |> Enum.map(fn str ->
        str
        |> String.trim()
        |> decode_body()
      end)
      |> Enum.filter(fn d -> d != :ok end)

    case handle_response(decoded) do
      # LLM finished generating, so we are informing the LiveView about it
      {text, :finish} ->
        send(pid, {__MODULE__, "last_chunk", text})

      # LLM generated text, so we are sending it to the LiveView process
      generated_text ->
        send(pid, {__MODULE__, "chunk", generated_text})
    end

    {:cont, {req, resp}}
  end


  defp handle_response(decoded) when is_map(decoded) do
    decoded
    |> Map.get("choices")
    |> List.first()
    |> Map.get("delta")
    |> Map.get("content")
  end

  defp handle_response(decoded) when is_list(decoded) do
    result =
      Enum.reduce(decoded, "", fn choices_map, acc ->
        case choices_map do
          :finish ->
            {acc, :finish}

          map ->
            acc <> handle_response(map)
        end
      end)

    result
  end

  defp decode_body(""), do: :ok
  defp decode_body("[DONE]"), do: :finish
  defp decode_body(json), do: Jason.decode!(json)

After receiving a response chunk, we need to do couple of string manipulations before decoding it with Jason. Note that decode_body() function returns :ok on empty string, and :finish when we reach the stop string (in our case it is "[DONE]".

After we decode the response, we need to extract the text generated by LLM. Here is the sample response decoded from LLM:

  %{
    "choices" => [
      %{
        "delta" => %{
          "content" => "•",
          "role" => "assistant",
          "token_id" => 6806,
          "tool_calls" => nil
        },
        "finish_reason" => nil,
        "index" => 0,
        "logprobs" => nil,
        "seed" => nil,
        "text" => "•"
      }
    ],
    "created" => 1721912534,
    "id" => "some-id",
    "model" => "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    "object" => "chat.completion.chunk",
    "usage" => nil
  }

Streamed LLM output might contain a single world or multiple words. When it contains a single world, we handle it as a map. When it contains many words in one response chunk, we need to extract the words together and put them together. That's what is going on in this function:

  defp handle_response(decoded) when is_list(decoded) do
    result =
      Enum.reduce(decoded, "", fn choices_map, acc ->
        case choices_map do
          :finish ->
            {acc, :finish}

          map ->
            acc <> handle_response(map)
        end
      end)

    result
  end

Here, we are looping through multiple outputs and extracting the text by recursively calling handle_response(map).

Output with multiple chunks of text will keep arriving. A list of chunks may arrive where the last statement will be "[DONE]" indicating the end of LLM output. When the last message arrives, we will decode it to :finish which we will send to the LiveView.

When we reach the end of output, we are doing to receive the :finish atom, but we are also going to receive the chunk as below, with usage that indicates how many tokens were used, and finish_reason with the finish reason instead of nil. The token usage and finish reasons are not to be ignored when you build your SAAS. Remember, this is the part where you multiple token cost by multiple X to make profit :)

  %{
    "choices" => [
      %{
        "delta" => %{
          "content" => " measurements",
          "role" => "assistant",
          "token_id" => 22323,
          "tool_calls" => nil
        },
        "finish_reason" => "length",
        "index" => 0,
        "logprobs" => nil,
        "seed" => 16606951688656440000,
        "text" => " measurements"
      }
    ],
    "created" => 1721827856,
    "id" => "8a8dcdd7e2d74-ARN",
    "model" => "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo",
    "object" => "chat.completion.chunk",
    "usage" => %{
      "completion_tokens" => 512,
      "prompt_tokens" => 93,
      "total_tokens" => 605
    }
  }

Now, let's handle the responses on the LiveView. There are two function that we will handle the incoming message from another process. One handles the LLM chunk(s) and other handles finishing chunk(s). As we receive the text, we are going to append it with the text we have received before.

When we receive the last chunk, then we are going to show the flash with "Finished generating".

  def handle_info({PhoenixPlayground.TogetherAi, "chunk", text}, socket) when is_binary(text) do
    socket =
      socket |> update(:text, &(&1 <> text))

    {:noreply, socket}
  end

  def handle_info({PhoenixPlayground.TogetherAi, "last_chunk", text}, socket)
      when is_binary(text) do
    socket =
      socket
      |> update(:text, &(&1 <> text))
      |> update(:loading, &toggle_loading/1)
      |> put_flash(:info, "Finished generating")

    {:noreply, socket}
  end

There is still a couple of more things we need to consider. We need to add error handling as well as message handler for Task.

Let's assume that the API call to TogetherAI will not succeed. In that case, we will need to add another handle_stream() function that will receive the response with error code. For the sake of simplicity, I put the handle_stream that receives the success 200 first, so it will be called first, if does not pattern match (e.g. different status code), it will go to this handle_stream where we assume it will be response with error and error message.

  defp handle_stream({:data, data}, {req = %Req.Request{}, resp = %Req.Response{status: _}}, pid) do
    error_msg = data |> Jason.decode!() |> Map.get("error") |> Map.get("message")
    send(pid, {__MODULE__, :error, error_msg})
    {:cont, {req, resp}}
  end

Remeber that we are calling spawning new process with Task.async()? We will need to handle other messages that the Task sends to its parent. Please refer to officials docs on Task.async.

 def handle_info({PhoenixPlayground.TogetherAi, :error, error_msg}, socket) do
    socket = socket |> put_flash(:error, error_msg)
    {:noreply, socket}
  end

  def handle_info(_msg, socket) do
    # message that come here unhandled are:
    # 1. {:DOWN, _ref, :process, _pid, :normal}
    # 2. {_ref, {:ok, response = %Req.Response{}}}
    # TODO: other Task.async responses

    {:noreply, socket}
  end

Now, run iex -S mix phx.server go to /home and enjoy!

Here is the repo for the project. https://github.com/azyzz228/elixir-llm-tutorial

Discussion:

more error handling should be added?
Handle the last chunk with Token usage info and finish_reason better

Free & simple CI/CD for Elixir Phoenix app

aziz abdullaev — Sat, 04 May 2024 19:42:38 +0000

In this article, I will show the implementation of simple cost- and dependency-free Continuous Integration / Continuous Development (CI/CD) flow to auto-deploy Phoenix app without downtime.

This article is based on Tom Delalande’s YT video with very little of my contribution to make it work with Phoenix application.

After good amount of hard work, you finally decide to deploy your app and realize that deploying your app is not an easy task. But, thanks to Docker, the deployment can be simplified a lot.

In this article, I will show how to implement simple ci/cd flow where you can push your code to your repo and have newest version of your Phoenix app re-deployed with 0 downtime and auto-configured SSL. The VM will check the Github repo every minute to see if anything has changed. If yes, new docker image will be built and deployed.

Virtual Machine & DB

Pick any cloud provider you like and spin up a database and virtual server with ubuntu. I am using DigitalOcean (Love it, hate aws and gcp) and PostgreSQL database on it. Make sure you allow connection from your virtual server to your database.

ssh into your virtual machine and set up connection with your Github account so that your VM can clone and pull your private repos. Basically, you will have to generate ssh key on virtual machine, then add the key to your Github account through Setting > Developer settings. Here is the link: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent

Install Docker using official guide here https://docs.docker.com/engine/install/

Go to your domain provider, and set type A @ (or any subdomain) record to forward to the public IP of your VM.

Create deployment folder that will contain the log file:

cd ~ (in DigitalOcean Ubuntu 20.04 VM, this results to /root)

mkdir deployment (=> /root/deployment)

VM part is mostly set and ready. We will come back to VM later with very small task.

Preparing Phoenix project

Generate release with `—docker` flag

Inside of your Phoenix project, generate release with Dockerfile by running the following command:

mix phx.gen.release --docker

If you are using NPM packages, then add these lines to install Node and all your dependencies after COPY assets assets.

Lines that handle assets:

COPY assets assets

ENV NVM_DIR=/root/.nvm
ENV NODE_VERSION 20.9.0

RUN curl https://raw.githubusercontent.com/creationix/nvm/v0.39.5/install.sh | bash \
  && . $NVM_DIR/nvm.sh \
  && nvm install $NODE_VERSION \
  && nvm alias default $NODE_VERSION \
  && nvm use default
ENV PATH="/root/.nvm/versions/node/v${NODE_VERSION}/bin/:${PATH}"

RUN mix cmd npm install --prefix assets
RUN mix assets.deploy

Add Caddyfile

We are going to use Caddy as a reverse proxy that can also generate SSL certificates for us. On our VM, Caddy will listen to incoming traffic from the user on ports 80 and 443 (HTTP and HTTPS), then forward that request to port 4000 where we will have our Phoenix running. Phoenix response will then be forwarded thru Caddy to the user. Meanwhile, Caddy will make sure all connection is secure and uses HTTPS.

Create /caddy folder, and Caddyfile inside:

#/caddy/Caddyfile 
your_cool_domain.com {
    reverse_proxy server:4000 
}

Why server:4000? Phoenix app will have the name server as docker service (see below).

Configure `docker-compose` file

services:
    server:
        build:
            context: .
            dockerfile: Dockerfile
        env_file:
            - ".env"
    caddy:
        image: "caddy:2.7-alpine"

        restart: unless-stopped
        ports:
            - "80:80"
            - "443:443"
            - "443:443/udp"

        volumes:
            - $PWD/caddy/Caddyfile:/etc/caddy/Caddyfile

add magical ci/cd scripts

Create a /scripts folder and add the following scripts:

deployer.sh

This script makes sure no two deployments are running at the same time, takes cares of logging deployment info, and runs a conditional-deploy scripts.

(Note: /root/deployment folder must already exist. Feel free to change it to whatever you like)

#!/usr/bin/env bash

export DOCKER_CONTEXT=default

LOCK_FILE="$(pwd)/your-app.lock"
cd /root/yetkazuv
flock -n $LOCK_FILE ./scripts/conditional-deploy.sh >>/root/deployment/deploy-app.log 2>&1

conditional-deploy.sh

This script checks if remote repo is different from local and runs server-deploy if necessary

#!/usr/bin/env bash

echo "$(date --utc +%FT%TZ): Fetching remote repository"
git fetch

UPSTREAM=${1:-'@{u}'}
LOCAL=$(git rev-parse @)
REMOTE=$(git rev-parse "$UPSTREAM")
BASE=$(git merge-base @ "$UPSTREAM")

if [ $LOCAL = $REMOTE ]; then
    echo "$(date --utc +%FT%TZ): No changes detected in git"
elif [ $LOCAL = $BASE ]; then
    BUILD_VERSION=$(git rev-parse HEAD)
    echo "$(date --utc +%FT%TZ): Changes detected, deploying new version: $BUILD_VERSION"
    ./scripts/server-deploy.sh
elif [ $REMOTE = $BASE ]; then
    echo "$(date --utc +%FT%TZ): Local changes detected, stashing"
    git stash
    ./scripts/server-deploy.sh
else
    echo "$(date --utc +%FT%TZ): Git is diverged, this is unexpected."
fi

server-deploy.sh

This scripts build a new version of our app inside docker. After new app is built, it then stops old one and substitutes it with new one

#!/usr/bin/env bash

git pull

BUILD_VERSION=$(git rev-parse HEAD)

echo "$(date --utc +%FT%TZ): Releasing new server version. $BUILD_VERSION"
echo "$(date --utc +%FT%TZ): Running build..."
docker compose rm -f
docker compose build

OLD_CONTAINER=$(docker ps -aqf "name=server")
echo "$(date --utc +%FT%TZ): Scaling server up..."
BUILD_VERSION=$BUILD_VERSION docker compose up -d --no-deps --scale server=2 --no-recreate server

sleep 30

echo "$(date --utc +%FT%TZ): Scaling old server down..."
docker container rm -f $OLD_CONTAINER

docker compose up -d --no-deps --scale server=1 --no-recreate server

echo "$(date --utc +%FT%TZ): Reloading caddy..."
CADDY_CONTAINER=$(docker ps -aqf "name=caddy")
docker exec $CADDY_CONTAINER caddy reload -c /etc/caddy/Caddyfile

Commit and push to remote Github repo all new files

Simplest way is:

git add .
git commit -m "add ci/cd after reading this article"

VM: final touches**

ssh into your VM

git clone (SSH method) your project by running

cd into your project folder

create .env file and add all ENV variables you are using while having at least these:

DATABASE_URL=
PHX_SERVER=true
PHX_HOST=your_domain.com
SECRET_KEY_BASE=

run docker compose up to have your app up and running

Now, let’s set a cron job that will run our deployer.sh every minute.

in your VM, run

crontab -e

Then choose the editor you are most comfortable with (i go with VIM), add this line:

 * * * * * /root/repo_name/scripts/deployer.sh 

Voila, now you can just push your code to Github and your server will pull the updates and re-deploy latest changes.

—

Discussion:

I strongly recommend the YouTube video by Tom
You can also deploy your DB, Plausible Analytics, and everything you would like in the same VM as other docker containers
you probably do not want to store all your ENVs in .env file permanently
log file can grow infinitely, so it’s a good idea to create cron job with another script that will delete old log files every X amount of time (like couple of weeks, months)

Network optimization (4x WS message size reduction) for sending lots of data from server to client (LiveView)

aziz abdullaev — Fri, 02 Feb 2024 18:43:05 +0000

In other article, I discussed about using Task.async_stream to concurrently fetch data in chunks and send to the client to render on a map.

When I implemented the aforementioned optimizations in my localhost, results were phenomenal. 12,000+ points would be fetched from my locally running DB and rendered within a second. However, when deployed, it was not nearly as fast. In fact, it was really slow. It was so bad that it scored 0 for performance in Lighthouse report.

I have analyzed every moving part of application to track down the bottleneck. My DB and VM were both deployed in the same country which resulted in little latency and fast query time. JavaScript (JS) execution time was not considered as I opened the page on the same machine (my laptop). When analyzing the timeline of JS execution along with the rendering, I found that it would take up to 1 second for the new chunk of data to
arrive for JS to execute.

The network was the bottleneck. And the slow network is known to be an Achilles heel of LiveView’s architecture.

My configs are:

Elixir 1.15.7
OTP 26
LiveView 0.20.1
Phoenix 1.7.10
Phoenix LiveView 0.18.18
Ecto 3.11
All running on my MacBook Air M2 8GB Ram

Network optimizations were needed to enhance the performance. Here is what I came up with:

Using single push_event as given in docs (duh)

I looked through the WebSocket (WS) messages and found that each message with 2000 projects would weigh 700kb. Looking at the message content, I found my first obvious mistake.

In Part 1, I sent the data using Enum.reduce:

  def handle_info({:data_received, data}, socket) do
    socket =
      Enum.reduce(data, socket, fn %Project{} = project, socket ->
        socket
        |> push_event("add_project", form_project_map(project))
      end)

    {:noreply, socket}
  end

This resulted in WS messages with repetitive event name with this pattern:

“add_project: [PROJECT], add_project: [PROJECT], add_project: [PROJECT]...”

So, the function was changed to this:

  def handle_info({:data_received, data}, socket) do
    socket =
      socket
      |> push_event("add_markers_for_project", %{data: compress(data)})


    {:noreply, socket}
  end

Message size was dropped from 750kb to 650kb.

Minimizing data being transferred

I dropped unused fields for the data I query from database. It was relatively easy, by just adding select: to the query.

      from i in Investment,
        where: not is_nil(i.latitude) and not is_nil(i.longitude),
        limit: ^count,
        order_by: i.id,  
    select: %{
          project_name: i.project_name,
          source_name: i.source_name,
          type: i.type,
          latitude: i.latitude,
          longitude: i.longitude
        }

Next, I shortened the length keys:

        select: %{
          n: i.project_name,
          s: i.source_name,
          t: i.type,
          lat: i.latitude,
          lon: i.longitude
        }

but wait, why do we need to have shortened keys when we can get rid of the keys altogether?

        select: [
          i.project_name,
          i.source_name,
          i.type,
          i.latitude,
          i.longitude
        ]

WS message size was reduced from 650kb to around 400kb:

All the data and messages transferred between LiveView and Client are all JSON encoded anyways, so basically our data is encoded to a long string that is then decoded on the client-side. What if we could compress the string before sending it?

Good news, I did not even need any dependencies, as :zlib comes with Erlang and is easy to use. After I JSON encoded my data, I compressed it using :zlib.compress. But the catch here is that :zlib.compress produces a binary, so I had to encode it to a string using Base64 encoding.

So, the compression was achieved with 3 lines of code without any dependencies:

  def compress(data) do
    data
    |> Jason.encode!()
    |> :zlib.compress()
    |> Base.encode64()
  end

This technique allowed to reduce the message size to only around 100kb (4x less!)

Now, the hardest part - handling the data on the client-side with JS.

There are couple of dependencies that need to be installed:

pako - for zlib-decompressing
js-base64 - for decoding Base64 encoded string.

Here is the function that decompresses zlib compressed + base64 encoded string:

const decompress_to_json = (base64string) => {
    let base64_decoded = Base64.atob(base64string)
    let charCodes = [];
    for (let i = 0; i < base64_decoded.length; i++) {
        charCodes.push(base64_decoded.charCodeAt(i));
    }
    let inflatedData = pako.inflate(charCodes, { to: 'string' });

    const data = JSON.parse(inflatedData);
    return data;
}

and my event handler for rendering the points on a map look like this:

        this.handleEvent("add_markers_for_project", ({ data }) => {

            const projects = decompress_to_json(data)
            for (let i = 0; i < projects.length; i++) {
                const project = objectifyProject(projects[i]);
                if (!project.latitude || !project.longitude) {
                    return;
                }
                addMarker(project, map, markersLayerGroup);
            }
        });

where objectify() simply creates object with keys from the array, i.e. from [_name_, _type_, _lat_, _lon_] to {name: _name_, type: _type_, lat: _lat_, lon: _lon_}

Discussion

Let’s talk about the tradeoffs. JS bundle size has increased because of the additional dependencies. And yes, we are doing a lot more operations on both server side elixir and client side, but is it worth performing more computations in order to optimize for network? Absolutely.

compress() function takes around 2.2ms to compress + encode 4000 entries.
decompress_to_json() takes around 20ms to decode + decompress 4000 entries. Execution time is negligible compared to the benefits it produces.

As a result, I was able to get my deployed map render in speed comparable to the one on the local host. It is almost instantaneous. Lighthouse score for performance is 93+.

Parsing data from CSV into PostgreSQL in Phoenix Framework project with Elixir

aziz abdullaev — Sun, 24 Dec 2023 16:37:42 +0000

In this article, I will discuss how to populate database (PostgreSQL in my case) from CSV file.

My configs are:

Elixir 1.15.7
OTP 26
LiveView 0.20.1
Phoenix 1.7.10
Phoenix LiveView 0.18.18
Ecto 3.11
Dockerized PostgreSQL
All running on my MacBook Air M2 8GB Ram

First, let’s install CSV parser. I am using NimbleCSV — created and maintained by the creator of Elixir, Jose Valim.

In mix.exs deps, we need to add:

def deps do
  [
    {:nimble_csv, "~> 1.1"}
  ]
end

Let’s create a context and schema which we are going to populate:

mix phx.gen.live Projects Investment investments name funder type status

Now, let’s go to the actual implementation which is really easy and straightforward.

In our seeds.exs file, we add:

alias NimbleCSV.RFC4180, as: CSV

"/Users/user/Documents/investments.csv"
|> File.stream!()
|> CSV.parse_stream()
|> Stream.each(fn [
                   project_name,
                   funder,
                   type,
                   status
                 ] ->
  %{
    project_name: project_name,
    funder: funder,
    type: type,
    status: status,
  }
  |> MyApp.Projects.create_investment()
end)
|> Stream.run()

First, we are importing NimbleCSV.RFC4180 (the most common type of CSV encoding — works well with Google Sheet exported CSVs) as CSV (make sure to double check your CSV file’s encoding). Then, we are creating a file stream, parsing it through the CSV parser. Then on each row, we are going to create an entry in our database.

Function that we pass to Stream.each takes a row as an array in which each element is a separate column of data. Then, we are going to create a map and pass it to our Context function (that was auto-generated for us).

Then, all we have to do is to run this script in terminal:

mix run priv/repo/seeds.exs

For the sake of intellectual debate, there is probably a lot of ways this can be improved.

We can add some asynchronous parallel spices into the script. One can create a few parallel processes that will asynchronously iterate the rows from certain indices. 1st process will iterate from 0th to 100th row, second will iterate from 100th to 200th, so on. Each process will create an array of data which we can then insert to DB with Ecto.Repo.insert_all.

File upload to AWS S3 or S3 compatible bucket from Phoenix LiveView using Elixir

aziz abdullaev — Sun, 10 Dec 2023 07:53:23 +0000

In this article, I will discuss how to implement LiveView file uploader from client directly to the DigitalOcean's Spaces (S3) bucket using Elixir and LiveView in your Phoenix application (but you can also use this tutorial to upload to AWS S3 or any S3-compatible bucket).

This article is not endorsed or paid by DigitalOcean. It is just the lack of resources on integration with other object storage buckets (than AWS S3). But also, DigitalOcean’s Spaces (disclaimer: in my opinion) a lot easier to use and has more straightforward pricing.

My configs are:

Elixir 1.15.7
OTP 26
LiveView 0.20.1
Phoenix 1.7.10
Phoenix LiveView 0.18.18
Ecto 3.11
Dockerized PostgreSQL
All running on my MacBook Air M2 8GB Ram

If you want to upload files from the server, please check out ex_aws, ex_aws_s3, and AWS S3 in Elixir with ExAws. Uploading from the server is very straightforward compared to uploading from the client mainly because ex_aws already includes all necessary request settings. If you want the user to first upload the file to the server (maybe you want to change the file a bit before storing it or generating thumbnail of the picture), then consider also using Waffle that comes with seamless integration with ex_aws.

However, if we want user to upload files and then store it in the S3 bucket without any changes to the file, it would be more efficient to enable users upload the file directly
to the S3 bucket (user -> s3) instead of uploading it to the server so that the server uploads it to the S3 (user -> our server -> s3). And this is what this article focuses on.

This article is built entirely on the following resources: 

External uploads guide from LiveView docs
Phoenix LiveView Uploads Deep Dive — 2020 article by Chris McCord (creator of Phoenix framework)
Liveview File Uploads to S3 — YouTube video by LiveView Mastery
Slightly modified S3 uploader (originally coming from Chris McCord) by LiveView Mastery. gist

Uploading file directly from the client (with LiveView) is challenging because we need to do multipart HTTP request and add security verification parameters that are stored in ENV of our Phoenix application. Luckily, Chris McCord has got us covered.

Let’s do initial setup.

Create an account in DigitalOcean and create Spaces Object Storage bucket. You will need to choose the location of the server, name of the bucket.
Inside of Spaces bucket page, go to “Settings” and add CORS configuration. For the sake of tutorial, you can put * to the origin to allow everyone an access then allow all methods, but you must immediately change the CORS configuration to the one that suits your needs best, and NEVER have * in the origin field
Then generate access key and store it somewhere safe. We will need access key so that our Phoenix application can connect and access the bucket.
Add the Spaces (S3) bucket credentials to the config.exs:

config :your_app, 
    access_key_id: “fake”, 
    secret_access_key: "fake", 
    bucket: "fake", 
    region: "your_s3_region"

Note: NEVER store your credentials in plain text or variable. You must handle the secrets through ENV variables, but that is not the focus of this article. Please refer to official docs on how to handle secrets in Elixir.

Now, let’s talk about overall architecture of things. We are going to upload the file to the Spaces (S3) bucket and will receive an URL to the file, then we are going to store that URL in database in the string field of the schema. For example, let’s say I am building an online menu for the restaurant. I will have menu_items schema and I want to upload pictures of prepared menu item there that the consumer will get. I will have to add field :image, :string (that will store the image URL from my Spaces bucket) to the schema menu_items.

When it comes implementing the file upload to the Spaces (S3) bucket in LiveView, there are several steps we have to do:

Get secrets from our Phoenix application and pass it to the client
Implement the file upload to the Spaces (S3) bucket from the client by adding some JavaScript
Store the file URL in database

Let’s jump into the code.

First, inside of liveview, we are going to enable the user to upload the file in the form and add a function that will fetch the secrets from Phoenix application and prepare required headers for the request client-side JavaScript will do.

In the LiveView, we are going to use allow_upload to allow the user upload. Then, we will add a function to allow_upload that will pass our secrets from Phoenix application to the client. We are going to call that function presign_upload.

 def mount(_, _, socket) do 
   socket =
      socket
      |> allow_upload(:image,
        accept: ~w(.jpg .jpeg .png),
        max_entries: 1,
        external: &presign_upload/2
      )

    {:ok, socket}
  end


  defp presign_upload(entry, %{assigns: %{uploads: uploads}} = socket) do
    meta = S3Uploader.meta(entry, uploads)
    {:ok, meta, socket}
  end

presign_upload will basically fetch secrets from out Phoenix application and pass them to the client. It will also prepare the headers that the S3 requires.

There is a dependency-free S3Uploader written by Chris McCord. If you go to the official LiveView docs on uploading directly to S3, you will notice that presign_upload() there has some logic which is not ideal to have on the LiveView. So, we are going to use LiveView mastery’s S3Uploader which is functionally the same as Chris McCord’s but handles the logic so that we in our LiveView our presign_upload looks like this:

  defp presign_upload(entry, %{assigns: %{uploads: uploads}} = socket) do
    meta = S3Uploader.meta(entry, uploads)
    {:ok, meta, socket}
  end

Instead of this:

defp presign_upload(entry, socket) do
  uploads = socket.assigns.uploads
  bucket = "phx-upload-example"
  key = "public/#{entry.client_name}"

  config = %{
    region: "us-east-1",
    access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
    secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
  }

  {:ok, fields} =
    SimpleS3Upload.sign_form_upload(config, bucket,
      key: key,
      content_type: entry.client_type,
      max_file_size: uploads[entry.upload_config].max_file_size,
      expires_in: :timer.hours(1)
    )

  meta = %{uploader: "S3", key: key, url: "http://#{bucket}.s3-#{config.region}.amazonaws.com", fields: fields}
  {:ok, meta, socket}
end

NOTE: To upload to the DigitalOcean’s Spaces bucket, you need to change the URLs on lines: 95, 113, 127 from "https://#{bucket()}.s3.#{region()}.amazonaws.com” to https://#{bucket()}.#{region()}.digitaloceanspaces.com/.

We add file uploading functionality for the user using <.live_file_input upload={@uploads.image} />

So, our LiveView HEEx will minimally look like this:

def render(assigns) do
    ~H"""
    <div>
      <form
        phx-submit="save"
        phx-change="validate"
      >

       <label for="name">name</label>
          <input
            type="text"
            name=“menu_item[name]"
            phx-update="ignore"
            id="name"
          />

      <label for={@uploads.image.ref}>image</label>
          <.live_file_input upload={@uploads.image} />

        <button
          type="submit"
        >
          Submit
        </button>

    </form>
       </div>
    “””
end

Important: You must bind phx-submit and phx-change on the form.

Otherwise the file upload will not work. Refer to docs.

When user will hit “Submit” button, client must get the meta data (with our secrets) and initiate the upload directly to the Spaces (S3) bucket. According to the official external upload guide from docs, we can implement that by adding this code to the app.js:

let Uploaders = {}

Uploaders.S3 = function(entries, onViewError){
  entries.forEach(entry => {
    let formData = new FormData()
    let {url, fields} = entry.meta
    Object.entries(fields).forEach(([key, val]) => formData.append(key, val))
    formData.append("file", entry.file)
    let xhr = new XMLHttpRequest()
    onViewError(() => xhr.abort())
    xhr.onload = () => xhr.status === 204 ? entry.progress(100) : entry.error()
    xhr.onerror = () => entry.error()
    xhr.upload.addEventListener("progress", (event) => {
      if(event.lengthComputable){
        let percent = Math.round((event.loaded / event.total) * 100)
        if(percent < 100){ entry.progress(percent) }
      }
    })

    xhr.open("POST", url, true)
    xhr.send(formData)
  })
}

let liveSocket = new LiveSocket("/live", Socket, {
  uploaders: Uploaders,
  params: {_csrf_token: csrfToken}
})

Now, we have a LiveView with the form where you can upload a file, and on submit button, client will automatically upload the file to the Spaces (S3). Now let’s handle the part how to actually store the file url.

Let’s handle validations and submission of the form. In the LiveView we need to add:

 

 def handle_event("validate”, _, socket) do
    {:noreply, socket}
  end

  def handle_event("save", %{“menu_item” => menu_item_params}, socket) do

       uploaded_files = consume_uploaded_entries(socket, :image, fn _, entry ->
        {:ok, S3Uploader.entry_url(entry)}
      end)

    menu_item_params = case Enum.empty?(uploaded_files) do
      true -> menu_item_params
      false -> Map.put(menu_item_params, "image", uploaded_files |> List.first())
    end

    socket =
      case Context.create_menu_item(menu_item_params) do
        {:ok, _} ->

          socket
          |> put_flash(:info, "menu_item created")
          |> push_patch(to: “/success”)

        _ ->
          socket
          |> put_flash(:error, "Could not create a new menu_item")
      end

  end

When the user hits the submit button, image is uploaded by the client-side JS code with metadata handed over by S3Uploader via presign_upload(). The metadata from S3Uploader also includes a name of the file (uuid) to use for saving it to the bucket for the file uploaded. On our LiveView part of handling the submission, we are using consume_uploaded_entries() and generating the URL the file that would have if uploaded to the Spaces S3 bucket. Because the file is the same and its uuid is the same, S3Uploader.entry_url(entry) will result in the URL (with the name of the file) that is identical to the one passed to client-side JS.

Next, we are going to check if any file was uploaded, then update the menu_item_params map. uploaded_files will be a list of URLs if we had many files uploaded, but as we permit only one file upload in allow_upload, our list will be either empty or have only one item.

Then, we are going to pass the parameters to the Context to save it in our database and handle the result of operation.

If the user has uploaded the file, menu_item_params will look something like this:

%{
    “name” => “delicious uzbek plov”,
    “image” => “https://tutorial-bucket.ams3.digitaloceanspaces.com/2d1622d5-328f-4b8b-b41d-44370bafe222.jpg”
}

There are also couple of small improvements we can add.

First, we can enable auto-upload. When the user chooses the file, our client-side JS will start uploading to the Spaces (S3) bucket right away. We can implement that with one line of code (thanks Chris). We will need to pass auto_upload: true to allow_upload function on mount:

 def mount(_, _, socket) do 
   socket =
      socket
      |> allow_upload(:image,
        accept: ~w(.jpg .jpeg .png),
        max_entries: 1,
    auto_upload: true,
        external: &presign_upload/2
      )

    {:ok, socket}
  end

Next, we can show the preview as well as progress of file upload to the user, by adding these lines to the view:

          <%= for entry <- @uploads.image.entries do %>
            <.live_img_preview entry={entry} width="75" />
            <div class="py-5">
                      <%= entry.progress %>%
            </div>
          <% end %>

What to do with the progress is up to your imagination.

If you have any suggestions or critique, please let me know.

Performance optimization when adding 12,000+ markers to the map that renders fast with Elixir, LiveView, and Leaflet.js

aziz abdullaev — Sun, 03 Dec 2023 20:54:10 +0000

The task was to have a map that renders the location of 12,000+ sites by adding a circle marker and popup window when clicked. And it must be very fast, ideally so fast that it seems it’s instantaneous. In this article, I will share my implementation with Elixir, LiveView, and Leaflet.js along with optimization techniques I came up with.

If we were to zoom out, There are two main tasks that we need to do:

Fetch data from database and pass it to the frontend
Render the data on a map

First part is handled by the server while rendering the map happens on the client side. Because there is a lot of data we need to fetch and render, optimizations are needed on both server and client sides.

To implement the map using Elixir, LiveView, and Leaflet.js, we need to do the following:

Elixir fetches data from the database and passes it to the LiveView
LiveView receives the data and passes it to the client (JavaScript (JS) Hook with push_event)
JS handles the received data by creating a map with Leaflet.js and adding marker

My configs are:

Elixir 1.15.7
OTP 26
LiveView 0.20.1
Phoenix 1.7.10
Phoenix LiveView 0.18.18
Ecto 3.11
Dockerized PostgreSQL
All running on my MacBook Air M2 8GB Ram

Let’s do the initial setup.

When the user opens the page, LiveView process is created, web socket connection established, and we can query the data we want in the LiveView. Our LiveView will contain a map created with Leaflet.js. Thanks to the great JS interoperability (interop) that comes with LiveView, we easily can do that.
 A little side note if English is not your first language (which is my case). JS interoperability means JS integration, basically how we can use JS with LiveView. Okay, let’s continue.

Here is our initial LiveView.

defmodule CoolMapWeb.HomeLive do

  def mount(_, _, socket) do
    {:ok, socket}
  end

 def render(assigns) do
    ~H"""
    <div class="container mx-auto w-[80%]">
      <div class="w-full h-[500px] p-20" id="map" phx-hook="Map"></div>
    </div>
    """
  end

Nothing interesting going on here except for having a <div> with phx-hook=“Map” — that is how JS interop is implemented in LiveView (see the official docs). Now, let’s create a hook that will allow us to use Leaflet.js to create the map.

To install Leaflet.js through NPM, go to /assets/vendor direction and run npm install leaflet

Inside of our app.js file:

import L from "../vendor/node_modules/leaflet";

const Map = {
    mounted() {
        const map = L.map("map").setView([41, 69], 2);
        L.tileLayer(
            "https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer/tile/{z}/{y}/{x}",
            {
                attribution:
                    "Tiles &copy; Esri &mdash; Esri, DeLorme, NAVTEQ",
                maxZoom: 16,
            }
        ).addTo(map);

    },
};

const Hooks = {
    Map,
};

// add Hooks here
let liveSocket = new LiveSocket("/live", Socket, { params: { _csrf_token: csrfToken }, hooks: Hooks })

Here is the result:

Let’s do the simplest implementation of the task. When the LiveView loads, it will request a data from a Context, then push it to the Leaflet.js. To pass the data from LiveView to Leaflet.js Hook, we need to use push_event function on the LiveView and handle_event on the Hook.

Here is the function from Context that will query the projects with coordinates for us:

  def list_projects do
    query = from p in Project, where: not is_nil(p.latitude), order_by: p.id and not is_nil(p.longitude)
    Repo.all(query)
  end

We will request the data on LiveView’s mount function and pass it to the JS side:

  def mount(_, _, socket) do
    projects = Context.list_projects() # array [%Project{}, %Project{}]
    socket = 
    socket 
    |> push_event("add_project”, %{projects: projects)

    {:ok, socket}
  end

Note: the push_event in this case will raise an error because the payload must not contain the structs. We will fix this issue later.

To receive the data on JS side, we can either add window listener or add event handler to the hook. I went with option #2.

const Map = {
    mounted() {
        const map = L.map("map").setView([41, 69], 2);
        L.tileLayer(
            "https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer/tile/{z}/{y}/{x}",
            {
                attribution:
                    "Tiles &copy; Esri &mdash; Esri, DeLorme, NAVTEQ",
                maxZoom: 16,
            }
        ).addTo(map);

        // event handler
        this.handleEvent("add_project", (project) => {
        // add markers
        });

    },
};

Now, let’s discuss the interesting part: optimizations.

Elixir is built on top of Erlang and is fantastic when it comes to concurrent programming and many processes. When I ran query on my laptop (both dockerized postgreSQL and Phoenix application), I get the following results:

Running the simple query function list_projects() takes around 187 ms when measured with :timer.tc function. My DB has around 54,000 entries but only around 12,300 have both latitude and longitude. So, list_projects() results in around 12,300 items in an array of results.

When I used list_projects() on my laptop, passed it to the LiveView, and added markers for the map, my laptop just froze and took around 10+ seconds to recover. Although it takes only 187ms when I run the function on iex terminal, there are several reasons why it takes so long to actually produce a page and map with markers. One of the main reasons is the fact that JS code runs on the client side, although LiveView is a process living on the server. So the solution would be to pass the data to the JS side in smaller chunks so that browser does not crash and/or freeze.

On top of that, using concurrent programming in Elixir, we can asynchronously run multiple functions that will fetch the data. If we were to have chunks of 6,000 entries, then we would spawn two processes: one queries data for entries 1st to 5,999th, and the second from 6,000th to 12,000th. We can make chunk even smaller, I decided to have chunks with the size of 2,000 entires. When we run multiple asynchronous functions, we do not know which function will finish running first, so we need a mechanism of passing data to the LiveView then to the JS whenever asynchronous function returns the data from database.

How can we do that? Use the most important feature of LiveView: LiveView is a process, so it has its own PID. So, we will pass a PID of the LiveView to querying async functions, and functions will Kernel.send() the data to the LiveView.

Here is my implementation of the Context function that will receive the PID of the LiveView, asynchronously do DB queries, and send it to the LiveView.

  def stream_projects(pid, offset \\ 0, batch_size \\ 2000, total_records \\ 14000) do
    batches = div(total_records, batch_size)

    async_stream =
      Task.async_stream(0..(batches - 1), fn batch ->
        current_offset = offset + batch * batch_size

        query =
          from p in Project,
            where: not is_nil(i.latitude) and not is_nil(i.longitude),
            limit: ^batch_size,
            offset: ^current_offset

        case Repo.all(query) do
          [] ->
            :done

          result ->
            send(pid, {:data_received, result})
        end
      end)

    async_stream
    |> Stream.run()

    :done
  end

In stream_projects function, we create a stream of async functions that fetch the data 0th to 2000th, then 2000th to 4000th, etc, then send the data to the PID of the LiveView. (Please refer to official documentation on Streams and async_streams)

Below is the screenshot of performance measurement of stream_projects(). Note: in my project, function name is different, but the content of the function is identical to stream_projects(). It only takes 66ms to run the function.

In LiveView, we need to call the stream_projects on mount, then add handle_info function to receive the data and pass it to the JS Hook.

  def mount(_, _, socket) do
    Projects.stream_all_projects(self())

    {:ok, socket}
  end


 # ADD THIS FUNCTION
  def handle_info({:data_received, data}, socket) do
    socket =
      Enum.reduce(data, socket, fn %Project{} = project, socket ->
        socket
        |> push_event("add_project", form_project_map(project))
      end)

    {:noreply, socket}
  end

  defp form_project_map(%Project{} = project) do
    %{
      latitude: project.latitude,
      longitude: project.longitude,
      name: project.name,
    }
  end

In handle_info, we are creating an accumulated socket with projects passed to the JS hook individually. We are also creating a map from the %Project{} structure.

Now, let’s add a full event handler on JS hook that will be able to create a marker on project’s location. By default, markers are rendered as SVGs and added to the map and HTML, when we have lots of new markers we add a lot of new nodes to the DOM (which is very costly performance wise). The key optimization with Leaflet.js is to use Canvas renderer for markers so that nodes are not created and added to the DOM (FYI https://stackoverflow.com/questions/37043791/plotting-140k-points-in-leafletjs).

const Map = {
    mounted() {
        const map = L.map("map").setView([41, 69], 2);
        L.tileLayer(
            "https://server.arcgisonline.com/ArcGIS/rest/services/Canvas/World_Light_Gray_Base/MapServer/tile/{z}/{y}/{x}",
            {
                attribution:
                    "Tiles &copy; Esri &mdash; Esri, DeLorme, NAVTEQ",
                maxZoom: 16,
            }
        ).addTo(map);


        let myRenderer = L.canvas({ padding: 0.5 });
        this.handleEvent(“add_project", (project) => {

            let marker = L.circleMarker(
                [project.latitude, project.longitude],
                {
                    renderer: myRenderer,
                    radius: 1,
                    width: 0,
                    color: “#ef4444”,
                    fillColor: “#ef4444”,
                    fillOpacity: 0.8,
                    fill: true,
                }
            )
                .addTo(map);
                .bindPopup(`<p>${project.name}</p>`);

    },
};

Here is the result of the map that renders around 12,000 points from the cold start:

Discussion and notes:

Error handling on stream_projects() does not exist
The total number of projects is known and does not change frequently — this allows us to manually set the total_records to 14,000 in stream_projects.
total_records number is larger than exact number of entries, so we may be doing some extra queries.
Possible improvement on JS hook is to add async handling and addition of markers to the map
The actual loading speed of the map when deployed will depend on bandwidth, DB query latency, and client’s hardware.

This is my first article on programming, take it with the grain of salt. Please reach out if you have any corrections or suggestions for my article.

Part 2 is available here https://dev.to/azyzz/network-optimization-for-sending-lot-of-data-from-liveview-to-client-using-pushevent-2nl

DEV Community: aziz abdullaev

WYSIWYG editor in LiveView with embedded images that are uploaded to S3 bucket

AI powered app (with open-source LLMs like Llama) with Elixir, Phoenix, LiveView, and TogetherAI

Setup

Implementation

Free & simple CI/CD for Elixir Phoenix app

Virtual Machine & DB

Preparing Phoenix project

Generate release with —docker flag

Add Caddyfile

Configure docker-compose file

add magical ci/cd scripts

Commit and push to remote Github repo all new files

VM: final touches**

Network optimization (4x WS message size reduction) for sending lots of data from server to client (LiveView)

Parsing data from CSV into PostgreSQL in Phoenix Framework project with Elixir

File upload to AWS S3 or S3 compatible bucket from Phoenix LiveView using Elixir

Performance optimization when adding 12,000+ markers to the map that renders fast with Elixir, LiveView, and Leaflet.js

Generate release with `—docker` flag

Configure `docker-compose` file