DEV Community: Micael Nussbaumer

Criar um Enum Ecto.Type

Micael Nussbaumer — Sun, 30 Jan 2022 14:26:00 +0000

Hoje vou mostrar um enumerador implementado como um Ecto.Type, que uso em quase todos os projectos de Elixir em que contribuo. É um tipo semelhante ao Ecto.Enum que permite utilizar átomos na lógica interna da aplicação, validar/transformar valores equivalentes na sua forma de caracteres, e ao mesmo tempo limitar os possíveis valores que determinado campo/coluna pode assumir. Como não há tradução actual para o tipo de "string", no sentido em que é utilizado em programação, irei utilizar o termo listagrama, no sentido que uma "string" é uma lista de caracteres, e gramma em latim é símbolo/letra/desenho (hexagrama, etc), ou seja, uma lista de "grammas".

Apesar do Ecto incluir um Ecto.Enum desde a versão 3.5, este sofre de alguns problemas:

é declarado em conjunto com o esquema no qual é utilizado, o que não permite que seja partilhado facilmente entre esquemas,
não gera informações de tipo para usar em especificações,
não permite lógica adicional no tratamento de parâmetros adicionais,
não permite utilizar números inteiros para a representação do mesmo ao nível da base de dados (apenas grupos de caracteres)

Mesmo assim é por si só melhor do que usar simplesmente string como tipo de coluna nos casos em que sabemos que a mesma apenas pode assumir uma lista limitada de valores.
Para ilustrar isto, vejamos o seguinte módulo:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, :string)
  end
end

Isto é um esquema totalmente válido, mas tanto quanto sabemos ao lê-lo é que o campo :action é um grupo de caracteres arbitrário. Neste caso, :action apenas pode assumir um número limitado de valores mas o esquema em si não nos transmite essa informação de forma alguma.

Claro que poderíamos criar restrições ao nível da base de dados, e em certos casos independemente de tudo o resto devemos certamente fazê-lo, mas isso continuaria a não providenciar informação alguma a nível do código da nossa aplicação sobre as possíveis restrições, nem ajudaria a transformar ou traduzir valores entre sistemas externos e a nossa representação interna para esses mesmos valores. Para obter isso necessitamos de emergir ao nível dos nossos esquemas quais sãos os valores possíveis para tais colunas.

Nesse caso, o Ecto.Enum ajuda-nos a fazer exactamente isso, por exemplo, utilizando o mesmo esquema anterior, mas agora atribuíndo um Ecto.Enum como tipo ao invés da simples :string

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, Ecto.Enum, values: [:bid, :request, :upload, :pay])
  end
end

Agora é visível que valores este campo pode assumir. Simplesmente por ler a definição do esquema sabemos que são :bid, :request, :upload e :pay.

Este tipo também nos permite então traduzir valores no formato de simples caracteres para o nosso formato interno de átomos, o que é bastante útil uma vez que se tivermos de tratar de validar submissões de dados externas ao nosso sistema, essas submissões serão certamente no formato de caracteres e não em tipos nativos a Elixir/Erlang, como é o caso de átomos. No entanto, após traduzidos, podemos ter a certeza que são não só válidos (há uma equivalência explicíta entre ambas as representações) mas que podemos escrever a nossa lógica interna com recurso apenas a átomos, o que para além de ser mais eficaz é também mais robusto.

Resumindo:

dá-nos a habilidade de transformar e validar parâmetros:


params = %{"user_id" => 1, "action" => "bid"}

Ecto.Changeset.cast(%User.Action{}, params, [:user_id, :action])

permite-nos, como mencionado anteriormente, usar estritamente a versão de átomo na nossa lógica interna, pois sabemos que após ser validado terá de conformar-se a essa representação:


def log_action?(%User.Action{action: action}),
    do: action in [:bid, :pay]

Como átomos em Elixir e Erlang são internamente representados como números inteiros, é bastante mais performante utilizá-los, tanto para serializar ou de-serializar. Grupos de caracteres necessitam de muito mais trabalho para serem comparados e avaliados.

O que o Ecto.Enum adiciona é por si só em grande parte dos casos suficiente. É sempre melhor do que usar definições de tipos que permitem grupos arbitrários quando sabemos que apenas determinados valores são válidos.

Mas se por exemplo, agora quisessemos utilizar este mesmo tipo que definimos no User.Action noutro esquema, teríamos de duplicar esta definição, e não haveria ligação efectiva entre ambos, a não ser que fossemos buscar os valores a algum módulo/função partilhada e, mesmo nesse caso, a ligação não é formal, o tipo não é o mesmo nem é possível representá-lo como tal.

Para além disso, não temos também acesso a funções de validação independentes do esquema. O tipo sendo declarado no esquema em si, é parte do mesmo e as suas validações ocorrem no contexto de um esquema de Ecto. Adicionar claúsulas específicas para tratar de casos especiais é também impossível ou bastante mais complicado do que necessita de ser.

Adicionalmente, caso queiramos guardar tais valores como números inteiros ao nível da base de dados, também não o podemos fazer uma vez que o Ecto.Enum assume apenas listagramas (strings) como a coluna ao qual está associado.

Por último, não gera um typespec para o tipo. Caso usemos dialyzer na nossa aplicação não temos forma de transpor as restrições definidas no esquema para especificações de tipos para documentação (e consistência de documentação das interfaces).

No entanto é, como referido, bastante melhor que utilizar apenas :string como tipo, e por vezes também não necessitamos de mais nenhuma funcionalidade.

Caso necessitemos, vejamos primeiro como poderíamos definir um Ecto.Type independente para este caso em particular :

defmodule User.Action.Type do
  @behaviour Ecto.Type

  def type, do: :string

  @valid_atoms [:bid, :request, :upload, :pay]
  @valid_strings Enum.reduce(@valid_atoms, [], fn t, acc -> [Atom.to_string(t) | acc] end)
  @valid_map Enum.reduce(@valid_atoms, %{}, fn t, acc -> Map.put(acc, Atom.to_string(t), t) end)

  @valid_inverse_map Enum.reduce(@valid_atoms, %{}, fn t, acc -> Map.put(acc, t, Atom.to_string(t)) end)

  @type t() :: unquote(Enum.reduce(Enum.reverse(@valid_atoms), &{:|, [], [&1, &2]}))

  def load(data), do: cast(data)

  def cast(data) when is_atom(data) and data in @valid_atoms, do: {:ok, data}
  def cast(data) when is_binary(data) and data in @valid_strings, do: {:ok, @valid_map[data]}
  def cast(_), do: :error

  def dump(data) when is_atom(data) and data in @valid_atoms, do: {:ok, @valid_inverse_map[data]}
  def dump(data) when is_binary(data) and data in @valid_strings, do: {:ok, data}
  def dump(_), do: :error

  def embed_as(_), do: :dump
  def equal?(data, data), do: true

  def equal?(data_1, data_2) do
    case {cast(data_1), cast(data_2)} do
      {{:ok, same}, {:ok, same}} -> true
      _ -> false
    end
  end
end

Analisando o módulo:

declaramos que o módulo implementa o comportamento (@behaviour) de um Ecto.Type (o que nos garante avisos e erros aquando da compilação no caso de nos esquecer-mos de implementar alguma função)
declaramos uma função type/0 que resolve no tipo interno que o módulo representa (o que será utilizado aquando da serialização para a base de dados). Neste caso resolve em :string (ou seja, a coluna na base de dados será uma coluna de texto - varchar, char, text, etc)
declaramos um typespec, que resolve numa lista contendo todos os átomos válidos para este tipo
utilizando essa lista de átomos válidos (@valid_atoms) geramos uma lista de listagramas que correspondem directamente às respectivas versões dos átomos, assim como dois atributos do módulo para guardar o mapeamento entre os mesmos, átomos->listagramas e listagramas->átomos
declaramos todas as funções requeridas pelo comportamento Ecto.Type - uma vez que temos uma lista compostas por todos os átomos e outra por todos os listagramas válidos, podemos utilizar essas listas como cláusulas de restrição nas funções requeridas e dessa forma facilmente definir como se devem comportar.

Como podemos ver, é simples criar um Ecto.Type que represente um grupo restrito de valores, ajudando-nos a tornar o nosso código mais assertivo e explícito (e por conseguinte mais compreensível também).

Permite-nos também transformar e validar termos independentemente de um qualquer esquema. Podemos por exemplo executar User.Action.Type.cast("bid") para validar o termo "bid", que neste caso seria válido visto ser a representação em listagrama do átomo :bid.

Caso copiar ou replicar este módulo com as devidas alterações sempre que necessitemos de um tipo novo para um outro grupo de valores não for um problema, podemos utilizar esta versão e fazer exactamente isso, utilizando isto como a "planta" para todos os outros.

Esta versão já demonstra algumas vantagens relativamente ao Ecto.Enum, nomeadamente a capacidade de fazer transformações & validações de termos independentemente de esquemas, assim como ser utilizado em vários esquemas, sem requerer duplicação de código ou código adicional para os manter actualizados - futuras alterações aos valores do tipo ocorrem apenas no módulo que o define.

Uma vez que controlamos também o funcionamento interno do mesmo poderíamos facilmente modificá-lo para que serializasse os valores como números inteiros. No entanto, o melhor seria criar uma macro que faça tudo isso por nós, permitindo-nos aceder essa funcionalidade com uma simples linha de código. Tendo isso em mente, o nosso objectivo é criar uma macro que nos permite escrever o módulo User.Action.Type anteriormente exposto, como:

defmodule User.Action.Type do
  use TypedEnum, values: [:bid, :request, :upload, :pay]
end

De forma a utilizá-lo num esquema de Ecto da seguinte forma:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, User.Action.Type)
  end
end

E caso queiramos lidar com átomos ao nível da aplicação mas serializá-los enquanto números inteiros, que possamos definir o mesmo tipo como:

defmodule User.Action.Type do
  use TypedEnum, values: [bid: 0, request: 1, upload: 2, pay: 3]
end

Vejamos então uma lista do que o nosso módulo necessita suportar:

a macro __using__, de forma que os módulos que queiram fazer uso dele, possam utilizar a palavra-chave use, como demonstrado nos exemplos do que queremos
que seja capaz de distinguir entre listas de átomos como valores válidos e listas de palavras-chaves, de forma a que possa intuitivamente implementar ora versões com base de número inteiro, ou de listagramas
permitir a quem o deseje implementar a possibilidade de escrever funções adicionais para cast/dump do comportamento Ecto.Type
introduzir automaticamente um typespec (especificação de tipo) que reflita todos os valores válidos na sua forma de átomo
que emerja erros aquando da compilação quando os valores utilizados na definição do tipo sejam inválidos

Dito isso, o nosso módulo final terá a seguinte forma:


defmodule TypedEnum do
  defmacro __before_compile__(_env) do
    # these are inserted in the before_compile hook to give opportunity to the
    # implementing module to define additional variations
    quote do
      def cast(_), do: :error
      def dump(_), do: :error
      defp get_term(data), do: data
    end
  end

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

  defp validate_int_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &valid_int_enum?/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def validate_string_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &is_atom/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def valid_int_enum?({k, v}),
    do: is_atom(k) and is_integer(v)

  def raise_error({:length, _}),
    do: raise "TypedEnum expects `:values` to be a list or keyword list with at least 1 element"

  def raise_error({:format, _}),
    do: raise "TypedEnum expects the format of `:values` to be a keywordlist with the atom version as the key and an integer as the value (e.g.: [atom_key: 1, another_possible: 2, ...]), or a list of atoms for the string enum version (e.g.: [:atom_key, :another_possible, ...])"

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

  def bind_as_stringed_version(values, mod) do
    quote bind_quoted: [atoms: values, mod: mod] do
      @before_compile TypedEnum

      strings = Enum.map(atoms, fn entry -> Atom.to_string(entry) end)
      mapped = Enum.zip(strings, atoms) |> Enum.into(%{})

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :string

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :validation_mappings, mapped)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings) :: list(t()) | list(String.t())
      def values(type \\ :atoms)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms), do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, String.to_atom(data)}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, Atom.to_string(data)}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: data

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings[data]
    end
  end
end

Para que quem queira utilizar esta macro tenha a possibilidade de incluir também cláusulas específicas para cast/dump/get_term, definimos uma macro especial, __before_compile__ que nos permite inserir código após o módulo ter sido compilado, mas antes da compilação ser dada como finalizada. Neste caso, através disso incluímos versões das funções load/dump/get_term que funcionam para quaisquer valores não válidos e resolve no átomo :error (que é o esperado pelo Ecto quando um valor não é válido).

Uma vez que a função load/1 simplesmente executa a função cast/1 não há necessidade de definir uma versão de erro para ela.

A, também especial, macro __using__:

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

Uma vez que é uma macro, irá ser avaliada aquando da compilação do código. Nela, utilizamos Keyword.fetch! para o termo :values das opções que forem passadas ao use, por exemplo use TypedEnum, values: [:bid, :request, :upload, :pay] significa que a variável opts será uma lista Keyword com uma chave :values, cujo valor será uma lista de átomos (:bid, :request, etc). Uma vez que a chave :values é essencial ao funcionamento da macro, caso não esteja presente nas opts queremos que surja um erro, e daí utilizarmos fetch!.

De seguida decidimos se devemos definir o enumerador com valores de números inteiros ou listagramas, e fazemo-lo avaliando a forma do valor da chave :values. Caso seja uma lista Keyword assumimos que seja números inteiros, pois será composta por pares de versão_átomo -> número_inteiro_correspondente. Caso seja uma simples lista de átomos, assumimos que o desejado é que o enumerador use listagramas como formato.

Dependendo disso fazemos algumas verificações para nos certificarmos que os valores têm o formato correcto. Tudo isto acontece aquando da compilação, logo qualquer erro que haja emergirá nessa altura, o que permite ao utilizador saber imediatamente se há algum problema.

Agora iremos fazer uma leitura do que acontece na versão com base em números inteiros. Visto ser mais complexa que a com base em listagramas acredito que compreendendo esta será fácil ler a outra. Sendo uma macro que utiliza quote, os conteúdos dos blocos de quote do end irão ser introduzidos nos módulos que utilizarem esta macro e é o que se passa nesses blocos que é o mais importante.

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
.            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

Neste bloco de quote usamos o atributo bind_quoted, que nos permite passar variáveis propriamente expandidas para serem utilizadas no interior do bloco. Neste caso, atoms_ints com os valores contidos em values (o que havia sido passado como chave :values à macro), e também mod, contendo o nome do módulo que executou a macro.

Primeiro incluímos imediatamente o atributo @before_compile, que faz com que após a compilação, mas antes da sua finalização, a macro __before_compile__ do módulo em questão seja executada, nesta caso o __before_compile__/1 do módulo TypedEnum. Depois utilizamos a lista de Keywords (atoms_ints) para gerar um conjunto de mapas e listas que nos auxiliarão a decidir que validações e transformações são necessárias quando as funções de comportamento Ecto.Type sejam executadas.

Dessa lista de values, geramos 4 mapas - se utilizarmos o exemplo anterior onde a chave :values assumia o valor de uma lista com a seguinte forma [val_1: 1, val_2: 2]), o resultado de atom_integer_map, string_integer_map, string_atom_map, integer_atom_map será respectivamente:

mapa com as mesmas chaves e valores que a lista Keyword, neste caso: %{val_1: 1, ...}
outro mapa idêntico mas agora com as chaves em formato de listagrama ao invés de átomos: %{"val_1" => 1, ...}
um mapa que mapeia as correspondências entre listagramas e versões em átomos: %{"val_1" => :val_1, ...}
por último, as correspondências entre números inteiros e as suas equivalências em formato de átomo, e.g: %{1 => :val_1, ...}

Com estes quatro mapas auxiliares será mais fácil resolvermos os valores do tipo dependendo se estamos a fazer uma validação/transformação (cast/1) ou uma serialização (dump/1), e tendo em conta o formato do termo original (n. inteiro, listagrama ou átomo).

De seguida geramos 3 listas adicionais, que nos facilitarão restringir as cláusulas das nossas funções:

para listagramas: ["val_1", ...]
para átomos: [:val_1, ...]
para números inteiros: [1, ...]

Colocamos todas estas variáveis auxiliares em atributos do módulo (o que significa que quem esteja a utilizar o mesmo, os pode aceder com a síntaxe usual, @nome_do_atributo), e utilizamos a lista de valores válidos nos seus diferentes formatos para providenciar uma função, values/1, que resolve em todos os valores válidos para o formato desejado.

As funções cast (e no nosso caso load também), são usadas para transformar termos do seu formato interno na base de dados, ou externos à nossa aplicação, no formato utilizado pela nossa aplicação, neste caso em átomos. Por conseguinte, se parte do código por algum motivo executar load/cast com valores já no formato correcto, a resolução dessas funções resultará no mesmo termo. Se no entanto utilizarmos termos no formato de listagramas ou números inteiros, se tiverem uma correspondência válida, serão convertidos no seu formato de átomo. Caso sejam inválidos emergirão um erro.

No caso da função dump, a lógica é em tudo semelhante, com a diferença que o formato final que procuramos é o de número inteiro (para serialização na base de dados), neste caso, listagramas e átomos, se tiverem uma correspondência válida, serão convertidos no seu formato de número inteiro, caso o termo a avaliar seja já um número inteiro e válido, a função resolverá com esse mesmo termo.

Uma vez que cada função load/1 e dump/1 tem a acompanhar um grupo de restrições, caso os valores passados não sejam válidos, as cláusulas que inserimos através do @before_hook serão as executadas (uma vez que aceitam qualquer termo e não têm guardas), e resultarão num erro.

Caso o utilizador do módulo, tenha adicionado cláusulas extra, devido à ordem final de todas elas no módulo compilado, essas serão tentativamente executadas antes das dos erros. Por exemplo, imaginemos que estamos a programar uma aplicação que necessita interagir com uma versão antiga da base de dados, em que ao invés de utilizarem o termo "val_1", estavam a utilizar o termo "valor_1". Actualmente "valor_1" já não é uma representação válida, por isso não queremos permitir que seja persistida pelo nosso sistema, mas ao mesmo tempo, tendo uma correspondência directa ao termo "val_1" que por sua vez é válido, queremos convertê-la caso ela surja como parâmetro em alguma situação.

Nesse caso poderiamos definir o nosso tipo da seguinte forma:

defmodule ExampleCallerModule do
  use TypedEnum, values: [val_1: 1, val_2: 2]

  def cast("valor_1"), do: {:ok, :val_1}
  def dump("valor_1"), do: {:ok, 1}
  defp get_term("valor_1"), do: :val_1
end

E isto permitir-nos-ia validar, traduzir e verificar a sua equivalência, sem no entanto necessitar de introduzir mais um item no enumerador, ou complicar significativamente o nosso programa de forma a suportar tal caso.

Adicionalmente às funções requeridas pelo comportamento Ecto.Type eu pessoalmente gosto de definir também uma dump!/1, que simplesmente executa a dump/1 normal, e caso a resolução seja qualquer outra coisa que uma tupla {:ok, _}, que emerja um erro.

A função get_term é simplesmente para facilitar a verificação de equivalência entre termos.

E isto é basicamente tudo o que é necessario. A versão em listagramas é em todo símile a esta, mas mais simples uma vez que apenas necessita de funcionar com átomos e listagramas.

Como última nota relativa ao módulo que escrevemos, optei por manter duas versões separadas para a versão listagrama e outra para a versão de números inteiros - isto porque apesar de haver partes duplicadas entre ambas, escrever apenas uma versão requereria uso de várias condicionais, e penso que o código resultante seria bem mais complicado e não linear de entender. Assim temos um pouco de duplicação mas cada parte trata apenas do que tem a tratar.

E agora que vemos como poderíamos implementar tal módulo para nos auxiliar, caso queiram podem utilizar um pacote já feito, disponível em hexdocs (ou github mnussbaumer/typed_enum. De qualquer das formas espero que tenha sido útil.

Creating an Ecto Enum Type

Micael Nussbaumer — Sun, 30 Jan 2022 14:25:00 +0000

portuguese version

Today I'm going to write about a little Ecto type I use regularly in most Elixir projects I work in. Basically, it's an enum wrapper that allows you to use a limited set of :atoms or "strings" values in your Ecto schemas definitions.

You might be wondering why a writing an enum type when Ecto ships with an Ecto.Enum type since v3.5. The main reason is the typing of the "type" itself and sharing this type between multiple schemas.

Ecto.Enum is great and very handy for most regular cases where you simply want to be able to deal with atom versions of strings in your schemas and Elixir logic, while being able to store them as strings automatically at the database level and be able to cast from strings to atoms - which is a pretty normal usecase when dealing with external parameters or inputs.

To show you what I mean lets say you have a schema like:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, :string)
  end
end

Now, this is a totally valid schema. But if you have a limited set of values that :action might assume, right now you have no way of ensuring that when dealing with casting / inserting records of this type of schema the strings that are passed on are valid and you couldn't tell from the schema that this field can only assume certain values.

You might create this constraint at the database level, and if it's an essential part of your business logic you probably should keep alongside such constraint (although it makes it more onerous to update and change), but even if you don't, it would be nice to at least in your business logic be able to treat this :action field as having only a value from a set of known, well defined values, without having to explicitly read the field/param and manually checking it.

This is where Ecto.Enum can be used. Applying it to our schema means:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, Ecto.Enum, values: [:bid, :request, :upload, :pay])
  end
end

In this case, we basically defined as valid values only that specific set of atoms, and by extension, when casting the schema. Any string counterpart that matches those atoms if converted would also be a valid value.

What this gives us is:

the ability to cast params and make sure the action field is only valid if it conforms to those values, ex:


params = %{"user_id" => 1, "action" => "bid"}

Ecto.Changeset.cast(%User.Action{}, params, [:user_id, :action])

The ability to deal strictly with the atom versions in our programs logic, by being sure that after casting or retrieving a record from the database it will have to conform, ex:


def log_action?(%User.Action{action: action}),
    do: action in [:bid, :pay]

Importantly, it also surfaces immediately that this field has a specific set of values (is not an arbitrary :string).

In many cases this is enough. But if we, for instance, want to use that same "action type" in another schema, then we have to duplicate its definition - and it's never really the same "type", we need to keep both (or more versions if used in more places) in synch. It also doesn't expose directly the normal cast/load/dump functions that can also be useful, in some contexts. For instance, you might want to cast a single string somewhere and depending on the success of such take a path on your code or another.

It also doesn't allow you to use integer versions for the underlying enum definition at the database level, nor do some useful tricks with casting & evaluating the value when dealing with "translations" - e.g. some external datasource uses some strings values that you don't want to store as they are, nor assume as valid for the field, but instead convert them to your app's version, or you simply assert their "equivalence", even if they have different representations - say one external source uses bidding but all others including yours use bid. You don't want to allow bidding to be stored or deal with it internally, but you want to be able to cast bidding as bid.

There's also the fact that it doesn't expose a @typespec that you can use, which can be useful as well, when using dialyzer (which you, most of the times, should be) or simply when writing @specs in your code's documentation (which you should always strive to do).

None of these are showstoppers though and if you don't need any of the additional functionality then just using Ecto.Enum is a significant improvement over bare strings and might cover all your requirements while being part of Ecto itself (meaning, no additional dependencies or code to maintain).

If we were to be writing a single Ecto.Type, following the examples before, we could do something such as:

defmodule User.Action.Type do
  @behaviour Ecto.Type

  def type, do: :string

  @valid_types [:bid, :request, :upload, :pay]
  @valid_strings Enum.reduce(@valid_types, [], fn t, acc -> [Atom.to_string(t) | acc] end)
  @valid_map Enum.reduce(@valid_types, %{}, fn t, acc -> Map.put(acc, Atom.to_string(t), t) end)

  @valid_inverse_map Enum.reduce(@valid_types, %{}, fn t, acc -> Map.put(acc, t, Atom.to_string(t)) end)

  @type t() :: unquote(Enum.reduce(Enum.reverse(@valid_types), &{:|, [], [&1, &2]}))

  def load(data), do: cast(data)

  def cast(data) when is_atom(data) and data in @valid_types, do: {:ok, data}
  def cast(data) when is_binary(data) and data in @valid_strings, do: {:ok, @valid_map[data]}
  def cast(_), do: :error

  def dump(data) when is_atom(data) and data in @valid_types, do: {:ok, @valid_inverse_map[data]}
  def dump(data) when is_binary(data) and data in @valid_strings, do: {:ok, data}
  def dump(_), do: :error

  def embed_as(_), do: :dump
  def equal?(data, data), do: true

  def equal?(data_1, data_2) do
    case {cast(data_1), cast(data_2)} do
      {{:ok, same}, {:ok, same}} -> true
      _ -> false
    end
  end
end

Breaking it down:

declare a behaviour of Ecto.Type for our module (this ensures we'll get warnings if we don't implement some functions that are expected to be defined for such behaviour)
Declare a function type/0 that returns the underlying type of the Ecto.Type (what is used to persist values of this type in the database), in this case :string (which ecto uses to abstract over all char, varchar, and text column types)
Declare a typespec, that returns the valid atoms this type represents
From the list of possible atoms, we create a list of their counterparts in string form, and two module attributes to store the mappings between atom<->string versions and vice-versa
Declare all required behaviour functions for the Ecto.Type, since we have lists of both atoms and strings representing values we consider valid, we can use these in guards, to cast them appropriately to their atom form when in string, and the same for dumping them from atoms to strings.

As you can see it's easy to create an Ecto.Type that provides us with a more constrained set of values and helps us being more assertive on our code about what we expect.

It also makes it so that now we can actually cast those values independently of a schema. We can just do User.Action.Type.cast("bid") to validate if it's a valid action according to the type.

If you don't mind the copy pasting and changing the @valid_types list whenever you want to add a new type it's fine to use it like this. You can just use this as a blueprint then change the module name, and the list of values whenever you need to define another enum.

This already has some advantages, besides being able to cast the values independently of schemas, the type itself is now independent of their declaration in a schema, and as such can be used in multiple schemas as well.

We could also easily make it use integers for its underlying representation, as in a conventional enum. But we can also write a macro that effectively does all of this for us with a single line of code. Our objective is to have a macro that enables us to write the previous User.Action.Type module as:

defmodule User.Action.Type do
  use TypedEnum, values: [:bid, :request, :upload, :pay]
end

Followed by using it on the schema:

defmodule User.Action do
  use Ecto.Schema

  schema("users_actions") do
    belongs_to(:user, User)

    field(:action, User.Action.Type)
  end
end

And if we want to deal with atoms in our inner logic, but store them as integers in the database, while still allowing the casting of stringed versions:

defmodule User.Action.Type do
  use TypedEnum, values: [bid: 0, request: 1, upload: 2, pay: 3]
end

Let's breakdown what our module will need to do:

Implement the __using__ macro, so that it can be "used" in the way we wrote in the previous examples
Be able to distinguish between a plain list of atoms or a Keyword list being passed as values to allow both integer and string based enums at the database layer
Allow the caller to write any additional cases for the castdump logic of the Ecto.Type behaviour
Implement automatically a typespec for it, reflecting all the valid atoms it can assume
Raise on invalid values being passed to provide compilation time blowups.

This is our final module and we'll break it down after:


defmodule TypedEnum do
  defmacro __before_compile__(_env) do
    # these are inserted in the before_compile hook to give opportunity to the
    # implementing module to define additional variations
    quote do
      def cast(_), do: :error
      def dump(_), do: :error
      defp get_term(data), do: data
    end
  end

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

  defp validate_int_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &valid_int_enum?/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def validate_string_enum(values) do
    with {_, true} <- {:length, length(values) > 0},
         {_, true} <- {:format, Enum.all?(values, &is_atom/1)} do
      :ok
    else
        error -> raise_error(error)
    end
  end

  def valid_int_enum?({k, v}),
    do: is_atom(k) and is_integer(v)

  def raise_error({:length, _}),
    do: raise "TypedEnum expects `:values` to be a list or keyword list with at least 1 element"

  def raise_error({:format, _}),
    do: raise "TypedEnum expects the format of `:values` to be a keywordlist with the atom version as the key and an integer as the value (e.g.: [atom_key: 1, another_possible: 2, ...]), or a list of atoms for the string enum version (e.g.: [:atom_key, :another_possible, ...])"

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

  def bind_as_stringed_version(values, mod) do
    quote bind_quoted: [atoms: values, mod: mod] do
      @before_compile TypedEnum

      strings = Enum.map(atoms, fn entry -> Atom.to_string(entry) end)
      mapped = Enum.zip(strings, atoms) |> Enum.into(%{})

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :string

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :validation_mappings, mapped)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings) :: list(t()) | list(String.t())
      def values(type \\ :atoms)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms), do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, String.to_atom(data)}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, Atom.to_string(data)}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: data

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings[data]
    end
  end
end

We declare the __before_compile__ macro that allows us to place some code in the module after everything else is in place, in this case we want to include some catch-all clauses. The reason we do it in the __before_compile__ hook is that this then enables a user of the module to implement their own versions of those functions, which can be useful for instance to include clauses that cast correctly values that come from third party systems that we can't change, but we do not want to allow on our db, instead translate them to our versions.

Because the load/1 function will already catch all possible terms and simply call cast/1 we do not include it there.

Then the actual __using__ macro.

  defmacro __using__(opts) do
    values = Keyword.fetch!(opts, :values)
    mod = __CALLER__.module

    is_int_enum? = Keyword.keyword?(values)

    case is_int_enum? do
      true ->
        :ok = validate_int_enum(values)
        bind_as_integer_version(values, mod)

      false ->
        :ok = validate_string_enum(values)
        bind_as_stringed_version(values, mod)
    end
  end

This is a macro, so it will be called at compile time. We fetch! the :values key from the options - since it's required we use the ! version in order to raise when it's missing. We then decide if the underlying enum will be integer or string based through a very simple heuristic. Are the values a keyword list or a plain list? If it's keyword list then it must be an integer based enum, otherwise a string based enum.

Depending on that we do a check on the form of the values to make sure they comply and then generate the actual code to fill the caller module. Remember that this is a macro, so these checks/raises will occur at compile time allowing us to signal to the user of our module any problem.

We'll only go through the integer version since it's the most complex, but you should be able to then understand easily the stringed version too. The validation helpers are plain functions, but since they're called in the context of a macro they will raise at compile time. Lastly, both bind_as_ functions do quote their contents so that these are placed in the calling module as code.

  def bind_as_integer_version(values, mod) do
    quote bind_quoted: [atoms_ints: values, mod: mod] do
      @before_compile TypedEnum

      atom_integer_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, atom_val, int)
        end)

      string_integer_map =
        Enum.reduce(atom_integer_map, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, Atom.to_string(atom_val), int)
        end)

      string_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, _}, acc ->
          Map.put(acc, Atom.to_string(atom_val), atom_val)
        end)

      integer_atom_map =
        Enum.reduce(atoms_ints, %{}, fn {atom_val, int}, acc ->
          Map.put(acc, int, atom_val)
        end)

      strings = Enum.map(atoms_ints, fn {atom_val, _} -> Atom.to_string(atom_val) end)
      atoms = Enum.map(atoms_ints, fn {atom_val, _} -> atom_val end)
      ints = Enum.map(atoms_ints, fn {_, int} -> int end)

      @behaviour Ecto.Type
      @impl Ecto.Type
      def type, do: :integer

      Module.put_attribute(mod, :valid_atoms, atoms)
      Module.put_attribute(mod, :valid_strings, strings)
      Module.put_attribute(mod, :valid_ints, ints)
      Module.put_attribute(mod, :validation_mappings, string_atom_map)
      Module.put_attribute(mod, :validation_mappings_atoms, atom_integer_map)
      Module.put_attribute(mod, :validation_mappings_strings, string_integer_map)
      Module.put_attribute(mod, :validation_mappings_ints, integer_atom_map)

      @type t() :: unquote(Enum.reduce(Enum.reverse(atoms), &{:|, [], [&1, &2]}))

      @spec values(:atoms | :strings | :ints) :: list(t()) | list(String.t()) | list(integer())
      @doc "Given a desired format returns the matching values for that format"
      def values(type \\ :atoms)
      def values(:ints), do: unquote(ints)
      def values(:atoms), do: unquote(atoms)
      def values(:strings), do: unquote(strings)

      @impl Ecto.Type
      def load(data), do: cast(data)

      @impl Ecto.Type
      @doc false
      def cast(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, data}

      def cast(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings[data]}

      def cast(data) when is_integer(data) and data in unquote(ints),
        do: {:ok, @validation_mappings_ints[data]}

      @impl Ecto.Type
      @doc false
      def dump(data) when is_atom(data) and data in unquote(atoms),
        do: {:ok, @validation_mappings_atoms[data]}

      def dump(data) when is_binary(data) and data in unquote(strings),
        do: {:ok, @validation_mappings_strings[data]}

      def dump(data) when is_integer(data) and data in unquote(ints), do: {:ok, data}

      @doc "Dumps but raises in case of non-valid data"
      def dump!(data) do
        case dump(data) do
          {:ok, value} ->
.            value

          _ ->
            raise Ecto.CastError,
              message: "Unable to dump:: #{inspect(data)} ::into:: #{inspect(unquote(mod))}",
              type: unquote(mod),
              value: data
        end
      end

      @impl Ecto.Type
      @doc false
      def embed_as(_), do: :dump

      @impl Ecto.Type
      @doc false
      def equal?(term_1, term_1), do: true
      def equal?(term_1, term_2), do: get_term(term_1) == get_term(term_2)

      defp get_term(data) when is_atom(data) and data in unquote(atoms),
        do: @validation_mappings_atoms[data]

      defp get_term(data) when is_binary(data) and data in unquote(strings),
        do: @validation_mappings_strings[data]

      defp get_term(data) when is_integer(data) and data in unquote(ints),
        do: data
    end
  end

We do bind_quoted passing the values as the variable atoms_ints and caller module as mod.

The generated code will include the @before_compile hook referencing this module where we defined it, TypedEnum. Then we use the keyword list of atoms_ints to generate a few different maps and lists that we'll use further down.

Because keyword lists of literal terms are literal proplists inside macros it makes them very ergonomic to use inside macros. They also provide a nice easy syntax to pass options as in this case.

So from the values keyword list (e.g.: [val_1: 1, val_2: 2]) we generate 4 maps:

A simple map version of the proplist, the prop key as map key and the integer as the value, e.g.: %{val_1: 1, ...}
Another in the same form but now the keys as strings, e.g: %{"val_1" => 1, ...}
Another string to atoms mapping, e.g: %{"val_1" => :val_1, ...}
And lastly a integer to atom mapping, e.g: %{1 => :val_1, ...}

These 4 maps will be used to fetch the right form of the enum depending on if we're loading or dumping it, and if the original data being passed in is in atom, string or integer form.

Then we generate 3 lists to aid us in adding guard clauses to our functions:

for strings, e.g: ["val_1", ...]
for atoms, e.g: [:val_1, ...]
for ints, e.g: [1, ...]

We put all these things as module attributes (so the caller module can access them if they want, with the @... syntax) and we use them to provide the helper function values(type), that allows the user to retrieve the valid enum representations in any of its valid forms.

The cast functions of the Ecto.Type (and in our case the load) are used to transform values from their representation in the database, or from external requests, into our internal representation, which are atoms. Following this then means that if we trie to load or cast an atom that is a valid representation we just return that same atom since it's already in its correct form. If on the other hand it's a string (comes from an external request perhaps), or an integer (loaded from the database layer), we, with the aid of the mappings we assembled before, convert those into their atom version.

For the dump it's the same logic, but in this case the final form we want is the integer form, so we convert atoms and strings into their integer version, and if it's already an integer and a valid one we use that.

Otherwise, because of the guards we set up on each function, if the value they're called with is invalid, the catch all clauses that we set on the @before_hook will kick in and error at the ecto layer.

This allows still a user of the module to implement intermediate versions. Say one client that interacts with this is using a legacy version of the enums, where instead of "val_1" they were using "some_prop" as its identifier. We could set a specific clause on our caller module handling that specific case, for example:

defmodule ExampleCallerModule do
  use TypedEnum, values: [val_1: 1, val_2: 2]

  def cast("some_prop"), do: {:ok, :val_1}
  def dump("some_prop"), do: {:ok, 1}
  defp get_term("some_prop"), do: :val_1
end

And this would allow us to cast it, and even test its equality automatically by defining as well the get_term version for it.

I also like to add a specific dump! version that just calls dump underneath and raises in case dump returns anything else than an :ok tuple. The get_term are just helpers to simplify the equality checks.

And this folks, is all there is to it. The stringed version is very very similar, but simpler because it only has to deal with atoms and strings.

I opted to keep them as two separate functions because although there's some overlap and duplication going on, I think that having a whole bunch of conditionals on the macro for generating functions depending on if it's the integer or stringed version, and different module attributes, etc would just make it less straightforward to read and change if the need arises.

Now that you've seen how it works, if you don't want to write it yourself you can just include the package I've you can find it here in hexdocs (or through github in mnussbaumer/typed_enum and use it straightaway. Otherwise hope you could take away something useful.

Getting Started with ImageEngine and Angular

Micael Nussbaumer — Thu, 19 Aug 2021 12:01:01 +0000

Intro

In this article we'll go through using @imageengine/angular package in a sample project to easily take advantage of ImageEngine's CDN and optimisation engine.

ImageEngine is a super fast CDN that exposes an accessible engine to optimise your app image assets. You can optimise your distributions (which we'll call Engines) in different ways - through ImageEngine's dashboard, defining settings for your Engine or specific paths in your Engine, or through simple query parameters (called directives in ImageEngine), that allow you to dynamically, on-the-fly, execute optimizations to your assets.

The part we'll be leveraging here, through this component, is the query parameters - although this is mostly hidden from you and you don't have to worry about it to use the component.

To deliver actual real optimizations to your images you need an ImageEngine Engine , nonetheless this package can still be used by itself for enabling lazy loading and preventing Cumulative Layout Shift without one.

You can check the small app we'll be building, and the GitHub repo. It's a very basic sample app to show you how to use the component and visualise the different fit types in relation to image size and container size. After we write our app you can also try out all other properties of the component by changing the input parameters used.

The slashed border around the image represents the size you pick from the size select, the image inside it is the actual image, displayed according to the type of fit you chose.

Summary:

Intro
Setting up the Angular Project
Defining our Layout and Content
Deploy to Github Pages
Create ImageEngine Engine
Conclusion

Setting up the Angular Project

To follow along you'll need to have Node.js, NPM (that comes with node.js), and Angular CLI installed.

The versions used for this tutorial are npm 7.15.1, node.js v16.3.0 and Angular CLI 12.1.1.
To deploy on GitHub pages you'll also need a Github account and git installed, the version used here was 2.25.1.

With those installed, from the command line run, (answering N to the prompt for Angular Router and choosing CSS for the styles):

ng new ie-angular-sample
cd ie-angular-sample
npm install @imageengine/angular

Now we need to add the @imageengine/angular module and the Forms module to our app imports.

Open src/app/app.module.ts and make sure it looks like this:

import { NgModule } from '@angular/core';
import { BrowserModule } from '@angular/platform-browser';
import { FormsModule } from '@angular/forms';

import { AppComponent } from './app.component';
import { NgxImageengineModule } from "@imageengine/angular";

@NgModule({
  declarations: [
    AppComponent
  ],
  imports: [
      BrowserModule,
      NgxImageengineModule,
      FormsModule
  ],
  providers: [],
  bootstrap: [AppComponent]
})
export class AppModule { }

FormsModule is included in angular - we need it for form bindings - and NgxImageengineModule is the module exported from the @imageengine/angular package.

Now we're ready to use the image component in our app.
To save some space we're not going to show here the CSS we'll be using, because it has also a style reset sheet included, but make sure to copy it over from styles.css (replace the contents of src/styles.css with those of that file).

Defining our Layout and Content

Now let's edit src/app/app.component.ts to:

import { Component, ChangeDetectorRef, AfterViewInit } from '@angular/core';
import { IEDirectives, IEFit } from "@imageengine/imageengine-helpers";

const OPTIONS = [
    [1920, 920],
    [960, 460],
    [480, 230],
    [240, 115]
];

const FITS: IEFit[] = [
    "stretch",
    "box",
    "letterbox",
    "cropbox"
];

@Component({
  selector: 'app-root',
  templateUrl: './app.component.html',
  styleUrls: ['./app.component.css']
})

export class AppComponent implements AfterViewInit {
    image_path: string = "assets/h-lightbox-3.jpeg";
    sizes: number[][] = [...OPTIONS];
    size: number = 3;
    fits: IEFit[] = [...FITS];
    fit: IEFit = "cropbox";
    width: string = `${OPTIONS[3][0]}px`;
    height: string = `${OPTIONS[3][1]}px`;
    src_url: string | null = "";

    constructor(private change_detector: ChangeDetectorRef) { }

    ngAfterViewInit(): void {
       this.set_src_url();
    }

    change_size(new_val: number): void {
        this.size = new_val;
        this.width = `${this.sizes[new_val][0]}px`;
        this.height = `${this.sizes[new_val][1]}px`;
        this.change_detector.detectChanges();
        this.set_src_url();
    }

    change_fit(new_val: IEFit): void {
        this.fit = new_val;
        this.change_detector.detectChanges();
        this.set_src_url();
    }

    set_src_url() {
        let img = document.querySelector(".ngx-ie-image");
        if (img) {
            this.src_url = img.getAttribute("src");
            this.change_detector.detectChanges();
        }
    }
}

We import some type helpers from @imageengine/imageengine-helpers, this is a package that is a dependency of @imageengine/angular and contains the type specifications that the component uses plus some helpers that that package utilises (although we're not going to need them here).

Our component is very simple, we have a few properties we'll use to drive our UI, some functions to be used as callbacks (change_size and change_fit) and a helper (set_src_url) to get the url that was computed and in actual use by the component to fetch the asset.

Let's also edit src/app/app.component.html to:

<div id="main-container" >
  <div class="image-holder" [style.minWidth]="width">
    <div style="display: flex; flex-flow: row wrap; width: 100%; justify-content: center; align-items: center;">
      <select [ngModel]="size" name="select-size" (ngModelChange)="change_size($event)">
          <option [value]="index" *ngFor="let opt of sizes, index as index">{{opt[0]}}px x {{opt[1]}}px</option>
      </select>
      <select [ngModel]="fit" name="select-fit" (ngModelChange)="change_fit($event)">
          <option [value]="opt" *ngFor="let opt of fits">{{opt}}</option>
      </select>
    </div>
    <p style="margin: 20px auto; color: white;">Generated URL: {{src_url}}</p>
    <ngx-imageengine [wrapper_styles]="{width: width, height: height}" [derive_size]="true" [path]="image_path" [directives]="{format: 'jpg', fit: fit, compression: 10, sharpness: 15}"></ngx-imageengine> 
  </div>
</div>

Besides the normal html and the select's bound to the component properties, we have our ngx-imageengine component and our focus will be on that.

We won't cover everything about the component here, but in case you're curious just read the package readme as it provides a much more in depth overview of both the properties and details to make the best use of it. Here we'll explain some of the most common and useful properties of the component.

Our web-app displays an image inside a rectangle element and allows us to set the size of this container rectangle and the fit we wish for our image (the way the image is placed and sized relative to the rectangle/sizes we define). It also shows us the resulting url generated to fetch that image taking into account the directives we set from ImageEngine's CDN. It also allows you to visualise how the type of fit affects the display of the image.

For the component properties we set:

wrapper_styles The component itself renders a wrapper div, with an img tag inside of it. We can pass specific styles to be inlined both on the wrapper and img tag (and classes as well). Here we're only defining the width and height according to the properties of our component (that reflect the chosen option). This will make the wrapper itself assume these dimensions.
derive_size We set this property to true to indicate that we want our component to automatically derive the desired sizes for the image from the size of the wrapper. Notice that this would also work if the wrapper was styled through normal CSS - in which case we wouldn't need to pass the wrapper_styles object.
path The relative path to the image asset.
directives This is an object specifying what directives we want to apply to our image when requesting it from ImageEngine's CDN. In this case we're specifically setting the image format to jpg, the fit to whatever is selected from the fit select input, a compression of 10 (interpreted as percentage, goes from 0 to 100, defaults to 0 when not specified) and sharpness of 15, to add a bit of sharpness to the image. All of these could be dynamically set as is the case with fit.

To see all options you can check the @imageengine/angular readme

Right now because we didn't specify a host from where to fetch the image the component will fetch it from its relative location on your app. If you visit the sample app or the Github Repository you might have noticed that the path includes "/ie-loader-images/" instead of /assets/ but if you're following this tutorial that is irrelevant (it's related to the Engine and CDN I'm using for this example).

To recap - we set a size for the wrapper, we tell our component to derive the image size from the wrapper, and we give it a few directives to optimise our image. Notice we didn't have to pass width and height directives since we're using derive_size.

Before seeing our app in action, let's add the remaining missing things.
Edit src/app/app.component.css to:

#main-container {
    width: 100vw;
    min-height: 100vh;
    background-color: transparent;
    padding-top: 50px;
}

.image-holder {
    background-color: transparent;
    min-width: 100vw;
    text-align: center;
}

.image-holder select {
    font-size: 18px;
    height: 40px;
    padding: 20px;
    font-weight: lighter;
    background-color: black;
    color: white;
}

Which is just some basic styling for our elements.

Let's also add our image, you can copy it from the GitHub repo.
Put it in the src/assets folder.

Once that is done we should be able to start our app with, from the command line on the root of our project:
npm run start

And then visiting localhost:4200.

If you now change the settings from the selects you should see how that affects the image and the url. If you change on the component itself the directives being passed you will also see them reflected on the url and on the image.

One thing to take into consideration is the fact that we're using device_pixel_ratio conversions, so if you're visualising the app in a monitor with higher resolution than 1:1 (like a retina display) the sizes you see being encoded in the url will respect that and get the correct dimensions for the intrinsic size you want to display.

You can turn that off and there's other properties as well to help you make the most out of ImageEngine easily.

This also means that some images might be requested at higher dimensions than your original images have. When this is the case ImageEngine won't apply crop directives, nor upscale the image and instead return the image in its original size (with other non-size related directives applied).

The component will take care of displaying them according to the chosen fit without you needing to do anything. You can see this by choosing 1920x920 as the size and cropbox fit. The original image is smaller in size than that, so ImageEngine will return it in its original dimensions without the cropbox factor we specified, but our component still displays it as the cropbox would look - because the original size is smaller than the final intrinsic size you might notice the image loose some quality.

You'll also see, if you change the size to a bigger one and then back to a smaller one, that the component changes from the smaller to the bigger one, but then keeps the bigger one as long as it's bigger than the new selection. This can be turned off by using the @Input force_size_recalculation with true.

It defaults to false because in practice , it means that if you have responsive and/or derive_size on, if the sizes change to a smaller one, instead of asking for a new image in those smaller sizes (that would need to be transferred) it keeps the sizing that's bigger - meaning it uses the already fetched image. Nonetheless if you're changing ratios between breakpoints for instance, and using cropbox you might actually want to force a refetch, in those cases you can turn that flag on, but in almost every case it's better, more performant to just keep force_size_recalculation as false.

One last note regarding the directives the component supports. Setting the letterbox colouring isn't possible directly through the component, but is easily achievable by just giving a background-color to the wrapper element and using the box fit method.

Deploy to Github Pages

Angular has support for deploying on GitHub pages with the help of a package named angular-cli-ghpages.

We won't cover how to create a GitHub repo, but once you have that, add your repo as a remote:

git remote add origin git@github.com:<YOUR_USERNAME>/<THE_NAME_OF_YOUR_REPO>.git

And substitute the values between <...> by your username and the name you gave the repo.
Now let's add the angular-cli-ghpages with:

ng add angular-cli-ghpages

Followed by:

ng deploy --base-href=/<THE_NAME_OF_YOUR_REPO>/

(if you have a custom domain associated with your GitHub pages, you need to add the flag cname to the command, like --cname=your_domain.com)

Now you should be able to check the website online from Github Pages, the address should be https://YOUR_GITHUB_USERNAME.github.io/THE_NAME_OF_YOUR_REPO.

You can see that it works even without an host Engine - obviously it won't do any real optimization to your assets until we add one.

Create ImageEngine Engine

Since now we have our resources online, we are able to create a CDN distribution for it with ImageEngine. You can sign up for a free trial of ImageEngine by following the instructions on the following video and using the address of your newly created website.

imageengine.io

Once that is done you should have an ImageEngine delivery address. We'll now add that to our component and re-deploy with the delivery address set. To do that change your component to have the property host set to whatever is the address you got previously.

<ngx-imageengine [wrapper_styles]="{width: width, height: height}"
                 [derive_size]="true"
                 [path]="image_path"
                 [directives]="{format: 'jpg', fit: fit, compression: 10, sharpness: 15}"
                 host="http://YOUR-ADDRESS-IN-IE.cdn.imgeng.in/">
</ngx-imageengine>

(if you have a domain and are running Github Pages with HTTPS then you need to enable it on your Image Engine settings as well)

Notice we put a forward slash at the end of the host address we just added. This is because the paths to work in Github Pages (at least with cname and base-href) need to be absent of leading slashes to be relative but we need it when prepending the domain. In this case it's simpler to add it to the host address.

And run again:
ng deploy --base-href=/<THE_NAME_OF_YOUR_REPO>/

Conclusion

And that's it, now you can see that the generated URL includes the domain. Because it works without an Engine as well (without optimising the images obviously) you can run and test locally your app, assert on the generated urls, and confirm it's working as expected, besides actually seeing how the images will look like. You can do the same in a staging environment without setting up multiple Engine's to deal with different origins.

To make it more production ready you would probably add an environment variable to control the host value, so that when building for production that is set and the ImageEngine CDN used, and when running locally it's not.

As you can see it's quite easy to add advanced image optimisations to your website. Exploring the potential of ImageEngine's CDN and Optimization Engine allow you to drastically reduce your image's payloads, leading to a much leaner and snappier experience for your end-users.

With the aid of this component you can easily create fully maintainable layouts and designs, that only require editing of your CSS/styles to be updated when they change, provide automatically the exact best fit and size according to that styling, prevent Cumulative Layout Shift, have responsiveness to screen orientation/size changes, lazy-loading and are aware of your end-user screen pixel ratio.

Hope you find it useful!

ImageEngine custom loaders for Next.js

Micael Nussbaumer — Tue, 29 Jun 2021 16:35:23 +0000

This article will guide you through writing custom Next.js loaders, specifically for ImageEngine’s distribution, but is applicable to any other situation where you need to customise the loaders. We’ll be building a grid layout image portfolio with the option to view the images in a full window lightbox. You can check the app we’ll be building here and the github repo.

Summary:

Intro
Creating a Next.js App
Defining our Layout and Content
Setting up Custom Loaders for use by our Next.js Image components
Deploying on vercel.com
Create a distribution with ImageEngine and set our vercel environment to use it
Conclusion

Intro

ImageEngine has custom image components for React and Vue, and in another post we went through how to use ImageEngine's custom image component in a Next.js.

Next.js <Image/> component is a pretty neat tool that is available in any Next.js project you might start. It allows one to set up a system of breakpoints, sizes and settings to apply to our image assets. It takes care of preventing Cumulative Layout Shift on page rendering when provided with the correct size information and optimize our assets according to buckets of resolution/viewport/image sizes.

It’s especially useful if you’re deploying on Vercel as it will be automatically enabled and performant. It works by generating optimized assets in terms of dimensions, and then caching those resources for subsequent requests on their edge servers. This for many cases will be enough and offers a much better experience than unoptimized assets.

On the other hand, if you’re not running on Vercel’s hosting, the performance of initial optimization is not as fast and can take a bit of time. If you’re not doing SSR and instead are deploying a static website build, as of now, you won’t be able to use their Image component at all.

It also works better when the assets you’re serving all fall into horizontal/square ratios, as it uses width to determine the optimized images. So if you have a reasonable mix of vertical and horizontally oriented images then it will be much harder to serve the minimum needed sizes for those vertical images.

If you’re serving image content in very high quantities, even when using Vercel.com then you might be interested in something that allows you to optimize your images up to the single pixel count, doesn’t rely on your server performance nor caching solutions, and offers very fast CDN distributions to deliver the best possible loading and viewing experience to your customers and users.

In this post we’ll review how to setup such a project by designing two different components with two different loaders, one for square thumbnails and one for a full window lightbox. In the process we’ll see how to write a custom Image loader for Next.js and some tricks for layout responsiveness.

Creating a `Next.js` App

To follow through you'll need to have nodejs and npm installed. The versions used for this tutorial are npm 7.15.1 and node v16.3.0. We’ll also install the vercel CLI.

On the folder where you'll be creating your project run:

npx create-next-app

Now let's install the ImageEngine Package:

npm i @imageengine/react

We should be able to start our app by doing

npm run dev

And visiting

http://localhost:3000

This should show us the default page. Let's remove the default templating index.js page has and replace it with:

import Head from "next/head";

export default function Home() {
  return (

      <div className="main-container">
        <Head>
            <title>ImageEngine Optimized Lightbox</title>
            <meta name="description" content="Next.js custom image loaders leveraging ImageEngine CDN for highly optimized images" />
            <link rel="icon" href="/favicon.ico" />
         </Head>
      </div>
  );
};

And change /pages/_app.js to the following

import Head from "next/head";
import "../styles/globals.css";

function IECustomLoader({ Component, pageProps }) {

    return (
        <>
          <Head>
        <link rel="icon" type="image/png" href="/favicon.png"/>
        <meta name="viewport" content="initial-scale=1.0" />
        <meta name="description" content="Learn how to use ImageEngine with Nextjs custom loaders to serve highly optimised image assets from your CDN to your users." />
        <meta property="og:title" content="ImageEngine with NextJS" />
        <meta property="og:description" content="Learn how to use ImageEngine with Nextjs custom loaders to serve highly optimised image assets from your CDN to your users." />
          </Head>
          <Component {...pageProps} />
        </>
    );
};

export default IECustomLoader;

Create a next.config.js on the root of our project and inside it:

module.exports = {
    env: {
    DISTRIBUTION: process.env.DISTRIBUTION
    },
    images: {
    deviceSizes: [1920, 1500, 1000, 500, 300],
    imageSizes: []
    }
};

Let's remove the things we don't need, delete pages/api folder, styles/Home.module.css and public/vercel.svg::

rm -rf pages/api
rm styles/Home.module.css
rm public/vercel.svg

Create a folder at the root level named components, and inside the public folder create a ie-loader-images folder.

We should now have the following structure:

...
pages /
  _app.js
  index.js

styles /
  global.css

components / 

public /
  favicon.ico
  ie-loader-images

You’ll also need to download the files in public/ie-loader-images and place them in /public/ie-loader-images/.

For this web page we’ll need 2 components, a thumbnail component and a lightbox component. Both components will receive a src and alt props for the image, an onClick function and the sizes of the viewport. The lightbox component will additionally receive two functions for moving to the next and previous image.

The CSS is important but to save space it won’t be written here, you can copy it over from globals.css.

As a side note, Next.js Image component fixes the size of the image to prevent Cumulative Layout Shift, but since we establish fixed size containers in CSS, that would have been addressed as well. The other interesting bit there is the small CSS trickery to get the hover effect to display the labels without JS. The remaining is basic styling.

Thumbnail Component

Create a file components/thumbnail.js with:

import Image from "next/image";
import { constructUrl } from "@imageengine/react/build/utils/index.js";

import { useState, useEffect, createRef } from "react";

function thumbnail_loader({ src, quality, distribution, width }) {
    let url = distribution + src,
    directives = {
        width: width,
        height: width,
        fitMethod: "cropbox",
        compression: 100 - quality,
        sharpness: 10
    };

    return constructUrl(url, directives);
};

export default function Thumbnail({ onClick, src, alt, window_sizes }) {

    let thumbnail_ref = createRef(),
    [width, set_width] = useState(null),
    [initial_size, set_initial_size] = useState(null);

    useEffect(() => {
    if (window_sizes) {

        let dimensions = thumbnail_ref.current.getBoundingClientRect(),
        n_width = Math.ceil(dimensions.width);


        if (!initial_size) { set_initial_size(n_width); }

        if (initial_size >= n_width) { set_width(initial_size); }
        else {

        set_initial_size(n_width);
        set_width(n_width);

        }
    }
    }, [window_sizes]);

    return (
    <div className="image-thumbnail" onClick={onClick} ref={thumbnail_ref}>
      {width ?
          <Image
            src={src}
            alt={alt}
            sizes={`(max-width: 320px) 300px, (max-width: 500px) 500px, (max-width: 1000) 1000px, (max-width: 1500) 1500px, ${width}px`}
            layout="responsive"
            objectFit="cover"
            objectPosition="center"
            width={width}
            height={width}
            loader={process.env.DISTRIBUTION ? (args) => thumbnail_loader({...args, distribution: process.env.DISTRIBUTION}) : undefined}
            quality={80}/> : null
        }
        <div className="image-details">
          <p>{alt}</p>
        </div>
    </div>
    );
};

Let’s go through the component part first. We receive 4 props onClick, src, alt, window_sizes. We create a DOM ref that we can use to get the actual HTML element of our wrapper, and we set two useState’s, one to store the current width, and another the store the initial width. Most times these will be the same.

Then we use useEffect with a dependency on the value of window_sizes. Meaning this will run on mount once and then any time the prop window_sizes changes. Inside this hook if the window_sizes has a value, we use the ref for getting the wrapper component dimensions, we round the value, set the initial_size if it’s not set yet, and, then, if the new width is bigger than the the initial_size we set both initial and width to this new size, and if not we keep the initial_size and set it as our width value.

The reason we do this is because if we have already fetched an image with bigger dimensions than the new size there’s no point in fetching the smaller one, since the bigger one can be displayed just as correctly and has more quality. On the other hand, if the window resize resulted in a thumbnail that is bigger than initially, we want to fetch a bigger image to keep the quality at a higher size. We’ll also replace the initial whenever that’s the case, so that resizing smaller and back into bigger won’t retrigger a new image.

Then our component itself is a wrapper with a property of onClick set to the handler passed down as a prop (that will open the lightbox) and the ref we created, so that we can use the ref to get the wrapper width.

Inside the wrapper we have the Next.js Image component, and we set src, alt, sizes (for the breakpoints), layout as responsive (so that it fetches new images if the width and sizes change), objectFit and objectPosition so that it forces the image to be displayed covering the full “square” and centered, quality, and in case there’s an environment variable DISTRIBUTION set, we use the specific loader, an anonymous function that calls the thumbnails_loader we defined at the top of the file with an additional argument, thedistribution url, otherwise we set it as null so the default one will be used.

When DISTRIBUTION is set that loader will be used to provide the actual url the Image component will use. By default the loader has access to 3 params provided by Next.js, the original src, quality value and width. In this case since it’s a square image, the width is enough, but we still need the distribution url, so that we can generate the right source for our image.

By using the constructUrl provided by @imageengine/react we can pass an object with properties that dictate how the CDN will provide the images. Here we pass width, height, fitMethod, compression (it’s the inverse scale from the quality) and sharpness to apply a small amount of sharpness to the final image.

note as of now, constructUrl isn’t directly exported by the package - which means that although probably unlikely, it’s place in the lib might change - nonetheless it will probably be made public in a future release, but otherwise, you can always either copy the specific logic for it, or use directives directly and build your own url.

Notice that this will generate the same url for all srcset sizes entries, and that is fine since the url codifies the needed settings itself, but when using ImageEngine we don’t really need to rely on srcset since we can generate exactly the sizes we want on-the-fly.

Lightbox Component

Now let's move to the lightboxcomponent, create the file components/lightbox.js:

import Image from "next/image";
import { constructUrl } from "@imageengine/react/build/utils/index.js";

function lightbox_loader({ src, quality, distribution, w, h }) {
    let url = distribution + src,
    directives = {
        width: w,
        height: h,
        fitMethod: "box",
        compression: 100 - quality
    };

    return constructUrl(url, directives);
};

export default function Lighbox({ onClick, src, alt, window_sizes, previous, next }) {

    return (
    <div className="lightbox" onClick={onClick}>
      <button className="previous" onClick={previous}/>
      <div className="image-lightbox">
        {window_sizes ? <Image
                  src={src}
                  alt={alt}
                  sizes={`(max-width: 320px) 300px, (max-width: 500px) 500px, (max-width: 1000) 1000px, (max-width: 1500) 1500px, ${window_sizes.w}px`}
                  objectFit="contain"
                  objectPosition="center"
                  width={window_sizes.w}
                  height={window_sizes.h}
                  loader={process.env.DISTRIBUTION ? (args) => lightbox_loader({...args, ...window_sizes, distribution: process.env.DISTRIBUTION}) : undefined}
          quality={90}
          />  : null}
      </div>
      <button className="next" onClick={next}/>
    </div>
    );
};

You’ll notice that this is very similar to the thumbnail component, but that we use instead the w and h values directly from the window_sizes prop, and we use a different objectFit and quality.

The reason we pass width and height to our loader is that since we want the image to fit the viewport, with both those values and the box fit, ImageEngine will be able to generate perfectly sized images for them, especially when they’re vertical.

In the intro I mentioned that cases where the image orientation is not horizontal, that ImageEngine can provide better optimisations and the reason is related to how the Next.js Image component works.

When we set deviceSizes: [1920, 1500, 1000, 500, 300] on our next.config.js what we’re saying is, we’ll have 5 buckets of width sizes we want to allow. By using the sizes property on the Image component we can make it so that given a certain width the browser will try to match it with the smallest possible srcset that covers that width. For horizontal or square images this works fine, as there’s a direct match between the max width and the bucket. But when the image is vertical, this no longer matches neatly. So if you open the lightbox with a vertical image and the viewport size is 1800px wide, but only 750px high, the image that will be retrieved will be the one matching the 1920 entry. This is actually a bigger image since it’s vertical, it’s width to match 1920 will make it quite bigger than it needs to be.

When using ImageEngine, in the same situation, with a fitMethod of box, ImageEngine’s engine will actually be smart enough to see that the image needs to be 750px high at most, and will resize it by that axis. So the result might be a 400px X 750px image, instead of 1920px X 3000px for instance.

If we had exact information for both width and height of the actual images, we could try to work around that limitation of the Next.js optimizer, by doing ourselves the calculations of what width would correspond to that maximum height - but since this isn’t normally available we usually can’t. Plus, that would need to change whenever we had changes on layout, styling, or on the source images.

With ImageEngine we don’t need to worry about that, since it’s always going to be the perfect fit for the dimensions we give it. Plus, in the case of the thumbnails, if we changed the size of the wrappers in CSS, it would still work automatically, whereas with the default Image component we might need to change our deviceSizes or do small adjustments. We could even make the thumbnail element be completely future proof if instead of providing only the width (as it was a square) we also provided the height of the wrapper, so that even if we changed the styling to a different ratio it would still work correctly. Since it’s a demo, we’re using only one side but there’s no reason to not use both as we already have the DOM element from which we can extract the height as well.

The last part is to define our main index.js file.

index.js entry point

Let’s replace the contents of pages/index.js:

import Head from "next/head";

import Thumbnail from "../components/thumbnail.js";
import Lightbox from "../components/lightbox.js";

import { useEffect, useState } from "react";

const IMAGES = [
    ["/ie-loader-images/h-lightbox-1.jpeg", "Harvested field with hay bales - Alentejo, Portugal © Micael Nussbaumer"],
    ["/ie-loader-images/h-lightbox-2.jpeg", "Family cycling and skating in abandoned Tempelhof Airport lane - Berlin, Germany © Micael Nussbaumer"],
    ["/ie-loader-images/h-lightbox-8.jpeg", "Group of hindu and muslim kids posing for a photo - New Delhi, India © Micael Nussbaumer"],
    ["/ie-loader-images/v-lightbox-4.jpeg", "Buddhist Monk Portrait with a statue of the buddhist mythological Seven Headed Naga  serving as background - Siem Reap, Cambodia © Micael Nussbaumer"],
    ["/ie-loader-images/h-lightbox-5.jpeg", "Traditional Nepalase Hindu Temple in one of the many lively city squares of Kathmandu, Nepal © Micael Nussbaumer"],
    ["/ie-loader-images/v-lightbox-6.jpeg", "Woman in traditional Nepalese clothing sitting in a valley in Pokhara, Nepal © Micael Nussbaumer"],
    ["/ie-loader-images/h-lightbox-7.jpeg", "Geometric pattern on a ceiling inside the Red Fort - New Delhi, India © Micael Nussbaumer"],
    ["/ie-loader-images/h-lightbox-3.jpeg", "Kids silhuetes in the sea near-shore close to sunset - Phu Quoc Island, Vietnam © Micael Nussbaumer"],
    ["/ie-loader-images/h-lightbox-9.jpeg", "Portrait of four man sitting during a pause in their badminton game - Hanoi, Vietnam © Micael Nussbaumer"]
];


export default function Home() {

    let [opened, set_opened] = useState(false);
    let [window_sizes, set_window_sizes] = useState(null);
    let resize_timer;

    const get_window_sizes = () => {
    let doc = document.documentElement;
    return {w: doc.clientWidth, h: doc.clientHeight};
    };

    const previous = (evt) => {
    evt.stopPropagation();
    if (opened > 0) { set_opened(opened - 1); }
    else { set_opened(IMAGES.length - 1); }
    };

    const next = (evt) => {
    evt.stopPropagation();
    if (opened < (IMAGES.length - 1)) { set_opened(opened + 1); }
    else { set_opened(0); }
    };

    const set_timing = () => {
    if (resize_timer) { clearTimeout(resize_timer); }
    resize_timer = setTimeout(
        () =>  {
set_window_sizes(get_window_sizes());
resize_timer = null;
    }, 2000
    );
    };

    useEffect(() => {
    window.addEventListener("resize", set_timing);
    set_window_sizes(get_window_sizes());

    return () => window.removeEventListener("resize", set_timing);
    }, []);

    return (
    <div className={`main-container ${opened || opened === 0 ? "no-overflow" : ""}`} >
      <Head>
            <title>ImageEngine Optimized Lightbox</title>
            <meta name="description" content="Next.js custom image loaders leveraging ImageEngine CDN for highly optimized images" />
            <link rel="icon" href="/favicon.ico" />
      </Head>
      <main>
        {IMAGES.map(([path, alt], index) => {
        return <Thumbnail key={`image-thumb-${index}`} src={path} alt={alt} onClick={(_evt) => set_opened(index)} window_sizes={window_sizes}/> ;
          })
          }
      </main>
      {opened || opened === 0 ? <Lightbox src={IMAGES[opened][0]} alt={IMAGES[opened][1]} onClick={(_evt) => set_opened(null)} window_sizes={window_sizes} previous={previous} next={next}/> : null}
        <footer>
          <a href="https://imageengine.io" target="_blank">ImageEngine</a>
          <a href="https://micaelnussbaumer.com" target="_blank">© Micael Nussbaumer 2021</a>

          <a href="https://nextjs.org" target="_blank">Next.js</a>
          <a href="https://vercel.com" target="_blank">Vercel</a>
        </footer>
    </div>
    );
};

In our index.js file we define an IMAGES array, that contains the images we’ll show. The contents are mostly straightforward. We use useState to set both the window size and if the lightbox is open. We define functions for getting the window size, for moving forward and backwards across the images, and to control any possible resizing of the viewport.

On first mount, we useEffect to set a listener on the window resize event and we set the original window sizes. Although we only have one page in our Next.js app, we also set a cleanup function on the useEffect by returning an anonymous function to remove the eventListener we added previously. This is just good form, because if you’re using React as a SPA with multiple pages and you don’t clean-up your useEffects, you might end with memory-leaks or multiple listeners being triggered (and most likely throwing exceptions since what they’ll be referring to won’t be around anymore).

Regarding the resize event we use a setTimeout and a control variable resize_timer. This is so that we don’t trigger multiple window_sizes changes - as that would trickle down to our components and trigger multiple fetches for different image sources as the dimensions changed, for instance if resizing the window on a desktop manually. At the same time, listening to resize events and updating the window_sizes takes care of refetching the correct size in case a user is seeing the website on a mobile and changes the display from vertical to horizontal for instance.

The JSX contents are pretty regular, we map our IMAGES array into Thumbnail’s components passing src, alt, window_sizes and a onClick handle.

Then when clicking one of the thumbnails, the opened state is updated to the index of that image in our array and we use that to both display our Lightbox component and to read the correct data for our image. On the Lightbox we pass the same props plus the previous and next functions.
Lastly we have a few footer items.
And that’s it.

Setting up ImageEngine

With this we can already host our page in vercel, by using their CLI from our project’s root folder executing:

vercel

And setting up the project settings, followed by:

vercel --prod

...to deploy.

You’ll now be able to see the website online. Take note of your production server url.
Once that is done it’s time to setup the ImageEngine distribution.

You can follow this video on how to easily signup for a trial

imageengine.io

After following those steps, take note of the delivery address, because we'll use it when building our project. We can do it through the Vercel dashboard for our project, or through the CLI when deploying:

vercel --build-env DISTRIBUTION=”https://our_ie_address.imgeng.in” --prod

With that, our custom loaders will be enabled, and if we visit our website and inspect the HTML source we should see that it’s now using our ImageEngine distribution for the images!

Conclusion

While Next.js’s Image component along with vercel’s automated infrastructure is really a great improvement over using non-optimized assets and will work well for many types of websites, there’s a few cases where ImageEngine is a better option overall.

Those include websites with very high traffic and image content. The more targeted, and almost pixel precise, sizes it allows one to deliver, can lower significantly the sizes of images being transferred. Websites with mixed orientation content in any significant quantity also benefit widely from it.

Another situation is, if you’re using Next.js but not deploying on vercel then you’ll probably want to use it too - the Image component works the best possible in vercel as its infrastructure is prepared to handle it specifically (they’re the authors of Next.js) but if you’re deploying on your own server then it’s a different matter and having it stored, prepared and cached over a very fast external CDN will help you achieve the best performance for your website.

Lastly, as we saw, by taking some measures when writing the logic for the loader, we can make it so that our components are automatically able to handle changes in layout, styling or structure without any tweaking or re-doing of the app settings or components logic while keeping the exact needed sizes for any dimensions we decide to use. This is important as it removes one friction point when contemplating a re-styling of the whole or parts of the app.

ImageEngine with Nextjs

Micael Nussbaumer — Thu, 10 Jun 2021 07:11:32 +0000

Using ImageEngine in a React.js project

In this blog post we'll cover how to use ImageEngine in order to significantly improve your image assets, increasing your site responsiveness and improving your loading times - easy mode.

You can check the final website we'll be building (requires JS, no cookies)

There's also a repository containing all the code

Why use ImageEngine (IE) to optimize your images?

While this post is particular to Next.js when exporting a static website - a case where Next.js inbuilt Image Loader, and its optimisations can't be used - ImageEngine only requires a query string on your image urls and can be used in any stack, even outside websites.

Next.js also has the capability of specifying custom image loaders for their inbuilt <Image/> component, and we can use those to reach an external CDN from which to load our images with our own logic - read how to use the custom loader with ImageEngine here - but here we'll use a pre-built component that handles most of the work for us.

If using Next.js with a Nodejs server and their inbuilt image optimisations, both the image optimisations and the serving of the assets will be done from the server side, while using ImageEngine none of it has to pass through your server with exception of the first serving - that coupled with the full range of customisable properties and highly optimised algorithms ImageEngine offers, makes it worth considering.

The only exception is in case you're deploying directly in vercel.com, their whole infra-structure supports Next.js inbuilt Image Optimizations (with automatic edge caching and good performance overall) which is good enough for most cases. Although even then, ImageEngine can provide significant bandwidth and improved response times savings over it, specially if you're serving a non trivial amount of images.

I'm going to walk you through implementing the website and deploying it on S3, using ImageEngine to optimise the delivery of its image content.

How can we use ImageEngine to improve our Nextjs app loading speed?

First let's go through the website so that you can make sense of what's happening and what we'll be building.

The website is designed to show the difference in file sizes, and consequently in loading times, between images distributed in their original versions and images distributed through ImageEngine's smart CDN. Notice that the images that comprise the repository are sourced from the same files, but by using ImageEngine we can do some neat things with them.

The thumbnails on the left are from images on the ImageEngine CDN, optimised through simple URL directives. This means that specifications such as width, height, compression, etc, are simply passed through query parameters on the URL the image tags use, without requiring any additional work.

The thumbnails on the right side are from the original resources.

If you click any thumbnail or file title from the sidebar, you'll see that for the ImageEngine side, the total size now increased to account for that, while the normal assets didn't - because they were already loaded.

Of note is that many times, when loading a website, there's no actual need of loading full sized images, so optimising thumbnails is immediately an advantage in the majority of cases - because it allows a much faster initial load it impacts directly Google’s Core Web Vitals scoring.

In the same way the thumbnails were optimised through directives, we can also optimise those being displayed on the "preview" area. From the sidebar select, Pick a type -> jpg, Pick a fit type -> cropbox and enter 600 as the width and 400 as the height directive. There's others, like compression, sharpness, and so on that aren't on the sidebar, but available to use.

You should see some significant improvements on the size without any perceivable quality loss. At 600px X 400px, it’s enough to cover retina when displaying images at 300px X 200px, which is roughly the size of our previews.

Click around on the thumbnails to see the time images take to be displayed between changes. Even though the regular images are already downloaded and cached by the browser you can still see significant differences (ImageEngine images on first load, or when changing directives have to be fully processed and transferred, and usually still render faster than the unoptimized ones already downloaded and cached!).

Let's start actually coding this.

Setting up the Next.js project

To follow through you'll need to have nodejs and npm installed. The versions used for this tutorial are npm 6.14.4 and node v10.19.0.

On the folder where you'll be creating your project run:

npx create-next-app

Now let's install the image engine package:

npm i @imageengine/react

We should be able to start our app by doing

npm run dev

And visiting

http://localhost:3000

should show us the default page. Let's remove the default templating index.js page has:

import Head from "next/head";

export default function Home() {
  return (
      <div className="main-container">
        <Head>
          <title>ImageEngine Optimized Assets</title>
        </Head>
      </div>
  );
};

And change /pages/_app.js to the following

import Head from "next/head";
import { ImageEngineProvider } from "@imageengine/react";
import "../styles/globals.css";

const delivery_address = process.env.DELIVERY_ADDRESS;

function LeanImages({ Component, pageProps }) {
    return (
    <ImageEngineProvider deliveryAddress={delivery_address}>
      <Head>
        <link rel="icon" type="image/png" href="/favicon.png"/>
        <meta name="viewport" content="initial-scale=1.0, width=device-width" />
        <meta name="description" content="Learn how to use ImageEngine in Nextjs to serve highly optimised image assets from your CDN to your users." />
        <meta property="og:title" content="ImageEngine with NextJS" />
        <meta property="og:description" content="Learn how to use ImageEngine in Nextjs to serve highly optimised image assets from your CDN to your users." />
      </Head>
      <Component {...pageProps} />
    </ImageEngineProvider>
    )
};

export default LeanImages;

If you have your dev server running you should see an error. Let's stop the server and create a next.config.js on the root of our project and inside it:

module.exports = {
    env: {
    DELIVERY_ADDRESS: process.env.DELIVERY_ADDRESS || "http://localhost:3000",
    NON_OPTIMIZED_ADDRESS: process.env.NON_OPTIMIZED_ADDRESS || ""
    },
};

Let's remove the things we don't need, delete pages/api folder, styles/Home.module.css and public/vercel.svg::

rm -rf pages/api
rm styles/Home.module.css
rm public/vercel.svg

We should now have the following structure:

pages /
  _app.js
  index.js
styles /
  global.css
public /
  favicon.ico

We're ready to start. Our website will be a single page, divided into 2 sections, one left sidebar and a main area, the latter further divided into 3 parts - the title and image information, the preview area, and the thumbnails section. From the sidebar we'll be able to choose certain parameters related to the image display, and we'll have a small list with actionable items.

When clicking one of those items it will display them in the main area, in two versions, one from our image engine distribution, and the other from our original assets.

Locally it will source both from the same place in our assets, but once online correctly from their origins.

We will also display the total transferred file size for each image and a cumulative for all images loaded, between optimized and non-optimized.

To share state between our components we’ll use useReducer so let's write its initial state, on our pages/index.js:

const initial_settings_state = {
    base_path: "/images",
    file_types: ["jpg", "jp2", "webp"],
    file_type: null,
    fit_types: ["stretch", "box", "letterbox", "cropbox"],
    fit_type: null,
    width: null,
    height: null,
    images: [
        "senja_norway.jpg",
        "buddha_shakyamuni.jpg",
        "vasnetsov_samolet.jpg",
        "great_wave_off_kanagawa.jpg"
    ],
    copyright: {
        "great_wave_off_kanagawa.jpg": ["Great Wave off Kanagawa", "By After Katsushika Hokusai - Restored version, Public Domain", ["https://commons.wikimedia.org/w/index.php?curid=5576388"]],
        "buddha_shakyamuni.jpg": ["Buddha Shakyamuni", "By Unknown author - Public Domain", ["https://www.metmuseum.org/collection/the-collection-online/search/75274", "https://commons.wikimedia.org/w/index.php?curid=39112914"]],
        "vasnetsov_samolet.jpg": ["The Flying Carpet", "By Viktor Mikhailovich Vasnetsov - belygorod.ru, Public Domain", ["https://commons.wikimedia.org/w/index.php?curid=1374733"]],
        "senja_norway.jpg": ["Island of Senja, Troms, Norway (August, 2014)", "By Ximonic (Simo Räsänen) - Own work, CC BY-SA 4.0", ["https://commons.wikimedia.org/w/index.php?curid=34693021"]]

    },
    show_copyright: false,
    loaded_images_ie: {},
    loaded_images_regular: {},
    selected_image: null,
    selected_image_url: null,
    force_load: false
};

Our reducer function will be:

function settings_reducer(state, action) {
    switch (action.type) {
    case "file_type":
    case "fit_type":
    case "width":
    case "height":
    case "force_load":
    case "show_copyright":
        state[action.type] = action.value;
        return {...state};

    case "selected_image":
        state.selected_image = action.value;
        state.selected_image_url = action.value ? `${state.base_path}/${action.value}` : null;
        return {...state};

    case "add_image_size":
        let state_key = build_type_key(action.image_type);

        state[state_key][action.image] = action.size;
        state[state_key] = {...state[state_key]};

        return {...state};
    default:
        throw new Error();
  }
};

You might see an error because build_type_key(action.image_type) isn’t defined yet, we’ll add it in a while.

Sidebar component

Now our sidebar component. Create a folder components on the root folder and inside it a sidebar.js file:

import React from "react";

export default function Sidebar({ state, dispatch }) {
    return (
    <div className="sidebar-wrapper">
      <div className="sidebar">
        <h2>Directives Optimizations</h2>
        <select id="file_type" name="file_type" value={state.file_type || ""} onChange={(event) => dispatch({type: "file_type", value: event.target.value})}>
          <option value="" >Pick a type</option>
          {state.file_types.map((type, index) => {
                return  <option value={type} key={`file_type-${index}`}>{type}</option>
          })}
        </select>
        <br/>
        <br/>
        <select id="fit_type" name="fit_type" value={state.fit_type || ""} onChange={(event) => dispatch({type: "fit_type", value: event.target.value})}>
          <option value="">Pick a fit type</option>
          {state.fit_types.map((type, index) => {
                return  <option value={type} key={`file_type-${index}`}>{type}</option>
          })}
        </select>
        <br/>
        <br/>
        <input type="text" name="width" id="width" placeholder="Width directive in px, e.g: 25" value={state.width || ""} onChange={(event) => dispatch({type: "width", value: event.target.value})}/>
        <br/>
        <br/>
        <input type="text" name="height" id="height" placeholder="Height directive in px, e.g: 50" value={state.height || ""} onChange={(event) => dispatch({type: "height", value: event.target.value})}/>
            <ul>
              {state.images.map((image, index) => {
                return <li key={`image-${index}`} className={`${image === state.selected_image ? "selected" : ""}`} onClick={() => dispatch({type: "selected_image", value: image})}>{image}</li>
               })}
            </ul>
      </div>
    </div>
    );
};

We haveselects, inputs and a list of elements. We set handlers to trigger on change for the inputs and dispatch the changes to our reducer, and click handlers on the list elements to dispatch the selection of an image.

Regarding the styling, to save some space I’ll link directly to the final css file. Either copy the whole file or otherwise add it piece by piece as I’ll mention the initial line (IL) and last line (LL) for each component. In this case IL0 - LL128.

The central part will be made of two sections, the top part - info and images previews - and the bottom thumbnail section.
Create three files on the components folder, copyright.js, thumbnails.js and image_preview.js.

Copyright component

On copyright.js:

export default function Copyright({ state, dispatch }) {
    if (!state.selected_image) {
    return <div className="main-area-image-info"></div>;
    } else {
        let copyright = state.copyright[state.selected_image],
            title = copyright[0],
            description = copyright[1],
            links = copyright[2];

        return (
            <div className="main-area-image-info">
              <h4>{title}<button className="collapse-btn" type="button" onClick={() => dispatch({type: "show_copyright", value: !state.show_copyright})}>{state.show_copyright ? "-" : "+"}</button></h4>
              {state.show_copyright ? <div>
                        <p>{description}</p>
                {links.map((link, index) => <a href={link} target="_blank" key={`link-${state.selected_image}-${index}`}>{link}<br/></a>)}
               </div> :
               null}
            </div>
        )
    }
};

styles/globals.css add from IL131 - LL166.

Thumbnails component

thumbnails.js

import { Image } from "@imageengine/react";
import { get_size, handle_image, NON_OPTIMIZED_ADDRESS } from "../js/utilities";

export default function Thumbnails({ state, dispatch }) {

    return (
    <div className="main-area-thumbnails">
      <div className="thumbnails-container optimized">
        {state.force_load && state.images.map((image, index) => {
                 let image_path = `${state.base_path}/${image}`,
                     thumb_id = `thumb_${image_path}`;
                 return (
                    <div className="thumbnail-image" key={`thumbnail-optimized-${index}`} onClick={() => dispatch({type: "selected_image", value: image})}>
                      <div className="thumbnail-size">
                        {get_size(state, "ie", thumb_id)}
                      </div>
                      <Image
                        src={image_path}
                        directives={{
                          outputFormat: "jpg",
                          height: 90,
                          width: 90,
                          fitMethod: "cropbox",
                          compression: 80
                        }}
                        onLoad={(event) => handle_image("ie", event.target, thumb_id, dispatch)}
                      />
            </div>
        );
        })}
      </div>
      <div className="thumbnails-container regular">
            {state.force_load && state.images.map((image, index) => {
              let image_path = `${NON_OPTIMIZED_ADDRESS}${state.base_path}/${image}`;
              return (
                  <div className="thumbnail-image" key={`thumbnail-regular-${index}`} onClick={() => dispatch({type: "selected_image", value: image})}>
                     <div className="thumbnail-size">
                        {get_size(state, "regular", image_path)}
                     </div>
                     <img src={image_path} onLoad={(event) => handle_image("regular", event.target, image_path, dispatch)} />
                  </div>
                );
        })}
      </div>
    </div>
    )
};

Here we'll use the Image component provided by the @imageengine/react package, and import a bunch of helper functions not yet defined.
We have two containers, on each of these containers we map the images we have to image elements - on the optimized one we build the full path, and create a unique id for this thumbnail. We display the size of that image with get_size and use ImageEngine’s Image component to build a custom image source.

On the regular one we do the same mapping but use a bare <img> tag. In these we don't create a thumb_id because the images will be these, but on the optimized ones in order to account correctly for the total size of files being loaded we need to differentiate between the thumbnails retrieved from ImageEngine and the previews.
We’re using state.force_load to control the rendering, this is a walk-around for a glitch that happens on first load where sometimes the thumbnails won’t trigger the loaded handler.

Helper functions

For the helper functions, create a directory named js, and inside it a file named utilities.js:

export const NON_OPTIMIZED_ADDRESS = process.env.NON_OPTIMIZED_ADDRESS;

export async function handle_image(image_type, img, selected_image_url, dispatch) {

    let head = await fetch(img.src, {method: "head"}),
        size = head.headers.get("content-length");

    if (size) {
      dispatch({type: "add_image_size", image_type: image_type, image: selected_image_url, size: parseInt(size)});
    }
}

export function get_size(state, type, url) {
    let size = state[build_type_key(type)][url];
    return  size || size === 0 ? display_size(size) : "No info";
};

export function calc_total_sizes(state, type) {
    let total_size = Object.values(state[build_type_key(type)]).reduce((acc, size) => acc + size, 0);
    return display_size(total_size);
};

export function build_type_key(type) {
    return `loaded_images_${type}`;
};

function display_size(size) {
    if (size > 1048576) {
      return (Math.round((size / 1048576 + Number.EPSILON) * 100) / 100) + "MB";
    } else if (size > 1024) {
      return (Math.round((size / 1024 + Number.EPSILON) * 100) / 100) + "Kb";
    } else {
      return size + "b";
    }
};

Let's go through each element here:

NON_OPTIMIZED_ADDRESS just proxies the value we will set as an ENV variable, to be picked from our next.config.js we created earlier.
handle_image is the function we're using when an image finishes loading. It takes 4 arguments:
- image_type - if it's ie or regular so we can distinguish between which kind of image was loaded
- img - which is the html img element itself
- selected_image_url - which is the key under which we to correlate with this image
- dispatch - which is our reducer's dispatch function.

Now, there isn't a straightforward way in JS to get an image size that works every time for local, cross-origin, and cached files, so our solution is to issue an head request for the same resource. We use the actual img element to retrieve the exact src attribute. The response to this request includes the size of the resource, under the content-length header, but without actually transferring its content.

get_size is an helper that retrieves the correct size, for a particular url of a given type from our state.
calc_total_sizes sums up the sizes of all resources we have retrieved for a given type.
build_type_key is a simple helper to return the correct state key for a given type.
display_size is a helper that formats an integer value representing bytes into either mb, kb or b

With this in place, let's add the css for our thumbnails section, IL169 - LL236

Image Preview component

Finally, on image_preview.js:

import { get_size, calc_total_sizes } from "../js/utilities.js";

export default function ImagePreview({ state, type, prefix,  children }) {
    let prefixed = `${prefix}${state.selected_image_url}`;

    return (
    <div className="image-container-wrapper">
      <div className="image-info-container">
        <div className="image-info-size">{state.selected_image_url ? get_size(state, type, prefixed) : "No info"}</div>
        <div className="image-info-size-total">Total: {calc_total_sizes(state, type)}</div>
      </div>
      <div className="image-container">
            {state.selected_image ?
              children :
              <p>Select an Image to preview</p>
        }
      </div>
    </div>      
    );
};

This one is a simple wrapper. If there's no selected image it shows Select an image to preview, and in case the size of the file hasn't been retrieved yet (it's an async function) it shows No info, if there's a selected image we use the children prop to display it.

Let's add the CSS styling for this part IL239 - LL336

Final index.js

Lastly, we're ready to finish our index.js file (omitting the reducer and initial state) :

import Head from "next/head";
import Sidebar from "../components/sidebar.js";
import ImagePreview from "../components/image_preview.js";
import Thumbnails from "../components/thumbnails.js";
import Copyright from "../components/copyright.js";

import { Image } from "@imageengine/react";

import { useReducer, useEffect } from "react";

import {
    build_type_key,
    handle_image,
    NON_OPTIMIZED_ADDRESS
} from "../js/utilities.js";


export default function Home() {
    const [state, dispatch] = useReducer(settings_reducer, initial_settings_state);

    useEffect(() => {
    setTimeout(() => dispatch({type: "force_load", value: true}), 25)
    }, []);

    return (
    <div className="main-container">
      <Head>
        <title>ImageEngine Optimized Assets</title>
      </Head>
      <Sidebar state={state} dispatch={dispatch} />
      <div className="main-area">
        <h2 className="page-title">ImageEngine (left) vs Unoptimized (right)</h2>
        <Copyright state={state} dispatch={dispatch} />
        <div className="main-area-images">
              <ImagePreview state={state} type="ie" prefix={""}>
                <Image
                  src={state.selected_image_url}
                  directives={{
                    outputFormat: state.file_type,
                    height: state.height,
                    width: state.width,
                    fitMethod: state.fit_type
                  }}
                  onLoad={(event) => handle_image("ie", event.target, state.selected_image_url, dispatch)}
                />
              </ImagePreview>
              <ImagePreview state={state} type="regular" prefix={NON_OPTIMIZED_ADDRESS}>
                <img src={NON_OPTIMIZED_ADDRESS + state.selected_image_url} onLoad={(event) => handle_image("regular", event.target, (NON_OPTIMIZED_ADDRESS + state.selected_image_url), dispatch)}/>
              </ImagePreview>
        </div>
        <Thumbnails state={state} dispatch={dispatch} />
      </div>
    </div>
    );
};

Besides the useReducer we also imported useState and used it here:

  useEffect(() => {
        setTimeout(() => dispatch({type: "force_load", value: true}), 25)
    }, []);

To show the thumbnails so their onload handle triggers properly.

On the first <ImagePreview> component we pass as a child an ImageEngine Image component, and we set both the src prop to our selected_image_url and the directives prop to an object containing the customisable properties we have in our sidebar. We could add further directives here. Because we wrapped our _app.js content with an <ImageEngineProvider> it will automatically be used by the <Image> component. In a normal situation we might want to debounce the changes to our reducer’s state so it doesn’t immediately request a new image.

In the second <ImagePreview> instead we pass a simple html <img> tag.

If you try to see the website it should be working, but the images won't be loading as we haven't them yet - you can download them from images and place them in public/images.
We aren't using yet an actual ImageEngine distribution, so we won't have any differences in sizes, except that the ImageEngine side will count each image twice as you pick them, once for the thumbnails, another for the previews - as we discussed earlier.

If you want to add responsiveness to the layout, there’s a few more lines of css to add, IL338 - LL372.

Bucket, ImageEngine distribution, static build and deploying

Now that we have our app ready, let's set up an AWS S3 bucket to host it. In this case we will host it bare on S3, as a static website, making the whole bucket public, but outside of a tutorial, this bucket shouldn't be public at all and instead use cloudfront in front (get it?). You can do it from the aws console interface, or using the aws cli. With the cli, change the bucket-name and use-west-2 region to something you want on the following commands:

aws s3 mb s3://bucket-name --region us-west-2
aws s3 website s3://bucket-name/ --index-document index.html --error-document 404.html

Because we created the bucket in us-west-2 region and set it as a static website, it will be hosted on the following url http://bucket-name.s3-website-us-west-2.amazonaws.com

If using AWS Console just create a new bucket and set it as a static website after it's created (from the Properties tab),

Now that we have an host we can refer to, let's create an ImageEngine distribution for it:

imageengine.io

changing the url address used in the video to that of your bucket. Then from your account panel, select Edit Engine and on the CORS Support section change the HTTP methods allowed value from GET, to GET,HEAD.

And that’s all for ImageEngine. Right now we don't have any files on our bucket so we don't have a website yet. Next.js allows you to build a static website that we can use in S3. Since we have the url for our bucket and for our ImageEngine distribution we can build our website using those values.

Let's add a specific command to our package.json that does it for us, under the scripts key, add "static": "next build && next export". Save the file.
From the command line we can now run (substituting the values for the ones you have):

DELIVERY_ADDRESS="https://your-image-engine.cdn.imgeng.in" NON_OPTIMIZED_ADDRESS="http://bucket-name.s3-website-us-west-2.amazonaws.com" npm run static

This will create all necessary files for a static website in the out directory of your app dir. The last step is to upload those files into our bucket. We can do that manually through the console interface, or using the cli:

aws s3 cp out s3://bucket-name/ --recursive --acl public-read

Notice we need to set the Access Control List to public-read. If doing the upload through the interface we would need to set it on the upload panel, before uploading them.

And that's it, maybe give it a minute and you should be able to visit http://bucket-name.s3-website-us-west-2.amazonaws.com and see it working!

Now that you have the ImageEngine distribution available you can also run it locally with DELIVERY_ADDRESS="https://your-image-engine.cdn.imgeng.in" npm run dev.

If you omit the DELIVERY_ADDRESS then it will serve the local assets for both parts as it did initially. This is the same if you're integrating ImageEngine into an existing website, you can do all the code changes to use the Image component, while still relying on the normal image urls, and once ready, build a version with the correct DELIVERY_ADDRESS (or if running on a nodejs server restart it with the proper env set). If you’re not using Nextjs, or need some more control on your image components, you can also build the query string for the directives and use it directly in any img element’s src, or in http requests.

Conclusion

It's easy to add ImageEngine to a website or when starting a new project, as it involves just attaching a query string parameter to your image src's. In the same way it's also easy to leverage new HTML image attributes, such as srcset/sizes to define particular viewport dimensions with specific optimisations, as again, it just requires a query string.

If you're serving a significant amount of images and you don't have an image optimisation pipeline in place, need finer grain control than things like Next.js image optimisations, aren't running a Nodejs web server, or don't want to worry about that part of your infrastructure, then ImageEngine makes a lot of sense - its CDN is really fast as is their on-the-fly transformation engine - on top of that given their work on adjacent areas - device display identification - any improvements to their algorithms are automatically applied to your existing sources, and new options when available easily integrated.

DEV Community: Micael Nussbaumer

Criar um Enum Ecto.Type

Creating an Ecto Enum Type

Getting Started with ImageEngine and Angular

Intro

Summary:

Setting up the Angular Project

Defining our Layout and Content

Deploy to Github Pages

Create ImageEngine Engine

Conclusion

ImageEngine custom loaders for Next.js

Summary:

Intro

Creating a Next.js App

Thumbnail Component

Lightbox Component

index.js entry point

Setting up ImageEngine

Conclusion

ImageEngine with Nextjs

Using ImageEngine in a React.js project

Why use ImageEngine (IE) to optimize your images?

How can we use ImageEngine to improve our Nextjs app loading speed?

Setting up the Next.js project

Sidebar component

Copyright component

Thumbnails component

Helper functions

Image Preview component

Final index.js

Bucket, ImageEngine distribution, static build and deploying

Conclusion

Creating a `Next.js` App