DEV Community: Felipe Araujo

It's not just about coding

Felipe Araujo — Sat, 25 Mar 2023 15:55:24 +0000

It's also about being pragmatic, understand tradeoffs, discuss ideas and make the better decisions based on business needs. Questions like "Should I use a NoSQL database?" or "Should I run Kubernetes?" need to be made in based on the business strategy. Of course, you also need to consider how many users this application will need to support in the next 12 months. Evolutionary architecture is very different from ingenious design.

But how do you get this information? You have to do the boring part of the job talking with stakeholders and domain experts. This career is also about extracting value from conversations. We have an entire knowledge domain based on how to deal with business complexity called Domain-Driven Design, and one of its foundations is to make a common language with your domain experts.

Being adaptive is also essential. It's not about following the Scrum framework by the book and saying that you work with Agile. It's about adapting your work based on the most important thing. For example, how can you focus on developing the next feature if the rest of your team is working on a critical incident? Especially if you work in the Spotify model, where the squad should be responsible for maintaining their own software running in production.

Ten years ago, containers were something new, but now they are part of any developer's daily work as DevOps culture has become more common. As developers, we can use and understand infrastructure tooling to make our daily work easier.

Bug management is also an important part of the job. How can we better evaluate the impact of a bug instead of just relying on the dev team? Production bugs don't need a database intervention, but they do need a deep investigation to act on the root cause and avoid them from happening twice. The best way to avoid this is to focus on the beginning of the development process with good unit test scenarios that represent the business rules. If possible, have a great integration test suite covering meaningful scenarios. You don't need integration tests for everything.

A meaningful part of software development is explaining ideas via text or tech diagrams. It's not necessary to follow UML by the book, but you should follow some conventions. You will also need to write tech decisions with ADRs or RFCs, opening the discussion to a larger group.

What about coding skills? They are fundamental to how you make all of the other parts and make a delivery. They are essential, but with a broader view, your code will focus on the right thing.

Interest links

O Poder do Módulo Task do Elixir -Task.Supervisor

Felipe Araujo — Fri, 17 Sep 2021 20:09:38 +0000

Caso prefira em inglês você pode encontrar aqui.

A publicação anterior foi uma intro sobre o módulo task e como ele funciona de uma forma geral.

Agora vamos tornar as coisas um pouco mais complexas em nossa jornada com Task.Supervisor e ver mais alguns conceitos do elixir.

Antes de avançar, vamos dar uma olhada da definição desses conceitos:

Supervisor

Um supervisor é um processo que supervisiona outros processos, que nós chamamos de child processes. Supervisors são usados para construir a estrutura de processos hierárquicos chamados supervision tree. Supervision trees provem tolerância a falha e encapsulam como nossas aplicação iniciam ou desligam.

Processes

No Elixir, todo código é executado dentro de processos. Os processos são isolados uns dos outros, são executados simultaneamente entre si e se comunicam por meio da passagem de mensagens.

Os processos do Elixir não devem ser confundidos com os processos do sistema operacional. Os processos no Elixir são extremamente leves em termos de memória e CPU (mesmo em comparação com os threads usados em muitas outras linguagens de programação). Por causa disso, não é incomum ter dezenas ou mesmo centenas de milhares de processos em execução simultaneamente.

Vamos usar Task.Supervisor para criar processos responsáveis por gerenciar o child process relacionado.

Criando o Playground

O primeiro passo é criar uma nova aplicação elixir com árvore de supervisão, mais informações sobre mix pode ser encontrada na documentação oficial.

mix new newsample --sup

Vamos ver o que foi criado:

  lib
  lib/newsample
  lib/newsample.ex
  lib/newsample/application.ex
test
  test/newsample_test.exs
  test/test_helper.exs
README.md
mix.exs

Nesse momento vamos trabalhar somente com os arquivos lib/newsample.ex e lib/newsample/application.ex.

O próximo passo será criar a mesma função do post anterior dentro do arquivo lib/newsample.ex com algumas pequenas mudanças para facilitar os testes.

Para forçar um comportamento explosivo usar um pattern matching dentro da função say_hello com o valor "alpha":

  def say_hello("alpha" = to), do: raise("Error to say hello to #{to}")

  def say_hello(to) do
    IO.puts("Hello #{to}")
    :ok
  end

  def process() do
    items = ["alpha", "beta", "gama"]

    Enum.map(items, fn item ->
      Task.async(fn ->
        say_hello(item)
      end)
    end)
    |> Enum.map(&Task.await/1)
  end

Hora de usar o elixir's interactive shell.

iex -S mix

E então executar:

iex(1)> self
#PID<0.145.0>
iex(2)> Newsample.process
Hello beta
Hello gama

22:01:54.276 [error] Task #PID<0.148.0> started from #PID<0.145.0> terminating
** (RuntimeError) Error to say hello to alpha
    (newsample 0.1.0) lib/newsample.ex:15: anonymous fn/2 in Newsample.process/1
    (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: #Function<1.13017213/0 in Newsample.process/1>
    Args: []
** (EXIT from #PID<0.145.0>) shell process exited with reason: an exception was raised:
    ** (RuntimeError) Error to say hello to alpha
        (newsample 0.1.0) lib/newsample.ex:15: anonymous fn/2 in Newsample.process/1
        (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
        (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
        (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.11.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> self
#PID<0.151.0>

Como podemos ver a exception foi propagada até a sessão do iex podemos checar comparando o PID antes #PID <0.145.0> e após #PID <0.151.0> executar o código.

Então, como podemos evitar esse comportamento explosivo?

Finalmente chegamos no momento de usar supervised task, hora de abrir o arquivo lib/newsample/application.ex.

  def start(_type, _args) do
    children = []

    opts = [strategy: :one_for_one, name: Newsample.Supervisor]
    Supervisor.start_link(children, opts)
  end

Hora de criar o Task.Supervisor como um child process

  children = [
    {Task.Supervisor, name: Newsample.TaskSupervisor}
  ]

E agora com essa supervised task a function process e inicie como async_no_link. Isso significa que não há vínculo entre esse child process e o parent.

  def process() do
    items = ["alpha", "beta", "gama"]

    Enum.map(items, fn item ->
      Task.Supervisor.async_nolink(Newsample.TaskSupervisor, fn ->
        say_hello(item)
      end)
    end)
    |> Enum.map(&Task.await/1)
  end

Se executarmos novamente tudo vai funcionar sem essa propagação de erro.

iex(1)> self
#PID<0.158.0>
iex(2)> Newsample.process
Hello beta
Hello gama

21:04:48.200 [error] Task #PID<0.161.0> started from #PID<0.158.0> terminating
** (RuntimeError) Error to say hello to alpha
    (newsample 0.1.0) lib/newsample.ex:6: Newsample.say_hello/1
    (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: #Function<1.39266525/0 in Newsample.process/0>
    Args: []
** (exit) exited in: Task.await(%Task{owner: #PID<0.158.0>, pid: #PID<0.161.0>, ref: #Reference<0.1303494724.713818114.156158>}, 5000)
    ** (EXIT) an exception was raised:
        ** (RuntimeError) Error to say hello to alpha
            (newsample 0.1.0) lib/newsample.ex:6: Newsample.say_hello/1
            (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
            (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
            (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
    (elixir 1.11.3) lib/task.ex:639: Task.await/2
    (elixir 1.11.3) lib/enum.ex:1411: Enum."-map/2-lists^map/1-0-"/2
iex(2)> self
#PID<0.158.0>

Como podemos ver, o PID é o mesmo antes e depois da execução.

Processes e Supervision trees são os componentes importantes do elixir para construir sistemas resiliente e tolerante a falhas.

O Poder do Módulo Task do Elixir - O Início

Felipe Araujo — Fri, 14 May 2021 20:46:20 +0000

Caso prefira em inglês você pode encontrar aqui.

Recentemente eu estava estudando mais profundamente como o módulo Task funciona e para consolidar meus estudos decidi escrever esse post.

Antes de começar, vamos ver a documentação do módulo task. O melhor lugar para fazer isso é a documentação oficial do elixir. Aqui nós temos:

Conveniences for spawning and awaiting tasks.

Tasks are processes meant to execute one particular action throughout their lifetime, often with little or no communication with other processes. The most common use case for tasks is to convert sequential code into concurrent code by computing a value asynchronously

Ótima definição mas me mostre o código!!!!

Iniciando uma operação async

Nós temos duas maneiras básicas para executar uma operação async usando Task. É possível usar Task.start e Task.async. Vamos ver como isso funciona na prática.

Task.start

Task.start(fn -> IO.inspect("Hello") end)

{:ok, #PID<0.114.0>}

Task.async

Task.async(fn -> IO.inspect("Hello") end)

%Task{
  owner: #PID<0.110.0>,
  pid: #PID<0.118.0>,
  ref: #Reference<0.626386777.2138832899.106529>
}

É possível ver que Task.start retorna uma tupla com :ok e PID, enquanto Task.async retorna uma struct Task. Ambas funções funcionam da mesma forma.

Geralmente precisamos esperar o resultado de algumas funções async para assim, poder executar a próxima ação. Vamos lá!

Esperando resultados

O exemplo anterior foi meio básico, vamos primeiro adicionar alguma um delay na execução para torna-lo mais complexo.

  Task.async(fn ->
    :timer.sleep(5000)
    IO.inspect("Hello")
    :ok
  end)

Como já visto o resultado será uma %Task{}. Para esperar a response temos duas opções Task.await e Task.yield. Vamos ver as diferenças:

Task.await

O tempo de timeout padrão é 5 segundos;
Dado um timeout, ele lança uma exceção;
Após o tempo limite ser atingido, a task é interrompida;
Você pode definir um tempo limite personalizado ou usar o atom:infinity.

Task.await(task)
Task.await(task, :infinity)

Exemplo de timeout

> task = Task.async(fn -> IO.inspect("Hello") ; :timer.sleep(10000); :ok end)

> Task.await(task)
"Hello"
** (exit) exited in: Task.await(%Task{owner: #PID<0.110.0>, pid: #PID<0.124.0>, ref: #Reference<0.3761442499.262406148.76432>}, 5000)
    ** (EXIT) time out
    (elixir 1.11.3) lib/task.ex:643: Task.await/2

Como podemos ver, um tempo limite é um pouco explosivo ao usar Task.await. Uma maneira de lidar melhor com isso é são Supervised Tasks.

Task.yield

O tempo de timeout padrão é 5 segundos;
Dado um timeout, ele retorna nil;
Usar o atom :infinity não é permitido como em Task.await;
Depois de um timeout atingido, mantém a task em execução;
É possível terminar uma task em execução usando Task.shutdown (task, shutdown \\ 5000).

> task = Task.async(fn -> IO.inspect("Hello") ; :timer.sleep(10000); :ok end)
> Task.yield(task)
nil
# Let's check again
> Task.yield(task)
{:ok, :ok}

Dado um resultado :timeout, a resposta será nil. Depois disso, podemos executar Task.yield novamente. Para evitar task de longa execução sem nenhum resultado, você pode usar Task.shutdown (task, shutdown \\ 5000).

Um exemplo mais completo

Temos uma lista de itens e é necessário executar algum processamento em todos itens da lista.

    items = ["alpha", "beta", "gama"]

    Enum.map(items, fn item ->
      Task.async(fn ->
        :timer.sleep(4000)
        IO.inspect("Hello #{item}")
        :ok
      end)
    end)
    |> Enum.map(&Task.await/1)
    |> function_to_handle_results()

Com essa abordagem, ainda temos a exceção quando acontece o timeout. No entanto, é possível lidar com isso de uma maneira melhor com Supervised Tasks, mas esse assunto será abordado na próxima publicação.

Conteúdo adicional

The Power of Elixir Task Module - Into Task.Supervisor

Felipe Araujo — Fri, 07 May 2021 17:37:32 +0000

The previous post was an intro about the task module and how it works in a more general way.

Now it's time to make things a little bit complex with Task.Supervisor and see some elixir concepts.

Before going ahead let's see some elixir base concepts:

Supervisor

A supervisor is a process that supervises other processes, which we refer to as child processes. Supervisors are used to building a hierarchical process structure called a supervision tree. Supervision trees provide fault-tolerance and encapsulate how our applications start and shutdown.

Processes

In Elixir, all code runs inside processes. Processes are isolated from each other, run concurrent to one another, and communicate via message passing.

Elixir’s processes should not be confused with operating system processes. Processes in Elixir are extremely lightweight in terms of memory and CPU (even compared to threads as used in many other programming languages). Because of this, it is not uncommon to have tens or even hundreds of thousands of processes running simultaneously.

We will use Task.Supervisor to create processes responsible for manage the related child.

Creating the playground

The first step is create a new elixir application with a supervision tree, more information about mix possibilities can be found in the official documentation.

mix new newsample --sup

Time to see what was created:

  lib
  lib/newsample
  lib/newsample.ex
  lib/newsample/application.ex
test
  test/newsample_test.exs
  test/test_helper.exs
README.md
mix.exs

For now we'll just work with lib/newsample.ex and lib/newsample/application.ex.

The second step is to create the same function from the previous post inside lib/newsample.ex with minor changes to execute our tests easily.

To force an explosive scenario we have a pattern matching with "alpha" value.

  def say_hello("alpha" = to), do: raise("Error to say hello to #{to}")

  def say_hello(to) do
    IO.puts("Hello #{to}")
    :ok
  end

  def process() do
    items = ["alpha", "beta", "gama"]

    Enum.map(items, fn item ->
      Task.async(fn ->
        say_hello(item)
      end)
    end)
    |> Enum.map(&Task.await/1)
  end

And finally, execute it using elixir's interactive shell.

iex -S mix

Then execute the code:

iex(1)> self
#PID<0.145.0>
iex(2)> Newsample.process
Hello beta
Hello gama

22:01:54.276 [error] Task #PID<0.148.0> started from #PID<0.145.0> terminating
** (RuntimeError) Error to say hello to alpha
    (newsample 0.1.0) lib/newsample.ex:15: anonymous fn/2 in Newsample.process/1
    (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: #Function<1.13017213/0 in Newsample.process/1>
    Args: []
** (EXIT from #PID<0.145.0>) shell process exited with reason: an exception was raised:
    ** (RuntimeError) Error to say hello to alpha
        (newsample 0.1.0) lib/newsample.ex:15: anonymous fn/2 in Newsample.process/1
        (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
        (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
        (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3

Interactive Elixir (1.11.3) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> self
#PID<0.151.0>

As we can see the raised exception was propagated until the iex session. It's possible to check it comparing the PID before #PID<0.145.0> and after #PID<0.151.0> run the code.

How we avoid this explosive error propagation?

Now we are in the step to use a supervised task, let's open lib/newsample/application.ex.

  def start(_type, _args) do
    children = []

    opts = [strategy: :one_for_one, name: Newsample.Supervisor]
    Supervisor.start_link(children, opts)
  end

Let's create our Task.Supervisor as a child processes

  children = [
    {Task.Supervisor, name: Newsample.TaskSupervisor}
  ]

And now use this supervised task in the process method and start as async_no_link. It means that there is no link between this child process and the parent.

  def process() do
    items = ["alpha", "beta", "gama"]

    Enum.map(items, fn item ->
      Task.Supervisor.async_nolink(Newsample.TaskSupervisor, fn ->
        say_hello(item)
      end)
    end)
    |> Enum.map(&Task.await/1)
  end

If we execute again everything will work correctly without this error propagation.

iex(1)> self
#PID<0.158.0>
iex(2)> Newsample.process
Hello beta
Hello gama

21:04:48.200 [error] Task #PID<0.161.0> started from #PID<0.158.0> terminating
** (RuntimeError) Error to say hello to alpha
    (newsample 0.1.0) lib/newsample.ex:6: Newsample.say_hello/1
    (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
    (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: #Function<1.39266525/0 in Newsample.process/0>
    Args: []
** (exit) exited in: Task.await(%Task{owner: #PID<0.158.0>, pid: #PID<0.161.0>, ref: #Reference<0.1303494724.713818114.156158>}, 5000)
    ** (EXIT) an exception was raised:
        ** (RuntimeError) Error to say hello to alpha
            (newsample 0.1.0) lib/newsample.ex:6: Newsample.say_hello/1
            (elixir 1.11.3) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
            (elixir 1.11.3) lib/task/supervised.ex:35: Task.Supervised.reply/5
            (stdlib 3.14) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
    (elixir 1.11.3) lib/task.ex:639: Task.await/2
    (elixir 1.11.3) lib/enum.ex:1411: Enum."-map/2-lists^map/1-0-"/2
iex(2)> self
#PID<0.158.0>

As we can see the PID is the same after and before the execution.

Processes and supervisors trees are important components of elixir to build resiliences and fault tolerance systems.

The Power of Elixir Task Module - The beginning

Felipe Araujo — Fri, 23 Apr 2021 16:35:37 +0000

Recently i was studying more deeply how the elixir Task module works and to consolidate my study i decided to write this post.

Before we start, let's check the definition of the Task module. The best place to do it is in the Elixir official documentation. There we have:

Conveniences for spawning and awaiting tasks.

Tasks are processes meant to execute one particular action throughout their lifetime, often with little or no communication with other processes. The most common use case for tasks is to convert sequential code into concurrent code by computing a value asynchronously

Good definition, but show me the code!!

Start an async operation

We have two basic ways to execute an async operation using Task. It's possible to use Task.start and Task.async. Let's see how it works.

Task.start

Task.start(fn -> IO.inspect("Hello") end)

{:ok, #PID<0.114.0>}

Task.async

Task.async(fn -> IO.inspect("Hello") end)

%Task{
  owner: #PID<0.110.0>,
  pid: #PID<0.118.0>,
  ref: #Reference<0.626386777.2138832899.106529>
}

It's possible to see that Task.start returns a tuple with :ok and a process id, while Task.async returns a Task struct. However, both work in the same way.

Sometimes we need to wait some async process result to execute the next step. Let's do it!

Waiting for results

The previous sample was very basic. Adding some delay will make it better.

  Task.async(fn ->
    :timer.sleep(5000)
    IO.inspect("Hello")
    :ok
  end)

As we know, the response will be a %Task{} struct. To await for a response we have two options: Task.await and Task. yield. Let's check their differences:

Task.await

Default timeout is 5 seconds;
Given a timeout, it throws an exception;
After the timeout is reached, the task is stopped;
You can define a custom timeout or use the atom :infinity.

Task.await(task)
Task.await(task, :infinity)

Timeout sample

> task = Task.async(fn -> IO.inspect("Hello") ; :timer.sleep(10000); :ok end)

> Task.await(task)
"Hello"
** (exit) exited in: Task.await(%Task{owner: #PID<0.110.0>, pid: #PID<0.124.0>, ref: #Reference<0.3761442499.262406148.76432>}, 5000)
    ** (EXIT) time out
    (elixir 1.11.3) lib/task.ex:643: Task.await/2

As we can see, a timeout is a little explosive when using Task.await. A way to handle it better is to use Supervised Tasks.

Task.yield

Default timeout is 5 seconds;
Given a timeout, it returns nil;
Using the atom :infinity is not allowed as on Task.await;
After the timeout is reached, it keeps the task running;
It's possible to finish a running task using Task.shutdown(task, shutdown \\ 5000).

> task = Task.async(fn -> IO.inspect("Hello") ; :timer.sleep(10000); :ok end)
> Task.yield(task)
nil
# Let's check again
> Task.yield(task)
{:ok, :ok}

Given a :timeout result, the response will be nil. After that, we can execute Task.yield again. To avoid long running tasks without any results, you have to use Task.shutdown(task, shutdown \\ 5000).

A more complete sample

We have a list of items and it's necessary to execute some work in all of them:

    items = ["alpha", "beta", "gama"]

    Enum.map(items, fn item ->
      Task.async(fn ->
        :timer.sleep(4000)
        IO.inspect("Hello #{item}")
        :ok
      end)
    end)
    |> Enum.map(&Task.await/1)
    |> function_to_handle_results()

With this approach, we still have the exception given a timeout is reached. However, it's possible to handle it in a better way with Supervised Tasks, but this subject will be covered in the next post.

DEV Community: Felipe Araujo

It's not just about coding

O Poder do Módulo Task do Elixir -Task.Supervisor

Criando o Playground

Então, como podemos evitar esse comportamento explosivo?

O Poder do Módulo Task do Elixir - O Início

Iniciando uma operação async

Esperando resultados

Um exemplo mais completo

Conteúdo adicional

The Power of Elixir Task Module - Into Task.Supervisor

Creating the playground

How we avoid this explosive error propagation?

The Power of Elixir Task Module - The beginning

Start an async operation

Waiting for results

A more complete sample

Additional content