DEV Community: Takuya Aoki

How do I improve performance in Elixir

Takuya Aoki — Thu, 23 Nov 2017 13:04:41 +0000

https://dev.to/tokoyax/a-little-story-about-the-no-command-1am

Continuation of the above article

Execution speed problem

Command no is slower than theyes command like this.

> yes | pv -r > /dev/null
[31.0 MiB/s]

>./no | pv -r > /dev/null
[85.6 KiB/s]

I got the following comment.

you've better to use buffer for writing. Actually, yes command does not call write (2) for each lines.

It seems to be caused by a large number of I/O, so it is better to use a buffer.

Try to improve.

Using IO List

https://www.bignerdranch.com/blog/elixir-and-io-lists-part-1-building-output-efficiently

According to the above article, when passing strings to IO.puts, Instead of passing only one combined string, storing the part of the string to the list has better performance.

If the same string is stored in the list, it's efficiently because the same string refers to the same address.

I fixed 'no.exs' to use a buffer.

no.exs

defmodule No do
  @moduledoc """
  Documentation for No.
  """

  def main(args) do
    args
    |> parse_args()
    |> run()
    |> wait()
  end

  def say(args) do
    {_, _, _, debug} = args
    message = buffer(args)
    case debug do
      true -> IO.inspect({:debug, message, self()})
      _    -> IO.puts(message)
    end
    say(args)
  end

  defp buffer(args) do
    {argv, process_num, _, debug} = args

    message = case debug do
      true -> "#{expletive(argv)} from process no.#{process_num}"
      _    -> expletive(argv)
    end

    buffer(args, message, [])
  end

  defp buffer(args, message, list) do
    # faster than append last
    # https://hexdocs.pm/elixir/List.html
    list = [message | list]
    cond do
      IO.iodata_length(list) >= 4096 -> list
      true -> buffer(args, message, list)
    end
  end

  defp expletive(_ = ''), do: "n\n"
  defp expletive(value), do: value

  defp parse_args(args) do
    {opts, argv, _} = OptionParser.parse(
      args,
      swithces: options_list(),
      aliases: aliases_list()
    )
    {argv, opts}
  end

  defp options_list() do
    [
      processes: :integer,
      debug: :boolean,
    ]
  end

  defp aliases_list() do
    [
      p: :processes,
      d: :debug,
    ]
  end

  defp run(args) do
    {argv, opts} = args

    num = case opts[:processes] do
      nil -> 1
      _   -> String.to_integer(opts[:processes])
    end

    Enum.each((1..num), fn(n) ->
      spawn_link(No, :say, [{argv, n, self(), opts[:debug]}]) end)
  end

  defp wait(args) do
    wait(args)
  end
end

Measure

before

> ./no | pv -r > /dev/null
[85.6 KiB/s]

after

> ./no | pv -r > /dev/null
[802KiB /s]

SUCCESS :D

Try parallel execution

> ./ no - p 4 | pv - r > /dev/null
[1.33 MiB / s]

Okay, let's run in parallel more.

> ./ no - p 10000 | pv - r > /dev/null
[5.00 MiB /s]
[3.16 MiB /s]
[1.84 MiB /s]
[458 KiB /s]
[466 KiB /s]
[569 KiB /s]
[724 KiB /s]
[805 KiB /s]
[1.27 MiB /s]
[6.17 MiB /s]
[15.3 KiB /s]
[1.13 MiB /s]

Although the result which speeded up was obtained, the output result was not constant.

I don't know why.

Summary

I got knowledge of speed improvement such as IO List.
It is simply fun to make faster.
The yes command is still faster.

Github: [https://github.com/tokoyax/no:title]

(reference)

A Little Story About the 'no' Command

Takuya Aoki — Wed, 22 Nov 2017 05:05:39 +0000

Inspired by this article, I made similar commands as Elixir exercises.

http://postd.cc/a-little-story-about-the-yes-unix-command/

This is the 'no' command to deny all of the world!!!

The specification is almost same as yes. Continue to output n to the standard output forever.

It makes it possible to use it like this.

> no
n
n
n
n
n
n
n
...

Alternatively,

> no | sh work_a_lot.sh

Your annoying boss said, "Work harder!! Work harder!!!"

OMG...It's the way to 'KAROSHI'.

This command should be useful at such times :D

Let's start

Environment

> elixir -v
[Async - threads: 10] [hipe] [kernel - poll: kernel - poll: kernel - poll: false] [dtrace]

Elixir 1.5.2

Project creation

> mix new no
> cd no

Edit `mix.exs`

To create an executable file with Elixir, you can use escript.

http://erlang.org/doc/man/escript.html

line: 9 Specify module to use escript
line: 22-24 It is better to write a function that returns the module name etc.

mix.exs

defmodule No.Mixfile do
  use Mix.Project

  def project do
    [
      app: :no,
      version: "0.1.0",
      elixir: "~> 1.5",
      escript: escript_config(),
      start_permanent: Mix.env == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger]
    ]
  end

  defp escript_config do
    [main_module: No]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      # {:dep_from_hexpm, "~> 0.3.0"},
      # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"},
    ]
  end
end

No.main/1 of the module specified here becomes the entry point.

Implementation

It's not surprising that it just loops normally.I added an option -p that generates a lot of processes and keeps outputting n for each.Also, I made it possible to see the PID with -d so that I could observe the parallel execution.

OptionParser Conveniently useful!

no.exs

defmodule No do
  @moduledoc """
  Documentation for No.
  """

  def main(args) do
    args
    |> parse_args()
    |> run()
    |> wait()
  end

  def say(args) do
    {argv, process_num, sender, debug} = args
    case debug do
      true -> send sender, {:debug, "#{expletive(argv)} from process no.#{process_num}", self()}
      _    -> send sender, {:ok, expletive(argv)}
    end
    say(args)
  end

  defp expletive(_ = ''), do: 'n'
  defp expletive(value), do: value

  defp parse_args(args) do
    {opts, argv, _} = OptionParser.parse(
      args,
      swithces: options_list(),
      aliases: aliases_list()
    )
    {argv, opts}
  end

  defp options_list() do
    [
      processes: :integer,
      debug: :boolean,
    ]
  end

  defp aliases_list() do
    [
      p: :processes,
      d: :debug,
    ]
  end

  defp run(args) do
    {argv, opts} = args
    num = case opts[:processes] do
      nil -> 1
      _  -> String.to_integer(opts[:processes])
    end
    (1..num)
    |> Enum.each(
      fn(n) ->
        spawn_link(No, :say, [{argv, n, self(), opts[:debug]}])
      end
    )
  end

  defp wait(args) do
    receive do
      {:ok, message} ->
        IO.puts message
      {:debug, message, pid} ->
        IO.inspect {message, pid}
    end
    wait(args)
  end
end

Build

When you have finished implementing build.
Run in a directory with mix.exs.

> mix escript.build

Executable file should be generated in the current directory.

Execution

In the current directory.

> ./no -p 100
n
n
n
n
n
...

Debug option available.

>.. / no - p 100 - d
{"n from process no. 47", # PID <0.122.0>}
{"n from process no. 47", # PID <0.122.0>}
{"n from process no. 47", # PID <0.122.0>}
{"n from process no. 40", # PID <0.115.0>}
{"n from process no. 47", # PID <0.122.0>}
...

Gotcha!!!!

How fast is it?

> ./no | pv -r > /dev/null
[93.7 KiB / s]

As for the yes command,

> yes | pv -r > /dev/null
[31.6 MiB / s]

no command is toooo slow!!!! xD
Calm down, I believe it will reverse in parallel processing!!

>./no - p 100 | pv -r > /dev/null
[1.12 KiB / s]

...Why?

Summary

It is good to reimplement the Linux command as a language practice subject
Many of the functions are reasonably sized
It is pleasant to go through processing with * |>
I don't know why performance is slow.
I will investigate later.
Registration to Hex
Registered -> http://blog.tokoyax.com/entry/elixir/register-to-hex

Github Repo: https://github.com/tokoyax/no

This article is translated from Japanese origin.

http://blog.tokoyax.com/entry/elixir/no-command

(reference)

my first post

Takuya Aoki — Wed, 15 Nov 2017 07:01:14 +0000

hello!!!!

DEV Community: Takuya Aoki

How do I improve performance in Elixir

Execution speed problem

Using IO List

Measure

Summary

A Little Story About the 'no' Command

This is the 'no' command to deny all of the world!!!

Let's start

Environment

Project creation

Edit mix.exs

Implementation

Build

Execution

How fast is it?

Summary

my first post

Edit `mix.exs`