DEV Community: vKxni

Why side projects die so quickly ...

vKxni — Sat, 13 May 2023 09:26:58 +0000

As a programmer, you've likely experienced the frustrating feeling of staring at a blank screen, unsure of what to do next. But then, out of nowhere, it happens.
Like a bolt of lightning, a flash of inspiration strikes, and suddenly you're buzzing with excitement.
You've got it! A brilliant idea for a new program or website that has the potential to be a game-changer. You can already imagine the reactions of your friends and followers on GitHub.
This is it!
But then, just as you're gaining momentum, something unexpected happens...

Scenario 1: The Quick Idea

Without hesitation, you dive in headfirst. You come up with a catchy name, bootstrap your files, and begin coding. The adrenaline is pumping through your veins as you are excitedly trying to bring your vision to life.
But as you get further into the project, you start to realize that things aren't going quite as smoothly as you'd hoped.

Error messages start popping up left and right, and you can't seem to make any progress.
You feel your motivation slipping away, and frustration sets in. You're a beginner, and maybe this project was just too ambitious for your skill level.
But you're determined not to give up just yet. You try to push through, but the more you try, the more complicated the project seems to become.

Finally, you throw in the towel and delete the project folder. It's just another failed attempt at bringing your idea to life.

Scenario 2: A Small Effort

Alternatively, maybe a good friend approaches you with an idea for a new program or app that sounds promising.
You're excited to work on it together and dive into planning. You spend countless hours discussing the details and mapping out the project's scope.
You bootstrap the main files, and the project is starting to take shape.

But as time goes on, you start to realize that your friend isn't as committed to the project as you are.
You're doing all the heavy lifting, and your friend is nowhere to be found. You feel frustrated and disheartened.
Maybe this was just a passing fancy for your friend, but you're still invested in the project.
You keep pushing, but the work seems to pile up, and progress slows to a crawl. Soon, even you begin to lose steam, and the project becomes yet another unfinished idea gathering dust on your hard drive.

Conclusion

At the end, it is crucial to take a step back and thoroughly evaluate any project idea before jumping in without proper consideration.
It's essential to consider factors such as your skill level, the complexity of the project, and the feasibility of completing it within your available time frame.
By doing so, you can avoid the common pitfalls of taking on too much at the outset, becoming overwhelmed, and ultimately giving up on the project.
It's also important to keep in mind that a successful project often requires collaboration and support from others, so it's essential to have a solid team or support system in place from the start.
With a well-planned approach, realistic expectations, and the right support, you can turn your project idea into a successful and fulfilling reality.

"Programming is the art of transforming frustration into flashes of inspiration."

Elixir Telemetry Logging

vKxni — Sat, 01 Oct 2022 09:32:50 +0000

Ever wondered how you can log your Ecto/Mix Events to a file or even to the console? Well, here is how!

A small showcase

⚠️ Warning:
This isn't valid SQL code, this is just logging the Code being executed by Ecto that is able to save/edit/delete/purge data.

Requirements:

Elixir v13.3.2 +
A working Database
Basic Knowledge of Elixir

$ elixir -v
Elixir 1.13.2 (compiled with Erlang/OTP 24)

Getting started

Here, as an example, I am using Ecto - however, you can use basically everything else (that might be similar) and directly skip to the Telemetry part.

Docs: https://hexdocs.pm/ecto/getting-started.html

Creating our project

$ mix new sqlt --sup

Adding packages

If you use another DB, like MySQL, then add that instead.

{:ecto_sql, "~> 3.0"},
{:postgrex, ">= 0.0.0"}

make sure to run mix deps.get to install them.

Generating our Repo

$ mix ecto.gen.repo -r SQLT.Repo

This command will generate the configuration required to connect to a database. The first bit of configuration is in config/config.exs:

config :sqlt, SQLT.Repo,
  database: "localhost",
  username: "root",
  password: "admin",
  hostname: "localhost"

Make sure to fill out the correct informations here.

NOTE: Your PostgreSQL database may be setup to

not require a username and password. If the above configuration doesn't work, try removing the username and password fields, or setting them both to "postgres".
be running on a non-standard port. The default port is 5432. You can specify your specific port by adding it to the config: e.g. port: 15432.

Configurating our Repo

The SQLT.Repo module is defined in lib/sqlt/repo.ex by our mix ecto.gen.repo command:

defmodule SQLT.Repo do
  use Ecto.Repo,
    otp_app: :sqlt,
    adapter: Ecto.Adapters.Postgres
end

We now have to add our Repo to our Application. For that being said, open your application.ex file.

Possible path: lib/sqlt/application.ex.
Within the children, add the following:

 children = [
  # add this
  {SQLT.Repo, []},
]

This piece of configuration will start the Ecto process which receives and executes our application's queries. Without it, we wouldn't be able to query the database at all!

There's one final bit of configuration that we'll need to add ourselves, since the generator does not add it. Underneath the configuration in config/config.exs, add this line:
config :sqlt, ecto_repos: [SQLT.Repo]

This tells our application about the repo, which will allow us to run commands such as mix ecto.create very soon.

Database Setup

Run the following command:

$ mix ecto.create

You should now see
The database for SQLT.Repo has been created.

NOTE: If you get an error, you should try changing your configuration in config/config.exs, as it may be an authentication error.

This command will generate a brand new migration file in priv/repo/migrations, which is empty by default:

defmodule SQLT.Repo.Migrations.People do
  use Ecto.Migration

  def change do

end

We will now create a table called people and also add some values to it.

defmodule SQLT.Repo.Migrations.People do
  use Ecto.Migration

  def change do
    create table(:people) do 
      add :first_name, :string
      add :last_name, :string
      add :age, :integer
    end
  end
end

To run this migration and create the people table in our database, we will run this command:

$ mix ecto.migrate

Creating a schema

Let's create the schema within our application at lib/sqlt/people.ex:

defmodule SQLT.People do
  use Ecto.Schema

  @moduledoc """
  Changesets allow filtering, casting, validation and definition of constraints when manipulating structs.
  There is an example of working with changesets in the introductory documentation in the Ecto module. 
  The functions cast/4 and change/2 are the usual entry points for creating changesets. 
  The first one is used to cast and validate external parameters, such as parameters sent through a 
  form, API, command line, etc. 
  The second one is used to change data directly from your application.
    - https://hexdocs.pm/ecto/Ecto.Changeset.html
  The remaining functions in this module, such as validations, constraints, 
  association handling, are about manipulating changesets. 
  """

  def changeset(person, params \\ %{}) do
    person
    # tell what parameters are allowed to be passted through 
    |> Ecto.Changeset.cast(params, [:first_name, :last_name, :age])
    # we expect values for fn + ln
    |> Ecto.Changeset.validate_required([:first_name, :last_name])
  end

  schema "people" do
    field(:first_name, :string)
    field(:last_name, :string)
    field(:age, :integer)
  end
end

Awesome! Our database is now fully working.

Adding telemetry

Telemetry is a lightweight library for dynamic dispatching of events, with a focus on metrics and instrumentation. Any Erlang or Elixir library can use telemetry to emit events. Application code and other libraries can then hook into those events and run custom handlers.

Docs: https://hexdocs.pm/telemetry/readme.html

Configurating our Application

Open your application.ex file (lib/sqlt/application)
and add the following line

 :ok = :telemetry.attach("sqlt-repo-handler", [:sqlt, :repo, :query], &SQLT.Telemetry.handle_event/4, %{})

above the children.

Your application.ex should now look like this:

defmodule Sqlt.Application do
  # See https://hexdocs.pm/elixir/Application.html
  # for more information on OTP Applications
  @moduledoc false

  use Application

  @impl true
  def start(_type, _args) do
   # here!! :)
    :ok = :telemetry.attach("sqlt-repo-handler", [:sqlt, :repo, :query], &SQLT.Telemetry.handle_event/4, %{})

    children = [
      {SQLT.Repo, []},
    ]

    # See https://hexdocs.pm/elixir/Supervisor.html
    # for other strategies and supported options
    opts = [strategy: :one_for_one, name: Sqlt.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Configurating Telemetry

Start by creating a telemetry.ex file at the following path: lib/sqlt/telemetry.ex

defmodule SQLT.Telemetry do
  require Logger

def emit(value) do 
    :telemetry.execute([:sqlt, :emit], %{value: value})
end

# *
def handle_event([:sqlt, :repo, :query], measurements, metadata, _config) do
    data = "#{metadata.query} #{metadata.params}"

    Logger.info(data)
    File.write("raw.sql", data)
  end
end

*
Here we handle "main event" for the telemetry handler defined in the application.ex file

:ok = :telemetry.attach("sqlt-repo-handler", [:sqlt, :repo, :query], &SQLT.Telemetry.handle_event/4, %{})

Once some event is executed, we log it to the console and also write the content into a separate file in our root directory.

We use the metadata variable with the query and params parameter, tho there are some more

:type - the type of the Ecto query. For example, for Ecto.SQL databases, it would be :ecto_sql_query

:repo - the Ecto repository

:result - the query result

:params - the query parameters

:query - the query sent to the database as a string

:source - the source the query was made on (may be nil)

:options - extra options given to the repo operation under :telemetry_options

Also, if you go through the code, you notice the handle_event function, this is where all the magic happens. Obviously you can remove the Logger (nothing will be logged to the console) and also change the behaviour (in my case writing the result to a .sql file).

Creating Events

In my case I am using Ecto Events, of course you can use something else.

I will now create some functions that is inserts, edits and also deletes data from the Database.

lib/sqlt/queries.ex

defmodule Query do
  require Ecto.Query

# here we create / insert data
def insert do
    firstname = IO.gets("Enter the firstname: ") |> String.trim()
    lastname = IO.gets("Enter the lastname: ") |> String.trim()
    person = %SQLT.People{first_name: firstname, last_name: lastname, age: 25}
    SQLT.Repo.insert(person)
  end

# here we edit our data
def edit do
    name = IO.gets("Enter the first_name that you want to find: ") |> String.trim()
    newname = IO.gets("Now enter the new first_name: ") |> String.trim()
    person = SQLT.People |> Ecto.Query.where(first_name: ^name) |> SQLT.Repo.one()
    changeset = SQLT.People.changeset(person, %{first_name: newname})

    case SQLT.Repo.update(changeset) do
      {:ok, person} ->
        IO.puts("Updated #{person}")

      {:error, error} ->
        IO.puts("Error #{error}")
    end
  end
end

🎉🎉🎉 YOU ARE AWESOME! 🎉🎉🎉
You have successfully created a custom event logger with Telemetry.

If you run one of the functions above, you will see the raw.sql being created. By running more functions, telemetry will be write them into that file.

I have uploaded the whole code to GitHub, just in case if you want to take a look at the whole code/repo.

https://github.com/vKxni/sqlt

Rate Limits Phoenix

vKxni — Thu, 22 Sep 2022 15:31:48 +0000

Ever wondered how to protect your APi/Backend from Spam? Here is how.

Requirements:

Elixir v13.3.2 +
Phoenix v1.6.10+
Basic Knowledge of Elixir

$ elixir -v
Elixir 1.13.2 (compiled with Erlang/OTP 24)

$ mix phx.new --version
Phoenix installer v1.6.10

Getting started

Create a new Phoenix Project

$ mix phx.new ratelimit --no-mailer --no-assets --no-html --no-ecto --no-dashboard

If you will be asked to install some dependencies, answer with y (yes).

Adding Dependencies

Lets add some dependencies that will help us doing all of this
mix.exs

defp deps do
    [
      # here is some other stuff

      # <-- add the stuff below here -->

      # http
      {:httpoison, "~> 1.8"},
      # rate limit
      {:hammer, "~> 6.1"},
    ]
  end

For the rate limits, we will use the following library
https://github.com/ExHammer/hammer

Install dependencies

$ mix deps.get

Project files

Now lets get our hands dirty by creating some helper functions.

lib/ratelimit/base.ex

defmodule Ratelimit.Base do
  use HTTPoison.Base

  @moduledoc """
  This handles HTTP requests without api key (basic requests).
  """

  def process_request_headers(headers) do
    [{"Content-Type", "application/json"} | headers]
  end
end

Now lets add a function that gets the IP of the user visiting our website.
lib/ratelimit/helper/getip.ex

defmodule Ratelimit.IP do
  @doc """
  Get the IP address of the current user visiting the route.
  Formatted as a string: "123.456.78.9"
  """
  # {:ok, String.t()} | {:error, :api_down}
  @spec getIP() :: {String.t() | :api_down}
  def getIP() do
    ip_url = "https://api.ipify.org/"

    case Ratelimit.Base.get!(ip_url) do
      %HTTPoison.Response{body: body, status_code: 200} ->
        body

      %HTTPoison.Response{status_code: status_code} when status_code > 399 ->
        IO.inspect(status_code, label: "STATUS_CODE")
        :error

      _ ->
        raise "APi down"
    end
  end
end

Awesome, now lets create a file that handles our ratelimits
lib/ratelimit/util/ratelimit.ex

defmodule RatelimitWeb.Plugs.RateLimiter do
  import Plug.Conn
  use RatelimitWeb, :controller

  alias Ratelimit.IP
  require Logger

  # two request / minute are allowed
  @limit 2

  def init(options), do: options

  def call(conn, _opts) do
    # call the ip function
    ip = IP.getIP()

    case Hammer.check_rate(ip, 60_000, @limit) do
      {:allow, count} ->
        assign(conn, :requests_count, count)

      {:deny, _limit} ->
        # Beep Boop, remove this in production
        Logger.debug("Rate limit exceeded for #{inspect(ip)}")
        error_response(conn)
    end
  end

  defp error_response(conn) do
    conn
    |> put_status(:service_unavailable) # set the json status
    |> json(%{message: "Please wait before sending another request."}) # return an error message
    |> halt() # stop the process
  end
end

Now we have to configure Hammer in our config files.
For that, open config/config.exs and add this line here:

# Config the rate limiter
config :hammer,
  backend: {Hammer.Backend.ETS, [expiry_ms: 60_000 * 60 * 4, cleanup_interval_ms: 60_000 * 10]}

Adding the Controller

Now we have to create a simple controller for our website.
lib/ratelimit_web/controllers/page_controller.ex

defmodule RatelimitWeb.PageController do
  use RatelimitWeb, :controller

  def index(conn, _params) do
    send_resp(conn, 200, "Hello there!")
  end
end

and we have to edit our lib/ratelimit_web/router.ex to the following

pipeline :api do
    # add the rate limit plug here
    plug RatelimitWeb.Plugs.RateLimiter
    plug :accepts, ["json"]
  end

scope "/api", RatelimitWeb do
    pipe_through :api

    # add this here
    get "/test", PageController, :index
  end

Start the Server

Now lets try to start our APi with the following

$ mix phx.server

After that, navigate to the following URL:
http://localhost:4000/api/test.
You should see the following:

Sending requests

Now to test our rate limits, send multiple request to the same URL, by just refreshing the page more than 2 times.

You will see something changing suddenly, like this:

🎉🎉🎉 You are awesome!

Additional Things

If you want to change the message, you can easily do this in the lib/ratelimit/util/ratelimit.ex file.

In production, remove the IP inspect at the Logger.debug.

The code can be found here: https://github.com/vKxni/ratelimit

REST APi in Phoenix

vKxni — Tue, 16 Aug 2022 09:35:00 +0000

Ever wondered how to write a small REST APi for your frontend or for public usage? - here is how.

Requirements:

Elixir v13.3.2 +
Phoenix v1.6.10+
Basic Knowledge of Elixir

$ elixir -v
Elixir 1.13.2 (compiled with Erlang/OTP 24)

$ mix phx.new --version
Phoenix installer v1.6.10

Getting started

Lets start with creating our Project with the following command

$ mix phx.new myApp --no-html --no-assets --no-mailer --no-dashboard --no-ecto

This will remove all the useless boilerplate for us - since APis do not need any HTML or CSS.

Now lets path into your folder:

$ cd myApp

Lets open our router first lib/myapp_web/router.ex, there, add the following code
get "/random", RandomController, :index

Your Router should now look like this:

defmodule MyAppWeb.Router do
  use MyAppWeb, :router

  pipeline :api do
    plug :accepts, ["json"]
  end

  # localhost:4000/api/{route}
  scope "/api", MyAppWeb do
    pipe_through :api

    get "/random", RandomController, :index
  end
end

Custom Controller

As defined in the router, we now need a Controller called RandomController.
For that, open the folder controllers (lib/myapp_web/controllers) and create a new file called random_controller.ex.

defmodule MyAppWeb.RandomController do
  use MyAppWeb, :controller

  def index(conn, _params) do
    send_resp(conn, 200, "hello world")
  end
end

Save this file and now try to start your server.

$ mix phx.server

Your server should be listening on http://localhost:4000/api/random. Try to enter that URL in your favourite browser. Your should now see "hello world" 😁.

Custom Files

It's not a good practice to write functions directly into your controller - specially in a bigger project - so lets write a helper function that handles everything for us.

Create a folder called helper with the following path: lib/myapp/helper.

Since this isn't web related, we will now use our main folder (not the myapp_web one).

In that folder, create a file called random.ex.

It should look like this:

defmodule MyApp.Random do
  def random_number do
   Enum.random(100..10000)
  end
end

Awesome, now lets go back to our random_controller.ex and modify it a little bit.

First of all, import our new helper file with its function.

import MyApp.Random

Then replace our "hello world" response with the function name random_number().

Like this:
send_resp(conn, 200, random_number()).

Your random_controller.ex should now look like this

defmodule MyAppWeb.RandomController do
  use MyAppWeb, :controller

  import MyApp.Random

  def index(conn, _params) do
    send_resp(conn, 200, random_number())
  end
end

Save the file and restart your server

$ mix phx.server

Go to the following URL: http://localhost:4000/api/random, you should now see a random number every time you refresh/enter the page. 🎉

Handling CORS

This is a more in-depth topic - but it's necessary if you want other people to use your API or even call your own APi from another port/ip.

We will use a library called corsica for this, lets install it by opening our mix.exs file in the root directory.

Add the following line into your deps function

{:corsica, "~> 1.2"}

Your mix.exs should now look like this

defmodule MyApp.MixProject do
  use Mix.Project

  def project do
    [
      app: :myapp,
      version: "0.1.0",
      elixir: "~> 1.13",
      description: "An example Web APi",
      elixirc_paths: elixirc_paths(Mix.env()),
      compilers: Mix.compilers(),
      start_permanent: Mix.env() == :prod,
      aliases: aliases(),
      deps: deps(),
      author: "github.com/vKxni",
      docs: [
        main: "readme",
        extras: [
          "README.md"
        ]
      ]
    ]
  end

  def application do
    [
      mod: {MyApp.Application, []},
      extra_applications: [:logger, :runtime_tools]
    ]
  end

  defp elixirc_paths(:test), do: ["lib", "test/support"]
  defp elixirc_paths(_), do: ["lib"]

  # Here !! :)
  defp deps do
    [
      # phoenix server
      {:phoenix, "~> 1.6.10"},
      {:telemetry_metrics, "~> 0.6"},
      {:telemetry_poller, "~> 1.0"},
      {:gettext, "~> 0.18"},
      {:plug_cowboy, "~> 2.5"},
      {:poison, "~> 5.0"},
      {:jason, "~> 1.3"},

      # add this line here
      {:corsica, "~> 1.2"} # <--
    ]
  end

  defp aliases do
    [
      setup: ["deps.get"]
    ]
  end
end

Once done, run the following command

$ mix deps.get

This might take some time depending on your connection.

Adding CORS to our Endpoint

Open the file called endpoint.exs in your myapp_web folder.
(lib/myapp_web/endpoint.exs)

Now in your Endpoint, add the following lines

 plug(Corsica,
    max_age: 600,
    origins: "*",
    allow_headers: ["accept", "content-type", "authorization"],
    allow_methods: ["GET", "POST"],
    allow_credentials: true,
    log: [rejected: :error, invalid: :warn, accepted: :debug]
  )

Your endpoint.exs should now look like this:

defmodule MyAppWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :xadara

  @session_options [
    store: :cookie,
    key: "_myapp_key",
    signing_salt: "g10HT1jr"
  ]

  # our CORS config starts here
  plug(Corsica,
    max_age: 600,
    origins: "*",
    allow_headers: ["accept", "content-type", "authorization"],
    allow_methods: ["GET", "POST"],
    allow_credentials: true,
    log: [rejected: :error, invalid: :warn, accepted: :debug]
  )
  # CORS ends here

  plug Plug.Static,
    at: "/",
    from: :myapp,
    gzip: false,
    only: ~w(assets fonts images favicon.ico robots.txt)

  if code_reloading? do
    plug Phoenix.CodeReloader
  end

  plug Plug.RequestId
  plug Plug.Telemetry, event_prefix: [:phoenix, :endpoint]

  plug Plug.Parsers,
    parsers: [:urlencoded, :multipart, :json],
    pass: ["*/*"],
    json_decoder: Phoenix.json_library()

  plug Plug.MethodOverride
  plug Plug.Head
  plug Plug.Session, @session_options
  plug MyAppWeb.Router # this here is important too
end

If something looks a bit different, don't worry, just add the new PLUG right below the session_options.

And you are done! Congratulations.