DEV Community: Nicol Acosta

API Prototypes with dbb: Another step to better prototypes

Nicol Acosta — Thu, 09 May 2024 06:20:22 +0000

Second Post to know the other features in dbb

Introduction

In the previous post i added more features that i added to dbb and in this time, its not different, i added more stuff to build even more good stuff

Admin UI

This stuff are inside since the last post hehe, but i forgot to mention in the post so here is the missing part

It's really simple, just open your server on the / route and will load the admin page, the API it's on the /api/v1 route so will not conflict with the other routes

Will show obviously the schemas, basic CRUD for any of them, with cool create/edit forms based on the datatypes, the basic that any schema scan admin needs to have

New datatype

In this case it's just one type, that it's list of values, you can define just one type or more, and will be saved in database in the same way

To define in your schema, just add mark your type inside the list brackets ["string"]

{
  "schemas": [
    {
      "name": "some_schema_name",
      "fields": {
        "name": "string",
        "flags": ["string"],
      }
    }
  ]
}

In the Admin UI will show a input with the elements on text separated by comma, so ["some", "value"] will be displayed as "some, value" and in the same format will save as list

Value generator

Maybe in some case you need generate some key or some like that, so you can define this generator format in the generate key in the json

{
  "schemas": [
    {
      "name": "some_schema_name",
      "fields": {
        "name": "string",
        "sku": "string",
      },
      "generate": {
        "sku": "$str(4)$-$num(4)$-$sym(4)$-$str_num(4)$-$any(4)$"
      }
    }
  ]
}

The way to create a generator it's simple, just define in the generator your format with the function with $ on start and end of the function, will works in this format $str(4)$ and you can use this generator functions str, num, sym, str_num, any

In the Admin UI just just with keep empty to generate the value

Relations

You can relate one schema to another using the in the relations key in the json

{
  "schemas": [
    {
      "name": "orders",
      "fields": {
        "estimated_delivery_time": "datetime"
      }
    },
    {
      "name": "order_product",
      "fields": {
        "order_id": "string",
        "product_id": "string"
      },
      "relations": {
        "order_id": "orders",
        "product_id": "products"
      }
    },
    {
      "name": "products",
      "fields": {
        "name": "string",
        "description": "string",
      }
    }
  ]
}

You can define also if some relation is mandatory adding :mandatory to your key name

Load relations in query

Why have relations if you can't load this related data in the same call, so of course you can, adding relations as query param with your relations to load (yes, allow multiple)

GET /api/v1/:schema?relations=relation_1,relation_2

Pretty simple to use

Admin UI basic auth

The last is a basic auth to Admin UI, simple to configure, with env vars, not much secure it's just to not have unsecured page in the system

ADMIN_AUTH_USERNAME: user
ADMIN_AUTH_PASSWORD: pass

Next steps

The next to complete it's create a more complete experience on the admin, add functions like operations on query calls, some seeds to populate the system as demo and more features to create a complete solution as prototype

API Prototypes with dbb: A little more than a prototype

Nicol Acosta — Mon, 23 Oct 2023 08:21:20 +0000

First Post to know the basic about dbb

Introduction

In the last post i added the basics about dbb and since the date, i added a little bit of features, trying to make more helpful and automated so, here it's the changes

Schema file validation

Now you the system wouln't start if your schema file hasn't the right format, this ensure all it's rightly configured in your file, not a great thing

Webhooks

Before this, the API was just a simple CRUD without any other kind of trick, now this changes all, because you can define, in any call to a specific action a webhook call to any endpoint, with the data in the body

{
  "schemas": [
    {
      "name": "table_name",
      "fields": {
        "some_field": "string"
      },
      "hooks": [
        {
          "events": ["index", "create", "update"],
          "url": "https://some.site/some/hook",
          "method": "get",
          "headers": {
            "key": "value"
          }
        }
      ]
    }
  ]
}

You can specify your webhook auth in case that it's needed

New Datatypes

Now you can define in your schema the UUID's and Datetimes (Or Date and Time)

{
  "schemas": [
    {
      "name": "table_name",
      "fields": {
        "date_field": "date",
        "time_field": "time",
        "datetime_field": "datetime"
      }
    }
  ]
}

Swagger UI

Now automatically deploys an endpoint with swaggerui to play with your new API directly from browser

To access you need open http://your_instance_url/api_docs/v1

Next steps

The next steps it's work in a admin panel that shows the records of the schema tables, possible triggers like webhooks and all related stuff related to the model specs and records, maybe more datatypes, a API call that not triggers hooks, possible auth for the admin panel, im planning this part so all it's possible

Summary

Admin panel
- Data from tables
- Triggers (hooks)
API call that no triggers hooks
More datatypes

Easy site Crawling in Elixir with ex_crawlzy

Nicol Acosta — Mon, 23 Oct 2023 08:03:00 +0000

History

The last year (2022) i make the new year purpose (the tradition it's do nothing) to create something that create incomes, sounds like the default "i want to create my startup" history that never success, but must exist a starting poit isn't so i decide to create my brand new first service that will start as free and progresively will add this pricing service

So this super effective brand new product was never released, because i had some personal issues with my free time so it's on-hold or i want to belive this xD, btw, i coded a lot of stuff, created and learned a lot because this service was focused on crawling, yes, crawling

My favorite language is Elixir so all my personal projects are on Elixir and also im a beliver of OpenSource, so i decided that my code wouldn't be just isolated on that private repo, i created something good and thing that maybe in the world, some dude alone without hope of humanity must need help crawling a site using Elixir because i think that everyone in their career, in some moment, needs defeat the crawling monster, so my library must make this task easy

The solution

Stop of history, and let's move to the solution

Crawling exist in Elixir, the classic http request and the libraries that make this more 'easy"

The main point it's that all the libraries that i found has the same point, just the call and returns the html as text, so the parsing and all the hard stuff basically it's the same

What is the difference on make my own call using some http request librar and use your library? NOTHING!!!

So i create my library that basically calls to the endpoint site and returns the content parsed as a Map, sounds great, and it's great

The library it's ex_crawlzy and gives a direct solution to this crawl based on css selectors, that it's the stuff that almost all world does when the crawl it's needed, so you just need your selectors and this library makes the hard work

The code

The same of all libraries, add the dependencies

def deps do
  [
    {:ex_crawlzy, "~> 0.1.1"}
  ]
end

And includes the classic way that gives all the crawling libraries, just giving the html as text

site = "https://example.site"

{:ok, html_content} = ExCrawlzy.crawl(site)

But here come's the interesting stuff, the library includes a function to parse the html_content if you give the css selectors in a map using a key: selector format

fields = %{
  # shortcut for use a function from ExCrawlzy.Utils
  body: {"div#the_body", :text}
#  module/function way
#  body: {"div#the_body", {ExCrawlzy.Utils, :text}}
#  body: {"div#the_body", fn content -> 
#   ExCrawlzy.Utils.text(content)
#  end}
}

{:ok, %{body: body}} = ExCrawlzy.parse(fields, html_content)

You can parse using a direct shortcut of one parser from ExCrawlzy.Utils module, a tuple {Module, :function} or directly a function that must be called when the field it's parsed

Want more

The solution it's there, but i know that you will need more, maybe more organization in the case that you have a big map to fill of data, so for have more organization i added the module ExCrawlzy.Client.Json that helps to define directly your crawler in a module and just call your YourModule.crawl/1 and will call, parse and return your data, the implementation it's easy

First let's define the module

defmodule ExampleCrawler do
  use ExCrawlzy.Client.Json

  add_field(:title, "head title", :text)
  add_field(:body, "div#the_body", :text)
  add_field(:inner_field, "div#the_body div#inner_field", :text)
  add_field(:inner_second_field, "div#inner_second_field", :text_alt)
  add_field(:number, "div#the_number", :text)
  add_field(:exist, "div#the_body div#exist", :exist)
  add_field(:not_exist, "div#the_body div#not_exist", :exist)
  add_field(:link, "a.link_class", :link)
  add_field(:img, "img.img_class", :img)

  def text_alt(sub_doc) do
    ExCrawlzy.Utils.text(sub_doc)
  end
end

If you check the code, in this cases you can define as callback a function from ExCrawlzy.Utils or a directly defined function in your module, the function text_alt/1 it's defined in the module, the crawler will check automatically if it's a defined function in module or a parser from utils module

And then just use it

site = "https://example.site"
{:ok, data} = ExampleCrawler.crawl(site)

List of elements

Let's supose that you're crawling a store, i want's to list the products in your crawl, you can define this crawler using the module ExCrawlzy.Client.JsonList

defmodule ExampleCrawlerList do
  use ExCrawlzy.Client.JsonList

  list_size(2)
  list_selector("div.possible_value")
  add_field(:field_1, "div.field_1", :text)
  add_field(:field_2, "div.field_2", :text)
end

This module defines 2 new definition macros list_selector/1 and you need define parent selector, the one that has the list of elements and list_size/1 will define how many elements of the list will take when it's parsing

Example of the html pattern of the list

<div class="parent_class">
  <div class="child_class">
    ...content
  </div>
  <div class="child_class">
    ...content
  </div>
  <div class="child_class">
    ...content
  </div>
</div>

And follows the same rules as the first crawler module

site = "https://example_list.site"
{:ok, data} = ExampleCrawlerList.crawl(site)

Adding http clients

Talking about security, a lot of sites detect robots (crawlers) on the calls and retrieves a forbidden response or other kind fo responses to ensure just real traffic incomes, to this, the library has pre-coded some simulated clients that helps to avoid some robot-detectors, but in case that your site needs other specific browser request headers, you can define it

site = "https://example.site"
clients = [
  [
    {"referer", "https://your_site.com"},
    {"user-agent", "Custom User Agent"}
  ]
]

{:ok, content} = ExCrawlzy.crawl(site, clients)

Or use the macro add_browser_client/1 in your crawler module sharing a list of tuples

defmodule ExampleCrawler do
  use ExCrawlzy.Client.Json

  add_browser_client([
    {"referer", "https://your_site.com"},
    {"user-agent", "Custom User Agent"}
  ])
  add_field(:field_1, "div.field_1", :text)
end

Testing

Test it's really easy, it's a http client so the testing it's a http client, you can use tesla for the testing part

First add this line to your test.exs file, must add specifically your module

config :tesla, ExampleCrawler, adapter: Tesla.Mock

This it's a test example, the html it's saved in the priv folder to have more organization, strongly recommended this step

defmodule ExampleCrawlerTest do
  use ExUnit.Case

  import Tesla.Mock

  setup do
    {:ok, content} =
      :your_app
      |> :code.priv_dir()
      |> then(&"#{&1}/test.html")
      |> File.read()

    mock(fn
      %{method: :get, url: "https://example_list.site"} ->
        %Tesla.Env{status: 200, body: content}
    end)

    :ok
  end

  test "list things" do
    site = "https://example_list.site"
    assert {:ok, data} = ExampleCrawlerList.crawl(site)
  end
end

Conclusions

The library it's usefull, easy to implement, a lot of stuff that solves, in a traditional way you need add the library floki that it's a css selector parser, so you can skip that

There it's stuff to get done, this library it's based on Declarative Development so there is a lot of stuff to develop to make more flexible and complete the crawling, like nested crawlers, so all its just a concept right now and im trying to find the time to add more and more stuff to this library and the other stuff that i want to work

Event Based System with Localstack (Elixir Edition): Notifing to SQS when a file its uploaded

Nicol Acosta — Tue, 22 Aug 2023 23:15:03 +0000

Configure SQS

Easy! if you read the past post "Uploading files to S3 with PresignedURL's" you know how to configure in the docker-compose.yml file, so add SQS to the localstack init its simple

The git diff its this

  localstack:
    image: localstack/localstack:latest
    ports:
      - 4566:4566
    environment:
      # own env vars
      BUCKET_NAME: files
+     BUCKET_QUEUE: new_files_queue
+     QUEUES: general_events_queue,example_queue
      # service env vars
      DEFAULT_REGION: us-west-2
      AWS_ACCESS_KEY_ID: test
      AWS_SECRET_ACCESS_KEY: test
-     SERVICES: s3
+     SERVICES: s3,sqs
      DISABLE_CORS_CHECKS: 1
      PROVIDER_OVERRIDE_S3: asf
      S3_SKIP_SIGNATURE_VALIDATION: 1
    volumes:
      - ./.localstack:/var/lib/localstack
      - ./init_localstack.sh:/etc/localstack/init/ready.d/init_localstack.sh

Full code here

And add this diff of the localstack_init.sh file

...
echo "########### Setting resource names as env variables ###########"
LOCALSTACK_DUMMY_ID=000000000000

+ guess_arn_for_sqs() {
+     local QUEUE_NAME=$1
+     echo "arn:aws:sqs:${DEFAULT_REGION}:${LOCALSTACK_DUMMY_ID}:$QUEUE_+ NAME"
+ }
+ 
+ create_queue() {
+     local QUEUE_NAME_TO_CREATE=$1
+     awslocal sqs create-queue\
+         --region $DEFAULT_REGION\
+         --queue-name $QUEUE_NAME_TO_CREATE
+ }
+ 
+ echo "########### Creating upload file event SQS ###########"
+ create_queue $BUCKET_QUEUE
+ BUCKET_QUEUE_ARN=$(guess_arn_for_sqs $BUCKET_QUEUE)
+ 
+ echo "########### Creating queues in SQS ###########"
+ IFS=','
+ read -ra Queues <<< "$QUEUES"
+ for q in "${Queues[@]}";
+ do
+   create_queue $q
+ done
+ 
echo "########### Create S3 bucket ###########"
awslocal s3api create-bucket\
    --region $DEFAULT_REGION\
...
      ]
    }'

+ 
+ echo "########### Set S3 bucket notification configurations ###########"
+ aws --endpoint-url=http://localhost:4566 s3api put-bucket-notification-configuration\
+     --bucket $BUCKET_NAME\
+     --notification-configuration  '{
+                                       "QueueConfigurations": + [
+                                          {
+                                            "QueueArn": "'"$BUCKET_QUEUE_ARN"'",
+                                            "Events": ["s3:ObjectCreated:Put"]
+                                          }
+                                        ]
+                                      }'
+ 
echo "########### List S3 bucket ###########"
awslocal s3api list-buckets
+ 
+ echo "########### Get S3 bucket notification configurations ###########"
+ aws --endpoint-url=http://localhost:4566 s3api get-bucket-+ + notification-configuration\
+     --bucket $BUCKET_NAME

Full code here

And add this config to the config.exs

aws_host = System.get_env("AWS_HOST")
aws_region = System.get_env("AWS_REGION")
aws_port = System.get_env("AWS_PORT")

config :ex_aws, :sqs,
  scheme: "http://",
  region: aws_region,
  host: aws_host,
  port: aws_port,
  base_queue_url: "http://#{aws_host}:#{aws_port}/000000000000/",
  new_files_queue: System.get_env("AWS_SQS_NEW_FILES_QUEUE"),
  general_events_queue: System.get_env("AWS_SQS_GENERAL_EVENTS_QUEUE")

In this config we are configuring the whole AWS part and also a personal config that we will use in next steps

Listening the Queues

Now we have SQS Configured, but, who is listening?

To listen a message broker the most used library is broadway, this library helps to create GenServer's that listens a specific queue and process message by message (or by chunks).

The fisrt queue that needs to be listened it's the new files queues.

Basically get's the file data and adds a message to the next queue, see the function BroadwayGeneralEvents.insert_message(data)

defmodule EventsArqBackend.QueueWorkers.BroadwayNewFileEvents do
  @moduledoc false
  use Broadway

  alias Broadway.Message
  alias EventsArqBackend.QueueWorkers.BroadwayGeneralEvents

  def start_link(_opts) do
    {module, opts} = producer_module()
    options = opts ++ [queue_url: queue_url()]

    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {module, options}
      ],
      processors: [
        default: []
      ]
    )
  end

  @impl true
  def handle_message(_processor, %Message{data: data} = message, _context) do
    decoded_data =
      case Jason.decode!(data) do
        %{"Message" => message} -> Jason.decode!(message)
        message -> message
      end

    %{"Records" => records} = decoded_data

    Enum.each(records, fn
      %{
        "eventName" => event_name,
        "eventSource" => "aws:s3",
        "eventTime" => event_time,
        "s3" => %{
          "bucket" => %{
            "name" => bucket_name
          },
          "object" => %{
            "eTag" => entity_id,
            "key" => object_key,
            "size" => object_size
          }
        }
      }
      when event_name in ["ObjectCreated:Put", "ObjectCreated:Post"] ->
        data = %{
          bucket_name: bucket_name,
          entity_id: entity_id,
          object_key: object_key,
          object_size: object_size,
          inserted_at: event_time
        }

        BroadwayGeneralEvents.insert_message(data)

      _ ->
        :ok
    end)

    Message.update_data(message, fn _data -> decoded_data end)
  end

  defp queue_url,
    do:
      "#{Application.get_env(:ex_aws, :sqs)[:base_queue_url]}#{Application.get_env(:ex_aws, :sqs)[:new_files_queue]}"

  defp producer_module, do: Application.get_env(:events_arq_backend, :broadway)[:producer_module]
end

And the other queue listener

defmodule EventsArqBackend.QueueWorkers.BroadwayGeneralEvents do
  @moduledoc false
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    {module, opts} = producer_module()
    options = opts ++ [queue_url: queue_url()]

    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {module, options}
      ],
      processors: [
        default: []
      ]
    )
  end

  @impl true
  def handle_message(_processor, %Message{data: data} = message, _context) do
    decoded_data =
      case Jason.decode!(data) do
        %{"Message" => message} -> Jason.decode!(message)
        message -> message
      end

    IO.inspect(decoded_data, label: "*****************")

    # do something with the data
    # send notification

    Message.update_data(message, fn _data -> decoded_data end)
  end

  def insert_message(data), do: SqsClient.add_message_to_queue(queue_name(), data)

  defp queue_name, do: Application.get_env(:ex_aws, :sqs)[:general_events_queue]
  defp queue_url, do: "#{Application.get_env(:ex_aws, :sqs)[:base_queue_url]}#{queue_name()}"
  defp producer_module, do: Application.get_env(:events_arq_backend, :broadway)[:producer_module]
end

So... 2 queues for the same event? why?

Exist a reason, normally all the queues that you use in sqs are the .fifo queues, and the queue for the s3 must be a default queue (no-.fifo-queue) and that's the reason that i created 2 queues, a regular queue for s3 events and a .fifo queue for the other ones

The full code of this changes it's here

In the next post we will:

Create a Channel on Phoenix
Integrate Phoenix Channels to a React project
Broadcast to all connected clients for some update

See you in the next posts

API Prototypes with dbb and my first attemp to join to OpenSource World

Nicol Acosta — Tue, 04 Jul 2023 00:29:15 +0000

The history

As Developers our main purpuse it's... solve problems, and a few times (sarcastic!) we have limited time to solve this problems and another few times (more sarcastic!!) we need build some kind of CRUD or something to store data

It's our basic day by day, create a new schema and the CRUD, some frameworks includes some tools to generate an schema and CRUD files, but another more hasn't this kind of tools and you need to create all by hand

With this escuses xD i decided to build... something... so i had this idea to build an universal API just for experimenting and then the ideas appear, the idea of an universal API it's something that sounds not much optimized so why not use as a quick api, something to test or an early stage product

The tool, dbb

With this purposes i created dbb, the name hadn't explanation, it's just an inspiration from bb8, sounds nice and sticky

All the core it's on the config file, that it's a simple json file

// example config.json
{
  "schemas": [
    {
      "name": "users",
      "fields": {
        "name": "string",
        "age": "number",
        "male": "boolean"
      }
    },
    {
      "name": "products",
      "fields": {
        "name": "string",
        "expiration": "datetime"
      }
    }
  ]
}

This tool it's a easy to configure API that works instantly, with defined schemas in the config file, all dynamic, no migrations, just a server restart and changes are ready to use

And the deployments it's even more easy, it's a public docker image, easy config with env vars

# example docker-compose.yml
version: '3.4'
services:
  prod:
    image: nicolkill/dbb:latest
    depends_on:
      - postgres
    ports:
      - 4001:443
    volumes:
      - ./config.json:/app/config.json    # important add the volume
    environment:
      PORT: 443
      ALLOWED_SITES: "*"
      CONFIG_SCHEMA: config.json             # must match with volume
      PHX_SERVER: true
      SECRET_KEY_BASE: example_SECRET_KEY_BASE

      # db config
      POSTGRES_USERNAME: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DATABASE: dbb_test_prod
      POSTGRES_HOSTNAME: postgres

  postgres:
    image: postgres:13.3-alpine
    environment:
      POSTGRES_DB: postgres
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_HOST_AUTH_METHOD: trust
    ports:
      - 5432:5432

Usage

Like all API's, exist a basic usage on how to use it, the basic routes and operations are

GET /:schema - get the list of records
GET /:schema/:id - get an individual record by id
POST /:schema - save the record to database, using the body
PUT /:schema/:id - updates the record data using the body (overrides the whole data)
DELETE /:schema/:id - deletes the record data

If you are interested on the project you can go to the github page and check the full docs

Limits

As a creator, i never tested this API on stress or performance, so i don't know how much will support, i know the basic Elixir API performance, maybe on a simple VPS will run with 1k/2k calls per hour, enough to a early stage product, on a Cluster will run even better so...

I think that serves it's purpose, save time

The future

At the moment it's an basic API CRUD, but in next movements i'll add an events configuration to call to a webhook when some record it's added/modified/deleted, this it's like a essential part to consider this project as "more" than a simple dynamic CRUD

Also i have more stuff that i want to add to the project and all it's added to the Roadmap

Event Based System with Localstack (Elixir Edition): Uploading files to S3 with PresignedURL's

Nicol Acosta — Thu, 09 Feb 2023 23:40:17 +0000

Introduction

If you are reading this maybe you know what its Phoenix, but let me explain a few things first:

Localstack: Its a local version of AWS, yes, S3, SQS, SNS, DynamoDB and more all available from in your local machine

We will create an a platform that uses a ReactJS Frontend that calls the api in Phoenix and Uploads files to S3 and then notifies the file upload to a SQS Queue and then we process the messages and notifies using PhoenixChannels to all the connected frontends

Complex? sounds like but its not a big deal, lets start it!

The real work

First steps

You need to create your base, an a Frontend and a Backend in Phoenix, maybe you have it so lets move to the important part

You need Docker and Compose to use this examples, strongly recommend because makes easy with almost 0 config on the Localstack side, so all the code must assume that you have your project dockerized and configured with compose

Adding Localstack

This is the main config on the docker-compose.yml file

  localstack:
    image: localstack/localstack:latest
    ports:
      - 4566:4566
    environment:
      # own env vars
      BUCKET_NAME: files
      DEFAULT_REGION: us-west-2
      AWS_ACCESS_KEY_ID: test
      AWS_SECRET_ACCESS_KEY: test
      # service env vars
      SERVICES: s3
      DISABLE_CORS_CHECKS: 1
      PROVIDER_OVERRIDE_S3: asf
      S3_SKIP_SIGNATURE_VALIDATION: 1
    volumes:
      - ./.localstack:/var/lib/localstack
      - ./init_localstack.sh:/etc/localstack/init/ready.d/init_localstack.sh

And this is the init_localstack.sh file content, a unique thing about localstack its that you can move all strings like an aws-cli tool, also the container deletes all the content and config once the container stops, so the script file must create all the resources that you need from Localstack

Basically the init file creates the bucket and add cors to ensure the calls must be allowed

#!/bin/bash

echo "########### Create S3 bucket ###########"
awslocal s3api create-bucket\
    --region $DEFAULT_REGION\
    --bucket $BUCKET_NAME\
    --create-bucket-configuration LocationConstraint=$DEFAULT_REGION

echo "########### Adding cors bucket ###########"
awslocal s3api put-bucket-cors\
    --bucket $BUCKET_NAME\
    --cors-configuration '{
      "CORSRules": [
        {
          "AllowedHeaders": ["*"],
          "AllowedMethods" : ["HEAD", "GET", "POST", "PUT", "DELETE"],
          "AllowedOrigins" : [
            "http://localhost:4000",
            "http://localhost:3000"
          ],
          "ExposeHeaders": []
        }
      ]
    }'

echo "########### List S3 bucket ###########"
awslocal s3api list-buckets

In this examples we will use a feature of aws named presigned url's that its a url with signatures that will allow to the read/write from any caller, to ensure your browser can reach the Localstack container, must add this line to your /etc/hosts file

127.0.0.1 localstack

Adding AWS to Phoenix

We will use the library ex_aws to manage all the AWS resources

Add this to your mix.exs

{:ex_aws, "~> 2.1"},
{:ex_aws_s3, "~> 2.0"}

And add this to your config files (adjust in dev.exs and runtime.exs)

config :ex_aws, :s3,
  scheme: "http://",
  region: System.get_env("AWS_S3_REGION"),
  host: System.get_env("AWS_S3_URL"),
  port: System.get_env("AWS_S3_PORT"),
  bucket: System.get_env("AWS_S3_BUCKET")

Generating the Presigned URL

This function will generate the Presigned URL

  def presigned_url_upload(object_key, opts \\ []) do
    get_presigned_url(:put, object_key, opts)
  end

  defp get_presigned_url(http_method, object_key, opts) do
    # this its for a configuration issue with localhost
    opts =
      if is_prod() do
        opts
      else
        opts
        |> Keyword.put(:virtual_host, is_prod())
        |> Keyword.put(:bucket_as_host, true)
      end

    ExAws.S3.presigned_url(
      config(),
      http_method,
      s3_bucket(),
      object_key,
      opts
    )
  end

Full code here

and sharing the file

  # on some controller

  # 30 mins
  @presigned_upload_url_max_age 60 * 30
  # 10 MB
  @presigned_upload_url_max_file_size 10 * 1_000_000

  @presigned_url_opts [
    ["starts-with", "$Content-Type", ""],
    {:expires_in, @presigned_upload_url_max_age},
    {:content_length_range, [1, @presigned_upload_url_max_file_size]}
  ]

  def get_presigned_url(conn, _) do
    {:ok, url} = S3Client.presigned_url_upload("some_file_name", @presigned_url_opts)

    conn
    |> put_status(200)
    |> json(%{url: url})
  end

Uploading files from Frontend

First lets add the form component, something simple

        <input
          type="file"
          onChange={handleFileChange}
        />
        {uploaded &&
        <span className="uploaded">
          Uploaded
        </span>
        }

And upload it when a file its selected, this function example must check the if its a valid file, then get the presigned url and upload to this presigned url using fetch on a put method

  const [uploaded, setUploaded] = useState(false);

  const handleFileChange = async (e) => {
    if (!e.target.files) {
      return;
    }

    // get the presigned url
    let response = await api.getPresignedUrl();

    if (response.status < 200 && response.status > 300) {
      // show error getting presigned url
      return;
    }

    const file = e.target.files[0];
    const {url} = response.body;

    // uploading the file with put request to the presigned url
    response = await fetch(url, {
      body: file,
      method: "PUT"
    });

    if (response.ok && response.status >= 200 && response.status < 300) {
      setUploaded(true);
      setTimeout(setUploaded.bind(this, false), 5000); // will remove the 
    }
  };

And its all, you are pushing files to s3, the config its more extense than the real code but, a lot of times integrate some library or platforms to our code its more config than realworld code

The full code of this project its here and also includes a project structure front/back in same repo with docker compose config, just check the Makefile to see the available commands

In the next post we will:

Add SQS Queues and add a Worker per Queue using Broadway
Configure S3 Notification to SQS Queue When a file its uploaded
Post Messages to a Queue using Elixir

Part 2 here!

See you in the next posts

The REAL implementation of Firebase Cloud Messaging on the Backend side

Nicol Acosta — Tue, 18 Oct 2022 05:25:58 +0000

Why?

If you want to implement push notifications all the firebase docs tells you that you need use the new http v1.1 api and the old one would be deprecated in a near future because this old api isnt generating more api keys (source here), so, use the new one its critic to new apps

If your backend isnt in the list of available api libraries or you just want to implement a direct without have all the another stuff that is on google apis libraries, this its your post

Prerequisites

Create your project on firebase console
Go to :
- Project Settings
- Service Accounts tab
- Generate new Private key and download the file

The core problem

If you already seen the send message to multiple devices in the post to fcm api url you can see that its needed an Bearer api token to authorize your notification and the problem its here, htf! can i create an api token, what its the page and here its the hard part, isnt in any place

Dont give me problems, give me solutions

Its simple but google wants a dude to read his docs, dont worry here's a summary, you need implement an jwt cycle in your backend and use the access token and refresh when expires, and yes, espires shortly, every hour exactly

the first step its, go to the Api Credentials tab in the Cloud Console

Change to your project in top, its important!

Your Firebase project already has created an a OAuth2.0 client, so you just need open it and see their data, you need 2 things

Add this domain to the Redirect URI's https://developers.google.com/oauthplayground
Copy the Client ID
Copy the Client Secret (Still count as one!)

And this its the secret!, open this the Google OAuthPlayground (the same link that you add it to redirect uri's)

What its this?

Its that, a playground to use OAuth2.0, check the instructions to see what its happening

In the page, open the settings panel (on your right! big gear icon) and check the box that says "Use your own OAuth credentials" and yes, here add your Client ID and Client Secret, then in the left in the bottom add the google fcm permission url

https://www.googleapis.com/auth/firebase.messaging

And click in Authorize APIs blue button, this will send you to a classic auth with google screen and you can allow or deny, obviously, click on allow

In the next screen all tokens its setted, just click in the Exchange authorization code for tokens blue button and will get the access_token and refresh_token (and more data) and and then will jump to the next screen, but you can go back, just click in this button

Go back and press the Refresh access token gray button, this its your destiny, your goal request example, you just need save the refresh token, client id and client secret and replicate in your backend the call to the api and its all, now you have all to send this precious push messages from your backend in a more easy way

Here is an implementation on Elixir

defmodule Gooogle.Auth.Jwt do
  @tesla_client Tesla.client([
                  {Tesla.Middleware.BaseUrl, "https://oauth2.googleapis.com"},
                  Tesla.Middleware.FormUrlencoded
                ])

  defp oauth_refresh_token, do: Application.get_env(:my_app, :firebase)[:refresh_token]
  defp oauth_client_id, do: Application.get_env(:my_app, :firebase)[:oauth_client_id]
  defp oauth_client_secret, do: Application.get_env(:my_app, :firebase)[:oauth_client_secret]

  @spec refresh_token() :: map()
  def refresh_token() do
    body = %{
      client_id: oauth_client_id(),
      refresh_token: oauth_refresh_token(),
      client_secret: oauth_client_secret(),
      grant_type: "refresh_token"
    }

    %{status: 200, body: body} = Tesla.post!(@tesla_client, "/token", body)

    Jason.decode!(body)
  end
end

FAQ's

According some google docs, the refresh token never expires
You can save on redis with timeout in memory or wait to another response that isnt an 200 and refresh in the moment

htf = how to freak