DEV Community: Lionel Marco

YAML for resume ATS sharing

Lionel Marco — Tue, 11 Jun 2024 13:20:31 +0000

Why not, Is there something better ?

Acronyms, yes the first time that we face of with one acronym we need to google it.

What means ATS: Applicant Tracking System, nowadays talent hiring company need to process many candidates as possible and to do that uses software that read resumes written in pdf, doc or html files.

But this formats were designed with the aim of presenting data, they have a lot of information about fonts type, font size, colors, margins, positions, etc.. and are not well designed for storing data, therefore are difficult to parse and extract information from them.

Many times we are facing with the button [Import you resume here], but after doing it we can saw how the system fail to import relevant topics.

Then we search the web for "How to write an ATS-friendly resume" and found a lot of opinionated recipes, but we need a infallible "Canonical" way.

The need of standardization appear.

So we need a human legible format, we need to separate data from presentation, we need an agnostic way to share our resumes.

What means YAML ?

Basically YAML is markup language, ....(you can google it if need more info).

Why choose yaml, because between all human legible format (txt, csv, xml, json) we have that some are flat (txt), and have lack of hierarchy (csv), some are verbose (xml), some are complicated for non programmers (json), and some are only for presentation (md).

YAML files does not have strange symbols, in a first glance is similar to a book index, all academic people are familiar with a book index, and is widely used in cloud environments.

We can write a yaml file with any standard txt editor, does not need to buy any specific software, all the computers and phones has one installed, also yaml files are easy to validate, search "yaml online validator" and is easy to parse.

The more important thing from a programmer point of view, is that there are software libraries for parsing in every computer language,
thus is easy to write import and export programs.

Until now sound good but how to start ??

Implement a new standard its'nt so easy, There are some things that we can start doing:

First write your resume:

YourCamelCaseName_role.ats.yml

Something like that:

HomerJSimpson_NuclearPlantOperator.ats.yml

Validate it, by example in: YAML validator

Store it somewhere places that has visibility: your GitHub account, your personal Blog.

After this, many companies will become aware of this behavior and will begin to develop other forms of data ingestion.

Also here is where big players company like Linkedin play an important role.

They must implement a feature that let us import and export our resumes in YAML format.
And also implement an ATS, api endpoint to share our resume with others:

www.linkedin.com/ats/HomerJSimpson

I know that not sound easy, but someone has to start, and some day it would be a standard.

Then recruiting-software companies Icims, Taleo, Jobvite, SmartRecruit, Bullhorn, etc...
will follow the initiative.

Let's see an example resume in yaml, to improve the likelihood of a widespread adoption I wrote a short version of the Elon Musk resume.

Elon Musk yaml resume

Please share and adopt it.

Arduino, GPS based Heliostat, Suntracker

Lionel Marco — Sat, 27 Apr 2024 22:59:32 +0000

The heliostat is a device that, via a mirror, address the sunlight to a target.
In change, the suntracker always is pointing to the sun.
The heliostat is useful to adress the light to the interior of a house.
The suntracker is useful to setup a solar panel and collect as much energy as be possible. The system shown here can work in any of the two modes.

How know where is the sun ?

Basically, there are two sun tracking alternatives: one is search the sun using light sensors, and the other is calculate the sun position.
The first alternative could be affected by clouds and rain, and for the second, we need a algorithm that takes in count the complexity of celestial mechanics. Every one has its pros and cons, this implementation uses the second method.

So to calculate the sun position a "C" language library is used:

NREL's Solar Position Algorithm (SPA)

The description say: "This algorithm calculates the solar zenith and azimuth angles in the period from the year -2000 to 6000, with uncertainties of +/- 0.0003 degrees based on the date, time, and location on Earth."

The device that can give us those needed parameters is a GPS, it can calculate datetime, location and altitude.
Yes GPS can give time, compared with a RTC (real time clock) they have many advantages, first it rely on a constellation of satellites, equipped with atomic clocks, so basically we have the precision of an atomic clock in our hands at an affordable price, also unlike of a RTC clock it does not need to be initialized.
Just turn on and in a few second we have date time, altitude and location.

Mechanical configuration

The mechanical configuration chosen is azimuth-elevation, dual axis. The vertical axis control azimuth, and the horizontal axis control elevation. The zero value of azimuth is in the side of the sunrise and 180 degrees in the side of the sunset. The zero value of elevation is when the mirror normal is pointing to the horizon, and 90 when the normal is pointing to up.

The system have a Pin hole to verify the right alignment with the sun.
So before working as heliostat the system need to be run as suntracker, and use the pin hole to test it and align to the sun.

Pointing to a target, the math behind

The target is where we want to address the sun light.
To do that we need to move the mirror at a specific position, this position
is the bisecting ray between the sun and the target.

The next image show how to transform from spherical coordinates (azimuth-elevation) to 3d cartesian (x,y,z).
Where:
"S" is the sun unit vector.
"t" is the target unit vector.
"n" is the mirror unit vector.
"γ" (gamma) is azimuth.
"α" (alpha) is elevation.

The equations translated to code:

The next link point to the lines of source code where the target vector is calculated:

Mirror Orientation Equations

The electronic setup

The system is arranged in two modules one in charge of human machine interface, read gps values, calculate sun position and command the next movement, the other module is in charge of the mechanics, run the motors trough the drivers and listen the zero position career sensors.

The human machine interface is done through telnet via wifi, or with serial trough bluetooth, using Android Serial Bluetooth Terminal

Master module

In order to run the sun position algorithm a 32bit processor is needed, I started with "lolin nodemcu esp8266" but sudenly have not enough space and then move to "Node MCU ESP32S". Also change the partition table to:4MB no OTA

Useful links for partition table:
https://espressif-docs.readthedocs-hosted.com/projects/arduino-esp32/en/latest/tutorials/partition_table.html
https://robotzero.one/arduino-ide-partitions/

Slave module

The second module is an Arduino Mega 2560 with RAMPS 1.6 shield.
The shield have connected two MKS expansor driver that control two MKS 6600 Drivers for the steps motors, and two optics end stops taken from a 3D printer.

The first module feed with commands the second trough an I2C connection. The GPS can be connected either to the Slave or to the Master. Trough the serial ports.

There isn't a wiring diagram because the connection are straightforward, also in the web there are a lot of tutorial of how to connect a GPS or I2C.

The code

There are two modules, the ESP 32 work as a master and the Mega 2560 work as slave:

ESP32

Arduino Mega 2560

If it code was useful to you and you build one of these,better than my wood an L shape steel, please give me a comment with the links to the images.
Thanks

Elixir Cowboy Websocket - Querying IOT devices by latitude longitude proximity

Lionel Marco — Fri, 23 Sep 2022 20:50:10 +0000

In our problem purpose statement we have
many devices with a known latitude an longitude location, and their constantly inform any change in their position.

Our objetive is send a message to the devices that are near to a given location.
Or another use will be if one device need to know who are the nearby devices inside a given radius.

Possible use cases

Send or retrieve messages from-to nearby connected devices
can be applied in many use cases:

Taxi cab fleet

Supouse that in a taxi cab fleet a new passenger request a trip,
they give us the pickup point, the system must to offer the trip to the closest cabs. Then just need to broadcast only to the cabs in the surrounding area.

Sensors mesh

In a sensor mesh one of the devices read an abnormal lecture value and want to know the values of the nearby devices inside a given radius.

Delivery food

One of the motorcycle of delivery company broke a tire and
then request help assistance to near partners, for them to distribute the food.

Dependencies
The application
Space Zonification
Filtering by distance
Devices Registry
Socket Handler

Dependencies

Showing the dependencies is the classic way to start an Elixir post.

  defp deps do
    [
      {:cowboy, "~> 2.4"},
      {:plug_cowboy, "~> 2.0"},
      {:jason, "~> 1.3"},
      {:geocoder, "~> 1.1"},
      {:geocalc, "~> 0.8"}
    ]
  end

The application

The application is the core where all the involved processes will be started.

defmodule IotDevicesApp do
  use Application
  def start(_type, _args) do

    children = [
      Plug.Cowboy.child_spec(
        scheme: :http,
        plug:  SocketApp.HttpRouter,
        options: [dispatch: dispatch(),port: 5000]
      ),
      Registry.child_spec(
        keys: :duplicate,
        name: Registry.IotApp
      )
    ]

    opts = [strategy: :one_for_one, name: IotDevicesApp.Application]
    Supervisor.start_link(children, opts)
  end

  defp dispatch do
    [
      {:_,
        [
          {"/ws/[...]", IotDevicesApp.SocketHandlerLocation, []},
          {:_, Plug.Cowboy.Handler, {SocketApp.HttpRouter, []}}
        ]
      }
    ]
  end  
end

Space Zonification

In our app we have thousand of devices spread along a State, County or City, so measure the distance with all the device is not performant, a coarse zonification is nedeed and then apply a fine graded filtering using distance.

First solution that come to mind is a zonificate by city, using a reverse geocoding service like
Google Maps
or OpenStreetMap

The geocoder library fit our needs:
Hexdocs
Github

def getZone({lon, lat}) do 
  {:ok, coord} = Geocoder.call({lat, lon}, provider: Geocoder.Providers.OpenStreetMaps)
  # OpenStreetMaps is only for testing, it is limited to 1 request by second.
  "#{coord.location.state}, #{coord.location.city}"
end

Testing the function :


times_square = {-73.985130,40.758896}
getZone(times_square)
Return : "New York, New York"

Filtering by distance

A distance function must to be implemented, some question are nedeed:

How far away are the neighbors ? 1km, 5km, 10km
Which level of accuracy we need 1m, 100m, 500m
Need for speed ?

Two options are presented:
A) The fast way work in flat earth calculating the "Euclidean distance" in degrees, 1 degrees at equator 69 miles

def distance({cx, cy}, {nx, ny}) do
  :math.sqrt(:math.pow(abs(cx - nx), 2) + :math.pow(abs(cy - ny), 2))
end

B) Correct and may be a litle slow way, due to many calculus are involved, apply ‘haversine’ formula from GeoCalc

def distance({cx, cy}, {nx, ny}) do  
  Geocalc.distance_between([cx, cy], [nx, ny])
end

Devices Registry

It is necessary to store an keep a record ot all the connected devices, the right place is a Registry working as publisher subscriber. The PID and the location of every new connection is stored in the Registry, it will be showed bellow, in the Socket Handler.

Elixir Registry
The Registry is based in ETS (Erlang Term Storage).
Erlang Term Storage

The socket Handler

The start point to understand socket handler is the documentation:Cowboy WebSocket

defmodule IotDevicesApp.SocketHandlerLocation do
  @behaviour :cowboy_websocket
  @degrees  0.5
  @meters   1500

  # When a client open the connection, it send to us his latitude and longitude
  # javascript: websocket = new WebSocket("ws://192.168.1.109:5000/ws?lon=-73.985130&lat=40.758896");
  def init(request, _state) do   

    vars = URI.query_decoder(request.qs) |> Map.new()    

    lon_float = String.to_float(Map.get(vars, "lon", 0.0))
    lat_float = String.to_float(Map.get(vars, "lat", 0.0))
    {:cowboy_websocket, request, {lon_float, lat_float}, %{idle_timeout: :infinity}}
  end


  # Called once the connection has been upgraded to Websocket.

  def websocket_init(location) do
    IO.puts("-------------websocket_init--------------")
    # this function have different PID that the init() function    
    # for that reason the PID registration is here
    key = getZone(location)
    Registry.IotApp
    |> Registry.register(key, location)

    {:ok, {key, location}}
  end

  # websocket_handle called when:
  # javascript: websocket.send(JSON.stringify({action:"updlocation", lon:11, lat:22})
  # javascript: websocket.send(JSON.stringify({action:"status", ...status})
  # javascript: websocket.send(JSON.stringify({action:"wakeupneighbors"})

  def websocket_handle({:text, json}, state = {key, location}) do
    IO.puts("-------------websocket_handle--------------")

    IO.inspect(state, label: "state")

    payload = Jason.decode!(json)

    case payload["action"] do
      "status" -> {:reply, reportStatus(payload["status"]), state}
      "updlocation" -> updateLocation(self(), key, {payload["lon"], payload["lat"]})
      "wakeupneighbors" -> {:reply, wakeupNeighbors(key, location), state}
      _ -> {:ok, state}
    end
  end


  # Here broadcast happen
  # websocket_info: Will be called for every Erlang message received.
  # Trigger from  wakeupNeighbors()
  def websocket_info(info, state) do
    {:reply, {:text, info}, state}
  end

  #----------- Next are utilities functions ---------------------

  def reportStatus(status) do
    IO.puts("-------------reportStatus--------------")
    IO.inspect(status ,label: "status")
    # Here put your logic
    {:ok, str} = Jason.encode(%{"action"=>"ack"})
    {:text, str}
  end

  def updateLocation(pid, oldkey, newlocation) do
    IO.puts("-------------updateLocation--------------")
    IO.inspect({pid, self()}, label: "PID")
    # this function have same PID that the websocket_init() function
    # IO.inspect(newlocation ,label: "newlocation")
    newkey = getZone(newlocation)

    IO.inspect(Registry.values(Registry.IotApp, oldkey, self()), label: "old location")
    Registry.unregister(Registry.IotApp, oldkey)
    Registry.register(Registry.IotApp, newkey, newlocation)
    IO.inspect(Registry.values(Registry.IotApp, newkey, self()), label: "new location")
    {:ok, str} = Jason.encode(%{"action" => "ack"})
    {:reply, {:text, str}, {newkey, newlocation}}
  end


  def wakeupNeighbors(key, location) do
    # this function have same PID that the websocket_init() function
    IO.puts("-------------wakeupNeighbors--------------")
    {:ok, message} = Jason.encode(%{"action" => "report_status"})

    Registry.IotApp
    |> Registry.dispatch(key, fn entries ->
      for {pid, neighbor} <- entries do
        IO.inspect(neighbor, label: "neighbor location")

        if pid != self() && isNeighbour(location, neighbor) do
          IO.inspect(pid, label: "      send msg to")
          Process.send(pid, message, [])
        end
      end
    end)

    {:ok, str} = Jason.encode(%{"action" => "ack"})
    {:text, str}
  end

  def isNeighbour(location, neighbor) do
    # @degrees > distanceEuclidean(location,neighbor)
    @meters > distanceHaversine(location, neighbor)
  end

  def distanceEuclidean({cx, cy}, {nx, ny}) do
    :math.sqrt(:math.pow(abs(cx - nx), 2) + :math.pow(abs(cy - ny), 2))
  end

  def distanceHaversine({cx, cy}, {nx, ny}) do
    Geocalc.distance_between([cx, cy], [nx, ny])
  end

  def getZone({lon, lat}) do
    {:ok, coord} = Geocoder.call({lat, lon}, provider: Geocoder.Providers.OpenStreetMaps)

    # OpenStreetMaps is only for testing, it is limited to 1 request by second.

    key = "#{coord.location.state}, #{coord.location.city}"
    IO.inspect(key, label: "getZone")
    key
  end

end

Elixir Plug Cowboy, WebSockets Broadcasting to React

Lionel Marco — Wed, 31 Aug 2022 20:10:05 +0000

WebSockets is the most effective way to push content to web clients, The connection between both sides start with a HTTP handshaking and then is upgraded to WebSockets, establishing a full duplex channel where both sides can send and receive data.

Thank to this, we can push data to client, without the client requesting it. What is very useful for stock quotes list, game server score tables, crypto currency prices, bets odd display, or whatever real time data.

Basically we have one emitter and many consumers.

In this article it will be implemented a stock quotes list,
where the stock values will be sent to every client in a specific time interval.

The structure will consist of:

Quotes store: where the stock prices will be stored, and also it will simulate price changes.
Scheduler: It will trigger the broadcast to every some time.
Clients registry: Every client have a PID, we need it as address where to send the updates.
Socket handler: Handle the clients communications.

In the client side is a React app that receive and display a quotes table.

1-Dependencies
2-Quote Store
3-The Scheduler
4-Socket Handler
5-The application
6-React Client

1) Dependencies

Showing the dependencies is the classic way to start an Elixir post.

  defp deps do
    [
      {:cowboy, "~> 2.4"},      
      {:plug_cowboy, "~> 2.0"},
      {:jason, "~> 1.3"}, 
    ]
  end

2) Quote Store

A Elixir Agent is used in order to store the stock quote prices and simulate price changes.

Elixir Agent

How the documentation say: It are useful when need to share or store state that must be accessed from different processes or by the same process at different points in time.

defmodule Quote do

  @derive {Jason.Encoder, only: [:symbol, :name, :price, :share, :arrow, :dif, :difp]}
  defstruct [:symbol, :name, :price, :share, :arrow, :dif, :difp]


  def new({symbol, name, price, share}) do # I love C++
    %Quote{symbol: symbol, name: name, price: price, share: share}
  end
end


defmodule QuoteStore do
  use Agent

  def initial_values() do
    [
      {"BRK.B", "Berkshire Hathaway", 289.30, 637},
      {"JPM", "JPMorgan Chase & Co.", 115.52, 342},
      {"BAC", "Bank of America", 34.41, 308},
      {"WFC", "Wells Fargo & Company", 44.30, 168},
      {"MS", "Morgan Stanley", 88.30, 151},
      {"HSBC", "HSBC", 31.41, 128},
      {"AXP", "American Express", 157.33, 118},
      {"GS", "The Goldman Sachs Group", 340.18, 116},
      {"AAPL", "Apple Inc. ", 167.23, 368},
      {"MSFT", "Microsoft Corporation", 276.44, 340},
      {"GOOG", "Alphabet Inc.", 114.77, 200},
      {"AMZN", "Amazon.com,", 133.62, 500},
      {"TSLA", "Tesla, Inc", 889.36, 120},
      {"NVDA", "NVIDIA Corporation ", 171, 512}
    ]
    |> Enum.map(&Quote.new(&1))
  end

  def start_link(quotes) do    
    IO.puts("QuoteStore.start_link, length(quotes): #{length(quotes)}")
    Agent.start_link(fn -> quotes end, name: __MODULE__)
  end

  def values do
    Agent.get(__MODULE__, & &1)
  end

  def update do    
    values = Enum.map(Agent.get(__MODULE__, & &1), fn q -> newValue(q) end)
    Agent.update(__MODULE__, fn _ -> values end)
    values
  end

  def newShare(value), do: value * (101..105 |> Enum.random()) / 100
  def newPrice(value), do: value * (90..110 |> Enum.random()) / 100

  defp newValue(q) do
    newprice = newPrice(q.price)
    newshare = newShare(q.share)

    dif = Float.round(newprice - q.price, 2)

    # IO.puts(dif)
    difp = dif / q.price * 100

    # 2% threshold
    arrow =
      cond do
        difp > 2 -> "u"
        difp < -2 -> "d"
        true -> "e"
      end

    %Quote{
      symbol: q.symbol,
      name: q.name,
      price: newprice,
      share: newshare,
      arrow: arrow,
      dif: dif,
      difp: difp
    }
  end
end

3) The Scheduler

The scheduler will be in charge of trigger the broadcasting process. Every specific time interval it send a message to all the PID stored in the Registry. The Registry is based in ETS (Erlang Term Storage). The PID of every new connection is stored in the Registry, it will be showed bellow, in the Socket Handler.

Elixir Registry
Erlang Term Storage

defmodule Scheduler do
  def start(interval) do
    IO.inspect(self(), label: "Scheduler Start")
    :timer.apply_interval(interval, __MODULE__, :send_quotes, [])

    {:ok, self()}
  end

  def send_quotes() do
    IO.inspect(self(), label: "Broadcast messages from")

    Registry.StockTickerApp
    |> Registry.dispatch("stock_ticker", fn entries ->
      for {pid, _} <- entries, do: send(pid, :broadcast, [])
    end)
  end
end

5) Socket Handler

The start point to understand socket handler is the documentation:Cowboy WebSocket

defmodule StockTickerApp.SocketHandlerTicker do
  @behaviour :cowboy_websocket
  require Logger

  # Not run in the same process as the Websocket callbacks.
  def init(request, _state) do
    {:cowboy_websocket, request, nil, %{idle_timeout: :infinity}}
  end

  # websocket_init: Called once the connection has been upgraded to Websocket.
  def websocket_init(state) do
    Registry.StockTickerApp |> Registry.register("stock_ticker", {})
    str_pid = to_string(:erlang.pid_to_list(self()))
    IO.puts("websocket_init: #{str_pid}")

    stime = String.slice(Time.to_iso8601(Time.utc_now()), 0, 8)
    {:ok, json} = Jason.encode(%{time: stime, quotes: QuoteStore.update()})
    {:reply, {:text, json}, state}
  end

  # websocket_handle: Will be called for every frame received
  # Can be used to retrieve info about some specific quote ex: "TSLA"
  # but in this case we don't needed.
  # def websocket_handle({:text, json}, state) do
  # end

  # websocket_info: Will be called for every Erlang message received.
  # the scheduler send messages to every PID stored in the Registry
  def websocket_info(:broadcast, state) do
    stime = String.slice(Time.to_iso8601(Time.utc_now()), 0, 8)
    {:ok, json} = Jason.encode(%{time: stime, quotes: QuoteStore.update()})
    {:reply, {:text, json}, state}
  end
end

3) The application

The application is the core where all the involved processes will be started.

defmodule StockTickerApp do
  use Application

  def start(_type, _args) do
    children = [
      {QuoteStore, QuoteStore.initial_values()},
      %{id: Scheduler, start: {Scheduler, :start, [5000]}},
      Plug.Cowboy.child_spec(
        scheme: :http,
        plug: StockTickerApp.Router,
        options: [dispatch: dispatch(), port: 5000]
      ),
      Registry.child_spec(
        keys: :duplicate,
        name: Registry.StockTickerApp
      )
    ]

    opts = [strategy: :one_for_one, name: StockTickerApp.Application]
    Supervisor.start_link(children, opts)
  end

  defp dispatch do
    [
      {:_,
       [
         {"/ws/[...]", StockTickerApp.SocketHandlerTicker, []}
       ]}
    ]
  end
end

6) React client

In the client side is a React app, where the Material UI library will provide us of a nice look and feel user interface. Material UI

There is many ways to implement WebSocket in React, this is the more simple,
just the raw implementation, without any external library.

//socket.js
const wsHost = "ws://192.168.1.109:5000/ws/chat";
export var websocket = new WebSocket(wsHost);

A parent child component relation will be entablished.
Two components, a Dashboard and a quotes_table.

Dashboard

The Dashboard, will handle the socket connection and store the received quotations in their state.

import React from 'react';
import Typography from '@material-ui/core/Typography';
import QuotesTable from "./quotes_table";
import {websocket} from "./socket.js";

class Dashboard extends React.Component {

    constructor(props) {
      super(props); 
      this.state = { status: "Disconnected", quotes:[]};  
    }

    setStatus = (evt,value) => {
      console.log("setStatus",value);//,Object.keys(evt));
      //console.log("isTrusted",evt.isTrusted);
      this.setState({status: value});
    }

    onMessage= (evt) => { 
      console.log("    websocket.onmessage:");
      var data = JSON.parse(evt.data);
      console.log("Data", data);
      this.setState({quotes: data.quotes, time : data.time});

    }

    componentDidMount() {

      console.log("Dashboard componentDidMount:");
      websocket.onopen = (evt)  => this.setStatus(evt,"Open");
      websocket.onclose = (evt)  => this.setStatus(evt,"Close");
      websocket.onerror = (evt) => this.setStatus(evt,"Error");
      websocket.onmessage = (evt)  => this.onMessage(evt);      

    }

    render() {        

       console.log("DashBoard: render", this.state.quotes.length);       
       return (

          <div  style={{margin: "0 auto",padding:"5px", border: "2px solid gray", maxWidth: "35%" , display: "flex", flexDirection: "column"}} >

           <QuotesTable quotes={this.state.quotes} style={{border: "1px solid yellow"}} />
           <Typography >
             Last update:{this.state.time}  ,Status: {this.state.status}           
           </Typography>  

         </div>
)}}

export default Dashboard;

Quotes table

An the end we have the QuotesTable, where all the stock quotes are showed with arrow and colors to show a well knowed visual interpretation.

import React from 'react';
import Table from '@material-ui/core/Table';
import TableBody from '@material-ui/core/TableBody';
import TableCell from '@material-ui/core/TableCell';
import TableHead from '@material-ui/core/TableHead';
import TableRow from '@material-ui/core/TableRow';
import ArrowDropUpIcon from '@material-ui/icons/ArrowDropUp';
import ArrowDropDownIcon from '@material-ui/icons/ArrowDropDown';

import { makeStyles } from '@material-ui/core/styles';

const useStyles = makeStyles({
  customTable: {
    "& .MuiTableCell-sizeSmall": {
      padding: "2px 6px 2px 6px "     
    },
    "& .MuiTableCell-head": {      
      fontWeight: "bold",    
      backgroundColor: "#00a0a0"
    }
    ,
    "& .MuiTableRow-root": {
      height: "40px"
    }
    ,
    "& .MuiTableRow-root:hover": {
      backgroundColor: "#c0c0c0"
    }
  },
});

function drawArrowIcon(d){
  if (d==="u") return (<ArrowDropUpIcon  style={{fill:"green", transform: "scale(1.5)"}}/>);
  if (d==="d") return (<ArrowDropDownIcon  style={{fill:"red", transform: "scale(1.5)"}}/>);
}

function getColor(d){
  if (d==="u") return ("green");
  if (d==="d") return ("red");
  return "black";
}

export default function QuotesTable(props) {

  const { quotes } = props;

  console.log("QuotesTable: render", quotes.length);

  const classes = useStyles();

  return (

      <Table classes={{root: classes.customTable}}
      style={{ width: "auto", tableLayout: "auto", border: "2px solid gray" }} size="small" >
        <TableHead  >
          <TableRow >
            <TableCell >Symbol</TableCell>
            <TableCell align="right">Price</TableCell>
            <TableCell align="right">Move</TableCell>
            <TableCell align="right">Dif</TableCell>
            <TableCell align="right">Dif%</TableCell>
            <TableCell align="right">Share</TableCell>
          </TableRow>
        </TableHead>
        <TableBody>
          {quotes && quotes.map((row) => (
            <TableRow key={row.symbol} onClick={() => { console.log("Open info for:",row.symbol); }} >
              <TableCell align="left">{row.symbol}</TableCell>
              <TableCell align="right">{row.price.toFixed(2)} </TableCell>
              <TableCell align="center" >{drawArrowIcon(row.arrow) }</TableCell>
              <TableCell align="right" style={{color: getColor(row.arrow)}} >{row.dif.toFixed(2)}</TableCell>
              <TableCell align="right" style={{color: getColor(row.arrow)}} >{row.difp.toFixed(2)}</TableCell>
              <TableCell align="right">{row.share.toLocaleString(undefined, {minimumFractionDigits: 2,maximumFractionDigits: 2})}</TableCell>
            </TableRow>
          ))}
        </TableBody>
      </Table>

  );
}

Working with Geospatial data in Elixir-Postgrex, Mimic Node-PgPromise behaviour.

Lionel Marco — Tue, 30 Aug 2022 19:22:12 +0000

A client required me to rewrite a Geospatial REST API from Node-Express to Elixir-Cowboy.The Front-end was wrote in React, it interchange json with the API. The geospatial information is stored in Postgis, it is a spatial database extension for Postgres.

The communication between the front-end and the back-end is using json, so does not need to load or unload any binary object from the database. Just only need the string representation of geometries objects, the classis WKT Well Know Text Representation

The data manager of Elixir is Ecto, but working with it would imply create many views and some time, work with binaries.The app have 5 modules where every one have more or less 10 sql queries, so rewrite almost 50 well writed and optimized queries, wasn't on my mind.

Searching the web I found an interesting article, that was my main source of inspiration : Elixir-Without-Ecto

1-Introduction
2-Working with geospatial data
3-Configure and Start
4-Postgrex Query Helpers
5-Named Parameters

1) Introduction

Query Helpers

The Back-end use PgPromise, so the option was create some function helpers to mimic his behaviour:


# For queries that must to return only 1 row,
# raise if are 0 or more than 1
row = DB.one!(sql, data_map)

# For queries that must to return only 1 row,
# return error if are 0 or more than 1
{status, row} = DB.one(sql, data_map)

# For queries that return 0 or multiple rows
rows = DB.any!(sql, data_map)

# Get affected rows for update, insert, delete, 
affected = DB.affected!(sql, values)

# For update, insert, delete, store procedures, functions
# return a map with operation status "fail" or "succes"
status =  DB.status(sql, values)

Working with named parameters

Another Pg-Promise very useful thing is work with named parameters stored in json.
With it we avoid to use list values with reference numbers: $1, $2.
In elixir Json is stored in maps, named parameters let us write things like the next:

#iex>
sql= "INSERT into books(name,author,pages)
            VALUES($(name),$(author),$(pages));"

values= %{ "name"=>"Lord of the Rings", "author"=>"John Ronald Reuel Tolkien", "pages"=>500}

DB.status(sql, values)

%{status: "success"}

Obtain columns as maps

Postgrex return every row as a list with values, if we run the next query:

# iex> 
sql = "Select 1 as id, "Jhon" as name"

Postgrex.query!(sql,[])

Will be get:

%Postgrex.Result{
  columns: ["id", "name"],
  command: :select,
  connection_id: 5849,
  messages: [],
  num_rows: 1,
  rows: [[1, "Jhon"]]
}

But sometimes is more usefull to have a list of maps, where every column have an access key:

# iex> 
sql = "Select 1 as id, 'Jhon' as name;"

DB.one!(sql,%{})

Will give me:

# iex> 
{:ok, %{"id" => 1, "name" => "Jhon"}}

2) Working with geospatial data

Some examples with geometries:

Select:


 def selectField(id) do

    sql ="select  json_build_object(
                'id', id::text,
                'name', name,
                'centroid', st_AsGeoJSON(st_centroid(boundary))::json ,
                'area', to_char(ST_Area(boundary::geography)/10000, 'FM999999999.00') || 'ha',
                'boundary', st_AsGeoJSON(boundary)::json ) as field FROM fields WHERE id=$(id);"

    DB.one!(sql, %{"id"=>id})

  end

The result will be a list of rows where every row is a map ready to encode and send:

[%{"field"=>
            %{
              "id" => "14",
              "name" => "Field name here",
              "area" => "28.18ha",
              "boundary" => %{
                "coordinates" => [
                  [
                    [-63.69112612, -31.394299467],
                    [-63.70054825, -31.393974797],         
                    [-63.69087755, -31.39156248],
                    [-63.69094669, -31.39416452],
                    [-63.69112612, -31.394299467]
                  ]
                ],
                "type" => "Polygon"
              },
              "centroid" => %{
                "coordinates" => [-63.695616434, -31.392755962],
                "type" => "Point"
              }    
            }
  }          
]

Insert a Polygon:

Load geometries to the database:

 def insertField() do

    json_polygon= %{
                    "coordinates" => [
                      [
                        [-63.694052, -31.389509],
                        [-63.694352, -31.395883],
                        [-63.691005, -31.396286],
                        [-63.690061, -31.389839],
                        [-63.694052, -31.389509]
                      ]
                    ],
                    "type" => "Polygon"
                  }

    values= %{"name"=>"Soy Beans", "polygon"=>json_polygon}

    sql = "INSERT into fields(name,boundary)
            VALUES($(name),ST_GeomFromGeoJSON($(polygon)));"

    DB.status(sql, values)
  end

Will return :


%{status: "success"}

Update some points:

def updateLocations() do

 values = %{ "id"=>[10,11,12], "long"=> -63.694052, "lat"=> -31.389509, "srid"=>4326}
 sql= "UPDATE locations SET point=ST_SetSRID( ST_Point($(lon),$(lat)),$(srid)) WHERE id=ANY($(id));"
 IO.inspect(DB.affected(sql, values), label: "Rows affected")
end

After running this code the output will return :


Rows affected : 3

3) Configure and Start

The first thing to do is configure the access to Postgres and then start the process.

Create file config/config.exs where the
the name key is really important, it will be used has reference to the running process.

Config file

#file: config/config.exs
import Config
config  :postgrex, config: [
                            name: :mydb, 
                            hostname: "localhost", 
                            username: "user",
                            password: "pasw", 
                            database: "gis",
                            port: 5432
                            ]

Start the process

defmodule AppGis.Application do
  use Application

  @impl true
  def start(_type, _args) do
    dbconfig = Application.fetch_env!(:postgrex, :config)

    children = [
      {Postgrex, dbconfig},
      Another process...
    ]

    opts = [strategy: :one_for_one, name: Appgis.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

4) Postgrex Query Helpers

defmodule DB do

  defmodule DBError do

    defexception message: "", plug_status: 422
  end

  # For arguments values inside map
  def query!(sql, values_map) when is_map(values_map) do
    {sql, values} = replace(sql, values_map)
    Postgrex.query!(:mydb, sql, values)
  end

  # For arguments values in a list
  def query!(sql, values) do
    Postgrex.query!(:mydb, sql, values)
  end

  # For queries that return 0 or multiple rows
  def any!(sql, values) do
    case query!(sql, values) do
      %{rows: []} ->
        nil

      %{rows: rows, columns: columns} ->
        Enum.map(rows, fn row -> mapify(columns, row) end)

      _ ->
        raise DBError, message: "returned multiple rows"
    end
  end

  # For queries that must to return only 1 row,
  # raise if are 0 or more than 1
  def one!(sql, values) do
    case query!(sql, values) do
      %{rows: []} ->
        raise(DBError, message: "returned 0 rows")

      %{rows: [row], columns: columns} ->
        mapify(columns, row)

      _ ->
        raise DBError, message: "returned multiple rows"

    end
  end

  # For queries that must to return only 1 row,
  # return error if are 0 or more than 1
  def one(sql, values) do
    case query!(sql, values) do
      %{rows: []} -> {:error, nil}
      %{rows: [row], columns: columns} -> {:ok, mapify(columns, row)}
      _ -> {:error, nil}
    end
  end

  # For update, insert, delete
  def affected!(sql, values), do: query!(sql, values).num_rows

  # For update, insert, delete
  # with operation status
  def status(sql, values) do
    case query!(sql, values).num_rows do
      0 -> %{status: "fail"}
      _ -> %{status: "success"}
    end
  end

  # Tranform from [[cols...]] and [[values...]] to [%{"col"=>value}]
  defp mapify(columns, row) do
    columns
    |> Enum.zip(row)
    |> Map.new()
  end


end

5) Named Parameters

In order to work with named parameters, a function is writted.

Basically receive:

values = %{"firstName"=>"Jhon", "phone"=>"(555) 456 789 522"}
sql= "INSERT into clients(name,phone) VALUES($(name),($(phone));"

And transform to:

values = ["Jhon","(555) 456 789 522"]
sql= "INSERT into clients(name,phone) VALUES($1,$2);"

A thing to have in count is that Postgrex need to be feeded with te right amount of arguments, not one more, not one less.
If a value is a map, it is encoded to json.

def replace(sql, mapvalues) do
  # Pattern to find keys $(keyname)
  pattern = ~r/\$\(([^)]+)\)/
  # Get unique keys  
  keys = Enum.uniq(Enum.map(Regex.scan(pattern, sql), fn [_ | [tail]] -> tail end))
  # Build values list
  values = Enum.reverse(Enum.reduce(keys, [], fn k, lk -> [encode(Map.get(mapvalues, k)) | lk] end))

  # Replace key names with their associate index
  sql =
    String.replace(sql, pattern, fn match ->
      key = String.replace(match, ["$", "(", ")", " "], "")
      idx = Enum.find_index(keys, fn x -> x == key end) + 1
      # IO.puts("$#{idx}")
      "$#{idx}"
    end)  
  {sql, values}
end


def encode(v) when is_map(v) do
  {:ok, str} = Jason.encode(v)
  str
end

def encode(v), do: v
end

JSON validation for Cowboy REST API - Mimic Celebrate Joi in Elixir

Lionel Marco — Mon, 15 Aug 2022 23:30:00 +0000

When we work in a REST API one of the first things to do is validate incomming data.

Suddenly I must to rewrite a REST API from Javascript to Elixir and find that some of the existing library does not feed my basic needs of simplicity and less code possible.
All of us who have a Node-Express background, have used Celebrate and Joi in order to validate the presence of key, their data type and their stored value.

In this article I will show you how to implement in Elixir a schema validator, trying to mimic a (minimal) functionality of Celebrate Joi.

1-Plugs and Midlewares
2-Route-Schema Validation
3-Function Clause Matching
4-Json Schema Validation

1) Plugs and Midlewares

There is a common architecture between Node-Express and Elixir-Cowboy, Express have middlewares and Cowboy have Plugs.
In both cases these are used for routing, body parsing, user authentication verification, cross-origin resource sharing, data logging, data validation, metrics request, etc..

How the documentation say :"An Express application is essentially a series of middleware function calls." Using middlewars

For example: A classic Node-Express midlewares chain

//file: app_router.js
// import boiler plate ...
app.use(cors(cors_configuration)); //CORS
app.use(express.json());//Json body parsing

app.use(isAuth);//user jwt authentication verification

app.use(function (err, req, res, next) { //  error handling
  if (err.name === "UnauthorizedError") {
    res.status(401).send("invalid token...");
  } else {
    next(err);
  }
});

app.use(attachUserId);// add extra information about the current user

app.use((req, res, next) => { // some debug 
  console.log("Method",req.method);  
  console.log("Url", req.originalUrl);
  console.log("Body:",req.body);  
  return next();
});

The same happens in Elixir-Cowboy, where plugs are chained one after other, like this:

# file: app_router.ex
# use/import boiler plate ...

  plug Corsica,  cors_configuration # CORS  
  plug :match  
    plug Plug.Telemetry, telemetry_configuration #  metrics request  
    # body parsing
    plug Plug.Parsers,    parsers: [:json],    pass: ["application/json"],    json_decoder: Jason

    # Json validation
    plug Plug.RouterValidatorschema, on_error: &__MODULE__.on_error_fn/2

    # user jwt authentication verification
    plug Plug.Authenticator, on_error: &__MODULE__.on_error_fn/2
    # add extra information about the current user
    plug Plug.AttachUserId, on_error: &__MODULE__.on_error_fn/2
    # some debug 
    plug Plug.PrintUrlMethod

  plug :dispatch

Celebrate is a middleware, the equivalent in Elixir is a Plug, let start writing a plug:

The next link is the started point to understand and learn about how to write a plug, I recommend read it and then come here: The Plug Specification

There are two ways to define it: function plugs and module plugs, I will choose the module plugs way.

2) Route-Schema Validation

Has programers we want to write the minimal possible lines of code. Bassically reduce all to a schema definition and then apply in our route.

Schema:

photo_book_schema = [
                      qty: {:number, 1, 200},
                      hardcover: {:boolean},
                      colors: {:options, ["bw", "color", "sepia"]},
                      client_name: {:string, 6, 20},
                      client_phone: {:string, 8, 25, ~r/^[\d\-()\s]+$/}
                    ]

Route:

plug(:match)
plug(Plug.RouterValidatorschema, on_error: &__MODULE__.on_error_fn/2)
plug(:dispatch)

post "/create/", private: %{validate: photo_book_schema} do
  status = Orders.create(conn.body_params)
  {:ok, str} = Jason.encode(%{status: status})
  send_resp(conn |> put_resp_content_type("application/json"), str)
end

def on_error_fn(conn, errors, status \\ 422) do
  IO.inspect(errors, label: "errors")
  {:ok, str} = Jason.encode(%{status: "fail", errors: errors})
  send_resp(conn |> put_resp_content_type("application/json"), str) |> halt()
end

Anxious for the code ?
Before to go deeper in the code, we learn a bite about Function Clause Matching.

2) Function Clause Matching

One of the powerfull tools of Elixir is pattern matching. It is applied in many places, thanks to that, we write less lines of code compared with another languages.

In this case is applied in function clauses, Elixir let us to write multiple definitions for the same function, every definitions is called a clause.

See the next example:

defmodule Discount do
  def calculate(type, price) do
  case type do
      :normal -> price * 1
      :high   -> price * 0.8
      :low    -> price * 0.9      
  end  
  end  
end

Instead we can write:

defmodule Discount do
  def calculate(:normal, price), do: price
  def calculate(:high, price), do: price * 0.8
  def calculate(:low, price), do: price  * 0.9  
end

Where the "case" statement is moved outside of the function and the virtual machine take the decision of which must to apply.All definition refer to the same function "calculate" with arity 2.

2.1) Tips.

Some tips when use function clause matching.

Group:

Always group clauses of the same function together,

Don't mix with others:


defmodule FooBar do
  def foo(:a), do: 'a'
  def foo(:b), do: 'b'
  def bar(arg), do: IO.puts arg
  def foo(:c), do: 'c'
  def foo(arg), do: IO.puts arg
end

Will receive a compiler warning:

warning: clauses with the same name and arity (number of arguments) should be grouped together, "def foo/1" was previously defined.

Correct way:


defmodule FooBar do
  def foo(:a), do: 'a'
  def foo(:b), do: 'b'
  def foo(:c), do: 'c'
  def foo(arg), do: IO.puts arg
  def bar(arg), do: IO.puts arg
end

Order:

The declaration order really matter:
Don't do:


defmodule FooBar do
  def foo(arg), do: arg
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
end

Will receive a compiler warning:

warning: this clause for foo/1 cannot match because a previous clause always matches

Correct way, sort for selective weight:

defmodule FooBar do  
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
  def foo(arg), do: arg
end

Match all:

Alway write a final match all clause.

Sometimes we receive a inexpectable argument value and then a error is triggered:
(FunctionClauseError) no function clause matching

Don't do:


defmodule FooBar do
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
end

Correct way:


defmodule FooBar do
  def foo(:b), do: 'b'  
  def foo(:c), do: 'c'
  def foo(_), do: "Something happen"
end

4) Schema Validation

The Plug.Router provide a way to passing data between routes and plugs

The Plug.conn struct

A analogy to better undertand the Plug.conn and the plug chain is think in a train locomotive with wagons, when every plug is a station where we load or unload data and the wagons are the fields of the struct.

The Plug.conn struct have a field named private, it will be seted with a key and a value.


  post "/create/" , private: %{validate: photo_book_schema} do  
     .....
  end

When the plug find the validate key inside the private field, the validation process start, calling the function validate_schema()


defmodule Plug.RouterValidatorSchema do

  def init(opts), do: opts

    def call(conn, opts) do
    case conn.private[:validate] do
      nil -> conn
      schema -> validate_schema(conn,schema, opts[:on_error])
    end
  end

The validate_schema() walk over the schema list and validate the keys inside conn.body_params

If all validations are successful returns an empty map.
Otherwise when validation fail, call the given on_error callback with a list of the collected errors.
The errors list have the shape [ %{"key1"=> "error"}, %{"key2"=> "error"}]


  defp validate_schema(conn, schema, on_error) do
  errors = Enum.reduce(schema, [], validate_key(conn.body_params))

  if Enum.empty?(errors) do
    conn
  else
    on_error.(conn, %{errors: errors})
  end
end

defp validate_key(data) do
  fn {key, test}, errors ->
    skey = Atom.to_string(key)
    cond do
      not Map.has_key?(data, skey) -> [%{skey => "missing key"} | errors]      
      is_nil(data[skey]) -> [%{skey => "value is nil"} | errors] # or just return errors if value can accept nil
      true ->
        case is_t(test, data[skey]) do
          :ok -> errors
          {:invalid, reason} -> [%{skey => reason} | errors]
        end
    end
  end
end

The validate_key(data) return a parameterized function that be in charge of apply every test to the key associated data.
The is_t(test,value) will perform the test and here is where we take advantage of function clause matching, the function test for boolean, number, string, string patterns, options, and is very easy add a new test for any desired data type.


defp is_t({:boolean}, value) do
  if (not is_boolean(value)) do
    {:invalid, "not is boolean"}
  else
   :ok
  end
 end

 defp is_t({:number}, value) do
  if (not is_number(value)) do
    {:invalid, "not is number"}
  else
   :ok
  end
end

defp is_t({:number, vmin, vmax}, value) do
  cond do
    not is_number(value)   ->  {:invalid, "not is number"}
    value <= vmin ->  {:invalid, "min value"}
    value > vmax ->  {:invalid, "max value"}
    true ->:ok
  end
end


 defp is_t({:string}, value) do
    if is_bitstring(value) do
      :ok
    else
      {:invalid, "not is string"}
    end
 end

 defp is_t({:string, lmin, lmax}, value) do
     cond do
       not is_bitstring(value)   ->  {:invalid, "not is string"}
       String.length(value) <= lmin ->  {:invalid, "min length"}
       String.length(value) > lmax ->  {:invalid, "max length"}
       true ->:ok
     end
 end

 defp is_t({:string, lmin, lmax, reg}, value) do
  cond do
    not is_bitstring(value)   ->  {:invalid, "not is string"}
    String.length(value) <= lmin ->  {:invalid, "min length"}
    String.length(value) > lmax ->  {:invalid, "max length"}
    not Regex.match?(reg,value) -> {:invalid, "regex not match"}
    true ->:ok
  end
end

defp is_t({:options, alloweds}, value) do
  cond do
    value not in alloweds -> {:invalid, "not allowed"}
    true ->:ok
  end
end

# ---------------------------
# Final match :
# ---------------------------
defp is_t(_value,_test) do
  {:invalid, "test match fail"}
end

How to add flask, gunicorn, packages to a distroless docker container

Lionel Marco — Thu, 25 Nov 2021 19:57:13 +0000

The flask, gunicorn, nginx, stack is a powerfull toolchain.There are many tutorial about how to dockerizing it, but it is different. We will do it from a distroless way, with a security concern point of view.

1-Distroless Concept
2-How to work with theys
3-Adding Packages
4-Security Scanning

1) Distroless Concept

What are distroless containers ?

Think in slim down linux distribution, where are removed all unused things.
Don't have shell or package manager, come without: apt, apk, pip, bash, etc..
When remove all unnecessary things get a bonus of size saving, halving or even more the final size.So is more faster to upload it. From a security concern we obtain a small surface attack area.

To get more info about it: Google Container Tools

The concept is clear: Just only add the application runtime and the accesories libraries.

2) How to work with theys

The gcr.io/distroless/python3 is a batteries not included toy.

The classic recipe resolve all in one saucepan:

FROM python:3.6
ADD . /app
WORKDIR /app
RUN pip install flask gunicorn
EXPOSE 8000
CMD ["gunicorn", "-b", "0.0.0.0:8000", "app"]

Now the build process change, is more like a crispy home fries recipe, first boil the potatoes in water, then drain and cool, add spices and finally send to oven.

First we will take a base system python:3.6-slim and call build-env, over this it will be added the needed packages. Second move the packages and our application to the distroless image gcr.io/distroless/python3.

To do it, here is where multistage build come to help us: Docker: Use multi-stage builds

2.2) Package installing

The packages to install:

#file:requirements.txt
flask==1.1.4
gunicorn>19

To build our images, two stages are needed:

# file: flask_gunicorn_distroless.dockerfile

FROM python:3.6-slim  AS build-env
WORKDIR /app

# First install packages to avoid re-run pip install 
COPY requirements.txt ./
RUN pip install --disable-pip-version-check -r requirements.txt --target /packages

COPY ./versions.py /app
COPY ./app.py /app
COPY ./standalone.py /app

FROM gcr.io/distroless/python3
WORKDIR /app
COPY --from=build-env /app /app
COPY --from=build-env /packages /packages
#instead of COPY --from=build-env /packages /usr/local/lib/python/site-packages

ENV PYTHONPATH=/packages
#instead of ENV PYTHONPATH=/usr/local/lib/python/site-packages

CMD ["standalone.py"]

Some Tips:

Inside distroless the nexts command does not works:

RUN mkdir /app
RUN cp ./source /app

Instead must to use:

WORKDIR /app it performs mkdir and cd implicitly.
COPY ./app /app will create target directory

2.2) See what is inside

The image gcr.io/distroless/python3 come with some preinstalled packages, just to see what is inside of it, a small python file is writed.

# file version.py
import sys
import os
import argparse
from pathlib import Path

print("Versions:------------------")
parser = argparse.ArgumentParser()
parser.add_argument('folder', type=str, help='The folder to list.', nargs='?', default=os.getcwd())

#--------------------------
import flask
print("Flask Version:",flask.__version__)

import gunicorn 
print("Gunicorn Version:",gunicorn.__version__)

#--------------------------
print("Getcwd: ",os.getcwd()) 
print("Sys.path count:",str(len(sys.path)))
print('\n'.join(sys.path))


def listFolders(str_path: str):
    dir_path=Path(str_path).expanduser()
    print(" Packages:------------------")
    try:
       for path in dir_path.iterdir():            
         if path.is_dir(): 
            print(path)
    except PermissionError:
       pass

def main(args):
    print('Folder:'+args.folder)
    listFolders(args.folder)


if __name__ == "__main__":
    main(parser.parse_args())
    #listFolders('/packages')

In the dockerfile change the last line to:

CMD ["versions.py", "/usr/lib/python3.9"]

Rebuild the image:

docker build --tag flask_gunicorn_distroless --file flask_gunicorn_distroless.dockerfile .

And execute:

docker run flask_gunicorn_distroless

In the console will be listed the paths and the pre installed packages.

:docker run flask_gunicorn_distroless
Versions:------------------
Flask Version: 1.1.4
Gunicorn Version: 20.1.0
Getcwd:  /app
Sys.path count: 5
/app
/packages
/usr/lib/python39.zip
/usr/lib/python3.9
/usr/lib/python3.9/lib-dynload
Folder:/usr/lib/python3.9
Folders:------------------
/usr/lib/python3.9/multiprocessing
/usr/lib/python3.9/zoneinfo
/usr/lib/python3.9/sqlite3
/usr/lib/python3.9/distutils
/usr/lib/python3.9/venv
/usr/lib/python3.9/http
/usr/lib/python3.9/xml
/usr/lib/python3.9/asyncio
/usr/lib/python3.9/collections
/usr/lib/python3.9/wsgiref
/usr/lib/python3.9/encodings
/usr/lib/python3.9/pydoc_data
/usr/lib/python3.9/ctypes
/usr/lib/python3.9/xmlrpc
/usr/lib/python3.9/test
/usr/lib/python3.9/email
/usr/lib/python3.9/importlib
/usr/lib/python3.9/curses
/usr/lib/python3.9/html
/usr/lib/python3.9/json
/usr/lib/python3.9/logging
/usr/lib/python3.9/dbm
/usr/lib/python3.9/lib-dynload
/usr/lib/python3.9/urllib
/usr/lib/python3.9/concurrent
/usr/lib/python3.9/unittest
/usr/lib/python3.9/__pycache__

2.3) Running Flask

Just for sanity check, see if our flask application can run:

#file:app.py
from time import gmtime, strftime   
from flask import Flask

application = Flask(__name__)

@application.route("/")
def index():    
    return {'status':'Available','time':strftime("%H:%M:%S", gmtime())  }    

if __name__ == "__main__":    
    application.run(host="0.0.0.0", port=5000,debug=True)

In the dockerfile change the last line to:

CMD ["app.py"]

Rebuild the image:

docker build --tag flask_gunicorn_distroless --file flask_gunicorn_distroless.dockerfile .

And execute:

docker run -p 80:5000 flask_gunicorn_distroless

In your preferred web browser open the adress: http://localhost, if all is ok, a json will be showed.

2.4) Running Gunicorn

In the classic recipe, the usual way to run Gunicorn is:

CMD ["gunicorn", "-b", "0.0.0.0:8000", "app"]

But now the ENTRYPOINT is /usr/bin/python3.9

So need to run gunicorn from inside python:

Gunicorn Custom Application

The idea is subclass gunicorn.app.base.BaseApplication and overload load_config and load.

# file:standalone.py
from app import application
import multiprocessing
import gunicorn.app.base

def number_of_workers():
    return  (multiprocessing.cpu_count() * 2) + 1

class StandaloneApplication(gunicorn.app.base.BaseApplication):

    def __init__(self, app, options=None):
        self.options = options or {}
        self.application = app
        super().__init__()

    def load_config(self):
        config = {key: value for key, value in self.options.items()
                  if key in self.cfg.settings and value is not None}
        for key, value in config.items():
            self.cfg.set(key.lower(), value)

    def load(self):
        return self.application

if __name__ == '__main__':
    options = {
        'bind': '%s:%s' % ('0.0.0.0', '8000'),
        'workers': number_of_workers(),
    }
    StandaloneApplication(application, options).run()

In the dockerfile change the last line to:

CMD ["standalone.py"]

Rebuild the image:

docker build --tag flask_gunicorn_distroless --file flask_gunicorn_distroless.dockerfile .

And execute:

docker run -p 80:8000 flask_gunicorn_distroless

In your preferred web browser open the adress: http://localhost, if all is ok, a json will be showed.

3) Vulnerability Scanning

We will use Clair to perform a static analysis of vulnerabilities.

1) Run the vulnerabilities database:

docker run -d --name db arminc/clair-db:latest

2) Run the service:

docker run -p 6060:6060 --link db:postgres -d --name clair arminc/clair-local-scan

3) Run the client:

Download from:Clair Scanner

Retrieve the image id:

docker image ls flask_gunicorn_distroless
REPOSITORY                  TAG       IMAGE ID       CREATED         SIZE
flask_gunicorn_distroless   latest    fa824d3ab0a6   3 minutes ago   70.8MB

Scan, passing the loopback our own ip, and the image id:

./clair-scanner_linux_386 --ip=127.0.0.1 fa824d3ab0a6 | grep Unapproved | wc -l

Get 0 vulnerabilities

Compare with the python:3.9-slim

Retrieve the image id:

docker image ls python:3.9-slim
REPOSITORY   TAG        IMAGE ID       CREATED       SIZE
python       3.9-slim   5e714d33137a   5 days ago    122MB

Scan, passing the loopback our own ip, and the image id:

./clair-scanner_linux_386 --ip=127.0.0.1 5e714d33137a | grep Unapproved | wc -l

Get 39 vulnerabilities:

React Color Scale Interpolation

Lionel Marco — Mon, 11 Oct 2021 23:40:52 +0000

Sometimes we have to work with colors, either for labels, icons, bar charts or for a choropleth maps.

It will show a way in which given two colors, the color between them will be calculated, or a scale with many colors will be generated.

For UI components will be use MATERIAL-UI library.

1-Color interpolation
2-Select
3-Slider
4-Full code

1) Color interpolation

Here the more important function is which, given two colors, them
calculate a resulting color. It will be more similar to one or the other,
acording to a conversion ratio, that will vary between 0 and 1.




  const interpolate = (start,end,ratio)=>{    
    const r = Math.trunc(ratio*end[0] + (1-ratio)*start[0])
    const g = Math.trunc(ratio*end[1] + (1-ratio)*start[1])
    const b = Math.trunc(ratio*end[2] + (1-ratio)*start[2])    
    return [r,g,b]
  }

For example, interpolating in the middle of Black and White:

interpolate([0,0,0],[255,255,255],0.5)

We get Gray:

[127,127,127]

2) Color Selector

The MATERIAL-UI library have a Select control, in this case we use to display a list of color and choice one of them.

There are two color selector, one for start and other for end.



          <Select  value={color1}                                    
                   name={'color1'}
                   onChange={ this.onColorChange}
                   renderValue={ showIcon } 
                 >                    
                     { colors.map( e =>  <MenuItem  key={e} value={e}><ColorIcon color={e} /></MenuItem> )}    
          </Select>

The colors come from a previously declared array:



const colors = ['#800080','#FF0000','#FFD700','#00FF00','#006400','#0000FF']; //purple,red,gold,darkgreen,blue

Color Icon

On every choice of our select an icon with their respective color is displayed.
The icon receive a props 'color', basically it is a rectangule filled with the given color.



function ColorIcon({color}) {
 return (
 <SvgIcon   viewBox="0 0 50 20" style={{ width: 50, height:20  }}>     
   <rect   fill={color} x={0} y='0' width={50} height={20}  ></rect>            
 </SvgIcon>
);
};

Show Icon

The select have a 'renderValue' props, wich give the flexibility of show another thing that only text.


 showIcon }



function showIcon(value) {
  return (  <ColorIcon color={value}/> );
}

3) Ratio Slider

The MATERIAL-UI library have a Slider control, the slider will control the ratio of color mix. Varying from 0 to 100, then will be re maped to 0-1.



          <Slider
            value={ratio}          
            onChange={ this.onRatioChange}
            step={10} min={0} max={100}
            valueLabelDisplay="auto"
            marks={[ {value: 0,label: '0%'}, {value: 50,label: '50%'}, {value: 100,label: '100%'}]}                                   
          />

Every change on the slider will update the state and trigger a new render:



  onRatioChange = (event, newValue) => {    
    this.setState(prevState => ({...prevState,ratio: newValue})); 
  };

4) Full code

Next the full code is showed, imports are ommited just for shortness.



function ColorIcon({color}) {
 return (
 <SvgIcon   viewBox="0 0 50 20" style={{ width: 50, height:20  }}>     
   <rect   fill={color} x={0} y='0' width={50} height={20}  ></rect>            
 </SvgIcon>
);
};


function showIcon(value) {
  return (  <ColorIcon color={value}/> );
}

const colors = ['#800080','#FF0000','#FFD700','#00FF00','#006400','#0000FF']; //purple,red,gold,darkgreen,blue

export default class ColorInterpolation extends React.Component {

  constructor(props) {
    super(props);
    this.state = {color1:colors[2],color2:colors[1],ratio:50}      
  };   

  onColorChange = (e) => {    
    const {name,value} = e.target;
    this.setState(prevState => ({...prevState,[name]: value}));  
  };

  onRatioChange = (event, newValue) => {    
    this.setState(prevState => ({...prevState,ratio: newValue})); 
  };  

  render() {   
    //console.log("Render");     
    const {color1,color2,ratio} = this.state;    
    const interpolatedColor=getColor(color1,color2,ratio);

    return (    

      <div style={{  
        maxWidth:'500px',         
        display: "flex" ,                    
        flexDirection: "column", 
        margin:'10px',padding:'20px',
        border: '2px solid grey',
        borderRadius:'4px'
        }}>

        <div style={{               
           display: "flex" ,                    
           flexDirection: "row",      
           alignItems: "center",
           justifyContent: "space-around" }}>

          <Typography> Source: </Typography> 
          <Select  value={color1}                                    
                   name={'color1'}
                   onChange={ this.onColorChange}
                   renderValue={ showIcon } >                    
                     { colors.map( e =>  <MenuItem  key={e} value={e}><ColorIcon color={e} /></MenuItem> )}    
          </Select>    
          <Typography> Target: </Typography>       
          <Select   value={color2}                                    
                    name={'color2'}
                    onChange={ this.onColorChange}
                    renderValue={ showIcon } >                    
                     { colors.map( e =>  <MenuItem  key={e} value={e}><ColorIcon color={e} /></MenuItem> )}
          </Select>    
          </div>  
          <Slider
            value={ratio}          
            onChange={ this.onRatioChange}
            step={10} min={0} max={100}
            valueLabelDisplay="auto"
            marks={[ {value: 0,label: '0%'}, {value: 50,label: '50%'}, {value: 100,label: '100%'}]}
          />
          <div style={{marginTop:'5px', display: "flex" , alignItems: "center", justifyContent: "center" }}>   
            <Typography > Interpolated:</Typography>        
            <Typography > {ratio}% </Typography>        
            <ColorIcon color={interpolatedColor} />
            <Typography >{interpolatedColor}</Typography>
          </div>    
          <div style={{marginTop:'5px', display: "flex" , alignItems: "center", justifyContent: "center" }}>   
            <Typography > Scale:</Typography>        
            { [0,10,20,40,60,80,100].map( (v,i)=> {
              let c =getColor(this.state.color1,this.state.color2,v)
            return <ColorIcon key={i} color={c} />})}
          </div> 
        </div>
    );
}
}

function getColor(c1,c2,ratio)
{

  const interpolate = (start,end,ratio)=>{    
    const r = Math.trunc(ratio*end[0] + (1-ratio)*start[0])
    const g = Math.trunc(ratio*end[1] + (1-ratio)*start[1])
    const b = Math.trunc(ratio*end[2] + (1-ratio)*start[2])    
    return [r,g,b]
  }

  const hexToRgb = (hex) =>   
     [ 
       parseInt(hex.substr(1,2),16),
       parseInt(hex.substr(3,2),16),
       parseInt(hex.substr(5,2),16),
     ];  

  const rgbToHex = (rgb) => '#' + rgb.map(x => {
    const hex = x.toString(16)
    return hex.length === 1 ? '0' + hex : hex
  }).join('');

  const rgbInterpolated = interpolate(hexToRgb(c1),hexToRgb(c2),ratio/100);
  return rgbToHex(rgbInterpolated);
}

React Multi State Submit Button

Lionel Marco — Thu, 07 Oct 2021 20:33:53 +0000

A multi state submit button that inter actuate according API response.

When a form with data is sended, we have two alternatives, do in a blindness way or give the user a feedback of what happen.

We gather information in fields inside a form and of course whe need a submit button.

The benefits of a submit button instead a normal button is that by default their behaviour is tied to ENTER key in mobile or desktop.

In order to give information of the resulting operation a button with five states is implemented:

normal, loading, warning, error, success

Every status explain by self.

For dialog an icons will be use MATERIAL-UI library.

1-Tuning Icons
2-Multistate Button
3-Dialog with Button
4-Flask Dummy route

1) Tuning Icons

The MATERIAL-UI library have a large collection of useful icons.

Three icons are needed:

CheckCircleIcon for submit and save confirmation.
SyncIcon for processing.
SyncProblemIcon for error and warning.

In a normal case to use the icons only need to import them :

import CheckCircleIcon from '@material-ui/icons/CheckCircle';
import SyncIcon from '@mui/icons-material/Sync';
import SyncProblemIcon from '@mui/icons-material/SyncProblem';

But in this case, we need to extract the path to make a fine tunnig, taking from here :

material-ui-icons

Or from the browser using the right click mouse to inspect the element and then copy the PATH from the SVG.

Loading Icon

We will take the path of SyncIcon and give a rotation animation :

//file: /src/controls/SButtonIcons.jsx
//extract

export function LoadingIcon(props) {
  return (
  <SvgIcon   viewBox="0 0 24 24" style={{ width: 24, height:24  }} > 
   <circle fill="#1976d2" cx="12" cy="12" r="10" />        
        <g transform="translate(2.2 2.2) scale(0.8)"   >
        <path        
         d= 'M12 4V1L8 5l4 4V6c3.31 0 6 2.69 6 6 0 1.01-.25 1.97-.7 2.8l1.46 1.46C19.54 15.03 20 13.57 20 12c0-4.42-3.58-8-8-8zm0 14c-3.31 0-6-2.69-6-6 0-1.01.25-1.97.7-2.8L5.24 7.74C4.46 8.97 4 10.43 4 12c0 4.42 3.58 8 8 8v3l4-4-4-4v3z' 
         fill="#FFFFFF" strokeWidth="0" >
         <animateTransform attributeName="transform" type="rotate" from="360 12 12" to="0 12 12" dur="1.5s" repeatCount="indefinite">
        </animateTransform>
        </path>
        </g>

  </SvgIcon>
);
}

Error-Warning Icon

We will take the path of SyncProblemIcon and give a property with the desired fill color.

//file: /src/controls/SButtonIcons.jsx
//extract
export function WarningIcon(props) {
    return (
    <SvgIcon   viewBox="0 0 24 24" style={{ width: 24, height:24  }} > 
     <circle fill={props.color ? props.color :'orange' } cx="12" cy="12" r="10" />        
        <g transform="translate(2.2 2.2) scale(0.8)"   >
        <path    
         d= 'M3 12c0 2.21.91 4.2 2.36 5.64L3 20h6v-6l-2.24 2.24C5.68 15.15 5 13.66 5 12c0-2.61 1.67-4.83 4-5.65V4.26C5.55 5.15 3 8.27 3 12zm8 5h2v-2h-2v2zM21 4h-6v6l2.24-2.24C18.32 8.85 19 10.34 19 12c0 2.61-1.67 4.83-4 5.65v2.09c3.45-.89 6-4.01 6-7.74 0-2.21-.91-4.2-2.36-5.64L21 4zm-10 9h2V7h-2v6z'
         fill="#FFFFFF" strokeWidth="0" >      
        </path>
        </g>
    </SvgIcon>
  );
  }

2) Multistate Button

The button have a status prop with five possible values: normal, loading, warning, error, success.

To use it, just add: <SButton status={status} />, the status is coming from the API response.

Internally the button has a view state, it can be: idle, wait, timeout

At the start the view is idle. When come a response from the API: warning, error, success, the view change to wait. The view handle how many seconds the warning, error or success icons are showed.

After 2 second of wait view, the button view change to timeout restoring the normal button, giving to the user another try. And the fair tale begin again.

//file: /src/controls/SButton.jsx

export default class SButton extends React.Component {
    constructor(props) {
      super(props);
      this.state = {view:'idle'};   
    }


   //Called immediately after updating occurs. Not called for the initial render.
   componentDidUpdate(prevProps, prevState, snapshot) {

      //console.log("SButton componentDidUpdate, props:",this.props.status);

      //Only listen to 'status' change,      
      if (prevProps.status === this.props.status) return; // avoid re call "componentDidUpdate" when view change

      // after firts attemp, change from 'timeout' to 'idle'
      if (this.props.status === 'loading' ) 
      {
        console.log("view: idle");   
        this.setState({view: 'idle'});
      }            

      // ['warning','error','success']
      if (this.props.status === 'warning' ||this.props.status === 'error' || this.props.status === 'success'){          

       this.setState({view: 'wait'});// console.log("view: wait");   
       this.timer = setTimeout(()=> {this.setState({view: 'timeout'})}, 2000);
      }
   }

   componentWillUnmount(){
     clearTimeout(this.timer); // console.log("Button componentWillUnmount");     
    }

   render() {      

      var icon;

      if (this.state.view==='timeout') 
      { //when timeout, set the normal color to light blue 

        icon = <CheckCircleIcon style={{ color: '#1976d2' }}/>
      }
      else //view==[idle or wait], or in first render
      {

      // first render  
      if ( !this.props.status || this.props.status==='normal') {icon = <CheckCircleIcon style={{ color: '#1976d2' }}/> }
      // after submit
      if (this.props.status==='loading' ) {icon = <LoadingIcon/>}
      if (this.props.status==='warning') {icon = <WarningIcon  /> }
      if (this.props.status==='error') {icon = <WarningIcon color={'red' }/> }
      if (this.props.status==='success') {icon = <CheckCircleIcon style={{ color: 'green' }}/> }
      }

      // To avoid re-click when status==='loading'
      // type={this.props.status==='normal'?"button":"submit"}

      return (
        <>
        <IconButton {...this.props}  type={this.props.status==='loading'?"button":"submit"}  > 
          {icon}
        </IconButton>       
        </>               
      );
  }
}

3) Dialog with Submit Button

Putting the button at work. For demonstration purposes, the button has been placed in a dialog that exemplifies the creation of a user.

When the user click the button, an action is dispatched, this submit the data using Axios.The async interaction of the dialogue with the API will be made with Axios and managed by Redux thunk.

The dialog make the API request with Axios, if the API response is 'success' it close by self, if not error advices are showed.

handleSubmit

When the user finish to fill fields, click the button and then one action is dispatched.

//file: /src/client/ClientCreateDlg.jsx
//extract

  const handleSubmit = (e) => {
    console.log("handleSubmit, data:",data);   
    e.preventDefault();  // prevent a browser reload/refresh
    props.dispatch(actClientCreate(data)); 

  };

Actions and Reducer

The main action is actClientsFormApi(data,url) implementing an async Axios request with Redux Thunk, that will be called when need to create, update or delete a client. In this case just will use actClientCreate(data)
Just for demonstration, all three point to the same API route, but in real scenario, every must to have their specific route.

//file: /src/client/ClientsActions.jsx
//extract

//-----------------------
// Form =>  Create, Update, Delete
//-----------------------
export function actClientCreate(d) {return actClientsFormApi(d,"clientsresponse")};
export function actClientUpdate(d) {return actClientsFormApi(d,"clientsresponse")};
export function actClientDelete(d) {return actClientsFormApi(d,"clientsresponse")};

//------------------------------------------------------
function actClientsFormApi(d,url) { 

  return dispatch => {
    dispatch(actClientFormSubmit());// for processing advice msg

    axios.post(process.env.REACT_APP_API_BASE_URL+url,d, {withCredentials: true})
    .then(response => { dispatch(actClientFormResponse(response.data));})
    .catch(error => { dispatch(actClientFormResponse({status:'error',msg:error.message}))})                    

  };
}

export const actClientFormInit = () => ({
  type: 'CLIENT_FORM_INIT'  
});
export const actClientFormSubmit = () => ({
  type: 'CLIENT_FORM_SUBMIT'  
});
export const actClientFormResponse = (response) => ({
  type: 'CLIENT_FORM_RESPONSE',
  payload : response
});

The reducer is very simple.

//file: /src/client/ClientReducer.jsx

const initialState = {
  formStatus : 'normal',
  formMsg: null
};


export default function ClientsReducer(state = initialState,action)
 {
  switch (action.type) {

    case 'CLIENT_FORM_INIT':             
        return {
          ...state,
          formStatus: 'normal',
          formMsg: '',          
        };   
    case 'CLIENT_FORM_SUBMIT':   
        return {
          ...state,
          formStatus: 'loading',
          formMsg: '',          
        };   

    case 'CLIENT_FORM_RESPONSE':    

      return {
        ...state,
        formStatus: action.payload.status,
        formMsg: action.payload.msg,

      };   

    default:  
      return state;
  }
}

The dialog

Working with dialogs, we have an isolated place to do things like create, update or delete items.
We will take the dialog from MATERIAL-UI:

import Dialog from '@material-ui/core/Dialog';

As said before, the dialog interactuate with the API dispatching one action.

props.dispatch(actClientCreate(data));

And then, feed the button status prop with the status coming from the API response.

<SButton status={status} />

The dialog is connected to redux store, listening the operation status from the API.

//file: /src/client/ClientCreateDlg.jsx
//extract
const mapStateToPropsForm = state => ({    
  status:state.clients.formStatus,
  msg:state.clients.formMsg,   
});

export default connect(mapStateToPropsForm)(ClientCreateDlg);

Basically the dialog have a form with just three fields, every change over it update the data state hook.

//file: /src/client/ClientCreateDlg.jsx


function ClientCreateDlg(props){

  const initial = {  name:'',phone:'', mail:'',}; 
  const [data, setData] = useState(initial);  

  //Mount - Unmount  
  useEffect(() => {
    props.dispatch(actClientFormInit());  //componentMount    

   return () => {
    props.dispatch(actClientFormInit());  //componentWillUnmount      
   };
 }, []);

   //componentDidUpdate  status listener  
   // When success, auto close after some time
  useEffect(() => {
    console.log("status:", props.status);
    var timer;
    if( props.status==='success') 
    {
       timer = setTimeout(() => { props.clbkClose()}, 1000);
    }
    return () => clearTimeout(timer);
     }, [props.status]); 


  const handleClose = () => {   //console.log("handleClose");     
      props.clbkClose();
  };

  const handleChange = (e) => {
    const {name,value} = e.target;
    setData(prevState => ({...prevState,[name]: value}));  
  };

  const handleSubmit = (e) => {
    console.log("handleSubmit:");
    console.log("  data:",data);   
    e.preventDefault();  // prevent a browser reload/refresh
    props.dispatch(actClientCreate(data)); 

  };

    const { status, msg } = props; // server API responses   

    var advice = null;         
    if (status === "loading") advice = "Procesing...";    
    if (status === "error") advice =  "Error: " + msg;   
    if (status === "warning") advice =  "Warning: " + msg;      
    if (status === "success") advice = "Data was saved."; 


    return (
        <Dialog onClose={handleClose}  open={true}>
        <div style={{minWidth:'300px', maxWidth:'400px',minHeight:'200px', padding:"2px",display: "flex" ,flexDirection: "column"}}>
        <DialogTitle ><ViewHeadlineIcon  />Create new Client:</DialogTitle>          
        <form onSubmit={handleSubmit} >

        <div style={{minWidth:'50%',boxSizing:'border-box',padding:"2px",display: "flex" ,flexDirection: "column",flexGrow:'1'}}>         
          <TextField name="name"   size="small" placeholder="Name"  onChange={handleChange} />  
          <TextField name="phone"  size="small" placeholder="Phone" onChange={handleChange} />  
          <TextField name="mail"   size="small" placeholder="Mail"  onChange={handleChange} />     
        </div>        

        <div style={{           
           display: "flex" ,                    
           flexDirection: "row",      
           alignItems: "center",
           justifyContent: "space-around" }}> 

        <SButton status={status} />    
        <IconButton  onClick={handleClose}  > <CancelIcon/></IconButton>   
        </div>    

        <Typography  variant='caption' style={{fontWeight:'600',textIndent:'6px'}} noWrap>
          {advice}
        </Typography>                       

        </form>
        </div>
        </Dialog>  

)

};

const mapStateToPropsForm = state => ({    
  status:state.clients.formStatus,
  msg:state.clients.formMsg,   
});

export default connect(mapStateToPropsForm)(ClientCreateDlg);

Opening the dialog

The dialog is open from a main control ClientsMain. A client add button, trigger the dialog opening. Then use conditional rendering to show or hide it.

//file: /src/client/ClientsMain.jsx
class ClientMain extends React.Component {

  constructor(props) {
    super(props);
    this.state = {dlgIsOpen:false}      
  };   

  // Handle Dlg Open
  openClientCreateDlg = () =>{
    this.setState({dlgIsOpen:true});
  }

  // Dlg Close Callback
  clbkDlgClose = () => {
    console.log("clbkDlgClose");
    this.setState({dlgIsOpen:false});
  };
   //----------------------------------

  render() {   
    //console.log("Client Main render");     
    var renderDlg = null;
    if (this.state.dlgIsOpen){
      renderDlg =  <ClientCreateDlg clbkClose={this.clbkDlgClose} />                                              
    }

    return (    
        <>        
        <IconButton onClick={this.openClientCreateDlg} color="primary"> <AddCircleIcon/>Add new Client</IconButton>           
       {renderDlg}
        </>    
    );
}
}

export default ClientMain;

4) Flask response dummy route

In order to simulate the results of the API endpoint, a route with random responses is implemented.


@app.route('/clientsresponse', methods=['POST','GET'])
def clientrandomresponse():

  # Real world
  #  json_content = request.get_json(silent=False)
  #  response = doSomething(json_content)
  #  return response

  # Simulation

  responses = [{ 'status': 'success'},
               { 'status': 'error', 'msg': 'Json required'},
               { 'status': 'error', 'msg': 'Missing field '},
               { 'status': 'warning', 'msg': 'Data validation fail'}]

  time.sleep(1) # Just to show Processing Spinner
  return responses[time.localtime().tm_sec%4]

Conclusion:

Working it that way we achieve a better interaction of the user with the App.
It is important to note that this button can also be applied in inline editing forms.

Get the full code from https://github.com/tomsawyercode/react-multistate-submit-button

React, injecting dialogs with Redux, CRUD dialogs with Axios Flask Api interaction.

Lionel Marco — Thu, 30 Sep 2021 19:26:38 +0000

React, injecting dialogs with Redux, CRUD dialogs with Axios Flask API interaction.

Whether we like it or not the dialogues are an important part of our application.They allow us to perform simple or complicated actions in a place apart from the main content.

Either to avoid complexity or laziness we always try to use the minimum amount of third-party libraries. In this case we will only use the classic MATERIAL UI who will be in charge of generating the dialogue.
MATERIAL UI providee a dialog whith basic behaviour, like: closing whith scape key or when click outside, it can go fullscreen and also darken the page behind.

We will control the opening and closing of the dialogue with Redux. Also the asynchronic interaction of the dialog with the API will be handled by Redux. The dialog make the API request, get the API response and then if is all is ok it close by self, if not error advices are showed.

Bassically reduce all to a single line of code:

Opening:
this.props.dispatch(showDialog(ClientEditDlg,{id,someCallback}))

Closing:
this.props.dispatch(actCloseDlg());

It is important to note that this methodology can be applied in other types of controls such as: toast, snack bar, banners, or also side columns contents.

1-Modal Root
2-Main Layout
3-Content Area
4-Activity Dialog

1) Modal Root component

The ModalRoot is an intermediate attendant component, that will render any component and arguments passed in his props. Placed in the main layout, will receive any dialog from any module.

//file: /src/modal/ModalRoot.jsx

const ModalRoot = ({ dlgComponent, dlgProps }) => {
  if (!dlgComponent) {
    return null 
  }  
  const Dlg = dlgComponent;//just for Uppercase component naming convention
  return <Dlg {...dlgProps} />
}

export default connect(state => state.modal)(ModalRoot);

Tied to his own Redux store, so any action distpached, will be listen and then triger a new render.

1.1) ModalRoot, Actions and Reducer.

Only need two action, one to open and one to close:

//file: /src/modal/ModalActions.jsx

export const showDialog = (dlgComponent,dlgProps) => ({type:  'SHOW_DLG',  dlgComponent, dlgProps });
export const closeDialog = () => ({type:  'CLOSE_DLG' });

The ModalRoot reducer is very simple, just two actions:

//file: /src/modal/ModalReducer.jsx

const initialState = {dlgComponent: null, dlgProps: {}}

export default  function ModalReducer(state = initialState, action) {
    switch (action.type) {
      case 'SHOW_DLG':
        return { dlgComponent: action.dlgComponent,  dlgProps: action.dlgProps}               
      case 'CLOSE_DLG':
        return initialState        
      default:
        return state
    }
  }

2) Main Layout

The ModalRoot component will be placed in the app main layout, common to all modules. In this case only use the module ClientsMain. But keep in mind that here will be placed the navigation bar and all modules, like notes, orders, invoces. The rendering choice of one or other, will be handled by routing or conditional rendering.

//file: /src/App.js

function App() {
  return (
    <>
    <ModalRoot/>
    <ClientsMain/>     
    </>
  );
}

3) Content Area

For demostration purposes we will work over a client directory with name, phone and mail. Where we can edit and delete every item, also add a new client. "The classic CRUD".

The files of the client module:

ClientsMain.jsx // Listing
ClientCreateDlg.jsx // Create new
ClientEditDlg.jsx // Edit
ClientDeleteDlg.jsx // Delete confirmation

ClientsActions.jsx //Redux Files
ClientsReducer.jsx //Redux Files

3.1) Data fetching

The client list will be retrieve with Axios from a Flask endpoint. When ClientsMain is mounted, trig data fetch from API, dispatching actClientsFetch().

Fetch Client actions:


//file: /src/clients/ClientsActions.jsx

export function actClientsFetch(f) { 
  return dispatch => {
    dispatch(actClientsFetchBegin());  // for loading message or spinner

    axios.post(process.env.REACT_APP_API_BASE_URL+"clientslist",f,{withCredentials: true} )
    .then(response => { dispatch(actClientsFetchSuccess(response.data.items));})
    .catch(error => { dispatch(actClientsFetchError({status:'error',msg:error.message+', ' + (error.response && error.response.data.msg)}))} );    
  };
}

export const actClientsFetchBegin = () => ({
  type: 'CLIENTS_FETCH_BEGIN'
});

export const actClientsFetchSuccess = items => ({
  type: 'CLIENTS_FETCH_SUCCESS',
  payload: { items: items  }
});

export const actClientsFetchError = msg => ({
  type: 'CLIENTS_FETCH_ERROR',
  payload: { msg: msg}
});

Fetch Clients reducer:

Next lines show a code extracted from the reducer.

//file: /src/clients/ClientsReducer.jsx 
// extract :

  case 'CLIENTS_FETCH_BEGIN':  // "loading" show a spinner or Loading msg      
      return {
        ...state,
        status: 'loading'        
      };
    case 'CLIENTS_FETCH_SUCCESS': // All done: set status and load the items from the API
      return {
        ...state,
        status: 'success',        
        items: action.payload.items,
        isDirty : false
      };
    case 'CLIENTS_FETCH_ERROR':  // Something is wrong
      return {
        ...state,
        status: "error",
        msg: action.payload.msg,
        items: []        
      };

Flask dummy route

Just to simulate a server request, a Flask route returning static data is implemented.

@app.route('/clientslist', methods=['POST','GET'])
def clientlist():
   clients= [    {'id':'1','name':'Client 1','mail':' user1@mail.com','phone':'555-555-111'},
                 {'id':'2','name':'Client 2','mail':' user2@mail.com','phone':'555-555-222'},
                 {'id':'3','name':'Client 3','mail':' user3@mail.com','phone':'555-555-333'},
                 {'id':'4','name':'Client 4','mail':' user4@mail.com','phone':'555-555-444'}]

   return {'items':clients}

3.2) Automatic reloading:

In order to get data consistence the clients Redux store have a isDirty flag, any action over the clients (create,update,delete) will trigger actClientsSetDirty() changing isDirty flag to TRUE and then trigger data reload.

List reload when data is dirty:

//file: /src/clients/ClientsMain.jsx

 componentDidUpdate(prevProps, prevState) {
    if (this.props.isDirty && this.props.status !== 'loading')    {
      this.props.dispatch(actClientsFetch());
    }    
  }

Triggering list reload

//file: ClientsActions.jsx

export const actClientsSetDirty = () => ({
  type: 'CLIENTS_SET_DIRTY'
});

4) Activity Dialog

The activity dialog is the component that would be injected in the modal root, in this case use the material dialog, but can be any thing: banner, toast, etc...

4.1) Activity Dialog, Actions and Reducer.

The activity could be: create, update or delete clients. Every activity have their related action. In this case all point to the same API route, but in real scenario, every must to have their specific route.

//file: /src/clients/ClientsActions.jsx 
// extract :

export function actClientCreate(d) {return actClientsFormApi(d,"clientsresponse")};
export function actClientUpdate(d) {return actClientsFormApi(d,"clientsresponse")};
export function actClientDelete(d) {return actClientsFormApi(d,"clientsresponse")};

function actClientsFormApi(d,url) { 

  return dispatch => {
    dispatch(actClientFormSubmit());// for processing advice msg

    axios.post(process.env.REACT_APP_API_BASE_URL+url,d, {withCredentials: true})
    .then(response => { dispatch(actClientFormResponse(response.data));
                        dispatch(actClientsSetDirty()) ;})
    .catch(error => { dispatch(actClientFormResponse({status:'error',msg:error.message+', ' + (error.response && error.response.data.msg)}))
                    })

  };
}

export const actClientFormInit = () => ({
  type: 'CLIENT_FORM_INIT'  
});
export const actClientFormSubmit = () => ({
  type: 'CLIENT_FORM_SUBMIT'  
});
export const actClientFormResponse = (resp) => ({
  type: 'CLIENT_FORM_RESPONSE',
  payload : resp
});

Next lines show a code extracted from the reducer, where are three action related to form submiting.

CLIENT_FORM_INIT initialize the formStatus to normal,
CLIENT_FORM_SUBMIT to show processing message,
CLIENT_FORM_RESPONSE is the API response that could be: 'error' or 'success'.

//file: /src/clients/ClientsReducer.jsx 
// extract :

 case 'CLIENT_FORM_INIT':             
        return {
          ...state,
          formStatus: 'normal',
          formMsg: '',          
        };   
    case 'CLIENT_FORM_SUBMIT':   
        return {
          ...state,
          formStatus: 'loading',
          formMsg: '',          
        };   

    case 'CLIENT_FORM_RESPONSE':

        return {
          ...state,
          formStatus: action.payload.status,
          formMsg: action.payload.msg,
       };

4.2) Activity API interaction

The API response is attended by CLIENT_FORM_RESPONSE. A formStatus is implemented to know the request results from the API. Also a formMsg for API error messages.

//file: /src/clients/ClientsReducer.jsx 
// extract :

  case 'CLIENT_FORM_RESPONSE':    

      return {
        ...state,
        formStatus: action.payload.status, //response from API
        formMsg: action.payload.msg        
      };

We have three activity dialogs:

ClientCreateDlg.jsx // Create new
ClientEditDlg.jsx // Edit
ClientDeleteDlg.jsx // Delete confirmation

The dialog make the API request, if all is ok it close by self, if not error advices are showed.

All have the same internal structure, the important thing to highlight is the formStatus.

When axios resolve the API response, it trigger CLIENT_FORM_RESPONSE. Then the operation result is stored in formStatus that could be: 'error' or 'success'.

For shortness only show 'ClientsCreateDlg'

//file: /src/clients/ClientsCreateDlg.jsx 
// extract :  

function ClientCreateDlg(props){

  const initial = {  name:'',phone:'', mail:'',}; 
  const [state, setState] = useState(initial);  
  const fullScreen = useMediaQuery('(max-width:500px)');// if width<500 go fullscreen

  useEffect(() => { //Mount - Unmount  
         props.dispatch(actClientFormInit());  //componentMount    
         //console.log("component Mount");
        return () => {
         props.dispatch(actClientFormInit());  //componentWillUnmount    
         // console.log("componentWillUnmount");
        };
      }, []);

  //componentDidUpdate  status listener  
  useEffect(() => {
    console.log("status Update", props.status);
    if( props.status==='success') props.dispatch({type: 'CLOSE_DLG' });  //trigger UnMount             
     }, [props.status]); 

  const handleChange = (e) => {
    const {name,value} = e.target;
    setState(prevState => ({...prevState,[name]: value}));  
  };

  const handleSubmit = (e) => {
    console.log("handleSubmit:",state)
    e.preventDefault();  // prevent a browser reload/refresh
    props.dispatch(actClientCreate(state)); 
  };

  const handleCancel = () => {    
    props.dispatch({type: 'CLOSE_DLG' });
  } ;  

    const { status, msg } = props; // server API responses   

    var advice = null;         
    if (status === "loading") advice = "Procesing...";    
    if (status === "error") advice =  "Error: " + msg;   
    if (status === "success") {  return null; }        

    return (
    <Dialog onClose={handleCancel} fullScreen={fullScreen} open={true}>
        <div style={{minWidth:'300px',padding:"2px",display: "flex" ,flexDirection: "column"}}>
        <DialogTitle ><ViewHeadlineIcon  />Create new client:</DialogTitle>          
        <form onSubmit={handleSubmit} >

        <div style={{minWidth:'50%',boxSizing:'border-box',padding:"2px",display: "flex" ,flexDirection: "column",flexGrow:'1'}}>         
          <TextField name="name"  size="small" placeholder="Name"  onChange={handleChange} />  
          <TextField name="phone"  size="small" placeholder="Phone" onChange={handleChange} />  
          <TextField name="mail"   size="small" placeholder="Mail"  onChange={handleChange} />     
        </div>        

        <div style={{ display: "flex", flexDirection: "row",alignItems: "center",justifyContent: "space-around" }}> 

        <IconButton  type="submit"  >   <CheckCircleIcon  color="primary"/> </IconButton>    
        <IconButton onClick={handleCancel}  > <CancelIcon/></IconButton>   
        </div>       
        <Ad l={advice}/>               
        </form>
        </div>
    </Dialog>
);
}

const mapStateToPropsForm = state => ({    
  status:state.clients.formStatus,
  msg:state.clients.formMsg,   
});

export default connect(mapStateToPropsForm)(ClientCreateDlg);

4.3) Flask response dummy route

In order to show the results of the API endpoint, a route with random responses is implemented.


@app.route('/clientsresponse', methods=['POST','GET'])
def clientrandomresponse():

  responses = [{ 'status': 'success'},
               { 'status': 'error', 'msg': 'Json required'},
               { 'status': 'error', 'msg': 'Missing field '},
               { 'status': 'error', 'msg': 'Data validation fail'}]

  return responses[time.localtime().tm_sec%4]   # only for demostration

Conclusion:

May be look complex to understand, there are two related mechanisms, one in charge of dialog injections, and other related to API interaction.
Usually an app can have many modules: clients, notes, order, and are used one at time, so all can share the same dialog root component.
In this way can open a dialog from whatever place.

Get the full code from https://github.com/tomsawyercode/react-redux-dialogs-crud

DEV Community: Lionel Marco

YAML for resume ATS sharing

Why not, Is there something better ?

What means YAML ?

Arduino, GPS based Heliostat, Suntracker

How know where is the sun ?

Mechanical configuration

Pointing to a target, the math behind

The equations translated to code:

The electronic setup

Master module

Slave module

The code

Elixir Cowboy Websocket - Querying IOT devices by latitude longitude proximity

Possible use cases

Table of Contents

Dependencies

The application

Space Zonification

Filtering by distance

Devices Registry

The socket Handler

Elixir Plug Cowboy, WebSockets Broadcasting to React

Table of Contents

1) Dependencies

2) Quote Store

3) The Scheduler

5) Socket Handler

3) The application

6) React client

Dashboard

Quotes table

Working with Geospatial data in Elixir-Postgrex, Mimic Node-PgPromise behaviour.

Table of Contents

1) Introduction

Query Helpers

Working with named parameters

Obtain columns as maps

2) Working with geospatial data

3) Configure and Start

4) Postgrex Query Helpers

5) Named Parameters

JSON validation for Cowboy REST API - Mimic Celebrate Joi in Elixir

Table of Contents

1) Plugs and Midlewares

2) Route-Schema Validation

2) Function Clause Matching

2.1) Tips.

Group:

Order:

Match all:

4) Schema Validation

The Plug.conn struct

How to add flask, gunicorn, packages to a distroless docker container

Table of Contents

1) Distroless Concept

2) How to work with theys

2.2) Package installing

Some Tips:

2.2) See what is inside

2.3) Running Flask

2.4) Running Gunicorn

3) Vulnerability Scanning

React Color Scale Interpolation

Table of Contents

1) Color interpolation

2) Color Selector

Color Icon

Show Icon

3) Ratio Slider

4) Full code

React Multi State Submit Button

A multi state submit button that inter actuate according API response.

Table of Contents

1) Tuning Icons

Loading Icon

Error-Warning Icon

2) Multistate Button

3) Dialog with Submit Button

handleSubmit