DEV Community: alxizr

Spotify: The Most Expensive Skip Button

alxizr — Sun, 19 Jan 2025 10:00:00 +0000

Hello everyone,

Thank you for joining in for this article. Today we will talk about your attention span and the lack of it.

Prologue

I'm baffled. Genuinely, utterly baffled. How, in this age of information and enlightenment, can so many people willingly throw their hard-earned cash at a service that offers virtually nothing of value? I'm talking about Spotify Premium, of course.

The Free Version: A Hidden Gem

Let's be honest, the free version of Spotify is a masterpiece of modern entertainment. You get access to millions of songs, curated playlists, and the occasional ad. But the ads? Honestly, a small price to pay for such a vast library. It's like those old-timey radio shows – a brief interruption before the next captivating tune.

The "Premium" Illusion

Now, let's talk about this "Premium" nonsense. For the exorbitant price of a decent coffee, you get… what exactly? The ability to skip a few more songs? Seriously? Do you lack the patience to listen to a song in its entirety? Are you so easily distracted that you can't endure a 3-minute commercial? If so, maybe you need to re-evaluate your relationship with time management.

The Download Delusion

And then there's the "download music" feature. Wow, groundbreaking. Now you can clog up your already overflowing phone with gigabytes of music files. Music files that will likely slow down your device, thanks to all those adorable cat videos and high-resolution Instagram photos you've been hoarding. And let's be real, how often do you actually listen to downloaded music anyway? You're probably just streaming it directly from the app, aren't you?

The YouTube Conundrum

If Spotify truly offered a compelling value proposition, YouTube would be a distant memory. But here we are, still watching music videos on YouTube. Why? Because it's free, it's incredibly versatile, and it doesn't require you to pay a monthly fee for the privilege of listening to music.

The Skip Scam

And let's not forget the "premium" feature that allows you to skip a measly five songs per hour. Five songs! In this day and age, five songs is barely enough to get through one of those endless "Top 40" playlists. It's like they're intentionally trying to frustrate you.

Epilogue

Look, I get it. Convenience is king in today's world. But paying for the privilege of listening to music uninterrupted? It's a ridiculous concept. The free version of Spotify offers everything you need and more. So, next time you're about to reach for your credit card to upgrade, ask yourself: Are you really that impatient? Are you really that desperate to avoid a 30-second commercial? I think you might be surprised by the answer.

Disclaimer

This article is written with a satirical and humorous tone. It is not intended to be taken entirely seriously.

Stay tuned for next
Like, subscribe, comment and whatever ...
Thank you & Goodbye

2 Gits, 1 User: Navigating the World of Multiple Git Accounts

alxizr — Sat, 18 Jan 2025 07:59:00 +0000

Hello everyone,

Thank you for joining in for this article. Today we will talk a little about passion, pain, mystery and a lot of history.

Prologue

I used to think my GitHub profile was a reflection of me, a digital diary of my coding journey. But then I started working, and suddenly, my coding life felt fragmented. My personal projects lived on one account, while my work contributions were locked away in another. It was like having two separate lives, and I didn't know how to connect them.

The Dilemma

Let's be real, juggling multiple Git accounts can feel like trying to keep track of a dozen different passwords. You've got your personal account for side projects, maybe a few from past jobs, and then there's the one for your current employer. It's a common developer struggle, and it can leave you feeling like your online coding resume is scattered across the internet like confetti.

I used to just go with the flow. I kept my personal projects on my personal laptop and used the company-provided machine for work. It seemed to work, but looking back, it feels like a missed opportunity.

The "Invisible" Contribution

At one point, I was working at a place where I racked up hundreds of thousands of commits. I was proud of my work, of course. But when I left, it hit me: a massive chunk of my coding journey was invisible. My online presence, my "developer resume" as some call it, was incomplete. In the pre-COVID days, a vibrant green contribution graph on GitHub was almost a requirement for a senior developer role. It showed dedication, a consistent work ethic, and a passion for coding.

The Challenge

So, how do you bridge this gap? How do you bring all those contributions together?

Unfortunately, there's no magic button to transfer contributions directly between accounts. GitHub doesn't work that way.

The "Portfolio" Account Solution

But here's what you can do:

The "Portfolio" Account: Create a dedicated GitHub account just for showcasing your best work. Think of it as your online coding portfolio.
Selective Mirroring: Don't just dump everything in there. Choose the projects that truly represent your skills: personal projects, open-source contributions, and maybe even some carefully selected work projects (check your company's policies first!).
Automation (for the tech-savvy): Tools like GitHub Actions can automate the process of mirroring changes from your other accounts, saving you time and effort.

Connecting the Dots

Connect the Dots: Link Your Accounts: Make it easy for people to find your work. Link your accounts on your LinkedIn profile, create a personal website, or build a dedicated portfolio page.
Tell Your Story: Don't just list projects. Write a compelling bio that explains your journey and highlights your key accomplishments across all your accounts.

Epilogue

Managing multiple Git accounts can be a headache, but with a bit of planning and the right strategies, you can reclaim your developer journey and build a portfolio that truly represents your skills and experience.

Disclaimer

Always double-check your company's policies regarding intellectual property and open-source contributions before sharing any work-related projects.

Stay tuned for next
Like, subscribe, comment and whatever ...
Thank you & Goodbye

Setup Load Balancer and Reverse Proxy w/ Nginx and Docker

alxizr — Fri, 20 Oct 2023 06:23:29 +0000

Hello everyone,

Thank you for joining in for this article. Today we will talk a little about one of the most popular solutions out there to handle load balancing and reverse proxying, which is Nginx. We will also make use of Docker compose and integrate it with a small nodejs application for our example.

prologue & motivation

My motivation for this articles is coming from a side project that i was working on not long ago, and I reached a point in the development roadmap where I wanted to start and test things. I wanted to have a complete CI/CD pipeline that takes care of the many different steps in the deployment of this application, and one of this steps is actually a stress test, but alas this cannot be achieved easily. The end goal is to build a microservice environment to support this application, but we are too far away from there, let alone thinking of it. I concluded that it all must exist in my workstation. All I have a laptop with Ubuntu Desktop OS installed, Docker and Docker Compose and with this we will win.

prerequisites

Before we begin, as always we want to know what is the minimum for us to be able to start, be efficient and productive.

setup

Please open vscode and open the build in terminal. You can use the short key

    ctrl + `

Now let's create a folder for this project and change directory into it

    mkdir nginx_demo
    cd nginx_demo

Now we want to have the context of this folder inside our vscode by restarting it within nginx_demo folder

    code -r .

setup example app

I want to point out that this app could be anything you want. It doesn't have to be Node.js nor a REST API. You can choose what ever programming language or framework that you feel comfortable with. I chose this example because it is minimalistic and most important is that the app is not the focus.
Let's start with our Node.js Express application. It is a very basic setup and we wouldn't need to configure much.

We will create a folder named app and initialize a Nodejs project inside it

    mkdir app
    cd app
    node init -y

Once the project done initilzing, we want to install couple of npm packages to setup our REST API. We will use Express.js a minimalistic web server for Node.js and cors package to handle CORS and persist it for our production setup.

    npm i -S express cors

For our development setup, we will use nodemon package and also install several packages for type inference.

    npm i -D nodemone @types/{node,express,cors,nodemon}

Now we want to create a file named index.js and add the following code to it. The code is very simple and straight forward. We are creating an express app and we are using cors package to handle CORS. We are also using the os package to get the hostname of the machine that is running the app, we will need it later. We are also using the express.json() and express.urlencoded() to handle the request body.

// index.js
const express = require("express");
const cors = require("cors");
const os = require("os");

const PORT = 3333;
const HOST = "0.0.0.0";

const app = express();
app
  .use(cors())
  .use(express.json())
  .use(express.urlencoded({ extended: true }));

app.get("/", (req, res) => {
  return res
    .json({
      message: "Hello from /",
      host,
      os: os.hostname(),
    })
    .status(200);
});

app.listen(
  PORT,
  HOST,
  console.log(`Example app listening at http://localhost:${PORT}`)
);

Let's test our, but beforehand we want to add a script to our package.json file to run our app using nodemon. Add the following line to the scripts section of the package.json file.

    "dev": "nodemon index.js",
    "start": "node index.js"

Now we can run our app using the following command

    npm run dev

You can open the browser and navigate to http://localhost:3333 and you should see the following response

{
  "message": "Hello from /",
  "host": "localhost",
  "os": "..."
}

Now we want to create Dockerfile for this application so we could dockerize it. Create a file named Dockerfile and add the following code to it. We are using the official node image from docker hub and we are copying the package.json file and installing the dependencies. We are also copying the rest of the files and folders and running the app using the npm start command.

# api dockerfile
FROM node:alpine
WORKDIR /app
COPY package*.json .
RUN npm install
COPY . .
EXPOSE 3333
STOPSIGNAL SIGTERM
CMD ["npm", "start"]

We can now build our image using the following command and test that it actually works, but I will leave it to you.

    docker build -t api .
    docker run -p 3333:3333 api

setup nginx as reverse proxy

We will change directory back to the root folder nginx_demo and we will create a folder named nginx and change directory into it.

    cd ..
    mkdir nginx
    cd nginx

We need to create 2 files for our nginx setup.

    touch nginx.conf Dockerfile

In this Dockerfile for our nginx image, we are using the official nginx image from docker hub and we are copying the nginx.conf file to the nginx configuration folder. We are also exposing port 80 and we are using the STOPSIGNAL directive to stop the container gracefully.

**# nginx **dockerfile
FROM nginx:alpine
COPY nginx.conf /etc/nginx/nginx.conf
EXPOSE 80
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]

Now we want to create a file named nginx.conf and add the following code to it.

# nginx.conf

# best practice to define the number of worker processes and events block
worker_processes 1;

events {
    worker_connections 1024;
}

# http block
http {

    sendfile on;

    # upstream servers that will handle the requests
    upstream myapi {
        # load balancing algorithm
        least_conn;

        # api is the name of my custom container
        # 3333 is the port that the app is listening to inside the container
        # custom Docker image name and port number are totally arbitrary
        server api:3333;
    }

    server {
        listen 80;

        location / {
            # the upstream name port 80 is proxying to
            proxy_pass         http://myapi;

            gzip_static        on;
            proxy_redirect     off;
            proxy_set_header   Host $host;
            proxy_set_header   X-Real-IP $remote_addr;
            proxy_set_header   X-Forwarded-Host $server_name;
            proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Let's explain what we have in this nginx.conf file. We are using the upstream directive to define a group of servers that will handle the requests. We are also using the server directive to define the server that will handle the requests. We are also using the location directive to define the location of the requests. We are also using the proxy_pass directive to pass the requests to the upstream servers.

setup docker compose

We will change directory back to the root folder nginx_demo and we will create a file named docker-compose.yml and add the following code to it.

We will create a setup that ensures that everything works and once we are done with it, we will discuss on some caveats and how to overcome them.

    cd ..
    touch docker-compose.yml

version: "3.8"

networks:
  nginx_demo:
    name: nginx_demo

services:
  api:
    image: api
    build:
      context: ./app
      dockerfile: Dockerfile
    ports:
      - 3333:3333
    networks:
      - nginx_demo

Now let's test the setup using the following command

    docker-compose up -d

This should provide us with a running docker container that is listening to port 3333. We can test it using the following command

    curl http://localhost:3333

Or we can open the browser and navigate to http://localhost:3333 and we should see the following response

{
  "message": "Hello from /",
  "host": "localhost",
  "os": "..."
}

We can see that everything is working as expected. Now we want to add the nginx service to our docker-compose.yml file. We will add the following code to it, but before we want to kill the running docker container.

    docker-compose down

We will add the nginx service to our docker-compose.yml file. We will add the following code to it.

mynginx:
  image: mynginx
  build:
    context: ./nginx
    dockerfile: Dockerfile
  ports:
    - 80:80
  networks:
    - nginx_demo

If we run the docker compose spin up command again, and try to access the browser in http://localhost:80 we will see that we are refered to the api service. This is because we are directly binding the port of the application and we are using the nginx service as a proxy. This is not optimal and can cause some issues.

The reason for it is if you remember our motivation for this article is to stress test our application. When we explicitly bind ports to the host machine, we are limiting the number of requests that can be handled by the application. We want to be able to scale the application and the nginx service independently. We will change the api service in the docker-compose file and remove the port binding between the host machine and the container.

services:
  api:
    ...
    # pay attention to this line
    ports:
      - 3333
    ...

Once we achieved this we could carry on and test the setup again. We will run the following command

    docker-compose up -d

pay attention that if you try open the browser and navigate to http://localhost:3333 you will get an error. This is because we are not binding the port to the host machine. We can test it using the following command

    curl http://localhost:3333

but if you try to open the browser and navigate to http://localhost:80 you will see that everything is working as expected.

{
  "message": "Hello from /",
  "host": "localhost",
  "os": "..."
}

scaling

By now we concluded that we have a working setup with nginx as a reverse proxy and we have only 1 instance of our application, which in this example a very simple REST API. We want to scale this application under the api service independently. We will change the docker-compose.yml file and add the following code to it.


services:
  api:
    ...
    # pay attention to this line
    scale: 5
    ...

In this change we are actually telling docker compose to spin up 5 instances of the api service. You need to pay attention to the little nuance that there are no port bindings between the host machine and the container. Remember that the API application only listents to port 3333 inside the container, but there could not be more than 1 instance of the application listening to the same port. This is why we are not binding the port to the host machine. We are relying on the nginx service to handle the requests and proxy them to the api service. In this case what will happen is that the docker will assing random ports to the api service and we will not be able to access them directly from the host machine unless we will use the docker inspect command to get the port number and use it to access the api service directly. We will not do that, but we will use the nginx service to handle the requests for us.

We can test it using the following command

    docker-compose up -d

LEt's see all the different instances work in real time. We will open the browser in http://localhost:80 and we will see that the response is coming from different instances of the api service. If you remember when we setup our api, we used the os build in package to get the hostname of the machine that is running the app. We can see that the hostname is different for each request and it is actually displaying the id of each running docker container.

{
  "message": "Hello from /",
  "host": "localhost",

  // different for each request
  // matching the id of the docker container
  "os": "..."
}

bonus

We can add ensurance layer to our setup so we will not be trying to use the api service in the nginx service before the api service is ready. We can solve this by adding the depends_on directive to the nginx service and we will add the following code to it.

services:
    mynginx:
        ...
        depends_on:
            - api
        ...

Another thing that we need to do to keep us safe, otherwise it is going to be a pain in the neck to debug, is to remove the ports directive from the api service. This little change will ensure that we will not be able to access the api service directly from the host machine.

we can also add the restart directive to the api service and we will add the following code to it.

services:
    api:
        ...
        restart: always
        ...

This is the final version of our docker-compose.yml file

version: "3.8"

networks:
  nginx_demo:
    name: nginx_demo

services:
  api:
    image: api
    restart: always
    scale: 3 # this could be any number as long your hardware can support it
    build:
      context: ./app
      dockerfile: Dockerfile
    networks:
      - nginx_demo

  mynginx:
    image: mynginx
    build:
      context: ./nginx
      dockerfile: Dockerfile
    ports:
      - 80:80
    networks:
      - nginx_demo
    depends_on:
      - api

epilogue

Thank you for reading this article. I hope you enjoyed it and learned something new. If you have any questions or comments, please feel free to reach out to me. I will be more than happy to help you out.

BYOB w/ Flask, Python & MongoEngine

alxizr — Wed, 31 Aug 2022 19:15:19 +0000

Hello everyone,

Thank you for joining in for this article. Today we will take a little look at one of the most popular programming languages out there which is Python and a minimalistic, yet very powerful web framework called Flask. We will also make use of a NoSql document database called MongoDB and integrate it with an ODM (Object Document Mapper) called MongoEngine

Prerequisites

Before we begin, as always we want to know what is the minimum for us to be able to start, be efficient and productive.

Visual Studio Code
Git Bash
Postman
Pip
If you are by any chance missing one of these tools, make sure to download and install it before continuing.

Prologue

In my day to day job I work mainly with Node.Js in a microservices environment, but sometimes we have tasks or entire epics that are data driven and we need to handle massive computes on the data we work on, node.js is not the right tool for it because of its single thread non blocking io nature, thus we need to pick the right tool for the job. In a microservice environment you can have many and different services that handle single responsibility and they may be implemented with a programming language or a framework outside your main tech stack. Here, in this case, comes Python & Flask. In this article we will see how we can achieve a working api in no time.

Setup

We will open visual studio code and setup our workspace. We will make use of a tool called Pipenv it is a package manager very similar to NPM in the Node.Js ecosystem and it is the new way of doing things in the python ecosystem. We will start by creating our project directory api.

mkdir api
cd api

Now, the first thing we need to do right away and I will talk about later in more details, is creating a -.env_ file for our workspace environment variables and set a value.

echo PIPENV_VENV_IN_PROJECT=1 >> .env

This step also can be done only once globally and it depends on the operating system you have. You can look in the docs where to update this environment variable globally.

Our next step is where it all begins. We will create our encapsulated environment with pipenv and within install all of our project dependencies.

pip3 install pipenv

We use pip version 3 to install this package globally so it will be available to us for other python projects and this is the only time where we need the use of pip.

Now let us activate the encapsulated environment and start installing packages.

pipenv shell

This command is actually creating and activating our virtual environment, something that is well known and used in the python ecosystem. The reason we needed our .env file to be present beforehand is that the default behavior of pipenv is to create the .venv folder in a different location in the file system, depends if you are using Windows, Linux or Mac. This way we make sure that everything that is related to the project is present within the project workspace.

You can see that by running this command we got our virtual environment present and also a file named Pipfile created, this file contains all the settings for the project in term of dependencies, dev dependencies scripts and versions much like package.json in Node.js.

Now we will install all the dependencies at once so we would not have to keep constantly switch contexts. You can obviously just copy and paste it in your terminal if you don't want to type it all.

pipenv install flask flask-restful flask-marshmallow marshmallow mongoengine flask-mongoengine python-dateutil flask-cors

Once installation is done, you will see that a new file was created named Pipfile.lock which describes all the dependencies and sub dependencies we installed and also states the exact versions of each dependency.

Now we create our entry file app.py inside of a src folder just to keep things clear. We then change directory into this folder and this is going to be our root for the project. Any additional folders and files we will create will be descendant of src.

mkdir src
cd src
touch app.py

Now we will create 2 folders that will represent our business logic for the project and in time when this project will scale, you will be able to follow the same pattern.

Our example will go around a post model crud api. The model will be constructed with nested schema, just a small addition from me to see how we handle nested documents in MongoDB.

# database related tree

mkdir database controllers
mkdir database/models

touch database/db.py
touch database/models/{metadata,post}.py

# controller related tree

mkdir controllers
mkdir controllers/posts
mkdir controllers/posts/dto

touch controllers/posts/dto/post.dto.py
touch controllers/routes.py
touch controllers/posts/{posts,post}Api.py

By now all we have is the basic skeleton for the project with no 1 line of code written. We will address it in the next step, I just want to make sure we all have the same folder tree.

- api #(project folder)
| - src
|    | - app.py
|    | - controllers
|         | - dto
|              | - post.dto.py
|         | - posts
|              | - postApi.py
|              | - postsApi.py
|         | - routes.py
|    | - database
|         | - db.py
|         | - models
|              | - post.py
|              | - metadata.py

Database Setup

In this step we will configure our database in the project. We will use MongoDB. There are different ways to consume this service. The most popular one is via the cloud offering called MongoDB Atlas, it is free and all you need to do is register for the service. I for one do not want to add a layer of latency while developing locally so I will choose Docker instead. This step is optional and you don't have to do it the same way as me, but I will provide the details on how to run local MongoDB docker container for development purposes. This is not a Docker tutorial and I do not want to change focus, so if you are not familiar with it, then use Atlas.


# docker-compose.yml file

version: "3"

services:
  db:
    restart: always
    container_name: mongodb
    image: mongo
    ports:
      - "27017:27017"
      - "27018:27018"
      - "27019:27019"
    volumes:
      - "./mongo_vol:/data/db"

# spin up a mongodb container by running this command in your terminal

docker-compose up -d

Once our database is up and running wether locally or in the cloud, we can carry on to configure it in our project. Let us open the db.py file inside the database folder and write some code.

We need to import into this file the utility to work with MongoDB and we installed a package called flask-mongoengine which will handle the database connection and operations.

from flask_mongoengine import MongoEngine

db = MongoEngine()

def init_db(app):

    # replace the host URI with your atlas URI if you are not working locally
    # this settings must be as close as posiible to the init app call!
    app.config["MONGODB_SETTINGS"] = { "host" : "mongodb://localhost:27017/products" }

    db.init_app(app)
    pass

Here we imported the package, initialized a database connection instance and provided the host uri to where our database is located alongside the name of our target database name. The last line of code says that the connection will be bound to the app once it is running.

Database Model Setup - Post

Our model for the example is a post. We want to describe how a post looks like in terms of property fields and divide some of them to a nested schema. Our post will consist of these properties: title, text, creation date, author's email, url. We will decide that the parent schema will hold the title and text fields and the rest will be nested to a child schema called metadata, like this:

    # Metadata schema

    import datetime
    from ..db import db

    class Metadata(db.EmbeddedDocument):
        url = db.URLField()
        email = EmailField()
        date = db.DateTimeField(default=datime.datime.utcnow)
        pass

Pay attention to the date field, where we initialize it with a default date & time only on creation. We will not be providing this value via the api. The model will create and set it.

    # Post schema

    from ..db import db
    from .metadata import Metadata

     class Post(db.Document):
         title = db.StringField()
         text = db.StringField()
         metadata = db.EmbeddedDocumentField(Metadata, default=Metadata)

         meta = {
             "collection" : "posts",
             "auto_create_index": True,
             "index_background": True,
             "inheritance": True
         }
         pass

Also notice that all of the fields are optional, except the metadata field. The Post model expects to receive a metadata object with its fields because we set the default flag. Your task is to change them to be required fields with the expression required=True inside the parentheses

As you can see, we have two models that each holds partial data about our post and together they are a whole. Now, using the Post class we will engage the database via our api. We do not need to do anything with the Metadata class. It is there for convenience purposes.

API Setup

Flask supports different ways of building web applications and APIs, starting from templates for static pages, through blueprints and views and going to rest. We will choose the restful approach and we already installed the appropriate package for it flask-restful.

The way to construct rest api with the restful package is by using classes that inherits from a base class called Resource. We will create 2 classes for our api. 1st class handles all generic requests and the 2nd class will handle all the specific request that we want to filter by post id for example. Obviously you can decide on the best approach for what ever your needs are.


# PostsApi.py file

from database.models.post import Post
from flask import request, Response
from flask_restful import Resource
import json

class PostsApi(Resource):

    # get all post from database
    def get(self):
        posts = Post.objects().to_json()
        return Response(
            posts,
            mimetype="application/json",
            status=200
        )


    # create new post
    def post(self):
        body = request.get_json()
        post = Post(**body)
        post.save()
        return Response(
            json.dumps({
                "message" : str(post.id),
                "status": "success"
            }),
            mimetype="application/json",
            status=201
        )

You can see here that the class has 2 methods named after the HTTP verbs they each are responsible to handle. The GET HTTP call to this endpoint /api/posts/ will return the list of posts in the database and a POST HTTP call to the same endpoint will hit the same API class to create a new document in our database. Pay attention as well to the way we are engaging the MongoDB database and it is with the Post class we created earlier. This is a posts api and we need the entity that is bound to it. Now lets see how to create a specific request api.


# PostApi.py file

from database.models.post import Post
from flask import request, Response
from flask_restful import Resource
import json

class PostsApi(Resource):

    # get one post by id from database
    def get(self, id):
        post = Post.objects.get_or_404(id=id).to_json()
        return Response(
            post,
            mimetype="application/json",
            status=201
        )


    ### delete one post
    def delete(self, id):
        post = Post.objects(id=id).delete()
        return Response(
            json.dumps({
                "id" : str(id),
                "success" : post
            }),
            mimetype="application/json", 
            status=200)


    # updates existing post by id
    def post(self, id):
        body = request.get_json()
        post = Post.objects(id=id).update_one(**body)
        return Response(
            json.dumps({
                "message" : post,
                "status": "success"
            }),
            mimetype="application/json",
            status=200
        )

As you can see that the biggest difference is with the argumenst the class methods get in their signature. The id parameter is a dynamic one and it comes from the URL. The Flask Resource class handles in fetching it and providing it to us inside the logic we write. Pay attention that this is not the way of doing things in production environment, we will get back to this later, after we are done with everything to test couple of things.

We are now left with setting up the routes for the api and we are done with this step.

    # routes.py file

    from flask_restful import Api
    from .postApi import postApi
    from .postsApi import postsApi

    def init_routes(app):
        api = Api(app)
        api.add_resource(postsApi, "/api/posts")
        api.add_resource(postApi, "/api/posts/<id>")

Pay attention that dynamic URL parameters in Flask are wrapped in diamond brackets, unlike in node.js where its prefixed with : the colon sign.

App Setup

So we worked backwards and we are finally here. The place where it all comes together like magic. We need to take all the different parts we created from scratch with our bare hands and make them stick. This magic will happen inside the app file.

from flask import Flask
from resources.routes import init_routes
from database.db import init_db
from flask_cors import CORS


# Init flask app with mongodb using mongoengine, flask-mongoengine 
app = Flask(__name__)
init_routes(app)
init_db(app)
CORS(app)


### run app watch mode
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000, debug=True, use_reloader=True)
    pass

What we are doing here is importing the database util function init_db, the routes util function init_routes, utilities from Flask to bind everything and make it rain (🌧️ 🌧️ 🌧️) ❗ (😀 😀 😀) make it run❗❗❗ Pay attention that in production you want to turn off the debug flag.

Now we are going back to the terminal and we want to have this Flask application running. Well, we have a little stop beforehand. remember that we installed pipenv❓ We want to add a script to run the application. Let us open the Pipfile and add a scripts section at the bottom.


[scripts]
dev = "python src/app.py runserver"

Now in the terminal where we initialized our virtual environment run this command


pipenv run dev

Congratulations! This is your first Flask Rest API and it was not too bad at all. Now you can open postman and run some tests agains the API. You now know that this is the enterprise way of using Flask for APIs, but there were couple of things that we skipped when we wrote the different api classes and they are error handling and input validation. In python, much like in many different programming languages you will and must encounter errors to fix them beforehand you go to production. Python uses try...except blocks for this purpose and the cool thing is that we can stack multiple except blocks up to handle different errors in our code. For example you can have one method that can throw server error (500 & up) or a user input error (400 <= 499).

I will create one example and your homework is to add it to all api class methods. I will choose the DELETE HTTP method and implement a generic exception.


### delete one post
def delete(self, id):
    try:
        post = Post.objects(id=id).delete()
        return Response(
                json.dumps({
                    "id" : str(id),
                    "success" : post
                }),
                mimetype="application/json", 
                status=200)

    except Exception as err:
        return Response(
                json.dumps({
                    "message" : str(err)
                }), 
                mimetype="application/json", 
                status=500)

That is all folks! It was a nice ride today. Hope you enjoyed and learnt something. Make sure to complete your homework and obviously read the documentation for Python, Flask and MongoEngine.

One last thing, some food for tought make sure to write code that checks also that the Database is working... wether you run mongodb installation, docker container or Atlas.

Next time, hopefully soon, I want to talk little about dependency injection in Python application focusing on Flask.

Stay tuned for next
Like, subscribe, comment and whatever ...
Thank you & Goodbye

Wait, Wait, Wait ... Now Go! ⌚⏳

alxizr — Sat, 28 Aug 2021 15:33:02 +0000

Hello everyone,

Thank you for joining in for this article. I know the title is a bit vague, but I would like you to know that we will not be going to talk about the GO programming language, but rather, make it today a little discussion about blocking your code, because sometimes you need to do it. One of the reasons, if not the most common one, is to avoid being blocked by an API's rate limiter.

Prerequisites

Before we begin, as always we want to know what is the minimum for us to be able to start, be efficient and productive.

Prologue

Let us address the matter at hand - we want, on purpose by all means, block our code or some part(s) of it from executing sequentially in a short period of time and carry on with our day, but why is that? Well, like I mentioned in the beginning, one of the main reasons, the one that I am familiar with at least the most, is to by pass the rate limiting rule of any public facing API.

Ever encountered this 429 Too Many Requests error ? Well now we will demonstrate how to avoid it and quite elegantly if I may add.

Use case

I would like to talk about the motivation on implementing this scenario because you might not need this at all. There are certain use cases where you will have to have this implementation in your code buried somewhere inside a helper function and no body knows it but you. The one use case we are addressing for our example is rate limiting. This is a far common use case than others and I have decided to use it for building today our example. I also want to make sure and emphasis that we are not going to implement a rate limiter on our service but we are going to deal with one on a remote API which has nothing to do with us rather.

Assume that you are assigned a task to get all the information about the buses in your city and in order to achieve this information successfully you need to communicate with an external API, probably provided by the bus company or the city or what ever. The issues is that the data set is fairly large, let's assume 1,000,000 records, and you cannot get all of them in one go, so you need to basically paginate your way in getting the job done, but alas, you get the data in chunks and in the 10th try to get the next chunk you receive an error stating that you executed too many request to the server and now need to take a break. We need to understand that when this error occurs, we failed to accomplish the task because we did not retrieve all the records.

Drill down on the solutions

There are more that enough ways to solve this matter. You can argue that for example you do not need the entire data set or you can manually re-run the function from the point it failed or maybe even argue that this is not your problem because you are not responsible on something you have no control of, true story by the way, but you do realize that you add +1 to your fails counter as a developer who should be able to solve any task handed to you.

We want to talk about the solution that will guarantee us 100 per cent success on this matter, we want it to be fully automatic and no human intervention is needed and, from my stand of point the most important aspect of the matter, we take full ownership on the task the way we are accountable. Accountability is by far one of the most valued traits any employee can have that managers love and appreciate (we will leave this for another talk).

So, by now we fully understand the what, where and who but we did not yet determined the how. If you consider it a bit, we actually only need to do some 4th grade math in order to find the time period we need to wait. In Physics there is a constant called 'T' for time period and is equal to 1 second divided by the frequency.

    T = 1 / f

This equation still does not answer our question. We need to understand what we are looking for in the equation. The easy part is the 1 second, we know this is a constant and we cannot change it. Let us try and understand what the 'f' for frequency stands for. The 'f' tells us how many executions or attempts we can have in 1 second period the way that the other side of the equation will remain true at all times.

Let's see an example: Assume that we can approach the remote API 300 times in one minute. Our equation is addressing seconds so firstly we need to convert it to seconds. One minute consist of 60 second, then we divide 300 attempts in 60 and we get back 5 attempts per one second.

   // 300 attmpts per one minute

   T = 300 / 60
   T = 5

Now we want to place this value in the equation:

   // T = 1 / f

   5 = 1 / f
   f = 1 / 5
   f = 0.20
   f = 200 milliseconds

As we can see here, in order to not violate the rate limiting rule, we must have up to 5 attempts at the API per one second or wait at least 200 milliseconds between the executions. This was not that hard, but wait, JavaScript is asynchronous in nature. How will we make it run synchronously and sequentially ? The real question we are actually asking is, how do we make it that the HTTP requests to the remote API will wait the minimum time period to delay in between the executions. This is where we are going to use the tool that is called BottleNeck.

Bottleneck is a lightweight and zero-dependency Task Scheduler and Rate Limiter for Node.js and the browser.

With the help of this tool we can apply some logic, and not that complex if I may add, to solve our 3 part problem that we noted above.

I will give my 2 cents on this tool and how it works from a bird's eye view in the simplest manner I can. The tool is instantiated with the use of a constructor that receives some arguments, the instance that is created holds a number of methods for particular use case. We will need the wrap method in this example. The wrap method receives a function as an argument and returns a throttled version of it.

Let us see it in action

We will open our visual studio code and create a new folder anywhere, I will do it on my Desktop. We will open the integrated terminal, then we will create a file for our code with 'touch main.js' and initialize our project with 'npm init -y'. Last step is installing the bottleneck and axios npm packages with 'npm i -S axios bottleneck' command.

  mkdir ratelimitmycode
  cd ratelimitmycode
  touch main.js
  npm init -y
  npm i -S axios bottleneck

I will use the json placeholder API for this example and you can change it to any other url that you want to test.

When we look at what we need to code here, we basically understand that we need an HTTP client and for that reason we installed axios and we need the target url.

  // main.js

  const axios = require('axios');
  const url = 'https://jsonplaceholder.typicode.com/todos/';


  const main = async () => {

    const res = await axios.get(url);
    const data = await res.data;

  } // main


  main();

As you can see in this piece of code, there is nothing fancy in it. All we do here is fetching the entire available data set the remote API has to offer under this endpoint. We will now implement a different data fetching approach based on a particular item id and see what happens.

  // main.js

  const axios = require('axios');
  const url = 'https://jsonplaceholder.typicode.com/todos/';


  const main = async () => {

    for (let i=1; i<=200; i++) {
      const res = await axios.get(url + i);
      const data = await res.data;
      console.log(data)
    } // for

  } // main


  main();

You can see that this particular endpoint might hold 200 records and we can address them with their unique id. In this piece of code we are looping 200 times to execute the same endpoint to retrieve a different piece of data.

We also need to remember that the particular API we are addressing has no rate limit rule turned on. Let us assume that there was a rate limit rule here and we would fail after several attempts. What would we do ? Well, we already answered this question earlier. We need to calculate the rate limit quota and act by it.

We will assume the following, the API has a rate limit of 300 requests per one minute and it holds 1 million records. As we already did the math, we need to have 5 requests per one second, so in total it will take us 2 days and 7 hours approximately to complete the fetching successfully. Do not be frightened because of this long time period. We will not be fetching 1 million records to begin with and we need to also understand that there could be very long time consuming tasks.

Given this information we know now that with the current implementation we have the task will fail. We will not be able never to fetch the entire data set from the remote API. We need to change the strategy and for this we have Bottleneck to help us.

    // import the package
    const { default: Bottleneck } = require("bottleneck");


    // create the instance with a constructor
    const limiter = new Bottleneck({
        minTime: 200, //ms
        maxConcurrent: 1,
    });

    // created a throttled function
    const throttled = limiter.wrap( fn );

We see in this piece of code only configurations. We will explain what we see. first of all we import the package, then we create an instance and passing some configuration options and finally we create a throttled version of the function to act by the rate limit.

We look at the configurations options and understand what we see

minTime property should hold the minimal time period we need to wait between executions and it is in milliseconds. By default it is set to 0.
maxConcurrent property holds the number of jobs can be executed in the same time. By default it is null, and it should not be null ever, you always must control this value. This property exist to make sure that in case we have one job or execution that is longer the minTime value we set, it will not allow more jobs to start on different threads because it can break all of our logic and math.

Once we constructed our instance we want to use the wrap method on our function that is responsible to get the data, we pass that function as an argument and get a new throttled function.

Let us see an implementation with the latest code snippet from earlier. We will attempt to get 200 records and see how long it takes us with the configurations we set.

  // main.js

  const { default: Bottleneck } = require("bottleneck");
  const axios = require('axios');
  const url = 'https://jsonplaceholder.typicode.com/todos/';

  const limiter = new Bottleneck({
        minTime: 200, //ms
        maxConcurrent: 1,
  });


  const main = async () => {

    const requests = [];

    for (let i=1; i<=200; i++) {
      const throttled = limiter.wrap( axios.get(url + i) );
      requests.push( throttled );
    } // for


    console.log( Date.now() );
    await Promise.all( requests );
    console.log( Date.now() );

    /* rest of your code goes here */

  } // main


  main();

You can see that there is a bit going on here. First of all we create a throttled function on the fly inside the loop with a particular argument and push it into an array of throttled functions. We then await on the requests array with Promise.all to run all requests in the same time so we would receive one response. The responses are ordered by the requests in the responses array. We print in the terminal the timestamps before and after the promises resolves and we will see what is the time difference in milliseconds. According to our math we should get a roughly 40 seconds in total to get 200 records, this may vary with your hardware and link and add couple more seconds to the total. Once you have the data you can do what ever you want with it and it has no more need for the remote API until the next time.

Pay attention that we do not await on the wrapped function. I want to say also that we do not need to this on the fly, but rather to create a function that is responsible to create the Http request object with axios and pass it to the 'wrap' method, but we will still need to pass the argument to the throttled function inside the loop. I chose to do it like this because it is a simple implementation, but in case we have a more complex logic then definitely we will create a helper function.

I hope this was very informative for you and would become very useful in the future. I also recommend reading the documentation of Bottleneck, it has more to offer than what we'd cover in this article.

In a personal note i would really appreciate if you could provide some feedback on what you are reading, this would help me a lot. I am talking about my English skills, or something that i missed to address in the article itself, whatever you find can be very valuable for me to improve.

Stay tuned for next
Like, subscribe, comment and whatever ...
Thank you & Goodbye

AFACICD Like A Boss - Azure Function Apps CI/CD pipelines w/ Azure DevOps 🚀 ⚡

alxizr — Mon, 26 Jul 2021 21:38:32 +0000

Hello everyone,

Thank you for joining in for this article about creating continuous integration and continuous deployment for Azure function apps using Azure DevOps. We will make it today a little discussion about taking it from end to end creating all that is necessary from scratch. We will see what Azure function apps are, how we can get started with it and where we use it all around with those little nuances that we may have missed here and there.

Prerequisites

Before we begin, as always we want to know what is the minimum for us to be able to start, be efficient and productive.

Azure comes with free tier so you can jump right in with no worries. If you already have an Azure account then we would only need to create a function app and the DevOps project.

Prologue

I would like to provide with a simple and short description about what our goal is right now. We are not going to write some robust piece of code that reinvents the wheel, but rather we will create and use the serverless service provided by Microsoft Azure cloud platform. I want to tell you about the reason why we actually do need sometimes to harness this ability instead of writing our own Node.js web api, or any other web framework for that matter such as .Net or Python, etc.

As you can understand, Azure Function Apps are the serverless implementation that Azure provides us. If you know Firebase Cloud Functions or AWS Lambda or any other similar service, it serves the same purpose. Of course we also need to keep in mind that there are different types and templates for Azure functions, and we are going to implement one running on Node.js and it will be triggered by HTTP calls. I know what you are thinking right now.. it sounds exactly like reaching out to an API endpoint but with one major difference and that is we don't write the entire API but we write only the piece of code for one particular endpoint!

Serverless abilities provide us the choice to get up and running with tasks that may be recursive over a time period like reaching out to a remote / 3rd party API to get data from and write it into our database, or maybe we need to run some logic when certain events are triggered, maybe using a message queue or file storage. No matter the reason is, there is an entire service dedicated for these kind of tasks and we need to know how to make them talk to each other. A big advantage using serverless is that in the long run it can actually saves us some money and it is actually considered a best practice! It does so for several reasons and I will note mine. First of all serverless functions are powered off until they are triggered, there are couple types of triggers as mentioned above. Second of all due to the reason that you do not need to write the entire API, you are left with plenty of time to work on other tasks. You are focused on specific business logic and need only to implement it and worry about anything else. This way you progress faster and accomplish more work in the same period of time. Third of all you need to understand that serverless service is a managed service provided by the cloud provider and its resource management for scale and elasticity is automatic and none of your concern. Azure actually lets you use the compute power for running serveless for 1 million executions for free before you start paying for the actual compute power and even then its very low. You can read more in the Azure function Apps pricing section. Forth of all and this is my last note is that the serveless functions can support multiple programming languages (major once at this point of time) and you do not need to worry for missing out this generous service.

Now that we got this out of our way, let's get down to business and do something, because all we did so far is talk (or read actually).

Make theory become reality

People who know me as a programmer know that i am not a GUI person and i like coding my way to the target, but since this is somewhat of a tutorial and i assume some of you are not familiar with all these services in Azure, we will then use a GUI to create what we need. In the future, when i will have more free time and not so busy with work, we will see some advanced stuff going on Azure and there is no GUI there boys and girls.

We will start with creating Azure function app with a Node.js template using Javascript programming language. We then inspect what the template consist of and explain some nuances. Once we get this out of the way we will proceed to Azure DevOps and do what is needed there and be done for today.

Let's open vscode and install the extension named Azure tools if you have not done it already. I provided the link to it above.
You may need to login to your Azure account to have the resources available to you. We then click the icon on the side bar and locate the functions section.

If we hover on the Functions row we will get a menu. We want to choose Create Function option, the symbol of the lightning.

We will be prompted that in order to have a function we need to create a project, so we will choose Create new project. This is a logical container that will hold all of our functions in one place. Obviously if you have an existing project, then we can select it and carry on from there.

We will create a folder on the Desktop or anywhere else you want and then we will choose the programming language, here we want to select JavaScript.

The next step is to choose the type of the function and here we want to choose HTTP Trigger. There are many types of functions and we will choose the one that fits the demand.

Now we need to name this function, usually I will name it after the project it belongs to and the business logic it implements, for instance _GetAllNotes, but since this is an example, I will leave it as it is for the default name.

The next step is to choose the authorization level, it means who or what can access this function. We have 3 options and we want to choose Anonymous. It means that anyone, anything, anywhere that can speak HTTP can access the function to be able to trigger it. Once you get the hang of it, you will decide the access rights for your functions. In actuality, we want to see and implement for the general use case and only after we are proficient with the tool, we would be able to implement for the specific use cases.

Now it is the final step we need to tell vscode that we want to open the project in the current window so we could be the root of the containing folder.

Congratulations! Your first Azure function. You can see that the template gave us a sample code that returns a text message. Nothing fancy. Nobody cares. Now what ? So, as i said earlier, we want to inspect the template we got, we need to understand the folder structure, we need to understand the different config files and eventually we will need to understand the actual code so we could manipulate it as we fancy.

As you can see in the image, the root folder is actually the project we opened during the process of creating the function. It contains files and folders. The folders we see are the vscode settings for the Azure functions project itself for local development and the function itself named as we chose during the function creation process.

Let us inspect the files we have in the root folder. The first file we inspect is the host.json, it holds the configurations for this specific Azure function project and holds 3 configuration options or more. This file holds global configurations that affect all the functions that are part of it. The first one is the version of the Azure functions runtime available to us for use and it is 2. The second option is actually also version, but there is a difference, because this version is pointing on the extension bundle of the function, basically, the programming language runtime you decided to work with and the library used for it. In Javascript actually there was a major update not long ago migrating from v2 to v3. The third option is logging and Azure provides us with a built in tool named Application Insights. We can choose to enable it if we want to use it, in my honest opinion, you should, even if you are using another tool of your own. The second file we inspect is the local.settings.json, it hold the minimum configurations to enable you as a developer, run and test your function code locally in your environment as if they were running in the cloud. You will need to provide the values for the storage account this project is using in order to store the deployed version of the code and the runtime you are targeting. You should also provide the local project you running in your computer, the value for the traffic whether to encrypt it or not.

This file, by the way, is ignored by GIT when you commit as it holds sensitive data that you are not interested to expose to the world.

The third file is proxies.json, we will not talk about this today as it is a bit of an advanced feature for our discussion today. I will explain what proxies are and in the future, which is not today, we will see them in action. So proxies are basically a tool to specify routes or endpoints in your function app project that are implemented by a 3rd party resource (maybe another function app that out of your responsibility). It allows you to present a microservices architecture under the hood while exposing one single and unified API for the consumers. There are, in addition to what we'd mentioned, another 2 major features to proxies, and it is the ability to modify the requests and responses of the API and also version your exposed API. Sounds very interesting and leaves us intrigued for the upcoming.

We are done with the files and now let us inspect the folders. In our newly created project we can see 2 folders. One named .vscode and the other is the name of our recently created function HttpTrigger1. If you are using vscode as i am then you can see the .vscode folder. We will ignore it because it holds the configuration for vscode to support the function app inside our local development environment. It is created when you create a project from the vscode Azure tools extension as we actually did earlier. This folder right off the bat, comes with 3 files and we are only interested in the index.js file as this is the file that will hold our actual code. The function.json file holds the configurations affecting this particular function only. Should you recall the global configurations file host.json, this file scope is the function only and that is it. This file is pretty self explanatory and we are ignoring it.

Deep Dive Function

I would like to take a closer look at the template code we received. You can see in the first line of the code the expression module.exports, in Node.js when we want to export by default we use it. Here the function exported as asynchronous exposed with 2 arguments, context and request. The request is holding the data we receive from the client consumer of the function and the context holds the invocation information of the function. As this is a managed service in Azure the function is running in a process of its own and we use the context object to pass information in and out the function. If you ever used Node.js web framework called Koa, then this is probably the most exact same.

In our example we are implementing a function that its trigger is an Http call. Methods like Http GET or Http POST are the way to consume this function. We are only left to write the code for our use case. One important thing we should always keep in mind when writing functions is that they are limited in their lifespan. A function can live only for a maximum of 15 seconds or 15000 milliseconds. So what ever you are implementing, make sure that its not time consuming. Now, before we proceed to the next part, if you take a glimpse in the file function.json, you can see that the allowed Http methods are GET and POST. We will leave it for testing later.

How to CI/CD

After we have created the project and the function we can actually use the vscode extension Azure tools to deploy it, this way we are creating the basis for the CI/CD that we want to implement. Let us go on and deploy the function and carry on for the next part.

Back in the Azure tools extension we will hover on top of Functions and click the Deploy button. We will choose our subscription. Then we will choose Create new function app in Azure advanced mode . We will provide the name we want. We then choose the runtime stack Node.js. We now select the OS Linux. Afterwards we want to choose the resource group we want this project be a part of. An important note: if you don't have a resource group already exist, do not create it here with the tool, but rather log into the Azure portal and do it there, it is very simple. From my experience there was always something wrong with creating the function and the resource group in one go. Next we choose the location near us to deploy the function to, i will choose west Europe. Now we need to choose the plan, here we are going on Consumption, which is basically free tier and then pay as you go once you go over the top, but know that when you will need to implement some serious stuff with Azure Functions App then Consumption plan is not the one you want to choose. We will talk about it in the future as this is out of this discussion scope. Next in line is the storage account for the functions app and you can create one on the fly if you do not already have and existing one that you want to use. Next is Azure Application insights and we will skip it now, we will enable it from the portal later.

Generally we do not want to do it this way only because we are not the only ones who may need to work on this function and we want to have a copy stored in case something bad will happen with our computer, but for the initial step it is inevitable or we could do it from the Azure portal, whatever you find easier.

Before we do anything we need to save the changes in our functions project. Back in vscode verify that we are on the master branch.

We want to commit everything. Inside the terminal we will execute git add . and the next command is git commit -m "first commit". We cannot push it anywhere because we do not have a repository in this project defined yet. This is where we all think about GitHub repositories and we are actually going to use Azure Repos for that, so let's go there (link provided above). If you do not have an Azure DevOps account then this is the time to create one, otherwise lets open a new repository. There is a major reason for you having the Azure DevOps account set and it is the ability to connect the functions app project to the deployment center automatically. There is an option to add it later manually, but this is can be a bit tedious, so pay attention please.

We will click on the button New Project located on the top right corner.

We will provide the name for the repository and a description which is optional, we will choose to set it public so we do not have to bother with authentication and authorization for the repository, not at this point at least and we will click the Create button at the bottom of the pop up screen.

Next step is to upload our function app's project into the repo, so once the repository project was created we were navigated to the Summary screen, we will locate the Repos option under the left hand menu and click it.

Since this is a clean repository we have 2 options here to add code, Either we clone the repo and start all from scratch or we can add an existing project, we will choose the later.

Locate the 2 commands under the Push an existing repository from command line title. We will copy the commands one by one and go back to our vscode terminal and execute it.

Once you are done a quick page refresh will show you the code.

As we created the repository and uploaded our project to it we are now ready to proceed and create the pipeline that will implement our CI/CD. Having the code here is not going to do much, no matter who is working on it. The first thing we will do now is create a pipeline that will eventually create a deployment bundle for our function. This pipeline should be triggered automatically when we commit new version of the code. We will work with the master branch now, but when you will take it to the next level, by all means, create couple more branches with triggers and pipelines for them as well.

Locate the Pipelines option under the menu on left hand side and click it. Once you see the pipelines screen click the Create Pipeline button on the bottom right hand side.

In the screen in front of us locate at the very bottom the link Use the classic editor and click it.

We now need to choose the source of our code and it is Azure Repos Git. We will verify that the repo project is the one we want and the branch we want to target. Finally click Continue.

The next step is to select a template. Basically we could go on with an empty job and add the relevant tasks manually, but luckily Azure was resourceful enough to think ahead and create a template just for this use case (Thanks guys!). In the search field look for the phrase function node and select Apply on the item in the results named Azure Functions For Node.js

Once we are inside the pipeline task section we can see all the tasks provided by the template. No need to touch anything here, but we need to make a change on the Build pipeline change the Agent Specifications to Ubuntu-20.04. If you remember when we deployed the function we set the OS to be Linux, we need to have the same environment for the pipeline as our runtime.

Next we locate the Triggers tab at the top and select Enable continuous integration and Batch changes while a build is in progress. This means that every time we push new code to the master branch this pipeline will be triggered. Finally click Save & Queue.

In the pop up window verify all the settings and click Save and Run.

The pipeline is now running and creating our bundle that will be used for deployment. Our next step is to create the deployment pipeline. In order to create it we need to locate the Releases under the Pipeline menu and click it. Since we do not have any release pipeline we need to create one by clicking the New Pipeline button.

This step is fairly easy because we do not need much done here for Azure function Apps deployment. But these release pipelines can benefit you a lot if you choose to dig a bit deeper and see what is hiding under the hood.

In the screen in front of us we can look for a template for this use case again as we did with our build pipeline. Search for the phrase function and click Apply on Deploy a function app to Azure Functions

Once we are inside the release pipeline screen the first thing we need to do is to fetch the build artifact the the build pipeline generated. On the left hand side there is a step called Artifacts, locate it and click add. You will see that on the right hand side there is a pop up window and we want to select there the artifact. Since we did not change the name of the build pipeline you should have the same name as mine that was generated by the template. Once selected click Add button at the bottom.

Still on the artifacts we now want to enable the continuous deployment trigger, so click on the lightning button and switch the radio button to enabled mode. To close the pop up window click the X on the top right hand side.

Now we will proceed to the Stages section on the right hand side of the screen and we will click the 1 job, 1 task link in the empty stage we already have Stage 1.

In this stage we will need to provide our Azure subscription, App Type which is Function app on Linux and the name of the Azure function we deployed earlier manually.

Once you are done filling this information click Save and afterwards Create release buttons at the top. What will happen now is that the release pipeline will start working and will take the code for the function we created as an artifact by the build pipeline and deploy it to the function app itself. Obviously nothing is going to be actually different with the function code because we did not change it. Well, we did not change it yet. All we care about now is to make sure that the pipeline is working and we will test everything later.

Epilogue

Hopefully by now the release pipeline is done and the deployment process was successful. Now let's test it from the far end of the code, so going back to vscode, we want to change the code to something simple.

module.exports = async function (context, req) {

    context.res = {
        body: "This is the change we made to the original function so we could test our ci/cd!"
    };

}

As you can see, nothing fancy. We now will use git to commit this change on the master branch. Make sure you have the Azure DevOps window opened in front of you so we would be able to see the triggering of the build pipeline and the release pipeline. Ok, now back in the vscode, open the terminal and execute the next commands:

git add .
git commit -m "first change to test cicd"
git push

In a second or two you should see the magic happens. If you want to test it to see that it is working as expected, it is very easy. Let us open the Azure portal and search for the phrase Function App in the search bar. We locate the project we created earlier and on the left hand side menu locate and click the Functions option. Now we will select from the list the name of our function and navigate to its dedicated screen. In the menu on the left hand side locate Overview and select it. In the middle of the screen there should be a button that says Get Function Url. Copy the url to a new tab and see what you get, or you can use postman for it. Hopefully the deployment was done and you get to see the updated version of the code.

One last thing before we are done for today. We wanted to set the Azure Application Insights that we skipped during the deployment of the function app. I wanted to show you one manual step in the process of creating a function app and selected the easier of them all. If you go back to the Function App page in the portal and select the project we created, in the menu on the left hand side look for Application insights under settings. Click enable on the switch, choose the storage account to store all the logs generated by the application insights that we created earlier for the project and click Apply.

Some final words, We saw in this discussion how we can go from 0 to 100 with Azure Function Apps and the minimum resources that it takes for it to work. Starting out from resource groups, moving to storage account and application insights that integrates into the function apps. We also saw how in a few and not so difficult steps, were able to create a build and release pipelines for the continuous integration and deployment process, and in the end of it all we were able to test it with no effort. I am planning to go deeper with this function apps soon to show the more advanced stuff we can implement and i hope you will enjoy and benefit it in some way.

Stay tuned for next
Like, subscribe, comment and whatever ...
Goodbye

😡 Agonies, Despair and self hosted Redash BI server on Microsoft Azure cloud platform 😋 part 2

alxizr — Mon, 21 Jun 2021 12:21:09 +0000

Hello everyone,

Thank you for joining in today for part 2 to talk about the open source BI tool named Redash and how to set up a self hosted Redash server in the cloud.
We finished part 1 with a working instance in our local development environment and now we want to proceed on to the cloud. We will continue on from where we left off and if for any reason at all you didn't read the first part, i would suggest that you do that first.

Moving on to the cloud (Azure)

Another thing i wanted to talk about is the base for this setup. In the website there is a section for setting up a self hosted instance, here is the link. In order to see the full picture we need also to visit the GitHub project. The issue i have with this specif setup is that in aims using a dedicated vm that runs the Redash using docker compose. This is a very wasteful approach in terms of resources and cost. We decided to change the approach for our use case utilizing Azure App Services that can run a multi container application with docker compose. We also need to remember that we want to utilize the cloud platform and not create all the services locally in the web app because of the management involved in it regarding the database. We want to use a managed service from Azure for it. Azure has it and its called Azure Database for PostgreSQL servers.

Let's list what we will need for our setup in Azure. As we already said, we will need a multi container web app under Azure App Services, we need a managed PostgreSQL database under Azure Database for PostgreSQL servers and of course we can't move forward without an account with Azure.

We will start from easy and proceed to the more challenging steps we need to accomplish in order to have a working instance.

Step 1 is to create a managed PostgreSQL server in Azure. In order to accomplish it locate the search bar on the top of the screen in Azure portal and type Azure Database for PostgreSQL servers. Click on the option and navigate to the dedicate page. Locate the create button with ➕ sign on the top left hand side and click on it. In the next screen we will need to select the plan we need for production use so we will choose 'Single server' and in the following screen you will need to fill the details for this instance. Choose a subscription and create new resource group name, provide a name for this instance, choose location that is near you, leave the data source as 'none' and choose the PostgreSQL version. Pay attention that in the 'Compute + storage' option to choose 'General Purpose' this is very important. At the bottom you can provide the credentials for the admin user to PostgreSQL server, save these details because we will need them soon. Click next until you get to 'Review and Create' and click create. This may take couple of minutes. In case your review fails then go back to the 'Tags' tab and fill in these values: BusinessOwner, BusinessUnit, Application, Environment if any of them are empty. Once we got it to get into the creation process we can go on to the next step, as i said it may take a few minutes and we want to be efficient and not to waste time.

screen shot 1

screen shot 2

screen shot 3

Step 2 is to create a multi container web application under Azure App Services. As before let's locate the search bar on the top of the screen in Azure portal and type App Services. Click on the option and navigate to the dedicate page. Locate the create button with ➕ sign on the top left hand side and click on it. You can see that the window we got is very similar to the one we had before when wanted to create a managed database server. Fill the details for this web app instance. Choose a subscription and the same resource group name you created in the previous step, provide a name for this web app instance, choose a region that is near you, under the publish option choose Docker container, under the operating system option choose Linux and choose a plan for the web app. Pay attention that we will not drill down yet with the web app configurations until we create it and proceed with this article. We want to understand what we need to proceed with it. The default is an empty Nginx docker instance.

Step 3 is to create a docker compose file that we can apply on the multi container web application. Now, in theory you could use the one that we already have from part 1 of this series and we are going to do this actually and then we will change what we need in order to fit it to our needs. We discussed about why we don't want to do it. This is the reason we create in the first step a managed PostgreSQL server. Hopefully by the time we are done with step 2, both the web app and the database server are created. You should see the progress under the bell icon 🔔 on the top right hand side in Azure portal. Once the web application created lets navigate to it. You can either look it up in the search bar or via the App Services screen pick it from the list. As we are in the specific web app management screen we need to check that the web app is working with a public IP address. You should be in the overview page, on the top left next to the menu, locate the 'browse' button and click on it. You should see a new web page that shows the default Nginx welcome page. We confirmed that the web app is up and alive. Now locate in the left hand side menu the option called 'App Service Logs'. We want to enable the log stream from docker into the web app itself. Under 'Application logging' choose 'file system' and under 'Quota' and 'Days' choose 50 (it doesn't really matter) and click save. Now locate the 'Deployment Center' in the same menu. In this screen under the 'Settings' tab first of all let's set the source to be container registry. Now under the 'Registry settings' we will choose container type to be docker compose, registry source to be Docker hub and the repository access to be public. In the config area paste the content of our docker-compose.yml file from the previous article and click save. Scroll up to the very top and click on the logs tab. You will be able to see the process of the image pulling and starting up docker compose. Once it is done you can click on the browse button to navigate to the public web page of this web application or if you didn't close the tab we opened earlier when we checked that the web app is alive just refresh it. You should at this point, in theory, see the same screen, Redash welcome window where you need to create the admin user, but we know that this is not the case. We need to go through the same process as before to create the management database and run the setup script. Well, we can't do that here! There was a time, not long ago, with the previous version of the Azure portal and the web application where we could specify a command that runs on the first startup, it is not available anymore. Let's see how we can fix our situation.

If we pause for a moment and review what happened until this point, we would basically think that not much is different from the local environment except for the fact that our 'local' environment is now a web app that is running in the cloud! Well, yes! And this is the exact experience i was trying to make you go through. Some of you who maybe paused in between the readings and read the information behind the links that I left would see that the process and the experience that Redash docs takes you is just insane. There are nuances to pay attention to and development environment is not the damn same as the production one, which is crazy, in my opinion! Why would i have 2 or more completely different configurations that are not keeping me in harmony with the service 😵 ?! We all know the first and simple rule - keep your development environment as close as you can to your production environment. You could also examine the docker compose configurations file carefully and find that even in development we didn't use the suggested development setup, we went all the way to prod from the start. Surprise! 💥💥💥

As we reveal this discovery, we now want to pay attention to the configurations themselves and we understand that we can omit the PostgreSQL database service along with the pgAdmin service. We could argue that we don't actually need the nginx service as well, but there is a reason its here, so we are going to leave it (we will talk about this later in the future). Now, what is about the Redis server ❓❓❓ Well , if you think that we should drop it as well together with the pgAdmin and PostgreSQL you are right, but this is actually would be the wrong thing to do. I will explain it momentarily. Let's fix the cloud version docker compose configurations file, update the multi container web application so it will do its thing and then fix our Redash server, but before that we need one very important piece of information and its the connection string for the managed PostgreSQL server. Navigate the instance we created and on the left hand side menu locate the option called connection string. This is the format of the connection string, just copy your details to replace the place holders.

There are couple of reasons we decided to leave the Redis service locally inside the web app running in a seperate container. The first is that Redis use case in the eyes of the Redash server is used for queueing and caching only, since it all happens in the memory, we need the memory to be available to it from the same machine. The second is the speed, caching or not, these containers are all in the same place and there is no latency caused because we need to be dependant on a remote service. The third is actually pricing. Spinning up a Redis resource or Azure container instances running Redis in a different remote location is a bit costly and not efficient.

    // Azure PostgreSQL connection string format

    postgresql://<your-admin-username>@<your-managed-server-name>:<your-password>@<your-host>:5432/redash?sslmode=require

As we have this connection string ready for us to use, let's update it first in the file. Locate the key REDASH_DATABASE_URL and update the value with the connection string. Pay attention that we do specify the name of the actual database name that we are looking to get access to and its 'redash' on purpose.

x-environment: &base_environment
  PYTHONUNBUFFERED: 0
  REDASH_WEB_WORKERS: 4
  REDASH_LOG_LEVEL: "INFO"
  REDASH_RATELIMIT_ENABLED: "false"
  REDASH_REDIS_URL: "redis://redis_server:6379/0"
  REDASH_MAIL_DEFAULT_SENDER: "redash@example.com"
  REDASH_ADDITIONAL_QUERY_RUNNERS: "redash.query_runner.python"

  REDASH_DATABASE_URL: "YOUR CONNECTION STRING GOES HERE"

Next we will locate the pgAdmin and the PostgreSQL services and delete them.

# delete these 2 services under the service object

services:
  # database
  postgresdb:
    image: postgres:alpine
    restart: always
    container_name: postgresdb_server_local
    ports:
      - "5432:5432"
    environment:
    POSTGRES_HOST_AUTH_METHOD: "trust"
    POSTGRES_USER: postgresuser
    POSTGRES_PASSWORD: postgrespassword
    POSTGRES_DB: redash
    volumes:
      - ./postgres-data:/var/lib/postgresql/data

  # pgAdmin
  pgAdmin:
    container_name: "pgAdmin_local"
    image: dpage/pgadmin4
    restart: always
    ports:
      - "11180:80"
      - "11443:443"
    environment:
    PGADMIN_CONFIG_ENHANCED_COOKIE_PROTECTION: "False"
    PGADMIN_DEFAULT_EMAIL: pguser@mail.com
    PGADMIN_DEFAULT_PASSWORD: pgpassword
    depends_on:
      - postgresdb

    volumes:
      - ./pgadmin:/var/lib/pgadmin
      - ./pgadmin/backup:/var/lib/pgadmin/storage

There is one small step to do before we forget and it is to delete the dependency on the local PostgreSQL service under the Redash server service. We also want to delete the port 8080 as we do not use it at all in the web app because we are using the nginx reverse proxy.

    # delete the postgresdb dependency
    services:
        # redash server
        server:
            <<: *base_redash
            command: server
            ports:
              - "5000:5000"
              - "5678:5678"

            # delete this 1 port
              - "8081:8080"
            depends_on:
              - redis_server

            # delete this dependency
              - postgresdb

We are left with this file content.

    # Azure App Services docker-compose.yml Version

    version: "3.9"

    x-environment: &base_environment
        PYTHONUNBUFFERED: 0
        REDASH_WEB_WORKERS: 4
        REDASH_LOG_LEVEL: "INFO"
        REDASH_RATELIMIT_ENABLED: "false"
        REDASH_REDIS_URL: "redis://redis_server:6379/0"
        REDASH_MAIL_DEFAULT_SENDER: "redash@example.com"
        REDASH_ADDITIONAL_QUERY_RUNNERS: "redash.query_runner.python"

        # do not forget to update this key
        REDASH_DATABASE_URL: "YOUR CONNECTION STRING GOES HERE"


    x-base_redash: &base_redash
        environment:
            <<: *base_environment
        image: redash/redash:8.0.2.b37747
        restart: always


    services:
        # redis
        redis_server:
            image: redis:alpine
            container_name: redis_server_local
            restart: unless-stopped

        # redash server
        server:
            <<: *base_redash
            command: server
            ports:
              - "5000:5000"
              - "5678:5678"
            depends_on:
              - redis_server

        # redash scheduler
        scheduler:
            <<: *base_redash
            command: scheduler
            depends_on:
              - server
            environment:
              << : *base_environment
              QUEUES: "celery"
              WORKERS_COUNT: 1

        # redash worker 1
        scheduled_worker:
            <<: *base_redash
            command: worker
            depends_on:
              - server
            environment:
              << : *base_environment
              QUEUES: "scheduled_queries"
              WORKERS_COUNT: 1

        # redash worker 2
        adhoc_worker:
            <<: *base_redash
            command: worker
            depends_on:
              - server
            environment:
              << : *base_environment
              QUEUES: "queries"
              WORKERS_COUNT: 2

        # redash worker 3
        scheduled_worker:
            <<: *base_redash
            command: worker
            depends_on:
              - server
            environment:
              << : *base_environment
              QUEUES: "schemas"
              WORKERS_COUNT: 1

        # nginx
        nginx:
            image: redash/nginx:latest
            ports:
              - "8080:80"
            depends_on:
              - server
            links:
              - server:redash
            restart: always

You can save it now and upload it to the web application. If you recall we did it in step 3. Just repeat the step with this new file. A spoiler - nothing is going to change yet.

Can you guess what our next step is to make things work ? I want to challenge you actually. I can tell you that you ladies and gentlemen have the power to fix the issue without me telling you what to do! I really want you to try to think about it for couple of minutes and solve it. Just for the sports. I promise that i will tell you what to do next even if you are not successful.

⏰ ⏰ ⏰ ⏰ ⏰ Count to 600: This is your time window to solve it on your own and you got 540 seconds more than you need.

Alrighty then! We will need 2 things. The first is our local setup and the second is the new connection string. What we are going to do is actually spin up the local environment with the use of the database service being in the cloud rather the one we declared locally. So update the connection string, open a terminal, run the commands from part 1 and go back to the web application, in the overview screen restart it and then proceed to blibk on the browse button and wait to see that we have what we asked for. Just a friendly reminder go through the exact process as we did in the previous part in case you encounter issues.

First command to run:

    # local environment version
    $ docker-compose -f docker-compose.yml up

Second command to run:

    # local environment version
    $ docker-compose -f docker-compose.yml run server create_db

You can understand that we are ignoring the local environment just for the sake of setting up the remote database. I will tell you that there is also another way to accomplish the same fix and it is to do a backup on the local PostgreSQL instance and restore it in the remote instance, but this approach is way to complex comparing to what we actually did for the fix. The backup and restore option is a great choice if you are not starting from scratch. Not our scenario here.

Well,
I hope you guys enjoyed it. It surely was a bumpy ride, but heck, we learned something. It's all about the nuances.

Learn Redash

At this point we have a working instance of Redash and we can start using it. We can add data sources, write queries, setup scheduled tasks or queries, create visualizations and more. I will refer you to the official website and the YouTube channel for more information. I urge you guys to go over the Redash docs and learn how to use it. It is handy 👍

Stay tuned for next
Like, subscribe, comment and whatever ...
Goodbye

😡 Agonies, Despair and self hosted Redash BI server on Microsoft Azure cloud platform 😋 part 1

alxizr — Thu, 17 Jun 2021 10:51:14 +0000

Hello everyone,

Thank you for joining in today to talk about the open source BI tool named Redash. We will focus today mainly on how to set up a self hosted Redash server in the cloud starting from a local environment setup. We will see the nuances hidden in the process and try to 'debug' on the fly.

Motivation

You may wonder about the article's title to be a bit... maybe negative ?! Well i *** you not but the amount of energy and time wasted on solving simple issues is tremendous. I am writing this article so you potentially and hopefully won't go into the rabbit hole as a public service.

Let's hit the road!

Prerequisites

Before we begin, as always we want to know what is the minimum for us to be able to start and be efficient and productive.

Visual Studio Code
Azure Account
Docker - You do need to be familiar with docker, docker compose and yaml configurations file. In case you are not, I will suggest visiting and reading this article.

General setup

We want to understand which services integrate with Redash so we can have a minimalistic working example. We will list now what we need for us for the purpose of this article and later we will see what and where we can add additional services that you may fancy for you custom self hosted setup.

Service List
- Nginx server
- Redis server
- PostgreSQL database
- Redash server
- Redash scheduler
- Redash worker x 3 (optional, minimum is 2)
- pgAdmin (optional, GUI for PostgreSQL)

As you can see there is quite a bit to it. We need all these services to be configured and to be available to work in harmony with one another so the goal to have a self hosted Redash is successful.

Local Environment

In this step we will configure our development environment that is going to be almost the exact same as our operational cloud environment. We will use docker compose for this purpose, so install it if you don't already have it installed. I provided a link earlier in the prerequisites section.

We are going to write a docker compose file that is going to be a bit hefty. We will not use external files (.env files), because when we are ready to proceed to the cloud setup, there is no option to be dependant on external files, this way we will make ourselves used to the same perspective.

Let's first understand what these services are for in the list we wrote earlier. Redash is a python based application, so as PostgreSQL and they work perfectly together. The reason we need a database for the Redash application itself is basically for the application management and maintenance. Redash is dependant on other data sources in order to display the BI you fancy. The use of Redis server in the setup is for caching and queuing purposes and Nginx server is there to be a reverse proxy. We will see that the nginx server is a customised one for the Redash purposes. We will take a look why we need the Redash workers later on.

No we will write our docker compose file, we will fetch all our services from docker hub. I would suggest that you explore it for the specific services that we are going to be using and also maybe other services that you are interested in. In my opinion the instructions written in docker hub are a great resource.

# docker-compose.yml

version: "3.9"

x-environment: &base_environment
    PYTHONUNBUFFERED: 0
    REDASH_WEB_WORKERS: 4
    REDASH_LOG_LEVEL: "INFO"
    REDASH_RATELIMIT_ENABLED: "false"
    REDASH_REDIS_URL: "redis://redis_server:6379/0"
    REDASH_MAIL_DEFAULT_SENDER: "redash@example.com"
    REDASH_ADDITIONAL_QUERY_RUNNERS: "redash.query_runner.python"
    REDASH_DATABASE_URL: "postgresql://postgresuser:postgrespassword@postgresdb/redash"


x-base_redash: &base_redash
  environment:
    <<: *base_environment
  image: redash/redash:8.0.2.b37747
  restart: always


services:
  # redis
  redis_server:
    image: redis:alpine
    container_name: redis_server_local
    restart: unless-stopped

  # database
  postgresdb:
    image: postgres:alpine
    restart: always
    container_name: postgresdb_server_local
    ports:
      - "5432:5432"
    environment:
      POSTGRES_HOST_AUTH_METHOD: "trust"
      POSTGRES_USER: postgresuser
      POSTGRES_PASSWORD: postgrespassword
      POSTGRES_DB: redash
    volumes:
      - ./postgres-data:/var/lib/postgresql/data

  # pgAdmin
  pgAdmin:
    container_name: "pgAdmin_local"
    image: dpage/pgadmin4
    restart: always
    ports:
      - "11180:80"
      - "11443:443"
    environment:
      PGADMIN_CONFIG_ENHANCED_COOKIE_PROTECTION: "False"
      PGADMIN_DEFAULT_EMAIL: pguser@mail.com
      PGADMIN_DEFAULT_PASSWORD: pgpassword
    depends_on:
      - postgresdb

    volumes:
      - ./pgadmin:/var/lib/pgadmin
      - ./pgadmin/backup:/var/lib/pgadmin/storage

  # redash server
  server:
    <<: *base_redash
    command: server
    ports:
      - "5000:5000"
      - "5678:5678"
      - "8081:8080"
    depends_on:
      - postgresdb
      - redis_server

  # redash scheduler
  scheduler:
    <<: *base_redash
    command: scheduler
    depends_on:
      - server
    environment:
      << : *base_environment
      QUEUES: "celery"
      WORKERS_COUNT: 1

  # redash worker 1
  scheduled_worker:
    <<: *base_redash
    command: worker
    depends_on:
      - server
    environment:
      << : *base_environment
      QUEUES: "scheduled_queries"
      WORKERS_COUNT: 1

  # redash worker 2
  adhoc_worker:
    <<: *base_redash
    command: worker
    depends_on:
      - server
    environment:
      << : *base_environment
      QUEUES: "queries"
      WORKERS_COUNT: 2

  # redash worker 3
  scheduled_worker:
    <<: *base_redash
    command: worker
    depends_on:
      - server
    environment:
      << : *base_environment
      QUEUES: "schemas"
      WORKERS_COUNT: 1

  # nginx - pay attention to the image name
  nginx:
    image: redash/nginx:latest
    ports:
      - "8080:80"
    depends_on:
      - server
    links:
      - server:redash
    restart: always

Once we finished writing our docker compose yml configurations file we are ready to spin up our local environment and see it in action. We want to open the terminal in the same directory where we saved our docker-compose.yml file and also open docker desktop. Now run the next command.

    $ docker-compose -f docker-compose.yml up

You can see that the docker images are pulled from docker hub and and all the links are created one after the other. In the end of the startup you should see all containers available and colored green to indicate that they are healty. You also can see that the terminal is now blocked because of all the logs we that are streamed to it. We did it on purpose actually, we needed to see the logs. Obviousely we could run it in detached mode. I guess that maybe you want to see if Redash is working now? Let's locate the nginx container inside docker desktop, hover on that row and click on the 'open in browser' button.

Surprise! it doesn't work!! But why ? We did everything right. Well that is the first neuance. There is one more command we need to run to make it work locally. As our terminal is busy with the logs stream, I want to open a new terminal and run inside of it a one time command. You can close it after we are done.

    $ docker-compose -f docker-compose.yml run server create_db

This command tells the Redash server to run a pre configured database setup script so it will be able to manage itself. You need to pay attention and make sure that this command ends successfully other wise no Redash for you!

I would like to take the opportunity now and talk about another neuance here. If you remember we are using a specific version of Redash image (look at the anchor declaration), other images, older ones, may fail and crash during the last command that we run to setup the database. In case that happens, pay attention what you need to do, the solution is very easy but it took us a very long while to figure it out. You will have to create a database named 'redash' manually inside the postgresql server instance. I suggest using the pgAdmin instance that we listed as one of our services availabe locally. You can either use the same details as i wrote here in the connection string and the environment values that i set for postgres or you can change it. What ever you like. So once you connected to the postgres server instance and created the 'redash' database, just run the last command again.

Now let's go back to the browser and refresh. VOILÀ! At this point you should be seeing blanc, fresh Redash instance welcome screen that is asking you to create the first account with username as email and password to take the role of an admin.

As agreed in the beginning, we are not going to actually create the dashboards and actually make use of the BI capabilities. We are only talking about the setup here. Once we are finished with also creating our Azure cloud environment for Redash, we will take a look at some other configurations we need in order to make use of it properly.

I want to discuss about the docker compose configurations file. You can see that we make use of YAML anchors here. We actually have 2 of them. One for describing a Redash base service and the other is for the environment variables we want to pass to each instance. The service named 'server' is the one that runs the Redash application and it gets everything that the base_redash object contains, the Redash workers on the other hand, well they need a specific configurations on top of the base_redash object. We obviousely could write everything inline but i guess you can understand that we would get a mile long configurations file. If we can reuse instead of replicate then this is what we will do. The workers fill in for particular roles and each of them needs a specific configurations to do it. We pass via the environment these configurations, as you can see we actually extend and overide the base_environemnt object with each worker instance. The special '<<' double chevron right sign is used inside the YAML language to merge and override key: value pairs in the same object. What we did was declaring key: value pairs in the global obejct and override it as we needed to and also we added additinal keys QUEUES, WORKERS_COUNT that holds different values per worker (again, i will recommend reading the article i linked in the beggining to understand YAML better).

As this discussion / guide is getting to be a bit long, I decided to split it to 2 parts. We will end now this part with a working local environment so you could explore a bit on your own and I will release the second part once i finish it hopefully over the wekend.

Part 2 is coming...
Stay tuned for next
Like, subscribe, comment and whatever ...
Goodbye

That time when you thought you knew Y(A)ML 😵

alxizr — Thu, 10 Jun 2021 12:21:31 +0000

Hello everyone,

Thank you for joining in for this article about YAML files. We will make it today a little lightweight tutorial about YAML configurations file. We will see what it is, how we can get started with it and where we use YAML files all around but maybe missed those little nuances.

Y(A)ML is a data serialization language and it is a strict superset of JSON (javascript object notation). It is data oriented structured language used as an input format for different software applications. We can deduct that the language in the end of the day consist of key:value pairs. YML's goal is to be more human readable in a clean and consise manner.

We often use tools available for us by interacting with a GUI interface, but we don't realize that under the hood there is nothing more than a YAML file that is storing our personal configurations for the given task. We will take a look in a couple of examples here today along side learning the language.

With YAML we have 2 main types: Scalar and Collection. When we were young and went to high school we had our physics class and we learned that a scalar consist only of value that describes a size, this is not very far fetch with YAML as well. It means that we can have only one unique key that can hold a value and if we use that same key again in our file, we will override the original value we set earlier. For example if we want to declare a variable (key) 'NAME' to the value 'Joey' then this variable, the key itself, is unique and we can use it globally in the file.

    # key : value
    NAME: Joey

If we are not careful and declare that variable again to a different value, for example 'Chandler' then the last instance will override the original value.

    NAME: Joey

    # ...
    # other
    # yaml
    # configurations

    NAME: Chandler
    # this line will be the only source of truth when the file is evaluated, thus overriding every instance of the key NAME beforehand

A collection is basically the same, it also consist of key:value pairs, but one key can hold multiple values. For example a list of names.

    # list
    NAMES: ["Joey", "Chandler", "Ross", "Phoebe", "Rachel", "Monica"]

Another way to describe the same NAMES list or sequence is as such

    # list or sequence
    NAMES:
      - "Joey"
      - "Chandler"
      - "Ross"
      - "Phoebe"
      - "Rachel"
      - "Monica"

A collection in YAML is described not only in the form of an array but also can be described with maps. For example if we want to describe a mailing address of a person. Let's keep it simple for now. The address consist of street name, street number, city, state, zip code. Let's see how we can convert this address to YAML, we will choose the address of a Pizza Hut somewhere in the USA.

    # yaml object
    address:
      street_name: North Mathilda Avenue
      street_number: 464
      city: Sunnyvale
      state: CA
      zipcode: 94085

As we can see here, we have a key named 'address' which holds multiple key:value pairs inside of it. You need to pay attention to the indentations. When we want to group multiple key:value pairs under one logical container that is the parent, we must indent them with preferred 2 space characters and each new line must be aligned vertically otherwise the YAML file will throw an error when it is ready to execute.

This particular description is called a 'Map'. The map name is 'address' and it holds several pieces of data that are in the usual form of key:value pairs. You also can pay attention and see that the values are not only of type 'String' but can also be 'Number', either integer or float and also can be boolean. By the way, for strings the quotes are optional. We can also define a Date variable but need to pay attention that the date format must comply to the ISO 8601 standard which looks like this: 'yyyy-mm-dd hh:mm:ss:sss'.

    # dates ISO 8601
    some_date: 2018-30-09
    some_datetime: 2020-10-01 09:10:30

Since we know that YAML consists of key:value pairs and is a superset of JSON, we are able to describe map objects json style.

    # json style map object in YAML
    person: { name: "Johnny", age: 35, single: true }

I would prefer not to mix these 2 styles because to be honest, Sometimes we write very large YML files that are mile long and talking from experience here, if i get something wrong it is very unpleasant to debug. The fact that you can do it doesn't mean that you need to do it.

So far what we talked about was the types and saw particular samples that are a bit plain. Let's see an example where we can start complexing things. In this first example we will see how we can combine maps and collections. Let's say that i want to represent a list of people and represent this list as a collection of map objects.

    people:
      # method 1 - JSON style map object
      - { name: Alex, age: 18, single: false }

        # method 2 - YAML map object
      - name: Eric
        age: 19
        single: true

        # method 3 - another YAML map object, pay attention to the line break
      - 
        name: "Sam"
        age: 22
        single: true

As we can see here in this example, we have declared a variable (key) named 'people' and it holds multiple objects that are of the same format. We also can see that the way we declared each map object is different and we use 3 different methods to describe the same format for a map object but they all look the same for the YAML. One point we need to mention is that we can nest as much as we need to. If the person object has a property that describes 'hobbies' for example, we can add it thus creating a list object containing a list. Let's see it in an example. I will use the previous collection as a reference.

    people:
      - name: Tamara
        age: 20
        single: true
        hobbies: [movies, sports, food]

      - name: Julia
        age: 25
        single: false
        hobbies:
          - movies
          - sports
          - food

        # pay attention to the nesting
      - 
        name: Elaine
        age: 29
        single: false
        hobbies:
          - movies
          - sports:
              - swimming
              - hiking
              - dancing
          - food

Thus far we covered the types and how to use them. Now we will take a look at some features YAML supports. We now will take a look at formatting. In case we have a key that should hold large amount of data such as description of the particular object there are 2 ways to format it. We will use either the chevron right '>' or the pipe '|' signs. The main difference between them is that the formatting is either preserved or not. The chevron-right '>' sign will not preserve formatting and the pipe '|' sign will preserve the formatting. The reason we use formatting is make it more readale for us as humans, YAML will render everything in one line under the hood. Let's see it in action

    # no formatting. the text is written in one line

    car:
      model: Toyota
      make: 2021
      description: "Awarded Green Car Journal's 2020 Green Car of the Year®, Corolla Hybrid even comes with an enhanced Hybrid Battery Warranty that lasts for 10 years from date of first use, or 150,000 miles, whichever comes first"

    # chevron right '>' sign will not preserve the formatting; no need for quotes

    car:
      description: ">"
        Awarded Green Car Journal's 2020 Green Car of the Year®, 
        Corolla Hybrid even comes with an enhanced Hybrid Battery Warranty that lasts for 10 years from date of first use, 
        or 150,000 miles, whichever comes first
      model: Toyota
      make: 2021

    # pipe '|' sign will preserve the formatting; no need for quotes

    car:
      description: "|"
        Awarded Green Car Journal's 2020 Green Car of the Year®, 
        Corolla Hybrid even comes with an enhanced Hybrid Battery Warranty that lasts for 10 years from date of first use, 
        or 150,000 miles, whichever comes first
      model: Toyota
      year: 2021

Congratulations! You now covered all the basics to start using YML like a pro for your everyday work. We have one other topic that we need to cover and will take a look at it in just a moment. I would like to point out a small nuances beforehand.

YAML also supports other features which we did not discuss here in this article and the reason i chose not to discuss about them is because the use cases that apply for these features are very marginal when you should justify the use for them. Features like tags that are used for explicit types, tuples, setting keys not as strings, paragraphs and more. You can read more about in the official YAML docs.

If you do want that i will demonstrate some examples then just let me know and i will make another short part focusing on these features.

The one big feature that YAML also has called anchors and i often see that people do not really use it because of different reasons. To be honest i don't really know what is so scary with anchors and think that the value we gain by using them is huge. Anchors enable us to duplicate configurations or content and even inherit properties across the entire file. Not only that we can replicate a piece of configuration but we can inject an override to a specific key that is already defined in the anchor, thus making it very flexible. I agree that if you have some small or basic configuration file then there is no reason to use it, but if we assume that the file will grow in its content then it does worth the extra work in setting up anchors.

The way we work with anchors is by using the '&' sign and the '*' sign.
The format to define an anchor is by declaring a key followed by the anchor name preceded with the '&' sign and then the value.

    mykey: &myanchor myvalue

You can pay attention to the fact that the key and the anchor name don't have to match. When we want to use the anchor we need to assign the anchor name preceded with the '*' sign as a value to another key.

    anotherkey: *myanchor

Example - YAML Anchors 1

    name: &actor Neo
    movie_charachter: *actor # movie_charachter will hold the value Neo

As we can see in this simple example this is not really why or when we should use anchors. We are not looking for the simple implementations of anchors. I usually use them when i want to configure an object that has multiple properties, or key:value pairs that should not change across the file everywhere we need to duplicate the instance. The way that we ue anchors with complex key:value pairs is by using the double chevron-left '<<' signs follow by the anchor.

Example - YAML Anchors 2

    # global car object that we want to use across
    car: &base_car
      year: 2021
      make: Toyota
      model: Corolla
      color: Grey

    # reuse the car object without changing anything
    corolla:
      <<: *base_car

    # reuse the car object and override one of the properties
    runx:
      <<: *base_car
      model: runx

    # reuse the car object and override several of the properties
    prius:
      <<: *base_car
      model: prius
      color: Red

    # reuse the car object, override property and add additional that doesn't exist in the original anchor
    camry:
      <<: *base_car
      model: camry
      seats: 5

As we can see in this example, we declared an anchor, used it in the YAML file in different places and also customized it. Pay attention that the customization can apply to nested properties as well. Just read the part where we talked about nesting and implement it. No need to rewrite it again. Each one of the map objects will look the same as the anchor with the adjustments that we added.

Let's talk about where you will encounter most likely YAML file configurations in your everyday work. As developers and/or devops engineers we encounter YAML configurations all the time when we need to use Docker, specifically Docker Compose and also in our CI/CD pipelines. These 2 examples are the most common ones.

Example - YAML Docker Compose

in this example we will take a look at a simple docker compose config file for a local development environment

    # docker compose example

    # simple key:value pair
    version: "3"

    # complex Map object with nested map objects, each nested object represents a service in docker compose
    services:
      # service Map object
      redis_sentinel:
        image: redis:alpine
        volumes:
          - sentinel_data:/data

      # service Map object
      redis_worker:
        image: redis:alpine
        environment:
          - ALLOW_EMPTY_PASSWORD=yes
        volumes:
          - worker_data:/data

    # another complex Map object with nested map objects. volumes used for persistent storage in docker. docker containers are ephemeral which means that they are not designed to run forever, especially in local dev environment and that is why we need to create a volume and bind the local host's file system into the docker container's file system
    volumes:
      # service Map object
      sentinel_data:
        driver: local
      # service Map object
      worker_data:
        driver: local

As we can see here in the example, we have a common use case for YAML configurations file that is written in a repetitive fashion. I am sure you are confident enough to try and rewrite this YAML configurations file all by yourselves. Let's give it a try

    # docker compose example
    redis_service: &base_redis
      image: redis
      volume: null

    volumes:
      sentinel_data:
        driver: local
      worker_data:
        driver: local

    version: "3"

    services:
      sentinal:
        <<: *base_redis
        volume:
          - sentinal_data:/data

      worker:
        <<: *base_redis
        volume:
          - worker_data:/data
        environment:
          - ALLOW_EMPTY_PASSWORD=yes

You can see that i didn't touch the global configurations for volumes because it is very specific to docker and usually you do not want to make it dynamic. However, we can see that the rewrite didn't make too much of a difference, but we need to remember first that this is a very basic docker compose configurations file and if you add another 'service' under 'services' you will see the impact. Also the base_redis anchor is very light. Imagine that we had 20 properties with nested properties of their own and how would our file would look like.

Example - YAML - CI

    # Travis ci example

    language: node_js
    node_js:
      - node
    env:
      global:
        - PATH=$HOME/.local/bin:$PATH
    before_install:
      - pyenv global 3.7.1
      - pip install -U pip
      - pip install awscli
    script:
      - yarn build
      - echo "Commit sha - $TRAVIS_COMMIT"
      - mkdir -p dist/@myapp/$TRAVIS_COMMIT
      - mv dist/*.* dist/@myapp/$TRAVIS_COMMIT/
    deploy:
      provider: s3
      access_key_id: "$AWS_ACCESS_KEY_ID"
      secret_access_key: "$AWS_SECRET_ACCESS_KEY"
      bucket: "my_project_bucket"
      region: "us-west-2"
      cache-control: "max-age=31536000"
      acl: "public_read"
      local_dir: dist
      skip_cleanup: true
      on:
        branch: master
    after_deploy:
      - chmod +x after_deploy_script.sh
      - "./after_deploy_script.sh"

As you can see the configurations in this file are self explanatory and you can see that the pattern is consistent. We use key:value pairs and most of the configurations are complex map objects

And that's all folks! This is all you need to know about YAML. From now on you can and should be more confident when dealing with YAML file configurations and maybe you will have the chance to improve existing files.

A cool tool that i sometime use to validate the YAML files i work on is this, it check spelling and configurations so you can make sure that you didn't miss anything and in the case you did then an error will be thrown 😄

By the way, if you asked yourself what YAML stands for, it is Yet Another Markup Language. 😄😄😄

Stay tuned for next
Like, subscribe, comment and whatever ...
Goodbye

Nginx Serving Compressed Multiple Static Files w/ Unique Paths using Docker for Improved Performance ⏩ ⏩ ⏩

alxizr — Sun, 06 Jun 2021 20:00:53 +0000

In this setup we will configure the nginx server to serve multiple static files on different paths with gzip, media, js and css enabled. This scenario may fit well with Microfrontends (MFE) architechture where each individual route may serve a complete isolated application's production build version and by implementing the concept we will discuss here in this article you will also gain performance improvement.
This is a simple starter guide on how to get you up and running and in case you should want to implement this kind of setup, you will have to add in your additional custom configurations that suits your needs best.

📓 📓 📓 Prerequisites 📓 📓 📓

🏃🏃🏃 Getting started guide 🏃🏃🏃

we will be working with vscode and the integrated bash terminal on windows 10

open vscode
open bash integrated termianl

  ctrl + shift + `

or you can choose the 'view' tab in the menu and then click on 'Terminal'

create a project folder and cd into it

  $ mkdir <project_name> && cd <project_name>

create a Dockerfile file

  $ touch Dockerfile

you will excuse me if i skip the .dockerignore file as we are not focusing on Docker here but rather on what we want to achieve with nginx.

create a nginx.conf file

  $ touch nginx.conf

We now will take a look on the overview of the configuration files and for a more detailed information we will have towards the end of the article or you can read the repo as well.

Dockerfile file content description

# base image to build on: we use a lightweight image
FROM nginx:alpine

# You may want to clear defaults on producion server - set working directory to nginx asset directory and then remove default nginx static assets; uncomment these next 2 lines
# WORKDIR /usr/share/nginx/html
# RUN rm -rf ./*

# repeat this line for each application you fancy serving on different route
COPY ./<app_build_dir> ./usr/share/nginx/html/<app_build_dir>

# override the default nginx configuration file with our own
COPY ./nginx.conf ./etc/nginx/conf.d/default.conf

# expose the port your server should support (80/443..)
EXPOSE <port>

Pay attention: If you come from any Nginx background with Docker, do not add this command in the Dockerfile file

ENTRYPOINT ["nginx", "-g", "daemon off;"]

The reason we don't add it is because we override the default configurations and we want to control the custom behaviour of our nginx server.

nginx.conf file content description

server {
    listen 80;                  # default port
    root /usr/share/nginx/html; # absolute paths only
    gzip on;                    # enable compression


    # nginx default config
    location / {
        index index.html index.htm;
    }


    # repeat this block as many times as you need
    # my application page
    location /<my_app> {
        index index.html index.htm;
        try_files $uri $uri/ /index.html;
    }

    # set up gzip config
    # set up assets (javascript, css, media files)
}

💪💪💪 Adding in our applications 💪💪💪

Before we move forward we would actually need to get our project(s) available for us to use and deploy on to the nginx server. We can use basically anything we want for serving html, css and javascript for example React or Angular, but i am not going to go and implement this extra step as this is out of focus, but rather mimic a production build of a frontend framework by just creating 3 folders with an index.html file.

Again in the terminal pointing to the root of the project run these 2 commands:

    $ mkdir {Home,Profile,Dashboard}

    $ touch {Home,Profile,Dashboard}/index.html

Now open all the index.html files, add some html skeleton for home, profile and dashboard applications.

Here is an example. Just copy and paste it and don't forget to update the heading for each directory.

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Profile</title>
  </head>

  <body>
    <nav>
      <li>
        <a href="//localhost:8080/home/"> Home </a>
      </li>
      <li>
        <a href="//localhost:8080/profile/"> Profile </a>
      </li>
      <li>
        <a href="//localhost:8080/dashboard/"> Dashboard </a>
      </li>
    </nav>

    <h1>_Change_This_ page</h1>
  </body>
</html>

I want to take a moment and explain what just happened in this last step. We are acting as we have 3 different frontend projects that we want to serve on 3 different routes. In this case we are just implementing the bare minimum for the static html because this article is not about the static assets but rather on how to use them with nginx.

⏰ ⏰ ⏰ Mini Recap ⏰ ⏰ ⏰

As for this stage we should have one project folder which contains 2 configuration files (Dockerfile and nginx.conf) and 3 folders (Home, Profile and Dashboard) with index.html files in them that represents a full blown production build application.

project
|
|_  Dockerfile
|_  nginx.conf
|
|___Home
|   |_  index.html
|
|___Profile
|   |_  index.html
|
|___Dashboard
    |_  index.html

The only part that is missing here, before we can apply our settings are the exact contents of the Dockerfile and the nginx.conf files. Let's quickly see what they are.

🔨 🔨 🔨 Filling the missing pieces 🔨 🔨 🔨

Dockerfile


    FROM nginx:alpine

    COPY ./Home ./usr/share/nginx/html/home
    COPY ./Profile ./usr/share/nginx/html/profile
    COPY ./Dashboard ./usr/share/nginx/html/dashboard

    COPY ./nginx.conf ./etc/nginx/conf.d/default.conf

    EXPOSE 80

nginx.conf


    server {
        listen          80;
        root            /usr/share/nginx/html;

        include         /etc/nginx/mime.types;
        default_type    application/octet-stream;

        gzip on;
        gzip_disable "msie6";

        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_buffers 16 8k;
        gzip_http_version 1.1;
        gzip_min_length 256;

        gzip_types  font/eot
                    font/otf
                    font/ttf
                    image/svg+xml
                    text/css
                    text/javascript
                    text/plain
                    text/xml
                    application/atom+xml
                    application/geo+json
                    application/javascript
                    application/x-javascript
                    application/json
                    application/ld+json
                    application/manifest+json
                    application/rdf+xml
                    application/rss+xml
                    application/xhtml+xml
                    application/xml;


        # home page
        location /home {
            index index.html index.htm;
            try_files $uri $uri/ /index.html =404;
        }

        # dashboard page
        location /dashboard {
            index index.html index.htm;
            try_files $uri $uri/ /index.html =404;
        }
        # profile page
        location /profile {
            index index.html index.htm;
            try_files $uri $uri/ /index.html =404;
        }

        # root - nginx default
        location / {
            index index.html index.htm;
        }


        # Javascript and CSS files
        location ~* \.(?:css|js)$ {
            try_files $uri =404;
            expires 1y;
            access_log off;
            add_header Cache-Control "public";
        }


        # Media: images, icons, video, audio, HTC
        location ~* \.(?:jpg|jpeg|gif|png|ico|cur|gz|svg|svgz|mp4|ogg|ogv|webm|htc)$ {
            expires 1M;
            access_log off;
            add_header Cache-Control "public";
        }


        # Any route containing a file extension (e.g. /devicesfile.js)
        location ~ ^.+\..+$ {
            try_files $uri =404;
        }
    }

🎹 🎹 🎹 Putting theory 2 practice 🎹 🎹 🎹

Now after we set up our Nginx server as for what and how it should behave, we will build our very own custom version of it and then run it to see that everything works as expected!

Let's open a terminal from inside the project root folder and run the next command. We would want to 'tag' our custom image so we can address it by name and not by its id, to do it simply by adding the -t flag.

    $ docker build -t <my_custom_tag_image> .

and for the docker less experienced ones, this is what you need to type and run, pay attention to spacing and the dot!

    $ docker build -t multi_nginx .

Next step is to run the image in a container and we will accomplish it by running the next command.

    $ docker run -p <out_port>:<in_port> <my_custom_tag_image>

Pay attention that we specify the our local port to a container's inner port which can be anything but defaults to 80 or 443, this is the exact port number we exposed inside the Dockerfile configuration file. (I used port 8080)

and again for the docker less experienced ones, this is what you need to type and run

    $ docker run -p 8080:80 multi_nginx

pay a little attention, if you do come from a docker background, to the fact we don't run the container in a detached mode with the -d flag because we want to see the nginx server logs every route it should serve.

Our final step is to open a browser and browse to the container root path on localhost:8080. In this demo we have 3 different paths that are isolated from one another home, profile and dashboard.

Thank you very much for staying all the way through this article and hope that you may have learned something. until the next time.

Stay tuned for next
Like, subscribe, comment and whatever ...
Goodbye