DEV Community: Herbz

Docker + Flask + Vue + Nginx + MongoDB- deployment and development in one package (3)

Herbz — Sun, 26 Apr 2020 19:40:36 +0000

Chapter 7: Initialize MongoDB

Now that we have a running backend and frontend, the next step is to get the database running so we can handle data better.
In our docker-compose.yml add following:

    mongodb:
        image: mongo:4.2.5-bionic
        container_name:  mongodb
        restart: always
        environment:
            - MONGO_INITDB_ROOT_USERNAME=root
            - MONGO_INITDB_ROOT_PASSWORD=example
        volumes:
            - './database:/data/db'
        ports:
            - "27017:27017"
            - "27018:27018"
            - "27019:27019"
        expose:
            - 27017
            - 27018
            - 27019

    mongo-express:
        image: mongo-express
        restart: always
        environment:
            # mongodb authentication
            - ME_CONFIG_MONGODB_ADMINUSERNAME=root
            - ME_CONFIG_MONGODB_ADMINPASSWORD=example
            # mongodb URI
            - ME_CONFIG_MONGODB_SERVER=mongodb
            - ME_CONFIG_MONGODB_PORT=27017
            # web login username and password
            - ME_CONFIG_BASICAUTH_USERNAME=mongo
            - ME_CONFIG_BASICAUTH_PASSWORD=express
        ports:
            - 8081:8081

This setup allows us to have mongodb and a nice web UI mongo-express running. For additional security, we have set password for our db, and we have also set a web login BASICAUTH for our mongo-express service.
Spin this up with docker-compose up --build
In the browser, we can access to our database with localhost:8081 via mongo-express.

Build a website monitor with docker + flask + twilio + vue + mongodb (part 1)

Herbz — Sun, 26 Apr 2020 19:40:14 +0000

I have been eager to learn docker for quite some time and finally got some quality (self-isolation) time to try it out. Turns out it is really as good as the internet says, and let's jump into it!

I have recently built a website but sometimes it may go offline due to a wrong deployment build or whatever reason. There are some online services that claims to notify you when your site is down, but I don't know how often they check your site etc.. So let's build our own website monitor!

Step 1: Python whatsapp messaging service by twilio

The basic idea is simple, we query our website, check the response, if the response is not as expected, we notify ourselves via sms or email (the traditional way) or via whatsapp (the better way).
Twilio offer free whatsapp messaging service, and who still uses SMS in 2020?? Sign up an account here: https://www.twilio.com/whatsapp and this video explained everything you need https://www.youtube.com/watch?v=98OewpG8-yw
Basically you need 1. twilio credentials, 2. link your whatsapp number to the twilio sender and 3. install the twilio module for python.
To do it more properly, let's also use virtual environment for python.

mkdir backend
pip install pipenv
pipenv shell
pipenv install twilio

You also need to set up few environment variables before it can work properly (and this is better for our docker app later than hardcode). Credentials can be found here: https://www.comm100.com/livechat/knowledgebase/where-do-i-find-the-twilio-account-sid-auth-token-and-phone-number-sid.html. ADMIN_PHONE_NUMBER is your phone number.

export TWILIO_ACCOUNT_SID='abcd'
export TWILIO_AUTH_TOKEN='1234'
export ADMIN_PHONE_NUMBER='+441234'

Alright, the whatsapp messaging service can be configured like this:

# backend/TwilioWhatsapp.py

import os
from twilio.rest import Client

# check document here if needed
# https://www.twilio.com/docs/libraries/reference/twilio-python/index.html
def send_whatsapp_message(to_whatsapp_number, message_body):
    client = Client() # without specifying credentials, it uses environment variables

    from_whatsapp_number = "whatsapp:+14155238886"

    to_whatsapp_number="whatsapp:" + to_whatsapp_number

    client.messages.create(body=message_body, from_=from_whatsapp_number, to=to_whatsapp_number)

if __name__ == "main":
    send_whatsapp_message(os.environ['ADMIN_PHONE_NUMBER'], "hello")

We can test it by python TwilioWhatsapp.py, you should get a whatsapp message if you have set up the whatsapp sandbox properly.

Step 2: check whether our site is live using `requests + beautifulsoup`

install beautifulsoup4 and lxml library
pipenv install beautifulsoup4 lmxl
Our python code would be

# backend/URLChecker.py

import requests
from bs4 import BeautifulSoup

chrome_headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.122 Safari/537.36'}

def find_text_from_URL(URL, text_to_find):
    response = requests.get(URL, headers=chrome_headers)
    soup = BeautifulSoup(response.content, "lxml")

    if text_to_find in soup.text:
        return True
    else:
        return False


def extract_text_from_URL(URL):
    response = requests.get(URL, headers=chrome_headers)
    soup = BeautifulSoup(response.content, "lxml")

    return soup.text


if __name__ == '__main__':
    print(extract_text_from_URL("https://google.com/"))
    print(find_text_from_URL("https://google.com/", 'Google'))

Basically we can use extract_text_from_URL() to see what your site normally returns and then use find_text_from_URL() to find whether the text is available. As far as I find, the vue.js rendered website does not return the proper response from requests, but it is anyway enough to use those texts.

Step 3: create a service that checks our website conditions every so often.

Assuming we will have multiple website to monitor in the future, we can create a simple site_to_monitor.json to store the information we need. When we actually want to allow other users to add new sites, we probably should migrate to a database.

# backend/site_to_monitor.json
[{
    "URL": "https://google.com/",
    "text_to_find": "Google",
    "phone_number": "+441234567"
}]

Now our monitor_service app

# backend/MonitorService.py

from TwilioWhatsapp import send_whatsapp_message
from URLChecker import  find_text_from_URL, extract_text_from_URL
import json
import time
import os

def monitor_service():
    while True:
        # every one hour check the service itself is working.
        send_whatsapp_message(os.environ['ADMIN_PHONE_NUMBER'], "From Downotifier: the monitoring service is fully functioning")

        for i in range(60):
            with open("site_to_monitor.json") as site_info:
                site_info = json.load(site_info)

            for site in site_info:
                if not find_text_from_URL(site['URL'], site['text_to_find']):
                    send_whatsapp_message(site['phone_number'], "Your site {} is down, please check it".format(site['URL']))
                    print(site['URL'] + " is down.")

            time.sleep(60)

if __name__ == "__main__":
    monitor_service()

Small wrap up: Now the monitoring service is ready, we shall prepare our backend API to allow user adding new website to monitor, see which text is currently returned by our service and also remove the old records if it is no longer necessary.

Part 2 of this article is here:

Docker + Flask + Vue + Nginx - deployment and development in one package (2)

Herbz — Sat, 11 Apr 2020 17:34:27 +0000

Last time I set up a flask environment with Nginx and it allows hot reload etc. So practically we can test our code in development and directly deploy it without any changes. (other than the Nginx file for a little bit).

Now we want to incorporate a front end, and I prefer Vue.js for its simplicity. As it turns out, we don't have to install anything locally on our host machine. Everything can be done in the container.

The idea is to run vue ui on our container, which will allow us to serve and build our web app later. And all we need to do is to ports our 8000 (vue ui) and 8080 (app).

Chapter 4: Vue.js frontend

Create a frontend folder
Create the Dockerfile

FROM node:lts-alpine

# install project dependencies
RUN npm install -g @vue/cli

# copy project files and folders to the current working directory (i.e. 'app' folder)
ADD . /frontend


# vue ui --headless --port 8000 --host 0.0.0.0
CMD [ "vue", "ui", "--headless" , "--port", "8000",  "--host", "0.0.0.0" ]

NOTE: I spent some time to figure it out that the vue ui has to be hosted on 0.0.0.0:8000, the localhost of container doesn't work. You can read more here: https://pythonspeed.com/articles/docker-connection-refused/

Then modify the docker-compose.yml by adding the following:

    frontend:
        build: ./frontend
        container_name:  frontend
        restart: always
        #  mount the volumes so we can change code and hot reload
        volumes:
            - './frontend:/frontend'

        #  port allows the host machine to link to container
        #  8000 for vue ui, 5000 for our vue app
        ports:
            - "8000:8000"
            - "5000:5000"

Now we can create a project using the vue ui in our frontend folder. A problem I noticed is that vue seems to have issue overwriting our frontend folder, so instead it is making a frontend/frontend nested folder. No big harm anyway.
Creating the project just using the default setting. Then in vue ui go to Tasks --> serve, change the parameters to be host: 0.0.0.0, port: 5000. And then serve.
Now go to localhost:5000, and your vue app should be up and running. Changing the file on your host machine will reflect the changes in real time, so we are good to develop.

PS: if you are having problem of modifying the frontend folder, use `sudo chmod -R 777 frontend/`

Chapter 5: Review the other code and fix

Now that we have both our frontend and backend connected, we need to make some modification.
In our nginx.conf:

server {
    listen 80;

    # https://www.digitalocean.com/community/tutorials/how-to-set-up-a-node-js-application-for-production-on-ubuntu-16-04
    #  listen to our npm server.
    location / {
        proxy_pass http://frontend:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }

    location /api {
        include uwsgi_params;
        uwsgi_pass backend:8080;
    }
}

Now that we listen to our uWSGI server at localhost:80/api and listen to our npm server at localhost:80/. We need to modify the route in our python app.py, simply by changing /helloworld/ to /api/helloworld/.

app.py

from flask import Flask
app = Flask(__name__)

@app.route('/api/hello')
def hello_world():
    return 'Still hot reloading!'

if __name__ == "__main__":
    app.run(host='0.0.0.0')

Chapter 6: communication between backend and frontend

Now, our frontend can communicate with our backend through localhost/api or backend:8080, but the backend connection is only made available between containers. When we access our frontend app via browser, backend will not be accessible!! Instead, we have to use the first option.
Luckily this doesn't mean we need to do anything different between development and deployment. All we need to do is to use the relative path such as /api/hello to access to our backend.
To test our connection, we can do a simple axios call from frontend to backend.

frontend/components/HelloWorld.vue

<template>
  <div class="hello">
    <h1>{{ msg }}</h1>
    <button v-on:click="test_axios">Test access to our backend</button>
  </div>
</template>


<script>
import axios from 'axios';

export default {
  name: 'HelloWorld',
  data() {
    return {

    }
  },
  props: {
    msg: String
  },
  mounted() {
    this.test_axios()
  },
methods: {
    test_axios: function () {
      const baseURI = '/api/hello'
      // const baseURI = 'http://localhost:80/api/hello'
      axios.get(baseURI)
      .then((result) => {
        console.log(result)
      }).catch(function (error) {
           console.log(error);
        });
    }
  },
}
</script>

Chapter 7: a simple user case for get, post requests.

So above we are still using a simple hello_world and it may not be a good enough starting point for anyone who wants a skeleton project to work with.
Let's modify our frontend a bit to make it more interesting.

frontend/HelloWorld.vue

<template>
  <div class="hello">
    <button v-on:click="test_get">Fetch info from our backend</button>
    <br />
    <input v-model="user_name" placeholder="user_name" />
    <br />
    <button v-on:click="test_post">Send your name to backend</button>
    <div>{{backend_response}}</div>
  </div>
</template>

<script>
import axios from "axios";

export default {
  name: "HelloWorld",
  data() {
    return {
      user_name: "",
      backend_response: ""
    };
  },
  props: {
    msg: String
  },
  mounted() {},
  methods: {
    test_get: function() {
      var baseURI = "/api/test_get";
      var vm = this;
      // const baseURI = 'http://localhost:80/api/hello'
      axios
        .get(baseURI)
        .then(response => {
          console.log(response);
          vm.backend_response = response.data.message;
        })
        .catch(function(error) {
          console.log(error);
        });
    },
    test_post: function() {
      var baseURI = "/api/test_post";
      var vm = this;
      // const baseURI = 'http://localhost:80/api/hello'
      axios
        .post(baseURI, {
          user_name: vm.user_name
        })
        .then(response => {
          console.log(response.data);
          vm.backend_response = response.data;
        })
        .catch(function(error) {
          console.log(error);
        });
    }
  }
};
</script>

Note here I use `var vm = this` for some arrow functions, this is a nice trick that will save you from being insane.. https://gist.github.com/JacobBennett/7b32b4914311c0ac0f28a1fdc411b9a7

And our python backend is also modified to accept get and post

backend/app.py

from flask import Flask, jsonify, request
from flask_cors import CORS, cross_origin
import json

app = Flask(__name__)
CORS(app)


@app.route('/api/test_get', methods=['GET'])
@cross_origin()
def test_get():
    return jsonify({'message': 'Hello, I am your backend'})

@app.route('/api/test_post', methods=['POST'])
@cross_origin()
def test_post():
    if request.method == "POST":
        response = request.get_json()
        result = "Hello " + response['user_name'] 
        return  result

if __name__ == "__main__":
    app.run(host='0.0.0.0')

Now you are all good!
Next chapter we will be connecting to a mongodb and I think we are done!

Docker + Flask + Vue + Nginx - deployment and development in one package (1)

Herbz — Fri, 10 Apr 2020 20:17:21 +0000

Building a minimalist Flask + uWSGI + Vue.js + Nginx app

with docker deployment

keep it to a good practice as possible.

Chapter 1. Backend (Flask + uWSGI)

First, establish a virtual environment use pipenv.

mkdir backend

pipenv shell`

pipenv install flask, uWSGI

Second, create a bare minimal flask app named as app.py

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
    return 'Hello, World!'

if __name__ == "__main__":
    app.run(host='0.0.0.0')

Third, create a wsgi.py

from app import app

if __name__ == "__main__":
    app.run()

Lastly, create a app.ini file for uWSGI configs

[uwsgi]
# the module itself, by referring to the wsgi.py file minus the extension, and the callable within the file, app:
module = wsgi:app

# Enable hot reload!
py-autoreload = 1

# Nginx to handle actual client connections, which will then pass requests to uWSGI. 
socket = :8080


master = true
processes = 4
threads = 2

#  giving the Nginx group ownership of the uWSGI process later on, 
#  so we need to make sure the group owner of the socket can read information from it and write to it.
chmod-socket = 660

# clean up the socket when the process stops by adding the vacuum option:
vacuum = true
die-on-term = true

After all of this, our python backend is ready, create a requirements.txt by

pipenv lock -r > requirements.txt

Now we just need to create a docker file, which will copy files and then install necessary libraries (as outlined in requirements.txt) and get our uWSGI server up and running.

# use python container image
From python:3.6-stretch

# set the working directory of the image filesystem 
WORKDIR /backend

# copy current directory to the working directory 
ADD . /backend

# Install the python dependencies
RUN pip install -r requirements.txt

# start the uWSGI
CMD ["uwsgi", "app.ini"]

In our backend folder, we should have these files now:

app.ini
app.py
Dockerfile
Pipfile
requirements.txt
wsgi.py

Chapter 3. Deployment (nginx)

### All we need to do is to re-direct request from outside to our uWSGI server in container. Luckily we can directly access to them using the image name.

for more details: http://nginx.org/en/docs/beginners_guide.html

The nginx listen to 80 port and direct calls to different locations such as "/diary" to different internal ports. (flask: 8080 in this case)

server {

    listen 80;

    location / {
        include uwsgi_params;
        uwsgi_pass flask:8080;
    }
}

Then, the Dockerfile

FROM nginx

# remove the default conf
RUN rm /etc/nginx/conf.d/default.conf

#  copy the nginx.conf in our filesystem into the image filesystem
COPY nginx.conf /etc/nginx/conf.d

Chapter 4: docker-compose to connect all containers

All the containers are in a virtual lan environment.

go to the root folder of the project, now you should have following folders

backend/
frontend/
database/
nginx/

Create a new file called docker-compose.yml

# check your version: https://docs.docker.com/compose/compose-file/
version: "3.7"


services:
# first container
    backend:
        # look for dockerfile and build the container
        build: ./backend
        container_name:  backend
        restart: always
        # set enviornemt variables
        environment: 
            - APP_NAME=FlaskApp
        #  mount the volumes so we can change code and hot reload
        volumes:
            - './backend:/backend'
        #  expose port on the container, only to other containers
        expose:
            - 8080

    nginx:
        build: ./nginx
        container_name:  nginx
        restart: always
        # port allows the host machine to link to container
        # pass 80 from host machine to 80 of the image filesystem
        ports:
            - "80:80"

NOTE: we added a volumes here that allows future hot-reload to happen. (together with the uWSGI's py-autoreload)
NOTE: The difference about ports and expose:
expose allow communication between multiple containers
ports map one of the container port to the host machine port (the real internet)