BYOB w/ Flask, Python & MongoEngine

Hello everyone,

Thank you for joining in for this article. Today we will take a little look at one of the most popular programming languages out there which is Python and a minimalistic, yet very powerful web framework called Flask. We will also make use of a NoSql document database called MongoDB and integrate it with an ODM (Object Document Mapper) called MongoEngine

Prerequisites

Before we begin, as always we want to know what is the minimum for us to be able to start, be efficient and productive.

Visual Studio Code
Git Bash
Postman
Pip
If you are by any chance missing one of these tools, make sure to download and install it before continuing.

Prologue

In my day to day job I work mainly with Node.Js in a microservices environment, but sometimes we have tasks or entire epics that are data driven and we need to handle massive computes on the data we work on, node.js is not the right tool for it because of its single thread non blocking io nature, thus we need to pick the right tool for the job. In a microservice environment you can have many and different services that handle single responsibility and they may be implemented with a programming language or a framework outside your main tech stack. Here, in this case, comes Python & Flask. In this article we will see how we can achieve a working api in no time.

Setup

We will open visual studio code and setup our workspace. We will make use of a tool called Pipenv it is a package manager very similar to NPM in the Node.Js ecosystem and it is the new way of doing things in the python ecosystem. We will start by creating our project directory api.

mkdir api
cd api

Now, the first thing we need to do right away and I will talk about later in more details, is creating a -.env_ file for our workspace environment variables and set a value.

echo PIPENV_VENV_IN_PROJECT=1 >> .env

This step also can be done only once globally and it depends on the operating system you have. You can look in the docs where to update this environment variable globally.

Our next step is where it all begins. We will create our encapsulated environment with pipenv and within install all of our project dependencies.

pip3 install pipenv

We use pip version 3 to install this package globally so it will be available to us for other python projects and this is the only time where we need the use of pip.

Now let us activate the encapsulated environment and start installing packages.

pipenv shell

This command is actually creating and activating our virtual environment, something that is well known and used in the python ecosystem. The reason we needed our .env file to be present beforehand is that the default behavior of pipenv is to create the .venv folder in a different location in the file system, depends if you are using Windows, Linux or Mac. This way we make sure that everything that is related to the project is present within the project workspace.

You can see that by running this command we got our virtual environment present and also a file named Pipfile created, this file contains all the settings for the project in term of dependencies, dev dependencies scripts and versions much like package.json in Node.js.

Now we will install all the dependencies at once so we would not have to keep constantly switch contexts. You can obviously just copy and paste it in your terminal if you don't want to type it all.

pipenv install flask flask-restful flask-marshmallow marshmallow mongoengine flask-mongoengine python-dateutil flask-cors 

Once installation is done, you will see that a new file was created named Pipfile.lock which describes all the dependencies and sub dependencies we installed and also states the exact versions of each dependency.

Now we create our entry file app.py inside of a src folder just to keep things clear. We then change directory into this folder and this is going to be our root for the project. Any additional folders and files we will create will be descendant of src.

mkdir src
cd src
touch app.py

Now we will create 2 folders that will represent our business logic for the project and in time when this project will scale, you will be able to follow the same pattern.

Our example will go around a post model crud api. The model will be constructed with nested schema, just a small addition from me to see how we handle nested documents in MongoDB.

# database related tree

mkdir database controllers
mkdir database/models

touch database/db.py
touch database/models/{metadata,post}.py

# controller related tree

mkdir controllers
mkdir controllers/posts
mkdir controllers/posts/dto

touch controllers/posts/dto/post.dto.py
touch controllers/routes.py
touch controllers/posts/{posts,post}Api.py

By now all we have is the basic skeleton for the project with no 1 line of code written. We will address it in the next step, I just want to make sure we all have the same folder tree.

- api #(project folder)
| - src
|    | - app.py
|    | - controllers
|         | - dto
|              | - post.dto.py
|         | - posts
|              | - postApi.py
|              | - postsApi.py
|         | - routes.py
|    | - database
|         | - db.py
|         | - models
|              | - post.py
|              | - metadata.py

Database Setup

In this step we will configure our database in the project. We will use MongoDB. There are different ways to consume this service. The most popular one is via the cloud offering called MongoDB Atlas, it is free and all you need to do is register for the service. I for one do not want to add a layer of latency while developing locally so I will choose Docker instead. This step is optional and you don't have to do it the same way as me, but I will provide the details on how to run local MongoDB docker container for development purposes. This is not a Docker tutorial and I do not want to change focus, so if you are not familiar with it, then use Atlas.

# docker-compose.yml file

version: "3"

services:
  db:
    restart: always
    container_name: mongodb
    image: mongo
    ports:
      - "27017:27017"
      - "27018:27018"
      - "27019:27019"
    volumes:
      - "./mongo_vol:/data/db"

# spin up a mongodb container by running this command in your terminal

docker-compose up -d

Once our database is up and running wether locally or in the cloud, we can carry on to configure it in our project. Let us open the db.py file inside the database folder and write some code.

We need to import into this file the utility to work with MongoDB and we installed a package called flask-mongoengine which will handle the database connection and operations.

from flask_mongoengine import MongoEngine

db = MongoEngine()

def init_db(app):

    # replace the host URI with your atlas URI if you are not working locally
    # this settings must be as close as posiible to the init app call!
    app.config["MONGODB_SETTINGS"] = { "host" : "mongodb://localhost:27017/products" }

    db.init_app(app)
    pass

Here we imported the package, initialized a database connection instance and provided the host uri to where our database is located alongside the name of our target database name. The last line of code says that the connection will be bound to the app once it is running.

Database Model Setup - Post

Our model for the example is a post. We want to describe how a post looks like in terms of property fields and divide some of them to a nested schema. Our post will consist of these properties: title, text, creation date, author's email, url. We will decide that the parent schema will hold the title and text fields and the rest will be nested to a child schema called metadata, like this:

    # Metadata schema

    import datetime
    from ..db import db

    class Metadata(db.EmbeddedDocument):
        url = db.URLField()
        email = EmailField()
        date = db.DateTimeField(default=datime.datime.utcnow)
        pass

Pay attention to the date field, where we initialize it with a default date & time only on creation. We will not be providing this value via the api. The model will create and set it.

    # Post schema

    from ..db import db
    from .metadata import Metadata

     class Post(db.Document):
         title = db.StringField()
         text = db.StringField()
         metadata = db.EmbeddedDocumentField(Metadata, default=Metadata)

         meta = {
             "collection" : "posts",
             "auto_create_index": True,
             "index_background": True,
             "inheritance": True
         }
         pass

Also notice that all of the fields are optional, except the metadata field. The Post model expects to receive a metadata object with its fields because we set the default flag. Your task is to change them to be required fields with the expression required=True inside the parentheses

As you can see, we have two models that each holds partial data about our post and together they are a whole. Now, using the Post class we will engage the database via our api. We do not need to do anything with the Metadata class. It is there for convenience purposes.

API Setup

Flask supports different ways of building web applications and APIs, starting from templates for static pages, through blueprints and views and going to rest. We will choose the restful approach and we already installed the appropriate package for it flask-restful.

The way to construct rest api with the restful package is by using classes that inherits from a base class called Resource. We will create 2 classes for our api. 1st class handles all generic requests and the 2nd class will handle all the specific request that we want to filter by post id for example. Obviously you can decide on the best approach for what ever your needs are.

# PostsApi.py file

from database.models.post import Post
from flask import request, Response
from flask_restful import Resource
import json

class PostsApi(Resource):

    # get all post from database
    def get(self):
        posts = Post.objects().to_json()
        return Response(
            posts,
            mimetype="application/json",
            status=200
        )


    # create new post
    def post(self):
        body = request.get_json()
        post = Post(**body)
        post.save()
        return Response(
            json.dumps({
                "message" : str(post.id),
                "status": "success"
            }),
            mimetype="application/json",
            status=201
        )

You can see here that the class has 2 methods named after the HTTP verbs they each are responsible to handle. The GET HTTP call to this endpoint /api/posts/ will return the list of posts in the database and a POST HTTP call to the same endpoint will hit the same API class to create a new document in our database. Pay attention as well to the way we are engaging the MongoDB database and it is with the Post class we created earlier. This is a posts api and we need the entity that is bound to it. Now lets see how to create a specific request api.

# PostApi.py file

from database.models.post import Post
from flask import request, Response
from flask_restful import Resource
import json

class PostsApi(Resource):

    # get one post by id from database
    def get(self, id):
        post = Post.objects.get_or_404(id=id).to_json()
        return Response(
            post,
            mimetype="application/json",
            status=201
        )


    ### delete one post
    def delete(self, id):
        post = Post.objects(id=id).delete()
        return Response(
            json.dumps({
                "id" : str(id),
                "success" : post
            }),
            mimetype="application/json", 
            status=200)


    # updates existing post by id
    def post(self, id):
        body = request.get_json()
        post = Post.objects(id=id).update_one(**body)
        return Response(
            json.dumps({
                "message" : post,
                "status": "success"
            }),
            mimetype="application/json",
            status=200
        )

As you can see that the biggest difference is with the argumenst the class methods get in their signature. The id parameter is a dynamic one and it comes from the URL. The Flask Resource class handles in fetching it and providing it to us inside the logic we write. Pay attention that this is not the way of doing things in production environment, we will get back to this later, after we are done with everything to test couple of things.

We are now left with setting up the routes for the api and we are done with this step.

    # routes.py file

    from flask_restful import Api
    from .postApi import postApi
    from .postsApi import postsApi

    def init_routes(app):
        api = Api(app)
        api.add_resource(postsApi, "/api/posts")
        api.add_resource(postApi, "/api/posts/<id>")

Pay attention that dynamic URL parameters in Flask are wrapped in diamond brackets, unlike in node.js where its prefixed with : the colon sign.

App Setup

So we worked backwards and we are finally here. The place where it all comes together like magic. We need to take all the different parts we created from scratch with our bare hands and make them stick. This magic will happen inside the app file.

from flask import Flask
from resources.routes import init_routes
from database.db import init_db
from flask_cors import CORS


# Init flask app with mongodb using mongoengine, flask-mongoengine 
app = Flask(__name__)
init_routes(app)
init_db(app)
CORS(app)


### run app watch mode
if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000, debug=True, use_reloader=True)
    pass

What we are doing here is importing the database util function init_db, the routes util function init_routes, utilities from Flask to bind everything and make it rain (🌧️ 🌧️ 🌧️) ❗ (😀 😀 😀) make it run❗❗❗ Pay attention that in production you want to turn off the debug flag.

Now we are going back to the terminal and we want to have this Flask application running. Well, we have a little stop beforehand. remember that we installed pipenv❓ We want to add a script to run the application. Let us open the Pipfile and add a scripts section at the bottom.

[scripts]
dev = "python src/app.py runserver"

Now in the terminal where we initialized our virtual environment run this command

pipenv run dev

Congratulations! This is your first Flask Rest API and it was not too bad at all. Now you can open postman and run some tests agains the API. You now know that this is the enterprise way of using Flask for APIs, but there were couple of things that we skipped when we wrote the different api classes and they are error handling and input validation. In python, much like in many different programming languages you will and must encounter errors to fix them beforehand you go to production. Python uses try...except blocks for this purpose and the cool thing is that we can stack multiple except blocks up to handle different errors in our code. For example you can have one method that can throw server error (500 & up) or a user input error (400 <= 499).

I will create one example and your homework is to add it to all api class methods. I will choose the DELETE HTTP method and implement a generic exception.

### delete one post
def delete(self, id):
    try:
        post = Post.objects(id=id).delete()
        return Response(
                json.dumps({
                    "id" : str(id),
                    "success" : post
                }),
                mimetype="application/json", 
                status=200)

    except Exception as err:
        return Response(
                json.dumps({
                    "message" : str(err)
                }), 
                mimetype="application/json", 
                status=500)

That is all folks! It was a nice ride today. Hope you enjoyed and learnt something. Make sure to complete your homework and obviously read the documentation for Python, Flask and MongoEngine.

One last thing, some food for tought make sure to write code that checks also that the Database is working... wether you run mongodb installation, docker container or Atlas.

Next time, hopefully soon, I want to talk little about dependency injection in Python application focusing on Flask.

Stay tuned for next
Like, subscribe, comment and whatever ...
Thank you & Goodbye