DEV Community: Valentine Sean Chanengeta

Flask Neon Kit: The extension which automatically generates CRUD endpoints

Valentine Sean Chanengeta — Mon, 02 Sep 2024 06:30:43 +0000

This is a submission for the Neon Open Source Starter Kit Challenge : Ultimate Starter Kit

My Kit

The flask-neon-kit is a Flask extension which generates CRUD endpoints out of the box from defined models of the Neon Postgres database. It empowers developers with resources' CRUD endpoints by just defining database models and instantiating the kit. This solution is built upon the foundation of the Flask-SQLAlchemy. This initiative was driven by the tedious nature of manually writing CRUD logic for every Flask application entity. It was also motivated by flask-mongo-crud, the package I am working on that automatically generates CRUD endpoints for MongoDB models.

Features of the solution are:

Automatically generates CRUD endpoints from defined model.
Allows to customize app base URL as well as each model’s url prefix.
Allows to paginate when getting many entities from database table.

Prerequisites for using the kit:

Familiarity with implementing Flask Applications.
Familiarity with integrating Flask Applications with Postgres database using Flask-SQLAlchemy.

Installation

pip install flask-neon-kit

Folder Structure of Basic Application

project_root
|    __init__.py
|    config.py
|    app.py
|
|____models
     |    professor_subject.py

Configuration

Empty init.py file required in project root directory.
ROOT_URL can be configured in application's configurations and it is optional.
If configured, it is used by all generated endpoints.
Code snippet in Basic Application section shows how to configure ROOT_URL.

Database Configuration (config.py):

class DbConfig(object):
    SQLALCHEMY_DATABASE_URI = "<NEON-POSTGRES-CONNECTION-STRING>"

Models:

Models directory is required in the project root directory:
If custom name Models directory is not defined, as:
```
app.config[MODELS_DIRECTORY] = "<CUSTOM_NAME>"
```
- then, default “models” directory will be used.
- This is where models files are defined.
- Inside these files declare models classes and their configurations such as:
  - model_url_prefix [OPTIONAL]
  - and those supported by Flask SQLAlchemy
- model_url_prefix allows to have unique URLs for each and every model.
- If this configuration is not defined, default configuration will be used.

Model Class Code Snippet (professor_subject.py):

class ProfessorSubject:
    model_url_prefix = "/professor-subject-test"

    id = db.Column(db.Integer, primary_key=True, autoincrement=True)
    professor_first_name = db.Column(db.String())
    professor_last_name = db.Column(db.String())
    subject_name = db.Column(db.String())

    # These are entity fields
    def __init__(self, professor_first_name, professor_last_name, subject_name):
        self.professor_first_name = professor_first_name
        self.professor_last_name = professor_last_name
        self.subject_name = subject_name

Basic Application

Code Snippet (app.py):

from flask import Flask, request
from flask_sqlalchemy import SQLAlchemy
from .config import DbConfig
from flask_neon_kit import FlaskNeonKit

app = Flask(__name__)
# If models directory is not defined, default "models" directory will be used
app.config["MODELS_DIRECTORY"] = "<CUSTOM_NAME>"
# If root URL is not defined, generated endpoints will not have a root URL
app.config["ROOT_URL"] = "/flask-neon-kit/v1"

app.config.from_object(DbConfig)

# Database related part
db = SQLAlchemy()
db.init_app(app)

flask_neon_kit = FlaskNeonKit()
flask_neon.init_app(app, request, db)

if __name__ == "__main__":
    app.run(debug=True)

Generated Endpoints Examples and HTTP Methods:

HTTP Methods supported are POST, GET, PUT and DELETE.
The following generated endpoints will be using model snippet defined earlier in Configuration as well Basic Application sections.
These endpoints are after application base URL: <IP_ADDRESS>:<PORT_NUMBER>
Basic complete URL looks like:

<IP_ADDRESS>:<PORT_NUMBER>/<ROOT_URL>/<MODEL_URL_PREFIX>/<RESOURCE_NAME>

or

<IP_ADDRESS>:<PORT_NUMBER>/<ROOT_URL>/<MODEL_URL_PREFIX>/<RESOURCE_NAME>/<RESOURCE_IDENTIFIER>
RESOURCE_NAME is automatically generated, and a developer can not customize it.
RESOURCE_IDENTIFIER should be the entity ID.
Given the generated URL:
localhost:5000/flask-neon-kit/v1/professor-subject-test/professor-subject/2:
- "/professor-subject" is the RESOURCE_NAME.
- "/2" is the RESOURCE_IDENTIFIER.
- In case ROOT_URL is not specified:
  - In the above URL, ROOT_URL is /flask-neon-kit/v1.
  - The URL without ROOT_URL is /professor-subject-test/professor-subject
- In case MODEL_URL_PREFIX is not specified:
  - In the above URL, MODEL_URL_PREFIX is /professor-subject-test
  - The URL without MODEL_URL_PREFIX is /flask-neon-kit/v1/professor-subject
- In case both ROOT_URL and MODEL_URL_PREFIX are not specified:
  - The URL without both is /professor-subject
  - It contains only the RESOURCE_NAME.

POST

Saves new entity into the database
Only saves one entity at a time.
Supports only JSON data.

JSON Request Payload Example:

{
    "professor_first_name": "Foo",
    "professor_last_name": "Bar",
    "subject_name": "Comp Science"
}

JSON Response Payload Example:

{
    "message": "professor-subject created successfully"
}

GET

Retrieves entities from the database.
Can retrieve only one entity or a list of entities.
To retrieve only one entity, include RESOURCE_IDENTIFIER in the URL.
This RESOURCE_IDENTIFIER will be used to identify that one entity.
If RESOURCE_IDENTIFIER is not provided, list of entities will be retrieved.
Developer can opt for pagination when retrieving many entities.
To paginate, provide the following query parameters:
- pagination = true. If "false", all entities will be retrieved from the database table.
- limit = "number of entities" per page. Should be integer greater than 0. If not provided and pagination is true, default value of 5 will be used.
- page = "page number". Should be integer greater than 0. If not provided and pagination is true, default value of 1 will be used.

JSON Response Payload Example for Paginated Entities:

{
    {
        "count": 2,
        "current_page": 1,
        "data": [
            {
                "id": 2,
                "professor_first_name": "John",
                "professor_last_name": "Doe",
                "subject_name": "Math"
            },
            {
                "id": 3,
                "professor_first_name": "Foo",
                "professor_last_name": "Bar",
                "subject_name": "Comp Science"
            }
        ],
        "has_next": false,
        "has_pagination": true,
        "has_prev": false,
        "limit": 5,
        "message": "professor-subject retrieved successfully",
        "total": 2
      }
}

PUT

Updates one entity at a time.
That entity is identified by RESOURCE_IDENTIFIER in the URL.
If there is no such entity in the database, " not found" response will be returned.

DELETE

Deletes one entity at a time.
That entity is identified by RESOURCE_IDENTIFIER in the URL.
If there is no such entity in the database, the endpoint will return a payload:
```
{
    "message": "<RESOURCE_NAME> not found"
}
```

Link to Kit

GitHub:

ValentineSean / flask-neon-kit

Flask Neon Kit Docs

Features

Automatically generates CRUD endpoints from defined model.
Allows to customize app base URL as well as each model's url prefix.
Allows to paginate when getting many entities from database table.

Installation

pip install flask-neon-kit

Configuration

Empty init.py file required in project root directory.
ROOT_URL can be configured in application's configurations and it is optional.
If configured, it is used by all generated endpoints.
Code snippet in Basic Application section shows how to configure ROOT_URL.
Models
- Models directory is required in the project root directory:
- If custom name Models directory is not defined, as
```
app.config[MODELS_DIRECTORY] = "<CUSTOM_NAME>"
```
  - then, default “models” directory will be used.
  - This is where models files are defined.
  - Inside these files declare models classes and their configurations such as
    - model_url_prefix [OPTIONAL]
    - and those supported by Flask SQLAlchemy
  - model_url_prefix allows to have unique URLs for each and…

View on GitHub

The kit was also uploaded on Python Package Index (PyPI)

Your Journey

Main objective of this solution is to minimize the tedious nature of manually writing common CRUD logic for every Flask application entity. There is also a plan to integrate some cool Neon features like data branches management in the future. This will enable developers to switch among their data branches to work with different versions of their data whilst in their application development environment. This should have been integrated in this version but seems Neon API currently has some limitations to support such features. I learned a lot during the course, and the concept of data branches is one of the features I am looking forward into exploring in the future.

Flask-SocketIO Load Balancing

Valentine Sean Chanengeta — Thu, 28 Dec 2023 23:10:01 +0000

Introduction

Flask-SocketIO is a Python library which enables bi-directional communications between a Flask written server and SocketIO clients. Starting with version 2.0, it can support multiple server instances behind a load balancer to spread the client connections among them. This gives the application the ability to scale so that it can support very large numbers of concurrent clients.

This article will layout how to deploy a Flask-SocketIO application with Docker as well as Nginx for load balancing and Redis Message Queue. Nginx will forward distributed client requests to Flask server instances. Since each of these instances are connected to only a subset of the clients, Redis Message Queue is used to coordinate broadcasting. Our instances will be deployed as Docker containers.

The following are steps to deploy the application:

Socket.IO Server Event
Redis Configuration
Docker Configuration
Nginx Configuration
Deployment Demo
Application Demo

Assumptions

Before diving in, this article assumes readers to have basic understanding of Docker, Docker Compose, Redis Message Queue especially with Pub/Sub interface as well as Flask and Nginx.

SocketIO Server Event



@socketio.on("send-message")
def send_message(data):
    sender = data.get("username")
    if sender is not None:
        sender = sender.upper()
    text_message = data.get("text_message")

    message = {
        "sender": sender,
        "text_message": text_message,
    }

    socketio.emit("receive-message", message)

In the above code snippet, Flask-SocketIO server is listening to the event send-message. When a client emits send-message event, the server will receive it with data, execute and then emits receive-message event which will be received by all clients listening to this event.

Redis Configuration

This component uses Redis Pub/Sub which has publishers (senders) and subscribers (receivers). Messages are categorized into channels without knowledge of any receivers subscribed to that channel. Subscribers express interest in one or more channels and only receive messages that are of interest. These subscribers have no knowledge of those channels' publishers. Flask-SocketIO provides this Redis feature out of the box. The following snippet shows message_queue argument passed to SocketIO constructor to enable the server connect to the Redis message queue. The argument's value is a Redis server URL.



socketio.init_app(app, cors_allowed_origins=["http://localhost:6001"], message_queue="redis://flask-socketio-load-balancing-redis:6379")

In this documentation, it is recommended to apply monkey patching at the top of the main script, even above imports. Since this application is using gevent, we apply monkey patching as follows:



from gevent import monkey
monkey.patch_all()

Docker Configuration

Dockerfile



FROM python:3.10-slim-buster

WORKDIR /app

COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt

COPY . .
ENTRYPOINT gunicorn -k geventwebsocket.gunicorn.workers.GeventWebSocketWorker -w 1 app:app -b 0.0.0.0:5003

This Dockerfile code shows that the application will be running on Gunicorn server.

Docker Compose



version: '3'

services:
  flask-socketio-load-balancing-redis:
    image: redis:latest
    container_name: flask-socketio-load-balancing-redis
    restart: always
    ports:
      - "6379"

  api:
    build:
      context: .
    depends_on:
      - flask-socketio-load-balancing-redis
    ports:
      - "5003"

  flask-socketio-load-balancing-nginx:
    image: nginx:latest
    container_name: flask-socketio-load-balancing-nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro

    depends_on:
      - api
    ports:
      - "5003:5003"

Nginx Configuration



server {
    listen 5003;

    location / {
        proxy_pass http://backend_instances;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }
}

upstream backend_instances {
    server api:5003;
}

This configuration enables Nginx load balancer to forward client requests to our multiple flask server instances. The upstream group defines server addresses Nginx will forward requests to. In the above snippet, we defined only one server, api:5003 (Flask API), which is our docker compose api service name. Since Nginx and Flask API are connected to same docker compose network, Nginx can directly communicate with Flask API using its service name and container internal port (5003). Nginx will be able to forward client requests to different API instances using different load balancing algorithms. Docker compose enables to run multiple instances (docker containers) from one service using the flag --scale [SERVICE_NAME]=[NUMBER_OF_INSTANCES]. Here we will run our containers as follows:



 docker-compose up -d --force-recreate --build --scale api=3 --remove-orphans

Docker Containers

After running our containers as above, we will have 5 containers (Redis Server, 3 Flask API Instances and Nginx Server) as shown in the image below:

Application Demo

I presented the application at Flask Conn before, it is available on Youtube:

GitHub (Flask Server):

ValentineSean / flask-socketio-load-balancing

Conclusion

This article showed how to deploy multiple instances of Flask-SocketIO server using docker. These instances will be running as docker containers. Nginx will be responsible for forwarding client requests to these instances and Redis will be broadcasting server events among all instances.

Is a portfolio website necessary for a junior backend developer and how can one showcase their skills as a backender?

Valentine Sean Chanengeta — Fri, 20 May 2022 01:28:24 +0000

VSChool Web App with Flask, Vue, Mongo, Deepgram and Dropbox

Valentine Sean Chanengeta — Tue, 12 Apr 2022 00:04:04 +0000

—

Overview of My Submission

Due to Covid19 global pandemic many tasks are now done online including educational tasks such as content delivery. Many schools now have some of the content delivered through platforms like Google Classroom, Microsoft Teams, etc. with teachers/instructors deliver content to students in different formats such as audio, videos or texts.

One of the challenges I noticed is that some of teachers' pre-recorded content might have some parts which are not clear to their audience. This might cause students to contact their teachers for clarity or ignore those parts which may result as a distraction.

This challenge can be solved using Deep gram's powerful weapons which was VSChool's motivation. VSChool web app is a simple platform which allows users to upload an audio file, generate transcriptions which are used to create subtitles for the video that can be played by users.

System Flow

user can create a subject and topics
subject have many topics
user can also upload an audio (MP3) for each and every topic
the system will generate transcriptions and a subtitles file (WEBVTT), the complete process will be explained later
it will then upload both MP3 and VTT files to Dropbox
it will get MP3 and VTT shared links and save them in the database
web app will fetch topics with shared links and render respective media
MP3 is rendered as a video with its subtitles are obtained from VTT file
hence, users will watch a video with subtitles generated by the system

The process of generating subtitles

app sends a POST request to a RESTful API with an MP3 audio and topic_id in the payload
API will rename the audio file to a topic_id as a new file name
it will also create a new directory named after a topic_id and stored in the media_files directory
the audio file will be saved in this new created directory
the audio file will then sent to Deepgram for transcription with utterances set to True
JSON response from Deepgram is written in a WEBVTT file for video subtitles
VTT file is named after topic_id too
it is then saved in a directory created earlier (the one named after topic_id)
the directory is then compressed to a ZIP file, in same parent folder
-then the original directory is deleted to only left with a ZIP file
ZIP file is uploaded to Dropbox
Dropbox destination folder is configured to automatically extract new ZIP files to become folders
the system will sleep for 10 seconds to wait for Dropbox to extract ZIP files
it will then send two requests to create shared links for MP3 and VTT files in new folders in Dropbox
the system will save these to created shared links to the respective topic
these shared links will then be used by the App to play the video with subtitles

How to Set up the environment

API:

Add '.env' file with credentials as follows:
DEEPGRAM_API_KEY=""
DROPBOX_ACCESS_TOKEN = ""
DROPBOX_APP_KEY = ""
DATABASEUSER=""
DATABASEPASSWORD=""
CLUSTER=""
DATABASE=""
HOST=""
JWT_SECRET_KEY = ""
Please fill blank "" with respective values
Create Python virtual environment in project's folder using command 'python -m venv myenv'
Activate virtual environment using command 'ENV_NAME\scripts\activate'
Install required libraries in 'requirements.txt' file using command 'pip install -r requirements.txt'
Start the app using command 'flask run'

App:

Prerequisites @vue/cli 4.5.12 and npm
Run command 'npm install' to install all required libraries
Run command 'npm run serve' to start the app

Demo Video

https://youtu.be/_93XB16LWUI

Submission Category:

Accessibility Advocates

Link to Code on GitHub

Vue App Link: https://github.com/ValentineSean/v-school-app

-Flask API Link: https://github.com/ValentineSean/v-school-api

Product Catalog Web App with Flask, Vue and Mongo

Valentine Sean Chanengeta — Fri, 14 Jan 2022 00:23:36 +0000

Overview of My Submission

Product Catalog web app is a simple catalog where user named Supplier will create products and Customer will browse these products on the platform.

The product has few information such as product name, category of the product, supplier of the product, to mention only 3.

Customer can search based on product name, category name or first name of the supplier (this is done by Flask with Atlas Search). After receiving search results, they can also filter those search results (This is done by Vue in the browser).

ROLES SUMMARY: Supplier -> (creates product); Customer -> (search among products, rate the product)

I built the app using Flask, Vue and MongoDB. Vue is responsible for displaying data stored in MongoDB to users and Flask is serving as an interface between MongoDB and Vue. Products' images are stored in Cloudinary using its public API.

Submission Category:

E-Commerce Creation

Link to Code

Flask API link: https://github.com/ValentineSean/product-catalog-api
Vue App link: https://github.com/ValentineSean/product-catalog-app

Additional Resources / Info

App demo Youtube link: https://youtu.be/DPZ_p9JU5SU
App link: https://product-catalog-vsc.netlify.app/
suppliers credentials: ["go@productcatalog.net", "john@productcatalog.net"]
customers credentials: ["foo@productcatalog.net", "java@productcatalog.net"]
all passwords are "123456"

DEV Community: Valentine Sean Chanengeta

Flask Neon Kit: The extension which automatically generates CRUD endpoints

My Kit

Features of the solution are:

Prerequisites for using the kit:

Installation

Folder Structure of Basic Application

Configuration

Basic Application

Generated Endpoints Examples and HTTP Methods:

POST

GET

PUT

DELETE

Link to Kit

GitHub:

ValentineSean / flask-neon-kit

Flask Neon Kit Docs

Features

Installation

Configuration

Your Journey

Flask-SocketIO Load Balancing

Introduction

Assumptions

SocketIO Server Event

Redis Configuration

Docker Configuration

Dockerfile

Docker Compose

Nginx Configuration

Docker Containers

Application Demo

GitHub (Flask Server):

ValentineSean / flask-socketio-load-balancing

Conclusion

Is a portfolio website necessary for a junior backend developer and how can one showcase their skills as a backender?

VSChool Web App with Flask, Vue, Mongo, Deepgram and Dropbox

Overview of My Submission

System Flow

The process of generating subtitles

How to Set up the environment

Demo Video

Submission Category:

Accessibility Advocates

Link to Code on GitHub

Product Catalog Web App with Flask, Vue and Mongo

Overview of My Submission

Submission Category:

Link to Code

Additional Resources / Info

Database structure and indexes images: