DEV Community: Juan Benito

The importance of the interfaces

Juan Benito — Tue, 30 Jun 2020 19:46:22 +0000

The interfaces are an important tool in the Object Oriented wolrd, but there are many developers who don't like to use it, and don't see the advantages they offer.

In order to explain the importance of the interfaces, we will follow the integration process between a big smartphone company called Fuwei and its technology provider, Famsung.

Fuawei - The smartphone company

Fuawei is one of the biggest technology companies in the world, it produces smartphones and sells them around the globe.

A few years ago, the direction board decided to externalize the chips production to Famsung, in order to allow the company to increase the production of smartphone.

Famsung has a cool REST API which allows clients to order new components and retrieve the provider specification. Fuawei decided to use this Web Service to ask for the chips needed.

Fuawei had a small technology team, which was in charte to create the technological requirements of the products, it was the CORE TEAM. As well to ensure the integration with Famsung was successfully, Fuawei created another team called the Integration TEAM.

Unluckily, The Integration Team didn't know much about interfaces so the first integration attempt get the following result.

import requests

from fuawei.fuawei_chip import FuaweiChip

class Fuawei:

    def create_phone(self):
        famsung_chip_data = requests.get('http://famsung/web-service').json()
        famsung_specification_parsed = self.parse_famsung_specification(famsung_chip_data)
        return self.create_fuawei_chip(famsung_specification_parsed)

    def parse_fansung_specification(self, data):
        # Many lines to parse from fansum format to any more friendly format
        return {}

    def create_fuawei_chip(self, data) -> FuaweiChip:
        # Many lines to build from our friendly format to FuaweiChip
        return FuaweiChip()

This code is shared between the Core Team, and the Integration Team.

The core team and the integration team fought many times, because of the many conflicts they suffered between each other, also those issues caused many bugs.

Now, in the present, Famsung and Fuawei relation is uncertain, the politics from their respective countries are not in good terms, and they have forbidden to work together. Finally, the Fuawei CEO has decided to assume the chip production in their own factories.

In order to replace Famsung provider, Fuawei is creating a Components Team. They have decided to use gRPC technology as the communication method because it is more efficient.

But the integration team is already using the Famsung REST API, in many parts of the product, refactor the whole product might cost them too much time.

What could the integration technology team do?

The Core and Integration teams have decided get a time off in order to get the best solution to this problem. It is going to be hard this time, but they don't want to do twice the same mistake.

Dangers

The logic is not isolated
Too coupled to third party libraries
The teams are not able to work confident
So easy to create bugs

For example: Imagine the requests or grpcio library get deprecated, in the first iteration you would have to migrate big part of the project too, in order to replace them to other libraries.

One of the core developers has recently read the Clean Architecture Book, and he would like to apply it to this situation in order to decouple the integration logic from the core logic.

The importance of the Intefaces

This time, they will create a couple of classes with the specific logic for each integration, they would like to keep the old Famsung Integration in case the necessities change in the future.

The integration team likes the solution proposed by the Core Team. But they they would like to add a few changes in order to work better.

The teams are going to divide the code, isolating the different parts for each team, this way they will avoid conflicts among different teams.

In order to coordinate both teams, they have decided to create an interface called TechProvider with an order_chip method, and it would be in charge of the Core Team, to prevent error in the core logic.

The Core Team will be in charge of the core logic and the provider contract, and the Integration Team will handle the integration logic with the different technology providers, always implementing the TechProvider interface.

These changes will add many advantages to the company.

Advantages

The teams will be able work without conflicts
They can create new integration with new provider without too much effort
They don't rely on specific technology (which can change and be deprecated)
The specific business logic is isolated avoiding generating undesired dependencies.

After a few months, they accomplished the task with the following result.

Code

Core Team Area

from fuawei.phone import Phone
from fuawei.tech_provider import TechProvider

class Fuawei:

    def __init__(self, provider: TechProvider):
        self.provider = provider

    def create_phone(self) -> Phone:
        chip = self.provider.ask_chip()
        return Phone(chip=chip)

import abc

from fuawei.fuawei_chip import FuaweiChip

class TechProvider(abc.ABC):

    @abc.abstractmethod
    def ask_chip(self) -> FuaweiChip:
        pass

Integration Team

import requests

from fuawei.fuawei_chip import FuaweiChip
from fuawei.tech_provider import TechProvider

class FamsungTechProvider(TechProvider):

    def ask_chip(self) -> FuaweiChip:
        return self._build_fuawei_chip(requests.get('http://fuawei-chips/web-service').json())

    def _build_fuawei_chip(self, data) -> FuaweiChip:
        # On these lines you would have a lot of parse work in order to extract all the necessary information to
        # build your fuawei chip

        return FuaweiChip()

import grpc

from fuawei.fuawei_chip import FuaweiChip
from fuawei.tech_provider import TechProvider

class FuaweiTechProvider(TechProvider):

    def ask_chip(self) -> FuaweiChip:
        fuawei_service = grpc.insecure_channel('fuawei/web-service')
        # Communication to the gRPC service of the Fuawei Provider
        return FuaweiChip()

Because of they know this code may help other developers, they decided to push the whole repository to github.

Conclusion

Structuring your code is key piece of software development, and the interfaces are an important tool to accomplish it.

The interfaces will keep your architecture organized and decoupled, they empower you to manage the dependencies you use in your project, and help you to test each part of your software project in a isolated way.

Also, a good architecture will help you to keep the technical debt low, it will avoid conflicts with other team members and even other teams, and it will allow you to work more confident in the features you are developing without being worried about creating bugs.

I have used another OOP principles in the example, I would recommend check the SOLID principles out if you don't know any of them.

Resources

https://github.com/juanbenitopr/fuawei

https://www.amazon.es/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164/

https://en.wikipedia.org/wiki/SOLID

Build Gunicorn from Scratch

Juan Benito — Sun, 07 Jun 2020 10:41:45 +0000

Many python developers use gunicorn to deploy their python applications, it is one of the most used python WSGI servers. In order to understand better how these servers works under the hood, I decided to create one on my own!

The server have the next main features:

Clean code: The project has educational purpose, in order to show how the wsgi specification works under the hood, the code is as explicit as possible, the code is decoupled in three different layers: Network Server, WSGI (wsgi specification), http2(domain entities)
It has to work!: The server has to be able to receive income requests, load the python application and return a valid response.

This project is not production ready, and it could contains bugs. Also, the project doesn't fulfill all the use cases and all the possible corner cases, it is not safe deploying our applications using it.

Funicorn

I decided to create Funicorn as the not production ready version of Gunicorn.

Overall Architecture

The architecture of the project is pretty simple, it contains three components:

Network components (Server and Request Handler): which handle the low level interaction with the OS.
WSGI Layer: It follows the wsgi specification, and it is able to talk to the python application.
Python application: It contains the business logic you want to perform with you application, it usually use a framework such as Flask or Django

Main Application

The main application is our server entry point, it receives all the necessary configuration to handle properly your application. Also, it is the place to handle the life of your server.

The funicorn class only receives the essential information to work properly. It passes the application_path parameters to the handler, and wrap the server features.

# funicorn/main.py
class Funicorn:

    def __init__(self, app_path: str, app_obj: str, host: str, port: int):
        FunicornHandler.app_path = (app_path, app_obj)

        self.server = TCPServer((host, port), FunicornHandler)

    def run(self):
        self.server.serve_forever()

    def stop(self):
        self.server.shutdown()
        self.server.server_close()

Server

The network server handle the low level network requirements. It establishes, keeps and finishes the network connection successfully, such as opening the network socket, binding the socket to the specific host and port, it also channel the information to the handler.

In order to prevent me from building the server from scratch and handling myself the network connection, I used the TCPServer from the socketserver python module, this module provides a few server classes with the basic features you would need.

Request Handler

The Funicorn Handler handle appropriately every request the server receive. it loads the python application, processes the data received by the server, and return the appropriate response.

My handler inherit from BaseRequestHandler which is a base class provided by the socket library of python.

# funicorn/main.py
class FunicornHandler(BaseRequestHandler):
    app_path: Tuple[str, str] = None

    def handle(self):
        application = load_application(self.app_path[0], self.app_path[1])

        http_request = self.request.recv(1024)

        request: Request = RequestDTO(http_request).request

        response: Response = application.process_request(request)

        response_data = ResponseDTO(response).http_data
        self.request.sendall(response_data)

Request and Response

In order to work easily with HTTP and network entities, I created a Request and Response classes, with all the necessary information.

The classes only isolate the request and response information such as headers, the method used and the body.

The body must be a file-like object, it is passed to the application and it must implement the method read, and write at least. I have modeled it using BytesIO.

# funicorn/http2.py
class Request:

    def __init__(self, method: str, uri: str, headers: Dict[str, Header], body: Body, query_param: Optional[str],
                 server_name: str, ssl=False, protocol: str='HTTP/1.1'):
        self.method = method
        self.headers = headers
        self.body = body
        self.uri = uri
        self.server_name = server_name
        self.query_param = query_param or ''
        self.scheme = 'https' if ssl else 'http'
        self.protocol = protocol

class Response:

    def __init__(self, status: str, headers: List[Header], body: Body, protocol: str):
        self.status = status
        self.headers = headers
        self.body = body
        self.protocol = protocol

class Body:

    def __init__(self, data: bytes = None):
        self._data = io.BytesIO(data) if data else io.BytesIO()

    def read(self):
        return self._data.getvalue()

    def write(self, data: bytes):
        return self._data.write(data)

    def __len__(self):
        return len(self._data.getvalue())

WSGI

The WSGI Application entity is the layer which wrap the python application and WSGI specification for the rest of the server. It receives a request entity, passes it to the python application and returns the processed response.

The main method is process_request , where the request is processed, and passed to the application through the environment variables. Finally, the response is built through the start_response method (defined in the specification), and returned to the Handler.

# funicorn/wsgi.py
class WSGIApplication:

    def __init__(self, application: Callable):
        self.application = application
        self._request: Request = None

        self._response_builder = ResponseBuilder()

    def process_request(self, request: Request) -> Response:
        self.request = request

        environ = WSGIEnvironment(request).environ

        http_body = self.application(environ, self.start_response)

        self.set_response_body(http_body)
        response: Response = self._response_builder.build()

        return response

    def start_response(self, status, response_headers: List[Tuple[str, str]], exc_info=None):
        if not exc_info:
            self.set_response_metadata(status, response_headers)
        else:
            raise exc_info[1].with_traceback(exc_info[2])

All the information provided by the request, it is wrapped in the environment variables in the following way.

There are many environment variables you need to provide to the Python Application In order to work correctly, there variables are defined in the python proposals, and much of them are mandatory.

Also, many of these variables are defined by the HTTP Protocol, each one provides important information to the application. I have only implemented the mandatory variables.

I won't explain these variables in more detail, because I consider there is already enough information about it on the web.

# funicorn/wsgi.py
class WSGIEnvironment:

    def set_cgi_environ(self):
        self.environ['REQUEST_METHOD'] = self._request.method
        self.environ['SCRIPT_NAME'] = ''
        self.environ['PATH_INFO'] = self._request.uri
        self.environ['QUERY_STRING'] = self._request.query_param
        self.environ['SERVER_NAME'] = self._request.server_name
        self.environ['SERVER_PROTOCOL'] = self._request.protocol
        self.environ['SERVER_PORT'] = '8000'
        self.environ['CONTENT_TYPE'] = self._request.headers.pop('CONTENT_TYPE', Header('', '')).value
        self.environ['CONTENT_LENGTH'] = self._request.headers.pop('CONTENT_LENGTH', Header('', '')).value
        self.environ.update({f'HTTP_{header.key}': header.value for header in self._request.headers.values()})

    def set_wsgi_environ(self):
        self.environ['wsgi.input'] = self._request.body
        self.environ['wsgi.input'] = self._request.body
        self.environ['wsgi.errors'] = sys.stderr
        self.environ['wsgi.version'] = (1, 0)
        self.environ['wsgi.multithread'] = False
        self.environ['wsgi.input_terminated'] = True
        self.environ['wsgi.url_scheme'] = self._request.scheme
        self.environ['wsgi.run_once'] = True

Python Application

The python application executed by our server is a simple Flask example. The application is able to receive GET and POST request.

GET: The application return "Hello World"
POST: The application return the same body from the request (an echo application)

# funicorn/examples/flask_example.py
app = Flask(__name__)

@app.route("/hola", methods=['GET', 'POST'])
def hello():
    if request.method == 'POST':
        return request.json
    return "Hello World"

Conclusion

Implementing this project was a great exercise because it forces you to understand many details you didn't know about some of the standard most used by the python community, and how it actually works under the hood, it is not magic, it is actually code!

I would recommend to anybody take a look to the respository where you can see the whole code, in order to take a better look to many details I was not able to summary here. You can clone it and play with it, as I commented before I tried to be as explicit as possible, but if you have any doubt let me know :)

Also, let me know in the comment section, if you liked the article and it helps you to understand how this kind of projects works.

Explaining WSGI and Python Applications

Juan Benito — Sat, 23 May 2020 11:19:50 +0000

WSGI is one of the concepts you read when you start developing your python web project. However is a widely used concept, it might be quite difficult to understand when you read it the first couple of times. this article will answer the next questions

What is WSGI?
Why was WSGI created?
How does WSGI help us when we deploy our python web projects?

Definition

WSGI means Web Server Gateway Interface, it is the standard which define the rules followed by the Application Servers, such as Gunicorn, and the Frameworks we use to develop our applications, such as Django.

History

When python started being popular as programming language, many developers wanted to use it for the prominent web world until then dominated by PHP. In order to use python code on the server, the community developed an Apache module called mod_python.

However mod_python was popular, many vulnerabilities and bugs came up. Finally, the community decided to create a formal specification.

This specification was defined in the proposals PEP 0333 and PEP 3333. The goals of these proposals was to create a standard solution safe and secure, but flexible enough to be used with different python technologies.

The found solution was WSGI.

Theory

This standard defined how an Application Server should load a Python Application and work together. It allowed developers to create different Application Servers which are able to work with any python application if they meet the WSGI specification.

Python Web Project communication diagram

The picture above illustrate how our application WSGI server and our python application communicate with each other. They are completely decoupled, they don't need to know each other, they only need to satisfy the WSGI standard.

The WSGI server is in charge to handle the information received by the peripherals of our server (it might be using a network socket, file, stdin...). The incoming information will be adapted to the WSGI standard, the server will load our python application and it will send the information to it. It will become the Gateway of our application.

The Python Application needs to understand how the WSGI server will load it, how he will receive the information, and how it will need to be returned. Our application will bootstrap itself in the appropriate way to be loaded by the server, and being able to communicate with it.

Code

The theory is fine, but as developers we like to see the theory translated into code. The examples below are an Application Server (Funicorn) and Python Application (Fango), neither of these programs are production ready, are not tested and might contains bugs. They are examples to illustrate how the standard works in a general way.

In order to get easier to understand, only the most important parts of the standard are implemented.

Application Server

# Sibling Fake of gunicorn
import os
import sys
from typing import List, Tuple


class Funicorn:

    def __init__(self, app_path: str):
        self.application = load_application(application_path)
        self.buffer_output = sys.stdout.buffer
        self.set_environ()

    def set_environ(self):
        self.environ = {**os.environ}

        self.environ['wsgi.input'] = sys.stdin.buffer
        self.environ['wsgi.errors'] = sys.stderr.buffer

    def write(self, data: bytes):
        self.buffer_output.write(data)

    def write_headers(self, status, headers: List[Tuple(str, str)]):
        self.write(f'Status: {status}\r\n'.encode())

        for hkey, hvalue in headers:
            self.write(f'{hkey}: {hvalue}\r\n'.encode())

            self.write('\r\n'.encode())

    def start_response(self, status, response_headers: List[Tuple(str, str)], **kwargs):
        self.write_headers(status, response_headers)

    def finish(self):
        self.buffer_output.flush()

    def run(self):
        response_body = self.application({**self.environ}, self.start_response)
        self.write(response_body)


if __name__ == 'main':
    application_path = 'path_to_fango.app'
    funicorn = Funicorn(application_path)

    funicorn.run()

Gist - Funicorn

Python Application

# Sibling fake of Django
from typing import Dict, Any


class Fango:

    def __call__(environ: Dict[str, Any], start_response) -> bytes:
        HELLO = b'Hello!'
        status = '200 OK'
        response_headers = [('Content-type', 'text/plain')]
        start_response(status, response_headers)
        return HELLO


app = Fango()

Gist - Fango

Conclusion

The WSGI standard boost the number of libraries created by the python community in the web applications fields, allowing the developers to decouple the server technology from the python application in a isolated way.

The framework developers who didn't need to worry about how to handle the socket connection, the income requests or concurrent connections, was able to spend the time in the creation of many python frameworks loved by many of us as Django, Flask, Pyramid... and so on.

If you would like to dive deeper about how WSGI works I would recommend you go directly to the proposals PEP 0333 and PEP 3333, they contains the whole specification.

Don't Use Django for Big Projects

Juan Benito — Sat, 16 May 2020 11:06:04 +0000

Django and Django Rest Framework are great tools for creating REST applications. However, when your application grow up keeping the technical debt low is really challenging. I have suffered these kind of situations many times using Django.

The goal of this article is sharing with you the pains I have found getting bigger and more complex my applicaitons.

DRY (Don’t Repeat Yourself)

This principle is used many times by the Django ecosystem, avoiding duplications in your codes is great, but you need to understand the necessities you are solving. Two different features may have some common parts now, but they might change in the future, understanding when and how your code should be isolated using different components would be the perfect solution, in order to avoid future problems (SRP).

In order to boost your productivity, Django (and its ecosystem) exceed the usage of DRY principle, it save you a lot of work, but the sideways are it generate many couples between different parts of your applications which will generate you future. We will explore when this principle is overused.

Models

The Django models are one of the core features of any Django Application, they provide you access to the database, and they save you a lot of work. Here Django use the models as core entity to avoid you repeat the same them in other components. Despite the good intention, coupling your business entities with your database entities is not a good path you want to walk.

What happen when your database change? you will need to change your models and your business core.

As well there are other Django and DRF components which base part of their features on these models, such as: ModelForms, ModelSerializers, ModelViewsets…

These components have different mission, not only store information or work business data, DAMN IT we have just coupled our database to the whole application (a little messy)

Furthermore you are able to use your models from any app of your project, so when you change any model, you probably are affecting things which shouldn’t be, as well as breaking business use-cases in other places, so… hell I hope you have a high test coverage to catch it, before going to production.

Django Model dependency diagram with two applications

Recap, using Django models you are coupling many components of your project, and likely you will lose track which are using them, business, representation, data… If you have many features to maintain good luck!

Serializers

Django Serializers are one of the most useful parts of the DRF library, they allow you serialize different kind of information without sweat, but we must use them carefully, if you mix different kind of entities and start nesting serializers, it will become messy.

The DRY principle invite use to reuse serializers in many use-cases, hoping the business use case is not going to change never…. After a few weeks we have started using those serializers with many other endpoints, and insider another serializers. Refactor the serializer is now an epic tasks. Don’t hear DRY here.

If the previous situation is tedious to resolve, when you add the complexity of the ModelSerializer it becomes almost impossible to solve. The ModelSerializer couple the representation logic to the database. Then, when your representation, or you database change you are in problems.

This might become the ModelSerializer the most terrible thing in the Django ecosystem.

Trust me, NESTING MODEL SERIALIZERS IS NOT A GOOD IDEA, querys to your database will get out under control quickly.

Views

DRF Views are quiet good tools to handle the income requests of your application.

The first problem appearear with GenericViews. Those views try to save you tons of work using a few attributes.

The queryset attribute is the most harmful one, tie your view to the database, and that’s not good. As well as It encourages you to custom the queryset for each use-case using get_queryset method. I discourage the use of this method, when you handle many use cases in the same view is complicated understand which data you want to retrieve each time.

The serializer_class attribute is useful at the beginning when you need a new serializer for each use case this will end up being a torture (same serializer with dynamics fields is worse).

One of the coolest features of DRF, at the beginning, are the filters, they allow you filter easily your set of data, and they are able to handle the most use cases. However when the fields are custom created as annotations, or nested filtering is necessary it will be much difficult to handle.

Django Views try to centralize all the possible logic inside the views, and it end getting too much logic you need to handle in the same place, with many conditions and no much code, but too much possible pitfalls. The nice views quickly become spaghetti code area.

GenericViews and family (Mixins, Viewsets, ModelViewsets…) are pretty nice when you don’t need custom features and many use-cases to handle, doing simple CRUDs is super easy, but when the features are hard you should look for another strategy.

Conclusions

Django and Django Rest Framework are amazing tools to create REST Applications super fast, in order to accomplish your impossible deadline. These advantages don’t come without Although sideways, to Django and DRF help you by coupling many parts of your application and generating many cool features on their own.

Structuring your application from the beginning of the project is an important part for creating a maintainable future. You can use different coding architectures and Design patterns to keep a robust and mantainable project.