DEV Community: Richard Devers

K8s / Nginx simple trick

Richard Devers — Fri, 07 Jan 2022 15:33:55 +0000

The issue

I recently have been through a simple yet interesting use case in my job.

For some reasons, network teams forbid developers computers to access mongoDb database directly.

Only the kubernetes platform have network access to the managed database service.

Basically:

The solution

Looking a the previous drawing, you may see where this is going, developers can access the k8s cluster and the k8s cluster can access to the managed db service ^^...

So... here is the plan:

So, let's deploy a simple nginx k8s service, configured to stream tcp connection from the port 8080 to the desired ip:port. Then use a port forward to redirect connection

Assuming your mondoDb listen on 100.101.102.103:27017

---

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: mongodb-proxy
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: mongodb-proxy
  ingress:
    - {}
  egress:
    - {}
  policyTypes:
    - Ingress
    - Egress

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb-proxy
data:
  nginx.conf: |
    worker_processes  auto;
    error_log  /var/log/nginx/error.log notice;
    pid        /tmp/nginx.pid;


    stream {
        server {
            listen  8080 so_keepalive=on;
            proxy_connect_timeout 2s;
            proxy_pass    stream_backend;
            proxy_timeout 10m;
        }
        upstream stream_backend {
          server 100.101.102.103:27017;
        }

    }

    events {
        worker_connections  1024;
    }

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: "mongodb-proxy"
  template:
    metadata:
      labels:
        app.kubernetes.io/component: "mongodb-proxy"
    spec:
      serviceAccountName: default
      securityContext: {}
      containers:
        - name: mongodb-proxy
          volumeMounts:
            - name: mongodb-proxy
              mountPath: /etc/nginx
          securityContext:
            readOnlyRootFilesystem: false
            runAsGroup: 1000
            runAsNonRoot: true
            runAsUser: 1000
          image: "nginx/nginx-unprivileged"
          imagePullPolicy: Always
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            # - name: https
            #   containerPort: 443
            #   protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 256Mi
      volumes:
        - name: mongodb-proxy
          configMap:
            name: mongodb-proxy

First, deploy this k8s template.

Note that:

the nginx config file is directly set within the template and put into the container using a configMap, thanks k8s.
the network policy is way too large (too lazy to write it , sorry)
the ngninx-unpriviled image have been used because of some security restrictions on our k8s cluster.

Now i just have to create a port-forward between my computer to the mongodb-proxy.

It should be something like:

kubectl port-forward mongodb-proxy 8080:8080

(i didn't test it, i use k9s to do that for me).

Now i can access my db directly from my computer using localhost:8080

using Firefox i got:

This solution is very easy to perform, can be adapt to lots of other backend and hold in a single small helm file.

Python libraries for quality programming

Richard Devers — Fri, 07 Jan 2022 14:39:25 +0000

pydantic

Pydantic is a library owned by Samuel Colvin which is an incredible developper. Go read his code, you'll learn.

From the doc:

Data validation and settings management using python type annotations.
pydantic enforces type hints at runtime, and provides user friendly errors when data is invalid.
Define how data should be in pure, canonical python; validate it with pydantic.

Basically, Pydantic class are supercharged dataclass.

Data validation, on-fly complex custom validation and/or transformation, de/serialization with optional key, attribute alias , secret management and so more can be accomplished with it. Cherry on the cake, it's mypy and Pylance compatible.

Just have a look at their users (spoiler: The NSA, AWS, Microsoft etc..)

Your application will be way more robust, you'll deliver way faster and you'll have a significant gain in serenity.

It's an absolute must use, a master class.

fastapi

fastapi is another master class, this time by Sebastián Ramírez.

From the doc:

- Fast: Very high performance, on par with NodeJS and Go (thanks to Starlette and Pydantic). One of the fastest Python frameworks available.
- Fast to code: Increase the speed to develop features by about 200% to 300%. *
- Fewer bugs: Reduce about 40% of human (developer) induced errors. *
- Intuitive: Great editor support. Completion everywhere. Less time debugging.
- Easy: Designed to be easy to use and learn. Less time reading docs.
- Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.
- Robust: Get production-ready code. With automatic interactive documentation.
- Standards-based: Based on (and fully compatible with) the open standards for APIs: OpenAPI (previously known as Swagger) and JSON Schema.

Everything is said. Fully async, based on Pydantic (of course!), easy to test, fast and so on... By far the best REST API framework i have worked with.

typer

Another one by Sebastián Ramírez, typer

From the doc:

- Intuitive to write: Great editor support. Completion everywhere. Less time debugging. Designed to be easy to use and learn. Less time reading docs.
- Easy to use: It's easy to use for the final users. Automatic help, and automatic completion for all shells.
- Short: Minimize code duplication. Multiple features from each parameter declaration. Fewer bugs.
- Start simple: The simplest example adds only 2 lines of code to your app: 1 import, 1 function call.
- Grow large: Grow in complexity as much as you want, create arbitrarily complex trees of commands and groups of subcommands, with options and arguments.

Built on Pydantic, typer allow you to create simple CLI within minutes and complex one in a few hours.

Easy to test, easy to build, another great one to simplify your life.

httpx

In my opinion, httpx is the best http client so far. sync/async support and everything you'll ever need. Easy to use. A great One.

httpx tests are easy to do using respx. You can mock it very simply, side_effect management etc..

Bonus: datamodel-code-generator

Ok so pydantic help you create powerfull class models?

What if you don't even have to write them?

That's what datamodel-code-generator propose.

You put a swagger file in input, you get a python module with every swagger models as pydantic class in it as output! Magic!

Modern Python setup for quality development

Richard Devers — Fri, 07 Jan 2022 11:41:46 +0000

In this article, i'll show you what is my common python setup.

Goals are:

High quality code standard
Respect of the python standard
Dev experience quality
Security
repeatability

We'll see:

The IDE
Python versions
Package and virtual environments
Code linter/formatter
Tests tools
Documentation
Pre-commit
Security

This is entirely subjective :)

IDE: VSCode

VSCode is a great IDE for python programming.

One of his best feature is his extensibility though extensions.

Here are some extensions i used:

For the general development environment

Remote-Container: Open any folder or repository inside a Docker container and take advantage of Visual Studio Code's full feature set. Great to ensure every dev environment through a team are identical.
Python: Python IntelliSense (Pylance), Linting, Debugging (multi-threaded, remote), Jupyter Notebooks, code formatting, refactoring, unit tests, and more. I don't use Jupyter so i cant talk about it. However Pylance is a great tool. Under the hood it used PyRights which is a super fast code analyzer (way faster than mypy!). Don't forget add this to your settings:
vscode-icons: Each folder type have his own well design icons. Big project with a lots of files are way more easy to understand with that.
Window Color: Automatically adds a unique color to each window's activityBar and titleBar. A project will always have the same color. Great when a teams works on multiple project, everybody see the same color.
GitLens: Now you can see who wrote that directly into the code

For data parsing/formatting/transformation...

There's lots of extensions for that depending on your use case (docker, helm, etc...)

I think these one are required by pretty much every project:

Better TOML
change-case
Code Spell checker
JSON Tools
XML Tools
YAML
vscode-base64
Sort JSON objects

Python versions: Pyenv

Pyenv is so far the best python versions manager.

Python package and venv: Poetry

Poetry is a tool for dependency management and packaging in Python. It allows you to declare the libraries your project depends on and it will manage (install/update) them for you.

Basically, it's pip under steroids. It works with the latest package file format pyproject.toml . His dependencies and build system are very reliable. You can still export requirements.txt if necessary. Virtual environments works like a charms. Multiple python version too. It solve a lot of pain point. This is a must have.

Code linter/formatter

flake8: Flake8 is a wrapper around these tools: PyFlakes pycodestyle Ned Batchelder's McCabe script

Flake8 runs all the tools by launching the single flake8 command. It displays the warnings in a per-file, merged output.

black: Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Blackened code looks the same regardless of the project you're reading. Formatting becomes transparent after a while and you can focus on the content instead.

Black makes code review faster by producing the smallest diffs possible.

isort: isort your imports, so you don't have to.

For the tests

pytest: Python allow to write way less code than using directly unittest. Lots of good library handle it very well. Some library extend it to be even more efficient and user friendly (pytest-mock, pytest-freezegun, pytest-sugar...)
nox: The first point of nox is to run your tests into multiple environment (multiple python version). In my opinion way better than tox

For the documentation: Mkdocs

Mkdocs allow you to write documentation using markdown format. Really easy to use and integrate. You can easily integrate your documentation to your CI/CD pipeline for automatically test it then deploy it.

Pre-commit

This is a must have. pre-commit allow you to run multiple scripts (a.k.a pre-commit) before your commit validation. There's a big catalogue of builtin pre-commit. Some will modify files. Some will just return warning or errors. You can automatically perform a TONS of tests/validation before committing anything.

This is basically a local CI pipeline run before the real one.

Here is an example of one .pre-commit-config.yaml config file i use, among others it will:

sort your import automatically
black will format your code
flake8 will warn you about style issue
Validate .gitlab-ci.yaml against my enterprise gitlab server.
check style for bash script
and so much more...

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.0.1
    hooks:
      - id: check-added-large-files
      - id: check-ast
      - id: check-builtin-literals
      - id: check-case-conflict
      - id: check-docstring-first
      - id: check-executables-have-shebangs
      - id: check-json
      - id: check-merge-conflict
      - id: check-symlinks
      - id: check-toml
      - id: check-vcs-permalinks
      - id: check-xml
      - id: check-yaml
        args: [--allow-multiple-documents]
      - id: debug-statements
      - id: detect-aws-credentials
        args: [--allow-missing-credentials]
      - id: destroyed-symlinks
      - id: end-of-file-fixer
      - id: fix-byte-order-marker
      - id: fix-encoding-pragma
        args: [--remove]
      - id: forbid-new-submodules
      - id: mixed-line-ending
        args: [--fix=auto]
      - id: name-tests-test
        args: [--django]
      - id: requirements-txt-fixer
      - id: trailing-whitespace
  - repo: local
    hooks:
      - id: black
        name: black
        entry: poetry run black
        language: system
        types: [python]
      - id: flake8
        name: flake8
        entry: poetry run flake8
        language: system
        types: [python]
  - repo: https://github.com/pycqa/isort
    rev: "5.9.1"
    hooks:
      - id: isort
        args:
          - --profile
          - black
          - --filter-files
  - repo: https://github.com/adrienverge/yamllint.git
    rev: v1.26.1
    hooks:
      - id: yamllint
        args: [-c=.yamllint.yaml]
  - repo: https://gitlab.com/devopshq/gitlab-ci-linter
    rev: v1.0.2
    hooks:
      - id: gitlab-ci-linter
        args:
          - "--server"
          - "https://your.gitlab.server" # Need env var GITLAB_PRIVATE_TOKEN with gitlab api read token
  - repo: https://github.com/commitizen-tools/commitizen
    rev: v2.17.11
    hooks:
      - id: commitizen
        stages: [commit-msg]
  - repo: https://github.com/jumanjihouse/pre-commit-hooks
    rev: 2.1.5 # or specific git tag
    hooks:
      - id: forbid-binary
      - id: shellcheck
      - id: shfmt

Security

bandit: Bandit is a tool designed to find common security issues in Python code.

To do this Bandit processes each file, builds an AST from it, and runs appropriate plugins against the AST nodes. Once Bandit has finished scanning all the files it generates a report.

safety: Safety checks your installed dependencies for known security vulnerabilities.

By default it uses the open Python vulnerability database Safety DB, but can be upgraded to use pyup.io's Safety API using the --key option.

gitleaks: Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.

Job is done but if someone have better alternative, i would try them.

Conclusion

I hope you leaned a thing or two in this article.

Using these tools can help you falling in some common pitfall.

Pylance warning & errors are great best practices teachers. Your code understanding and therefore quality will greatly progress understanding and correcting them.

Python use case - Pagination

Richard Devers — Thu, 06 Jan 2022 17:28:45 +0000

The use case

A common use in development is to get some data from an external backend.

For performance reason, most implements pagination.

Let see how we can get all result from a backend.

For these examples, i'll use httpx to perform the http operation. It's an excellent client, you should use it.

Note that these are quick examples, not suitable for production environment (no pydantic, no security, no retry, no error management etc...)

In my examples i only print the lenght of the results, however, feel free to do something more interesting ;)

The backend

https://api.instantwebtools.net/v1/passenger?page=0&size=10

It provides a paginated API of a fake airline company.

Result looks like that:

{
    "totalPassengers": 19675,
    "totalPages": 1968,
    "data": [
        {
            "_id": "5ff393986feae02b6c22b251",
            "name": "Deepanss",
            "trips": 0,
            "airline": [
                {
                    "id": 3,
                    "name": "Cathay Pacific",
                    "country": "Hong Kong",
                    "logo": "https://upload.wikimedia.org/wikipedia/en/thumb/1/17/Cathay_Pacific_logo.svg/300px-Cathay_Pacific_logo.svg.png",
                    "slogan": "Move Beyond",
                    "head_quaters": "Cathay City, Hong Kong International Airport, Chek Lap Kok, Hong Kong",
                    "website": "www.cathaypacific.com",
                    "established": "1946"
                }
            ],
            "__v": 0
        },...
    ]
}

The async implementation

A common implementation would look like that:


import asyncio
import typing as t

import httpx


async def get_passengers(
    page: int = 0,
    size: int = 10,
) -> httpx.Response:
    print(f"Getting page n°{page}")

    response: httpx.Response = httpx.get(
        f"https://api.instantwebtools.net/v1/passenger?page={page}&size={size}"
    )
    response.raise_for_status()

    return response


async def get_all_passengers() -> None:
    page = 0
    result = []
    while True:
        response = await get_passengers(page=page)
        response_data = response.json()

        result += response_data["data"]
        if page >= response_data["totalPages"]:
            break
        page += 1

    print(f"total result:{len(result)} ")


if __name__ == "__main__":
    asyncio.run(get_all_passengers())

Output should look like:

Getting page n°0
Getting page n°1
Getting page n°2
Getting page n°3
Getting page n°4
Getting page n°5
Getting page n°6
Getting page n°7
Getting page n°8
Getting page n°9
Getting page n°10
total result:110

The async for implementation

This implementation use async generator in order to use async for .

import asyncio
import typing as t

import httpx


async def get_passengers(
    page: int = 0,
    size: int = 10,
) -> httpx.Response:
    print(f"Getting page n°{page}")

    response: httpx.Response = httpx.get(
        f"https://api.instantwebtools.net/v1/passenger?page={page}&size={size}"
    )
    response.raise_for_status()

    return response


async def get_all_passengers() -> t.AsyncGenerator[t.Any, None]:
    page = 0

    while True:
        response = await get_passengers(page=page)
        response_data = response.json()

        for passenger in response_data["data"] or []:
            yield passenger

        if page >= response_data["totalPages"]:
            break
        page += 1


async def get_all_passengers_async_for() -> None:
    result = [passenger async for passenger in get_all_passengers()]
    print(f"total result:{len(result)} ")


if __name__ == "__main__":
    asyncio.run(get_all_passengers_async_for())

This implementation is based on generator. Since the async get_all_passengers method yield a result, it become an AsyncGenerator.

AsyncGenerator implements the __aiter__ special method, that's why you can perform an async for on like in get_all_passengers_async_for .

In order to test it, you'll have to mock the __aiter__ attribute.

Example from the official documentation

>>> mock = MagicMock()  # AsyncMock also works here
>>> mock.__aiter__.return_value = [1, 2, 3]
>>> async def main():
...     return [i async for i in mock]
...
>>> asyncio.run(main())
[1, 2, 3]

Conclusion

I hope you enjoy learning this article :)

You have two way to handle pagination into your code, depending on your use case :)

Python import - The law of Demeter

Richard Devers — Tue, 04 Jan 2022 10:15:51 +0000

The Demeter Law

Python import can be a little tricky. The law of Demeter guideline and his applications to python import can help you mastering them and also writes more user-friendly, testable and reliable code.

According to wikipedia, the law of Demeter (or principle of least knowledge) is a software developing guideline that can be summarized this way:

Each unit should have only limited knowledge about other units: only units "closely" related to the current unit.
Each unit should only talk to its friends; don't talk to strangers.
Only talk to your immediate friends.

If we want to apply these to python import (the python module would be the unit), it would become something like:

Each module should only import public object of submodule, but only from the closest submodule. (aka "immediate friends")

The method

In order to accomplish that, the key is to declare explicitly in each module (though the _init_.py) which object are exposed. Then these objects are accessible to the parent module folder, which can also re-expose them if necessary.

The benefices of this method are:

Your _init_.py files become some kind of "documentation" of which objects should be accessed or not from other modules.
You fully control your import system, which is a key to loose-coupling.
The final user (other developper using your module) experience is enhanced, import are short and clear.

A downside is that it is more verbose.

First example

Assuming this setup:

project
│   __main__.py
└───demeter
│   │   __init__.py
│   └───module1
│       │   __init__.py
│       │   code1.py

# demeter/module1/code1.py

class MyClass1:
    pass

With that setup, we can access directly MyClass1 class from _main_.py:

# __main__.py

from demeter.module1.code1 import MyClass1

class1 = MyClass1()

This works, however this is a demeter's law violation: _main_.py import MyClass1 from demeter/module1/code1.py which is not an "immediate friend".

A good smell to detect it are long import. When applying the law of demeter, import are usually pretty short.

A cleaner way to do that would be...

Explicitly expose MyClass1 from module1 to upper level module:


# demeter/module1/__init__.py

from demeter.module1 import code1 as code1

MyClass1 = code1.MyClass1

Re-expose it to the upper level:

# demeter/__init__.py

from demeter import module1

MyClass1 = module1.MyClass1

Use it

# __main__.py

from demeter import module1

MyClass1 = module1.MyClass1

That way:

_main_.py only have knowledge of the demeter module and the demeter module only have knowledge about the module1 module. The principe of least knowledge is respected
modules (demeter and module1) explicitly expose what can be used from them(explicit is better than implicit, https://www.python.org/dev/peps/pep-0020/).

Second example

Assuming this extension of the previous setup:

project
│   __main__.py
└───demeter
│   │   __init__.py
│   └───module1
│       │   __init__.py
│       │   code1.py
|       └───module2
│       │   __init__.py
│       │   code2.py

# demeter/module1/module2/code2.py

class MyClass2:
    pass

To use MyClass2 in _main_.py we would...

Expose it to module1:

# demeter/module1/module2/__init__.py

from demeter.module1.module2 import code2 as code2

MyClass2 = code2.MyClass2

Re-expose it but this time from module1:

# demeter/module1/__init__.py

from demeter.module1 import code1 as code1
from demeter.module1 import module2 as module2

MyClass1 = code1.MyClass1
MyClass2 = module2.MyClass2

and...

# demeter/__init__.py

from demeter import module1

MyClass1 = module1.MyClass1
MyClass2 = module1.MyClass2

Finally, use it from your _main_.py:

# __main__.py

from demeter import module1

MyClass1 = module1.MyClass1
MyClass2 = module1.MyClass2

Whole module example

The law of demeter can also be applied to whole module.

The previous example could be something like:

# demeter/module1/code1.py

class MyClass1:
    pass

# demeter/module1/module2/code2.py

class MyClass2:
    pass

# demeter/module1/module2/__init__.py

from  demeter.module1.module2  import  code2  as  code2

# demeter/module1/__init__.py

from  demeter.module1  import  code1  as  code1
from  demeter.module1  import  module2  as  module2

MyClass1 = code1.MyClass1

# demeter/__init__.py

from  demeter  import  module1

MyClass1 = module1.MyClass1


# demeter/__main__.py
import  demeter

module1 = demeter.module1

MyClass1 = module1.MyClass1
MyClass2 = module1.module2.code2.MyClass2

This second method looks very similar to the bad practice example, isn't it?
However, the key points are that every desired module and object are explicitly exposed.

So you still benefice the full control of your module system and you respect the principle of least knowledge.

VSCode, Pylance, Pyrights

If you use VSCode and Pylance (which use Pyrights under the hood), this method will avoid you to have some reportUnknownMemberType errors

Conclusion

I hope you learned something reading this article (my first one uWu ^^).

It's up to you to choose if you prefer expose full module or only object. I guess it depends on the context, the project, the people etc....

Even if it look a little "too much" and require more code and more rigorous , these methods really enhance control and quality on the long run as well as deeply understanding (and thinking about ;) ) responsibility of every part of your code.