DEV Community: Roman Imankulov

Refiner

Roman Imankulov — Thu, 10 Aug 2023 13:04:57 +0000

I created a Refiner, an open-source project that automatically fixes grammar and stylistic errors. It can also adjust the tone and formatting.

That is addictively useful. Now, pretty much all my git commit messages, comments to Jira tickets, and Slack comments pass through the wisdom of this tool. The post you are reading has also gone through a series of refinements.

How does it help me?

It helps minimize any non-native speaker quirks. Sometimes, I struggle with complex grammar constructions or end up using expressions that don't quite exist. I'm sure my English writing has a noticeable Russian accent, even if I can't hear it. The refiner helps me make my writing sound more natural to native speakers.

Also, it's a lifesaver when I'm tired. There are times when my brain power is low and I can't express my thoughts clearly. My writing ends up sounding clumsy. For example, after a long day of research, when I need to share my findings with colleagues. Also, when I'm tired, my text can unintentionally come across as pessimistic or aggressive. The "friendly" and "positive" refiners help me make my text more appropriate without sounding fake.

It even does some of the work for me. I recently added an experimental "Finish the sentence" feature. Sometimes, we write things that are easy to predict. When I catch myself writing one of these clichés, I stop my sentence midway and add XXX to let the refiner complete it. Surprisingly, it works well for things like installation instructions and model definitions.

How does it work?

The refiner is basically a wrapper around OpenAI GPT. It takes the prompt, sends the request to OpenAI, and then formats the response to show the difference.

Why not use ChatGPT?

Initially, I used ChatGPT for this task, but I encountered two issues:

I had to repeat the same prompt over and over again, with slight contextual modifications.
More importantly, I had to proofread the entire text generated by GPT to make sure it didn't add any nonsensical stuff and still "sounded like me."

With Refiner, I save time by not having to type the prompt and only reviewing the highlighted parts of the text.

How can you use it?

Since it's an open-source tool, you can run it yourself. The source code is available at https://github.com/imankulov/refiner. The only thing you need to pass is an OpenAI API token.

I also set up a testing installation at https://refiner.roman.pt/, where you can test it for free without SMS and registration. At least, for now.

How much does it cost?

I was initially concerned about the cost of making it free and public. How much would it be worth for me to use it regularly?

I estimated the number of used tokens here, and checked the prices here. Given this text as an example, it has roughly 600 tokens. Instructions take about 100-200 tokens depending on the number of refiners. Therefore, reformatting the text would take about 800 input tokens and 600 output tokens.

The cost of editing this text is less than a quarter of a cent.

I'm fine with running the tool without any guardrails, as long as it's just me, my friends, and anyone who stumbled upon this blog post and finds the tool helpful.

Django Plausible Proxy

Roman Imankulov — Wed, 27 Apr 2022 10:34:15 +0000

I released Django Plausible Proxy, a Django application to proxy requests and send server-side events to Plausible Analytics.

Plausible is a lightweight and open-source web analytics platform, a privacy-friendly alternative to Google Analytics. I started using it for Django side projects about a month ago and loved its minimalistic interface and "exactly what I need" type of reports.

Initially, the integration code lived inside the project, but as it grew in functionality, I extracted it in a separate package that is available on PyPI and GitHub.

Proxying

Proxying allows a project owner concerned about missing data to see a complete picture. See Adblockers and using a proxy for analytics for the detailed outline of the problem and solution.

When installed and configured in settings.py and urls.py, the app proxies the HTTP requests as such:

https://<yourdomain.com>/js/script.js -> https://plausible.io/js/script.js
https://<yourdomain.com>/api/event    -> https://plausible.io/api/event

Server-side events

Sending events from Python code lets you keep track of events that can't be recorded otherwise, such as API requests. In the code, it looks like this:

from plausible_proxy import send_custom_event
...
send_custom_event(request, name="Register", props={"plan": "Premium"})

Less red tape on reviewing PRs with the Ship / Show / Ask strategy

Roman Imankulov — Mon, 27 Dec 2021 15:02:24 +0000

The PR model discourages quick iterations. Effectively, it kills small refactorings and documentation improvements.

When PR approvals are obligatory, I build hierarchies of PRs or isolate refactorings in separate commits. Still, I feel it's too complicated, and often I choose not to do anything.

Rouan Wilsenach observes that not all code changes were born equal and require the same ceremony before the merge:

"Ship" -- No PR. Small and safe changes that can go directly to the mainline.
"Show" -- Open a PR and merge when automated tests pass. Then notify others to share your knowledge or ask for improvement advice. Btw, @gitlab has a magic checkbox "Merge when pipeline succeeds".
"Ask" -- Open a PR and wait for the review. It works well when you're not sure you're taking the right approach and need a second opinion.

The code author has the freedom to decide in which category the code change falls, which sounds like unprecedented blasphemy in teams with trust issues.

On the other hand, I worked in a team where we naturally applied this approach. We felt incredibly productive and nothing bad happened.

Read more about this approach and how to make it work for your team.

Ship / Show / Ask

Don't let dicts spoil your code

Roman Imankulov — Mon, 27 Dec 2021 14:27:13 +0000

How often do your simple prototypes or ad-hoc scripts turn into fully-fledged applications?

The simplicity of organic code growth has a flip side: it becomes too hard to maintain. The proliferation of dicts as primary data structures is a clear signal of tech debt in your code. Fortunately, modern Python provides many viable alternatives to plain dicts.

What's wrong with dicts?

Dicts are opaque

Functions that accept dicts are a nightmare to extend and modify. Usually, to change the function that takes a dictionary, you must manually trace the calls back to the roots, where this dict was created. There is often more than one call path, and if a program grows without a plan, you'll likely have discrepancies in the dict structures.

Dicts are mutable

Changing dict values to fit a specific workflow is tempting, and programmers often abuse this functionality. In-place mutations may have different names: pre-processing, populating, enriching, data massage, etc. The result is the same. This manipulation hinders the structure of your data and makes it dependent on the workflow of your application.

Not only do dicts allow you to change their data, but they also allow you to change the very structure of objects. You can add or delete fields or change their types at will. Resorting to this is the worst felony you can commit to your data.

Treat dicts as the wire format

A common source of dicts in the code is deserializing from JSON. For example, from a third-party API response.

>>> requests.get("https://api.github.com/repos/imankulov/empty").json()
{'id': 297081773,
 'node_id': 'MDEwOlJlcG9zaXRvcnkyOTcwODE3NzM=',
 'name': 'empty',
 'full_name': 'imankulov/empty',
 'private': False,
...
}

A dict, returned from the API.

Make a habit of treating dicts as a "wire format" and convert them immediately to data structures providing semantics.

Use serializer and deserializer to convert between wire format and internal representation

The implementation is straightforward.

Define your domain models. A domain model is simply a class in your application.
Fetch and deserialize in the same step.

In Domain-Driven Design (DDD), this pattern is known as the anti-corruption layer. On top of semantic clarity, domain models provide a natural layer that decouples the exterior architecture from your application's business logic.

Two implementations of a function retrieving repository info from GitHub:

❌ Returning a dict

import requests

def get_repo(repo_name: str):
    """Return repository info by its name."""
    return requests.get(f"https://api.github.com/repos/{repo_name}").json()

The output of the function is opaque and needlessly verbose. The format is defined outside of your code.

>>> get_repo("imankulov/empty")
{'id': 297081773,
 'node_id': 'MDEwOlJlcG9zaXRvcnkyOTcwODE3NzM=',
 'name': 'empty',
 'full_name': 'imankulov/empty',
 'private': False,
 # Dozens of lines with unnecessary attributes, URLs, etc.
 # ...
}

✅ Returning a domain model


class GitHubRepo:
    """GitHub repository."""
    def __init__(self, owner: str, name: str, description: str):
        self.owner = owner
        self.name = name
        self.description = description

    def full_name(self) -> str:
        """Get the repository full name."""
        return f"{self.owner}/{self.name}"

def get_repo(repo_name: str) -> GitHubRepo:
    """Return repository info by its name."""
    data = requests.get(f"https://api.github.com/repos/{repo_name}").json()
    return GitHubRepo(data["owner"]["login"], data["name"], data["description"])

>>> get_repo("imankulov/empty")
<GitHubRepo at 0x103023520>

While the example below has more code, this solution is better than the previous one if we maintain and extend the codebase.

Let's see what the differences are.

The data structure is clearly defined, and we can document it with as many details as necessary.
The class also has a method full_name() implementing some class-specific business logic. Unlike dicts, data models allow you to co-locate the code and data.
GitHub API's dependency is isolated in the function get_repo(). The GitHubRepo object doesn't need to know anything about the external API and how objects are created. This way, you can modify the deserializer independently from the model or add new ways of creating objects: from pytest fixtures, the GraphQL API, the local cache, etc.

☝️ Ignore fields coming from the API if you don't need them. Keep only those that you use.

In many cases, you can and should ignore most of the fields coming from the API, adding only the fields that the application uses. Not only duplicating the fields is a waste of time, but it also makes the class structure rigid, making it hard to adopt changes in the business logic or add support to the new version of the API. From the point of view of testing, fewer fields mean fewer headaches in instantiating the objects.

Streamline model creation

Wrapping dicts require creating a lot of classes. You can simplify your work by employing a library that makes "better classes" for you.

Consider dataclasses

Starting from version 3.7, Python provides Data Classes. The dataclasses module of the standard library provides a decorator and functions for automatically adding generated special methods such as __init__() and __repr__() to your classes. Therefore, you write less boilerplate code.

I use dataclasses for small projects or scripts where I don't want to introduce extra dependencies. That's how the GitHubRepo model looks like with dataclasses.

from dataclasses import dataclass

@dataclass(frozen=True)
class GitHubRepo:
    """GitHub repository."""
    owner: str
    name: str
    description: str

    def full_name(self) -> str:
        """Get the repository full name."""
        return f"{self.owner}/{self.name}"

When I create Data Classes, my Data Classes are almost always defined as frozen. Instead of modifying an object, I create a new instance with dataclasses.replace(). Read-only attributes bring peace of mind to a developer, reading and maintaining your code.

... or consider Pydantic

Recently Pydantic, a third-party data-validation library became my go-to choice for model definition. Compared with dataclasses, they are much more powerful. I especially like their serializers and deserializers, automatic type conversions, and custom validators.

Serializers simplify storing records to external storage, for example, for caching. Type conversions are especially helpful when converting a complex hierarchical JSON to a hierarchy of objects. And validators are helpful for everything else.

With Pydantic, the same model can look like this.

from pydantic import BaseModel


class GitHubRepo(BaseModel):
    """GitHub repository."""
    owner: str
    name: str
    description: str

    class Config:
        frozen = True

    def full_name(self) -> str:
        """Get the repository full name."""
        return f"{self.owner}/{self.name}"

You can find some examples of me using Pydantic, in my post Time Series Caching with Python and Redis.

For legacy codebase, annotate dicts as TypeDict

Python 3.8 introduced so-called TypedDicts. In runtime, they behave like regular dicts but provide extra information about their structure for developers, type validators, and IDEs.

If you come across the dict-heavy legacy code and can't refactor everything yet, at least you can annotate your dicts as typed ones.

from typing import TypedDict


class GitHubRepo(TypedDict):
    """GitHub repository."""
    owner: str
    name: str
    description: str

repo: GitHubRepo = {
    "owner": "imankulov",
    "name": "empty",
    "description": "An empty repository",
}

Below, I provide two screenshots from PyCharm to show how adding typing information can streamline your development experience with the IDE and protect you against errors.

PyCharm knows about the value type and provides autocomplete

PyCharm knows about the missing key and issues a warning

For key-value stores, annotate dicts as mappings

A legitimate use-case of dict is a key-value store where all the values have the same type, and keys are used to look up the value by key.

colors = {
    "red": "#FF0000",
    "pink": "#FFC0CB",
    "purple": "#800080",
}

A dict, used as a mapping.

When instantiating or passing such dict to a function, consider hiding implementation details by annotating the variable type as Mapping or MutableMapping. On the one hand, it may sound like overkill. Dict is default and by far the most common implementation of a MutableMapping. On the other hand, by annotating a variable with mapping, you can specify the types for keys and values. Besides, in the case of a Mapping type, you send a clear message that an object is supposed to be immutable.

Example

I defined a color mapping and annotated a function. Notice how the function uses the operation allowed for dicts but disallowed for Mapping instances.

# file: colors.py
from typing import Mapping

colors: Mapping[str, str] = {
    "red": "#FF0000",
    "pink": "#FFC0CB",
    "purple": "#800080",
}

def add_yellow(colors: Mapping[str, str]):
    colors["yellow"] = "#FFFF00"

if __name__ == "__main__":
    add_yellow(colors)
    print(colors)

Despite wrong types, no issues in runtime.

$ python colors.py
{'red': '#FF0000', 'pink': '#FFC0CB', 'purple': '#800080', 'yellow': '#FFFF00'}

To check the validity, I can use mypy, which raises an error.

$ mypy colors.py
colors.py:11: error: Unsupported target for indexed assignment ("Mapping[str, str]")
Found 1 error in 1 file (checked 1 source file)

Take dicts under control

Keep an eye on your dicts. Don't let them take control of your application. As with every piece of technical debt, the further you postpone introducing proper data structures, the more complex the transition becomes.

Structure Flask project with convention over configuration

Roman Imankulov — Wed, 05 May 2021 08:42:45 +0000

When people ask me what a should beginner Python developer choose, Flask or Django, to start their new web project, all other things being equal, I always recommend Django.

Unlike Flask, Django is opinionated. In other words, every time you need to make a choice in Flask, Django has the answer for you. Among others, Django outlines the practical project structure out of the box.

With Django, you split your project into applications. The applications themselves have a well-defined structure: views.py, models.py, all those things. What maintains and enforces the design is the convention over configuration approach: you put your code to specific well-known locations, and the framework discovers them.

Flask doesn’t impose anything like that. You can start with a single file. When the application grows, it’s up to you to provide the application structure. As a result, the app can quickly turn into an unmaintainable mess.

Official Flask documentation, following the non-opinionated principle, leaves it for developers to decide. Chapters Larger Applications Modular Applications with Blueprints don’t cover the topic entirely. To close the gap, people created their own guidelines. Google “flask project structure” to find a plethora of variants and suggestions. For example, there is a chapter a better application structure in the monumental Flask Mega-tutorial.

What bugs me about any Flask solution is the lack of support for conventions. If you’re like me, you have a function app() that manually imports and configures all the things from all the packages and blueprints. Looks familiar? This is dirty.

# file: myproject/app.py

def app():
    from myproject.users.controller import blueprint as users_blueprint
    from myproject.projects.controller import blueprint as projects_blueprint
    ...
    # Initialize extensions
    db.init_app(flask_app)

    ...
    # Register blueprints
    flask_app.register_blueprint(users_blueprint)
    flask_app.register_blueprint(projects_blueprint)
    ...

For every extension, you call init_app. For every application with a well-defined structure, you make a dance, adding a couple of lines with imports and registrations.

To somehow address the issue, I created a Python package roman-discovery. The package lets you declaratively define the conventions of your application and run a discover() function to apply their rules. It's not specific to Flask, but I created it primarily with Flask in mind.

For example, assuming that you store all the blueprints in the myproject/<app>/controllers.py, that’s how you can automatically register all of them.

from flask import Blueprint
from roman_discovery import ObjectRule, discover
from roman_discovery.matchers import MatchByPattern, MatchByType

blueprint_loader = ObjectRule(
    name="Flask blueprints loader",
    module_matches=MatchByPattern(["myproject.*.controllers"]),
    object_matches=MatchByType(Blueprint),
    object_action=flask_app.register_blueprint,
)

discover(import_path="myproject", rules=[blueprint_loader])

There is a roman_discovery.flask module you can use as a source of inspiration, or, if you don’t mind applying my conventions, use it as is.

from roman_disovery.flask import discover_flask

app = Flask(__name__)
app.config.from_object("myproject.config")
discover_flask("myproject", app)

The latest line will do the following.

Scan myproject/*/controllers.py and myproject/*/controllers/*.py to find blueprints and attach them to the Flask application.
Import all files in myproject/*/models.py and myproject/*/models/*.py to help flask-migrate find all the SQLAlchemy models to create migrations.
Scan all files in myproject/*/cli.py and myproject/*/cli/*.py to find flask.cli.AppGroup instances and attach them to Flask’s CLI.
Scan top-level myproject/services.py, find all the instances that have init_app() methods, and call obj.init_app(app=app) for each of them.

It’s still new, lacks documentation, examples, and tests, but I hope it can already become helpful for you.

Follow roman-discovery on GitHub, also to know why the package has an uncommon “roman-” prefix.

imankulov / deescovery

Discover packages and classes in a python project

Deescovery

Micro-framework initialization problem

Micro-framework-based projects are clean while they're small. Every micro-framework codebase I've seen, has a mess in the project initialization. With time, create_app() becomes filled with ad-hoc settings, imports-within-functions, and plug-in initializations.

The Application Factory Pattern, proposed, for example, in the official Flask documentation, and the Flask Mega-Tutorial, legitimize this approach.

The nature of create_app() leaves no place for the open-closed principle. We update this module every time we add a new plug-in, a new blueprint, or a new package.

# myproject/__ini__.py
#
# A common Flask application. The code is based on the Flask Mega-Tutorial.
def create_app(config_class=Config):
    app = Flask(__name__)
    app.config.from_object(config_class)

    db.init_app(app)
    migrate.init_app(app, db)
    login.init_app(app)
    mail.init_app(app)
    bootstrap

…

View on GitHub

21 tools to document your Python project

Roman Imankulov — Wed, 17 Mar 2021 15:53:15 +0000

Overview of tools and services to document your Python web application from installation instructions to public API. How to make sure API documentation is in sync with your code. How to serve internal documentation and keep it private.

Why documentation matters

One of the best questions you can ask a team before joining it is whether it has any documentation. As for restaurants, dirty bathrooms say dirty kitchen; for software companies, poor documentation smells rusty software design and poor processes.

Below, I will share what I know about the tools for creating API and project documentation. I will not touch on the questions of processes; it's a separate topic. Here, only tools and services. My default context for a project is a web project exposing an HTTP API and likely written in Python.

API documentation

No matter, if the API is public or private, it needs to be documented. We have several options on the plate.

Dropbox Paper or Google Docs. The most straightforward approach is to write all the docs manually. At Doist, for API drafts we used Dropbox Papers and shared them across the team. The advantage of this approach is that you can API-first without any code yet, iterate quickly, don't need to install anything or convert between the formats, and can quickly gather the feedback in-place.

On the flip side, this approach gets unwieldy quite fast. You end up with multiple Dropbox papers laying around, with different formats, and besides, these documents get out of sync with the actual API quite fast. While convenient, I wouldn't recommend Dropbox Paper or Google Docs for anything but the draft versions of the API.

API draft in a Dropbox Paper

Slate and friends. The next step towards a more controlled evolution of the API documentation is a tool like Slate and the API documentation in git. It's still text that you need to write manually in Markdown, but Slate provides a structure, a slick renderer, and lets you publish your API as an independent website.

Todoist API documentation is made with Slate

One problem with Slate is that it couples the code, the UI, and your documentation in a single repository. Their way of installing the tool is cloning the entire repo and replacing their sample content with yours. There is no simple way to upgrade your application to the new version of the API. You can clone again and copy your files if your only changes are only on the documentation side, but if you found yourself tweaking the UI, the upgrade becomes more challenging.

Another problem, and arguably, a much bigger one, is that it takes significant efforts and diligence to keep the documentation and the actual API in sync. If someone adds a new field to the object, they need to remember to update the documentation accordingly. With Slate, there's not much you can do beyond remembering, but later we'll see how we can address this issue by generating documentation from the code.

OpenAPI specification

Who cares about WYSIWYG editors or markdown when you write YAML! Everybody loves YAML, and the OpenAPI ecosystem lets you take advantage of your passion. Let's talk more about OpenAPI and what it brings to the table.

Everybody loves yaml. Source

OpenAPI in 2021 should be considered as a standard de-facto for API specification language. The #1 reason to adhere to OpenAPI beyond pure love to YAML, is the tooling that lets you write, validate, test, mock, document your API, render it with a website, generate API specifications from code, and generate API clients from specifications.

OpenAPI editors

There are two ways to write OpenAPI specifications. The most straightforward approach is to write the specification manually. Coupled with OpenAPI mocking services, it's also the fastest way to unblock your client-side developers and let them work with the API that doesn't exist yet.

To get the feeling of how OpenAPI looks like, you can start with an online editor at editor.swagger.io.

Online swagger editor

I wouldn't recommend using an online editor for anything serious, though. Fortunately, you can have the same functionality in your IDE or editor with an extension.

When I'm in VScode, I use an OpenAPI Editor from 42Crunch.

OpenAPI editor in VSCode

In PyCharm, I use an OpenAPI Specifications plugin from JetBrains. The plugin from 42Crunch is also available if you prefer it.

OpenAPI editor from JetBrains in PyCharm

OpenAPI renderers

OpenAPI renderers take your YAML specification and turn it into a website with nicely formatted documentation.

Swagger UI is the most popular renderer. It goes even further than generating browsable documentation and lets you play with your API right from the browser.

Swagger UI is purely client-side. It runs a JavaScript code that reads the specification from a remote URL, parses it, and converts it to the interface.

If you installed an IDE extension, you don't need to install anything else. All extensions come with a command opening the Swagger UI right in the editor window: that's what you've just seen in the previous screenshots. If you want to install it, though, you can find all the different options in their installation instructions.

If you have Docker installed locally, the easiest way to get started with a separate Swagger UI is to run it with a single command:

docker run -p 8080:8080 swaggerapi/swagger-ui

The Swagger UI will be available at http://localhost:8080.

ReDoc, an alternative to Swagger UI. It has a slick three-column interface, similar to Slate. The API looks cool, but an API playground is provided only in their paid hosted version.

ReDoc supports extra fields to your attributes, so-called (vendor extensions)[https://github.com/Redocly/redoc#swagger-vendor-extensions]. These fields provide more options to control how to render the documentation. For example, you can add a key x-codeSamples to your request specs and provide code samples in different programming languages. If you want something similar to Slate but generated automatically, that's probably the closest you can get.

As with Swagger UI, Docker is the fastest way to run ReDoc:

docker run -p 8080:80 redocly/redoc

ReDoc will be available at http://localhost:8080.

ReDoc serving a file from localhost

Generate OpenAPI from the code

Some frameworks can generate OpenAPI specs for you. You can stop worrying about the divergence between your specification and the implementation. The spec is built from the code, and it's always up-to-date!

FastAPI is the framework that comes to mind first. It thinks, lives, and breathes OpenAPI and lets you express quite advanced OpenAPI constructs with Python.

Django-REST-Framework is a framework that can do everything for your API. With this saying, when using it, I found that if you don't use model serializers, it's close to impossible to write a decent API specification from the code. Eventually, I found that writing a plain YAML file from scratch was faster and easier. This way or another, here's the schema API guide, and here's the guideline on how to render the documentation.

apispec is not a framework, but a library that provides a Pythonic interface to OpenAPI constructs. It has multiple integrations with different tools and frameworks, including Flask, Pyramid, aiohttp, and Falcon. The list of integrations is available on the ecosystem page.

Internal documentation

What to write. While the target audience of the API documentation is client-side developers, the primary audience of the internal documentation is the developers of the service itself.

There, we outline things like installation and configuration instructions, architectural choices, or guiding principles. Internal documentation can help navigate the code, deploy things to production, add a new dependency, apply a database migration, etc.

Where to store. I prefer to keep the internal service documentation in the same repository as the service itself in a separate directory, like docs. The closer the documentation is to the code, the less the chance to get it out of sync with the implementation. Don't overthink it. Something as simple as a bunch of markdown files in the directory can work. Other developers can browse the documentation right from their editor or use the GitHub interface.

Rendering the internal documentation

The next step in streamlining the experience with the documentation is to turn this documentation into a standalone website. Here, any static site generator can help, but Python world provides a few "traditional" options that are suited the best for the documentation and integrated with other tools and services.

Sphinx is a standard de-facto in the Python world. Powerful, but a bit on the challenging side to master. With this saying, most of the projects with extensive documentation prefer Sphinx over alternatives.

MkDocs tries to make writing documentation a delightful process. Unlike Sphinx, MkDocs uses Markdown under the hood. The ecosystem of Markdown is rich and spans well beyond the scope of Python. Plugins, editors, formatters, converters, everything is at your service. MkDocs configuration is straightforward. In its simplest form, it's a single YAML file with the project name and a list of pages.

# File mkdocs.yml
site_name: My Project
nav:
  - Home: index.md
  - About: about.md

My choice for Python project documentation is MkDocs with the mkdocs-material theme.

Other renderers

The beauty of internal documentation is that you're not limited to doc-specific frameworks; you can use any static site generator. Many of them have themes specifically suitable for documentation websites. I provide below some examples.

Pelican, the #1 static site generator written in Python.

Hugo is a popular Golang-based framework for building sites. Hugo runs this blog. Documentation templates.

Docusaurus is a static site generator from Facebook. Their core focus is helping to get the documentation right and well, but they made it possible to build any website, as it is a React application.

Next.js takes another step further away from documentation / static site generators camp. It's not quite a static site generator, but rather a fully-fledged React-based framework that happens to be able to export pages to HTML. If you are a friend of JavaScript and React, it can be an option, but it's overkill as a starting point.

Serving the internal documentation

Anything that can serve a static website, can serve your documentation. If you don't want to make your documentation public, opt-in for a solution that can make your documentation private. Below I will provide some solutions that you should fit your needs.

Read the Docs is the standard de-facto for serving technical documentation, especially popular among Open Source projects. It supports Sphinx and MkDocs out of the box, supports multiple versions of the documentation and localized versions. The project readthedocs.com provides commercial support and serves both public and private documentation.

GitHub pages is the natural choice if you keep your projects and documentation on GitHub. Starting with the Pro or Team tariff plans, you can make your pages visible to your collaborators only.

Gitlab pages provides the same functionality, but unlike GitHub, private websites are available even on the free account. There is a Gitlab group pages with examples of projects sharing documentation, including mkdocs, Sphinx, among dozens of others.

netlify integrates well with CI/CD workflows and is free for public sites. The pro version lets creating password-protected sites, and the business plan provides Role-based access control.

In all three cases, you build your documentation within the CI/CD pipeline and build static pages to the server.

My stack

To wrap up the discussion about the documentation, I will share my preferences for the stack.

API specification format: OpenAPI.
OpenAPI spec editor: PyCharm.
To generate the API from the code: FastAPI.
To render the API: Swagger UI.
To write documentation: MkDocs.
To serve documentation: nothing (read from the editor) or Read the Docs.

If you like it, please share it and tag me. I'm @rdotpy. If you want to discuss it, drop me a line. My DMs are open.

Linter for Python architecture

Roman Imankulov — Tue, 23 Feb 2021 07:51:05 +0000

How do you enforce architecture for your Python and Django projects other than in code reviews or guidelines?

The goal of the architecture is to organize and constrain the dependencies between the components of your project. For example, in a regular Django project, views depend on models, and not vise versa. If you have utility functions, they shouldn't depend on any specificities of your project.

I am playing with import-linter, and it looks promising.

What import-linter does

In Python projects, we define dependencies by imports. If module A imports module B, then A depends on B. The import-linter lets you specify the rules that declaratively constrain that dependency flow. In a config file, you define so-called contracts. For example, one contract can say, "in my projects, models.py must not have any imports from the views.py."

[importlinter]
root_package = myproject

[importlinter:contract:models_views]
name = Models don't import views
type = forbidden
source_modules =
    myproject.views
forbidden_modules =
    myproject.models

If your models.py contains something like from . import views, the linter raises an error.

At the moment, there are three types of contracts:

Forbidden modules: checks that one set of modules are not imported by another set of modules.
Independence: checks that a set of modules do not depend on each other.
Layers: enforces a layered architecture, where higher layers may depend on lower layers, but not the other way around.

If you can't express your architecture with these contracts, or you find yourself writing too many similar rules, you can write yours, and it's straightforward.

How import-linter works

I love the elegance of the model behind the import-linter. To analyze the project, it builds an import graph. Your contract takes it and checks if any edges violate the contact rules. Is there an edge between module X and module Y? Are there any loops? Is there a path from X to Y, etc.?

Behind the scenes, it uses the library grimp that, in its turn, is based upon NetworkX.

Kind of summary

Things that I like:

There is no magic in the project, and the graph-based model is both transparent and powerful.
It's quite fast even on large projects.
It doesn't enforce defining the architecture for the entire application. On a legacy project, you can start small.
You can define your own architecture rules with a Python class, and it's straightforward.
Works with pre-commit.

Things that left me confused or slightly disappointed:

It looks like a limited set of contract types makes you write different contracts for something that logically should be one contract.
Documentation uses generic like "foo" or "project_one". Opinionated examples for a simple CRUD application or a typical Django or Flask project would help a lot.

I am still at the early stage of adoption but already added it to my Python cookiecutter. Let's see if it sticks.