DEV Community: Sergey Piskunov

N+1 variations of a Singleton in Python

Sergey Piskunov — Mon, 07 Aug 2023 18:08:55 +0000

Intro

There are cases when we need to have one and only one instance of a particular class in our program. We may want this instance to encapsulate state and logic, accessible from multiple locations within the application. Such scenarios are ideal for object that consume significant resources and serve as a central access point to specific resources. Examples include database connections, logging helpers, and application settings. The Singleton is the creational design pattern that shows us how to create such an object. Let's see how many ways we may implement a Singleton pattern (and its variations) in Python and try to analyze the pros and cons of each.

When discussing the concept of "one and only one instance" it is crucial to consider the specific boundaries of that uniqueness. These boundaries are commonly referred to as namespaces or scopes.

The logic behind the Singleton pattern includes the following:

Defining a namespace to hold an instance.
Upon an object creation request, check if an instance already exists within the chosen namespace. If it does, return the existing instance; otherwise, create a new instance and save it within that namespace.

In Python, namespaces can be defined at different levels, including a module, a function, or a class. Therefore, the actual implementations of this pattern will be based on these options, depending on where the unique instance is intended to be stored.

Let's assume we have the following class, which we need to turn into a Singleton:

# example.py

class Server:
    """This is a helper class providing a way to send requests 
    to some third-party service. We want to use it across 
    the whole application as the single access point to that 
    third-party service"""

    def __init__(self, hostname: str):
        # Let's assume that the initialization phase of this 
        # helper includes some hard work in creating the 
        # connection to the particular server. 
        # To avoid repeating this work, it holds that connection 
        # in the instance attribute for reusing in the future.
        # Let's keep our example as simple as this:
        self.connection = f"connection to the {hostname}"

        # Let's print a notification to indicate that our 
        # 'connection' was established by the particular 
        # Server instance
        print(f"{id(self)} connected to the {hostname}")

    def ping(self):
        # Instead of making real requests somewhere, 
        # let's just print a notification.
        print(f"{id(self)} sent request via {self.connection}")

In Python, objects are considered the same if they have the same identifier. However, we will not only compare the IDs but also check the actual state of our instances by calling instance.ping() and observing the actual server being requested.

Let's use the Python shell to run our examples.

>>> from example import Server
>>> srv = Server(hostname="test.server")
140262246251248 connected to the test.server
>>> srv.ping()
140262246251248 sent request via connection to the test.server

Notice that the helper class establishes the connection right after initialization. The log messages contain instance IDs, which indicate the exact instance that established the connection and performed the request.

Let's begin with the simplest way to convert our Server class into a Singleton.

Module-level variable

In Python, when a module is imported, its contents are executed only once, and subsequent imports of the same module will refer to the already loaded module object. This behavior ensures that the module's variables, functions, and classes are shared across different parts of the code, providing a singleton-like behavior.

# module_level_variable.py

class Server:
    """ The actual implementation is in the Intro section """

srv = Server(hostname="test.server")

Python Shell test:

>>> from module_level_variable import srv
140138551301488 connected to the test.server
>>> srv.ping()
140138551301488 sent request via connection to the test.server
>>> from module_level_variable import srv
>>> srv.ping()
140138551301488 sent request via connection to the test.server

Despite re-importing, we still have the same initialized instance sending requests already.

That is indeed the most straightforward implementation, where no modification of the target class is needed. Python itself provides accessibility and state persistence for free. However, it's worth noting that the connection occurs right after the class is first imported, which can be disadvantageous in some cases. We may prefer initializing the instance at a specific time rather than immediately after importing.

Now, let's explore how we can eliminate this initialization disadvantage.

Module-level variable with instance getter

We may initialize instances when needed using a module-level function responsible for returning either a new or already instantiated object from the global namespace.

# module_level_instance_getter.py

class Server:
    """ The actual implementation is in the Intro section """


instance = None


def get_instance(*args, **kwargs):
    global instance
    if instance is None:
        instance = Server(*args, **kwargs)
    return instance

Python Shell test:

>>> from module_level_instance_getter import get_instance
>>> srv = get_instance(hostname="test.server")
139961814546704 connected to the test.server
>>> srv.ping()
139961814546704 sent request via connection to the test.server
>>> srv2 = get_instance(hostname="test.server")
>>> srv2.ping()
139961814546704 sent request via connection to the test.server

We can update this instance getter to work with any class passed as a parameter. That will allow you to turn any class into a singleton by keeping them in a global mapping and storing created singletons in the global variable.

# module_level_multiple_instances_getter.py

class Server:
    """ The actual implementation is in the Intro section """


singletons = {}

def get_singleton(cls, *args, **kwargs):
    global singletons
    if cls not in singletons:
        singletons[cls] = cls(*args, **kwargs)
    return singletons[cls]

Python Shell test:

>>> from module_level_multiple_instances_getter import get_singleton, Server
>>> srv = get_singleton(Server, hostname="test.server")
139938106238832 connected to the test.server
>>> srv_2 = get_singleton(Server, hostname="test.server")
>>> srv.ping()
139938106238832 sent request via connection to the test.server
>>> srv_2.ping()
139938106238832 sent request via connection to the test.server

One may ask why not make the get_instance a class method. Let's see.

Class-method instance getter

We can use the class namespace as a scope containing the instance. Let's add the get_server class method to our helper class.

# class_method_getter.py

class Server:
    """ The actual implementation is in the Intro section """

    _instance = None

    @classmethod
    def get_instance(cls, *args, **kwargs):
        if cls._instance is None:
            cls._instance = cls(*args, **kwargs)
        return cls._instance

Python Shell test:

>>> from class_method_getter import Server
>>> srv = Server.get_instance(hostname="test.server")
139953073616112 connected to the test.server
>>> srv.ping()
139953073616112 sent request via connection to the test.server
>>> srv_2 = Server.get_instance(hostname="test.server")
>>> srv_2.ping()
139953073616112 sent request via connection to the test.server

Notice how easily we can break this (and the previous) implementation by omitting the get_instance call:

>>> srv_3 = Server.get_instance(hostname="test.server")
139953073616112 connected to the test.server
>>> srv_3.ping()
139953073616112 sent request via connection to the test.server
>>> srv_4 = Server(hostname="test.server")
139953073617168 connected to the test.server
>>> srv_4.ping()

There is a hack allowing us to create a single entrypoint for instance creation.

Module-level instance getter with a nested class

Python allows us to enclose the whole class definition within a particular function which may be a single entrypoint for getting the class instance.

# module_level_getter_nested_class.py

instance = None        

def get_instance(*args, **kwargs):

    class Server:
        """ The actual implementation is in the Intro section """

    global instance
    if instance is None:
        instance = Server(*args, **kwargs)
    return instance

Python Shell test:

>>> from module_level_getter_nested_class import get_instance
>>> srv = get_instance(hostname="test.server")
140111140043504 connected to the test.server
>>> srv.ping()
140111140043504 sent request via connection to the test.server
>>> srv_2 = get_instance(hostname="test.server")
>>> srv_2.ping()
140111140043504 sent request via connection to the test.server

While the solution works, it is not commonly used due to its inflexibility. Furthermore, the use of those getters is considered non-Pythonic. The tight coupling of Singleton-related logic with the target class appears to break the Single Responsibility Principle. Fortunately, there are better approaches to address these issues and improve the implementation of the Singleton pattern.

Class decorator

The multiparadigm nature of Python enables us to create class decorators that can encapsulate the Singleton-related behavior.

# class_decorator.py

def singleton(cls):
    _instance = None

    def wrapper(*args, **kwargs):
        nonlocal _instance
        if _instance is None:
            _instance = cls(*args, **kwargs)
        return _instance
    return wrapper

@singleton
class Server:
    """ The actual implementation is in the Intro section """

Python Shell test:

>>> from class_decorator import Server
>>> srv = Server(hostname="test.server")
140568956844784 connected to the test.server
>>> srv.ping()
140568956844784 sent request via connection to the test.server
>>> srv_2 = Server(hostname="test.server")
>>> srv_2.ping()
140568956844784 sent request via connection to the test.server

While we achieved cleaner, more readable, and untangled code, we may encounter issues with child classes. To ensure a more robust approach, let's explore other solutions that can effectively handle inheritance and child classes.

Base class

We can store the instance within a class variable and implement the Singleton-related logic in the __new__ method.

# base_class.py

class Singleton:
    _instance = None

    def __new__(cls, *args, **kwargs):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
        return cls._instance

class Server(Singleton):
    """ The actual implementation is in the Intro section """

Python Shell test:

>>> from base_class import Server
>>> srv = Server(hostname="test.server")
139871502953360 connected to the test.server
>>> srv.ping()
139871502953360 sent request via connection to the test.server
>>> srv_2 = Server(hostname="test.server")
139871502953360 connected to the test.server
>>> srv_2.ping()
139871502953360 sent request via connection to the test.server

The resulting subclassed Server class can be further subclassed if needed, and these subclasses will continue to act as singletons. However, this solution has an issue: the 'connected to the test.server' log message appears twice. This approach does not allow us to run initialization lazily, only during the first call. That may be an issue for resource-heavy initialization.
Indeed, there is room for improvement. Let's continue exploring more sophisticated approaches to implement the Singleton pattern.

Metaclass

We instantiate our Singleton class from the type in this approach, which is required to achieve proper metaclass behavior in Python. In the case of a metaclass, the __call__ method is called first, making it an excellent place to intercept instance creation.

# metaclass.py

class Singleton(type):
    _instance = None

    def __call__(cls, *args, **kwargs):
        if cls._instance is None:
            cls._instance = super().__call__(*args, **kwargs)
        return cls._instance

class Server(metaclass=Singleton):
    """ The actual implementation is in the Intro section """

Python Shell test:

>>> from metaclass import Server
>>> srv = Server(hostname="test.server")
140421527038288 connected to the test.server
>>> srv.ping()
140421527038288 sent request via connection to the test.server
>>> srv2 = Server(hostname="test.server")
>>> srv2.ping()
140421527038288 sent request via connection to the test.server

This solution looks like the most Pythonic way to implement a Singleton. It is readable, pluggable, works for subclassed objects, and may be a good choice if not for one but.

In the real application we may want to have connectors to multiple servers, which of them acting as independent singleton.

>>> from metaclass import Server
>>> srv = Server(hostname="test.server")
140654371744080 connected to the test.server
>>> srv.ping()
140654371744080 sent request via connection to the test.server
>>> srv_2 = Server(hostname="second.test.server")
>>> srv_2.ping()
140654371744080 sent request via connection to the test.server
>>> srv.ping()
140654371744080 sent request via connection to the test.server

Notice how the srv instance's connection gets "overridden" by the newly created instance. This occurs because of the primitive-type variable _instance, which can hold only one instance at a time. Let's see how we can deal with that.

Multiton

Multiton is a variation of the Singleton pattern where we can store multiple instances based on certain criteria. In our case, we can use a dictionary with a key consisting of the class name and class arguments to resolve the issue encountered in the previous section effectively.

# multiton.py

class Multiton(type):
    _instances = {}

    @classmethod
    def _generate_instance_key(cls, args, kwargs):
        # This implementation of a unique key may be sensitive 
        # to complex objects passed as parameters. Feel free to 
        # override this method for the target class to fit 
        # your specific use-case.
        return f"{cls}{args}{sorted(kwargs)}"

    def __call__(cls, *args, **kwargs):
        key = cls._generate_instance_key(args, kwargs)
        if key not in cls._instances:
            instance = super().__call__(*args, **kwargs)
            cls._instances[key] = instance
        return cls._instances[key]


class Server(metaclass=Multiton):
    """ The actual implementation is in the Intro section """

Python Shell test:

>>> from multiton import Server
>>> srv = Server(hostname="test.server")
139800681970704 connected to the test.server
>>> srv.ping()
139800681970704 sent request via connection to the test.server
>>> srv_2 = Server(hostname="test.second.server")
139800681969888 connected to the test.second.server
>>> srv_2.ping()
139800681969888 sent request via connection to the test.second.server
>>> srv.ping()
139800681970704 sent request via connection to the test.server

It appears that we have found the most effective implementation so far, but let's take a moment to consider a different approach.

Monostate

We have been trying to reuse the same instance all this time, but Alex Martelli notes that we should focus on the shared state and behavior rather than the shared identity. He proposed using the Monostate pattern instead, where there may be multiple instances, but they share the same __dict__ special method's contents.

# monostate.py

class Monostate(type):
    _shared_state = {}

    def __call__(cls, *args, **kwargs):
        obj = super().__call__(*args, **kwargs)
        obj.__dict__ = cls._shared_state
        return obj

class Server(metaclass=Monostate):
    """ The actual implementation is in the Intro section """

Alex actually calls it a nonpattern because of a lack of evidence of widespread usage. It is a bit less intuitive than the classic Singleton approach, and the actual implementation may differ according to the use case. However, I think there is room for such an approach as well.

Thread safety

So far, we have achieved a robust implementation of the Singleton (Multiton) pattern using a metaclass. However, it appears that our implementation may misbehave in a multithreaded context. When object creation occurs within code shared by multiple threads, there can be a race condition. One thread might start instantiating an object, and then another thread takes control. Since the object is not yet fully created, the second thread starts creating it as well. In the following test, this issue was reproduced with the help of the deliberately slowed-down __init__ method. That may be the case for a heavy initializing logic.

# thread_test.py

import threading
import time

class Server(metaclass=Monostate):
    def __init__(self):
        time.sleep(0.1)


result_set = set()

def create_instance():
    instance = Server()
    result_set.add(str(instance))


threads = [
    threading.Thread(target=create_instance)
    for _ in range(5)
]

for thread in threads:
    thread.start()

for thread in threads:
    thread.join()

print(f"Created {len(result_set)} instance(s):")
print('\n'.join(result_set))

Output:

Created 5 instance(s):
<__main__.Server object at 0x7f2f8d72b8b0>
<__main__.Server object at 0x7f2f8d72ba30>
<__main__.Server object at 0x7f2f8d72b7f0>
<__main__.Server object at 0x7f2f8d72b730>
<__main__.Server object at 0x7f2f8d72b970>

Depending on the use case, we should decide whether to make our instance unique within each thread or unique across all threads.

Thread confined Multiton

By utilizing this approach, we can associate the state with a particular thread. However, it's crucial to understand that this would no longer be a Singleton pattern at the global application level. Instead, each thread will have its unique instance of the object.

# thread_confined.py

import threading


class Multiton(type):
    _local = threading.local()

    @classmethod
    def _ensure_local_namespace(cls):
        if not hasattr(cls._local, "_instances"):
            cls._local._instances = {}

    @classmethod
    def _generate_instance_key(cls, args, kwargs):
        # This implementation of a unique key may be sensitive 
        # to complex objects passed as parameters. Feel free to 
        # override this method for the target class to fit 
        # your specific use-case.
        return f"{cls}{args}{sorted(kwargs)}"

    def __call__(cls, *args, **kwargs):
        cls._ensure_local_namespace()
        key = cls._generate_instance_key(*args, **kwargs)
        if key not in cls._local._instances:
            instance = super().__call__(*args, **kwargs)
            cls._local._instances[key] = instance
        return cls._local._instances[key]

Thread safe Multiton

To avoid race conditions, we must lock access to the shared _instances dictionary while an instance is being instantiated. That may be easily achieved with the Lock object from the Python standard library.

# thread_safe.py

import threading


class Multiton(type):
    _instances = {}
    _lock = threading.Lock()

    @classmethod
    def _generate_instance_key(cls, args, kwargs):
        # This implementation of a unique key may be sensitive 
        # to complex objects passed as parameters. Feel free to 
        # override this method for the target class to fit 
        # your specific use-case.
        return f"{cls}{args}{sorted(kwargs)}"

    def __call__(cls, *args, **kwargs):
        instance_key = cls._generate_instance_key(*args, **kwargs)
        # Double-checked locking technique
        if instance_key not in cls._instances:
            with cls._lock:
                if instance_key not in cls._instances:
                    instance = super().__call__(*args, **kwargs)
                    cls._instances[instance_key] = instance
        return cls._instances[instance_key]

class Server(metaclass=Multiton):
    """ The actual implementation is in the Intro section """

Output:

Created 1 instance(s):
<__main__.Server object at 0x7f0c6e16b730>

One may ask why if instance_key not in cls._instances is called twice. Technically, it would be sufficient to make that check once after the lock is acquired. However, the case when this check results in True will occur for the very first call only. All subsequent calls will result in False indicating that the lock was acquired unnecessarily. Acquiring locks unnecessarily in a class/method can lead to slow code that is hard to identify, so it's crucial to acquire locks only when necessary to avoid performance issues.

Epilogue

As we are aware, there is no universal solution, and nearly every approach has its limitations and disadvantages. Some articles even label the Singleton as an antipattern. Among the concerns, such as "difficult to comprehend," "challenging to read," and "violates the Single Responsibility Principle," one particular concern holds significant meaning. The constraint of utilizing only a single instance may complicate testing in scenarios where we genuinely require multiple tests against various states of that instance. However, this problem can be alleviated by structuring the application in a way that allows external injection of the instance. During regular operations, the block of code can utilize the Server(metaclass=Multiton) instance, yet during tests, the Server() instance can be externally injected.

We must also bear in mind that, in certain cases, the namespace housing unique instances could be overwritten by invoking importlib.reload(module_name). This action would result in the re-instantiation of instances.

Additionally, depending on the specific implementation, extra steps might be necessary to dispose of instantiated singletons. Even if we no longer require those instances, the scope retaining references to our singletons could hinder the garbage collector from eliminating them.

In the context of simple projects or prototypes, one might resort to module-level variables for maintaining unique instances. Nevertheless, a metaclass approach is preferable for more intricate scenarios, particularly when lazy evaluation is imperative. When crafting a common library without a clear vision of its usage, opting for a thread-safe approach is advisable.

Anyway, despite of added complexity and risks, it is better to understand the idea behind the Singleton pattern. Its ability to provide a single access point to some resource with lazy instantiation outweighs its drawbacks, at least for languages like Python.

Bibliography

Model Thinking course from the software engineer’s perspective

Sergey Piskunov — Sat, 04 Mar 2023 20:30:28 +0000

I want to give a review on the "Model Thinking" course, which I have recently completed using Coursera platform. Although this 8-week online course from Michigan University aims to make us better citizens of the world and decision-makers in general, I will share my thoughts about how that course's material may be applied to software engineering. I'm going to discuss several course sections that were the most relevant for me.

Why do we model?

As the first quote from this course's material, I'd like to put this one: "Models are the new Lingua Franca which allow people from different domains to communicate their ideas simply and clearly.". We all know from Domain Driven Design how crucial for a software project is to have a ubiquitous language between engineers and non-technical colleagues. Enriching our documentation with models makes communication between team members even more effective.

"We don't want to let the models tell us what to do. We want the models to help us make better choices. Smart people use both models and experience and not just blindly use models.". Models are not a silver bullet, though; we should understand their applications and restrictions, making them just yet another tool but not a replacement for common sense and experience.

One of the most widespread examples of using models in a software engineer's routine is explaining the complexity of algorithms. We usually do not say "The Bubble Sort" is slow, but the "Merge Sort" is faster because each person has their own understanding of fast and slow. We say the "Bubble Sort" has O(n^2) time complexity, and the "Merge Sort" has the O(n log n). So during decision-making, we operate not with our emotions and feelings about the speed of algorithms, but we use their mathematical models. That gives us a fair performance comparison and understanding of possible applications of those algorithms.

Sorting and peer effects.

Here we can find an exciting model of social behavior, which may explain, for example, how we adopt new technologies within the industry (or new ideas within a team). "Granovetter's Model - The tail wags the dog. The people at the tail of the normal distribution are extremists, who sometimes drive what happens because of the chain reaction of thresholds of other people. Collective action is more likely if there are lower thresholds (the "tail" which leads to a cascade effect of "wagging the dog") and there is more variation in thresholds (which increases the probability of the chain reaction).".

Standing Ovation Model - Granovetter's model, in which the cascade effect is amplified by "expert" opinion, the opinion of a friend, a celebrity, or other influencers.

Aggregation.

This section explains a lot of concepts, which a regular software developer may face in a day-to-day work, like the Bell Curve, Standard Deviation, and Mean. We may often see those terms in performance-testing reports or monitoring dashboards.

Decision models.

I have found this section useful as it gives us decision-making tools to deal with complex subjects with multiple characteristics - "Multi-Criterion Choice Models". Imagine you must choose a framework or library among a dozen options. While each candidate has a dozen of aspects. A comparison table may be the right tool for that, but you may need to learn how to make the results of such comparison even more expressive using weights and distances. Also, one of the essential things in this section is a general recap on the Probabilistic Theory, Decision Trees — a crucial grounding for every engineer.

Models of people. Thinking Electrons.

This section may not give us concrete technical advice, but it sheds light on how people behave. The authors show examples of people's rational, irrational, and rule-based actions and different types of biases. That may help us to understand people better and thus to be better team players.

Linear models.

The Linear Models section gives us many insights not only about reading the output of some types of statistical analysis (for example, Regression Output), but about ways to explain the raw data you have. For example, you may find that the more execution threads do you add, the fastest your code becomes. You may assume that the speed of your code linearly depends on the amount of resources you have, and you may plan your server's capacities according to that assumption. But one of the crucial caveats here is to be careful about extrapolating your linear data outside the data range. At some point, you may notice that adding more X does not give you more Y.

"Correlation is not causation" is also important to remember during any historical data analysis and attempts to predict the future using that data. The authors give you a wise idea that to optimize your system even more, you might need a so-called "New Reality". To have better results, you are just building the new system instead of optimizing an old one. "The New Reality - is moving outside the box, explained by linear models and coefficients to the new system. Instead of endless optimizing of the SQL queries, try to re-align your data so that it may fit into the NoSQL paradigm."

Economic growth.

What kind of economic-related knowledge may we reuse in our software engineering paths? Our engineering paths consist not only of technologies we know now but of how we evolve as professionals and grow. That growth has much more in common with economic growth than you may think. Assuming that our skill set is a kind of our personal GDP, we may apply the economic growth models to ourselves! The basic economic growth model suggests that Sustainable growth requires innovation. Let me quote the excerpt: "You can't just do more. At some point, the rate at which things fall off and the rate at which things increase is just going to even out. If you look at the data on what makes for really successful people, people who are very successful in their careers continue to learn.". That is especially important in the age of replacing some jobs with Machine Learning systems and other types of automation.

"Growth requires creative destruction". Another example is the evolution of programming languages. I'm not going to start the Holy War related to the specific languages, but I want to say, that there is a mathematical model, showing us that, at some point, it is better to create (or adopt) the new fancy programming language instead of cluttering a 20-year-old one with new concepts, while also trying not to break backward compatibility and its idiomatic style.

Diversity and innovation.

This section describes a thing we face almost every day at our work. The "No Free Lunch Theorem". I could not find words better than the author's explanation - "No Free Lunch Theorem tells us that no single heuristic is always effective. A heuristic that works great for one problem class may be ineffective for another problem type... All algorithms that search the same number of points with the goal of locating the maximum value of a function defined on a finite set perform exactly the same when averaged over all possible functions." That may come in handy during the design and optimization of algorithms/business logic. This section has some additional problem-solving tip as well - "When we go about solving problems, the first thing we do is we encode them. We have some representation of the problem. That representation determines how hard the problem will be.". That is why it is not a good idea to start writing code immediately after reading the task description. It's better to invest decent time in design and preparation, pseudo-coding, and leave the coding itself as the less important routine stage. We may find in this section a mathematical proof of why teams are better at problem-solving than individuals - because of diversity, perspectives, heuristics, and recombination of those new ideas. You'll always uphold a command work after hearing that explanation.

Networks.

This is again something one may have studied in college, but worth repeating. Networks and graphs are literally everywhere, from the computer network itself and distributed systems within those networks to the object inheritance and dependency graphs in the source code. The McCabe code complexity analysis, graph databases, and graph query languages are also based on the theory explained in this section.

Conclusion.

I have reviewed a little more than half of the sections of the "Model Thinking" course—only those I have found the most applicable to my personal path of a software engineer. However, your list of favorite concepts may differ because the knowledge you'll get after taking this course is rather a meta-knowledge that may be applicable to a wide variety of cases in your life. So, I encourage not only software engineers, but all to take that course.

Links

Coursera "Model Thinking" course

Source code structure checklist

Sergey Piskunov — Mon, 02 Jan 2023 18:50:02 +0000

This is my checklist for a source code structure. Of course, there is no silver bullet and different domains/technologies/languages have their specific requirements and limitations. So, here is the language-agnostic and domain-agnostic set of questions I usually ask myself while assessing the existing code base and before creating a new one. I do not try to get all of those boxes checked altogether, but I should understand why any of them is unchecked and determine whether it is a “red flag” for the particular project.

1. Does the source code tell you “what does the service do” instead of “which framework or library it is based on”?

This will check how easy it is to open the code and quickly find a place for a new feature, make a bug fix or check how the particular part of the business logic works.

We are used to read the code much more often than to write it. So we need to squeeze the maximum out of our programming language’s expressiveness. I’m talking not only about the proper names for variables, classes etc., but about the structure of files and directories as well. We often build our projects on top of a certain framework which imposes a structure to the entire project. Switching to the new framework often leads to re-writing the whole code base. Steve McConnell in his book “Code Complete” describes this effect as “writing in programming language” opposed to “writing with help of the programming language”. In that book, you may find a very profound explanation of how to write a program for people, not for computers.

Try to make the file tree self-explanatory, and try to extend generic terms, like utils, models, validators, and so on. As for me, is better to see the utils directory with a single file per each helper-object instead of the single utils file with all helpers inside and with dozens of lines of code. I am also not a big fan of shortened entity names. I’d rather spend yet another microsecond reading a complex name and catching the concept, rather than reading a shortened but ambiguous one. All of those points sound obvious, but save hours of time in the long run.

2. Can you easily run the project or its test suite locally with a simple command?

This will check how easy it is to pick up the project by a team members (especially newcomers).

We usually do not want to spend an hour in order to find all possible startup options and distract our colleagues by asking corresponding questions. Some kind of command automation utility may help.

I use good old Makefile where I keep all needed complex commands associated with shortcuts like: image, deploy, bootstrap, test, clean, etc. I also add a “help section” to it, which prints a brief explanation of all commands and a couple of examples so that any team member can just type make help in the command line and recall all details. It is much easier to run make image test clean or something like that instead of typing long docker or docker-compose related commands and try to remember commands specific to each project. I am also happy when the CI/CD config may run those Makefile commands. That helps to keep all possible ways to run and manage application in a single place.

3. Can you add support for the "--dry-run" flag without a significant redesign of the project itself and its tests?

This will check whether your business logic is tightly coupled with the database and external services.

By this “dry run” mode I mean the ability to run the service with isolated or mocked external dependencies. We need this not only to perform manual testing and debugging but also to increase testability and extensibility of the project. This topic is described in details in the “Architecture Patterns with Python” book by Harry Percival and Bob Gregory. As written in the book - "patching out the dependency you’re using makes it possible to unit test the code, but it does nothing to improve the design. For that, you’ll need to introduce abstractions". As for unit testing, there is two most popular approaches for testing external calls: London-school TDD using Mocks (patching the call in place) and the Classic Style using Fakes (Fakes are working implementations of the thing they’re replacing). Dynamic languages such as Python allow you to simplify the structure and just monkey-patch all needed calls. But, strictly speaking, mocking is a code smell and may lead to complicated tests.

My choice is - Mocks for small or short-term solutions and Fakes for the medium/big and long-lasting ones.

4. Can you add support for the new version of the API/Model without a significant redesign of the project itself and its tests?

This check is like the previous one but examines extensibility of the business logic-related parts of the application.

Not the severe point, but it is better to think about the place for different versions of the same entities ahead of time and about how they may live together within the particular code base.

In my personal experience, the initial version of the API or business logic entities seems the most proper one and stable, but the stable requirements are a myth. The task to add a new API version usually comes when the service is already deployed to the production and making such a change involves lots of steps with further refactoring. As proposed in the famous tweet: “Make the change easy (warning: this may be hard), then make the easy change.”

5. Can you add support for the new interface, e.g. CLI without a significant redesign of the project itself and its tests?

This will check whether your business logic is tightly coupled with the representation layer.

If so, it would be much harder to add a new external interface or to build a healthy testing-pyramid. Loose coupling of the business logic with the representation layers (actual HTTP endpoints in case of a web application) helps us to simplify unit tests and to prepare to switching to another framework. That switch might look like an impossible case, but in my opinion, it is often almost impossible to painlessly switch to a better framework just because of the tightly coupled code. Not because we do not want to switch to the better framework. It is usually not just a major refactoring, it means rewriting the major portion of the code (and tests), which is not desirable at all.

I’m going to suggest here any kind of approach based on the inversion of control. The Hexagonal architecture or any of its variants should work. Such kind of code structure makes it easy to separate business logic from representation and persistence layers or to add new layers as the Web interface besides existing CLI Interface and vice versa. Although using such approaches for small or short-term solutions may be an overkill.

6. Are there any unused/unnecessary dependencies in the source code?

This will check whether the code base depends on things it doesn't use.

Although we mostly use open-sourced dependencies, the risk of having related security issues is still there. Also, each added dependency does not decrease the compile time/startup time/binary size (depending on the particular programming language) of the service. For the same reasons, it is better not to keep testing-related dependencies in a production image/binary. There are also cases when we add dependency on the whole package if we actually need one or two helper functions from it. In most cases, it is better to copy-paste those helper functions into your code base and get rid of the extra dependency.

7. Does the source code have a “gentleman set” of root-level helper files?

This will check, I'd say, the general "maturity" of the repository.

There is a set of commonly used files, which doesn't influence the functionality itself but helps us to maintain the codebase in a unified manner.
Such files are: .gitignore, .dockerignore, .editorconfig, CHANGELOG.md, CONTRIBUTING.md, explicit configs for static code analyzers? And by the way... the actual version of the Readme file. No joke. Let’s end the practice when the main contributors to the readme file are newcomers, who found that steps described in the Readme file do not work anymore...

Afterword

Note that I deliberately skipped any checks for unit-tests as they deserve a separate checklist.

One of the main goals of such a strict inspection is to increase the readability and maintainability of the project. This is extremely important for long-lasting projects. In most cases, we shouldn’t bother a lot about how easy it will be for a processor to parse our code base and create a nice abstract syntax tree from it. But we should bother about how much time it will take for a newcomer/your colleague/yourself to find the right place for a new feature or a bug fix. Let’s be honest, human beings are not so good at keeping a lot of objects and abstractions in memory simultaneously. We shouldn’t neglect our natural peculiarities and help ourselves in any ways possible.

Bibliography:

McConnell, Steve. 2004. Code Complete, 2nd Edition
Percival, Harry and Gregory, Bob. 2020. Architecture Patterns with Python