DEV Community: João M.C. Teixeira

Separating business and administrative logic - example 1

João M.C. Teixeira — Tue, 28 Nov 2023 23:06:31 +0000

Hi,

(I am re-posting here a post I made in 2019 on my blog)

A quick view on business and administrative logic separation.

A dummy example where we use decorators to check if the input path exists before assigning it to the class attribute. Context managers could accomplish the same, but with the decorator, checking takes place before even executing the function.

def check_folder_exists(func):
    """
    Decorator for class property setters
    """
    @wraps(func)
    def wrapper(self, folder):

        if not Path(folder).exists():
            raise FileNotFoundError(f'Path not found: {folder}')

        else:
            return func(self, folder)
    return wrapper


class Parameters:
    def __init__(self, **kwargs):
        self.kwargs = kwargs

    @property
    def db_folder(self):
        return self._db_folder

    @db_folder.setter
    @check_folder_exists
    def db_folder(self, folder):
        self._db_folder = folder

Below is another snippet from my code where I use a combination of three contexts to handle exceptions hierarchically.

for step in config_reader.steps:

    with GCTXT.abort_if_exception(), \
            GCTXT.query_upon_error(XCPTNS.ConfigReadingException), \
            GCTXT.abort_if_critical_exceptions(CONFREADER.critical_errors):

        step()

Do you find this useful for your projects? Let me know.

Cheers,

Type hints in Python: to be or not to be

João M.C. Teixeira — Tue, 04 Apr 2023 09:50:25 +0000

Hi everyone,

Using type hints in Python is an active controversy nowadays. Are they good, or are they not? Should we use them, or should we not?

In one of the projects I was lead developer, someone asked today about adopting type hints. I will share with you my opinion. It may help you to decide whether to use type hints or not. GitHub discussion below:

https://github.com/haddocking/haddock3/discussions/637

Cheers

Simple password generator in Python

João M.C. Teixeira — Thu, 03 Mar 2022 16:39:51 +0000

Hi everyone,

I published today a very simple implementation of a password generator using Python through the command-line. Yet, there are some interesting options.

We can discuss about the implementation details.

https://github.com/joaomcteixeira/passgen

Modular, extensible, maintainable, and testable functional strategy pattern - part 1

João M.C. Teixeira — Fri, 10 Sep 2021 15:09:54 +0000

The Strategy Pattern is likely the most used design pattern. Who said it? Well, I said it. The strategy pattern concept is simple. At a given moment, the execution workflow must take one from different paths depending on the evaluation of the current state, being state a variable or a group of variables.

There are countless situations to apply the strategy pattern; however, we will focus on analysing a single variable to control the execution flow.

We have variable var1 and, depending on the boolean result of different evaluations, var1 is processed differently.

var1 is int -> process it as an int
var1 is long string -> process the long string
var1 is short string -> process the short string

What could be the most extensible, maintainable, modular, and testable way to implement a strategy pattern here? Let's see different options:

1) if-statements are the most easily grasping strategy, likely used by all novice programmers but also advanced ones (when coding for very straightforward cases 😉).

if isinstance(var1, int):
     # do some logic here
elif isinstance(var1, str) and len(var1) > 100:
     # do other logic here
elif isinstance(var1, str) and len(var1) <= 100:
     # do yet another logic here
else:
     # everything before was false
     # do this other thing

The case presented above might be a feasible quick implementation for some simple cases. But here, we imagine all operations are long and complex because we are developing an extensive program ☺️. So, what are the problems of a direct if/elif/else implementation as the one presented above?

if-statements will become too lengthy and maybe nested as the number of options grows.
Also, having the business logic directly under the if clauses render that code challenging to test. Surely, impossible to test on its own.
Adding more options will just make the code less readable, especially if implementations are lengthy.
And, the code is not modular because you cannot use those logic blocks anywhere else as they are hardcoded directly under the if-clauses.

How can we improve from this situation?

The first step you can do to improve the code's modularity and testability is to encapsulate each logical process in its separate function:

if isinstance(var1, int):
     process_is_int(var1)
elif isinstance(var1, str) and len(var1) > 100:
     process_is_long_string(var1)
elif isinstance(var1, str) and len(var1) <= 100:
     process_is_short_string(var1)
else:
     do_else_logic(var1)

By doing this, you will be able to test the different functions in their dedicated unit-tests. Also, your code becomes more modular because you can re-use those functions anywhere else; even have them as a public, well-documented library API.

The second step towards improving the quality of your code is to transform your asserting statements into, also, individual functions.

if is_int(var1):
     process_is_int(var1)
elif is_long_string(var1):
     process_is_long_string(var1)
elif is_short_string(var1):
     process_is_short_string(var1)
else:
     do_else_logic(var1)

Notice how you can now use the evaluation functions (is_int, is_long_string, and is_short_string) anywhere else in the code and test them properly. Also, suppose you want to add additional features to those functions in a later stage, for example, new assertions or logging. In that case, you can do it easily without the need to modify the if/elif block.

(side comment) Once you reach this point of understanding, your brain will likely spark the light of using a dictionary to control the flow. Probably you have read about that strategy somewhere. I have used it also. Using dictionaries to control flow is productive in some cases, but its use won't excel in this particular case. So I will discuss using dictionaries to control flow in other posts of my series (keep an eye on them) and won't talk about them here. (end of side comment)

Finally, there is a way that out-stands the previous two, both if-else blocks and that crazy dictionary strategy I told you about but didn't explain 😉.

The implementation design that best suits our example (in my experience and opinion) is using a for loop. Yes, a for-loop. How?

First, we create a list of tuples where each tuple has two values: 1) the evaluation function (boolean logic) and 2) the business logic. The for loop will loop through the list and operate on the first function that evaluates to True.

# note how our logic is getting defined in separate functions
# as we evolve from if-statements to this pipeline-like method
options = [
    # (if_this_is_true, do_this).
    (is_int, process_is_int),
    (is_long_string, process_is_long_string),
    (is_short_string, process_is_short_string),
    (lambda x: True, process_the_else_logic),
    ]

for evaluation, function in options:
    if evaluation(var1):
        function(var1)
        # the loop does not need to continue
        # after a function is executed
        break

In case the else logic does not receive var1 as an argument, you can move it to the else method of the for-loop:

options = [
    (is_int, process_is_int),
    (is_long_string, process_is_long_string),
    (is_short_string, process_is_short_string),
    ]

for evaluation, function in options:
    if evaluation(var1):
        function(var1)
        break
else:
    process_the_else_logic()

Benefits of this last approach? Several:

The for-loop executes the boolean logic only once and only if the previous evaluated to False, so there are no losses against the if-else block.
You can easily change the order of the process by changing the order of the tuples in the list without the need to be moving large pieces of code around.
You can add/remove new functionalities by adding/removing tuples in the list without changing the engine.
It is effortless to read and understand what the code will do. Understanding how the code will do it is another story. For that you would need to read into each function. Congratulations, you have separated the what from the how.

Honestly, I see no flaws in this method. I think it really is extensible, maintainable, modular, and testable. If you think differently or know other possible examples, let me know in the comments. One can apply the strategy pattern to countless situations. This is one of them. Likely, you have heard of the strategy pattern when implementing OOP, but life is not always about OOP despite what they told you 😛

But this is not the whole story. For example, many times, we use if-statements to evaluate the state of multiple variables. And, of course, that is also possible using the for loop strategy presented here. But asserting the state of multiple variables is a more complex case that would require the use of functools.partial, so we leave that discussion for another post.

I hope you enjoyed this quick yet profound discussion, which I am sure will help you design larger and more maintainable software packages. Despite not being skilled in other languages besides Python, I trust you can apply this same (or similar) idea in other realms of programming.

Cheers,

Express your intentions in your coding - attributes that are not

João M.C. Teixeira — Sat, 28 Aug 2021 10:13:58 +0000

In the previous post of this series, I suggested prefixing the underscore _ to variables intended to be short-lived. Here I will discuss another example where variable names lie on their intentions.

Pay attention: people are addicted to the self.

If you are coding a class, you have some logic in a method, and you are defining a variable within that method, do not use self. if that variable has no usage outside that method! 😌

When you assign a variable to self. you are telling the readers of your code that variable (better said, that attribute) has another purpose somewhere else in the execution workflow. Maybe it is even part of the public API. Therefore, someone editing your method will automatically trigger the alarms and search the codebase for a place where that attribute appears before refactoring or recoding your method. The maintainer of your code will get a bit stressed if she/he finds no place anywhere else where that attribute is used. What is happening here?

So, be mindful and express your intentions in your coding. That is, if you define a variable inside a class method, don't use self. if the scope of that variable does not go beyond that method. Instead, use self. indeed if that variable is an actual attribute part of the private or public API of the class.

class MegaClass:

    def method_1(self, arg1, arg2):
        """Write a docstring ;-)"""
        var1 = # some logic with arg1. `var1` dies after the method
        self.var2 = # some logic with var1 and arg2
        # self.var2 is an actual attribute of MegaClass part of its API.

Hope this example is clear and you can use it to better express your intentions in your coding 😄

Express your intentions in your coding - short lived variables

João M.C. Teixeira — Fri, 27 Aug 2021 16:51:20 +0000

Short Friday post. Let's talk a bit about implementation intentions and how you reflect those in your code.

The more I code, and the more I review others code, some little things start to "stress" me out a bit 😄 but it's positive stress, I've to say.

Have you considered how others read and understand the scope of your variables? More and more, I try to express my intentions when naming variables. And no, I am not talking about long names or short cryptic names. Here, I am talking about the time and the length (scope) a variable will have in the execution/implementation. Let's see. Imagine the following block of code:

var1 = # some logic

# some logic defined in 2 or 3 lines 
# using var1

# the code continues on and var1 is never used again.

Another more rigorous example could be:

var1 = # some long-line logic about 80 chars
# one shorter line using var1

# the code continues on and var1 is never used again.

How do you know that var1 is not used anywhere else in the implementation when you read these lines? You don't (until you search for it and find it's not present anywhere else). So here comes the question: How can we inform the reader that var1 is used only in the immediate scope of the following lines?

Here's what I do: prefix the variable name with _. Like this, when reading _var1, we know its scope will end in the subsequent short implementation. Of course, you can argue we could use _ only instead of _var1. That is true, and you are right. But sometimes, it is nice to name variables to give them a purpose. Also because the pure underscore _ is most often used for not used variables.

You can also use _var1 to temporarily store the result of some logic to facilitate the reading in the following lines. Here's the example I used in my previous post, where I use _tmp to simplify the next logic step.

# reduce the long call to a short temporary variable
_tmp = someobj.somemethod(run["run_dir"])
somejob = Path(
    rundir,
    f'{fileroot}_{_tmp}_{somevar}w',
    )

That's all for this Friday short post. Let me know your comments,
Cheers,

Make readable string formatting

João M.C. Teixeira — Thu, 19 Aug 2021 09:31:05 +0000

I run into this example some days ago while reading someone's else code (variable names were modified to some... or var...):

somejob = "{}{}{}{}{}{}{}{}".format(
    rundir,
    "/",
    fileroot,
    "_",
    someobj.somemethod(run["run_dir"]),
    "_",
    str(some_var),
    "w"
    )

What is wrong with this example?

First, can you actually understand the final shape of the somejob string? For me it is just utterly cryptic. How many {} are there?

So, what is wrong with this case and how can we improve it?

The problem with this string formatting example is that constants are being passed as variables, hiding the shape of the string. It becomes much easier to read if the developer defines the constant sub-strings in the main string:

somejob = "{}/{}_{}_{}w".format(
    rundir,
    fileroot,
    someobj.somemethod(run["run_dir"]),
    str(somevar),
    )

Still, because in this particular case we are using paths, I would go further and advocate for using pathlib.Path instead of the hardcoded /:

# reduce the long call to a short temporary variable
_tmp = someobj.somemethod(run["run_dir"])
somejob = Path(
    rundir,
    f'{fileroot}_{_tmp}_{somevar}w',
    )

I truly believe these modifications improve code reading and facilitates the life of maintainers. What do you think? Does the last example reads better than the first one?

Cheers,

I write about...

João M.C. Teixeira — Wed, 18 Aug 2021 14:49:05 +0000

I like to write about and discuss Python implementations, designs, strategies, patterns, and alike. I don't intend to compose tutorials nor explain the basic concepts. Instead, I want to engage with you in fruitful discussions and explore more advanced topics. I currently have two post series here at DEV.to:

In the first series, I comment on Pythonic tips, tricks, or behaviours I encounter in daily projects. I dedicate the second series to long, highly contextualized discussions and conceptualizations on specific topics - most likely software architecture. I don't write in any defined order; I post as things come to mind.

I used to post on another blog - Pythonic Thoughts Snippets -, but I am migrating my programming writing activity entirely here to DEV.to.

Time needed to search in a list or set in Python

João M.C. Teixeira — Wed, 18 Aug 2021 11:10:31 +0000

The time it takes to search an item in a list depends on the size of the list and the position of the item in it. While for set the time cost is maintained. Here, we do not consider the time needed to create the lists and sets.

l100 = list(range(100))
s100 = set(range(100))
l100k = list(range(100_000))
s100k = set(range(100_000))

%%timeit
100 in l100

895 ns ± 31.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%%timeit
100 in s100

25.8 ns ± 1.56 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

%%timeit
100_000 in l100k

874 µs ± 22.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%%timeit
100_000 in s100k

25.6 ns ± 0.812 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

%%timeit
1 in l100k

33.5 ns ± 0.589 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

%%timeit
1 in s100k

26.4 ns ± 1.13 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

Python Singletons

João M.C. Teixeira — Tue, 23 Feb 2021 18:51:40 +0000

A long time has passed since I last wrote a Singleton. Definitively, we rarely write singleton patterns in Python because we can represent singletons as variables in a module, and import them everywhere - see my strategy for logging library wide.

However, how should we construct a singleton class which represents a state that is initialized (loaded/read) together with the class first instantiation and for which we don't want the whole loading process to be repeated in case the singleton class is instantiated again at a different stage of the runtime?

Well, here it is:

class MyClass:
    _state = None

    def __new__(cls, *args, **kwargs):

        if cls._state:
            print('State already exists, returning existing state.')
            return cls._state

        elif cls._state is None:
            cls._state = super().__new__(cls)
            return cls._state

    def __init__(self, arg):
        self.arg = 1


a = MyClass(1)
print(a.arg)  # 1
b = MyClass(2)
print(b.arg)  # 1

assert a is b  # True

Enjoy and Cheers,

Fibonacci without recursiveness in Python - a better way

João M.C. Teixeira — Mon, 25 Jan 2021 15:16:39 +0000

¡Short post! (not so short after the edit)

Some time ago we had a great discussion in a post (see also comments) about From 100% to 0% CPU with memoization applied to Fibonacci series.

From that post comes the idea that explaining recursive functions using the Fibonacci example might fail the very purpose of Python in the first place.

What about solving the Fibonacci problem this way?

EDIT 20 Feb 2021

The original for loop was for i in range(2, i -1):, I corrected it to the following bellow. Thanks to @paddy3118 for spotting that repetition of i.

# first definitions
i = 50  # number of elements in the final Fibonacci series
fib = [0, 1, 1]  # fixed seed
fiba = fib.append  # short cut to the append method

# Calculate the Fibonacci series
for _ in range(3, i): fiba(fib[-2] + fib[-1])

@paddy3118 in his/her comment also suggested the use of numpy to store the values and just fill the results in the indexes. Obviously, that requires using numpy and that might not be an option, however, if it is, let's investigate how much time it costs to use that approach:

This is how I understood @paddy3118 comment, please Paddy correct me I am wrong.

i = 50
fib = np.zeros(i, dtype=np.int)
fib[0:3] = [0, 1, 1]
for k in range(3, i):
    fib[k] = fib[k - 2] + fib[k - 1]

In my machine with Jupyter %%timeit I receive:

The slowest run took 5.29 times longer than the fastest. This could mean that an intermediate result is being cached.
10000 loops, best of 5: 23 µs per loop

From which: 100000 loops, best of 5: 2.42 µs per loop refer to the array creation and 10000 loops, best of 5: 21.5 µs per loop to the for loop.

While if I %%timeit the approach with lists, we have for the whole process:

100000 loops, best of 5: 4.66 µs per loop

Actually, it is much faster to perform this operation with lists. Maybe contrarily to expectations?

Is that what you were referring to @paddy3118? Thanks for your comment, I enjoy it a lot!

Second EDIT

On a later discussion with a friend, we saw that if the list grows considerably, the append version becomes much slower than the version where the list is defined beforehand (see example below). Hence, if the number i is known, and that number is large, the version presented next is much faster.

This latter version avoids the list being (internally) copied over to larger allocated memory blocks as new values are appended. So, depending on the size of the target lists you expect to produce and on the performance requirements, you could choose one or the other. I believe this last version is more in line with what @paddy3118 was referring to before.

i = 50_000
fib = [0 for _ in range(i)]
fib[1:3] = [1, 1]

for k in range(3, i):
    fib[k] = fib[-2] + fib[-1]

Runs in:

100 loops, best of 5: 4.1 ms per loop

While the first version with i = 50_000 runs in:

10 loops, best of 5: 68.8 ms per loop

I started this post to discuss that the Fib series might not be a good example to explain recursion. We ended up finding a very nice situation where Fib can (maybe) serve as a more adequate example. I will look forward to writing a small post on how lists work internally in Python and explain the behavior here observed.

Cheers,

Python Project Template

João M.C. Teixeira — Sat, 23 Jan 2021 02:12:08 +0000

I have updated my Python Project Template repository. This is a fully working template where you will find configured continuous integration services with GitHub actions, Codecov, tox, ReadTheDocs, and others.

You can use this template as a reference to configure your own CI protocol or as an actual GitHub template for your repository.

I also discuss many aspects on how to structure a Python application/library.

Let me know your thoughts,

https://github.com/joaomcteixeira/python-project-skeleton