DEV Community: Humberto A Sanchez II

Towards Effortless Python Configuration Files Version 3

Humberto A Sanchez II — Tue, 26 Nov 2024 20:33:33 +0000

Introduction

This is the final article in this series. This implemenation seeks to fix the main disadvantage of boiler plate code that I described in the previous article. I call this implementation a dynamic property class.

Class Representation

The following class diagram shows the DynamicConfiguration reusable class and the supporting data structures needed for a developer to use this functionality. It still provides all the basic functionality of version 2 including auto-boot strapping, creation of missing sections and key values.

Developer Code Description

I am going to present the full source code for an application seeking to use this class. I am using the previous properties that we have been discussing in the previous 3 articles.

from codeallybasic.DynamicConfiguration import DynamicConfiguration
from codeallybasic.DynamicConfiguration import KeyName
from codeallybasic.DynamicConfiguration import SectionName
from codeallybasic.DynamicConfiguration import Sections
from codeallybasic.DynamicConfiguration import ValueDescription
from codeallybasic.DynamicConfiguration import ValueDescriptions
from codeallybasic.SecureConversions import SecureConversions

from codeallybasic.SingletonV3 import SingletonV3

from ByteSizedPython.ImpostorEnumByName import ImpostorEnumByName
from ByteSizedPython.PhoneyEnumByValue import PhoneyEnumByValue

LOGGER_NAME:    str = 'Tutorial'
BASE_FILE_NAME: str = 'config.ini'
MODULE_NAME:    str = 'version3properties'

DEFAULT_PHONEY_ENUM_BY_VALUE:  PhoneyEnumByValue  = PhoneyEnumByValue.FakeBrenda
DEFAULT_IMPOSTOR_ENUM_BY_NAME: ImpostorEnumByName = ImpostorEnumByName.High

GENERAL_PROPERTIES: ValueDescriptions = ValueDescriptions(
    {
        KeyName('debug'):    ValueDescription(defaultValue='False', deserializer=SecureConversions.secureBoolean),
        KeyName('logLevel'): ValueDescription(defaultValue='Info'),
        KeyName('phoneyEnumByValue'):  ValueDescription(defaultValue=DEFAULT_PHONEY_ENUM_BY_VALUE.value,  enumUseValue=True),
        KeyName('impostorEnumByName'): ValueDescription(defaultValue=DEFAULT_IMPOSTOR_ENUM_BY_NAME.name,  enumUseName=True),
    }
)

DATABASE_PROPERTIES: ValueDescriptions = ValueDescriptions(
    {
        KeyName('dbName'): ValueDescription(defaultValue='dbName'),
        KeyName('dbHost'): ValueDescription(defaultValue='localhost'),
        KeyName('dbPort'): ValueDescription(defaultValue='5342', deserializer=SecureConversions.secureInteger),
    }
)

CONFIGURATION_SECTIONS: Sections = Sections(
    {
        SectionName('General'):  GENERAL_PROPERTIES,
        SectionName('Database'): DATABASE_PROPERTIES,
    }
)

class ConfigurationPropertiesVersion3(DynamicConfiguration, metaclass=SingletonV3):
    def __init__(self):

        self._logger: Logger = getLogger(LOGGER_NAME)

        super().__init__(baseFileName=BASE_FILE_NAME, moduleName=MODULE_NAME, sections=CONFIGURATION_SECTIONS)

Lines 45-50 are the code that you have to write. Essentially, your are just ensuring that you pass the file name, module name, and the configuration sections. This Sections type comes from the DynamicConfiguration module.

Lines 21-28 and 30-36 are ValueDescriptions dictionaries. The KeyName is the property and points to a ValueDescription. Notice that the indicator on how to persist an enumeration is moved from the previous implementation's decorator to a boolean attribute in a ValueDescription.

Implementation Code Description

If you look closely at the class diagram for DynamicConfiguration you will see that it implements two Python magic methods. They are the __getattr__(self, name)__ and __setattr__(self, name, value)__ methods.

__getattr__(self, name)__ Allows a developer to define behavior when a class consumer attempts to access a non-existent attribute.
__setattr__(self, name, value)__ Allows a developer to define behavior for assignment to an attribute.

The following is the code for __getattr__. This looks very much like the decorator we used in version 2. The key work happens on the call on line 14 to the protected method _lookupKey(). It returns a full description of the attribute so that we can simulate the attribute retrieval.

    def __getattr__(self, attrName: str) -> Any:
        """
        Does the work of retrieving the named attribute from the configuration parser

        Args:
            attrName:

        Returns:  The correctly typed value
        """

        self._logger.info(f'{attrName}')

        configParser: ConfigParser     = self._configParser
        result:       LookupResult     = self._lookupKey(searchKeyName=KeyName(attrName))
        valueDescription: ValueDescription = result.keyDescription

        valueStr: str = configParser.get(result.sectionName, attrName)

        if valueDescription.deserializer is not None:
            value: Any = valueDescription.deserializer(valueStr)
        else:
            value = valueStr

        return value

The following is the__setattr__() implementation. Notice the support for enumerations in lines 22-27 and the write-through feature in line 30.

    def __setattr__(self, key: str, value: Any):
        """
        Do the work of writing this back to the configuration/settings/preferences file
        Ignores protected and private variables uses by this class

        Does a "write through" to the backing configuration file (.ini)

        Args:
            key:    The property name
            value:  Its new value
        """

        if key.startswith(PROTECTED_PROPERTY_INDICATOR) or key.startswith(PRIVATE_PROPERTY_INDICATOR):
            super(DynamicConfiguration, self).__setattr__(key, value)
        else:
            self._logger.debug(f'Writing `{key}` with `{value}` to configuration file')

            configParser: ConfigParser  = self._configParser
            result:       LookupResult  = self._lookupKey(searchKeyName=KeyName(key))
            valueDescription: ValueDescription = result.keyDescription

            if valueDescription.enumUseValue is True:
                valueStr: str = value.value
                configParser.set(result.sectionName, key, valueStr)
            elif valueDescription.enumUseName is True:
                configParser.set(result.sectionName, key, value.name)
            else:
                configParser.set(result.sectionName, key, str(value))

            self.saveConfiguration()

Accessing and Modifying Properties

Accessing and modifying properties is exactly the same as in version 2.

    basicConfig(level=INFO)

    config: ConfigurationPropertiesVersion2 = ConfigurationPropertiesVersion2()

    logger: Logger = getLogger(LOGGER_NAME)

    logger.info(f'{config.debug=}')
    logger.info(f'{config.logLevel=}')
    logger.info(f'{config.phoneyEnumByValue=}')
    logger.info(f'{config.impostorEnumByName=}')
    logger.info('Database Properties Follow')
    logger.info(f'{config.dbName=}')
    logger.info(f'{config.dbHost=}')
    logger.info(f'{config.dbPort=}')
    logger.info('Mutate Enumeration Properties')
    config.phoneyEnumByValue = PhoneyEnumByValue.TheWanderer
    logger.info(f'{config.phoneyEnumByValue=}')
    config.impostorEnumByName = ImpostorEnumByName.Low
    logger.info(f'{config.impostorEnumByName=}')

The above snippet produces the following output.

INFO:Tutorial:config.debug='False'
INFO:Tutorial:config.logLevel='Info'
INFO:Tutorial:config.phoneyEnumByValue=<PhoneyEnumByValue.FakeBrenda: 'Faker Extraordinaire'>
INFO:Tutorial:config.impostorEnumByName='High'
INFO:Tutorial:Database Properties Follow
INFO:Tutorial:config.dbName='example_db'
INFO:Tutorial:config.dbHost='localhost'
INFO:Tutorial:config.dbPort=5432
INFO:Tutorial:Mutate Enumeration Properties
INFO:Tutorial:config.phoneyEnumByValue=<PhoneyEnumByValue.TheWanderer: 'The Wanderer'>
INFO:Tutorial:config.impostorEnumByName='Low'

Conclusion

The source code for this article is here. See the support class SingletonV3. See the implementation of

Advantages

Easy type safe access to application properties
Reusable parent class for different implementations
Data structure driven code to add new sections and configuration keys
No boiler plate code for properties

Disadvantages

Since there are no actual properties implemented we do not get IDE support for them
Additionally, because of the lookup methods for keys, different keys in different sections cannot have the same name

Towards Effortless Python Configuration Files Version 2

Humberto A Sanchez II — Tue, 26 Nov 2024 20:20:28 +0000

Introduction

In the previous article, we managed to create a simulated property dataclass to manage configuration values for an application. However, it was basically just a template that we had to re-implement for each usage. In version 2, of my work I succeeded in implementing a reusable class.

It requires a pair of decorators on the property definitions.
Plus, the implementation requires a Sections data type that contains a Section entry for each configuration section.
Each section requires a list of ConfigurationNameValue entries

Class Representation

The following class diagram shows the basic reusable class and the data structures needed for a developer to use this functionality.

Subclass Configuration Properties

The developer starts the process by subclassing Configuration Properties as follows.

BASE_FILE_NAME: str = 'config.ini'
MODULE_NAME:    str = 'version2properties'

class ConfigurationPropertiesVersion2(ConfigurationProperties, metaclass=SingletonV3):

    def __init__(self):
        self.logger: Logger = getLogger(LOGGER_NAME)

        super().__init__(baseFileName=BASE_FILE_NAME, moduleName=MODULE_NAME, sections=CONFIGURATION_SECTIONS)

        self._configParser.optionxform = self._toStr    # type: ignore

        self._loadConfiguration()

The call to super creates a fully qualified path to the configuration file. The code follows a subset of XDG Base Directory Specification. The code first tries XDG_CONFIG_HOME, then HOME, and finally uses the current directory if both fail. In line 13, the developer calls the protected method to get the configuration parser ready. Additionally, that call will

Ensures the configuration file exists, if not it creates an empty one
Creates missing sections
Creates missing keys in the sections

The previous capabilities then result in lots of reusable code and bootstraps configuration files. Additionally, we make the class a Singleton to mitigate creation costs as we instantiate it across our application code base.

Define Sections

Notice the parameter sections in the super call. This definition of sections looks like this:

from codeallybasic.ConfigurationProperties import Sections

CONFIGURATION_SECTIONS: Sections = Sections(
    {
        SectionName('General'):  SECTION_GENERAL,
        SectionName('Database'): SECTION_DATABASE,
    }
)

The above is a dictionary where the key is a section name and the value is a section.

Define Section

A Section is just a list of ConfigurationNameValue entries. A ConfigurationNameValue is a data class with 2 values, a PropertyName and its default value. Here are our sections.

from codeallybasic.ConfigurationProperties import Section
from codeallybasic.ConfigurationProperties import ConfigurationNameValue
from codeallybasic.ConfigurationProperties import PropertyName

SECTION_GENERAL: Section = Section(
    [
        ConfigurationNameValue(name=PropertyName('debug'),                           defaultValue='False'),
        ConfigurationNameValue(name=PropertyName('logLevel'),                     defaultValue='Info'),
        ConfigurationNameValue(name=PropertyName('phoneyEnumByValue'),  defaultValue=DEFAULT_PHONEY_ENUM_BY_VALUE.value),
        ConfigurationNameValue(name=PropertyName('impostorEnumByName'), defaultValue=DEFAULT_IMPOSTOR_ENUM_BY_NAME.name),
    ]
)

SECTION_DATABASE: Section = Section(
    [
        ConfigurationNameValue(name=PropertyName('dbName'), defaultValue='example_db'),
        ConfigurationNameValue(name=PropertyName('dbHost'), defaultValue='localhost'),
        ConfigurationNameValue(name=PropertyName('dbPort'), defaultValue='5432'),
    ]
)

Notice that I have specified two new enumeration properties. One where we wish to persists is value and the other its name.

Here are there enumeration definitions.

class PhoneyEnumByValue(Enum):
    TheWanderer = 'The Wanderer'
    Mentiroso   = 'Mentiroso'
    FakeBrenda  = 'Faker Extraordinaire'
    NotSet          = 'Not Set'

    @classmethod
    def deSerialize(cls, value: str) -> 'PhoneyEnumByValue':

    @classmethod
    def deSerialize(cls, value: str) -> 'PhoneyEnumByValue':

        match value:
            case PhoneyEnumByValue.TheWanderer.value:
                phoneyEnum: PhoneyEnumByValue = PhoneyEnumByValue.TheWanderer
            case PhoneyEnumByValue.Mentiroso.value:
                phoneyEnum = PhoneyEnumByValue.Mentiroso
            case PhoneyEnumByValue.FakeBrenda.value:
                phoneyEnum = PhoneyEnumByValue.FakeBrenda
            case _:
                raise Exception('Unknown PhoneyEnumByValue')

        return phoneyEnum


class ImpostorEnumByName(Enum):
    Low       = 0.1
    Medium = 0.5
    High     = 1.0
    NotSet = -1.0

We'll see how these affect the property definitions in a developer class

Property Definitions

String properties are defined as follows.

    @property
    @configurationGetter(sectionName='General')
    def debug(self) -> str:
        return ''       # never executed

    @debug.setter
    @configurationSetter(sectionName='General')
    def debug(self, newValue: str):
        pass

What we have gotten rid of is the boiler plate to access the configParser to get and set the values. What we added was the configurationGetter and configurationSetter decorators. I am not going to cover the implementation details of the decorators and leave this as an exercise for the reader. These decorators take care of interacting with the configuration parser to get and set values. On setting values the configurationSetter decorator does the write-through.

Developers define integer properties as follows.

    @property
    @configurationGetter(sectionName='Database', deserializeFunction=int)
    def dbPort(self) -> int:
        return -1

    @dbPort.setter
    @configurationSetter(sectionName='Database',)
    def dbPort(self, newValue: int):
        pass

Notice that the configurationGetter decorator has an optional parameter. It is a function that takes the string property value and converts it to an appropriate typed value before returning to the caller properties. This can be applied to float properties.

Enumeration properties where we want to persist the enumeration name are defined as follows:

    @property
    @configurationGetter(sectionName='General')
    def impostorEnumByName(self) -> ImpostorEnumByName:
        return ImpostorEnumByName.NotSet      # Never executed

    @impostorEnumByName.setter
    @configurationSetter(sectionName='General', enumUseName=True)
    def impostorEnumByName(self, newValue: ImpostorEnumByName):
        pass

Besides using the appropriate decorators note that to persist the enumeration name use the enumUseName parameter and set it to True.

The following is an enumeration property where the developer want to persist its value. Notice that the setter decorator indicates that it is an enumeration. Additionally, notice that the developer must provide a deserialize method that can convert the value to the specific enumeration value.

    @property
    @configurationGetter(sectionName='General', deserializeFunction=PhoneyEnumByValue.deSerialize)
    def phoneyEnumByValue(self) -> PhoneyEnumByValue:
        return PhoneyEnumByValue.NotSet      # Never executed

    @phoneyEnumByValue.setter
    @configurationSetter(sectionName='General', isEnum=True)
    def phoneyEnumByValue(self, newValue: PhoneyEnumByValue):
        pass

Accessing and Modifying Properties

Accessing and modifying properties is exactly the same as in version 1.

    basicConfig(level=INFO)

    config: ConfigurationPropertiesVersion2 = ConfigurationPropertiesVersion2()

    logger: Logger = getLogger(LOGGER_NAME)

    logger.info(f'{config.debug=}')
    logger.info(f'{config.logLevel=}')
    logger.info(f'{config.phoneyEnumByValue=}')
    logger.info(f'{config.impostorEnumByName=}')
    logger.info('Database Properties Follow')
    logger.info(f'{config.dbName=}')
    logger.info(f'{config.dbHost=}')
    logger.info(f'{config.dbPort=}')
    logger.info('Mutate Enumeration Properties')
    config.phoneyEnumByValue = PhoneyEnumByValue.TheWanderer
    logger.info(f'{config.phoneyEnumByValue=}')
    config.impostorEnumByName = ImpostorEnumByName.Low
    logger.info(f'{config.impostorEnumByName=}')

The above snippet produces the following output.

INFO:Tutorial:config.debug='False'
INFO:Tutorial:config.logLevel='Info'
INFO:Tutorial:config.phoneyEnumByValue=<PhoneyEnumByValue.FakeBrenda: 'Faker Extraordinaire'>
INFO:Tutorial:config.impostorEnumByName='High'
INFO:Tutorial:Database Properties Follow
INFO:Tutorial:config.dbName='example_db'
INFO:Tutorial:config.dbHost='localhost'
INFO:Tutorial:config.dbPort=5432
INFO:Tutorial:Mutate Enumeration Properties
INFO:Tutorial:config.phoneyEnumByValue=<PhoneyEnumByValue.TheWanderer: 'The Wanderer'>
INFO:Tutorial:config.impostorEnumByName='Low'

Conclusion

The source code for this article is here. See the support class SingletonV3. See the implementation of ConfigurationProperties

The result of the implementation left me satisfied as a consumer of the code. I was able to get and set typed properties. It did not remove as much code as I had initially hoped for. However, it did provide me with reusable code. However, it did prompt me to write a Live Template in PyCharm to get me over generating individual properties.

Advantages

Easy type safe access to application properties
Reusable parent class for different implementations
Data structure driven code to add new sections and configuration keys

Disadvantages

Still lots of boiler plate code
I have feeling that using decorators was a misuse of them

My next post I implemented what I call dynamic properties. It entirely removed all the boiler plate code and still maintained the aforementioned advantages.

Towards Effortless Python Configuration Files Version 1

Humberto A Sanchez II — Tue, 26 Nov 2024 20:17:47 +0000

Introduction

As noted in the previous article the simplistic version is rife with problems including scalability, maintainability and extensibility.

A simple extension from Version Ø is to try and hide the Python configuration details behind a property class. That is implement a pseudo data class that exposes a set of properties that allow a developer to simply do property set and get calls to retrieve and persist property values.

From the maintainers point of view this implementation should support the following capabilities.

Allow auto creation of configuration sections if they are missing
Allow auto creation of property values if they are missing
The properties should be implemented as both read-through and write-through.
In order to avoid the startup costs for the above as an application instantiates this class throughout the application this class should be a Singleton.

Class Representation

The following UML Class diagram describes a class that would meet the requirements in the introduction. The ConfiguratonProperties class meets requirements 1 & 2 with the protected methods .createMissingSections and .createMissingKeys

Create Implementations

Create Missing Sections

The following code shows the implementation. Notice that additional sections require code updates to this method

SECTION_GENERAL:  str = 'General'
SECTION_DATABASE: str = 'Database'

def _createMissingSections(self):
    """
    Create missing sections.  Add additional calls for
    each defined section
    """
    self._createMissingSection(SECTION_GENERAL)
    self._createMissingSection(SECTION_DATABASE)

The missing section code is as follows.

def _createMissingSection(self, sectionName: str):
    """
    Only gets created if it is missing
    Args:
        sectionName: The potential section to create
    """
    hasSection: bool = self._configParser.has_section(sectionName)
    self.logger.info(f'hasSection: {hasSection} - {sectionName}')
    if hasSection is False:
        self._configParser.add_section(sectionName)

Create Missing Keys

The following code shows the implementation. Again note if we add an additional section the developer must add an additional loop for the new section.

GENERAL_PREFERENCES: Dict[str, str] = {
    'debug':    'False',
    'logLevel': 'Info'
}

DATABASE_PREFERENCES: Dict[str, str] = {
    'dbName': 'example_db',
    'dbHost': 'localhost',
    'dbPort': '5432'
}

def _createMissingKeys(self):
    """
    Create missing keys and their values.  Add additional calls for
    each defined section.
    """
    for keyName, keyValue in GENERAL_PREFERENCES.items():
        self._createMissingKey(sectionName=SECTION_GENERAL, keyName=keyName, defaultValue=keyValue)
    for keyName, keyValue in DATABASE_PREFERENCES.items():
        self._createMissingKey(sectionName=SECTION_DATABASE, keyName=keyName, defaultValue=keyValue)

The missing key code is as follows. Notice any missing keys are immediately persisted.

def _createMissingKey(self, sectionName: str, keyName: str, defaultValue: str):
    """
    Only gets created if it is missing.  The configuration file is updated
    immediately for each missing key and its value

    Args:
        sectionName:   The section name where the key resides
        keyName:       The key name
        defaultValue:  Itsß value
    """
    if self._configParser.has_option(sectionName, keyName) is False:
        self._configParser.set(sectionName, keyName, defaultValue)
        self._saveConfiguration()

Class Properties

Sample implementations for requirement 3 follow.

String Properties

Notice that setting a property writes-through to the configuration file by setting the property and immediately persisting it. Reading properties are effectively read-through because of how we immediately write set properties.

@property
def dbName(self) -> str:
    return self._configParser.get(SECTION_DATABASE, 'dbName')

@dbName.setter
def dbName(self, newValue: str):
    self._configParser.set(SECTION_DATABASE, 'dbName', newValue)
    self._saveConfiguration()

Integer Properties

Integer properties use the .getint method to retrieve the value. When setting the property the developer must manually convert it to a string.

@property
def dbPort(self) -> int:
    return self._configParser.getint(SECTION_DATABASE, 'dbPort')

@dbPort.setter
def dbPort(self, newValue: int):
    self._configParser.set(SECTION_DATABASE, 'dbPort', str(newValue))
    self._saveConfiguration()

Boolean Properties

Boolean properties use the .getboolean method to retrieve their value. When setting the property the developer must manually convert it to a string.

SECTION_GENERAL:  str = 'General'

@property
def debug(self) -> bool:
    return self._configParser.getboolean(SECTION_GENERAL, 'debug')

@debug.setter
def debug(self, newValue: bool):
    self._configParser.set(SECTION_GENERAL, 'debug', str(newValue))
    self._saveConfiguration()

Enumeration Properties

I will not cover enumeration properties in this article. There are two ways to persist them, by their name or by their value. Each mechanism requires a slightly different way to deserialize the values back to an enumeration type.

Accessing and Modifying Properties

The following code snippet demonstrates how to access and modify the properties.

from logging import Logger
from logging import getLogger
from logging import basicConfig
from logging import INFO

LOGGER_NAME: str = 'Tutorial'

basicConfig(level=INFO)

config: ConfigurationProperties = ConfigurationProperties()

logger: Logger = getLogger(LOGGER_NAME)

logger.info(f'{config.debug=}')
logger.info(f'{config.logLevel=}')
#
logger.info('Change the values and show them')
#
config.debug = True
logger.info(f'{config.debug=}')
config.logLevel = 'Warning'
logger.info(f'{config.logLevel=}')
#
logger.info('**** DataBase Section ****')
logger.info(f'{config.dbName=}')
logger.info(f'{config.dbHost=}')
logger.info(f'{config.dbPort=}')
#
logger.info('Change db values and print them')
config.dbName = 'ozzeeDb'
config.dbHost = 'ozzeeHost'
config.dbPort = 6666

logger.info(f'{config.dbName=}')
logger.info(f'{config.dbHost=}')
logger.info(f'{config.dbPort=}')

The above snippet produces the following output

INFO:Tutorial:Configuration file existed
INFO:Tutorial:hasSection: True - General
INFO:Tutorial:hasSection: True - Database
INFO:Tutorial:config.debug=True
INFO:Tutorial:config.logLevel='Warning'
INFO:Tutorial:Change the values and show them
INFO:Tutorial:config.debug=True
INFO:Tutorial:config.logLevel='Warning'
INFO:Tutorial:**** DataBase Section ****
INFO:Tutorial:config.dbName='ozzeeDb'
INFO:Tutorial:config.dbHost='ozzeeHost'
INFO:Tutorial:config.dbPort=6666
INFO:Tutorial:Change db values and print them
INFO:Tutorial:config.dbName='ozzeeDb'
INFO:Tutorial:config.dbHost='ozzeeHost'
INFO:Tutorial:config.dbPort=6666

Conclusion

The source code for this article is here. The support class SingletonV3 is here

The result of the implementation initially left me satisfied as a consumer of the code. I was able to get and set typed properties. However, as the maintainer of the code I had to manually update code data structures and code loops whenever I added new sections and new properties. Additionally, all I really got from this is a mechanism/pattern to use whenever I needed new configuration properties in different applications.

Advantages

Easy type safe access to application properties
Invoking the singleton in different parts of my application provided consistent and reliable access to properties regardless which part of the application modified values

Disadvantages

Updates to add new properties was tedious
Lots of boiler plate code
No reusability across various applications. Essentially, I just had a template

See my next post that documents an alternate implementation to address the disadvantages I listed, while keeping the advantages.

Towards Effortless Python Configuration Files Version Ø

Humberto A Sanchez II — Tue, 26 Nov 2024 20:16:04 +0000

Introduction

We write many Python applications that require customization via external properties or applications for which we wish to customize or affect behavior with non-hardcoded properties and/or runtime configuration properties. Various Google searches for solutions yield tutorials that point us to code samples that while practically work, do not scale appropriately for real world applications.

These set of articles document my journey through various implementations as I repeatedly refactored and reimplemented on my journey to a simple, maintainable, and easily extensible mechanism to manage application configuration properties.

Version Ø

The tutorial versions I found were simply expansions of the code snippets that the library developers provide to prove that their implementations work. While this is sufficient to provide a proof of concept these snippets do not scale in the application real world.

An example of this is the following code snippet.

import configparser

def create_config():
    config = configparser.ConfigParser()

    # Add sections and key-value pairs
    config['General'] = {
      'debug': 'True', 
      'log_level': 'info'
    }
    config['Database'] = {
      'db_name': 'example_db',
      'db_host': 'localhost', 
      'db_port': '5432'
    }

    with open('config.ini', 'w') as configfile:
        config.write(configfile)


if __name__ == "__main__":
    create_config()

While this code snippet certainly allows us to persist our configuration values it leaves us with the problem of reading these persisted values. Again, the implementation developer snippets provide us with sample code on how to retrieve these values as depicted in the following code snippets.

import configparser

def read_config():
    config = configparser.ConfigParser()

    config.read('config.ini')

    # Access values from the configuration file
    debug_mode = config.getboolean('General', 'debug')
    log_level = config.get('General', 'log_level')
    db_name = config.get('Database', 'db_name')
    db_host = config.get('Database', 'db_host')
    db_port = config.get('Database', 'db_port')

    # Return a dictionary with the retrieved values
    config_values = {
        'debug_mode': debug_mode,
        'log_level': log_level,
        'db_name': db_name,
        'db_host': db_host,
        'db_port': db_port
    }

    return config_values

if __name__ == "__main__":
    # Call the function to read the configuration file
    config_data = read_config()

    # Print the retrieved values
    print('Debug Mode',       config_data['debug_mode'])
    print('Log Level',         config_data['log_level'])
    print('Database Name', config_data['db_name'])
    print('Database Host', config_data['db_host'])
    print('Database Port', config_data['db_port'])

Conclusion

I see many issues in the above code. While it is perfectly acceptable for small scripts, the code suffers from the usage of string values, related to actual Python variable names and their potential proliferation across a large code base. While this is potentially mitigated by the use of global constants, my opinion is that this is not scaleable because it does not follow a basic software design principle espoused by Andrew Hunt and David Thomas, in their seminal book, The Pragmatic Programmer and fails the DRY principle, aka Do not Repeat Yourself.

The source code for this article is here.

See my next post that documents an initial implementation to solve some of the issues I outlined.

Python Project Creation on Mac OS X

Humberto A Sanchez II — Wed, 16 Oct 2024 20:45:25 +0000

Introduction

PyFabricate is a Mac OS X application that simplifies the creation of Python projects. Since I authored it, obviously it is an opinionated version of what I think a Python project should look like. Additionally, I have a certain toolset that I use to enable me to write Python Mac OS X applications. However, I tried to create a bit of customization by exposing the templates from which I create project files. More on this later.

Overview

PyFabricate uses a wizard-like interface to query for the project details.

Opinionated Assumptions

The developer uses HomeBrew to install the opinionated dependencies.
- Python is managed via pyenv (For use in the created virtual environments)
- direnv is installed (optional)
The project name is the same as the module package name

Installation

Get the current release from here.

Copy this file to your Mac OS applications folder. The application is Apple code signed so you will get a simple message you have to acknowledge the first time you execute it.

Project Skeleton Details

The following is a sample of the initial skeleton that PyFabricator creates. It does now show the virtual environment that it creates. More on this later.

Essentially, PyFabricate creates the src and tests directories. In the project root directory it creates the following files:

.envrc
.gitignore
.mypi.ini
.python_version
.LICENSE
README.md
pyproject.toml
requirements.txt

Additionally, PyFabricate creates all the python package files (__init__.py) in all the appropriate places. And finally, it places two sets of logging configurations files, one for your application and one for your unit tests.

At the end of the fabrication, PyFabricate creates a Python virtual environment. Because of technical limitations PyFabricate cannot use a pyenv version of Python to create the virtual environment. Thus, it depends on a homebrew installed version.

Wizard UI

The wizard UI is a 4 step process where PyFabricate gathers sufficient project details that it uses to populate the skeleton templates. The templates use the following self-explanatory substitution tokens.

Token Name	Description
PROJECT_NAME	Used to construct the base project directory and is the Python module name
DESCRIPTION	Placed as the description in the generated `pyproject.toml`
OWNER_NAME	Placed as the author name and maintainer name in the generated `pyproject.toml`
OWNER_EMAIL	Placed as the email address for botht the author and the maintainer in the generated `pyproject.toml`
KEYWORDS	Used as the project keywords in the generated `pyproject.toml`
PYTHON_VERSION	Used in the generated .gitignore file to exclude the directory that contains the virtual environment. Additionally appended as the directory name for the generated virtual environment. For example, '`pyenv-3.12.0`'

The following is an annotated graphic of the pyroject.toml template.

Wizard UI Step 1 - Introduction

This step is an introduction to PyFabricate. After this step, PyFabricate does a best effort to ensure that its execution dependencies are met. If not, the developer cannot proceed past this step.

Wizard UI Step 2 - Project Details

At this step PyFabricate collects project specific information. Most of this serves to populate the pyproject.toml template.

Wizard UI Step 3 - Base Directory

PyFabricate assumes that the developer keeps all of his Python projects in a single subdirectory. This is an opinionated view and is simply an organizational preference.

Wizard UI Step 4 - Python Version

PyFabricate queries pyenv for the developer installed Python versions. These are the only ones from which PyFabricate can create Python virtual environments.

Action Log

At the end of the wizard, PyFabricate presents a log as it creates the project. The following is an example log.

Customization

As mentioned earlier the developer can customize some aspects of the project generation. PyFabricate stores the templates it uses in $HOME/.config/pyfabricate/templates as noted below.

For example, a developer may want to use a difference license. Simply, replace the LICENSE.template file

Wrap Up

PyFabricate is currently only at version 0.5.0. I am currently investigating adding additional features and removing limitations.

Signing Git Commits on Mac OS via SmartGit

Humberto A Sanchez II — Mon, 09 Sep 2024 21:18:15 +0000

Introduction

Between this article and this article, I figured out how to sign my GitHub commits.

Essentially it showed me how to:

Check the GPG installation
Check the keys
Configure a single repository to use a signing key
Verify signed commits
It pointed me here for instructions on how to add my GPG key to GitHub

In SmartGit you can configure signing by typing:

Smartgit —> Settings —>Git config

You will see the following dialog. Enter your information.

However, commits with SmartGit failed because pinentry (pin entry, get it !) was trying to popup a terminal based UI to ask me for the signing key passphrase.

Because I use homebrew it is rather confusing as to where to configure gpg, the gpg-agent, and pinentry. Lots of the docmentation is for Linux and to make matters worse I am on Mac OS.

Earlier documentation suggested I had to update gpg.conf to configure pinentry.

Finally found this article that correctly pointed to gpg-agent.conf

Actually it said:

~/.gnupg/gpg-agent.conf

But I deduced that the homebrew configuration files resided here:

/opt/homebrew/etc/gnupg

The file did not exist so I created it. Voila, on my next check-in SmartGit popped up the OS X pin entry dialog

My Cliff's notes on key generation

gpg --full-generate-key

List the key to get the signing key

gpg --list-secret-keys --keyid-format LONG

Export public to import into GitHub

gpg --armor --export [my key] > public.gpg

Verify signed commits

git log --show-signature

Prototype to Maintainable Code

Humberto A Sanchez II — Mon, 20 Jun 2022 17:49:09 +0000

Introduction

The impetus for this post is that I noticed that my approach to writing production software applications was to write code in some kind of sandbox that was outside the repository that I intended to check in.

Goal

My goal was to first prove that I could accomplish the task I needed to do and verify it was correct with no regards as to the code aesthetics or how easy it would be to modify it. The primary purpose was to get something to work. However, it was always with the attitude that the initial code version was in no way shape or manner the final releasable product.

Prototype

I was trying to solve a problem that I discussed in a blog post. I was not totally familiar how within Python:

I was supposed to launch an external process and capture the output.
After capturing the output I needed to parse it
And produce something that was consumable by the OS X diskutil unmount command.

The idea was that the utility would unmount a set of problematic volumes (aka, some pesky time machine volumes).

Source Code Version 1

In my typical fashion, I authored the code delineated in the following figure that met the requirements I just specified.

As I was writing this I noticed that the variables:

output
tmLines, and
timeMachineVolumes

all declared that they were a list of strings. If ever there was a chance for bugs that is a classic way to introduce them.

Additionally, I am not a big fan of reams and reams of unfettered code. In my opinion, that is unmaintainable. The original code gives hints that it can be refactored. We can deduce this because before each block of code there is a comment that describes its functionality.

The Better Way

The first step was to create specific types for the various transformations. This is really just syntactic sugar but serves to make code more readable and less error prone when refactoring. The following figure shows the transformation.

Next, I needed to re-arrange the code as methods in a class. The following figure shows how I rearranged the original code to make it more understandable for future versions of me.

The results of this refactoring resulted in three methods

• _runExternalMountCommand
• _extractTimeMachineLines
• _extractTimeMachineVolumes

These three methods are called sequentially from the .execute driver method.

Summary

What you should learn is first make your code work correctly. Next, remember that either a future you or someone who did not author the original code will at some point have to update and/or fix it. Thus, construct your code so that it is understandable to humans. Leave explicit hints as to the function of the source code. I would go as far as to say to write code that reads as English and is self documenting. Some people refer to this as literate programming.

I published the original version here

What is an Object Model

Humberto A Sanchez II — Sun, 18 Apr 2021 19:31:56 +0000

Introduction

What is an object model? This document explores this medium to complex computer science topic. It first explains in plain english and then pictorially a simple real world model. It concludes with a more complex and purely abstract model invented by the internet.

Document Management

This source Libre Office document resides at the following URL: https://hsanchezii.wordpress.com

This document’s status is: Work-in-progress.

Acronyms

DOM - Document Object Model. This is the data representation of the objects that comprise the structure and content of a document on the web.
HTML - Hyper Text Markup Language. This is the language in which most websites are written. It is used to create web pages and make them functional.

What is an Object Model

An object model is a computer science mechanism employed in order to be able to define or replicate real world “things”, aka objects in terms that a computer program can understand and manipulate.

Real World Model

Let us suppose we wanted to build an object model for a book. See the figure "Simple Book Model". The way to read this is as follows:

A Book has a table of contents and chapters.
A Chapters has Paragraphs
A Paragraph has Sentences
Sentences have Words
Words consist of a fixed set of Letters

The Internet's Document Object Model

The figure "Simple Web Dom" is a simplified and incomplete representation of an HTML DOM

It says:

A Documents consists of a Head tag and a Body tag
A Body may contain
- Titles (realistically you only have a single Title tag)
- Paragraphs
- Tables
- UnOrderLists
- Divs
Tables consist of Rows
Rows consist of Table Rows and Table Data

Summary

With these simple examples you can see it is easy to think in terms of object models.

Refactoring -- An Example

Humberto A Sanchez II — Tue, 12 Jan 2021 15:49:11 +0000

Introduction

What is refactoring? Why would a developer need to do a refactor? How do I refactor? I'll strive to answer these questions.

What is Refactoring

Refactoring is developers rewriting code to improve its structure. Developers make small incremental changes and verify functionality has not changed. Unit tests are the single most important enabler of refactoring. They provide automatic verification. They allow the developer to refactor and improve the underlying code with impunity.

Motivation

Developers refactor in order to make code testable, easier to understand, and easier to update. This code restructuring does NOT change its external behavior.

Refactoring Step by Step

PreRefactor Class

The following is a wxPython dialog in pre-refactoring mode. A cursory look at the code outline shows a curious mix of code functionality. It is violating the basic Single Responsibility Principle. Look at the code highlighted in the following figure:

Single Responsibility Principle

Classes should honor the single responsibility principle. This principle says that a class should do a single thing and only a single thing and do it well.

More formally it states, that every class in a computer program should have responsibility over a single part of that program's functionality and it should encapsulate it.

I assert that being a dialog this class should only do or handle user interface functionality. Specifically, functionality required by a wxPython dialog. All other things should be deferred to another “chunk of code” or better yet another class.

I assert that the following methods are basic functionality that manipulate which tip text to display and do not directly belong in a user interface class.

• ._getCurrentTipText() 
• ._incrementTipNumber()
• ._normalizeTip()

Post Refactor Class

Extract Functionality

Thus, the first step is to extract the tip manipulating functionality into a separate class. This makes it necessary to create a class named TipHandler.

In literate programming this name is self-explanatory. Literate programming is a way to write computer code that is understandable by a human. In the past, programmers wrote programs and code that was optimized for a computer. With today’s advances in computer speed and optimization it is perfectly possible with no penalty to write a verbose program that reads like prose.

Notice that in the following figure the TipHandler has the following non-UI functionality:

• Knows the current tip number
• Knows the current tip text
• Can properly increment the current tip number

Pure UI Class

Once we extract the above functionality from the UI class we wind up with a pure UI class as illustrated below.

At first glance the following class methods do not appear to be UI only functionality.

• ._getTipText()
• ._normalizeTip()

UI Constructor

Let us first look at the class constructor in the following image.

The UI class has effectively encapsulated the TipHandler functionality, so that it can delegate to it. This allows the UI class to be oblivious to the TipHandler inner workings.

`TipHandler` Unit Tests

The following figure depicts the unit tests necessary to functionally test the newly extracted class. These tests verify:

• How the tip count is computed
• Retrieving the correct current tip number
• Retrieving the correct current tip text
• How to properly increment the tip number and account for wrap-around

Conclusion

Thus, what we have accomplished is:

• A simple refactor that turns one class into two
• Demonstrated the single responsibility principle
• Demonstrated literate programming
• Provided an example of unit tests for the extracted functionality

What I am looking forward to in Python 3.9

Humberto A Sanchez II — Wed, 26 Aug 2020 21:39:27 +0000

Strings get removeprefix() and removesuffix()
New ways to merge or update dicts with the '|' (merge) and '|=' (update) operators

Ambivalent on

I use type hinting a lot. So I have to import typing.List in order to do types like List[str]. 3.9 will let you use the built in so the above will look like list[str];