DEV Community: Scott Rallya

Mastering Software Architecture Visualization with the C4 Model

Scott Rallya — Fri, 19 May 2023 03:56:07 +0000

Disclaimer

Note, the majority of this article was written by me but I did employ AI tools to help me generate the outline and some of the major talking points. I do appreciate the use of AI tools to help me bring you great content like this, and I hope you don't mind me employing such tools to help me convey complex ideas like software architecture visualization in a simple, easy-to-understand manner. Thank you for your consideration.

Introduction

Software architecture is difficult, and visualizing complex architectures can be a challenging process. In this article we'll examine the C4 Model for software architecture visualization and how it simplifies the process of visualizing complex architectures. We'll discuss a history of the model, its benefits and limitations, and we will examine the 4'Cs, Context, Container, Component, and Code, and explain the hierarchal nature of the model. Next, we will explore the use of the Structurizr DSL for creating C4 model diagrams. We will follow up with a conclusion and a recap of what we discussed.

Introduction to the C4 Model: What it is and why it's beneficial?

The C4 model is a software architecture visualization model invented by Simon Brown, a software architect and author, as a lightweight and scalable approach to communicate software architecture effectively. The model emphasizes a hierarchal view of the architecture at four levels: context, containers, components, and code.

The Context level provides a birds-eye view of the system, giving an overview of system boundaries, external actors (users), and their interactions with the system.

The Container level focuses on the major architectural building blocks of the system. This can include web servers, databases, desktop apps, microservices, or external services such as APIs. It communicates how these systems interact with each other.

The Component level zooms into a container, showing the internal structure and building blocks of that container and how those building blocks relate to one another. It focuses on the key responsibilities within the container and how they interact with one another.

The Code level provides a detailed view of the code structure of the component, showing details such as the classes, modules, and packages. This aids developers in understanding the actual implementation of these components.

The C4 model strives to promote simplicity and clarity in visualization, using a small set of symbols and notations to represent the different elements. This allows for readability of the diagrams. As a communication tool, the C4 model enables all stakeholders, not just developers, to use the model to understand and communicate about the structure of the software system. Project managers and product owners can use the Context and Container levels to discuss the high-level overview of the system, while Software Architects, Designers, and Implementers can use the Component and Code level to focus on the low-level details.

In addition, the C4 model allows for documentation support, creating documentation alongside the code and helping to capture the architecture effectively. It can easily be updated as the architecture and design of the system changes, thereby ensuring that the documentation always reflects the current state of the system.

Why Do We Need the C4 Model

The Challenge of Software Architecture Visualization

Modern software is a complex combination of many moving pieces and parts that often interact in sometimes unpredictable ways. Being able to visualize and model the containers and components of a software architecture effectively and document the code well is an essential step in any project. Traditional models and methods of modeling software architecture have attempted to accomplish this with some success. Traditional textual documentation, which involves writing detailed descriptions of the software architecture including its components and interactions, can be effective but does come with its disadvantages. As the architecture changes sometimes the documentation remains an afterthought and doesn't change with the system. It can be time consuming and its hard to visualize a complex system from a textual description.

Other methods such as UML Diagrams and Data Flow Diagrams (DFDs), provide visual representations of the software architecture and the flow of data within the system respectively. These capture a visual representation of the software architecture and provide high level insights into the design and architecture of the system, but they too come with their own sets of disadvantages. UML diagrams are very technical and are usually only useful to the implementers, designers, and architects working on the system. If a non-technical stakeholder needed to communicate about the system, they would find it difficult to look at such a diagram and effectively understand the relationships and meaning of the symbols without studying UML and the meaning of the various arrows and symbols used to denote relationships such as "Is-A" and 'Has-A'".

The Role of Software Architecture Visualization

The role of software architecture visualization, as employed by the C4 Model, is to facilitate the high level visualization of complex architectures so that all stakeholders, both technical and non-technical, can communicate effectively about the design and implementation of complex systems. It facilitates communication, collaboration, decision-making, and maintaining a shared understanding among stakeholders. As the system grows in complexity, the diagram and software visualization can grow in complexity as well, enabling a continued understanding of the system.

Brief History of the C4 Model

The C4 Model, as mentioned above was created by Simon Brown and first introduced in a blog post titled "The C4 software architecture model" in 2011. Since then, Brown has gone on to develop and speak extensively about the merits and benefits of the C4 model. The inspiration behind the C4 model lies in various sources, including the ideas of the 4+1 architecture model by Philippe Kruchten and the work of Ivar Jacobson on use case diagrams and the Unified Modeling Language (UML).

Key Advantages of the C4 Model

The C4 model aimed to be a lightweight approach to more traditional heavyweight approaches, such as the use of UML diagrams and large architectural diagrams. It focuses on simplicity and clarity for its visual representations. It accomplishes this by focusing on 4 abstraction levels, Context, Containers, Components, and Codes. These levels allow architects, developers, and other stakeholders to visualize the architecture at various levels of detail and to discern details that are relevant to them depending on the stated objective.

Limitations of the C4 Model

Despite its benefits, the C4 model does have some limitations and drawbacks. Some of these include limited detail at the code level, meaning it doesn't capture some fine-grained details when focusing closer on implementation details. It lacks a formal notation, so how it is interpreted can vary from one use case to another use case. It has trouble capturing dependency management between components or containers. While it can be depicted informally, more complex relationships might require additional documentation on the part of the user of the C4 model. Finally, it struggles with modeling dynamic behavior and focuses more on static aspects of modeling software visualization. Still, despite these drawbacks the C4 model offers a powerful set of abstractions and structure for defining a software architecture visualization framework for modeling complex systems.

Overview of the C4 Model

Hierarchy of the C4 Model

The C4 model comprises a hierarchal model of abstractions that centers around 4 levels, each with different intended audiences. These are the Context level, the Container level, the Component level, and the Code level. Each of these can be viewed as subcategories or nested within the each, the Context level gives way to the Container level, the Container level to the Component level, and the Component level to the Code level. As we get deeper and deeper into the abstractions, more and more details are revealed to us about the software architecture.

Context Level

At the top level we have the highest level of abstraction, the Context level. This level answers the question, "What is the system and how does it interface with the external world? How does it interact with actors, or users?" It provides a very high-level overview of the overall software architecture. This view of the system can be understood by both technical and non-technical stakeholders, providing valuable insights into the high-level architecture of the system.

Container Level

The next level of abstraction is the Container level, and shows the "containers" of the system. Containers are not to be thought of in the traditional sense of Docker containers, or lightweight, standalone, executable units that encapsulate software and its dependencies. Rather, containers here refer to anything like web severs, databases, mobile apps, desktop apps, filesystems, microservices, external APIs, etc. This level is still useful for both technical and non-technical stakeholders, but provides a more detailed look at the internals of the software architecture. Here we are able to define various interactions between the containers, for example, do the web servers read and write to the database or do they interface with the external APIs? How do the web servers interact with microservices? How does the database interact with the file system? All this is modeled at the Container level.

Component Level

As we dive deeper we start to get into more technical details and start to delve into the realm that is more exclusively for designers, architects, and the software implementers. Next we have the Component level, and this is what we get when we zoom in on a container and see the building blocks that it comprises of. For example, suppose we zoom in on an ETL Pipeline and look at the components that make it up. We might see an extraction component, a validation component, a transform component, a loading component, and a monitoring and logging component. This is the layer of services, controllers, repositories, etc. and provides a more granular view of the structure of the overall system architecture. This is especially important for the architects and the developers because it provides details necessary for developing the actual code.

Code Level

The final level of the C4 model is the code level and represents the individual code elements within the software architecture. These are the tiniest elements of the model, and represent specific classes, modules, interfaces, and other code elements. It provides an extremely detailed view of the architecture and is useful for developers implementing the code and answering questions like, "What does this class look like" and "How do I implement this interface" and "What is the relationship between these classes?"

Using Structurizr To Design a Machine Learning C4 Model

Structurizr, available at https://structurizr.com/ and also has a free plan available with a single workspace available, is a tool that allows you to build out a C4 model using its Structurizr DSL to outline the Containers, Components, and Contexts of the architecture. It is a valuable tool for designing and visualizing a software architecture. In this example we'll build out a simple Machine Learning Pipeline, I mean a very basic one, featuring an ETL Pipeline and a model that trains images that classifies it as a image of planet earth or not. To begin, lets define our work space.

Unfortunately, there is no syntax highlighting for structurizr as far as I can tell so I'll format it as best as I can.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

}

Here we define our Workspace. We give it a name, "ML Pipeline", and a description.

Next, we declare a model.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
    }

}

Next, we're declare our user and our software system.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
        user = person "ML Engineer"

        mlSystem = softwareSystem "ML System" {

        }

        user -> mlSystem "Uses"
    }

}

We can render the image and produce the following Context diagram:

We can define a few Containers now for our ETL engine and our ML engine and define a reliance on the ML engine to use the ETL engine.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
        user = person "ML Engineer"

        mlSystem = softwareSystem "ML System" {
            etlEngine = container "ETL Engine"
            mlEngine = container "ML Engine"
        }

        user -> mlSystem "Uses"

        mlEngine -> etlEngine "Loads data from"
    }

}

Rendering this gives us the following Container diagram:

Now we can define our components for the ETL Engine and ML Engine respectively, and their relationships.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
        user = person "ML Engineer"

        mlSystem = softwareSystem "ML System" {
            etlEngine = container "ETL Engine" {
                extractionComponent = component "Extraction Component"
                validationComponent = component "Validation Component"
                transformationComponent = component "Transformation Component"
                loadingComponent = component "Loading Component"
                monitoringAndLoggingComponent = component "Monitoring and Logging Component"

            }
            mlEngine = container "ML Engine" {
                dataLoadingComponent = component "Data Loading Component"
                featureExtractionComponent = component "Feature Extraction Component"
                modelLoadingComponent = component "Model Loading Component"
                modelTrainingComponent = component "Model Training Component"
                modelValidationComponent = component "Model Validation Component"
            }
        }

        user -> mlSystem "Uses"

        mlEngine -> etlEngine "Loads data from"

        extractionComponent -> validationComponent "Sends data to for validation"
        validationComponent -> transformationComponent "Sends data to for transformation"
        transformationComponent -> loadingComponent "Sends data to for loading"

        dataLoadingComponent -> featureExtractionComponent "Loads data into for feature extraction"
        featureExtractionComponent -> modelLoadingComponent "Sends data into model for loading"
        modelLoadingComponent -> modelTrainingComponent "Sends model into component for training"
        modelTrainingComponent -> modelValidationComponent "Sends model into component for validation"
    }

}

And we have two diagrams, the ETL Engine component diagram and the ML Engine component diagram

Now we have a fully realized and defined software architecture for our simple ML pipeline. The Structurizr tool doesn't actually allow you to dive into the Code level at all. In fact, Simon Brown recommends you not generate the Code level often because it changes so frequently and you often generate the code diagrams from your IDE rather than the other way around.

Conclusion

In this article we examined the complexities of software architecture and why software architecture visualization makes sense. In particular, we examined the C4 model for software architecture visualization and its merits and some of its drawbacks. We looks at its hierarchal nature and its four levels, Context, Container, Component, and Code, and explained what each one entails. We then provided an examination of the Structurizr DSL for documenting and architecting a software visualizing using a simple ETL and ML engine pipeline as an example.

Now that you have a better understanding of the C4 model, you have the necessary knowledge and tools at your disposal to use the C4 model to better visualize your complex software architectures. As always, please feel free to reply with your comments, questions, or concerns. I hope you enjoyed this article, and please feel free to read my other articles on Dev.to. Thank you for reading, and have a wonderful day!

References

C4 Model - The C4 Model Website, outlining the model in great detail.

Structurizr - The Structurizr website, where you can sign up for free and create your own software visualizations.

Creating a DSL In Python

Scott Rallya — Wed, 17 May 2023 18:31:46 +0000

Introduction

Domain-Specific Languages, or DSLs, are specialized programming languages designed to solve specialized problems within a specific domain. DSLs provide a concise and expressive syntax that is tailored to address the specific needs and challenges of a given problem. DSLs empower developers to express complex concepts and operations in an intuitive manner. Through using DSLs, developers can focus on the problem at hand without dealing with unnecessary details.

There are several advantages to creating a DSL in Python. Python's flexibility and expressiveness make it ideal for hosting a domain-specific language. It's rich ecosystem of libraries and tools provides a solid foundation for creating unique and specialize DSLs that can integrate seamlessly with existing codebases.

In this article, we will explore the process of implementing a simple DSL in Python. We're explore the core concepts, examine the necessary components, and guide you through the basic steps of implementing your own DSL. By the end, you will have a clear understanding of how to design, implement, and utilize DSLs to improve your Python applications!

Understanding DSLs

DSLs are specialized languages designed to address complex problems in a simple and intuitive manner for specific domains. They offer a concise syntax tailored to the particular needs of the application. This approach brings several advantages, such as improved readability, expressiveness, extendibility, and ease of use.

Internal vs External DSLs

There are two categories of DSLs, internal DSLs and external DSLs.

Internal DSLs

Internal DSLs are hosted within a language itself and leverage the syntactical features of its host language to define a specialized syntax. These DSLs leverage the flexibility and expressiveness of its host language and as a result are relatively easy to implement.

External DSLs

External DSLs, on the other hand, define their own syntax and grammar which stands alone from a host language. These require a dedicated parser and interpreter to properly handle the parsing and execution of the language. Advantages of external DSLs include providing more control over the design and more flexibility, but at the disadvantage of increase complexity of implementation.

Design Principles in Creating DSLs

There are certain design principles to follow when creating DSLs. Following these principles ensures that a DSL is intuitive to use, expressive, and efficient in its execution.

Simplicity: The DSL should define a clear and concise syntax that is easy to understand and fits within the problem domain.
Expressiveness: A DSL should strive to capture the problem domain well, defining the operations and concepts necessary to achieve the specified results.
Readability: DSLs should be readable by developers who write and maintain the codebase. Using meaningful keywords and consistent naming conventions ensures a clear and readable design.
Compositionality: Composition of DSL constructs allows building of complex and meaningful components from simpler ones, thus promoting code reuse.
Error Handling: Proper error handling is essential to ensure data integrity and inform users of your DSL when errors occur and how to respond to them.

Setting up Your DSL Environment

In order to implement your DSL, it is necessary to properly set up your environment to support its development. This involves choosing the appropriate libraries and tools that will enable you to create and execute your DSL.

Choosing Libraries

Python offers a number of libraries for creating DSLs. One option is ply.

ply is an implementation of the lex and yacc parsing tools for Python and enables the creation of lexers and parser in Python. It provides a clear way to define grammar rules and handle tokenization and parsing of your DSL code. By utilizing ply , you can easily define the structure and behavior of your DSL.

Installation

Provided you have pip installed, installing ply is as easy as running the following command:

pip install ply

With ply installed you are now ready to define the syntax for building and interpreting your DSL!

Define the DSL Syntax

Defining the syntax of the DSL you wish to implement is a critical step in the implementation. Identifying the language constructs, keywords, and expressions that will comprise your DSL is necessary in order to understand how you will implement it. By designing a clear syntax, you will enable your users to express their intensions clearly and accurately.

Identifying the Syntax

First identify the problem domain and understand the operation and concepts necessary to support your DSL. You will have to consider what actions, conditions, and calculations the user of your DSL will want to perform. For example, let us suppose we want to create a simple DSL for defining vectors, matrices, and carrying out simple operations such as vector addition and matrix multiplication. You might define the DSL as follows:

vector v1 = [1, 2, 3]
vector v2 = [4, 5, 6]
matrix m1 = [[1, 2], [3, 4]]
matrix m2 = [[5, 6], [7, 8]]

vector v3 = v1 + v2
matrix m3 = m1 * m2

This defines a very simple DSL for defining vectors and matrices and carrying out simple operations. We will enforce certain constraints in the design of our DSL using the numpy language to handle addition and multiplication operations and enforce shape constraints.

Creating Grammar Rules

Once you have a clear understanding of the syntax you want to utilize, you can proceed to define grammar rules in ply. These rules specify the structure and semantics of what constitutes a valid expression the DSL.

Using ply, we define token names and regular expressions to tokenize the incoming input stream. You would also define grammar rules that define how these tokens can be combined to form valid code expressions.

First, let us import the necessary libraries we will need to use.

import ply.lex as lex
import ply.yacc as yacc

import numpy as np

Next, we'll need to define our tokens

# Token definitions
tokens = (
        'IDENTIFIER',
        'NUMBER',
        'VECTOR_ID',
        'VECTOR',
        'MATRIX_ID',
        'MATRIX',
        'PLUS',
        'MULTIPLY',
        'LPAREN',
        'RPAREN', 
        'LBRACKET',
        'RBRACKET',
        'COMMA', 
        'EQUALS',
        'PRINT')

# Ignored characters
t_ignore = ' \t'

# Token regular expressions
t_PLUS = r'\+'
t_MULTIPLY = r'\*'
t_LPAREN = r'\('
t_RPAREN = r'\)'
t_EQUALS = r'='
t_LBRACKET = r'\['
t_RBRACKET = r'\]'
t_COMMA = r','

We declare a variable store for storing our variables. It is a simple dictionary at the moment, but can be expanded to be a more elaborate object as the DSL demands.

# Variables
variables = {}

Next we define some tokens for things like newlines, our print statement, our vector and matrix declarations, and our identifiers.

# Token definition for newline, print, vector and 
# matrix identifiers, generic identifiers, and numbers
def t_NEWLINE(t):
    r'\n+'
    t.lexer.lineno += t.value.count('\n')

def t_PRINT(t):
    r'print'
    t.type = 'PRINT'
    return t

def t_VECTOR_ID(t):
    r'vector\s+[a-zA-Z_][a-zA-Z_0-9]*'
    return t

def t_MATRIX_ID(t):
    r'matrix\s+[a-zA-Z_][a-zA-Z_0-9]*'
    return t

def t_IDENTIFIER(t):
    r'[a-zA-Z_][a-zA-Z_0-9]*'
    t.type = 'IDENTIFIER'
    return t

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

We want to be able to define a program for our DSL, essentially a series of statements that can be executed in sequence. We define the structure of the grammar as a comment and define how we want to handle the statement within the code. Here we just pass for now.

# ---- PROGRAM ----
def p_program(p):
    '''program : program statement
               | statement'''
    pass

Now we want to focus on how we handle parsing vectors. We need to be able to assign a vector to a variable and store that variable in the variable table.

First, let's focus on the assignment portion: vector v1 = <expression>

def p_statement_vector_assignment(p):
    'statement : VECTOR_ID EQUALS expression'
    variable_name = p[1].split()[1]
    variables[variable_name] = p[3]
    p[0] = (variable_name, p[3])

Here we define a statement that is defined as VECTOR_ID token, an EQUALS token, and an <expression>. We take the token at p[1], corresponding to VECTOR_ID, and split it to obtain the variable name. In the above example, that would be v1. We then assign variables[variable_name] to the value of <expression>. We then assign p[0] as a tuple of (variable name, value).

From here, we can define our vector expression as a series of functions. They are define as follows:

def p_vectordef(p):
    'expression : LBRACKET vector_values RBRACKET'
    p[0] = np.array(p[2])

def p_vector_values_single(p):
    'vector_values : NUMBER'
    p[0] = [p[1]]

def p_vector_values_multiple(p):
    'vector_values : NUMBER COMMA vector_values'
    p[0] = [p[1]] + p[3]

The overview of this code is as follows. p_vectordefis an expression that looks at statements in the form of [ <vector_values> ] . p_vector_values_single handles single values, in this case, just a number. Finally, p_vector_values_multiple handles multiple values, in the form of 1, 2, 3, 4. Notice how we reference vector_values from within P_vector_values_multiple? This allows it to recursively call itself until it terminates at a NUMBER token.

With this code in place, we can now parse and store statements in the form of vector v1 = [1, 2, 3].

Matrices are defined similarly, with a few additional functions to handle rows and row values.

# ----- MATRIX -----
def p_statement_matrix_assignment(p):
    'statement : MATRIX_ID EQUALS expression'
    variable_name = p[1].split()[1]
    variables[variable_name] = p[3]
    p[0] = (variable_name, p[3])

def p_expression_matrix(p):
    'expression : MATRIX'
    p[0] = p[1]

def p_matrix(p):
    'expression : LBRACKET matrix_rows RBRACKET'
    p[0] = np.array(p[2])

def p_matrix_rows_single(p):
    'matrix_rows : row'
    p[0] =[p[1]]

def p_matrix_rows_multiple(p):
    'matrix_rows : row COMMA matrix_rows'
    p[0] = [p[1]] + p[3]

def p_row(p):
    'row : LBRACKET row_values RBRACKET'
    p[0] = p[2]

def p_row_values(p):
    'row_values : NUMBER'
    p[0] = [p[1]]

def p_row_values_multiple(p):
    'row_values : NUMBER COMMA row_values'
    p[0] = [p[1]] + p[3]

# ----- END MATRIX -----

The above code allows us to parse matrices in the form of matrix m1 = [[a1, b1, c1,....,z1], [a2, b2, c2...,z2], ...., [an, bn, cn, ....zn]].

With this in place, we can now define some operations, such as addition and matrix multiplication. We can also define a print statement to print the values of our variables.

First, we have to define an expression for retrieving IDENTIFIERS from the variable table.

def p_expression_identifier(p):
    'expression : IDENTIFIER'
    variable_name = p[1]
    if variable_name in variables:
        p[0] = variables[variable_name]
    else:
        print(f"Error: Variable '{variable_name}' not in variable table")

Here if we encounter a IDENTIFIER token, we look to see if the variable identified by IDENTIFIER is in the variable identifier. If it is we retrieve its value and store it in p[0] else we print an error message.

Addition and multiplication are straight forward as well, with multiplication using numpy's matmul method:

def p_expression_add(p):
    'expression : expression PLUS expression'
    p[0] = p[1] + p[3]

def p_expression_multiply(p):
    'expression : expression MULTIPLY expression'
    p[0] = np.matmul(p[1],p[3])

Likewise, the print statement can be easily defined as well.

def p_statement_print(p):
    'statement : PRINT LPAREN IDENTIFIER RPAREN'
    variable_name = p[3]
    if variable_name in variables:
        print(variables[variable_name])
    else:
        print(f"Error: Variable '{variable_name}' not in variable table")

Here we parse statements in the form of print(<IDENTIFIER>). We retrieve IDENTIFIER and look it up in the variable table. If it is present, we print the value of that variable, otherwise, we print an error message.

We conclude with a simple error handler:

def p_error(p):
    print("Syntax error: ", p)

To run the program, we build the lexer and parser.

# Build the lexer and parser
lexer = lex.lex()
parser = yacc.yacc()

And define our DSL code:

# Parsing and executing DSL code
dsl_code = """
vector v1 = [1, 2, 3]
vector v2 = [4, 5, 6]
print(v1)
print(v2)

matrix m1 = [[1, 2], [3, 4], [5, 6]]
matrix m2 = [[5, 6, 7], [7, 8, 9]]

print(m1)
print(m2)

vector v3 = v1 + v2
matrix m3 = m1 * m2

print(v3)
print(m3)
"""

Finally, we can call the parse method on the parser object on dsl_code to see the output of our program.

parser.parse(dsl_code)

With that, we have a fully functionally DSL in Python

Conclusion

In this article, we explored domain-specific languages. We examined two types of DSLs, internal and DSL, as well as the criteria that defines a good DSL. Furthermore, we looked at the advantages of creating a DSL for domain specific problems. We then dived into the process of designing and implementing a DSL using the Ply librarying, which provides lexing and parsing capabilities in the Python language.

We begun by defining the tokens of our DSL. These form the basic building blocks of the language, such as numbers, identifiers, and keywords like vector and matrix. We leveraged regular expressions to define token rules and used Ply's lexer tok tokenize our sample DSL code.

We then proceeded to design our grammar rules to define the syntactical and semantic structure of our language. We created rules for declaring vectors and matrices and assigning them to variables. Next, we proceeded to define operations for performing addition and matrix multiplication on the variables we stored, as well as an operation for a print statement. Through parsing the code, we construct an abstract syntax tree (AST) that represents the structure of the DSL code.

By learning how to implement a DSL in Python and how to leverage the Ply library, you know have the necessary knowledge and tools to create your own domain specific languages. Whether it is for data engineering, rule-based systems, game design, or other use cases, a well-constructed DSL can greatly facilitate the development of software and applications, enhancing productivity and code expressiveness.

Now it's time to apply what you've learning and start exploring the possibilities of building your own DSLs! With these tools at your disposal, you have the flexibility and power to create powerful DSLs and improve the way you tackle complex problems across a wide variety of domains.

Thank you, and happy DSL development!

Full code available at: https://github.com/fractalis/devto-articles/blob/main/python-dsl/matrix-dsl.py

How to write a basic rule engine in Python

Scott Rallya — Mon, 15 May 2023 01:34:36 +0000

While there are many existing rules engines in Python, such as the amazing rule-engine framework, I thought it would be an interesting exercise to utilize ChatGPT to help me write my own basic rule engine from scratch. So, without further ado - I present to you PYROSE - PYthon Rule-based Operation System Engine.

The basic structure of our rule system is as follows. At the basic foundation of engine we have Facts. Facts are simple objects that represent information we want stored in our system and can contain any kind of information that is relevant to the design of the constraints by which our rules-engine operates. Let us begin by defining a very simple structure for our Facts

from typing import Any

class Fact:
    def __init__(self, **kwargs: Any):
        self.__dict__.update(kwargs)

Here we define a very simple object that can be instantiated with any number of keyword arguments to define our fact. We then update our object used the self.__dict__.update method to add attributes to our object corresponding to the keywords passed to the initializer. We can initialize a fact as follows,

person_fact = Fact(name="John Brown", age="35", occuptation="Software Developer")

Accessing a attribute is as simple using dot notation as if you were access a member or method on the object.

person_fact.age # Returns 35

Now that we have established our Fact object, we need to define a Condition by which to apply our Fact object. Our Conditions are very simple in design as well to start off with. The initializer takes two parameters, a name and an evaluation function. The evaluation function will be applied to a Fact and return a bool. It also contains a method, evaluate, which will take a Fact and return a bool, calling the objects evaluation_function on the Fact. Here is the outline of our Condition class:

from rule_engine.fact import Fact

from typing import Callable, Any, Dict, List

class Condition:
    def __init__(self, name: str, evaluation_function: Callable[[Fact], bool]):
        self.name = name
        self.eval_func = evaluation_function

    def evaluate(self, fact: Fact) -> bool:
        return self.eval_func(fact)

Following a similar suit, we can define our Action class. An Action class takes in a name parameter and a Callable parameter into its initializer. The Callable corresponds to the Action's execution function, what is executed if all the Conditions of the Rule are True. It also contains an execute method which calls the execute function on the given Fact, giving a None type as a result. Here is the definition of the Action class:


from typing import Callable, Any, Dict, List
from rule_engine.fact import Fact

class Action:

    def __init__(self, name: str, execution_function: Callable[[Fact], None]):
        self.name = name
        self.exec_func = execution_function

    def execute(self, fact: Fact) -> None:
        self.exec_func(fact)

So far we've defined our Fact, our Condition, and our Action. We can combine these together to form our Rule class. This will be the main driving force behind our Engine. The Rule class will initially be constructed with a single Action and Condition. Two methods, add_condition and add_action, will allow you to add additional conditions and actions to the Rule as you need.

Finally, a third method, evaluate, will take in a list of Facts. It defines a fact_generator which takes a list of conditions and a list of facts. For each fact, it maps each of the conditions' eval_func against the fact. It then reduces this list to a single boolean value, and if this value is true, we yield the fact.

We then call the fact_generator function, wrap it in a list to get the list of all True facts, and if the length is greater than 0, we iterate through the list of true facts. For each true fact, we iterate through a list of the rules actions and call the action's exec_func on the true fact.

The complete definition of the Rule class is as follows:


from rule_engine.condition import Condition
from rule_engine.action import Action
from rule_engine.fact import Fact

from typing import Any, List
from functools import reduce

class Rule:
    def __init__(self, condition: Condition, action: Action):
        self.conditions = [condition]
        self.actions = [action]

    def add_condition(self, condition: Condition) -> None:
        self.conditions.append(condition)

    def add_action(self, action: Action) -> None:
        self.actions.append(action)

    def evaluate(self, facts: List[Fact]) -> Any:
        def fact_generator(conditions: List[Condition], facts: List[Fact]):
            all_conditions_true = True
            for fact in facts:
                results = map(lambda condition: condition.eval_func(fact), conditions)
                all_conditions_true = reduce(lambda x, y: x and y, results)

                if all_conditions_true:
                    yield fact

        true_facts = list(fact_generator(self.conditions, facts))

        if len(true_facts) > 0:
            for fact in true_facts:
                for action in self.actions:
                    action.exec_func(fact)

You can use the rule engine as follows:


from rule_engine.fact import Fact
from rule_engine.condition import Condition
from rule_engine.action import Action
from rule_engine.rule import Rule

age_cond = Condition(name="Age>=21", evaluation_function=lambda fact: fact.age >= 21)
occupation_cond = Condition(name="Occupation==Software Developer", evaluation_function=lambda fact: fact.occupation == "Software Developer")

print_action = Action(name="Print Fact", execution_function=lambda fact: print("Name: {} Age: {} Occupation: {}".format(fact.name, fact.age, fact.occupation)))

john = Fact(age=25,name="John Brown", occupation="Software Developer")
sarah = Fact(age=35,name="Sarah Purple", occupation="Data Engineer")
barry = Fact(age=27, name="Barry White", occupation="Software Developer")

rule = Rule(condition=age_cond, action=print_action)
rule.add_condition(occupation_cond)

rule.evaluate([john, sarah, barry])

All in all, this is a very basic demo of a very simplistic Rules engine. However, I think it can definitely be built and improved upon in any number of ways, so I hope that you enjoyed reading and using what you learned in your own projects. If you have any questions, comments, suggestions or ideas, please feel free to reach out to me. Thank you for reading.

Acknowledgments:

Thanks to John for catching some unnecessary code duplication! Catch out John's articles.