DEV Community: Scott Rallya

The Research Foundation: What I'm Building and Why the Methodology Is the Message

Scott Rallya — Mon, 15 Jun 2026 15:02:59 +0000

As developers, we build systems that adapt to users. But what does it look like when the system adapts to the learner's cognitive profile — not just their preferences or pace, but their actual neurological constraints?

I've been building a self-directed PhD curriculum in Applied AI/ML and logging every session: energy levels, focus quality, what clicked, what didn't, what the AI did that made the difference. The goal is a five-year longitudinal dataset of one neurodiverse learner using AI as primary educational scaffolding.

Here's the hypothesis driving everything — and what I learned when I actually started studying.

Here's What I'm Actually Trying to Prove

The other night I sat down to study linear algebra at what I'd call energy level 2 out of 5 — functioning but effortful, like moving through moderate resistance. Not a good day to learn anything. I opened my curriculum, started reading about fields and vector spaces, and something happened that I didn't expect.

I generated eight conceptual questions.

Not exercises I'd been assigned. Not prompts from the material. Questions I produced autonomously, across multiple topics, while cross-referencing two documents — at low energy, late in the evening, after a full day of work. One of those questions was meta: is this worth recording?

The fact that I even asked that second question is, I'm realizing, part of the research.

The hypothesis

I have a PhD thesis. Here's the central claim:

Augmented intelligence — AI operating as a bidirectional cognitive extension, calibrated to a learner's individual personality and cognitive profile — enables learning outcomes that traditional institutional models of education failed to produce. For neurodiverse learners whose cognitive profiles create significant friction at the task-initiation and conception-to-execution interface, this augmentation doesn't merely improve outcomes: it enables doctoral-level engagement that was previously structurally inaccessible.

That's the formal version. Here's what it means in plain language:

I've tried to learn this stuff before. Online courses. Graduate programs. I'd sit down, follow along, execute the steps, pass the tests — and retain almost nothing. I couldn't explain what I'd done. I could run Gaussian elimination but couldn't tell you what it meant. The knowledge lived in my fingers, not my head.

The programs called this my fault. The solution they offered was "spend more time reading." I did. Same result. Eventually I stopped.

What I'm arguing is that this wasn't a learning failure. It was an interface failure. The structure of institutional education — passive content delivery, scheduled assessments, no adaptation to cognitive state, no scaffolding for task initiation — is structurally incompatible with how my mind works. And that the thing that's changing this isn't willpower or more effort. It's AI.

Not AI as a search engine. Not AI as an autocomplete tool. AI as something more like an externalized cognitive scaffold — one that holds the thread when my attention drifts, externalizes the working memory I can't reliably maintain, and closes the gap between what I can conceive and what I can actually execute.

The theory comes from Clark & Chalmers' Extended Mind Thesis (1998): the tools we use to think with are continuous with our cognitive systems, not separate from them. My glasses aren't separate from my vision; they're infrastructure for it. The AI isn't separate from my thinking; it's infrastructure for it.

What I'm trying to show, over five years of documented study, is that this infrastructure enables doctoral-level engagement that institutional structures — for this learner, with this profile — did not.

The methodology is the message

Here's the part that I keep finding strange and fascinating: the way I'm producing evidence for this hypothesis is by doing the thing the hypothesis describes.

Every session is logged. Energy level, focus quality, mood, cognitive load. Every concept I struggled with, every question I generated, every moment of confusion and resolution. The questions I asked at energy:2 late on a Thursday evening are data. The fact that I generated eight of them without being prompted is data. The meta-question — is this worth recording? — is data about the research process itself.

The formal term for this is autoethnography: using rigorous first-person documentation as research methodology. I'm the researcher and the primary research subject simultaneously. The data I'm generating is a longitudinal record of one neurodiverse adult learner using AI as a primary educational tool — over what will be a five-plus year span.

To my knowledge, this kind of longitudinal dataset doesn't exist in the academic literature on AI education. That's part of what makes it worth doing.

The strange part — the part I find genuinely interesting — is that the research process itself demonstrates the hypothesis. I figured out the hypothesis through a Socratic dialogue session with an AI, working through my own cognitive experience systematically until a claim emerged. The derivation process is data. The hypothesis and the method are the same object.

What studying at energy level 2 actually revealed

Back to that Thursday session. Here's what happened, and why I think it's relevant to the research.

I was working through the foundations of linear algebra — specifically the concept of a field, which is the mathematical structure that underlies the number systems we use. Fields have eleven formal axioms. I didn't try to memorize them. Instead I worked on building one generative question: what can I do with these numbers, and can I always undo it?

That question — can I always undo it? — reconstructs most of what you need to know. Real numbers form a field because you can always divide (except by zero). Integers don't because dividing two integers often exits the set: 2 ÷ 3 isn't an integer. One question. Two examples. The structure follows.

But here's what I noticed while doing this: I would understand something completely while reading it, and then it would be gone the moment my attention moved. Not fuzzy — gone. I'd have to go back.

I've experienced this my whole life but never had a framework for it. What I know now, after logging it and thinking about it more carefully, is that there are two different things happening: processing and encoding. I can process content well — comprehension in the moment is fine. The transfer from working memory to long-term storage is where things break down. That transfer requires executive function support — sustained attention, active elaboration — and executive function is exactly what ADHD impairs.

So I'm not a bad learner. I have a specific interface incompatibility between my cognitive profile and passive content delivery. That's a design problem, not a character flaw.

And then something else came up, which I didn't expect at all.

The visualization thing

Someone asked me to describe what it looks like when I try to visualize a mathematical structure — say, a vector space. What I described was: something between a visual translation and a textual representation. Blurry. Non-static. It forms and then immediately dissipates. Not a picture, not words, but something in between that never quite stabilizes.

Memory palace techniques — where you build a mental house and attach concepts to rooms — don't work for me. The structure won't stay still long enough to attach anything to.

This turns out to be a recognized phenomenon. There's a spectrum from vivid voluntary mental imagery (hyperphantasia) to essentially no voluntary mental imagery (aphantasia). Most people assume everyone visualizes the same way they do. They don't.

The interesting part for the research: spatial-relational intuition and visual imagery are separable. You can navigate the relationships between abstract structures without forming a crisp internal picture of them. I described functions as "floating boxes with inputs and outputs" — that's spatial-relational thinking, not visualization. The channel exists; it just doesn't produce photographs.

What this means for curriculum design: the default assumption in most math education is that learners can build and manipulate internal visual representations. Diagrams in textbooks are meant to help you generate your own internal picture. If that generation isn't available, the diagram has to be the representation — exoskeleton rather than scaffold. The curriculum needs to build around that, not through it.

I'm logging all of this. Not because I have answers yet, but because the question of how cognitive profile shapes learning trajectory — when that trajectory is mediated by AI — is exactly what RQ5 is asking.

Why any of this matters for what I'm building

Hearth & Code isn't primarily a curriculum company. It's a research project that happens to be producing a platform.

The research question that drives everything: can AI calibrated to an individual learner's cognitive and personality profile enable learning outcomes that institutional structures failed to produce?

The answer I'm betting on is yes. The platform is how I test it. The PhD is the research log.

Every design decision in H&C comes back to this: not "what would make this feel like a good educational app," but "what does the evidence from a learner actually navigating this material actually show needs to happen?"

Right now the evidence is N=1. Mine. That's the point of starting here — to build the measurement infrastructure, establish the methodology, and generate the first data before trying to generalize.

The five-year plan is to have enough longitudinal data to make claims with research grounding, and a platform that has been designed around evidence rather than intuition. Intuition informed by lived experience is how I started. Evidence is how I'll finish.

What's next

The first module — Linear Algebra for ML — is in progress. The content is generated, the first session is logged, and the Checkpoint 1 computational exercises are the immediate next step. Python and NumPy, not paper. (Paper doesn't work for me. Now I know why, specifically, and it's in the research log.)

Phase B of the curriculum generation is queued: modules M02 through M18, covering multivariate calculus, probability, information theory, computability theory, algorithms, programming language theory, type theory, systems, databases, and security. All of it will be generated with the same learner-centered design principles: concrete worked examples, specific numbers, computational exercises, and no notation without explanation.

The PhD repo is public: github.com/hearthandcode/phd-applied-ai

The session logs are in there. The module content is in there. The thesis is in there, unfinished and honest about it. If you're curious what self-directed doctoral study actually looks like when logged rigorously and built in public, that's the place to look.

And if any of what I described about visualization, or the processing-encoding gap, or the interface incompatibility with conventional education — if any of that resonates with your own experience — I'd genuinely like to hear about it. That's also data.

Hearth & Code is building a gamified, adaptive, AI-native educational platform. The PhD curriculum is the research foundation. Both are built in public. If you're interested in where education is going when AI is the infrastructure rather than the tool, subscribe.

I designed a 67-module AI/ML doctoral curriculum for my ADHD brain — here's the architecture

Scott Rallya — Sat, 13 Jun 2026 22:28:51 +0000

I'm a software engineer who took a 6-year break from the field for mental health reasons. When I came back, I found an industry that had been transformed by AI — and a skills gap I needed to close before I could build what I actually wanted to build.

So I built a doctoral-level self-study curriculum. Here's what that actually looks like.

The curriculum

phd-applied-ai is 67 modules across 6 phases:

Phase 0 — Math foundations: Linear algebra, calculus, probability, statistics, information theory
Phase 1 — CS fundamentals: Computability, complexity, algorithms, type theory, distributed systems
Phase 2 — ML foundations: Classical ML through transformers, reinforcement learning, graphical models
Phase 3 — Advanced AI: LLMs, fine-tuning, diffusion, multi-agent systems, interpretability, safety
Phase 4 — Specialized research: AI in education, adaptive learning, ethics, affective computing, neurodiversity
Phase 5 — Thesis and defense

Every module includes: a theory digest at doctoral level, a working code project, a mastery rubric, and a dated study log. The whole thing is CC BY-NC-SA and live on GitHub now.

The par scoring system

Traditional grades and deadlines are incompatible with variable-energy learning. I replaced them with par scores — a borrowing from golf.

Each module has defined par criteria: qualitative standards that describe what "good enough to move on" looks like. You hit par when you hit it. Over par means you went deep. Under par means you moved fast. Neither is a failure. There's no "you're behind" state.

I have ADHD and bipolar disorder. Deadline-based pacing has never matched how my brain actually operates. The par system makes that mismatch structurally irrelevant.

The advisor committee

Five AI advisor personas, each calibrated to a distinct OCEAN personality profile:

Dr. Chen (ML Theory) — high Conscientiousness, high Openness; demands formal precision
Dr. Kowalski (Systems/Architecture) — high Agreeableness, low Neuroticism; asks what breaks at scale
Dr. Williams (Education/Pedagogy) — high Agreeableness, high Openness; asks if this serves the learner
Dr. Okonkwo (AI Ethics) — high Openness, low Agreeableness; asks whose values are encoded
Dr. Patel (Applied AI/Industry) — high Extraversion, low Conscientiousness; asks if it ships

Running a module debrief means defending your understanding against five different critical lenses simultaneously. The adversarial diversity surfaces things passive review misses — a high-Conscientiousness reviewer catches methodological gaps that a high-Openness reviewer would skip past.

The personas are model-agnostic prompts in the repo. Drop any PERSONA.md into Claude, GPT-4o, or a local Ollama model as a system prompt.

The research angle

I'll be logging energy, focus, and mood for every study session over the next 5+ years. The AIED research community has almost no longitudinal data on how neurodivergent learners navigate self-directed doctoral education. Mine will be public — released as the curriculum progresses, committed to the repo as session logs.

The curriculum is also the proof-of-concept for Hearth & Code, an adaptive AI-native learning platform I'm building. Every module I study is a prototype of what the platform will eventually offer. The data I collect as a learner is the early research base.

Three design decisions I'd like feedback on

Module sequencing in Phase 0–1 — I put statistical foundations depth (M02–M04) before gradient-based methods. Does that sequencing look right to people who've studied this material?
Thesis framing — I'm treating this as design research on a neurodivergent adaptive learning system, not traditional ML benchmarking. Methodological thoughts welcome.
OCEAN advisor model — any prior art on personality-modeled AI tutors or adversarial committee simulations?

Repo: github.com/hearthandcode/phd-applied-ai
Blog: hearthandcode.substack.com

Mastering Software Architecture Visualization with the C4 Model

Scott Rallya — Fri, 19 May 2023 03:56:07 +0000

Disclaimer

Note, the majority of this article was written by me but I did employ AI tools to help me generate the outline and some of the major talking points. I do appreciate the use of AI tools to help me bring you great content like this, and I hope you don't mind me employing such tools to help me convey complex ideas like software architecture visualization in a simple, easy-to-understand manner. Thank you for your consideration.

Introduction

Software architecture is difficult, and visualizing complex architectures can be a challenging process. In this article we'll examine the C4 Model for software architecture visualization and how it simplifies the process of visualizing complex architectures. We'll discuss a history of the model, its benefits and limitations, and we will examine the 4'Cs, Context, Container, Component, and Code, and explain the hierarchal nature of the model. Next, we will explore the use of the Structurizr DSL for creating C4 model diagrams. We will follow up with a conclusion and a recap of what we discussed.

Introduction to the C4 Model: What it is and why it's beneficial?

The C4 model is a software architecture visualization model invented by Simon Brown, a software architect and author, as a lightweight and scalable approach to communicate software architecture effectively. The model emphasizes a hierarchal view of the architecture at four levels: context, containers, components, and code.

The Context level provides a birds-eye view of the system, giving an overview of system boundaries, external actors (users), and their interactions with the system.

The Container level focuses on the major architectural building blocks of the system. This can include web servers, databases, desktop apps, microservices, or external services such as APIs. It communicates how these systems interact with each other.

The Component level zooms into a container, showing the internal structure and building blocks of that container and how those building blocks relate to one another. It focuses on the key responsibilities within the container and how they interact with one another.

The Code level provides a detailed view of the code structure of the component, showing details such as the classes, modules, and packages. This aids developers in understanding the actual implementation of these components.

The C4 model strives to promote simplicity and clarity in visualization, using a small set of symbols and notations to represent the different elements. This allows for readability of the diagrams. As a communication tool, the C4 model enables all stakeholders, not just developers, to use the model to understand and communicate about the structure of the software system. Project managers and product owners can use the Context and Container levels to discuss the high-level overview of the system, while Software Architects, Designers, and Implementers can use the Component and Code level to focus on the low-level details.

In addition, the C4 model allows for documentation support, creating documentation alongside the code and helping to capture the architecture effectively. It can easily be updated as the architecture and design of the system changes, thereby ensuring that the documentation always reflects the current state of the system.

Why Do We Need the C4 Model

The Challenge of Software Architecture Visualization

Modern software is a complex combination of many moving pieces and parts that often interact in sometimes unpredictable ways. Being able to visualize and model the containers and components of a software architecture effectively and document the code well is an essential step in any project. Traditional models and methods of modeling software architecture have attempted to accomplish this with some success. Traditional textual documentation, which involves writing detailed descriptions of the software architecture including its components and interactions, can be effective but does come with its disadvantages. As the architecture changes sometimes the documentation remains an afterthought and doesn't change with the system. It can be time consuming and its hard to visualize a complex system from a textual description.

Other methods such as UML Diagrams and Data Flow Diagrams (DFDs), provide visual representations of the software architecture and the flow of data within the system respectively. These capture a visual representation of the software architecture and provide high level insights into the design and architecture of the system, but they too come with their own sets of disadvantages. UML diagrams are very technical and are usually only useful to the implementers, designers, and architects working on the system. If a non-technical stakeholder needed to communicate about the system, they would find it difficult to look at such a diagram and effectively understand the relationships and meaning of the symbols without studying UML and the meaning of the various arrows and symbols used to denote relationships such as "Is-A" and 'Has-A'".

The Role of Software Architecture Visualization

The role of software architecture visualization, as employed by the C4 Model, is to facilitate the high level visualization of complex architectures so that all stakeholders, both technical and non-technical, can communicate effectively about the design and implementation of complex systems. It facilitates communication, collaboration, decision-making, and maintaining a shared understanding among stakeholders. As the system grows in complexity, the diagram and software visualization can grow in complexity as well, enabling a continued understanding of the system.

Brief History of the C4 Model

The C4 Model, as mentioned above was created by Simon Brown and first introduced in a blog post titled "The C4 software architecture model" in 2011. Since then, Brown has gone on to develop and speak extensively about the merits and benefits of the C4 model. The inspiration behind the C4 model lies in various sources, including the ideas of the 4+1 architecture model by Philippe Kruchten and the work of Ivar Jacobson on use case diagrams and the Unified Modeling Language (UML).

Key Advantages of the C4 Model

The C4 model aimed to be a lightweight approach to more traditional heavyweight approaches, such as the use of UML diagrams and large architectural diagrams. It focuses on simplicity and clarity for its visual representations. It accomplishes this by focusing on 4 abstraction levels, Context, Containers, Components, and Codes. These levels allow architects, developers, and other stakeholders to visualize the architecture at various levels of detail and to discern details that are relevant to them depending on the stated objective.

Limitations of the C4 Model

Despite its benefits, the C4 model does have some limitations and drawbacks. Some of these include limited detail at the code level, meaning it doesn't capture some fine-grained details when focusing closer on implementation details. It lacks a formal notation, so how it is interpreted can vary from one use case to another use case. It has trouble capturing dependency management between components or containers. While it can be depicted informally, more complex relationships might require additional documentation on the part of the user of the C4 model. Finally, it struggles with modeling dynamic behavior and focuses more on static aspects of modeling software visualization. Still, despite these drawbacks the C4 model offers a powerful set of abstractions and structure for defining a software architecture visualization framework for modeling complex systems.

Overview of the C4 Model

Hierarchy of the C4 Model

The C4 model comprises a hierarchal model of abstractions that centers around 4 levels, each with different intended audiences. These are the Context level, the Container level, the Component level, and the Code level. Each of these can be viewed as subcategories or nested within the each, the Context level gives way to the Container level, the Container level to the Component level, and the Component level to the Code level. As we get deeper and deeper into the abstractions, more and more details are revealed to us about the software architecture.

Context Level

At the top level we have the highest level of abstraction, the Context level. This level answers the question, "What is the system and how does it interface with the external world? How does it interact with actors, or users?" It provides a very high-level overview of the overall software architecture. This view of the system can be understood by both technical and non-technical stakeholders, providing valuable insights into the high-level architecture of the system.

Container Level

The next level of abstraction is the Container level, and shows the "containers" of the system. Containers are not to be thought of in the traditional sense of Docker containers, or lightweight, standalone, executable units that encapsulate software and its dependencies. Rather, containers here refer to anything like web severs, databases, mobile apps, desktop apps, filesystems, microservices, external APIs, etc. This level is still useful for both technical and non-technical stakeholders, but provides a more detailed look at the internals of the software architecture. Here we are able to define various interactions between the containers, for example, do the web servers read and write to the database or do they interface with the external APIs? How do the web servers interact with microservices? How does the database interact with the file system? All this is modeled at the Container level.

Component Level

As we dive deeper we start to get into more technical details and start to delve into the realm that is more exclusively for designers, architects, and the software implementers. Next we have the Component level, and this is what we get when we zoom in on a container and see the building blocks that it comprises of. For example, suppose we zoom in on an ETL Pipeline and look at the components that make it up. We might see an extraction component, a validation component, a transform component, a loading component, and a monitoring and logging component. This is the layer of services, controllers, repositories, etc. and provides a more granular view of the structure of the overall system architecture. This is especially important for the architects and the developers because it provides details necessary for developing the actual code.

Code Level

The final level of the C4 model is the code level and represents the individual code elements within the software architecture. These are the tiniest elements of the model, and represent specific classes, modules, interfaces, and other code elements. It provides an extremely detailed view of the architecture and is useful for developers implementing the code and answering questions like, "What does this class look like" and "How do I implement this interface" and "What is the relationship between these classes?"

Using Structurizr To Design a Machine Learning C4 Model

Structurizr, available at https://structurizr.com/ and also has a free plan available with a single workspace available, is a tool that allows you to build out a C4 model using its Structurizr DSL to outline the Containers, Components, and Contexts of the architecture. It is a valuable tool for designing and visualizing a software architecture. In this example we'll build out a simple Machine Learning Pipeline, I mean a very basic one, featuring an ETL Pipeline and a model that trains images that classifies it as a image of planet earth or not. To begin, lets define our work space.

Unfortunately, there is no syntax highlighting for structurizr as far as I can tell so I'll format it as best as I can.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

}

Here we define our Workspace. We give it a name, "ML Pipeline", and a description.

Next, we declare a model.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
    }

}

Next, we're declare our user and our software system.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
        user = person "ML Engineer"

        mlSystem = softwareSystem "ML System" {

        }

        user -> mlSystem "Uses"
    }

}

We can render the image and produce the following Context diagram:

We can define a few Containers now for our ETL engine and our ML engine and define a reliance on the ML engine to use the ETL engine.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
        user = person "ML Engineer"

        mlSystem = softwareSystem "ML System" {
            etlEngine = container "ETL Engine"
            mlEngine = container "ML Engine"
        }

        user -> mlSystem "Uses"

        mlEngine -> etlEngine "Loads data from"
    }

}

Rendering this gives us the following Container diagram:

Now we can define our components for the ETL Engine and ML Engine respectively, and their relationships.



workspace "ML Pipeline" "Machine Learning Pipeline Example" {

    model {
        user = person "ML Engineer"

        mlSystem = softwareSystem "ML System" {
            etlEngine = container "ETL Engine" {
                extractionComponent = component "Extraction Component"
                validationComponent = component "Validation Component"
                transformationComponent = component "Transformation Component"
                loadingComponent = component "Loading Component"
                monitoringAndLoggingComponent = component "Monitoring and Logging Component"

            }
            mlEngine = container "ML Engine" {
                dataLoadingComponent = component "Data Loading Component"
                featureExtractionComponent = component "Feature Extraction Component"
                modelLoadingComponent = component "Model Loading Component"
                modelTrainingComponent = component "Model Training Component"
                modelValidationComponent = component "Model Validation Component"
            }
        }

        user -> mlSystem "Uses"

        mlEngine -> etlEngine "Loads data from"

        extractionComponent -> validationComponent "Sends data to for validation"
        validationComponent -> transformationComponent "Sends data to for transformation"
        transformationComponent -> loadingComponent "Sends data to for loading"

        dataLoadingComponent -> featureExtractionComponent "Loads data into for feature extraction"
        featureExtractionComponent -> modelLoadingComponent "Sends data into model for loading"
        modelLoadingComponent -> modelTrainingComponent "Sends model into component for training"
        modelTrainingComponent -> modelValidationComponent "Sends model into component for validation"
    }

}

And we have two diagrams, the ETL Engine component diagram and the ML Engine component diagram

Now we have a fully realized and defined software architecture for our simple ML pipeline. The Structurizr tool doesn't actually allow you to dive into the Code level at all. In fact, Simon Brown recommends you not generate the Code level often because it changes so frequently and you often generate the code diagrams from your IDE rather than the other way around.

Conclusion

In this article we examined the complexities of software architecture and why software architecture visualization makes sense. In particular, we examined the C4 model for software architecture visualization and its merits and some of its drawbacks. We looks at its hierarchal nature and its four levels, Context, Container, Component, and Code, and explained what each one entails. We then provided an examination of the Structurizr DSL for documenting and architecting a software visualizing using a simple ETL and ML engine pipeline as an example.

Now that you have a better understanding of the C4 model, you have the necessary knowledge and tools at your disposal to use the C4 model to better visualize your complex software architectures. As always, please feel free to reply with your comments, questions, or concerns. I hope you enjoyed this article, and please feel free to read my other articles on Dev.to. Thank you for reading, and have a wonderful day!

References

C4 Model - The C4 Model Website, outlining the model in great detail.

Structurizr - The Structurizr website, where you can sign up for free and create your own software visualizations.

Creating a DSL In Python

Scott Rallya — Wed, 17 May 2023 18:31:46 +0000

Introduction

Domain-Specific Languages, or DSLs, are specialized programming languages designed to solve specialized problems within a specific domain. DSLs provide a concise and expressive syntax that is tailored to address the specific needs and challenges of a given problem. DSLs empower developers to express complex concepts and operations in an intuitive manner. Through using DSLs, developers can focus on the problem at hand without dealing with unnecessary details.

There are several advantages to creating a DSL in Python. Python's flexibility and expressiveness make it ideal for hosting a domain-specific language. It's rich ecosystem of libraries and tools provides a solid foundation for creating unique and specialize DSLs that can integrate seamlessly with existing codebases.

In this article, we will explore the process of implementing a simple DSL in Python. We're explore the core concepts, examine the necessary components, and guide you through the basic steps of implementing your own DSL. By the end, you will have a clear understanding of how to design, implement, and utilize DSLs to improve your Python applications!

Understanding DSLs

DSLs are specialized languages designed to address complex problems in a simple and intuitive manner for specific domains. They offer a concise syntax tailored to the particular needs of the application. This approach brings several advantages, such as improved readability, expressiveness, extendibility, and ease of use.

Internal vs External DSLs

There are two categories of DSLs, internal DSLs and external DSLs.

Internal DSLs

Internal DSLs are hosted within a language itself and leverage the syntactical features of its host language to define a specialized syntax. These DSLs leverage the flexibility and expressiveness of its host language and as a result are relatively easy to implement.

External DSLs

External DSLs, on the other hand, define their own syntax and grammar which stands alone from a host language. These require a dedicated parser and interpreter to properly handle the parsing and execution of the language. Advantages of external DSLs include providing more control over the design and more flexibility, but at the disadvantage of increase complexity of implementation.

Design Principles in Creating DSLs

There are certain design principles to follow when creating DSLs. Following these principles ensures that a DSL is intuitive to use, expressive, and efficient in its execution.

Simplicity: The DSL should define a clear and concise syntax that is easy to understand and fits within the problem domain.
Expressiveness: A DSL should strive to capture the problem domain well, defining the operations and concepts necessary to achieve the specified results.
Readability: DSLs should be readable by developers who write and maintain the codebase. Using meaningful keywords and consistent naming conventions ensures a clear and readable design.
Compositionality: Composition of DSL constructs allows building of complex and meaningful components from simpler ones, thus promoting code reuse.
Error Handling: Proper error handling is essential to ensure data integrity and inform users of your DSL when errors occur and how to respond to them.

Setting up Your DSL Environment

In order to implement your DSL, it is necessary to properly set up your environment to support its development. This involves choosing the appropriate libraries and tools that will enable you to create and execute your DSL.

Choosing Libraries

Python offers a number of libraries for creating DSLs. One option is ply.

ply is an implementation of the lex and yacc parsing tools for Python and enables the creation of lexers and parser in Python. It provides a clear way to define grammar rules and handle tokenization and parsing of your DSL code. By utilizing ply , you can easily define the structure and behavior of your DSL.

Installation

Provided you have pip installed, installing ply is as easy as running the following command:

pip install ply

With ply installed you are now ready to define the syntax for building and interpreting your DSL!

Define the DSL Syntax

Defining the syntax of the DSL you wish to implement is a critical step in the implementation. Identifying the language constructs, keywords, and expressions that will comprise your DSL is necessary in order to understand how you will implement it. By designing a clear syntax, you will enable your users to express their intensions clearly and accurately.

Identifying the Syntax

First identify the problem domain and understand the operation and concepts necessary to support your DSL. You will have to consider what actions, conditions, and calculations the user of your DSL will want to perform. For example, let us suppose we want to create a simple DSL for defining vectors, matrices, and carrying out simple operations such as vector addition and matrix multiplication. You might define the DSL as follows:

vector v1 = [1, 2, 3]
vector v2 = [4, 5, 6]
matrix m1 = [[1, 2], [3, 4]]
matrix m2 = [[5, 6], [7, 8]]

vector v3 = v1 + v2
matrix m3 = m1 * m2

This defines a very simple DSL for defining vectors and matrices and carrying out simple operations. We will enforce certain constraints in the design of our DSL using the numpy language to handle addition and multiplication operations and enforce shape constraints.

Creating Grammar Rules

Once you have a clear understanding of the syntax you want to utilize, you can proceed to define grammar rules in ply. These rules specify the structure and semantics of what constitutes a valid expression the DSL.

Using ply, we define token names and regular expressions to tokenize the incoming input stream. You would also define grammar rules that define how these tokens can be combined to form valid code expressions.

First, let us import the necessary libraries we will need to use.

import ply.lex as lex
import ply.yacc as yacc

import numpy as np

Next, we'll need to define our tokens

# Token definitions
tokens = (
        'IDENTIFIER',
        'NUMBER',
        'VECTOR_ID',
        'VECTOR',
        'MATRIX_ID',
        'MATRIX',
        'PLUS',
        'MULTIPLY',
        'LPAREN',
        'RPAREN', 
        'LBRACKET',
        'RBRACKET',
        'COMMA', 
        'EQUALS',
        'PRINT')

# Ignored characters
t_ignore = ' \t'

# Token regular expressions
t_PLUS = r'\+'
t_MULTIPLY = r'\*'
t_LPAREN = r'\('
t_RPAREN = r'\)'
t_EQUALS = r'='
t_LBRACKET = r'\['
t_RBRACKET = r'\]'
t_COMMA = r','

We declare a variable store for storing our variables. It is a simple dictionary at the moment, but can be expanded to be a more elaborate object as the DSL demands.

# Variables
variables = {}

Next we define some tokens for things like newlines, our print statement, our vector and matrix declarations, and our identifiers.

# Token definition for newline, print, vector and 
# matrix identifiers, generic identifiers, and numbers
def t_NEWLINE(t):
    r'\n+'
    t.lexer.lineno += t.value.count('\n')

def t_PRINT(t):
    r'print'
    t.type = 'PRINT'
    return t

def t_VECTOR_ID(t):
    r'vector\s+[a-zA-Z_][a-zA-Z_0-9]*'
    return t

def t_MATRIX_ID(t):
    r'matrix\s+[a-zA-Z_][a-zA-Z_0-9]*'
    return t

def t_IDENTIFIER(t):
    r'[a-zA-Z_][a-zA-Z_0-9]*'
    t.type = 'IDENTIFIER'
    return t

def t_NUMBER(t):
    r'\d+'
    t.value = int(t.value)
    return t

We want to be able to define a program for our DSL, essentially a series of statements that can be executed in sequence. We define the structure of the grammar as a comment and define how we want to handle the statement within the code. Here we just pass for now.

# ---- PROGRAM ----
def p_program(p):
    '''program : program statement
               | statement'''
    pass

Now we want to focus on how we handle parsing vectors. We need to be able to assign a vector to a variable and store that variable in the variable table.

First, let's focus on the assignment portion: vector v1 = <expression>

def p_statement_vector_assignment(p):
    'statement : VECTOR_ID EQUALS expression'
    variable_name = p[1].split()[1]
    variables[variable_name] = p[3]
    p[0] = (variable_name, p[3])

Here we define a statement that is defined as VECTOR_ID token, an EQUALS token, and an <expression>. We take the token at p[1], corresponding to VECTOR_ID, and split it to obtain the variable name. In the above example, that would be v1. We then assign variables[variable_name] to the value of <expression>. We then assign p[0] as a tuple of (variable name, value).

From here, we can define our vector expression as a series of functions. They are define as follows:

def p_vectordef(p):
    'expression : LBRACKET vector_values RBRACKET'
    p[0] = np.array(p[2])

def p_vector_values_single(p):
    'vector_values : NUMBER'
    p[0] = [p[1]]

def p_vector_values_multiple(p):
    'vector_values : NUMBER COMMA vector_values'
    p[0] = [p[1]] + p[3]

The overview of this code is as follows. p_vectordefis an expression that looks at statements in the form of [ <vector_values> ] . p_vector_values_single handles single values, in this case, just a number. Finally, p_vector_values_multiple handles multiple values, in the form of 1, 2, 3, 4. Notice how we reference vector_values from within P_vector_values_multiple? This allows it to recursively call itself until it terminates at a NUMBER token.

With this code in place, we can now parse and store statements in the form of vector v1 = [1, 2, 3].

Matrices are defined similarly, with a few additional functions to handle rows and row values.

# ----- MATRIX -----
def p_statement_matrix_assignment(p):
    'statement : MATRIX_ID EQUALS expression'
    variable_name = p[1].split()[1]
    variables[variable_name] = p[3]
    p[0] = (variable_name, p[3])

def p_expression_matrix(p):
    'expression : MATRIX'
    p[0] = p[1]

def p_matrix(p):
    'expression : LBRACKET matrix_rows RBRACKET'
    p[0] = np.array(p[2])

def p_matrix_rows_single(p):
    'matrix_rows : row'
    p[0] =[p[1]]

def p_matrix_rows_multiple(p):
    'matrix_rows : row COMMA matrix_rows'
    p[0] = [p[1]] + p[3]

def p_row(p):
    'row : LBRACKET row_values RBRACKET'
    p[0] = p[2]

def p_row_values(p):
    'row_values : NUMBER'
    p[0] = [p[1]]

def p_row_values_multiple(p):
    'row_values : NUMBER COMMA row_values'
    p[0] = [p[1]] + p[3]

# ----- END MATRIX -----

The above code allows us to parse matrices in the form of matrix m1 = [[a1, b1, c1,....,z1], [a2, b2, c2...,z2], ...., [an, bn, cn, ....zn]].

With this in place, we can now define some operations, such as addition and matrix multiplication. We can also define a print statement to print the values of our variables.

First, we have to define an expression for retrieving IDENTIFIERS from the variable table.

def p_expression_identifier(p):
    'expression : IDENTIFIER'
    variable_name = p[1]
    if variable_name in variables:
        p[0] = variables[variable_name]
    else:
        print(f"Error: Variable '{variable_name}' not in variable table")

Here if we encounter a IDENTIFIER token, we look to see if the variable identified by IDENTIFIER is in the variable identifier. If it is we retrieve its value and store it in p[0] else we print an error message.

Addition and multiplication are straight forward as well, with multiplication using numpy's matmul method:

def p_expression_add(p):
    'expression : expression PLUS expression'
    p[0] = p[1] + p[3]

def p_expression_multiply(p):
    'expression : expression MULTIPLY expression'
    p[0] = np.matmul(p[1],p[3])

Likewise, the print statement can be easily defined as well.

def p_statement_print(p):
    'statement : PRINT LPAREN IDENTIFIER RPAREN'
    variable_name = p[3]
    if variable_name in variables:
        print(variables[variable_name])
    else:
        print(f"Error: Variable '{variable_name}' not in variable table")

Here we parse statements in the form of print(<IDENTIFIER>). We retrieve IDENTIFIER and look it up in the variable table. If it is present, we print the value of that variable, otherwise, we print an error message.

We conclude with a simple error handler:

def p_error(p):
    print("Syntax error: ", p)

To run the program, we build the lexer and parser.

# Build the lexer and parser
lexer = lex.lex()
parser = yacc.yacc()

And define our DSL code:

# Parsing and executing DSL code
dsl_code = """
vector v1 = [1, 2, 3]
vector v2 = [4, 5, 6]
print(v1)
print(v2)

matrix m1 = [[1, 2], [3, 4], [5, 6]]
matrix m2 = [[5, 6, 7], [7, 8, 9]]

print(m1)
print(m2)

vector v3 = v1 + v2
matrix m3 = m1 * m2

print(v3)
print(m3)
"""

Finally, we can call the parse method on the parser object on dsl_code to see the output of our program.

parser.parse(dsl_code)

With that, we have a fully functionally DSL in Python

Conclusion

In this article, we explored domain-specific languages. We examined two types of DSLs, internal and DSL, as well as the criteria that defines a good DSL. Furthermore, we looked at the advantages of creating a DSL for domain specific problems. We then dived into the process of designing and implementing a DSL using the Ply librarying, which provides lexing and parsing capabilities in the Python language.

We begun by defining the tokens of our DSL. These form the basic building blocks of the language, such as numbers, identifiers, and keywords like vector and matrix. We leveraged regular expressions to define token rules and used Ply's lexer tok tokenize our sample DSL code.

We then proceeded to design our grammar rules to define the syntactical and semantic structure of our language. We created rules for declaring vectors and matrices and assigning them to variables. Next, we proceeded to define operations for performing addition and matrix multiplication on the variables we stored, as well as an operation for a print statement. Through parsing the code, we construct an abstract syntax tree (AST) that represents the structure of the DSL code.

By learning how to implement a DSL in Python and how to leverage the Ply library, you know have the necessary knowledge and tools to create your own domain specific languages. Whether it is for data engineering, rule-based systems, game design, or other use cases, a well-constructed DSL can greatly facilitate the development of software and applications, enhancing productivity and code expressiveness.

Now it's time to apply what you've learning and start exploring the possibilities of building your own DSLs! With these tools at your disposal, you have the flexibility and power to create powerful DSLs and improve the way you tackle complex problems across a wide variety of domains.

Thank you, and happy DSL development!

Full code available at: https://github.com/fractalis/devto-articles/blob/main/python-dsl/matrix-dsl.py

How to write a basic rule engine in Python

Scott Rallya — Mon, 15 May 2023 01:34:36 +0000

While there are many existing rules engines in Python, such as the amazing rule-engine framework, I thought it would be an interesting exercise to utilize ChatGPT to help me write my own basic rule engine from scratch. So, without further ado - I present to you PYROSE - PYthon Rule-based Operation System Engine.

The basic structure of our rule system is as follows. At the basic foundation of engine we have Facts. Facts are simple objects that represent information we want stored in our system and can contain any kind of information that is relevant to the design of the constraints by which our rules-engine operates. Let us begin by defining a very simple structure for our Facts

from typing import Any

class Fact:
    def __init__(self, **kwargs: Any):
        self.__dict__.update(kwargs)

Here we define a very simple object that can be instantiated with any number of keyword arguments to define our fact. We then update our object used the self.__dict__.update method to add attributes to our object corresponding to the keywords passed to the initializer. We can initialize a fact as follows,

person_fact = Fact(name="John Brown", age="35", occuptation="Software Developer")

Accessing a attribute is as simple using dot notation as if you were access a member or method on the object.

person_fact.age # Returns 35

Now that we have established our Fact object, we need to define a Condition by which to apply our Fact object. Our Conditions are very simple in design as well to start off with. The initializer takes two parameters, a name and an evaluation function. The evaluation function will be applied to a Fact and return a bool. It also contains a method, evaluate, which will take a Fact and return a bool, calling the objects evaluation_function on the Fact. Here is the outline of our Condition class:

from rule_engine.fact import Fact

from typing import Callable, Any, Dict, List

class Condition:
    def __init__(self, name: str, evaluation_function: Callable[[Fact], bool]):
        self.name = name
        self.eval_func = evaluation_function

    def evaluate(self, fact: Fact) -> bool:
        return self.eval_func(fact)

Following a similar suit, we can define our Action class. An Action class takes in a name parameter and a Callable parameter into its initializer. The Callable corresponds to the Action's execution function, what is executed if all the Conditions of the Rule are True. It also contains an execute method which calls the execute function on the given Fact, giving a None type as a result. Here is the definition of the Action class:


from typing import Callable, Any, Dict, List
from rule_engine.fact import Fact

class Action:

    def __init__(self, name: str, execution_function: Callable[[Fact], None]):
        self.name = name
        self.exec_func = execution_function

    def execute(self, fact: Fact) -> None:
        self.exec_func(fact)

So far we've defined our Fact, our Condition, and our Action. We can combine these together to form our Rule class. This will be the main driving force behind our Engine. The Rule class will initially be constructed with a single Action and Condition. Two methods, add_condition and add_action, will allow you to add additional conditions and actions to the Rule as you need.

Finally, a third method, evaluate, will take in a list of Facts. It defines a fact_generator which takes a list of conditions and a list of facts. For each fact, it maps each of the conditions' eval_func against the fact. It then reduces this list to a single boolean value, and if this value is true, we yield the fact.

We then call the fact_generator function, wrap it in a list to get the list of all True facts, and if the length is greater than 0, we iterate through the list of true facts. For each true fact, we iterate through a list of the rules actions and call the action's exec_func on the true fact.

The complete definition of the Rule class is as follows:


from rule_engine.condition import Condition
from rule_engine.action import Action
from rule_engine.fact import Fact

from typing import Any, List
from functools import reduce

class Rule:
    def __init__(self, condition: Condition, action: Action):
        self.conditions = [condition]
        self.actions = [action]

    def add_condition(self, condition: Condition) -> None:
        self.conditions.append(condition)

    def add_action(self, action: Action) -> None:
        self.actions.append(action)

    def evaluate(self, facts: List[Fact]) -> Any:
        def fact_generator(conditions: List[Condition], facts: List[Fact]):
            all_conditions_true = True
            for fact in facts:
                results = map(lambda condition: condition.eval_func(fact), conditions)
                all_conditions_true = reduce(lambda x, y: x and y, results)

                if all_conditions_true:
                    yield fact

        true_facts = list(fact_generator(self.conditions, facts))

        if len(true_facts) > 0:
            for fact in true_facts:
                for action in self.actions:
                    action.exec_func(fact)

You can use the rule engine as follows:


from rule_engine.fact import Fact
from rule_engine.condition import Condition
from rule_engine.action import Action
from rule_engine.rule import Rule

age_cond = Condition(name="Age>=21", evaluation_function=lambda fact: fact.age >= 21)
occupation_cond = Condition(name="Occupation==Software Developer", evaluation_function=lambda fact: fact.occupation == "Software Developer")

print_action = Action(name="Print Fact", execution_function=lambda fact: print("Name: {} Age: {} Occupation: {}".format(fact.name, fact.age, fact.occupation)))

john = Fact(age=25,name="John Brown", occupation="Software Developer")
sarah = Fact(age=35,name="Sarah Purple", occupation="Data Engineer")
barry = Fact(age=27, name="Barry White", occupation="Software Developer")

rule = Rule(condition=age_cond, action=print_action)
rule.add_condition(occupation_cond)

rule.evaluate([john, sarah, barry])

All in all, this is a very basic demo of a very simplistic Rules engine. However, I think it can definitely be built and improved upon in any number of ways, so I hope that you enjoyed reading and using what you learned in your own projects. If you have any questions, comments, suggestions or ideas, please feel free to reach out to me. Thank you for reading.

Acknowledgments:

Thanks to John for catching some unnecessary code duplication! Catch out John's articles.