A Guide to Navigating Development, Avoiding Bugs, and Conquering Spaghetti Monsters
In this article, I’ll guide you through the ever changing world of development that may cause anxiety. I’m going to share my experience in navigating through different requirements, choosing the right architecture, and adapting it as the project grows. My goal as a developer is to create maintainable, readable, and flexible code, which I hope to share with you in this article.
I have been working with JavaScript/TypeScript since 2014, and my coding style has been influenced by the free and expressive nature of these languages. The principles and concepts I discuss in this article are applicable not only to JavaScript and TypeScript, but can be adapted to other programming languages as well.Disclaimer
Also, I think It's important to note that there are numerous approaches to solving problems and implementing software solutions. When faced with opposing opinions or different ways of doing something, finding a balance between them can lead to a more optimal solution. I believe that the answer often lies in the middle ground, as I will show in this article.
I know that each developer may have their own unique way of accomplishing tasks, and there is no one-size-fits-all solution. Your experiences and perspectives are valuable, and I welcome your input and feedback. If you have alternative approaches, suggestions, or additional insights, please feel free to leave a comment or reach out. Collaboration and sharing of ideas contribute to the growth and improvement of the development community as a whole.
The purpose of this article is to offer insights, theoretical perspectives, and principles that can enhance your software development journey. I encourage you to adapt and modify these principles to suit your specific needs and context. Software development is a creative process, and finding your own path within the spectrum of opinions can lead to remarkable outcomes.
Overview
In this article, I’ll walk you through layered architecture, one layer at a time. This won't be your typical, by-the-book layered architecture; instead, it incorporates additional layers that aim to find a middle ground by combining the best aspects of existing architectures. We'll start at the top, adopting the layered or onion approach, and gradually transition to a hybrid component architecture as the complexity of the business logic intensifies. Rest assured, we'll maintain the integrity of the Business Layer (BL) throughout this transformation.
Before diving into the specific layers, it's important to note that while this approach primarily focuses on backend applications, it can also be used for frontend development. However, it is important to acknowledge that this approach lacks sections/layers for UI components, screens, and screen routers. With that in mind, let's explore each layer and its theoretical foundations to gain a deeper understanding of their roles and significance.
App Boot Layer (ABL): This layer is important for setting the stage for the application. In my explanation, I'll delve into the theoretical foundations and discuss why proper initial start up and shut down processes are essential. Understanding the significance of this layer will help you appreciate the smooth start and graceful exit of the application you are developing.
Controller Layer (CL): Further on, I'll discuss the role and significance of the entry point in an application. I'll explore the theoretical aspects of how actions and requests are received and processed in this layer. By understanding the central hub's purpose and the separation of business logic from the transport framework, we'll gain insights into designing efficient and maintainable software.
Business Layer (BL): This is the heart of the application containing the business logic. In this section, I'll explore the theoretical concepts of the Use Case Layer (UCL) and Service Layer (SL). I'll discuss the importance of orchestrating functions and implementing reusable, value-driven code that drives the core functionality.
Data Access Layer (DAL): Descending deeper into the architecture, I'll explore the theoretical aspects of interacting with various data sources. I'll explain the significance of the DAL as a bridge between our business logic and the data. Understanding the theoretical foundations of seamless interaction and manipulation will give you the knowledge to design robust data access mechanisms.
Library Layer (LL): Here I’ll talk about the theoretical benefits of the code snippets and utilities. I'll discuss the art of creating modular, reusable code without over-engineering or compromising simplicity. Understanding the theoretical principles of this layer gives you the opportunity to leverage existing resources and enhance your efficiency as a developer.
Util Layer (UL): Finally, I'll explore the theoretical foundations of the utility functions. I'll discuss the significance of tools like logging, environment management, and error handling. Understanding the theoretical aspects of these functions empowers you to make informed decisions and adopt best practices throughout the development process.
App Boot Layer (ABL)
In the vast jungle of application development, there are moments when I need to prepare the application before it can emerge into the world. Similarly, when the time comes to shut down the application, I often find myself needing to perform necessary cleanup or conceal my creation from the outside world until all ongoing processes are completed.
Within the ABL, I have the power to execute processes asynchronously, whether in sequential order or in parallel. This layer allows me to define various methods of initiating the application. For instance, in the realm of backend development with Node.js, I can make decisions such as excluding the RestAPI interface during unit testing or bypassing database connections when they are not necessary.
Example:
Let's consider an example scenario:
Starting up
When I launch an application, my first priority is to establish a seamless connection to the database and to execute any necessary database migrations. This ensures that the data infrastructure is ready for action. While the connection to the database is being established, I take the opportunity to register an internal EventBus, powered by the reliable RxJS library. I have full confidence in registering it because internal EventBus is triggered by the code that is located in business logic.
Moving on to the next phase of initialization, I focus on activating interfaces that allow the application to interact with the outside world. This entails enabling RestAPI, MessageQueue, and CronJob functionalities simultaneously. By doing so, I unlock the full potential of the application, enabling it to effectively communicate and serve various external systems and users.
Shutting down
When the time comes to shut down my application, such as during a version release, it is never advisable to forcefully terminate it. Doing so would risk a potential loss of valuable information and terminating unfinished processes.
While some environments, like AWS, Azure, or Google may provide mechanisms for redirecting traffic to new instances, they are not foolproof solutions, these mechanisms may not cover all incoming signals. For instance, message queues or cron jobs may still be active in the background, while only RestAPI calls are redirected.
The safest approach to shutting down a Node.js application is to deactivate all listeners and let ongoing processes finish gracefully.
Lastly, it is wise to log any errors that occur during the application shut down process. This practice helps to identify and address potential issues, even in the final moments of the application's lifecycle.
Controller Layer (CL)
In the CL, I aim to consolidate all entry points of the application. Meaning that any action invoked within the application must first pass through the CL. By establishing the centralised control, it becomes much easier to track the origin of any process. Furthermore, this established control serves as an effective boundary, preventing any transport framework I use from invading my business logic.
My application has various entry points, including RestAPI, MessageQueue, CronJob, and EventBus. In this layer, I establish a router that directs messages or invocations to their respective functionalities. In my opinion, it is essential to create readable routes and consolidate them in one location to avoid overlooking any of them. Each route is associated with a handler, which should exclusively execute a single function from the BL. This separation is crucial because multiple transports may trigger the same handler, such as both RestAPI and MessageQueue invoking the identical functionality. By keeping the business logic out of the controller, I avoid code duplication and minimize code smell.
Controller handlers play a crucial role in handling data received from end users. This is where I perform validation and, if necessary, transform the data. It is essential to conduct these validation and transformation steps within the handlers themselves. The reason is that each controller is tailored to its specific transport system, be it RestAPI, MessageQueue, or any other. Consequently, the data structure associated with each transport system may differ, needing specific validation and transformation procedures within the respective handlers. By encapsulating these processes within the handlers, we ensure that the data is appropriately validated and transformed based on the requirements of each individual controller and its associated transport system.
Business Layer (BL)
The BL is responsible for handling the core business logic. It is important to keep this layer independent of any frameworks to ensure that the business logic stays unaffected by the specific implementation details of any frameworks. To effectively manage the complexity of the BL layer, I have adopted an approach inspired by the Onion Architecture:
Use Case Layer (UCL): focuses on defining and orchestrating the specific use cases or business scenarios of the application. It represents the high-level actions or operations that the system needs to perform to fulfil the business requirements.
Service Layer (SL): encapsulates less complex, technical solutions that are reusable across the BL.
Component Layer (CompL): encapsulates complex reusable logic, combining the rules and principles of both the UCL and the SL.
Repository Layer (RL): responsible for abstracting the data persistence and retrieval operations from the underlying data storage. It provides an interface to interact with the data storage while shielding the BL from the specific details of the data storage mechanism.
Model Layer (ML): it represents the domain-specific data structures used within the BL. It encapsulates the core concepts of the application domain, ensuring a cohesive representation of the business objects and their relationships.
Use Case Layer (UCL)
Every time I dive deep into the development process, I’m reminded of the importance of having good and descriptive names for functions. I'm trying to enforce this rule consistently, although I am aware that it may not be possible to follow it 100% of the time. With this understanding, I have introduced a dedicated layer - Use Case Layer (UCL) - to accommodate situations where the naming convention can be relaxed. The UCL serves as a repository for functionalities derived from the product owner, encapsulating the dynamic business logic that can evolve based on market demands.
Through the establishment of the UCL, I am able to strike a balance between the need for clear and descriptive function names and the reality of complex functions that may require more concise names. This specialised layer enables me to effectively accommodate both scenarios. Furthermore, I embrace the inclusion of comments within the UCL, as they enhance the documentation and understanding of the code within this layer.
The main distinction I've observed between the SL and the UCL is that the UCL undergoes changes driven by dynamic market demands, while the SL represents the technical solutions for the current market requirements, and ideally, when the market demands shift, ideally, new functionalities are introduced in SL to meet these evolving needs. In the UCL, I ensure that I don't include code that should be in the SL; instead, I focus on calling and orchestrating functions from the SL. My goal is to make the functionality of a UCL function evident by simply reading the names of the SL functions it calls, creating a seamless narrative flow similar to reading a book. Although the names of UCL functions can be short and unclear, I try to accompany them with thorough documentation and requirements to ensure that it’s clear how the intended feature functions.
Example:
In my backend system, I have an authorization endpoint that handles various tasks. Its responsibility is to verify the existence of a user with the provided password, check if the user is active, and validate the 2FA code entered during the login process. While naming this function based on all the tasks it performs would result in a lengthy name, I've decided to simply call it login for simplicity and ease of understanding. I'll place this function in the authorization-use-case.ts file, which is located in UCL. It's important to note that I won't provide the actual implementation code here which belongs in the SL.
// src/business/use-case/authorization-use-case.ts
expose const authorizationUseCase = {
login: (params: {userName: string, password: string, code2fa: string}) => {
const {userName, password, code2fa} = params
const user = userService.findUserBy({userName, password}).catch(undefined)
if(!user) throw new Error('Username or password incorrect')
const isUserActive = userService.verifyActiveUser({user})
if(!isUserActive) throw new Error('User is not active')
const isCodeValid = code2faService.isValid({user, code2fa})
if(!isCodeValid) throw new Error('Two factor code invalid')
return authorizationService.generateAuthToken({user})
}
}
A great example of the business logic that belongs in the SL is the verifyActiveUser function. This logic is not subject to frequent changes based on market demands. The status of a user's activity is stored within the user business model, making it inherently tied to the business model itself. Since every part of the application depends on the business model, it becomes apparent that market demands can indirectly influence changes in the SL. However, I firmly believe that if I define my business model effectively, market demands will have minimal impact on the SL.
This underscores the importance of dedicating sufficient effort and time to defining a robust and reliable business model. By investing in the initial stages of creating a solid foundation, I can minimise the need for frequent changes in the SL, ensuring stability and consistency throughout the application.
Service Layer (SL)
In the SL, I focus on writing reusable code with business value. My goal is to ensure that the logic implemented here is simple and serves a single purpose - keeping complexity low. It is crucial to follow a naming convention here, function name needs to accurately describe the function's purpose. If a function's name becomes too long, it is an indication that the function may be too complex. Although it may not always be possible to strictly stick to this rule when following a horizontal layer architecture, there might come a time when we need to transition to a component based architecture of writing.
Component Layer (CompL)
In the UCL and SL, I follow some straightforward rules and make a conscious effort not to mix them. However, I understand that it can be challenging to write code that fully follows these rules. That's where the CompL comes into play as a helpful solution. In this layer, I combine the rules from the UCL and SL, employing UCL rules in the public parts of the code and SL rules in the private parts of the code. The main idea behind this approach is to encapsulate complex and reusable logic, ensuring its modularity and integrity.
Example:
In this scenario, I have an invoice entity with a status property. The available statuses for this entity are: DRAFT, CREATED, SUBMITTED, and CANCELLED. Given the strict rules governing status changes, which are already complex, I find it beneficial to encapsulate the status rules within a dedicated component. By doing so, I can expose a validation function that checks if the current invoice can be updated to a desired new status.
To achieve this, I create an invoiceStatus component that exposes a function named canChangeTo({invoice: Invoice, status: string}). The purpose of this function is to hide the intricate logic that determines whether a particular status change is allowed. It's important to note that the logic behind canChangeTo function is not intended to be reusable in other layers; it should only be accessed through the exposed entry points of the component.
Ideally, this component would not use the logic from other layers, including layers below and next to it (RL, SL), because the result may generate one big spaghetti monster.
src
...
├── business
│ ├── component
│ │ └──invoice-status
│ │ ├──rule
│ │ │ ├──draft.ts
│ │ │ ├──created.ts
│ │ │ ├──submitted.ts
│ │ │ └──cancelled.ts
│ │ ├──index.ts # here we hae all exposed endpoints
│ │ └──service.ts # has a switch by rule to select rule
│ ├── model
│ ├── repo
│ ├── service
│ └── use-case
...
Repository Layer (RL)
The RL is where I take responsibility for abstracting the data persistence and retrieval operations from the underlying data storage. It's my job to provide an interface that allows me to interact with the data storage while shielding the BL from the specific intricacies of the data storage mechanism.
Within the RL, I focus on implementing the business logic responsible for persisting and retrieving data. Regarding RL for storing data, the underlying method used is not as crucial as the ability to store it securely and retrieve it whenever needed. I can have a range of options available, including databases, files, memory, FTP, third-party systems, RestAPIs, GraphQL, and more.
It's also important for me to establish boundaries and keep frameworks at bay when it comes to our business logic. To achieve this, I have DAL that offers this flexibility. This allows me to switch the data type of any entity simply by changing the DAL implementation.
During my work in this layer, I recognized the need to introduce a custom light Object-Relational Mapping (ORM) framework. This framework is specifically designed for my business logic and doesn't rely on the third-party ORMs used in the applications. While it may not be a perfect solution, it serves well for handling simple queries. For more complex queries, I turn to the DAL to find an appropriate solution.
Model Layer (ML)
As a developer, I understand that ML plays a key role in the application. It serves as the foundation that defines and shapes the entire application. Any changes made to the ML can potentially impact every other layer, as they all depend on it.
Initially, when the application is not yet complex, I may not need to differentiate between the Model, Entity, and Transport Object. However, it's important to be aware of these tools and their potential usage at any given moment. I must be cautious to ensure that I don't inadvertently compromise my model without evaluating whether the solution lies in modifying the Entity or Transport Object.
Entity: represents the object definition for the DAL. It serves as a means to represent the data stored in the application.
Transport Object: represents the object definition for the CL. It comes into play when I need to redefine what is sent to the end user without modifying the underlying Model.
These distinctions give the flexibility to tailor the data representation according to specific requirements without directly altering the model.
Entity
In the overview diagram, I have defined entities in the DAL. When it comes to storing models in a database, I consider which properties are necessary to store and which ones can be calculated or derived. For instance, if I have a model with properties like firstName, lastName, and fullName, I don't necessarily need to store the fullName in the database. I can always calculate it by concatenating the firstName and lastName values. This approach allows me to optimise the storage and retrieval process while maintaining the necessary data integrity.
Transport Object
As for the Transport Object, it serves as an object definition that responds to requests made through the CL. It provides a way for me to format data differently or conceal certain information before sending it to the end client. The Transport Object allows me to tailor the data presentation according to specific requirements or privacy considerations. By doing so, I ensure that the client receives the most relevant and appropriate information while maintaining data security and confidentiality.
Data Access Layer (DAL)
This layer is dedicated to managing the storage and retrieval of data. It provides flexibility to store data in various ways, including databases, files, memory, FTP, third-party systems, RestAPIs, GraphQL, and more.
It's crucial to understand that this layer serves as a wrapper for the frameworks I use to manipulate data. The framework's implementation should remain contained within this layer and should not spill into the RL where the business logic resides. By maintaining this separation, I ensure that the core business logic remains unaffected by the specifics of the data manipulation framework.
Furthermore, I mentioned that I use a custom Object-Relational Mapping (ORM) implementation in the RL. As a result, I need to map my ORM implementation with any third-party ORM that I utilise in the DAL. This mapping ensures compatibility and seamless integration between the custom ORM and the external ORM, allowing for efficient data access and manipulation throughout the application.
Library Layer (LL)
In this layer, my focus is on encapsulating code that has the potential to be used in other applications, making it easily extractable. I approach it as if I'm developing a third-party library, ensuring that the code is modular and reusable. By doing so, I can extract and reuse modules from this layer in other applications without unnecessary complications.
It's important to strike a balance between convenience and over-engineering. While I aim to make the code convenient to write and extract, I must avoid creating a completely separate external library for every reusable piece of code. I also want to avoid embedding code directly into my business logic, because embedded code is hard to reuse.
Finding the middle ground is key. I want to identify and encapsulate code that has broad applicability and can benefit multiple projects. By doing this, I can maximise code reuse, simplify maintenance, and streamline development across different applications.
Util Layer (UL)
In this layer, I focus on storing functionalities that don't contain any specific business logic. These functions are designed to be used across all layers of the application without any restrictions. Some common examples of functionalities that belong to the UL include a logger, environment variables management, and basic error messages.
The UL serves as a central repository for these utility functions, providing convenient access to commonly used features that are not tightly coupled to any particular business domain. By separating these functionalities into a dedicated layer, we promote code reusability, maintainability, and modularity throughout the application.
For instance, a logger utility allows us to easily log messages and events across various components and layers of the application, providing valuable insights during development, debugging, and monitoring. Similarly, an environment variables utility helps manage configuration settings specific to different deployment environments, enabling greater flexibility and scalability. Additionally, a basic error handling utility assists in capturing, handling, and reporting errors consistently throughout the application.
By consolidating these non-business-specific functionalities in the UL, I create a cohesive set of tools that can be utilised across the application, streamlining development and promoting efficient code organisation.
Conclusion
In this article, I have explored the architectural layers of software development and discussed strategies for creating robust and maintainable systems. I began by understanding the importance of the App Boot Layer (ABL), where I start up and shut down the application efficiently. Moving to the Controller Layer (CL), I established a centralised control point for all entry points and separated business logic from transport frameworks. In the Business Layer (BL), I orchestrated use cases and implemented reusable, value-driven code. The Data Access Layer (DAL) acted as a bridge between business logic and data sources, while the Library Layer (LL) encapsulated modular, reusable code. Finally, the Util Layer (UL) focused on essential tools like logging and basic error handling. I hope that navigating through these layers gives you insights into building scalable and resilient software architectures.
Folder structure example for node.js application
src
├── app-boot
│ └── init
├── business
│ ├── component
│ ├── model
│ ├── repo
│ ├── service
│ └── use-case
├── controller
│ └── express
├── dal
│ └── typeorm
│ └── entity
├── lib
│ └── typeorm
│ ├── migration
│ └── subscriber
└── util
Art credit in this post goes to the dynamic duo of human creativity and artificial intelligence, with a special shoutout to Dream by WOMBO.Art Disclaimer
Top comments (2)
Definitely an interesting read, especially the idea of adding a specific Library Layer for future endeavours.
How would you potentially separate what would be stored in that layer compared to the util layer which could also provide a lot of reusability between multiple projects?
Thanks for the question, it looks like I was not quite clear in that layer, so I will try to update that section.
The idea for the Library Layer is to write code and use it as any npm library out there. So if I want to write some reusable logic and use it across all layers like I use functionalities from the Util Layer, I would create another wrapper around this reusable logic inside my Util Layer. This way, when I extract the logic from the Library Layer I would end up changing only code in the Util Layer, and the rest of my business logic is protected from changes.
The same goes for any other layer DAL or Controller Layer. I do this with many npm packages. I try to find balance, I marry some libraries (use them deeply in my code) and others I try to keep away.