Introduction
I cannot stress enough the importance of documentation for system design. It acts as a comprehensive roadmap, guiding the entire project by clarifying architecture, functionality, and relationships between components. Moreover, it ensures collaboration, reduces ambiguity, and facilitates maintenance. In the fast-paced environment of software development, where deadlines are getting tighter and the scope broader, finding time to craft efficient system documentation proves challenging. However, neglecting this aspect is a mistake in the long run.
Picking the Format
There are multiple forms of documentation, each for its purpose, some of these are text and spreadsheet documents, diagrams, code comments, wikis (knowledge bases), and API documentation. In my years working with software and dealing with a few of these, I’ve found, through experience, that the industry tends to prefer text and diagram documentation, with code comments as the runner-up. Code comments are especially popular among developers, but they are typically internal and very situational, limiting their shareability.
I've encountered my fair share of text documentation, which has a dissertation-like aspect and is common in conversations with non-technical stakeholders due to its readability. Although those can be used for the technical aspects of system design, they fall short of providing a holistic view of the entire project. As a consequence, spotting logical inconsistencies and potential issues becomes challenging in such dense documents. Additionally, this type of document has a major drawback, the language barrier.
I am not a native English speaker, I am fluent in English today, but that has not always been the case. While on a regional scale, your team might not have issues with it, you must consider that in this industry, you can work with customers and teams all across the globe. Chances are, some of the people you talk to – more than you can imagine – will not be native or fluent speakers of the same language as you. Sometimes, even if they are, there can be some miscommunications due to accents and regional idioms. Therefore, this article will explore the form of documentation I’ve found to be the most efficient throughout my career: the visual one!
Draw it up!
There is the most cliché saying of all times: “A picture is worth a thousand words”. This is a very well-known adage for a reason, and I can safely say it exists in many places around the world meaning the same thing and transcending language barriers. This is true in many professional areas, not only those related to images or storytelling, and in technology and system design, it is no different!
Before we get started, let’s get some terminologies in place. When talking about visual system design, we normally work with diagrams. There are several types of diagrams as well, each for a different purpose and with a different style.
Types of Diagrams
Diagrams are normally built following the directives from the Unified Modeling Language (UML), which is the industry standard for visual documentation. They can be divided into two main categories: Structural and Behavioral.
Structural diagrams: are responsible for shaping the overall structure of a system, by utilizing static objects and indicating their relationships. Acts as a direct guide for the required technology in a project. The main structural diagrams include the Class, Object, Package, Component, Composite Structure and Deployment diagrams.
Behavioral diagrams: focus on the more dynamic aspects of the system, establishing the interaction between the system components and how they behave over time. These diagrams help visualize the flow of data, communication, and execution within a system. The most common behavioral diagrams are the Use Case, Activity, Sequence, Communication, State Machine, and Interaction diagrams.
When working with system design, I typically incorporate aspects from both categories in the same diagram to give it a complete meaning. I found this approach to be more accessible and faster to spin up. Our goal here is to avoid investing a lot of time in building overly technical documentation that will end up requiring additional explanations or a lengthy document to accompany it. This, in turn, enhances our efficiency and production speed.
Despite the flexibility to mix and match, if you are doing so, it is important to ensure that the final diagram makes sense and is understandable by anyone not directly involved in the project. This implies that individuals don’t need to have the full project context to understand its contents. I often find myself adding some aspects of sequence and interaction diagrams into a component diagram or incorporating architecture into either of these, depending on the system scope and how I believe it's most effective to convey the use cases within that scope.
As the focus of this article is to guide how to produce efficient documentation, there won’t be much detail on the specifics of each diagram type. I will, nonetheless, give you a practical example of interaction and architecture diagrams (architecture diagrams are based on the component diagram) and show how they can interact with each other. Before we get into it, though, it's essential to decide where you'd like to create this documentation.
Diagraming tools
There are plenty of tools out there that can facilitate drawing up a system design. It all comes down to your own preference, but truth be told, pen and paper can do just as good as a job, so there is no excuse not to draw it up. Of course, using digital tools has its advantages in keeping things clean and I would encourage their usage as they easily allow sharing and collaboration with others.
The focus of this article is not to review specific diagramming tools in detail. However, whilst there are many excellent resources available, some of the tools I've used before and recommend are draw.io and lucidcharts. Both have a decent free tier, integrate nicely with most cloud storage solutions, and work very well for most diagram types.
Sequence diagrams, on the other hand, stand as an outlier. For these, I like to use code-based tools such as sequencediagrams.org as they facilitate some of the more intricate details of this type of diagram and can very easily be converted to different tools.
Once you've selected your tool, you're prepared to embark on planning your system design! It's worth noting that all the diagrams in this article were crafted using draw.io. Nevertheless, they can seamlessly be replicated in any other diagramming tool of your choice.
The Architecture
Assuming you already have all system requirements, as well as user stories, it is time to start thinking about architecture. There are plenty of different ways to draw up the architecture diagrams. I don’t intend to explore in detail how to plan an architecture. However, I will give an overview to help illustrate the point. For that, I’d like to bring you a more practical example, and a small one at that, as you’ll see how valuable a full system design can be, even for the smallest of systems.
Consider the following user story: “I’d like to send data of multiple formats to a 3rd party system”. From this story, we can infer that the input data will have at least two different formats, and this data will have to be transformed into a format expected by the 3rd party system. We can achieve this with a microservice that exposes a secure endpoint to receive data, transforms it, and maps it into the schema expected by the integrated system.
This seems simple enough and you could probably do it all while keeping this system design in your mind alone. But, if you consider the following situation: you built this and six months later the 3rd party system was updated and now you need to go back and review the integration. Well, you’ll probably waste a lot of time just getting to understand what the initial requirements were and how the service was implemented. Or you can follow along and draw a design that will get you back on track even after so long, better yet, you could even delegate this task and not have to worry about it.
Below, you can see what a diagram for this system could look like (note that I will be using AWS services as a frame of reference, but this can be translated to any cloud computing service):
Does this diagram bother you? If you said yes, good! While this architecture will accomplish the user story and is probably the first architecture that comes to your mind, it seems amiss. And that is only clear because we drew this.
Looking at this, we can tell what the reliability issues are. If the 3rd Party Service is unavailable for any period, your incoming data will be lost in its entirety. Also, if you ever want to attach a custom domain to the service endpoint, you will have to review the entire process, not to mention the security risks of making the Lambda service endpoint public for it to receive messages directly.
All of that will be clear once you start development and will most likely cause you to start spinning off additional services to overcome these challenges, which in turn may cause you to have to review the code multiple times and re-deploy everything several times more. However, since we have the drawing we can visualize all these complications without setting up a service or even starting to code.
Now let's review this architecture and make it more robust and reliable taking best practices into account.
It is already looking a lot better, right? Now you can tell we have a system that won’t stop in case of failure, but more importantly, that can recover from failure. This can be expanded even further including more services to improve resiliency and maintainability. Of course, this diagram as it stands is very technical and is aimed at developers. If you are ever to share this with non-technical stakeholders you will need some additional notes on what each of these services do.
Moving on to the next steps, while the architecture will give us a bird’s eye view of the system, the workflow diagram will give us a more fine-tuned perspective of what the integration service will look like, which is invaluable for developers to understand the system and plan for the code challenges.
Workflow Diagram
This diagram is an amalgamation of the activity and component diagrams, providing a perspective on the interaction between components and overall system behavior. I have had a fair bit of success expanding on this by including architecture elements and adding some almost pseudo-code sections. That made the understanding so much better for developers and stakeholders alike.
I had opportunities to test this form of diagram in real production situations, where I had to analyze complex system logic alongside non-technical stakeholders. Having a workflow diagram sped up the development process by leaps and bounds, with fewer iterations of logic adjustments compared to the times I did not have such a diagram.
Continuing our example, see below what the workflow diagram would look like in this scenario.
The diagram above should cover all you need to build this microservice, it even has some aspects of a sequence diagram in the interaction with the 3rd Party Service, and should be fairly easy to read. Of course, in this context, the data mapping information should be elsewhere, but if you wanted to, you could add a JSON object as a note somewhere on the diagram to keep it all centralized in a single place. Although this should be enough to illustrate for now.
The more seasoned developers must have noticed by now that there is a stress point in this flow, and it becomes apparent thanks to the diagram. If you pay attention to the Authorization flow, which follows a standard authentication mechanism to request an access token and use this token for further requests into the system. In the flow above, we are requesting a new token every time we process an event. Though this would work out fine in testing scenarios, once this microservice starts receiving thousands of message events all at once, this volume of requests would cause significant stress to the 3rd Party Service Authorization endpoint. Prompting the security mechanism from the service to block additional requests. Furthermore, these tokens are normally valid for several hours, sometimes even days, and requesting a new token, every time data has to be sent as part of an integration, is not only a bad practice but can cause the service to shut down.
Once again, we can see that because of the holistic view that a flow like this can provide of the system, even before we touch a piece of code. The alternative to this would be to start developing until we are done, and of course, our tests worked out without issues because we just sent 2 or 3 messages at a time while testing and adjusting, but once this gets deployed to production and more messages start coming in, the system would eventually stop. You would then have a distressed customer questioning why their services are being overburdened and, in some cases, incurring additional costs due to unexpected request volume on an endpoint that is not prepared for it.
Now, going back to the diagram, we can easily fix that by introducing a cache that will keep the token throughout its expiration period and, therefore, only request a new one when strictly necessary. Here is how this flow could look like:
Much better right? When talking about coding this microservice, if you follow the diagram you should note that each rectangle can be directly translated into a function of the code and, as expected, the lozenges are directly translated into conditional flows (if or switch statements). This allows for a high-level view of the code and even gives an idea of what kind of libraries you might need. The best part about this diagram is that it’s very easy to read and provides developers with all the directions they’ll need for implementation. With this, you can finally start coding or assign this to your team and let them get to work!
This flow has room to grow and it’s just an example of how we can achieve some simple documentation that will save a lot of time in the long run. Just make sure to keep track of any changes. This way, once you are finished, you will know where to resume if needed — whether due to a new customer requirement, platform upgrade, maintenance, debugging, or any other reasons for which you, or the current system owner, may be required to engage with this project.
Conclusion
Having clear documentation will always be beneficial no matter the size of the project. It will be your guideline throughout your project life cycle and allow for a more reliable and maintainable system. Although documenting small systems may initially seem counterintuitive, I encourage you to embrace it. Maybe you will take some time at first, quite possibly even more time than coding itself, but as you practice and improve this skill you’ll see yourself doing it faster and faster, and whenever you run into a complex system filled with intricate details, you'll be glad you fostered this ability. So don’t give up and keep drawing!
Top comments (0)