DEV Community: Matthew Brookson

Pagination Pitfalls: Preventing Data Loss

Matthew Brookson — Mon, 25 Sep 2023 08:08:21 +0000

I recently ran into a peculiar problem at work. I had written a database query to bulk load millions of records from a data warehouse which worked by paginating over a large dataset. Everything seemed to be working okay... until we noticed some data was missing.

I initially wondered if it was a bug with the database (naive and unlikely I know) and after some investigation we finally discovered the problem. Pagination and ordering. It’s obvious in hindsight and felt like a silly mistake, so I wanted to share my learning.

The Problem: Ordering by a Non-unique Column

When working with databases and paginated queries, it's essential to consider the order in which your data is retrieved. If you're ordering your results by a non-unique column, you might encounter a common issue – missing rows. This happens because pages of data may return the same row that was already included in a previous set, leading to gaps in your results.

Imagine you have a table of user records and you want to display them in alphabetical order by last name. If you're fetching these records in chunks for a paginated user list, and two users have the same last name, you could end up with one of those users appearing on two pages and one of those users missing from your paginated results altogether.

Let's break down why this happens:

Initial Order: You start by ordering your data by the non-unique column (in this case, the last name) and fetching the first page.
Paginating: As you move through subsequent pages, you request the next set of data, but since the order is not unique, the database may return rows that were already included in the previous pages.
Missing Rows: This redundancy in rows can lead to some records not being displayed at all in your paginated results. Users with the same last name as others may end up being skipped, causing gaps in your list.

The Solution: Multi-column Ordering

To avoid missing rows in your paginated database queries, you can introduce multi-column ordering. Instead of relying solely on a non-unique column, add another column that provides a unique order. This could be using a combination of columns that are unique together, or to be safe you can use an auto-incrementing primary key, UUID or a timestamp that you know is guaranteed to be unique.

Here's how it works:

Ordering by Multiple Columns: When you order your data by multiple columns, you ensure that the combination of these columns is unique. In our example, you might be able to order by both last name and first name. This combination is far less likely to produce duplicate rows, but may not be impossible depending on your implementation. To be 100% sure, you could order by last name and user ID. This way we’re guaranteed that the ordering is unique.
Consistent Pagination: Now, when you paginate through your data, you're guaranteed to receive a distinct set of records on each page. There's no risk of missing rows, even when two users share the same last name.

Conclusion

This is such a fairly simple concept in the end, and super simple to implement. That said, it’s also easy to miss and it could cause all sorts of problems with missing data. In the world of paginated database queries, ensuring that your results are complete and accurate is crucial.

By implementing multi-column ordering, combining a non-unique column with a unique one, you can prevent missing rows and provide a seamless, error-free experience for your users. So, next time you're working on a paginated query, remember the importance of multi-column ordering to ensure the integrity of your data retrieval process.

The most productive tech-stack I've ever used

Matthew Brookson — Tue, 19 Sep 2023 20:43:59 +0000

I've been a software engineer for about 10 years now, and along the way I've developed a various side-projects that have tried, failed and been buried in the side-project graveyard.

Back in the day .NET was my go-to for everything. It's where I started and all I really knew. I used the ASP.NET MVC framework, jQuery and bootstrap. This was pretty cutting-edge back then, but times have changed. Build times were slow, deployment was complicated and slow. I was also pretty new to software development so I didn't even know about things like CI/CD.

In recent years the state of web technologies has changed and improved massively. Yes, there's a hell of a lot more choice available. New npm packages every day (sometimes feels like there's even a new whole framework released every day) and there's a lot to keep on top of, but it's also the best and most accessible time ever to get into web development.

So what's the stack?

These days I have a new go-to stack that feels like my most productive setup yet. With all the examples below I always prefer to use TypeScript. Let's run through the tech and I'll share my thoughts on why these tools are awesome!

Next.js

Probably the most popular React-based full-stack framework in use right now? It's fairly easy to learn, has a huge community and loads of help and advice online, and just does the job really well.
nextjs.org

tRPC

This library is truly amazing. Gone are the days of building API endpoints, then manually making calls from your frontend to these APIs, trying your best to keep them in sync. tRPC removes this friction entirely. Simply write a function that returns data on the backend and it provides a type-safe function that you can call from your frontend. Want to rename a property? Just refactor/rename and it updates everywhere. It's magical!
trpc.io

React Query

Manage all your loading state, caching and more, with a super simple and powerful library that handles all your data fetching needs. tRPC ships with a React Query integration and the two go perfectly hand-in-hand.
tanstack.com/query/v3

UI Components

This is pretty generic as I've tried a few tools and like them all. Mantine, Chakra, Material UI or shadcn/ui are all awesome. shadcn/ui is particularly awesome as it is all based on TailwindCSS so it's super customisable. In any case, using a library of components means you can get started with your main value proposition and feature development immediately without reinventing the wheel. All these library have nice defaults and are fairly easy to customise to your liking.
mantine.dev
chakra-ui.com
mui.com
ui.shadcn.com
tailwindcss.com

Prisma

A powerful type-safe database ORM that supports multiple databases. This is now my default choice as again, it's super simple to integrate, manages migrations nicely, and the type-safe queries are fantastic.
prisma.io

T3 stack

Special mention here to the T3 stack which is an awesome tool to help you get a project started with Next.js, Prisma, Tailwind and NextAuth (or any combination of the above).
create.t3.gg

Deployments and Hosting

For deployments I have two main preferred choices. Vercel, which is particularly suitable for Next.js projects, is free to use for hobbyists and small projects but you must consider its serverless nature and also potential for high costs later down the line. My other choice which I've used for a couple of recent projects is Railway, which has the ability to host and auto-scale docker containers and databases with very minimal setup. So far I've found it to be amazing and they are regularly releasing new features and updates. Both these options hook into GitHub and automatically build and deploy commits pushed to your remote main branch.
vercel.com
railway.app

Closing thoughts

With this stack I can now get a project created and deployed in a matter of minutes, in development in a matter of hours, and deployed with a core set of product features in days to weeks. It's truly amazing how fast it is possible to move with this stack. Just check out blogfactory.dev as an example of something I've been working on recently. It's a new AI startup that helps indie hackers like yourself generate blog posts for their own products, using OpenAI tools under the hood.

Don't get me wrong, this isn't a one-size-fits-all. I wouldn't necessarily use just this stack for literally everything and anything. For example, for my full time role in a large team of many engineers with a plethora of micro-service backends there's more to consider. I still love the .NET platform and there are many other great tech stacks for various different purposes. But for indie hackers who want to get their first tens, hundreds or thousands of users, this feels like near perfect solution.

P.S. Seriously, check out blogfactory.dev...

Is fragile software harming your user retention?

Matthew Brookson — Tue, 18 Jul 2023 10:51:29 +0000

The importance of resilient software in relation to user experience and retention

When it comes to keeping users engaged and coming back for more, nothing beats a software system that can handle the heat and deliver a consistently delightful experience. So, let's strap in and explore why resilient software is a game-changer for user retention.

Expectations of a smooth software experience

People have grown accustomed to software that just works. Your users don’t even notice that your software is working as they expect a seamless experience, where everything works like a well-oiled machine. However, when things don't work as expected frustration creeps in, leaving a lasting impression and potentially driving users away. Consistent uptime is a key factor in retaining users' loyalty and preventing them from seeking alternatives. All businesses have competitors and therefore users have options. If your software system experiences regular downtime, users won't think twice about abandoning ship and seeking greener pastures. By investing in resilient software, you're investing in the user experience, making it a top priority to keep your users happy and engaged.

Tip of the iceberg

You're using a software application, and suddenly it goes down. Annoying, right? Well, inconvenience is just the tip of the iceberg. A system failure can have serious consequences, potentially impacting revenue streams and even compromising user health in certain contexts. Whether it's an e-commerce site, a healthcare application, a bank, or anything in between, downtime is never a good thing. It’s sometimes critical, and always at least a buzzkill that can lead to frustrated users jumping ship.

WTF is going on?

Imagine navigating a software system where information seems to change its mind more often than a chameleon changes colour. Inconsistent data is a recipe for user confusion and frustration. Users rely on accurate and up-to-date information to make informed decisions and trust the system they're using. By providing a consistent and reliable flow of information, resilient software ensures users can rely on the data they interact with, maintaining trust and engagement.

Not all software systems require immediate consistency though. For certain scenarios, eventual consistency can be a viable approach and is often preferred from a technical point of view in a distributed system. By carefully considering the level and timing of consistency required, you can strike a balance that aligns with your users' expectations. It's crucial to communicate transparently about the degree of consistency provided, aiming to make it as seamless as possible and also ensuring users understand and accept any potential delays or discrepancies.

How to keep your users’ trust

Graceful error handling

I’m not saying things never go wrong. Errors are an inevitable part of any software system, but how you handle them makes all the difference. Instead of presenting users with cryptic error messages or leaving them in the dark, resilient software takes the high road. Sometimes you can handle errors silently and recover from issues without the user ever knowing. Otherwise, display meaningful and useful error messages, guiding users through issues and offering solutions or alternatives. By gracefully handling errors, you can better manage expectations, empower users to overcome obstacles and maintain their trust in your software.

Modularisation can minimise breakages

One small breakage shouldn't bring down the entire show. By breaking features into modules within your software you can ensure that a glitch in one area doesn't cascade and cause a catastrophic failure. Modularised code makes it far easier to maintain, particularly in big teams and large software systems. This approach allows you to isolate issues, fix them swiftly, and minimise the impact on the overall user experience. Having well organised code can even prevent creating bugs on the first place.

Observability, Monitoring, and Alerting

As with any complex system, maintaining a watchful eye is essential. Observability, monitoring, and alerting mechanisms play a vital role in discovering and resolving issues promptly. By proactively monitoring your software's performance, you can identify bottlenecks, anomalies, and potential failures before they wreak havoc on the user experience. Swift issue resolution leads to happier users, improved retention rates, and a reputation for reliability.

Conclusion

Building and maintaining resilient software is not only crucial for a good user experience but also a powerful driver for user retention. From minimising downtime and inconsistencies to gracefully handling errors, every aspect of resilient software contributes to keeping users happy and engaged. By investing in the robustness and reliability of your software, you can build a loyal user base that sticks around.

Designing resilient software architecture with the Transactional Outbox Pattern

Matthew Brookson — Thu, 13 Jul 2023 07:56:19 +0000

The title is a bit of a mouthful, I know. In this post we're going to explore the Transactional Outbox Pattern, a software architecture design pattern that focuses on building resilient software systems. As experienced developers, we know that failures are inevitable, particularly in distributed systems with communication between multiple services. It’s crucial therefore to build systems that can gracefully recover from these failures. So, let's dive in and learn how the Transactional Outbox Pattern can help us achieve just that. I'll also briefly preface this by acknowledging this is not the only way to achieve this and there are some downsides to consider which we'll cover too.

Understanding the Transactional Outbox Pattern

What's the Transactional Outbox Pattern?

The Transactional Outbox Pattern enables reliable message-based communication and transactional consistency, ensuring that your messages are persisted first and subsequently get delivered even in the face of failures. With this pattern, you can design systems that bounce back from errors and keep on ticking. Now let’s see how it works.

The Outbox Table: Storing Outgoing Messages

The idea behind the Transactional Outbox Pattern is to store messages in a special table in your application's primary database - the Outbox table. By doing this, we can ensure that each message is persisted and no messages gets lost, even if your system encounters a hiccup along the way. Once the message has been persisted in the Outbox table, a separate process should periodically scan the table for new messages and publish them to their intended destination.

But why I hear you ask? Let’s find out…

Transactions for Atomicity and Consistency

In the software architecture world, atomicity and consistency are like wine and cheese - a perfect combination.

Atomicity refers to the "all-or-nothing" nature of a transaction. In a software system, a transaction is a unit of work encapsulating a series of operations, such as reading from or writing to a database. Atomicity ensures that either all the operations within a transaction are completed successfully, or none of them are. If any operation fails, the entire transaction is rolled back, and the system returns to its previous state. This guarantees data consistency and prevents partial updates that could lead to inconsistent or incorrect results.

Consistency means data remains in a valid state before and after a transaction. In a consistent system, data is subject to predefined rules or constraints and adheres to some intended business logic. When a transaction is committed it must preserve data integrity and maintain any defined relationships or constraints. For example, if a transaction involves updating related data in multiple tables, consistency ensures that all updates are applied correctly and that referential integrity is maintained.

Now picture this: You have a critical task to perform, and it involves sending messages to different components in your system. Messages are often in the form of events which are essentially facts about something that has happened within the system. However, consider this. What happens if there is a failure in the process after the message has been published and our state never gets persisted? The consumer now receives a “fact” in a message and assumes everything is true, even if it may not be. Essentially, until any state changes have been persisted the fact can’t really be considered a fact and therefore the message should not be published..

The Transactional Outbox Pattern provides a transactional mechanism that allows you to store messages with the main transaction, guaranteeing atomicity and consistency. This ensures that messages are either successfully stored along with the main operation or rolled back together if any part fails. It's like having a safety net that catches your messages when things don't go as planned.

Reliable Message Brokers: Ensuring Reliable Delivery

To complete our resilient system, we need a reliable message broker that guarantees message delivery. Popular options like RabbitMQ, Apache Kafka, or AWS Simple Queue Service (SQS) come to the rescue here. These message brokers ensure that your messages reach their intended recipients, even if failures occur during the process, like a trustworthy postman who never loses a letter.

Best Practices and Considerations

Designing resilient software systems requires thoughtful planning and attention to detail. Here are some best practices and considerations to keep in mind as we implement the Transactional Outbox Pattern:

Handling Failures in the Message Publishing Process

Let's face it - things can still go wrong during the message publishing process. Your message destination service or broker may go down or there may be network issues, for example. The Transactional Outbox Pattern has your back in these situations. Imagine you’re working on a critical payment processing feature, and suddenly, your payment gateway becomes unavailable. With the Transactional Outbox Pattern, you can ensures messages are stored atomically, allowing you to retry the payment processing flow when the gateway is back up and running.

Ensuring Exactly-Once Delivery

Duplicate messages can cause problems. However, the Transactional Outbox Pattern guarantees exactly-once delivery. How? By employing techniques like deduplication and idempotency. So, even if a network hiccup causes a message to be published twice, your system can recognise the duplicate when storing in the Outbox and handle it gracefully, preventing unintended duplicate messages being stored and sent. Of course, it’s always a good idea to consider duplicate messages on the consumer side too. This can be done by handling messages with idempotency so they can be received multiple times successfully or by ensuring if a message has already been seen it is not processed a subsequent time.

Observability

As is true for any distributed system, good monitoring and alerting are essential. Monitoring provides insights into system performance and behaviour, allowing proactive identification of issues and optimisation opportunities. Alongside monitoring, robust alerting ensures prompt notifications of critical events, enabling quick response and resolution. With effective monitoring and alerting in place, you can maintain the stability, availability, and reliability of your distributed system.

Potential Downsides

While the Transactional Outbox Pattern offers numerous benefits for building a resilient software architecture, it's important to be aware of its potential downsides. One consideration is the potential impact on database load. Storing outgoing messages in the Outbox Table adds an additional write workload to the database, especially during high-volume periods. Subsequently, it also increases the read workload because a periodic job has to query for new messages and lock them so they're not consumed by multiple processes.

Another thing to consider is some frameworks are well equipped to not require the Transactional Outbox Pattern. Take Entity Framework from .NET for example. It allows you to implement a unit of work pattern to wrap changes in a transaction and exposes an OnSaveChanges hook from the DbContext which allows you to run some code after the transaction is committed. This means that you can solve the atomicity and consistency issue without needing to persist the messages to your database first. That said, you may decide that it's still valuable to persist published messages for auditing amongst other reasons.

Conclusion

We’ve explored the Transactional Outbox Pattern and discovered its power in designing resilient software systems. By leveraging this pattern in your software architecture, handling failures, ensuring exactly-once delivery, and implementing observability, you can build robust systems that maintain data consistency and gracefully recover from disruptions. As with anything in software development, there are many considerations and this isn't a one-size-fits-all solution. Certainly try it out and take the time to consider whether this is the right fit for your problem.

Now, armed with the Transactional Outbox Pattern and your newfound knowledge, go forth and create software that stands the test of time and most importantly leads to satisfied users.

Tell, don't ask: Domain-driven code refactoring

Matthew Brookson — Mon, 10 Jul 2023 20:14:13 +0000

I bang on about domain-driven design a lot. It’s an interesting and vast topic and can be difficult to know where to start. Especially if you’re working on an existing project with lots of existing code.

Turns out there are some simple but effective ways to start implementing some DDD concepts into any codebase immediately.

One of these is “Tell, don’t ask”. Let’s discuss.

// Domain object
class Person {
  private isWearingSocks: boolean;

  canPutOnSocks() {
    return !this.isWearingSocks;
  }

  putOnSocks() {
    this.isWearingSocks = true;
    this.publish(new PersonPutOnSocksEvent());
  }

  removeSocks() {
    this.isWearingSocks = false;
    this.publish(new PersonRemovedSocksEvent());
  }
}

// Calling code
let canPutOnSocks = person.canPutOnSocks();

if (canPutOnSocks) {
  person.putOnSocks();
} else {
  throw new PersonCannotPutOnSocksError();
}

Here we have a person instance and some logic to decide whether the person wear some socks. There is nothing wrong with this at a first glance. The code works and the rules are checked as they should. But let’s think about it a little more.

Can a person put socks on?

What if we have two places where we call the putOnSocks function? Can the person put socks on? We would probably need to check using an if statement everywhere we want to call this function in case they can’t. Code duplication isn’t always a bad thing, but this is certainly something to keep in mind.

Why? Why? Why?

Another consideration is that although we’re checking whether the person can put socks on, if they can’t we are throwing a PersonCannotPutOnSocksError error. This is also fine, but it’s not very helpful. Why can’t they put socks on?

Breaking the rules

Most importantly, what we’ve discovered here is some domain logic. More specifically, there’s an invariant which is a validation rule that must be enforced by the domain object for it to be in a valid state.

In this case we have a rule that a person cannot put socks in if they already have socks on. The rule was already there in the original code but it was outside of the person class and would need to be repeated everywhere to ensure the rules were followed. If the rules are not enforced we would publish two consecutive PersonPutOnSocksEvent events. Can you put socks on if you haven’t taken the previous socks off? (Alright smarty pants, maybe you technically could but not in my made-up scenario). And what about removing socks? Can you remove socks if you’re not wearing any? Certainly not! So we definitely shouldn’t be able to raise consecutive PersonRemovedSocksEvent events, but the code allows us to do just that. We’re essentially leaving our code open to logic bugs.

Let’s fix it.

Tell, don’t ask

Let’s refactor this code using the “Tell, don’t ask” approach.

// Domain object
class Person {
  private isWearingSocks: boolean;

  private canPutOnSocks() {
    return !this.isWearingSocks;
  }

  putOnSocks() {
    if (this.isWearingSocks) {
      throw new PersonAlreadyWearingSocksError();
    }
    this.isWearingSocks = true;
    this.publish(new PersonPutOnSocksEvent());
  }

  removeSocks() {
    if (!this.isWearingSocks) {
      throw new PersonNotWearingSocksError();
    }

    this.isWearingSocks = false;
    this.publish(new PersonRemovedSocksEvent());
  }
}

// Calling code
person.putOnSocks();

Firstly you’ll probably notice there is less logic in the calling code. Why’s that? Essentially we’ve encapsulated our domain logic, moving it inside the domain object itself rather than calling it all from outside. This may seem trivial, but there are some major benefits.

Privacy first

Since we’re no longer asking whether the person can put on socks we are able to make the canPutOnSocks function private. This isn’t required and we could still expose this function if needed, but what it highlights is that we have more fine grained control over what behaviours and functionality we want to expose from our domain objects. Generally speaking, we should aim to expose as little as possible.

Enforcing invariants

By encapsulating the domain logic inside the person class we now have better control over the invariants and can enforce them within the class itself. For example, now we can tell the person class to put on socks without asking first. The domain object is now responsible for ensuring that the person is not already wearing socks. Now the logic is encapsulated we can more easily throw a specific error too since we know why the person cannot put on socks - because they’re already wearing some. The same logic applies to removing socks as mentioned in the previous section.

Since we’re now encapsulating this logic we’re also ensuring that the correct domain events are published. We can no longer publish consecutive PersonPutOnSocksEvent or PersonRemovedSocksEvent events. We would need to tell the person to remove their socks before putting socks on.

Testing

Before we had logic outside of our domain object. How would we test it? We’d need to test our calling code to ensure the invariants are enforced. What if we miss one? What if there’s a mistake or a conflict of rules? Now we’re refactored the code, our encapsulated domain logic is easily testable in one place.

To conclude

Hopefully this over-simplified example demonstrates why the “Tell, don’t ask” approach can be useful when writing code. It’s a simple and effective way to write concise domain objects, enforce invariants and make your code robust and testable.

It’s something you can implement whether you’re starting a project from scratch or making changes to an existing codebase. You don’t need to rewrite everything but you can introduce this pattern as small, incremental refactors.

If you found this interesting or useful then drop me a follow on Twitter @matthewbrookson.

You can find more about domain-driven design in my ebook at https://mbrookson.uk/products/dicovering-ddd. Check it out!

Level up your terminal: How to configure autocomplete in your macOS terminal

Matthew Brookson — Mon, 10 Jul 2023 20:06:21 +0000

The terminal is a powerful tool. It's useful for many tasks, from creating, copying, moving and deleting files, to executing scripts and running programs. However, its default settings are not particularly user-friendly, and as software engineers we're used to our code editors helping us out.

Don't despair! With some simple changes you can drastically improve the zsh terminal experience by adding two super useful features - autosuggestions and syntax highlighting. And if you've just got a new MacBook like me then this is the perfect time to level up your setup!

Follow these simple steps to get set up:

Use Homebrew to install zsh plugins

Install zsh-autosuggestions
Install zsh-syntax-highlighting



brew install zsh-autosuggestions
brew install zsh-syntax-highlighting

More plugins, including these two, can be found in the zsh community project repo on Github.

* This step assumes you already have Homebrew installed. I'll do a post on this soon! If you don't use Homebrew then check out the docs linked above for alternative ways to install.

Add to your `.zshrc` file

If it doesn't already exist, create this file in the root of your user directory. The easiest way to do this is to run touch ~/.zshrc

Add the following lines to the .zshrc file to import the code for the plugins



source $(brew --prefix)/share/zsh-autosuggestions/zsh-autosuggestions.zsh
source $(brew --prefix)/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

For more information about this file and other zsh configuration files, see this post.

Active the zsh changes

For your changes to take effect either restart your terminal window or run this command to force zsh to reload its sources in .zshrc:



source ~/.zshrc

And that's it! 🎉

Try it out for yourself. Now when you type in the terminal you should see syntax highlighting. Let's take the mkdir test command for example.

Typing m shows in red to highlight this is not a valid command.

Continuing to type mkdir turns it green, highlighting that this is a valid command! Complete running this command with mkdir test.

Now this command has been run, the next time we type m the auto-suggestions has remembered the previous command and suggests it to us. Just hit the right arrow to auto-complete
the suggestion!

I hope this small change facilitates your workflow and makes using your zsh terminal shell more enjoyable!

For even more options to enhance your terminal check out projects like oh-my-zsh and the Fish shell. However, the simple changes outlined in this post are a great addition to level up your terminal
experience.

The value of value objects

Matthew Brookson — Mon, 10 Jul 2023 14:29:15 +0000

Value objects are a fundamental concept from domain-driven design that allow us to model values as rich objects. The idea is that representing values as primitives (i.e. string, number, date etc) can mean we are lacking context and meaning in our software. This is sometimes referred to as "primitive obsession". Instead, we can model concepts in our domain as value objects that encapsulate both data and behaviours.

Value objects are immutable objects that are distinguishable only by the state of their properties. They have no unique identifier. If a value object’s properties are the same as those on another value object of the same type they are considered equal. To check if two value objects are considered equal we must compare all of their properties.

The value of value objects

Now we know what value objects are, let’s take a look at some benefits of using them.

Take the concept of money for example. Money could be modelled as a simple number, but what behaviour are we hiding? Can the amount be negative? What about currency? Money is always of a specific currency. Can we add two currencies together? Probably not as different currencies have different values and simply adding doesn't account for exchange rates.

If you have $1 and I have $1, we don’t care or know who has which \$1. If we exchange that money, we both end up with the same amount of money. In the real world it makes no difference who has which physical bit of money, and in a computer system there isn’t even any physical money. Money has no unique identity, and so long as the properties of the money are the same we can treat them as equal. A value object may therefore be better suited here than a number.

We could create a value object that has a currency and amount property.

enum Currency {
  USD = 'USD',
  GBP = 'GBP',
}

type MonetaryAmountProps = {
  currency: Currency;
  amount: number;
};

class MonetaryAmount {
  get currency(): Currency {
    return this.props.currency;
  }

  get amount(): number {
    return this.props.amount;
  }

  constructor(private props: MonetaryAmountProps) {}

  equals(other: MonetaryAmount): boolean {
    return this.currency === other.currency && this.amount === other.amount;
  }

  add(other: MonetaryAmount): MonetaryAmount {
    if (other.currency !== this.currency) {
      throw new Error('Cannot add amounts with different currencies');
    }

    return new MonetaryAmount({
      currency: this.currency,
      amount: this.amount + other.amount,
    });
  }

  // more operators like subtract and multiply could exist here...
}

To create an instance of a MonetaryAmount we can see that a currency and amount are required via the constructor, intrinsically linking the two values. Since currency is defined as an enum we know we can only create a monetary amount with a valid currency.

Next, rather than adding two numbers together that could represent two different currencies, we expose an add() function on the value object which performs the add operation. This function performs a check to ensure the currencies of the two monetary amounts being added are the same - if not then an error is thrown. The domain behaviour is encapsulated and ensures that business rules are not broken.

Adding two monetary amounts together returns a new instance of MonetaryAmount because value objects are immutable. This immutability is important because it ensures that when two values objects are created equal they will always be equal. The equality of two value objects can be checked in this example via the equals() function.

To those familiar with it, this may just sound like object oriented programming. In a way, you're right. But the important thing here is we're specifically modelling value objects based on behaviours that are specific to a domain. This is a contrived example but when you have lots of business rules this a great way of encapsulating logic that's easy to understand and reason with.

Want to learn more?

You can try using value objects for yourself by taking an existing primitive value in your codebase and rethinking what domain logic you may be hiding by modelling it as a primitive. You may be surprised how many primitive values actually have more meaning than first meets the eye!

If you enjoyed this post then be sure to check out Discovering Domain-Driven Design, my ebook where I cover more about value objects and other DDD concepts that can help to model business logic and build complex software.

What is domain-driven design?

Matthew Brookson — Mon, 10 Jul 2023 14:26:06 +0000

Developing software is challenging. It's not just writing code. It's everything from discovering a problem, doing process and product analysis, developing requirements, writing code and iterating on the process.

As software engineers, most of us have experienced a “spaghetti” or "big ball of mud" codebase. A codebase with no clarity, where there are dependencies in all directions. This is a symptom of code that is often highly coupled, fragile and difficult to test and maintain. A change to one part of the code may have a big knock-on effect.

There is often a common cause for this - developers are trying to build software solutions with a lack of business knowledge and understanding. When this happens we end up with software that does not truly represent business functionality. A traditional waterfall approach to developing software can be a root cause of this because it focuses on one person or team gathering information and requirements, then passing to engineers to build and deliver. There's a huge gap of knowledge transfer and collaboration.

Domain-driven design approaches software development with a focus on engineering teams gaining a rich understanding of business rules and processes from domain experts. It involves engineers being involved early on in the discovery phase and working closely with the product teams and domain experts directly. The aim is to understand as much about the domain and the business as possible. This has many positive impacts.

It provides context
It aligns the engineering team with the business
It reduces ambiguity and information lost in transit
It helps to improve communication because a common language can be established
It facilitates changes because everyone's on the same page
It decreases complexity of inherently complex problems

Although there are technical patterns engineers can use when writing code, it’s important to remember that domain-driven design is not just about code. It is not purely a technical concern. It is a wider concept that can be utilised by other team members including product teams and key project stakeholders. It requires engineering teams to seek understanding of the problem space before trying to solve the problem, and modelling the domain using that understanding. It involves establishing a common ubiquitous language that improves communication and helps to create software solutions that are clearly aligned to business processes.

If we align the software we build to the business we can write better, more easily maintainable and testable software. We can also communicate more effectively, thus enabling us to tackle software complexities when solving complex problems.

For more information, check out my ebook "Discovering Domain-Driven Design" for a distilled overview of domain-driven design and how you can use it to build better software.

Are your tests valuable?

Matthew Brookson — Mon, 10 Jul 2023 13:06:15 +0000

Do you write automated code tests? If you do, why do you test the things you do? What techniques do you use?

In this post I'll cover why we should test code and the different ways you can write valuable tests.

Why do we test?

First, let's think about what automated test provide. What's the point?

Confidence

Tests give us confidence. When we write code we want to be confident that it's working as intended. You might be sure your code is working at the time of writing it and not see the value in spending further time writing tests, the benefits really increase over time. As our application and code changes, tests give us confidence that the changes made do not unintentionally break existing functionality. This is valuable because it means we can move more quickly without worrying whether there have been unintended consequences and enables us to ship a high quality product more regularly.

Cost

Automated tests are inexpensive compared to manual testing. You can run thousands of automated tests in seconds. It would be extremely difficult to manually test your entire application every time a change was made. Also as your application grows you would have to hire more manual testers who's salary you have to pay. On the other hand, automated tests can test your entire application on every small change and run cheaply or even for free. For example, it's simple and free to set up automated tests for a Node project using Github Actions!

Reliability

Of course there is still a human element to automated tests because they must be written by someone in the first place. However, since the tests are run by a computer they can be repeated over and over again reliably and produce the same results. If you test manually there is much more room for human error.

Types of automated testing

There are three main types of testing we can utilise.

Unit tests
- Test individual parts of your application in isolation. They often involve mocking dependencies so you can truly test the specific implementation of the code in question. These are most useful for testing complex logic.
- e.g. Test a service that depends on a database repository performs the correct logic, but mock the repository so we can test the service in isolation.
Integration tests
- Test that several units work together and use their actual dependencies rather than mocks. Generally integration tests will test input and output of a function.
- e.g. Test a service that depends on a database repository without mocking. This way we can test the service performs the expected actions on the repository and returns the correct result from the database.
E2E tests (end-to-end tests)
- Test the full stack of your application by imitating how an external person or program would interact with it.
- e.g. Spin up an instance of the real application, click a button on the screen and test that the user is presented with the expected result, like it has displayed an alert or navigated to a new screen with the expected content.

So how do we make tests valuable?

Over the last year I've been thinking more and more about what makes tests actually valuable. It's a valid concern that tests can add a slight overhead at the time of writing them, so we want to make sure our tests are going to provide value in the long run.

For a while I thought it was vital to test every line of code and try to achieve 100% unit test coverage. I've recently changed my opinion on this though, and here's why.

It's important to consider time vs value. As we discussed earlier, we write tests to give us confidence. With that in mind, do we really need to take the time to write a test for a very simple function we have when we're already confident it works fine just by looking at it?

I believe, rather than aiming for a blanket 100% test coverage, you should aim to test areas which are particularly critical or areas that have complexity and you need extra assurance that things are working properly. I also believe integration and E2E tests provide the more value than unit tests. Firstly, your users are the most important factor of your business. This means it's important to test your application from the view point of the end user to ensure it's working as they would expect.

Last year I had to perform a large .NET solution refactor which involved moving almost all files in the solution. Unit tests mostly still passed, even though the entire application was broken, because they were only testing the individual units in isolation. The real value was having integration tests. Once these had passed we had the confidence that the application was actually working and could deploy it.

Closing thoughts

Tests are super important and can definitely be valuable in the long run. Here are some key points to take away and consider in your existing and future projects.

Write unit tests for complex logic but don't worry about testing every single line of code in other places.
Write integration and E2E tests for the majority of your application to give you the greatest amount of confident your application is working as expected.
When writing tests, consider how much value and confidence you are getting for the amount of time it takes to write. It might seem a pain to spend a few extra hours writing tests, but it could save you 10x that in manual testing or bug fixing in the future!

I've been inspired by recently by Kent C. Dodds who is the author of the https://testingjavascript.com course and https://testing-library.com library which I highly recommend for front-end testing. Also check out his explanation of the "testing trophy" model rather than the more common "testing pyramid".

The secret to writing code that speaks for itself

Matthew Brookson — Mon, 10 Jul 2023 12:47:43 +0000

In the world of programming, documentation is essential for developers to understand how a particular codebase works. It's a key component of a successful engineering team. It helps developers to understand the purpose of the code and how it works so they can contribute to projects, adding features and fixing bugs. There are several ways to document code, including comments, README files, and wikis like Confluence or Notion. The problem with this type of documentation, particularly those that are external from the code, is that once a document has been created it often loses it's connection with the project and is not maintained over time as the code evolves. However, well-written code itself can be a form of documentation. This blog post explores how well-written code can serve as documentation.

As a developer, you might be familiar with the concept of "self-documenting code" or "readable code". The idea is simple: when you write code that is easy to read and understand, you are essentially documenting via the code itself. This means that other developers who work with your code (including future you) will be able to understand what your code does and how it does it, without needing to read through a separate document or ask you for clarification. In other words, well-written code can be its own documentation.

So what makes code "well-written"? Here are a few characteristics:

Clear Naming Conventions

One of the most important aspects of readable code is clear, consistent naming conventions. This means using descriptive names for variables, functions, and classes that accurately reflect their purpose and functionality. For example, instead of using generic names like "foo", "bar" “x” and “y”, use names that describe what the variable or function actually does. By using descriptive names, you can make it much easier for someone else to understand your code without needing to read through lengthy comments or documentation. It's always a good sign if your code reads almost like a natural sentence, just like you might speak it out loud. Making sure to use ubiquitous language is also key to ensure the same terminology and concepts mean the same thing throughout the code and the rest of the business.

Well-written code can reduce the need for further excessive documentation, making it easier for developers to get up to speed and contribute to the project.

Modularity

Well-written code is often modular, meaning it is broken down into smaller, more manageable pieces. This makes it easier to understand and modify, since each piece has a specific purpose and can also be tested independently. Bounded contexts are also relevant here, making your code more cohesive and enforcing boundaries around related domain concepts. Breaking down your code into modules makes it clearer where code should live and can make it considerably easier for newcomers to get started on a project. Modules can also make it easier to reuse code in future projects or to share code with other developers. This can save you and others a lot of time and effort in the long run.

Consistency

Well-written code is consistent, both in terms of coding style and overall structure. This means that it adheres to industry best practices and follows established patterns and conventions. By following established conventions, you can make it easier for other developers to understand your code and to work with it. This can also make it easier for you to understand and work with code written by others, since you will be familiar with the conventions they are following. There are many successful and well known design patterns and practices that you can follow. Check out patterns.dev for some great explanations and examples!

Conclusion

When you write code that meets these criteria, you are essentially creating documentation at the same time. This can be especially valuable for larger, more complex projects, where understanding the overall architecture and functionality can be difficult and engineers come and go over time taking their knowledge with them. Of course, even the best-written code can still benefit from additional documentation and this post is by no means advocating against that; in fact I highly encourage it. Code should be supplemented with comments, external documentation, and other resources to ensure that developers have all the information they need to work effectively.

However, by focusing on creating well-written, self-documenting code, you can greatly reduce the need for extensive further documentation and make it easier for others to work with your code. Naturally, if code is self-documenting it is updated as it changes, whereas additional separate documentation can often get outdated unless it's maintained with discipline.

Also note that the concepts in this post, including ubiquitous language and bounded contexts, are fundamental concepts from the domain-driven design philosophy.

So the next time you sit down to write some code, remember that the code itself can be its own documentation if it is well-written and easy to understand. Your future self and other developers will thank you!

Language matters: The power of ubiquitous language

Matthew Brookson — Mon, 10 Jul 2023 08:30:00 +0000

I’m not talking about programming languages here (although some languages have features that support domain-driven design better than others). I’m talking about the power of ubiquitous language.

In the world of software development, effective communication is paramount. Miscommunication can lead to misunderstandings, bugs, and ultimately, a software product that falls short of meeting users' needs. That's where ubiquitous language comes in - a language that spreads to every corner of your software development process.

But what exactly is ubiquitous language? Think of it as a bridge that connects developers, domain experts, and stakeholders, enabling seamless and meaningful conversations about the software's purpose, features, and behaviour. Ubiquitous language is a common vocabulary or terminology that everyone can understand and use to discuss the domain-specific concepts and processes.

Imagine you're building a new e-commerce platform. You’re using technical jargon like "data transfer object" or "entity" in your code, then trying to communicate with non-technical people around that. It usually goes one of two ways - you start using that technical jargon with them and they have no idea what you’re talking about, or you try to translate the concepts you think about as a developer into concepts the person you’re talking to understands. This is very challenging, a huge cognitive load and from my experience always leads to miscommunication.

Ubiquitous language encourages you to speak the language of the business domain. Talk about "products," "customers," and "orders" instead of technical jargon. Talk about "add to cart" rather than "create basket item", or "checkout" rather than "create order". Learn about the domain concepts and what they mean. By aligning the language you use with the stakeholders and domain experts, you're fostering a shared understanding that can shape the software's design and behaviour more accurately.

But the power of ubiquitous language goes beyond communication. It influences the structure of your software. By using domain-specific terminology consistently in your code, you create a codebase that reflects the language of the business, making it easier to maintain, extend, and reason about. With this approach, your code reads in a way that’s easy to speak about with other people in the business.

Now, this doesn't mean that ubiquitous language is a silver bullet that instantly solves all communication and software development challenges. It requires effort, collaboration, and a genuine desire to understand the domain deeply. It demands active participation from developers and domain experts alike, continuously refining and evolving the shared language as the understanding grows.

Remember, language matters. Create software that speaks the language of the business, avoiding technical jargon and promoting a deeper understanding of the domain. In the realm of software development, ubiquitous language is the secret ingredient that unlocks the potential for building high-quality, domain-aligned software solutions.

DEV Community: Matthew Brookson

Pagination Pitfalls: Preventing Data Loss

The Problem: Ordering by a Non-unique Column

The Solution: Multi-column Ordering

Conclusion

The most productive tech-stack I've ever used

So what's the stack?

Next.js

tRPC

React Query

UI Components

Prisma

T3 stack

Deployments and Hosting

Closing thoughts

Is fragile software harming your user retention?

The importance of resilient software in relation to user experience and retention

Expectations of a smooth software experience

Tip of the iceberg

WTF is going on?

How to keep your users’ trust

Graceful error handling

Modularisation can minimise breakages

Observability, Monitoring, and Alerting

Conclusion

Designing resilient software architecture with the Transactional Outbox Pattern

Understanding the Transactional Outbox Pattern

What's the Transactional Outbox Pattern?

The Outbox Table: Storing Outgoing Messages

Transactions for Atomicity and Consistency

Reliable Message Brokers: Ensuring Reliable Delivery

Best Practices and Considerations

Handling Failures in the Message Publishing Process

Ensuring Exactly-Once Delivery

Observability

Potential Downsides

Conclusion

Tell, don't ask: Domain-driven code refactoring

Can a person put socks on?

Why? Why? Why?

Breaking the rules

Tell, don’t ask

Privacy first

Enforcing invariants

Testing

To conclude

Level up your terminal: How to configure autocomplete in your macOS terminal

Use Homebrew to install zsh plugins

Add to your .zshrc file

Active the zsh changes

The value of value objects

The value of value objects

Want to learn more?

What is domain-driven design?

Are your tests valuable?

Why do we test?

Confidence

Cost

Reliability

Types of automated testing

So how do we make tests valuable?

Closing thoughts

The secret to writing code that speaks for itself

Clear Naming Conventions

Modularity

Consistency

Conclusion

Language matters: The power of ubiquitous language

Add to your `.zshrc` file