How to set up a slim Project Architecture that scales

Over the years people have reinvented this wheel dozens of times, from Domain Driven Design to hexagonal architecture, with some vertical slicing in between. And as always there are pros and cons to each but ultimately you could do everything with every single architecture. The most important part for scaling a backend seems to be bounding by business context, this keeps the scope of a single feature small and doesn’t lead to a single repository folder with hundreds of repositories.

For my most recent projects I started adopting vertical slicing, or at least my version of it, that was influenced by context boundaries and my first experience with it was great. A downside of this, which for many is a no go, is that I am coupling heavily between the API and the code structure. If you read my last post (I switched REST for RPC, and it feels great), you know that I think that this isn’t as bad as it's always made out to be, since most backends will only ever serve one API. And an upside of this is the discoverability, just clicking through the codebase you know exactly what does what, without reading a line of code. So this is the logical continuation of this, but this time I want to go into more specifics how and why I do those things the way I do.

The Premise

For simplicity sake, let’s start the all time favorite for tutorials, a ToDo app.

For this we need an endpoint to create, update and delete an entry. These endpoints need to check the auth status and communicate with a database. Then, just to see how we interact with a different business context, let's say we also have a leaderboard that shows the users with the most finished ToDo’s, that is another endpoint that can just return the leaderboard. I think this should roughly cover what most API’s do.

The Scaffolding

As always we need some sort of project structure, for this example I will be using typescript (I figured it's the easiest to read for most) with gRPC, but it just as easily translates to other languages. Then my base directory structure is a standard monorepo structure, on the root level I just have a “/app” and a “/lib”. If you use a nx monorepo, this is just the default structure, but even without it I came to like it as a nice start. In the app folder for now I only have a single todo folder, that serves as the entrypoint for our backend.

In the library folder things get more interesting, this is where the meat of our project lives. Since we are generating our APIvia gRPC we first need a place to store our proto files. If you wanna do the same thing with an openAPI spec first approach, this will work exactly the same. I chose to place my definitions under shared/API-interfaces, I try to avoid polluting the root library folder with anything that is not a business domain, so I just put them under shared, you could also create a core root library and put it there, but this is just my preference. Next up we have our generated files from those definitions. I put them under shared/sdk, but once again matter of preference, I can see those being generated inside the domains themself, which I tend to avoid to minimize the generated content in there, or again at the root level. The final two folders I create in there are an auth folder, which has our interceptor that validates the user as well as a helper function we will use later, and a db folder that, well, handles our database. This leaves us with our two root level libraries, defined by their domain: leaderboard and todo. All in all that leaves us with this general shell of a project

The Domain implementation

With this all out of the way, let's get started with the important architectural decisions. Every domain is divided into two sub-libraries, ‘core’ and ‘service’. Inside the core library we handle the setup and configuration for this, that means: 

• Database migrations

• Schema definition files

• Possible configs for third party APIs

• Etc.

The service library implements the service/controller linked to this domain. At the root level of this service we have a single service.ts file, this is just the orchestrator it implements each endpoint, handles the network specific stuff like loading metadata from the request and then passes everything needed to the handler for the specific endpoint. That means that each endpoint is usually only one line long.

Then each endpoint has its own directory named after the functionality it handles, for example a get-todo endpoint would have a folder named get-todo. This way you can instantly see which functionality is implemented where. Everything that this endpoint uses in this domain is implemented here, based on how averse you are to files this could either be a single file, or, if you are like me and like files for organization, every functionality gets its own file. So what are those functionalities:

• command.ts: the orchestrator of this functionality, this is the handler, that the service.ts calls and is the only part that is necessary, the rest may or may not exist, based on what you need

• mapper.ts: this file contains all your mappings, from the API models, to internal models, to database models. Disclaimer: I prefer to manually write mappers, I like the safety that comes with it, if you use automappers you probably won’t need this

• models.ts: this file contains all you intermediate model definitions, if you just need to map the API model to the database model, you probably won’t need this

• query.ts: this file contains all your database queries that this endpoint needs. And yes, that means every query for this endpoint, no shared queries between endpoints or modules. (A little more explanation why later)

Those are at least the ones I find myself using the most, but there is no strict rule, need a comparer? Add it. Need a call to another API? Just add it here.

Maybe you already spotted and got hung up a little on the name command.ts, since most of the time this would be called handler. But calling this code a handler makes it sound like it will only ever handle an API event, or at least that's what this name screams to me. A command is something that can be used everywhere and that is exactly what those commands are in this context. I will expand upon this in more detail when I will shortly move over to code examples, but in this system, the todo Domain is not allowed to directly interact with the leaderboard domain and vice versa. But when closing or reopening a todo Item, we need a way to change the leaderboards. For this we can just call a command from the leaderboard domain that does this for us. Now we have a clear ownership over different parts of the database, if anything ever happens to the leaderboard, we know it happened inside the leaderboard library.

The implementation

For this example I used connectrpc for the gRPC stuff and drizzle for the database interactions, but I won’t go into details about how they work, since this is about architecture, not typescript libs. Also the code I am about to show you is definitely spotty, to keep the examples clean I did not include any kind of error handling, data validation, testing, etc., so please keep that in mind and if you want to see how to integrate this just let me know and I will do a follow up.

With all that said, let's get into the specifics. Let's take a look at the service implementation for the todos:

import type {
  CreateToDoRequest,
  DeleteToDoRequest,
  GetToDosRequest,
  ToDoService,
  UpdateToDoRequest,
} from "../../shared/sdk/todo/v1/todo_handler_pb";
import type { ServiceImpl, HandlerContext } from "@connectrpc/connect";
import { getUserId } from "../../shared/auth";
import { updateToDoCommand } from "./update-todo";
import { createToDoCommand } from "./create-todo";
export const todoService: ServiceImpl<typeof ToDoService> = {
  async createToDo(req: CreateToDoRequest, context: HandlerContext) {
    return await createToDoCommand(req, getUserId(context));
  },
  async updateToDo(req: UpdateToDoRequest, context: HandlerContext) {
    return await updateToDoCommand(req, getUserId(context));
  },
  ...
};

As you can see, this service doesn’t do much, it uses the helper function of our auth implementation to get the users id from the context and passes this along with the request to the handlers. This is important as the system grows larger, the amount of endpoints per service increases, so if you later on have 50+ endpoints in a single service, and have 10+ lines of implementation for each endpoint, this gets unwieldy fast. I try to keep it to 3 lines tops per endpoint, but usually a single line is enough.

Then we reach the heart of the system, a command, lets start with a simple on, the createToDo

import type {
  CreateToDoRequest,
  CreateToDoResponse,
} from "../../../shared/sdk/todo/v1/todo_handler_pb";
import { dbToResponse, requestToDb } from "./mapper";
import { insertToDo } from "./query";

export async function createToDoCommand(
  params: CreateToDoRequest,
  userId: string
): Promise<CreateToDoResponse> {
  const data = requestToDb(params, userId);
  const result = await insertToDo(data);
  return dbToResponse(result!);
}

Again, not much to see here, but it shows exactly how a command is supposed to work, it’s mostly responsible for orchestrating how the data flows. In this example it uses the mapper to transform our request object into the correct format for the database, inserts this data via our query into the database, and finally maps the database response to the response object. If you are interested, here is the implementation of the mapper

import { create } from "@bufbuild/protobuf";
import {
  CreateToDoResponseSchema,
  ToDoItemSchema,
  type CreateToDoRequest,
  type CreateToDoResponse,
} from "../../../shared/sdk/todo/v1/todo_handler_pb";
import type { NewToDo, ToDo } from "../../core/db/schema";

export function requestToDb(
  request: CreateToDoRequest,
  userId: string
): NewToDo {
  return {
    userId: userId,
    title: request.title,
  };
}

export function dbToResponse(entry: ToDo): CreateToDoResponse {
  return create(CreateToDoResponseSchema, {
    item: create(ToDoItemSchema, {
      id: entry.id,
      title: entry.title,
      isCompleted: entry.completed,
    }),
  });
}

But the only thing special here is how to create a response item that protobuf can handle, but that's not the point. And lastly we have our query

import { db } from "../../../shared/db";
import { todos, type NewToDo, type ToDo } from "../../core/db/schema";

export async function insertToDo(todo: NewToDo): Promise<ToDo | undefined> {
  const [newEntry] = await db.insert(todos).values(todo).returning();
  return newEntry;
}

Well, that's even simpler, depending on how you interact with your database, this is maybe a line longer or two, but usually that's it. And that is the beauty of this architecture. If someone new starts working on this project and you tell him to check why inserting a ToDo item isn’t working, he will be pretty fast to spot where the logic is and isn’t having a hard time navigating through multiple layers of Controllers, Services and repositories. But let's take a quick look at one slightly more complicated example, that I already teased earlier, updating todo. Again nothing too complicated, but what we do need to do here is load the currently saved item, save the new one and then check to see if we need to update the leaderboard.

So lets take a look at our command

import type {
  UpdateToDoRequest,
  UpdateToDoResponse,
} from "../../../shared/sdk/todo/v1/todo_handler_pb";
import { handleLeaderboard } from "./leaderboard-handler";
import { dbToResponse, requestToDb } from "./mapper";
import { getToDo, updateToDo } from "./query";

export async function updateToDoCommand(
  params: UpdateToDoRequest,
  userId: string
): Promise<UpdateToDoResponse> {
  const data = requestToDb(params, userId);
  const existing = await getToDo(params.id, userId);
  if (existing == null) {
    throw new Error("item does not exist");
  }
  const result = await updateToDo(data, params.id, userId);
  if (result == null) {
    throw new Error("error while updating to do");
  }
  const success = await handleLeaderboard(existing, result, userId);
  if (!success) {
    throw new Error("error while updating the leaderboard");
  }
  return dbToResponse(result);
}

Well at least this is a tiny bit more involved, so let's take a closer look. The first thing we see that's different is that we have a new file, leaderboard-handler.ts. Given that we just talked about that we have to update the leaderboard somehow, that's probably what this is for, and also once again, if you just look at the files for this command, you immediately have an idea what's going on. Next up we also now have to queries instead of one, one for loading the existing one and the other for saving. So the handler itself, like before it’s handling the data flow and this time it has some minimal validation built in, but no real logic since it's only responsible for the data flow. So it maps the request to a db object, like we saw before and then it loads the existing data. And here comes the point about why unique queries are a good thing. Usually you would now search for the correct repo, inside this repo search for a getToDoById or something similar and use this, because DRY is important. But here you can’t do that, since there is no global repo, instead you have to write this query yourself, so let's take a look at that

import { and, eq } from "drizzle-orm";
import { db } from "../../../shared/db";
import { todos, type ToDo, type UpdateToDo } from "../../core/db/schema";
import type { ToDoCompleted } from "./model";

export async function updateToDo(
  todo: UpdateToDo,
  id: string,
  userId: string
): Promise<ToDo | undefined> {
  const [updatedItem] = await db
    .update(todos)
    .set(todo)
    .where(and(eq(todos.id, id), eq(todos.userId, userId)))
    .returning();

  return updatedItem;
}

export async function getToDo(
  id: string,
  userId: string
): Promise<ToDoCompleted | undefined> {
  const [result] = await db
    .select({ completed: todos.completed })
    .from(todos)
    .where(and(eq(todos.id, id), eq(todos.userId, userId)));
  return result;
}

Well interestingly enough since this is a hyperspecialized query and we know that the only thing that interests us is the completed flag, we can now load only the completed flag, making the whole thing run just a tiny bit faster.

Now finally let’s take a look at the last piece of the puzzle, how to interact with another module. For this I created a new library under lib/todo/integration/leaderboard. This library is going to be really small and will be just a thin wrapper around our leaderboard module. Now when we want to use a command from the leaderboard module, we create a new file in our wrapper. For this example we want to call the updateLeaderboardCommand, so we create a new update-leaderboard.ts with the following content

 

import {
  updateLeaderboardCommand,
  type UpdateLeaderboardRequest,
} from "../../../leaderboard";

export async function updateLeaderboardIntegration(
  params: UpdateLeaderboardRequest
) {
  return await updateLeaderboardCommand(params);
}

We created this wrapper to have a clear integration layer between our modules, now when we look inside it, we can directly see which commands are getting used, also if you use a monorepo tool like nx, you can define which libs are allowed to interact with which ones, this separation allows you to clearly define these rules. So now finally the leaderboard-handler.ts

import type { ToDo } from "../../core/db/schema";
import { updateLeaderboardIntegration } from "../../integration/leaderboard/update-leaderboard";
import type { ToDoCompleted } from "./model";

export async function handleLeaderboard(
  existing: ToDoCompleted,
  result: ToDo,
  userId: string
) {
  if (existing.completed == result.completed) {
    return true;
  }
  if (result.completed) {
    return await updateLeaderboardIntegration({
      userId: userId,
      increase: true,
    });
  } else {
    return await updateLeaderboardIntegration({
      userId: userId,
      decrease: true,
    });
  }
}

Now this part of the code only calls our integration function, making this module completely self contained. So how exactly does our project look like now?

All in all I would say still pretty organized, you can directly see where which command is implemented and what which file does.

Commands without an endpoint

So far it looks like our API definition completely describes our project, but this is only partially true. Our UpdateLeaderboardCommand actually has no endpoint assigned to it, if you were really careful, you may have noticed that our integration imports the UpdateLeaderboardRequest type directly from the updateLeaderboardCommand folder, not from the sdk folder. For now our command does not need to be public, it is only ever getting called from the updateToDo command. So for this case I created a model.ts, defined the request and result objects in there and that's it. If at some point you need to create an endpoint for it, you just have to call this command from the service, probably change out the request and result object and you are done, so the only difference between those two is probably where the models are defined.

Scaling in the future

Now this is the point that most backends never reach but people love talking about anyway. Let's say our ToDo app explodes in popularity, we already covered the performance benefits of hyperspecialized queries, the ease of onboarding new developers into this, the expandability of the code itself, since each endpoint is a tiny self contained feature, but of course we have missed the big point, Microservices. So now we want to have two APIs instead of one, the leaderboard and the todo api. How do we separate them? Well our monorepo approach from the start does most of the heavy lifting already, technically, we can just add a second app folder, set it up to only serve the leaderboard service, remove the leaderboard service from the existing one and you are good to go. But of course that's dirty, because now you still have leaderboard code in your todo API. Well, I guess if you deploy an update independently from one another you could run into problems. This is where our integration layer comes in to save the day. You just need to replace the function call with an API call and you are done, easy.

But what about

… circular dependencies? Since each library is allowed to call each other library, what if the leaderboard suddenly needs to load some todos? 

That is a very good question and the answer is pretty simple: just don’t. Circular dependencies are not limited to this architecture, they are a known problem and people found ways to handle them, usually by creating a new domain that will then orchestrate the data handling, resolving those. 

… tight coupling? You are supposed to avoid tight coupling, but now your services directly depend on one another.

Yeah, you are supposed to avoid a lot of stuff people still do. But, as always it depends if you are supposed to avoid tight coupling. If you have to build a rather complex backend with a multitude of APIs that all need to work completely independently of one another then yeah, this is probably not the right fit for you. But if you build a small to medium sized backend, where it's alright that to remove a feature you touch a different one, then it's probably not so bad.

So with this all said and done, I hope you enjoyed the read. If you are curious here is a link to the github repo of the API I started to build for this, to grab my examples from.

Till next time.