DEV Community: Denny Christochowitz

Slire: A Minimal Repository Layer for Node.js + MongoDB/Firestore

Denny Christochowitz — Mon, 15 Dec 2025 06:32:59 +0000

Over the last few years building Node.js backend services on MongoDB and Firestore, I kept running into the same pattern:

Every team rolls their own “repository layer” on top of the database.
Everyone re‑implements the same things: multi‑tenancy, soft deletes, timestamps, versioning, audit trails.
Business logic ends up littered with { tenantId, _deleted: { $ne: true } } filters and hand‑rolled update operators.

At the same time, full‑blown ODMs (like Mongoose and similar libraries) often try to abstract the database too much. They introduce their own query DSLs and life‑cycle hooks, can make it harder to use advanced database features (like MongoDB aggregations), and you still end up dropping down to the native driver for anything non‑trivial.

I wanted something in between.

That’s why I built Slire.

Slire is a small Node.js library that gives you a consistent, type‑safe repository layer over MongoDB and Firestore. It handles the boring but important parts of data access—like scope, soft deletes, timestamps, versioning, and tracing—while staying out of the way when you need the full power of the native driver.

What Slire is (and isn’t)

Slire is intentionally minimal. It doesn’t try to be an ODM or replace your database driver.

Instead, it:

Wraps your existing MongoDB or Firestore collections.
Gives you a small, well‑typed API for the most common CRUD and query operations.
Automatically enforces scope (e.g. per‑tenant data isolation).
Manages common consistency fields for you:
- Soft delete (_deleted)
- Timestamps (_createdAt, _updatedAt, _deletedAt)
- Versioning (_version)
- Optional per‑write trace data (who did what, when)
Exposes helpers like applyFilter and buildUpdateOperation so you can safely use the native driver for complex operations, while keeping behavior (scope, soft delete, versioning, tracing) consistent.

What it doesn’t try to do:

No custom query DSL—just plain MongoDB/Firestore filters.
No magic model lifecycle hooks or hidden queries.
No attempt to make MongoDB “feel like SQL” or vice versa.

If you like the expressiveness of the native drivers but are tired of rewriting the same repository boilerplate, this is for you.

For the full rationale and how this approach compares to ODMs and other heavier data‑access abstractions, see:

👉 Why Slire?

Quickstart: a scoped Task repository

Slire is built around repository factories. You define your domain type and a small factory function that wires it to your database client and scope.

Let’s say you have a simple Task type and a multi‑tenant MongoDB app:

// task.ts
export type Task = {
  id: string;
  tenantId: string; // scope
  title: string;
  status: 'todo' | 'in_progress' | 'done' | 'archived';
  dueDate?: Date;
  _createdAt?: Date;
};

You can create a repository factory like this:

import { MongoClient } from 'mongodb';
import { createMongoRepo } from 'slire';
import type { Task } from './task';

export function createTaskRepo(client: MongoClient, tenantId: string) {
  return createMongoRepo<Task>({
    collection: client.db('app').collection<Task>('tasks'),
    mongoClient: client,
    scope: { tenantId }, // enforced on all reads/updates/deletes
    options: {
      softDelete: true,
      traceTimestamps: 'server',
      version: true,
    },
  });
}

Usage in a service:

const taskRepo = createTaskRepo(mongoClient, 'tenant-123');

// Create a task (id, scope, timestamps, version, and trace are handled for you)
const id = await taskRepo.create({
  title: 'Draft onboarding guide',
  status: 'todo',
});

// Update – only allowed fields, `_updatedAt` and `_version` are handled automatically
await taskRepo.update(id, { set: { status: 'in_progress' } });

// Scoped read (only returns a task if it belongs to tenant-123 and isn’t soft-deleted)
const task = await taskRepo.getById(id, { id: true, title: true, status: true });

You still have full access to taskRepo.collection for complex queries or aggregations, but you don’t have to manually wire scope/soft‑delete/versioning every time.

For more details, check the full 👉 README.

Designing a clean data access layer

Slire is more than just a set of helpers; it's built around some core data‑access design principles that help keep your application code maintainable as it grows—whether you use Slire itself, talk directly to native drivers, or build a similar repository layer of your own.

1. Explicit dependencies instead of global repositories

Instead of injecting a huge TaskRepo everywhere and calling methods ad‑hoc, Slire encourages narrow, explicit ports:

// task-repo.ts
export type TaskRepo = ReturnType<typeof createTaskRepo>;

const TaskSummaryProjection = { id: true, title: true, status: true } as const;
type TaskSummary = Projected<Task, typeof TaskSummaryProjection>;

export type GetTaskSummary = (id: string) => Promise<TaskSummary | undefined>;
export type SetTaskStatus = (id: string, status: Task['status']) => Promise<void>;

export function makeGetTaskSummary(repo: TaskRepo): GetTaskSummary {
  return (id) => repo.getById(id, TaskSummaryProjection);
}

export function makeSetTaskStatus(repo: TaskRepo): SetTaskStatus {
  return (id, status) => repo.update(id, { set: { status } });
}

Your business logic then depends on simple functions, not on a giant repository interface:

type CompleteTaskInput = { taskId: string; projectId: string; userId: string };

type CompleteTaskDeps = {
  getTaskSummary: GetTaskSummary;
  setTaskStatus: SetTaskStatus;
  // other domain-level capabilities…
};

export async function completeTaskFlow(
  deps: CompleteTaskDeps,
  input: CompleteTaskInput
): Promise<{ task: TaskSummary; projectProgressChanged: boolean }> {
  const { getTaskSummary, setTaskStatus } = deps;

  const task = await getTaskSummary(input.taskId);
  if (!task) throw new Error('Task not found');

  if (task.status === 'done') {
    return { task, projectProgressChanged: false };
  }

  await setTaskStatus(input.taskId, 'done');

  // …maybe call other ports here (e.g. notify manager, recalc project)

  const updated = await getTaskSummary(input.taskId);
  return { task: updated!, projectProgressChanged: true };
}

Why this helps:

Business logic is easy to test with simple stubs/mocks.
Changing how you fetch data (e.g. switching from MongoDB to Firestore or changing query shapes) doesn’t require touching the orchestration logic.
It’s clear what each use case depends on (getTaskSummary, setTaskStatus, etc.).

I go much deeper into this here:

👉 Data Access Design Guide

2. Client-side “stored procedures” without giving up control

Sometimes you need to perform batch operations or complex updates that don’t fit nicely into simple CRUD methods. For example, recomputing per‑project task summaries once a day.

With Slire, you can write a client‑side stored procedure that:

Uses MongoDB’s aggregate directly for performance and flexibility.
Still benefits from Slire’s applyFilter and buildUpdateOperation helpers, so you don’t forget about scope, soft deletes, timestamps, or versioning.

Here’s a (simplified) example:

export async function recomputeProjectTaskSummaries({
  mongoClient,
  tenantId,
  now = new Date(),
}: {
  mongoClient: MongoClient;
  tenantId: string;
  now?: Date;
}): Promise<void> {
  const taskRepo = createTaskRepo(mongoClient, tenantId);
  const projectRepo = createProjectRepo(mongoClient, tenantId);

  await mongoClient.withSession(async (session) => {
    await session.withTransaction(async () => {
      const summaries = await taskRepo.collection
        .aggregate<{
          _id: string;
          openTaskCount: number;
          completedTaskCount: number;
          overdueOpenTaskCount: number;
          nextDueDate?: Date;
        }>(
          [
            { $match: taskRepo.applyFilter({}) },
            {
              $group: {
                _id: '$projectId',
                openTaskCount: {
                  $sum: {
                    $cond: [{ $in: ['$status', ['todo', 'in_progress']] }, 1, 0],
                  },
                },
                completedTaskCount: {
                  $sum: { $cond: [{ $eq: ['$status', 'done'] }, 1, 0] },
                },
                overdueOpenTaskCount: {
                  $sum: {
                    $cond: [
                      {
                        $and: [
                          { $in: ['$status', ['in_progress']] },
                          { $lt: ['$dueDate', now] },
                        ],
                      },
                      1,
                      0,
                    ],
                  },
                },
                nextDueDate: {
                  $min: {
                    $cond: [{ $in: ['$status', ['todo', 'in_progress']] }, '$dueDate', null],
                  },
                },
              },
            },
          ],
          { session }
        )
        .toArray();

      await projectRepo.collection.bulkWrite(
        summaries.map((s) => ({
          updateOne: {
            filter: projectRepo.applyFilter({ _id: s._id }),
            update: projectRepo.buildUpdateOperation({
              set: {
                openTaskCount: s.openTaskCount,
                completedTaskCount: s.completedTaskCount,
                overdueOpenTaskCount: s.overdueOpenTaskCount,
                hasOverdueTasks: s.overdueOpenTaskCount > 0,
                nextDueDate: s.nextDueDate ?? null,
              },
            }),
          },
        })),
        { session }
      );
    });
  });
}

Here:

The query is pure MongoDB (aggregate with $match, $group, $min, etc.).
taskRepo.applyFilter(...) ensures you only touch active tasks in the right tenant.
projectRepo.buildUpdateOperation(...) ensures _updatedAt, _version, and _trace are applied consistently.

This is a good example of Slire’s “native‑first with guardrails” philosophy.

How it fits into a larger architecture

Slire is designed to be composed:

Use createTaskRepo / createProjectRepo in application services.
Wrap repositories into domain-specific data access modules (createTaskDataAccess, createProjectDataAccess) that expose only the operations your business logic needs.
Optionally, compose those modules into a single createDataAccess factory for convenience in HTTP handlers, jobs, and tests.

The Data Access Design Guide goes into:

Explicit dependencies vs injecting repositories everywhere.
The “sandwich method” for clean read/process/write flows.
Specialized data access functions and adapters.
Using specifications without over‑abstracting your queries.
Deciding between unified vs modular factories.

If you care about keeping your services maintainable as they grow, I think you’ll find it useful even beyond Slire itself.

Status, roadmap, and how to try it

Slire is currently at v1.0.2. It’s still young, but the core API is stable and:

Tested against MongoDB (via mongodb driver) and Firestore (@google-cloud/firestore).
Uses TypeScript and targets modern Node.js (20+).
Comes with examples, a detailed README, and design docs.

You can install it today:

pnpm add slire
# or
npm install slire

👉 GitHub: dchowitz/slire

👉 Docs: Why Slire?, Data Access Design Guide, and Audit Trail Strategies with Tracing

I’d love feedback, questions, or ideas:

Is this a pattern that would help your team?
What’s missing for you to try it in a side project or production service?
Do you have war stories from ODMs or ad‑hoc repos that Slire could help with?

Drop a comment, open an issue, or ping me on LinkedIn—I'd be happy to chat and improve this together. 🙌

TypeScript Enums vs. String Unions: What's the Deal?

Denny Christochowitz — Mon, 14 Apr 2025 19:24:37 +0000

Enums in TypeScript can be a bit of a double-edged sword. They look neat, give you nice tooling, and let you group related constants. But they also come with quirks, runtime implications, and compatibility oddities. If you've ever scratched your head wondering whether to use an enum or just go with string unions, this guide is for you.

Let's break it all down.

The Classic Enum

Here's a classic numeric enum in TypeScript:

enum Priority {
  low = 0,
  medium = 1,
  high = 2,
  critical = 3,
}

You can use it like this:

const p = Priority.high;
console.log(Priority[p]); // 'high'

🔄 Reverse Mapping

With numeric enums, TypeScript generates a reverse mapping, so you can go from 2 to 'high'.

But there's a catch:

const key = Priority[2]; // inferred as `string`, not 'low' | 'medium' | ...

You lose type safety unless you cast it:

const key = Priority[2] as keyof typeof Priority;

Or wrap it in a helper:

function getEnumKeyByValue<E extends Record<string, string | number>>(
  enumObj: E,
  value: E[keyof E]
): keyof E | undefined {
  return (Object.keys(enumObj) as (keyof E)[]).find(
    (k) => enumObj[k] === value
  );
}

Why Some Devs Avoid Enums

Enums in TypeScript compile to real JavaScript objects. That means:

They increase bundle size.
Structurally identical enums are not compatible.
You have to remember all the quirks (like reverse mapping only working for numeric enums).

They're not bad — just something to use with eyes open.

The String Union Alternative

Instead of an enum, you can write:

type Priority = 'low' | 'medium' | 'high' | 'critical';

No runtime cost. Super simple. But now you don’t have a runtime object to loop over or validate against.

The Best of Both Worlds: Enum-Like Pattern

Here’s the slick trick: define a const array and infer the type from it.

const priorityValues = ['low', 'medium', 'high', 'critical'] as const;
type Priority = typeof priorityValues[number];

✅ Zero runtime bloat
✅ Type-safe union
✅ You can iterate over priorityValues
✅ You can validate at runtime

function isPriority(input: string): input is Priority {
  return priorityValues.includes(input as Priority);
}

🔁 What About Object.fromEntries with Strong Typing?

You can build a typed object from a list of keys using Object.fromEntries. Here's how:

const entries: [Priority, string][] = priorityValues.map(
  (key) => [key, key.toUpperCase()] // just an example transformation
);

const priorityLabels = Object.fromEntries(entries) as Record<Priority, string>;

Now priorityLabels.low is strongly typed, and accessing an invalid key will be a type error.

Or you can make it reusable with a helper:

function fromEntriesTyped<K extends string, V>(
  entries: readonly (readonly [K, V])[]
): Record<K, V> {
  return Object.fromEntries(entries) as Record<K, V>;
}

const entries = priorityValues.map((key) => [key, key.toUpperCase()] as const);
const priorityLabels = fromEntriesTyped(entries);

A Reusable Enum Helper

Want to standardize this? Here's a tiny utility:

export function createEnum<const T extends readonly string[]>(values: T) {
  return {
    values,
    includes: (val: unknown): val is T[number] => values.includes(val as T[number]),
    type: null as unknown as T[number],
  };
}

Usage:

const priorityEnum = createEnum(['low', 'medium', 'high', 'critical'] as const);
type Priority = typeof priorityEnum.type;
const PriorityValues = priorityEnum.values;
const isPriority = priorityEnum.includes;

Now you’ve got everything: type safety, runtime values, validation, and no surprises.

TL;DR: When to Use What

Situation	Go With
Small list, no runtime use	✅ String union
You want runtime access/iteration	✅ Const array + type
Need reverse mapping	✅ Numeric enum
Need labels or metadata	✅ Custom object

Enums are fine — but if you're aiming for minimalism, clarity, and flexibility, that const array + type combo is a real MVP.

Happy typing! 🧑‍💻

How to fight repetitive tests in JS/TS with parameterisation

Denny Christochowitz — Sun, 22 Jan 2023 16:13:03 +0000

This short post demonstrates a helpful technique that can significantly simplify writing unit tests in JavaScript and TypeScript. I'm using TypeScript here with Jest as the testing framework. The approach presented works equally well for Mocha and should also work for other similar test runners.

So, what is it about?

When writing unit tests, we often need to pass a variety of input data to our component under test (a function or class - often called system under test, or shortly SUT) and check how it behaves.

We quickly have tests that look very similar because we usually copy and paste them and make minor adjustments. Data-driven tests can remedy this situation and help to keep the test code clear and easier to change. A simplified example will illustrate this. Let's assume we have a Type Predicate isPerson.

Tests for that might look like this:

type Person = {
  firstName: string;
  lastName: string;
};

function isPerson(input: any): input is Person {
  return (
    input &&
    typeof input.firstName === "string" &&
    input.firstName.length > 0 &&
    typeof input.lastName === "string" &&
    input.lastName.length > 0
  );
}

Tests for that might look like this:

it("accepts a valid person", () => {
  const input = { firstName: "John", lastName: "Doe" };
  expect(isPerson(input)).toBe(true);
});

it("accepts a valid person, ignoring unknown properties", () => {
  const input = { firstName: "John", lastName: "Doe", foo: 42 };
  expect(isPerson(input)).toBe(true);
});

it("rejects null", () => {
  expect(isPerson(null)).toBe(false);
});

it("rejects undefined", () => {
  expect(isPerson(undefined)).toBe(false);
});

it("rejects an empty object", () => {
  expect(isPerson({})).toBe(false);
});

it("rejects an array", () => {
  expect(isPerson([])).toBe(false);
});

it("rejects true", () => {
  expect(isPerson(true)).toBe(false);
});

it("rejects false", () => {
  expect(isPerson(false)).toBe(false);
});

it("rejects a number", () => {
  expect(isPerson(42)).toBe(false);
});

it("rejects a string", () => {
  expect(isPerson(42)).toBe(false);
});

it("rejects, if firstName is missing", () => {
  const input = { lastName: "Doe" };
  expect(isPerson(input)).toBe(false);
});

it("rejects, if firstName is empty", () => {
  const input = { firstName: "", lastName: "Doe" };
  expect(isPerson(input)).toBe(false);
});

it("rejects, if lastName is missing", () => {
  const input = { firstName: "John" };
  expect(isPerson(input)).toBe(false);
});

it("rejects, if lastName is empty", () => {
  const input = { firstName: "John", lastName: "" };
  expect(isPerson(input)).toBe(false);
});

What is immediately noticeable here are the many similar negative test cases ("rejects..."). Overall, the test methods look pretty much the same, except for the input and the test description.

I don't mind having non-DRY code in tests, as each test should work independently and depend as little as possible on others. Nevertheless, it is tedious to adapt each test in case the interface of the SUT or the test code itself changes during the development.

To remove some DRY-ness, we could combine some tests, for example, for primitive types:

it("rejects primitive types", () => {
  expect(isPerson(null)).toBe(false);
  expect(isPerson(undefined)).toBe(false);
  expect(isPerson(true)).toBe(false);
  expect(isPerson(false)).toBe(false);
  expect(isPerson(42)).toBe(false);
  expect(isPerson("foo")).toBe(false);
});

This style is sufficient for one-liner tests but can quickly get out of hand for more complex test bodies. Some people think having multiple assertions in a test is terrible, but I don't see it as black and white.

Let us now reformulate this test into a data-driven test. We achieve this by passing the input to the test code as a parameter. Thanks to the functional nature of Javascript, this is relatively easy:

const primitives = [null, undefined, true, false, 42, "foo"];

primitives.forEach((input) => {
  it(`rejects primitive ${JSON.stringify(input)}`, () => {
    expect(isPerson(input)).toBe(false);
  });
});

We basically iterate over the possible input values and execute each test individually, as in the example shown at the beginning. To do this, we wrap the test in a function that takes an input value as a parameter.

However, I want to reveal one disadvantage of this approach. Your favourite IDE likely refuses to run single tests wrapped in this way. The built-in parser will not recognize them. But as long as we group tests in describe scopes, we can still execute them in one go using IDE augmentations, which is sufficient for me (I work with WebStorm).

Thanks to JSON.stringify(input) for building our test name, we can easily include complex test input:

[
  ...primitives,
  {},
  [],
  { lastName: "Doe" },
  { firstName: "", lastName: "Doe" },
  { firstName: "John" },
  { firstName: "John", lastName: "" },
].forEach((input) => {
  it(`rejects ${JSON.stringify(input)}`, () => {
    expect(isPerson(input)).toBe(false);
  });
});

By doing that, we condensed our original test suite significantly. To extend that, let's include the positive test cases as well.

[
  ...[
    ...primitives,
    {},
    [],
    { lastName: "Doe" },
    { firstName: "", lastName: "Doe" },
    { firstName: "John" },
    { firstName: "John", lastName: "" },
  ].map((i) => [i, false] as const), // <- tuple [input, result=false]
  [{ firstName: "John", lastName: "Doe" }, true] as const, // <- tuple [input, result=true]
  [{ firstName: "John", lastName: "Doe", foo: 42 }, true] as const, // <- tuple [input, result=true]
].forEach(([input, result]) => {
  it(`${result ? "accepts" : "rejects"} ${JSON.stringify(input)}`, () => {
    expect(isPerson(input)).toBe(result);
  });
});

The idea is the same, except a tuple is first built from each input, consisting of the input itself and the expected result. So the test is parameterized based on the input and the desired result.

Many developers (me included) may find that example already too much of a good thing because the higher complexity in preparing the test input is not worth it here. And if we had considerably more positive test cases, we could combine them separately into one parameterized test.

Finally, I want to show you a slightly more complex but simplified example for the sake of brevity. Here we extend our type Person with an address specification and test a fictitious function isPersonWithAddress.

type Person = {
  firstName: string;
  lastName: string;
};

type Address = {
  street: string;
  postalCode: string;
  city: string;
};

type PersonWithAddress = Person & { address: Address };

const primitives = [null, undefined, true, false, 42, "foo"];
const notANonEmptyString: any[] = [
  null,
  undefined,
  "",
  42,
  true,
  false,
  [],
  {},
];

const samplePersonWithAddress = {
  firstName: "John",
  lastName: "Doe",
  address: {
    street: "Mainstreet 42",
    postalCode: "8043",
    city: "Zurich",
  },
};

[
  { prop: "firstName", invalid: notANonEmptyString },
  { prop: "lastName", invalid: notANonEmptyString },
  { prop: "address", invalid: primitives },
  { prop: "address.street", invalid: notANonEmptyString },
  { prop: "address.postalCode", invalid: notANonEmptyString },
  { prop: "address.city", invalid: notANonEmptyString },
].forEach(({ prop, invalid }) => {
  invalid.forEach((value) =>
    it(`rejects ${prop} = ${JSON.stringify(value)}`, () => {
      // we use lodash's `set` to mutate an object property
      const input = _.set({ ...samplePersonWithAddress }, prop, value);
      expect(isPersonWithAddress(input)).toBe(false);
    })
  );
});

The initial situation in this example is always the same input (samplePersonWithAddress) for the function under test (isPersonWithAddress). In a nested loop, we iterate over invalid values for each property of the test input. Effectively, this is a parameterization in two dimensions: First, the property of the input value to modify and second, invalid values.

Conclusion

Parameterized tests are a valid approach to keep the test code cleaner and DRY. But they are not a silver bullet. Just because it's relatively easy to generate many combinations of input data doesn't mean we should always do so. First and foremost, it is crucial to test modules individually. That way, components that depend on others can be tested with only some possible combinations of input since we already know that the modules they depend on are tested thoroughly. Besides, this would otherwise quickly lead to an exponential combination of possible input data, which is undoubtedly different from what we strive for (meaningful tests). Overall, it is better to align the test parameterization with the logic inherent in the particular module.

A Good Read - Crafting Interpreters

Denny Christochowitz — Sun, 18 Sep 2022 18:56:12 +0000

tl;dr - "Crafting Interpreters" by Robert Nystrom is an excellent inspirational read about a topic an average software engineer (like me) doesn't have to deal with every day.

The book left me with the urge to tell you about it, so here we are - and to invent yet another programming language which is just stupid given the many languages out there and my limited time at hand - but still...

See for yourself: https://craftinginterpreters.com

Read on if you want to get to know more about this gem.

Why did I read this book?

Software engineering is a broad area and ever-evolving. Working in this field, my day-to-day job covers only a tiny slice of topics therein. Which I believe is true for most software engineers out there.

Now and then, I feel the urge to explore topics outside of my technology-wise comfort zone. Mainly stuff I got in contact with way back during my CS classes, knowledge almost buried and lost in my nebulous wetware.

Reading "Crafting Interpreters" by Robert Nystrom was one of those excursions. When I got hold of this book, it was a permanent companion during my summer vacation. I love it, and here is why.

Interesting Topic

Building compilers was one of my favourite classes back then, despite my choice to deepen my studies in computer graphics later.

Approachable

Building interpreters, compilers, or other tooling around programming languages is widely perceived as black art. However, the author's relaxed and engaging writing style makes the book impressively accessible to anyone with basic programming skills, despite its depth and details.

Hands-on

The common thread of this book is to have you written a complete interpreter by the end.

I believe that is a novum amongst other literature in this field, mainly focusing on theoretical concepts. "Crafting Interpreters" lets you code along, building an interpreter for Lox, a dynamically typed language with first-class functions and basic object-oriented features if you want.

In fact, it does it twice. The first part walks you through the implementation of both the interpreter and virtual machine in Java - kind of as a warm-up, giving you the general concepts and ideas. The second and bigger part of the book re-implements the whole shebang in just C with a focus on performance, without all the conveniences a higher-level language brings, for example, data structures (linked lists, hash tables) or memory management behind the scenes (garbage collection).

Refreshing Perspective

The second part of the book reminded me that there is software out there that is not bottlenecked by unnecessary DB roundtrips or chatty service contracts when we look at performance - things I often have to deal with in my job.

When building an interpreter for a programming language and the virtual machine executing the users' programs, you create a piece of software that runs solely in-process, where performance degrades mainly through poor algorithms and data structures. Something you wouldn't start investigating when expensive database queries are the limiting factor or remote procedure calls define the performance baseline.

I found it very refreshing to see optimisations, like the clever use of bit manipulations, e.g. in the case you need to compute the modulo of a number which always happens to be a power of 2, or taking data locality into account, or the cost of pointer indirections.

The book has a lasting effect on me as I sometimes see myself (pointlessly) wondering if the data used in the hot code path I just wrote fits nicely into a cache line or has to be fetched slowly from RAM.

Awesome Quality

The book is beautifully typeset and illustrated. You can feel the love and passion Robert Nystrom has put into this work. In that regard, it is way above most other software engineering books I have read. I could imagine that even readers who are not so much interested in the topics explained would skim through the book and then suddenly find themselves start reading it.

Only Small Downsides

There are no serious downsides, in my opinion. And those mentioned below didn't bother me greatly.

Implementing an interpreter is one thing. Ensuring it behaves along a programming language's semantics is another. Unfortunately, the aspect of testing is skipped altogether, something you'd have to deal with when inventing a real language beyond toy purposes.

Additionally, and primarily due to the nature of the book carrying the complete implementation in its pages, there are some boring passages regarding programming chores like header files, includes and imports.

Summary

A refreshing book if you're interested in the inner workings of interpreters or if you need some brain stimuli.

Thanks, Robert Nystrom, for this book and the many inspirations within! ❤️

Dear readers, I'd love to hear in the comments about books which impressed you lately! 👋

Consistent Hashing Explained with React+SVG

Denny Christochowitz — Mon, 10 Jan 2022 15:58:03 +0000

This post explains the principles behind "Consistent Hashing" with the help of some interactive React + SVG demos here and there.

The source of the interactive demos can be found in the accompanying GitHub repo.

The Problem

Consistent hashing originally was applied in the late 90s for caching websites. The goal was to have a shared cache for many nearby users, e.g. on a university campus. If one of these users requested a website, the cache would first be checked and only in case of a cache miss the request would be routed to the server hosting the website. The apparent benefits of such a cache are an overall better user experience due to reduced response times and less internet traffic.

However, the catch is that a single machine can hardly provide enough memory for storing cached websites. Depending on the number of users accessing websites through a shared cache, hundreds of servers or higher magnitudes are required in this regard. A shared website cache thus comprises a lot of servers on which the cached websites are somehow distributed.

The naive approach to look up a particular website in the cache would be to iterate over all the involved servers and look if it's there, which is obviously not very optimal. It would be nice if we had some sort of lookup function that tells us which server to ask for a given website right away.

f(URL) -> server

Hash Functions

Luckily there are hash functions which will help us here. A hash function maps values of an arbitrarily large domain (e.g. strings representing website URLs) to a smaller domain with a restricted set of values (e.g. 32-bit integers) and comes with these properties:

cheap to compute
deterministic - the same input always results in the same output
kind of random behaviour - maps input randomly across possible values in target domain without noticeable correlation

You find a comprehensive list of hash functions here.

Note that there is a class of hash functions called cryptographic hash functions with some additional properties:

it is infeasible to generate a hash function input that yields a given hash value (i.e. to reverse the process that generated the given hash value)
it is infeasible to find two different hash function inputs with the same hash value

Since our problem of determining the cache server based on a URL is free of any security concerns, we're good to go with a simple non-cryptographic hash function. Of course, any cryptographic hash function would work - but with the downside of a higher computing cost.

Now let's assume we have chosen a suitable hash function h, which gives us a 32-bit integer for an arbitrary input string (all the demos below use xmur3). How do we map the hash value to our set of a few hundred or thousand cache servers, considering that the number of cache servers can change over time?

Naive Approach

Given that we have m servers addressed from 0 to m-1, the most straightforward way to get a server associated with a specific URL would be:

server = h(URL) % m

Applying the modulo here works reasonably well if the number of cache servers is known in advance and unlikely to change over time. But if m changes (e.g. a server goes down, or we have to add a couple more servers to increase our cache capacity), potentially all URLs cached so far would be reassigned to another server and invalidated. While that may seem borderline acceptable for our use case of caching websites, it is not. If the number of servers on which data is distributed is constantly changing, applications will suffer drastically as affected data parts have to relocate frequently.

🤓 Applying the modulo is a common technique to map potentially large integers to a smaller domain. Change the number of nodes in the demo below. You can observe that often almost all of the URLs would be reassigned to another node.

Consistent Hashing

Consistent caching is a surprisingly simple approach (once you get it) that keeps the redistribution of URLs to servers to a minimum. Even if the number of cache servers m changes over time, most of our cached websites stay assigned to the same cache server.

Let's briefly rephrase our problem statement in a more general fashion and stick to this terminology for the rest of this post.

We want to evenly distribute an arbitrary amount of data to a limited set of nodes so that a change in their number causes a minimal set of data to be reassigned to other nodes.

Let's define d as the key identifying a certain piece of data (e.g. a URL representing a website) we want to associate with a node n. Furthermore, let's assume we use a suitable hash function h.

The main twist of consistent hashing is that in addition to hashing the keys (a shorter way of saying applying the hash function to the keys), we also hash the node identifiers (something unique like an URL or an IP address). That way, we have both our keys and nodes represented as hash values.

A key d is then associated with that node, whose hash value is the nearest successor to the hash value of d. If there is no such node (which can certainly happen), the node with the overall minimum hash value is taken. That means we basically wrap around by forming a hash ring (the end of the hash space connects to the start).

Put another way, we clockwise search for the next hashed node h(n) on our hash ring starting from our hashed key h(d).

With consistent hashing, only k/m nodes are reassigned on average, where k is the number of keys, and m is the number of nodes.

🤓 The demo below shows three nodes and a key on our hash ring. The wide arc represents the key's partition, with an arrow pointing to the assigned node. You can fiddle around by entering other key values.

You can ignore the suffix _0 in the shown node identifiers for now. I'll explain it in the next section.

(Note that this demo and the following ones are pre-bundled in Glitch. If you want to poke around the sources, have a look at the GitHub repo. See the last section about the reasons for pre-bundling.)

🤓 The following demo shows nine nodes, of which three are active. The current key is assigned to node-11. Turn off this one and afterwards node-13. Observe how the key gets reassigned. Play around, toggle other nodes and try out different keys.

You may have noted that the distribution of nodes on the hash ring in the demos is not so bad, given that we place them randomly. Well, I cheated a bit to make the visualization easier to understand and to let the nodes not overlap each other. This brings us to the next topic.

Virtual Nodes

This basic version of consistent hashing - while certainly better than the (modulo-based) naive one - still has some drawbacks:

Due to hashing, an even distribution of nodes on the hash cannot be guaranteed so that the space (partition size) between two adjacent nodes can vary to a high degree. It is possible to have partitions that are very small or large.
Similarly, the keys may not get distributed uniformly on the hash ring, resulting in empty or overcrowded partitions.

To mitigate these issues, real-world implementations of consistent hashing often represent a node multiple times on the hash ring via virtual nodes. This can be done simply by hashing the concatenation of a node identifier with a number. For example, if we wanted to have each node represented three times on the hash ring, a node identifier node-11 could be described with the virtual identifiers node-11_0, node-11_1 and node-11_2. (I applied this naming schema in the demos, in case you're wondering.)

Alternatively, instead of having virtual node identifiers according to the virtual node count, we could also apply different hash functions to each node identifier as described in this excellent Stanford lecture notes. However, since this approach is more involved I used the naming scheme for simplicity.

Instead of having the same virtual node count for each of our server nodes, we could also think about a different number of representations for nodes on the hash ring depending on their capacity (e.g. CPU or storage). Nodes with a higher capacity could be configured to have more virtual nodes, summing up to a larger partition on the hash ring and a higher probability of keys assigned.

🤓 The demo below shows the effect virtual nodes have on the partition size. It emphasizes all belonging partitions of the selected node. Initially, each node is represented only by a single virtual node as in the previous demos. Go ahead and try out cranking up and down the number of virtual nodes!

Implementation Notes

I won't walk you through the implementation of consistent hashing or any of the demos shown in this post. That would go beyond the scope I've planned for this write-up. Instead, just some short general remarks. (If you're interested in more implementation details, let me know in the comments. Maybe I'll then find time for a follow-up post.)

To make the node lookup as fast as possible, we should undoubtedly refrain from sequentially iterating over all our (virtual) nodes and computing their hashes each time we wan't to lookup the node assigned to a key. A good approach would be to store the nodes in a data structure optimized for fast retrieval. Particularly the task "Here is a key hash; return the smallest of all your current node hashes greater than that." should perform well.

A binary search tree (BST) is an excellent option here. The BST would be sorted by node hashes and additionally each node hash would be associated with the corresponding node identifier for a reverse lookup of the (virtual) node based on the found hash. Adding or removing a node and adjusting the number of virtual nodes would update the binary search tree accordingly.

Another data structure in need would be a map, which allows us to look up a physical node based on a virtual one.

Finally, the very essential operations a consistent cache must provide to be useful (in Typescript notation):

type ConsistentHash = {
  addNode(node: string): void;
  removeNode(node: string): void;
  lookupNode(key: string): string;
};

This would assume a fixed virtual node count, either as implementation detail or as a parameter during initialization. If we wanted more flexibility in this regard, i. e. adjusting the virtual node count at runtime, we could extend our consistent hash API with:

type ConsistentHash = {
  //...
  setVirtualNodeCount(count: number, node?: string): void;
};

This way, we are able to set the virtual node count per single node or globally.

Looking for a finger exercise? Why don't you try to implement consistent hashing then?

Summary

Consistent hashing as an approach originated from the problem of building an efficient distributed cache for websites and has found wide adoption in a broad range of distributed system scenarios.

Data partitioning is undoubtedly one of the main applications of consistent hashing, but there are other limited resources a node in a distributed system can have (besides storage capacity). For example, if you wanted to design a large scale chat application with millions of users, you would quickly realize that the number of web socket connections a single server can handle is limited. Thus, assigning web clients to web socket servers is yet another use case consistent hashing can handle.

Take care & happy coding 🙌

Meta Note

I wanted to write a short explanatory text sprinkled with some interactive demos.

Given that all demos in this post (except the first) exceed the amount of code I'm willing to write in an online IDE (capable of showing previews here on dev.to), I was kind of lost at first and wondered how to embed these interactions. After some tries, I eventually decided to deploy them as pre-bundled static websites to Glitch. And yes, I'm very aware this is not how Glitch wants you to use it.

I wished I could simply import the demos in an MDX-like way, as these are all React components. That feature, along with some fine-grained control over the imported component's size, would really be awesome.

Very interested to hear about your approaches regarding embedding apps for demo purposes in your posts!

React + Firebase: A Simple Context-based Authentication Provider

Denny Christochowitz — Sat, 29 May 2021 15:23:38 +0000

This post showcases a quick and easy way to make a Firebase-authenticated user available throughout your React web app.

We are using here plain React with Typescript, and no extra state management library like Redux involved.

Firebase offers us to register a callback that gets called every time a user is authenticated or signed out to get notified about the current authentication situation.

import firebase from "firebase/app";

firebase.auth().onAuthStateChanged((user) => {
  if (user) {
    console.log("authenticated", user);
  } else {
    console.log("signed out");
  }
});

We thus could implement a React component that is interested in the authenticated user quite straightforward like this:

import * as React from "react";
import firebase from "firebase/app";

function CurrentUser() {
  const [user, setUser] = React.useState<firebase.User | null>(null);

  React.useEffect(() => {
    const unsubscribe = firebase.auth().onAuthStateChanged(setUser);
    return unsubscribe;
  }, []);

  return <div>{user?.displayName || "unauthenticated"}</div>;
}

Our React component facilitates React.useEffect to register the Firebase onAuthStateChanged callback once after it was mounted. The effect returns the unsubscribe callback from onAuthStateChanged, ensuring that we don't run in any memory leaks.

Additionally, we have a state for the current user which's setter happens to match the callback signature perfectly.

This works just fine if only a single component in your React app is interested in the authentication state. Duplicating the state and effect for each other component would be cumbersome.

But more importantly, this approach works only for permanent (not conditionally rendered) components in our app's render tree, since otherwise, they might miss the initial authentication state because onAuthStateChanged only notifies changes.

One way to tackle this is to provide the authentication state globally utilizing a React context and companion hook. Let's start with the context first:

// FirebaseAuthContext.tsx
import * as React from "react";
import firebase from "firebase/app";

type User = firebase.User | null;
type ContextState = { user: User };

const FirebaseAuthContext =
  React.createContext<ContextState | undefined>(undefined);

const FirebaseAuthProvider: React.FC = ({ children }) => {
  const [user, setUser] = React.useState<User>(null);
  const value = { user };

  React.useEffect(() => {
    const unsubscribe = firebase.auth().onAuthStateChanged(setUser);
    return unsubscribe;
  }, []);

  return (
    <FirebaseAuthContext.Provider value={value}>
      {children}
    </FirebaseAuthContext.Provider>
  );
};

export { FirebaseAuthProvider };

Few things to note here:

User is a type alias for the authenticated Firebase user returned by onAuthStateChanged. The callback is called with null if no user is authenticated.
ContextState is a type alias for the value provided by our context FirebaseAuthContext.
We do not expose FirebaseAuthContext directly. Instead we expose FirebaseAuthProvider which encapsulates FirebaseAuthContext.Provider and a onAuthStateChanged subscription. It's quite similar to the CurrentUser implementation above.

Now let's define a simple hook that gives components interested in the authenticated user access to it:

// FirebaseAuthContext.tsx
// ...

function useFirebaseAuth() {
  const context = React.useContext(FirebaseAuthContext);
  if (context === undefined) {
    throw new Error(
      "useFirebaseAuth must be used within a FirebaseAuthProvider"
    );
  }
  return context.user;
}

export { FirebaseAuthProvider, useFirebaseAuth };

Our hook useFirebaseAuth simply facilitates React.useContext to access the previously defined context. We explicitly check for undefined to catch possible misuses as early as possible.

FirebaseAuthProvider usually is instantiated only once in an App, typically near the root in order to give all components below the opportunity to access the user via useFirebaseAuth. Here's a simple (constrived) example:

// example.ts
import * as React from "react";
import { FirebaseAuthProvider, useFirebaseAuth } from "./FirebaseAuthContext";

// ...initialize firebase

function App() {
  return (
    <FirebaseAuthProvider>
      <UserName />
      <UserEmail />
    </FirebaseAuthProvider>
  );
}

function UserName() {
  const user = useFirebaseAuth();
  return <div>{user?.displayName || "unauthenticated"}</div>;
}

function UserEmail() {
  const user = useFirebaseAuth();
  return <div>{user?.email || "-"}</div>;
}

A few things to note:

Firebase initialization is left out for the sake of brevity. You can check it out here if you haven't already.
The hook can be used by any component below FirebaseAuthProvider regardless of nesting level.
Every notification of onAuthStateChange triggers a re-render.
If your app manages state with Redux or a similar library, you may be better off handling the authentication state there as well.

I've found this approach very simple to implement and apply. It is based on Kent C. Dodds excellent blog post "How to use React Context effectively". You should definitely go and check it out for a more detailed description and some more background info.

Thanks for reading 🤗

If you liked it and don't mind, give it a ❤️

Take care & happy coding 🙌

Photo by Markus Spiske on Unsplash

7 Reasons Why I Favour Feature Slices

Denny Christochowitz — Thu, 29 Apr 2021 09:54:51 +0000

Photo by Thiago Cardoso on Unsplash

There are two main approaches to structuring your app's code: horizontal layers and vertical feature slices.

Not long ago, I had a hard time implementing a new feature in a .NET Core WebApi project which happened to be structured by layer. I was thinking, "Why again a layered codebase?" when I realized that most brownfield projects I've contributed to so far as a developer followed the layered approach. I'd always have a hard time with these kinds of codebases. The reason is, when I work on a feature, it just feels natural to me to co-locate all involved components rather than spreading them to a maximum degree across the whole codebase.

I don't want to explore the reasons leading to layered codebases in the first place here - I believe they are manifold. Instead, I will explain the benefits of feature slices from my perspective as someone engineering line-of-business applications based on .NET backends for most of the time.

Before we dive into the various aspects positively influenced by feature slices, let's shortly recap both approaches and establish a common baseline.

Or, just skip the intro and jump to the sections directly:

First impression
Navigation
Mental overhead
Documentation
Single responsibility
Overgeneralization
Pragmatism

A Short Recap

Technical aspects and infrastructure are the main drivers for the layered approach, regardless of whether we look at the frontend or backend part of an application. Here, components are strictly organized according to their kind.

For example, when working on a typical .NET Core WebApi, we find namespaces (or even projects) dedicated to controllers (exposing HTTP endpoints), data transfer objects, services implementing business logic, business objects, and data access.

A React app following this structure would have its code organized into folders containing components, hooks, state management, data access (talking to Web APIs), etc.

On the other hand, feature slices reflect the high-level use cases or topics of an app, with top-level namespaces or folders named accordingly, e.g. Login, BrowseProducts, or Checkout.

That does not imply that there is anarchy in terms of component responsibilities within a feature slice. Typically, there too will be components encapsulating business logic, data access etc. The difference is that these components live side by side as they collectively provide a particular feature. In other words, the layers are still intact but not etched into the project's structure.

Of course, if a feature is very complex, breaking the structure down into several sub-features is very common. Likewise, the layered approach: It is often possible to identify the features the app is providing, be it through namespacing or component names within a specific layer.

Both approaches also go by other names. For example, onion architecture and horizontal slices refer to layered structures, and vertical slices refer to feature sliced structuring.

Having the basics out of the way, the remainder of this post will outline why I favour feature slices.

Reason 1 — First Impression

Remember the last time you skimmed over someone else's source code trying to figure out what it tries to achieve?

If you find yourself digging deep in folders or even single files to get a rough idea of the features implemented, you probably look at a structure-by-layer. However, the high level how, infrastructure-wise, becomes evident after only a short reading time. Often it is hard to tell one layered codebase from another, especially when looking at those built on the same tech stack.

With feature slices, on the other hand, the source code naturally reveals the implemented features, as the topmost folders are named accordingly. Infrastructure concerns are hidden within each slice and do not block the view of the essential.

That said, I consider the what more important than the how when skimming over codebases.

Reason 2 — Navigation

We developer constantly navigate codebases: when implementing a new feature, extend existing ones, searching for the right line to set a breakpoint etc. Of course, today's IDEs help a lot in this regard with powerful search features or filters no one wants to miss.

The truth is we don't always know what to search for. Often we end up exploring the codebase by clicking through its folders, which is especially true for developers who are new to a codebase but probably also valid for those who have been around for a longer time and returning to work after an intense weekend or vacation.

Feature slices help to navigate the codebase efficiently. We drill down the folders by feature (and subfeatures), and the main components in charge reveal themselves.

In contrast, a layered structure needs more brainpower as the components within a layer are not always clearly cut-by-feature. We have to remember which logic got where and sometimes, several files need an investigation before we find the right one.

Things get worse the more complex the source code is. But to be fair, complexity affects both approaches. An excellent layered codebase potentially outperforms a poorly structured feature-sliced one in terms of navigation effectiveness.

For me, one of the biggest pain points when working with layered codebases is that I have to jump back and forth between those layers when working on a single topic. The sort of client projects I usually work on has accumulated code over several years. Their sheer amount of folders and files makes navigation a constant annoyance considering their layered structure.

Reason 3 — Mental Overhead

This aspect is closely related to navigation. Navigation requires us to have the way (the sequence of clicks necessary to reach a particular file) in mind. Likewise, when adding or moving components, we need an overall idea of where to put what.

Working with feature slices in this regard is a no-brainer as this structuring approach keeps the mental overhead to a minimum. Once we find the right location to work on (which is easy, as explained in the last section), we rarely need to navigate away from it during the work on a distinct feature. As a result, we see ourselves less exposed to unnecessary distractions and can longer hold the flow.

On the other hand, layered codebases require us to have our navigation system turned on the whole time. We switch back and forth between layers, and choosing the correct places for new components requires conscious decisions. These things turn our focus away from the real problem at hand, disturbing the flow.

Imagine we are in the middle of implementing the backend part of this new feature. We have to write a repository class, and it needs an interface so that we can mock it away when unit testing the consuming service. Questions arise:

Is it ok to put the interface right beside the repository, or was there a different place for all the data access interfaces? (Yes, there are still projects out there who prepare themselves for the unlikely event of swapping out the persistence layer somewhen in the future.)

Same with the data types returned by the repository. They should go where the interface goes, right? Or is there a special place for them too? Oh wait, there is already a similar type; it only needs two additional properties. Do we reuse it here? We could make it a base class (which the author would never do, btw). And so on.

Sure, once we get used to it and visited the layer places several times, placed new code here and modified others there, we can keep this kind of distractions to a minimum. But still, what remains is this nagging thought that things could be much easier.

Reason 4 — Documentation

While this one is debatable, I would argue that vertical feature slices potentially result in less code in need of documentation. At least a developer is not encouraged to write more comments than necessary.

Is that a good thing? I'd say yes because documentation should only explain things that are not obvious as good code usually can speak for itself: the reasoning behind a chosen approach, the constraints considered, the trade-offs been made, and of course, the explanation (or reference to it) of a particular algorithm used.

In my experience, it is pretty common for layered codebases to contain comments which try to compensate for the distribution of code belonging to a specific topic or feature: "This is used by ..."

Feature sliced code does not require such explanations. At first glance, it may seem a minor point, but for me, seeing those bread crumb comments which strive to make up for the lost cohesion reveals a major structural flaw.

Reason 5 — Single Responsibility

Feature slices help to keep components' responsibilities focused. To put it another way: With layering, it is easier to take shortcuts, leading to a violation of the single responsibility principle (SRP).

Why is that? Well, this is more of a subtle matter. A layered codebase tempts us to leave the SOLID path as it is much easier to put code for a new feature into an existing component instead of beginning a new one. The reason is that the components of the same infrastructural type (controller, service, repository) live side-by-side. For an inexperienced developer, it's not much of a difference. For others, it's just convenient if in a hurry or the SOLID defences are down for some reason (fatigue, stress, distractions).

That said, feature sliced code not automatically adheres to the single responsibility principle. It just allows the developer to follow more easily here. An awareness must nevertheless be present. Otherwise, you might fall into the trap of writing components spanning multiple layers, e.g. business logic entangled with data access, which can hurt testability.

One example of how features slices foster single responsibility is the registration code for dependency injection (i.e. mapping of interfaces to concrete implementations, factories; Autofac is a prominent representative in .NET stacks). Where should this registration code ideally go?

Most of the time, I've seen layered codebases (but not only those) with one massive registration class plugging together each and every piece required to get things going, no matter the layer or feature.

Feature slices provide a natural way out of these big-ball-of-mud-registrations. Each slice can have a dedicated registration class nicely encapsulating all the nitty-gritty details of the feature's inner workings (e.g. an Autofac module). A concise top-level registration class only has to pick up the feature registrations — a big plus for maintainability.

Reason 6 — Overgeneralization

Another observation related to the last one (SRP) is that layered code promotes generalization since it is more likely to find similarities between components in the same layer. Often we then feel the urge to treat those with the powerful cure called generalization.

While applying generalization may improve code in some situations, we must be careful and consider passing on this kind of refactorings if the components have nothing in common but parts of their structure. Too often, seemingly simple changes to one feature break another because generalization assumptions for those two made in the past are no longer valid today.

Generalization is a double-edged sword. It may help in some regards, but it shoots us in the foot if not watched closely. And sleep with an extra eye open if in layered mode.

Reason 7 — Pragmatism

I can imagine most of us came across this component in a business layer whose sole purpose is to delegate to data access code and often only mirrors the corresponding part of the repository API in question, without even any data mapping. Such kind of component is called anaemic. It does not contribute any functionality; it just adds more bloat to the code.

Anaemic components are a code smell, usually found in layered codebases. Often, dogmatic obeyance (in the form of coding guidelines or strong opinions of leading developers) to the dependency chain (e.g. WebApi Controller -> Business Layer -> Data Access Layer) prohibits skipping a layer if circumstances would allow it.

Not seldomly, we see codebases split up into several modules, one for each layer, and the allowed dependencies between them restricted (e.g. by conventions or tools like NDepend) to enforce the dogmatism described above.

As a result, we see a good amount of anaemic stuff lingering around and an otherwise unnecessary amount of modules.

From a technical perspective, it is sufficient to have the codebase partitioned into modules according to the deployment targets and the shared stuff. Since a typical application only has a limited set of deployment targets, such as CLIs or web services, the number of modules required is usually low.

In defence of those extra modules, we hear arguments like: "We can reuse the business logic elsewhere in the future." or "We can swap out our persistence layer if required! How cool is that?!".

In my opinion, these are just weak attempts defending hopeless cases of YAGNI. I consider these extra modules (per layer) as physical namespaces and completely pointless.

In contrast, feature slices stand in the way of these common malpractices due to their intrinsic nature being vertical spikes penetrating all layers.

Each slice can start simple and evolve only to the necessary degree of complexity. Why not letting a WebAPI controller action talk to a repository directly if there is absolutely no logic in between?

Superfluous modules naturally feel out of place in feature sliced codebases. Their uselessness should become evident to the strongest advocate of physical namespacing. Of course, unless someone suggests having a module per slice 😱.

Conclusion

The choice between a layered and a vertically sliced codebase significantly impacts its maintainability and developer productivity.

Even though it is not always immediately apparent what positive effects feature sliced codebases bring to the table, we should not dismiss this structuring approach lightly.

Compared to layering, I find feature sliced codebases:

are more accessible - what is where? - emphasize the what over the how
contain fewer bloat - start simple, complexity evolves as required
are better to maintain
improve developer productivity
foster pragmatism
are more fun to work with 😀

That doesn't mean that feature slices are the silver bullet to all our problems. Reasonable design decisions, feature partitioning, and lots of discipline are nevertheless in great demand to make our projects succeed.

Thanks for reading this! I'm keen to hear about your experiences and opinions regarding codebase structuring.

Take care & happy coding!

Footnote 1

For reasons of simplicity, I use the term component synonymously for all sorts and levels of encapsulated logic: classes, functions and compounds of those.

back

Footnote 2

The term module refers to a compile target here. In .NET, for example, those are called assemblies.

back

Develop, Build, and Deploy Microsoft SQL Databases on macOS

Denny Christochowitz — Wed, 09 Dec 2020 21:05:31 +0000

Photo by Rodion Kutsaev on Unsplash
Updated: 2020-12-16

Intro

In my day to day work as a consultant on customer projects the typical tech stack involves having a Microsoft SQL Server (MSSQL) for persistence with several ASP.NET Core web APIs on top, and a web-based UI, which nowadays I like writing in React.

Being in home office mode for several months now, I am developing on my personal MacBook Pro - in a Parallels Windows VM. The reason for the latter is the lack (so far) of cross-platform tooling for building Visual Studio database projects and deploying the resulting DACPACs. Our team successfully applied the target-state deployment approach for quite a while and never saw the need for an alternative migration script-based approach (which would not have the Windows platform limitation).

Thanks to a curiosity-driven investigation of one of my colleagues it turns out that the complete toolchain is available now cross-platform which enables us to build and deploy DACPACs on Linux and macOS alike. This post summarizes the steps required to prepare the tools for macOS.

Requirements: Basic knowledge of SSDT, dotnet CLI, and docker.

Develop, Build & Publish Database Projects in ADS

Until recently developing MSSQL database projects was only possible with Visual Studio in conjunction with the SQL Server Data Tools (SSDT).

Azure Data Studio (ADS) is a cross-platform SQL Server Management Studio (SSMS) replacement and provides a similar set of features. Its recent SQL Database Projects extension turns it into a full IDE for MSSQL database projects including build and deployment capabilities.

Download and install ADS. If you're already familiar with VS Code, then you'll quickly feel at home with ADS too. Once installed open the extensions view, search for SQL Database Projects (preview version 0.4.1 at time of writing) and install it as well.

That's all required for working with MSSQL database projects in macOS (and Linux) from the safety of an IDE with publishing targets somewhere in the cloud.

If you want to automate things, or if you prefer to work in the shell, or like to run a local MSSQL instance for developing purposes read ahead.

Building Database Projects in the Shell

Download and install .NET Core or .NET 5.0 if you not already have as we're going to facilitate the dotnet CLI.

A database project build is now just a few keystrokes away:

dotnet build <path-to-sqlproj> \
    /p:NetCoreBuild=true \
    /p:NETCoreTargetsPath=<path-to-build-targets>

The one interesting part is the /p:NETCoreTargetsPath parameter, which points to the SQL Database Projects extension folder from the last section. On macOS this folder is usually /Users/<username>/.azuredatastudio/extensions/microsoft.sql-database-projects-0.4.1/BuildDirectory. This folder contains the ingredients which allows msbuild to cope with .sqlproj files.

The original article describing this approach recommends copying the extension's BuildDirectory folder somewhere to make your scripts independent of the extension, which might be necessary for build servers where no ADS is present. However, this worked not for me. I got a very unspecific error which I was not able to resolve (resp. not willing to put much time into). Instead, I referenced the original BuildDirectory folder directly, which worked just fine.

You will find the resulting build output (the DACPAC file) in ./bin/Debug relative to the .sqlproj file.

Some remarks:

The /p:NETCoreTargetsPath parameter worked only if the given path is absolute.
If you open existing DB projects in ADS, then ADS will modify the projects as described here. They say that the projects will continue to work in SSDT. However, I've not checked this yet.
If your existing DB project contains pre- or post-deployment scripts, then you will have to replace all backslashes in paths with slashes. The build will fail otherwise.

Prepare a local MSSQL instance

If we want to deploy the DACPAC to a local running MSSQL instance, we have to create one first with the help of docker. Feel free to skip this section if you've done this before.

Install Docker Desktop for Mac.

Then pull the docker image for MSSQL server:

docker pull mcr.microsoft.com/mssql/server:2019-latest

After that create a container with a data volume mounted to a host folder:

docker run -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=<strong-pwd>' \
    -p 1433:1433 \
    --mount type=bind,source=$(pwd)/data,target=/var/opt/mssql/data \
    --name hi-mssql -h hi-mssql \
    -d mcr.microsoft.com/mssql/server:2019-latest

The data volume ensures that DB files and transaction logs survive a container shutdown. Note that in the specific case of the MSSQL docker image, binding host volumes with the -v parameter did not work.

Ad-hoc queries can be executed this way:

docker exec -it hi-mssql /opt/mssql-tools/bin/sqlcmd \
    -S localhost -U sa -P "<strong-pwd>" \
    -Q "SELECT 'Hello from SQL Server ' + CAST(SERVERPROPERTY('Edition') AS VARCHAR) + '!'"

If in any case, MSSQL is not starting up correctly, I recommend peeking into the logs. Often MSSQL is just rejecting a not strong enough password. You might find it handy to observe live SQL server log output with this command (in a dedicated terminal):

docker logs hi-mssql -f

Let's close this section with a good entry point for official documentation around docker and MSSQL.

Deploy a DACPAC

As a prerequisite for deploying DACPACs from the terminal we need to install the sqlpackage tool first.

Assuming that we have moved the downloaded folder to ~/sqlpackage, make sure the command file is executable:

chmod +x ~/sqlpackage/sqlpackage

Since sqlpackage is not a known app to macOS, we have to completely turn off macOS's security policy right before the first time we run it, and enable it again afterwards:

sudo spctl --disable-master
sqlpackage
sudo spctl --enable-master

It is NOT recommended to leave spctl disabled! Once re-enabled spctl will no longer complain when we execute sqlpackage again. Remember to repeat these steps if you've updated the SQL Database Projects extension.

Finally, let's extend the PATH variable by adding the following line to our ~/.zshrc:

path+=~/sqlpackage

Reload the shell. Now we are ready to deploy a DACPAC!

The general parameterization of sqlpackage goes something like this:

sqlpackage /Action:Publish /SourceFile:<dacpath-path> /Profile:<publish-profile-path>

The publish profile specifies the target DB server and various deployment options. We can pass most of those as dedicated parameters to sqlpackage. However, using a publish profile is more convenient most of the time as the configuration possibilities are quite overwhelming.

Sometimes you're only interested in the migration script sqlpackage derives from the DACPAC and the target database. In that case, change the Action parameter to Script.

Parameters and a publish profile also can be used together complementing each other. That is handy, if you don't want to have the credentials in the publish profile (which gets normally committed into source control). I've found the following invocation useful:

sqlpackage /Action:Publish \
    /SourceFile:<path-to-dacpac> \
    /TargetServerName:localhost \
    /TargetUser:sa \
    /TargetPassword:"<sa-pwd>" \
    /TargetDatabaseName:<db-name> \
    /Profile:<path-to-publish-profile>

For more details about sqlpackage have a look at the documentation.

Conclusion

Thanks to the latest cross-platform tooling from Microsoft we are now able to develop, build, and deploy DACPACs to MSSQL databases on macOS (and Linux) without having to rely on a virtual machine running Windows.

I hope you enjoyed this post. Any feedback welcome!