DEV Community: Shalvah

Dealing with domain modelling mismatches on external services

Shalvah — Sun, 23 Nov 2025 03:35:59 +0000

Integrating with external services is a pain. The technical challenges—reliability, performance, caching, downtime, and so on—are well-known and often solvable with infrastructure. But a less-obvious non-technical challenge is domain modelling mismatches.

Domain model mismatches happen because people see the same world differently. To you, it's a House; to them, it's a Building. To you, it's a User; to system A, a Customer; to system B, it's a Client. And the way we describe our world defines how we interact with it.

I've found that these mismatches are especially tough in these cases:

Multiple entrypoints. When entities in your system may be created or modified via other systems. Example: Your app has “Customers” that can sign up directly, but your sales/finance team can also create clients in Stripe for invoicing. You need to sync these new users into your system.
Bidirectional sync. When you need to also sync entities created in your system to external systems. For instance, you may need to push some data about your users to your helpdesk such as Intercom or Zendesk, so agents can have richer context about a user when providing assistance.
Partial domain coverage. When each system only covers a slice of your overall picture. For example, your sales CRM stores sales data, your billing system stores payment info, your helpdesk stores support tickets, and your analytics system stores usage activity. To your system, all these form a complete picture of the “Customer” model.

Here's a sample system that might have to deal with all these cases. It stores a User object, some parts of which it syncs to some other systems, and some parts which it syncs back from others. Each of the other systems only stores a few pieces of the User object.

Of course, ideally, you want to avoid these. And you especially want to avoid them occurring on the same project. But you can't always avoid it.

I've worked on projects that somehow checked all of these. In our case, we wanted to do things such as tracking a user's journey from way before they actually became users, so we allowed them multiple entrypoints into our system.

I'll explore some of the ways mismatches show up, with some real examples I've encountered.

Naming

Different systems may use different names.

Examples:

We had a model called an Appointment. On our calendar tool (Calendly), it was an Event; on our sales tool (Pipedrive), an Activity; on our contact tool (Aircall), a Call.
Similarly, a Customer on our end was a Calendly Invitee, a Pipedrive Person, and an Aircall Contact.

Sometimes these are just differences in nomenclature, with no differences in behaviour. In such cases, the only headache is keeping track of the names, especially when they clash with something else in your system.

Other times, the names reflect the difference in semantic purposes and implicit context that these models carry. Calendly uses the term "event" because their domain is concerned with the generic act of scheduling something to happen at some time; we use "appointment" because we are interested in the time, but also the meeting with someone. Similarly, a call in Aircall focuses only on a conversation via a specific medium, and does not have to be scheduled. And Pipedrive Activities are not necessarily appointments; they only represent an action taken in the sales process.

Apart from the models themselves, the actions available might have different names: Appointment.create vs Event.schedule, Invoice.mark_as_paid vs Transaction.pay, etc. Once again, these could be significant behaviour differences or not.

Overall, these semantic differences are usually pointers to more serious behavioural differences, as we'll see below.

Spelling

The same thing might be spelt (or **spelled) differently.

The least significant difference, but it's not to be overlooked. Spelling mistakes can cause bugs if the right tooling isn't in place—for example, checking if event[:status] == :cancelled (British spelling) when Calendly spells it canceled (American).

Structure

One system might represent some information using one field, another might split it across multiple fields or even multiple models.

If the mapping is straightforward, this isn't too annoying; you just need to remember to do the conversion at the service boundaries.

Examples:

We needed to pull in card transactions from an external card provider. On our end, we just cared about the time the user performed the transaction. However, the provider's API had multiple timestamps: authorisationTime, paymentTime, paymentLocalTime, and createdAt, fields with different meanings. We don't care as much about the distinction between these, so we built a transactionTime in our system based on a combination of these.
A user's sales journey was tracked in Pipedrive typically as a Lead object, which transitioned into a Deal object. Even though Pipedrive had a Person object, sales agents tended to track information about the user right on the Deal object. So syncing users to our backend meant we had to pull in data from the Deal as well.

Validations

One service might allow things another rejects, and vice versa.

Examples:

When creating customer accounts for testing, we often used an email ending with a .test TLD. Our new customer support tool, Dixa, rejected such domains, so when we began importing our existing users into their system, we would run into failures.
Our sales agents sometimes signed users up with phone numbers only, and a "fake" email (for instance, older people who weren't very Internet-savvy). In many cases, these emails were technically valid, but not allowed by Dixa, so we were unable to import these users into it.
Pipedrive allowed agents to create "Activities", which mapped to Appointments on our end, but did not enforce the presence of a date and time.

Flows

The lifecycles and workflows of corresponding domain models might be different across services.

A model in one system might go through states or transitions that don't exist on yours, or vice versa. This hurts especially in a many-to-one setup, where every new provider has its own state machine, and it might not.

Sometimes, an external model may have states or properties that you have no idea about. Sometimes, no one on the team knows!

Examples:

On Calendly, an event can only have two states: active or canceled. On our end, our Appointments could be created, rescheduled, canceled, or completed.
We added a wallet provider who performs KYC on each user. They track this via their User model's status. However, our internal user model was much simpler than their KYC workflow, so we couldn't track the KYC status on it.
We had a bank partner that would send us transactions labelled deleted and hidden. Somehow these made their way into our own Transaction model, but there was never any real clarity on what those meant.
We were syncing card transactions from a card provider. At first, we synced all transactions, but later realized we only cared about Settled transactions. But then we found out a card transaction's lifecycle is more complex, and it can be Settled multiple times, or never, or with a different amount. This flow was new to us, so we displayed wrong info to customers in some cases. Only after a few reports did we dive deep into understanding the lifecycle and deciding how we would model it.

Restricted and allowed behaviours

A service might attribute certain behaviours and rules to their models that you don't.

These can either limit what you can do or force you to add extra validations.

Examples:

Dixa, being a customer service platform, allows multiple emails and phone numbers to be associated to a user. We did not. This created a problem when syncing these users to our system.
We allowed multiple users to share a phone number, but Dixa does not.
Calendly events are nigh-immutable. Rescheduling an event cancels the old one and creates a new one. Calendly sends a separate webhook for cancelled, and one for created. On our end, this looked like two separate actions.
Pipedrive activities are very mutable. This caused us a ton of grief, as the sales agents did a lot of strange (or normal but unexpected) things that regularly caused problems in our system. For instance, we synced only Activities of type "Consultation" to our system (as Appointments). But sometimes, the activity type may be changed later, which wrecks our model. Also, an agent might delete an Activity, which was okay (we can just cancel the appointment on our end), but meant we could not fetch any information about it from the Pipedrive API.

Identifiers

The same resource might be identified by different keys on different systems.

Examples:

Calendly: The email is the unique identifier for an event attendee. However, sometimes our users used a different email to book the appointment on Calendly, which meant we couldn't find them in our database!
Dixa requires a user to have at least one of email or phone. This makes sense for their domain (they're a customer contact platform), but led to a duplicates situation for us: if an unsynced user contacted us via email and via phone separately, Dixa would create this as two separate accounts, one with their email, and one with their phone.
Pipedrive did not require emails or phone numbers for users, or enforce uniqueness of those. This led to a ton of duplicate users (at some point, over two thousand).
Dixa supports an external_id field, allowing us to uniquely link a user on their end with one on ours. But this only worked for users created in our system, and synced to theirs. For users created on their end, we had to use heuristics to match them uniquely to a user on ours.

Scheduling

Different systems might operate on a different schedules.

Examples:

Some of our banking providers would notify us of new transactions in real time, which is nice, so our database is always up to date. Other providers provide us a list of new transactions once a day, and others once a week. This is a modelling mismatch—to us, a BankStatement is a report of all transactions on your bank account; to them, it's a report of at least all transactions up to the past week.

Dealing with domain model mismatches

First, think again. Before adding a new domain model, ask, "Do we need this? Does this entity represent an actual object intrinsic to our domain, or is it merely a record we mirror from elsewhere?" I've seen setups with domain models representing external entities that had no semantic value to the service. This led to the rest of the codebase being coupled to these unnecessary models. If the model isn't core to your domain or doesn't affect any key business logic, then avoid maintaining yours. In this case, sometimes all you need is a few lines of code mapping one field to another.

Next, pick your battles. Now that you've decided your own model is probably needed, you must decide how much of a mismatch you're willing to tolerate. Should you try to follow their architecture? Should you relax some of the constraints on your end? Should you assume any future partners will follow the same model? It is tricky, especially when you don't yet know much about the domain. One strategy is to just go with how the external service models it, but this can have the long-term consequence of locking you in.

The best engineers I've worked with have had the ability to come up with a model that was flexible enough to allow for change or different external providers, without over-engineering things. At the very least, you'll have to:

research further into the domain and the external service's model to understand use cases, limitations, and comparison with your business case
communicate and brainstorm with product and business leads to better understand their vision

On the technical side, some things we've done include:

external_reference and external_service columns. In cases where a single model on our end might map to multiple entities on other services, we used a separate table, consisting of reference, service, and entity.
JSON(B) columns to dump some additional service-specific fields that may be relevant only for that field. Some people find this controversial, but I've found worked better than having sparse columns for each service provider and was easier to maintain than entity-attribute-value.
decoupling models that represent a sub-process or sub-component of another, such as in our KYC mismatch case above

Consider separating domain models, especially useful when workflows/structures don't match. For instance, in the case of the KYC lifecycle above, we created a User::KYC model to track specifically the KYC process for that user. On our provider, this information was still a part of the User object, but for us, it allowed us to examine the KYC process independently without needing to clutter our User model. It also served as a form of Bounded Context.

Embrace the workarounds. You will run into cases where there's no "clean" solution, and it's okay to do something unorthodox. Some examples:

Scheduling mismatch: Calendly webhooks sometimes were delayed, so to prevent users seeing missing data, we implemented a workaround where we would call their API on demand, to "prefetch" the expected webhook. And since we could potentially be processing the webhook at the same time, we had to handle this race condition with a mutex.
Behaviour mismatch: Since Calendly models appointment rescheduling as cancelling + creating new, we had to at some point implement a "waiting" loop in our webhook processor to check if an appointment had just been rescheduled or truly cancelled. In a related problem, we implemented something called intents to get around some restrictions on rescheduling.
Identifiers mismach: Since there was no exact way to prevent Pipedrive duplicates, we ended up adjusting our sync process to also look for and merge duplicates whenever syncing.

Workarounds must be accompanied by documentation. Document key differences and details of the integration. This applies to any aspect of external service integrations, but can be super helpful for domain modelling, because there are so many decisions, small and large. Every kind of documentation helps, from code comments to full Wiki pages. Future developers who need to touch this code can see at a glance that a certain external service represents information in this weird way, so this is why we must do this random piece of witchcraft.

Establishing strong domain boundaries is also key. As much as possible, aim to keep the mapping between different systems at the outer edges of your system. Your core domain models should stay agnostic of the providers you choose, to an extent, and then a Translation Layer can deal with figuring out the various kinks.

I've also found that setting up good tooling can help smooth things over. Custom lint rules can help, from complex rules that enforce inter-service dependencies to simple ones that correct all spellings of "Cancelled" to "Canceled". Good integration libraries also help. I'm increasingly of the opinion that it's better to make your own API client, because the external SDKs often come with the assumptions of their model that don't translate cleanly to yours.

And don't forget observability. Make your code shout at you. Let your system tell you about its state. Don't just set up logs, metrics, and traces; store information that is useful, such as change history, event history, etc, and build useful tools and admin panels to explore that information. For our syncs, we invested a lot of effort in making our webhook processing as observable as possible: we could explore, track, and replay webhook payloads; we could see the result of a specific webhook, and what entities it had created or modified. This greatly aided us in finding the errors caused by mismatches. In the case where we needed to prefetch items from Calendly, we added metrics to tell us how often we had to do this, how often it was actually useful, and the overall impact on our system.

Lastly, architect for change. I've learnt that migrations are inevitable. Assume that someday we will have to migrate to another provider, or APi, or infrastructure, and so on. So don't architect to the current provider, but also don't make your architecture as generic as possible. Instead, focus making it robust, observable, documented and clear enough that changing it is not an expensive process—its dependencies are not hidden, its tradeoffs are understood, and its structure is clear.

In general, domain driven design has several patterns that help here. Of course, they come with their own tradeoffs, so choose appropriately. I'm personally still exploring many of these patterns to see how I can leverage them without taking on the overhead.

All decisions are wrong, but some are better

Shalvah — Tue, 04 Nov 2025 17:16:32 +0000

Everything is fucked, no systems are sound, all decisions are wrong, and we should destroy all software.

I've noticed this recurring trend at my last several companies:

We have an existing project. People take jabs at it, commenting on how it's "a mess" and "legacy code". I've seen this applied to decades-old systems as well as three-year-old ones.
We start a new project. People are happy, commenting about how it's nice to get a fresh start, stick to "clean code", and avoid making the mistakes made on the other project.

These are perfectly reasonable attitudes. Software systems grow in complexity, and some can turn into messes really quickly. Working on a greenfield project is always great—who doesn't like a fresh start? A decade ago, I was the one advocating for and proposing software rewrites. Let's clean up the mess!

But there's a last part of the loop:

Eventually, people start realizing: We may not make the mistakes of the other project, but we'll make our own set of mistakes. Our code ends up being legacy code.

But here's the thing: it's the circle of life!

There are no perfect systems. Our code evolves, has to handle more cases, becomes legacy. It becomes full of cruft, tradeoffs and pain. How do we avoid this?

First, know that these two things are true:

All decisions are wrong. All architectures are inadequate. All abstractions are poor. All nontrivial code is hard to read.
Some decisions are less wrong than others. Some architectures are better than others. Some abstractions are better. Some code is easier to read than others. Some systems are easier to work with.

However, the line between "wrong" and "less wrong" is more than just your feeling of legacy and confusion. Unless you start a new project every three years, you'll still end up with that feeling. There must be a better way.

All decisions are wrong

You'll never architect the perfect system. Oh, using integers for IDs is bad because of enumeration and overflow? How about UUIDs? Oh, that's bad for storage and search performance? How about some other random ID format? And so on, and so forth. There will always be some bottleneck, some situation that you can't fix.

During design/architecture discussions, I tell folks, "We need to accept that whatever decision we take is the wrong one. We just need to pick one we can live with." Once you go beyond CRUD, you begin to solve problems where you have to play by certain constraints. Because of this, you can't merely pick an idealized solution.

But some are better

That said, it is undeniable: there are clearly some ways that are worse. Fetching all items from a database and letting the client paginate it is obviously a worse decision than implementing pagination on the backend. But in real-world business cases, we deal with a lot more non-obvious situations. Should we normalize or inline this data? Should we use strings or integers here?

My litmus test is this: If you claim that a certain architecture/design is worse than another, you must be able to:

point out its problems (with data, preferably)
point out its strengths (Yes, you need to try to understand why it was chosen in the first place.)
explain why the alternative fits this situation better
point out the problems of the alternative

In summary, it is not enough to say "This code sucks." You must be able to say why it sucks, why it might have been chosen, point out a better option, and acknowledge the tradeoffs involved.

It's not just about what you do, it's about how you do it

In the best systems I've worked on, there were some unconventional things that had to be done. These were things that might make a new joiner immediately go "WTF". And yet they did not make the system a pain to work with. How?

First, picking and sticking with conventions. I used to be very anti-convention, feeling that they stifled creative freedom and experimentation. That can be true. But I've also come to see that going with a convention, even a poor one, makes things easier for everyone. We don't have to judge these situations afresh every time, deciding again on each one. We can save our energy for the important decisions.
Next, documenting assumptions, decisions and details. Make things easy to find. Document the conventions. Document when and why you stray from these conventions. One of the biggest things that makes a codebase "age" quickly is developers doing strange things and leaving no explanation. Write, write, write. Write comments, write pull request descriptions, write wiki docs, draw diagrams. Leave breadcrumbs. Even if the system is a mess, make it an understandable mess. Better overdocumented than undocumented.
Finally, an attitude of constant improvement. The best systems I worked with recognized that the system was a living, evolving being. Rather than fixating on "getting it right" from the start, we were open to suggestions for improvements. We encouraged people to fix things they saw wrong, or propose new conventions if they solved problems we were having. Sometimes, we had a running list of technical improvements we wanted to work on, and encouraged folks to pick up tasks from there.

The human cost

We struggle with these legacy systems, not merely because they're technically worse, but because they overload us. They either require more developers, or they place a larger mental burden on each developer. They require more intricate solutions to common problems, or more effort to change or adjust things. One of my worst experiences was a project where changing a simple label in the UI took me three days. It really sucked, and had me questioning my skills.

So whether architecting our new system, or living with the old, an important question to ask is How can I reduce the human cost here? How can I make things easy to discover? How can I make the system easy to change?

Making non-atomic actions atomic using intents

Shalvah — Mon, 27 Oct 2025 23:08:28 +0000

Context

At a previous job, we built a service we called our operations platform. Its aim was to unify the SaaS tooling used by our ops teams with domain data from our core business, so we could directly see how the ops efforts contributed to our growth. A key feature here was an appointment booking system, where users could book consultations with our sales team. For a start, we chose to store appointments in our database, but rely on Calendly for the actual scheduling. I'm not sure this was the right choice, but the key reason was that our sales team depended on several of Calendly's features, and we needed to move fast.

With this approach, the sales team continued to work in Calendly, and we embedded the Calendly UI in our app for users to schedule appointments. We would then receive webhooks from Calendly and create the appointment in our backend. Our backend served as the source of truth for our app.

As one might expect, we encountered several issues in this approach. This post is about one of those.

Constraints

One of the features our sales team relied on was Calendly's pooling feature (round robin). They could create pools consisting of several agents, and a customer who booked an appointment would be assigned to one of the agents in the pool. However, if the customer later rescheduled, the customer would keep the same agent they were assigned to. The business wanted to change this, so that reschedulings would go back into the pool.

Calendly didn't provide any option for this. The only solution was to do it in two steps: cancel the existing appointment on Calendly and book a new one, thereby picking a new agent from the pool. But this was untenable: we needed the process to be seamless for our users.

Solution

To solve this, I came up with a high-level API called PendingCancellations. A PendingCancellation represents an intent to cancel an appointment. This intent may be acted upon (thereby cancelling the appointment) or discarded. We would still do rebooking in multiple steps, but with a PendingCancellation, it would look like this (at a high level):

create PendingCancellation (intent to cancel old appointment)
 -> book new appointment
  -> execute PendingCancellation (actually cancel old appointment)

For the implementation, we wanted to present this as a single step to our customers. We also need to keep in mind this detail: appointments are actually created on Calendly (which calls them "events"), then synced to our database via webhooks.

So the full implementation looked like this:

User clicks "Reschedule" in our app
The app tells our backend to create a PendingCancellation, linked to the user and appointment in our database.
Then the app redirects to Calendly for booking a new appointment (not rescheduling)
User completes the booking
Backend receives Calendly's event.created webhook.
We process the webhook. If there is a PendingCancellation, we go ahead and cancel the old appointment and create the new one.

Failure modes

This was a fine API. But, of course, I also had to think about failure modes:

What happens if the user changes their mind and exits without completing the new booking?

Nothing. The old appointment remains active. All unprocessed PendingCancellations are ignored and cleaned up after a few hours. This is the beauty of the intent system: if it isn't explicitly acted upon, it has no effect.

What happens if the user enters this flow (clicks "Reschedule") multiple times?

We only allow for one PendingCancellation per appointment. If there is an existing PendingCancellation, we simply update its timestamp.

What about transactional guarantees (consistency)? Since this is not an atomic operation, what happens if one action fails (cancelling the old one or booking the new one)?

This was tricky. Normally, we could wrap the whole operation on our backend in a database transaction (cancel old + create new). This way, if one failed, there would be no side effects, and we could safely retry the webhook.

However, since the appointments are first created on Calendly before being synced to our database, we don't have control over the "create" part. Regardless of what happens in our database transaction, the new appointment already exists on Calendly, and it might cause confusion for a user if they saw no change in their app.

So I chose to do something different: when we receive the webhook, first create the new appointment before cancelling the old one. Also don't bother wrapping it in a transaction. That way, if the cancellation failed, we would end up showing the user both old and new appointments. This is undesirable, of course, but, to my mind, it was better to see 2 appointments than 0 (if cancel was executed first).

In hindsight, I'm not sure this was the better decision. Seeing 2 appointments after rebooking might be worse, since it might tempt the user to try cancelling the old one. It also meant we had to make the webhook processing idempotent so we could safely retry the webhook without creating a third appointment. 😄

Dealing with limitations

We still had a few more challenges to solve. A key one was linking the new appointment from Calendly to the old one in our database. Essentially, when we receive Calendly's event.created webhook, how do we know if this is a rebooking? Calendly's UI doesn't allow us to attach any metadata to the event (such as old_appointment_id=abc), so we have no way of differentiating between our rebooked appointments and truly new appointments.

The only thing we had was the user ID (sort of—but that's a story for another time). We also knew two things:

a user can only have one appointment of a given type at any time
the new and old appointments must have the same type

So I worked with that: when we receive a new event booking webhook, fetch all PendingCancellations for that user. From there, we use heuristics: find the most recent PendingCancellation matching the new appointment's type, and pick that. This worked reliably.

Conclusion

I'm really proud of the PendingCancellation API. It's a nice, lightweight solution that fits neatly into our constraints and is generally reliable. It of course comes with tradeoffs, but is generally pleasant to work with.

Two sources of inspiration I had were Stripe's PaymentIntents and Android Intents. They're not exactly similar, but the name "Intent" conveys a desire to perform an action, without any actual commitment. It is entirely possible for the desired action to expire or be invalidated before it can be completed, and the intent should handle this gracefully. I was also inspired by the concept of change requests, something we already used quite effectively in our codebase.

DIY Smart home project: Presence-activated lights

Shalvah — Thu, 01 May 2025 12:43:25 +0000

Background

I got these Antela smart bulbs a while ago. They connect to my home network, and I control them from my phone via either the Tuya Smart app (Tuya is a Chinese IoT service provider) or Google Home. I got the smart bulbs rather than smart switches because they allow me to customize the brightness and colour, a feature I found helpful when my eyes were super sensitive to light after eye surgery.

I also recently began to use Home Assistant as my smart home software. It's an open-source and more powerful alternative to Google Home/Amazon Alexa/Apple HomeKit/Samsung SmartThings. With those vendor platforms, you're limited to what they support. With Home Assistant, you can build your own setups and connect things as you wish, whether it's totally DIY or consumer-ready products from vendors. You install it onto an always-on computer (I have mine running on a Raspberry Pi 5) and connect it to your other smart devices, and it manages them for you according to your configuration

I recently got an idea for a project. In my typical daily workflow, I spend most time in my office and living room area, occasionally dashing to the bedroom to get something. I find it a minor annoyance having to stop to turn on the lights, so I wanted to have the bedroom lights turn on whenever I walked in.

This means I need some sort of motion- or proximity-activated light. In the world of smart homes, there are always many different options. There are consumer-ready solutions, such as this light—plug it in, and that's all. When you come close, the light comes on. There are kits such as the Everything Presence One that only provide the sensor + networking functionality—plug it in, connect to your network, and it shows up as a device in your Home Assistant instance. From there, you can configure Home Assistant to turn on your lights based on the device state.

I wanted to go a little more DIY. Instead of buying the presence sensor kit, I would build mine. My plan was to get an actual presence sensor element (such as this) and set that up to send data to Home Assistant, which can then control my lights.

Schematic

Here's the setup:

The presence sensor detects human presence in an area.
The microcontroller (physically wired to the sensor) powers the sensor and relays readings from it wirelessly to my Home Assistant instance.
Home Assistant turns on or off my lights when needed, depending on the readings from the sensor and my configured automation.

Instead of manually programming the microcontroller, I'll run ESPHome on it, for two reasons:

ESPHome is made by the makers of Home Assistant, so it already integrates with it
ESPHome simplifies the programming so you can do everything with a few lines of YAML. No code required.

Equipment

For the microcontroller, I got an ESP32, specifically the ESP32-C3-DevKitM-1U (that's a mouthful!). The ESP32 naming convention still confuses me (somewhat helpful article and chips comparison), but here are my reasons:

I got an ESP32 because I've seen it recommended a lot. It's small and cheap, easy to program to do "one thing". Other options could be the Raspberry Pi Pico W (which ESPHome also supports) and an Arduino (no idea about support).
I got the C3 series because it has WiFi and ESPHome recommended it as the low-power version.
I got the DevKit version because it comes with some conveniences for noobs: USB-to-serial converter, voltage regulator, GPIO pins already exposed, and reset & boot buttons.

For the sensor, I got a passive infrared (PIR) sensor, Adafruit’s BS412. It has a great range of several meters, and is easy to wire.

Aside: Active vs Passive IR sensors

I initially got the Vishay TSSP93038SS1ZA infrared sensor, but this was a mistake. During testing, I realized that this is an active infrared sensor. It detects a source of IR light pulsating at a specific frequency, like an IR LED or a TV remote, which means it needs a dedicated IR emitter.

Active IR systems are good for "light barriers": you have an emitter at one end and the receiver at the other. When something blocks the beam coming from the emitter, you trigger something. This kind of system is used in elevator doors, automatic taps, soap dispensers, towel dispensers, hand dryers, item counters on conveyor belts etc. But it wouldn’t work well for my desired setup; I need to sense the presence of a human within a given area (not only in a straight line). PIR sensors detect natural radiation of the human body, and are a better fit for the presence sensing application.

A light barrier simply detects when the light source is interrupted. Source

A presence sensor has a wider, 3D detection area, and works by sensing the IR radiation from humans. Source

Getting used to the ESP32 and the sensor

From here, I could jump straight to installing ESPHome on the ESP32 and configuring it. But of course I had to first program it manually and test some basic behaviour. There are multiple ways to program an ESP32, such as with Python or via the Arduino IDE, but I followed Espressif's Getting Started guide, which uses their official C framework, ESP-IDF.

I did get stuck a few times, but nothing too serious. The only other microcontroller I've programmed is the Raspberry Pi Pico Zero, and that was much easier, because the it uses Python and has simpler guide docs. I'm new to ESP32, and it seems more powerful but more complex. ESP-IDF is in C and has fewer beginner guides.

It also took me a while to find the right pinout diagram, but it helped that the ESP32 had labels on the actual board (DevKit feature?).

To connect the BS412 to the ESP32, I followed the product page and datasheet [PDF]: pin 1 and 2 to GND (ground), pin 3 to 3.3V, and pin 4 (output) to a GPIO pin. (If you have trouble identifying which pin is which, there's a tiny tab on the sensor's head. Check the datasheet for the cross-section of the circular head to see the numbering of each pin relative to the tab.)

I also connected a LED to another GPIO as an indicator (note: this isn't necessary, since the DevKit comes with an RGB LED on GPIO8). Then I wrote an IDF program to read its value and light the LED. Interestingly, the PIR sensor has a HIGH output when it detects motion, different from the active IR sensor, which has LOW output when it detects an IR signal.

#include <stdio.h>
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "driver/gpio.h"
#include "esp_log.h"

static const char *TAG = "demo";

#define PRESENCE_SENSOR_PIN 19
#define INDICATOR_PIN 10

void app_main(void)
{
    gpio_reset_pin(PRESENCE_SENSOR_PIN);
    gpio_set_direction(PRESENCE_SENSOR_PIN, GPIO_MODE_INPUT);
    gpio_reset_pin(INDICATOR_PIN);
    gpio_set_direction(INDICATOR_PIN, GPIO_MODE_OUTPUT);

    while (1) {
        int sensor_value = gpio_get_level(PRESENCE_SENSOR_PIN);

        if (sensor_value == 0) {
            ESP_LOGI(TAG, "No motion detected (LOW)");
            gpio_set_level(INDICATOR_PIN, 0);
        } else {
            ESP_LOGI(TAG, "Motion (HIGH), %d", sensor_value);
            gpio_set_level(INDICATOR_PIN, 1);
        }

        vTaskDelay(pdMS_TO_TICKS(500));  // Wait for 500ms before reading again
    }
}

Aaaand this works. Nice.

Tuning the sensor

Now we begin to tweak and configure it so it works how we want.

The key sensor parameter to be tuned is the on-time, the length of time the signal stays HIGH after detecting motion. There's also a lock time, which is a minimum amount of time the signal will stay LOW before entering another HIGH phase.

The sensor works in phases:

Motion detected → output goes HIGH and stays like this for at least the on-time
Motion stops → after a short time (on-time exceeded), output goes LOW
Lock time starts now → during this, no new detection possible
After lock time ends → PIR is ready to detect again

Presence sensors are usually used in such phased setups: when a person enters the area, turn on the lights. Then turn them off if the person has not moved for a while. If the person is still in the area, they'll move and the lights will come back on.

The sensor's datasheet explains how to adjust the on-time, but the lock time is not configurable for this sensor. Luckily, it's small enough (2.3 seconds) that it doesn't matter. Adjusting the on-time is done by adjusting the potential difference between pin 2 and ground using a voltage divider setup, diagrammed below:

The datasheet has the recommended resistor values for the resistors in order to achieve different on-times. The shortest on-time is 2 seconds, which you get by connecting the pin directly to ground (0 V difference). The longest on-time is 1 hour (potential difference of 0.5 * the supply voltage). For my case, I wanted an on-time of around a minute, so I used a couple of resistors that added up to 310 kΩ as R2 (R1 is always 1 MΩ).

An alternate way to adjust the on-time is by not adjusting it, but using software to override the effect. You can add a delay in your programming, which is essentially implementing a state machine: when the sensor goes HIGH, your program sets its internal state to HIGH and keeps it that way until your desired on-time expires, ignoring any sensor changes during that time. I eventually switched to this. More on that later.

Switching to ESPHome

The next step: install ESPHome onto the microcontroller, and program it via ESPHome's YAML config. There are 3 stages to this (see Getting Started):

Prerequisite: Install ESPHome Device Builder add-on in Home Assistant
Install ESPHome on the microcontroller
Configure the device to connect to your WiFi
Add the device as a Home Assistant entity

The installation is mostly straightforward: plug it in via USB (if it has a serial-to-USB chip, which the DevKits do), and then use ESPHome Device Builder to flash it with the firmware (for which you'll likely be redirected to ESPHome Web).

I had trouble with this though. For anyone who runs into this in future (or myself), I'm documenting my issues and fixes in asides. Click to see the details.After running “Prepare for first use”, I kept getting “An error occurred. Improv Wi-Fi Serial not detected”. I think that ESPHome Web is supposed to use Improv WiFi to send my WiFi credentials to the newly flashed microcontroller, but somehow that was failing. I eventually found an alternative on the forums: the flashing process also turns my ESP32 into a temporary WiFi access point, and all I needed to do was connect to that, then visit the gateway address (highlighted below). That page showed me a form where I could enter the credentials for my home network, and that was done. From there, it turned off the AP, then took about 2 minutes before showing me a newly discovered device in ESPHome.

Once that is done, you should see the newly discovered device in ESPHome, from where you can "Take Control".

Taking control installs a basic ESPHome config onto the device, prompting you for some initial config such as a friendly name.

I ran into another issue here. It complained about the device not being able to connect to my WiFi, even though I had already done that in the previous step. I guess the WiFi configuration in ESPHome Web and ESPHome Device Builder are different, but I'm not sure why it didn't prompt me to add the WiFi credentials to the config first before trying to connect. The resolution for this was clicking "Edit" to edit the newly generated config YAML file, and then add a section containing my WiFi credentials ([docs](https://esphome.io/components/wifi.html\)\), then click "Save" and "Install".Finally, the device was successfully set up, and my device had a basic config:

Now time to configure my sensor. Here's what I added to the file at first:

binary_sensor:
  - platform: gpio
    pin: GPIO19
    device_class: occupancy
    name: "Bedroom Presence"
light:
  - platform: status_led
    name: "Presence detected"
    pin: GPIO10

This declares a Binary Sensor, a sensor which can be in one of two states (on/off). This component allows you to use any HIGH/LOW sensor without needing a custom integration for it. The occupancy device class tells Home Assistant how to interpret the states.

The second section declares a Status LED. ESPHome automatically knows to use it as an indicator—turn it on when the sensor is on, and vice versa.

This is beautiful. A few lines of YAML and I had this working!

Automating with Home Assistant

The final piece was to hook it up with Home Assistant. To do this, I went to Home Assistant's Settings -> Devices. The newly added ESPHome device had been discovered, so I clicked Add, assigned it to my Bedroom, and we were ready.

I went to add the automation, and discovered Home Assistant already comes with an inbuilt automation for "Motion-activated lights". I tried to use this at first.

But this didn't match my use case, since I don't want to turn on the lights every time, just from the afternoon until when I go to bed. So I made mine instead. Two actually—one to turn on the lights when the sensor detects a person, and one to turn them off when the detection status clears. I set them up to only work from 1 pm to 12 am.

You could also do this with only one automation, but I found it easier to wrap my head around this.

And voila! It worked. But there's more...

Power issues and more tweaking

The next thing I tried to do was make this mobile. Powering microcontroller boards is always a challenge. Thus far, I had simply plugged it into an old USB charger, which delivered the 5V needed. But I wanted to try a different location for the sensor, so I wanted to see if I could power it off battery.

I connected a 9V battery to the GND and 5V inputs (the DevKits come with a voltage regulator that ensures the board only gets 5V), and this worked...for about 2 days, and then the battery was empty! That was unsettling. I thought microcontrollers were supposed to be energy-efficient? Well, it was time for me to do some more tuning and learning.

First, I got rid of the LED. It was only meant to be a temporary aid during development; no need to use that extra power.

I also ended up getting rid of the voltage divider setup and going back to the original circuit, where pin 2 was connected to ground, and on-time was the default 2s. To implement the on-time in software, I added a delay in ESPHome.

binary_sensor:
  - platform: gpio
    pin: GPIO19
    device_class: occupancy
    name: "Bedroom Presence"
    filters:
      - delayed_off: 45s

I don't know if this saves a lot of battery, but I'm much happier with the software version of on-time. It makes the hardware connection simpler, it's easier to maintain and adjust, and it's much more flexible. (Alternatively, I could add this delay into the Home Assistant automation, but I preferred to have it as part of the device, so the state of the sensor always matched the state of the lights).

Still, I learnt (thanks to ChatGPT and forums) that the common 9V batteries are pretty bad for powering microcontrollers. Despite providing a large voltage, they have low capacity (~500–600 mAh), worse than a typical 1.5V AA battery. Additionally, some energy is wasted in the voltage regulator.

And that's when I learnt about sleep mode. ESP32 can go into sleep and be woken up after a certain time, a sensor, or a touch. ESPHome supports this via the Deep Sleep component.

I only needed the sensor to be active from 1 pm to 1 am, so there was no point keeping it awake outside of that time. I experimented with things, and ended up adding a config like this:

deep_sleep:
  id: deep_sleep_1
  sleep_duration: 12h

time:
  - platform: sntp
    id: sntp_time_1
    timezone: "Europe/Berlin"
    on_time:
      # At 01:01 pm every day, block deep sleep
      - seconds: 0
        minutes: 1
        hours: 13
        then:
          - deep_sleep.prevent: deep_sleep_1
      # At 01:01 am every day, enter deep sleep
      - seconds: 0
        minutes: 1
        hours: 1
        then:
          - deep_sleep.allow: deep_sleep_1
          - deep_sleep.enter:
              id: deep_sleep_1
              until: "13:00:00"
              time_id: sntp_time_1

There are three parts here:

A deep_sleep configuration to sleep for 12 hours, called deep_sleep_1
An on_time trigger that enters deep_sleep_1 at 1 am every day (I gave a buffer of +1 minute).
An on_time trigger that blocks deep sleep at 1 pm every day.

As long as my controller remains on, it will switch in and out of deep sleep at these times. But since sometimes I may unplug or disconnect my device to work on it, it might not be on when the time trigger comes around. So I also added an on_boot trigger to check the time whenever the ESP32 is turned on, and enter deep sleep if needed.

esphome:
  # ...
  on_boot:
    then:
      - delay: 1min
      - if:
          condition:
            and:
              - time.has_time
              - lambda: "return id(sntp_time_1).now().hour >= 13;"
          then:
            - logger.log: "It is past 1pm. Sensor activated."
            - deep_sleep.prevent: deep_sleep_1
          else:
            - logger.log: "It is not yet 1pm. Entering deep sleep."  
            - deep_sleep.allow: deep_sleep_1
            - deep_sleep.enter:
                id: deep_sleep_1
                until: "13:00:00"
                time_id: sntp_time_1

The deep_sleep.prevent action (which blocks deep sleep) isn't really necessary in both triggers, but I kept it in there as a sort of failsafe.

As with most things here, there are multiple ways to achieve this configuration. This approach looks a little messy, but I found it relatively easy to reason about for me.

So now, when the time rolls around, or I plug in my ESP32 before 1 pm, I see logs like these:

It enters deep sleep and we can see that it has an "expected disconnect" from the ESPHome API. During this time, the device shows as "Unavailable" or "Unknown" in Home Assistant (although sometimes it just sticks on the last known state). And then when the wakeup time rolls around, it is able to reconnect:

Deep sleep is pretty cool. I'm currently testing with a new battery to see how long it lasts in this setup. I may eventually switch back to USB power, though, since the sensor won't be moving around, so battery power is less necessary.

Final thoughts

Finishing

At the end, the product still looks like a work in progress. It's on a breadboard, which is used for prototyping, and there are jumper wires everywhere. To make this a truly polished product, I could:

move this from a breadboard to a custom PCB (printed circuit board)
replace the DevKit with a regular ESP32, which is more energy-efficient
make a custom case and mount for the whole thing (good use case for 3D printing)

However, these are secondary to my current goal, and I don't have the time to dedicate to them, so I'll move on for now.

Debugging and observability

A tricky thing about working with microcontrollers is debugging. In the initial development phase (when it's plugged in to your laptop and you're coding), it's easy, since you can log things to your console. But when it's out on its own, running in the wild, it's like a black box. The device might randomly stop working, and you have no idea why. Microcontollers are very tiny devices so they don't store logs, which means you'll never know what exactly happened in the past. You can only connect to your laptop and try to reproduce it.

Using a LED as an indicator to show the current state is a tiny help. A thing I want to try in future is having the ESP32 wirelessly send logs to another device, then having that device store the logs.

Buying parts

A surprising challenge here was figuring out the best place to get parts (cost and speed).

I started with Conrad (also because they have a store near me), but their selection is a bit limited and prices are higher (for example, an ESP32 DevKit costs €22). This is similar to what you get if you shop on Amazon.

Next up were European/German retailers like mouser.de, digikey.de, Distrelec, and Reichelt. These were super cheap (ESP32 DevKit for €9.31, and PIR sensor for €1.85), but had shipping times of around 5 days.

It turns out that most of these electronics parts come either from China or the US (and that's also probably from China). I considered going directly to AliExpress for maximum cost savings, but I find the site much harder to navigate, and the shipping would take even longer.

I ended up using Mouser. Shipping took around 4 days, but it took me 2 weeks to find time for the project again. That's the annoying thing about hardware development—when you don't have a part, you're forced to pause, and by the time the part arrives, you've switched contexts. This tweet is so accurate:

i’m just sitting here with like 6 stalled hardware projects waiting for physical components to arrive. yall live like this?

— yung perf papi (@ken_wheeler) April 8, 2025

### Credits

Thanks to Biodun for helping me understand some of my initial requirements and being a sounding board.

In defence of complexity

Shalvah — Tue, 24 Dec 2024 11:05:18 +0000

I noticed a pattern when I join a new company. I'm being onboarded, someone is explaining their systems to me, and they make a joke about how it's unnecessarily complex and probably could be simplified. We laugh, but I often think to myself, "Maybe, maybe not".

I'm a strong advocate for simplicity in software design, but in this article I want to try to explain, with some personal examples, why we should have a different attitude to complexity.

Minimum Necessary Complexity

Every engineering problem has a minimum necessary complexity associated with any solution.

Yes, sometimes, they are over-engineered, and the level of complexity is higher than it needs to be, but this is not something easily determinable. Even the simplest of solutions to any non-trivial problem will need to overcome a certain amount of complexity. As software architects, our job is to determine how much complexity is appropriate and how we wish to distribute it.

Distributing complexity

Take an example of a text-based note-taking app such as Google Keep. Compared to richer organizers such as OneNote, Keep has always been more basic, which makes it appealing for simple cases—the equivalent of a set of digital sticky notes. In the early days, Keep only supported text. If you wanted to add an image, you would have to upload it somewhere else and then insert the URL.

As the developer, you might be fine with this. It's a win-win: your software remains simple to use, while users can still put in whatever they want, as long as they can convert it to text. But supposing you want to provide a better experience for your users and so you add image support. This will undoubtedly raise the complexity of your software. Now you need to think about file storage, backwards compatibility (keeping all notes text-based internally), presentation (how should images appear), interface (UI for uploading images), limits, abuse, and so on.

Another example is Twitter's "Tweet all" feature (writing a thread at once). Users can post one tweet and then reply to it, and so on. But Twitter added the ability to post multiple tweets that would then be linked as one thread. Again, we're trading some simplicity in our system for a better UX.

Of course, the amount of complexity that will be involved in both cases will vary dependeing on factors like your tech stack and architecture, as well as your skills. But there will be some.

Taking complexity on: Simple does not mean "not complex"

I call this the act of taking on complexity. We don't have to, but we want to reduce friction for our end users and make things "simple" to them. So we step in and shift some of this complexity to our end.

So, yes, something that appears simple to the user may in fact belie a lot of complexity. Adding a new feature, supporting alternatives, adding redundancy, caching to improve performance and so on.

There are also non-product reasons to add complexity, chief among them probably being security. Rate limits, authorization, RBAC, MFA, and the like all add complexity.

Complexity happens in various forms. It could be:

combinatorial (if you have 6 possible toggles, this is actually 2⁶ possible scenarios)
infrastructure
performance
system architecture
code architecture

You can't build good (intuitive, reliable, secure, insert-adjective-here) software without these two things:

the willingness to take on complexity
the discernment to know what complexity should be taken on.

Modern civilization runs on complex systems. Water and power delivered to your house, plumbing, telecommunications, Internet, traffic lights, airports, manufacturing of cans and bottles—these are examples of complex systems we've come to take for granted, because of the simplicity of their outputs.

Complexity's problems

But yes, complexity does have problems...

Cost

For one, it's costly, especially when it's intentional and well-designed. It takes time, effort and the right skills. This is why there's another option: offload it to an external provider. This lets you achieve the final goal, such as a better UX, without the full investment. The cost is still borne by you (monetarily), but your core system can remain "simple".

Offloading complexity also works as a starting point, similar to starting with a cloud provider before moving to self-hosted. Especially when you don't have the needed skills, using an external provider gives you a chance to get progressively familiar with the challenges of such systems, until you get to a point where you can finally take it on.

Growth

A second problem is that complexity grows. Complex systems beget complex systems. Once you've accepted the complexity, problems that could be previously "simple" may now take additional work. For example, improving performance in a monolith would be fairly straightforward; with microservices, you pay the price for the extra network calls, which means taking more extreme measures like aggressive caching, switching network protocols and serialization formats, and the like.

This growth is often by accident. You could start out with an architecture that makes sense for current and future use cases, but then the product evolves in unforeseen ways, trapping you in an unplanned mess. Unfortunately, this happens; you can't always avoid this.

Accidental complexity

And this leads us to our main enemy. Accidental complexity happens when a process is complex in a way we didn't expect. This usually indicates a mismatch between our mental model of how complex the process should be versus the requirements forced on us.

A common example today is deployment of web apps, especially with tools like Kubernetes. Sometimes people adopt Kubernetes because they've heard about the reliability features, and then they get frustrated by all of the extra work that it imposes on them. The complexity may be necessary, but only because Kubernetes is engineered towards a specific class of problems. And to this user, who doesn't care about those problems, all of this feels accidental.

Avoiding accidental complexity is hard, especially when it's not created by us. Sometimes the complexity is necessary, but we're just poorly informed. The best option is probably to get properly informed, weigh it up and decide if it is worth it for us. If it is, can we convert it into intentional complexity, via proper planning, documentation, and changing our approach? If not, it probably makes sense to migrate to a different solution.

A personal story

An example of complexity that sticks with me is a project from a previous job. We had an appointment booking system in our app, which was simply an embedded Calendly widget. We did this because we didn't want to deal with the vagaries of scheduling. Calendly was good at this stuff—timezones, round-robin scheduling, managing availability, group scheduling, etc. Unfortunately, they require you to use their UI for scheduling, hence the widget.

But even with this, we still had a lot of complexity to deal with.

We had to sync appointments from Calendly into our system, but our models did not match 1:1
We had to do this sync via listening to their webhooks. Let's just say: there's a reason people are trying to build startups out of webhook processing.
Their API was limited in some annoying ways, so things that should actually be simple weren't.
Worse, we had a second external service where agents could set up appointments on their own. This provider had a different model from Calendly and an even more frustrating API.
Even worse, an appointment booked via Calendly could also show up in this other service and be managed from there. (PS: "Single source of truth" is a lie.)
The icing on the cake was that the process was driven by humans, so you couldn't rely on them not doing stupid things. They would often do random things that would cause undefined behaviour, such as changing an item mid-processing (with no change history).

Complex? As fuck.

But was this really necessary? Mostly.

Primarily because this was a process that existed before we began incorporating it into our backend, so we couldn't simply rip it up and build it from scratch in an ideal manner. Also, we didn't have the resources (people or time) to work on a whole appointment scheduling + CRM system.

As the lead developer on this system, I spent days just thinking about the complexity of it all, looking for ways we could reduce it. I rewrote a bunch of the code several times. Wrote tons of comments and documents describing how the system worked.

Managing complexity

So the panacea is not necessarily rejecting complexity outright, but knowing how to manage it. Some things I've learnt that help:

Discernment: It's super important to know what complexity is worth taking on, and where to distribute it in your stack. Some kinds of complexity, such as combinatorial, are best avoided as much as possible. Take the time to assess the situation and decide whether it's worth it.
Documentation: Be sure to document things. Describe:
- the desired results
- the path we're taking to achieve them
- why Everything from comments in code to diagrams and external docs helps here.
Standardization: It helps if you can stick to common conventions. This means you only need to document the places where you diverge.
Observability: Complex systems can and will fail in surprising ways. A system is only as good as the confidence you have in it. You must invest in your ability to understand your systems (here's my free book on Observability Basics). This means architecting the system at a low and high level in such a way that changes can be understood and problems debugged easily. This entails:
- thinking about failure modes and consequences
- making the system verbose via logs, metrics and traces
- collecting metrics to track certain paths
- building features like history and changelogs to track changes
- adding the ability to observe and test behaviour in production (eg via feature flags or parallel experiments)
- adding the ability to simulate specific scenarios This isn't easy, and also adds complexity. But they will increase your confidence in the system. For example, in my Calendly story, we often didn't even have IDs we could rely on for looking up models, but had to use heuristics like "how long ago was this scheduled?" For cases like this, I used metrics, logs and occasional spot checks to compare the results of the heuristics with what was expected.
Iteration: Don't be afraid to rewrite things. This architecture may have made sense last year, but we've grown and now it's hard to keep up. There's never a perfect setup.
"Git gud": Finally, you have to level up in your stack. Being knowledgeable in your tools will give you more options and save you from building things afresh. Additionally, not all implementations will be readable or easy to understand; building up your expertise will make them more approachable.

Conclusion

Always question complexity. But don't assume it's always unnecessary. And when you need to take it on, be equipped for how to manage it.

Reflections on Year 1 of my engineering studies

Shalvah — Fri, 18 Oct 2024 00:11:18 +0000

I've finally completed the first year (of three) of my Bachelor's degree in Engineering at IU Hochschule. Technically speaking, it's my second first year, since I quit school some years ago.

Some background:

It's a general engineering degree. There are electives for various specializations in the final year, but to the best of my knowledge, the degree you get is a B.Eng.
It's a three-year program full-time, but flexible if you want to do part-time. You can spread it out over 4 or 6 years, or even longer. It took me just under a year to complete the first full-time year, but I expect the next two years will go slower, as there's more content, and most of it is new to me.
I chose to study Engineering, not Computer Science, because I already work as a software professional. CS would fill some gaps in my knowledge, but mostly in a theoretical way. There'll be some learnings, but I don't think it's worth a multiyear program. Plus, engineering still fascinates me.
It's entirely online, including exams. This has pros and cons, some of which I'll discuss.

General thoughts

School still sucks. Like many other things in life, it's a broken system. Curriculum. Course materials. Lecturers. Exams. The system will favour some people and not others. It rewards conformity.
School is awesome. Formal education is a great concept. My entire programming journey has been informal (I'd say "self-taught", but we all learnt from the works of others). It's great to have a curriculum, that structures things and tells you where to go next. It relieves you of a huge burden.

Things I've (re-)learnt:

Focus on the journey. This came after working hard on some courses but ultimately having a less-than-stellar score on the exam. I have the privilege of not needing this to get a job. It means I can afford not to worry too much about grades. Instead, I want to optimize for learning as much as I can, even if my grades suffer. It seems obvious, but it's hard for me as a completionist and a perfectionist. This has meant:
- intentionally skipping some sections of a textbook because I saw no use in them except for passing an exam
- not being stressed out about memorising random things like dates, which have no bearing on my actual skill as an engineer
- finding extra materials and challenges that go well beyond the course content, even when they prolong the length of the course
Learning takes time. In the first several months, my priority was "progression": finishing one course ASAP and moving to the next. But I ultimately realized that learning takes time, and there's no way to short-circuit it. Sometimes, just doing nothing after studying helps you learn better than moving directly to the next thing.
Progress is not linear. Another thing that's deeply personal to me, who gets easily frustrated if there are no signs of progress. I have to accept that I will forget things, I won't understand everything, I will need to go look up old courses, I will jump back and forth, and that's okay.
Continuous tweaking. At first, I was worried I didn't know how to study since I get easily distracted. Over time I've found that I can do whatever works, rather than trying to find the optimal process. I've tried different things and made a note of what works in what situation. And I feel free to adjust it if it isn't working. Some things I've tried:
- studying seated/standing/lying down
- studying outside/in a restaurant/in a park/at a bus stop/on a train
- studying with no music/chill music/fun music (this one does NOT work)
- studying in bursts/for long periods
- studying via a printed textbook/phone/tablet/desktop
- writing notes by hand/typing
Others can help: I'm super independent and love "figuring it out", so it took effort to learn to ask for help. But it's worth it. I've gotten help and learnt useful techniques from my classmates. It's always illuminating to see how others approach things.
I can help others. In the same vein, I've tried to help my classmates when I can, explaining concepts, giving directions, or providing tips. I even became a de facto class representative when we needed to complain to the school administration.

Things I've enjoyed

Learning. Learning is so cool. Especially when it's getting to know something you didn't before that explains how some other thing you were aware of works. Human beings have made so many awesome things.
Connecting dots. Even better, I've enjoyed seeing my knowledge compound. One delightful experience for me was visiting a technology museum while studying a course on Automation Technology, which got me through a rut I was stuck in. And every time I find myself able to understand a course by building on things I learnt from another course feels like a lightbulb going off in my head. These moments make me feel more confident in the curriculum.
Inspiration and experimentation. I love how really studying a course sometimes inspires me to build or try something. For instance, my Linear Algebra course inspired me to make the vector graphing library I used in this post and this post. My IoT course showed me some of the real benefits of smart devices, and I have a bunch of ideas written down to try sometime. My Production Engineering course has me itching to try 3D printing.

Things I wish were better

Classroom environment. I still wish I could be in a classroom (I get this urge a lot when watching lecture videos from MIT's OpenCourseWare). I miss having more free time and being surrounded by smart, motivated folks with similar goals. But, truth be told: I'm romanticizing it a bit. I'd bet half of my class back in my first program didn't know what the hell they were doing. Plus, most of us didn't maximize the opportunity. We were young and foolish. Oh, and being in a classroom sounds cool now, but actually having to be in one for several hours a day is not.
Labs. The program is online-only, but obviously you can't become an engineer by reading alone. There has to be some practice. Thus far, everything we've done has been theoretical, but I think there's a provision for "simulations" in Year 2 or 3. Still, I think the course materials have done a poor job of encouraging a hands-on approach. They've seemed like "theory dumps", with the occasional picture or text case study. I would like if they included suggested practical exercises at least. I've had to give myself such exercises, but I just do it for things that pique my curiosity.
Exams. The program unfortunately has only one final exam, which can range in difficulty from "lol, that was too easy" to "what the fuck." It's an unfair system (although you can retry if you fail). I picked up a habit from a classmate of making up midterm exams for myself.
The curriculum. There's been some useless stuff, both for me personally and for the overall program. Some courses are mandatory for everyone but focus on niche stuff that people in those industries will be specially trained in. Some courses are merely "memorize-and-dump". I'm mostly lucky, though; my interests are in electronics, and I think our program is biased towards that.

Overall I'm happy. I think the next semester is going to be pretty tough. I'm quite excited about the upcoming Control Engineering course, but less enthused about the three Mechanics/Materials courses. For now, I hope to take things a bit slow by experimenting with some interesting learnings from Signals and Systems, and perhaps doing a boring nontechnical course. (Update: I decided to step back and work on some small projects for each of my courses, to help solidify my knowledge.)

Most importantly, I'm grateful. Years ago when I quit school, I worried that my mental health would prevent me from holding a full-time job. And when I started up school again, I was scared I couldn't do it. Yet here I am, job in hand, and one year of school down. Grateful.

Exploring software design problems and solutions: Transactions and side effects

Shalvah — Fri, 19 Jul 2024 07:30:32 +0000

As practice in thinking and talking about practical software design approaches (at the high and low level), I want to explore some scenarios from first principles. The goal of this exercise is to analyse problems, the solutions we apply to them, and the new problems those bring up. I encourage you to pause the article at certain points and think through it yourself. Code examples are Ruby/Rails-ish, but should be easy enough to follow.

Importantly, these thoughts are subjective, as a lot of design is. Feel free to question my reasoning, and leave a comment to let me know what you think.

We start with a simple system for a shopping website: you have an endpoint where you want to create an account and a cart for a customer at the same time.

def call
  account = Account.create!(...)
  cart = Cart.create!(account) 
  CartItems.create_many!(cart, ...) # Add items to the cart
  # Maybe also create some other things ...
end

There's a problem here, which is that if the cart item creation fails, you end up with inconsistent state. The endpoint will return an error, and the account exists, and maybe the cart does, but it's empty.

We can solve this by having the caller retry on failure. But we need to change Account.create! to Account.find_or_create! to prevent duplicates, and so on. What problems does this bring?

Creation logic gets more confusing and messy, as now another non-trivial yet orthogonal concern (idempotency) has been mixed in.
All future changes to this endpoint need to know about and carry on with the find_or_create! pattern, or risk incorrectness. This gets more difficult to enforce if we extract parts of the endpoint into other methods or classes.
If the caller does not retry, you're left with orphaned entities (such as the Account which will never be used).

We know the right answer here: use database transactions! If one step fails, the transaction is rolled back, so nothing is persisted. But before we jump to that, I want to explore why.

You don't have to use a database transaction; you could also write your custom rollback implementation.

def call
  @account = Account.create!(...)
  @cart = Cart.create!(@account)
  @cart_items = CartItems.create_many!(@cart, ...)
rescue
  @cart_items.delete_all if @cart_items.present?
  @cart.delete if @cart
  @account.delete if @account
end

I'm not judging. There may be valid reasons for this. You may want to avoid nested, sequential, or long-running transactions, or you just work in a weird place where they shun database transactions.

But let's step back. Why is this frowned upon?

Major reason: It's still susceptible to failure. If there's a bug in your code or something goes wrong in your database while you're trying to delete the orphaned entities, you're screwed.

Minor reason: It leaves traces. For instance, the deleted entities will still likely take up disk space until the disk is purged. Any autoincrementing sequences (like IDs) will also likely not reset.

This is why we in the industry agreed to rely on database transactions for this. Instead of treating the data layer as a dumb store and trying to handle everything ourselves, we rely on it to help us ensure correctness, which allows us to leverage its ACID guarantees. This means that when we open a transaction, the database can promise us that all the operations in that transaction will be considered one atomic operation. They will succeed together or fail together. Database transactions also help us mitigate concurrency problems.

We've made progress. Databases which have ACID guarantees (typically SQL databases) are really good at this. We should lean on them.

So now we have:

def call
  in_transaction do
    account = Account.create!(...)
    cart = Cart.create!(account) 
    CartItems.create_many!(cart, ...)
  end
end

But things can get murky. Suppose we want to send an email to the user to verify their email after we create the account. Let's say we do this by enqueueing an async job which is pushed to Redis or a separate database and then processed by Sidekiq.

def call
  in_transaction do
    account = Account.create!(...)
    SendVerificationEmail.perform_later(account.id)
    # or
    # SendVerificationEmail.perform_later(account.email)
    cart = Cart.create!(account) 
    CartItems.create_many!(cart, ...)
  end
end

We have a problem again. If the account creation succeeds, but the cart creation does not, the whole transaction will be rolled back. But the SendVerificationEmail job will have already been enqueued! It will either fail (no account found), or succeed (if we passed the account email directly).

This is what happens when you introduce a side effect. I see side effects as a change in a different system outside the control of the current executor (the database). Here, it is an async job, but it could also be, for instance, a synchronous API call to a payment provider to charge a user's card. How do we solve this?

Let's consider some options. The simplest fix is moving the side effect outside of the transaction.

def call
  account = in_transaction do
    account = Account.create!(...)
    cart = Cart.create!(account) 
    CartItems.create_many!(cart, ...)
    account # Return account
  end

  # If the transaction was rolled back, the method would exit.
  # This means that, at this point, account definitely exists.
  SendVerificationEmail.perform_later(account.id)
end

If you can, you should do this. But this isn't always possible. For instance, maybe the account creation logic is not directly in this method, but encapsulated in a service class somewhere in your application:

class AccountCreator
  def call
    account = Account.create!(...)
    SendVerificationEmail.perform_later(account.id)
  end
end

and then used here and other places:

def call
  in_transaction do
    account = AccountCreator.call(...)
    # Maybe you also have a CartCreator
    cart_with_items = CartCreator(account, ...)
  end
end

The AccountCreator is probably used in a few other places, like the normal sign up flow. It's not so easy to extract the email sending from the transaction without affecting all these places. And this could happen for many other service classes: they innocently initiate their own external calls without knowing we're in a transaction.

In some ways, this is a silly problem. This is not a case of race conditions, possible error scenarios or high-level design problems. This is purely because of how we organize our code. If we choose to not use service classes, and rather inline this, it takes us back to the previous version, and we can now simply move the job to the end.

But it's a real problem. Code structure matters. The design and domain model we choose at the high level influences how we structure our code, and this in turn locks us into certain paths of development.

So how is this solved in the Rails' world? To solve this, Rails has a callback called after_commit. We could use it like this:

# It only works in models, so we must put the job enqueueing in our Account model
class Account
  # This method will be called whenever anyone does Account.create!
  after_commit :send_verification_email, on: :create

  def send_verification_email
    SendVerificationEmail.perform_later(id)
  end
end

With this we can solve the problem of nested service classes: move all external system calls into after_commit in the model. But you can probably already spot some problems this raises:

Actions are now coupled to models rather than business processes, which means they are always invoked, even when not relevant. send_verification_email will always be called, even in some cases where we may not want to (for example, backfilling accounts via a one-off script). We could add filters in the callbacks, but this increases the amount of logic they hold.
Dependencies are now hidden. It's easy to add a new flow for creating accounts that calls Account.create!, without realizing that it will also send emails.
Logic is fragmented. Rather than having each service class handle a business process, some of the logic needs to be moved into the model. The code becomes harder to follow, and we end up with hidden side effects. And remember, code structure matters. We will pay the price.
Visibility (and thus debugging) becomes harder, since we're now relying on a chain of responsibility handled by the framework, not our code, thus introducing a new layer of indirection. Stack traces will now jump from your code deep into the framework, before getting back to your code. And when multiple callbacks or multiple models are involved, it's hard to be sure of the order in which they were executed. I have personally wasted several minutes of my life hunting down the source of some external change. I had to do a mix of code reading and runtime debugging, tracing it across several boundaries, and I can tell you, it sucks.

One more big problem is that after_commit can have surprising behaviour when you have nested transactions.

A way to improve on this is with the library after_commit_everywhere. This library makes it possible for you to use after_commit outside models, so now the logic can remain in the corresponding service class.

class AccountCreator
  include AfterCommitEverywhere

  def call
    account = Account.create!(...)
    after_commit { SendVerificationEmail.perform_later(account.id) }
  end
end

I much prefer this, as we can now keep related concerns together. But I still see some minor problems:

after_commit_everywhere depends on monkey-patching, which feels like a hack.
It doesn't solve the problem of indirection. It in fact makes it a bit worse, as we've now added another layer on top the framework.
Is this a leaky abstraction? I don't know. Someone could argue that it is, as it means the service class needs to know/care about the existence of a database transaction.

But overall, after_commit_everywhere seems like a good solution for most cases.

Taking an assessment of where we are:

We introduced transactions to solve the problem of incorrectness due to partial failure of a multi-step process.
We introduced after_commit to solve the problem of inconsistency due to nested service classes interacting with external systems during a transaction.
We introduced after_commit_everywhere to solve the fragmentation of regular after_commit.

Great! Let's backtrack a bit and consider some alternative approaches to after_commit.

I haven't seen this done anywhere yet, but in theory you could make each service class declare or return its side effects, and leave it to the caller to execute them. This could look something like this:

Result = Data.define(:item, :side_effects)

class AccountCreator
  def call
    account = Account.create!(...)
    Result.new(
      item: account, 
      side_effects: [
        lambda { SendVerificationEmail.perform_later(account.id) },
      ]
    )
  end
end

# ... In the original caller
def call
  @side_effects = []
  in_transaction do
    result = AccountCreator.call(...)
    @side_effects.concat(result.side_effects)
    cart_with_items = CartCreator(result.item, ...)
  end

  @side_effects.each(&:call)
end

I like this because it separates side effects clearly. But now it relies on the caller to explicitly dispatch those side effects. Our custom Result class makes it harder to miss, but it's still fairly easy to forget to write that last line. It also gets pretty unwieldy with nesting, as each nested service class will have to pass to its parent not just its result and side effects, but also those from all other service classes it invoked.

Let's consider yet another alternative: If we look at the problem some more (enqueueing external jobs), we can see that it arises due to control. The queue system which handles the SendVerificationEmail job is outside the control of our database, so these two systems cannot synchronize. By moving the job to after the transaction, we're saying, "Since this external system (queued jobs) is outside the control of our transaction, let's wait until the database relinquishes control back to us." But what if we brought the jobs under the control of our database?

If we changed our queue system to store jobs in the same database, the SendVerificationEmail job would become part of the current transaction, and so it would not be visible to our queue executor until the transaction commits. And if the transaction is rolled back, so is the job. Problem solved.

I think this is a fair solution for a small app, but suboptimal for larger ones. Database jobs are a different kind of workload from application records. Storing them in the same database means your job executions can impact your regular application (for example, queue workers constantly polling your database could affect app performance, jobs could contribute to database disk bloat).

So let's iterate on this. How about if we put the jobs under our database's control, but only temporarily? This is what the transactional outbox pattern essentially is:

Rather than enqueuing jobs to our queue system, we store them as messages in a table in our app's database. This allows our database to retain control over these messages, and roll them back if the transaction is rolled back.
We then have a separate process (for example, a cron job) which is responsible for checking for new messages regularly and enqueueing them as jobs in our queue system. The table of messages is our outbox, containing messages from our application to our queue system.

It could look like this:

# Now we write messages
class AccountCreator
  def call
    account = Account.create!(...)
    # JobMessage is the model for our outbox table
    JobMessage.create!(class: SendVerificationEmail, params: [account.id])
  end
end

# A cron that runs every n seconds, and enqueues the jobs
class EnqueueJobsFromMessages
  def call
    JobMessage.each do |message|
      message.class.send(:perform_later, params)
      message.delete
    end
  end
end

Transactional outbox is awesome. We are able to synchronize our queue system to the app database while still keeping them decoupled.

Of course, it has its problems too!

We've now introduced infrastructure complexity. We've added one more component in between our application and queue workers, and every new component is a new point of failure.
We'll likely have an increase in latency (probably tiny though), since now we don't process jobs immediately, but wait for the cron job to pick them up. Our throughput may also decrease since batch jobs like this creates a temporary bottleneck until they are fanned out to the workers.
We also need to watch for concurrency issues in the cron job. With enough usage, we will quickly run into funny situations such as different instances of the cron job overlapping and then trying to enqueue the same messages. A more robust version of this cron job would look like:

class EnqueueJobsFromMessages
  def call
    in_transaction do
      # FOR UPDATE will lock each selected message so another cron job instance doesn't pick it up
      JobMessage.select("FOR UPDATE SKIP LOCKED").each do |message|
        message.class.send(:perform_later, params)
        message.delete
      end
    end
  end
end

What about side effects which aren't jobs? For example, calling an external service API. To make these work with transactional outbox, we'll need to wrap them in jobs, which also means we can no longer process them synchronously. At that point, I'd say we need to rethink our paradigm and consider whether we needed these to be synchronous in the first place. It makes me wonder about durable workflows, something I hope to explore in the future.

Final thoughts

And now I'll call it a day. We could still expand on these with more solutions and problems, both common and novel. For me, this exercise in deconstructing the patterns we use demonstrated again that we make solutions to common problems which introduce problems themselves. They don't exist in isolation, and for every solution we pick we must pay a price. Sometimes they increase complexity, sometimes they require a change of paradigm, sometimes they reduce visibility, but sometimes they're good enough.

I don't think this is a surprise, as we all know there are tradeoffs. Still, there are patterns that we've come to universally accept/reject, and it's always interesting to dig into the why, so we can more confidently pick one over the other. Adios!

(PS: This article was inspired by poring over this.)

Lessons for me on leadership, from a film about apes

Shalvah — Sun, 21 Apr 2024 19:25:13 +0000

Watching Dawn of the Planet of the Apes gave me some thoughts about dictatorship, and more broadly, leadership. Mostly regarding at work, but also in other areas of my life. These are things that are also present in the real world, but somehow, it took a movie on apes for me to grasp them.

Dictators (both Caesar and Koba) take power. They don't wait around for a consensus that they're the leader. They take decisions, and you must accept it (or take a chance and rebel). This—taking one-sided decisions—is something I suck at. I always try to be "reasonable" and listen to others' opinions. This is nice (it makes me a great colleague) but makes me poor in situations where consensus is unclear, or where strong leadership is needed.

This assumption of power is connected to confidence. For it to work, the dictator must not entertain doubt within themselves. They must believe completely in the legitimacy of their rule. For Caesar and Koba to be effective, they couldn't afford to have moments of repentance. No impostor syndrome. No wondering, "Who even gave me the right to be in charge? What if these people realize I'm no more special than they are?" Nope, you move forward. Even when you seize power via a coup, you must not doubt that you're right.

Dictators create their own authority. Sometimes through fear, sometimes through respect.

I think that these things are why dictatorship works, and also why it doesn't work. A dictator is always taking a gamble. Sure, they don't depend on consensus, but that also means consensus could someday overthrow them. Or another dictator could rise against them.

Translating this to my work (and life): I'm often reluctant to take power. I'm fine with being appointed or elected to lead, but I rarely decide to be in charge of others. I'd rather do shit myself than ask people over whom I have no explicit authority (eg superiors, peers, even juniors that don't explicitly report to me) to do it. But that's because I play it safe. I don't like taking a gamble, taking the risk of people refusing to cooperate or my leadership being questioned or my decisions backfiring. This is something I should improve on.

My ability to listen to others' opinions is gold, and I should not lose it. But at the same time, I should be ready to matter-of-factly take decisions when needed, and not always shy away.

Postscript: Interestingly, the feedback I got in my last peer review kinda amounted to this. Positive feedback talking about how I'm a great engineer and teammate, and critical feedback saying I could improve in confidence. It makes sense now.

Learn SVG by drawing an arrow

Shalvah — Thu, 11 Jan 2024 17:45:41 +0000

How SVG works

The common way of representing digital images is via pixels. Take an image and break it down into small squares ("pixels"), where each square has a single colour. If you have many tiny squares, the colours blend in, and the image looks smooth to the human eye. If you don't have enough squares, the image looks rough and jagged, and you can see the edges of the individual squares. This is the difference between high-resolution screens/images (more pixels) and low-resolution ones (fewer pixels).

But there's also vector graphics (SVG = Scalable Vector Graphics). They don't use pixels, but instead work by drawing the image on the display, using geometry (maths). For example, an arrow in PNG would consist of many, many squares lined up together in the shape of an arrow, while in SVG, it could consist of a line starting from some coordinates (x1, y1), then turning and turning until the shape of the arrow head is complete.

Both types of graphics have their use cases. Adobe has a good introductory explainer on raster graphics vs vector graphics.

(PS: all of the illustrations in this article are SVGs. If you try zooming into them, you'll see that they don't get pixelated.)

This article will demonstrate SVGs by constructing an arrow in HTML/JavaScript.

Drawing with geometry

SVG shapes are defined based on geometry. When you define an <svg> element, you get an invisible Cartesian plane; there are x and y axes, and all points you will draw in this SVG are specified relative to an origin. Note that by default, x increases to the right as normal, but y increases downwards, and the origin is at the top left. I've made a grid to illustrate this:

xy5010015020025030050100150200250300

For instance, to draw a circle, we specify the coordinates of its centre (cx and cy), and the value of its radius; for a line, we specify its start (x1, y1) and end (x2, y2) coordinates; for a rectangle, start coordinates (x, y), width and height:

<svg width="300" height="300" viewBox="0 0 300 300">
  <circle cx="150" cy="100" r="50" 
    style="fill: none; stroke: red; stroke-width: 2px;">
  </circle>
  <line x1="50" y1="50" x2="200" y2="200" 
    style="fill: none; stroke: black; stroke-width: 2px;">
  </line>
  <rect x="100" y="150" width="50" height="75"
    style="fill: none; stroke: green; stroke-width: 2px;">
  </rect>
</svg>

xy5010015020025030050100150200250300

(Note that the grid lines and axes are not automatically added; these are separate SVG lines I added to make things clearer.)

You may have noticed the style attributes; SVGs can be styled like regular HTML elements, with inline CSS, classes, and selectors.

We've used <circle>, <rect> and <line>, but at the basic level, you can construct any shape by describing its path (the <path> element). Here's one way to draw the same shapes as above:

<svg width="300" height="300" viewBox="0 0 300 300">
  <!-- line -->
  <path d="M 50,50 L 200,200"
    style="fill: none; stroke: black; stroke-width: 2px;">
  </path>

  <!-- rectangle -->
  <path d="M 100,150 l 50,0 l 0,75 l -50,0 Z"
    style="fill: none; stroke: green; stroke-width: 2px;">
  </path>

  <!-- circle -->
  <path d="M 100,100 A 50 50 0 0 1 200,100 A 50 50 0 0 1 100,100"
    style="fill: none; stroke: red; stroke-width: 2px;">
  </path>
</svg>

xy5010015020025030050100150200250300

The d attribute has specific syntax, which I like to think is for instructing the "artist" on how to move their pen.

For the line: M 50,50 L 200,200 says "**Move your pen to x1=50,y1=50, then draw a straight **Line until you get to x2=200,y2=200**".
For the rectangle, M 100,150 l 50,0 l 0,75 l -50,0 Z draws a line from corner to corner: "**Move your pen to x1=100,y1=150, then draw a straight **line until you get to x2=x1 + **50,y2=y1 + **0, then another **line until x3=x2 + **0,y3=y2 + **75, then another **line until x4=x3 *- 50*,y4=y3 + **0, then connect back to the start point (Z)".
For the circle, M 100,100 A 50 50 0 0 1 200,100 A 50 50 0 0 1 100,100 moves to 100,100, and then draws one arc (A) with a radius of 50 until it gets to 200,100. Then from that point, it draws another arc with the same radius to connect to the start point. (I won't explain the Arc syntax in detail because it's fairly complex.)

I like this article for a quick explainer on the syntax, and of course MDN for a full reference.

<path> is useful when you need to draw an irregular shape. It can get super complex, though; for regular images, you'll probably use some graphic design software that generates it for you. You'll likely only need to work with paths directly when animating something or making some custom graphic element, although you may want to use the Canvas API instead.

Manipulating SVGs with JavaScript

There's no special SVG JavaScript API; we must use the DOM APIs, similar to other HTML elements. An important difference is that you need to use document.createElementNS() instead of document.createElement(), otherwise the elements won't be rendered.

For instance, here's a small class for this:

class Svg {
  static NAMESPACE_URI = 'http://www.w3.org/2000/svg';

  constructor(domElementId, width, height) {
    this.$svg = document.createElementNS(Svg.NAMESPACE_URI, 'svg');
    this.$svg.setAttribute('xmlns', Svg.NAMESPACE_URI);
    this.$svg.setAttribute('width', width);
    this.$svg.setAttribute('height', height);
    this.$svg.setAttribute('viewBox', `0 0 ${width} ${height}`);
    document.querySelector(`#${domElementId}`).appendChild(this.$svg);
  }

  add(elementType, attributes = {}, styles = {}) {
    let $element = document.createElementNS(Svg.NAMESPACE_URI, elementType);
    Object.entries(attributes).forEach(([k, v]) => {
      $element.setAttribute(k, v);
    });
    Object.entries(styles).forEach(([k, v]) => {
      $element.style[k] = v;
    });
    this.$svg.appendChild($element);
    return $element;
  }
}

<div id="container"></div>

<script>
let root = new Svg(`container`, 300, 300);
root.add(`path`,
  {d: `M 50,50 L 200,200`},
  {strokeWidth: `2px`, stroke: `green`}
);
root.add(`circle`,
  {cx: `150`, cy: `100`, r: `50`},
  {strokeWidth: `2px`, stroke: `red`, fill: 'none'}
)
</script>

For a full-featured API, you can use a library such as Snap.svg or SVG.js.

Constructing an arrowhead

To add an arrowhead to this line, let's go with the <path> approach. Think about the shape of an arrowhead for a moment. How do you move your pen when drawing this?

Tip: There's an easier way to draw an arrowhead in SVG (skip straight to the "marker" section). But if you want to have fun with the maths or go deeper, enjoy!

To draw an arrowhead as a path, we need four things: the coordinates of the starting point P, and those of the corner points Q, R, and S. Then our SVG path will be (assuming the pen is already at P): L Qx,Qy L Rx,Ry L Sx,Sy L Px,Py. However, we only know the coordinates of the starting point P.

We also know the vertical height of the arrowhead, h, and its base, b, because an arrowhead is an isosceles triangle, and we can pick whatever height and base we want to make it look good.

So let's do a little trigonometry to find the unknowns. For each unknown point, we will calculate the "delta" (Δ) of its coordinates (the distance from P in x and y).

To find R, we can use the angle between line PR and the horizontal.

This is a right-angled triangle, so this gives us \Delta R_x = h \cos \theta and \Delta R_y = h \sin \theta. But the second diagram shows that the angle \theta is also present in the smaller P-triangle (vertically opposite angles are equal), so \tan \theta = \frac{\Delta P_y}{\Delta P_x}.

So we have

\theta = \tan^{-1} \frac{\Delta P_y}{\Delta P_x}

\Delta R_x = h \cos \theta

\Delta R_y = h \sin \theta

This gives R's coordinates in terms of P's (R_x = P_x + \Delta R_x and R_y = P_y + \Delta R_y).

To find Q, we take the angle \alpha between the arrowhead's base and the horizontal, which gives us \Delta Q_x = b \cos \alpha and \Delta Q_y = b \sin \alpha

Now, this angle adds up with \theta to make 90° (or π/2 radians), so we have that:

\Delta Q_x = b \cos (\frac{\pi}{2} - \theta)

\Delta Q_y = b \sin (\frac{\pi}{2} - \theta)

And Q is:

Q_x = P_x - \Delta Q_x

Q_y = P_y + \Delta Q_y

We subtract the x-delta, since the line \overrightarrow{\rm PQ} is going to the left (x is decreasing).

S is the reflection of Q, so it has the same deltas as Q, but in the opposite direction. For S, the y-delta is negative, since \overrightarrow{\rm PS} is going downwards (decreasing y).

S_x = P_x + \Delta Q_x

S_y = P_y - \Delta Q_y

And here's all this as a JavaScript function (with h = 10, b = 5):

function makeArrow(destination, start = { x: 0, y: 0 }) {
  let height = 10;
  let base = 5;

  let P = destination;

  // I'd normally write this out as "theta", but JavaScript supports Unicode identifiers, nice 😎
  let θ = Math.atan((destination.y - start.y)/(destination.x - start.x));
  let α = Math.PI / 2 - θ;

  let ΔQ = { x: base * Math.cos(α), y: base * Math.sin(α) }
  let Q = { x: P.x - ΔQ.x, y: P.y + ΔQ.y }
  let S = { x: P.x + ΔQ.x, y: P.y - ΔQ.y }

  let ΔR = { x: height * Math.cos(θ), y: height * Math.sin(θ) }
  let R = { x: P.x + ΔR.x, y: P.y + ΔR.y }

  root.add(`path`,
    { d: `M ${start.x},${start.y} L ${P.x},${P.y} L ${Q.x},${Q.y} L ${R.x},${R.y} L ${S.x},${S.y} L ${P.x},${P.y}` },
    { strokeWidth: `2px`, stroke: `black` }
  );
}

makeArrow({x: 100, y: 200});

Here's an interactive playground. You can edit the values to see how this arrow behaves. Can you spot its limitations?

xy5010015020025030050100150200250300

Arrow end (x, y):

Arrow start (x, y):

Two things you may have noticed:

The arrowhead doesn't handle rotation correctly. For instance, makeArrow({ x: 30, y: 0 }, {x: 100, y: 200}) produces an inverted arrowhead.
A minor annoyance is that the arrowhead actually points beyond the destination. For instance an arrow to (100, 200) will have its arrowhead a few units past (100,200).

We can adapt the calculations to account for these scenarios, but that's enough maths for today. Instead, we'll switch to using the <marker> element, which can automatically handle these issues for us.

Using `<marker>`

The good news is that SVG already has an element for this. The <marker> element is meant for placing "markers" on a shape. For instance:

arrowheads (duh) or some other shape you want to put at the end of a line
path breakpoints
resize handles on an element
drag-and-drop handles

The marker approach builds on the "manual" maths we've already done, but fixes these two limitations:

By setting orient="auto-start-reverse" on the marker, the arrowhead will be rotated appropriately to match the line's placement.
By setting marker-end={url-to-the-marker} on the line, the arrowhead will end exactly at our destination.

The marker approach comprises a few things: first, a <marker> element which has an id. We set orient="auto-start-reverse", and set the markerHeight and markerWidth to our triangle height (10) and 2 * base (5), respectively.

<marker
  markerWidth="10" markerHeight="10" orient="auto-start-reverse" 
  id="arrowhead" style="stroke-width: 2px; stroke: black;">

</marker>

Within the <marker> element, we add a nested <path> describing the arrowhead (only the triangle, not the rest of the arrow). We don't have to use <path>, by the way; we could also use three <line>s.

<marker ...>
   <path d="...something..."></path>
</marker>

How do we determine d? Well, we can use the same values we calculated earlier, but we don't need to. One benefit of <marker> is that its location on the grid doesn't matter. It can even be in another file; it will be rotated and rendered properly for any element it's attached to. This means our path does not need to depend on the actual coordinates of the line (P, Q, R, S). Instead, we can pretend P is at (0, 0) and we're facing upwards. This allows us to directly use the base and height as our deltas, giving us:

Q_x = P_x - b = -b

Q_y = P_y + 0 = 0

R_x = P_x + 0 = 0

R_y = P_y + h = h

S_x = P_x + b = b

S_y = P_y - 0 = 0

And so we have the arrowhead path, which will remain the same regardless of the line:

<marker ...>
   <path d="M 0,0 L -5,0 L 0,10 L 5,0 Z"></path>
</marker>

Finally, a <line> for the rest of the arrow as usual, setting its marker-end to the id of the marker:

<line 
  x1="0" y1="0" x2="100" y2="200"
  marker-end="url(#arrowhead)" 
  style="stroke-width: 2px; stroke: black;">
</line>

Markers are reusable elements; we can draw many different lines which have the same marker, and it will be rendered and appropriately rotated for each. You can also set marker-start instead (or both), if you wish to have the arrow on the other end.

Porting this to JavaScript:


let height = 10;
let base = 5;

let $marker = root.add(`marker`,
  {
    markerWidth: base * 2,
    markerHeight: height,
    orient: `auto-start-reverse`,
    id: 'arrowhead'
  },
  {strokeWidth: `2px`, stroke: `black`}
);

let $path = document.createElementNS(Svg.NAMESPACE_URI, 'path');
$path.setAttribute(`d`, `M 0,0 L -${base},0 L 0,${height} L ${base},0 Z`)
$marker.appendChild($path);

function makeArrowWithMarker(destination, start = {x: 0, y: 0}) {
  root.add(`line`,
    {
      x1: start.x,
      y1: start.y,
      x2: destination.x,
      y2: destination.y,
      'marker-end': `url(#${$marker.id})`,
    },
    { strokeWidth: `2px`, stroke: `black` }
  );
}

makeArrowWithMarker({x: 100, y: 200});

And finally, here it is in action:

xy5010015020025030050100150200250300

Arrow end (x, y):

Arrow start (x, y):

Building a PHP client for Faktory, Part 6: Higher-level usage

Shalvah — Sat, 25 Nov 2023 23:43:51 +0000

We have the low-level tools for connecting to the Faktory server, executing commands, fetching data, logging, and so on. Now, I'll work on the high-level API this Faktory library will expose. I plan to make it flexible enough for a user to use directly, or maybe plug it into their framework.

We need to provide a producer API (enqueuing jobs) and a consumer API (fetching and executing jobs).

Producer API

For the producer, we can expose a base Job class and a Dispatcher which enqueues jobs.

My proposed usage looks like this:

// Create a dispatcher
$dispatcher = Dispatcher::make(
    logLevel: \Monolog\Level::Debug,
    hostname: 'tcp://localhost',
);
// Dispatch a job now
$dispatcher->dispatch(SendWelcomeEmail::class, [$userId]);
// Schedule a job for later
$dispatcher->dispatch(SendWelcomeEmail::class, [$userId], delaySeconds: 60);
// Enqueue multiple instances in bulk
$dispatcher->dispatchMany(
  SendWelcomeEmail::class, [[$user1, $user2]], 
  delaySeconds: 60
);

I believe in convenience, so I'll also provide a global dispatcher:

Dispatcher::configure(
    logLevel: \Monolog\Level::Debug,
    hostname: 'tcp://dreamatorium',
);
SendWelcomeEmail::dispatch($userId);
SendWelcomeEmail::dispatchIn(seconds: 60, $userId);
SendWelcomeEmail::dispatchMany(
  inSeconds: 60, [[$user1], [$user2]]
);

Okay, implementing. The job class allows a user to specify options for the the job payload sent to Faktory:

abstract class Job
{
  public static ?string $queue = null;

  public static int $retry = 25;

  public static int $reserveFor = 1800;

  public static array $custom = []; // From the Faktory spec; allows users to add custom data

  public static function dispatch(...$args)
  {
    Dispatcher::instance()->dispatch(static::class, $args);
  }

  public static function dispatchIn(?int $seconds, array $args = [])
  {
    Dispatcher::instance()->dispatch(static::class, $args, delaySeconds: $seconds);
  }

  public static function dispatchMany(array $args, ?int $inSeconds = null)
  {
    Dispatcher::instance()->dispatchMany(static::class, $args, delaySeconds: $inSeconds);
  }
}

The Dispatcher creates and passes configuration to the Faktory client, and transforms job operations (enqueueing/scheduling) to the appropriate Faktory operation:

class Dispatcher
{
  public function dispatch(string $jobClass, array $args = [], int $delaySeconds = null)
  {
    $jobPayload = static::toJobPayload($jobClass, $args, $delaySeconds);
    $this->client->push($jobPayload);
  }

  public function dispatchMany(string $jobClass, array $argumentsListing, int $delaySeconds = null)
  {
    $basicPayload = static::toJobPayload($jobClass, [], $delaySeconds);

    $jobPayloads = [];
    foreach ($argumentsListing as $index => $arguments) {
      $jobPayloads[] = array_merge($basicPayload, [
        "jid" => "{$basicPayload['jid']}_{$index}",
        "args" => $arguments,
      ]);
    }

    return $this->client->pushBulk($jobPayloads);
  }

  protected static function toJobPayload(string $jobClass, array $args, int $delaySeconds = null)
  {
    return PayloadBuilder::build(
      jobType: $jobClass,
      args: $args,
      queue: $jobClass::$queue,
      retry: $jobClass::$retry,
      reserveFor: $jobClass::$reserveFor,
      delaySeconds: $delaySeconds
    );
  }
}

In the above snippet, I've omitted some code from the dispatcher, mainly boilerplate like constructors and managing the underlying client. You can view the full code on GitHub.

The PayloadBuilder class is a helper class to generate the needed payload shape for the Faktory server:

class PayloadBuilder
{
  public static function build(
    string $jobType,
    array $args,
    ?string $queue,
    ?int $retry,
    ?int $reserveFor,
    ?int $delaySeconds,
  ): array
  {
    $payload = [
      'jid' => 'job_' . bin2hex(random_bytes(12)),
      'jobtype' => $jobType,
      'args' => $args,
    ];

    if ($queue) {
      $payload['queue'] = $queue;
    }
    if (is_null($retry)) { // 0 is a possible value, meaning no retries
      $payload['retry'] = $retry;
    }
    if ($reserveFor) {
      $payload['reserve_for'] = $reserveFor;
    }

    if ($delaySeconds) {
      $executionTime = (new \DateTimeImmutable('@'.(time() + $delaySeconds)));
      $payload['at'] = $executionTime->format(\DateTimeInterface::ATOM);
    }

    return $payload;
  }
}

That's it for the dispatcher. Next up: retrieving and executing jobs. For this, I'm creating an Executor class. It's quite similar to the Dispatcher in terms of managing the client and having a global instance, so I'll omit those parts. Here's the key logic::

class Executor
{
  public function start(array $queues = []): void
  {
    $this->client->connect();

    $queuesListening = empty($queues)
      ? "all queues" : ("queues" . implode(",", $queues));
    $this->logger->info(sprintf("Listening on %s", $queuesListening));

    while (true) {
      $jobPayload = $this->getNextJob($queues);
      $this->processAndReport($jobPayload);
    }
  }

  /**
   * Process a retrieved job, and ACK or FAIL it to Faktory.
   */
  public function processAndReport(array $jobPayload): bool
  {
    try {
      $this->process($jobPayload);
      $this->reportSuccess($jobPayload);
      return true;
    } catch (Throwable $e) {
      $this->reportFailure($jobPayload, $e);
      return false;
    }
  }

  /**
   * Process a retrieved job. This merely instantiates the job and executes it.
   * Any exceptions thrown by the job are not handled.
   */
  public function process(array $jobPayload): void
  {
    $jobInstance = $this->instantiateJob($jobPayload);
    $jobInstance->process(...$jobPayload['args']);
  }

  /**
   * Manually fetch the next job to be executed from the specified queues.
   * Faktory will block for a few seconds if no job available, then return null.
   * The $retryUntilAvailable parameter forces the executor to try again when this happens.
   */
  public function getNextJob(array $queues = ["default"], bool $retryUntilAvailable = true): array|null
  {
    while (true) {
      $job = $this->client->fetch(...$queues);
      if ($job || !$retryUntilAvailable) return $job;
    }
  }

  protected function reportSuccess(array $jobPayload)
  {
    $this->logger->info(sprintf("Processed job result=success class=%s id=%s", $jobPayload['jobtype'], $jobPayload['jid']));
    $this->client->ack(["jid" => $originalJobPayload["jid"]]);
  }

  protected function reportFailure(array $jobPayload, Throwable $exception)
  {
    $this->logger->info(sprintf("Processed job result=failure class=%s id=%s", $jobPayload['jobtype'], $jobPayload['jid']));
    $this->client->fail([
      "jid" => $originalJobPayload["jid"],
      "errtype" => $exception::class,
      "message" => $exception->getMessage(),
      "backtrace" => explode("\n", $exception->getTraceAsString()),
    ]));
  }

  protected function instantiateJob(array $jobPayload): Job
  {
    $class = $jobPayload['jobtype'];
    return new $class;
  }
}

I'll bring this article to a close here, because I got interrupted multiple times and ended up spending several months on it. 😅 I'm off to tackle the next stages; here's the code thus far.

Thus far, we have a basic working API. I'm not yet sure how it would fit into a production app, but it's good for a start. In the next phase, I'd like to address some limitations of the current setup.

Exploring concurrent rate limiters, mutexes, semaphores

Shalvah — Mon, 11 Sep 2023 22:39:11 +0000

This is a dump of my learnings and experiments while going down a little rabbit hole.

Concurrent rate limiters

I was studying Sidekiq's page on rate limiters. The first type of rate limiting mentioned is the concurrent limiter: only n tasks are allowed to run at any point in time. Note that this is independent of time units (e.g. per second), or how long they take to run. The only limitation is the number of concurrent tasks/requests.

So I asked myself, how would I implement a concurrent rate limiter? I'm fairly familiar with locking (via Redis and the database, for instance), so that was what came to mind, but in its usual form, that only works as a mutex (number of allowed tasks, n = 1). I wasn't sure about how to implement that when n > 1. Decided to dig into it from first principles.

Concurrency control scenarios

In this case, that meant stepping back to think about concurrency control in general, and the scenarios I know of.

The first scenario is process-local: you have multiple threads within a process, and you want to ensure only n threads can access a resource at once. I already knew how to do this:

when n = 1, use a mutex. Only the thread with the lock on the mutex can execute; others have to wait.
when n > 1, use a semaphore. I wasn't too familiar with semaphores, so I decided to brush up.

Semaphores are similar to mutexes, but they are less about guaranteeing exclusive access and more about keeping track of who has access. A semaphore starts out with a fixed number of "permits". A thread can request a permit (similar to acquiring a lock), and that reduces the number of available permits. When all permits are in use, any requesting threads will have to wait until one is released. In this sense, a semaphore is kinda like a bouncer at a club—it regulates the number of people who can get in.

Semaphores via mutexes

There are many semaphore implementations available for Ruby. I decided to implement one myself. The key thing is that the semaphore governs access to a resource (the number of permits), so we need a way to ensure the semaphore can do this job safely. I used Ruby's native Mutex to achieve this:

class Semaphore
  def initialize(max_permits)
    @max_permits = max_permits
    @used_permits = 0
    @mutex = Mutex.new
  end

  def permit(&block)
    acquire
    block.call
    release
  end

  private

  def acquire
    acquired = false
    until acquired
      @mutex.synchronize do
        acquired = permit_acquired?
      end

      sleep 0.05 unless acquired
    end
  end

  def permit_acquired?
    if @used_permits < @max_permits
      @used_permits += 1
      return true
    end

    false
  end

  def release
    @mutex.synchronize do
      @used_permits -= 1 if @used_permits > 0
    end

    puts "#{@max_permits - @used_permits} permit(s) available"
  end
end

Usage:

semaphore = Semaphore.new(2)

t1 = Thread.new do
  semaphore.permit do
    puts 'Thread 1 acquired semaphore'
    sleep rand(1..3)
    p "Thread 1 releasing"
  end
end

t2 = Thread.new do
  semaphore.permit do
    puts 'Thread 2 acquired semaphore'
    sleep rand(1..3)
    p "Thread 2 releasing"
  end
end

t3 = Thread.new do
  semaphore.permit do
    puts 'Thread 3 acquired semaphore'
    sleep rand(1..3)
    p "Thread 3 releasing"
  end
end

[t1, t2, t3].map(&:join)

Sample output:

Thread 1 acquired semaphore
Thread 2 acquired semaphore
"Thread 2 releasing"
1 permit(s) available
Thread 3 acquired semaphore
"Thread 1 releasing"
1 permit(s) available
"Thread 3 releasing"
2 permit(s) available

This approach checks whether there are any available permits and returns one if so. Otherwise, it will sleep for 0.05 seconds and check again. The mutex guarantees that we can safely increment or decrement the number of permits without race conditions. This is a basic implementation, btw; one important thing missing is wait timeouts—we shouldn't have to wait forever.

Also note that there is no expiry on a permit—a client could get a permit and refuse to release it! Apparently, that's by design; semaphores don't control what you do with the permit. The onus is on you to be responsible with it.

Fair semaphores

This approach suffers from unfairness. Suppose thread A has been waiting for a permit to become available, and finally, another thread releases one. If at that moment, thread A is still sleeping, and a new thread (thread B) is launched, B might acquire the permit instead of A. In essence, an unlucky thread could wait for a very long time (or forever!) while newer threads get a permit. This is like if the bouncer selected people at random, instead of who's been waiting the longest.

Also, the constant sleeping and waking up (polling) is suboptimal. We're giving the Ruby interpreter and OS more work to do (constantly scheduling and waking up the thread) when there might not be actually any permits available, and we're potentially taking time away from threads which have actual work to do.

So I decided to try a different approach, using Ruby's Thread::Queue. Thread::Queue is a queue data structure designed for concurrent use; requesting an item (via #pop) will either return an item if one is available, or block until another thread adds one to the queue (via #push). So we model the list of permits as a queue (you can put anything you want in the queue, as long as it's the same number as the permits). Acquiring = popping, releasing = pushing.

class Semaphore
  def initialize(max_permits)
    @permits = Thread::Queue.new([1] * max_permits)
  end

  def permit(&block)
    acquire
    block.call
    release
  end

  private

  def acquire
    @permits.pop
  end

  def release
    @permits.push(1)
    puts "#{@permits.size} permit(s) available"
  end
end

This works too (and the code is also much shorter). I'm not a 100% certain the waiting threads are served in order of who asked first (the docs don't say exactly that), but I think that's the case.

Condition variables

After this, I took a look at the semaphore class in the popular library, concurrent-ruby to see how they implement it, and I learnt about something new: condition variables. And Ruby comes with this included!

The name sounds super technical, but it's quite approachable in reality: a condition variable lets you tell other threads waiting on a resource that it's now available. It's meant to be used with a mutex. Instead of having the thread constantly poll, as in my initial implementation, the thread sleeps forever (or with a timeout), and gets woken up by the condition variable when a new permit is available. Here's the new implementation:

class Semaphore
  def initialize(max_permits)
    @max_permits = max_permits
    @used_permits = 0
    @mutex = Mutex.new
    @condition_variable = ConditionVariable.new
  end

  def permit(&block)
    acquire
    block.call
    release
  end

  def acquire
    @mutex.synchronize do
      until permit_acquired?
        @condition_variable.wait(@mutex)
      end
    end
  end

  def permit_acquired?
    if @used_permits < @max_permits
      @used_permits += 1
      return true
    end

    false
  end

  def release
    @mutex.synchronize do
      @used_permits -= 1 if @used_permits > 0
    end
    @condition_variable.signal

    puts "#{@max_permits - @used_permits} permit(s) available"
  end
end

Rather than polling (sleep-wake-check-sleep), we just sleep (@condition_variable.wait), and then when another thread is done, they call @condition_variable.signal, which will wake up the first waiting thread (so it's fair, yay). It reminds me a bit of events in JavaScript.

Distributed concurrency control

Okay, good diversion; now, back to concurrent scenarios. We've looked at process-local scenarios. The second scenario is system-local, but separate processes. This is, for example, when you have multiple web server processes running on the same machine, and you want to control access to a shared resource.

Processes don't share memory, so our mutex/semaphore can't live inside any single process. We have to use an external datastore, such as a cache (eg Redis), or a database (PostgreSQL). You could even use a file or what-have-you.

A third scenario is in a distributed system: the processes are on separate machines. This is an extension of the above case, so the same approaches as above apply, but obviously in a distributed way (the datastore has to live on an external location all the processes can access).

Okay, so how could we implement mutexes (n = 1) here?

Aside: things to keep in mind

When dealing with locks/mutexes, you want to avoid starvation (ie, an unruly/crashed client holding on to the lock and blocking everyone else). A common (but imperfect) way to avoid this is to set a lock expiry (how long a given client can hold a lock). This way, if a client is unable to release the lock (for instance, if they crashed), it will automatically be released after a while.

You also want to limit contention, typically by setting a wait timeout (how long a client should wait for when another client is using the lock). If you don't set a wait timeout, you could end up with other processes hanging forever because a lock is in use. Sometimes that might be desired, but more likely you probably want to quit and try again later. Either way, you should decide on your wait timeout policy for your clients.

Mutex with Redis

A common pattern is to set a key in Redis, something like SET some-lock-key some-value NX EX expiry-in-seconds.

NX will set the key only if it doesn't already exists. If the key already exists, it means another process has the lock, and you need to retry.
EX (or PX) sets a time when the lock expires, so a crashed process doesn't keep on hanging on to the lock
To release the lock, you can use DEL to delete the key. (But you shouldn't! See below.)

This works, but has a few flaws:

Processes need to poll Redis until they get the lock, or give up. We already saw why polling is suboptimal.
Lock expiry is, at best, a guess. You're hoping all your clients will finish in that time. But if one process somehow doesn't finish in time, the lock would erroneously expire, and you could end up with two concurrent processes (!)
If the above happens, and the first process tries to release the lock with DEL, it would delete the lock now held by the second process. The Redis docs have details on how to correctly delete a lock.
In a distributed Redis cluster, a lock could be acquired multiple times. Distributed locking is probably a topic for another day, though. Patterns like Redlock are suggested (but also criticized).

Mutex with PostgreSQL

For SQL, I like Postgres' advisory locks. Running SELECT pg_advisory_lock(123) will give this client a lock called 123, and all other clients who run that same statement will have to wait until the first client releases the lock.

With Postgres' advisory locks, you don't need a lock expiry, since the lock is bound to the session or transaction. If the client crashes, PG will release the lock. Wait timeouts aren't directly supported, but you can achieve this by using the lock_timeout setting (combined with transaction-level locks if you don't want global wait timeouts):

BEGIN; -- Start transaction
SET LOCAL lock_timeout = '10s'; -- Error if no lock acquired after 10s
SELECT pg_advisory_xact_lock(123); -- Get a transaction-level lock

-- Execute the rest of your code

COMMIT; -- End transaction; lock is released automatically

MySQL also has advisory locks (although they are not transaction-bound) and wait timeouts:

SELECT GET_LOCK('my_lock', 10); -- Error if no lock acquired after 10s
SELECT RELEASE_LOCK('my_lock');

I love Redis, but I think I prefer SQL databases for mutexes. Since the Redis API does not expose any concept of a lock, we try to emulate it with the SET NX pattern or more complicated algorithms, which is why we have to do the "lock expiry" dance. PostgreSQL, on the other hand, has locks as a first-class function, which means it can provide better guarantees, such as the fact that locks will always be released when the session ends.

Semaphore with Redis

How about semaphores (n > 1)? At first I was thinking of something using Redis transactions (MULTI) and WATCH, something like this (not valid code):

WATCH semaphore-permits-used
max = GET semaphore-permits-max
used = GET semaphore-permits-used
return unless used < max # No permits available

MULTI
INCR semaphore-permits-used
EXEC

WATCH will fail the transaction if the semaphore-permits-used is modified by another client, so this serves as a mutex for the permits. But this implementation seems pretty complex to me; it involves us switching between our app code and Redis multiple times (or making this a Redis script). i haven't had the chance to try it yet, though.

Turns out, you can implement a semaphore in Redis quite simply with blocking list operations (akin to what we did with Thread::Queue in Ruby):

First, put n items in a list in Redis (say semaphore-available-permits)
To acquire a permit, call BLPOP semaphore-available-permits. This will pop one item from the list. If there's none available, it will block until some other client pushes one. You can also specify a wait timeout: BLPOP semaphore-available-permits <wait-timeout>.
To release a permit, call RPUSH semaphore-available-permits. If there are clients waiting for a permit, the longest waiting client will automatically get the newly released permit (so it's a fair semaphore).

It's still a bit ugly, because that first step is crucial (otherwise clients would wait forever). The best approach there is to either have each client check if the semaphore has been initialized (e.g. by checking if a certain key exists), and initialize it themselves if not; alternatively, you could have an explicit initialization step that creates the permits when your app starts up.

Semaphore with Postgres

Unfortunately, advisory locks don't help here. We need a good ol' database table to keep track of our semaphores.

CREATE TABLE global_semaphores (
  semaphore_name TEXT,
  max_permits INTEGER,
  used_permits INTEGER DEFAULT 0,
)

To kick things off, we create the semaphore with capacity n = 4:

INSERT INTO global_semaphores ("my_semaphore", 4)

Checking out and releasing a permit is straightforward: update the used_permits count. But, to ensure exclusive access, we must use a transaction:

UPDATE global_semaphores 
  SET used_permits = used_permits + 1
  WHERE semaphore_name = "my_semaphore"
  AND used_permits < max_permits;

The UPDATE statement will automatically lock the matching row (our semaphore) until it finishes executing, so no transaction needed (unless we want to specify a local timeout). All we need to do is check if the row was updated; if so, we have our permit.

To release the permit, it's the reverse:

UPDATE global_semaphores 
  SET used_permits = used_permits - 1
  WHERE semaphore_name = "my_semaphore";

One downside here is that there's no easy way to block while waiting for a new permit to be available. We'll have to rely on polling.

I find it quite interesting the differences in approach between Redis and PG here: an explicit list of permit items (Redis) vs a counter (Postgres). I think you could use either approach in both, but it would be more complicated. (I actually had an initial implementation in Postgres that used multiple rows and transactional isolation, but it was def more complex.) Postgres' transactional guarantees make it easier to work with a single counter (= a single row), while Redis' list data structure and blocking options make that approach straightforward.

Putting it all together: concurrent rate limiter

We're almost there! Actually, we're there. A concurrent rate limiter is essentially a semaphore. The max number of permits = the max number of concurrent tasks.

However, the Sidekiq version also includes lock expiry, so I spent some time thinking about it. My conclusion: there's no "nice" way to do it. The permit expiry approach I could think of (or find in the wild) was:

When a client gets a permit successfully, it must record that permit and its acquisition time (in a hash in Redis, or a row in a table in Postgres)
When the client releases the permit, it can then delete the hash entry or row
You either have an external process that regularly checks for permits that are too old and force-releases them, or have a newly-connecting client do that check themselves.

I think that's it for that exploration. It was quite interesting racking my brain about semaphores and guarantees. My current conclusions are that I'd prefer to use Postgres for mutexes and Redis for semaphores. It was also a revelation that semaphores don't provide any guarantee of expiry, so you must program carefully around that.

High and low cardinality

Shalvah — Sun, 13 Aug 2023 12:36:14 +0000

Let's talk about cardinality. Can we talk about cardinality please, Mac? I've been dying to talk about cardinality with you all day.

In maths, cardinality is the number of elements in a set. A = {2, 4, 5, 8} -> cardinality of A is 4. It's quite similar in software engineering—cardinality is a rough idea of how many distinct values are in a set.

For example, timestamp fields (such as created_at) are often high-cardinality, because they will likely have many different values. By contrast, a field like day_of_week is low-cardinality, because it can only have 7 different values. An enum field like status is also probably low-cardinality (active, inactive, and maybe a few others). Boolean fields/flags have very low cardinality—just two values. User-specific data, like user IDs or names, are high-cardinality.

Note: there's another common usage of cardinality in software, which refers to relationships in database entities, but that's not what I'm talking about here.

Why does cardinality matter? It has effects on data storage and access patterns.

Data access

A good example of where cardinality affects access is relational database indexes.

You might have heard that you should add an index for any fields you want to query on. So, supposing you have a users table, you might add an index on created_at and status, so your queries can be faster.

Here's the thing—for most simple queries, the database only uses one index[note 1]. So in a query like this:

SELECT * FROM users 
WHERE status = 'active' 
AND created_at >= '2023-08-01 00:00:00'

the database has to decide which index it will use, and it's fairly likely it will pick the one on created_at.

Why's that? Because the database engine maintains statistics about the tables, and it may have noticed that the status column usually only has a few different values, while created_at has a wide variety. This means that if it uses the index on created_at, it could filter out a good portion of the rows in the table, and then scan each row to check if it matches the status.

Imagine we have 25,000 users. Most (say 17,000) are active. If the database used the status index, it would have 17,000 rows that it needs to then loop through and check the creation timestamp. But if it used the created_at index and a time range, it could end up with maybe 2,000 users created in the last month. That's a lot fewer rows to check.

Of course, this doesn't always work as I've described. The engine doesn't know your application like you do. A different query, a different time range, different data patterns (such as most users from a certain period being inactive), or outdated intenral engine statistics might make the engine's chosen index perform worse, or might make it to pick an entirely different algorithm.

An important point: cardinality is relative! Or more accurately, selectivity is. Selectivity is cardinality (distinct values in a set) relative to the overall size of the set. The status column might have only 6 distinct values, but if there are only 12 rows, that's a good selectivity ratio, as there's a good chance we could filter out half of the rows with one query.

Database indexing is an involved topic, and I won't delve further here. I'm a big fan of this article, which talks more about cardinality, selectivity, and many different reasons your index might not be used.

Data storage

It's related to access, but more specifically: if we know that a set has a small range of values, we can store it more efficiently (which could mean we can also query it more efficiently). This is what ClickHouse's LowCardinality column type does.

For instance, suppose we have a status column as a String. If we insert three rows with status of "active", we'll be storing a 6-byte string 3 times. But if status is designated as LowCardinality, the database can avoid this by storing active once, and assigning it a small integer ID (like what we did in byte packing). Then every new row gets that ID stored, meaning we'd only store 1 byte per row.

In a database like ClickHouse, used for storing huge amounts of data, this is quite valuable. Here's a good demonstration of the space savings, and here's an example where it improves query speed.

As an end user, it's also helpful to know when a tool you use expects a value to be low- or high-cardinality, even when they don't use those terms.

An example: in Datadog, metrics can have tags; for a metric such as "number of payments processed", you could attach a tag like status, with values such as status:successful and status:failed. However, you're charged for every metric + tag combination, so you pay for every new value of status that you send. If you introduce a status:pending, you'll pay (literally) for that!

This means your tags should not have unbounded cardinality. A tag like payment_amount is asking for trouble. Instead, if you really need to capture values like that, you could put the payment amount in buckets (0 - 100, 100 - 500, etc).

Notes

1. In theory, it could use more than one index (and it does sometimes), but because of the way indices work, that would be more complicated and potentially slower. But for more complex queries, like JOINs, multiple indexes help.

DEV Community: Shalvah

Dealing with domain modelling mismatches on external services

Naming

Spelling

Structure

Validations

Flows

Restricted and allowed behaviours

Identifiers

Scheduling

Dealing with domain model mismatches

All decisions are wrong, but some are better

All decisions are wrong

But some are better

It's not just about what you do, it's about how you do it

The human cost

Making non-atomic actions atomic using intents

Context

Constraints

Solution

Failure modes

What happens if the user changes their mind and exits without completing the new booking?

What happens if the user enters this flow (clicks "Reschedule") multiple times?

What about transactional guarantees (consistency)? Since this is not an atomic operation, what happens if one action fails (cancelling the old one or booking the new one)?

Dealing with limitations

Conclusion

DIY Smart home project: Presence-activated lights

Background

Schematic

Equipment

Aside: Active vs Passive IR sensors

Getting used to the ESP32 and the sensor

Tuning the sensor

Switching to ESPHome

Automating with Home Assistant

Power issues and more tweaking

Final thoughts

Finishing

Debugging and observability

Buying parts

In defence of complexity

Minimum Necessary Complexity

Distributing complexity

Taking complexity on: Simple does not mean "not complex"

Complexity's problems

Cost

Growth

Accidental complexity

A personal story

More problems

Managing complexity

Conclusion

Reflections on Year 1 of my engineering studies

General thoughts

Things I've (re-)learnt:

Things I've enjoyed

Things I wish were better

Exploring software design problems and solutions: Transactions and side effects

Final thoughts

Lessons for me on leadership, from a film about apes

Learn SVG by drawing an arrow

How SVG works

Drawing with geometry

Manipulating SVGs with JavaScript

Constructing an arrowhead

Using <marker>

Further reading

Building a PHP client for Faktory, Part 6: Higher-level usage

Producer API

Exploring concurrent rate limiters, mutexes, semaphores

Concurrent rate limiters

Concurrency control scenarios

Semaphores via mutexes

Fair semaphores

Condition variables

Distributed concurrency control

Aside: things to keep in mind

Mutex with Redis

Mutex with PostgreSQL

Semaphore with Redis

Using `<marker>`