DEV Community: Neighbourhoodie

NH:STA S01E02 OpenPGP.js

Maddy — Wed, 01 Apr 2026 08:00:00 +0000

This post is part of a series on our work for the Sovereign Tech Agency (STA). Our first post in the series explains why and how we are contributing to various open source projects.

About the project

OpenPGP.js is a pure, Open Source OpenPGP implementation written in JavaScript. Its main use-case is enabling PGP workflows in web-based email systems, but as JavaScript is available on almost all devices these days, its utility is universal.

Our contributions

We started out by introducing a fuzz testing suite to the project. Fuzz testing is a form of unit testing, but instead of relying on manually crafted input and comparing it to the desired output, fuzz testing generates a near infinite number of permutations for input data to find rare implementation bugs. For security-related software, this is an important aspect of a complete automated testing suite.

We then focussed on making the project more approachable for new contributors by:

improving the documentation for first-time contributors
adding a high-level description of the project’s architecture
and improving the general contribution guidelines.

Finally, we started work on migrating certain core modules from JavaScript to TypeScript, to make crucial parts of the project more type-safe.

Reflections from the team

Here’s a short interview with Neighbourhoodie developer Alba Herrerías Ramírez, who runs our STF programme and worked on OpenPGP.js:

What was the most surprising thing working on this project?

Alba: I’m not sure if it’s ‘surprising’ but something I found pleasant was their user documentation, it’s great, I would like to see more projects paying this detail to docs.

What was especially challenging about this project?

OpenPGP.js have been planning to release v6 for a long time and our work got stuck in the middle (since they requested us to base our work in the v6 branch). We needed to accommodate the project’s timelines.

Conclusion

In summary, we could play to our strengths here and help a web-based project and we could build upon our work with Sequoia-PGP. There is lots to be done on the OpenPGP.js project and we hope we get another chance at helping them along.

Find out more about the work we do by visiting the Neighbourhoodie Blog.

NH:STA S01E01 Sequoia-PGP

Maddy — Wed, 25 Mar 2026 09:00:00 +0000

This post is part of a series on our work for the Sovereign Tech Agency (STA). Our first post in the series explains why and how we are contributing to various open source projects.

In our first project with the STA — and first episode of the season — we take a closer look at sequoia-git and use our frontend skills to help the project.

About the project

Sequoia-PGP is an OpenPGP (Open Pretty Good Privacy) implementation in Rust. Its focus is on safety and correctness by using a memory-safe language. PGP has been the backbone for many encryption tasks for decades and this Rust implementation takes this ecosystem into the future.

While work on the core library is progressing with sufficient speed, the lead maintainers are responsible for many ancillary tasks that are important to help the rest of the world to get on board. By relieving them of these tasks, we created space for the project to concentrate on what they do best: write security software without distractions.

Our contributions

Our first task was to review the sequoia-git subproject. Sequoia-git builds on top of git and allows you to cryptographically make sure that code changes you might incorporate into your software come from a trusted set of developers. We gave it a spin and presented the team with a comprehensive report where we believe the tool could be improved in terms of setup, first run, documentation, error handling and overall useability.

Secondly, we developed from scratch a contributing guide for Sequoia-PGP. Open Source projects thrive on the contributions they receive and constantly recruiting and onboarding new developers is a core requirement for any Open Source project maintainer. To make this easier on everybody, we ourselves became first-time contributors and wrote up everything we had to learn to get started. Now anyone who’d like to get started helping out the project can walk in our footsteps and won’t require as much direct help from the current maintainers, so they can focus on other important tasks.

Next, we provided Sequoia-PGP with a modern Frontend Design and Reusable Styling. Our prime goal here was to produce a system that was easy to maintain for many years by people who are not primarily web developers. This led us to eschew many of the modern best-practices designed for folks who do web development day in and day out, but these projects often come with a high learning curve and have to receive regular updates. By going back to web development foundations and minimal tooling, we achieved a modern, better maintainable and better looking website for Sequoia-PGP. This again removed significant time away from the core maintainers while making the project more approachable for newcomers.

In addition to the successful completion of these milestones, this was also our very first project with the Sovereign Tech Agecy and aside from working on the project itself, we also established the blueprint for all following projects. We thank the Sequoia-PGP team for their patience while we worked out the system as we went along.

Conclusion

In summary, we are very happy we managed to help Sequoia-PGP on a more sustainable path for their very important mission. We learned a lot about the OpenPGP ecosystem as a result, and as it turned out, not a moment too soon. Tune in next time when we cover our work on the OpenPGP.js project.

Find out more about the work we do by visiting the Neighbourhoodie Blog.

Neighbourhoodie and The Sovereign Tech Agency

Maddy — Wed, 18 Mar 2026 09:00:00 +0000

We’ve been doing something incredibly exciting for the last couple of years, that we’re getting around to sharing on DEV: Neighbourhoodie is an official Implementation Partner of the Sovereign Tech Agency (STA), formerly the Sovereign Tech Fund, and their Tech Resilience Program.

Hold up, what does all that mean? Let’s back up a little:

In 2021, a security issue (or vulnerability) was made public in the Log4j library. This is business as usual in the software world, security vulnerabilities are found every day, they are being reported, issues are fixed, new releases come out and then everyone can update.

That time, things went a little different: the vulnerability was disclosed without anyone having a chance to fix it, let alone update their systems to the latest version.

In addition, this vulnerability affected anyone running Log4j in a way that anyone anywhere can run unverified code on a target system. This is 10 out of 10 bad.

Two more compounding factors made this issue even worse:

It turns out, Log4j is used everywhere. Not a day goes by where anyone doing any digital work is not using a system that uses Log4j. Small companies, big companies, governments, everyone is using Log4j.
Log4j was maintained by a very small team that was working part time and unpaid.

One of many government agencies, the German Bundesamt für Sicherheit in der Informationstechnik (Federal agency for IT security) classified this as an “extremely critical threat situation”.

In response to this, the German government founded the Sovereign Tech Agency to run programs that help avoid issues like this in the future.

One of these programs is the Tech Resilience Program, formerly called the Bug Resilience Program:

The Bug Resilience Program proactively increases the resilience of open source software infrastructure and empower small and medium-sized open source projects. The goal is to lower their risk of harboring bugs and improve their capacity to respond to bugs as they are discovered. The program provides services to OSS projects, such as helping projects deal with technical debt, working on known security issues, performing code security audits to reduce high-risk vulnerabilities, as well as offering a bug & fix bounty platform to discover, responsibly report, and fix bugs.

So far so good, but where does Neighbourhoodie come into the picture?

How it works

As mentioned above, we are an Implementation Partner. That means we work directly with Open Source projects and help them be more resilient in the face of security issues. We are collaborating with projects in the open and help them address their highest need issues. These vary widely from directly working on the code to improving processes, to internal documentation that helps onboard more folks, and sometimes it means taking on a bunch of chores to free up the core maintainers to focus on high-value work.

Here’s an (abbreviated for clarity) outline of the kind of work we are doing with the projects:

Analysis
1. Organisational Preparation
2. Technical Preparation
  1. Review software dependencies
  2. Review project: Code & Tests, including CI
Improvement
1. Testing
  1. Add test coverage
  2. Introduce or expand automated testing
  3. Increase testing matrix
2. Release Engineering
  1. Stable versioning
  2. Automate releases
  3. Audit access control for release automation
3. Software Development
  1. Help fix high-impact issues
  2. Help review outstanding contributions
  3. Improve documentation for first time contributors
  4. Improve contribution guidelines more generally
  5. Improve developer experience
  6. Help recruit more contributors

While all this is very abstract, we’ll share some of the concrete work we have done with actual projects in later blog posts.

Neighbourhoodie are honoured to have already worked on diverse and essential projects, you can more about on our blog:

We’ll be publishing these stories here on DEV in the coming weeks.

How can you join?

The Sovereign Tech Agency’s Tech Resilience Program is open for your applications.

We can also help you directly

Many companies and products have crucial dependencies on small and potentially understaffed and underfunded Open Source projects. We can help identify and improve the ones that are important to your business. Our friendly sales team is happy to help. Book a call with us today!

How to Sync Anything: Building a Sync Engine from Scratch — Part 3

Maddy — Wed, 11 Mar 2026 14:08:40 +0000

Last time we learned how to efficiently decide what needs syncing. This time we will learn how to version our data.

Let’s jump right in!

In the example from part two, with stories updating after they’ve been created, and with apps being able to request a list of current stories at any time, we already saw how to deal with different versions of a story in our calculation of a delta for an app to download.

This third part explores different version schemes and discusses their respective trade-offs. Before we go into them, let’s briefly look at what kind of technique we are looking for.

Given two objects with the same ID, we need to be able to tell which one came after the other, so we know which one is the most recent one.

If we have more than two, we need to be able to put them into an ordered list, so we know which ones come after others, and so we can know which one is the most recent, or latest. And we need to be able to tell if a more recent version of an object is an ancestor of an older one.

Increasing Integers

The first and most natural idea for denoting versions of a document is increasing integers. That’s just a fancy way of saying “Version 1”, “Version 2”, “Version 3”, and so on.

The advantage here is that these version numbers are very easy to understand for humans and computers alike.

So what’s the problem?

Imagine the client device and the server updating one of our story objects independently. Our story is at “Version 1”. The server then creates “Version 2” because that’s the next in the series.

The client also creates a “Version 2”, but we have no way of knowing whether the contents of the versions are the same. When we try to synchronise the client and the server now, we run into trouble.

We need to find something better to describe our versions.

Timestamps

The next common idea for denoting versions of a document is a timestamp, a snapshot of a clock on some device, Sat Jul 23 02:16:57 2005, or 12569537329. For example, we could have story objects with a property updatedAt that we assign the timestamp of the last update to. And when we first create the object, we also record the current time.

Aside: there are many formats for timestamps with properties ranging from easy to read for humans to easy to process for computers. Whichever you choose depends on a set of tradeoffs for your application and a discussion of the merits of one format against another is outside of the scope of this article. If you just want to pick one that’s generally good, go with ISO 8601.

When we have two story objects with the same ID, we can compare their timestamps to figure out which one is the latest version of our story. As time has the convenient property of monotonically increasing, in other words: clocks only ever go forward, and this will forever be true.

Or will it? Before you go on, we’d like you to read this fantastic collection of falsehoods that programmers believe about time and the sequel. We’ll wait right here.

Hold music is playing

Say we are editing our address book on our phone. First we add a contact’s new phone number. Phone numbers are icky to type so once we are done and compare it to where they wrote it down, we see we swapped two digits.

Under the hood we recorded the current timestamp of the phone into our object versions, so we know which one was the latest. We fix the swapped digits quickly, no harm done. Or is there?

As we’ve seen in The Falsehoods, that is not always true. In fact, even companies with near infinite engineering and operations resources, like Google, can run into trouble with this.

There are a number of things that could conceivably happen, like a rogue time server telling the phone’s clock to adjust to an earlier time, or a daylight savings time switch (lucky they happen at night!); even Apple’s iPhones kept having issues with alarms set for January 1st for a while.

In our example, this means the phone number with the typo will survive as the latest edit to our object, and not the correct one. This is not what we wanted.

Whatever the exact scenario, this is plausible and has documented occurrences in small and large-scale systems: you can’t rely on timestamps to guarantee the order of two items, even if the timestamps were generated on the same device as they lead to data loss.

Ok. How can we improve on timestamps?

Before we find out, we need to introduce one more new concept: conflicts.

Embracing Conflicts

To illustrate conflicts we need to expand our example a little bit. Imagine our app is not just a consumption app for people who want to read our stories, but it is part of a content management system (CMS) where story authors can update their stories.

Our system as designed so far already works in this case; we can have the server request latest story updates from the app of all authors, and it can then send updates to any apps of our readers.

Now imagine this happening: an editor uses the CMS to edit a story by an author to fix a typo, while the author updates the same story with latest developments and we are using a timestamp to signify which version is the latest.

Even if we now have two devices with perfectly synchronised clocks (which, as we’ve learned already, we can’t guarantee we have), this shows a larger problem: what happens if two people make changes to the same object on two different devices roughly at the same time?

It doesn’t help that we know which one was updated after the other, either we pick the latest story developments and keep the typo, or vice versa. This is traditionally called a conflict.

DUN. DUN. DUN

If you are anything like me (which you are probably not, but bear with me), you don’t like the idea of conflicts. I am, as they call it, conflict averse. And while embracing conflicts might be a decent strategy in real life, in computing, we are usually trained that conflicts are bad. Very bad.

That is until we start learning about distributed systems. That’s a mighty term for a lot of ideas and concepts, but it usually boils down to: you have two or more computers connected by a network of some sort and you are trying to make it so some piece of data looks the same on both computers.

Now the tricky part is this: either computer can fail at any time, and the network can have a multitude of failure scenarios (c.f. The Fallacies of Networked Computing) that range from making the other computer appear very far away, or very busy, to turned off.

Distributed systems is a large field with a lot of applications, but that must not scare us. On the one hand, we do have a distributed system: two devices and a server, all connected over the internet.

This fits the “two or more computers connected by a network” definition. But instead of introducing a whole new field of computing into our discussion, we’ll just learn a few best practices from distributed systems and then be on our way, so no worries!

As opposed to single-machine computing, which is what we usually do, where we’ve learned that conflicts are a bad thing™, in distributed computing conflicts are a natural state of data. That doesn’t mean we can just ignore them because they are “natural”. It means we cannot ignore the fact that these things exist, or try to build up an illusion that these things don’t exist. We have to embrace conflicts.

We’ve already seen we can create a conflict, now we can look how to resolve one. Maybe in our scenario above, the typo fix is in the second paragraph and the new developments in the story are at the bottom.

If we are storing a story paragraph by paragraph, for example, we could have a conflict resolution procedure that checks if the two edits we have are in separate paragraphs, and if yes, updates our object with both new versions of the respective paragraphs and then stores the result back into our object.

This is commonly referred to as a merge. If you’ve used git or other source code version control systems (especially ones of the distributed variety), you might have seen merge strategies that do exactly that.

In this case, a computer can decide what the right final, non-conflicted version of our story is.

But now imagine both author and editor fixed the typo, but the author decided to use a different word altogether, while the editor just corrected the spelling, and both are doing this at the same time.

In that case, it is impossible for a computer to know which is the correct version. We could apply a policy in our app that in case of a conflict the author’s changes should take precedent. Or editors get the preferred treatment. Whichever it is, this can be a viable strategy for specific apps.

As mentioned before, these are all examples, and we are trying to become experts in sync more generically. So we need a better solution.

And here is a bit of a bummer: there are some types of conflicts that no computer can ever solve. We will need human intervention. There is no way around this, but at the end of the day, our brains are a lot more sophisticated than computers and conflict resolution is where this plays out.

Despite this, I hope you feel a little more comfortable with conflicts.

As an added bonus, now we are equipped to learn how to improve on timestamps for versioning our objects.

Vector Clocks

If you start searching for techniques to solve the issues with timestamps from a wall clock, you will eventually find a mention of vector clocks, for a variant, see Lampert Timestamps. A vector clock also works with timestamps — though its source is not a wall clock, like we humans use, but a so-called logical clock.

When using vector clocks as version specifiers, we not only send logical timestamps with our objects, but also the state of the logical clock, so that later, when it is time (pun intended) to compare two objects, we can calculate which one came first. Great!

Just one more thing.

Remember when we talked about the author and the editor of a story fixing a typo and adding to the story at the same time? We now know how to detect and how to resolve this conflict.

But what if the typo fix both create result in the same text? With timestamps or vector clocks, a conflict is still generated, and our subsequent merge procedure has got nothing to do. So we can handle this too, but since the resulting story is the same, wouldn’t it be nice to not create a conflict here in the first place?

I realise this example is a little bit contrived, but I hope by staying within our example project, we can see the problem here. Where the problem would actually make our lives harder is when we don’t have a single sync server, but a cluster of sync servers, which we might need for scaling to higher and higher loads of our system.

Explaining all this takes time, and this tutorial is already getting long, so we are going to skip the details of this. I hope you forgive me. If you don’t, I hope this gif of a squirrel makes up for it:

Content Addressable Versions

To solve the problem of same edits generating conflicts, we need to look at another technique: Content Addressable Versions. The idea here is this: take the contents of an object, pass it through a hash function, and use the resulting hash as the version. If we now make the same change on two devices, we don’t generate a conflict. Hooray!

We have one old problem though: Content Addressable Versions are not ordered. There is no way to take two of those versions and know from their value which came before the other. This is unfortunate, but we can solve this.

Instead of replacing the version with each object update, we keep an ordered list of versions along with the document. When we update the object, we put that Content Addressable Version at the top of the list. Then we can know which version came before another by traversing our list.

Before this is generally useful, we must add one pragmatic trade-off. If we update our objects a lot, our list of versions gets very long, and we are storing a lot of data, and moving said data across devices, and at some point we are losing all the benefits we covered when we discussed deltas above.

As a result, we make this list a bounded list, e.g., it can have at most “X” entries. We can even make this length adjustable depending on our application. Anything between 10 and 1000 is reasonable for general purpose applications, so we’ll have to play with our app to see what’s best.

The trade-off here is avoiding unnecessary conflicts at the expense of a little more storage space, and we get to decide how much storage we want to spend for this benefit.

You may think that the list of versions is rather clunky compared to a neat solution like Vector Clocks. But being able to avoid conflicts where possible becomes a priority when our server becomes a cluster of servers that all work independently to handle our application load and we want to present a consistent set of data to our users.

Storing Conflicts Efficiently

And then, finally, one last thing. When we are storing our ordered list of versions, it looks like this, simplified to use numbers instead of our Content Addressable Versions:

We have five versions; now we are creating a conflict. In order to signify we have two versions that are in conflict, we put them in a sub-list at the top of our list:

We now know that version “6” and version “5” are in conflict. Now imagine instead of resolving the conflict first, we create another one:

Now we have versions “7” and “6” in conflict, and that conflict being in conflict with version “5”. Heavy stuff!

We are representing versions in a tree structure. It allows us to:

store conflicts efficiently, and
resolve them recursively, so regardless of how many conflict we have, we can always get to a non-conflicted state eventually.

Putting it All Together

Here is a summary of our algorithm, without the explanation of why we do all the things, summing up all of the above:

There are two devices, A and B, that want to sync data from A to B.
Read high watermark checkpoints from both devices, if they exist.
Start reading the update sequence on A from the recorded high watermark or from the beginning, if none exists.
- for each ID/version pair:
- is it a delete?
  - Yes: store the delete locally (see below) on B.
- do we have the ID/version pair already locally on B?
  - No: fetch ID/version from A and store it locally (see below) on B.
- store high watermark of current update sequence on A and B.

Store it Locally:

Check version list to see if the new version is a direct successor of the currently latest version on B.
- Yes: add the new version to the data store on B.
- No: create a conflict.

Note: I’m glossing over a few details here that involve tracking of deleted conflicts, which are outside of the scope of this tutorial.

Conclusion

Thank you for reading! I hope you learned a lot about the pitfalls of synchronising data reliably and that you are not discouraged to tackle your own sync solutions based on the principles we’ve explored here.

With all the knowledge you’ve gained here, I hope you feel empowered to build your own sync engine!

Please leave any feedback in the comments on Mastodon.

Without you knowing, I explained to you the why and how of the CouchDB Replication Protocol, which allows seamless peer-to-peer data sync between any number of peers, including all the scenarios we’ve explored above.

The CouchDB Replication Protocol is implemented in CouchDB itself, so that covers our server component. Then there is the PouchDB project implementing the same protocol in JavaScript targeted at Browser and Node.js applications; that covers your clients and dev servers.

Addendum

There are two related technologies I’d be amiss to not mention: Operational Transforms and CRDTs.

Operational Transforms

If you’ve ever seen collaborative text editing on Etherpad, Google Docs, or similar, that is powered by a technology called Operational Transforms. It is designed to let any number of people collaborate on text at the same time. It can deal with a certain level of network instability, but generally, it requires clients to be connected at all times. If you go and keep editing, your changes can be integrated later, but not indefinitely later.

In addition, it is designed for text, not for generic objects, so for true offline capabilities of generic data objects, Operational Transforms are less useful. Check them out if you need a solution for mostly connected text.

CRDTs

Conflict-Free Replicated Data Types or CRDTs are specialised data structures designed for use in distributed systems and they have a lot of the properties we’ve discussed in this tutorial, but most notably, they do not have a concept of conflicts. How great is that?! Very, in fact.

Alas, everything in programming is trade-offs, so what do we trade for being able to have conflict-free data structures? Well, they are specialised data structures, like sets and counters, and not generic object representations like JSON. So, we’ll have to buy into a whole world of these specialised data structures, and maybe we have a hard time mapping our application objects to them. If you think your application can benefit from CRDTs, I strongly encourage you to try them out, but be aware of the trade-offs.

An earlier version of this post used to be published on the now defunct hood.ie blog. That earlier version had received reviews from Naomi Slater, Katharina Hößel and Jake Archibald. My thanks to them.

Discover more ways to work with CouchDB — the syncing database — on the Neighbourhoodie blog, where we regularly publish tips, guides and tutorials.

It’s time for the 2025—2026 Annual Apache CouchDB User Survey!

Maddy — Wed, 04 Feb 2026 15:01:32 +0000

CouchDB is an open source database made by a global team of volunteers. It’s shaped by its users and community through everything from contributions to project feedback — and now is the best time to get involved.

📝 Be a participant in the 2025—2026 Annual Apache CouchDB User Survey

You’ll have the opportunity to let us know what you use most, love most and need most from your CouchDB.

We’ll also be back here when we’re ready to share the insights. If you’re curious what others have to say or want to learn what they have running alongside CouchDB in their stack, sign up for our newsletter to get the results as they come out.

We can’t wait to hear from you!

How to Sync Anything: Building a Sync Engine from Scratch — Part 2

Maddy — Wed, 19 Nov 2025 08:00:00 +0000

In this part, we will learn how to efficiently find out what data needs to be synchronised.

Say we have a news app that runs on mobile devices and a server that publishes new stories. Blogs and RSS are good real-world examples.

The scenario is this: our app starts for the first time and there are no stories available for the user to read on the device. So, the app asks the server to send the latest set of stories to the device. End result: our users get to read some stories.

Later, when our apps starts for the second time, your app asks for the latest set of stories.

And here is the first interesting bit: we don’t want to send the app the stories it already has.

Why?

Four reasons:

They are already on the device, so it’s redundant
It costs us server processing time
It costs our users bandwidth
It adds response-time latency to the experience

As a general rule of thumb, in user experience design: don’t let the user wait unnecessarily or they will get frustrated and stop using your app, your store, whatever.

So what’s the solution?

We need to request new stories from the server in a way that says “I’ve got all stories up to this point. Give, give me anything that’s new”.

We’ll call the difference between “all stories” and “stories that exist on the client” the delta (i.e. difference).

The delta is what we need to get from the server.

Calculating the Delta

To ask for the delta efficiently, we need two ingredients:

Our app needs to store its state somewhere, so it knows what stories it already has. We’ll call this the high watermark (for reasons that will become clear soon). In a native app, this could be stored in a device-local database or file. In a website or web app, this could live in browser storage systems like localStorage, or IndexedDB.
Our server needs to look up the list of all stories, sorted by when they were published. It needs to be able to do do this efficiently. And it needs to be able to send back any range of stories, from any specified start date to the present moment.

A naive implementation on the server side would be something equivalent to retrieving all stories from the database, sorting them by date in memory, before sending the result to the app. If the app sends a high watermark, the server would only send stories that come after the high watermark.

While this definitely works, it is not very efficient.

All stories have to be loaded from the database, sorted and finally filtered to match the range the client is interested in.

While a server with only a few apps requesting only a handful of stories can easily do this, we should be looking for optimisations here. Something that will work with large data sets.

How? Traditionally, we would add an index to the database.

The Index

An index has two things going for it:

Order: things will be stored in index order on disk, so reading things (e.g. stories sorted by publish date) in that order is very efficient.
Support for ranges: reading only part of an index from any point in the index range (e.g. all stories after a certain date) is very efficient.

Maybe the database already has an auto-increment integer as a primary key. If so, each new story gets an ID integer that is one higher the previous ID.

If this is the case, the server is already prepared and the app only needs to store the highest ID it gets from the server in the initial request. The app can then send that ID as the high watermark for subsequent requests.

When the app receives the new batch of stories, it stores the new highest ID as the next high watermark, and so on.

Here’s what this looks like:

Updates

Now, let’s imagine our stories are sometimes updated. This is fairly common. Typos need to be fixed, corrections posted, new story developments need to be added, and so on.

In this case, using an auto-incrementing ID is not a good solution.

Not only do new stories need to get a new ID, or more precisely a higher ID, but also: updates to a story need to get a higher ID, so that they are included in the calculation of the next delta.

To solve this, we need a new table: updates.

This updates table provides a secondary index that we use for recording updates only. If you add one record for every update, pointing back to the corresponding story that was updated, the auto-incrementing ID functions as an update sequence.

Now, instead of passing back the story ID as a high watermark, the client can pass back the last update sequence it got. (We have to tell the client the latest update sequence with every request for this to work.)

When the server receives an update sequence from the client as the high watermark, all it needs to do is send back every story that corresponds to every update with a higher update sequence ID in the updates table.

In this example, the client first receives five stories and the update sequence ID of “5”. When the client sends the “5” back as the high watermark, the server notices that there's an update record with ID “6” and sends back the only the corresponding story, which happens to be the story with ID “3”. The client also now gets the update sequence “6”, which is recorded locally as the high watermark.

In our case, our app might choose to handle this information by adding an “updated” marker to the list of stories, so our user knows the story has been updated with new information, even if it was previously read.

So far so good!

But we need to handle one more case for updates: what happens when a story has been updated twice?

This is how it would look in our current system:

This looks very similar to the previous image, with one difference: the delta includes story three twice. Thing is, we only really need the latest update. That previous update will be discarded by the client, so we're wasting server resources, bandwidth, and UX latency sending it.

In our example scenario, this isn’t too bad. But imagine we're downloading thousands of objects with hundreds of updates.

Not great.

So what do we do about it?

The update sequence index we’re using is called a composite index. This means more than one data item defines the index range and how it is sorted. In our case, we have an auto-increasing change ID plus a static story ID.

We need to make one more change to our composite index to solve this multiple update problem.

Let’s make it so that the story ID column in the updates table is unique. Every time we try to write to that table, if we see there’s an existing entry with the story ID we’re about to record a change for, we delete the old row.

If we do this, there will only ever be one update record per story, and it will always correspond to the latest update.

Here’s what that looks like:

What happened here?

Between the first and second requests, two changes were made to story three. Because the story ID column on the updates table is unique, we only have the final update recorded, which is why the update with ID “6” is missing.

The end result? Our client only receives the final update for story three.

Deletes

There’s one last thing before we move on: deletes.

In our app, we want to treat deletes as updates, so that deletes can be sent to the client like any other sort of change.

Let’s take the same example we’ve already been using and instead imagine that between the first and second client requests, story four is deleted from the server.

In this case, we’d want to record this deletion in the updates table, along with an update ID. We can then send this information back to the client as if it were a new story, updating the high watermark as we do so.

Here’s what that would look like:

At this point, the app has several choices. The simplest is probably just to delete the story locally.

The Road to CRUD

Congratulations! Now we know how to communicate new stories, story updates, and story deletions to our app. In other words, we can sync Create, Read, Update, and Delete (CRUD) operations.

And here’s a cool thing: we’ve done this generically enough that a story can be any sort of object we might want.

In this post, we’ve spoken about different versions of the same object. For example, using 3, 3*, and 3** to refer to the first, second, and third versions of story three. We’re also talking about versions when we talk about the deleted and non-deleted version of story 4.

This concept — versions — is very important. So we’ll take a closer look at that in post three of this three part miniseries.

Discover more ways to work with CouchDB — the syncing database — on the Neighbourhoodie blog, where we regularly publish tips, guides and tutorials.

How to Sync Anything: Building a Sync Engine from Scratch — Part 1

Maddy — Wed, 12 Nov 2025 08:00:00 +0000

There’s an old saying I paraphrased in this by now ancient tweet[sic]:

“Friends don’t let friends build their own {CRYPTO, SYNC, DATABASE}.” — @janl on September 24th, 2014

What do I mean by that?

Well, it’s very hard to get these things right. Additionally, not getting them right will mean a lot of unhappy developers and end-users. In other words, these things are best left to the experts.

However.

Brought to its logical conclusion, we’ll end up with a situation where nobody is doing crypto, sync or databases anymore, because no new people learn these things. So there are exceptions.

You can be an exception.

Let’s become experts in one of these things! Specifically, for the purposes of this series of posts: sync.

Or, to put it another way: let’s look at a bunch of different problems and solutions that relate to making data available offline.

I’m picking a specific field (web frontend / backend sync) by way of example, but know that the same concepts apply to any two or more sites you want to synchronise.

The Future

When I first wrote this article, Progressive Web Apps (PWAs) were just being announced as maybe becoming a thing in the future.

That future has long since arrived. What has not arrived is the logical next step that comes after caching your static assets: how to make your data available when a web app is not connected to a server.

You already know how to store your website’s assets in a ServiceWorker cache. You already have some idea about how to store server-delivered content in IndexedDB. And you might know how to locally store user-entered data (think an address book, note taking, favourites, and so on) in IndexedDB or localStorage so you can push it to the server when a network connection becomes available.

We can already see a few different scenarios:

server always pushes (news)
client always pushes (notes)
both push (social, email, or multi-device access of individual notes, and all forms of group data sharing.)

Let’s have a look at the different techniques required to make this all work. We’ll see which applies to which use-case while we go through everything, step-by-step.

The Scene: Background Sync

In his introductory talk about PWAs, Jake Archibald shows an example of a chat application. Towards the end, he uses the Background Sync API to send new messages “later”, whenever the browser thinks it has an actual internet connection (as opposed to having a network but not thinking there’s an internet connection).

This is a great UX experience: the interaction is done as far as the user is concerned. There is still an indicator that the message hasn’t reached the recipient yet, but there is no need to keep the user waiting for any network operations.

In the same talk, Jake explained that the browser (and sometimes mobile operating system) mechanics that determine whether a device is online are not very useful. That’s because they only cover the connection from the device to the wifi router or cell tower. But there are a lot of steps between there and the final web server. For example: ISP routers, transparent proxies, satellite uplinks, just to name a few.

Now imagine we are sending a message within a Background Sync event as shown in Jake’s talk, but we’re on a fast train that goes through a bunch of tunnels in fast succession. Or we are on a conference or hotel wifi. Or we’re a large event with thousands of other people.

Our phone might get a request/response going, and as a result, wakes up Background Sync and tries to send our message. But by the time the message gets sent, we don’t get any requests going because the network became unavailable again. Background Sync will then re-try at a later time. Which is great, and exactly what we want. We don’t have to worry about whether the message is sent, because we know it will be, eventually.

Now. Let’s look behind the scenes.

The Cycle of (the) Web

Even though there are many hops at which things can go wrong in a HTTP request/response cycle, there are generally two parts: the request and the response.

Each part can fail, so we have to account for the following scenarios:

the request fails
the request makes it, but the response fails

Scenario one is neatly covered by Background Sync, but what about scenario two?

Imagine we are sending a message. The server accepts it and sends it onwards to the final recipient. Then, the server responds to the device, saying it handled the message correctly.

If that response fails, Background Sync will consider the message sending request as failed and will re-try it later. At that time, the server accepts the message and sends it to the recipient again. And now we’ve created a mess, because the recipient is left wondering why there are two messages that are exactly the same.

And if we are really unlucky, our recipient will get an infinite stream of the same message because the same problem is happening over and over and over again.

Let’s see how we can avoid this.

There are solutions for this exact problem that we can employ in the server portion of our app. But we’ll get to that later. For now, let’s look at how to solve this more generically, so that we can solve the same problem when it comes up in very different contexts, not just when sending messages.

Identity

In programming, we generally work with bits of data wrapped up in objects. If we want to be able to refer to an object at a later time, we must be able to reference it unambiguously. Sometimes that means giving something a name (like an auto incrementing number) and sometimes we can derive an identity from a number of uniquely identifying properties of the object.

For example: in an address book, assuming no two people have the same name (a bad assumption, but we’ll get to this shortly), the identity (or ID) of an object could be derived from the combination of the first name and the last name. That would be an example of what is known in database circles as a natural key.

The advantage of a natural key is that you don’t need to store any extra data on the object to be able to refer to it later. It also means your data is easier to de-duplicate.

Natural Key Disadvantages

There are two disadvantages to using natural keys: changes in the natural key and the need for uniqueness.

Key Changes

Say your natural key is somebody’s email address. Email addresses change, and you need to be able to deal with this change.

For example, if you update someone’s email address, but another object was using that as the natural key to refer to your object, that second object will need to update its reference.

While this type of change might be infrequent, other natural keys can change more frequently.

This is opposed to an opaque key, or surrogate key, that is a number (like an account number, a social security number, a phone number) or string of random characters (e.g. a UUID). Every time you’ve seen an auto incrementing ID column in an SQL database, that is a surrogate key.

Uniqueness

In our address book example, when your natural key is derived from last name and the first name, we may hit a problem. What if you know two people named Jane Smith? Our natural key is not unique, so if we try to look up “Jane Smith”, we can’t be sure which object to return.

Surrogate Key Advantages

Surrogate keys have the advantage of being agnostic to any changes in our objects’ data.

The disadvantage is that they are opaque, so there’s no natural relationship between an object’s ID and its data. The ID 43135 doesn’t tell you a whole lot about a user record for Jane Smith.

While these IDs are usually only used by computers, sometimes they leak out into the real world. Sometimes we even expect people to remember them and manipulate them. Not ideal. In addition, they make debugging and logging harder, as devs are now forced to map IDs to objects’ natural data to see anything useful.

Moving Forwards

Since we are looking at how to make data available offline, and since that means storing data on an end-user device and a server, we have multiple copies of all our data. So we need to be sure that any operations can unambiguously reference that data.

So, when creating a new object that we give it a unique ID. And we can’t guarantee this with natural keys. A new Jane Smith could be added on the server and the user’s device while they can’t talk to each other because one is offline. That would spell all sorts of trouble for us.

Unless any of our data lends itself to be a natural key without the disadvantages, we use surrogate keys.

Specifically we use UUIDs, because they have the convenient property that even though you still can’t guarantee the uniqueness of natural data on disconnected devices, the chance of assigning the same UUID to different objects on two or more devices is so unlikely, we practically don’t even have to consider it a possibility.

With that sorted, we have a new problem. We have multiple sets of data, which may or may not diverge from each other in various ways. In my next post, I’ll show you how to figure out what’s changed, what needs syncing, and in what direction.

Discover more ways to work with CouchDB — the syncing database — on the Neighbourhoodie blog, where we regularly publish tips, guides and tutorials.

How to Sync Anything

Maddy — Wed, 05 Nov 2025 13:40:23 +0000

In this article I’ll discuss a common naive solution to replication, why it doesn’t work, and what the building blocks of a good solution look like. Having established this theoretical framework, my next article will look at how CouchDB provides many of those building blocks such that replicating from it into any other system is relatively painless.

Something I have seen happen surprisingly often in my career as a developer is that teams end up having to implement replication, but they don’t acknowledge that this is what they’re doing, and so end up with ad-hoc solutions that don’t really work in production. When I’ve worked at product companies this usually takes the form of integrating internal systems, such as caching, search indexing, or making incremental updates in single-page apps. Sometimes it means having to integrate with external services, for example copying a company’s user profiles into a newly adopted CRM system and keeping them up to date going forward.

As a consultant in the public sector, I’ve seen many projects whose entire purpose was to integrate several existing systems and make sure they all agree on the exact state of their data. For example, the permissions for accessing casework documents are driven by roles stored in the HR department’s Active Directory. Or, some accounting information needs to be saved in two places at once because two organisations got merged and are in the process of splicing their systems together. Or, in the tech infrastructure world, you might have a complicated build pipeline involving generation of artifacts from many different input sources and you want it to react to upstream changes promptly.

All of these are, at heart, replication problems: the goal is to make some target data store (a cache, an index, an external HR system, a package repository) in some sense mirror the state of a source data store. I am using the term “data store” in the loosest possible sense here, to mean any stateful representation of some of the information in your system. Often these will be honest-to-god databases — an RDBMS, a document store, etc. — but you really ought to think of any part of your system that accumulates state in this way.

There are many different techniques for solving this problem, and one’s choice of solution is often constrained by the fact that many of the products people deploy to store and distribute their system’s data are not designed to make replication easy to implement or reliable to operate, especially when numerous heterogeneous systems need to be integrated. A lot of the more “obvious” solutions are riddled with pitfalls that stem from the fact it is fundamentally quite hard to keep two stores in sync when one or both of them keep changing, when there’s no globally agreed ordering of events, and when updates can fail, get re-ordered or lost. Many of the problems I’ve seen in production happen because of a naive approach to these problems, or because engineers ignore them entirely.

The Easy Solution: ETL

Before getting into the details, I’ll quickly mention one solution that works really well for certain situations. Given that updating the target system incrementally from the source is hard to do correctly, the simplest possible solution is to not bother doing that at all. Instead, you rebuild the target from scratch periodically. You locate all the data of interest in the source, pull it out, transform it as necessary, and write it into the target, which starts from a blank state on each run.

This process is generally known as extract, transform, load (ETL) and is widely used in analytics and data warehousing. It is very easy to make it correctly mirror the state of the source, because the target does not retain state for very long. If you find a bug in the data copying logic, you fix it, deploy the fix, and the state gets corrected the next time the process runs, because you’re building a fresh copy of your data every time.

The Pros and Cons of ETL

The downside of this approach is that it’s very slow, because you have to scan all your source data every time. In fact it gets slower as your system grows, which can produce costs that don’t scale in the same way as your revenue sources. It also has bad consistency; if the target is rebuilt nightly or weekly, then it is never fully in sync with the source, and may even be internally inconsistent as your source data changes while it’s being copied.

For some applications, this is not a problem. Many analytics use cases do not benefit from real-time access to data and may even give worse results in this mode. They can also make use of cheaper archival storage technology and specialised processing tools so that their operational cost doesn’t grow too badly as your source data grows.

There is also an operational benefit that comes from decoupling. The ETL approach typically doesn’t need custom logic built into application code to make it work. Application developers can get on with building features, while a separate team maintains the system the extracts what they want from the application database, as long as the data extraction does not cause excessive load and interfere with end user traffic. This lets both groups operate quite independently without causing one another friction.

So, if you don’t mind the target being slightly stale, and taking a long time to update, ETL is a great solution. But what happens when that’s not an option?

Ad-Hoc Replication

ETL may be conceptually very simple, but it has some major downsides. It cannot maintain real-time consistency with the source system, and it’s also extremely wasteful to scan through all your data on every rebuild, when most of it will not have changed since the last run. To address both these issues, developers will often want a more incremental approach to keeping the target up to date.

This is where things typically start to go wrong. To make incremental sync work correctly, it is usually necessary to think about the big picture of how you want the source and target to interact and design or adopt a protocol that fits all these needs. However, many of these replication projects are developed in an ad-hoc way, with little bits of code being added to handle particular scenarios, and the interactions and failure modes of these small changes are never looked at holistically.

The “Waiting for Changes” Approach

A very common approach to ad-hoc replication, especially when used to implement caching or search indexing, is to put hooks or event listeners into the application that react to changes in the data, and forward those changes to the target system. Those hooks may be put into the data model, so you can be sure that any code path that changes a record will invoke the hooks. However, this means it’s impossible to invoke the code in the data model without causing these side effects to the target system, which might be undesirable, especially for testing. Conversely, if the hooks are put into the application’s request handling tier, this keeps the data layer uncoupled from the target system, but it’s much harder to make sure you cover every code path that might result in a record being changed. In some frameworks, it’s not uncommon for developers to debug and fix production issues by invoking the data model through an interactive shell, and any changes made this way may bypass any application logic built atop the model.

This is the first source of inconsistency in these systems: the method of detecting changes to the source data is itself lossy. Because of this, there is no way of knowing whether any given record in the source system is accurately mirrored in the target without scanning the entire data set, i.e. without ending up with an ETL implementation.

What Happens in a Rollback?

The second major source of inconsistency comes from what those event hooks actually do with the change they are observing in the source data. When writing to the target system, they might attempt to relay the full state of the source record, or just the fields that changed in the source record, depending on the data models of the source and target and what facilities their framework provides. The latter assumes that the target record was already in sync with the previous state of the source record, which might not be the case, especially when other failure modes are accounted for, and this can result in lost updates. Conversely, the event hook might be implemented in such a way that it can be activated by changes in the model that have been sent to the database, but are part of a transaction that has not yet committed. If the transaction ends up being rolled back, then you end up with uncommitted data in the target system. This can be especially hard to clean up because the target state results from source state that should never have been recorded and will never be explicitly deleted.

More Ad-Hoc Replication Inconsistencies…

Because the source and target are often distinct systems, the target may be unavailable when the source is changed, so the event needs to be retried somehow. The target may reject the update because its data model has different constraints to the source. For example, it might not recognise certain email addresses as valid, or it might assume emails are used to identify users and must be unique, which may not be the case in the source. These sorts of validation constraints mean that sometimes, data from the source cannot be represented in the target’s data model.

If the target system is a remote third-party API, with high latency relative to the system’s own resources, it can also fail for network-related reasons, enforce rate limits, or just add too much latency to apply updates to it inside the scope of normal end-user requests. Therefore the copying of data to the target system is often pushed into a background job queue and is processed asynchronously from where the change originated in the source system.

This is the final contributor to this ad-hoc approach failing to produce consistency. If changes in the source data are processed asynchronously and may fail and be retried, they will end up being processed in a different order to how they happened. In general, arbitrary updates to data do not commute, that is, you cannot reorder them without changing their effects. If somebody updates a record and then later deletes it, the replicator will become confused if it tries to delete the record from the target and then update it.

...and Edge-Cases

One additional edge case that emerges in ad-hoc replication is that while the event hooks deal with changes made to the data since their introduction, they do not deal with all the legacy data from before that time. This necessitates a completely different approach to copy over all the existing data, and then keep the target up to date using event listeners. Deciding how to switch from the former to the latter is very difficult because the source data is changing all the time, so it’s hard to execute this correctly without downtime to stop anyone changing the source during the switch-over. Having a whole different code path for a one-off event makes it likely that code path will have defects that go undetected and require future ad-hoc fix-ups to the target data.

This approach is sometimes likened to change data capture (CDC) or event sourcing, but it is strictly weaker than both of them. CDC requires a reliable way of identifying which changes have not yet been replicated, and in event sourcing the primary way of recording updates is to write them to a global consistently ordered log, much like a database’s write-ahead log (WAL), which is then consumed to produce various representations of the system state.

Ad-hoc replication is characterised by reactions to changes that can be ignored, fabricated, misinterpreted, reordered, dropped, or executed multiple times, with no reliable way to check and ensure that the target is in sync with the source. It inevitably produces inconsistent states that are hard to debug, not least because there is no formal definition of what state the target should be in, given some state of the source. There is never one root cause to why the target is in a bad state; the reality is that this whole approach is almost guaranteed to produce inconsistency, and needs to be replaced with something else. So what does that something else look like?

Diff and Patch

Broadly speaking, replication systems can be classified into two camps: state-based, and operation-based. State-based systems work by examining the current state of the source and target, and making changes to remove any differences between them. Operation-based systems rely on distributing a consistent sequence of events, or a log, to all the systems and having them build their local state off this log. This is how event sourcing works, and how consensus algorithms like Raft work.

Many real-world systems use some combination of these approaches, but here I’ll be emphasizing a state-based approach, with some optimisations enabled by event logs. State-based replication tends to be more applicable to disparate systems that only expose their current state to clients and don’t have any way of sharing their event histories such that you could use them as a log.

The first ingredient of a replication system that actually works™ is something I hinted at in the previous section. There must be some well-defined way of determining whether a record in the target is consistent with its corresponding source record. This includes the possibility that the target record should not exist, because it is absent or deleted from the source. If it is not in a consistent state, then a description of how it differs from the intended state must be produced, i.e. which operations would be needed to bring the target record into line with the source.

It is possible this sounds kind of “obvious”; how can you say the target is in an incorrect state if you cannot define what the correct state looks like? However, I have personally seen too many systems where the team did not have a satisfactory answer to this question, and solving this was a prerequisite to making any further progress on the problem.

You can think of this operation as a sort of “diff and patch” function. The diff tells you how the source and target are out of sync, and is empty if they are in sync. The patch tells you how to change the target to make the diff become empty. For example, imagine we have two systems that contain data on accounts in a social network, and each of them have a certain account in the following state:

Source                      Target
------                      ------

{                           {
  "username": "alice",        "username": "alice",
  "location": "london",       "location": "berlin",
  "following": [              "following": [
    "bob",                      "bob",
    "carol",                    "dave",
    "dave"                      "eve"
  ]                           ]
}                           }

The diff between the target and the source would be:

The location field changes from berlin to london
carol is added to the following set
eve is removed from the following set

The patch that removes these differences would be:

Set the target's location field to london
Add carol to the target's following set
Remove eve from the target's following set

The exact details of the diff and patch will depend on what data representation and interface each system actually provides; if the target is a document store, the patch might reduce to a single write containing the whole new state, but if it’s a SQL database it will be possible to update specific columns and add/remove specific rows as required. It is usually desirable to express the diff and patch in the smallest number of operations against each system to maximise performance.

This sync function must be idempotent i.e. it should be safe to run any number of times against record in the source system to bring its corresponding target record up to date. This immediately fixes several problems with ad-hoc replication:

Replicating a change always works by comparing the full state of the source and target records, rather than assuming the target matched the previous source state and copying some random changes over. This makes sure your event listeners always leave the target in the correct state.
It’s not affected by event handlers being reordered by asynchronous processing, retries, and so on. The diff/patch function always does the same thing whenever it is run: it does whatever is needed to match the target match the source.
If it fails due to a transient network error, a rate limit, etc, it can be retried indefinitely on an arbitrary schedule and will eventually succeed and produce the right state, as long as it inspects the state of the source record when it runs rather than retaining the state from when it was first scheduled.
The same code path can be used for copying over all legacy data, and for reacting to future events. You either run the diff/patch function against all existing records, or against a record you know just changed. Eventually this will result in all source records being synced.
There is a well-defined notion of what it means for a target record to be in an incorrect state, and furthermore, the system itself can identify such states and correct them automatically. If the detection is found to be incorrect, it can be patched and re-run over existing records to resynchronise them.

Methods for Comparing

Notice that we now have the same code path being used to handle legacy data, new changes, and retrying in case of failures. This is an example of crash-only design, where you have one code path that handles “normal” operation and failure states. The function’s starting assumption is that the target is in a bad state that needs to be identified and corrected, rather than assuming some good state and invoking an exceptional code path if an error happens. Using the same code path for all eventualities gives you a simpler system that is much more robust.

This is the core building block of several replication programs. Git’s push command works by comparing each local branch with its corresponding remote branch, figuring out which commits the remote branch is missing, and copying them over. Rsync works by comparing the file size and modification time of each source and target file (or their checksums if so instructed), and copying the source data to the target if those differ. CouchDB replication works by comparing which revisions the source and target have for each doc, and copying over any that the target is missing. In each system, the pattern is the same: compare the source and target records, and transfer whatever data is needed to make them equal.

A further desirable property of the diff function is that it is fast. This is not quite as important as making it fast to determine which records out of your whole data set are out of sync, which we will cover below, but is still beneficial especially in cases that require a full scan of the source data set.

For example, Rsync compares files on their size and modification time, which are small bits of metadata that are cheap to read, instead of reading the content of each file, which would be much slower. This assumes that if two files have the same size and modification time, then they very probably contain the same content. This is especially likely given that Rsync will set the mtime of the target to match the source, after it copies the data over and verifies both files have the same checksum. After doing this, it is relatively safe to use the mtime as a proxy for the fact that the data was checked last time Rsync itself updated the target.

Optimising Compare Behaviour

In Git, comparing two branches is cheap because the way the commit history is stored makes it inherently efficient to identify which commits exist in one branch and not in another. It doesn’t have to employ a “short cut”, it has a data model that directly makes its desired operations cheap. CouchDB has a similar internal structure that makes it cheap to compare the revs of a document between the source and target, so it can minimise which revs need to be copied over without comparing the content of all the revs and sending all that content over the network.

When designing your own diff functions, it is worth looking for places where you might be able to skip comparing something expensive by taking short-cuts or using better data structures. However, if multiple systems and/or people have the ability to write to the target system, this isn’t always possible and your diff function will need to be “paranoid” and assume the target could be in any random state unless you’ve proven otherwise.

Tell Me the News

Just having a reliable idempotent sync function has already solved a lot of our replication problems. In principle, you could run this function whenever you want against all source records, and this would give you eventual consistency in the target system. However, most source records don’t change most of the time, so scanning through all of them will typically be very wasteful and become slow to react to changes as they happen. To make a replication system practical, we need some way of identifying which records have changed recently. I’ll briefly discuss a couple of common ways of achieving this.

The Record Flagging Method

The first approach is to put a marker/flag on each record when it is changed, and to remove the marker after the diff/patch function has re-synchronised the record to the target. This is relatively simple to implement using a counter; when the record is changed, the counter is incremented, and when it is re-synchronised, the counter is reset to zero unless it was incremented during sync. This makes sure that changes made while the record is being synchronised are not ignored, and instead initiate another execution of the sync function. Conditionally resetting the counter can be implemented using compare-and-swap (CAS) to avoid having to pre-emptively lock the source record and prevent end users interacting with it while it’s being synced to another system.

How the sync function is executed is completely up to you — after all, it is idempotent and therefore safe to run at any time. If a lot of changes are likely to happen to a record over a short span of time, you might choose to schedule the sync function to run once, a short time interval after the first change is detected, so it picks up all the changes and doesn’t need to be executed for each individual change. If the target benefits from sending changes to it in batches, you can use a cron job to find all the records marked as changed and sync them all at once. Or, if you need less replication latency, you can execute the sync function immediately on each change.

However, keep in mind that you should only allow one instance of the sync process to run on a given source record at a time; if multiple processes are trying to sync the same record at the same time they may end up getting confusing diffs and producing an inconsistent state in the target.

To deal with the transfer of legacy data, or to deal with changes and bug fixes in the sync function itself, you can mark all existing records to execute sync against all of them. You will probably want a background process to handle syncing records in batches, just so you can control the throughput and amount of load this generates on the source and target systems, rather than trying to re-synchronise all your data immediately. The outstanding backlog can be easily monitored by counting the number of source records marked for sync.

The Event Log Method

The second common method for signalling which records need syncing is to have the application add events to a log whenever a record is changed, and have another process consume this log and use it to execute the sync function against the appropriate records. Whereas the previous approach is a form of change data capture, this technique corresponds to event sourcing. It is typically implemented using background job queues, a database write-ahead log, a message broker like RabbitMQ or an event streaming system like Kafka. However you implement it, the core idea is the same: write to the log when a record is changed, and consume from the log to invoke the sync function.

Note that for this to work, it is critically important that the log is complete i.e. it faithfully reports all changes to the source data. The log should not discard an event until the sync function reports that it has been successfully handled, i.e. that the target is up to date as of that event. Is it not necessary for the log to preserve event order, indeed in many systems it is not possible to construct a total global order in which source data changes happened. The log is just there to provide a signal about which records need to be synced, so that you do not have to scan your entire data set to find this out.

…and How to Choose Which

Whether using record markers or an event log, you must ensure that a change to a record is only considered to have been processed after the sync function reports that it has been successfully handled. Since the sync function is idempotent, this means your change notification does not need exactly-once delivery, it is fine for it to have at-least-once delivery. In the worst case, the sync function runs multiple times for the same event, determines the target is already in sync, and has no further work to do. If the system has at-most-once delivery, or the sync function fails to retry on transient failures, it will end up not conveying all changes to the target, and the source and target will drift out of sync.

How can CouchDB help?

As a distributed database, CouchDB is built from the ground up to make replication reliable and efficient. Not only is replicating databases between CouchDB instances as simple as a single API call, it gives you the building blocks necessary to replicate from CouchDB into any other system you like with relatively little complexity. In a future article we’ll look in detail at how to use the CouchDB _changes feed to reliably replicate your CouchDB data to other places.

Discover more ways to keep your CouchDB project efficient on the Neighbourhoodie blog, where we regularly publish tips like this, plus guides and tutorials.

Everything You Need to Know About CouchDB Database Names

Maddy — Wed, 29 Oct 2025 08:00:00 +0000

Naming a database might not sound like an exciting activity. But it can be, if you know all the considerations that go into naming a database in CouchDB. Let’s start with the restrictions.

Restrictions

CouchDB database names have restrictions in terms of which characters can be used. Based on these restrictions, a database name:

must begin with a lowercase letter from a to z, no diacritics etc.
Each character in the name must be one of:
1. a lowercase letter from a to z
2. a number from 0-9
3. an underscore, dollar sign, open or closed parenthesis
4. the plus and minus signs
5. a slash
May be no longer than 238 characters.

Or expressed as a Regular Expression: ^[a-z][a-z0-9_$()+/-]{238}$

The collection of special characters might seem unfamiliar at first. We’ll explain how they come together further down.

First, we talk about one of them, the slash, or /. It is used in URLs and in UNIX-like file systems to denote hierarchy, like a subdirectory.

Database & File System Limits

In CouchDB 1.x, a database was represented by a single file in the file system. If you had a database called people, CouchDB would store all associated data in a file called people.couch. And all .couch database files are stored in the same directory on your file system (it can be found in the CouchDB configuration under [couchdb] database_dir).

CouchDB does not put a practical limit on how many databases there can be on a server (other than the theoretical ~43²³⁸), but file systems do. While those limits are getting higher and higher, in the times of CouchDB 1.x, you had to consider how many databases you would create to get good performance out of your file systems. Some file systems get really slow when there are more than 2¹⁶ or 2³² files in a single directory.

To make sure you can create more databases than a file system limits you to, CouchDB allows you to add slashes to database names. It will create actual subdirectories in the file system, so you can avoid having too many files in a single directory.

For example, a database called user/32/14/55187 will be stored in the datatabase_dir as user/32/14/55187.couch.

Splitting Databases with Sharding

CouchDB 2.0 introduced database sharding, the splitting up of single databases into multiple .couch files, which are stored each in their own directory per shard range. A shard range is expressed as a subdirectory which is named after the range, which goes from 00000000 to ffffffff. For example, a database with four shards (q=4) occupies the following shard ranges:

00000000-3fffffff
40000000-7fffffff
80000000-1bffffff
1c000000-ffffffff

A database with just one shard occupies the full range:

00000000-ffffffff

For databases with a single shard, which are common in the database-per-user pattern, all database files are stored in the same directory on the file system, and the same rules as with CouchDB 1.x apply.

But the more shards you have per database, the fewer actual files there are in each shard subdirectory in the file system. So you’ll be reaching at which point the file system introduces slowness at a later point, but it is still worth considering if you have a very large number of databases.

Incidentally, the shard ranges explain why the database name is limited to 238 characters. In the past, file system paths could be at most 255 (2⁸) characters long. But since CouchDB 2.0 and onwards always includes the shard range, we have to subtract 17 characters (2x8 for the beginning and end of the shard range plus 1 for the dash in the middle).

More On Special Characters

And where do the other special characters come in? It is pretty simple actually. When deciding which characters should be allowed in database names, the CouchDB developers surveyed all common file systems and collected all their respective restrictions about what characters could be included in file names. The result is the list of characters allowed in a CouchDB database: all these characters are allowed as part of a file name on any modern file system.

That said, we usually recommend keeping it to [a-z][a-z0-9-_/].

Discover more ways to keep your CouchDB project efficient on the Neighbourhoodie blog, where we regularly publish tips like this, plus guides and tutorials.

First Steps: Sharding in CouchDB

Maddy — Wed, 22 Oct 2025 07:00:00 +0000

While other databases out there might shard, CouchDB is one of the few that does it automatically and saves you the annoying — read error-prone — work of setting it up yourself. Being unique in this way, it’s a topic you may not know too well. The arcane-sounding term (especially if it reminds you of the prismatic variety) doesn’t need to conjure confusion or intimidation. In this post, we’re going to take a deeper look at scaling CouchDB with shards: what sharding is, plus why and how to do it.

What Sharding Is & Why It’s Useful

Any given central processing unit (CPU) — the thing on which your database lives — has a physical limit to how much processing it can do at one time.

By extension, there’s a limit to how big your database can get if you still want to do useful things with it if it can only run on a single core.

Databases each provide their own method to scale with your data and request load as they increase, enabling higher and higher volumes of traffic, more and more reads and writes. Broadly speaking, databases handle this load in one of two ways: scaling vertically or horizontally.

Vertical Scaling

This usually means throwing more resources at the machine running your database. If you rely on cloud computing, that might mean going up an EC2 machine size, or if you are running your own machines, migrating your database to a new server with more CPUs or CPU cores. Faster network connection on the new server will also help. However you choose to do it, the goal is to increase processing capacity provided by hardware and/or infrastructure.

The major pro of this approach is that by staying focused on the hardware level, you don’t have to deal with the complexities of distributed databases.

The cons are that each step you take up in computing power comes with a price tag, and you’re limited in terms of how far you can scale up by how big the biggest available machine at the time is.

Horizontal Scaling

The other way you can take things mainly means splitting the data you have into logical partitions. These parts can then be distributed across multiple CPU cores and even across multiple computing nodes. By doing so, you can change the way data is processed to make more efficient use of existing resources rather than having to buy ever higher-performance servers. If you opt for a horizontal scaling strategy, then adding hardware resources looks like adding more of the same kind of servers onto which data are distributed.

The biggest pro here is that you can scale beyond the capacity of a single machine. Also, combining multiple smaller machines is cheaper than splurging on the one really powerful machine you’d need to scale vertically.

The usual con in a horizontal scaling scenario is that distributed databases add a lot of complexity. I say usually because CouchDB’s clustering features make a lot of this go away, and instead your cluster looks like a single, but powerful and redundant server.

CouchDB is especially good at horizontal scaling. One of the ways it works around physical limitations is by using shards, the logical parts it can create and distribute. Shards also mean CouchDB has the performance benefit of shorter access paths. If you logically split shards along document ids, for example, then you can get much faster read and write performance since each shard now has a log(1/number-of-shards) shorter path to the document itself. This means you can get away with less powerful — read cheaper — hardware.

Another benefit CouchDB gains by using shards is that you can even further distribute your required computing across multiple nodes when replicas are all on their own machine.

Databases That Shard

Let’s approach databases that shard in terms of two categories: relational and NoSQL.

In relational databases — as the name suggests — various pieces of data are closely coupled with one another. This makes sharding inherently difficult because there is a lower limit to how data can be de-coupled and split in a schema. Because it’s not what they were strictly designed to do, popular relational databases like Postgres, MySQL and MSSQL rely on third-party sharding extensions. Citus for Postgres, for example, enables sharding and uses reference tables replicated to all nodes to get around distributing data relations. One notable exception to this is OracleDB, which might be the only SQL database with native sharding capabilities.

Now for the NoSQL databases. Noteworthy in this category is MongoDB, which does shard, but not automatically —you have to enable it. Also, confusingly, MongoDB conflates sharding and setting up a cluster, which can be a bit confusing.

So, CouchDB remains somewhat an outsider in that it’s one of the few database systems that shards automatically and transparently, as in: your application is none the wiser and just benefits from the additional hardware resources as you grow.

Sharding in CouchDB

In CouchDB, a shard is a file on disk with a .couch ending. You can also configure how many replicas your database will make of each shard in a scenario where you have more than one node.

When it comes to configuring your shards, there are two important values to keep in mind:

One is q, which is the number of shards per database. In CouchDB, sharding can be controlled per database and doesn’t have to be an overarching instance configuration. You can have one database in four shards, one in two shards and another with one shard, for example. The default value is 2, and we’ll cover guidelines for a limit per shard a bit later on.
The other parameter is n, which is the number of shard replicas. Shard replicas make sure you have multiple, independent copies of your data distributed across multiple nodes — which is when this parameter becomes interesting — but no node can have more than one replica of the same shard. For a single node, by default n = 1. Having multiple replicas of the same shard on a single node doesn’t have benefits for data resilience and would just waste disk space.

A database can be stored in 1…n shard files. Each shard is still limited by a single CPU, but 2 shards can be handled by 2 separate CPU cores. 4 shards can be handled by 4 cores, 8 shards by 8 cores, and so on. Very large databases can have a q of 16, 32, 64 or more.

For a cluster, the default number of shard replicas or n value is 3. So for example if you have q = 2 and n = 3 (which is the default in the cluster) you would have 6 shards in total, per database.

In CouchDB shard ranges are hashed from the document IDs using the CRC 32 hash, which achieves a relatively uniform hash distribution. What this means is that you don’t need to worry about your ID distribution, because CouchDB hashes your document IDs before adding them to a shard. The result is that shards won’t be — or will very rarely be — different in size. Usually shards are fairly evenly balanced (unless you have a lot of attachments or conflicts, but that is not a sharding concern).

Here's an example showing two shards in a database:

Shard 1: Hashed IDs 00000000-7fffffff
Shard 2: Hashed IDs 80000000-ffffffff

One shard has the hexidecimal range 00000000-7fffffff, and the other shard has the range from 80000000-ffffffff. All your document IDs will be hashed to fit into one of these ranges and each ID here has a ~50% chance of landing in either one.

A major benefit of sharding in CouchDB is more efficient view indexing. By having multiple parts of your data, you can process view indexes in parallel. CouchDB can spawn q couchjs processes, effectively doubling the throughput for each doubling of q. This works as long as the respective amount of CPU cores are available.

Doing Stuff With CouchDB Shards

Let's quickly have a look at some cool stuff you can do with shards. The most basic thing you can do is to send an HTTP GET request to your database to check its sharding parameters:

curl -s http://mynode1:5984/mydbname

{
  …
  "cluster": {
    "q": 2,
    "n": 3,
    "w": 2,
    "r": 2,
  }
  …
}

Here we can see the cluster key which tells us the value of q and of n. We can also see values for w and r, read and write quorums which, for the sake of staying on topic, we won’t go further into in this article.

We can also find out where exactly shard replicas are placed in a cluster by calling the mydbname/_shards endpoint:

curl -s http://mynode1:5984/mydbname/_shards

{
  …
  "shards": {
    "00000000-7fffffff": [
      "mynode1@localhost",
      "mynode2@localhost",
      "mynode3@localhost",
    ]
  },
  …
}

We can see that the first shard, with shard range 00000000-7fffffff is stored on node 1, 2 and 3 — exactly as it should be. In a scenario with four nodes where n = 3, it might (correctly) be 1,2 and 4.

One cool thing you can do is split an existing shard into two smaller ones using the _reshard endpoint:

curl
  -X POST http://mynode1:5984/_reshard/jobs
  -H "Content-Type: application/json"
  -d '{
    "type": "split"
    "db": mydbname",
    "range": "00000000-7fffffff"
    }'
  -s

[{"ok":true, …}]
#q will now be 3 for this database
#Keep in mind that shard ranges are now
#unbalanced unless you split the other range too.

In this request, we’re asking CouchDB to split the shard with range 00000000-7fffffff on this database:

By adding a shard we’re changing the q value to 3
And end up with three shards:
- One untouched shard, in its original size
- Two smaller shards split from one shard, equal in size to each other but half the size of the untouched shard

While this is very neat to do, don’t forget that leaving our CouchDB as-is after sending this request and keeping shards of differing sizes would not be ideal, and can lead to degraded performance. We’d want to split our other shard too and have q = 4, and as a rule of thumb, when increasing the number of shards, always ensure we’re splitting all shards and keeping them roughly equal in size.

Reducing your q value is not possible in CouchDB out-of-the-box: the _reshard endpoints only support splitting shards up, not down. But if you were hoping to read the contrary we have good news! At Neighbourhoodie we developed a little NodeJS CLI tool called couch-continuum that reduces the number of shards for you.

The procedure is not too difficult, and you could still reduce shards manually if you wanted to. It basically consists of creating a new, replica database with the new q value, replicating your data there, and then destroying the original source database, recreating it with the new q value and replicating the data back. This solution will incur some downtime. You could avoid that by updating your application code to point toward the replica database, but that’s a topic a bit beyond the scope of this article. Suffice it to say it requires some consideration.

Things to Consider in Production

The first thing we recommend is to start with the default q value of 2 (since CouchDB 3 onwards) and split your shards with the _reshard endpoint as your database grows. This takes away a lot of the stress of choosing your q value up-front, because as we’ve seen, it’s much easier to go up in q values than to go down. Providing a larger q value than you need would mean consuming more resources than you need to, for no clear benefit. CouchDB will tend to consume the resources you give it, and a large q value will set it on a hungry path. We’ll look at rough guidelines for a limit per shard in the next section.

Another rule of thumb: n should always be at least equal to the number of nodes in your cluster, but not more than 3. It’s kind of self-explanatory: you’d want your system to make use of the breadth of infrastructure you’ve made available to it, you can’t just have a full copy of all data on each shard. And also, having three copies means that if one node fails for some reason, you still have redundant access to your data, so while you wait for that node to recover or be replaced, you can still lose one more node and your application is, again, none-the-wiser. It is very unlikely that you need to protect against even more cascading errors beyond that.

Rough Guidelines For a Limit Per Shard

As a very general rule of thumb, and depending on your attachment sizes, we recommend increasing your shard limit or q value when each shard has:

1GB — 10GB active data size per shard, or
1M — 10M documents, whichever comes first.

Additionally, if you have 1000s of requests and one of your CPU cores peaks at 100%, you might run into the limit. If you observe this level of usage, it’s sensible to upscale.

Good To Know Before Sharding

There are potential downsides to sharding, depending on your use case or frequent processes you might need your system to run:

View joins
Mango
_all_docs
and _changes

tend to cause more resource usage, as each request needs to talk to q shards. Partitioning can help by collecting data with a common prefix on a single shard instead of distributing the data evenly across shards. We’ve written about how partitioning works and which use cases it’s suited to: check it out if your project will be join or _all_docs-heavy, for example.

Splitting Shards

We mentioned this before, but it’s important that your shards are of a similar size to one another in order to avoid performance quirks. You should always split shards quadratically:

2 → 4
4 → 8
8 → 16
16 → 32
…

Consider Document Design

Shards help you do some pretty cool stuff with CouchDB! But if you’re not configuring CouchDB correctly, or if you don’t have a sound and reasonably considered approach to document design, sharding won’t help you mitigate the impact of that.

One of the most important things to think about early in CouchDB is how you’ll achieve a consistent and scalable document model. CouchDB can do a lot of things, but it can’t do everything, so getting your docs in a row ( 🦆!) is your best way to ensure steady performance from the outset. We’ve written up a couple of tips to get you going:

Conclusion

Shards serve multiple purposes in CouchDB — they’re essential to its scaling strategy and data availability, and a key part of using CouchDB as a distributed system. Most importantly, you mostly won’t have to think about shards as CouchDB handles them for you transparently. Only when your application becomes a major success will you have to take them into account, and even then they’re pretty straightforward to handle and your application will not need a rewrite.

We recommend diving more deeply into the topic in the official CouchDB Docs.

This article owes a huge debt to Olivia Hugger and is based on her presentation Sharding in CouchDB for Fun & Profit which you can watch, in full, on our YouTube channel.

If you would like insights to help optimise your architecture, need a CouchDB monitoring tool, or don’t yet know what you need — give us a call. We are curious and would love to help you and your project be successful.

Discover more ways to keep your CouchDB project efficient on the Neighbourhoodie blog, where we regularly publish tips like this, plus guides and tutorials.

Sharding in CouchDB: Reducing the Number of Shards

Maddy — Wed, 15 Oct 2025 08:00:00 +0000

In contrast to increasing the number of shards for a database, reducing the number of shards is not a built-in operation. In addition, as shard splitting is only available in CouchDB 3.x and later, this advice is good for version 2.x as well.

couch-continuum

Neighbourhoodie has built the couch-continuum tool that automates the bulk changing of database parameters, including the number of shards for a database.

This tool can both increase and decrease the number of shards for a database in both CouchDB 2.x and 3.x.

There is just one caveat: it can not operate without taking the original database offline for the duration of its restore. So you can only do this during a maintenance window.

Discover more ways to keep your CouchDB project efficient on the Neighbourhoodie blog, where we regularly publish tips like this, plus guides and tutorials.

Sharding in CouchDB: Increasing the Number of Shards

Maddy — Thu, 09 Oct 2025 08:00:00 +0000

This advice is only true for CouchDB 3.0.0 or later. Next week, we’ll cover increasing the number of shards in CouchDB 2.x.

Shard Splitting

Increasing the number of shards in CouchDB is implemented by a technique called shard splitting. It allows you to specify for any one shard in a database to be split into two equal-sized shards.

The Reshard Endpoint

CouchDB does all the hard work for you: a single request to the /_reshard endpoint will start the process. Shard splitting is a background process that can go on while your CouchDB cluster is fully operational. Once the split shards are available, CouchDB starts using them to serve requests instead of the original shard, which then gets deleted.

Note that shard copies are replicated across your cluster. We recommend you split one shard at a time. In addition, we recommend to split all shards of a database around the same time in order to guarantee consistent performance.

Discover more ways to keep your CouchDB project efficient on the Neighbourhoodie blog, where we regularly publish tips like this, plus guides and tutorials.