DEV Community: Anthony Dodd

Distributed State Management - Without the "E"

Anthony Dodd — Wed, 07 Apr 2021 06:37:42 +0000

Come, let us journey into the depths of software, into the heart of application data and state management. We will explore an event-less pattern, relying only on our trusty old friends Postgres and gRPC.

On this journey, we will explore how a software team can manage their data in a distributed microservices architecture, where state "ownership" is divided between various microservices.

Setting the Stage

To set the stage, let's say that the software team we are journeying with is in the business of hosting a key/value store for customers — let's call it KVX — and runs everything in the cloud on a Kubernetes platform.

This team of ours loves Postgres (who doesn't), is pretty good with gRPC, but is not at all into the world of events, event processing, EDA, none of that.

This is presented as our challenge on this journey. Anything to do with events seems to be a serious blocker. We need to find a way to build a robust platform, but if our solution has anything to do with eventing patterns, then we've failed.

Break it Down

If we can't dig into anything related to the "E" word, then we are left with a somewhat narrow scope. Various technologies like Kafka, Nats, RabbitMQ, and any of the myriad other tools in the ecosystem which may help on this front, all of that is off the table.

So let's get down to the basics. First principals. What are we even trying to do? What is the problem which needs to be solved? Here's what we know:

We need to propagate state changes from one transactional service to others, maybe even call a few external cloud provider APIs and the like.
We're pretty good at handling requests using gRPC.
We can persist data in Postgres using transactions pretty well.
Responding to customer requests, we're pretty good at that too.

So what's the problem then? Well, this whole microservices thing. Often times when a microservice finishes updating its database tables, we need to do some asynchronous work. Ok, we've recorded what the user has requested, but now we need to actually make some things happen. We need to provision some KVX instances, we need to make some calls to our cloud provider to provision some infrastructure, we need to make some real things happen.

To state the problem succinctly: we need to drive the state of our system to match the state which the user has requested. The aspired state needs to become the current state.

Take a Shot

Given our constraints about the "E" word and all, our options are somewhat limited. The only tools we really have to work with are our trusty old Postgres database, gRPC ... and well, that's about it. Let's see what we can do.

When a user tells us that they want to provision a new KVX instance on our cloud platform, we need to be sure that we actually record their request in Postgres. This algorithm is simple enough:

A request comes in on one of our gRPC services, so we handle the request,
Next we open a Postgres transaction (we're good at this) and write some data,
Once we've committed our transaction, we respond to the user telling them that they've received their request and we'll get started on provisioning their new KVX instance right away.

We've responded to the user, we know what they want. Now what? How do we do something with that data in Postgres? It's just sitting there.

Work Templates

Because we are avoiding anything to do with the "E" word, we need to keep our solution simple. Along with our standard data models, we will be creating records in our database called "work templates", and we will store these records in a table called work. To make things even more simple, let's say we store the specification of the work which needs to be done as JSON in a JSONB column in our work table. Let's say the table looks like:

                                     Table "work"
 Column |           Type           | Nullable |             Default
-------------+--------------------------+----------+----------------------------------
 id     | bigint                   | not null | nextval('work_id_seq'::regclass)
 time   | timestamp with time zone | not null | now()
 spec   | jsonb                    | not null |
Indexes:
    "work_pkey" PRIMARY KEY, btree (id)

With this approach, we will need to have some tasks in our code which poll Postgres periodically to check for work to be done. If we're feeling advanced, we could use Postgres LISTEN/NOTIFY to make things a bit more real-time. Our worker tasks will need to lock a row in the work table in order to process it, maybe just using a simple DELETE FROM work WHERE id=ANY(SELECT id FROM work ORDER BY id LIMIT 1) RETURNING *; query within a transaction.

While that record is held in limbo by our transaction, we can now safely begin our work without worrying about another worker attempting to process the same record. If some error takes place and we need to rollback, that work will be available for later processing. If we break down our units of work small enough we can apply the following pattern:

Grab a record from our work table, and hold onto that record in our transaction.
Do the work described therein. Maybe we call a few other microservices, call Kubernetes, update a few other database rows.
If there is additional work which needs to be performed, but we want it to be its own isolated unit of work, we can write a new record to our work table describing the work to be done, which will be picked up later by another worker.
Finally we will commit our transaction, and that unit of work will now be done.

Nice! A successful eve... I mean, work processing implementation. Let's look at the edge cases.

Errors

We've got a few different types of errors that we need to worry about here. On the highest of levels, we can categorize our errors as being either transient or intransient errors.

Transient errors are things like network blips, VM restarts, pods being suddenly rescheduled in Kubernetes and work being interrupted. The idea is that these errors are not permanent. The next time your code is able to execute, things will quite likely work as they should.

Intransient errors are errors which will not resolve themselves. Someone misspelled the name of our werk table, someone forgot to commit a transaction, maybe some boolean logic was wrong, so on and so forth. An engineer will typically need to fix the bug and deploy the updated code. Once the code has been deployed, the expectation is that the system will begin to work as needed once again.

Great, so error handling is covered. Worst case scenario, we need to deploy some bug fixes, but then things will be back to normal.

Idempotence

This is a slightly more of a difficult issue. In order to make our workers idempotent, we will need to ensure that our algorithms expect to be retried. We will need to account for the fact that the various other microservices or external APIs (cloud providers, Kubernetes &c) may have already been successfully called as part of this unit of work, but that a failure may have taken place after the fact.

It is important to note that retires are critical for state consistency. If we are actually attempting to drive our system's state to match the user's requested state, then we can't just give up at the first sign of trouble. Think about how terrible of an experience that would be for the user. What's worse? Think about all of the inconsistent half-state we would have sitting around. Consider this:

If we run into an error after successfully provisioning a new EC2 instance for a user in AWS, but then we just decided to abandon our work and not drive it to completion because we ran into some transient error after the fact, we would end up with lots of orphaned EC2 instances in AWS ... and we would also end up with a LARGE bill at the end of the month.
Similarly, if we are making gRPC requests to another microservice from our worker, and that peer microservice has already committed a transaction as part of our gRPC request, but then we decide to rollback our work because of some transient error after the fact, we now have a partially propagated state change. We can't just leave it there. What if that service was responsible for billing our customers. They get charged even though their KVX service was never provisioned? Not a good idea.

The point is hopefully now clear. Retries are an integral part of ensuring that we reliably propagate state changes throughout the system. In order to make this work properly, we need to ensure that our algorithms are crafted to expect such conditions.

As a final note on this subject, if one of our units of work is not safe to retry — E.G., sending emails — it is often the case that email providers offer their own idempotency mechanism to guard against duplicate requests. If such is not available, then we are by definition dealing with an "at most once" constraint, and the algorithm must be crafted to ensure that the critical section will never be retried. The solution can be simple: commit your transaction first, then try to send the email. If it fails, well ... the user will have to request that a new email be sent. Really, just use an email provider which provides an idempotent interface.

Synchronization

Awesome, we've covered some serious ground, we even have an idempotent eve... I mean, worker system. What next? Well, we need to be sure that we avoid race conditions.

Race conditions are endemic to distributed systems. Engineers need to stay vigilant to ensure that their algorithms are race condition free. This is especially a problem with distributed systems because the scope is no longer isolated to an individual process. We are now dealing with lots of different microservices, all of them communicating with each other at various points in time, updating state and making various sorts of changes to the system.

Many different conditions can trigger race conditions. Nearly any user-facing feature is subject to the possibility of race conditions. These can be especially dangerous when we are dealing with money, infrastructure, and other things which are more than just "data".

We are using Postgres for everything, so this is actually an easy problem for us to solve. Any work initiated in our system related to a customer's KVX instance will simply cause the has_active_work column to be set to true for the respective KVX instance. This update takes place as part of the transaction initiated by a user request, and the update takes place directly on the database row for that specific KVX instance.

If a user attempts to make another request to change some aspect of their KVX instance, we can kindly inform them that they still have some active work taking place for their KVX instance, and that they should try again when it is finished. Presenting a nice UI on top of this data is quite simple.

Once a worker task has finished all of its work, as part of its transaction it will set that has_active_work column to false for the target KVX instance.

Problem solved. We lock out race conditions before they have a chance to get into the system. However, we need to remain vigilant! Bugs can always creep in. A little documentation and peer review can go a long way to keep this pattern robust.

Stigma

Wow, that's got to be everything right? We've got a production grade eve... I mean, worker system now. But what will everyone else say about this?

There is some stigma associated with this approach. Typically, the criticism is that this approach espouses tight coupling between services. If a worker in service-abc (a microservice) needs to call service-xyz (another microservice), then we have created a dependency chain. In order for service-abc to function, service-xyz and any other services it depends upon must be available. This happens to also be true for any external 3rd party APIs such as AWS, Kubernetes, email providers, or any other such services which our workers in service-abc depend upon.

Practically speaking, this is inescapable for 3rd party resources which a team might depend upon. If a team needs to orchestrate the AWS API ... it doesn't matter how deep you burry that code under layers of abstraction ... sooner or later, the AWS API will need to be called. Is anyone really trying to remove that constraint? What would that even practically mean? A new dependency, that's what it would mean.

Maybe this isn't such an issue after all. We can generate alerts and observability data when we are having trouble accessing our dependencies. Maybe having explicit dependencies can be a good thing. Let's roll with that perspective for now.

Scale

Perhaps a criticism might be that the "work template" pattern just won't scale. All of those gRPC calls. All of that dependency management. However, none of these are concrete arguments. Data flowing over a network is going to be common to any distributed system (practically speaking), and dependencies will always exist no matter where you put them or how deep you burry them.

"Well ... Postgres won't be able to handle all of those worker queries at scale." I would counter by saying that Postgres can be quite surprising. Benchmarks would be needed; however, there is truly no doubt that even a single Postgres database instance can handle a surprising scale of throughput. Maybe if this were to become an issue, we could look into CockroachDB, TimescaleDB, or one of the other alternatives.

PID Controllers

A final note that I would like to make on this subject overall is that this worker system we've defined reminds me a bit of a simple proportional controller — a type of PID controller — which is far more common in the embedded systems space.

The templates of work can be seen as an error signal. The system is currently in state "x", but the user wants it to be in state "y", do some work to bring the system into state "y". With this model, an entire system can be seen as a series of controllers reconciling their observed state with the desired state, and then they take action to drive the system to the desired state.

Conclusion

We did it. All that's left to do is to give some feedback. What do you think? Would you build a system like this? What are we missing?

Any and all feedback is welcome! Cheers 🍻

Trunk 0.7.0 | Stable Pipeline API | Future Goals

Anthony Dodd — Thu, 08 Oct 2020 03:30:22 +0000

Build, bundle & ship your Rust WASM application to the web.

Github repo
Release notes 0.7.0
Release notes 0.6.0
We now have a brew package available. Just brew install trunk. Works on MacOS, Linux & Windows (WSL). Woot woot!

This release covers the 0.6.0 & the 0.7.0 releases of Trunk. We'll get to the nitty-gritty of these releases shortly, but I wanted to take a moment to talk about where Trunk is at now and where myself and others in the community would like to take Trunk in the future.

First of all, and most importantly, I believe we have finally found a perfect pattern for declaring asset pipelines in the source HTML (typically a project's index.html). The pattern which is now the standard as of the 0.7.0 release is as follows: <link data-trunk rel="{pipelineType}" data-other data-attrs data-here />. Let's break this down.

Every asset pipeline, everything Trunk does in relation assets, is now controlled by normal HTML link tags with the special data-trunk attribute. This simple mechanism makes clear which links are intended for Trunk to process and which are not.
Trunk links must have the rel attribute, a living standard HTML attribute which normally declares the relationship of the referenced resource to the HTML document. In our case, this represents the Trunk pipeline type. See the Trunk README #assets section for more details on supported assets and their attributes.
Speaking of attributes, all pipeline types will require some number of attributes. Most require the standard href attribute to reference some target file. Others take non-standard attributes which are always prefixed with data-. E.G., the rel="rust" pipeline type is an optional pipeline type, if omitted Trunk will use the Cargo.toml in the source HTML's directory; however, if your cargo project exposes multiple binaries, you will need to specify which binary Trunk should use. This pipeline type supports the data-bin="bin-name" attribute for exactly that reason. Check out the aforementioned assets section in the README for more details.

Awesome! I'm very excited about how much this pipeline API has evolved. It is VERY extensible (what can't you specify via data-* attributes?), and my hope is that this pipeline API will ultimately become the 1.0 API for Trunk. However, Trunk is a young project, and still has a long way to go. Let's talk about the future.

Trunk's Future

There are lots of great features the Trunk community has been discussing, a few notable ones:

support for automatic browser reloading via WebSockets or SSE. This is definitely par for the course as far as web bundlers are concerned.
WASM HMR (hot module reloading). This is just an extension of the above, however there is a lot of awesome potential here. Building Rust WASM web applications is quite a lot of fun these days.
inline CSS, JS and other applicable asset types. This will be an easy extension to the new pipelines API discussed above. For most of these asset types, it will be a simple data-inline attribute, and Trunk should be able to generate the necessary code to have the associated asset inlined.
CSS components pattern. This is something which I personally think would be pretty cool. For those of you that remember EmberJS from back in the day, they had a nice feature where one could just place their CSS right next to their components, and they would be concatenated and served for you. Easy lift for Trunk, and folks might find it quite useful.
A BIG LIFT: a Trunk library which will allow folks to declare assets directly in their Rust code right next to their WASM components. Already lots of discussion on this feature, still some planning and design work to do.

I've said a lot, so I'll say one last thing here: Trunk is an excellent project to get involved with! The new pipeline API has come along with an awesome refactor of the internal layout of the code. Adding new asset pipelines and pipeline extensions is easy and enjoyable! This community would be even better with you involved! Cheers mate! Let's do this!

the nitty-gritty

0.7.0

changed

All assets which are to be processed by trunk must now be declared as HTML link elements as such: <link data-trunk rel="rust|sass|css|icon|copy-file|..." data-attr0 data-attr1/>. The links may appear anywhere in the HTML and Trunk will process them and replace them or delete them based on the associated pipeline's output. If the link element does not have the data-trunk attribute, it will not be processed.

fixed

Fixed #50: the ability to copy a file or an entire dir into the dist dir is now supported with two different pipeline types: <link data-trunk rel="copy-file" href="target/file"/> and <link data-trunk rel="copy-dir" href="target/dir"/> respectively.

removed

The manifest-path option has been removed from all Trunk subcommands. The path to the Cargo.toml is now specified in the source HTML as <link rel="rust" href="path/to/Cargo.toml"/>. The href="..." attribute may be omitted, in which case Trunk will look for a Cargo.toml within the same directory as the source HTML. If the href attribute points to a directory, Trunk will look for the Cargo.toml file in that directory.

0.6.0

added

Closed #59: Support for writing the public URL (--public-url) to the HTML output.

fixed

Closed #62: Improved handling of file paths declared in the source index.html to avoid issues on Windows.
Closed #58: The output WASM file generated from the cargo build is now determined purely based on a JSON build plan provided from cargo itself. This will help to provide a more stable pattern for finding build artifacts. If you were running into issues where Trunk was not able to find the WASM file built from cargo due to hyphens or underscores in the name, that problem should now be a thing of the past.
The default location of the dist dir has been slightly modified. The dist dir will now default to being generated in the parent dir of cargo's target dir. This helps to make behavior a bit more consistent when executing trunk for locations other than the CWD.
Fixed an issue where paths declared in a Trunk.toml file were not being treated as relative to the file itself.

Trunk 0.5.0 | Proxy System

Anthony Dodd — Wed, 23 Sep 2020 13:21:01 +0000

Build, bundle & ship your Rust WASM application to the web.

Trunk now ships with a built-in proxy which can be enabled when running trunk serve. There are two ways to configure the proxy, each discussed below. All Trunk proxies will transparently pass along the request body, headers, and query parameters to the proxy backend.

proxy cli flags

The trunk serve command accepts two proxy related flags.

--proxy-backend specifies the URL of the backend server to which requests should be proxied. The URI segment of the given URL will be used as the path on the Trunk server to handle proxy requests. E.G., trunk serve --proxy-backend=http://localhost:9000/api/ will proxy any requests received on the path /api/ to the server listening at http://localhost:9000/api/. Further path segments or query parameters will be seamlessly passed along.

--proxy-rewrite specifies an alternative URI on which the Trunk server is to listen for proxy requests. Any requests received on the given URI will be rewritten to match the URI of the proxy backend, effectively stripping the rewrite prefix. E.G., trunk serve --proxy-backend=http://localhost:9000/ --proxy-rewrite=/api/ will proxy any requests received on /api/ over to http://localhost:9000/ with the /api/ prefix stripped from the request, while everything following the /api/ prefix will be left unchanged.

config file

The Trunk.toml config file accepts multiple [[proxy]] sections, which allows for multiple proxies to be configured. Each section requires at least the backend field, and optionally accepts the rewrite field, both corresponding to the --proxy-* CLI flags discussed above.

As it is with other Trunk config, a proxy declared via CLI will take final precedence and will cause any config file proxies to be ignored, even if there are multiple proxies declared in the config file.

The following is a snippet from the Trunk.toml file in this repo:

[[proxy]]
rewrite = "/api/v1/"
backend = "http://localhost:9000/"

Announcing YBC | Yew Bulma Components

Anthony Dodd — Fri, 18 Sep 2020 20:20:17 +0000

YBC is a Yew component library based on the Bulma CSS framework.

YBC encapsulates all of the structure, style and functionality of the Bulma CSS framework as a set of Yew components. YBC also ships with support for the Yew Router, adding Bulma-styled components which wrap the Yew Router components for clean integration.

As a guiding principal, YBC does not attempt to encapsulate every single Bulma style as a Rust type, let alone the many valid style combinations. That would be far too complex, and probably limiting to the user in many ways. Instead, YBC handles strucutre, required classes, functionality, sane defaults and every component can be customized with any additional classes for an exact look and feel.

To get started with YBC, have a look at the Getting Started guide in the README. A few pertinent highlights:

YBC works out of the box with Bulma CSS. Add <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.0/css/bulma.min.css"/> to your HTML, and then you're ready to start using YBC.
YBC also supports full customization using Bulma's recommended customization pattern. Details here. TL;DR, use Trunk for building & bundling your app. It will handle compiling your scss/sass, which is what you will use for customizing Bulma.

Let me know what you think. The hope is that this crate will make building web apps with Rust WASM that much easier. Cheers!

Trunk 0.4.0 | Layered config (Trunk.toml), JS Snippets & release binaries

Anthony Dodd — Fri, 18 Sep 2020 20:17:18 +0000

Build, bundle & ship your Rust WASM application to the web.

Pertinent Highlights

In addition to CLI arguments and options, Trunk now supports layered configuration via Trunk.toml & environment variables.
There is an example Trunk.toml to the root of the repository showing all possible config values along with their defaults. Link above.
JS snippets generated by wasm-bindgen are now fully supported by Trunk. They should have been since the 0.1.0, but I overlooked that feature :). Docs for wasm-bindgen JS snippets.
Release binaries will now be uploaded to the Github release page for every future release. This is great for CI/CD. Brew formulae for Mac/Linux, and Choco package for Windows are in the works.

Announcing Trunk — Build, bundle & ship your Rust WASM application to the web.

Anthony Dodd — Wed, 26 Aug 2020 03:53:45 +0000

I am happy to announce the very first release of Trunk. Trunk is a CLI tool, written in Rust, which provides a simple, zero-config pattern for building Rust WebAssembly applications, bundling application assets (sass, css, images &c) and shipping it all to the web.

Trunk is designed for creating progressive, single-page web applications, written in Rust, compiled to WebAssembly, without any JS (though today JS is still needed for loading WASM modules). Trunk follows a simple paradigm: declare an index.html file describing the single page of your application, then Trunk will parallelize bundling assets declared in your HTML, will build your WASM app, hash resources for cache control ... all without any extra config files.

If you are interested in getting involved (which I hope you are), I would love for you to help out. There are lots of great features planned, and many still in the design phase. I hope you will stop by, give the issues a read, share any thoughts if you feel so inclined, and if you want to write some code, please do!

”Pack your things, we’re going on an adventure!” ~ Ferris

Announcing async-raft v0.5.0: Rebased onto Tokio & Renamed to `async-raft`

Anthony Dodd — Tue, 18 Aug 2020 03:07:53 +0000

I am pleased to announce the release of async-raft v0.5.0.

The TLDR:

Everything Is Now Based on Tokio
Singular & Well Defined API (thanks to async/await, async_trait & tokio)
Automatic Log Compaction
Joint Consensus Overhaul
Project Renamed to async-raft (was previously actix-raft)

For more info, here are the release notes https://github.com/async-raft/async-raft/releases/tag/v0.5.0