DEV Community: David Zhang

Fathom: On RPC

David Zhang — Tue, 07 Jun 2022 22:49:55 +0000

This is the third part of a series of devlogs. I invite you to take a look at the first post for the project brief if you're confused about why Fathom exists at all.

You can also keep track of the project's progress at the GitHub repository for the project.

Hello! It's been a hot minute. I lost power for a week, and had a whole host of other issues to deal with these past few days. Nevertheless, here we are!

In honour of the finalization of the HTTP/3 spec (IETF RFC 9144) yesterday, I figured I'd announce (such as it is) my decision to write a 'custom' RPC implementation over QUIC, HTTP/3's new transport protocol intended to replace TCP. This has been the result of several days' thinking and experimentation, and I'd like to discuss my reasoning for this decision in this post. At their most basic, it boils down to performance, comprehension, and control.

However, before we start, I want to give a brief explanation of what RPC is, and why it's important. Simply, RPC stands for Remote Procedure Call, and is an abstraction over some form of call-and-response mechanism that allows two (or more) processes to interact with each other, much like class methods allow two objects to interact with one another. As long as one object (or process) implements an interface, other entities can interact with that object or process through that interface. It's like Object Oriented Programming except your objects are processes!

What that out of the way, let's talk about the biggest reason to create a custom RPC framework is needed. Most simply, it is control. I want to be able to plunge through the depths of the RPC code to understand why something performs the way that it does, and then be able to do something about it. It is my experience so far that the larger the library and the greater its complexity, the more opaque it is to the developer. Creating a custom framework permits me to implement the absolute simplest code necessary to achieve the functionality that I need without having to worry about everything else that comes with the library.

Everything else follows from that decision. Latency, throughput, and power consumption (all grouped together into 'performance') are more easily understood and dissected when you have control over the code base and can freely plunge the depths whenever you feel like it, making changes as best suits your use case. And the comprehensibility of a single-use code base is almost always superior to that of a general purpose library.

This might all sound like NIH syndrome, and indeed I certainly detect some whiffs of that kind of reasoning myself. However, since this is a learning project, I am prepared to tolerate some of that kind of reasoning here, at least for the moment.

Whilst that may be the case, I think it is nevertheless important to have a discussion of what this means for the project as a whole. First and foremost, this will severely impact the velocity of the project. The time before user-facing features will be visible are extended indefinitely until the basic RPC framework is complete, a non-trivial task by itself. This has knock-on effects, including the increasing likelihood of waning interest of you the reader as the project moves more slowly than is perhaps desirable. It also increases the likelihood that something else will come up, and (having limited functionality), I will drop this project in favour of something else.

These are all important considerations to make, and I am uncertain how to handle them. If you have any experiences and stories about make/use decisions, or anything else you'd like to share, please leave a comment!

Moving on from RPC, I think it's worthwhile to discuss how things have been going in a more general sense. For one thing, you may note that there appears to have been a consolidation of effort into app-server, especially in the test branch. This is a phenomenon that I have noticed often in my projects, where all efforts at organizing things top-down inevitably fail. You'd think I'd have learned by now!

I think that this is a symptom of a sort of uninformed flailing that I tend to do when I don't really know what's going on. Early in a project, when the true complexity of what the project seeks to accomplish is not evident and only the broad-strokes components are visible, it's easy to partition things into logical sections. However, this kind of partitioning inevitably affects the way I think about modules, with mental frames that have to be 'loaded' and 'unloaded' when moving between them. Just like context switches, except defined artificially with no real benefit early on.

As projects grow, it's nice to be able to reason about small units of code independently (just as with microservices), but early development really seems to suffer; even more than I initially thought. With Fathom, every change I made in terms of API primitives or separation of concerns inevitably led me to touch many separate crates all at once, imposing multiple context switches and massive overhead. So I decided to once again contract the code, this time entirely within app-server and app-desktop. With the current pace of development, anything that can improve the pace of development is very welcome.

As for the tasks for the coming days, the priority is going to be getting the basic RPC system up and running. I have (many) ideas about how it's going to work, and the next post will probably be a write-up about it design. The goal here is to have a final, runnable example working by the end of the week, and a write-up complete and posted sometime next week.

What's Your Preferred RPC Mechanism and Why?

David Zhang — Thu, 26 May 2022 20:58:23 +0000

What it says on the tin. What kind of mechanism do you prefer for your web APIs? Where would you use it? Where wouldn't you?

Examples:

gRPC
JSON over HTTP (RESTful, usually)
Binary over WebSockets
Custom UDP (e.g. for real-time game networking?)

Fathom: So... Microservices

David Zhang — Thu, 19 May 2022 18:31:03 +0000

This is the second post in a series of devlogs. I invite you to go take a look at the first post for the project brief if you're confused about why Fathom exists at all.

As I've begun this project, the most pressing concern has been exploring the architecture design space, and selecting one with that will inhibit growth the least as Fathom grows. The phrasing here is deliberate, since any architectural decisions made now are invariably made with incomplete information, their founding assumptions made partially or completely incorrect.

The project brief explicitly describes a microservices-based architecture, which presents an interesting challenge. In spite of its many benefits, microservices by their nature have an up-front cost implicit in their adoption; that of the bring-up/strike (to use an AV term) of the services repeatedly during early development. More concretely, since Fathom barely does anything at the moment, it is guaranteed that inter-module interfaces will need to be refactored or rewritten. Until that API can stabilize at least a little, it will be necessary to take down and restart multiple services on every debug-compile-run cycle; a significant drag on productivity and a source of developer friction.

This consideration has implications, in that it is currently (in my mind at least) better to continue with a monolithic application until Fathom reaches MVP (at the least) at which point I'll have an existing API to analyze and work from, if only for the reduced development friction. This does not mean, however, that we cannot co-opt some strategies from microservices in order to improve Fathom's architecture. Indeed, the current app/endpoint/service/base split across crate (Rust's unit of compilation) boundaries and the explicit per-crate declaration of dependencies should go a long way to enforcing a clear separation that may be useful when it comes time to split off individual microservices. We can (and indeed must) go a bit further by establishing a few simple rules about which crates can depend on which other crates:

Apps (such as the server crate) may depend only on endpoints.
Endpoints may depend only on the comm crate
Services may depend only on the comm crate.

Aside from architecting the individual components of the back-end, the API that the web interface uses to communicate with Fathom has been on my mind. A number of approaches are available, from REST with JSON, to Websockets, to gRPC. In every case, the question is one of per-message efficiency, especially for binary data. However, they all nevertheless share one problem that I am as yet not sure how to deal with: that of defining the message types across both sides of the application. The ideal situation would be to generate them out of an external spec, but the details of that remain unclear to me. Furthermore, with the current separation between the web and rest crates, any attempt at defining an API in rest would cause web to depend upon it, violating rule 2. Adding another shared dependency ws-comm that auto-generates stubs for both sides is certainly an option, however.

Either way, the selection of a web<->server communication method is currently blocking all further progress, so that's going to be the focus of the next few days. I'm growing increasingly fond of the ws-comm approach, though I am wary of the increased complexity of that that will come with it. Once that's done, I hope to have a basic log-in screen, followed by a browser-only file tree populated by drag & drop implemented over the coming week. We'll see how that goes!

If you want to keep track of my moment-to-moment progress, feel free to take a look at the GitHub repository for the project.

Also, if you have any questions, comments, or would like to share your own experience with what' I'm talking about, please feel free to leave a comment!

Fathom: Trying a New Hard Thing.

David Zhang — Wed, 11 May 2022 02:59:41 +0000

So named because I cannot fathom the fathomless depths of this rabbit trail to fathom the brilliant complexity of building web apps.

More seriously, hi 👋! Welcome to the first of what will hopefully be a series of posts about a new project I'm working on in order to learn a whole bunch of exciting web technology. This first post will discuss what the project is about, why it exists, and some rough ideas about how it's gonna go down. Subsequent posts will focus more on the actual project, with research notes and project updates on what is likely to be a once-a-week schedule.

So without further ado, let's begin!

So what is Fathom going to be?

Fathom is going to be a multi-tenant file sync service that is quick to deploy and comes with desktop clients supporting file placeholders. That is to say, it:

Works as a file sync service like OneDrive/Dropbox/etc.
Can be deployed on a home server (or a Raspberry Pi).
Supports multiple users with their own private directories.
Allows certain directories to be shared between users.
Provides sharing features so users can show off their prized Korean recipe collection from the comfort of their own home.
Comes with a Windows desktop sync client that makes use of placeholder files, so you don't have to have all the files on your computer.
Provides a web interface for convenient CRUD (Create, Read, Update, Delete) operations on the go, and for managing the app & users.
Correctly detects merge conflicts and prompts users to deal with it.

All that seems like a lot of work, which begs the question:

Why?

To learn! And maybe get a home cloud solution that does everything I want it to. To share this experience with others. I've spent most of my time working on projects that lived only on one computer, and I feel it's high time I get up to date on web, or at least have something useful to contribute to discussions about the web. Documenting the process in public will hopefully help others who want to embark upon a similar journey, and certainly allow for some amount of schadenfreude as I pull my hair out over some weird heisenbug.

With all that said, I'm targeting a few specific learning goals which will be the underpinning of this entire endeavour. They are to:

learn how to use Docker & Kubernetes.
learn how to create and work with microservices.
learn how to create an application-specific web server with Rust's Tokio/Tower/Hyper libraries.
learn how to create posts just like this one, only better.

... I don't know about you, but that second list seems like it could be accomplished with a lot less work. But! Fathom has features that I want in a way that I'm interested in working on, and when it comes to learning, what captures your attention will easier to actually sit down and work on (or so I've found). Granted, I might just find this project too big to chew on, but we'll cross that bridge if we get to it.

In any case, on to the

Now What?

Now I get to cracking figuring out just how this project is going to take shape. There will likely be a fair few 'research report' style posts, as well as status updates and design documents in the coming months as I work things out. Certainly, the feature list I mentioned at the start of the post sets out a list of features that are going to be necessary, such as account management and storage.

The next concrete step is going to be getting a basic HTTP server running in Rust, followed by a super basic file browser for drag-and-dropped directories that doesn't actually upload anything but file and directory names. I'll hopefully have a better idea about what kind of work needs to be done, and what components are going to be needed by that point.

This is intentionally a very, very early place to start talking about a project, and many projects don't really make it past this phase. However, I think even that is important to document since I've definitely encountered feelings of shame or insufficiency when a project I was thinking about had to be put aside for something else. If this project succeeds, great! Documentation of the entire process (warts and all) will be available for public scrutiny. If it fails, also great! Fathom will join the countless other examples strewn across the internet of projects that never took off; living (or dead) proof that it's alright for these kinds of things to fail.

Irregardless, I'm off to work on this thing.

PS: If you want to keep track of my moment-to-moment progress, feel free to take a look at the GitHub repository for the project.

PPS: How was the post? Too long, too short, too empty, too dense? Please do let me know!