DEV Community: Aakarshit Uppal

Book Review: Designing Data-Intensive Applications

Aakarshit Uppal — Sun, 29 Dec 2024 16:30:00 +0000

⭐⭐⭐⭐ | Mandatory reading for every engineer

After seeing major recommendations from a lot of respected folks¹, I finally decided to read DDIA this year. And right off the bat, I’ll just reaffirm many of the things I heard/read — I agree that it should be mandatory reading for engineers. I also agree that it should be read multiple times, once every few years as you progress through your career. I for one feel quite unlucky not to have come across this gem of a book sooner, but at the same time really glad that I did now!

For me the greatest value the book provided was in the form of formalization & reaffirmation — as a Tech Lead with almost 6 years of professional experience working with distributed systems, albeit more on the user end of the spectrum, I’ve collected a lot of incidental knowledge that typically comes with such experience. And reading through DDIA, one of my main observations was that Kleppmann does a really great job at formalizing this knowledge. Working at the same organization for the past few years, I have left a long trail of decisions in my wake. For me DDIA provided great validation for some of those decisions — giving me more confidence in my ability in making them, often with a healthy dose of “Whew! I sure managed to dodge that invisible bullet!” In other words, DDIA helps you reflect on your knowledge and decisions through a very clear lens of formalized knowledge.

DDIA helps you reflect on your knowledge and decisions through a very clear lens of formalized knowledge

If you’re someone with relatively less experience, a lot of things might not click the way they would if you had more experience, but just trust me when I say that you’ll be much better off knowing them now than later. The fact that the concepts & knowledge I picked up from the book have already helped me multiple times in the past few months, a couple times in very critical situations, should be evident enough. And I’m fairly positive that they’ll continue to do so for years to come.

DDIA also teaches you what you should beware of — acting as a sort of a reference of things that can and will go wrong when working with modern systems. This aspect of it is invaluable in my opinion. Imagine being able to spend just a few hours of your life to gain knowledge of things that folks only get to learn over years, often through ending up in very tricky situations. This knowledge allows you to create and improve a personal framework for architectural decision making.

Imagine being able to spend just a few hours of your life to gain knowledge of things that folks only get to learn over years

Another impact it had on me was igniting an interest in the construction and internals of distributed systems, such as databases, brokers etc. I have a feeling that this is likely to have a major impact on my career trajectory in the near future. It also widened the sphere of my understanding — I recently attended P99 Conf, and was pleasantly surprised by how much of it I was able to absorb; the same goes for Dist Sys blogs I’ve been reading recently.

All this said, the book isn’t perfect in every way — at times the focus seems to deviate from things that most require it — but that might just be an artefact of my personal experiences and preconceptions. I also found it to be a demanding read in some sections, either by virtue of trying to cram too much into too little, or at times the structuring of contents requiring tracing too many references in order to grasp some concepts. Chapter 9 comes to mind as the main offender in this regard². But note that this is only relative to the rest of the book — DDIA is by far the easiest and smoothest to read technical book I’ve read. Kleppmann (along with Chris Riccomini) is working on the second edition³ currently, and in addition to being excited by all the new concepts that’ll be covered, I’ll also look forward to some of these being addressed to make for an even better book.

… inviting you to break through the traditional approaches of designing distributed systems and give a fair consideration to the alternatives, lest they be missed in the noise

Another major goal that Kleppmann seems to accomplish is expanding your worldview on how to view and design distributed systems, inviting you to break through the traditional approaches and give a fair consideration to the alternatives, lest they be missed in the noise. I suspect his ideas played an important role in the development of things such as AtProto⁴, indicating the continuing achievement of this goal. The discussion about ethical considerations in the end was also a welcome surprise: Kleppmann didn’t have to, but he still did, which I respect and admire. It will push me to consider the consequences of the things I build or help build in a more holistic manner, and also give some bandwidth to reflecting, both internally and externally, on such consequences in my body of (future) work.

As I hinted at the start — DDIA is definitely the kind of book that gives you more every time you read it, the most deserving of this claim, at least of all the books I’ve read. I’m certainly planning to revisit it in a couple years or so. And I think you should too.

Thanks for reading! Feel free to share this with others on the butterfly site, the elephant site, the ~~bird~~ unicode character site, or maybe even the orange site?

Originally published at https://aksh.dev on December 29, 2024.

A very special thanks to Phil Eaton — he played a major role in reigniting my passion for reading via his book club, while helping me discover some iconic books. Make sure to check it out and join future runs! ↩︎
Murat describes this well in his review of chapter 9: “The chapter tries to do too much. Almost an entire semester of distributed systems content is force-jammed into this chapter … For a more manageable load, much of this chapter should be moved to other chapters.” ↩︎
Chris Riccomini on LinkedIn: Designing Data-Intensive Applications, 2nd Edition ↩︎
Kleppmann posted about this on Bluesky, and has also co-authored the paper Bluesky and the AT Protocol: Usable Decentralized Social Media. As of writing he is serving as an “Advisor to the Bluesky team” (taken from his Bluesky profile description). ↩︎

The Joy of Hackable Tools

Aakarshit Uppal — Sat, 27 Jan 2024 00:15:00 +0000

I just got my first neovim related MR merged!

#FeelsGoodMan

This got me thinking about the joy that is hackable tools: tools that allow you not just to customize, but to hack around and create stuff that you want. Customizability is becoming more and more an expectation these days, even if only at a very elementary level: it’s uncommon, for example, to find a website that doesn’t have a dark mode toggle. And I have really enjoyed customization with a lot of things, such as my android phone¹, my OS (arch, btw), my terminal², my browser³, my editors, my shell, the list goes on.

However, customization can only take you so far. Here I was customizing lualine for my neovim setup, excited about adding Code::Stats xp⁴ to it for a light dose of self gratification. But alas it wasn’t to be. After banging my head against some plugin configurations, it became clear that it was more than just a configuration issue. I was up against a bug.

Remembering lazy’s dev directory option from ThePrimeagen’s neovim revamp stream⁵, I decided to take a look under the hood of the codestats plugin⁶ itself. After tripping myself up by adding both the short name and the directory where lazy had cloned it by default (yep, it loops), I finally got a print statement in the plugin code to show up in neovim. Awesome! A few print statements later, I had the bug cornered, and a couple minutes later I had submitted a PR⁷.

The barrier to entry: A single line change

This little incident brought joy to me. The barrier to entry was actually that low: just a single line change and you’re off to the races. All of my lua knowledge was just the Learn X in Y minutes page⁸ that I had gone through quickly before starting my setup. And getting my hands dirty with the plugin code made neovim plugin creation seem so much more approachable. If I wanted x feature I could just write my own plugin, et voilà, the feature is ready!

Getting pretty good at lua

The debate on whether switching to vim et al is going to improve your productivity, or just lead to time being wasted in setting it up, is timeless. Even though I myself had been tending to the former side of this debate already, the thing that finally tipped me over was an unexpected one: the joy that comes from being able to get into the code that makes your tool work and change how a part of it functions. This joy is highly underrated in my opinion, and makes the debate entirely one-sided, at least for me.

I had been hearing a lot about Obsidian. More and more people I tend to agree with or look to for inspiration had been talking about it. I can’t place the tipping point exactly, but it was probably a mix of No Boilerplate’s pitch⁹, jonhoo’s usage of it¹⁰ and kepano’s philosophy¹¹. But there was the thing of Notion hosting all my current knowledge-base, work journal and task management. So when the official migration plugin¹² was released, I decided to take the plunge.

Obligatory Obsidian Graph Screenshot

I exported my workspace, selected it in the migration tool, and gleefully looked at all my notes getting translated and The Graph™ populating. However, once I went to see a couple notes, all was not well. My ToC were gone, and so were the intricately chosen emojis for each page. But most importantly, all my code blocks were weird and unhighlighted. I was taken aback for a moment. It was not to be after all. I remembered seeing a github link for the migration plugin, and went to see if there was an issue for this. Sure enough, there was one¹³, but it looked like it was already resolved. After double-checking the versions etc., I submitted another one¹⁴. Now to wait.

And then it clicked. It’s an open source plugin! I don’t have to wait, I can just fix it myself! Hmm, it’s typescript. I have coded all of 0 lines of javascript in my life. But maybe I can still give it a go? I just have to find and fix a bug. And the plugin being well-written, with all steps being nicely structured as separate functions, inspired some confidence. But now I’ll have to set it up :/ I don’t want a bunch of node modules on my laptop, eww. But it’s 2023, surely I can just open it up in some cloud IDE? Maybe codespaces won’t bug out this time and actually successfully open a project for once?

It did. A few minutes of tinkering around (it wasn’t a couple hours at all), and I had figured out the commands and the built package file and where to place it to make it work in my obsidian. A couple iterations of change + build + copy-paste later, I had the code blocks formatting fixed.

That joy again! The possibilities were endless: even though not that simple this time due to my complete lack of knowledge of the JS ecosystem, I could hack around and fix anything I wanted, or even add features to make things work the way I wanted: which is exactly what I did over the next few days¹⁵. Even though my MR was rejected¹⁶ in favour of the actual Notion migration contributor’s much more elegant and sensible fix¹⁷, I went on to add support for random things that bothered me about the imported notes¹⁵.

Official Version vs My Hacked Version

Again, there is the debate of whether you should get into Obsidian, with the added overhead of setting it up with plugins that might go unmaintained any day and overpriced or over-complicated syncing, or just stay with Notion and focus on doing your thing instead. Here it’s easier to tend to the former side for me. But when I went through the experience of hacking away and adding a bunch of features that resulted in an importing experience that was arguably an upgrade from Notion even visually, there was no doubt left.

Imported note before hacking | Original Notion page | Imported note after hacking

And even though right now I’m writing this blog in Notion (sorry for not giving you a perfect ending¹⁸), I’m going to switch away this year to Obsidian without a doubt. I’ve already enlisted my wife as a test subject (albeit with partial success) by forcing her to switch over from Evernote (eww) and Notion to Obsidian and setting up hacky syncing via dropbox. It’s even part of my resolution for 2024, so of course it’s bound to happen no matter what.

Can't let those 24 people down

But for now I’m going to proceed with my neovim setup, with part of me hoping to encounter a bug that I can just go ahead and fix and be on my way with a bunch of joy.

Thanks for reading, and don't forget to spread the joy! Share this on the ~~bird~~ unicode character site, the elephant site or maybe even the orange site?

Originally published at https://aksh.dev on January 21, 2024.

A_K_S_H_ | XDA Forums ↩
Cycling tabs in recently used order · kovidgoyal/kitty · Discussion #3720 · GitHub ↩
userchrome, anyone? ↩
Code::Stats ↩
Neovim Config - Part 1 - Lazy Vim - YouTube ↩
GitHub - liljaylj/codestats.nvim: Code::Stats plugin for Neovim ↩
Fixed get_xp & get_level by aksh1618 · Pull Request #3 · liljaylj/codestats.nvim · GitHub ↩
Learn Lua in Y Minutes ↩
Hack your brain with Obsidian.md - YouTube ↩
Hardware and software [2024 edition] - YouTube ↩
File over app — Steph Ango, see also: Obsidian's manifesto ↩
Free your notes - Obsidian ↩
Notion code blocks are imported without line breaks · Issue #82 · obsidianmd/obsidian-importer · GitHub ↩
Notion code blocks still being imported without line breaks · Issue #107 · obsidianmd/obsidian-importer · GitHub ↩
I got so excited that I went completely overboard and actually created a release! Release 1.3.1-aksh-0.1.0 · aksh1618/obsidian-importer · GitHub ↩
Notion: Fix codeblocks being imported without line breaks by aksh1618 · Pull Request #108 · obsidianmd/obsidian-importer · GitHub ↩
fix newlines in notion codeblocks by joshuatazrein · Pull Request #118 · obsidianmd/obsidian-importer · GitHub ↩
I did end up editing it in Obsidian before publishing, so I guess there was a happy ending after all! I wanted to go for footnotes instead of links for this one to enhance readability, and a couple of Obsidian plugins came in really handy ↩