DEV Community: Jonathan E. Magen

A Compendium of Gen-Z Terminology in Technical Contexts

Jonathan E. Magen — Wed, 29 May 2024 22:20:54 +0000

I wonder if others like me, other non-sociologists, also find the whole notion of a generation irrelevant. Seems that more folks use generational designations to broadly label, disrespect, or deride others. So I (an elder-millenial, for those looking to label) would prefer to tell you how much my Gen-Z colleagues have taught me. Hopefully, when others have the chance to develop the appreciation I have gained.

To facilitate this, I have collected yet another list of terms used by my contemporaries. However, thre are two differences. One difference is that I only include that which I have personally experienced. The other is that these expressions have been presented specifically in the context of technology and computing.

Any essential expressions which are missing, are missing because I've never run into them myself.

Expressions of Words and Speech

Bye: A response to something exceedingly humorous, as if to say something is so funny that a politely exiting the situation is one's only reprieve.
Cap / No Cap: A lie. One might use this as others might say "for real" or "honestly" as in the example "That code worked on my machine, no cap!"
Dead: When someone says "I'm dead" they wish to indicate they have experienced extreme laughter or amusement. Seems to descend from the expression "died laughing".
Drag: To harshly criticize to the point of insult. A self-deprecating example might be "Thank goodness I have ESLint to drag my code."
Extra: Intense or otherwise noticable, usually positive. Can be used sarcastically in sentences such as "With so much company surveillance software on my computer, no wonder the fan is always extra."
FR: Voiced or written abbreviation for "for real".
Gatekeep: To hoard useful knowledge. This differs from its previous usage of denying others' belonging in a given group.
Here for it: Used to indicate a satisfactory approval such that it is, alone, enough motivation to justify one's presence.
High-Key and Low-Key: Either way this is an admission of approval or adoration in the face of relative distate within a given social group. It might be used conspiratorially to express an unpopular opinion "I low-key love Java". Alternatively you can use it as a proud declaration of enthusiasm in opposition to the conventional wisdom when saying "I high-key think Windows is right for this".
Hits different: If something hits different, it is uniquely positive. You may here it as "I thought Python had great docs but the Rust documentation hits different."
I love that for you: This term indicates both sincere and personal enthusiasm. It may be paired with a compliment in phrases such as "You are so good at explaining algorithms. I love that for you!" Though, it may also be used alone as a compliment.
IYKYK: An abbreviation, usually written, for "if you know, you know". This most often refers to insider knowledge or understanding, but could also highlight the presence of an in-joke.
Jail: Strong disapproval in response to learning of something undesirable. It most likely originated as shorthand for "that's so terrible, it should be imprisoned". If you were to tell how a cloud platform deleted virtual machines without warning, you might hear the "jail" in response.
Main Character / Main Character Energy: A person who has main character energy has become, positively, the center of attention. Someone who is the prime mover of, or instigator in, a situation.
MFRN / MFW: Written abbreviations for "my face right now" and "my face when". Always coupled with a graphical depiction such an expressive emoji, GIF, or other image.
Obsessed: A spoken declaration of enthusiastic approval usually sprinkled throughout conversation. Those who hear it may safely macroexpand it to "I'm obsessed with that". It could be used in the same way as someone listening would say "uh-huh" or "go on" to informally confirm that their attention remains focused on the speaker.
Sending me: Used to indicate when someone or something induces amusement, as in "Our tech lead's source comment in the integration code absolutely sent me."
Sheesh: An exclamation that someone is "looking good" or seems impressive. Often used alone as a reaction or for emphasis in usages like "Did you see her bugfix? Sheesh!"
Simp: Anything or anyone noticeably enamored with a given subject. Can be used appreciatively or cheekily in sentences like "The Rust compiler catches so many bugs for me, what a simp!"
Sus: Short for "suspicious" and used to indicate distrust or disbelief. You might hear it stated that "It's sus because the tests pass but it produces the wrong output in production."

Graphical or Pictoral Expressions

It has been my experience that GIF reactions or messages are usually employed sparingly. On the other hand, emoji may be used liberally albeit with differing connotations.

💀: See: Dead
😭: As if to day: "I'm laughing so hard that I'm crying!"
✍️: Communicates that the sender is paying attention so closely, it's as if they're taking notes.

If SAFe was a programming language, would you use it?

Jonathan E. Magen — Mon, 20 Feb 2023 14:05:22 +0000

Even with all of the many criticisms of the Scaled Agile Framework (SAFe), people still seem to use it. So let's do a thought experiment and ask the question, "if SAFe were a programming language, would you use it?"

To answer this question, let's hypothesize a satirical programming language.

Introducing SafeScript: A DSL for building valuable software using the SAFe methodology

SafeScript is a domain-specific language (DSL) geared toward building valuable software using the SAFe methodology.
It is a statically typed, purely functional, expression-based language executed by a special interpreter called the Execution Train Engineer (ETE).

Providing the SafeScript ETE with a SafeScript program enables you to get good, cheap, and fast execution thanks to its fully reversible design.

Design principles

Modular and functional: The core building blocks of SafeScript are functions and modules.
Functions can be contained within modules and modules are contained within files. Logic may be contained within either functions or modules, but modules cannot be within functions, though don't forget they are in files. Logic may not live outside of a function unless inside a module in the form of bindings. In this essay

All functions are anonymous by default: Functions are the backbone of your application, the workhorses of all modern software. Just as developers are the workhorses of a software product's construction! Because of this similarity, functions are anonymous by default, just like SAFe treats developers as nameless and faceless.

Value is a top-level concept: As part of SafeScript we’ve made val a keyword so it becomes very clear where your software adds value. This makes it very easy for static analysis to determine how much business value a program adds by simply counting the prevalence of val keywords.

Syntax forged in the fire of conceptual theft: Just like SAFe borrows a little bit from various project management frameworks, as well as adapting concepts from Agile software development, the syntax of SafeScript borrows from other languages. Mostly, it descends from OCaml, Elixir, and a car crash.

Total side-effect conversion: All side-effecty program units are automatically converted to side-effect free code using a process called total disjoint removal. This functionally prohibits non-essential operations such as printing to the screen, which you probably didn't need to do anyhow. If management wanted developer input, they'd ask for it.

Objects, done right and done safely: Object systems are a fantastic language feature which help to promote productive linkages between related data and behavior. However, they can also be a source of bugs, so SafeScript has a built-in safety net which uses pre-emptive inspection to actually optimize away any fallible code.

Security from the get-go: The SafeScript compiler ships with an approval engine as part of the SafeScript Workflow Approval Platform (WAP). This means that it will only accept programs which have been pre-approved. To gain approval, submit your program to the appropriate team by opening a ticket. Please be sure to include the business rationale and a funding stream. You will receive approval within 5-10 business days, following which you may use the --approve compiler flag. This will activate all instances of the approve keyword in your source code.

Deliberately branchless: While most programs make heavy use of conditional logic in the form of if statements, SafeScript has no such concept. In SafeScript, you make a plan when you write your code, and you stick to it no matter what.

Examples

While there is no hello world equivalent for SafeScript (why would there be, you should be getting to work and delivering value instead of standing around gabbing all day), the following is a valid program.

using [Core]

val panda <~ "bamboo" do
  Core.export(panda)
end

Yes, Lisp is useful for DevOps and automation

Jonathan E. Magen — Fri, 14 Oct 2022 12:34:44 +0000

I originally published a less-good version of this post here.

In this article, I will show you modern examples of Lisp which can help you with DevOps tasks and automation in general. No, I'm not joking. No, I'm not drunk.

In no uncertain terms, the power of Lisp is ideally suited for automation. This is not just because of how expressive it is. It's also because of how straightforward Lisp is for solving real-world problems.

Now, let's dive in.

1: Racket's new Zuo

Zuo is a huge step forward for both process automation as well as using Lisp for general scripting!

Since Zuo is basically just a tiny Racket wrapped around an automation-friendly core, it's ideal for scripting build systems! Zuo is new in Racket 8.6 but it's already making a splash amongst language aficionados who also need to get stuff done. Pay special attention to ways in which zuo/build can help replace make.

2: High-level automation with Bass

From building container images to automating builds, Bass has you covered! I originally found out about Bass from this Lobsters post and have been following it ever since.

Bass lang is a bespoke, "low-fidelity" Lisp which runs on top of Docker's buildkit. Unlike Zuo, it's a Lisp dialect designed and written exclusively for the purposes of making common project tasks cacheable. It does this through the enduring magic of thunks, or unevaluated functions. Make sure to check out it's growing standard library.

3: Configure your server fleet with Adams

Sure existing options are cool, but have you tried not using YAML?

While Adams might seem little more than a stylistic choice in favor of S-Expressions, this library packs a punch. Come for the efficient code-reuse, stay for the declarative nature of configs. As a bonus feature, it's written in Common Lisp and therefore has a ton of extension potential.

Now go forth and be awesome! Wishing you the best of luck as you Lisp!

Update 2024-05: Common Lisp for shell scripting

Is it time to look past Git?

Jonathan E. Magen — Wed, 06 Jul 2022 13:44:32 +0000

I clearly remember the rough days of CVS and how liberating the switch to Subversion felt all those years ago. At the time I was pretty sure that nothing would ever get as good as Subversion. As it turns out, I was wrong, and Git showed it to me! So, having learned distributed version control concepts and embraced Git, I was pretty zealous about my newfound super powers. Again, I felt sure that nothing would ever surpass it. Again, it turned out I was wrong.

At the time of this writing, Git's been with us for over a decade and a half. During that time the ecosystem has absolutely exploded. From the rise of GitHub and GitLab to the myriad of new subcommands (just look at all this cool stuff), clearly Git has seen widespread adoption and success.

So what's wrong with Git?

Well for starters, I'm not the only person who thinks that Git is simply too hard. While some apparently yearn for the simpler Subversion days, my frustration comes from working with Git itself, some of which others have been pointing out for the last decade.

My frustrations are:

I find the git-* family of utilities entirely too low-level for comfort. I once read a quote that Git is a version control construction kit more than a tool to be used directly. I think about this quote a lot.
Git's underlying data model is unintuitive, and I love me a good graph data structure. Conceiving of a project's history as a non-causal treeish arrangement has been really hard for me to explain to folks and often hard for me to think about. Don't get me started on git rebase and friends.
I can't trust Git's 3-way merge to preserve the exact code I reviewed. More here and here.

Two interesting projects I've discovered recently aim to add a Git-compatible layer on top of Git's on-disk format in order to simplify things. If not to simplify the conceptual model, then at least to provide an alternative UX.

One such project is the Gitless initiative which has a Python wrapper around Git proper providing far-simpler workflows based on some solid research. Unfortunately it doesn't look like Gitless' Python codebase has had active development recently, which doesn't inspire much confidence.

An alternative project to both Git and Gitless is a little Rust project called Jujutsu. According to the project page Jujutsu aims, at least in part, to preserve compatibility with Git's on-disk format. Jujutsu also boasts its own "native" format letting you choose between using the jj binary as a front-end for Git data or as a net-new tool.

What have we learned since Git?

Well technically one alternative I am going to bring up predates Git by several years, and that's DARCS. Fans of DARCS have written plenty of material on Git's perceived weaknesses. While DARCS' Haskell codebase apparently had some issues, its underlying "change" semantics have remained influential. For example, Pijul is a Rust-based contender currently in beta. It embraces a huge number of the paradigms, including the concept of commutative changes, which made DARCS effective and ergonomic. In my estimation, the very concept of change commutation greatly improves any graph-based models of repositories. This led me to have a really wonderful user experience when taking Pijul out for a spin on some smallish-local projects.

While Pijul hasn't yet reached production-quality maturity, the Fossil project has a pretty good track record. Fossil styles itself as the batteries-included SCM which has long been used as the primary VCS for development of the venerable SQLite project. In trying out Fossil, I would rate its usability and simplicity as being roughly on par with Pijul, plus you get all the extras like a built-in web interface complete with forums, wiki, and an issue tracker.

Ultimately I prefer the Pijul model more than Fossil's but still can't realistically see either of them replacing Git.

Why I'm not ditching Git (yet)

While my frustrations have lead me to look past Git, I've come to the conclusion that I can't ditch it quite yet. Git's dominance is largely cemented and preserved by its expansive ecosystem. It's so much more than the Git integration which ships standard in VSCode and even more than the ubiquity of platforms like GitHub and GitLab. It comes down to hearts and minds insofar as folks have spent so much time "getting good at Git" that they lack the motivation to switch right now. Coupled with the pervasiveness of Git in software development shops, it's hard to envision any alternatives achieving the requisite critical mass to become a proper success.

Granted, just as Git had a built-in feature to ease migration from Subversion, Fossil has a similar feature to help interface with Git. That certainly helps, but I honestly don't know if it'll be enough.

Illustrating the duality of closures and objects

Jonathan E. Magen — Tue, 14 Jun 2022 01:10:07 +0000

I'd heard it before but this time it really sunk in. Let me explain a bit more.

I was recently reading the first few chapters of Software Design for Flexibility (SDF). Doing so reminded of the duality between closures in Functional Programming (FP) and object instances in Object-Oriented Programming (OOP). When I'd heard it previously this connection had seemed tenuous at best and irrelevant at worst. However when I considered this duality this last time, I was able to internalize its significance.

I further chatted about this realization with my friend Reuven Lerner who quoted one of his teachers saying that "closures are outside-in objects" and it blew my mind. As a concession to Reuven being one of the world's most successful Python trainers, code examples below are written in typed Python.

Let's dive in!

What is an object?

When you ask this question to different communities you will get different answers. For example a Smalltalker will likely leverage message passing in their definition while the Java kids' explanations might hinge on inheritance and encapsulation. Still, a Lisper will likely get very excited to tell you about generic functions and the importance of the metaobject protocol. None of these responses would be incorrect, strictly speaking.

For our purposes, we're going to use a more linguistically neutral definition and say that:

An object is a logical grouping of related data and behavior.

In other words, an object is the "what" and the "how" packaged together. In the words of Hanson and Sussman, the authors of SDF, the data and state of an object are "curated" together so that they can subsequently influence the actions of the object.

Assume our problem domain has the concept of a "pipeline job". How might we represent this entity using a classical object system like the one found in Python?

from dataclasses import dataclass


@dataclass
class Pipeline:
    job_id: int
    source: str
    destination: str

    def execute(self) -> str:
        message = f"PIPELINE[{self.job_id}] {self.source} -> {self.destination}"
        print(message)

        return message

Note that the string construction is merely a stand in for the business logic of our pipeline for the sake of simplicity.

That seems simple enough, right? With this approach we can also hold onto one or many Pipeline instances and execute them whenever we please!

# Conjure up a pipeline
my_pipeline = Pipeline(1, "src", "dest")
# Use it whenever
my_pipeline.execute()

What is a closure?

A closure is a special function which retains references to the environment in which it was defined. Let's look at a simple example!

external = 5

def vanilla_func(x: int) -> int:
    return x + 10

def closure_func(x: int) -> int:
    return x + external

In the above example, we define three variables, where the first is a number and the latter two are functions. The first function, vanilla_func, simply adds 10 to its argument. The second one, closure, adds 5 to its argument. You'll also likely notice that closure_func references external which is defined outside its body. Since external is not in the parameter list of closure_func but appears in the body, it is a free variable.

Since closure_func references a free variable from the (lexical) environment in which it was defined, it is a closure as that reference persists for the life of closure_func. Hence, we say that a function is a closure because it closes over the variables in its environment! If my explanation wasn't quite right and this doesn't make sense to you yet, there's some great material out there to help

An entire article, replete with examples, illustrating closures in Python using regular old functions.
A fantastic explainer on closures with JavaScript examples over on the MDN Web Docs.
An article using TypeScript examples with some excellent illustrations.

In languages which support FP, functions can be returned from other functions. So let's see what our pipeline conception from earlier would look like when modeled as a closure. Again, with a Python example:

def make_pipeline(job_id: int, source: str, destination: str) -> Callable[[], str]:

    # closure is here
    def execute_pipeline() -> str:
        message = f"PIPELINE[{job_id}] {source} -> {destination}"
        print(message)

        return message

    # return the closure as a thunk
    return execute_pipeline

In the above example, make_pipeline returns a closure, execute_pipeline which is enriched with the same data and behavior as the Pipeline object from the previous section.

# Conjure up a pipeline
my_pipeline = make_pipeline(1, "src", "dest")
# Use it whenever
my_pipeline()

Wrap-up discussion

We can immediately see some surface-level parallels between the closure and object approaches to expressing our domain's pipeline concept. The two Python code examples make it clear that you can achieve the same level of ahead-of-time data and behavior curation with closures as you can with objects. In short, we've highlighted a duality linking OOP object instances with FP closures. In my gut, I believe this should be explored and formalized further.

If you like this stuff, please reach out to me since I would love to discuss it further!

A modern, ergonomic Unix shell configuration with Fish

Jonathan E. Magen — Sun, 21 Nov 2021 18:11:28 +0000

Like many others, I spend enough time in the shell that even small improvements can yield huge productivity gains. This post is an attempt to catalog several of quality-of-life enhancements I've made over recent months to my shell setup.

History

Like most folks I know, I began my command-line life with good 'ol Bash and actually still do most of my shell scripting in Bash. Even if I eventually rewrite the command in another language, many of my tools begin life as a Bash script.

At some point, perhaps ~10 years ago, I was introduced to ZSH with the venerable Oh My ZSH framework. For me, this was a huge awakening because it taught me about how much shell improvements can impact your life. I gradually evolved my ZSH setup to use package managers like antigen and began to invest more in creature comforts.

Since that time, while the productivity gains have compounded, friends recommended Fish to me, the Friendly Interactive SHell. For my workflows this was the major boost which catapulted my shell configuration from "good enough" to actually "working well".

Why Fish?

The first thing I noticed was that the default configuration of Fish was equivalent to about 30-45 lines of ZSH config. So even starting fresh with Fish was already roughly on-par with my previous ZSH configuration. Moreover, I find that the modern Fish dialect, especially its error-case semantics, are vastly superior considering the frequency with which things go wrong. In this way, Fish gives me a much more solid foundation on which to build a proper environment for myself.

Diving into my Fish config

The core of any Fish config is the config.fish file found at ~/.config/fish/config.fish on most systems.

Mine looks something like this:

#!/usr/bin/env fish

# Configure my shell's prompt
starship init fish | source

# Make it easy to switch to commonly-used directories
zoxide init fish | source

# Set up my ruby environment with rbenv
status --is-interactive; and source (rbenv init -|psub)
set -gx PATH "$HOME/.rbenv/bin" $PATH

# Use VS Code as the default editor and block until the file is closed
set -x EDITOR 'code --wait'

# Make skim do the right thing by default
set -x SKIM_DEFAULT_COMMAND "fd --type f"

## Elixir and Erlang
# Elixir package timeout to prevent stalled downloads
set -x HEX_HTTP_TIMEOUT 59
# Elixir IEx history config
set -x ERL_AFLAGS '-kernel shell_history enabled'

## Rust
# Enable a shared Cargo target directory to cut down on disk usage
set -x CARGO_TARGET_DIR $HOME/.cargo/shared_target
# Ensure that Cargo binaries are in the PATH
set PATH $HOME/.cargo/bin/ $PATH

# Helpful expansions
abbr -a -g gp git push
abbr -a -g gc git commit -asm
abbr -a -g gst git status

# Helpful aliases
command -q exa; and alias la="exa -abghl --git --color=automatic"
command -q exa; and alias ll="exa -bghl --git --color=automatic"

# Fisher!
if not functions -q fisher
    set -q XDG_CONFIG_HOME; or set XDG_CONFIG_HOME ~/.config
    curl https://git.io/fisher --create-dirs -sLo $XDG_CONFIG_HOME/fish/functions/fisher.fish
    fish -c fisher
end

## Node
# Volta config
set -gx VOLTA_HOME "$HOME/.volta"
set -gx PATH "$VOLTA_HOME/bin" $PATH

Analysis

In order of appearance, here are some of the tools which feature in the above Fish config:

The Starship prompt is a terrific alternative to slower prompts written in pure shell. It supports a whole variety of status features, is quite performant, and has easy-to-understand configuration options.
I use zoxide to quickly switch to recently used directories, or to paths which I use particularly commonly. It can be hard to remember to use z instead of cd but once you get the hang of it you'll never go back.
My setup relies heavily on rbenv since I use Ruby for automation all over the place.
Tools like skim work great with others like fd to provide additional functionality when it comes to locating files and filtering through search results. I also use skim's sk command pretty heavily in other scripts when interactivity is required.
Even though ls is a good-old standby, I prefer the aliases for ll and la that rely on exa. These aliases default to showing important information like git status and can even prevent me from needing to launch subsequent, exploratory commands.
I really fancy the Fisher package manager for installing Fish libraries, tools, and commands!
Since I'm a big fan of toolchains as code, I've fallen in love with volta for managing my Node.js and Yarn versions. Don't forget commands like volta pin in Node.js projects to lock in versions to your package.json file!

Conclusion

So I guess you could say that Fish really hooked me. Get it? Anyway, thanks for reading this post and feel free to reach out to me with your favorite shell tricks and quality-of-life improvements! I really love this stuff and would love to hear from you.

The coming era of OS kernel modularity

Jonathan E. Magen — Wed, 29 May 2019 15:28:33 +0000

There used to be a debate between folks about whether monolithic kernels or microkernels were the superior architecture. It might have been interesting, but it got pretty ugly at times. While the flame war raged on, research and development continued in parallel within both the academic and industrial worlds. Looking back on it now, it's clearly not merely a matter of monolith vs. microkernel architecture. Rather, it's a larger discussion about code modularity.

The last few years have shown us a few key trends hidden among the often-overlooked activity in the world of Operating System (OS) development. Chief among them is the idea that the future need not look like the past. From Google's Fuchsia project to BPF in Linux, the era of kernel modularity has arrived.

Modular kernels currently under development

If you keep your ear to the ground, you'll inevitably hear the excited rumblings of folks watching the not-so-secret development of Google's new open source Fuchsia operating system. The Fuchsia kernel is called Zircon and is written in C++. Zircon is notable for a variety of reasons, not the least of which is the eschewing of the traditional POSIX standard in favor of what it's supporters refer to as "more modern" conventions. Of particular note is the intentional deprecation of dotdot filesystem conventions for security reasons. Anyone interested in what else Google's Fuchsia team considers to be modern conventions should be referred to the Fuchsia docs, with the most popular destination being The Book. This recent set of notes from China may also be of interest.

Though Google Fuchsia has forsaken POSIX, not all modular kernels have followed suit. For example, the Redox project has opted to continue the POSIX tradition while clearly going for a microkernel approach. Since the goal of Redox is to make a reliable, secure, general purpose OS, the team chose to write everything in the Rust programming language. This seems like a great idea as Rust has really wonderful safety guarantees and very ergonomic error handling.

On the Rust front, Redox isn't the only game in town as the new Tock OS receives more attention for targeting embedded and low-power systems. The website touts the ability for Tock to run on everything from industrial sensors to security keys to watches. If you're into IoT and value security and reliability, keep an eye on Tock.

Moreover, there's even an experimental research OS called Manticore, also written in Rust, aiming to explore the world and concepts of parakernels.

No discussion of modular kernels would be complete without a mention of the Genode project. Genode isn't an actual kernel so much as a microkernel construction kit. Even though it's written in C++, Genode claims that it offers a security-first approach to OS development with sights set on both IoT as well as more traditional workloads.

The monolith strikes back

Despite its roots as a monolithic kernel, Linux was always able to sidestep an entire class of problems by leveraging dynamically linked kernel modules at run time. However, while this has proven a successful strategy, the discussion of modularity has shifted with the advent of kernel support for running programs written using the Berkeley Packet Filter (BPF) mechanism. Originally developed for controlling network IO, BPF has since been extended (eBPF) to allow for attaching filters to various OS functions. The upshot of ePBF support in Linux is that it allows for tying code deep into the kernel without modifying the kernel itself.

Brendan Gregg has a terrific compilation of eBPF learning resources available on his blog and the IO Visor project also maintains a pretty solid overview. Also, there is a performance-oriented BPF book set to be published in October of 2019.

Even though they're certainly smaller than classical OSes, the newer world of Unikernels borrows heavily from the research world's development of nanokernels carrying with them the concept of a LibraryOS. Standing tall among them is the venerable MirageOS project, a library OS written entirely in OCaml. For people interested in the "mini-monolith" (minilith?) approach of Unikernels looking to develop in languages other than OCaml, the Unikernels website maintains a list of projects from which you can pick anything from Go to BEAM languages.

Another exciting option, geared more toward traditional development, is the UniK project started by my friend Idit Levine, which can take almost any existing program and produce a Unikernel from it. There are examples for common languages and platforms such as Node.js, Python3, and even Java. One of the cooler things about UniK is how clean the code is, you can read through the (Go) source and understand a great deal about how it provides a pluggable backend supporting flavors such as rumprun and even Firecracker.

What about a language VM as the kernel?

There have been multiple attempts to write OS kernels in interpreted or semi-compiled languages. Two of the most notable are actually the now defunct JNode and JX projects. However, one additional effort from the past also stands out as significant.

The Singularity OS research project at Microsoft has been discontinued now for the better part of a decade, but has continued to intrigue me whenever I think about it. An attempt to make an entire OS in nothing but C# and a minimal VM, it was rumored to have heavily leveraged a Just-In-Time (JIT) compiler to provide runtime profile and optimizations far beyond what a traditional Ahead-Of-Time (AOT) compiled kernel could offer. Additionally, the supposed memory safety of such managed languages could offer some terrific security guarantees like runtime memory inspection and execution supervision.

In recent months, I have watched the development of WebAssembly (WASM) and the announcement of the WebAssembly System Interface (WASI) with great interest. While tools like wasmer make it easy to run applications compiled to WASM, I was just exposed to this effort by the NanoVM folks to run WASM unikernels. Pretty cool.

Great, so what's the point?

There's clearly a larger discussion happening about code and system modularity in a variety of areas. As people explore techniques like microservice architectures (which are NOT a solution for every problem), the community continues to develop a variety of viewpoints. Therefore, it seems only natural that OS development would have a contribution to make to such discussions and that it would be influenced in return. Let's see what develops (pun intended).

Graceful degradation: Harvest and Yield in the age of microservices

Jonathan E. Magen — Tue, 30 Apr 2019 01:19:15 +0000

Terms like "graceful degradation" are coming into vogue alongside the emerging discipline of chaos engineering and evolving understandings of what it means to make a distributed system fault-tolerant.

Take, for example, the case of Netflix, where the term "chaos engineering" got its start. If they experience a hiccup and can't stream their normal catalog of movie titles, they might display a polite error message and instead offer a more limited selection of "featured titles". If your personalized recommendations can't be retrieved as they normally would, the platform might opt to show you the 50 most popular movies as chosen by viewers. The point is that, to their credit, Netflix will rarely "fail hard" but instead will "fail soft" by gracefully degrading the functionality they provide to ensure a reasonable user experience.

In short, systems which degrade gracefully are those which prioritize availability over completeness. This is a hard, but often-necessary tradeoff when evaluating service quality.

Yet, prioritizing availability over other core functionality is nothing new to the distributed systems community. The CAP theorem, otherwise known as Brewer's Theorem, has long shown us that in the presence of network partitions within a distributed system, you must choose between that system's availability (ability to serve clients) and its data consistency (all parts of the system must agree on the same values at the same time). Therefore, it should be no surprise that a subsequent paper, by Armando Fox and Eric Brewer, lays out a foundation for following through on this sort of tradeoff.

In Harvest, Yield, and Scalable Tolerant Systems, Fox and Brewer say quite a lot in just a few short pages. Leaving aside the fascinating statements that the authors make regarding the interpretation of the CAP theorem (strong and weak!), the paper introduces the super-useful concepts of harvest and yield.

In short, Fox and Brewer borrow from engineering and define the two terms as such:

We assume that clients make queries to servers, in which case there are at least two metrics for correct behavior: yield, which is the probability of completing a request, and harvest, which measures the fraction of the data reflected in the response, i.e. the completeness of the answer to the query.

So now we have two definitions:

Yield: a measure of a distributed system's ability to provide an answer.

Harvest: a measure of that answer's completeness.

To illustrate this concept, let's consider a simplified scenario involving a constellation of four microservices feeding data to the fictional online store for Captain Carolina's Dive Shop. Now, Captain Carolina cares just as much about designing effective software systems as she does about selling scuba lessons or surfboards so her engineers have split things up into four logical units.

Search service: Provides a searchable accounting of the store's product inventory.
Shopping cart service: Facilities for aggregating orders based on product inventory.
Account service: Managing everything from first-time marketing traffic through customer preferences and contact details.
Recommendation engine: Based on browsing habits and purchase history, provides the sort of "you might like" suggestions someone would come to expect from an online store.

Every user who comes to the site is a potential customer so we want to make sure that we provide them with the best experience possible. However, we also know that not all software works all the time and even with backing-service SLAs sitting squarely in the four-nines region, no service works perfectly all the time. Therefore, we may use the dual concepts of harvest and yield to reason about the behavior of Captain Carolina's website storefront.

Since things have been broken up into the aforementioned microservices, any, but not all of them, can fail and the site will still be up. In the event that the shopping cart fails, the site will still work such that visitors may still search the catalog. If the search service fails, users can still browse. If the account service fails, users can always purchase as a guest. If recommendations can't be provided, it doesn't impede the general operation of the store and merely sacrifices a chance to up sell.

By isolating and compartmentalizing logic, we have also isolated and compartmentalized failure. Furthermore, in this example we have traded harvest (completeness) for yield (availability).

While Brewer and Fox remind us that yield is usually measured in nines of uptime, they don't provide much guidance on harvest. In the world of microservices, you might define harvest as a measure of responding services:

Harvest = Responding Services / Total Services

Or more compactly:

H = R / S

Therefore, if just one of Captain Carolina's services went down, then we could concludes that H = 3/4 = 0.75. In fact, it would only be if all of our services became unavailable, H = 0/4 = 0, that our yield would be compromised!

Wrapping up

This article uses microservices as an example even though a microservices architecture isn't for every product, team, or problem. Generally, I hope that the paired terms of harvest and yield help you to have better conversations about service quality in distributed systems. If you liked anything about this post, please go ahead and read the paper!