DEV Community: Matt Butcher

The sidecar pattern in Kubernetes describes a single pod containing a container in which a main app sits. A helper container (the sidecar) is deployed alongside a main app container within the same pod. This pattern allows each container to focus on a single aspect of the overall functionality, improving the maintainability and scalability of apps deployed in Kubernetes environments. From gathering metrics to connecting to data sources (a la Dapr), sidecars have found a notable place in the cloud-native developer’s toolbox. Sidecars are designed to run alongside your apps continuously and do not scale down to zero. Wouldn't it be great if they did? In this article, we introduce scaling sidecars to zero in Kubernetes.

Zero Cost Sidecars in Kubernetes

WebAssembly (Wasm) and containers will peacefully co-exist and be complementary technologies. While containers offer an efficient way to package entire apps with dependencies, Wasm provides a lightweight, secure, and fast-executing environment that can help scale apps, making serverless Wasm workloads the ideal partner for long-running container apps. In fact, Solomon Hykes (founder of Docker) said this five years ago

Before we explore one such example of Wasm and containers, a word about efficiency.

Maximizing Efficiency With the Sidecar Pattern

A common criticism of the sidecar pattern is its inefficiency. The underlying problem with sidecars is that the sidecar containers must remain operational throughout the lifespan of the main app, leading to potential resource wastage. Consider an app with three sidecars (so four total containers). A typical deployment sets the replica count to 3. So deploying a single app results in 12 long-running containers — three replicas of each of the 4 containers. All 12 of those processes consume CPU and memory all the time.

With SpinKube, there’s a cool way to enjoy all of the benefits of a sidecar without the resource consumption.

Wasm apps written using Spin apps follow a design pattern called serverless functions in which the app is started once when a request comes in. The app handles the request and then shuts down. If four requests come in simultaneously, then four copies of the app are started. When zero requests come in, no copies of the app are running. They are, in this sense, “zero cost”.

Spin apps, based on their Wasm underpinnings, are also lightweight. They cold-start in under a millisecond (as opposed to the dozens of seconds it takes a container to start up). They consume fewer resources at runtime, using around 2-5M of memory and minuscule amounts of CPU. And because they only run while processing a request, most live no more than a few hundred milliseconds.

Spin apps are a great candidate technology for implementing sidecars. And SpinKube makes it possible.

Spin Apps as Sidecars

First, talking about how an app and its sidecars are connected is good. We’ll take a trivial scenario from Dapr. In that ecosystem, a main process uses HTTP or gRPC to communicate with its sidecars. You can almost think about it as the microservice architecture applied to a Kubernetes pod. Say we have an example with one app querying a sidecar service for an HTTP response. In this scenario, the main app periodically needs to perform an HTTP request to the other service. Both are long-running servers, but the main app creates an HTTP client that talks to the sidecar’s HTTP server. That sidecar HTTP server does its internal routing and executes the correct business logic to handle the request.

With a Spin app, there is no reason for the sidecar to need a long-running process. After all, it only needs to handle requests from one other client: the main app. This is a perfect candidate for a Spin app.

When this app is deployed, the main app (in a container) is run in the container runtime, and it executes as a server, always on, always listening.

The Spin app sidecar is deployed in the same pod as the container app, but it is scheduled onto a Spin runtime instead of a container runtime.

The Spin app is deployed, but it is not (properly speaking) running.

When a new request comes into the main app, its HTTP server receives and handles the request. That main app contacts the sidecar at some point over a local HTTP request. When the request to open the network connection happens, the containerd Spin shim (the thing in containerd that handles Spin app invocation) starts a new instance of the Spin app to handle the request object it received from the main app. The new instance of the Spin app then runs to completion, returns a response object, and shuts down.

The important thing to note here is that the Spin app only runs when handling the request. After that, all the resources it uses, including CPU and memory, are freed up again.

Running 4 or 12 of these sidecars per pod can be done efficiently. In fact, it’s preferable to run all of those sidecars in the same Spin instance, meaning they share their resource allocations even more efficiently. In theory, one could run over 1,000 sidecars per main app, but it’s unlikely that there’s a practical use case where this is the best design.

Creating Our Spin App Sidecar

We, begin by using the Spin template - in this case the (Rust HTTP template)[https://github.com/fermyon/spin-rust-sdk] to get us started:

cd $HOME
spin new -t http-rust --accept-defaults spin-app-sidecar
cd spin-app-sidecar

We then add some business logic to the sidecar. In this case, telling the main-container-server app what the current time is:

use spin_sdk::http::{IntoResponse, Request, Response};
use spin_sdk::http_component;
use chrono::Local;

/// A simple Spin HTTP component.
#[http_component]
fn handle_spin_app_sidecar(_req: Request) -> anyhow::Result<impl IntoResponse> {
    Ok(Response::builder()
        .status(200)
        .header("content-type", "text/plain")
        .body(Local::now().format("%Y-%m-%d %H:%M:%S").to_string())
        .build())
}

As you can see above, we are using the chrono library to obtain and help with formatting the time. To resolve dependencies, run the following command:

cargo add chrono

Our spin-app-sidecar app is now ready to build and push.

We will now use a GitHub Personal Access Token (via the GH_PAT and GH_USER variables in our CLI) to push the app to a registry. First, we generate a GitHub Personal Access Token and set write:packages in your our GitHub user interface. We then set the GH_PAT and GH_USER variables in our CLI and then push the app to a registry:

# Store PAT and GitHub username as environment variables
export GH_PAT=YOUR_TOKEN
export GH_USER=YOUR_GITHUB_USERNAME

# Authenticate spin CLI with GHCR
echo $GH_PAT | spin registry login ghcr.io -u $GH_USER --password-stdin
# Push container server app to the registry
spin registry push --build ghcr.io/$GH_USER/dapr-integration/spin-app-sidecar:1.0.1

We can now use spin kube scaffold to generate a .yamlfile based on the deployed app:

spin kube scaffold --from ghcr.io/$GH_USER/spin-app-sidecar:1.0.1 \
    --out spin-app-sidecar.yaml

The above command will create a spin-app-sidecar.yaml file with the following contents (note, we have replaced the static username with the $GH_USER variable here for your convenience):

apiVersion: core.spinoperator.dev/v1alpha1
kind: SpinApp
metadata:
  name: spin-app-sidecar
spec:
  image: "ghcr.io/$GH_USER/spin-app-sidecar:1.0.1"
  executor: containerd-shim-spin
  replicas: 2

Deploy the app using the .yaml file:

kubectl apply -f spin-app-sidecar.yaml

Scheduler Overhead

So far we’ve seen why Spin apps make excellent sidecars. We’ve stayed at a fairly high level. But we should be aware of what happens under the hood. While the apps themselves take no CPU or memory, containerd has to do a little more work, and it does this using a low-level Spin shim.

The Spin shim listens for inbound requests for a Spin app and then starts the relevant serverless function to handle the request. Of course, this requires a small amount of memory deep in the Kubernetes stack, but it is still lighter than the work containerd must do to start a container.

The situation is different in Fermyon Platform for Kubernetes, in which Wasm is not scheduled through containerd, and one process per node handles thousands upon thousands of Wasm apps.

But again, even up to a thousand Spin sidecars can be scheduled using fewer resources than one container-based sidecar.

Running More Apps in Your Cluster

Thanks to SpinKube’s ability to run Spin apps side-by-side with containers, Spin apps can be used in exciting ways. Here, we’ve taken a fresh look at the sidecar pattern that is popular with service meshes and distributed API services like Dapr. And what we’ve seen is that Spin apps make an excellent alternative to older containerized sidecars. Spin apps are faster and more efficient, meaning you can run more apps in your cluster. This translates not only to increased density but also to smaller clusters and saved money.

Local Service Chaining

Spin's local service chaining functionality allows developers to write applications as a network of chained microservices, whereby a "self-request" (an internal component request) can be passed in memory without ever leaving the Spin host process. Although it may limit how deployments can be arranged, local service chaining is highly efficient, depending on the nature of the microservices. It's important to highlight this as a viable strategy for enhancing the integration of helper workloads alongside long-running apps.

Conclusion 🎉

These new approaches to orchestration can minimize CPU and memory usage, ultimately allowing for higher app density per cluster and significant cost savings. In addition, more efficient operations and reduced startup times equate to faster machine-to-machine communication and improved end-user experience.

You can get started building a Spin app over at the QuickStart guide or learn more about the other things you can do with Spin and Kubernetes over at the SpinKube site. Drop a comment if you have suggestions about patterns around sidecars in Kubernetes!

The History and Evolution of WebAssembly in Kubernetes

Matt Butcher — Thu, 16 May 2024 12:00:00 +0000

Discover how WebAssembly and Kubernetes evolved together to create ultra-efficient, high-performing modern cloud environments.

Kubernetes (K8s) is the established container orchestration solution. From data centers to pools of devices, it has shown itself nearly ubiquitous. Every major cloud provider offers hosted K8s, yet it can be conveniently run on a small cluster of Raspberry Pi’s. K8s has lived up to its initial promise: Bring container orchestration to everyone.

WebAssembly (Wasm) has taken a different path to success. Originally conceived as a way to extend Web browser language support beyond JavaScript, it has outgrown its initial promise—by a lot. BBC and Amazon use Wasm in their embedded streaming players. Prime uses Wasm to increase stability and speed when updating its app for more than 8,000 device types. Shopify makes commerce extensible with Wasm. Envoy Proxy uses Wasm as a plug-in framework. And, Disney Streaming use Wasm in their Disney+ Application Development Kit (D+ ADK).

To us, the most exciting application of Wasm is on cloud and edge. To do that well, Wasm needs to run inside of K8s.

Why Wasm on the Server Side?

There are three characteristics that Wasm has, because of its browser heritage, that are highly desirable for a server-side platform:

A strong security sandbox
Multi-arch, multi-OS, and multi-language support
High performance and efficiency during execution

The browser requires a security sandbox to prevent a security breach from a Wasm binary downloaded off the Internet. On the cloud (and edge) side, this same requirement is necessary. Just as is the case with virtual machines (VMs) and containers, Wasm binaries must be able to execute in a multi-tenant environment without posing a risk either to the host or to other guests running on the same service.

The browser also requires that a single Wasm binary be able to run on any operating system and CPU architecture that a browser supports. In the Wasm world, dozens of languages (including biggies like JavaScript, Python, Go, C/C++, and Rust) can compile to Wasm, and then run on any platform. On the server side, this is a fantastic boon, because developers themselves do not need to target the specifics of a production environment. They simply deliver the application, and platform engineers can deploy it onto whatever operating system and architecture combo they see fit (we like to demo this by deploying an application into a mixed K8s cluster with Arm and Intel worker nodes).

Finally, in the browser space, performance is paramount. Website visitors are fickle and impatient. Consequently, Wasm needs to download quickly, cold-start in the blink of an eye, and remain performant during the entirety of a user’s visit. Likewise, it must consume few resources. None of us wants a webpage taking up gigs of memory just to paint a form on the screen!

And this performance point is what sets Wasm apart on the cloud. For workloads like serverless functions, event-driving architectures, and steps in a processing chain, we want our code to cold start instantly, consume as few resources as possible, and execute at near-native speeds. Wasm is the only technology that fits this bill while still providing a robust sandbox and tremendous language, architecture, and OS support.

There are, of course, many ways to run Wasm binaries. Spin, the developer tool for building server-side Wasm can run in a wide array of environments, from embedded systems to large Nomad clusters. The most exciting environment, though, is K8s.

The Road to Running Wasm in K8s

Most of us at Fermyon have a deep background in the container and K8s space. We’ve been part of the CNCF Technical Oversight Committee (TOC), contributed and maintained many projects, and even written some children’s books on the matter. Even years ago, we understood how important it was to run Wasm inside of K8s.

We knew that for Wasm to be successful inside of K8s, it would be an anti-pattern to merely package up a Wasm runtime inside of a container, and deploy that. Doing this simply obviates every single reason to run Wasm – start-up time is slow, you’re locked into a particular OS and operating system, and there is no clear way to attach external services to the Wasm file without first proxying them through the container layer. So from the beginning, we set our sights higher: we wanted to run Wasm inside of K8s. Not on top of it.

Our first two attempts were not so successful, though.

Back in 2020, those who would go on to become the Fermyon founding team built an (unreleased) experimental Wasm runtime for K8s called WOK (Wasm On K8s). The idea, at the time, was to replace containerd with a Wasm runtime. We made it a few months into this project before realizing that the scope was huge.

That experience, combined with some work we did with Virtual Kubelet, inspired us to try a second approach: Virtualize a Kubelet worker which runs Wasm. Krustlet (a mangled portmanteau of Rust Kubelet) worked reasonably well for running Wasm binaries, but had a major limitation: It did not integrate well into any of the rest of the K8s ecosystem. Everything from volume mounts to service meshes was simply not available to a Krustlet application. Over time, we came to realize that once again we had bitten off more work than we could manage.

The old saying about the third time being the charm certainly held up here. While we were building Spin, Microsoft started a project called runwasi, which took the sensible approach to support Wasm inside of containerd by writing a containerd Shim. This meant that one could simply write a regular Pod specification, with all the usual bells and whistles, and have some or all of the images reference Wasm binaries instead of containers.

With this tool and several others (including the awesome KWasm project), Microsoft, Fermyon, SUSE, Liquid Reply, and others were able to put together an awesome K8s toolkit: SpinKube.

Why We’re Excited About Spinkube

SpinKube provides all of the tools you need to run Spin Wasm applications inside of your K8s cluster. It works on every major K8s distribution, and from data center to Raspberry Pi (and probably smaller), it’s got you covered.

What makes SpinKube so special? Serverless functions have been around for a decade. While developers have made it clear over and over again that Lambda-style functions are both faster to write and easier to maintain, the runtime solutions for serverless have always been cobbled together from ill-suited technologies. K8s has seen a share of serverless platforms — Knative, OpenWhisk, Fn Project, and OpenFaaS — but in all cases, they were based on container technologies that made them slow and resource-hungry. In one recent conversation I had with an OpenWhisk user, they noted that it takes 37 seconds to cold start a single function. Contrast that with Spin, which comes in at under half of a millisecond.

Research suggests that the average K8s node runs only 30 containers. Since each container is long-running, resource-hungry, and scoped to peak resource consumption, the size (and cost) of a K8s node starts to rise with density.

SpinKube does not suffer from this problem. In fact, it is trivial to deploy 250 apps per node in a SpinKube cluster. Whether you have that many apps or not, the main point is that serverless-style Wasm is far more efficient to operate than containers in your K8s cluster.

Add to that Spin’s “blinking cursor to deployed application in two minutes or less” developer story, and it becomes clear why SpinKube can turn into a major boon both for developer and operations teams.

One of the best parts about SpinKube, though, is that Spin applications are deployed directly into K8s nodes alongside containers. This means you can:

Run containers as sidecars to Wasm (or vice versa)
Mount volumes, secrets, and config maps to a Spin app
Use your service mesh, sidecars, telemetry, and other existing K8s services
Continue to manage your policy control and security the same way you always had
Use your K8s monitoring and dashboard systems unaltered

But if you’re really interested in efficiency and performance, there’s one more chapter to this story.

Fermyon Platform for Kubernetes Brings Hyper-Elasticity to You

In 2022, we built Fermyon Cloud, our hosted developer-centered cloud service. The goal was to make it trivially easy for a developer to deploy (for free!) a Spin application into a highly stable production environment.

At the time of this writing, a single node in our own Fermyon Cloud cluster hosts 3,000 user-supplied applications.

We can scale from zero instances of an app up to tens of thousands of instances in under a second. And we’re accomplishing all of this on AWS 2xl-sized VMs! Fermyon Cloud is downright cheap to operate and enables us to have a very generous free tier.

We wanted to bring that density to you.

So we took the engine inside of Fermyon Cloud — the massively multi-tenant Wasm sandboxing environment called Cyclotron — and we combined it with SpinKube to create our enterprise offering: Fermyon Platform for Kubernetes.

Fermyon Platform for Kubernetes is great for:

Hosting workloads that need to scale up rapidly — even if unpredictably (as in the case of trending articles or sudden sales surges)
Hosting a large number of serverless functions
Providing a multi-tenant system with strong performance or density needs
Managing sophisticated pipelines such as ETL workloads
Replacing Lambda, Azure Functions, Google Cloud Functions, or other similar “not in K8s” serverless architectures
Amping up the performance and scalability of workloads that once ran on OpenWhisk, OpenFaaS, Knative, or Fn Project

Conclusion

K8s was originally thought of as a container orchestrator. And Wasm was originally a browser technology. Good technologies meet their design intent. Great technologies exceed them. And both of these have proven to be great technologies.

Check out SpinKube to start building ultra-high-performing serverless functions that can run side-by-side with your existing containerized applications in K8s.

It Took Me 20+ Years To Learn This Lesson About Dev

Matt Butcher — Wed, 01 May 2024 18:29:57 +0000

I started my software development career as a web developer. I worked at companies large and small, including stints at HP and Google. At Microsoft, I was a principal engineer. I left to become co-founder and CEO of Fermyon.

I’m not one to dispense advice often. Our individual experiences are varied, and the anecdotal evidence of one person’s experience rarely generalizes well. But having made the same mistake over and over again in my career (and having observed others do the same), I feel like I have one good piece of advice to share:

Remember that every line of code you write is a line of code you will support.

This maxim carries more inside of it than appears at first glance. I’ll share what it says about:

Coding as artisanship
Not Invented Here Syndrome
The cost of complexity
Reducing future needs for support
And why your future you (and others) will thank you for the code you’re producing today

Coding as artisanship

Let’s start with an optimistic application of the maxim. Artisanship is a word perhaps often associated with bygone eras. It implies dedication to a craft, where mastery emerges over time and by virtue of hard work and frequent activity. Unlike the term “engineering,” which suggests a practice heavy on planning, artisanship implies a hands-on effort. Producing code, with its debugging and testing and refactoring, is a hands-on effort.

Building good software requires skill, knowledge, and a continual desire to improve. As you build your code, take time to do it well. Yes, “perfection is the enemy of the good,” and you need to code to get the job done. But following good coding conventions, naming variables well, and thinking through the problem you are solving are all practices that will make it easier to maintain your code over the longer term. In short, when you approach coding as a hands-on process resulting in mastery acquired over time, you are an artisan.

It’s a pain to support poor code. It is intellectually stimulating to maintain good code. Artisanship focuses us on producing good code as part of our daily practice.

"Not Invented Here" is your enemy

We all know the feeling. Sure, there’s a library or tool that does this. But you can do it better!

Every software engineer has done this at least once. It 15 years of my career before I finally learned this lesson. I rewrote everything from template engines to low-level data structures. And I always had some kind of justification:

Sure, there were other libraries, but they were too big/bloated/complicated/special purpose…
I had a novel way of doing things that other people hadn’t thought of
I didn’t trust/know/feel comfortable with someone else’s library
I’d just read a book/article/blog post that inspired me
I could learn more by doing it myself than by using someone else’s (true, but at a cost far greater than I realized)
I didn’t have time to look

In all of these cases, I discounted two things:

How much time others had spent solving the problem already. (A mark of that code’s maturity)
How much time future-me would have to spend fixing, tweaking, generalizing, and maintaining that code

Seasoned engineers call this Not Invented Here (NIH) Syndrome. And it is a pitfall that leads to duplicating other peoples work while taking on new maintenance burden. So when you find yourself in that moment where you could pick a library or tool off-the-shelf, but you think you just might be able to do it better, ask yourself how passionate you are about maintaining that code for the next several years.

Are there exceptions to this application? Absolutely! But the exceptions should be rare, and after you’ve genuinely explored whether existing solutions are good enough to get the job done.

The more complex your code, the harder it is to debug

I worked on a package manager that had a complex string parser inside. In a bout of inspiration, I wrote the entire parser in one sitting. Moreover, I did it in just a couple of functions. And even while those functions weren’t terribly long, they were conceptually complex. Expressed using technical terms, my code had very high cyclomatic complexity. There were lots of different paths through these two functions, making it hard to take in at a glance all the different potential results of using this pair of functions.

In my defense, I wrote tests. And I wrote at least a little bit of documentation. And my code was impeccably formatted. Best of all, the code functioned very well, and I didn’t have to touch it for years (nor did any of the other dozen developers who worked on the project).

But none of that made up for the fact that it was hard to understand.

Then one day, we got a CVE. In a particularly devious attack, a hacker could force the parser to allocate more memory than the system had. After three or four years of not touching (or thinking about) this code, I had to go fix it. As my eyes scanned the lines, it felt completely foreign. And over the next few days I had to pick apart my own code to try to figure out what it was doing and how I could fix what turned out to be a well-hidden bug.

Demeter’s Law is a good software practice to follow. Named after the Greek goddess Demeter, it suggests that limiting the cyclomatic complexity (the number of decision trees) in code is a virtue attained by splitting code into useful units. While this is less of a real law than just a rule of thumb, it is a good one. It may force you to write more lines of code (as you break complex tasks down into functions), but this is virtuous if it results in more debuggable and maintainable code.

Writing test for your code now means supporting your code less later

One of the most audacious things I have ever heard a software developer say—and this was from a programmer with more than 20 years of experience—was that “requiring unit tests means telling your team you don’t trust them.” The first time he said this to me, I was astonished. What a huge and improbable leap! In every important industry we have, from doling out bills at the bank to filling bottles at the pharmacy to fastening labels to a newly created pair of jeans, any good system has a series of quality checks.

Code should have that too.

And what’s cool is that with code, it’s actually remarkably easy to add quality checks. We just write code to test our code!

While the developer I quoted above cited trust as the main reason, I have found that in my own career, I’d cite a different reason for not writing tests: Being in a rush. I’ve found a few good antidotes for being in a rush, and those are almost always external:

Use a code coverage tool to make sure that I am covering at least 80% of my code. And that each time I write something new, that new code doesn’t drop test coverage beneath that 80% level.
Require tests before any code passes code review. I am a HUGE believer in the value of code review, and this is one area that a review can really help with You may find other ways to help you stick to a testing regimen. A friend of mine and excellent coder, Adam Reese, once told me that he found it intellectually challenging to write elegant and exacting tests. Viewing tests as a challenge—a puzzle to be solved—helped him stay motivated to keep to his coverage goals.

Regardless of your strategy, if you don’t test now, you’ll be debugging later. Possibly in much more urgent circumstances with more dire consequences.

Future you won’t remember what present you is thinking

So document it all!

We humans have a funny bias. We think that we will remember tomorrow everything that happened today. And we think that when we look back (perhaps even years from now) we will recall our mental state when an event happened. The thing is, we know this isn’t true. Most of us have a hard time recollecting what we had for lunch a few days ago.

I’m an ex-philosopher. The early modern philosopher David Hume expressed eloquently that he doubted that we should even consider ourselves to be “the same person” today as we were yesterday. Our minds are remarkably fluid, and Hume worried we tend to overstate how similar present me is to past me or future me.

Future you will not remember the nitty details of why you wrote this code the way you did today, or why you named that variable fhr or what you intended that poorly named function to do or why you left the comment // FIXME later on line 235.

The best way you can combat the limitations of your own memory is to make your code clear. That means:

Comment liberally
Name things well
Write higher level documentation
Write useful commit messages in your version control system

Future you will thank you. Or at least not be angry with you.

Corollary: If other people can’t understand it, it will always be your problem

So document it in a way that other people understand.

Really, if you are doing a good job at the last one, you probably have this one covered anyway. But here’s where the problem can get more insidious (or at least more annoying).

Earlier I told the story of the string parser I wrote. And how there was a CVE some years later. And that I had to fix it. What I didn’t mention is that I wasn’t even really working on that project when I was called in to fix it. I had moved on to other things.

But because it was not clear how my code functioned, nobody else felt like they could fix that bug. I had to halt all my work on another project for a few days while I came back to this one and fixed my code.

What’s the lesson there? If my code isn’t easy to understand, it will come back to haunt me. Because other people will not be able to understand it. And they will git blame it. And they will hunt you down. And they will refuse to let you have coffee and pizza until after you fix the problem. It will be unpleasant.

Know what’s better? Making your code so easy to understand that somebody else can fix your code without you even having to know about it.

Conclusion

I suggest the following maxim as a way to push yourself to improve your own code:

Remember that every line of code you write is a line of code you will support.

It will save future you time and energy. It will prevent frustration in your peers. It will make it easier to transform today’s code into tomorrow’s code. And it will probably make everyone else think you are one of those uber-coders as well.

There is a craft to code, an artisanship. And the best way to attain a high degree of mastery is to think carefully about what you are building, and then pour that thought into code.

I’ll say it once more: It’s a pain to support poor code. It’s a delight to support good code.

Build a Slack Emoji Bot in 20 Lines of Code

Matt Butcher — Fri, 26 Jan 2024 20:56:29 +0000

My colleague Rajat recently built a Behind The Scenes photo sharing webapp. The idea behind this app was simple - At Fermyon, we’re a fully remote company. We tend to use Slack for all our communication and we have a channel named #behind-the-scenes where we share photos from all our travel to conferences, and meetups etc. The idea was to create a web app that was a front end to the photos shared on that channel. (You can check out this tutorial on how Rajat built this using Spin and Nuxt.js)

A cool feature of this workflow was that not all photos shared on the channel were displayed on the Behind The Scenes app. Photos would only appear on the site if they were ‘signed off’ by the person who posted the image. This sign off method was rather clever - It would be via a specific Slack emoji reaction! I loved that little implementation detail so set out to see how it was built out.

Here’s how you can can build a webhook that is triggered with a Slack emoji reaction, in less than 20 lines of code!

Create a Slack App

To build this webhook you first need to create a Slack app.

1. Go to api.slack.com and click on ‘Your Apps’

2. Click the ‘Create New App’ button and add an App Name (you can change this later) and also a workspace that you want to develop your app in. Ensure you have the permissions to add App integrations to Slack. Personally, I prefer having a private Slack workspace for testing out bots as I’d rather not alarm my colleagues with an erratic Slack bot 🙃

3. Nice! Your Slack app is created. We now need to add an Event Subscription to it so that it can detect when an emoji reaction is added to a message. Click on the ‘Add features and functionality’ section and then the ‘Event Subscriptions’ button

4. The Slack API has an interesting way to ensure only ‘Verified’ URLs can respond to Events. Essentially, their API sends a HTTP POST request to your endpoint, that contains a `challenge` parameter. They expect your endpoint to respond with the same `challenge` value to verify your endpoint.

This is what the payload looks like:



{
    "token": "Jhj5dZrVaK7ZwHHjRyZWjbDl",
    "challenge": "3eZbrw1aBm2rZgRNFdxV2595E9CY3gmdALWMmHkvFXO7tYXAYM8P",
    "type": "url_verification"
}

See the documentation here for more info.

Hold on, I need a backend endpoint to verify the Events API and then when someone adds an emoji to a Slack message.

Here’s where Spin comes in 👉🏽

Create an endpoint using Spin

Spin is an open source developer tool for building and running serverless applications powered by WebAssembly. To install Spin, check out the guide here.

1. Once you’ve installed Spin, let’s create a new app. This guide has code samples in Python but you can choose any of the other languages in the Spin SDK to get started:



spin new 
Pick a template to start your application with: http-py (HTTP request handler using Python)
Enter a name for your new application: reaction-emoji-bot
Description: 
HTTP path: /...



cd reaction-emoji-bot
code .

2. That was easy! Now let’s get our hands dirty with some code. Open the folder in your favourite code editor and look for `app.py` which is the starting point for your code. We have to first write code to verify the Slack Events API. We can do so by parsing the `request` object for the `challenge` parameter and returning it via the `response`



def handle_request(request):
    data = json.loads(request.body)
    challenge = data.get('challenge', None)
    return Response(200,
                    {"content-type": "text/plain"},
                    bytes(challenge, "utf-8"))

3. The code above is straightforward - You can extract the `challenge` parameter by parsing the `request.body` . The response is literally just the `challenge` parameter. The Slack API is actually flexible in how you can send this value back. According to them you can verify the request's authenticity and then respond in plaintext or with `application/x-www-form-urlencoded` or even in JSON.

Given that this needs to be a ‘live’ endpoint - we have to build and deploy this. Go back to your terminal and type in spin build which will build your Spin app. In case you’ve built the above using TypeScript, do a npm install before to install some dependancies.
Now that the Spin app is built, deploying it to the cloud is straightforward. Just do a spin deploy and the app deploys to the Fermyon Cloud. (Assuming you’ve already logged in to the Fermyon Cloud)



> spin deploy 
Uploading reaction-emoji-bot version 0.1.0 to Fermyon Cloud...
Deploying...
Waiting for application to become ready.................. ready
Available Routes:
  reaction-emoji-bot: https://reaction-emoji-bot.fermyon.app (wildcard)

Congratulations! Your endpoint is now live. All we need to do now is give the appropriate permissions for your Slack app to access messages, and then write code to listen to the emoji-added event.

Listening to the Events API

1. Let’s first give your Slack App permissions to read messages on Slack. On api.slack.com, click on ‘Oauth & Permissions’ in the left menu that’s in the ‘Features’ section.

2. Scroll down to the ‘Scopes’ section and add the following scopes to your Slack App. Depending on what your app does, it may require more permissions.



chat:write, im:history, incoming-webhook, reactions:read

You also need to subscribe to bot events. In the ‘Event Subscriptions’ section, scroll down to ‘Subscribe to bot events’ and add the following.



message.im, reaction_added

Again, if you expand the scope of your bot, ensure you add the relevant events.

3. Now that your app has permissions, you can write some code in your Spin app to listen to events such as a reaction emoji that’s added to a message. Go back to your code editor and add the following code:



def handle_request(request):
    json_data = request.body.decode('utf-8')
    # Parse the JSON string
    data = json.loads(json_data)

    # Access the 'reaction' value
    reaction_value = data['event']['reaction']

    # Print the result
    print(reaction_value) 

    # do something

The code again is straightforward. When any event occurs, there is a HTTP POST request with a JSON payload that looks something like this (I’ve redacted some tokens)

The event parameter has the type of event as well as the specifics, in this case the reaction



{
    'token': 'XXXXXXXX',
    'team_id': 'XXXXXXXX',
    'context_team_id': 'XXXXXXXX',
    'context_enterprise_id': None,
    'api_app_id': 'XXXXXXXX',
    '**event**': {
        **'type': 'reaction_added',**
        'user': 'XXXXXXXX',
        **'reaction': 'white_check_mark',**
        'item': {
            'type': 'message',
            'channel': 'XXXXXXXX',
            'ts': 'XXXXXXXX'
        },
        'item_user': 'U06CRMCJW',
        'event_ts': 'XXXXXXXX'
    },
    'type': 'event_callback',
    'event_id': 'XXXXXXXX',
    'event_time': 17035280,
    'authorizations': [{
        'enterprise_id': None,
        'team_id': 'XXXXXXXX',
        'user_id': 'XXXXXXXX',
        'is_bot': True,
        'is_enterprise_install': False
    }],
    'is_ext_shared_channel': False,
    'event_context': '4-XXXXXXXX'
}

The Python code above just parses the JSON to extract the reaction. You can choose to do anything once you’ve received an event with the type of emoji reaction that has been added.

For example: Approve a post if someone adds a ✅ emoji.

4. Slack requires the endpoint for verification to be the same as the endpoint that is listening to any events. So we can route a request based on the payload. Just add this `if` condition. Again, if you are listening to multiple events it might be better to use a `switch` case or a different method of routing. Check out Rajat’s code here where he’s done the same



from spin_http import Request, Response, http_send

import json

def handle_request(request):

    json_data = request.body.decode('utf-8')

    data = json.loads(json_data)

<span class="c1"># Check if it's a URL verification request



    if 'challenge' in data:

        challenge = data.get('challenge', None)

        return Response(200, {"content-type": "text/plain"}, bytes(challenge, "utf-8"))

    else:

        # It's an event notification

        # Access the 'reaction' value

        reaction_value = data['event']['reaction']

        print(reaction_value)

    <span class="c1"># Do something fun with the emoji reaction here

You’re almost there! Let’s `spin build` and `spin deploy` your app. Once that’s done, go to your Slack Apps and add your app to the workspace to test it out. In your app’s page, click on ‘Install App’ in the left menu and then choose the Slack Workspace and channel your bot can post to.

6. You can double check to see if the bot has been added to the channel

To debug your code or look at the logs, you can login to [cloud.fermyon.co](http://cloud.fermyon.co)m and click on the app that you’ve just deployed. Click on logs on the left and you will see that all your print statements and events are logged here in real-time. You can even have a custom domain for this endpoint by clicking on the Custom Domains section in the left menubar.

Next Steps

That was a simple tutorial on how you can create a simple webhook that listens to an emoji reaction. You can expand this to do all sorts of cool Slash commands in Slack. You can also use this with custom emojis. You’ll notice in the first screenshot at the top of this page, there’s a 🍇 emoji. That’s the reaction emoji we used (Ask us why in the comments, there’s a fun story there 😄)

Also, check out how fast the responses are! This is because Spin uses WebAssembly which has no cold-starts. This makes it ideal for Slack bots and webhooks.

*In Part II of this article, we will continue this and see what we can do when a user adds a emoji to a message. *

Let us know what type of Slack bots you are building. We’d love to hear from you <3

Git Prom! My Favorite Git Alias

Matt Butcher — Fri, 05 Jan 2024 23:22:30 +0000

I work on a lot of GitHub repositories. In most of these, I make a fork of the repository, do my own coding, and then open PRs against the main repository.

Like any avid Git user, over time I have accumulated a number of Git aliases. But my favorite (and one of the most frequently used) is one I call git prom.

Since Git setups tend to differ, let me give a quick high-level about how I set up my clones and why.

How I Set Up My Local Clones

Typically, when working on an upstream repository at GitHub, I start with the main repo, clone it, and work on a clone. Then I open PRs to suggest merging my changes back into upstream.

So we're tracking two repos: the upstream and my own clone. I use the following naming convention:

My own repo is always called my (git clone my URL)
Upstream is always origin (git remote add origin URL)

For example, here's a snippet of my Git config for the spin repository:

[remote "origin"]
    url = git@github.com:fermyon/spin.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[remote "my"]
    url = git@github.com:technosophos/spin.git
    fetch = +refs/heads/*:refs/remotes/my/*

I find this really useful because then origin is always the "real" project, and my is always my copy. Easy to remember.

Some of my friends call upstream upstream and their copy origin. I find that use of origin confusing, but if it works for you, it's totally fine. I'll point out where you need to make changes in the alias.

The Problem: Updating my branch to the last from upstream

For a longer-running PR, it is often the case that I need to periodically update my branch with the latest from the upstream repository.

The best way to do this is to:

Pull the upstream repo
Rebase my own work on top of it

I wanted a fast way to do this that didn't involve typing lots of flags (and hence introducing typo opportunities).

The Solution: The `git prom` alias

I wrote a quick Git alias that is in my main .gitconfig file (and is therefore global):

[alias]
    prom = "!git pull --rebase origin $(git remote show origin | grep 'HEAD branch' | cut -d' ' -f5)"

First, let me explain what it does. Then I'll go into the specific command a little more.

The name gives it away; prom is shorthand for Pull Rebase Origin Main. So it pulls and rebases origin's (or upstream's) main branch.

Say I'm on a feature branch called feat/prom-example, and I've been working away on it. In the meantime, upstream's main branch has been merging new PRs. I need to re-sync. I'll use git prom to rebase my current branch on the latest main:

$ git prom
remote: Enumerating objects: 1180, done.
remote: Counting objects: 100% (327/327), done.
remote: Total 1180 (delta 327), reused 327 (delta 327), pack-reused 853
Receiving objects: 100% (1180/1180), 536.46 KiB | 2.87 MiB/s, done.
Resolving deltas: 100% (691/691), completed with 124 local objects.
From github.com:fermyon/spin
 * branch              main       -> FETCH_HEAD
   0405da75..74cd215a  main       -> origin/main
Updating 16dd1339..74cd215a
Fast-forward
 .devcontainer/Dockerfile                     |    3 -
 .devcontainer/devcontainer.json              |    1 -
 .envrc                                       |    1 +
# and more changes

At the end, I have the latest version of the upstream's main branch pulled, and then my own changes rebased on top of that.

This is a very common part of workflow, so it's nice to have a nice mnemonic alias that is more typo-resistent than the usual full command.

How The Command Works

Even though the workflow is very common, the alias is surprisingly complex. Let's take a quick look at what it's doing.

prom = "!git pull --rebase origin $(git remote show origin | grep 'HEAD branch' | cut -d' ' -f5)"

In essence, when I am on a branch feat/prom-example and I want to pull and rebase, the basic command is git pull --rebase origin main. And in most cases, the prom alias could have just been prom = "!git pull --rebase origin main".

But... it is not always the case that the upstream main branch is named main. Some still use the antiquated master. Others use their own systems for what the main branch is. (I worked on one that set the main branch name to whatever the current major release number was, e.g. v2.)

So to get around this limitation, we need to run a bit of ad hoc shell processing: $(git remote show origin | grep 'HEAD branch' | cut -d' ' -f5)

This does the following:

Run git remote show origin to see the branches that are upstream (remember: for me, origin is the name of the upstream repository. If you use a different convention, adapt accordingly).
grep 'HEAD branch' gets us the name of the branch that upstream currently considers to be the "HEAD". For GitHub, this is the branch that is the default main branch.
And cut -d' ' -f5 says "split the returned line based on ' ' (space) characters and return me the 5th (-f5) field.

So git remote show origin | grep HEAD returns

  HEAD branch: main

Note the leading whitespace. The cut command returns five fields:

[EMPTY]
[EMPTY]
HEAD
branch:
main

And we just want field 5. If we were to run just this subcommand in a terminal, here's what the output would look like:

$ git remote show origin | grep HEAD | cut -d' ' -f5
main

A Quick Note for Windows (non-WSL) Users

The alias I use works fine on macOS, Linux, WSL, and probably most UNIXy OSes. But it will not work on Windows outside of WSL. I don't happen to know how to do this in Powershell, so your best bet may be to just hard code in the default branch name with !git pull --rebase origin main.

If you know a Windows solution, please post it in the comments below. I'd love to know this.

Conclusion

My git prom alias abstracts away the specifics of git pull --rebase origin main. It still works with esoteric or unconventional upstream repos as long as I follow my origin and my conventions.

If you use a different convention (like upstream instead of origin), then all you need to do is change occurrences of origin to whatever convention you use.

Cover image generated using Bing's version of DALL-E using the prompt: "Using a blueprint illustration style, draw a terminal window used by a software developer."

DEV Community: Matt Butcher

See how to combine Rancher Desktop and Wasm for ultra-fast serverless

Running Serverless Wasm Functions on the Edge with k3s and SpinKube

I wrote up a post about how to easily build A2A-style agents. I also talk a bit about why I like A2A more than MCP.

Serverless A2A with Spin

Open Telemetry is quite a powerful tool with Wasm!

Unlocking Otel in WebAssembly - Wasm I/O 2025

We use a lot of programming languages at Fermyon -- TypeScript, Go, Rust, Python.... But for the core tooling, we chose Rust. Here's why.

Why We Chose Rust For Spin

This demo is a great example of how to use an AI/LLM for something that gives immediate value to a specific context.

AI at The Edge With Fermyon Wasm Functions

Great introduction to Fermyon's new Wasm Functions platform.

Rawkode's Hands-on Intro to Fermyon Wasm Functions

Wasm IO is coming up in a few weeks, and this will be a big topic there.

Looking Ahead to WASIp3

Jasmine Mae for Fermyon ・ Mar 17

Scaling Sidecars to Zero in Kubernetes

Zero Cost Sidecars in Kubernetes

Maximizing Efficiency With the Sidecar Pattern

Spin Apps as Sidecars

Creating Our Spin App Sidecar

Scheduler Overhead

Running More Apps in Your Cluster

Local Service Chaining

Conclusion 🎉

The History and Evolution of WebAssembly in Kubernetes

Why Wasm on the Server Side?

The Road to Running Wasm in K8s

Why We’re Excited About Spinkube

Fermyon Platform for Kubernetes Brings Hyper-Elasticity to You

Conclusion

It Took Me 20+ Years To Learn This Lesson About Dev

Coding as artisanship

"Not Invented Here" is your enemy

The more complex your code, the harder it is to debug

Writing test for your code now means supporting your code less later

Future you won’t remember what present you is thinking

Corollary: If other people can’t understand it, it will always be your problem

Conclusion

Build a Slack Emoji Bot in 20 Lines of Code

Create a Slack App

1. Go to api.slack.com and click on ‘Your Apps’

3. Nice! Your Slack app is created. We now need to add an Event Subscription to it so that it can detect when an emoji reaction is added to a message. Click on the ‘Add features and functionality’ section and then the ‘Event Subscriptions’ button

4. The Slack API has an interesting way to ensure only ‘Verified’ URLs can respond to Events. Essentially, their API sends a HTTP POST request to your endpoint, that contains a challenge parameter. They expect your endpoint to respond with the same challenge value to verify your endpoint.

Create an endpoint using Spin

1. Once you’ve installed Spin, let’s create a new app. This guide has code samples in Python but you can choose any of the other languages in the Spin SDK to get started:

Listening to the Events API

1. Let’s first give your Slack App permissions to read messages on Slack. On api.slack.com, click on ‘Oauth & Permissions’ in the left menu that’s in the ‘Features’ section.

2. Scroll down to the ‘Scopes’ section and add the following scopes to your Slack App. Depending on what your app does, it may require more permissions.

3. Now that your app has permissions, you can write some code in your Spin app to listen to events such as a reaction emoji that’s added to a message. Go back to your code editor and add the following code:

You’re almost there! Let’s spin build and spin deploy your app. Once that’s done, go to your Slack Apps and add your app to the workspace to test it out. In your app’s page, click on ‘Install App’ in the left menu and then choose the Slack Workspace and channel your bot can post to.

6. You can double check to see if the bot has been added to the channel

Next Steps

Git Prom! My Favorite Git Alias

How I Set Up My Local Clones

The Problem: Updating my branch to the last from upstream

The Solution: The git prom alias

How The Command Works

A Quick Note for Windows (non-WSL) Users

Conclusion

4. The Slack API has an interesting way to ensure only ‘Verified’ URLs can respond to Events. Essentially, their API sends a HTTP POST request to your endpoint, that contains a `challenge` parameter. They expect your endpoint to respond with the same `challenge` value to verify your endpoint.

You’re almost there! Let’s `spin build` and `spin deploy` your app. Once that’s done, go to your Slack Apps and add your app to the workspace to test it out. In your app’s page, click on ‘Install App’ in the left menu and then choose the Slack Workspace and channel your bot can post to.

The Solution: The `git prom` alias