DEV Community: Kevin Wu

Serverless Design Principles

Kevin Wu — Fri, 25 Mar 2022 05:15:48 +0000

Introduction

Serverless is becoming more and more popular these days. It has always
been an interesting space for me because I\'m interested in the
stateless functional style of programming. I started working on Lambda
functions essentially as my first project as an SDE at Amazon back in
2017, worked with one of our purely serverless data storage services at
Amazon Fashion, eventually made my way to the actual Lambda backend
team, and now at Microsoft I find myself working with Azure Functions
again. Obviously, Lambda has not really existed for that long, so I
think I\'ve basically maxed out on possible number of years of
Serverless experience. I hope to highlight some of the more interesting
differences between more \"classic\" design with servers and serverless
design.

Motivations

I wanted to start with why we would use Lambda/Functions over classic
VMs or Kubernetes clusters. The original motivation for Lambda was
mostly to save costs, but later on we noticed some efficiencies that
could only be realized at Lambda scale.

Saving costs

There are a lot of web services out there that generally serve less than
1 transaction per second, but were still costing a lot in VM usage. The
idea was that there should be an efficient way for us to schedule a lot
of these web services on the same hardware so that we can save a lot of
money.

Efficiencies at scale, minimizing resource contention with data

After running Lambda for a while, Lambda realized there were ways that
we could schedule work really efficiently. Lambda has the data to
analyze different workflows to see what their bottleneck resource is,
and schedule them such that the functions would not have to contend for
the same resource at the same time. For example, it would be optimal to
schedule a memory-heavy workflow with a compute-heavy workflow, so they
are much less likely to contend for the same resources. There is an
excellent talk by Marc Brooker about this you can find here
https://youtu.be/xmacMfbrG28?t=1310.

It\'s just easier

I\'d be remiss if I didn\'t include this, but a lot of the time, Lambda
and its ilk are just the easiest services to set up, requiring little
knowledge of server infrastructure, and that makes it much easier to use
to a broad audience. Last year, I threw together a demo of a bookstore
to give a talk on design for the CS department at my alma mater, and it
was just easier to use API Gateway backed by Lambda and DynamoDB, so I
didn\'t have to really think about servers at all.

Key ideas

These are some key ideas that will come up repeatedly in our best
practices. I\'m highlighting these specifically because they differ from
traditional \"serverful\" architecture.

Concurrency

We use the number of concurrent invocations to talk about the scale of a
Function, not requests per second or capacity, which is more
traditional. You can calculate your concurrency by multiplying your
requests per second by the expected latency of the request, see
Little\'s Law. There is
actually quite a lot of content in Marc\'s talk that I linked to earlier
about this if you want a deeper dive on why this is the case. The main
reason is that concurrency is a measure that doesn\'t depend on hardware
(hence serverless right?). Concurrency also takes into account how
efficiently you\'re responding to requests rather than just how many
requests you\'re getting. One common ticket I\'d see at Lambda would be
a team wondering why they were being throttled when they had low
requests per second (more on this later).

Cold starts

The big issue with serverless that people like to talk about is higher
latency from cold starts. A cold start¹ is essentially when your
execution environment needs to both prepare for the execution, and then
actually perform the execution. At Lambda, we called these stages Init
and Invoke. It\'s not uncommon to see cold starts that are over 10
seconds, especially if you\'re not careful. I\'ve also seen many tickets
about this in my time at Lambda.

Quick detour on code/data reuse.

I also wanted to include a quick interlude about how persistent
resources get reused because I think it\'s practical for those new at
writing these Functions.

Class members

We talked a little bit about init and invoke. Intuitively, you\'d
think well I\'m creating some resources, let\'s say a DB client, during
the init phase, but how does it get reused? Do I instantiate a new
client per request, or can we reuse resources across invokes? The
answer, thankfully, is that you do in fact reuse these resources
across invokes. If you initialize your DB client outside of your
function definition, it will be reused the next time you invoke, so
members of your class probably behave as you would expect coming from a
more serverful background. Let me illustrate this with a quick example.

class Function:
    db_client = None

    def handle_invoke(event, context):
        if db_client is None:
            db_client = DbClient()

        return db_client.get_some_stuff()

On your first invoke, you will instantiate a new DbClient{.verbatim},
but on subsequent requests to the same execution environment,
db_client{.verbatim} will continue to already be instantiated, so you
will be able to get_some_stuff{.verbatim} directly.

The more perceptive readers may notice a potential problem here. If
you\'re following security best practices using ephemeral credentials
that expire after some hours, you could experience credentials issues by
reusing these clients. If you\'re using the AWS SDK, this will be
generally handled for you, but otherwise it\'s something to keep in mind
as you\'re developing.

File System

This is a quick one. If you use EFS with Lambda or by default with Azure
Functions, your invokes all share the same file system. This was
originally meant for machine learning workflows, where people have
relatively large training models they want to load, but has plenty of
other uses.

Best practices

Here are a collection of best practices that I found to be less commonly
reported, but are really helpful with design.

Keep functions short (like less than 2 minutes)

If you read the Azure Functions best practices, they say that you should
keep your functions short because they timeout. This is true, of course.
Lambda used to time out at 5 minutes, but now has extended to 15
minutes. So you might think, well, if I limit my function to 10 minutes,
I should be pretty safe. Unfortunately, there is more nuance here.
Earlier, I introduced the concept of concurrency and Little\'s Law.

::: center
L=\lambdaW{=latex}
:::

L{.verbatim} is the concurrency. λ is the effective arrival rate
(requests per second), and W{.verbatim} is the wait time (latency).

For example, the largest throttling point for Lambda in one of the
regions is 3000 concurrent invokes. If you consider a 10 minute function
(600 seconds), we can calculate the λ, or requests per second at which
you\'ll be throttled.

::: center
3000=λ * 600

λ = 3000/600

λ = 5
:::

Oops, just 5?

From the equation itself, you can see that the higher the latency, the
lower the requests per second your function will be able to handle. Even
using the default numbers, the λ decreases unintuitively quickly as W
increases.

Generally speaking, Lambda is designed to execute short functions, and
other tools such as AWS Batch or ECS are better suited for longer
running jobs.

Cold start related

Let\'s dig a little deeper on the steps to init. Generally speaking,
no matter the serverless environment, we have to do these things.

Acquire an execution sandbox
Pull the code/executable into the sandbox
Start the executable runtime e.g. JVM or CLR
Run your init code.

Keep functions small (like under 50MB)

We\'ve also already talked about cold starts. One of the most
unintuitively slow parts of a cold start is actually pulling your
code/binary. I suggested the limit of 50MB or so mainly because lots of
people insist on using Java or C# to write functions, but in reality if
you\'re using an interpreted runtime like python or node, you can easily
keep your code under 1MB. I mean in this case Java and C# will be like
50x slower, and that\'s just to download the executable.

Use an interpreted runtime for more predictable results

While we\'re on the topic of runtimes, try to use a fast runtime. JVM
and CLR have a reputation of taking a long time to initialize. I would
routinely see such functions take upwards of 10 seconds to initialize.
While JVM and CLR languages generally execute faster than node or
python, (if you\'re following the earlier advice about function
duration), you are spending a much higher proportion of your time in
init causing more latency instability when you do hit a cold start. In
my experience, node has been a good choice for having a more consistent
experience. You can also bypass the runtime completely and pick a
compiled language like Golang or Rust². I\'ve also been on teams that
used Golang to great effect with Lambda as well.

Be mindful of price

It\'s easy to get lost in the ease of using Functions and forget how
much you\'re spending. I\'ve accidentally spent thousands of dollars (of
Amazon\'s money) in just a few days. The pricing model is a constant
cost per invoke and then a rate on GBs, where GB is how much memory
you\'ve allocated (not how much you\'re using) and seconds of invocation
rounded to the nearest millisecond. This means the less you use, the
less you pay, but also conversely, the more you use, the more you pay.
It\'d be good to look up how much it would cost to get a VM or
Kubernetes to do the same job and make sure you\'re willing to pay the
excess. I\'ve found that the point where Lambda starts costing more
happens at a much lower concurrency than people generally think. Of
course, functions do more than VMs. They scale automagically, do OS
patching, etc, so it may be worth it to you, but you should at least
know how much you\'re paying for that.

Assorted Tips

Lambda currently doesn\'t charge for inits under 10s

Lambda doesn\'t really like people spreading this particular tidbit, but
this is relatively widely known now. In order to optimize cold start
times, your init phase is generally run on a more powerful sandbox
and you don\'t have to pay for it. The caveat is that if you spend
more than 10 seconds in the init phase, the sandbox is restarted and
you will be charged for init. There is a blog post that went
relatively viral about this phenomenon:
https://hichaelmart.medium.com/shave-99-93-off-your-lambda-bill-with-this-one-weird-trick-33c0acebb2ea.

Init can happen before you actually make an invoke request

This one surprises customers from time to time, but in order to optimize
cold starts, the platform can initialize your execution environment well
before you actually make an invoke. Especially if you make use of the
dependency injection features. Reworking the previous example a little
bit to show a common way that this comes up:

class Function:

    def __init__():
        self.logger = SomeCustomLogger()
        self.db_client = DbClient()
        logger.logInfo("db_client is initialized")

    def handle_invoke(event, context):
        return db_client.get_some_stuff()

If we log the initialization of db_client{.verbatim}, you can see that
log statement, even if you did not make a request.

Don\'t use Task without returning something in Azure Functions (when you can)

This is more off a quirk of how async/await works in dotnet, but plain
Task{.verbatim} functions are syntactic sugar for void{.verbatim}.
But since execution metadata is saved in the Task{.verbatim} object,
if you use just plain Task{.verbatim}, you return void{.verbatim}
and you lose all your execution metadata. For example, if you return
Task{.verbatim}, all Exceptions are suppressed because there is
no way to return them to the caller. For that reason, I suggest at least
returning something like Task<bool>{.verbatim}, which will throw
Exceptions as expected.

Actually, within Lambda we had several levels of cold starts,
which I get into a little bit later, but we\'re keeping it simple
for now. ↩
Shoutout to https://github.com/awslabs/aws-lambda-rust-runtime.
But probably you\'ll have issues with Rust\'s really large compiled
binaries. ↩

Moving to Windows from Linux Pt 1

Kevin Wu — Tue, 05 Oct 2021 11:17:20 +0000

I've started recently at Microsoft and being a die-hard Linux user for some time, I missed a lot of the cool features I had been using on both Linux and MacOS. In this post, I'm going to talk about some of the tools that I use day to day and how I replaced them on Windows. Even though I used Windows growing up, I was not very technical then, so I still feel like I'm new to the Windows world, so do let me know if there are any other tools that I should know about. Hopefully, this can help some other Linux users who are trying to get comfortable in a Windows environment. I wasn't able to cover all of the tools that I found in this post, so I'll be adding more parts to this as I have time.

Package manager (pamac, brew -> scoop)

Lets start with how we can install the software that I will be talking about in later sections!

Most Linux users are used to some sort of package manager, and I have used quite a lot of them over the years like apt, dnf, yum, pacman, nix + home-manager, and homebrew + linuxbrew. After trying so many, personally, I felt that choosing the path of least resistance is usually the easiest for me because the software tends to be more compatible with each other if I don't mix them or use the one the system intended. Manjaro recommends pamac, so I use that. And on Mac, Homebrew is the staple in this category.

I know actually these installers are somewhat different. Pamac and its kin actually install onto the system (which is why you need sudo), but I like Homebrew's approach of installing at the user level (which is why you should not use sudo). I am the only user on any of my computers (as are most people these days), and I like the idea of not messing with my system when I don't need to, so I generally prefer to install things as a user. The good people at Apple and Microsoft work hard, after all, to make the System OS stable, and I don't like to mess with their work. Actually, it's even more annoying on Windows because you can't just sudo, you have to open an entire new window as Admin (without any of your user customizations) in order to install software on the system.

In the Windows world, I found that there's chocolatey, scoop, and winget that are generally used for this purpose. I don't really have access to winget yet, so I ruled that one out pretty quickly. Chocolatey installs on the system rather than as the user, so I chose scoop since I prefer to install as a user. That being said though, I did install chocolatey anyway because their repositories tend to have more software and sometimes that's the only way for me to get software.

(Pseudo) Tiling Window Manager (pop-shell, amethyst -> komorebi)

Do you ever get lost in all the windows that you have open? Tiling Window Managers help you keep your windows organized so you can spend more time doing things rather than arranging your windows. I also like that you can drive your entire desktop experience using your keyboard.

More die-hard Linux users use full-on window managers e.g. i3 or awesome, but I actually prefer to still have a desktop environment, so I use scripts that emulate some of the most important behaviors for me. For most linuxes, I use either pop-shell on GNOME or krohnkite on KDE. On macOS, I use amethyst.

Pop shell is pretty different from most tiling window managers, but I think it finds a good balance between usability and feature set. On the other hand though, it's hard to find a direct replacement for it. Ultimately (tldr), I settled on komorebi for Windows.

At first, I used the Microsoft PowerToys, where I found an issue about emulating pop-shell. Eventually in that issue, I found a list of various projects trying to emulate tiling window managers on Windows. At first, it was pretty overwhelming because there are a ton of little-used projects. They were mostly either really young or no longer maintained. Luckily (or maybe not so luckily), I mostly had trouble actually installing a number of them due to being pretty inexperienced with installing OSS on Windows, so I was able to narrow it down quickly to a few projects that had relatively easy install instructions. I started off using workspacer, which worked really well out of the box, but I had trouble configuring it because I didn't know a lot of dotnet or C#.

I liked komorebi the best because it was using a familiar technology (rust, cargo, autohotkey) and was intuitive to install and configure how I needed it. Since I have to use dotnet and C# for work, I imagine that eventually it'll be easier for me to use workspacer, so I think it might be interesting to try that one again later.

System stats monitor (system-monitor, iStat Menus, powerline stats -> Xmeters)

Do you ever run a command and wonder if it actually worked? Especially on an ssh session, I find it comforting to see that the computer is actually doing something. It's also nice to have some system stats visible at all times so that when your computer freezes, you have a general idea as to why that happened.

Currently when on Linux, I use system-monitor or the built-in KDE widgets. Alternatively, on an ssh session, I use tmux and powerline plugins. On macOS, I use iStat Menus, which work decently well but don't really fit in with the rest of the more elegant macOS aesthetic, or at least the free version.

This was a pretty quick fix for me. I just searched it quickly on google and came up with XMeters.

System monitoring software (Activity Monitor, Gnome System Monitor -> ProcessExplorer)

I'm normally not super picky about this kind of software, and having used Windows throughout my childhood, I am pretty familiar with Task Manager, but there is no way to even search for a process. While googling this issue, I found that a lot of people use Process Explorer, which has a lot of features that Task Manager lacks. I'm honestly not sure how Windows users have gone this long without even the ability to search for a process built-in.

Set Up Grpc Web Server With AWS

Kevin Wu — Sat, 01 May 2021 12:20:57 +0000

Intro

I was asked on one of my last posts about gRPC, how does one host a gRPC web server. I thought it would be a good opportunity to learn what the AWS customer experience is, currently, for setting up a simple gRPC web server. If you are advanced at gRPC and AWS already, you could probably set it up just using this post: https://aws.amazon.com/blogs/aws/new-application-load-balancer-support-for-end-to-end-http-2-and-grpc/, which is what I am using to do this. But if you need some guidance on how to use these components, like I did. The following may be helpful to you.

Setting up an EC2 instance

My first step is setting up an EC2 instance, which will run our gRPC web server. From the AWS console,

select EC2.
select "Instances"
select "Launch Instances"
Next, you can configure your EC2 instance. In case you are unfamiliar, here is what I did:
1. Select Amazon Linux 2 AMI on 64-bit (x86)
2. Choose t2.micro (Because it's free)
3. Click review and launch and then, on the next page launch
Make sure that you add TCP on port 50051 as an open inbound port for your security group.

Set up an ALB for my EC2 instance

Technically, you already have a web server you could host your gRPC web application on, but I wanted to try setting up a load balancer as well.

Click "Load Balancers" on the left
Click "Create Load Balancer"
Click "Application Load Balancer"
1. Enter any name (I chose "test-alb")
2. Under Listeners, choose HTTPS and port 50051
3. Under Availability Zones, select two AZs but make sure the one that your EC2 instance was created in is one of them. For me that was us-west-2d, but you can check on the EC2 console where yours is.
4. Click Configure Security Settings

Creating a certificate

This was one of the toughest parts for me, since I didn't actually have a real certificate to use. Luckily, you can create a self-signed certificate and use it here just fine, but it requires a little extra work. Note this is not a best practice, but it will work for this little demo. First, on my computer I ran¹:

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout example.key -out example.crt -subj "/CN=example.com" -addext "subjectAltName-DNS:example.com,DNS:www.example.net,IP:10.0.0.1"

Then I tried to use this certificate in various ways, by adding it in the Load Balancer creation UX with both ACM and IAM, and also adding it into ACM through the ACM UX and then trying to choose it here. Unfortunately, they all didn't work for me. Instead what did work for me was importing it into IAM and choosing it here. This is not a best practice, but it's what we're doing here for convenience.

aws iam upload-server-certificate --server-certificate-name test-cert --certificate-body file://example.crt --private-key file://example.key

Then, I was able to find this certificate in the Load Balancer creation UX.

Load Balancer Creation Continued

For Security groups, I just chose the security group my EC2 instance belonged in.
For Configure Routing, I created a new target group (test-target-group)
1. Choose Instance for target type
2. Protocol: HTTPS
3. Port 50051
4. Protocol Version: gRPC
Register targets: I chose my EC2 instance here.
Click review and then create.

Running a gRPC server on the instance

Next, we want to start some gRPC server on port 50051 so we can do an end to end test.

I'm starting by running rustup so I can create a simple tonic server.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
cargo new server
cd server

Next, I open up Cargo.toml and added tonic and tonic-health. e.g.

[dependencies]
tonic = "0.4"
tonic-health = "0.3.1"
tokio = { version = "1.5.0", features = ["full"] }

Then cargo build. This failed for me because it was missing cc, which is strange because it looks like glibc is actually installed. But, I just ran sudo yum groupinstall "Development Tools" which installs a collection of tools, and it worked after that.

Once that's done, open up src/main.rs and put in

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (mut health_reporter, health_service) = tonic_health::server::health_reporter();
    health_reporter
        .set_service_status("".to_string(), tonic_health::ServingStatus::Serving)
        .await;

    let addr = "0.0.0.0:50051".parse().unwrap();

    tonic::transport::Server::builder()
        .add_service(health_service)
        .serve(addr)
        .await?;

    Ok(())
}

Start your server with cargo run

Testing a request to the Load Balancer

If everything is set up correctly, you should now be able to use evans to access your web server at the load balancer url or even the url for your ec2 instance directly e.g.

evans --host <your url> --proto health.proto -p 50051

You can get the health.proto file here:
https://github.com/hyperium/tonic/blob/master/tonic-health/proto/health.proto

Conclusion

Hopefully, this will help other people who are interested in setting up a gRPC web server on AWS. There doesn't seem to be a lot of documentation out there on this specifically, so I hope that this article will be useful.

This actually didn't work for me on my macOS machine because the openssl typically in the path on macOS is actually LibreSSL. You should instead brew install openssl and use that binary's absolute path. ↩

Designing an Online Book Store

Kevin Wu — Mon, 12 Apr 2021 10:38:07 +0000

Introduction

I've been given the opportunity recently to do a small tech talk at a Hackathon taking place at my alma mater. The goal is to both teach how software engineers build something as well as demonstrate it to them live. I wanted to pick a topic that would be somewhat representative to students of what I think most engineers work on, but also something that I had experience with. Having worked in e-commerce for a long time, I decided implementing a store using AWS would be the best way to demonstrate that. Maybe it's generic, but it is an easy-to-understand use-case because most college students have some experience using an online store and building something using cloud tools is likely to be helpful for them in their career in any kind of software engineering (or, at least for the hackathon). It also covers the kind of skills that people want to capture in a systems design interview question (for what that's worth).

This will be the corresponding blog post in which I cover the same content except it focuses more on the experience of designing the system and my thought processes as I make those decisions. The talk will focus more on the how-to aspect of setting up the project and using the associated tools. Therefore, in this post, you see the more disorganized process of me designing something and maybe backtracking later and changing it a little etc.

Requirements

Let's start with what do we want the store to be able to do.

We should be able to browse products. We can implement ~3 products in our example.
We should be able to add some products to cart, and this cart should persist across browsing sessions. There are a lot of definitions of "session" in the front-end world, but I mean session as in the layman's terms here e.g. closing your browser and revisiting the same url.
At the end of our "session", we should be able to buy the items in our cart and that order should be logged somewhere.

I think those are the basic elements for a store. Of course, I'm skipping a lot of details for simplicity here. I've worked most of my career to this point in Order Validation and Fulfillment, which both are conveniently ignored here.

Data store

I like to start my design with how will we store data. Given the requirements that we've identified, we should identify what kind of data we need to store.

Product information. In order to browse products we need to display information about these products for the customer. Let's call this the catalog.
Session information. We need to know what products customers have in their cart for the checkout stage.
Order information. We need to know what customers ordered and how to fulfill that order. We need this to actually ship the product to customers but also for legal and tax reasons.

There are lots of ways to store data and data storage is a complicated topic in itself, but I am planning to do all of this with DynamoDB. The main reason is that I want to help students with their Hackathon. Therefore it's simplest to choose some kind of serverless solution to decrease the complexity of the project. This leaves either DynamoDB or S3, and DynamoDB makes the most sense for these things.And although serverless SQL is starting to exist nowadays, I think that NoSQL makes the most sense for the Hackathon too because we don't have the time to do the proper kind of table design during a Hackathon that SQL really requires.

We still have to decide exactly how we want to structure our data, so the "data store" aspect of this design is not done, but one of the advantages of NoSQL is that we can decide this later or even as we implement, so I'm moving on to the API design.

API -- Or, a brief detour into the definition of an API

At least when I was in college, I didn't really know what API meant, so here's a quick and dirty explanation. I see a lot of people trying to explain what an API is in light of the Oracle V Google Supreme Court case, and they always just say API stands for Application Programming Interface. I feel that doesn't really help at all, so I will try to describe it differently. In short, software tends to be architected in layers, and an API is the general catch-all term for the ways software layers interact.

So, for example, in python, in order to write to a file, you could use the following code:

with open("text.txt", "w") as file_handler:
    file_handler.write("Some string")

Writing files is complicated and depends on your Operating System, your File System, and even the actual kind of disk you have. Generically speaking, you make a request to your Operating System, which in turn makes a request to your File System, which in turn, makes a request to your actual hardware.

Something like this:

But python takes care of this for you, you write this same code regardless of whether your working with:

macOS, APFS, NVME SSD;
Windows, NTFS, HDD;
or; Linux, ext4, SATA SSD

That snippet of python code is described as the File API for python. It's how python programs interact with Files. This allows programmers to create complicated systems without necessarily understanding how each piece of it works as long as they understand the concepts the API designer has created for them.

API Design

If the data storage section are the nouns in this project, the APIs are the verbs. What can we do with these data? Let's break these down by the requirements again.

Browsing Products

Primarily, we need a way to read the catalog data. Let's call this

getProduct: (productId: str) -> (Product)

It will take a productId which will be an uuid and return a JSON representation of your product.¹

Saving a customer's cart session

For this one, I expect customers to need to add products to their cart;

addToCart: (customerId: str, productId: str) -> ()

be able to list the items in cart;

getCart: () -> List<Product>

and remove items from cart;

removeFromCart: (customerId: str, productId: str) -> ()

Note from this, you start to see the benefit of creating these APIs and not using the database directly. You could just read from the database using their API from the client side, but this way, you limit the ways that a customer can interact with the data.

Checkout

For checkout, we will be needing the same getCart that we created earlier. In addition to that, we'll need a way to create orders:

makeOrder: (customerId: str) -> (orderId: int)

makeOrder takes a customerId, and returns an int representing the orderId, which will be visible to the customer, so I've chosen a simple int to display to them.

For these APIs, I'm planning to use API Gateway backed by Lambda because I think serverless will be the easiest to use for the Hackathon (but also I'm biased).

Customer Interaction

Generally, in this part of the design, we want to imagine the way that customers would interact with this system. For a web application this would be the web page or pages that they would use to shop. Often as a software engineer, this part of the design is usually done in advance. Luckily for the end users, typically Product Managers and/or UX Designers will design the general interface, and engineers only implement it.

Since I have limited front end experience, having only worked in the front end for a few projects here and there and that front end is generally on the client side and doesn't have too much interaction with cloud architecture. I'm probably going to just make a quick website for demo purposes but not really include it in this design process.²

My next post will be the how-to post on how to set this up using AWS.

Why an uuid instead of just a number? It can be a problem for customers to be able to guess your product ids, but it can also be used by competitors to guess how many products you have. See https://en.wikipedia.org/wiki/German_tank_problem ↩
Plus it took me two hours to embed the html for that one graphic in this post, so maybe I shouldn't be teaching front end development. ↩

Oncall superstition and the Return to Mean Bias

Kevin Wu — Mon, 12 Apr 2021 10:22:37 +0000

I've been on call this week, so I haven't had a lot of time to write something out. But one thing I noticed relating to on call is the superstition that: acknowledging that there haven't been a lot of pages will, ironically, cause you to get paged. I'm not sure that there are many people who literally believe that or if it's just a bit of a running joke in the industry. But why is this belief so widespread?

One thing I've enjoyed during my oncall this week was watching the Medlife Crisis youtube channel and particularly learning about how on call works for doctors. One of his videos was about how bias affects medical research ¹. While, watching this, I learned of a particular bias called Return to Mean Bias that describes this phenomenon. In fact, he uses basically the same example in his video with his own on call shifts that he works.

The logic behind this is simple. If you assume that if you graph your number of incidents over time, you'd fluctuate around a mean value. Then, when you are experiencing a slow on call without a lot of incidents, you are probably in the middle of one of the lull periods in your graph, and therefore more likely to notice that it has been slow. Eventually, though, the number of incidents will have to return to that mean value, so after a lull you're actually more likely to have a rush of incidents.

Take a look at this visual lovingly designed by my girlfriend:

The blue line is the mean number of incidents. When you have a slow period, you'll be about to ramp up back toward the mean as long as the mean has not changed.

That's it. It was interesting to learn that this on call superstition is a probability trick. I will hopefully have more interesting content when my shift is over.

https://www.youtube.com/watch?v=ooWA4tM_gUs ↩

Setting Up a gRPC Protobuf Server With Tonic

Kevin Wu — Mon, 12 Apr 2021 10:18:26 +0000

Recently I've been starting to work with gRPC and protobuf. Before this, I was used to using just plain JSON and HTTP.

gRPC is a system developed by Google for fast RPC-style communication between miroservices. It uses HTTP/2 as its transport protocol, and by default, it uses protobuf (analogous to JSON) as its serialization format.

I really like the idea of having a strong contract between the client and server. I've always been interested in type safety and proofs as code. Traditionally, with JSON, the only contract between client and server is that the reply more or less is in the JSON format and will fit the JSON specification, but you can get no guarantees regarding the structure or data that is transferred. With protobuf, the client and server agree on a contract beforehand with proto files that define a structure and datatypes. This would allow us to create more powerful proofs that span more than one service in a miroservice architecture. And of course, gRPC promises to be a performant protocol because it sends nearly as few bytes as possible for each message, which generally translates to speed over the network.

This post documents my experience setting up an example project with rust and tonic, which is maintained by one of my coworkers, Lucio. I will set up a simple server, add some other APIs and set up a second server to do health checks.

Set up a new cargo project

$ cargo new grpc-demo
    Created binary (application) `grpc-demo` package

$ cd grpc-demo
$ cargo add tonic

If you're used to rust, this shouldn't be difficult to follow. It's the general way that one starts a project in the rust world. You may need to install cargo-edit in order to have cargo add, but it's well worth it for managing dependencies in your cargo project.

Set up a simple tonic server

I figured I should start with the helloworld example in the tonic examples directory. I started with a few more dependencies that I noticed are being used in the example.

cargo add anyhow
cargo add tokio

Later on, I realized that tokio generally wants to be added with a list of features, so I've adjusted the Cargo.toml in my project, so that it looks like:

[dependencies]
anyhow = "1.0.40"
tokio = {version = "1.4.0", features = ["full"]}
tonic = "0.4.1"

Creating the protobuf files

At this point, I was creating the proto files in my project. I've never used protobuf before, so I expected this to be one of the most challenging sections.

My first instinct was to create the proto files in the src folder with the main.rs file.
It ended up looking like this:

syntax = "proto3";

package helloworld;

service Greeter {
    rpc SayHello (HelloRequest) returns (HelloReply);
}

message HelloRequest {
    string name = 1;
}

message HelloReply {
    string message = 1;
}

Learning to include the protobuf files into our `main.rs`

When returning to my main.rs, I added

pub mod hello_world {
    tonic::include_proto!("helloworld");
}

But this didn't work out. My linter, rust-analyzer, gave me an error "OUT_DIR not set, enable "load out dirs from check" to fix"". This had recently been a bug with rust-analyzer, but even with building with cargo build, I still got "environment variable OUT_DIR not defined". I knew that this had something to do with how tonic generates code for the proto files.

The original message from rust-analyzer was confusing to me because I remembered turning it on, but I couldn't find it anymore. Rust-analyzer is always changing quickly, so my first instinct was that maybe they had changed the configuration recently. Looking at the issues for rust-analyzer, I found one that said the setting had been renamed to runBuildScripts. This one happened to be on in my configuration, so I decided it might be better to look into the other error message.

After googling the other error message, I noticed a few issues on Github where people had the wrong directory structure, and this seemed very likely to me because I had just guessed the directory structure when I put my proto files in src.

Looking through the tonic documentation, it looked like I would need to compile the proto files.
This being my first time working with generated code and deviating from the default build process for rust, I quickly realized that I'd have to create a build.rs file in the project root (not src), and include

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::compile_protos("proto/helloworld.proto")?;
    Ok(())
}

This also required me to cargo add -B tonic-build, create a new proto directory for the proto files, and move helloworld.proto into that directory. With all of those changes, my code started to build.

One more interesting thing I found is that normally you can't make a function in a trait async. I got the error message

functions in traits cannot be declared `async`
`async` trait functions are not currently supported

but tonic gets by this with the #[tonic::async_trait] macro on the trait.

Finally, writing out our simple server

From there on out, everything else was straightforward. I simply needed to create a struct that implements the hello function that you want, and start the server. My final result looked like this:

use hello_world::{greeter_server::{Greeter, GreeterServer}, HelloReply, HelloRequest};
use std::net::SocketAddr;
use tonic::{Request, Response, Status, transport::Server};

use anyhow::Result;

pub mod hello_world {
    tonic::include_proto!("helloworld");
}

#[derive(Default)]
pub struct MyGreeter {}

#[tonic::async_trait]
impl Greeter for MyGreeter {
    async fn say_hello(&self, request: Request<HelloRequest>) -> std::result::Result<Response<HelloReply>, Status> {
        let reply = HelloReply {
            message: format!("Hello {}!", request.into_inner().name),
        };

        Ok(Response::new(reply))
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let addr: SocketAddr = "127.0.0.1:8000".parse()?;
    let greeter = MyGreeter::default();

    Server::builder()
        .add_service(GreeterServer::new(greeter))
        .serve(addr)
        .await?;

    Ok(())
}

Testing the server

After running the server with cargo run, I needed a way to test that the server works. I had heard of an interesting tool called evans, so I decided to use this.
It took me a while to figure out the right parameters to query the server, especially because tonic doesn't seem to support gRPC reflection right now, and there are few examples out there.

What ended up working for me was:

evans --proto proto/helloworld.proto -p 8000

This was particularly confusing because there is an argument called --path, which doesn't work.

However, after figuring out how evans works, I was able to call my service.

helloworld.Greeter@127.0.0.1:8000> call SayHello
name (TYPE_STRING) => "kev"
{
  "message": "Hello \"kev\"!"
}

It works!

Adding a new API

To check my understanding, I wanted to add a new API that is like Greeter but says goodbye instead.

I started with adding my new rpc and messages in the proto files.

syntax = "proto3";

package helloworld;

service Greeter {
    rpc SayHello (HelloRequest) returns (HelloReply);
    rpc SayGoodbye (GoodbyeRequest) returns (GoodbyeReply);
}

message HelloRequest {
    string name = 1;
}

message HelloReply {
    string message = 1;
}

message GoodbyeRequest {
    string name = 1;
}

message GoodbyeReply {
    string message = 1;
}

I added a new function in main.rs.

use hello_world::{
    greeter_server::{Greeter, GreeterServer},
    GoodbyeReply, GoodbyeRequest, HelloReply, HelloRequest,
};
use std::net::SocketAddr;
use tonic::{transport::Server, Request, Response, Status};

use anyhow::Result;

pub mod hello_world {
    tonic::include_proto!("helloworld");
}

#[derive(Default)]
pub struct MyGreeter {}

#[tonic::async_trait]
impl Greeter for MyGreeter {
    async fn say_hello(
        &self,
        request: Request<HelloRequest>,
    ) -> std::result::Result<Response<HelloReply>, Status> {
        let reply = HelloReply {
            message: format!("Hello {}!", request.into_inner().name),
        };

        Ok(Response::new(reply))
    }

    async fn say_goodbye(
        &self,
        request: Request<GoodbyeRequest>,
    ) -> std::result::Result<Response<GoodbyeReply>, Status> {
        let reply = GoodbyeReply {
            message: format!("Goodbye {}!", request.into_inner().name),
        };
        Ok(Response::new(reply))
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let addr: SocketAddr = "127.0.0.1:8000".parse()?;
    let greeter = MyGreeter::default();

    Server::builder()
        .add_service(GreeterServer::new(greeter))
        .serve(addr)
        .await?;

    Ok(())
}

I did a sanity check with evans to show that it worked:

helloworld.Greeter@127.0.0.1:8000> show service
+---------+------------+----------------+---------------+
| SERVICE |    RPC     |  REQUEST TYPE  | RESPONSE TYPE |
+---------+------------+----------------+---------------+
| Greeter | SayHello   | HelloRequest   | HelloReply    |
| Greeter | SayGoodbye | GoodbyeRequest | GoodbyeReply  |
+---------+------------+----------------+---------------+

helloworld.Greeter@127.0.0.1:8000> call SayGoodbye
name (TYPE_STRING) => kev
{
  "message": "Goodbye kev!"
}

Create a health-check service that runs on a second server and port.

Once again, I started by creating a new proto file called health.proto with my new HealthCheck service defined in the same proto directory:

syntax = "proto3";
package health;

service HealthCheck {
    rpc isHealthy(HealthCheckRequest) returns (HealthCheckReply);
}

message HealthCheckRequest {

}

message HealthCheckReply {
    bool isHealthy = 1;
}

Then, I edited the build.rs file to include this new proto file:

fn main() -> Result<(), Box<dyn std::error::Error>> {
    tonic_build::compile_protos("proto/helloworld.proto")?;
    tonic_build::compile_protos("proto/health.proto")?;
    Ok(())
}

This time I had to cargo add futures in order to import try_join, but it was relatively straightforward to implement a new HealthChecker struct, which just always returns true. I then set up a new server instance in main and used try_join! to run them both concurrently.

use healthcheck::{
    health_check_server::{HealthCheck, HealthCheckServer},
    HealthCheckReply, HealthCheckRequest,
};
use hello_world::{
    greeter_server::{Greeter, GreeterServer},
    GoodbyeReply, GoodbyeRequest, HelloReply, HelloRequest,
};
use futures::try_join;
use std::net::SocketAddr;
use tonic::{transport::Server, Request, Response, Status};

use anyhow::Result;

pub mod hello_world {
    tonic::include_proto!("helloworld");
}

pub mod healthcheck {
    tonic::include_proto!("health");
}

#[derive(Default)]
pub struct MyGreeter {}

#[tonic::async_trait]
impl Greeter for MyGreeter {
    async fn say_hello(
        &self,
        request: Request<HelloRequest>,
    ) -> std::result::Result<Response<HelloReply>, Status> {
        let reply = HelloReply {
            message: format!("Hello {}!", request.into_inner().name),
        };

        Ok(Response::new(reply))
    }

    async fn say_goodbye(
        &self,
        request: Request<GoodbyeRequest>,
    ) -> std::result::Result<Response<GoodbyeReply>, Status> {
        let reply = GoodbyeReply {
            message: format!("Goodbye {}!", request.into_inner().name),
        };
        Ok(Response::new(reply))
    }
}

#[derive(Default)]
pub struct HealthChecker {}

#[tonic::async_trait]
impl HealthCheck for HealthChecker {
    async fn is_healthy(
        &self,
        _request: Request<HealthCheckRequest>,
    ) -> std::result::Result<Response<HealthCheckReply>, Status> {
        Ok(Response::new(HealthCheckReply {
            is_healthy: true
        }))
    }
}

#[tokio::main]
async fn main() -> Result<()> {
    let addr: SocketAddr = "127.0.0.1:8000".parse()?;
    let greeter = MyGreeter::default();
    let health_addr: SocketAddr = "127.0.0.1:9000".parse()?;

    let greeter_server =Server::builder()
        .add_service(GreeterServer::new(greeter))
        .serve(addr);

    let health_server = Server::builder()
        .add_service(HealthCheckServer::new(HealthChecker::default()))
        .serve(health_addr);
    try_join!(greeter_server, health_server)?;

    Ok(())
}

Finally, I tested it with evans again, and it showed that both services were up and running:

helloworld.Greeter@127.0.0.1:8000> call SayHello
name (TYPE_STRING) => kev
{
  "message": "Hello kev!"
}

helloworld.Greeter@127.0.0.1:8000> 

health.HealthCheck@127.0.0.1:9000> call isHealthy
{
  "isHealthy": true
}

Setting up a static site with Hugo, Gitlab and Netlify

Kevin Wu — Mon, 12 Apr 2021 10:11:03 +0000

How fitting that the first post on a blog is how to essentially set one up. This is my first time using Hugo, so this post is targeting absolute beginners with experience using Unix.

Install Hugo

The first thing you need to do is install the hugo CLI. I did this via Nix, but you can find the instructions for your platform on their site https://gohugo.io/getting-started/installing/.

Nix:

nix-env -i hugo

What I really liked about hugo is that it is simple to use for just writing plain markdown and builds very quickly compared to other static site generators like Gatsby or Jekyll. This really follows in the tradition of golang, which hugo is written in.

Set up local git repository

Next, you can create a new static site with

hugo new site my-site

This should create a directory called my-site and fill it with a template to get started on your site. Then inside the directory, you can create an interactive preview of your site using hugo server. This hosts a local version of your site on port 1313 on localhost. It was kind enough to bind the address to your localhost too so only you should be able to see it. It should look something like this:

$ hugo server

Start building sites … 

                   | EN  
-------------------+-----
  Pages            | 10  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  1  
  Processed images |  0  
  Aliases          |  2  
  Sitemaps         |  1  
  Cleaned          |  0  

Built in 54 ms
Watching for changes in /Users/kvwu/workspace/kvwu-blog/{archetypes,content,themes}
Watching for config changes in /Users/kvwu/workspace/kvwu-blog/config.toml
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop

Notice how fast it builds! 54ms is quite good. This is really good for a personal blog where I don't want to waste a lot of time watching the build process run.

You can initialize git at this point in the my-site directory with git init.

The next thing that I did was set up a nice theme. You can find a lot of great themes available at https://themes.gohugo.io/. I just found one that I liked and added it as a submodule of my new site repository.

git submodule add $THEME_URL themes/$THEME_NAME
# For example for my theme
git submodule add https://github.com/vaga/hugo-theme-m10c.git themes/m10c

This copies the repository at the $THEME_URL in the themes directory, and should have added themes/$THEME_NAME to staging. You can take a look at the theme by adding theme = $THEME_NAME to your config.toml and looking at http://localhost:1313/, and if it looks good to you, commit these changes.

git commit -m 'add theme'

I also found it useful to edit some other information in config.toml.
I changed the author and params.description to better fit my needs. These changed the values on the sidebar for my theme. I suppose you could play around and find which configurations change your page.

Set up Gitlab remote repository

Gitlab is a site that allows you to host a git repository online. You could also use Github, but I used Gitlab, so my instructions will be for that.

As far as I know, there isn't a great way to set up a remote repository on Gitlab through CLI, and it's not easy to describe how to use their website with words. Their instructions on how to do this can be found here https://docs.gitlab.com/ee/user/project/working_with_projects.html#create-a-project.

Once you have set up a remote repository, you can add your local repository to it

git remote add origin git@github.com:username/projectpath.git
git push

Set up Netlify

Netlify is a service that will host your site for you. This particular section suffers from the same problem as the Gitlab one. Netlify does have a cli, but I used their web UI. However, luckily for us, Netlify provides really detailed instructions on their site for how to host your site: https://gohugo.io/hosting-and-deployment/hosting-on-netlify/.

The tutorial, at least at time of writing, shows how to set up using Github but it's pretty easy to set up with Gitlab as well by just clicking Gitlab instead of Github when prompted.

When prompted to select your repo, select the repo you just created and your master branch. This will create a hook so that whenever you push your master branch, you will deploy a new version of the website.

One issue that I encountered with the compatibility was that my theme required at least hugo 0.55, but Netlify was using 0.54 by default. In this case you can create a netlify.toml file e.g.

[context.production.environment]
HUGO_VERSION = "0.81.0"

to specify your hugo version. I chose 0.81 because that was the version that I had installed, which I think should lead to the most consistent experience. You can find out which version you're using by running hugo version.

DEV Community: Kevin Wu

Serverless Design Principles

Introduction

Motivations

Saving costs

Efficiencies at scale, minimizing resource contention with data

It\'s just easier

Key ideas

Concurrency

Cold starts

Quick detour on code/data reuse.

Class members

File System

Best practices

Keep functions short (like less than 2 minutes)

Cold start related

Keep functions small (like under 50MB)

Use an interpreted runtime for more predictable results

Be mindful of price

Assorted Tips

Lambda currently doesn\'t charge for inits under 10s

Init can happen before you actually make an invoke request

Don\'t use Task without returning something in Azure Functions (when you can)

Moving to Windows from Linux Pt 1

Package manager (pamac, brew -> scoop)

(Pseudo) Tiling Window Manager (pop-shell, amethyst -> komorebi)

System stats monitor (system-monitor, iStat Menus, powerline stats -> Xmeters)

System monitoring software (Activity Monitor, Gnome System Monitor -> ProcessExplorer)

Set Up Grpc Web Server With AWS

Intro

Setting up an EC2 instance

Set up an ALB for my EC2 instance

Creating a certificate

Load Balancer Creation Continued

Running a gRPC server on the instance

Testing a request to the Load Balancer

Conclusion

Designing an Online Book Store

Introduction

Requirements

Data store

API -- Or, a brief detour into the definition of an API

API Design

Browsing Products

Saving a customer's cart session

Checkout

Customer Interaction

Oncall superstition and the Return to Mean Bias

Setting Up a gRPC Protobuf Server With Tonic

Set up a new cargo project

Set up a simple tonic server

Creating the protobuf files

Learning to include the protobuf files into our main.rs

Finally, writing out our simple server

Testing the server

Adding a new API

Create a health-check service that runs on a second server and port.

Setting up a static site with Hugo, Gitlab and Netlify

Install Hugo

Set up local git repository

Set up Gitlab remote repository

Set up Netlify

Learning to include the protobuf files into our `main.rs`