DEV Community: Kevin McDonald

My Favorite Interview Question

Kevin McDonald — Mon, 22 Sep 2025 10:00:00 +0000

Interviews are a strange dance. As an interviewer, you’re trying to get a signal on a candidate’s skills, experience, and personality in a very short amount of time. As a candidate, you’re trying to showcase your best self while under pressure. It’s a tough situation for everyone involved.

Over the years, I’ve asked a lot of interview questions. Some have been great, some have been duds. But I always come back to my absolute favorite question:

“How does the internet work?”

I know what you’re thinking. That’s a huge, open-ended question! And you’re right, it is. That’s exactly why I love it. You also might be thinking that this is a super common question. I agree! But unlike most questions, this one is still extremely useful despite its popularity.

Why this question is so powerful

This question is a fantastic tool for a few reasons:

It’s a blank canvas. The candidate can start anywhere they want. Do they start with the physical layer? The application layer? The philosophy of interconnectedness? It gives them a chance to show me what they think is important.
It reveals their depth of knowledge. The question can be answered at many different levels of abstraction. A junior engineer might give a high-level overview, while a senior engineer might be able to dive into the nitty-gritty details of TCP handshakes and BGP routing.
It tests communication skills. Can the candidate take a complex topic and explain it in a clear, concise way? This is a crucial skill for any engineer, especially in a team environment.
It’s a conversation starter. The question often leads to a natural back-and-forth conversation. I can ask follow-up questions to probe deeper into areas where the candidate seems knowledgeable or to help them along if they’re struggling.

Sometimes, the sheer open-endedness of the question can be a bit overwhelming for a candidate. If they’re struggling to start or getting too philosophical, I’ll give them a more concrete prompt:

“What happens when you type google.com into your browser and press Enter?”

This usually gets the ball rolling. Here’s a breakdown of the kind of answer I’m looking for, with varying levels of detail depending on the candidate’s seniority.

What I expect from most candidates

For any software engineering role that involves web development, I expect the candidate to be able to walk me through the following steps:

DNS Lookup: The browser needs to translate the human-readable domain name google.com into a machine-readable IP address. This involves checking the browser’s cache, the OS’s cache, and then querying a DNS resolver, which has its own upstream sources and cache.
TCP Connection: Once the browser has the IP address, it establishes a TCP connection with the server. This involves establishing a connection using TCP’s three-way handshake (SYN, SYN-ACK, ACK).
HTTP Request/Response: The browser sends an HTTP GET request to the server. The server processes the request and sends back an HTTP response, which includes the HTML, CSS, and JavaScript files that make up the Google homepage.
Rendering: The browser parses the HTML and renders the page. It also executes the JavaScript, which might make additional requests to the server to fetch more data.

What I hope for from senior candidates

For a more senior role, especially at a company that deals with networking or infrastructure, I’m hoping for a deeper dive into some of the following topics:

ARP (Address Resolution Protocol): Before the browser can even send a packet to the router, it needs to know the router’s MAC address. This is where ARP comes in.
DHCP (Dynamic Host Configuration Protocol): - How did our computer get an IP address in the first place? A senior candidate might explain that the machine likely requested an IP address, a subnet mask, the default gateway’s IP, and the DNS server’s IP from a DHCP server on the local network.
NAT (Network Address Translation): - To conserve the limited global supply of IPv4 addresses, most home and office networks use a single public IP address for many devices. NAT is the mechanism that allows a router to translate between private, internal IP addresses and that single public one.
BGP (Border Gateway Protocol): How does my request find its way from my local network to Google’s servers, potentially halfway across the world? Mentioning BGP shows an understanding that the internet is a ’network of networks’ and that routing between these large autonomous systems is a complex, solved problem.
HTTP/2 and HTTP/3 (QUIC): Is the connection really just a single TCP connection? A senior candidate might discuss the limitations of TCP, like head-of-line blocking, and how newer protocols solve them. Pointing out that since the destination is Google, the connection is likely using HTTP/3 (which runs on QUIC over UDP) is a huge plus.
TCP Keep-Alives: How does the connection stay open for subsequent requests? What are the trade-offs of keep-alives?
Caching: Where does caching happen in this whole process? DNS caching, browser caching, CDN caching… there are many layers.
Pre-flight Requests (CORS): If the page makes requests to other domains, the browser might need to make a pre-flight OPTIONS request to check if the cross-origin request is allowed.
Packet Breakdown: What does an actual packet look like? What are the different headers (Ethernet, IP, TCP, HTTP)?

It’s not about getting it “right”

I want to be clear: I don’t expect any candidate to know everything about the internet. That’s impossible. What I’m looking for is a candidate who has a solid foundation, can reason about complex systems, and is curious to learn more. How does the candidate behave when I press into an area that they don’t know a lot about? Are they curious? Are they willing to say they don’t know? Do they make things up? All of that is very useful for evaluating skill and knowledge level.

Sometimes a candidate’s answer goes in a completely unexpected direction. I once had a candidate describe the hardware interrupts and OS context switches that happen just from typing ‘google.com’. Even though it was a tangent, it was a valuable signal that showed me how they reason about systems from the hardware up.

This question is a fantastic way to gauge all of those things. It’s a journey we go on together, and I often learn something new from the candidate’s answer.

What’s your go-to interview question? I’d love to hear it.

Visualizing the Internet (2025)

Kevin McDonald — Mon, 16 Jun 2025 00:00:00 +0000

For the past couple of years, I’ve been creating visualizations of the internet’s physical infrastructure. This project pieces together data from a few sources, and for me, seeing this data visualized together is compelling. These maps show the undersea fiber optic cables that form the backbone of global connectivity and the Internet Exchange Points (IXPs) where networks meet. This year, I’m thrilled to announce a major evolution of the project. Instead of static images and pre-rendered videos, the Internet Map is now a fully interactive, animated map that you can see online.

You can explore the new map live at map.kmcd.dev.

The new version lets you take control. You can pan, zoom, and step through time from the earliest days of the subsea cable network to the latest deployments in 2025. A new statistics panel provides a detailed snapshot for any given year, offering a richer understanding of how our connected world has grown.

What You’re Looking At

The map visualizes two critical components of the internet’s physical layer.

A Note on What You’re Seeing (and Not Seeing)

It’s important to note that this map visualizes publicly available data, which doesn’t capture the full picture of global connectivity. The city peering information, for instance, is sourced from PeeringDB, which tracks publicly advertised connections at IXPs. A vast amount of internet traffic also flows through private peering arrangements and paid transit links that are not publicly documented and therefore do not appear here.

Similarly, the map focuses on the intercontinental backbone of submarine cables. It does not show the incredibly dense web of terrestrial fiber optic cables that run under our streets and alongside major roads. While that data would be fascinating, visualizing it would be overwhelming, and acquiring a complete dataset is nearly impossible as network providers rarely share this proprietary information.

Submarine Cables

The lines snaking across the ocean floors are submarine communications cables. These bundles of fiber optic strands are the high-speed data arteries that connect continents. Laying and maintaining them is a modern marvel of engineering, involving everything from specialized cable-laying ships to underwater robots for repairs. As you explore the map, you can see how the web of these cables has become denser over time, enabling the global, real-time communication we now take for granted. By the start of 2025, the network has grown to 599 cables, spanning a staggering 1,602,092 kilometers.

A Physical Target: Vulnerabilities and Sabotage

While these cables are heavily armored, especially in shallower coastal waters where most damage occurs, their isolation on the seabed makes them vulnerable. For decades, the most common threat has been accidental damage from fishing trawlers and dragged anchors. However, in recent years, a more alarming trend has emerged: intentional sabotage. The increasing frequency of suspicious cable cuts suggests that these vital arteries of communication are becoming targets in geopolitical conflicts, a reality that may have brought many new visitors to this map.

Here are just a few of the many recent incidents:

The Balticconnector Pipeline and Cable Damage (October 2023): The Balticconnector gas pipeline and two telecom cables between Finland and Estonia were damaged by a dragged anchor from the Hong Kong-flagged ship Newnew Polar Bear. China later admitted its vessel was responsible but claimed it was an accident.
The C-Lion1 and BCS East-West Interlink Cuts (November 2024): Two key telecom cables in the Baltic Sea, C-Lion1 (Finland-Germany) and BCS East-West Interlink (Lithuania-Sweden), were severed. The Chinese-owned cargo ship Yi Peng 3 was the primary suspect after it was observed making anomalous movements in the area.
The Christmas Day Baltic Cable Cuts (December 2024): On Christmas Day 2024, the Estlink 2 power cable and other telecom cables between Finland and Estonia were damaged by a dragged anchor. Finnish authorities seized the suspected vessel, the oil tanker Eagle S, which was identified as part of Russia’s “shadow fleet”.
The Matsu Islands Blackout (February 2023): Two undersea cables connecting Taiwan to its outlying Matsu Islands were severed by Chinese vessels, leaving the 14,000 residents with severely disrupted internet for over 50 days. The incident highlighted the societal impact of such disruptions and was seen as part of a broader pressure campaign by China.
The Trans-Pacific Express Cable Cut (January 2025): The major Trans-Pacific Express international cable was cut near Taiwan, with suspicion falling on the Chinese-owned cargo ship Shunxin 39. The vessel had a history of using multiple identities to evade tracking and sailed erratically over the cable’s location before the incident.
The Red Sea Cable Disruption (February 2024): Three critical cables in the Red Sea were severed by the anchor of the sinking cargo ship Rubymar, which had been struck by a Houthi missile. The incident disrupted a significant portion of Europe-Asia data traffic and highlighted the vulnerability of infrastructure in contested maritime chokepoints.
The Gulf of St. Lawrence Sabotage (December 2023 & 2024): A subsea cable connecting Nova Scotia and Newfoundland was deliberately cut in December 2023 and again in December 2024. Evidence showed an “angle grinder cut” through the steel-wrapped cable, confirming sabotage, though the perpetrator and motive remain unknown.

These examples represent only a fraction of such incidents, which have escalated in frequency and impact in recent years.

Internet Exchange Points (IXPs)

The circles on the map represent cities with Internet Exchange Points. If submarine cables are the interstate highways of the internet, then IXPs are the bustling, hyper-connected metropolitan areas where all the traffic is headed.

An IXP is a physical data center, or a set of connected data centers, where many different networks can physically plug into each other to exchange traffic directly. This process is called “peering.”

So why do hundreds of networks choose to gather in the same buildings in cities like Frankfurt or Amsterdam? The answer is a powerful network effect that you could call digital gravity. The value of an IXP is determined by the networks present there. Once a major network joins, it becomes exponentially more attractive for others to join as well.

The most powerful sources of this gravity are large content providers like Google, Meta, Apple, and Netflix. These companies have an “open peering policy.” In essence, they are saying to any Internet Service Provider (ISP): “Connect with us directly here at the IXP, and we will give your customers a faster, better path to our services.”

This creates a powerful win-win scenario:

The ISP wins because their customers get lightning-fast, low-latency access to YouTube, Google Drive, or Apple’s App Store. This makes the ISP’s own service more valuable and competitive.
Google and Apple win because they get to deliver their content without paying high fees to third-party backbone carriers. Every byte of data they serve over a direct peering connection is money saved.

This economic incentive is the engine that drives the growth you see on the map. ISPs flock to the IXPs where the big content providers are, which in turn attracts more content providers, creating a feedback loop of ever-increasing capacity and value. This is why a handful of cities have become global hubs with staggering traffic volumes, while others remain smaller, regional nodes.

The World in 2025: A Snapshot

The animation and data now extend to 2025, revealing significant ongoing investment. In this year alone, 31 new cables were added (or are promised very soon), stretching over 144,320 kilometers , enough to circle the earth three and a half times.

Some of the longest and most impactful new cables of 2025 include:

Bifrost (19,888 km): A monumental project directly connecting Singapore to North America via Indonesia, the Philippines, and Guam. It’s a joint effort by Meta, Keppel, and Telin to bolster connectivity across the Asia-Pacific region.
Echo (17,184 km): Another critical trans-Pacific cable, built by Google and Meta, that forges a new, resilient path from the U.S. to Singapore, also landing in Guam and Indonesia. This cable deliberately avoids the crowded northern routes to increase network diversity.
Firmina (14,517 km): A Google-led cable enhancing the North-South America connection. It runs from the U.S. East Coast to Argentina, with landings in Brazil and Uruguay, dramatically improving access to Google services in South America.
TPU (13,470 km): A Google-owned cable system connecting the U.S. with Taiwan and the Philippines, bolstering trans-Pacific capacity.
JUNO (11,710 km): A cable system by Seren Juno Network connecting Japan to the U.S., utilizing advanced technology to offer a high number of fiber pairs and enhance communication resiliency.

Regional Peering Powerhouses

Looking at the total peering capacity reveals a clear global hierarchy. Europe remains the undisputed leader, with an incredible 1.5 Pbit/s of capacity. Asia and North America follow with robust networks of their own, while South America shows impressive growth.

This massive regional capacity is concentrated in a few key metropolitan hubs. The list of top peering cities shows just how vital they are to the global network:

Amsterdam, NL : 200 Tbit (+12.7 Tbit)
Frankfurt, DE : 166 Tbit (+8.19 Tbit)
São Paulo, BR : 157 Tbit (+3.16 Tbit)
London, GB : 113 Tbit (+3.21 Tbit)
Tokyo, JP : 90.2 Tbit (+2.18 Tbit)

The growth in a city like São Paulo is remarkable and shows the increasing investment in internet infrastructure in South America, directly supported by new cables like Firmina.

How It’s Made: The New Tech Stack

The transition to an interactive map required a complete overhaul of the technology stack. The previous versions, which relied on generating static SVG images, faced several challenges. It was difficult to dynamically size the lines representing cables and find the right balance of detail for country borders; too much detail slowed the map down, while too little looked simplistic when zoomed in.

The solution was to adopt a tile-based methodology, where map tiles at different levels of detail are fetched dynamically as a user zooms—the same concept used by Google Maps. I was faced with a choice: implement this highly complex tiling logic myself or use a well-supported library. Since the project’s focus was on data visualization, I opted for the more direct path by using Leaflet, a powerful library for creating dynamic and interactive maps.

The back-end Go scripts that gather and process the data from sources like TeleGeography and PeeringDB were largely unchanged, only needing a few new fields in the JSON output to power the new front end.

Beyond the core technology, I also wanted to incorporate a few small details to improve the user experience and make the map feel more intuitive and personal:

Personalized View : The map automatically geolocates your region and centers the initial view there. My hope is that the map feels familiar and relevant the moment you open it, no matter where you are in the world.
Persistent ‘About’ Section : The ‘About this map’ panel remembers its state. If you close it, it stays closed on subsequent visits and refreshes until you decide to open it again.
Relative City Sizing : The size of each city circle is relative to its total peering bandwidth, giving an immediate visual sense of where the major hubs of connectivity are concentrated.
Interactive Highlighting : Hovering over or clicking on any cable or city highlights it and brings it to the forefront. This small detail makes it much easier to focus on and explore individual parts of the network.

Closing Thoughts

This project continues to be a fascinating exploration of the physical reality of our digital world. By making the map interactive, I hope to provide a more powerful tool for anyone curious about the immense and intricate infrastructure that underpins our daily lives. As global data demand soars, the growth of these subsea cables and peering exchanges will only become more critical. Explore the map, watch the internet grow, and see for yourself how the world gets connected.

This project was a significant undertaking, and I’ve been thrilled to see it shared in various places online. If you choose to share it, I only ask that you please provide attribution by linking back to the project, just as I give credit to my own data sources, TeleGeography and PeeringDB.

Thank you for reading, and please feel free to explore and share the map with others! map.kmcd.dev

HTTP QUERY and Go

Kevin McDonald — Wed, 04 Jun 2025 10:00:00 +0000

You’re likely familiar with the HTTP methods, GET and POST , the workhorses of HTTP. These have both worked surprisingly well and have provided well-defined caching behavior for over a quarter of a century. However, neither of these solves the problem of complex request parameters without completely throwing out the caching semantics of GET. This is where a concept like a QUERY method comes in, and it’s not just a thought experiment; it’s an area actively being explored by the IETF HTTP Working Group.

But, why?

The standard HTTP methods serve us well, except the ones that don’t… but that’s a topic for another day. GET is great for simple data retrieval, but its reliance on URL parameters makes it cumbersome for complex queries or large sets of input parameters. URLs have practical length limits, and embedding deeply nested structures is awkward.

This often leads developers to use POST for what are semantically read-only query operations. Many widely-used protocols effectively do this:

GraphQL typically uses POST requests with JSON bodies to send queries.
gRPC and gRPC-Web typically rely exclusively on POST.
Older protocols like SOAP and XML-RPC almost exclusively use POST to encapsulate their operations, including data retrieval. This has been an issue for a long time!

While using POST works, it’s a compromise. POST traditionally implies an action that might change state on the server, which means intermediaries (read: caches) and even client-side logic might treat these “query-via-POST” requests with undue caution, forgoing caching or automatic retries that would be safe for a truly read-only operation.

This is precisely the problem that a dedicated safe method with a request body aims to solve. The IETF HTTP Working Group is discussing such a method in a draft titled “A Safe HTTP Method with a Request Content”. As the draft states:

The QUERY method provides a solution that spans the gap between the use of GET and POST. As with POST, the input to the query operation is passed along within the content of the request rather than as part of the request URI. Unlike POST, however, the method is explicitly safe and idempotent, allowing functions like caching and automatic retries to operate.

While the draft uses “QUERY” as a candidate name (among others), the core idea is what we’re exploring: a method for safe, idempotent data retrieval that can carry a payload. For the rest of this article, we’ll continue to use “QUERY” to represent this concept and show how you can implement such a custom method in Go today.

It’s crucial to remember that until such a method is formally standardized and widely adopted, using custom HTTP methods can impact interoperability. However, for internal APIs, tightly controlled systems, or as a forward-looking experiment, they can be very useful.

Server-Side with Go

Go 1.22 introduced enhancements to http.ServeMux that allow you to register handlers for specific HTTP methods and paths more directly. Let’s build a server that handles our custom QUERY method:

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
)

func queryHandler(w http.ResponseWriter, r *http.Request) {
    // The new ServeMux handles method checking based on registration.

    body, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, "Error reading request body", http.StatusInternalServerError)
        return
    }
    defer r.Body.Close()

    // In a real application, you'd parse the query from the body
    // (e.g., JSON) and fetch data accordingly.
    fmt.Fprintf(w, "Received your QUERY request with body: %s\n", string(body))
    log.Printf("Handled QUERY request with body: %s", string(body))
}

func main() {
    mux := http.NewServeMux()
    // Register handler specifically for QUERY method on /data path
    mux.HandleFunc("QUERY /data", queryHandler)

    log.Println("Server starting on port 8080, handling custom QUERY method...")
    if err := http.ListenAndServe(":8080", mux); err != nil {
        log.Fatal(err)
    }
}

In this server:

We use mux.HandleFunc("QUERY /data", queryHandler) to directly associate the queryHandler with our custom QUERY HTTP method for the /data path.
The queryHandler reads the request body, where the complex query parameters would reside.

Congratulations, we’ve made an API with an endpoint no browser can use. So useful. Since we can’t test with a browser, let’s write a client to pair with this server.

Client-Side, also with Go

Here’s how a Go client can send a QUERY request:

package main

import (
    "fmt"
    "io"
    "log"
    "net/http"
    "strings"
)

func main() {
    queryPayload := `{"filters": {"status": "active", "category": "electronics"}, "fields": ["name", "price"]}`
    client := &http.Client{}

    // Create a new request with the custom "QUERY" method
    req, err := http.NewRequest("QUERY", "http://localhost:8080/data", strings.NewReader(queryPayload))
    if err != nil {
        log.Fatalf("Error creating request: %v", err)
    }
    req.Header.Set("Content-Type", "application/json") // Important for body processing

    resp, err := client.Do(req)
    if err != nil {
        log.Fatalf("Error sending request: %v", err)
    }
    defer resp.Body.Close()

    fmt.Println("Response Status:", resp.Status)
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        log.Fatalf("Error reading response body: %v", err)
    }
    fmt.Println("Response Body:", string(body))
}

The client uses http.NewRequest("QUERY", ...) to specify the custom method and sends the queryPayload in the request body.

Client-Side with cURL

Although web browsers won’t randomly make QUERY calls, many other tools can use QUERY. Many also support arbitrary HTTP verbs, although this isn’t typically leveraged due to filters from load balancers, firewalls, etc.

Anyway, here’s what that request looks like with cURL:

$ curl -X QUERY -d '{"filters": {"status": "active", "category": "electronics"}, "
fields": ["name", "price"]}' http://localhost:8080/data -v
* Host localhost:8080 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:8080...
* Connected to localhost (::1) port 8080
> QUERY /data HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Length: 89
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 89 bytes
< HTTP/1.1 200 OK
< Date: Sat, 31 May 2025 15:47:45 GMT
< Content-Length: 129
< Content-Type: text/plain; charset=utf-8
< 
Received your QUERY request with body: {"filters": {"status": "active", "category": "electronics"}, "fields": ["name", "price"]}
* Connection #0 to host localhost left intact

QUERY vs. GET and POST - A Clearer Separation

Let’s summarize the distinctions with our (potentially future-standard) QUERY method in mind:

GET : Used for retrieving data. Parameters are typically sent in the URL. GET requests must be safe and idempotent. They generally don’t have a body.
POST : Used for submitting data to be processed, often resulting in a change of state or side effects on the server (e.g., creating a new resource). Parameters are sent in the request body. Not necessarily safe or idempotent.
QUERY (custom/proposed): Used to request data (like GET) but with the ability to send a complex query in the request body (like POST). Crucially, it is defined as safe and idempotent (like GET). This explicitly tells intermediaries and clients that the request has no side effects and can be cached or retried automatically.

Caching Behavior: GET, POST, and QUERY

HTTP caching is vital for performance. A method’s characteristics (especially safety and idempotency) directly influence its cacheability.

1. GET

Behavior : GET requests are inherently cacheable. Caches readily store and serve responses to GET requests if caching headers (like Cache-Control, Expires, ETag) allow. This is a foundational aspect of HTTP performance.

2. POST

Behavior : POST requests are generally not cacheable by default. Since POST can have side effects, caching responses could lead to unintended consequences or stale data if the action isn’t repeated.

3. QUERY

The IETF draft emphasizes that a method like QUERY is “explicitly safe and idempotent, allowing functions like caching and automatic retries to operate.” This is key.

Behavior (Current Custom Method): Because our QUERY method is non-standard today, caches will not cache it by default. They typically only automatically consider standard safe methods like GET. To make responses to a custom QUERY request cacheable:
- The server must send explicit caching headers (e.g., Cache-Control: public, max-age=3600). These headers signal the cacheability of the response.
- Intermediary caches (CDNs, reverse proxies) might need specific configuration to recognize and cache responses for this custom method.
- The Vary HTTP header is important if the response depends on the request body content.
Behavior (Future Standardized Method): If a method like QUERY becomes a recognized HTTP standard, caches would likely treat it similarly to GET for caching purposes, provided it’s implemented according to its safe and idempotent semantics. This would be a major advantage, allowing complex queries with bodies to be cached as effectively as GET requests.

Key Takeaway for QUERY Caching

The intent of a QUERY method is to be cacheable. If using a custom method today, you must provide explicit caching directives. The ongoing standardization effort aims to make this caching behavior more automatic and universally understood, unlocking performance benefits for complex, body-inclusive queries that are currently often forced into less cache-friendly POST requests. How long will this process take? Who knows! These kinds of changes can take decades.

What’s Next?

The discussion around a safe HTTP method with a request body is an exciting development. By understanding its purpose and experimenting with custom methods like QUERY in Go, we can better appreciate the nuances of HTTP and prepare for potential future standards that will make our APIs more robust and performant.

Some projects have already taken to adding support for the QUERY method. Take a look at NodeJS: Support for ‘QUERY’ method, which added support last year. As demonstrated earlier in this article, using QUERY is already possible in Go. This general support is often true for other languages and libraries as well, since custom HTTP methods are a feature of the HTTP specification.

The other aspect of moving this forward is a bit more nebulous: bureaucracy. There is still work and review to be done to graduate the draft into an official RFC from the IETF. But fear not. There’s actually steady changes being made to the draft document. You can tell this from the git history on the httpwg’s repo. At the time of writing May 19th was when the last change was made, so I’m certain that this hasn’t been forgotten about.

FauxRPC

Kevin McDonald — Tue, 20 Aug 2024 10:00:00 +0000

I would like to introduce FauxRPC, a powerful tool that empowers you to accelerate development and testing by effortlessly generating fake implementations of gRPC, gRPC-Web, Connect, and REST services. If you have a protobuf-based workflow, this tool could help.

Why FauxRPC?

Faster Development & Testing: Work independently without relying on fully functional backend services.
Isolation & Control: Test frontend components in isolation with controlled fake data.
Multi-Protocol Support: Supports multiple protocols (gRPC, gRPC-Web, Connect, and REST).
Prototyping & Demos: Create prototypes and demos quickly without building the full backend. Fake it till you make it.
Improved Collaboration: Bridge the gap between frontend and backend teams.
Plays well with others: Test data from FauxRPC will try to automatically follow any protovalidate constraints that are defined.

How it Works

FauxRPC leverages your Protobuf definitions to generate fake services that mimic the behavior of real ones. You can easily configure the fake data returned, allowing you to simulate various scenarios and edge cases. It takes in *.proto files or protobuf descriptors (in binpb, json, txtpb, yaml formats), then it automatically starts up a server that can speak gRPC/gRPC-Web/Connect and REST (as long as there are google.api.http annotations defined). Descriptors contain all of the information found in a set of .proto files. You can generate them with protoc or the buf build command.

Get Started

FauxRPC is available as an open-source project. Check out the documentation and examples to get started. Here’s a quick overview, but be sure to check the official documentation for the most up-to-date instructions:

Install via source

go install github.com/sudorandom/fauxrpc/cmd/fauxrpc@latest

Pre-built binaries

Binaries are built for several platforms for each release. See the latest ones on the releases page.

Use Descriptors

Make an example.proto file (or use a file that already exists):

syntax = "proto3";

package greet.v1;

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

Create a descriptors file and use it to start the FauxRPC server:

$ buf build ./example.proto -o ./example.binpb
$ fauxrpc run --schema=./example.binpb
2024/08/17 08:01:19 INFO Listening on http://127.0.0.1:6660
2024/08/17 08:01:19 INFO See available methods: buf curl --http2-prior-knowledge http://127.0.0.1:6660 --list-methods

Done! It’s that easy. Now you can call the service with any tooling that supports gRPC, gRPC-Web, or connect. So buf curl, grpcurl, Postman, Insomnia all work fine!

$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/greet.v1.GreetService/Greet
{
  "greeting": "dream"
}

Server Reflection

If there’s an existing gRPC service running that you want to emulate, you can use server reflection to start the FauxRPC service:

$ fauxrpc run --schema=https://demo.connectrpc.com

From BSR (Buf Schema Registry)

Buf has a schema registry where many schemas are hosted. Here’s how to use FauxRPC using images from the registry.

$ buf build buf.build/bufbuild/registry -o bufbuild.registry.json
$ fauxrpc run --schema=./bufbuild.registry.json

Multiple Sources

You can define this --schema option as many times as you want. That means you can add services from multiple descriptors and even mix and match from descriptors and from server reflection:

$ fauxrpc run --schema=https://demo.connectrpc.com --schema=./example.binpb

Multi-protocol Support

The multi-protocol support is based on ConnectRPC. So with FauxRPC, you get gRPC, gRPC-Web and Connect out of the box. However, FauxRPC does one thing more. It allows you to use google.api.http annotations to present a JSON/HTTP API, so you can gRPC and REST together! This is normally done with an additional service that runs in-between the outside world and your actual gRPC service but with FauxRPC you get the so-called transcoding from HTTP/JSON to gRPC all in the same package. Here’s a concrete example:

syntax = "proto3";

package http.service;

import "google/api/annotations.proto";

service HTTPService {
  rpc GetMessage(GetMessageRequest) returns (Message) {
    option (google.api.http) = {get: "/v1/{name=messages/*}"};
  }
}
message GetMessageRequest {
  string name = 1; // Mapped to URL path.
}
message Message {
  string text = 1; // The resource content.
}

Again, we start the service by building the descriptors and using

$ buf build ./httpservice.proto -o ./httpservice.binpb
$ fauxrpc run --schema=httpservice.binpb

Now that we have the server running we can test this with the “normal” curl:

$ curl http://127.0.0.1:6660/v1/messages/123456
{"text":"Retro."}⏎

Sweet. You can now easily support REST alongside gRPC. If you are wondering how to do this with “real” services, look into vanguard-go. This library is doing the real heavy lifting.

What does the fake data look like?

You might be wondering what actual responses look like. FauxRPC’s fake data generation is continually improving so these details might change as time goes on. It uses a library called fakeit to generate fake data. Because protobufs have pretty well-defined types, we can easily generate data that technically matches the types. This works well for most use cases, but FauxRPC tries to be a little bit better. If you annotate your protobuf files with protovalidate constraints, FauxRPC will try its best to generate data that matches these constraints. Let’s look at some examples!

syntax = "proto3";

package greet.v1;

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string greeting = 1;
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

With FauxRPC, you will get any kind of word, so it might look like this:

{
  "greeting": "sufficient"
}

This is fine, but for the RPC, we know a bit more about the type being returned. We know that it sends a greeting back that looks like “Hello, [name]”. So here’s what the same protobuf file might look like with protovalidate constraints:

Now let’s see what this looks like with protovalidate constraints:

syntax = "proto3";

import "buf/validate/validate.proto";

package greet.v1;

message GreetRequest {
  string name = 1 [(buf.validate.field).string = {min_len: 3, max_len: 100}];
}

message GreetResponse {
  string greeting = 1 [(buf.validate.field).string.pattern = "^Hello, [a-zA-Z]+$"];
}

service GreetService {
  rpc Greet(GreetRequest) returns (GreetResponse) {}
}

With this new protobuf file, this is what FauxRPC might output now:

{
  "greeting": "Hello, TWXxF"
}

This shows how protovalidate constraints enable FauxRPC to generate more realistic and contextually relevant fake data, aligning it closer to the expected behavior of your actual services. As another example, I will show one of Buf’s services used to manage users, buf.registry.owner.v1.UserService. Here’s what the UserRef message looks like:

message UserRef {
  option (buf.registry.priv.extension.v1beta1.message).request_only = true;
  oneof value {
    option (buf.validate.oneof).required = true;
    // The id of the User.
    string id = 1 [(buf.validate.field).string.tuuid = true];
    // The name of the User.
    string name = 2 [(buf.validate.field).string = {
      min_len: 2
      max_len: 32
      pattern: "^[a-z][a-z0-9-]*[a-z0-9]$"
    }];
  }
}

So let’s make our descriptors for this service, start the FauxRPC server and make our example request:

$ buf build buf.build/bufbuild/registry -o bufbuild.registry.binpb
$ fauxrpc run --schema=./bufbuild.registry.binpb
$ buf curl --http2-prior-knowledge http://127.0.0.1:6660/buf.registry.owner.v1.UserService/ListUsers
{
  "nextPageToken": "Food truck.",
  "users": [
    {
      "id": "c4468393f926400d8880a264df9c284a",
      "createTime": "2012-03-06T12:15:03.239463070Z",
      "updateTime": "1990-10-29T13:12:31.224347086Z",
      "name": "jexox",
      "type": "USER_TYPE_STANDARD",
      "description": "Tattooed taxidermy.",
      "url": "http://www.productexploit.name/synergies/target"
    },
    {
      "id": "0e4ca24f4ff54761b109daab0da1bea2",
      "createTime": "1955-05-16T02:37:30.643378679Z",
      "updateTime": "1923-08-28T04:28:43.330711919Z",
      "name": "ya0",
      "type": "USER_TYPE_STANDARD",
      "state": "USER_STATE_INACTIVE",
      "description": "Helvetica.",
      "url": "https://www.centralengage.info/markets/scale/e-commerce/exploit",
      "verificationStatus": "USER_VERIFICATION_STATUS_UNVERIFIED"
    }
  ]
}

Hopefully, this gives you a good idea of what the output might look like. The better your validation rules, the better the FauxRPC data will be.

What’s left?

FauxRPC is already great for some use cases but it’s not “done” as there’s more to do to make it better. I have plans to add the ability to configure stubs for each RPC method. This will allow you to define specific responses or behaviors for each RPC, giving you more control over the simulated service. I hope this will make it easier to iterate on protobuf designs without needing to actually implement services until later.

Stay Tuned

I made a documentation website to organize documentation. I think it looks pretty good for how quickly I threw it together. The code for FauxRPC lives on GitHub at sudorandom/fauxrpc. It’s a little thin now but there’s a lot that I can write about in there. I’m actively developing FauxRPC and have many exciting features planned for the future. This is early on for this project but it has come together as a coherent and useful program for me extremely quickly. So please try it out and let me know your feedback and suggestions. Stay tuned for updates!

... and don't forget to star the repo on GitHub. It helps more than you know!

Tracking the Wins

Kevin McDonald — Tue, 30 Apr 2024 00:00:00 +0000

Mental health is hugely important for software engineers. When I first started coding, the thrill of solving problems and getting that hit of dopamine from the instant feedback loop was incredible. It's easy to forget those initial challenges and accomplishments as you progress, though, and problems that used to look like unscalable walls turn into speed bumps. That's why I believe regularly reflecting on your work is so valuable. Otherwise, you might find yourself dwelling on the struggles and forgetting the successes you've achieved along the way.

Writing down the wins

Here's how I track my progress: Every week or two, I set aside some time to quickly document my achievements, even small ones. I keep it concise to avoid procrastination. I have some anxiety around tasks with too many unknowns, so a simple, mechanical process works best for me. I'm also a very forgetful person, so I usually have to review artifacts created by the work that I do, like following digital breadcrumbs that I leave around. Here's a list of where I might find these things:

issues I've worked on
pull requests
emails
edits to documentation
slack messages

The more concrete evidence, the better, but some achievements don't leave clear evidence so be sure to capture interesting anecdotes as they happen. Anything that has allowed me to progress forward should be highlighted, even if it is a result of compromise or realizing that you made a mistake.

I also apply a similar strategy to my personal life. Did I stay on top of my Danish lessons and language learning apps? Did I have periods of being ill during this period? Was I physically active, by cycling, exercising, or simply taking the stairs? Tracking these things helps me see the broader picture.

Quarterly reviews

To pair with the weekly tracking. I conduct a more in-depth reflection every 3 months. Here, I add my thoughts to these accomplishments. I create a narrative of the past quarter, encompassing both highs and lows, as well as the inevitable "neutral" experiences. Three months seems like a good timeframe to identify any significant shifts in my attitude or goals. It allows me to assess my progress on past goals and set new ones for the future. I believe this time frame is MUCH better than the yearly cycle of New Year resolutions. Twelve months is way too long. One month is usually not enough to make good progress on goals or to establish habits. Three months seems just right to me. So here's the list of what might appear on a quarterly review:

A "narrative" of the last three months using concrete examples from my journal to back it up.
Report on the progress of previous goals.
Create new goals or revise the older goals if needed.
Pictures and video of important moments.
Reflect on big family budget items and see if they align with my priorities in life.

Yearly themes

Yes, I am a Cortexan and podcast episodes outlining yearly themes are among my favorites. Yearly themes give a framework for me to push myself to do things I want to do while allowing me to adjust (every quarter) exactly what it means to accomplish my theme. Sometimes life throws additional challenges. For example, it's silly to fail at achieving a goal of "running a mile every day" if you break your leg. This year is "The Year of Building". Here's what I wrote about my theme shortly before the year began:

I want to build some cool things. I want to make some useful open source projects. I want to implement some neat solutions at work and actually prove them out. My follow-through sometimes lacks when I lose interest in things. I want to make more cool blog posts.

I also want to build on my Danish by continuing my Danish classes, self study with and using it more "in the wild".

I think I'm doing a good job with this one. You may notice though that the latest batch of blog posts didn't start in January. I "slacked" on this goal for over a month but in my theme framework, it's fine because I was ramping up my Danish classes again. My yearly theme allows me a lot of grace on the day-week-month scale while hopefully keeping me honest on the yearly scale and allowing for adjustment of individual goals to match reality.

Try it yourself

Journaling and goal-setting are extremely personal things so your process may vary a lot from mine. But one thing is likely true of everyone: by keeping track of your wins, both big and small, you can maintain motivation and see your progress over time. Give it a try and see how it impacts your software engineering (and life) journey!

Making Greppable Code

Kevin McDonald — Tue, 16 Apr 2024 00:00:00 +0000

Have you ever felt like you're sifting through lines of code, desperately searching for that elusive function or variable? Fear not, for there's a way to make your code more discoverable: greppable naming conventions.

What is Greppability?

Drawing inspiration from the powerful Linux command grep used for text searching, greppable code refers to code that's easy to search through using clear and meaningful names. Just like grep helps you quickly find specific text within a file, greppable names allow you to efficiently locate relevant code sections by searching for terms that reflect their purpose.

Crafting Greppable Names

Here are some key principles to follow:

Specificity: Ditch generic names like processData() or utils. Opt for terms that reflect the specific content (cleanAndValidateOrderData(), orderId).
Priority: It is far more important for filenames, class names and methods to have specific names than it is for variables inside of functions. Variables inside of functions already have a lot of implied context, like the filename, class (if your language uses classes) and the function that the variable appears in. All of that extra context should be greppable but the variables probably don't need to be.
Consistency: Maintain a consistent naming style throughout your codebase (e.g. camelCase or snake_case) and always use the 'standard' for your language or framework.
Verbs when naming methods: Methods describe actions. Use verbs at the beginning of method names to convey their purpose (calculateOrderTotal(), updateCustomerRecord()).
Abbreviations with Caution: Use abbreviations sparingly and only for widely recognized terms (e.g. HTTP, XML). Overuse can hinder readability. If your variable names look like 2010s-era tech startups, you're probably doing it wrong.
Dynamic Dispatch: Doing some more complex programming patterns like dynamic dispatching can greatly hurt the ability for the code to be greppable. It can also add indirection so be sure you use these methods sparingly and add more documentation where needed to help lost souls who are tracing code through the codebase.

The Search Advantage of gRPC

Here's where things get interesting for APIs. Compared to RESTful APIs with their predefined resource structures, gRPC (and other RPC-based APIs) offer greater flexibility in naming methods. This freedom allows you to leverage greppable naming conventions to their full potential.

For instance, in a RESTful API, an endpoint for updating a user profile might be named /users/:id. While functional, this doesn't explicitly convey the action performed. In a gRPC API, however, you could have a method named UpdateUserProfile(UpdateRequest request), which clearly describes the purpose and is much easier to search for in the code.

The Greppability Benefit

Meaningful names not only make code easier to understand but also significantly reduce maintenance time. When you (or another developer) revisit your codebase later, clear names can make traversing the codebase much nicer. I think the term "greppability" has a clear meaning and can come up in many contexts: in code, data, documentation, etc. Do you think this a good word for a developer to have in their vocabulary?

Building APIs with Contracts

Kevin McDonald — Tue, 09 Apr 2024 00:00:00 +0000

In today's interconnected world, APIs (Application Programming Interfaces) are the glue that connects computers. They allow different applications to talk to each other, share data, and perform actions. However, traditional methods of creating APIs can lead to challenges, especially when dealing with versioning changes and integrating complex systems. This is where contract-based APIs come in, offering a more robust and reliable approach and taming some of the wildness that exists on the web.

The Power of Pre-defined API Contracts

Imagine building a house without a blueprint. It would be chaotic and prone to errors. A contract-based API is like a detailed blueprint for communication between applications. It defines exactly what data can be exchanged, in what format, and what actions can be performed. This pre-defined agreement unlocks several advantages for developers and applications:

Improved Developer Experience:
- Developers on both sides (client and server) have a clear understanding of what's expected, making integration smoother.
- Automated Documentation: Contracts serve as self-documenting artifacts, reducing the need for manual documentation creation and maintenance. This saves development time and ensures documentation stays in sync with the actual API implementation.
Reduced Errors:
- Mismatched data formats or API changes become less likely, leading to fewer bugs and headaches. Contracts act as a validation layer, catching potential issues early in the development process.
Easier Integration:
- Contracts act as a single source of truth, simplifying the process of connecting different services. Developers can quickly understand how to interact with the API without extensive back-and-forth communication.
Streamlined Development:
- Contract-based APIs often enable tools to automatically generate code for both client and server implementations based on the defined contracts. This eliminates manual coding and boilerplate, allowing developers to focus on core functionalities.

By leveraging pre-defined contracts, you can build more robust, reliable, and maintainable APIs that streamline development efforts and improve overall communication within your application ecosystem.

Protobuf: The Language of APIs

In my world, the foundation of many contract-based APIs lies in Protocol Buffers (protobuf). It's a language-neutral data format specifically designed for structured messages exchanged between applications. Protobuf offers several advantages:

Smaller Message Sizes: Protobuf messages are compact and efficient, leading to faster transmission and reduced bandwidth usage.
Faster Parsing: Parsing protobuf messages is significantly faster compared to traditional formats like JSON or XML.
Cross-language Compatibility: Protobuf definitions are language-agnostic. Code for interacting with the API can be generated for many programming languages.

Defining service contracts in protobuf involves creating .proto files. These files specify the structure of messages (data fields and types) and define the service methods (requests and responses) that an API offers.

Here's a basic example of a .proto file defining messages for a user and an address:

syntax = "proto3";

message User {
  string name = 1;
  int32 id = 2;
  string email = 3;
  Address address = 4;
}

message Address {
  string street = 1;
  string city = 2;
  string state = 3;
  string zip = 4;
}

In this example, the User message has fields for name, ID, email, and an Address message. The Address message itself has fields for street, city, state, and zip code. These defined message structures ensure consistent data exchange between applications.

gRPC: Building APIs on a Solid Foundation

gRPC (gRPC Remote Procedure Call) is a high-performance framework that builds upon protobuf's strengths. It provides a powerful way to implement remote procedure calls, allowing applications to interact using clients generated for each language using (usually) types and semantics that make sense to the language. Here's how gRPC leverages protobuf:

Protobuf Messages for Communication: gRPC uses protobuf messages to define the request and response data exchanged between client and server.
Generated Code for Seamless Interaction: gRPC automatically generates server and client stubs (boilerplate code) from the .proto files. This eliminates manual coding and ensures consistency between client and server implementations.

Introducing Services and Request/Response Types with gRPC

Now let's expand on the concept of services within a .proto file. We can define a service called UserService with methods for user management:

syntax = "proto3";

service UserService {
  rpc CreateUser(CreateUserRequest) returns (User) {}
  rpc GetUser(GetUserRequest) returns (User) {}
}

message CreateUserRequest {
  User user = 1;
}

message GetUserRequest {
  int32 id = 1;
}

This example defines a UserService with two methods: CreateUser and GetUser. Each method takes a specific request message and returns a response message. The CreateUserRequest message contains a User object to be created, while the GetUserRequest specifies the user ID to retrieve. This clear separation of request and response types further enhances the contract between client and server. You should also notice just how clear the intention is for the methods of UserServer. A reader of this spec doesn't have to do a mapping of extremely vague words like "POST" to more understandable actions like "create". Also notice how the CreateUser and GetUser methods are greppable, making it trivial to locate uses of these RPCs, even across several repositories.

Alternatives

While Protobuf and gRPC are a powerful duo, there are other contract-based API solutions to consider:

OpenAPI (Swagger): This is a popular specification for defining RESTful APIs, offering a standardized way to document and interact with web services.
Thrift: Similar to protobuf, Thrift is a language-neutral protocol for defining service contracts. It supports various RPC protocols beyond gRPC.
Avro: This JSON-like data format uses schemas to ensure reliable data exchange. It's often used with Apache Kafka for streaming data pipelines.

The choice between these options depends on your specific needs. gRPC and protobuf excel for high-performance, RPC-based communication, while OpenAPI is better suited for existing RESTful APIs. Thrift is also there. Avro has some dynamic typing features and has self-describing messages but is also slower for the same reason.

Conclusion

Contract-based APIs offer a significant advantage in building robust and scalable communication between applications. protobuf and gRPC provide a powerful combination for defining clear contracts and generating efficient code. By leveraging these technologies, you can streamline API development, improve developer experience, and ensure seamless integration within.

Dropping Unknown Fields in ConnectRPC

Kevin McDonald — Tue, 02 Apr 2024 00:00:00 +0000

gRPC, with its focus on performance and language neutrality, remains a popular choice for building microservices and APIs. But when exposing your gRPC service to the internet, there are a few security considerations to account for. Protobuf, the serialization format often used with gRPC, offers various encoding options that can significantly impact your service's security posture.

One crucial optimization for internet-facing gRPC services is customizing the behavior towards unknown fields. I've talked about unknown fields in a previous post, so read that one if unknown fields are still a mystery to you and then come back here. By default, protobuf messages can contain fields that are not defined in the current version of the proto schema. While convenient for development and can help with forward compatibility, this poses a security risk in a public environment.

Here's why you should consider dropping unknown fields when exposing gRPC to the internet:

Preventing Malicious Data: Unknown fields can be exploited by malicious actors to inject unexpected data into your service. This could lead to potential security vulnerabilities like code injection or unexpected behavior.
Ensuring Compatibility: Uncontrolled unknown fields can cause compatibility issues if your clients are using different versions of the proto schema. Dropping them enforces stricter adherence to the defined message format.
Improving Performance: Skipping unknown fields during message parsing can lead to performance gains, especially when dealing with large datasets.

How to Drop Unknown Fields

Here is how you can drop unknown fields while using the standard proto.UnmarshalOptions struct provided by the google.golang.org/protobuf/proto package. Here's how to do it in your Go code:

import (
    "google.golang.org/protobuf/proto"
    ...
)

// Configure unmarshalling options to discard unknown fields
opts := proto.UnmarshalOptions{
    DiscardUnknown: true,
}

// Use the options when unmarshalling incoming messages
msg := &MyMessage{}
err := proto.Unmarshal(data, msg, opts)
if err != nil {
    // Handle error
}

By setting the DiscardUnknown field to true in the proto.UnmarshalOptions struct before unmarshalling incoming messages, you ensure that any unknown fields are ignored. This helps mitigate the security risks associated with unknown fields while processing internet-facing gRPC requests.

How to Drop Unknown Fields in Connect RPC Servers

package main

import (
    "log"
    "net/http"

    "golang.org/x/net/http2"
    "golang.org/x/net/http2/h2c"
    "go.akshayshah.org/connectproto"
)

func main() {
    greeter := &GreetServer{}
    mux := http.NewServeMux()
    path, handler := greetv1connect.NewGreetServiceHandler(
        greeter,
        // Add an option that customizes protobuf marshalling/unmarshalling behavior
        connectproto.WithBinary(
            proto.MarshalOptions{},
            proto.UnmarshalOptions{DiscardUnknown: true},
        ),
        // Add an option to customize JSON marshalling/unmachalling
        connectproto.WithJSON(
            protojson.MarshalOptions{},
            protojson.UnmarshalOptions{DiscardUnknown: true},
        )
    )
    mux.Handle(path, handler)
    log.Fatal(http.ListenAndServe(
        "localhost:9000",
        h2c.NewHandler(mux, &http2.Server{}),
    ))
}

In this example, connectproto.WithBinary ensures only messages with defined fields are processed, enhancing the security of your gRPC service. connectproto.WithJSON does the same thing but with JSON.

Additional Considerations

While dropping unknown fields is a valuable security practice, it's important to consider potential trade-offs:

Backward compatibility: Clients using older versions of the proto schema will encounter errors if they rely on previously defined unknown fields.
Logging and Debugging: Dropping unknown fields might make it harder to identify the source of unexpected behavior during development or debugging.

In such cases, it's recommended to document these trade-offs and have a clear versioning policy for your gRPC service and client applications.

Conclusion

Exposing gRPC services to the internet requires careful security considerations. By customizing protobuf encoding options, specifically by dropping unknown fields using proto.UnmarshalOptions, you can significantly improve the security posture of your service. Remember to weigh the benefits against potential drawbacks and implement a solution that aligns with your specific needs.

RESTless: Web APIs After REST

Kevin McDonald — Tue, 26 Mar 2024 00:00:00 +0000

The "RESTful API" has been the workhorse of the web for many years. It has been an ever-changing religion with tenants that developers try their hardest to adhere to. But as web applications evolve, user demands grow and our industry experience with API design grows, it's time to re-evaluate this approach. This article explores the limitations of REST and delves into modern alternatives that can unlock a world of possibilities beyond.

Objects? More like "objnoxious"

Imagine building a social media API endpoint to retrieve a user's feed. Using a single REST object to represent a feed item can get messy. This object would need to encompass:

User information (username, profile picture)
Post content (text, images, videos)
User interactions (likes, comments, shares) with timestamps
Additional data like post visibility or author verification status

This "feed" object becomes bloated, especially if the feed contains many posts. Fetching and updating this complex object for every feed interaction can be inefficient. You can split the object up into many child objects but you are likely creating the need for clients to make more requests and greatly increasing the complexity of the API.

This highlights a limitation of REST: forcing real-world entities (like a social media feed) into rigid object structures with a strict hierarchy can lead to cumbersome data management.

Versioning is weird

Let's say you introduce a new field to your product data model in a REST API. Versioning in the path (e.g., /api/v2/products) forces you to update every single endpoint URL that uses that data. Versioning each path by adding a query parameter (e.g., /api/products?version=2) or header seems more targeted, but what if only a specific endpoint needs the new version? Do you maintain a list of versions per endpoint? Do you bump the version for the entire API and default to the latest version? This is open to interpretation and every solution seems awkward to me.

Limited Options, Clever Solution

REST only offers a handful of methods (GET, POST, etc.) to handle data. There are 9 methods in total. Let's talk about each one.

CONNECT
DELETE
GET
HEAD
OPTIONS
PATCH
POST
PUT
TRACE

Most of these are used in niche, hyper-specific ways. I suspect most developers don't know what HEAD, TRACE, OPTIONS, and CONNECT do. None of these are incredibly useful when developing web APIs. So let's ditch them. Let's also ditch PATCH because it's just PUT in a trenchcoat.

Now we have: GET, POST, DELETE, and PUT. Sweet, we're left with enough methods to make a CRUD application. Roughly speaking:

Create = POST
Read = GET
Update = PUT
Delete = DELETE

Wow! Such simple. So elegant. I'm sure glad that everything web developers do boils down to these 4 simple actions... Oh wait, that's totally wrong. There are so many more actions you can do to an object. Just think of how crazy it would be if you were using a programming language where you could only have four pre-defined methods in each class. Think of all of the extra container classes and weird abstractions we'd make on top of that. This sounds like utter insanity and is exactly what REST gives us.

Let's stop pretending that there are only 9 (but actually 4) things you can do to a resource.

Inefficiency of JSON

JSON is slow and inefficient. It'd be insane if we built the internet around this format. Wait, we did? Really? JSON is wasteful in many ways. It's text-based, which has an inherent cost in payload size and processing. JSON also will include key names over and over again and the length of the keys directly translates to longer payloads. This is not ideal if we're trying to save on data transfer. Protobuf, on the other hand, is a compact and efficient binary format specifically designed for data serialization. Even factoring in gzip, JSON loses out to encoded protobuf in pretty much every way: CPU usage, memory usage, message size and speed. This just shows that there are better formats than JSON.

OpenAPI?

OpenAPI is a specification for describing RESTful APIs. It acts as a contract between API providers and consumers, defining the available resources, their properties, and the allowed operations (GET, POST, PUT, DELETE) for each. A common way OpenAPI is used is by generating OpenAPI specifications from the source of the backend service. Depending on the level of library integration these tools can automatically discover the HTTP method, route, request and response types, etc. This can help keep documentation up-to-date compared to manually creating OpenAPI spec, which is pretty incredible.

Okay, now here's where I get philosophical. If we're going to have a declarative specification for our APIs, I believe the specification should be the source of truth rather than the output. In my mind, OpenAPI specification should be the very first thing you write and agree upon and the servers and clients should be. However, many people don't do this because the tooling isn't amazing. Developers have grown fond of specific libraries and frameworks to develop our APIs so the target for generating language/framework/library code is vast and appears to be an incredibly hard problem. OpenAPI tries to solve so many problems at once. It's coming in after we've designed our APIs and is trying to describe what's already there. It's an afterthought.

I've attempted to go down the route of using OpenAPI to generate server stubs and clients. It didn't end well. There were too many issues generating clients and servers, even when the OpenAPI specification was valid. So I had to edit our OpenAPI spec to fit the limitations of the code generators just to get code generation to work. And that was just the beginning of my problems. I contend that this is a natural result of the design goals of the project. OpenAPI was not designed for generating code this way as a priority, so with many complex scenarios, it can be unclear how to map the spec to the semantics of the language/framework/library. OpenAPI wasn't designed for that, it was only designed to describe the API, not the server or client that produces or consumes it. Now consider just how many "targets" for client and server stubs you want. Now consider that target libraries and the OpenAPI spec itself are evolving. This results in a compatibility matrix from hell.

So what other options do we have?

REST has served us well, but the modern web demands more. Let's explore the exciting alternatives that offer a range of benefits and functionalities:

GraphQL

The "Choose Your Own Adventure" API. Still with the social media app example, imagine fetching only the user's name and profile picture for the feed view, and then requesting their full profile and friend list separately when a user clicks on their profile. GraphQL allows you to specify exactly the data you need for each view, reducing unnecessary data transfer.

This method also has its downsides like making the backend API extremely complex.

gRPC

The "Cut the Drama" API. Consider a mobile game that communicates with a game server. gRPC allows you to define remote procedures (like attackEnemy or usePowerUp) that the client can call directly on the server. This removes the need for complex REST resource mapping and makes the communication intent clear. There are variants of gRPC like ConnectRPC that allow for leveraging of HTTP GET requests so you can fully leverage browser caching.

WebSockets

Need real-time updates like a live chat or stock ticker? WebSockets offer a persistent two-way communication channel, ideal for constantly flowing data between client and server. This is different from REST's request-response cycle, allowing for a more dynamic connection.

Server-Sent Events (SSE)

SSE allows the server to push updates to the client without the client needing to constantly ask. Imagine live sports scores or social media notifications. SSE is simpler to implement than WebSockets, but is one-way (server to client).

Here's a conclusion that summarizes the key points and offers a final thought:

The Verdict: REST vs. the Rest

REST APIs have served us faithfully for years, but as our applications become more complex and data-hungry, it's worth considering the alternatives. GraphQL offers flexibility in data fetching, gRPC provides clear and efficient communication, WebSockets enable bidirectional real-time data flow and great browser support, and SSE simplifies server-to-client updates.

The choice ultimately depends on your specific needs. But remember, the API landscape is ever-evolving. HTTP/2 and HTTP/3 have opened up some new functionality that we have yet to fully tap into with our API designs. So, keep an open mind, explore the options, and don't be afraid to break free from the comfy (but maybe slightly threadbare) jeans of REST when a more fitting alternative emerges.

Making gRPC more approachable with ConnectRPC

Kevin McDonald — Tue, 05 Mar 2024 00:00:00 +0000

Introduction

gRPC is an open-source framework for building high-performance applications that communicate with each other. gRPC is language-neutral, meaning clients and servers can be written in different programming languages, and it offers features like authentication, streaming, and load balancing. Also, because it is built using protocolbuffers it gives an amazing way to define contracts in a much more clear way than bolting on swagger to your API after the fact. However, gRPC has some limitations that restrict its usage. It requires special gRPC clients and support for HTTP/2 (which still lacking in some areas), and it doesn't work from Javascript.

Support for Javascript has been solved in a couple of ways. Let's discuss the two approaches!

gRPC-Web

gRPC-Web is a variant of gRPC and gRPC client for Javascript that avoids HTTP/2-specific features like HTTP trailers. This is an incredibly practical solution to the problem. And it does work really well, but I still have my reservations on some of the implementation details.

gRPC-Web doesn't fix the "this doesn't look like any HTTP API I've ever seen" issue that I have with gRPC in general. In other words, I want to be able to send a normal cURL example to someone. gRPC-Web doesn't work for that without special gRPC-specific clients or tooling.

Additionally, I don't like how gRPC-Web is typically deployed. You usually are forced into a proxy that can convert gRPC into gRPC-Web. Instead, I prefer the gRPC-Web implementation to sit alongside the actual gRPC server. The protocol isn't so different from the normal gRPC version so it shouldn't be too much work to add support to existing gRPC server implementations. I know that the popular gRPC library tonic supports gRPC-Web out of the box.

gRPC Transcoding

The idea with transcoding is to annotate your protobuf service methods with HTTP verbs and path patterns that can map a more REST-like API to gRPC. Many solutions allow you to provide a separate config file so you aren't required to have the HTTP annotations making a mess of your protobuf files. Google has a service that can use this mapping and provide a REST-like API on top of your gRPC service. and several gRPC proxies can do this kind of transcoding as well like gRPC-Gateway and envoy.

Here's a simple version of what the annotations can look like:

syntax = "proto3";

package status.v1;

import "google/protobuf/empty.proto";
import "google/api/annotations.proto";
import "google/rpc/status.proto";

service Status {
  rpc GetStatus(google.protobuf.Empty) returns (google.rpc.Status) {
    option(google.api.http) = {
        get: "/v1/status"
    };
  }
}

You can see how status.v1.Status.GetStatus maps to GET /v1/status. Thanks, protobuf options!

However, most ways of deploying this in Go weirdly use proxies, creating a new network hop and a decoding/encoding step. Additionally, transcoding ruins a lot of the benefits you have from a contract-based interface that gRPC provides. Generated clients generally can't interpret the google.api.http options so if you want to keep to a contract-based model you have to rely on converting your protobuf file to OpenAPI and generating clients based on that. I don't generally prefer this method because it adds extra complexity. However, it can be a really good way to support existing APIs by "swapping out" traditional HTTP handles with gRPC.

ConnectRPC

Let me introduce ConnectRPC. I believe it elegantly solves all of the issues I have with the gRPC ecosystem. ConnectRPC is a series of libraries for building browser and gRPC-compatible APIs. With ConnectRPC as the server, you get support for three protocols: gRPC, gRPC-Web and the so-called "Connect" protocol. These three protocols are all served from a single ConnectRPC server by simply using the HTTP content-type header, which gRPC and gRPC-Web clients already send. Let me break down where you might use each protocol:

Microservice communication: Connect or gRPC
Publically exposed API: Connect, gRPC or gRPC-Web depending on client language support
Clients running in environments without HTTP/2 support: Connect or gRPC-Web
Sending simple API examples to colleagues: Connect

Hopefully, I convinced you to at least give ConnectRPC a try. Now, get started with it! Here are the getting started docs for Go. It does a great job at detailing how you make a Go server and client implemented with ConnectRPC.

Multiple Protocol Support

So, ConnectRPC is running three different protocols? Isn't that overkill? You can make everything work with the Connect protocol, but there are benefits to supporting all three.

We can use the gRPC protocol to call into this ConnectRPC service using normal gRPC tooling like grpcurl:

$ grpcurl -plaintext \
          -proto greet/v1/greet.proto \
          -d '{"name": "Jane"}' \
          127.0.0.1:8080 \
          greet.v1.GreetService.Greet
{
  "greeting": "Hello, Jane!"
}

If you have the reflection API enabled, you can omit the -proto option.

$ grpcurl -plaintext \
          -d '{"name": "Jane"}' \
          127.0.0.1:8080 \
          greet.v1.GreetService.Greet
{
  "greeting": "Hello, Jane!"
}

Now here's where the magic is. In addition to gRPC-specific tooling, I can also use generic HTTP tools with ConnectRPC servers, like curl:

$ curl -XPOST \
       -H"Content-Type: application/json" \
       -d '{"name": "Jane"}' \
       "http://127.0.0.1:8080/greet.v1.GreetService/Greet"

This shows how the "I want to be able to send a normal cURL example to someone" desire from above is completely fulfilled.

I also want to point out that there is also the buf curl command which is a CLI tool that allows you to call ConnectRPC services using all three protocols (gRPC, gRPC-Web, Connect).

$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet
{
  "greeting": "Hello, Jane!"
}

This last command actually uses the Connect protocol. We can pass in --protocol to use the gRPC-Web or the gRPC protocol instead:

$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet \
           --protocol=grpcweb
{
  "greeting": "Hello, Jane!"
}
$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet \
           --protocol=grpc
{
  "greeting": "Hello, Jane!"
}

Digging Deeper (optional)

This is an optional section where we will dig deeper into what is happening here with this last command to show what is happening under the hood. We will see how server reflection works with gRPC and a couple uses of it. We can see more details of the previous buf curl command by adding -v at the end:

$ buf curl --http2-prior-knowledge \
           -d '{"name": "Jane"}' \
           http://127.0.0.1:8080/greet.v1.GreetService/Greet -v
buf: * Using server reflection to resolve "greet.v1.GreetService"
buf: * Dialing (tcp) 127.0.0.1:8080...
buf: * Connected to 127.0.0.1:8080
buf: > (#1) POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo
buf: > (#1) Accept-Encoding: identity
buf: > (#1) Content-Type: application/grpc+proto
buf: > (#1) Grpc-Accept-Encoding: gzip
buf: > (#1) Grpc-Timeout: 119999m
buf: > (#1) Te: trailers
buf: > (#1) User-Agent: grpc-go-connect/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#1)
buf: } (#1) [5 bytes data]
buf: } (#1) [23 bytes data]
buf: < (#1) HTTP/2.0 200 OK
buf: < (#1) Content-Type: application/grpc+proto
buf: < (#1) Date: Sat, 02 Mar 2024 06:41:48 GMT
buf: < (#1) Grpc-Accept-Encoding: gzip
buf: < (#1) Grpc-Encoding: gzip
buf: < (#1)
buf: { (#1) [5 bytes data]
buf: { (#1) [244 bytes data]
buf: * Server reflection has resolved file "greet/v1/greet.proto"
buf: * Invoking RPC greet.v1.GreetService.Greet
buf: > (#2) POST /greet.v1.GreetService/Greet
buf: > (#2) Accept-Encoding: identity
buf: > (#2) Content-Type: application/grpc+proto
buf: > (#2) Grpc-Accept-Encoding: gzip
buf: > (#2) Grpc-Timeout: 119994m
buf: > (#2) Te: trailers
buf: > (#2) User-Agent: grpc-go-connect/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#2)
buf: } (#2) [5 bytes data]
buf: } (#2) [6 bytes data]
buf: * (#2) Finished upload
buf: < (#2) HTTP/2.0 200 OK
buf: < (#2) Content-Type: application/grpc+proto
buf: < (#2) Date: Sat, 02 Mar 2024 06:41:48 GMT
buf: < (#2) Greet-Version: v1
buf: < (#2) Grpc-Accept-Encoding: gzip
buf: < (#2) Grpc-Encoding: gzip
buf: < (#2)
buf: { (#2) [5 bytes data]
buf: { (#2) [38 bytes data]
buf: < (#2)
buf: < (#2) Grpc-Message:
buf: < (#2) Grpc-Status: 0
buf: * (#2) Call complete
{
  "greeting": "Hello, Jane!"
}
buf: < (#1)
buf: < (#1) Grpc-Message:
buf: < (#1) Grpc-Status: 0
buf: * (#1) Call complete

This is a little overwhelming at first so I will break down the two different calls that are being made here. Because we aren't passing the protobuf file (or descriptors) as the --schema option we are missing some information needed to generate the protobuf messages from the given JSON string. So that's the first call that buf curl will make, using the Server Reflection API:

buf: > (#1) POST /grpc.reflection.v1.ServerReflection/ServerReflectionInfo
buf: > (#1) Accept-Encoding: identity
buf: > (#1) Connect-Accept-Encoding: gzip
buf: > (#1) Connect-Protocol-Version: 1
buf: > (#1) Connect-Timeout-Ms: 119999
buf: > (#1) Content-Type: application/connect+proto
buf: > (#1) User-Agent: connect-go/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#1)
buf: } (#1) [5 bytes data]
buf: } (#1) [23 bytes data]
buf: < (#1) HTTP/2.0 200 OK
buf: < (#1) Connect-Accept-Encoding: gzip
buf: < (#1) Connect-Content-Encoding: gzip
buf: < (#1) Content-Type: application/connect+proto
buf: < (#1) Date: Sat, 02 Mar 2024 06:33:57 GMT
buf: < (#1)
buf: { (#1) [5 bytes data]
buf: { (#1) [244 bytes data]
buf: * Server reflection has resolved file "greet/v1/greet.proto"

It acquired a version of the greet.proto file from the server. Now our client has everything it needs to make the actual request where it calls the actual service.

With HTTP, that service lives at POST /greet.v1.GreetService/Greet. Here's what the call looks like:

buf: * Invoking RPC greet.v1.GreetService.Greet
buf: > (#2) POST /greet.v1.GreetService/Greet
buf: > (#2) Accept-Encoding: identity
buf: > (#2) Content-Type: application/grpc+proto
buf: > (#2) Grpc-Accept-Encoding: gzip
buf: > (#2) Grpc-Timeout: 119994m
buf: > (#2) Te: trailers
buf: > (#2) User-Agent: grpc-go-connect/1.14.0 (go1.21.6) buf/1.29.0
buf: > (#2)
buf: } (#2) [5 bytes data]
buf: } (#2) [6 bytes data]
buf: * (#2) Finished upload
buf: < (#2) HTTP/2.0 200 OK
buf: < (#2) Content-Type: application/grpc+proto
buf: < (#2) Date: Sat, 02 Mar 2024 06:41:48 GMT
buf: < (#2) Greet-Version: v1
buf: < (#2) Grpc-Accept-Encoding: gzip
buf: < (#2) Grpc-Encoding: gzip
buf: < (#2)
buf: { (#2) [5 bytes data]
buf: { (#2) [38 bytes data]
buf: < (#2)
buf: < (#2) Grpc-Message:
buf: < (#2) Grpc-Status: 0
buf: * (#2) Call complete
{
  "greeting": "Hello, Jane!"
}
buf: < (#1)
buf: < (#1) Grpc-Message:
buf: < (#1) Grpc-Status: 0
buf: * (#1) Call complete

I wanted to show you how the server reflection works because it is a really good strength of gRPC. You can expose an API and have it be completely discoverable. Tools can automatically use this discovery mechanism... but so can humans. Look at these commands with grpcurl:

$ grpcurl -plaintext 127.0.0.1:8080 list
greet.v1.GreetService

$ grpcurl -plaintext 127.0.0.1:8080 describe
greet.v1.GreetService is a service:
service GreetService {
  rpc Greet ( .greet.v1.GreetRequest ) returns ( .greet.v1.GreetResponse );
}

$ grpcurl -plaintext 127.0.0.1:8080 describe .greet.v1.GreetRequest
greet.v1.GreetRequest is a message:
message GreetRequest {
  string name = 1;
}

We can see all the available services using the list and describe commands. And if you pass an object to the describe command you can dig down into message definitions. Protobuf works well here as the API contract for our services.

Conclusion

ConnectRPC offers a compelling solution for building gRPC servers that support multiple protocols. It seamlessly integrates gRPC, gRPC-Web, and the Connect protocol, allowing clients written in various languages and environments to interact with your service.

Here are the key takeaways:

ConnectRPC eliminates the limitations of traditional gRPC by supporting HTTP/1.1 and Javascript environments.
It provides a familiar REST-like API through the Connect protocol while still leveraging the benefits of Protobuf for contracts.
Multiple protocol support with a single server simplifies deployment and reduces complexity.
Existing gRPC tooling can still be used for server reflection and making gRPC requests.
The Connect protocol itself can be used with generic tools like curl, making API exploration a breeze.

If you're looking for a flexible and future-proof way to build gRPC APIs, ConnectRPC is definitely worth considering. I highly recommend checking out the getting started guide for Go for a hands-on approach to using ConnectRPC.