DEV Community: Kerno

Into the Multicloud-Verse with Crossplane

Kerno — Wed, 20 Sep 2023 14:27:00 +0000

Welcome to this issue of Activation Function. Every other week, we introduce you to a new and exciting open-source backend technology (that you’ve probably only kind of heard about… ) and explain it to you in 5 minutes or less so you can make better technical decisions moving forward.

In this issue, we’ll explore Crossplane, an open-source framework to provision and manage cloud resources across any cloud provider (aka a multi-cloud control plane) using the magic of Kubernetes.
But wait... Why the heck would I use Kubernetes to do this? Let’s find out.

TL;DR:

Crossplane is an incubating CNCF project created in 2018 by Upbound.
Crossplane lets you provision & manage infrastructure in a cloud-agnostic way using the Kubernetes API by abstracting the underlying cloud provider. It offers support for all major cloud providers and can be extended to support any third-party tool running pretty much anywhere.
Crossplane is built to serve both Ops and Dev teams. Platform teams can create an interface that application teams can self-serve cloud resources without worrying about implementation details.

Why do I need a multi-cloud control plane?

Unless you’ve lived under a rock for the past 10+ years, just like the rest of us, you’re probably hooked on this awesome thing called managed cloud services. It’s convenient to get started with, easy to scale, and, in many cases, cost-efficient.

Initially, the game was simple – You needed a managed cloud service, you got it from your main cloud provider, and that was it.

And then, the cloud wars got REAL, and things went from 0 to 100 real quick!

You now have so many options to choose from. Mixing and matching (aka adopting a multi-cloud strategy) allows you to pick whatever best fits your use case, budget, compliance, geography, etc.

But of course, there are always trade-offs. In this case, it’s complexity.

Running across multiple cloud providers is a mess—fragmented tooling, no portability, different UIs/UXs, no standardization, etc. You get the picture…

Wouldn’t it be nice to leverage whatever resources best fit your needs because, you know... we’re all so unique and get a single pane of glass, ideally a single API, to manage and abstract away all the complexity?

Of course you do!

Enters Crossplane

At its simplest, Crossplane is an abstraction layer that lets you provision and orchestrate cloud resources across multiple vendors in a declarative way using a single unified API.

Think of it as a universal remote control for cloud services.

In addition to its declarative and unified nature, Crossplane brings 2 major advantages:

Workload portability – This allows dev teams to build applications that can run on any cloud provider without any modifications. We’ll break this down later in the how it works section.

Reconciliation loop – This ensures that the state of your deployment matches the state of the configuration you passed in.

Does this all sound familiar? Well, it should, as this is what Kubernetes does for containers.

In fact, Crossplane is essentially extending Kubernetes beyond the cluster and turning it into a universal Control Plane. It has adopted the Kubernetes declarative resource management model and leverages the Kubernetes API, etcd, and controllers to manage external cloud services.

Definition: A Control Plane is a system that provides cloud infrastructure management, routing, and orchestration.

Given this tight integration with Kubernetes, Crossplane plays nicely with most tools in the Kubernetes ecosystem (e.g., Argo CD) AND can run directly on top of an existing Kubernetes cluster without requiring any changes.

And there you have it… That’s why Kubernetes.

[Note] Kubernetes manages containers (pods) and the resources they consume across nodes. Crossplane manages workloads (container, serverless, others) and the resources they consume across cloud providers or on-premise environments.

How does Crossplane work?

To use Crossplane, you must first install its control plane in a Kubernetes Cluster.

Note: Crossplane must be installed in a Kubernetes cluster—either where your applications run or a new one.

Crossplane extends the Kubernetes API using custom resource definitions (CRDs) known as Providers responsible for creating and managing external cloud-provider-specific services (e.g., an RDS database). Crossplane has a Provider for every major cloud, including AWS, GCP, and Azure, but you can also build custom Providers using the provider template.

Once your Provider is installed, you must configure it with the necessary security credentials and endpoint details to communicate with your cloud provider's API.

Once that’s all set, you’re ready to start creating resources (e.g., an RDS instance) to be managed by the Provider. These are called Managed Resources, and you create them by declaring a Kubernetes custom resource (CR) that specifies things like the size, type, and any other configurations specific to the cloud resource you want to spin up.

This is where the Provider’s reconciliation mechanism kicks in, and the magic happens. The Provider’s controller detects the change (desired state), and the Provider communicates with the cloud provider's API and attempts to reconcile the actual state of the cloud resource with the desired state declared in the CR, in this scenario, creating an RDS instance.

[Note] Crossplane Managed Resources are just Kubernetes resources, which means that you can use any Kubernetes tool to monitor and query the state of these resources

But what if you need to spin up other components alongside the main one (e.g., networking, security, RBAC, etc.)? Compositions are here to help!

Compositions let you create multiple individual Managed Resources as a single object known as Composite Resources (XR). Like Managed Resources, XRs have a reconciliation loop that will sync their desired and actual state.

[Note] Before being able to create Composite Resources, you’ll need to create a Composite Resource Definition (XDR), which defines the schema for a custom API you’ll use to create your XRs.

Crossplane for Devs

Crossplane was created to serve both Administrators (e.g., Platform Engineers) and Developers. So far, we’ve explored the administrator side of things, but what’s in it for devs?

This is where portability comes into play. With the help of Crossplane, Developers can define workloads without worrying about implementation details, environment constraints, and policies.

They do this by leveraging Claims.

Claims are the primary way developers interact with Crossplane. Claims access the custom APIs the platform team defines in a Composite Resource Definition to create the developer's desired resource.

[Note] Claims are like Composite Resources. The difference between Claims and composite resources is that Crossplane can create Claims in a namespace, while composite resources are cluster-scoped.

Crossplane Vs. Terraform

Like Crossplane, Terraform lets you dynamically provision and declaratively make infrastructure changes across cloud providers and build reusable components. However, there are a couple of differences worth pointing out:

State Management – Crossplane offers out-of-the-box automated reconciliation (aka drift correction) leveraging Kubernetes controllers and etcd. On the other hand, unless you’re using a Terafform cloud, you’ll need to configure your own backend and manage the state of your resources manually.
Policy-Driven Provisioning – Crossplane provides a native policy framework to enable policy-driven provisioning and management. On the other hand, Terraform requires external tools to do the same.

That being said, Terraform is a much more established technology with a huge ecosystem of tools and a large community. Additionally, Terraform is a standalone tool, whereas Crossplane requires you to run Kubernetes. Finally, the lack of a preview feature similar to Terraform's dry-run/plan feature to assess and verify changes in advance is another drawback of Crossplane.

So which solution should you go with? Well, as usual, it depends.

Are you ready to go all-in on Kubernetes? Do you already have expertise in HCL? Do you like managing infrastructure using APIs? Does your existing tooling (e.g., CI/CD tooling) play nice with Kubernetes? Is the Terraform Open Source license an issue?

Where to next

Crossplane is still a fairly new project; therefore, resources and documentation are still limited. That being said, here are a few resources to get you started on the right path:

Documentation

Articles & Tutorials

An Introduction to Crossplane – The original and first introduction written by the creators of Crossplane
vRelevant Blog – A blog written by Nate Reid covering various cloud computing topics, including Crossplane.
First steps with Crossplane

Books

Videos & talks

Communities

People to follow

Viktor Farcic – Developer Advocate at Upbound
Bassam Tabbara – Crossplane co-creator
Jared Watts – Crossplane co-creator
Illya Chekrygin – Crossplane co-creator
Matthias Lübken – PM at Upbound

That’s it, folks! I hope this gave you an overview of Crossplane. There are a lot of technical nuances to effectively implementing Crossplane and tailoring it to your needs, so I strongly recommend looking at the documentation.

Until next time!

P.S. Did you enjoy this content? Sign up here.

Programming the Kernel with eBPF

Kerno — Tue, 05 Sep 2023 09:23:55 +0000

Welcome to this issue of Activation Function. Every month, we introduce you to a new and interesting open-source backend technology (that you’ve probably only kind of heard about… ) and explain it to you in 5 minutes or less so you can make better technical decisions moving forward.

In this issue, we’ll explore eBPF (Extended Berkeley Packet Filter), an exciting new technology that makes programming the kernel flexible, safe, and accessible to developers.

TLDR

eBPF is a mechanism that makes the kernel dynamically programmable without modifying the source code.
eBPF is safe, fast, incredibly flexible, and extensible.
eBPF has been running in production for over half a decade at internet scale on millions of servers.
eBPF use cases range from observability, networking, security, tracing, and profiling.

[IMPORTANT NOTE] eBPF is now a standalone term that doesn’t stand for anything. You'll see the term BPF in Linux source code, and you'll see BPF and eBPF used interchangeably in tooling and documentation. The original BPF is sometimes referred to as cBPF (classic BPF) to distinguish it from eBPF.

Why program the Kernel?

The kernel can oversee and control the entire system, which, on the one hand, makes it the ideal place to implement networking, security, and observability capabilities and, on the other hand, makes it very risky to fiddle with.

As a result, innovation at the kernel level has been super slow! After all, as the saying goes,

With great power comes great responsibility.

Up until recently, if you wanted to add functionality to the kernel, you had two options:

Try to change the kernel's source code and convince the community that the change is required, which, as you can imagine, took ages.

Load kernel modules (LKMs) can be risky on many levels… (security, performance, stability, compatibility, and the list goes on.) and costly to maintain as every kernel version upgrade can break them. Enters eBPF

Enters eBPF

‍
eBPF is a mechanism that makes the kernel dynamically programmable, kind of like how JavaScript lets you dynamically change the behavior of a webpage. Here is a great high-level explanation by Brendan Gregg:

eBPF does to Linux what JavaScript does to HTML. (Sort of.) So, instead of a static HTML website, JavaScript lets you define mini-programs that run on events like mouse clicks, which are run in a safe virtual machine in the browser. And with eBPF, instead of a fixed kernel, you can now write mini-programs that run on events like disk I/O, which are run in a safe virtual machine in the kernel. In reality, eBPF is more like the v8 virtual machine that runs JavaScript rather than JavaScript itself. eBPF is part of the Linux kernel.

Although eBPF is far from completely replacing LKMs, it sets itself apart by bringing great flexibility while mitigating risk by putting solid safety and controls in place.

How we got here?

1992 – Van Jacobson wanted to troubleshoot network issues, but existing network filters were too slow. He and his team developed BPF (Berkeley Packet Filter) to be fast, efficient, and easily verifiable to run in the kernel safely.

BPF was a great technology, but it had a few limitations that became apparent over the years as networking technology evolved. Among other things:

It wasn't adapted to modern processors and multi-processor systems.
It was stateless, which made it a bad fit for complex packet operations.
It took a lot of work to extend for developers.

2014 – Alexei Starovoitov introduced the extended BPF (eBPF) design that took things to a whole new level by:

Overhauling the BPF instruction set to take advantage of modern hardware.
Introducing Helper functions that eBPF programs can call to interact with the system.
Introducing the bpf() system call so that user space programs can interact with eBPF programs in the kernel.
Introducing the eBPF verifier which ensures that an eBPF program is loaded only if it’s safe to run.
Moving beyond packet filtering and opening the door for many use cases around networking, observability, security, tracing, and profiling.

Today, eBPF is a general-purpose compute engine within the Linux kernel that allows you to hook into, observe, and act upon anything happening in the kernel

Check out this talk by Alexei Starovoitov for an in-depth history of eBPF.

How does eBPF work?

Before we delve into this, it’s essential to understand the difference between the kernel and user space in Linux.

Here is a quick rundown:

The Linux kernel is the software layer that sits between your applications and the hardware they run on in a layer called the user space.
The user space is unprivileged; therefore, it can’t access the hardware directly.
When an application requires something from the hardware, it will need to request the kernel, which is privileged, to do it on its behalf using the system call (syscall) interface.
The kernel then relays the request to the hardware, coordinating concurrent requests and ensuring everything runs smoothly.

Alright, back to eBPF. Without going the rabbit hole, here is how it works on a high level:

Step #1 | Program Development
You can write your own eBPF program using a tool like bpftrace that provides an easy-to-learn high-level language or the BPF Compiler Collection (BCC) Python framework. The program is then compiled into bytecode.

[Note] As a beginner, you don’t need to write eBPF code from scratch, as BCC comes with over 70 tools you can use out of the box. Here is a glimpse of what you have at your disposal:

Step #2 | Program Verification
The bytecode runs through the eBPF verifier inside a VM to ensure it will not harm the system before being loaded into the kernel.

[Note] The verification process is quite complex. You can read more about it here. Although much work has gone into improving and simplifying it, you can still run into strange errors when developing your program. If you need help, check out the eBPF Slack community channel.

Step #3 | Program Attachment
The verified program is loaded into the kernel and attached to predefined hook points before being further JIT compiled at runtime into native machine instructions to ensure maximum performance.

Step #4 | Program Execution
The program is triggered on predefined events and helper functions are called.

Maps are then used to pass data between the kernel and user space or other eBPF functions and to maintain the state.

[Note] eBPF program becomes active when loaded into the kernel. You don’t need to reboot the machine, restart existing processes, or change anything about other applications.

eBPF in production

Since its inception in 2014, eBPF capabilities have continued to grow, supported by 300+ kernel developers and major tech players, including Netflix, Meta, Google, Cloudflare, DoorDash, and many others, running eBPF-based tools in production for over half a decade 24/7 at internet scale on millions of servers. Let's look at some examples:

LinkedIn uses eBPF for Infrastructure Observability.
Netflix has developed a network observability sidecar called Flow Exporter, which uses eBPF tracepoints to capture TCP flows in near real-time.
Cloudflare uses eBPF for network security, performance monitoring, and network observability Apple uses eBPF for kernel security monitoring.
Meta uses eBPF to process and load balance every packet coming into their data centers.
DoorDash uses eBPF for application monitoring.
Digital Ocean and Cruise use eBPF for GPU performance monitoring.

And the list goes on.

eBPf limitations

No technology is perfect, and eBPF isn't an exception. Let's discuss a few current limitations you should be aware of:

eBPF was initially released in a limited capacity in 2014 with Linux 3.18. You need at least Linux 4.4 or above to use eBPF fully.
Despite much work, eBPF portability between kernel versions and distributions is still not 100% there.
eBPF is still a pretty complex technology that isn't easy to grasp for the average developer. Anyone working with eBPF will need a solid knowledge of networking and kernel inner workings.
eBPF is still in the early stages of expanding to other OS ecosystems, with Windows leading the charge .
Despite great efforts by the community and large companies like Google to secure eBPF, it's still vulnerable to cyber attacks.

eBPF-based projects

Despite a lot of effort put in by the community to make eBPF more accessible, there reality is that it's still quite a complex technology to work with for the majority of developers.

The good news is that if you want to leverage the power of eBPF, there are a growing number of projects that can help you do that without writing eBPF programs:

Falco is a behavioral activity monitor designed to detect anomalous activity in applications.
Tetragon provides eBPF-based transparent security observability combined with real-time runtime enforcement.
Parca helps you track memory, CPU, I/O bottlenecks broken down by method name, class name, and line number over time.
Cilium is an open source project that provides eBPF-powered networking, security and observability.
Calico Open Source is designed to simplify, scale, and secure container and Kubernetes networks.
Hubble is a fully distributed networking and security observability platform for cloud native workloads.
pwru is an eBPF-based tool for tracing network packets in the Linux kernel with advanced filtering capabilities.
Pixie is an open source observability tool for Kubernetes applications.
Kerno provides the best developer experience to monitor and troubleshoot distributed cloud-native applications quickly and autonomously.

And the list goes on.

Where to next?

If eBPF sounds like your cup of tea, and you're interested in exploring further, you're in luck, as many great free resources are available. Here are a few:

Documentation

Articles & Blogs

Tutorials

bcc Python Developer Tutorial. This tutorial is about developing bcc tools and programs using the Python interface
Learn eBPF Tracing: Tutorial and Examples.
The bpftrace One-Liner Tutorial - Learn bpftrace for Linux in 12 easy lessons, where each lesson is a one-liner you can try running

Books

Videos & talks

Communities

People to follow

That's it for today. I hope this gave you a good idea of what eBPF is and how you can use it for your next project. There is still SO MUCH to unpack when it comes to eBPF, so I encourage you to go out and explore!

Until next time!

Redefining Time Series Analytics @ Pinterest & Scaling Dev Tools @ Meta

Kerno — Tue, 22 Aug 2023 14:48:56 +0000

Welcome to Activation Function – Each week we help you discover, understand, and draw inspiration from interesting open-source backend technologies used by top software companies.

Sounds like your cup of tea? Great! Subscribe here

Alright, let's dive in.

Issue 001 | TScript & Goku to Streamline Time Series Analytics for Observability

TScript is a language the Pinterest engineering team designed to streamline time series database queries and analytics. The goal was to create a language that encapsulates DB queries as variables, allows for multiline entries, makes the queries expandable, provides a filter for DB queries, and is verbose enough for the user to quickly understand the syntax from a time series context.

Goku was the DB chosen by the Pinterest engineering team after evaluating the performance of Ganglia, Graphite, and OpenTSDB. Goku has four distinct advantages over OpenTSDB (the leading contender):

Goku uses an inverted index engine - a significant efficiency improvement over OpenTSDB.
Goku implements Gorilla compression - 12x compression.
Goku’s engine computes near the storage layer allowing for parallel processing and, thus, faster response times.
Goku uses thrift binary over JSON (OpenTSDB) - faster queries at scale.

TScript leverages multiple features, making it easier for developers to create queries and dashboards that visualize the results. Combining explicit syntax and graphing functionalities, the following snippet (See Figure 1.2) will return a dashboard (See Figure 1.3) with appropriate markings.

The team's key challenge was data processing after it was returned from the DB. By preallocating memory using NumPy NaN values, it’s possible to consolidate DataFrames and achieve a considerable performance improvement.

Scaling developer tools - Meta’s approach

Meta has an extensive codebase; at scale, the company has experienced challenges with the current best-in-class tools (Ex: Git, GitHub) - Sapling was born.

Sapling provides multiple features aimed at working with multiple repositories, simplifying UI, and augmenting operations we’ve taken for granted. For example, it’s optimized for lazy loading the files of a repository, thus making it efficient for retrieving large amounts of files and making it possible to work on massive sets of files without local storage. The drawback is that this requires a constant connection to the web. The second major update was layering a set of features/commands on top of Git - Branching and merging now allow for concurrent development. As an engineer awaits code review, they’re able to keep working on their code.

Meta engineers push code to production on a regular basis. At scale, removing friction between developers and them running their code is a key undertaking for leadership. The engineering team has released the second version of Buck - Buck2. It’s an open-source large-scale build system that benchmarked 2x as fast at completing builds as Buck1.

Static code analysis plays a vital role at Meta. By incorporating several platforms into the process, Meta engineers have validated their code and mitigated crashes and potential issues - Infer, RacerD, and Jest.

Infer is easy to install and deploy within Java, C, C++, and Objective-C applications. It’s possible to run validations on portions of a project; here’s an example of validating a branch called “feature” and a “main.”

RacerD was a project that stemmed from the need for concurrency analysis at scale. Engineers at Meta released an MVP in 2017 with the following features in mind:

Prioritizing a high signal detection ensures that numerous alerts don't overwhelm developers. High false positives can lead to "alert fatigue," where developers begin to ignore warnings, potentially overlooking real issues.

An interprocedural analysis approach indicates a deeper view of the codebase. By tracking data races through nested procedure calls, the tool can understand the context better, capturing the bigger picture of potential data races in larger, modular projects.

Eliminating the need for manual annotations streamlines the analysis process. Manual annotations can be error-prone, tedious, and become outdated as code evolves. Automating this or using tools that infer these relationships ensures consistency and reduces maintenance overhead.

Speed is crucial in modern software development, especially with CI/CD pipelines. Rapidly analyzing and reporting on a massive codebase ensures the development workflow remains uninterrupted, promoting higher code quality and faster release cycles.

Differentiating between joint (coarse-grained locking) and rare (fine-grained synchronization) in product code underscores the primary focus on understanding and analyzing coarse-grained locking. This pragmatic approach ensures the most frequent potential issues are addressed first, leading to more stable software.

The resulting tool showcases the possibility of static concurrency analysis for rapidly evolving, vast codebases like Meta's. Its genius is automating the understanding of threads, locks, and memory, eliminating the need for manual, error-prone human input. This automation seamlessly integrates into multiple developers' workflows. Core features like the News Feed, which preemptively addressed over 1,000 concurrency concerns, wouldn't be sustainable without such efficiency and scale in a concurrent environment. This represents the true impact of next-gen analysis.

You may have noticed that Infer isn’t available for languages other than C, C++, Java, and Objective-C. As JavaScript remains a dominant language in the web vertical, Meta needed a testing framework that addressed this segment - Jest.

Jest was created with simplicity in mind and currently supports projects that use the most popular runtimes, such as Node.js. Jest optimizes testing efficiency by ensuring tests maintain a unique global state, enabling parallel execution without hiccups. Jest smartly prioritizes previously failed tests to expedite the process and adjusts the test sequence based on file execution times. This strategy saves time and ensures the most crucial issues are addressed first, showcasing intelligent test management.

📝 References