DEV Community: Hugo Martins

Abstractions can be expensive

Hugo Martins — Sat, 10 Aug 2024 00:00:00 +0000

Daniel Lemire wrote in Performance tip: avoid unnecessary copies, comparing Bun to Node.js, about a particular case where copying data becomes expensive:

It turns out that the copy was not happening as part of base64 decoding but in a completely separate function. There is an innocuous function in Node.js called StringBytes::Size which basically must provide an upper on the memory needed by the Buffer.from function.

As shown in the benchmarks, in the piece, Node.js becomes much slower than Bun when computing the same thing (e.g decoding base64). Even though both of these runtimes use the same library, an unsuspicious copy of data deep in the code of Node.js made it more expensive and slower.

I’ve encountered a similar situation in Go. A service was misbehaving and using too much memory. Profiling and investigating, I narrowed it down to the fact that it was using io.ReadAll to transform a response into a slice of bytes ([]byte). In itself, there is nothing wrong with this idea but there’s something particular about io.ReadAll (source):

func ReadAll(r Reader) ([]byte, error) {
    b := make([]byte, 0, 512)
    for {
        if len(b) == cap(b) {
            // Add more capacity (let append pick how much).
            b = append(b, 0)[:len(b)]
        }
(...)

Inside a for loop, ReadAll will validate that the buffer it created (512 bytes) has enough capacity to store the data. If it doesn’t, it duplicates the size of the existing buffer. This happens until the buffer has enough space to store all of the data. It inherently means we will use a lot more memory than the original size of the data. All of this might be unexpected.

If we don’t know the size of the data a priori, there’s not much we can do to reduce this (I believe) but, if we do know the size of the buffer, io.Copy might be a good alternative. It allows you to specify the size of the buffer, before writing data to it, avoiding the memory-intensive capacity increases.

This was true up until, at least, Go 1.20.4. In recent versions, the implementation has changed. But it doesn’t change the underlying fact that this abstraction could potentially be expensive memory-wise, or even contribute to a program performing poorly or misbehaving. As Lemire points out:

The story illustrates why our software is slower than it should be. We have layers of abstractions to fight against. Sometimes you win, sometimes you lose.

These layers are there for a reason, but they are not free.

To make matters worse… these abstraction layers often thicken over time… and the friction goes up.

It is not that the layers are unnecessary: they do good work. They allow us to avoid reimplementing the same things over and over again. They allow us to build on top of several individuals’ expertise and they support frameworks that allow us to be more productive…but they are not always free , there’s a cost associated with these abstractions. And it is important to keep an eye (and monitoring) open to the possibility that sometimes, only sometimes, they are more expensive than we are willing to accept.

On What Lexers Do

Hugo Martins — Tue, 21 Feb 2023 15:31:16 +0000

A couple of weeks ago, I’ve started reading Writing An Interpreter In Go. Reading this book means I’ll be building an interpreter for a language called monkey (you can follow my implementation). It is a topic that I’ve always been curious about but never actually took the plunge and decided to study it.

I’ve dabbled a little bit with it in university. But we never actually wrote any of the internal tools. We always used generators for the code and focused more on the concepts. Rest assured, I can’t remember most of it by now.

While reading this book, I reconnected with the basic concepts of how interpreters and compilers work. I’ve been taking notes and trying to expand my knowledge with other sources. I’ve always heard that to know something, you need to be able to explain it. I’m trying to do that here.

Lexers transform source code into symbols (or tokens). They execute what is called “lexical analysis”. From Writing an Interpreter in Go:

(…) from source code to tokens, is called “lexical analysis”, or “lexing” for short. It’s done by a lexer (…)

Wikipedia defines “lexical analysis” as “the process of converting a sequence of characters (…) into a sequence of lexical tokens (…)”. This aligns well with what we read in the book.

So, when we execute a lexer, we transform the source code (text) into a series of data structures (tokens). It is an integral component of writing interpreters (or compilers) as we need to transform source code into a representation that we understand and can process in later stages (e.g. parsing).

As an example, in Go, if we have the statement str := "string", we are in the presence of three tokens: an identifier token (str), an assignment token (:=) and a string token (string). Tokens are meaningful data structures , allowing us to approach source code as data structures, rather than dealing with text inside of interpreters.

Lexers also help us ignore elements of source code that aren’t relevant (e.g. whitespaces in some languages) and focus just on the elements that actually matter to us. After lexing, all irrelevant whitespace has been stripped off. For example, in C or Go, I wouldn’t expect whitespace to appear as tokens after lexing. In Python, where whitespace is relevant, I expect it to be taken into account in lexing.

One concept I was particularly interested in, as I read, was the fact that lexers need to look-ahead. This isn’t always done the same way, it depends on the approach used, but it is integral to lexing, to be able to resolve conflicts.

As an example, reading a “=“ sign might represent different things. It can be an equal sign (e.g. assignment) or it can be the start of a comparison (e.g “==“). Without looking ahead, it is impossible to resolve conflicts and accurately know what a given character represents in source code.

Lexical analysis is only but a small step, in the process of interpreting source code and its results, that will need to be fed into a parser, which will perform additional steps with the tokens.

It has been a good experience reading these initial sections of Writing An Interpreter In Go and finally feel I’m starting to understand the basics of these concepts.

On Unit Testing in Temporal

Hugo Martins — Wed, 28 Dec 2022 00:00:00 +0000

I’ve been using Temporal, “a developer-first, open source platform that ensures the successful execution of services and applications”, for troubleshooting purposes for a while now. We use it quite extensively at Datadog. Recently, I’ve started actually implementing something on top of Temporal and was amazed at the simplicity of its testing framework.

At first I was a bit confused, as I’m interested in Go and could only find documentation focusing on Java. Eventually, I found How to test Workflow Definitions in Go, a great introduction to the testing framework that Temporal offers with testsuite.

It all starts with a WorkflowTestSuite, in which we can initialize a TestWorkflowEnvironment. This TestWorkflowEnvironment is powerful and we immediately get access to a few important functions, such as AssertExpectations or ExecuteWorkflow. These allow you to do things as diverse as checking that all expected calls (to mocks) have been made and to execute a workflow inside a test environment. We can use RegisterActivity or RegisterWorkflow as well, to ensure our Activities and Workflows are available in the environment we’ll execute our tests in.

Unfortunately, as pointed out in Temporal’s documentation, “unless the Activity invocations are mocked(…) the test environment will execute the actual Activity code including any calls to outside services." This is particularly harmful if the Activities being invoked execute code that makes calls to endpoints on a network because:

it makes the tests unable to execute without connecting to the network;
it makes for unreliable and inconsistent behaviour of the tests;
and it doesn’t allow us to test multiple failure modes.

For this reason, Temporal’s testing framework offers methods to mock Activities in a TestWorkflowEnvironment such as OnActivity. This works well when the Activities are implemented internally but I couldn’t get it to work when the Activities are implemented in a third-party package.

I read a bit of the temporalio/samples-go repository, which houses a collection of samples and accompanying tests, to understand what needs to be done. It is seems simple enough:

type UnitTestSuite struct {
    suite.Suite
    testsuite.WorkflowTestSuite
}

func (s *UnitTestSuite) TestSampleWorkflow() {
    // Initialize test environment for UnitTestSuite.
    env = s.NewTestWorkflowEnvironment()

    // Register needed Activities.
    env.RegisterActivity(externalpackage.ExternalActivity)

    // Mock Activities.
    env.OnActivity(externalpackage.ExternalActivity, mock.Anything, mock.Anything).Return(nil)

    // Execute Workflow and Assert.
    env.ExecuteWorkflow(SampleWorkflow)
    (...)
}

I was surprised it was this easy. I confess I was expecting a little bit more complexity. That might be due to my lack of knowledge about the internals of Temporal, but I have to say I was quite excited about how easily I was able to make progress.

Understanding 'kubectl explain'

Hugo Martins — Tue, 29 Mar 2022 00:00:00 +0000

kubectl explain can be used to get more information about which fields a given resource needs, as well as the meaning behind those fields. It can be used in mainly two ways: either for getting information about the fields of a resource (e.g. kubectl explain <resource>) or by getting information about a specific field of a resource (e.g. kubectl explain <resource>.<field/s>).

Together with api-versions and api-resources it can quickly provide a lot of insight into Kubernetes and what needs to be written on a manifest to properly create or edit objects.

Examples

Getting information about root fields of Pods:

$ kubectl explain pods
KIND: Pod
VERSION: v1

DESCRIPTION:
     Pod is a collection of containers that can run on a host. This resource is
     created by clients and scheduled onto hosts.

FIELDS:
   apiVersion <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata <Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   spec <Object>
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

   status <Object>
     Most recently observed status of the pod. This data may not be up to date.
     Populated by the system. Read-only. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

Getting information about the spec of Pods:

kubectl explain pods.spec
KIND: Pod
VERSION: v1

RESOURCE: spec <Object>

DESCRIPTION:
     Specification of the desired behavior of the pod. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status

     PodSpec is a description of a pod.

FIELDS:
   activeDeadlineSeconds <integer>
     Optional duration in seconds the pod may be active on the node relative to
     StartTime before the system will actively try to mark it failed and kill
     associated containers. Value must be a positive integer.

   affinity <Object>
     If specified, the pod's scheduling constraints

   automountServiceAccountToken <boolean>
     AutomountServiceAccountToken indicates whether a service account token
     should be automatically mounted.

   containers <[]Object> -required-
     List of containers belonging to the pod. Containers cannot currently be
     added or removed. There must be at least one container in a Pod. Cannot be
     updated.

   dnsConfig <Object>
     Specifies the DNS parameters of a pod. Parameters specified here will be
     merged to the generated DNS configuration based on DNSPolicy.

   dnsPolicy <string>
     Set DNS policy for the pod. Defaults to "ClusterFirst". Valid values are
     'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS
     parameters given in DNSConfig will be merged with the policy selected with
     DNSPolicy. To have DNS options set along with hostNetwork, you have to
     specify DNS policy explicitly to 'ClusterFirstWithHostNet'.

(...)

It explains what this field represents, as well as all the fields that can be configured inside this specific field, and their meanings. It can be a powerful tool to understand Kubernetes. A sort of man for Kubernetes itself.

It can be particularly interesting to use kubectl explain to inspect, when you forget, essential fields in Kubernetes manifests, or internal details.

What Can Be Done With 'kubectl run' Command?

Hugo Martins — Fri, 18 Mar 2022 00:00:00 +0000

kubectl run is a command that can “create and run a particular image in a pod”, which means that it will start a pod with a particular image. It can also create the pod with some particular details by using combinations of flags. We can see the code for it in pkg/cmd/run/run.go.

Interesting Flags

--image specifies the image to be executed.
--command allows to overwrite the command field in the container with whatever arguments are sent after --.
--env to set environment variables in the pods, in the form of key:value and one variable per --env flag.
--expose and --port to create a ClusterIP associated with the pod.
--labels which allows us to configure labels for the specific pod. Labels are accepted with only one flag, separated by a comma.
--dry-run for testing purposes.

Examples

From documentation:

kubectl run nginx --image=nginx
kubectl run hazelcast --image=hazelcast/hazelcast --env="DNS_DOMAIN=cluster" --env="POD_NAMESPACE=default"
kubectl run hazelcast --image=hazelcast/hazelcast --labels="app=hazelcast,env=prod"
kubectl run nginx --image=nginx --command -- <cmd> <arg1> ... <argN>

Use Case

While I don’t see much use in kubectl run for dealing with production grade systems, it can be a great resource for testing out different configurations of pods and creating a bunch of pods if there’s a need for it. It can also be a great introductory command for when first starting to deal with Kubernetes because it eases the interaction with Kubernetes without having to deal with YAML right away.

Due to its --output flag, it can also be a great way to use kubectl to create pods, in a stable way, from scripts because we can have stable and reproducible outputs from it. The fact that we can use a --dry-run flag is also a big plus.

Essential Fields in Kubernetes Manifests

Hugo Martins — Tue, 15 Feb 2022 00:00:00 +0000

I recently started a quest to complete CKAD in the next few months, by May 2022. As I’ve explained in that previous essay, “I have spent quite some €€€ enrolling in it and I feel that it can still teach me a lot of relevant concepts about Kubernetes that will be useful” on my day-to-day.

While studying for CKAD, through Kubernetes Certified Application Developer (CKAD) with Tests, I’ve come to realize the importance of understanding the syntax of manifests in Kubernetes. Subconsciously, I obviously already knew this - the same way I know how important it is to dominate the syntax of a given programming language - but it is too easy to fall into a pattern of copy-pasting-and-changing, or simply filling in the gaps in already existing manifests.

Manifests in Kubernetes are the baseline of describing and defining resources, that we can then create and edit afterwards. Manifests represent the object specification describing “its desired state, as well as some basic information about the object (such as a name).” ¹ These manifests are most often described in .yaml files.

In essence, there are four essential fields in Kubernetes manifests that must be present in all manifests. These are: apiVersion, kind, metadata and spec. Each of these might have widely varying values populating them.

As an example, a starting point for a Kubernetes manifest would be:

apiVersion:
kind:
metadata:
spec:

apiVersion

apiVersion allows us to define what version of the API a given resource is going to be using. It can be simply v1, which means it will be part of the core API specified at /api/v1. It can also be <name>/<version>, for example batch/v1, specifying that at resource uses an API that is under /api/<name>. ²

We can find more about what APIs and versions exist on a given cluster by executing kubectl api-version:

$ kubectl api-versions
admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
(...)

Results will differ from cluster to cluster, and between Kubernetes versions. We can have custom APIs, disabled APIs, or recent APIs could’ve been implemented in different Kubernetes versions.

ReplicationController and ReplicaSet are two popular objects that clarify this and differ in the apiVersion. ReplicaController is a component of the core API in v1 so we would write apiVersion: v1 in its spec. ReplicaSet, which evolved from ReplicaController, is a component of a more recent API served at apps/v1. This sort of versioning is incredibly powerful and flexible, allowing Kubernetes to evolve while keeping a lot of backwards compatibility.

kind

kind represents the kind of object that is specified via a manifest. Each kind of resource will be available on a particular API. This makes it essential that the specified kind and apiVersion match on a specific manifest.

We can inspect which kind we can use for objects by executing kubectl api-resources:

kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
apiservices apiregistration.k8s.io/v1 false APIService
controllerrevisions apps/v1 true ControllerRevision
daemonsets ds apps/v1 true DaemonSet
deployments deploy apps/v1 true Deployment
replicasets rs apps/v1 true ReplicaSet
statefulsets sts apps/v1 true StatefulSet
(...)

With kubectl api-resources we can quickly see as well which apiVersion needs to be specified in order to specify a particular resource.

metadata

metadata describes information of an object that allows for the unique identification of that object. When creating a manifest, this field should have at least an associated name. Usually we will also see a field named labels.

spec

spec defines the desired state of the object in Kubernetes. It will vary widely between different resources and API versions, which means that it can be tricky to figure out - or memorize - all the needed fields.

I’ve found that there’s one instance of a resource that can be created without a spec field which is namespaces. If we create a namespace with only apiVersion, kind and metadata, creating the namespace with kubectl create, Kubernetes will accept that manifest but it will create the namespace internally with an appropriate spec. As an example:

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace

Running kubectl create results in:

$ kubectl create -f my-namespace.yaml
namespace/my-namespace created

Executing kubectl get will allows us to see the injected spec field:

$ kubectl ns my-namespace -o yaml
apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
  (...)
  selfLink: /api/v1/namespaces/test
  uid: f1b901a6-31d6-457a-aaaf-0cb6d600d52c
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

All of this information can be confirmed in Kubernetes' own documentation by reading Required Fields. Although this isn’t a deep exploration of manifests, having solid bases can be extremely important to understand what has been built on top of this. Reasoning about this structure also provides a glimpse at the baseline that provides so much flexibility to Kubernetes, allowing it to have 50+ components out-of-the-box and a lot of extensibility via custom resources.

Describing a Kubernetes object ↩︎
API Overview ↩︎

Certified Kubernetes Application Developer (CKAD) in 2022

Hugo Martins — Sun, 06 Feb 2022 00:00:00 +0000

May 2021 in sunny Portugal. I virtually attended KubeCon + CloudNativeCon Europe 2021, writing a series of daily highlights about this experience. It focused on the talks I most enjoyed, what I learned and technologies I’d keep an eye on. I even took the time to write some tweets, even though I barely use Twitter.

#KubeCon setup. Starting off with LitmusChaos office hours. pic.twitter.com/c7Q24gl6Dd

— Hugo Martins (@caramelomartins) May 4, 2021

At the time, I was using Kubernetes extensively. I was working as an SRE, which was really more a Platform Engineer, working on tooling in the Software Development Life Cycle ™️ domain, following the hype of Internal Platform Teams. Kubernetes was quickly becoming, if it wasn’t already, the de facto standard framework to build these sorts of platforms on. Kubernetes serves as the underlying “OS” of sorts, while a lot of applications and customization are baked on top of it.

Due to the type of work I had been doing, and because it is a domain that interests me and I want to keep working on it, I ended up making the decision to enroll in the Certified Kubernetes Application Developer (CKAD). I racked in an hefty discount from attending KubeCon + CloudNativeCon Europe 2021 and had an entire year to study for, register and pass CKAD. What could go wrong?!

It was all very promising and hopeful but I ended up not being able to study for CKAD and ended up not registering to take the exam after almost a year. To make matters worse, CKAD went through a series of program changes just four months after I enrolled. It now includes questions about Helm, more security-oriented aspects, deployment strategies and CRDs, among other changes.

Although I no longer feel the same motivation and excitement towards passing the exam, mostly because I’m not working as closely with Kubernetes anymore, I have spent quite some €€€ enrolling in it and I feel that it can still teach me a lot of relevant concepts about Kubernetes that will be useful, as well as help me improve my kubectl proficiency, which I still use everyday.

Because of the reasons above, I’m going to try and complete Certified Kubernetes Application Developer (CKAD) in 2022, more specifically, in the last few months I have until May. This means that I’ll have to dive into Kubernetes concepts and tooling, while studying, and write more about it which I’m hoping I’ll share in the form of small snippets and notes.

I don’t want it to become a reference guide to Kubernetes but I believe that sharing notes and small essays might help me get more motivated towards studying for CKAD. In this way, I believe, I’ll probably start a small series of notes on Kubernetes. I’m not yet sure on what this series will be in terms of structure, or its name, but I’m hopeful that it will give me the extra boost I need in order to successfully study and complete CKAD.

Locks, Deadlocks and Liquidbase

Hugo Martins — Thu, 30 Dec 2021 00:00:00 +0000

“Long time no see”, heh? I’ve been busy with other undertakings - such as switching jobs - which means I’ve had less time to write. The emptiness in here since October is explained by that but I’m back and I bring some extraordinary fun with Liquidbase and its locking mechanism.

Liquidbase is an open source library designed to improve versioning of database schema changes. You can check its source code but, in essence, it helps controlling versioning across database schema changes, similar to Flyway, and it is written in Java, making it ideal for bootstrapping Spring applications.

Recently, I’ve had a nasty encounter with Liquidbase, due to its use in Spinnaker, in particular clouddriver. It envolves corrupt locks and deadlocks caused by those.

Liquidbase relies on a changelog, composed of changesets, to reliable and accurately know which changes have already been executed or haven’t been executed. Liquidbase uses a table in the database, named DATABASECHANGELOG, to track these changesets, whether they have already been executed and when they have been executed.

Because Liquidbase is aware that multiple instances of the same application might be booting up at the same time and might be reading DATABASECHANGELOG at the same time, or conflicting times, it also has a table named DATABASECHANGELOGLOCK. In this table, instances that are currently reading the database can share locks and manage themselves out of concurrency issues.

Whenever an application with Liquidbase boots up, it will read DATABASECHANGELOGLOCK and check for an existing lock. If the lock has been acquired by another instance, it will wait until the lock has been freed to proceed with acquiring the lock and perform the necessary operations.

Unfortunately, as we know, this behaviour is prone to situations of corrupt locks being left in the database, for instances that are no longer requiring the lock or even running, leaving all other instances locked out and unable to boot. In essence, creating a deadlock because all instances are waiting for the existing lock to be freed, while the original instance has probably terminated a long time ago.

This is exactly what happens with Liquidbase if an instance happens to be unexpectedly terminated without freeing its lock. Any other instance that tries to boot up won’t be able to acquire the lock and will enter an infinite loop, until the lock is finally free.

I’m assuming this is a common occurrence because it is even documented in Liquidbase’s documentation:

If Liquibase does not exit cleanly, the lock row may be left as locked. You can clear out the current lock by running liquibase releaseLocks which runs UPDATE DATABASECHANGELOGLOCK SET LOCKED=0

Unfortunately, I didn’t know this before hand. When I encountered this issue, it made for a fun afternoon of chasing ghosts…until I saw somewhere online that there might be a corrupt lock still in the database.

If you happen to encounter this error and see instances booting up and looping infinitely with Waiting for changelog lock…please make sure you aren’t witnessing a deadlock due to a corrupt lock in your database.

Devops Principles of Feedback: Understanding What's Going On

Hugo Martins — Mon, 04 Oct 2021 00:00:00 +0000

A few months ago, I’ve written at length about DevOps themes, in essays such as Three Ways of DevOps and DevOps Principles of Flow: Deliver Faster, based on reading The DevOps Handbook. Since then, I’ve had this essay half-baked, sitting on my drafts and haven’t been able to complete it up until now.

In this essay, I want to focus our attention on The Principles of Feedback. I’ve previously summarized them as:

The Principles of Feedback are the guiding principles behind the Second Way. At their core, they enable a “fast and constant” flow of feedback from production all the way back to engineers. These principles focus on getting the necessary ambience and tools in place for engineers to monitor their work and quickly react to any adverse situations. They focus on amplifying feedback from the operational side of software to the development side of software.

In a simple analogy, if The Principles of Flow focus on the left-to-right flow of information (e.g. from “dev” to “ops”), The Principles of Feedback guide the right-to-left flow of information (e.g. from “ops” back to “dev”).

Principles

We make our system of work safer by creating fast, frequent, high quality information flow throughout our value stream and our organization, which includes feedback and feedforward loops. This allows us to detect and remediate problems while they are smaller, cheaper and easier to fix; avert problems while they are smaller, cheaper, and easier to fix; avert problems before they cause catastrophe; and create organizational learning that we integrate into future work. (The DevOps Handbook, pp. 27)

To achieve what the above quote proposes, The DevOps Handbook outlines the following principles:

Working Safely within Complex Systems: We must create the necessary environment in which we can work in our systems “without fear, confident that any errors will be detected quickly, long before they cause catastrophic outcomes (…)”.
See Problems as They Occur: We need to ensure we have telemetry to accompany our production environment and be able to effectively monitor any incidents long before they affect customers. We can also use telemetry to validate we are achieving our desired goals.
Swarm and Solve Problems to Build New Knowledge: Swarming, from Wikipedia, “is a collective behavior exhibited by entities, particularly animals, of similar size which aggregate together, perhaps milling about the same spot or perhaps moving en masse or migrating in some direction.” In the technological sense, this means that when an issue, even the smallest of issues, occurs, all elements of a team must unite to resolve the problems as quickly as possible.
Keep Pushing Quality Closer to the Source: Ensure that quality is enforced by our peers, rather than top-down bureaucratic processes.

Telemetry Everywhere

A direct, but at times complex, way of abiding by the two initial principles, is to ensure that telemetry is available to detect and solve problems in real-time. For this to be a possibility, we should have centralized systems for telemetry infrastructure used by the entire organization. We should also ensure that applications have appropriate logging that can help production issues.

Having this infrastructure in place helps use telemetry to guide problem-solving, rather than relying on semi-blind troubleshooting approaches. Apart from using telemetry to solve problems, we should also push for telemetry to be used for production metrics that are outside the scope of daily work such as, for example, number of successful logins. These metrics can then be used as SLIs that contribute to more effective SLOs.

Another important aspect of telemetry is that it cannot be a walled garden. We must ensure that telemetry can be self-service, providing visibility to developers about the telemetry of their applications, extending this feedback loop to everyone inside the organization. This increases transparency and guarantees that everyone has a shared vision of “reality”.

With telemetry available, we need to ensure that we use it to better anticipate problems and achieve our goals, this means statistical analysis, usage of anomaly detection techniques and automated alerts.

This fast feedback loop, with telemetry and infrastructure for self-service, allows for deployments to be safer, for developers to follow their work and see how their applications behave, have developers share on-call rotas and even manage their services in production - instead of having to rely on a dedicated operations team.

Review & Coordinate Close to the Source

We should aim at creating review and coordination processes as close to the source code as possible. For example, avoiding change approval processes entirely, fearlessly cutting on bureaucratic processes, instead relying on peer-review processes, coordination and scheduling of changes, without formal approvals. Code reviews have been proven to improve code readability, a sense of ownership, as well as code transparency and contributions. We should review the review process, to ensure that we are enabling the effectiveness of reviews, rather than creating a process that doesn’t ensure the quality of the source code.

Reviewing and coordinating closer to the source also enables teams that swarm whenever an issue occurs to troubleshoot it also contributes to the creation of shared knowledge, that becomes distributed throughout the team, allowing teams to be more effective and efficient.

Whenever possible bottom-up coordination is much better than top-down mandates that are based on bureaucratic processes.

Conclusions

An organization that has a well set up infrastructure, which provides self-service telemetry, aligned with the autonomy of self-sufficient teams is much better equipped to face the challenges that modern development teams are facing every day.

These practices - outlined throughout this essay - are the core of the underlying practices that enable organizations to ensure there are feedback and feedforward loops in place that allow for errors to be cheaper, smaller and caught earlier.

I hope in the future to write more about the last way of DevOps, The Principles of Continual Learning, which are by far my favourite guidelines of the entire book because they are, at this moment, perhaps the lesser-known guidelines and the ones that are less put in practice.

Another Way of Accessing Host Resources in Minikube

Hugo Martins — Mon, 20 Sep 2021 00:00:00 +0000

A few years ago, I wrote an essay about this topic titled “How to Access Host Resources in Minikube Pods?”. In it, I have described how to access host resources in Minikube. Something I thought should be easy but, at the time, it really wasn’t.

Back then, I’ve described an approach, based on a Github issue, relying on listing kernel routing tables, that allowed us to fetch the IP of the host machine inside Minikube. I described why this worked, going so far as sharing a few snippets of code from the Kubernetes source code.

While this has served me well for the last few years, I’ve recently discovered that this approach is a bit outdated - the documentation on that essay doesn’t even exist anymore. I wrote this when Minikube was v1.6.2 and it is now v1.23.1…a lot has happened. After searching a bit more, I’ve understood that Minikube now has a proper solution for this, documented in Host access.

This approach applies for Minikube bigger than v1.10 and it essentially relies on using host.minikube.internal in order to connect to the host Minikube is running on. According to the documentation, “minikube v1.10 adds a hostname entry host.minikube.internal to /etc/hosts.” I’ve tested this and it seems to actually work, while being much easier and reliable.

As with the initial approach, the same caveat applies:

The service running on your host must either be bound to all IP’s (0.0.0.0) and interfaces, or to the IP and interface your VM is bridged against. If the service is bound only to localhost (127.0.0.1), this will not work.

KubeCon Europe 2021: Highlights #4

Hugo Martins — Fri, 07 May 2021 09:27:59 +0000

Today was the last day of KubeCon Europe 2021. As I have done in the last three days, it is only right that I write about what today was like for me. You can find my highlights for previous days in KubeCon Europe 2021: Highlights #1, KubeCon Europe 2021: Highlights #2 and KubeCon Europe 2021: Highlights #3. These are a series of notes written as a stream of consciousness without much editing. This year's edition of KubeCon Europe was once again completely virtual.

Sessions

When Prometheus Can’t Take the Load Anymore by Liron Cohen was the first good session I watched today. This session focused on showcasing some issues with Prometheus such as scalability issues, no real high availability, no global view of information and no long term storage. Then Liron Cohen explained, in quite some detail, three potential solutions: M3, Cortex and Thanos. In the end, they used Thanos because of the simplicity of its architecture when compared with M3 and Cortex. I had no idea about the complexity that is needed to have a highly available Prometheus which is why this session was so enlightening for me.

On a smaller scale, SIG CLI: Intro and Updates by Maciej Szulik, Katrina Verey and Jeff Regan, was also interesting because I wanted to know more about this SIG. Updates with better integration between kubectl and kustomize were presented, as well as some other tooling that I was unaware of such as kui and krew. A funny thing I discovered was that they used a pattern in kubectl development, which I have seen before, of coupling commands (from Cobra) with business logic. They are now trying to go back on that decision, which is something I've also faced before because coupling the framework with the business logic ends up creating a lot of interoperability issues in the business logic.

Next up is Battle of Black Friday: Proactively Autoscaling StockX by Mario Loria and Kyle Schrade. This session started as a story of going from manual scaling to proactive automated scaling. It was a wonderful session about proactive scaling, going further than Cluster Autoscaling and HPA. They've developed a system that can easily handle spikes in traffic of more than 500% in less than 30 seconds by having an endpoint consuming events from Kubernetes and then, based on those events, having a CronJob warming up and cooling down more resources, proactively, A great example of how to deal with the delays of cluster autoscaling and HPA when your spikes are not gradual, non-linear and unannounced. Also a great example of keeping systems simple, with a single endpoint and a CronJob to manage the warming up and cooling down proactively.

How We are Dealing with Metrics at Scale on GitLab.com by Andrew Newdigate. Showcased a solution for mitigating low precision, unactionable and flappy alerting. It leverages key metrics, such as Apdex, Requests per Second, Errors per Second and Saturation, as SLIs to then be able to monitor if SLOs are being met. Metrics are centralized and shared in a metrics catalogue that defines key metrics, per-service SLOs, inter-service dependencies, and generates Prometheus and Thanos rules, as well as, Grafana dashboards. Using Grafana as a launchpad to other components can also help with context during on call.

One great discovery for me today was Gateway API presented in Gateway API: A New Set of Kubernetes APIs for Advanced Traffic Routing by Harry Bagdi and Rob Scott. It provides a set of resources in Kubernetes that improve on what already exists with Ingresses. It focuses on improved routing and can handle much more complex scenarios, is more expressiveness and extensive. It can handle canary deployments (traffic splitting) much more easily, header matching and multicluster traffic. It also supports HTTP, UDP and TCP routing through simple and expressiveness resources. I believe this is a truly ground-breaking thing in this niche.

Cert-Manager Beyond Ingress – Exploring the Variety of Use Cases by Matthew Bates was a nice introduction to a lot of use cases of cert-manager that I was unaware. Apparently, you can use cert-manager in almost all layers of Kubernetes. Pod to Pod, Kubelet, kubeadm and webhooks. I had no idea cert-manager had all of this reach.

High Throughput with Low Resource Usage: A Logging Journey by Eduardo Silva featured a deep dive into the complexities of log collectors and processors such as Fluentd. It also went into some detail on how Fluent Bit can optimize data handling and I/O with data serialization and buffer management via chunks. There are a couple of slides that are particularly enlightening in terms of the complexity that a log collect and processor has to handle.

I still had a few more sessions on "to watch" such as K8s Labels Everywhere! Decluttering With Node Profile Discovery. by Conor Nolan, Isolate the Users! Supporting User Namespaces in K8s for Increased Security by Mauricio Vásquez, Application Autoscaling Made Easy With Kubernetes Event-Driven Autoscaling by Tom Kerkhove, and Choose Wisely: Understanding Kubernetes Selectors by Christopher Hanson. Given that the sessions are available on-demand on KubeCon's platform I will potentially watch them later next week. There was just so much to see that you can't get everywhere, otherwise, you'd be awake for 4 days straight.

Cool Tech

ketpn: "Cloud-native application life-cycle orchestration".
M3: "Uber’s Open Source, Large-scale Metrics Platform for Prometheus"
Cortex: "Horizontally scalable, highly available, multi-tenant, long term Prometheus".
Thanos: "Open source, highly available Prometheus setup with long term storage capabilities".
kui: "hybrid command-line/UI development experience for cloud-native development".
krew: " plugin manager for kubectl command-line tool".
Virtual Clusters: "a new architecture to address various Kubernetes control plane isolation challenges".
Goldilocks: "utility that can help you identify a starting point for resource requests and limits".
Rook: "Production ready management for File, Block and Object Storage".
Ceph: "uniquely delivers object, block, and file storage in one unified system".
Gateway API: "a collection of resources that model service networking in Kubernetes".
Fluent Bit

KubeCon Europe 2021: Highlights #3

Hugo Martins — Thu, 06 May 2021 08:02:59 +0000

I wrote about personal highlights from KubeCon Europe 2021 yesterday and the day before. I think it is only right if I write about it a third time. For those that are unaware, I’m writing about my experience in KubeCon Europe 2021 and today is Day 3. These will be a series of notes written as a stream of consciousness without much editing. This year’s edition of KubeCon Europe was once again completely virtual.

Sessions

I started my day with some leftover sessions from yesterday. In this case, I started with Zero Pain Microservice Development and Deployment with Dapr and KubeVela by Hongchao Deng which went about presenting a solution for improving workload deployments with KubeVela and Dapr as sidecar for metrics. Two things were noticeable for me in this session: more and more tools being built to ease and abstract away the complexity of Kubernetes, in this case, KubeVela helps Platform Teams create “application-centric” resources in Kubernetes; and, once again, there’s this trend of using sidecars for deployment auxiliary technology with their applications.

In the same vein, Turning Your Cloud Native Apps Inside Out With a Service Mesh by Adam Zwickey and Liam White presented another use case of using Envoy as a sidecar to transparently create service meshes that can be managed by a Platform Team, leaving developers to focus exclusively on their own services. Have I said that there’s a trend to use sidecars with auxiliary technology? Envoy seems to be front and center in this. Above we just saw using Dapr for metrics, for example.

One of the things I enjoy about these conferences are the introductory sessions such as Introduction and Deep Dive Into Containerd by Kohei Tokunaga and Akihiro Suda where I have learned a bit more about how containerd internals function - for example, apparently you can easily build containerd clients - or CoreDNS Deep Dive: Building Custom Plugins by Yong Tang where I have understood that we can build plugins for CoreDNS, which I had no idea about. It is a great way of getting a quick introduction to a specific technology or learning about extensibility in a project.

Why Use Managed Kubernetes?: It’s Dangerous to Go Alone! by Seth McCombs is a candidate for my favourite session in this year’s KubeCon. Seth tried to dismantle the notion that you frequently hear at these conferences about “the right way” to do things. For most use cases, using managed Kubernetes services is aptly justified because, by having a team of experts in some cloud provider working day and night on these services, they can guarantee much more stability and reliance than we’d otherwise be able to. It is much more important to focus on building cool things, and features that can contribute to the core business rather than wasting time managing Kubernetes. There are cases where running your own Kubernetes is justified, particularly if you need customizations, but managed services usually contribute to fewer costs, more reliability and more time to build features. End users don’t necessarily care about whether you are running Kubernetes or using AWS/GCP. Another thing that I noticed, and I was talking about in yesterday’s updates, was Seth’s reluctance in managing your own etcd - it truly must be a pain. End users don’t necessarily care about running on Kubernetes. Seth ended the session with a great quote: “Build cool stuff. In whatever way works for you. With the tools that work for you”. Kubernetes should be just one more tool in your toolbox.

I also had a chance to watch three good sessions on security topics. Uncovering a Sophisticated Kubernetes Attack in Real-Time by Jed Salazar and Natália Réka Ivánkó where they suggested that security could be more like SRE. They have showcased a solution that leverages https://ebpf.io/ and https://cilium.io/ for security monitoring in Kubernetes (again, with sidecars in pods!!). This solution allows to proactively acting whenever specific actions are executed in Kubernetes clusters, moving security from something that is always on fire to being something proactive and monitored in real-time. Hacking into Kubernetes Security for Beginners by Ellen Körbes and Tabitha Sable was an extremely creative session where I’ve heard things such as: “We are made of stars but your RBAC shouldn’t be.” More than being simply creative, these sessions expanded my understanding of the attack surface in workloads running in Kubernetes. Misconfigured RBAC, powerful tools configured and running pods, potential vulnerabilities in Kubernetes APIs, image vulnerabilities. There’s so much to take care of! A few suggestions have been left about having audit logs in Kubernetes APIs, leveraging admission controllers and CI scanning for vulnerabilities. Finally, The Art of Hiding Yourself by Lorenzo Fontana presented security issues in Kubernetes from the perspective of someone trying to hide inside a given system. It talks about ways to hide process activity, network activity and storage activity, and even pods!! What fascinated me the most was how fairly trivial it is to hide these things, as long as you know how to circumvent the common ways of monitoring these things. Sometimes you need to look at things from a different angle to be able to see clearly. This session leveraged Falco for real-time security monitoring in Kubernetes.

Finally, one of the sessions I was most excited about was Sidecars at Netflix: From Basic Injection to First Class Citizens by Rodrigo Campos Catelin and Manas Alekar. I am a big fan of Netflix’s engineering culture which prompted my enthusiasm. This is a story of how Netflix evolved from an architecture where EC2 VMs stored application code, along with auxiliary technology for logging, metrics and networking, through a solution with Titus where each EC2 VM was running everything in containers, and now with Kubernetes and trying to advance Kubernetes sidecars. They have laid out some sidecar issues that they are currently facing such as no startup order guarantees, no shutdown order guarantees, can’t be straightforwardly used with Jobs and can’t use them with initContainers. Interestingly enough, the thing I was most fascinated by was KEP 753. I had no idea but it seems that there have been some proposals to advance sidecars to First Class Citizens. The proposal didn’t go anywhere but there are still “pre-proposal” efforts to start a new proposal and introduce sidecars in Kubernets’ specification. I was amazed because there are so many instances of companies using sidecars in production, I’ve mentioned a few of them in the last couple of days, but there’s no real standard/specification around sidecars in Kubernetes.

Cool Tech

Crossplane: “Compose cloud infrastructure and services into custom platform APIs”.
KubeVela: “Make shipping applications more enjoyable”.
OpenKruise: “A Kubernetes extended suite for application automations”.
Dapr: “Simplify cloud-native application development”.
kubespray: “Deploy a Production Ready Kubernetes Cluster”.
nerdctl: “Docker-compatible CLI for containerd, with support for Compose”.
eBPF: “eBPF is a revolutionary technology that can run sandboxed programs in the Linux kernel without changing kernel source code or loading kernel modules”.
Cilium: “eBPF-based Networking, Observability, and Security”.
yugabyteDB: “Open source, cloud native relational database for powering global, internet-scale apps”.
Titus: “a container management platform that provides scalable and reliable container execution and cloud-native integration with Amazon AWS”.