DEV Community: Ethan J. Jackson

Developers won’t test if it’s too hard

Ethan J. Jackson — Thu, 10 Sep 2020 16:14:15 +0000

Projects need a story for local development. With some projects, the development environment is cobbled together and evolves organically. With others, a dedicated team designs and manages the dev environments.

I've noticed a common theme in the ones I like. Good development environments make me confident that my code will work in prod while giving me fast feedback. These sorts of environments are fun to code in, since I can easily get into the flow — I don't have to stop coding to wait for my code to deploy, or wait until it's merged to do a "proper" test.

Recently, I've been thinking of test usefulness and test speed as the fundamental tradeoff in development environments. It's easy to build a robust testing environment that's slow to use. Or an environment that gives fast feedback, but forces you to test in a higher environment. The challenge is finding the right balance between the two.

In this post I'll explore why it's hard to have both, and what some companies have done about it.

How deployment pipelines help

Before focusing on development environments, let's take a look at deployment pipelines through the lens of test usefulness and test speed.

Most companies test code in a series of environments before deploying to production. In a well-designed deployment pipeline, you have high confidence that changes will work in prod once they reach the end of the pipeline.

In the ideal world, you'd be able to instantly tell, with 100% certainty, whether changes will work in production.

However, this is impossible to build in. Instead, we create deployment pipelines where development is quick to test in, but isn't as similar to production. Once things are working in development, developers deploy them to staging, which is as similar to production as possible, for final testing.

Note: Some companies have different names for these environments, or more environments, but the concept generally applies

Staging environments: high confidence, slow feedback

The most common reason bugs slip through to production is that they're tested in environments that aren't similar to production. If you don't test in an environment that's similar to production, then you can't really know how your code will behave once it gets deployed.

Staging environments are the last place changes are tested before going live in production. They mimic production as closely as possible so that you can be confident that a change will work in production if it works in staging. The similarities between staging and prod should go deeper than just what code is running — VM configuration, load balancing, test data, etc should be similar.

Staging environments live in the upper left of our "test usefulness vs speed" spectrum. They give you high confidence that your code will work in prod, but they're too difficult to do active development in.

However, they hold a nugget of wisdom: the key to getting useful test results in development environments is to make them similar to production. For the rest of this post, I'll focus on dev-prod parity as a proxy for test usefulness since the former is easier to evaluate.

Let's dig a bit deeper into staging environments to see what we do (and don't) want to replicate in development environments.

How is staging similar to production?

Here are some common ways that staging environments match production.

They run the same deployment artifacts (e.g. Docker images) for services.
They run with the same constellation of service versions. If you test with a dependency at v2.0 in staging, it better not be v1.0 in production.
They run on the same type of infrastructure (e.g. on a Kubernetes cluster running in AWS, where the worker VMs have the same sysctls).
They have realistic data in databases.
They're tested with realistic load.
They run services at scale (e.g. with multiple replicas behind load balancers)
If the application depends on third-party services (like Amazon S3, Amazon Lambda, Stripe, or Twilio), they make calls to real instances of these dependencies rather than mocked versions.

The relative importance of these factors varies depending on the application and its architecture. But it's useful to keep in mind the factors that you deem important, because you may want your development environment to mimic production in the same way.

Why not just use staging for development?

Developing directly in the staging environment ruins the principle of having a final checkpoint before deploying to production, since it would be dirtied by in-progress code that's not ready to be released.

But putting that aside, developing via a staging environment is slow:

The environment is shared by all developers, so testing is blocked for all developers if any broken code is deployed.
Deploying is slow because it requires going through the full build process, even if you're just making a small change.
Debugging is difficult since the code is running on infrastructure that developers aren't familiar with.

Whitepaper: How Cloud Native kills developer productivity

Development environments: a sea of tradeoffs

Development environments don't need to be perfect replicas of productions to be useful. Parity with production follows the Pareto principle: 20% of differences account for 80% of the errors. Plus, deployment pipelines provide a "safety net", since even if a bug slips through development, it'll get caught in staging.

This lets us cut some of the features of staging that decrease productivity during development. But what should we cut?

The sweet spot for development environments is the shaded area around "ideal".
We want our development environments to be much faster to test in than staging,
and we're willing to sacrifice a bit of "test usefulness" to get that.

Here are some common compromises teams make, allowing them to operate in the ideal area.

Problem: Slow preview time

Nothing breaks your flow like having to wait 10 minutes to see if your change
worked. By the time you're able to poke around and see that your change didn't work,
you've already forgotten what you were going to try next.

Solution: Hot reload code changes

Docker containers are great since they let you deploy the exact same image that you tested with into production. However, they're slow to build since they don't handle incremental changes very well. Doing a full image build to test code changes wastes a lot of time.

Docker volumes let you sync files into containers without restarting them. This, combined with hot reloading code, can get preview times for code changes down to seconds.

The downside is that this workflow doesn't let you test other changes to your service. For example, if you change your package.json, your image won't get rebuilt to install the new dependencies.

Solution: Have a separate development environment per developer

It's tempting to share resources in development so that there's less to maintain, and less drift between developers. But the potential for developers to step on each other's toes and block each other outweighs the conveniences, in my opinion.

The most common thing that drifts when using isolated environments is service versions. If your development environment boots dependencies via a floating tag, images can get stale without developers realizing it. One solution is to use shared versions of services that don't change often (e.g. a login service).

Problem: Cumbersome debugging

Previewing code changes is only one part of the core development loop. If the
changes don't work, you debug by getting logs, starting a debugger,
and generally poking around. Too many layers of abstraction between the
developer and their code make this difficult.

Solution: Use simpler tools to run services

Even if you use Kubernetes in production, you don't have to use Kubernetes in development. Docker Compose is a common alternative that's more developer-friendly since it just starts the containers on the local Docker daemon. Developers boot their dependencies with docker-compose up and get debugging information through commands like docker logs.

However, this may not work for applications that make assumptions about the infrastructure setup. For example, applications that rely on Kubernetes operators or a service mesh may require those services to run in development as well.

Solution: Run code directly in IDE

In traditional monolithic development, many developers run their code directly
from their integrated development environment (IDE). This is nice because IDEs have integrations with tools such as
step-by-step debuggers and version control.

Even if you're working with containers, you can run your dependencies
in containers, and run just the code you're working on via an IDE. You can
then point your service at your dependencies by tweaking environment variables.
With Docker Desktop, containers can even make requests back to the host
via docker.internal.host.

The downside of this approach is that your service is running in a
substantially different environment, the networking is complicated, and
versions of dependencies like shared libraries tend to drift.

Implementation challenges

Sometimes, you're forced to make compromises because it's just too hard to build the
perfect development environment. Unfortunately, most companies need to invest
in building custom tooling to solve the following problems.

Working with non-containerized dependencies, like serverless: Some teams just point at a shared version of serverless functions, which quickly gets complicated if they write to a database.
Too many services to run them all during development: Applications get so complex that the hardware on laptops isn't sufficient. Some companies run just a subset of services or move their development environment to the cloud.
Development data isn't realistic: Because production data contains sensitive customer information, many development environments just use a small set of mock data for testing. Some teams set up automated jobs that back up and sanitize production data. Others point their development environments at databases in staging, which tend to be more similar to production.

Conclusion

Development environments are usually an afterthought compared to staging and
production. They evolve haphazardly based on band-aid fixes. But developers
spend the bulk of their time in development, so I think they should be
designed consciously by making tradeoffs between test usefulness
and test speed.

Unnecessary differences between development and production cause bugs to slip through
to staging and production. Therefore, differences between development and production should be intentional, and designed to speed up development.

What does your ideal development environment look like? What tradeoffs does it make?

Resources

Try Blimp for booting cloud dev environments that hot reload your changes instantly.

Whitepaper: How Cloud Native kills developer productivity

Why Eventbrite runs a 700 node Kube cluster just for development

Why SREs should be responsible for development environments

Why Eventbrite runs a 700 node Kube cluster just for development

Ethan J. Jackson — Thu, 27 Aug 2020 20:48:19 +0000

In Part 1 of this interview with Remy DeWolf, a principal engineer on the DevTools team at Eventbrite, we discussed what information factored into Eventbrite's decision to move their development environment to the cloud.

The DevTools team at Eventbrite set out to build yak because they had too many services to run locally. They knew they wanted to move their development environment to the cloud, but it ended up yak had additional benefits that ranged from sharing environments to transitioning to working remotely due to COVID.

In this post, we dig into how yak works, what it's like for devs to use it, and how it's been received.

Q&A

How is the Eventbrite application architected?

This is a common story that you will find in a lot of startups. The founding engineers built a monolith and the strategy was to build features fast and capture the market. It was a very successful approach.

As the company grew over time, having a large team working on the monolith became challenging. When reaching a certain size, it was also harder to keep scaling vertically.

Over time, some of the monolith was migrated over to microservices. Now, new services are generally containerized, and the monolith is containerized in dev but not in production.

What prompted you to rehaul your dev envs and what problems did you set out to solve?

See How did you decide it was time to build yak? from Part 1 of the interview.

How did you convince your company?

In the beginning, we partnered with developers to help us focus on the most important features and also to keep them excited about the technology. Specifically, there was a great interest to learn more about Kubernetes. We also added instrumentation to our developer tools so we could measure how much time developers were wasting.

Once we understood how much time the developers spent waiting or dealing with issues, we had to make a call between spending money on cloud computing or wasting engineer time. We presented our plan to the CTO and we got the green light to move forward.

How Cloud Native kills developer productivity

Making the switch to microservices but think it’s too good to be true? Or you already made the switch but you’re starting to notice that local development is harder than it used to be. You’re not alone. Read more of this whitepaper

What’s the developer workflow like with yak?

Every morning, a developer has two options:

Reconnect to their previous session: this takes a few seconds and they can resume their work from where they left it the previous day.
Update their local branch and update their remote docker images: this takes 5-7 minutes to get the environment updated.

From this point, all their containers are running remotely and they can work through the day. Here are some of the common operations:

Change code locally: Changed files are automatically synced over to their remote containers. It usually takes a few seconds for the changes to be available. We use rsync for this, which is very efficient. To keep it simple, we do a one-way sync (from laptop to remote container).

Note: this flow is much faster than the standard flow of building/pushing/deploying images, which in practice is hard to get under a minute for large applications.

Debug code: Developers can add breakpoints in their code and attach to a running container to get a live debugging session. We provided a command that wrapped kubectl attach under the hood.
Run tests: Developers can run unit tests locally, but any tests that require dependencies (such as a DB or Redis) can be run remotely in a pod. For integration tests, they can run tests in a specific pod and connect to the other services directly.

Most of the frontend changes are done locally and don’t require the cloud. The cloud is very useful for backend development and running various tests.

How does it work?

Every developer has their own namespace where they manage their remote developer environment. Kubernetes does the heavy lifting and yak simplify the management of their containers.

How does the DevTool team interact with the development environments?

Usually, we don’t need to interact with the dev env but only focus on the big picture, such as managing the clusters, and adding more features to the tools.

We do support the developers, so sometimes we would directly connect to a namespace to troubleshoot some issues, via standard Kubernetes commands like kubectl logs.

What’s the ongoing maintenance burden?

In the beginning, it was a lot of work because we had a few issues where we didn’t understand the root cause, and we were also weak with the user documentation.

Over time, we got this under control. The documentation was revamped and we built a good knowledge base about the most common errors.

How has the environment changed from when you first designed it?

Our approach was always to deliver incremental value by first focusing on a minimum viable product (MVP) and adding features over time. For these reasons, we made many changes from the original design.

Here are a few interesting changes:

Our infrastructure was running on one EKS cluster originally. At one point, we had 700 worker nodes, and 14,000 pods running. We ran into performance and rate-limiting issues that made us reconsider this single-cluster approach. Over time, we switched to a multi-cluster architecture where each cluster had no more than 200 nodes.
Syncing the code directly into running containers could sometimes cause the container to crash if the changes made the application fail the probe check. After iterating a few times on how to solve this problem, we decided to set up a sidecar container that is responsible for syncing the code.
To persist data over time (for example to save the MySQL database files of a developer) we use Statefulsets backed by EBS volumes. However, AWS has a limitation around EBS volumes -- an application running in a pod on EKS must be on a node in the same availability zone (AZ) as the EBS volume. To solve this problem, we partitioned our EKS nodes per availability zone and we used taints to make sure that our Statefulset would be in the same AZ.

Have there been any unexpected benefits?

Yes,

Sharing environments:
Have you ever heard a developer say "but it worked for me locally when I ran the tests?" Consistency improves by running in the cloud. The ability to share developer environments proved to be very helpful when trying to understand test failures or work on issues that were hard to reproduce.
Working globally:
We have a globally distributed team but most of the test/QA infrastructure is in the US. Simple operations like resolving the application dependencies or downloading a Docker image requires a lot of networking round trips. If the network latency is poor, these operations are slow.

By running on the cloud, the developer opens a connection to some container (with port forwarding or by getting a shell) and then they can run their commands from the same AWS region where the rest of the infrastructure is located. For our engineers based outside of the US, being able to develop on the cloud has been a huge improvement.
Transitioning during COVID:
When COVID happened, all developers switched to working remotely. For some, this meant sharing home internet connections with other households or moving back to their family. It would have been extremely difficult or impossible for some of them to run the developer environment locally. Operations such as pulling Docker images or resolving application dependencies would require gigabit of data on a daily basis. By developing on the cloud, the transition to remote work was fairly seamless and the developers were able to continue their work from home.

Kelda and Eventbrite

Kelda has collaborated with Eventbrite for a long time. We first met when we were building the predecessor to Blimp, which moves your Docker Compose development environment into the cloud. Eventbrite had already built yak internally, and we were trying to make a general solution. We’ve been trading ideas ever since.

Check out Blimp to get the benefits of yak without having to build it yourself!

References

Part 1 of the interview: Why managing dev environments is a full time job at Eventbrite

Read Blimp commands and usage in the Docs

See if you're making any of these 5 common Docker Compose mistakes

Remy DeWolf's Medium

Why managing dev environments is a full time job at Eventbrite

Ethan J. Jackson — Thu, 13 Aug 2020 16:31:16 +0000

Deciding when to invest in developer productivity improvements is hard. If you’re on the ops side of things, you’re usually concerned about production and releases. If you’re a developer, you’re concerned about getting new features out as quickly as possible.

Usually, teams make development productivity improvements in two situations. Either the fix is so small that you can just do it in addition to your other work, or development is so painful that making changes has ground to a halt.

However, there’s still a large murky middle ground: how do you decide that it’s worth investing in a large change to your development workflow before development has ground to a halt?

Remy DeWolf spent three years making these sorts of decisions as a principal engineer on the DevTools team at Eventbrite. He was part of the decision to build yak, which moved Eventbrite’s development environment into the cloud. This was a highly calculated decision since it cost a few EC2 instances per engineer and yak was built from scratch.

In this first post, we’ll dig into how Remy made this tough decision, and got buy-in from the rest of the company. In our next post, we’ll get into the nitty-gritty on how their remote development environment works, and what it’s been like for developers.

Read part 2 of this interview about Eventbrite’s specific setup and 3 unexpected benefits of remote dev environments.

Q&A

How is the Eventbrite application architected?

As the company grew over time, having a large team working on the monolith became challenging. And after a certain size, it was also harder to keep scaling the monolith vertically.

Over time, some of the monolith was migrated over to microservices. New services are generally containerized, and the monolith is containerized in development but not in production.

What’s your development environment setup now?

Every engineer runs ~50 containers which corresponds to the monolith, the microservices, the data stores (MySQL, Redis, Kafka…) and various tools (logging, monitoring).

Developers use yak (which we built internally) to deploy and manage their remote containers.

We use AWS EKS for the Kubernetes clusters, in which every developer has their own namespace. We have hundreds of developers and many EKS clusters.

yak is very similar to blimp since it enables the engineers to manage their remote containers without exposing them to the complexity of Kubernetes.

How did you decide it was time to build yak?

Before yak, each developer ran their development environment locally on their laptop. However, the development environment became so big that it slowed down developer laptops.

The main issue was that you might not realize that this was an issue because it was creeping one service at a time.

Once we added instrumentalization to our tools, we started to understand the scale of the problems. Moving to the cloud is expensive but when we were able to put it side by side with the wasted engineering time, the decision was easy for us.

Another goal of yak was to make Kubernetes easy for developers. We kept it as minimal as possible and the configuration files are plain Kubernetes manifest files. The intent was to feed developer curiosity so they learn more about Kubernetes over time.

What areas do you recommend tracking regarding developer productivity?

Whenever possible, align the developer productivity goals with the business. Every DevTool team should understand how they contribute to the company goals and vice versa. If this is unclear, I would start with that.

Next, make sure that developer productivity is part of the plan, not an afterthought. For example, some engineering teams move to microservices and only track the number of services and the uptime in production. These are great metrics, but they’re incomplete. They will generate inconsistency and the developer experience will suffer over time.

In terms of which metrics to pick, there is no general recommendation. It’s important to understand how developers work, understand how frequently they perform critical tasks, and instrument the tools that they use. With this data, you will be able to identify the most important areas to invest and track the progress over time.

I would also recommend having a metric about mean time to recovery (MTTR). If a developer is completely stuck, how would you bring them back to a clean state so they can resume their work? For this one, if you run the developer environment locally, you will have many different combinations of OS/tools/versions resulting in many different issues. If you are on the cloud and use a generic solution (e.g. Docker + Kubernetes), this problem will be much easier to solve.

How did you collect feedback at Eventbrite?

We had many channels:

Instrumentation into the tools. Every time a developer would build, run, or deploy docker images we would send metrics. Similarly, every CI job would do the same. Then we would generate some dashboards for the metrics to track and measure the progress over time. If you are using a tool like Sumologic or Datadog, it’s very easy to send custom metrics and build dashboards.
Quarterly engagement surveys.
Demos: invite other engineers to show them the progress and engage with them.
New hires: these new employees bring a fresh perspective and they are not afraid to ask questions and challenge the status quo.
Networking: build relationships with other developers (coffee breaks, office visits, lunches, etc..)

Can you give some examples of developer productivity OKRs?

Time to start the developer environment is under x min

This time is usually wasted time, so it’s important to track it and improve it. If the dev stack is unreliable or slow, it would be captured in this OKR.
Engagement is over x%

If you send an engagement survey every quarter, you can have an OKR to make sure the trend is upward. Seeing a drop would mean that the team might not be working on the most relevant projects.
Average time from commit to QA/Prod

This one will capture the CI/CD pipeline effectiveness. If you experience some flaky tests or deployment errors in the pipeline, it would negatively impact the key results.

Over time, some OKRs will be exhausted, so consider renewing them over time. For example, if your survey always has the same questions, developers will eventually stop responding. Also if an OKR has been greatly improved, it’s a good time to shift priorities.

In my personal experience, I would focus on a few OKRs instead of having too many. Sometimes by trying to please everybody, you will not have a big impact. Some projects might require the full team focus, which can temporarily impact other OKRs. This would be a calculated strategy as these projects would bring huge improvements when delivered.

Are there any warning signs people should look out for in order to know their developer productivity is suffering?

This is where it’s important to have good metrics and monitor them over time. You should be able to feel the pulse of your developers by looking at different data points. Ideally, you would tie these to your OKRs and review the progress every sprint and make adjustments.

If you don’t have this data there are still warning signs that productivity is suffering:

Increase in support cases and/or requests for help. If developers need external help to do their work, this is a sign that a process is too hard to use or not well documented.
On the other hand, I’d be worried if you find out that some processes aren’t working properly but nobody reported them to your team. You want developers to be always looking for improvements and not accepting a broken process.

Kelda and Eventbrite

Check out Blimp to get the benefits of yak without having to build it yourself!

References

Read part 2 of this interview about Eventbrite’s specific setup and 3 unexpected benefits of remote dev environments.

See Blimp commands and usage in the Docs

Read 5 common Docker Compose mistakes

Remy DeWolf's Medium

Why SREs Should be Responsible for Development Environments

Ethan J. Jackson — Sat, 01 Aug 2020 16:35:28 +0000

Let's discuss an extremely common anti-pattern I've noticed with teams that are relatively new to containers/cloud-native/kubernetes, etc. More so than when building traditional monoliths, cloud-native applications can be incredibly complex and, as a result, need a relatively sophisticated development environment. Unfortunately, this need often isn't evident at the beginning of the cloud-native journey. Development environments are an afterthought – a cumbersome, heavy, brittle drag on productivity.

The best teams treat development environments as a priority and devote significant DevOps/SRE time to perfecting them. In doing so, they end up with development environments that "just work" for every developer, not just those who are experienced with containers and Kubernetes. For these teams, every developer has a fast, easy-to-use development environment that works for every developer every time.

What's a development environment?

Before we go further, let's get on the same page about what we mean by a development environment in this context. When working with cloud-native applications, each service depends on numerous containers, serverless functions, and cloud services to operate. For this post, a development environment is a sandbox in which developers can run their code and dependencies for testing. It's not the IDE, compiler, debugger, or any of those other tools.

Whitepaper: Why Cloud Native kills developer productivity

Sound Familiar?

You're working on a new project or planning to modernize an old one. The team has read all about the whiz-bang nifty new cloud-native technologies, like containers, Kubernetes, etc. So, you decide to take the plunge and build a cloud-native app.

The team realizes that a core group of DevOps/SREs will be necessary to get everything running in a scalable, reliable, and automated setup. Site reliability engineers are hired/trained and get to work. They setup up Kubernetes, CI/CD, monitoring, logging, and all of the other tools we've learned are critical for a modern application.

Everyone knows that it's the DevOps/SRE team's job to get all of this stuff up and running. However, development environments aren't top of mind. The site reliability team considers it their duty to focus on production and CI/CD – Development is the developer's job. At the same time, the developers think it's their job to deliver application features, not to maintain infrastructure. It's not really anyone's responsibility to focus on developer experience before CI/CD, so it's neglected.

Unfortunately, an ad hoc approach to development environments tends to emerge. Whenever there's a new service, whatever developer happens to be working on it, realizes they need some way to boot their dependencies and test their code. They Google around and figure that Docker Compose is a reasonable way to do this. They copy and paste some example, tweak through trial and error until it's working, and move on. The quality fo this initial compose file ranges widely depending on the DevOps knowledge of the engineer who happened to write it. Sometimes it's pretty solid; sometimes, it's brittle and slow.

Worse, this process repeats. Every time there's a new service, it gets a new git repository, and some new engineer finds themselves writing a compose file. Perhaps this new file is copied from an existing project. Perhaps it's developed from scratch. Either way, now we have two compose files that need to be maintained and updated as the app changes over time. This process repeats and repeats until all services have their own ever so slightly different configuration files that are a nightmare to maintain.

As a result of this (all too common) process. We see several typical issues:

Development environments are unmanageable. They spread across dozens of repositories in dozens of subtly different copy-and-pasted docker-compose files. Keeping these up to date in a fast-changing application is impossible.
Development environments are incomplete. They only deal with containers because they are the easiest for an individual developer to get up and running with docker-compose. Everything else developers need to test (serverless functions, databases, specialty cloud services) requires manual effort.
Developers waste time focusing on things that aren't their specialization. Just as most backend engineers can't CSS their way out of a paper bag, there's no reason for every frontend/AI/data engineer to be experts on the current DevOps trends. Developers shouldn't spend time configuring and debugging development environments — they should spend time building features.

Managed Development Environments

So how do we avoid this all-too-common scenario? The good news is that it's not particularly challenging to do so if you're intentional and proactive. The best teams tend to follow a couple of principles to ensure a great experience.

Clear Responsibility:

There's a team that is explicitly responsible for providing development environments for all developers. That team can be the DevOps/SRE team, or a dedicated developer productivity team. The key is that it's someone's job to focus on this issue. Furthermore, that person is likely someone with a large amount of DevOps expertise that will produce better outcomes more efficiently.

Central Management:

The development environment must be managed centrally by the site reliability team responsible for it. A single git repository contains all of the configuration and scripts necessary for a developer to get going. When the site reliability team changes something, they do so once in that central repository, and all developers benefit. Furthermore, typically, the development environments run in a centrally managed cluster in the cloud. As a result, it's easy for the site reliability team to ensure things work consistently for everyone, and debug problems when they do arise.

Full Automation:

Their development environments are fully automated. A single command brings up everything a developer needs to test their code. Developers don't need to do nearly any manual setup work beyond the code changes they're actively working on.

Conclusion

Achieving these goals isn't easy. It requires a significant and sustained investment from the site reliability team, and buy-in from developers and management to succeed. However, while the cost can be significant, it's small relative to the wasted time and effort saved by giving every developer a fast environment that just works every time. At Kelda, we're working hard to make this dream attainable for every developer.

References

Try Blimp to see how you can improve development speed

Read more about Docker internals -- see how registry credentials are stored.

Tutorial: How to Use Docker Volumes to Code Faster

By: Ethan Jackson

Originally: https://kelda.io/blog/devops-should-manage-development-environments/

How We Cut Our Docker Push Time by 90%

Ethan J. Jackson — Thu, 23 Jul 2020 14:52:18 +0000

At Kelda we're building Blimp, a version of Docker Compose that runs in the cloud. Our goal is to improve the development productivity by providing developers with an alternative to bogging down their local systems with loads of resource-hungry Docker containers.

We've put a lot of engineering effort into supporting all of the Docker Compose fields commonly used during local development, such as volumes, ports, and build. In this post I'll talk a bit about what we've gleaned from the experience as it relates to Docker Compose's build functionality.

services:
    service:
        build: .

When a service has a build field, Blimp builds your images locally, and pushes them to the cloud so that they can be pulled by the development environment. This push can be frustratingly slow, especially on home networks. Waiting 30 minutes for the image to upload before being able to start developing was just unacceptable to us.

To be fair, Docker already has some image optimizations built in, but it didn't do exactly what we wanted out of the box. So, we set out to optimize the push process. To achieve this, we had to dive deep into Docker's image push API.

In this post, I'll cover:

What exactly happens when you do a docker push
How we used this to build our pre push feature and decrease image push times by 90%.

Images are Layers

Before diving into the image push API, you first need to understand what a Docker image actually is.

It's common for developers to think of Docker images like operating system images or ISOs -- a static snapshot of a filesystem that represents the container. Really though, Docker images are quite a bit more sophisticated than that.

A Docker image is made up of layers of filesystems. Put simply, each line in a Dockerfile can be thought of as a layer, and the sum of all the layers the Dockerfile defines is the resulting image.

For example, in the following, FROM python is telling Docker to lay the foundation of our image with the existing Python layers. Likewise, COPY . . creates a new layer which contains all the files in . (i.e. the current working directory, which is referred to as the build context), and overlays them on top of any existing layers.

FROM python
COPY . .
CMD python app.py

The Python base image is 934MB. Assuming that the user is copying in 2MB of files, the base image would make up 99% of the resulting image!

This provided us with a really interesting opportunity to optimize. Why should we waste a user's precious bandwidth pushing this entire image, when often times the vast majority of it is already available from public sources?

Our solution is to have users only push the bits of the image that are unique to their build, and then automatically fetch the rest directly from the base image's registry (e.g. DockerHub), which has plenty of bandwidth.

Bringing it back to the Python example above, we want to make it so that the python layers aren't uploaded over the user's network. Instead, our servers will "pre push" the layers from our high bandwidth servers. Then, the user's docker push just needs to push the layer for COPY . ..

The good news for us was that out of the box, Docker only pushes the layers that don't already exist in the registry. Each layer has a digest, which represents the contents of the layer. These digest IDs are used before pushing to figure out if the registry already has that layer -- if it does, then the client doesn't bother pushing the layer's contents.

But we still had to design a way to prepopulate the base image layers in the registry so that the Docker Push API would reuse them.

The Docker Push API

Docker pushes images in two parts: first it uploads the layers described above. Then, once all the layers are uploaded, it uploads the signed manifest, which references the layers and has some additional metadata.

Simple Layer Caching

Each layer upload starts off with a HEAD request that checks whether the layer already exists in the registry.

If the layer already exists in the registry, then the registry responds with a 200 OK response, and the Docker client doesn't bother pushing it again. In these situations, docker push shows the following output:

6b73f8ddd865: Layer already exists

If the layer doesn't exist, then the registry responds with 202 Accepted, along with the URL that should be used for uploading the layer. The client then uploads the image in chunks via PATCH requests, or directly via a single PUT request.

This layer checking only works when the layers in question exist in the same repository as the image being pushed. So blimp/backend:1 and blimp/backend:2 can share layers, but blimp/backend:1 can't share layers with blimp/another-image:1 (without taking advantage of another API, that I'll describe now).

Cross Repository Mounts

You may have seen the following output when running docker push before. This output means that the push is making use of cross repository mounts, which is a cool feature to cache layers across multiple images.

e1c75a5e0bfa: Mounted from library/ubuntu

This feature was introduced in Docker Registry v2.3.0. Cross repository mounts allow clients to inform the registry that they know about another image in the registry that may share the same layer, and that the registry should try using the layer from that image rather than going through the full upload process.

When Docker receives this request, it first makes sure that the client has pull access to this other repository. If the client has access, and the layers match up, the registry sends back a 201 Created response. Otherwise, it sends a 202 Accepted response, and the client goes through the full upload process described above.

Optimizing Blimp

If you use a custom Docker image for development, Blimp will automatically build and push the image when you start up your sandbox. The image for each service is pushed to blimp-registry.kelda.io/<sandboxID>/<service>:<imageID>, where is a unique identifier for your sandbox, and is a hash to make sure we always run the latest version of your image.

As a reminder, our goal for looking into all this is to make it so that when you push this image, you only have to push the "unique" layers that can't be pulled from more efficient sources.

Initial Design

At first, we wanted to make use of cross repository mounts. This would let all our users share the same base images, so we would only have to push the base image for the very first user that references it. Plus, it'd set us up to build private image caches for teams so that they could share layers from their Dockerfile other than the base image.

We were hoping to do something like this:

Analyze the image's Dockerfile to find out what its base image is.
Send a request to our server to push this base image to the registry with the name blimp-registry.kelda.io/public/<image>:<tag>.
Tag the base image locally with blimp-registry.kelda.io/public/<image>:<tag> so that Docker would provide it as a cross repository mount.
Push the image with docker push.

Unfortunately, step 3 didn't actually cause Docker to provide the pre pushed base image as a cross repository mount. Docker only updates its list of images used for cross repository mounts on the first time a layer is pushed or pulled.

We considered giving users push access to the public repo, but we deemed that too insecure. We also considered ditching docker push entirely in favor of go-containerregistry, but that would have entailed making a significant change to go-containerregistry in order to show image push updates.

So, we went back to the drawing board.

Revised Design

After giving up on cross repository mounts, we asked: why bother with cross repository mounts when we could just push directly to the user's repository?

Although our servers would have to push a copy of the base image for each user, this is still much more efficient than having the user push it directly from their laptop since the bandwidth between our servers and the registry is so much higher.

Ultimately, that's what we settled on. The repository for each service (blimp-registry.kelda.io/<sandboxID>/<service>) always has a base tag that our servers push the base image to. The registry then automatically references it during the normal push API outlined above whenever the user pushes their image -- no icky manipulation of Docker's state necessary.

Putting it all together, this is what happens when Blimp pushes a locally built image:

The Blimp CLI parses the reference to the base image from the image's Dockerfile.
The Blimp CLI tells the Blimp servers to push the base image to blimp-registry.kelda.io/<sandboxID>/<service>:base.
The Blimp CLI builds the image, using the same base image.
The Blimp CLI pushes the full image to blimp-registry.kelda.io/<sandboxID>/<service>:<imageID>.
Docker goes through the layers one by one, and pushes them. If the layer is from the base image, the registry notices and instructs the CLI to skip the push.
For the layers not in the base image, Docker does the full upload process.

Conclusion

At Blimp, we want to make moving your development environment to the cloud as seamless as possible. One of our design principles is that the move should use the exact same config, and not require any changes to your workflow. Although we could have users work around the push slowness by prebuilding and pushing images to a shared public repository, that would violate our design goals. Building this feature was a fun deep dive into Docker internals, and a big step towards making the onboarding process to Blimp seamless.

References

See how fast it is yourself! Try an example

Read more about Docker internals -- see how registry credentials are stored.

Read the spec for the Docker Push API

By: Christopher Cooper

Using Data Containers To Boot Your Development Environment In Seconds

Ethan J. Jackson — Thu, 09 Jul 2020 14:21:06 +0000

One of the most time consuming parts of booting a Docker development environment is initializing databases. The data container pattern gets around this obstacle by taking advantage of some less known features of volumes. With data containers, you can easily distribute, maintain, and load your database's seed data.

Data containers are a commonly overlooked tool for building the nirvana of development environments: booting the environment with a single command that works every time.

You're probably already using volumes to save you some time working with databases during development; if one of your development containers crashes, volumes will prevent you from losing the database's state. But interestingly, Docker volumes have some cool quirks that we can leverage for the data containers pattern.

In this post, I'll:

Explain why data containers are the best way to initialize your databases.
Explain how data containers work by taking advantage of some unusual behavior that isn't present in other volume implementations, such as Kubernetes volumes.
Walk you through a brief example on how to do it.

Just want the code?
Get it here and boot it with docker-compose up (or blimp up!)

Standard Techniques for Initializing Databases

When developing using Docker, there are three approaches developers commonly use for setting up their databases. All of them have serious drawbacks.

1) Initialize Your Database By Hand

Most people start by setting up their databases by hand. But this has several serious drawbacks:

This approach can be very time-consuming. For example, it's easy to spend an entire day copying the data you need from staging and figuring out how to seed your database with it. And, if you lose your volume, you have to do it all over again.
It's hard to sustain over time. I once worked with a team that dreaded destroying their database since they knew they'd have to re-initialize it later. As a result, they would do their best to avoid avoiding working on certain projects so they could spare themselves the pain of destroying and initializing their database.

2) Initialize Your Database Using a Script

Using a script can save you a lot of manual work. But it comes with its own set of headaches:

The script may take a while to run, slowing down the environment boot time.
In the rush of all the other work developers have to do, it's easy to put off maintaining the script. As the database's schema changes over time, the script breaks, and then you have to spend time debugging it.

3) Use a Remote Database

Using a remote database -- typically your staging database -- is certainly faster than running scripts or initializing your database by hand. But there's a big downside: you're sharing the database with other developers. That means you don't have a stable development environment. All it takes is one developer mucking up the data to ruin your day.

A Better Way: Data Containers

Data containers are containers that store your database's state, and are deployed like any other container in your Docker Compose file. They take advantage of some quirks of Docker volumes to copy the data from the container into the database, so that the database is fully initialized when it starts.

To see how volumes can speed up your development work with databases, let's take an example from the Magda data catalog system. Here's a snippet from the Magda Docker Compose file:

services:
  postgres:
    image: "gcr.io/magda-221800/magda-postgres:0.0.50-2"
    volumes:
      - 'db-data:/data'
    environment:
      - "PGDATA=/data"

  postgres-data:
    image: "gcr.io/magda-221800/magda-postgres-data:0.0.50-2"
    entrypoint: "tail -f /dev/null"
    volumes:
      - 'db-data:/data'

volumes:
  db-data:

When you run docker-compose up in the Magda repo, all the Magda services start, and the Postgres database is automatically initialized.

How It Works

This setup takes advantages of two features of Docker volumes:

1) Docker copies any files masked by volumes into the volume. The Magda example has the following in its Docker Compose file.

  postgres-data:
    image: "gcr.io/magda-221800/magda-postgres-data:0.0.50-2"
    entrypoint: "tail -f /dev/null"
    volumes:
      - 'db-data:/data'

When postgres-data starts, it mounts a volume to /data. Because we built the gcr.io/magda-221800/magda-postgres-data image to already have database files at /data, Docker copies those files into the volume.

2) Volumes can be shared between containers. So any files written to db-data by postgres-data are visible in the postgres container because the postgres container also mounts the db-data volume:

  postgres:
    image: "gcr.io/magda-221800/magda-postgres:0.0.50-2"
    environment:
      - "PGDATA=/data"
    volumes:
      - 'db-data:/data'

Putting this all together, when you run docker-compose up, the following happens:

Docker copies /data from postgres-data to the db-data volume
Docker starts the postgres container.
The postgres container starts, and boots with the data in its data directory.

In short, instead of having to spend a lot of time repeatedly initializing your databases by hand or creating and maintaining scripts, with remarkably little work you are good to go. You've got a fully automated system in place that will work every time.

Benefits

This approach has 3 major benefits:

It's a huge timesaver for developers -- booting is now super quick, and developers don't have to manually add data or create and maintain scripts.
It ensures that everyone on your team is working with an identical set of data. To get the newest version of the data, all you need to do is docker pull the data image, just like any other container. In fact, docker-compose will do that for them, so they don’t even have to think about it when onboarding.
It's easy to automate. There's a lot of existing tooling for automating Docker builds, which you can take advantage of with this approach.

Downsides

The main downside of this approach is that it can be hard to maintain the data container. Maintaining it by hand has the same downsides as initializing the database manually or with scripts -- the data can get stale as the db schema changes.

Teams that use this approach tend to generate their data containers using CI. The CI job snapshots and sanitizes the data from production or staging, and pushes it to the Docker registry. This way, the container generation is fully automated, and developers don't have to worry about it.

Conclusion

Data containers are a cool example of how Docker Compose does so much more than just boot up containers. Used properly, it can substantially increase developer productivity.

We're excited to share these developer productivity tips because we've noticed that the development workflow has become an afterthought during the move to containers. The complexity of modern applications requires new development workflows. We built Blimp so that development teams can quickly build and test containerized software, without having to reinvent a development environment approach.

Resources

Check out another trick for increasing developer productivity by using host volumes get rid of container rebuilds.

Try an example with Blimp to see how easily development on Docker Compose can be scaled into the cloud.

Read common Docker Compose mistakes for more tips on how to make development faster.

How to Develop Your Python Docker Applications Faster

Ethan J. Jackson — Thu, 02 Jul 2020 00:49:16 +0000

Docker has many benefits that make deploying applications easier. But the process of developing Python with Docker can be frustratingly slow. That's because testing your Python code in Docker is a real pain.

Luckily, there's a technique you can use to reduce time you spend testing. In this tutorial, we'll show you how to use Docker's host volumes and runserver to make developing Python Docker applications easier and faster.

(If you're a Node.JS developer, see How to Develop Your Node.Js Docker Applications Faster.)

How Host Volumes and Runserver Can Speed Up Your Python Development

As every Python developer knows, the best way to develop your application is to iterate through short, quick cycles of coding and testing. But if you're developing using Docker, every time you change your code, you're stuck waiting for the container to rebuild before you can test.

As a result, you end up with a development workflow that looks like this:

You make a change.
You wait for the container to rebuild.
You make another change.
You wait some more.

And if your team uses CI/CD, so you're constantly running your code through automated tests? You're going to be spending even more time waiting for the container to rebuild.

Coding and waiting and coding and waiting is not a recipe for developer productivity -- or developer happiness.

But there's a way to modify a container's code without having to rebuild it. The trick is to use a Docker host volume.

Host volumes sync file changes between a local host folder and a container folder. If you use a host volume to mount the code you're working on into a container, any edits you make to your code on your laptop will automatically appear in the container. And as you will see in the next section, you can use the runserver package to automatically restart your application without having to rebuild the container -- a technique known as "live reloading."

The result: instead of wasting lots of time waiting for your containers to rebuild, your code-test-debug loop is almost instantaneous.

Example: Using Host Volumes and Runserver in Python Docker Development

The idea of using a host volume to speed up your Python coding might seem a little daunting, but it's pretty straightforward.

To demonstrate this, let's use a Python example: django-polls, a basic poll app that’s part of Django’s introductory tutorial. To clone the repo:

$git clone https://github.com/kelda/django-polls

The repo assumes you are using Docker Compose. You can also use
Blimp, our Compose alternative that scales to the cloud.

Here's the docker-compose.yml file for django-polls:

version: '3'
services:
  web:
    build: .
    command:
      - sh
      - -c
      - "./wait-for-postgres.sh && python manage.py migrate ; python manage.py shell < init-db.py ; python manage.py runserver 0.0.0.0:8000"
    ports:
      - "8000:8000"
    depends_on:
      - db
    volumes:
      - ".:/code"
  db:
    image: "postgres:12"
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_USER=polls
      - POSTGRES_PASSWORD=polls
      - POSTGRES_DB=polls

This file tells Docker to boot a container, the Django application, and a Postgres database where the application stores the poll. It also tells Docker to mount a host volume:

volumes:
    - ".:/code"

As a result, Docker will mount the ./ directory on your laptop, which contains the code you're developing, into the container at /code.

Next, you need to set up your Docker container so that whenever you edit your code, Docker automatically restarts your Python application. That way, your application will always use the latest version of your code.

If you are creating a Django app, the easiest way to do that is to have your .yml file tell Docker to use runserver, Django's development web server:

  web:
    build: .
    command:
      - sh
      - -c
      - "./wait-for-postgres.sh && python manage.py migrate ; python manage.py shell < init-db.py ; python manage.py runserver 0.0.0.0:8000"

As a result, whenever you modify your code on your laptop, runserver restarts the process without rebuilding the container.

In short, by using a host volume and runserver, you can set up your Python application's container so it automatically syncs code changes between the container and your laptop. If you didn't do this, you'd have to rebuild the container every single time you made a change to your code.

Over time, this technique can substantially speed up your Python development. For example, we've heard from users that it's not uncommon for container rebuilds to take 5-30 minutes. With host volumes and runserver, your code sync is almost instantaneous. Imagine what your day would look like if you could save yourself 5-30 minutes every time you modify and test your code.

Syncing Your Own Code When Developing a Python Application

Now that you've seen how to use this technique in a sample application, the rest of this tutorial will show you how to enable code syncing in one of your existing Python projects.

Prerequisites

Just like the example above, your Python project should include the following:

A git repo that contains your code
A Dockerfile that builds that code into a working container
A docker-compose.yml file you use to run that container

How to Configure Your Container to Automatically Sync Your Python Code

1) Locate the folder in your Docker container that has your code. The easiest way to figure out where your code is stored in your container is to look at your Dockerfile's COPYcommands. In the django-polls example, you can see from the Dockerfile that the container expects the code to be in /code:

FROM python:3
RUN apt update && apt install -y netcat && rm -rf /var/lib/apt/lists/*
ENV PYTHONUNBUFFERED 1
RUN mkdir /code
WORKDIR /code
COPY requirements.txt /code/
RUN pip install --no-cache-dir -r requirements.txt

2) Find the path to the folder on your laptop that has the same Python code.

3) Add a host volume to your docker-compose file. Find the container in your docker-compose file that you want to sync code with, and add a volume instruction underneath that container:

volumes:
  "/path-to-laptop-folder:/path-to-container-folder"

4) Make sure your Docker Compose file configures your container for live reloading. In the django-poll example, you implemented it by using runserver as your web server:

  web:
    build: .
    command:
      - sh
      - -c
      - "./wait-for-postgres.sh && python manage.py migrate ; python manage.py shell < init-db.py ; python manage.py runserver 0.0.0.0:8000"

5) Run Docker Compose or Blimp. Now all you need to do is either run docker-compose:

$ docker-compose up

Or if you're using Blimp:

$ blimp up

As a result, Docker will update the container's code with the code that's on your laptop.

Now that your container is set up to use a host volume and runserver, whenever you modify the Python code on your laptop, your new code will automatically appear in the container.

Conclusion

At first, the idea of using host volumes to sync the Python code on your laptop with your container might seem a little weird. But once you get used to this workflow, you'll see how much more efficient it is. With just a few tweaks to your Docker containers' set up, developing your Python Docker app is easier and faster.

Resources

Try the Python example on Blimp

Read How to Develop Your Node.Js Docker Applications Faster

Read about common mistakes with host
volumes
that can slow down your application

Check out Blimp, our team's project to improve developer productivity for Docker Compose.

5 Common Mistakes When Writing Docker Compose

Ethan J. Jackson — Wed, 24 Jun 2020 18:50:12 +0000

When building a containerized application, developers need a way to boot containers they're working on to test their code. While there are several ways to do this, Docker Compose is one of the most popular options. It makes it easy to:

Specify what containers to boot during development
And setup a fast code-test-debug development loop

The vision is that someone writes a docker-compose.yml that specifies everything that's needed in development and commits it to their repo. Then, every developer simply runs docker-compose up, which boots all the containers they need to test their code.

However, it takes a lot of work to get your docker-compose setup to peak performance. We've seen the best teams booting their development environments in less than a minute and testing each change in seconds.

Given how much time every developer spends testing their code every day, small improvements can add up to a massive impact on developer productivity.

(Credit XKCD)

Mistake 1: Frequent Container Rebuilds

docker build takes a long time. If you're rebuilding your container every time you want to test a code change, you have a huge opportunity to speed up your development loop.

The traditional workflow for working on non-containerized applications looks like this:

Code
Build
Run

This process has been highly optimized over the years, with tricks like incremental builds for compiled languages and hot reloading. It's gotten pretty fast.

When people first adopt containers, they tend to take their existing workflow and just add a docker build step. Their workflow ends up like this:

Code
Build
Docker Build
Run

If not done well, that docker build step tosses all those optimizations out the window. Plus, it adds a bunch of additional time-consuming work like reinstalling dependencies with apt-get. All of this adds up into a much slower test process than we had before Docker.

Solution: Run your code outside of Docker

One approach is to boot all your dependencies in Docker Compose, but run the code you're actively working on locally. This mimics the workflow for developing non-containerized applications.

Just expose your dependencies over localhost and point the service you're working on at the localhost:<port> addresses.

However, this is not always practical, particularly if the code you're working on depends on things built into the container image that aren't easy to access from your laptop.

Solution: Maximize caching to optimize your Dockerfile

If you must build Docker images, writing your Dockerfiles so that they maximize caching can turn a 10 minute Docker build into 1 minute.

A typical pattern for production Dockerfiles is to reduce the number of layers by chaining single commands into one RUN statement. However, image size doesn't matter in development. In development, you want the most layers possible.

Your production Dockerfile might look like this:

RUN \
    go get -d -v \
    && go install -v \
    && go build

This is terrible for development because every time that command is re-run, Docker will re-download all of your dependencies and reinstall them. An incremental build is more efficient.

Instead, you should have a dedicated Dockerfile specifically for development. Break everything into tiny little steps, and plan your Dockerfile so that the steps based on code that changes frequently come last.

The stuff that changes least frequently, like pulling dependencies, should go first. This way, you don't have to build the entire project when rebuilding your Dockerfile. You just have to build the tiny last piece you just changed.

For an example of this, see below the Dockerfile we use for
Blimp development. It follows the techniques described above to shrink a heavy build process down to a couple of seconds.

FROM golang:1.13-alpine as builder

RUN apk add busybox-static

WORKDIR /go/src/github.com/kelda-inc/blimp

ADD ./go.mod ./go.mod
ADD ./go.sum ./go.sum
ADD ./pkg ./pkg

ARG COMPILE_FLAGS

RUN CGO_ENABLED=0 go install -i -ldflags "${COMPILE_FLAGS}" ./pkg/...

ADD ./login-proxy ./login-proxy
RUN CGO_ENABLED=0 go install -i -ldflags "${COMPILE_FLAGS}" ./login-proxy/...

ADD ./registry ./registry
RUN CGO_ENABLED=0 go install -i -ldflags "${COMPILE_FLAGS}" ./registry/...

ADD ./sandbox ./sandbox
RUN CGO_ENABLED=0 go install -i -ldflags "${COMPILE_FLAGS}" ./sandbox/...

ADD ./cluster-controller ./cluster-controller
RUN CGO_ENABLED=0 go install -i -ldflags "${COMPILE_FLAGS}" ./cluster-controller/...

RUN mkdir /gobin
RUN cp /go/bin/cluster-controller /gobin/blimp-cluster-controller
RUN cp /go/bin/syncthing /gobin/blimp-syncthing
RUN cp /go/bin/init /gobin/blimp-init
RUN cp /go/bin/sbctl /gobin/blimp-sbctl
RUN cp /go/bin/registry /gobin/blimp-auth
RUN cp /go/bin/vcp /gobin/blimp-vcp
RUN cp /go/bin/login-proxy /gobin/login-proxy

FROM alpine

COPY --from=builder /bin/busybox.static /bin/busybox.static
COPY --from=builder /gobin/* /bin/

One final note: with the recent introduction of multi-stage
builds, it's now possible to create Dockerfiles that both have good layering and small images sizes. We won't discuss this in much detail in this post, other than to say that the Dockerfile shown above does just that and as a result is used both for Blimp development as well as production.

Solution: Use host volumes

In general, the best option is to use a host volume to directly mount your code into the container. This gives you the speed of running your code natively, while still running in the Docker container containing its runtime dependencies.

Host volumes mirror a directory on your laptop into a running container. When you edit a file in your text editor, the change is automatically synced into the container and then can be immediately executed within the container.

Most languages have a way to watch your code, and automatically re-run when it changes. For example, nodemon is the
go to for Javascript. Check out this post for a tutorial on how to set this up.

It takes some work initially, but the result is that you can see the results of your code changes in 1-2 seconds, versus a Docker build which can take minutes.

Mistake 2: Slow Host Volumes

If you're already using host volumes, you may have noticed that reading and writing files can be painfully slow on Windows and Mac. This is a known issue for commands that read and write lots of files, such as Node.js and PHP applications with complex dependencies.

This is because Docker runs in a VM on Windows and Mac.
When you do a host volume mount, it has to go through lots of translation to get the folder running on your laptop into the container, somewhat similar to a network file system. This adds a great deal of overhead, which isn't present when running Docker natively on Linux.

Solution: Relax strong consistency

One of the key problems is that file-system mounts by default maintain strong consistency. Consistency is a broad topic on which much ink has been spilt, but in short it means that all of a particular files reader's and writers agree on the order that any file modifications occurred, and thus agree on the contents of that file (eventually, sort of).

The problem is, enforcing strong consistency is quite expensive, requiring coordination between all of a files writers to guarantee they don't inappropriately clobber each other's changes.

While strong consistency can be particularly important when, for example, running a database in production. The good news is that in development, it's not required. Your code files are going to have a single writer (you), and a single source of truth (your repo). As a result, conflicts aren't as big a concern as they might be in production.

For just this reason, Docker implemented the ability to relax consistency guarantees when mounting volumes. In Docker Compose, you can simply add this cached keyword to your volume mounts to get a significant performance guarantee. (Don't do this in production ...)

volumes:
    - "./app:/usr/src/app/app:cached"

Solution: Code syncing

Another approach is to setup code syncing. Instead of mounting a volume, you can use a tool that notices changes between your laptop and the container and copies files to resolve the differences (similar to rsync).

The next version of Docker has Mutagen built in as an alternative to cached mode for volumes. If you're interested, just wait until Docker makes its next release and try that out, but you can also check out the
Mutagen project to use it without waiting. Blimp, our Docker Compose implementation, achieves something similar using Syncthing

Solution: Don't mount packages

With languages like Node, the bulk of file operations tend to be in the packages directory (like node_modules). As a result, excluding these directories from your volumes can cause a significant performance boost.

In the example below, we have a volume mounting our code into a container. And then overwrite just the node_modules directory with its own clean dedicated volume.

volumes:
  - ".:/usr/src/app"
  - "/usr/src/app/node_modules"

This additional volume mount tells Docker to use a standard volume for the node_modules directory so that when npm install runs it doesn't use the slow host mount. To make this work, when the container first boots up we do npm install in the entrypoint to install our dependencies and populate the node_modules directory. Something like this:

entrypoint:
  - "sh"
  - "-c"
  - "npm install && ./node_modules/.bin/nodemon server.js"

Full instructions to clone and run the above example can be found here.

Mistake 3: Brittle Configuration

Most Docker Compose files evolve organically. We typically see tons of copy and pasted code, which makes it hard to make modifications. A clean Docker Compose file makes it easier to make regular updates as production changes.

Solution: Use env files

Env files separate environment variables from the main Docker Compose configuration. This is helpful for:

Keeping secrets out of the git history
Making it easy to have slightly different settings per developer. For example, each developer may have a unique access key. Saving the configuration in a .env file means that they don't have to modify the committed docker-compose.yml file, and deal with conflicts as the file is updated.

To use env files, just add a .env file, or set the path explicitly with the env_file field.

Solution: Use override files

Override files let you have a base configuration, and then specify the modifications in a different file. This can be really powerful if you use Docker Swarm, and have a production YAML file. You can store your production configuration in docker-compose.yml, then specify any modifications needed for development, such as using host volumes, in an override file.

Solution: Use extends

If you're using Docker Compose v2, you can use the extends keyword to import snippets of YAML in multiple places. For example, you might have a definition that all services at your company will have these particular five configuration options in their Docker Compose file in development. You can define that once, and then use the extends keyword to drop that everywhere it's needed, which gives you some modularity. It's painful that we have to do this in YAML but it's the best we have short of writing a program to generate it.

Compose v3 removed support for the extends keyword. However, you can achieve a similar result with YAML anchors.

Solution: Programmatically generate Compose files

We've worked with some engineering teams using Blimp that have a hundred containers in their development Docker Compose file. If they were to use a single giant Docker Compose file it would require thousands
of lines of unmaintainable YAML.

As you scale, it's okay to write a script to generate Docker Compose files based on some higher-level specifications. This is common for engineering teams with really large development environments.

Mistake 4: Flaky Boots

Does docker-compose up only work half the time? Do you have to run
docker-compose restart to bring up crashed services?

Most developers want to write code, not do DevOps work. Debugging a broken development environment is super frustrating.

docker-compose up should just work, every single time.

Most of the issues here are related to services starting in the wrong order. For example, your web application may rely on a database, and will crash if the database isn't ready when the web application boots.

Solution: Use depends_on

depends_on lets you control startup order. By default, depends_on only waits until the dependency is created, and doesn't wait for the dependency to be "healthy". However, Docker Compose v2 supports combining depends_on with
healthchecks. (Unfortunately, this feature was removed in Docker Compose v3, instead you can manually implement something similar with a script like wait-for-it.sh)

The Docker documentation recommends against approaches like depends_on and wait-for-it.sh. And we agree, in production, requiring a specific boot order for your containers is a sign of a brittle architecture. However, as an individual developer trying to get your job done, fixing every single container in the entire engineering organization may not be feasible. So, for development, we think it's OK.

Mistake 5: Poor Resource Management

It can get tricky to make sure that Docker has the resources it needs to run smoothly, without completely overtaking your laptop. There's a couple things you can look into if you feel like your development workflow is sluggish because Docker isn't running at peak capacity.

Solution: Change Docker Desktop allocations

Docker Desktop needs a lot of RAM and CPU, particularly on Mac and Windows where it's a VM. The default Docker Desktop configuration tends not to allocate enough RAM and CPU, so we generally recommend tweaking the setting to over-allocate. I tend to allocate about 8GB of RAM and 4 CPUs to Docker when
I'm developing (and I turn Docker Desktop off when not in use to make that workable).

Solution: Prune unused resources

Frequently people will unintentionally leak resources when using Docker. It's not uncommon for folks to have hundreds of volumes, old container images, and sometimes running containers if they're not careful. That's why we recommend
occasionally running docker system prune which deletes all of the volumes, containers, and networks that aren't currently being used. That can free up a lot of resources.

Solution: Run in the cloud

Finally, in some cases even with the above tips, it may be impossible to efficiently run all of the containers you need on your laptop. If that's the case, check out Blimp, an easy way to run Docker Compose files in the cloud.

What should you do?

TDLR; To improve the developer experience on Docker Compose, I'd encourage you to

Minimize container rebuilds.
Use host volumes.
Strive for maintainable compose files, just like code.
Make your boots reliable.
Manage resources mindfully.

What a mysterious bug taught us about how Docker stores registry credentials

Ethan J. Jackson — Tue, 23 Jun 2020 14:44:29 +0000

We recently ran into a mysterious bug that required hours of digging into the
arcane details of Docker's registry credentials store to figure out. Although
in the end the fix turned out to be easy, we learned a thing or two along the
way about the design of the credentials store and how, if you're not careful,
it can be configured insecurely.

Blimp, sometimes needs to pull private images from a
Docker registry in order to boot those images in the cloud. This typically
works fine, but unfortunately, when some users started
Blimp, they were getting the following error message:

Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentials

At first, we were completely baffled by this cryptic message and had no clue it
was related to our handling of credentials. To understand how we figured it
out, first you need to know a little about how modern Docker credentials are
handled.

Docker's External Credentials Store

The recommended way to store your Docker credentials is in an external
credentials store. In your Docker config file, which is usually located at
~/.docker/config.json, there are two fields you can use to configure how
Docker gets and stores credentials: credsStore and credHelpers.

credsStore tells Docker which helper program to use to interact with the
credentials store. All helper programs have names that begin with
docker-credential- -- the value of credsStore is the suffix of the helper
program.

For example, if you work on a Mac laptop, you might decide to use the Mac OS
keychain. The name of the helper program to use the keychain is
docker-credential-osxkeychain. So your config.json would include the
following:

{
  "credsStore": "osxkeychain"
}

If you want to see what credentials Docker currently has for you, you can use list. For example:

docker-credential-osxkeychain list

The result is a list of pairs of servers and usernames. For example:

{
  "http://quay.io":"kklin",
  "https://index.docker.io":"kevinklin"
}

You may also notice credHelpers in your config.json. These helpers are similar
to credsStore, but are used to generate short lived credentials. For example,
if you use gcr, gcloud installs a credHelper that uses
your Google login to get tokens. This way, Docker never has your Google
credentials directly -- the docker-credential-gcloud acts as a middleman
between Docker and your Google credentials.

Once again, here's the error message our users were getting:

Get https://1234.dkr.ecr.us-east-1.amazonaws.com/v2/blimp/blimp/manifests/v0.1: no basic auth credentials

We were able to run the docker-credential-osxkeychain list and get
commands to see the credentials for 1234.dkr.ecr.us-east-1.amazonaws.com, so
why were we getting an error that there weren't any credentials??

In the Beginning: Docker Stores Your Registry Password In Your Config File

It turns out that external credentials stores weren't
added to Docker until version 1.11,
in 2016. Before 1.11, Docker stored credentials via a config field called
auths. This field is stored in the same file as the credStore: ~/.docker/config.json.

Whenever you logged into a registry, Docker would set the value of auths to
your password. For
example,
your config file might contain the following:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
        },
        "localhost:5001": {
            "auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
        }
    }
}

What we learned the hard way is that there's a quirk with Docker's login
command. When you log in using docker login, Docker adds an entry via the
credsStore and in auths, using slightly different server names. Your
credentials are properly stored in the credentials store, but the entry in
auths doesn't contain the username or password. The result looks something
like this:

{
"auths": {
  "https://index.docker.io/v1/": {}
}

The problem is that Blimp grabs credentials from both auths and
credsStore. So it was passing two copies of the credentials to the Docker
image puller -- one with the correct username and password, and one without the
password at all.

Unfortunately, Docker preferred the https:// version of the credential, and
attempt to pull the image with the empty credential. Thus, the no basic auth credentials error.

Once we figured out that the problem was that an empty duplicate entry was
getting added to the insecure store, it was easy to fix the
problem. All we
needed to do was add an if statement to skip empty credentials:

    addCredentials := func(authConfigs map[string]clitypes.AuthConfig) {
        for host, cred := range authConfigs {
            // Don't add empty config sections.
            if cred.Username != "" ||
                cred.Password != "" ||
                cred.Auth != "" ||
                cred.Email != "" ||
                cred.IdentityToken != "" ||
                cred.RegistryToken != "" {
                creds[host] = types.AuthConfig{
                    Username:      cred.Username,
                    Password:      cred.Password,
                    Auth:          cred.Auth,
                    Email:         cred.Email,
                    ServerAddress: cred.ServerAddress,
                    IdentityToken: cred.IdentityToken,
                    RegistryToken: cred.RegistryToken,
                }
            }
        }
    }

A Potential Docker Credentials Security Risk

In the process of uncovering this bug, we noticed a potential security risk
that you may not be aware of. As we learned, it's best practice to use an
external store to store your external registry credentials. However, depending
on how and when you installed Docker it's possible you could still be using the
legacy auths method. If you are, your ~/.docker/config.json might look
something like this:

{
    "auths": {
        "https://index.docker.io/v1/": {
            "auth": "YW11cmRhY2E6c3VwZXJzZWNyZXRwYXNzd29yZA=="
        },
        "localhost:5001": {
            "auth": "aGVzdHVzZXI6dGVzdHBhc3N3b3Jk"
        }
    }
}

This may look reasonable secure, the passwords appear to be a garbled bunch of
gibberish. Surely those passwords are encrypted, right?

Guess again. All Docker did was encode the passwords using base64. And as David
Rieger pointed out on Hacker
Noon,
base64

may look like encryption on first glance, but it's not. Base64 is a scheme
for encoding, not encryption. You can simply copy the base64 string and
convert it to ASCII in a matter of seconds.

That seemingly secure password of aGVzdHVzZXI6dGVzdHBhc3N3b3Jk? All you need to do to read the password is base64 decode it:

$ echo aGVzdHVzZXI6dGVzdHBhc3N3b3Jk| base64 -D
hestuser:testpassword

The Moral of Our Story: Double Check Your Docker Credentials' Security

So that's the bad news: if Docker config file isn't properly set up, Docker is
storing your credentials password in plain text.

The good news is that it's easy to fix the problem.

All you and your team members need to do is take a quick look at
~/.docker/config.json. If it contains an auths password, get rid of it and
switch over to using a credentials store. To do so, just download the
appropriate docker-credential- helper for your system, and update the
credsHelper field in ~/.docker/config.json.

Hope that helps!

Read the Top 5 common mistakes when writing Docker Compose

How to Develop Your Node.Js Docker Applications Faster

Ethan J. Jackson — Wed, 10 Jun 2020 00:35:06 +0000

Docker has revolutionized how Node.js developers create and deploy applications. But developing a Node.js Docker application can be slow and clunky. The main culprit: the process for testing your code in development.

In this article, we'll show a tutorial and example on how you can use Docker's host volumes and nodemon to code faster and radically reduce the time you spend testing.

How Host Volumes and Nodemon Can Speed Up Your Node.js Development

One of the irritating things about testing during development with Docker is that whenever you change your code, you have to wait for the container to rebuild. With many Node.js applications, this can chew up a lot of time.

As a result, you end up with a development workflow that looks like this:

You make a change.
You wait for the container to rebuild.
You make another change.
You wait some more.

And if you have CI/CD and are continually running your code through automated tests? You're going to be spending even more time waiting for the container to rebuild.

This process gets pretty tedious. And it's hard to stay in the flow.

But there's a way to change a container's code without having to rebuild it. The trick is to use a Docker host volume.

Host volumes sync file changes between a local host folder and a container folder. If you use a host volume to mount the code you're working on into a container, any edits you make to your code on your laptop will automatically appear in the container. And as you will see in the next section, you can use the nodemon package to automatically restart your application without having to rebuild the container -- a technique known as "live reloading."

The result: instead of having to spend lots of time waiting, your code-test-debug loop is almost instantaneous.

Example: Using Host Volumes and Nodemon in Node.Js Docker Development

The idea of using a host volume to speed up your Node.js coding might seem a little intimidating. But it's pretty straightforward to do.

To demonstrate this, let's use a Node.js example:
Node-todo, a simple to-do application
created by scotch.io. To clone the repo:

$git clone https://github.com/kelda/node-todo

The repo assumes you are using Docker Compose. You can also use
Blimp, our alternative to Compose that runs in the cloud.

Here's Node-todo's docker-compose.yml:

version: '3'
services:
  web:
    build: .
    ports:
      - "8080:8080"
    depends_on:
      - mongo
    volumes:
      - "./app:/usr/src/app/app"
  mongo:
    image: "mongo"
    ports:
      - "27017:27017"

This file tells Docker to boot a container, the Node.js application, and a MongoDB database where the application stores the to-dos. It also tells Docker to mount a host volume:

volumes:
  - "./app:/usr/src/app/app"

As a result, Docker will mount the ./app directory on your laptop, which contains your code, into the container at /usr/src/app/app.

Now, all you need to do is ensure that whenever you've edited your code, your Node.js application restarts so it's using your latest code. That's where nodemon comes in.

nodemon is a Node.js package that automatically restarts an application when it detects file changes in one or more specified directories. Once you've changed your code on your laptop/desktop, nodemon detects that change and restarts the process without rebuilding the container.

To make this happen, you need to tell Docker to set the entrypoint to nodemon instead of node.js. You do that in the Dockerfile:

FROM node:10-alpine
ENV PORT 8080
WORKDIR /usr/src/app
COPY . /usr/src/app

RUN npm install -g nodemon
RUN npm install

ENTRYPOINT ["nodemon", "/usr/src/app/server.js"]

In short, by using a host volume and nodemon, you can set up your Node.js application's container so it automatically syncs code changes between the container and your laptop. If you didn't do this, you'd have to rebuild the container every single time you made a change to your code.

Over time, this technique can substantially speed up your Node.js development. For example, we've heard from users that it's not uncommon for container rebuilds to take 5-30 minutes. With host volumes and nodemon, your code sync is almost instantaneous. Imagine what your day would look like if you could save yourself 5-30 minutes every time you change and test your code.

Syncing Your Own Code When Developing a Node.js Application

Now that you've seen how it works in a sample application, let's walk through how to enable code syncing in one of your existing Node.js projects.

Prerequisites

Just like the example above, before you get started, we recommend your Node.js project includes the following:

A git repo that contains your code
A Dockerfile that builds that code into a working container
A docker-compose.yml file you use to run that container

How to Configure Your Container to Automatically Sync Your Node.js Code

1) Locate the folder in your Docker container that has your code. The easiest way to figure out where your code is stored in your container is to look at your Dockerfile's COPYcommands. In the Node-todo example, its Dockerfile tells Docker to put the code in . /usr/src/app:

COPY . /usr/src/app

2) Find the path to the folder on your laptop that has the same Node.js code.

3) Add a host volume to your docker-compose file. Find the container in your docker-compose file that you want to sync code with, and add a volume instruction underneath that container:

volumes:
  "/path-to-laptop-folder:/path-to-container-folder"

4) Switch from using node.js to nodemon. In the Node-todo example, you implemented it via Dockerfile commands:

RUN npm install -g nodemon
RUN npm install

ENTRYPOINT ["nodemon", "/usr/src/app/server.js"]

As a result, Docker will install nodemon with npm install -g nodemon and change the entrypoint from
node to nodemon.

5) Run Docker Compose or Blimp. Now all you need to do is either run docker-compose:

$ docker-compose up

Or if you're using Blimp:

$ blimp up

Docker will overwrite the container's code with the code that's on your laptop.

Now that you've modified your project so it uses a host volume and nodemon, any changes you make to your Node.js code on your laptop will now automatically appear in the container.

Conclusion

Using host volumes to link your Node.js code on your laptop with your container can take a little getting used to. But it'll make developing your Node.js Docker apps easier and faster.

Originally posted on: https://kelda.io/blog/develop-nodejs-docker-applications-faster/

10 DockerCon 2020 Talks for 5/28

Ethan J. Jackson — Wed, 27 May 2020 14:37:23 +0000

DockerCon Live 2020 is a free event online on 5/28! We're really excited to learn more about the container ecosystem.

In addition to Kelda Co-founder Ethan's talk about Docker Compose in the Cloud with Blimp (at 2:00 PDT/5:00 EDT), we handpicked 10 talks we're interested in watching because of their practical knowledge!

1. Docker Desktop + WSL 2 Integration Deep Dive

10:30am-11:00am PDT

Simon Ferquel, Senior Software Developer, Docker

Have you ever wondered how Docker Desktop on Windows works with WSL 2 to provide a better developer experience? This talk will dive deep into the Docker Desktop and WSL architectures and show how they fit together.

2. Best Practices for Compose-managed Python Applications

11:00am-11:30am PDT

Anca Lordache, Engineer, Docker

It can be tricky to get your multi-tier Python application up and running deterministically for development. Managing the versions, dependencies and configuration takes up time that you could be using to code. Containers and Docker Compose solve this and give you a deterministic development environment that's quick to get up and running and easy to move to production.

This talk will show you some best practices for Python projects with Docker Compose, including:

How to bootstrap your project
An example development workflow with debugging and automated testing
How to make your builds reproducible and optimized

All this should make your Python development experience quicker and better!

3. How To Build and Run Node Apps with Docker and Compose

11:00am-11:30am PDT

Kathleen Juell, Developer, Digital Ocean

Containers are an essential part of today's microservice ecosystem, as they allow developers and operators to maintain standards of reliability and reproducibility in fast-paced deployment scenarios. And while there are best practices that extend across stacks in containerized environments, there are also things that make each stack distinct, starting with the application image itself.

This talk will dive into some of these particularities, both at the image and service level, while also covering general best practices for building and running Node applications with database backends using Docker and Compose.

4. Become a Docker Power User With Microsoft Visual Studio Code

11:30am-12:00pm PDT

Brian Christner, Docker Captain & Co-Founder, 56K.Cloud, host of @thebytepodcast

In this session, we will unlock the full potential of using Microsoft Visual Studio Code (VS Code) and Docker Desktop to turn you into a Docker Power User. When we expand and utilize the VS Code Docker plugin, we can take our projects and Docker skills to the next level. In addition to using VS Code, we streamline our Docker Desktop development workflow with less context switching and built-in shortcuts. You will learn how to bootstrap new projects, quickly write Dockerfiles utilizing templates, build, run, and interact with containers all from VS Code.

5. Tinkertoys, Microservices, and Feature Management: How to Build for the Future

12:00pm-12:30pm PDT

Heidi Waterhouse, Principal Developer Advocate, LaunchDarkly

Lots of us aren’t developing tidy, discrete features that are easy to manage. How do you plan to move from a tangle of interconnected features to something that you can test and deploy each part of? How do you manage the combinatorial complexity of individual feature testing? Join us for an overview on the conceptual basis of designing for feature management. It sounds simple to say that we will build one feature at a time, give it an API interface and allow it to connect with other features and microservices. The implementation is anything but simple. This talk explores how you can start migrating your existing features and services to a more modular, testable, and resilient system. Since containers are not state-aware, how do you make changes to their presentation without needing to rebuild them entirely? With feature flags, your container can be stable and your presentation dynamic. How can you test a distributed architecture on your laptop? How can you simulate partial outages? This talk is going to touch on some of the best practices that you can use to bring new life to your brown fields.

6. New Docker Desktop Filesharing Features

1:00pm-1:30pm PDT

Dave Scott, Technical Staff, Docker

Developers need fast edit-compile-test cycles to maximise their productivity. Source code is often edited in an IDE on the Mac or Windows host and shared directly with containers where it can be executed. Since the containers are running in helper VMs, the files must be accessed remotely or copied, which can lead to performance problems, lengthening the edit-compile-test cycle and lowering developer productivity. In this talk I'll describe recent changes to Docker Desktop to make file sharing faster and more reliable. We have a completely new implementation on Windows which replaces CIFS / SMB and on Mac we have an integration of "mutagen" which performs automatic two-way synchronisation of source code and build artefacts. This talk will contain a deep dive into these new features and demonstrate how to use them effectively.

7. Simplify All the Things with Docker Compose

1:30pm-2:00pm PDT

Michael Irwin, Application Architect, Virginia Tech

As you probably know by now, containers have revolutionized the software industry. But, once you have a container, then what? How do you run it? How do you help someone else run it? There are so many flags and options to remember, ports to configure, volume mappings to remember, and don't even get me started with networking containers together! While it's possible to do all of this through the command line, don't do it that way! With Docker Compose, you can create an easily shareable file that makes all of this a piece of cake. And once you fully adopt containers in your dev environment, it lets you setup your code repos to allow the simplest dev onboarding experience imaginable: 1) git clone; 2) docker-compose up; 3) write code. In this talk, we'll talk about several tips to help make all of this a reality. We'll start with a few Docker Compose basics, but then quickly move into several advanced topics. We'll even talk about how to use the same Dockerfile for dev and prod (we've all been there by having two separate files)! As an added bonus, we'll look at how to use Docker Compose in our CI/CD pipelines to perform automated tests of the container images built earlier in the pipeline! We'll have a few slides (because we have to explain a few things), lots of live demos (show it in action!), and maybe a few other surprises as well! Let's have some fun and help simplify all the things with Docker Compose!

8. Dev and Test Agility for Your Database With Docker

2:00pm-2:30pm PDT

Julie Lerman, Software Coach, The Data Farm

Agile practices teach us how to deal with evolving applications but so often the data store is overlooked as a component of your application lifecycle. Database servers are monolothic, resource intensive and mostly viewed as set in stone. Encapsulating your database server in a container and your database in a storage container can dramatically lighten the load and make your database as agile as your model and other processes. And you can even use a serious enterprise class database like SQL Server this way. This session will show how to benefit from using a containerized version of SQL Server for Linux during development and testing. We'll also address concerns about data that needs to be persisted. You'll also get a peek at the DevOps side of this, including using images in your CI/CD process.

9. How to Use Mirroring and Caching to Optimize your Container Registry

3:30pm-4:00pm PDT

Brandon Mitchell, Docker Captain and DevOps Solutions Architect, BoxBoat

How do you make your builds more performant? This talk looks at options to configure caching and mirroring of images that you need to save on bandwidth costs and to keep running even if something goes down upstream.

10. Your Container has Vulnerabilities. Now What?

3:30pm-4:00pm PDT

Jim Armstrong, Product Marketing Director Container Security, Snyk

Originally posted at: https://kelda.io/blog/dockercon-2020-talks/

Docker Experience

Ethan J. Jackson — Mon, 04 May 2020 18:57:34 +0000

I'm working on a project to reduce Docker’s resource usage.

Please fill out this 5-min survey to help me improve the Docker development experience!

I'll be raffling a $150 Amazon gift card as thanks.

https://forms.gle/2ggShw236EjBdtxn7