We've all heard the jokes from practically anyone with their development environments on Docker for Mac: it makes your Mac sound like a jet plane at takeoff.
However, their counterpart developers on Linux simply go 🤷🏻♂️ and develop with no such problems.
Why is Docker for Mac a quantifiably worse experience than running Docker on a Linux machine? We'll explore the reasons in this post.
Containers vs. Virtual Machines
First, a word on container architecture and how it differs from your standard Virtual Machine (VM).
Generically speaking, both are similar in that you're running "computers inside your computer". The difference comes in how this occurs.
(Source: https://wiki.aquasec.com/display/containers/Docker+Architecture)
Container Disk Changes
As you can see above, Containers make use of your Host OS and its kernel, and therefore are "closer to the iron". For example, in order for a Container to read/write from your Host OS hard drive, it has to:
- Mount the disk on the Container natively (i.e. it has direct access to the disk on the Host OS thanks to the kernel)
- Work as if you were directly on the Host OS
VM Disk Changes
VM's run an additional operating system on top of your Host Operating System, as well as an additional abstraction layer (called the Hypervisor) for the "Guest OS" to talk to the Host OS. For example, in order for a VM to read/write from your Host OS hard drive, it has to:
- Mount the disk on the Guest OS
- Mount the disk from the Host OS on the Hypervisor
- Have the Hypervisor synchronize changes between both
Docker for macOS
Now, while it may be called Docker for macOS, it is architecturally different than Docker on Linux.
(Source: https://collabnix.com/how-docker-for-mac-works-under-the-hood/)
As you can see above, instead of accessing the Host OS directly, Docker for macOS instead has to spin up its own Linux VM.
It then can only access the kernel of that VM, which then has to go through the steps above to synchronize the disks of your Containers and the Host OS.
Whereas Docker for Linux essentially has a direct line to the Host OS (and, by extension, the disk, network, GPU, etc), Docker for macOS has to go through several abstractions to do low-level tasks.
Development Machine Implications
Your typical Docker development setup is usually as follows:
Host OS
- Developer tooling (IDE's, text editors, linters, etc)
- Source code editing
- Source control
Container
- Application code and dependencies
- Some kind of hot or live reloading mechanism when code changes
- A copy or reference to the host OS source code
The jet plane taking off when you do a docker-compose up on macOS? It's your Host OS' resources hard at work to synchronize low-level I/O (specifically disk and network) between the Host OS and Containers; this is on top of having to run the Containers themselves.
This is also why you see the Hyperkit process usually consuming much of your CPU even at idle. All that synchronizing work between these layers is not trivial!
Options for macOS
This is the part where many would tell you to "just develop on Linux". While it is true that Docker on Linux is the architecture as intended (and therefore is the best experience), simply switching is not tenable for most folks.
The options below will get you closer to parity with the experience on Linux. At the very least, the jet plane taking off might only happen every once in a while versus all the time.
Docker for Mac Edge Build (with Mutagen)
As of this date, Docker has a blessed approach for minimizing resource consumption on disk changes using something under the hood called Mutagen. You won't have to worry about the details though, as they package it as part of the Docker for Mac Edge build.
Instructions
- Install Docker for Mac Edge Build
- In the Docker UI, go to Resources => File Sharing and specify what folders you want mounted to Docker containers
Pros
- "Blessed" by the Docker team
- Minimal setup -- use docker-compose and docker files as normal
- Drastically reduces CPU on hot/live reloads
- File changes are considerably faster
Cons
- Still working out the kinks
- Requires Docker Edge Build
docker-sync
An alternative that's been on the scene for several years now is called docker-sync.
docker-sync is essentially a container running in parallel with your own containers whose job is to efficiently let your container know when files change. It is, in effect, another abstraction layer to speed up the process.
Instructions
- Install docker-sync
- Modify your docker-sync.yml according to your dev setup
Pros
- Works on across Docker platforms
- Drastically reduces CPU on hot/live reloads
- File changes are considerably faster
Cons
- Configuration modifications needed
- More docker resources consumed due to additional parallel containers
- Additional orchestration needed to spin your containers up and down
- Occasionally has syncing issues; i.e. the container doesn't get updated with Host OS changes, and it needs restarting
In Summary
Docker was mainly built with Linux in mind. As it proved out its utility, it was eventually ported to macOS and Windows.
Since both operating systems are vastly different than Linux under the hood, virtualization was the only feasible way to get things working. This unfortunately results in these low-level inefficiencies that we otherwise take for granted.
With Mutagen being packaged as part of Docker for Mac in the future, there is hope for macOS developers that the "jet engine" problem starts to diminish.
However, as it currently stands, the best developer experience for Docker still remains its native Linux.
Top comments (29)
An alternative I found was to use NFS:
medium.com/@sean.handley/how-to-se...
firehydrant.io/blog/nfs-with-docke...
jeffgeerling.com/blog/2020/revisit...
I used to work with Docker for MacOS (with NFS volume) and indeed it's much faster than use the default volume map but even NFS volume is very, very slow compare to Docker on Linux, it's unbelievable.
Good links, Pedro. We'd looked into this as well, but it seemed docker-sync was a smidge faster and slightly more reliable. NFS looked solid overall though!
We use nfs to setup our devenv across all of our developers computers (linux/macos), it works fine :)
@ericnograles read this!!
docs.docker.com/docker-for-mac/osx...
Basically adding the "delegated" flag to docker volumes will help AMAZINGLY with CPU and I/O performance. Don't just trust me on it, try it!
Thanks for the link! SUPER interesting -- although if I'm reading that article above properly, it sounds like it'd be more of a
cached(i.e. host is the authority) setup?How has your experience been day-to-day? i.e. do the container volumes ever get out of sync, and if they do, is there a quick mechanism to get them back to parity (without having to straight up
docker-compose down && docker-compose up -d?Inspired by Windows WSL which runs a Linux VM that is integrated with Windows. I decided to setup a Linux VM on Mac and install my Docker dev environment on the Linux VM. It actually runs a lot better than my native Mac Docker environment. I'm using Parallels Desktop which is a lot more performant than Virtual Box so that could also help things as well. I haven't seen or heard of anyone else doing this but it was the only way for me to get my dev environment to run from my Mac without running out of resources.
Right there with you! I actually use a VirtualBox VM and just hook my host tooling up to it via SSH. Works so much better than Docker for Mac!
Awesome! You should write a post about :)
I used to work with Docker on MacOS for almost two years but some weeks ago I just gave up. After a comparison that I did running Docker on Linux, the MacOS version (Including the use of NFS volumes) is too slow.
If you can get rid of MacOS and use Linux, go ahead, I'm pretty sure, you won't regret.
This article is very interesting. I knew that Docker for Mac OS was different and running inside a VM, but not with such details.
However IMHO there is a little bit of exageration: it is working correctly on my machine, and the overhead in comparison with the Linux setup is not such a pain.
Thanks for sharing, Eric.
I think it's a case of YMMV here. We have a pretty I/O heavy setup with our front-end (i.e. webpack) and, without mitigation, it constantly chews through resources. Checking my Activity Monitor right now, at idle, my docker.hyperkit on my MBP sits at 40%...and that's with help from docker-sync.
Without mitigation, it's not unusual for it to sit at 90%+, especially when you're saving source code a bunch from the host.
On my work MBP with large projects it's up to about 50 times slower than on Linux running IO heavy apps, particularly those which have to monitor for changes or recursively search directories.
Great article! Built a solution to this problem that runs your local containers on remote linux machines: getporter.dev - I would love your honest thoughts on how this compares to your experience with Mutagen.
I had less issues running docker inside an Ubuntu VM than directly in Mac 😟
This makes some sense, unfortunately. If indeed the case generally, Docker devs should suggest such a solution, rather than providing a substandard VM of their own for MacOS. I'm hearing these reports after learning of "Doki malware" and I'm losing confidence in the product. Or should I say I'm losing faith in the organization behind the product.
Part of me agrees with you. However, they (Docker) are definitely in a tough spot. Whereas Microsoft was cooperative in creating a bridge to their kernel with WSL-2, Apple generally keeps macOS under lock and key. i.e. Docker for Mac will be at the mercy of the macOS Hyperkit.
The fact that the new Docker for Mac will ship with Mutagen def shows me that they hear us and are doing whatever they can to alleviate the situation.
fwiw, I've taken the Docker Edge Build with Mutagen for a spin and it's been very good! Once they work out the kinks, hopefully it consistently improves the experience for everyone.
The heating and cooling issues you're talking about will hopefully go away anyway when Macs switch to ARM
Is it not better to just install vm on your Mac, spin up a Linux OS and then install docker on that?
Better performing, definitely.
However, it does come with the additional tax of more networking overhead in the form of port forwards to/from the host/guest, or reverse proxying for more complicated setups.
I keep revisiting these posts every few months. Looks like Mutagen integration isn't happening now so I think I'll try the Linux VM as well.
Eric, thanks for the docker edge tip!.....
I had this issue for a month and at some point, I thought about taking my mac to a repair shop but this really helps me .....
Keep it up...