Ben Taylor

Posted on Apr 22, 2021 • Updated on Jun 22, 2021

Running Vagrant on an M1 Apple Silicon using Docker

#docker #vagrant #linux #arm64

I've recently started at a new job, and I'm fortunate enough to have a new M1 Macbook Air. I'd heard from the wider community that everything "just works" on the M1, even though it's not an Intel chip. "Even Docker" they said. "Wow" I thought, this will be seamless, incredible.

So I sit there on my first day ready to get started. NPM installs everything fine, my python packages all install and both the backend and frontend are running, too easy! Then I hit an issue. This application requires a service which is run using Vagrant on Virtualbox.

Virtualbox very much doesn't support Apple Silicon. It's unclear whether Virtualbox will ever support Apple Silicon. The preferred alternative is a product by VMWare which also doesn't (yet) support Apple Silicon. I'm in a bit of a pickle.

But! Never fear! I can see that Vagrant also supports Docker. So over the next three days I dive into documentation, I frantically search the web for "vagrant docker M1" and "vagrant docker config" and "vagrant docker network issue" and "vagrant docker systemctl". After shaving at least three seperate yaks, and configuring a monstrosity I got it all running! Here's some lessons I learned.

Disclaimer: I am very much not a linux / ops / devops person. My experience with linux is limited but there's not much help out there for this issue so I'm doing my best.

Docker and Vagrant are frenemies

Docker and Vagrant have very different philosophies about how you should run your development setup. Docker's ideal is a minimalist setup, with just enough to run the single process you need. If you want more processes, Docker wants you to create more containers. Vagrant prefers a maximalist style, install everything on the one virtual machine and get it all going together.

To get Docker going you'll need a Dockerfile this describes the setup needed to create the Docker container. For our Docker container to be friends with Vagrant we're going to configure it as if its a traditional linux machine. By default you'll need at least sshd (which lets you SSH into the machine). I also needed systemd (which runs services). This is very much not how you should do Docker normally.

Here's the Dockerfile I used:

# Docker image to use with Vagrant
# Aims to be as similar to normal Vagrant usage as possible
# Adds Puppet, SSH daemon, Systemd
# Adapted from https://github.com/BashtonLtd/docker-vagrant-images/blob/master/ubuntu1404/Dockerfile

FROM ubuntu:18.04
ENV container docker
RUN apt-get update -y && apt-get dist-upgrade -y

# Install system dependencies, you may not need all of these
RUN apt-get install -y --no-install-recommends ssh sudo libffi-dev systemd openssh-client

# Needed to run systemd
# VOLUME [ "/sys/fs/cgroup" ]
# Doesn't appear to be necessary? See comments

RUN apt-get -y install puppet

# Add vagrant user and key for SSH
RUN useradd --create-home -s /bin/bash vagrant
RUN echo -n 'vagrant:vagrant' | chpasswd
RUN echo 'vagrant ALL = NOPASSWD: ALL' > /etc/sudoers.d/vagrant
RUN chmod 440 /etc/sudoers.d/vagrant
RUN mkdir -p /home/vagrant/.ssh
RUN chmod 700 /home/vagrant/.ssh
RUN echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1I2YJ6oXevct1YeS0o9HZyN1Q9qgCgzUFtdOKLv6IedplqoPkcmF0aYet2PkEDo3MlTBckFXPITAMzF8dJSIFo9D8HfdOV0IAdx4O7PtixWKn5y2hMNG0zQPyUecp4pzC6kivAIhyfHilFR61RGL+GPXQ2MWZWFYbAGjyiYJnAmCP3NOTd0jMZEnDkbUvxhMmBYSdETk1rRgm+R4LOzFUGaHqHDLKLX+FIPKcF96hrucXzcWyLbIbEgE98OHlnVYCzRdK8jlqm8tehUc9c9WhQ==" > /home/vagrant/.ssh/authorized_keys
RUN chmod 600 /home/vagrant/.ssh/authorized_keys
RUN chown -R vagrant:vagrant /home/vagrant/.ssh
RUN sed -i -e 's/Defaults.*requiretty/#&/' /etc/sudoers
RUN sed -i -e 's/\(UsePAM \)yes/\1 no/' /etc/ssh/sshd_config

# Start SSH
RUN mkdir /var/run/sshd
EXPOSE 22
RUN /usr/sbin/sshd

# Start Systemd (systemctl)
CMD ["/lib/systemd/systemd"]

If you don't need systemd in your setup then you can remove that and change the RUN of sshd to:

CMD ["/usr/sbin/sshd", "-D"]

All of this sets up a Docker container which doesn't work like a regular Docker container. It runs more like a virtual machine. This means it will be difficult to manage using the normal Docker commands. Once you take it down it will also be difficult to get up again. Best to control it with Vagrant. I've had to delete the container and re-create it with Vagrant many times.

Vagrant is pretty easy to configure

To get Vagrant running using your Dockerfile you just need to add a little bit of configuration:

Vagrant.configure(2) do |config|

# ... your existing config

  # Custom configuration for docker
  config.vm.provider "docker" do |docker, override|
    # docker doesnt use boxes
    override.vm.box = nil

    # this is where your Dockerfile lives
    docker.build_dir = "."

    # Make sure it sets up ssh with the Dockerfile
    # Vagrant is pretty dependent on ssh
    override.ssh.insert_key = true
    docker.has_ssh = true

    # Configure Docker to allow access to more resources
    docker.privileged = true
  end

# ...

end

You may need to override some other settings here. The override variable allows you to change top-level settings, while the docker variable allows you to setup docker-specific stuff. Have a read of the vagrant docker provider and vagrant docker provisioning documentation.

Once you've set this up, you can run your Vagrant machine with:

$ vagrant up --provider=docker

And if you need to do extra things inside the machine, you can ssh in with:

$ vagrant ssh

You should have sudo access as well if you need it.

Vagrant doesn't like to run multiple boxes

When I first ran Vagrant, before I realised Virtualbox wouldn't work it started and then failed. Then when I tried to run vagrant with the Docker provider it gave me errors about not having multiple boxes. I also couldn't delete my box because virtualbox wasn't running.

You can just delete the files in .vagrant to resolve that. It seems like Vagrant doesn't have a good way of resolving this itself.

You can re-provision Vagrant

Sometimes I found my setup would break, in my case it wasn't too hard to just run the Vagrant provisioning script again.

$ vagrant up --provision

Once that was complete some of my issues got resolved. It's not a good idea to blindly do this, but in my case I'm just trying to get it running.

Docker's public network bridge feature is broken

It doesn't seem to work on Mac! So if your Vagrant config has something like:

config.vm.network "public_network"

Try to see if you can replace it with:

config.vm.network "private_network", type: "dhcp"

In my case this substitution was sufficient, but you may have other requirements.

Your linux binaries need to be arm64

I ran into some issues because our provisioning scripts assumed that the architecture of the system would be amd64 (Intel and AMD 64-bit processors). It was a pretty safe bet that the CPU was amd64 up until Apple introduced their M1 chip!

Running linux on the M1 chip uses the arm64 architecture instead. As a handy shortcut you can get the architecture of the current machine in linux with:

$ dpkg --print-architecture

I used this in shell scripts by setting a variable:

readonly ARCH=`dpkg --print-architecture`

And then substituting that in when binaries were downloaded and installed, usually that was relatively simple, e.g.:

wget http://somedomain.com/some_binary_linux_amd64.zip

Became

wget "http://somedomain.com/some_binary_linux_${ARCH}.zip"

I learned a lot - but I still don't know much

I learned a lot about Docker (and how not to do things) Vagrant (and how to hack it) and some various linux tricks trying to get this all working. However I'm very much not a linux, Docker or Vagrant person, I'm just trying to make things work. So if I've written anything here horribly wrong, or extremely misguided, please throw down a comment.

If this helped you I'd love to hear it! I got stuck googling around for how to do this, and I imagine there are other people out there in the same boat.

Top comments (13)

jamalbendadi • Jun 12 '21 • Edited

I don't have a “/sys/fs/cgroup” folder on my Mac m1 and I believe this is giving me an error because if I leave the volume line out it starts up normally. This is the error:
Command: ["docker", "run", "--name", "project_default_1623498440", "-d", "-p", "0.0.0.0🔢80", "-p", "5678:8080", "-p", "6000:1433", "-p", "6001:3306", "-p", "5432:1080", "-p", "127.0.0.1:2222:22", "--privileged", "b6cedafbcc778e04751de4e9d489d598a097c9e8032a2cd8518e9a4fe43bbb9f", {:notify=>[:stdout, :stderr]}]

Stderr: docker: Error response from daemon: OCI runtime create failed: invalid mount {Destination:[ Type:bind Source:/var/lib/docker/volumes/12fac82322215174e740bef9cd4f05ef753399de075f0f650dbe2e3d3e673ccb/_data Options:[rbind]}: mount destination [ not absolute: unknown.

Could this really be the cause of it or is it something else?

Ben Taylor • Jun 20 '21

Hi, it looks like there's a bug in the script I posted. The quote marks around that part "/sys/fs/cgroup" were the wrong kind (the curly kind like “, instead of the kind you use in code like "). Not sure how that happened, sorry! If you fix that up it should work.

ingrid • Jun 16 '21

I am having the same problem! Could you fix it?

jamalbendadi • Jun 18 '21

I just left the line out and it ran correctly, but if that is not a good idea for you, maybe this might help you
github.com/moby/moby/issues/30723

Ben Taylor • Jun 20 '21

Hi Ingrid, I've left a comment on how to fix that issue above. I've also fixed it in the article.

Ben Taylor • Jun 22 '21

I've updated the Dockerfile to comment out the line VOLUME [ "/sys/fs/cgroup" ]. Initially I somehow had the wrong syntax for that line which seems to have just ignored it. Locally mine works without that line so I think it's unnecessary.

Aldrin Navarro • Jan 19 '22

Thanks for sharing Ben. I use Vagrant too out of necessity rather than convenience with the current state of ARM/Apple Silicon support.

"Vagrant doesn't like to run multiple boxes"
This section isn't really clear to me.

Is vagrant box remove <box_name> not working? But since a vagrant box isn't required, then maybe Docker (desktop) is more useful for pruning/deleting.

How about running vagrant destroy -f <vm_name> before making changes to the Vagrantfile? AFAIK vagrant commands read this file on execution. So it takes extra caution when to do changes or at least comment out some lines on the Vagrantfile, destroy the vm, then make the changes on Vagrantfile, and run vagrant again.

Then when I tried to run vagrant with the Docker provider it gave me errors about not having multiple boxes.

I'm not sure I understood this correctly. No errors presented here as well. Perhaps a log should lead you somewhere.

Some places I look for and things I do for clues,

Running vagrant with --debug flag
Check for logs of the VM provider .vagrant/machines/<vm_name>/<provider>/**/*.log

But then again, Docker and Vagrant are two different species. So you are always on the edge.

Ben Taylor • Jan 22 '22

What happened was my Vagrantfile was setup to run using Virtualbox, but I didn't have it on my machine. Vagrant created the box but then errored out. It wasn't able to destroy the box because Virtualbox wouldn't run. It was just one of those awkward scenarios where tools expect certain pre-requisites.

Lea H • Sep 20 '21

Hey! This was awesome!
Any idea on how to solve a non responsive ssh connection? At first creation it works great, but when I do vagrant halt and then vagrant up it is unable to ssh reconnect, and vagrant ssh delivers nothing.

Ben Taylor • Sep 23 '21

I'm not sure sorry! I'd guess that sshd is stopping, but I'm not sure why that would be. I found that if I took down this box it wouldn't come up, so I would just recreate it instead. This is not really a setup I would recommend, rather something I had to do to get an existing project to work!