DEV Community

Cover image for Setup OSDev Environment on macOS
Vincent Yang
Vincent Yang

Posted on • Originally published at blog.vyang.org

Setup OSDev Environment on macOS

Why OSDev on macOS works (if you stop fighting defaults)

The failure mode is always the same: you follow an OSDev tutorial, hit make, and end up with a binary that either won’t boot, won’t link, or isn’t even in the format your bootloader expects.

macOS is a great editor and workflow environment. But it is a terrible default build target for classic OSDev.

This guide builds a layered mental model for the “why”, then gives three setups (from lightest to heaviest) that I’ve used successfully: a cross-toolchain on macOS, a Docker build container, and a Linux VM via Lima.


The two hard constraints (the stack you can’t ignore)

1) CPU architecture: aarch64 vs x86

Modern Macs (M1/M2/M3/M4) are aarch64 (ARM64) machines. Most OSDev examples, bootloaders, and “hello kernel” walkthroughs targets i686/x86_64.

  • What breaks: Native clang/ld produce ARM machine code by default.
  • Why it matters: An x86 emulator (or real x86 hardware) can’t execute ARM opcodes. If your kernel is the wrong ISA, you won’t get “a little wrong” behavior—you’ll get a hard stop.

If you are targeting ARM on purpose, you still have the second constraint.

2) Executable format: Mach-O vs ELF

macOS uses Mach-O. Most OSDev toolchains and boot flows assume ELF.

  • ELF: The common format for Linux/BSD tooling and OSDev build systems.
  • Mach-O: The native macOS format, tightly integrated with Apple’s linker/loader and platform conventions.

  • What breaks: Bootloaders, kernels, and link scripts in OSDev projects often expect ELF sections/symbol layout and ELF headers.

  • Nuclear option (don’t do this): Write your own linker (or a Mach-O loader in your boot path) just to make the native toolchain fit.


Three working setups (from native → isolated → maximum compatibility)

Option 1: Cross-compiler on macOS (fastest feedback loop)

This is the “keep everything local” setup:

  • Editor: macOS
  • Build toolchain: i686-elf-* or x86_64-elf-*
  • Output: ELF binaries for your target

Path A: Homebrew (the quick start)

If all you need is a usable cross toolchain, Homebrew is the shortest path.

brew install i686-elf-binutils i686-elf-gcc
Enter fullscreen mode Exit fullscreen mode
  • Pros: Minutes to set up.
  • Cons: Less control. Once you start building a libc, switching freestanding/hosted modes, or pinning specific versions, you’ll want a source build.

Path B: Build from source (the “I want it correct” path)

This is the common OSDev approach: build binutils first, then build GCC against it.

Why this order matters

  • Binutils provides as (assembler), ld (linker), and friends.
  • GCC will call into those tools when producing final objects/binaries.

Binutils example (i686-elf)

# Native compiler toolchain for building tools
xcode-select --install

# Fetch sources
wget https://ftp.gnu.org/gnu/binutils/binutils-2.45.tar.gz
tar -xvf binutils-2.45.tar.gz

# Out-of-tree build keeps your source directory clean
mkdir -p binutils-2.45/build
cd binutils-2.45/build

# Key flags:
# --target=i686-elf    => produce tools that emit i686 code and ELF output
# --prefix=...         => install into an isolated directory
# --disable-nls        => disables Native Language Support (fewer deps, faster build)
# --disable-werror     => warnings won't fail the build on strict compilers
../configure \
  --target=i686-elf \
  --prefix=/usr/local/cross \
  --disable-nls \
  --disable-werror

# macOS doesn't ship `nproc` by default
make -j"$(sysctl -n hw.ncpu)"
make install
Enter fullscreen mode Exit fullscreen mode

Repeat the same pattern for GCC, pointing it at the same --target and --prefix.


Option 2: Docker (portable + reproducible)

Docker gives you a clean Linux build environment while keeping your source on the host.

  • What you gain: A pinned, shareable toolchain across machines.
  • What you lose: Some performance (especially if you force x86 emulation on Apple Silicon).

Minimal Dockerfile skeleton

FROM --platform=linux/amd64 ubuntu:latest

RUN apt-get update && apt-get install -y \
    build-essential \
    wget \
    nasm \
    qemu-system-x86

# Toolchain install/build goes here:
# - install prebuilt x86_64-elf toolchain packages, OR
# - build binutils+gcc from source inside the image

WORKDIR /osdev
CMD ["/bin/bash"]
Enter fullscreen mode Exit fullscreen mode

Build + run

docker build -t osdev-env .

# -v maps your current folder into the container
docker run --rm -it -v "$(pwd)":/osdev osdev-env make
Enter fullscreen mode Exit fullscreen mode

Why --platform=linux/amd64 matters
On Apple Silicon, this forces an x86_64 userland. That can increase compatibility with prebuilt toolchains, but it may run slower due to emulation.


Option 3: Linux VM via Lima (maximum compatibility)

If you want the least “mystery meat” behavior, stop fighting macOS and build inside Linux.

  • This is the nuclear option for compatibility: it matches the environment most OSDev docs assume.
  • Workflow: edit on macOS, build/run in Linux, attach VS Code via SSH.

Install Lima

brew install lima
Enter fullscreen mode Exit fullscreen mode

Create and enter a VM

limactl start --name osdev template://ubuntu
limactl shell osdev
Enter fullscreen mode Exit fullscreen mode

Mount your source directory (host ↔ guest)

Edit the instance config:

limactl edit osdev
Enter fullscreen mode Exit fullscreen mode

Add a writable mount:

mounts:
  - location: /path/to/your/os/source/code
    writable: true
Enter fullscreen mode Exit fullscreen mode

VS Code integration

Add to your SSH config so your editor can see Lima hosts:

Include ~/.lima/*/ssh.config
Enter fullscreen mode Exit fullscreen mode
  • Editor UI: macOS
  • Build tools + QEMU: inside the VM

Run QEMU inside the VM

sudo apt-get update
sudo apt-get install -y qemu-system-x86

# Example: adjust to your boot flow / image layout
qemu-system-x86_64 -m 512M -net none -kernel /path/to/your/os/kernel
Enter fullscreen mode Exit fullscreen mode

Conclusion

Pick the lightest setup that gives you deterministic results:

  • Cross-compiler: Best day-to-day speed on macOS.
  • Docker: Best reproducibility for teams and CI.
  • Lima VM: Best compatibility when you’re deep in toolchain/bootloader land.

If you want a full Linux host on Apple Silicon without virtualization, Asahi Linux is the endgame.

Experiments & Resources

To keep this post focused on the core concepts, I’ve hosted the full code bundles, and early-access development logs on my personal site. If you're looking to run these you can find everything here:

View Project Files & Early Access

Top comments (0)