DEV Community

Cover image for Setup OSDev Environment on macOS
Vincent Yang
Vincent Yang

Posted on Originally published at blog.vyang.org

Setup OSDev Environment on macOS

Why OSDev on macOS works (if you stop fighting defaults)

The failure mode is always the same: you follow an OSDev tutorial, hit make, and end up with a binary that either won’t boot, won’t link, or isn’t even in the format your bootloader expects.

macOS is a great editor and workflow environment. But it is a terrible default build target for classic OSDev.

This guide builds a layered mental model for the “why”, then gives three setups (from lightest to heaviest) that I’ve used successfully: a cross-toolchain on macOS, a Docker build container, and a Linux VM via Lima.


The two hard constraints (the stack you can’t ignore)

1) CPU architecture: aarch64 vs x86

Modern Macs (M1/M2/M3/M4) are aarch64 (ARM64) machines. Most OSDev examples, bootloaders, and “hello kernel” walkthroughs targets i686/x86_64.

  • What breaks: Native clang/ld produce ARM machine code by default.
  • Why it matters: An x86 emulator (or real x86 hardware) can’t execute ARM opcodes. If your kernel is the wrong ISA, you won’t get “a little wrong” behavior—you’ll get a hard stop.

If you are targeting ARM on purpose, you still have the second constraint.

2) Executable format: Mach-O vs ELF

macOS uses Mach-O. Most OSDev toolchains and boot flows assume ELF.

  • ELF: The common format for Linux/BSD tooling and OSDev build systems.
  • Mach-O: The native macOS format, tightly integrated with Apple’s linker/loader and platform conventions.

  • What breaks: Bootloaders, kernels, and link scripts in OSDev projects often expect ELF sections/symbol layout and ELF headers.

  • Nuclear option (don’t do this): Write your own linker (or a Mach-O loader in your boot path) just to make the native toolchain fit.


Three working setups (from native → isolated → maximum compatibility)

Option 1: Cross-compiler on macOS (fastest feedback loop)

This is the “keep everything local” setup:

  • Editor: macOS
  • Build toolchain: i686-elf-* or x86_64-elf-*
  • Output: ELF binaries for your target

Path A: Homebrew (the quick start)

If all you need is a usable cross toolchain, Homebrew is the shortest path.

brew install i686-elf-binutils i686-elf-gcc
Enter fullscreen mode Exit fullscreen mode
  • Pros: Minutes to set up.
  • Cons: Less control. Once you start building a libc, switching freestanding/hosted modes, or pinning specific versions, you’ll want a source build.

Path B: Build from source (the “I want it correct” path)

This is the common OSDev approach: build binutils first, then build GCC against it.

Why this order matters

  • Binutils provides as (assembler), ld (linker), and friends.
  • GCC will call into those tools when producing final objects/binaries.

Binutils example (i686-elf)

# Native compiler toolchain for building tools
xcode-select --install

# Fetch sources
wget https://ftp.gnu.org/gnu/binutils/binutils-2.45.tar.gz
tar -xvf binutils-2.45.tar.gz

# Out-of-tree build keeps your source directory clean
mkdir -p binutils-2.45/build
cd binutils-2.45/build

# Key flags:
# --target=i686-elf    => produce tools that emit i686 code and ELF output
# --prefix=...         => install into an isolated directory
# --disable-nls        => disables Native Language Support (fewer deps, faster build)
# --disable-werror     => warnings won't fail the build on strict compilers
../configure \
  --target=i686-elf \
  --prefix=/usr/local/cross \
  --disable-nls \
  --disable-werror

# macOS doesn't ship `nproc` by default
make -j"$(sysctl -n hw.ncpu)"
make install
Enter fullscreen mode Exit fullscreen mode

Repeat the same pattern for GCC, pointing it at the same --target and --prefix.


Option 2: Docker (portable + reproducible)

Docker gives you a clean Linux build environment while keeping your source on the host.

  • What you gain: A pinned, shareable toolchain across machines.
  • What you lose: Some performance (especially if you force x86 emulation on Apple Silicon).

Minimal Dockerfile skeleton

FROM --platform=linux/amd64 ubuntu:latest

RUN apt-get update && apt-get install -y \
    build-essential \
    wget \
    nasm \
    qemu-system-x86

# Toolchain install/build goes here:
# - install prebuilt x86_64-elf toolchain packages, OR
# - build binutils+gcc from source inside the image

WORKDIR /osdev
CMD ["/bin/bash"]
Enter fullscreen mode Exit fullscreen mode

Build + run

docker build -t osdev-env .

# -v maps your current folder into the container
docker run --rm -it -v "$(pwd)":/osdev osdev-env make
Enter fullscreen mode Exit fullscreen mode

Why --platform=linux/amd64 matters
On Apple Silicon, this forces an x86_64 userland. That can increase compatibility with prebuilt toolchains, but it may run slower due to emulation.


Option 3: Linux VM via Lima (maximum compatibility)

If you want the least “mystery meat” behavior, stop fighting macOS and build inside Linux.

  • This is the nuclear option for compatibility: it matches the environment most OSDev docs assume.
  • Workflow: edit on macOS, build/run in Linux, attach VS Code via SSH.

Install Lima

brew install lima
Enter fullscreen mode Exit fullscreen mode

Create and enter a VM

limactl start --name osdev template://ubuntu
limactl shell osdev
Enter fullscreen mode Exit fullscreen mode

Mount your source directory (host ↔ guest)

Edit the instance config:

limactl edit osdev
Enter fullscreen mode Exit fullscreen mode

Add a writable mount:

mounts:
  - location: /path/to/your/os/source/code
    writable: true
Enter fullscreen mode Exit fullscreen mode

VS Code integration

Add to your SSH config so your editor can see Lima hosts:

Include ~/.lima/*/ssh.config
Enter fullscreen mode Exit fullscreen mode
  • Editor UI: macOS
  • Build tools + QEMU: inside the VM

Run QEMU inside the VM

sudo apt-get update
sudo apt-get install -y qemu-system-x86

# Example: adjust to your boot flow / image layout
qemu-system-x86_64 -m 512M -net none -kernel /path/to/your/os/kernel
Enter fullscreen mode Exit fullscreen mode

Conclusion

Pick the lightest setup that gives you deterministic results:

  • Cross-compiler: Best day-to-day speed on macOS.
  • Docker: Best reproducibility for teams and CI.
  • Lima VM: Best compatibility when you’re deep in toolchain/bootloader land.

If you want a full Linux host on Apple Silicon without virtualization, Asahi Linux is the endgame.

Experiments & Resources

To keep this post focused on the core concepts, I’ve hosted the full code bundles, and early-access development logs on my personal site. If you're looking to run these you can find everything here:

View Project Files & Early Access

Top comments (0)