How to set up an ARM64 playground on Ubuntu 18.04

With the recent announcement of Apple Silicon (Apple laptops shifting to the 64 bit ARM architecture), it's a great time to finally learn ARM64!

Since actual ARM64 systems are a bit hard to come by, here's how to set up a basic dev playground on a standard Ubuntu 18.04 x64 system. We'll be able to compile, disassemble, execute and debug ARM64 programs with it.

A note on terminology: Many of the tools installed will be named "aarch64" which is effectively equivalent to "ARM64" for our purposes.

If you are interested in systems topics, feel free to follow me on twitter where I post about stuff like this :)

Install tools

Install cross compilers + toolchain

First, let's install a toolchain for cross compiling C and C++.

sudo apt install gcc-8-aarch64-linux-gnu
sudo apt install g++-8-aarch64-linux-gnu

This gives us (among other things):

• aarch64-linux-gnu-gcc-8: Cross compiler for C
• aarch64-linux-gnu-g++-8: Cross compiler for C++

Install QEMU

Our x64 system won't be able to run binaries produced by this toolchain natively, so we need to emulate. QEMU is a high quality emulator (and more) that is able to run binaries of different architectures in an emulated userspace environment.

sudo apt install qemu

This gives us (among other things):

• qemu-aarch64: Userspace emulator for ARM64 binaries

Install GDB with support for multiple architecture

We can use QEMU along with GDB to debug our binaries, but we need a special version of GDB with support for non-native architectures.

sudo apt install gdb-multiarch

This gives us:

• gdb-multiarch

Cross compile a program

Now we have everything we need to cross compile C/C++ into ARM64 binaries, look at the generated assembly, and run/debug the binaries. Let's start with compiling.

Here's a small C++ program called arm64main.cpp.

#include <iostream>

int main() {
    std::cout << "Hello from ARM64!\n";
}

Here's how to compile it.

$ aarch64-linux-gnu-g++-8 -o arm64main arm64main.cpp -static
$ file arm64main                 
arm64main: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=7b1bbf64436de3f0e268d1d8ab93d2123d4dcaef, not stripped

Note the -static flag. We need this because the cross compiler by default generates dynamic binaries that rely on an ARM64 version of the dynamic linker, which we don't have. Generating static binaries is an easy way around this, since we're just playing around.

Cool! Now we have an ARM64 binary we can disassemble, execute, and debug.

Disassemble a binary

We're interested in learning the ARM64 architecture, so it's important to be able to disassemble binaries, so we can see the ARM64 instructions that it contains.

We can do this with aarch64-linux-gnu-objdump which we got with the toolchain. For binaries compiled from C++, we can also use the aarch64-linux-gnu-c++filt utility to demangle the symbol names.

$ aarch64-linux-gnu-objdump -d arm64main | aarch64-linux-gnu-c++filt

This disassembles the binary. For example, the main function looks like this:

000000000040090c <main>:
  40090c:       a9bf7bfd        stp     x29, x30, [sp, #-16]!
  400910:       910003fd        mov     x29, sp
  400914:       f00007e0        adrp    x0, 4ff000 <free_mem+0x20>
  400918:       910a0001        add     x1, x0, #0x280
  40091c:       b0000ae0        adrp    x0, 55d000 <_GLOBAL_OFFSET_TABLE_+0x48>
  400920:       f9436800        ldr     x0, [x0, #1744]
  400924:       94005a2d        bl      4171d8 <std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*)>
  400928:       52800000        mov     w0, #0x0                        // #0
  40092c:       a8c17bfd        ldp     x29, x30, [sp], #16
  400930:       d65f03c0        ret

Pretty neat! Now we can learn about the architecture by looking up what these instructions do in the manual.

Execute and debug a binary

The last functionality we want for our playground is to be able to execute, and ideally debug our binaries. We can do this using QEMU.

We can execute our binary with qemu-aarch64.

$ qemu-aarch64 ./arm64main
Hello from ARM64!

In fact, we can actually also invoke the binary like a native one:

$ ./arm64main
Hello from ARM64!

This works because QEMU registers itself as an interpreter for ARM64 ELF binaries via binfmt_misc.¹

In addition to simply executing, we can also debug and step through the ARM64 binary with QEMU and GDB.

We will need two terminal windows for this.

In the first window, run QEMU with the -g flag which will spawn a debug server on a port.

$ qemu-aarch64 -g 1234 ./arm64main

In the second window, attach to the server using GDB.

$ gdb-multiarch ./arm64main
(gdb) target remote :1234
Remote debugging using :1234
0x00000000004007c4 in _start ()
(gdb)

Nice! GDB has attached to QEMU's debug server, and now we do much of our normal debugging activities. We can set breakpoints, and examine register and memory state.

(gdb) break main
Breakpoint 1 at 0x400920
(gdb) continue
Continuing.

Breakpoint 1, 0x0000000000400920 in main ()
(gdb) bt
#0  0x0000000000400920 in main ()
(gdb) x/i $pc
=> 0x400920 <main+20>:  ldr     x0, [x0, #1744]
(gdb) info reg x0
x0             0x55d000 5623808
(gdb) x/8x $sp
0x40008003f0:   0x00800400      0x00000040      0x00493bf4      0x00000000
0x4000800400:   0x00000000      0x00000000      0x00400810      0x00000000
(gdb) set disassemble-next-line on
(gdb) si
0x0000000000400924 in main ()
=> 0x0000000000400924 <main+24>:        2d 5a 00 94     bl      0x4171d8 <_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc>
(gdb)
0x00000000004171d8 in std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) ()
=> 0x00000000004171d8 <_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc+0>:      fd 7b be a9     stp     x29, x30, [sp, #-32]!
(gdb)
0x00000000004171dc in std::basic_ostream<char, std::char_traits<char> >& std::operator<< <std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) ()
=> 0x00000000004171dc <_ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc+4>:      fd 03 00 91     mov     x29, sp

There's many other resources for learning GDB, so I won't go into detail here.

Conclusion

That's it! We now have a dev playground which lets us compile, disassemble, execute, and debug ARM64 binaries so we can experiment and learn about the ARM64 architecture.

If you enjoyed this guide, feel free to follow me on Twitter where I tweet about C++ and other interesting low level topics.

https://twitter.com/offlinemark

---

---

1. https://en.wikipedia.org/wiki/Binfmt_misc ↩

Comments

[Fenix]

Thanks for sharing

[Tingwei]

Thank you !!!

[rikercd]

Maybe I'm missing something obvious here, but I am unable to install the gcc package in an ubuntu 18.04 devcontainer. Says it is not found. Any ideas?