Introduction
Sometimes you look at a backtrace in gdb and find ?? where the function names should be.
#0 0x0000aaaaaaaa0894 in ?? ()
#1 0x0000aaaaaaaa0994 in ?? ()
?? means that gdb could not find a function name for that address. This article covers four reasons you get ??:
- The application has been
stripped - The function is inside a library and is not public
- The location of the libraries was not given with
set sysroot - When a core dump is opened on the host, gdb cannot match it with the application
The test environment is ARM64 Linux (BusyBox) running on QEMU, with an Ubuntu 24.04 host.
Reason 1: The Application Has Been stripped
Applications shipped in products are sometimes run through strip to make the file smaller. strip removes the following two things:
| What is removed | Contents | What gdb uses it for |
|---|---|---|
| Symbols | The mapping between function and variable names and addresses | Turning addresses into function names |
| Debug information | Line numbers, the types and locations of variables, and so on (added with -g) |
Showing line numbers and the values of variables |
The symbols that function names come from are gone, so every function in the application becomes ??.
(No debugging symbols found in ./sensor-stripped)
...
Program received signal SIGSEGV, Segmentation fault.
0x0000aaaaaaaa0894 in ?? ()
(gdb) bt
#0 0x0000aaaaaaaa0894 in ?? ()
#1 0x0000aaaaaaaa0994 in ?? ()
#2 0x0000fffff7e184c4 in ?? () from /lib/libc.so.6
#3 0x0000fffff7e18598 in __libc_start_main () from /lib/libc.so.6
#4 0x0000aaaaaaaa0770 in ?? ()
The file command tells you whether a file has been stripped. If it says stripped at the end, it has.
$ file sensor-stripped
sensor-stripped: ELF 64-bit LSB pie executable, ARM aarch64, ... stripped
Fix: Keep the Unstripped Application or a Debug Information File
Even if the application on the target stays stripped, if you keep the unstripped application from the same build on the host, gdb on the host can show function names and line numbers when you connect through gdbserver.
(gdb) bt
#0 print_average () at sensor.c:33
#1 0x0000aaaaaaaa0994 in main () at sensor.c:51
Instead of keeping the whole unstripped application, you can also put just the debug information in a separate file.
objcopy --only-keep-debug sensor sensor.debug
objcopy --add-gnu-debuglink=sensor.debug sensor-stripped
The first line extracts only the debug information and symbols into sensor.debug. The second line writes a marker into the stripped application saying "the debug information is in sensor.debug." gdb follows this marker and loads sensor.debug (see "Debugging Information in Separate Files" in the gdb manual: https://sourceware.org/gdb/current/onlinedocs/gdb.html/Separate-Debug-Files.html).
With either method, what you keep must come from the same build. With something rebuilt later, the instructions and addresses may shift, and they cannot be matched correctly.
Note: Just Leaving Out -g Does Not Give You ??
If you build without -g, the symbols remain, so function names still appear. What you lose are line numbers and variables. You get ?? only when strip removes the symbols as well.
Reason 2: The Function Is Inside a Library and Is Not Public
Sometimes the application itself is not stripped, but only the frames inside a library are ??.
(gdb) bt
#0 0x0000fffff7e92b1c in ?? () from /lib/libc.so.6
#1 0x0000aaaaaaaa0874 in reset_average () at sensor.c:27
#2 0x0000aaaaaaaa093c in main () at sensor.c:46
Line 27 in #1 is the line that calls memset(), so #0 is inside memset(). Even so, its name is ??.
The C library may keep only the names of the functions it exports. memset() actually works by running an implementation function chosen to suit the CPU, and that implementation function is not exported. So its name is shown as ??.
Fix: Look at the Calling Line
Even if you do not know the name of the function inside the library, bt tells you which line of your own code called it. In the example above, sensor.c:27 in #1 leads you to the memset() that caused the problem.
If you do not need to follow it into the library, the shortcut is to skip the ?? frames and look at the topmost frame in your own code.
Reason 3: The Location of the Libraries Was Not Given with set sysroot
An application runs using shared libraries such as the C library. gdb on the host reads the function names inside a library from the library's file.
When you open a core dump with gdb on the host alone, you need to tell gdb with set sysroot where on the host the target's library files are. If you do not, gdb cannot read the library files, and the frames inside the libraries become ??.
When you are connected through gdbserver, gdb receives the library files from the target through gdbserver and reads them, even without set sysroot.
Fix: Prepare a Directory Laid Out Like the Target and Specify It
gdb treats the directory specified with set sysroot as / on the target. For example, if you specify ~/sysroot, gdb looks for /lib/libc.so.6 on the target at ~/sysroot/lib/libc.so.6 on the host.
(gdb) set sysroot ~/sysroot
(gdb) file ./sensor
(gdb) core-file core
So inside the directory you specify, you need to place the libraries at the same paths as on the target. The gdb manual (Commands to Specify Files: https://sourceware.org/gdb/current/onlinedocs/gdb.html/Files.html) says:
If you use set sysroot to find executables and shared libraries, they need to be laid out in the same way that they are on the target, with e.g. a /bin, /lib and /usr/lib hierarchy under path.
Use the same library files as the ones running on the target. Specify set sysroot before loading the core dump.
Reason 4: When a Core Dump Is Opened on the Host, gdb Cannot Match It with the Application
Even after specifying set sysroot as in the fix for Reason 3, you may still get ??. Here is an example where only the libraries were placed there, not the application. Even though the application is given with file, neither function names nor line numbers appear.
(gdb) set sysroot ~/appdbg-study/initramfs
(gdb) file ./sensor
Reading symbols from ./sensor...
(gdb) core-file core
warning: Can't open file /mnt/ch05/sensor during file-backed mapping note processing
...
#0 0x0000aaaae5490894 in ?? ()
(gdb) bt
#0 0x0000aaaae5490894 in ?? ()
#1 0x0000ffff81c884c4 in ?? ()
#2 0x0000aaaae5490770 in ?? ()
Cause
Before gdb on the host applies function names, it checks that the application in the core dump and the application given with file are the same. What it uses for this check is the start of the application (the ELF headers), and gdb looks for it in this order:
- Inside the core dump
- In the application at the same path as on the target, inside the directory specified with
set sysroot
If it is in neither place, function names become ??.
In this environment, it was in neither place.
-
The ELF headers were not in the core dump: whether they are included is decided by the
0x10bit of/proc/<pid>/coredump_filter(bit 4 inman 5 core, "Dump ELF headers."). On the target, the value was00000023, and this bit was not set -
The application was not placed in the sysroot: the libraries were there, but the application (
/mnt/ch05/sensor) was not
This default is decided by the kernel configuration CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS. In the kernel's Kconfig (fs/Kconfig.binfmt), it defaults to enabled, but the ARM64 defconfig disables it. In kernels distributed by board makers, it was also disabled in NXP's imx_v8_defconfig and in Jetson Linux's defconfig / tegra_defconfig.
Fix: Make One of the Two Available
| Fix | What to do | When to do it |
|---|---|---|
| Place the application in the sysroot | Inside the directory specified with set sysroot, place the application at the same path as on the target too |
After the crash, on the host |
| Put the ELF headers in the core dump | Set coredump_filter to 0x33
|
Before the crash, on the target |
If you set coredump_filter in the shell, it applies to the applications run from that shell.
echo 0x33 > /proc/self/coredump_filter
./sensor
With a core dump taken this way, function names and line numbers appeared even without placing the application in the sysroot.
#0 print_average () at sensor.c:33
33 printf("%s average: %d\n", avg.sensor->name, avg.sum / avg.num);
(gdb) bt
#0 print_average () at sensor.c:33
#1 0x0000aaaac4cd0994 in main () at sensor.c:51
You can change the default by adding coredump_filter=0x33 to the kernel boot options, or by enabling CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS and rebuilding the kernel.
Note that what goes into the core dump is only the ELF headers, not the whole files. With either fix, the shared libraries must be placed inside the sysroot at the same paths as on the target.
Summary
| Reason | Where ?? appears |
Fix |
|---|---|---|
The application has been stripped |
Every function in the application | Keep the unstripped application from the same build, or a debug information file |
| A library function that is not public | Frames inside the library | Look at the calling line in your own code |
The location of the libraries was not given with set sysroot
|
Frames inside the libraries | Prepare a directory laid out like the target and specify it with set sysroot
|
| gdb cannot match the core dump opened on the host with the application | Every function in the application | Place the application in the sysroot, or set coredump_filter to 0x33
|
When you see ??, first check which frames are ??; that gives you an idea of which reason applies. If everything, even inside the application, is ??, it is Reason 1 or 4. If only the frames inside libraries are, it is Reason 2 or 3.
Want to Learn More?
This article is based on my book, Hands-On Linux Application Debugging with QEMU.
In the book, with no real hardware, using just one PC, you debug applications for embedded Linux both on the target and from the host. You run applications with bugs put in on purpose and use tools to find the line that causes each one.
- Stop an application in gdb, look at its variables, and get a backtrace. Find the line that overwrote a value with a watchpoint
- Connect from the host with gdbserver and VS Code to debug
- Open the core dump of a crashed application, both on the target and on the host
- In a multithreaded application that has stopped, see which thread is waiting for what
- Find memory errors that do not cause a crash with ASan and UBSan
Top comments (0)