In the last part of my first article on how GCC compiles, I explained the System V AMD64 ABI convention for passing parameters, it's time to explain how a function returns a result. But before that, I am going to explain the mechanism used by GCC to allocate some space in stack memory. Stack memory is a built in mechanism for x86 processors. It comes with specific instructions (push , pop ) to save and restore some data and a specific register rsp ( stack pointer register). It is used by compilers (like GCC) to store local data in functions
To illustrate the return mechanism and the stack memory allocation, I will use the following program:
#include <stdio.h>
// 1. Return via register (Standard scalar type)
int return_integer() {
return 42;
}
// 2. Return via hidden pointer (Large struct)
// The caller provides a memory address where the result should be stored
typedef struct {
long buffer[4];
} LargeData;
LargeData return_large_struct() {
LargeData ld = {{1, 2, 3, 4}};
return ld;
}
// 3. Return via multiple registers (Small struct optimization)
// On x86_64, GCC places 'first' in RAX and 'second' in RDX
typedef struct {
long first;
long second;
} PairOfLongs;
PairOfLongs return_pair() {
PairOfLongs p = {100, 200};
return p;
}
int main() {
int i = return_integer();
LargeData ld = return_large_struct();
PairOfLongs p = return_pair();
printf("Integer: %d\n", i);
printf("Large: %ld, %ld\n", ld.buffer[0], ld.buffer[3]);
printf("Pair: %ld, %ld\n", p.first, p.second);
return 0;
}
This program will be compiled with GCC and the binary generated will be disassembled by objdump using the following commands
gcc -g -fno-stack-protector -z execstack -no-pie -o a3 ./a3.c
objdump -M intel --show-raw-insn -d a3 > a3.objdump
The option ( -M intel ) forces the intel convention in the disassembling process. I prefer the intel convetion over the ATT convention.
1. Stack memory allocation
Most of the functions compiled with gcc start with the same 4 lines (see below)
00000000004011bc <main>:
4011bc: f3 0f 1e fa endbr64
4011c0: 55 push rbp
4011c1: 48 89 e5 mov rbp,rsp
4011c4: 48 83 ec 40 sub rsp,0x40
endbr64 (end branch 64 bits) is a security checkpoint used by modern INTEL and AMD processors, It is part of a hardware protection technology called Control-flow Enforcement Technology (CET). It stops "Jump-Oriented Programming" (JOP): Attackers often try to redirect the flow of a program to existing snippets of code to do things they weren't designed to do. endbr64 prevents these unauthorized redirects. It’s a hardware-level gate because it is checked directly by the processor. We will come back on this subject in another article.
push rbp save the base pointer register to the stack
mov rbp,rsp stores the value of the stack pointer register to the base pointer register. By doing this we create a stack frame.
sub rsp,0x40 allocates a certain amount of memory for the variables of the function (here 0x40 = 64 bytes). GCC computes the necessary space for all the variables declared in the function and allocates some stack memory space by changing the value of the stack pointer register (rsp). One can save some data to the stack by the push instruction ( ex push rax ) and can retrieve data from the stack by using the pop instruction ( ex pop rax). push and pop modifies the stack pointer register (rsp), push by decreasing its value, and pop by increasing its value. The stack pointer register stores 64 bits values on a 64 bit OS, 32 bits values on 32 bits OS, etc.
2. How a function compiled by GCC returns some result
Depending on the size of the results returned, GCC uses one of the following 3 mechanisms:
1) the result is returned by a register (RAX, EAX, AX, AL)
2) the result is returned by a hidden pointer
3) the result is returned on several registers (for floating pointers)
3. Return by a register
0000000000401136 <return_integer>:
401136: f3 0f 1e fa endbr64
40113a: 55 push rbp
40113b: 48 89 e5 mov rbp,rsp
40113e: b8 2a 00 00 00 mov eax,0x2a
401143: 5d pop rbp
401144: c3 ret
00000000004011bc <main>:
...
4011c8: b8 00 00 00 00 mov eax,0x0
4011cd: e8 64 ff ff ff call 401136 <return_integer>
4011d2: 89 45 fc mov DWORD PTR [rbp-0x4],eax
This is the most common return mechanism, a value is returned by a single register (here eax). Notice that even in a 64 bits OS, the value returned can be a 64 bits value, a 32 bits value, a 16 bits value or even a 8 bits value.
mov DWORD PTR [rbp-0x4],eax moves the value of the register eax (the result of the function return_integer) to a specific memory location in the stack
2. Return by a hidden pointer
0000000000401145 <return_large_struct>:
401145: f3 0f 1e fa endbr64
401149: 55 push rbp
40114a: 48 89 e5 mov rbp,rsp
40114d: 48 89 7d d8 mov QWORD PTR [rbp-0x28],rdi
401151: 48 c7 45 e0 01 00 00 mov QWORD PTR [rbp-0x20],0x1
401158: 00
401159: 48 c7 45 e8 02 00 00 mov QWORD PTR [rbp-0x18],0x2
401160: 00
401161: 48 c7 45 f0 03 00 00 mov QWORD PTR [rbp-0x10],0x3
401168: 00
401169: 48 c7 45 f8 04 00 00 mov QWORD PTR [rbp-0x8],0x4
401170: 00
401171: 48 8b 4d d8 mov rcx,QWORD PTR [rbp-0x28]
401175: 48 8b 45 e0 mov rax,QWORD PTR [rbp-0x20]
401179: 48 8b 55 e8 mov rdx,QWORD PTR [rbp-0x18]
40117d: 48 89 01 mov QWORD PTR [rcx],rax
401180: 48 89 51 08 mov QWORD PTR [rcx+0x8],rdx
401184: 48 8b 45 f0 mov rax,QWORD PTR [rbp-0x10]
401188: 48 8b 55 f8 mov rdx,QWORD PTR [rbp-0x8]
40118c: 48 89 41 10 mov QWORD PTR [rcx+0x10],rax
401190: 48 89 51 18 mov QWORD PTR [rcx+0x18],rdx
401194: 48 8b 45 d8 mov rax,QWORD PTR [rbp-0x28]
401198: 5d pop rbp
401199: c3 ret
00000000004011bc <main>:
...
4011d5: 48 8d 45 d0 lea rax,[rbp-0x30]
4011d9: 48 89 c7 mov rdi,rax
4011dc: b8 00 00 00 00 mov eax,0x0
4011e1: e8 5f ff ff ff call 401145 <return_large_struct>
Because the result returned by the function is a large structure with a size far greater than 8 bytes, the register mechanism can't be used.
see what happens before the call, with the 2 instructions lea rax,[rbp-0x30] and mov rdi,rax rdi receives rbp-0x30 which corresponds
to an address in the stack. Even if the function return_large_struct has no parameter, rdi is filled with a value which is then used in the function return_large_struct.
Using RDI as a pointer is why this mechanism is called the hidden pointer mechanism.
In this mechanism, the called function performs zero stack allocation for the result. It treats the address provided in RDI as a raw memory pointer. The called function essentially
acts as a 'data mover,' writing its local results directly into the memory frame that was already reserved by the caller. And in our example the caller has reserved data in the stack.
The only "real" instructions of the function return_large_struct are the 4 following lines
401151: 48 c7 45 e0 01 00 00 mov QWORD PTR [rbp-0x20],0x1
401158: 00
401159: 48 c7 45 e8 02 00 00 mov QWORD PTR [rbp-0x18],0x2
401160: 00
401161: 48 c7 45 f0 03 00 00 mov QWORD PTR [rbp-0x10],0x3
401168: 00
401169: 48 c7 45 f8 04 00 00 mov QWORD PTR [rbp-0x8],0x4
401170: 00
By default, objdump prioritizes human-readable assembly over raw opcode alignment. When an instruction's length causes a mismatch with the internal column formatting, it may wrap the remaining bytes to a new line, hence 00 on a new line after the instruction line. Note that the 00 bytes appearing on separate lines are not instructions, but rather the hexadecimal representation of the immediate data values encoded directly within the machine instructions.
The following lines have only one purpose: to return the information to the caller. See how the compiler uses the register rcx and the stack to store data
40117d: 48 89 01 mov QWORD PTR [rcx],rax
401180: 48 89 51 08 mov QWORD PTR [rcx+0x8],rdx
401184: 48 8b 45 f0 mov rax,QWORD PTR [rbp-0x10]
401188: 48 8b 55 f8 mov rdx,QWORD PTR [rbp-0x8]
40118c: 48 89 41 10 mov QWORD PTR [rcx+0x10],rax
401190: 48 89 51 18 mov QWORD PTR [rcx+0x18],rdx
3. Return by multiple registers
000000000040119a <return_pair>:
40119a: f3 0f 1e fa endbr64
40119e: 55 push rbp
40119f: 48 89 e5 mov rbp,rsp
4011a2: 48 c7 45 f0 64 00 00 mov QWORD PTR [rbp-0x10],0x64
4011a9: 00
4011aa: 48 c7 45 f8 c8 00 00 mov QWORD PTR [rbp-0x8],0xc8
4011b1: 00
4011b2: 48 8b 45 f0 mov rax,QWORD PTR [rbp-0x10]
4011b6: 48 8b 55 f8 mov rdx,QWORD PTR [rbp-0x8]
4011ba: 5d pop rbp
4011bb: c3 ret
00000000004011bc <main>:
...
4011e6: b8 00 00 00 00 mov eax,0x0
4011eb: e8 aa ff ff ff call 40119a <return_pair>
4011f0: 48 89 45 c0 mov QWORD PTR [rbp-0x40],rax
4011f4: 48 89 55 c8 mov QWORD PTR [rbp-0x38],rdx
Here what the compiler does is pretty straightforward, it uses the registers rax and rdx to return the values to the caller function.
Right now , we've seen the 3 main ways used by gcc to return values from a function. but if we use some compilation options , we'll see that a lot can change.
4. what happens when we change some compilation options
Now we are going to use the following command to compile
gcc -g -fno-stack-protector -z execstack -no-pie -O3 -o a3_o3 ./a3.c
objdump -M intel --show-raw-insn -d a3_o3 > a3_o3.objdump
The result is really different from what we had previously
0000000000401050 <main>:
401050: f3 0f 1e fa endbr64
401054: 48 83 ec 08 sub rsp,0x8
401058: ba 2a 00 00 00 mov edx,0x2a
40105d: bf 02 00 00 00 mov edi,0x2
401062: 31 c0 xor eax,eax
401064: 48 8d 35 99 0f 00 00 lea rsi,[rip+0xf99] # 402004 <_IO_stdin_used+0x4>
40106b: e8 d0 ff ff ff call 401040 <__printf_chk@plt>
401070: b9 04 00 00 00 mov ecx,0x4
401075: ba 01 00 00 00 mov edx,0x1
40107a: 31 c0 xor eax,eax
40107c: 48 8d 35 8e 0f 00 00 lea rsi,[rip+0xf8e] # 402011 <_IO_stdin_used+0x11>
401083: bf 02 00 00 00 mov edi,0x2
401088: e8 b3 ff ff ff call 401040 <__printf_chk@plt>
40108d: b9 c8 00 00 00 mov ecx,0xc8
401092: ba 64 00 00 00 mov edx,0x64
401097: 31 c0 xor eax,eax
401099: 48 8d 35 82 0f 00 00 lea rsi,[rip+0xf82] # 402022 <_IO_stdin_used+0x22>
4010a0: bf 02 00 00 00 mov edi,0x2
4010a5: e8 96 ff ff ff call 401040 <__printf_chk@plt>
4010aa: 31 c0 xor eax,eax
4010ac: 48 83 c4 08 add rsp,0x8
4010b0: c3 ret
4010b1: 66 2e 0f 1f 84 00 00 cs nop WORD PTR [rax+rax*1+0x0]
4010b8: 00 00 00
4010bb: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0]
...
00000000004011b0 <return_integer>:
4011b0: f3 0f 1e fa endbr64
4011b4: b8 2a 00 00 00 mov eax,0x2a
4011b9: c3 ret
4011ba: 66 0f 1f 44 00 00 nop WORD PTR [rax+rax*1+0x0]
00000000004011c0 <return_large_struct>:
4011c0: f3 0f 1e fa endbr64
4011c4: 66 0f 6f 05 74 0e 00 movdqa xmm0,XMMWORD PTR [rip+0xe74] # 402040 <_IO_stdin_used+0x40>
4011cb: 00
4011cc: 48 89 f8 mov rax,rdi
4011cf: 0f 11 07 movups XMMWORD PTR [rdi],xmm0
4011d2: 66 0f 6f 05 76 0e 00 movdqa xmm0,XMMWORD PTR [rip+0xe76] # 402050 <_IO_stdin_used+0x50>
4011d9: 00
4011da: 0f 11 47 10 movups XMMWORD PTR [rdi+0x10],xmm0
4011de: c3 ret
4011df: 90 nop
00000000004011e0 <return_pair>:
4011e0: f3 0f 1e fa endbr64
4011e4: b8 64 00 00 00 mov eax,0x64
4011e9: ba c8 00 00 00 mov edx,0xc8
4011ee: c3 ret
There is no call of the different functions in the main function because the main already contains the result of all these functions.
With optimization (-O3): The compiler realizes, "Why call ? I'll just put the instructions for return_integer right here inside main.
So if there is no call and the results of the functions have been integrated directly in the main function why do we still find the 3 functions (return_integer, return_large_struct, return_pair) in the binary ?
GCC and the linker often preserve the original function code for a few critical reasons.
1. The "Export" Requirement (Symbol Visibility)
By default, in C, functions have external linkage. This means that another object file or a shared library might technically be able to call return_integer. Even if your current main doesn't use the function anymore, GCC cannot guarantee that no one else will, so it keeps the code in the binary to ensure the symbol remains resolvable.
2. Debugging and Profiling
If you compile with debug symbols (even if you don't use -g, some remnants remain), the linker tries to maintain a map between the source code functions and the machine code. If it deleted the function entirely, tools like gdb or perf (profilers) would not be able to "find" the function, making it impossible to step through or profile the original code.
3. The "Ghost" Code Phenomenon
Think of the binary as a storage container. When you compile:
GCC generates the object code for every function defined in a3.c.
The linker then combines these. If you haven't explicitly asked the linker to "garbage collect" unused code (a feature called Dead Code Elimination), it will simply place all generated code into the .text section of the final binary.
If you want a truly minimal binary where these "orphaned" functions are physically removed, you can use linker flags that perform Dead Code Elimination. You can add these flags to your compilation command:
gcc -O3 -ffunction-sections -fdata-sections -Wl,--gc-sections a3.c -o a3_min
5. Epilogue
By disassembling the binary (with objdump), one precisely see what gcc creates during the compilation process. In this article, I explained the stack allocation mechanism and the way GCC returns a result from a function and I showed that some compilations options could really change the binary code.
6. Disasm of the binary generated by GCC
Below you find the importants parts of the disassembly of the binary generated by gcc:
0000000000401136 <return_integer>:
401136: f3 0f 1e fa endbr64
40113a: 55 push rbp
40113b: 48 89 e5 mov rbp,rsp
40113e: b8 2a 00 00 00 mov eax,0x2a
401143: 5d pop rbp
401144: c3 ret
0000000000401145 <return_large_struct>:
401145: f3 0f 1e fa endbr64
401149: 55 push rbp
40114a: 48 89 e5 mov rbp,rsp
40114d: 48 89 7d d8 mov QWORD PTR [rbp-0x28],rdi
401151: 48 c7 45 e0 01 00 00 mov QWORD PTR [rbp-0x20],0x1
401158: 00
401159: 48 c7 45 e8 02 00 00 mov QWORD PTR [rbp-0x18],0x2
401160: 00
401161: 48 c7 45 f0 03 00 00 mov QWORD PTR [rbp-0x10],0x3
401168: 00
401169: 48 c7 45 f8 04 00 00 mov QWORD PTR [rbp-0x8],0x4
401170: 00
401171: 48 8b 4d d8 mov rcx,QWORD PTR [rbp-0x28]
401175: 48 8b 45 e0 mov rax,QWORD PTR [rbp-0x20]
401179: 48 8b 55 e8 mov rdx,QWORD PTR [rbp-0x18]
40117d: 48 89 01 mov QWORD PTR [rcx],rax
401180: 48 89 51 08 mov QWORD PTR [rcx+0x8],rdx
401184: 48 8b 45 f0 mov rax,QWORD PTR [rbp-0x10]
401188: 48 8b 55 f8 mov rdx,QWORD PTR [rbp-0x8]
40118c: 48 89 41 10 mov QWORD PTR [rcx+0x10],rax
401190: 48 89 51 18 mov QWORD PTR [rcx+0x18],rdx
401194: 48 8b 45 d8 mov rax,QWORD PTR [rbp-0x28]
401198: 5d pop rbp
401199: c3 ret
000000000040119a <return_pair>:
40119a: f3 0f 1e fa endbr64
40119e: 55 push rbp
40119f: 48 89 e5 mov rbp,rsp
4011a2: 48 c7 45 f0 64 00 00 mov QWORD PTR [rbp-0x10],0x64
4011a9: 00
4011aa: 48 c7 45 f8 c8 00 00 mov QWORD PTR [rbp-0x8],0xc8
4011b1: 00
4011b2: 48 8b 45 f0 mov rax,QWORD PTR [rbp-0x10]
4011b6: 48 8b 55 f8 mov rdx,QWORD PTR [rbp-0x8]
4011ba: 5d pop rbp
4011bb: c3 ret
00000000004011bc <main>:
4011bc: f3 0f 1e fa endbr64
4011c0: 55 push rbp
4011c1: 48 89 e5 mov rbp,rsp
4011c4: 48 83 ec 40 sub rsp,0x40
4011c8: b8 00 00 00 00 mov eax,0x0
4011cd: e8 64 ff ff ff call 401136 <return_integer>
4011d2: 89 45 fc mov DWORD PTR [rbp-0x4],eax
4011d5: 48 8d 45 d0 lea rax,[rbp-0x30]
4011d9: 48 89 c7 mov rdi,rax
4011dc: b8 00 00 00 00 mov eax,0x0
4011e1: e8 5f ff ff ff call 401145 <return_large_struct>
4011e6: b8 00 00 00 00 mov eax,0x0
4011eb: e8 aa ff ff ff call 40119a <return_pair>
4011f0: 48 89 45 c0 mov QWORD PTR [rbp-0x40],rax
4011f4: 48 89 55 c8 mov QWORD PTR [rbp-0x38],rdx
4011f8: 8b 45 fc mov eax,DWORD PTR [rbp-0x4]
4011fb: 89 c6 mov esi,eax
4011fd: 48 8d 05 00 0e 00 00 lea rax,[rip+0xe00] # 402004 <_IO_stdin_used+0x4>
401204: 48 89 c7 mov rdi,rax
401207: b8 00 00 00 00 mov eax,0x0
40120c: e8 2f fe ff ff call 401040 <printf@plt>
401211: 48 8b 55 e8 mov rdx,QWORD PTR [rbp-0x18]
401215: 48 8b 45 d0 mov rax,QWORD PTR [rbp-0x30]
401219: 48 89 c6 mov rsi,rax
40121c: 48 8d 05 ee 0d 00 00 lea rax,[rip+0xdee] # 402011 <_IO_stdin_used+0x11>
401223: 48 89 c7 mov rdi,rax
401226: b8 00 00 00 00 mov eax,0x0
40122b: e8 10 fe ff ff call 401040 <printf@plt>
401230: 48 8b 55 c8 mov rdx,QWORD PTR [rbp-0x38]
401234: 48 8b 45 c0 mov rax,QWORD PTR [rbp-0x40]
401238: 48 89 c6 mov rsi,rax
40123b: 48 8d 05 e0 0d 00 00 lea rax,[rip+0xde0] # 402022 <_IO_stdin_used+0x22>
401242: 48 89 c7 mov rdi,rax
401245: b8 00 00 00 00 mov eax,0x0
40124a: e8 f1 fd ff ff call 401040 <printf@plt>
40124f: b8 00 00 00 00 mov eax,0x0
401254: c9 leave
401255: c3 ret
Top comments (0)