PLEASE NOTE: Despite the provocative title, this article is strictly an educational deep-dive into ransomware internals and memory forensics. I’m showing you how the tech works under the hood—I am NOT telling you to develop ransomware or do anything illegal.
In this blog post, we’re going to explore Compiler Optimizations—what they are, how they work, and why sometimes they might not be as helpful as you think for your code.
So, let’s start with the basics: what exactly is compiler optimization, and how does it work?
note : we use
-O3 in this article
_
1 ~ what in the world is compiler optimization ?
in the design theory of it the fellow developers noted that the main goal of CO , is to make the code " more efficient, without changing its original functionality " and of course the primary goals are typically to improve execution speed, reduce memory usage, minimize code size, or lower power consumption.
now lets take a look at how do they work
2 ~ compilation pipeline
OK cool now you understand how does the pipeline look like , but i focused at the ir (intermediate representation) optimization specifically (DEAD STORAGE ELIMINATION) since its the main topic of this blog .
3 ~ taking a look at how does the compiler work with the -O3 flag and what affects does it do on the code .
ok , lets see this through a disassembler to understand whats really happening .
take this example first :
#include <stdio.h>
#include <string.h>
int secret(int *ptr) {
return *ptr;
}
int main() {
int data = 42;
int *y = &data;
int val = secret(y);
memset(&data, 0, sizeof(data));
//printf("value was: %d, and now after memset: %d\n", val, data);
return 0;
}
gef➤ disas main
Dump of assembler code for function main:
0x0000000000001040 <+0>: endbr64
0x0000000000001044 <+4>: xor eax,eax
0x0000000000001046 <+6>: ret
End of assembler dump.
gef➤
as you see nothing if this asm was code it will be just a return 0; as you see it xor eax to return zero and ret .
because there is no output from this program
first lets see the Dead Store Elimination (DSE)
This targets writes to memory.
The "Store": memset(&data, 0, ...) is a command to write zeros to memory.
Why it's "Dead": Because the variable data is never being used again after that line. The compiler sees you writing a value that will never be used by anyone.
Result: The compiler deletes the memset call
entirely.
now lets take a look at Dead Code Elimination (DCE)
This targets computations and logic that don't affect the program's output.
The "Code": The variable val = secret(y) and the initialization int data = 42.
Why it's "Dead": Once we commented out the printf, the variable val became useless. It doesn't influence the return value of main (which is hardcoded to 0), and it doesn't trigger any I/O.
Result: The compiler deletes the secret function call and the int data variable itself.
\
what happens if there was a printf ? :
something very interesting happens , fr your going to be surprised by how good are now day compilers in optimizing code
\
now lets check it with a printf
#include <stdio.h>
#include <string.h>
int secret(int *ptr) {
return *ptr;
}
int main() {
int data = 42;
int *y = &data;
int val = secret(y);
memset(&data, 0, sizeof(data));
printf("value was: %d, and now after memset: %d\n", val, data);
return 0;
}
0x0000000000001060 <+0>: endbr64
0x0000000000001064 <+4>: sub rsp,0x8
0x0000000000001068 <+8>: xor ecx,ecx
0x000000000000106a <+10>: mov edx,0x2a
0x000000000000106f <+15>: mov edi,0x2
0x0000000000001074 <+20>: lea rsi,[rip+0xf8d] # 0x2008
0x000000000000107b <+27>: xor eax,eax
0x000000000000107d <+29>: call 0x1050 <__printf_chk@plt>
0x0000000000001082 <+34>: xor eax,eax
0x0000000000001084 <+36>: add rsp,0x8
0x0000000000001088 <+40>: ret
****
the optimized assembly shows that the compiler removed all of that code. Instead of actually executing those functions at runtime, it realized that secret(y) always returns 42 and that data is only used in the subsequent printf. Since memset(&data, 0, sizeof(data)) simply sets data to zero and nothing else uses it, the compiler replaced the values directly in the printf call. In other words, the instructions for assigning 42, calling secret(), and zeroing data are gone — everything is hardcoded constants passed straight to printf .
thats amazing tbh , cool asl .
But how can some compiler optimization cause some security issues ?
4 ~ security issues !!
#include <string.h>
#include <stdio.h>
void process_login() {
char password[64];
strcpy(password, "_thx_for_reading");
printf("عيد مبارك \n");
memset(password, 0, sizeof(password));
}
int main() {
process_login();
return 0;
}
the problem with this code if it has been compiled with optimizations the memset that zeros out the buffer in the memory will just be optimized under the rule that nothing after the memset touches the buffer again .
whats the impact ? :
1 : If our program has a separate bug (like a buffer over-read), an attacker can trick the program into sending back more data than it should. Since the password was never wiped, it's just sitting in the "garbage" area of the RAM, ready to be leaked in a response.
2 : core dumps and memory analysis
3 : If another function runs right after process_login, it will re-use the same stack space.
If that second function has a vulnerability (like a "format string" bug), an attacker can peek at the local variables of that function.
Because the memory wasn't cleared, they aren't just seeing the new function's data—they are seeing the leftovers of our password.
see you in part 2 .... : )
Top comments (0)