Smashing the Stack: A Deep Dive into Buffer Overflows

The C language has no runtime bounds checking. Declare a 16-byte array, write 100 bytes into it, and the CPU will comply without complaint — overwriting whatever happens to occupy the adjacent memory. In most cases that means a crash. In specific, controllable cases, it means an attacker chooses what the CPU executes next.

Buffer overflows have been understood since the Morris Worm in 1988. They remain relevant today not because the underlying bug is subtle — it isn’t — but because the techniques required to exploit them against modern mitigations have become increasingly sophisticated. Understanding the basic mechanism is the prerequisite for understanding everything that came after.

The Stack Frame

When a function is called, the compiler arranges a stack frame: a region of the stack holding the function’s local variables, saved registers, and the address the CPU should return to when the function completes.

On x86_64, the stack grows downward — from higher addresses toward lower. Local variables are allocated by subtracting from the stack pointer. Buffers, however, are written to from lower addresses upward. This directional opposition is what makes overflows dangerous: writing past the end of a buffer writes toward the saved return address.

A typical x86_64 stack frame layout after the function prologue (with frame pointer, compiled without optimization):

High addresses
┌──────────────────────────────┐
│  ... caller's stack frame    │
├──────────────────────────────┤  ← rbp + 16
│  Return Address (8 bytes)    │  ← the address of the instruction after call
├──────────────────────────────┤  ← rbp + 8  (wait — see note below)
│  Saved RBP (8 bytes)         │  ← rbp
├──────────────────────────────┤
│  Local buffer[16] (16 bytes) │  ← rbp - 16
├──────────────────────────────┤
│  ... more locals if any      │  ← rsp
Low addresses

One clarification on what “allocates the stack frame”: the OS determines the total size and base address of the stack segment for the process. The frame itself is created by the function prologue — compiler-generated instructions that run at function entry — not by the OS. The OS has no visibility into individual function calls.

Modern compilers also don’t always use a frame pointer. With -O2 or higher, GCC omits rbp as a frame pointer (it becomes a general-purpose register), making the frame layout flatter but harder to read in a debugger. For this walkthrough, we assume -fno-omit-frame-pointer.

The Vulnerable Code

#include <stdio.h>
#include <string.h>

void check_authentication(char *input) {
    char buffer[16];

    /*
     * strcpy copies bytes from input until it hits a null byte.
     * It has no knowledge of the destination buffer's size.
     * Other functions with the same problem: gets(), scanf("%s"),
     * sprintf() with an unbounded format string.
     */
    strcpy(buffer, input);

    printf("Access Denied.\n");
}

int main(int argc, char **argv) {
    if (argc > 1) {
        check_authentication(argv[1]);
    }
    return 0;
}

The safe alternative isn’t strncpy — it doesn’t null-terminate when the source is longer than the limit, which trades an overflow for a read of uninitialized memory. Use snprintf(buffer, sizeof(buffer), "%s", input) or strlcpy (available on BSD/macOS and as a glibc extension) instead.

The Assembly: Prologue and Epilogue

What the compiler emits for check_authentication (AT&T syntax, simplified):

Prologue (executed at function entry):

push   rbp          ; save caller's base pointer onto the stack
mov    rbp, rsp     ; set current stack top as new frame base
sub    rsp, 16      ; allocate 16 bytes for buffer[]

Epilogue (executed at function exit):

leave               ; equivalent to: mov rsp, rbp; pop rbp
ret                 ; pop 8 bytes from stack top into RIP and jump there

The ret instruction trusts whatever 8 bytes sit at the current stack pointer to be the legitimate return address. It doesn’t validate them. If those bytes have been overwritten, the CPU jumps to wherever they point.

Reproducing the Overflow

Modern compilers enable several mitigations by default. To observe the raw overflow behavior, compile with protections disabled:

gcc -o vuln vuln.c \
    -fno-stack-protector \   # disable stack canaries
    -no-pie \                # disable position-independent executable
    -z execstack \           # mark stack as executable (for shellcode demo)
    -g                       # include debug symbols for GDB

Run with a 24-byte input to confirm it reaches the return address:

python3 -c "import sys; sys.stdout.buffer.write(b'A'*24 + b'B'*8)" | \
    xargs ./vuln
# Segmentation fault (core dumped)

In GDB, you can watch the overwrite happen:

gdb ./vuln
(gdb) run $(python3 -c "print('A'*24 + 'B'*8)")
(gdb) info registers rip
# rip: 0x4242424242424242  ← 'BBBBBBBB' in hex

The stack layout at the time of ret:

• Bytes 0–15: buffer[16]
• Bytes 16–23: saved rbp (8 bytes)
• Bytes 24–31: return address — overwritten with 0x4242424242424242

Total distance from buffer start to return address: 16 + 8 = 24 bytes. This is the offset — the exact number of padding bytes needed before the address you want to inject.

Shellcode Injection

With the return address under control, the question becomes: what address do you inject, and what’s there?

The classic technique places machine code directly in the buffer, then overwrites the return address to point back into it. The payload structure:

[ NOP sled ] [ shellcode ] [ padding ] [ &buffer on stack ]

NOP sled (0x90): The x86 NOP instruction does nothing and advances to the next byte. Prepending shellcode with a sequence of NOPs creates a landing zone — if the injected address points anywhere into the sled, execution slides into the shellcode. This compensates for slight stack address variation between runs.

Shellcode: Raw machine bytes that exec /bin/sh or open a reverse shell connection. A minimal execve("/bin/sh") shellcode on x86_64 is around 27 bytes.

Target address: The approximate address of the NOP sled on the stack — found by examining $rsp in GDB when the buffer is in scope.

One common misconception: this gives you a shell running as the current user, not root. Privilege escalation requires a separate step — typically chaining this into a SUID binary or a kernel vulnerability.

Modern Mitigations

Compiling without protections is necessary for the demo above but not representative of what you’d face on a production system. Three mitigations are standard:

Stack Canaries (-fstack-protector-all): The compiler inserts a random value (the canary) between the local buffer and the saved rbp, then checks it before executing ret. If the overflow has modified it, the program calls __stack_chk_fail() and aborts. The canary value is read from _dl_random at process startup — random per execution but not cryptographically generated in the sense of a CSPRNG output.

NX / W^X (-z noexecstack, default on most Linux systems): The stack’s page table entries have the NX (No-Execute) bit set. Jumping to shellcode on the stack triggers a fault before any shellcode byte executes. This breaks the classic injection technique entirely.

ASLR + PIE: ASLR randomizes the base addresses of the stack, heap, and loaded libraries at process start. But without PIE (Position-Independent Executable), the main binary itself loads at a fixed address — making ASLR only partially effective. Compiling with -pie -fPIE randomizes the binary’s own load address. Together, ASLR + PIE means you can’t hardcode addresses for any part of the process memory.

These three together close the door on the technique described above. The return address you’d want to inject is at an unknown location, the stack isn’t executable, and the canary will fire before ret if you’ve overwritten through it.

Return-Oriented Programming (ROP)

Modern exploits work around NX by not injecting any code at all. Instead, they hijack the return address to point to gadgets: short sequences of legitimate instructions already present in the binary or its loaded libraries, each ending in ret.

By chaining gadgets, an attacker can construct arbitrary computation using code that was already marked executable — so NX provides no defense. A minimal example:

gadget 1: pop rdi; ret     → load a value into rdi (first argument register)
gadget 2: <address of "/bin/sh" string in libc>
gadget 3: ret2libc         → call system() or execve() in libc

The attack called ret2libc was the first widely understood ROP variant: overwrite the return address to jump directly into system() in libc (which is executable, non-randomized without ASLR, and already loaded). Pass it the address of the string /bin/sh (also in libc) as an argument. No shellcode, no NX violation.

Against full ASLR + PIE, ROP requires an additional information leak — a separate vulnerability that reveals a runtime address, from which the attacker can compute the positions of gadgets. This is why modern exploitation is almost never a single bug: it’s a chain where one vulnerability provides a leak and another provides control flow hijack.

Tools like ROPgadget and pwntools automate gadget discovery and chain construction. Understanding the stack frame mechanics above is the prerequisite for understanding why those tools work.