Bypassing NX: How Return-Oriented Programming (ROP) Works

Stack buffer overflows have been understood since the 1980s, and the defenses against the simplest variant — injecting shellcode and jumping to it — have been standard for over a decade. Mark the stack non-executable, and any code an attacker writes there will trigger a fault the moment the CPU tries to fetch it as an instruction.

Return-Oriented Programming (ROP) is the answer to that defense. Instead of injecting code, an attacker reuses code that’s already in memory — the program’s own compiled instructions, or those in loaded libraries. No new bytes need to be executable. The technique is general enough that arbitrary computation is possible using only these pre-existing fragments.

This article explains ROP from first principles: what gadgets are, how chains are constructed, what the x86_64 calling convention has to do with it, and where the technique breaks down. If you’ve read the previous article on stack buffer overflows, the material picks up directly where that left off. If not, a brief recap of the prerequisites follows.

Prerequisites: What ROP Builds On

Buffer overflows and the return address: When a function returns, the ret instruction pops an 8-byte return address off the stack and loads it into RIP (the instruction pointer). If an attacker can overwrite stack memory past a buffer’s bounds — via strcpy, gets, or similar unchecked copy functions — they can overwrite that return address and redirect execution.

NX / W^X: Modern systems mark the stack (and heap) as non-executable at the page table level using the NX bit (bit 63 of the x86_64 Page Table Entry). If RIP points into a non-executable page, the CPU raises a #PF page fault before fetching any instruction bytes. Shellcode injected onto the stack can’t execute. The program’s .text segment and loaded libraries must remain executable — they’re where the actual code lives.

ROP’s insight: the .text segment is full of instructions. If you can redirect RIP into the middle of existing code sequences and control the stack, you can chain together small instruction fragments to do whatever you want, all within executable memory.

What Is a Gadget?

A gadget is a short sequence of instructions ending in ret. The simplest useful example:

pop rdi
ret

To understand why this matters, look at what ret actually does in x86_64. The opcode 0xC3 is equivalent to:

RIP ← [RSP]     ; load 8 bytes at stack pointer into instruction pointer
RSP ← RSP + 8   ; advance stack pointer

Every ret instruction reads the next address from the stack and jumps there. If the attacker controls the stack contents, they control where every ret goes. This is the core of ROP: the stack pointer becomes the program counter.

By placing a sequence of gadget addresses on the stack, an attacker creates a chain. Each gadget runs its few instructions, hits ret, and the CPU fetches the next gadget address from the stack. Execution threads through memory jumping from gadget to gadget, doing useful work at each stop.

Unintended Gadgets

x86 uses variable-length encoding — instructions range from 1 to 15 bytes. This means you can find valid instruction sequences by reading from an offset into an existing instruction. The bytes were put there by the compiler for a different purpose, but the CPU has no concept of “instruction boundaries” in memory. It decodes whatever bytes it finds.

For example, if a function contains the byte sequence 48 89 c7 c3 at some offset, reading from offset +2 gives c7 c3, which decodes as mov edi, ... followed by ret on some interpretations — a gadget the compiler never intended to create. Tools like ROPgadget and ropper scan executable segments for all such sequences automatically.

The x86_64 Calling Convention Problem

On 32-bit x86, function arguments were passed on the stack. Calling system("/bin/sh") meant placing the string pointer at a specific stack offset — straightforward to fake with a crafted overflow payload.

The System V AMD64 ABI, used on 64-bit Linux, passes the first six arguments in registers: rdi, rsi, rdx, rcx, r8, r9. Only additional arguments spill to the stack.

This means calling system("/bin/sh") requires:

1. A pointer to the string /bin/sh loaded into rdi
2. A jump to system()

A direct ret2libc — overwriting the return address with system()’s address — fails because rdi contains whatever garbage was there at the time of ret. You need a gadget to set it first.

Building a ROP Chain: Step by Step

1. Find the /bin/sh String

The string /bin/sh is present in libc — you can verify this with:

strings -tx /lib/x86_64-linux-gnu/libc.so.6 | grep "/bin/sh"
# 1d8698 /bin/sh

The offset 0x1d8698 from libc’s load base gives you the runtime address once you know where libc is loaded (more on that shortly).

2. Find system() in libc

readelf -s /lib/x86_64-linux-gnu/libc.so.6 | grep " system@"
# 0000000000050d70  system@@GLIBC_2.2.5

Offset 0x50d70 from libc base.

3. Find a pop rdi; ret Gadget

ROPgadget --binary ./vuln --search "pop rdi"
# 0x0000000000401233 : pop rdi ; ret

If the binary isn’t PIE (not compiled with -pie -fPIE), this address is fixed across every execution — ASLR doesn’t randomize the main binary’s .text segment without PIE. Gadgets in the binary itself are stable; gadgets in libc are randomized unless you have a leak.

4. Stack Alignment

Before calling any function through a ROP chain, rsp must be 16-byte aligned at the point of the call instruction’s equivalent. Many functions use SSE instructions like movaps internally, which fault on unaligned stack access. If a ROP chain crashes inside system() rather than before it, misalignment is usually the cause.

The fix: insert a bare ret gadget (0xc3 at any convenient executable address) as a one-instruction pad before the call target. A single ret advances rsp by 8, correcting 8-byte misalignment.

[ pop rdi ; ret  ]   ← gadget address
[ &"/bin/sh"     ]   ← argument loaded into rdi by pop
[ ret            ]   ← alignment pad (advances rsp by 8, fixes 16-byte alignment)
[ &system()      ]   ← jumped to by the alignment pad's ret

5. The Stack Layout

Assuming a 16-byte buffer followed by 8 bytes of saved rbp, the return address sits at offset 24:

6. Execution Trace

1. Vulnerable function’s ret fires. Pops 0x401233 into RIP. RSP → +32.
2. CPU executes pop rdi at 0x401233. Loads /bin/sh address into rdi. RSP → +40.
3. CPU executes ret at 0x401235. Pops alignment pad address into RIP. RSP → +48.
4. Alignment pad ret fires. Pops system() address into RIP. RSP → +56.
5. system() executes with rdi pointing to /bin/sh. A shell starts under the current process’s user ID — not root, unless the binary is SUID.

The ASLR Problem

The chain above uses hardcoded libc addresses. In practice, ASLR randomizes libc’s load base at every execution, making those offsets wrong each run.

Two important qualifications:

PIE vs. no-PIE: ASLR randomizes the stack, heap, and libraries. It does not randomize the main binary’s load address unless the binary is compiled as Position Independent (-pie -fPIE). Without PIE, gadgets in the binary’s own .text section have fixed addresses. This is why CTF challenges often use binaries compiled without PIE — the gadgets are stable even with ASLR on.

Bypassing ASLR requires a leak: To use libc gadgets under ASLR, you need to know libc’s base address at runtime. This requires a separate vulnerability — an information leak — that reads a pointer from a known location and reveals where memory was mapped. Common targets:

• GOT (Global Offset Table): After the first call to a libc function (e.g., puts), the GOT entry for that function holds the resolved runtime address. Reading a GOT entry with a format string vulnerability or an arbitrary read primitive gives you a libc address, from which you can compute all other offsets.
• /proc/self/maps: Readable by the process itself; exposes exact memory maps but requires a file read primitive.

The typical two-stage attack: stage 1 leaks a GOT entry to determine libc base; stage 2 uses the computed libc base to build the final ROP chain and trigger the shell.

Other Mitigations ROP Faces

Full RELRO (Relocation Read-Only): Marks the GOT read-only after program startup. This prevents GOT overwrite attacks and makes the GOT leak-only — you can still read it, but can’t write fake addresses into it.

Stack canaries: Still relevant. A canary between local variables and the return address will fire before ret if the overflow path crosses it. ROP is only useful if you can reach the return address — many overflow paths must go through the canary.

SafeStack / ShadowCallStack: Compiler mitigations (Clang’s SafeStack, ARM’s Shadow Call Stack) maintain a separate, protected stack for return addresses. An overflow of the regular stack can’t reach the shadow stack. These make ROP significantly harder but aren’t universally deployed.

Control Flow Integrity (CFI): Compiler or hardware mechanism that validates indirect branches against a pre-computed set of allowed targets. Coarse-grained CFI (checking that an indirect call targets a valid function start) is widely deployed. Fine-grained CFI that restricts ret targets is less common but exists. Intel CET (Control-flow Enforcement Technology) adds hardware shadow stack support for ret protection.

Practical Tools

ROPgadget: Scans a binary for gadgets and can automatically attempt to build simple chains.

ROPgadget --binary ./vuln --rop --badbytes 0a

pwntools (Python): The standard library for writing exploit scripts. Handles offset calculation, packing addresses, gadget lookup, and interaction with the target process.

from pwn import *

elf    = ELF('./vuln')
libc   = ELF('/lib/x86_64-linux-gnu/libc.so.6')
proc   = process('./vuln')

pop_rdi = 0x401233                   # from ROPgadget, stable without PIE
bin_sh  = libc.address + 0x1d8698   # requires known libc base
system  = libc.address + 0x50d70
ret_pad = 0x40101a                   # bare ret for alignment

payload  = b'A' * 24                 # 16 buffer + 8 saved rbp
payload += p64(pop_rdi)
payload += p64(bin_sh)
payload += p64(ret_pad)
payload += p64(system)

proc.sendline(payload)
proc.interactive()

ropper: Alternative gadget finder with a richer query interface.

checksec: Inspect a binary’s enabled mitigations before spending time on a chain that’s blocked by full RELRO or CET.

checksec --file=./vuln

ROP is a good example of why security mitigations have to be understood in combination rather than in isolation. NX closes one attack surface; ASLR raises the bar on another; stack canaries block a specific exploitation path. Each was a meaningful improvement when introduced. The response to each was a technique that worked around it, often by exploiting the feature’s own assumptions. Understanding each layer — what it protects, what it assumes, what defeats it — is how you evaluate whether a given system is actually hardened or just has the right flags set.