Heap Buffer Overflow and Allocator Exploitation

Definition

A heap buffer overflow writes past the end of a heap allocation, corrupting whatever lies adjacent — a neighboring object's contents or the allocator's own chunk metadata. Allocator exploitation is the craft of converting that corruption (or a related free-list corruption) into a controlled allocation primitive: making malloc hand back a pointer the attacker chose, which becomes an arbitrary write. This is class 2 of the six memory-corruption bug classes, and unlike a stack overflow it has no canary in its path — the heap's defenses live in the allocator, not the frame.

Why it matters

Heap overflows are a leading source of remote code execution in servers, browsers, and media/file parsers, and the gateway to the "House of *" allocator-exploitation lineage that defines CTF and real-world userland pwning. Three transferable lessons:

The target is the neighbor, not the buffer. A heap overflow is only interesting because of what sits next to the overflowed chunk — an adjacent object's function pointer, or the allocator's size/free-list metadata. Exploitation is a layout problem (heap grooming) before it is a write problem.
The allocator is a programmable machine. glibc's ptmalloc, Windows' segment heap, and jemalloc each maintain free lists and chunk metadata in-band. Corrupt that metadata and the allocator becomes a weird machine: a few malloc/free calls return overlapping or arbitrary chunks. Mastering one allocator's internals is the real skill; the overflow is just the way in.
Version is everything. glibc 2.31 vs 2.34 vs 2.39 differ enormously: __malloc_hook/__free_hook (the classic write targets) were removed in 2.34; tcache "safe-linking" landed in 2.32. An exploit is written against a specific allocator version, and so is its defense. This is why "heap exploitation" is really "this-allocator-this-version exploitation."

The defender pair is exploit-mitigations; the structural cure is bounds-checked containers and memory-safe languages.

How it works

Heap allocators carve memory into chunks. In glibc each chunk has an in-band header (prev_size, size with low flag bits) followed by user data; when a chunk is freed, the allocator reuses the start of its user area to store free-list pointers (fd/bk). The exploit chain:

Layout. Groom the heap so an overflowable chunk sits immediately before a chosen victim — either an object with a code pointer/length, or a free chunk whose metadata you want.
Overflow. Write past the source chunk's user data into the next chunk's header (size/flags) or, if it is free, its fd/bk pointers.
Corrupt. Either overwrite the victim object's contents directly, or forge allocator metadata (a fake size, a poisoned free-list next).
Trigger. Call malloc/free so the allocator consumes the corrupted metadata and links an attacker-chosen address into a bin.
Allocate-to-arbitrary. A later malloc of that size returns the attacker-chosen address. Writing to that "allocation" is an arbitrary write — point it at a GOT entry, a hook (glibc ≤ 2.33), an exit/FILE handler, or an app function pointer → control flow.

Adjacent-object overwrite, the simplest case, needs no metadata trickery:

struct obj { char name[32]; void (*action)(void); };
struct obj *a = malloc(sizeof *a);     // chunk A
struct obj *b = malloc(sizeof *b);     // chunk B, allocated just after A
strcpy(a->name, attacker_string);      // no bounds check: overflows A into B->action
b->action();                           // calls attacker-controlled pointer

The metadata case — glibc tcache poisoning — is the modern workhorse, sketched:

p = malloc(0x20); q = malloc(0x20)
free(q); free(p)                 # tcache[0x20] freelist:  p -> q
<overflow or UAF overwrites p's fd (tcache next) with &target>   # safe-linking aware
malloc(0x20)  -> p
malloc(0x20)  -> target          # allocator returns an attacker-chosen address
*target = value                  # arbitrary write primitive

The bug is the overflow; the exploit is the allocator faithfully obeying corrupted metadata. The target is almost never the overflowed buffer — it is the chunk or free-list entry beside it.

Techniques / patterns

Fingerprint the allocator and version first. glibc (ptmalloc), Windows segment heap, jemalloc (Android/FreeBSD), tcmalloc/PartitionAlloc (Chrome), kernel SLUB — each has a different metadata layout and technique set. On glibc, the exact 2.x version selects which primitives survive.
Groom before you overflow. Allocate/free to force the victim object (one carrying a code pointer or a length field) immediately adjacent to the overflowable buffer.
Prefer overwriting a length/size field. Turning a small bounded overflow into a corrupted length converts a relative write into an absolute, larger read/write — a far stronger primitive than the original overflow.
Leak before you write. Allocator exploitation almost always needs a heap and libc base leak first to defeat ASLR; an info-leak primitive (OOB read) usually precedes the write.
Know the write targets per version. GOT entries, __free_hook/__malloc_hook (glibc ≤ 2.33), __exit_funcs/TLS destructors, and File-Stream-Oriented-Programming (_IO_FILE vtables) for post-hook glibc.
tcache is the first stop. On modern glibc the tcache (per-thread cache) has the weakest checks and the most reliable reuse — start there before fastbin/unsorted/largebin.

Variants and bypasses

Allocator exploitation spans 5 families — one allocator-agnostic, three glibc-internal, two other allocator worlds.

1. Adjacent-object overwrite (allocator-agnostic)

Overflow into the contents of the next chunk — a function pointer, object pointer, or length field. No metadata forgery; works on any allocator. The simplest and often most reliable primitive.

2. glibc tcache poisoning

Corrupt a tcache free-list next pointer (via overflow or UAF) so the next malloc of that size returns an attacker-chosen address. The dominant modern glibc primitive. Safe-linking (glibc 2.32+) XOR-encodes next with chunk_addr >> 12, so a naive overwrite needs a heap-address leak first.

3. glibc fastbin / unsorted / largebin attacks ("House of *")

The classic lineage: forge a fastbin fd to return a fake chunk (House of Spirit), abuse the unsorted-bin bk to write a large pointer to a target (unsorted-bin attack), or corrupt largebin bk_nextsize for a controlled write (largebin attack). Mitigated piecemeal by size/alignment and bin integrity checks across glibc versions.

4. Windows segment heap and LFH

Windows 8.1+ default. The Low-Fragmentation-Heap front-end plus segment-heap back-end use _HEAP_ENTRY metadata with header encoding/cookies and LFH bucket randomization. Exploitation favors adjacent-object overwrite and LFH "bucket spray" over metadata forgery, since the metadata is encoded.

5. jemalloc (Android, FreeBSD)

Region/run/chunk structure with mostly out-of-line metadata. Exploitation targets adjacent regions within a run and the occasional in-line metadata — central to Android browser and media-codec exploitation.

Impact

Ordered by typical severity:

Remote code execution. Heap overflow in a network service, browser, or media/file parser reachable by attacker content — a leading RCE class.
Local privilege escalation. Heap overflow in a setuid binary or kernel allocator (SLUB) yielding privileged execution.
Sandbox escape. Heap corruption in a broker/IPC process to cross an isolation boundary; a chain link.
Arbitrary read/write primitive. Allocate-to-arbitrary or a corrupted length field yields controlled memory access — the engine of the rest of the chain.
Information disclosure. Over-reading an adjacent chunk (or a corrupted length) leaks heap/libc pointers that defeat ASLR.

Detection and defense

Ordered by effectiveness:

Memory-safe languages and bounds-checked containers. Rust slices, std::vector::at, std::span with bounds checks, and std::string (vs strcpy) remove the overflow at the source. The structural fix.
AddressSanitizer in CI. ASan surrounds heap allocations with poisoned redzones, so an overflow traps at the offending write with allocation + access stacks — not at a later allocator crash. The highest-leverage control for unavoidable C/C++.
Hardened allocators. glibc tcache integrity checks + safe-linking; Chrome PartitionAlloc (per-type partitions, guard pages, freelist hardening); Windows segment-heap metadata encoding; GWP-ASan and page-heap/guard-page allocators that put each allocation on its own page so an overflow faults immediately. These raise cost without fixing the bug.
Coverage-guided fuzzing. libFuzzer/AFL++/OSS-Fuzz on parsers find heap overflows daily — the modal way real heap bugs are discovered before ship.
Compiler mitigations. -D_FORTIFY_SOURCE=3 catches a subset of overflowable mem*/str* calls with known sizes; heap cookies detect some metadata tampering at free time.
Hardware memory tagging. ARM MTE assigns adjacent allocations different tags, so an overflow that crosses into the next granule traps on the tag mismatch.
Runtime / EDR telemetry. The post-overflow chain (heap spray, GOT/hook overwrite, anomalous control-flow) leaves EDR-visible signals.

What does not work as a primary defense

Stack canaries. The heap has no canary in the write path; SSP is irrelevant to heap overflows.
ASLR alone. Defeated by an info leak, which heap exploits routinely obtain first. ASLR raises the bar; it does not close the class.
Removing one write target (e.g., __malloc_hook in glibc 2.34). Closes that target, not the primitive; FSOP, exit handlers, and GOT remain. The allocate-to-arbitrary write just retargets.
_FORTIFY_SOURCE as a guarantee. It only covers calls where the compiler knows the destination size; dynamic-size copies and custom loops slip through.

Practical labs

Run only against owned lab environments or authorized engagements.

Catch a heap overflow with AddressSanitizer

cat > heap.c <<'EOF'
#include <stdlib.h>
#include <string.h>
int main(void) {
    char *a = malloc(32);
    char *b = malloc(32);          // adjacent victim
    strcpy(a, "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA_overflow_into_b");  // > 32
    return b[0];
}
EOF
gcc -fsanitize=address -g -o heap heap.c && ./heap
# Expected: ASan "heap-buffer-overflow" with the write location and both allocations.

Confirm the allocator version your primitives depend on

ldd --version | head -1
# glibc 2.31 (Ubuntu 20.04) vs 2.35 (22.04) vs 2.39 (24.04) change which heap
# techniques work and which write targets (__malloc_hook etc.) still exist.

Inspect live chunk metadata

# In gdb with pwndbg or gef installed against a small malloc/free program:
#   gef>  heap chunks        # list chunk headers (size, flags, fd/bk)
#   gef>  heap bins          # tcache / fastbin / unsorted / small / large bins
# Watching a freed chunk's fd pointer is how you understand tcache poisoning.
echo "Use pwndbg/gef 'heap chunks' and 'heap bins' on a toy malloc/free target."

Practice the techniques safely

git clone https://github.com/shellphish/how2heap
# how2heap is the canonical, version-tagged glibc heap-technique playground:
# tcache_poisoning.c, fastbin_dup.c, unsorted_bin_attack.c, house_of_*.c —
# each a minimal, runnable demonstration against your local glibc. Owned-lab only.

Practical examples

CVE-2021-3156 — Sudo "Baron Samedit" heap overflow → root. An off-by-one in Sudo's argument parsing corrupts the heap; despite ASLR and canaries it yields local root on nearly every Linux distro of the era. Mature code is not immune.
CVE-2023-4863 — libwebp heap buffer overflow (in-the-wild 0-day). An out-of-bounds heap write in WebP image decoding, reachable by simply rendering an image, hit Chrome, Electron apps, and countless libwebp embedders simultaneously — a blast-radius lesson in shared-library heap bugs.
CVE-2020-0796 — "SMBGhost" (Windows SMBv3). An integer overflow in SMBv3 decompression produces an undersized allocation and a controlled heap overflow in srv2.sys, giving wormable RCE/LPE — a segment-heap-era Windows kernel heap overflow.
glibc tcache poisoning in practice. The modal CTF/real userland chain: leak heap + libc, poison a tcache next past safe-linking, allocate over a hook/GOT/FILE-vtable, get a one-shot. The how2heap tcache_poisoning.c is the minimal model.
ASan catches a parser heap overflow in CI. A fuzzer feeds a malformed length field; the memcpy overruns its destination; ASan pinpoints the write and the fix is a bounds check — the modal caught-pre-deploy outcome.

memory-corruption — the parent; heap overflow is class 2 of the six bug classes.
use-after-free-and-dangling-pointers — the temporal heap cousin; shares the reclaim/grooming machinery.
double-free-and-allocator-corruption — the free-list-corruption sibling and the "House of *" lineage's other root.
stack-buffer-overflow — the spatial cousin on the stack, where the canary lives.
exploit-mitigations — the structural defender pair (hardened allocators, MTE, CET, ASLR).
rop-and-ret2libc — where an allocate-to-arbitrary write usually pivots once it controls a code pointer.
EDR / Process Correlation — the runtime signals a heap-exploit chain emits.
Attacker-Defender Duality — overflow research and allocator-hardening engineering are the same problem from opposite chairs.

Suggested future atomic notes

double-free-and-allocator-corruption
tcache-poisoning-and-safe-linking
house-of-techniques-glibc
windows-segment-heap-internals
arm-mte-and-memory-tagging

Future atomic notes are listed as <span class="unresolved-link" title="Unpublished or unresolved: wikilinks">wikilinks</span> even when the target file does not exist yet, so they register as forward-links in Obsidian.

References

Foundational: MITRE CWE-122 — Heap-based Buffer Overflow — https://cwe.mitre.org/data/definitions/122.html
Research / Deep Dive: Microsoft Security Response Center — A proactive approach to more secure code (memory-safety statistics) — https://msrc.microsoft.com/blog/2019/07/a-proactive-approach-to-more-secure-code/
Official Tool Docs: AddressSanitizer (heap-buffer-overflow detection) — https://clang.llvm.org/docs/AddressSanitizer.html
Research / Tooling: shellphish — how2heap (canonical, version-tagged glibc heap-technique reference) — https://github.com/shellphish/how2heap
Hardening: ARM — Memory Tagging Extension (MTE) — https://developer.arm.com/documentation/108035/latest/