57 A field guide to memory errors

57.1 What a memory error is — four promises#

A memory error is an access to an object that breaks one of that object’s promises. It is also called a memory-safety bug. The standard has no such word — it simply writes “undefined behaviour” for each accident separately. So this chapter first sorts errors by what was broken.

Chapter 36 defined an object as “a region of memory that holds a value”. Four things attach to that region.

The four are independent. A read inside the size is still an error if the lifetime has ended, and a read inside the lifetime is still an error if the type is wrong. So when you ask “is this access right?”, you ask four times — is that byte inside the region, is the region still alive, may it be read with this type’s eyes, and did this pointer come from that region?

★ And all four share one consequence. Breaking any of them is undefined behaviour (chapter 56). There is no promise that the program dies on the spot, and no promise that a strange value comes out. That is why the chapter’s catalogue of errors (“What the tools catch, and what they miss”) prints only what the tool called the error, not what the program did. The two demonstrations that do print what a program outside the contract did (_FORTIFY_SOURCE and strict aliasing) show one run of one build, not a behaviour that will repeat.

57.2 Breaking the size — buffer overflow#

A buffer overflow is writing past the end of a container — the array or block set aside to hold the data, that is, the buffer. The bytes that overflow do not vanish; they overwrite whatever was next to it. Going past the end in the reading direction is called an out-of-bounds read. The off-by-one, the write that overwrites a neighbour and the read through which someone else’s memory leaks out were each shown in chapter 39. Here they are regrouped by kind and by defence.

57.2.1 What overflows — five kinds#

The fourth row is where an integer accident turns into a memory accident. n * size is size_t arithmetic, so it is not UB when it overflows; it quietly wraps around (chapter 6). As a result the allocation succeeds, and the program starts writing as if n elements fit. That is why C23′s checked arithmetic (chapter 96) belongs wherever sizes are multiplied.

57.2.2 Why it is dangerous — it is not only data that is overwritten#

An overflow does not only overwrite neighbouring variables. A function’s stack frame also stores the address to return to when the function ends (chapter 66). Overflowing a local array can reach that address, and then, at the moment the function ends, the program goes wherever the overwritten value points. If the input can decide that value, the person who sent the input decides where the program goes. That was the road the Morris worm took in 1988 (chapter 43).

This book does not cover how that road is walked. One thing is worth knowing — a buffer overflow does not end with one wrong value; it is an accident that can hand the flow of the program to someone else. That is why the defences below are stacked in layers.

57.2.3 The layers of defence#

Defence is not a single layer. The upper layers keep overflows from happening; the lower layers keep an overflow that did happen from spreading.

Two of the middle layers, _FORTIFY_SOURCE and the stack protector, can be checked with one program. A program that strcpys seventeen bytes into an 8-byte container was built three ways and run.

/* Buffer overflow --- strcpy 17 bytes (including the null) into an 8-byte container.
 * The source string is chosen at run time, so the compiler cannot notice in advance. */
#include <stdio.h>
#include <string.h>

int main(int argc, char **argv)
{
    (void)argv;
    char name[8];
    const char *src = argc > 5 ? "ok" : "0123456789abcdef";
    strcpy(name, src);              /* a function that does not take the container's size */
    printf("copied %zu bytes into an 8-byte buffer\n", strlen(name) + 1);
    return 0;
}

no protection              ran to the end: copied 17 bytes into an 8-byte buffer
-fstack-protector-strong   stopped: *** stack smashing detected ***: terminated
-D_FORTIFY_SOURCE=3 -O2    stopped: *** buffer overflow detected ***: terminated

The first line is the most frightening line in this section. The build without protection ran to the end without incident and calmly reported seventeen bytes in an eight-byte buffer. The nine bytes that overflowed overwrote something; it simply was not used again in this run. The build with the stack protector stopped where the function ends, because it saw the canary had changed. The _FORTIFY_SOURCE build stopped earlier — at the very call to strcpy — because the compiler knew that name has size 8 and substituted a version of strcpy that checks the size.

57.2.4 A buffer that carries its length#

The second layer of defence is design. The trouble with strcpy(dst, src) is that dst is a single pointer, so the receiving side does not know the size of the container. Functions that take the size separately (strncpy, snprintf) expect the caller to write the size correctly. If you make the pointer, the length and the capacity into one bundle, every write passes through one of the bundle’s functions, and the checking gathers in that one place.

/* A buffer that carries its length --- one shape that prevents overflow *by structure*.
 * Pass a single pointer and the receiver does not know the size of the container (strcpy's problem).
 * Pass pointer and length as one bundle, and every write is checked in the same place. */
#include <stdckdint.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct {
    char  *data;
    size_t len;     /* bytes written */
    size_t cap;     /* size of the container */
} buf;

/* Grow the container if there is no room. On failure, false --- the old contents stay as they were. */
static bool reserve(buf *b, size_t extra)
{
    size_t need;
    if (ckd_add(&need, b->len, extra)) return false;    /* refuse if the length computation overflows */
    if (need <= b->cap) return true;

    size_t cap = b->cap ? b->cap : 8;
    while (cap < need)
        if (ckd_mul(&cap, cap, 2)) return false;

    char *p = realloc(b->data, cap);    /* receive the result in another variable first */
    if (p == NULL) return false;        /* on failure b->data is still alive (C23 7.24.3.7p3) */
    b->data = p;
    b->cap  = cap;
    return true;
}

static bool append(buf *b, const char *s)
{
    size_t n = strlen(s);
    if (!reserve(b, n)) return false;
    memcpy(b->data + b->len, s, n);
    b->len += n;
    return true;
}

/* Size computation when borrowing an array of n --- if n * size overflows, a small block comes back. */
static void *alloc_array(size_t n, size_t size)
{
    size_t bytes;
    if (ckd_mul(&bytes, n, size)) return NULL;
    return malloc(bytes);
}

int main(void)
{
    buf b = {0};
    const char *words[] = {"length ", "travels ", "with ", "the ", "pointer"};
    for (size_t i = 0; i < sizeof words / sizeof words[0]; i++)
        if (!append(&b, words[i])) { free(b.data); return 1; }
    printf("%.*s\n", (int)b.len, b.data);
    printf("len %zu, cap %zu\n", b.len, b.cap);
    free(b.data);

    void *huge = alloc_array(SIZE_MAX / 2, 4);  /* the product overflows size_t */
    printf("alloc_array(SIZE_MAX / 2, 4): %s\n", huge ? "got a block" : "refused (size overflow)");
    free(huge);
    return 0;
}

length travels with the pointer
len 31, cap 32
alloc_array(SIZE_MAX / 2, 4): refused (size overflow)

Three places are worth reading. reserve computes “current length + added length” with ckd_add and the size to grow to with ckd_mul — the size computation overflow (the fourth row of Table 57.2) is stopped here. The result of realloc is first received in another variable. On failure the old block stays as it was (C23 7.24.3.7); receiving straight into b->data would lose the only address of the old block at the moment of failure — a leak. The last line of the output comes from asking for an array of SIZE_MAX / 2 elements to show a product that overflows. Without the check, malloc would have returned a small block as a success.

The shape is not new. Length-prefixed strings (Pascal strings), the slices of other languages and C++‘s std::span are all the same idea. C simply has no such type in the standard, and it can be made from one structure as above (proven, the library of chapter 101, takes that road).

/* generated code */
if ( ++p == pe )
    goto _test_eof;

57.3 Breaking the lifetime#

An object’s lifetime is set by its storage duration (chapter 45). An automatic object ends with its block; a dynamic object ends with free. Lifetime accidents go in two directions — touching it after it ended (dangling), and not ending what should be ended (leaking).

57.3.1 Leaks — losing the last pointer#

A memory leak is losing the last pointer to a borrowed block, so that the block can never be returned. In the standard’s eyes it is not an accident — not returning memory is not UB, and when the program ends the operating system usually reclaims everything. So in a short command-line tool it does no harm. The harm is to programs that run for a long time. In servers, daemons, editors and browsers a leak eats memory bit by bit until it finally brings the machine to a halt.

Most leaks take three shapes: overwriting (p = malloc(...) twice with no free in between), early return (an error path leaves without returning what it borrowed earlier), and confusion about ownership (the receiver and the giver each think “the other will return it”). The third is why the ownership rule of chapter 46 is needed.

★ “Memory grows even though nothing leaks” is a different problem. Fragmentation is covered in chapter 99.

57.3.2 Dangling pointers and use after free#

A dangling pointer is a pointer that still points at an object whose lifetime has ended. Reading or writing through it is a use after free. As chapter 46 showed, the allocator soon hands a returned slot to another request, so a use after free reads or overwrites someone else’s object that is there now.

The standard adds one more sentence here. The second paragraph of C23 6.2.4 says that “if a pointer value is used in an evaluation after the object the pointer points to (or just past) reaches the end of its lifetime, the behavior is undefined”, and goes on: “the representation of a pointer object becomes indeterminate when the object the pointer points to (or just past) reaches the end of its lifetime”.³

It takes neither a read nor a write to make this UB. After free(p), using the value of p at all — comparing it with == to another pointer, converting it to an integer to print it — is outside the contract. The bits are unchanged, but the value is no longer a “value”. The rule has a nickname — pointer zap. The reason for a sentence this strict lies in the provenance of chapter 38. A pointer is not just a number called an address; it is a value that also carries which object it came from, and when that object disappears, half of the value disappears. The practical conclusion is simple: do not use a freed pointer again — not even to compare it. The habit free(p); p = nullptr; is of some use, if only half, precisely because nullptr is a “value”.

57.3.3 Double free and invalid free#

The contract of free is two sentences. Passing a null pointer does nothing. Passing a pointer that was never handed out (the middle of a block, a stack address, an address already returned) is UB.⁴ The allocator keeps bookkeeping in front of and behind its blocks (chapter 99), so returning the wrong address corrupts the bookkeeping, and corrupted bookkeeping poisons every allocation after it, one by one. That is why the place where the accident happens and the place where it blows up are often far apart.

57.3.4 The address of an automatic object that has ended#

A function’s local variable ends its lifetime when the function ends. Return its address and the receiver holds a dangling pointer from the very start.

int *make(void)
{
    int v = 7;
    return &v;          /* the compiler warns --- and the warning is right */
}

GCC and Clang both warn about this shape. Store the address in a variable once and return that, and the warning goes away but the accident stays. The same trap comes in less visible shapes — the compound literal &(struct pt){1, 2} inside a block ends its lifetime when the block ends, and an array member of the temporary object created by a call to a function returning a structure ends when that expression ends.

57.3.5 realloc — the old pointer and size zero#

realloc frees the old object and returns a new one. The new address may equal the old one, but that is chance; the old pointer and all its aliases become dangling. Going on using the old pointer in the belief that “I only grew it” is the most common realloc accident.

And one place changed in C23. realloc(p, 0) is now undefined behaviour.⁶ Before, implementations split between “free the block and return null” and “return a block of size zero”, and because you could not know which, both leaks and double frees happened. Rather than choosing one, the committee moved it outside the contract. To shrink to zero, call free.

57.4 Breaking initialisation and type#

57.4.1 Indeterminate values — reading a place never written#

An uninitialised automatic variable and a block just returned by malloc hold an indeterminate value (chapter 24). Reading it comes with no promise about what value comes out, and for some types the read itself is UB. The reason it is not “just some garbage value” is optimisation. The compiler treats a variable never written as something that may be any value, and may give two different answers when the same variable is read twice.

57.4.2 Non-value representations — bits that are not a value of the type#

For some types not every bit pattern is a value. For bool only 0 and 1 are values; if the place holds 2, that bit pattern is a non-value representation. Earlier standards called it a “trap representation” — reading it might spring a trap. C23 changed the name and kept the rule: reading it through an lvalue that is not of character type is UB.⁸ Chapter 49 met this trap when reading another member of a union. Writing the byte in with memcpy is inside the contract; the contract is left at the moment it is read as bool.

Conversely a structure or union as a whole is never a non-value representation (paragraph 6 of the same subclause). Whatever its padding bytes hold, copying a structure whole is safe. But the values of those padding bytes are unspecified, so comparing two structures with memcmp can report structures with equal values as different (chapter 47).

MD_Update(&m,buf,j);              /* line 247 */
...
#ifndef PURIFY
    MD_Update(&m,buf,j); /* purify complains */   /* line 467 */
#endif

57.4.3 Strict aliasing — seen again#

Chapter 14 called breaking strict aliasing “the act of telling the compiler something untrue as if it were true”. The rule is in the seventh paragraph of C23 6.5.1. An object’s stored value may be accessed only through an lvalue matching the object’s effective type; the exceptions are the same type with signedness or qualifiers changed, a structure or union that has it as a member, and character types.¹⁰ The effective type of a declared object is its declared type; for an object with no declared type, such as a malloc block, it is the type of the lvalue last used to store into it.

How this rule meets optimisation can be seen in one short program. A function taking an int * and a float * was given the same address as both arguments.

/* Strict aliasing --- what happens when an int lvalue and a float lvalue designate the same place.
 * The two parameters of store_both have *different types*. The compiler may read this as "an int object
 * and a float object do not overlap" (C23 6.5.1p7) --- and on that premise not read *ip again. */
#include <stdio.h>

[[gnu::noinline]] static int store_both(int *ip, float *fp)
{
    *ip = 1;
    *fp = 0.0f;
    return *ip;             /* compiler: "a float store cannot change an int -> 1" */
}

int main(void)
{
    int x = 42;
    int r = store_both(&x, (float *)&x);    /* outside the contract: writes an int object as a float */
    printf("returned %d, x is now %d\n", r, x);
    return 0;
}

-O0                        returned 0, x is now 0
-O2                        returned 1, x is now 0
-O2 -fno-strict-aliasing   returned 0, x is now 0

The build with optimisation off and the -fno-strict-aliasing build read x again after the second write and returned 0. The -O2 build returned 1 — it trusted that “writing a float cannot change an int” and did not read *ip again. Yet, as x is now 0 on the same line shows, memory already holds 0. The value the function returned and the memory say different things. GCC 14.2 and Clang 19.1 produced these three lines identically.

-fno-strict-aliasing is the option that turns off the strict-aliasing assumption — that lvalues of different types do not reach the same object. The Linux kernel builds with it switched on in its top-level Makefile (KBUILD_CFLAGS += -fno-strict-aliasing) — kernel code has many idioms that break the rule, so it chose to give up the optimisation the rule offers. The same place also has -fno-delete-null-pointer-checks (CVE-2009-1897 of chapter 56 is the reason). An option widens the contract only for that build. Move the code to another build and the accident comes back.

57.4.4 The same bits through other eyes, inside the contract#

When the same bits have to be read as another type — looking at the bits of a floating-point number, or turning bytes received from the network into an integer — there are two roads inside the contract.

/* Two roads *inside the contract* to read the same bits as another type --- memcpy and a union.
 * A pointer cast (*(uint32_t *)&f) breaks strict aliasing. These two do not. */
#include <stdint.h>
#include <stdio.h>
#include <string.h>

static uint32_t bits_by_memcpy(float f)
{
    uint32_t u;
    static_assert(sizeof u == sizeof f, "float must be 32 bits here");
    memcpy(&u, &f, sizeof u);       /* only moves bytes --- one instruction once optimised */
    return u;
}

static uint32_t bits_by_union(float f)
{
    union { float f; uint32_t u; } pun = { .f = f };
    return pun.u;                   /* reading another member: allowed in C (C23 6.5.3.4 footnote) */
}

int main(void)
{
    const float samples[] = {1.0f, -2.5f, 0.1f};
    for (size_t i = 0; i < sizeof samples / sizeof samples[0]; i++) {
        float f = samples[i];
        printf("%5.2f  memcpy 0x%08x  union 0x%08x\n",
               (double)f, (unsigned)bits_by_memcpy(f), (unsigned)bits_by_union(f));
    }
    return 0;
}

 1.00  memcpy 0x3f800000  union 0x3f800000
-2.50  memcpy 0xc0200000  union 0xc0200000
 0.10  memcpy 0x3dcccccd  union 0x3dcccccd

memcpy only moves bytes, so it breaks no type rule. With optimisation on, the compiler turns the call into a single instruction — it costs almost nothing. Reading another member through a union is allowed in C (a footnote to C23 6.5.3.4 calls this “type punning” and says only that the result may be a non-value representation).¹¹ C++ does not allow it, so for a header shared by both languages memcpy is the safe choice. The pointer cast *(uint32_t *)&f is the same accident as the aliasing demonstration above.

57.4.5 Misalignment#

Converting a pointer to a type that needs stricter alignment is UB at the moment of the conversion if the result is not correctly aligned.¹² This holds even without a dereference (chapter 38).

The reason this accident lives so long is x86. x86 usually handles misaligned integer reads quietly. So code that reads *(uint32_t *)(buf + 1) from a byte buffer runs fine on x86 for years, then dies with a bus error the day it moves to a machine that demands alignment (older ARM, some RISC-V, digital signal processor (DSP) chips). It is not safe on x86 either — if the compiler, assuming “this is aligned”, chooses a vector instruction that requires alignment, it dies on the spot on x86 too. To take a value out of a byte buffer, use the memcpy above.

57.5 Breaking provenance — provenance seen again#

Chapter 38 showed provenance. A pointer carries, together with the number called an address, the origin “which object it came from”, and pointer arithmetic cannot leave that object’s range (up to one past the end). So even if two neighbouring variables x and y sit side by side in memory and &x + 1 == &y comes out true, changing y through &x + 1 is an error — the address is the same, the provenance is not. As that chapter showed, an optimising compiler assumes that “two pointers of different provenance do not overlap” and moves code accordingly.

Seen through provenance, the other accidents of this chapter read differently. A buffer overflow is reaching a neighbouring object with one object’s provenance, and a use after free is using the provenance of an object that is gone. On a machine whose hardware carries provenance itself (CHERI), both are stopped the same way — by the bounds and validity tag attached to the pointer (Appendix N).

There are two places where the provenance rule bites even inside one object.

• Overlapping copies. memcpy is UB when the two regions overlap.¹³ When they can overlap, as when shifting elements within an array by one, use memmove. The two parameters of memcpy carry restrict — a promise that “these two do not overlap”, which the implementation trusts in choosing a fast copy.
• The promise of restrict. If you put restrict on your own function, you promised that while the function runs, an object modified through one pointer is not reached through another (chapters 27 and 39). Breaking it is the same UB.

57.6 At a glance — UB, unspecified and implementation-defined on the memory side#

The accidents of this chapter are gathered with their places in the standard, each labelled with one of the three branches of chapter 56 (UB, unspecified, implementation-defined).

The “UB” rows also appear one by one in the list of Annex J.2. There is no need to memorise that list (chapter 56). The twelve rows of this table are the places that code handling memory actually steps on.

57.7 What the tools catch, and what they miss#

For each accident a small program was written and run with sanitizers on (chapter 18). The output below is the name the tool gave when each program ran. The demonstration is built to fail if a tool cannot name the expected error — so the catching tools the text names for each error are re-checked on every build. The source printed is one of them, the use after free.

/* Breaking the lifetime, reading --- read a block that was already returned. */
#include <stdlib.h>

[[gnu::noinline]] static int peek(const int *p) { return *p; }

int main(void)
{
    int *p = malloc(sizeof *p);
    if (p == NULL) return 1;
    *p = 7;
    free(p);
    return peek(p);         /* the value of p is now indeterminate (C23 6.2.4p2) */
}

== size
  stack_overflow.c  ASan: stack-buffer-overflow
  heap_overflow.c   ASan: heap-buffer-overflow
== lifetime
  use_after_free.c  ASan: heap-use-after-free
  double_free.c     ASan: attempting double-free
  bad_free.c        ASan: attempting free on address which was not malloc()-ed
  leak.c            LSan: detected memory leaks
  return_local.c    ASan: stack-use-after-return
== initialization
  bad_bool.c        UBSan: load of value 2 into bool
  uninit.c          ASan+UBSan: nothing reported
  uninit.c          MSan: not offered by this compiler
== type
  misaligned.c      UBSan: load of misaligned address
== overlap
  overlap.c         ASan: memcpy-param-overlap

Most are caught by ASan (AddressSanitizer). ASan inserts a check before every memory access at build time and puts a poisoned zone (red zone) on both sides of every object. Overflows are caught by touching that zone, and use after free by keeping returned blocks in quarantine for a while and noticing the touch. Leaks are caught by LeakSanitizer, which scans at exit for blocks still reachable, and the 2 in a bool and the misalignment by UBSan.

★ The lines to notice are the two for uninit.c. ASan and UBSan do not look at uninitialised reads. The tool that does is MemorySanitizer (MSan), and on this machine only Clang had it — run the same script with Clang 19.1 and the second of those lines becomes MSan: use-of-uninitialized-value. MSan works properly only when the whole program (including the libraries it uses) is built with the same option, so in practice other tools such as Valgrind are also used to fill this gap.

The limits all the tools share are worth writing down too.

• They see only the paths that ran. An accident on a line no test passed through is not caught.
• They hardly see strict-aliasing violations. Build the aliasing demonstration above with ASan and UBSan and nothing is reported. This accident is found by reading the code.
• They are slow. ASan makes execution roughly twice as slow and uses more memory. So it goes on in test builds and off in release builds. What guards release builds is the layered defence above.

57.8 Summary — one line per accident#

The last column leads into the next chapter. Catching accidents is the tools’ job, but keeping them from arising is the job of the design that decides who returns memory, and when. The next chapter covers that design — the habits and data structures for governing lifetime in C, and the ways other languages took this job from people and gave it to machines (reference counting and garbage collectors), with what they cost.