85 The five bugs shipped for fifty years

What to know first

chapter 51, Errors and contracts · errors and contracts

chapter 42, Strings · the danger of strings

chapter 61, The terrain of the standard library · the traps of the standard library

Looking back

Chapter 42 said the functions handling strings “do not know the size of the vessel”, and chapter 51 said “a failure not confirmed becomes a thing that never happened”. Then why do such problems still remain — is half a century not more than enough time to mend them?

A. Not because they cannot be mended but because mending them breaks all the code already written. The moment one more size parameter is put into strcpy’s signature, every C program in the world stops compiling. The standard is an institution that must protect existing code (chapter 61′s reason for “thin and old” appears here too), so instead of removing dangerous functions it has taken the road of placing better functions beside them. So the choice comes over to the programmer — it is, in effect, a language in which knowing what is dangerous and choosing accordingly is itself skill.

The need for this chapter, and its context

Part 12 introduces the author’s library, and its first chapter does not mention the library. That is this part’s most important arrangement: state the problem first, and reach for the tool five chapters later. And to put this chapter here, chapters 51 (contracts) and 61 (the standard library’s traps) had to come first. The preceding eighty-three chapters were, in a sense, the material for these five bugs.

By the end of this chapter

This part’s statement of the problem. We confirm with actually running code why C has kept shipping the same five classes of bug for half a century — that it is not the programmer’s carelessness but the shape of the API. Only after all five have been seen does the name proven appear again. Not introducing the tool first is this part’s principle.

Q. Then are these five a list of mistakes beginners make?

A. No. These are mistakes the skilled keep making too, and that point matters. If people who know the whole grammar still slip in the same places, the cause is not the person but the shape of the tool. A function that does not take a size has no way of checking a size, and a function whose return value may be thrown away with nothing happening will one day be thrown away. This chapter takes that “shape” apart one at a time.

The questions this chapter answers

Would it not be better to use another language entirely to avoid such problems?

85.1 One — string functions do not know the size of the vessel

The oldest and most exploited class. Exactly as seen in chapter 42.

char buf[64];
strcpy(buf, name);            /* how long is name? strcpy does not ask */
strcat(buf, ", welcome!");    /* and how much room is left now? */

strcpy’s signature has no destination size. What is not there cannot be checked, so this function will happily write to the 200th byte of a 64-byte vessel. What gets wrecked depends on what the compiler placed after it, and commonly that is the function’s return address (recall chapter 44′s picture of the stack).

The modern prescription is to use the editions that take a size — snprintf is representative. And here is a second trap. Functions that take a size quietly truncate when it overflows.

examples-en/ch85/truncate.c

#include <stdio.h>
#include <string.h>

/* The common way: a path assembled into a fixed buffer.
   Inside this function the compiler cannot know the length of dir or name. */
static int build_path(char *out, size_t cap, const char *dir, const char *name) {
    return snprintf(out, cap, "%s/%s", dir, name);   /* if it does not fit, it cuts quietly */
}

int main(void) {
    char path[24];

    build_path(path, sizeof path, "/var/log", "app.log");
    printf("fits      : %s\n", path);

    build_path(path, sizeof path, "/var/log/service/http", "access.log");
    printf("truncated : %s\n", path);
    printf("            length=%zu, buffer=%zu\n", strlen(path), sizeof path);

    /* to find out whether it was truncated, the return value must be looked at */
    int need = build_path(path, sizeof path, "/var/log/service/http", "access.log");
    if (need >= (int)sizeof path)
        printf("detected  : needed %d bytes, had %zu\n", need + 1, sizeof path);
    return 0;
}

Output

fits      : /var/log/app.log
truncated : /var/log/service/http/a
            length=23, buffer=24
detected  : needed 33 bytes, had 24

Look at the third line. What was to be made was /var/log/service/http/access.log, and what remained in hand is /var/log/service/http/a. The program neither stopped nor warned. If this string is a file path it opens the wrong file, if a command it becomes a different command, if a log the record of an accident is cut without a sound. A truncated path is not a short path but a wrong path.

Counter-example. Treating truncation as success

snprintf(path, sizeof path, "%s/%s", dir, name);
open_file(path);          /* nobody asked whether it was truncated */

snprintf in fact gives the answer — it returns the length that would have been needed. If that value is at least the vessel’s size it was truncated (the example’s last line is that check). The problem is that this check is optional. Throw the return value away and the compiler says nothing. That leads straight into the second bug.

In practice. The compiler catches only what it can see

Something that really happened while making this example. At first snprintf was called directly inside main with literal arguments, and gcc caught it.

error: ‘%s’ directive output truncated writing 10 bytes
       into a region of size 2 [-Werror=format-truncation=]
note: ‘snprintf’ output 33 bytes into a destination of size 24

An excellent diagnosis. Yet moving the same call inside a function called build_path made the warning vanish. The moment a function boundary is crossed the compiler cannot know the real lengths of dir and name. In a real program strings come from files or from the network, so cases where the compiler can help are rather rare. A warning is a free review, not a guarantee (chapter 17).

85.2 Two — there is no device that makes you confirm failure

char *p = malloc(n);
p[0] = 'x';                   /* malloc gives null on failure */

C’s ways of reporting failure are two. Return a sentinel value (null, -1, EOF), or leave the reason in the global variable errno. Neither can compel a check. Code that throws the return value away is perfectly legal, and errno is global state that must be read at exactly the right moment, before the next call overwrites it (chapter 61).

examples-en/ch85/unchecked.c

#include <stdio.h>
#include <stdlib.h>

/* Reading a port number out of a configuration line — the version that does not check for failure */
static int read_port_careless(const char *line) {
    int port = 8080;              /* the default */
    sscanf(line, "port=%d", &port);   /* the return value is not looked at */
    return port;
}

/* the same work — the version that checks for failure */
static int read_port_checked(const char *line, int fallback) {
    int port;
    if (sscanf(line, "port=%d", &port) != 1) {
        printf("  (parse failed, keeping %d)\n", fallback);
        return fallback;
    }
    return port;
}

int main(void) {
    const char *good = "port=9000";
    const char *typo = "prot=9000";     /* a typo */

    printf("careless good: %d\n", read_port_careless(good));
    printf("careless typo: %d\n", read_port_careless(typo));
    printf("checked  typo: %d\n", read_port_checked(typo, 8080));

    /* strtol is the same: it reports failure as a value, but nobody forces you to look */
    long n = strtol("abc", nullptr, 10);
    printf("strtol(\"abc\") = %ld\n", n);
    return 0;
}

Output

careless good: 9000
careless typo: 8080
  (parse failed, keeping 8080)
checked  typo: 8080
strtol("abc") = 0

That careless typo returned 8080 is this section’s heart. There was a typo in the configuration and the program quietly fell back to the default. On the surface nothing happened, and months later only the question “why is the setting not taking effect?” remains. That strtol("abc") gives 0 is the same pattern — failure and “a real 0” come back as the same value.

A common misconception. “Failure is exceptional, so it can be handled later”

The premise that failure is rare is wrong to begin with. A file may not exist, input carries typos, disks fill, networks break — every place where the program touches the outside world is a point of failure. And the real reason “later” is dangerous lies elsewhere. Code that ignored a failure does not stop but keeps running. A wrong value flows into the next calculation, into the function after that, and by the time the problem finally shows itself it blows up far from its cause. Chapter 51′s “fail early” returns here.

85.3 Three — `printf` believes exactly what you tell it

Chapter 61 took the grammar of the format string apart. That grammar has one structural weakness — the type is written twice. Once in the format (%d) and once in the argument (the variable’s type). If the two go out of step the language cannot prevent it, because as seen in chapter 58 type information does not ride along into variadic arguments.

Today’s compilers catch this. The real message is like this.

warning: format ‘%d’ expects argument of type ‘int’,
         but argument 2 has type ‘double’ [-Wformat=]
    3 |     printf("%d\n", 3.0);
      |             ~^     ~~~
      |              |     |
      |              int   double

But only this far. The moment the format becomes a variable — the moment a multilingual message is taken from a table or a log format is received from a configuration — the compiler has nothing left to look at.

Counter-example. Code that takes the format as a variable

const char *fmt = load_message("greeting");   /* a format taken from a table */
printf(fmt, count);                           /* no warning. no check either */

Not a single warning comes from this code. Because a way of knowing whether format and arguments match does not exist at compile time. Worst is when the format is user input, which becomes the format string vulnerability seen in chapter 61.

85.4 Four — who frees this

char *s = build_message();    /* must this be freed? the type says nothing */

As learned in chapter 45, dynamically taken memory must be released by somebody exactly once. Yet a char * a function returned may be any of four things.

Just allocated — it must be freed.
Pointing at a buffer the caller gave — it must not be freed.
A string literal in a read-only place — freeing it is an accident.
A static buffer the next call will overwrite — it must neither be freed nor held for long (chapter 61′s strtok was such).

The types of the four cases are all the same. The answer is in the documentation, and documentation goes out of step with code as a matter of course. Here arise chapter 45′s three accidents — a leak (nobody frees), a double free (both free), and use after free (somebody still points at it after freeing).

Counter-example. An API whose type does not state ownership

const char *lookup(int code);        /* a literal? an allocation? a static buffer? */
char       *format_time(time_t t);   /* must this be freed? */

It cannot be known from the name and type alone. Every place that uses this API must remember the documentation, and if even one forgets it becomes one of the three accidents above. That the discipline is entrusted to human memory is the essence of the problem.

85.5 Five — a callback nobody can type-check for you

qsort(a, n, sizeof *a, cmp);   /* cmp takes const void* */

qsort takes a comparison function through a void * interface in order to sort any type. In a language with no generics this is nearly the only way, but the price is the complete abandonment of type checking. Whatever you cast to inside the comparator, the compiler believes you.

examples-en/ch85/cmp_bad.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* Counterexample: it compares only the first character. The types match perfectly and there is no warning */
static int cmp_first_char(const void *a, const void *b) {
    const char *const *x = a;
    const char *const *y = b;
    return (*x)[0] - (*y)[0];
}

/* The right version: it compares the whole string */
static int cmp_full(const void *a, const void *b) {
    const char *const *x = a;
    const char *const *y = b;
    return strcmp(*x, *y);
}

static void show(const char *label, const char **v, size_t n) {
    printf("%s:", label);
    for (size_t i = 0; i < n; i++) printf(" %s", v[i]);
    printf("\n");
}

int main(void) {
    const char *src[] = {"pear", "apple", "peach", "apricot"};
    const char *v[4];
    size_t n = sizeof src / sizeof src[0];

    memcpy(v, src, sizeof src);
    qsort(v, n, sizeof v[0], cmp_first_char);
    show("first-char", v, n);

    memcpy(v, src, sizeof src);
    qsort(v, n, sizeof v[0], cmp_full);
    show("full      ", v, n);
    return 0;
}

Output

first-char: apple apricot pear peach
full      : apple apricot peach pear

The first-char comparator’s types match perfectly, it compiles without a single warning, and it does not die. It is only that peach and pear are in the wrong order — seeing only the first letter, the two were judged “equal” and the rest was left to chance. This class of bug is found last of all, because it gives a quietly wrong answer.

In practice. Attacks aiming at a data structure’s worst case

There is a performance trap in the same place. Widely used sorting and hashing implementations are fast on average but slow down sharply on particular inputs, and the technique of an attacker deliberately making such inputs to paralyse a server (an algorithmic complexity attack) was organised in a 2003 paper¹ and used in real attacks. Attacks of the same family aiming at hash collisions brought down several web frameworks at once in 2011. They were events showing that a data structure’s worst case is itself a security problem, and so today’s libraries take as their defaults sorting with a guarantee even in the worst case (introsort) and hashes using a random seed — we see them in the flesh in chapter 92.

85.6 And a sixth — bytes have types

We said five, but one more must be added to be fair. It is the strict aliasing seen in chapter 13. A hand-written parser that peers into a byte buffer through pointers of different widths runs perfectly at -O0 and quietly gives a different answer at -O2. It is the representative of the “bug that appears only in release” seen in chapter 17, and the clause to which the Linux kernel surrendered with a single flag.

Gathering the six into one table gives this part’s sketch.

Recap

problem	what C gives	what is needed
buffer overflow and truncation	string functions that do not know the size	strings that carry their length, writes that do not truncate
unconfirmed failure	sentinel values and `errno`	errors that come as values, a compile refusal if discarded
format mismatch	a `printf` that believes the format string	placeholders that take the type from the argument
unclear ownership	a `char *` that means four things	different types for owning and borrowing
unchecked callbacks	the `void *` interface	documented contracts and worst-case-guaranteed algorithms
the hidden type of bytes	UB on breaking the aliasing rule	a byte type the rule exempts

Table 86.1

Q. Would it not be better to use another language entirely to avoid such problems?

A. That too is an answer, and many places really went that road (chapter 1). But the places where C must be used still remain — operating systems, firmware, the floor layer other languages lean on, and projects where decades of code have already piled up. What can be done in such places is not to change the language but to change the shape of the API. The right-hand column of the table above is not a list of items requiring a new language but things that can be made by design within C. From the next chapter we see that design.

The library that implements that right-hand column as it stands is the proven this book has leaned on. We first met it in chapter 43 and its name has come up a few times since, but treating it head on begins now. The next chapter is installation and a first program — and why this library has the shape of “nothing to install”.

Notes

Scott A. Crosby and Dan S. Wallach. 2003. Denial of Service via Algorithmic Complexity Attacks. In Proceedings of the 12th USENIX Security Symposium. USENIX Association, Washington, DC. usenix.org/conference/12th-usenix-security-symposium ↩

← PrevContentsNext →