102 Build and test — how a project is put together by many hands

102.1 What changes when there is a build#

A build tool takes three jobs off your hands.

The middle one is this section’s subject. What happens without it is worth causing on purpose.

#!/bin/sh
# Reproduce the stale-artifact accident.
#
#   Widen GREET_MAX from 16 to 32 in the header and touch only greet.c.
#   The rule with no dependencies rebuilds greet.o alone: greet now writes
#   trusting 32 while main.o still reserves the old 16. The buffer overflows.
set -eu
cd "$(dirname "$0")"

for kind in broken fixed; do
    printf '== Makefile.%s\n' "$kind"
    make -s -f "Makefile.$kind" clean >/dev/null
    make -s -f "Makefile.$kind" CFLAGS="-std=c23 -Wall -Wextra -g -fsanitize=address$([ $kind = fixed ] && printf ' -MMD -MP')" >/dev/null 2>&1
    printf '   first          : %s\n' "$(./demo 2>&1 | head -1)"

    sed -i 's/#define GREET_MAX 16/#define GREET_MAX 32/' greet.h
    touch greet.c
    make -s -f "Makefile.$kind" CFLAGS="-std=c23 -Wall -Wextra -g -fsanitize=address$([ $kind = fixed ] && printf ' -MMD -MP')" >/dev/null 2>&1
    out=$(./demo 2>&1 | head -3 || true)
    printf '   after widening : %s\n' "$(printf '%s' "$out" | head -1)"
    printf '%s' "$out" | grep -q 'stack-buffer-overflow' && printf '   → ASan: stack-buffer-overflow\n' || true

    sed -i 's/#define GREET_MAX 32/#define GREET_MAX 16/' greet.h
    make -s -f "Makefile.$kind" clean >/dev/null
done

== Makefile.broken
   first          :   GREET_MAX=16  len=15
   after widening : =================================================================
   → ASan: stack-buffer-overflow
== Makefile.fixed
   first          :   GREET_MAX=16  len=15
   after widening :   GREET_MAX=32  len=31

Take apart what happened. The size of the container lives in the header.

#define GREET_MAX 16
void greet(char *out);

greet.c fills that many, and main.c reserves that many. Both files trust the same number. And yet the rule in Makefile.broken says only this.

%.o: %.c
  $(CC) $(CFLAGS) -c -o $@ $<

Nowhere in it does it say “this object file also depends on greet.h”. So widen the header from 16 to 32 and touch only greet.c, and make rebuilds greet.o alone. The new greet writes 31 characters and the closing NUL — 32 bytes — while the old main.o still reserves a 16-byte container.

★ The result is not slowness but a stack buffer overflow. The demonstration catches it on the spot under the sanitizer (chapter 18). Uncaught, this program would have become the kind that “sometimes gives odd values”.

CFLAGS = ... -MMD -MP
-include main.d greet.d

102.2 How files are divided — the header is the contract, the source the circumstances#

Files are not divided because one grew long. The criterion is keep together what changes together, and separate what changes separately. Split what changes together and every fix walks across several files; join what changes separately and one side keeps forcing the other to be rebuilt.

On top of that sits one circumstance peculiar to C — the header is the contract and the source the circumstances (chapter 53). What stands in the header is the promise the caller leans on; what stands in the source is how that promise is kept. The how may change; change the promise and other people break.

★ That last row has one widely used exception: writing a very short function in a header as static inline (from C99 on). Copies are still made, but the compiler usually folds them in and an unused one raises no warning. Even so, the body of that function is now visible in the header, so anything still likely to change belongs in the source.

★ The third row is the most expensive at scale. Write a struct’s members into the header and that layout becomes a public promise, so adding one member becomes rebuilding every caller. Large projects therefore often publish the name and hide the innards.

/* header: a name, and no innards */
typedef struct config config;
config *config_open(const char *path);
int     config_get_int(const config *c, const char *key, int fallback);
void    config_close(config *);

The caller does not know how many bytes config is. So it need not know, and does not break when it changes. The price is that it cannot be used as a value, only through a pointer.

102.3 The boundary between public and internal#

As a project grows its headers fall into two kinds — those put outside and those used only within. The best-known case of pinning that boundary down through file layout is the Linux kernel. It split the declarations that go out to user space into include/uapi/, with the inner headers including those. Two reasons are given for the split — reducing the tangle between headers, and making what is a public promise plain to the eye.

102.4 Folder layout — what the conventions are for#

The names differ a little from project to project but the kinds are much the same. The point is not a list to memorise but what each one is for.

★ Keeping build/ apart repays more than it looks. With artefacts scattered beside the source, nobody can say “which configuration made this .o”, and you cannot build under two configurations at once. Separate the place and build/debug and build/release can live side by side.

102.5 Kinds of test — what each one catches#

“Writing a test” is not one activity. Each kind catches a different accident.

This book has already shown two of them. The sanitizer of chapter 18 is the partner of fuzzing, and the “failing shell” of chapter 92 is the technique of making a unit test deliberately walk the path that rarely runs.

102.6 Golden tests — freezing the answer in a file#

Of the kinds, golden repays most as things grow, because any program with output can have one almost for free. Keep the input in a file, freeze the output you have checked to be right as an answer file, and from then on the machine compares with diff.

PostgreSQL uses this on a large scale. Under src/test/regress/ it keeps sql/ for the inputs, expected/ for the answers and results/ where this run’s output lands, and leaves the mismatches in regression.diffs. The tool doing the comparison is nothing special: it is diff.

The same thing can be stood up in a dozen or so lines.

#!/bin/sh
# A golden test - freeze the right answer in a file and let the machine compare.
#   Feed in/X.txt, then diff what comes out against expected/X.txt.
#   --accept freezes the current result as the answer (the first time, and after
set -eu
cd "$(dirname "$0")"
cc -std=c23 -Wall -Wextra -o wordcount wordcount.c
mkdir -p results
fail=0 n=0
for input in in/*.txt; do
    name=$(basename "$input" .txt)
    ./wordcount < "$input" > "results/$name.txt"
    n=$((n + 1))
    if [ "${1:-}" = "--accept" ]; then
        cp "results/$name.txt" "expected/$name.txt"
        continue
    fi
    if ! diff -u "expected/$name.txt" "results/$name.txt" > "results/$name.diff"; then
        printf '  ✗ %s\n' "$name"
        sed -n '3,6p' "results/$name.diff" | sed 's/^/      /'
        fail=$((fail + 1))
    else
        printf '  ✓ %s\n' "$name"
        rm -f "results/$name.diff"
    fi
done
[ "${1:-}" = "--accept" ] && { printf 'froze %d answers\n' "$n"; exit 0; }
[ "$fail" -eq 0 ] && printf '%d golden tests passed\n' "$n" || { printf '%d mismatched\n' "$fail"; exit 1; }

  ✓ empty
  ✓ spaces
  ✓ two-lines
3 golden tests passed

Inputs and answers frozen like this are called fixtures — the provisions a test stands on. Their properties and traps are taken up separately in chapter 103.

Three things are worth noticing.

1. The answer is not written by hand. --accept freezes the current result. Written by hand it comes out wrong, and a wrong answer turns the test backwards.
2. A failure shows only the lines that differ. diff already does that work.
3. So this test earns its keep even when you do not know “what is right” — because what it asks is has it changed since yesterday.

102.7 How much, and what not to test#

Is there an answer to “how much should I test”? There is one case that shows the upper end.

SQLite states the numbers in its own testing document¹ — as of 3.42.0 (May 2023), the library itself is about 155.8 KSLOC while the test code and scripts come to 92,053.1 KSLOC, that is 590 times the library. The frames that run the tests — the harnesses — are not one but four (the TCL tests, TH3, the SQL Logic Test and dbsqlfuzz), and it states that the core, in its default configuration, holds 100% branch coverage under TH3 — the qualification is in the original, and it is not “the whole library, always”.

★ What is to be learned here is not “do 590 times”. It is two things.

1. Why keep several frames — where one approach shuts its eyes, another looks. Ten tests written from different thinking beat a hundred written from the same.
2. The quantity of tests is a result, not a goal — SQLite tests that much because it claims to be “a database that runs anywhere”. A different goal gives a different answer.

And the question on the other side matters more.

102.8 Working together — what the machine refuses and what people look at#

Once several people are working, one more question appears. Who stops what has gone wrong?

The answer divides in two. What the machine refuses (a gate) and what people look at (a review). Mix them and both get worse — give a person what a machine can count and the person tires; give a machine what a person must judge and the rule refuses the wrong things.

102.9 What has to be written down#

Last, the work of making a place for others to come into. Documents earn their keep not by being many but when each answers a different question.

★ The last row is the one most often missing and the most expensive. Months later, the person who wants to undo that decision does not know what was weighed to reach it, and so walks again a road already examined and discarded.

That was what goes where. The next chapter looks at how to work on top of it — the vocabulary and procedure used when several people keep a thing turning, and how to tell whether a check really bites.