57 The preprocessor and the translation phases

57.1 Macros — turning a name into a stream of tokens

#define NAME content is the directive “from now on, when you meet the token NAME, replace it with content.” There is also a form taking arguments (#define MAX(a,b) ...), which looks like a function but is not — it is substitution, not a call, with neither type checking nor any guarantee about evaluation order.

57.1.1 Parentheses — the first thing anyone using macros learns

With a function the argument arrives after being calculated into one value, but with a macro the tokens as written are planted into the body. So the operators inside the argument and the operators in the body meet in one place, and chapter 20′s precedence divides them. We see four traps together with their actual expansions.

/* Why a macro's arguments must be wrapped in parentheses — the places where
   the intent and the expansion part ways. xstr() prints "what did it expand
   into, then" alongside, so it can be checked. */
#include <stdio.h>

#define str(...)   # __VA_ARGS__
#define xstr(...)  str(__VA_ARGS__)

#define SQ_BAD(x)   x * x           /* no parentheses */
#define SQ_HALF(x)  (x) * (x)       /* only the arguments wrapped */
#define SQ(x)       ((x) * (x))     /* arguments and the whole wrapped — the canonical form */

#define NEG_BAD(x)  -x
#define NEG(x)      (-(x))

/* a macro that uses an argument twice — when it meets an argument with a side effect */
#define MAX(a, b)   ((a) > (b) ? (a) : (b))

int main(void)
{
    /* ── (1) leave the argument unwrapped — precedence cuts in ────── */
    printf("SQ_BAD(1+2)  -> %-16s = %d   (9 was intended)\n",
           xstr(SQ_BAD(1+2)), SQ_BAD(1+2));
    printf("SQ_HALF(1+2) -> %-16s = %d\n",
           xstr(SQ_HALF(1+2)), SQ_HALF(1+2));

    /* ── (2) leave the whole unwrapped — the outer operator cuts in ─ */
    printf("100/SQ_HALF(2) -> %-14s = %d   (25 was intended)\n",
           xstr(100/SQ_HALF(2)), 100/SQ_HALF(2));
    printf("100/SQ(2)      -> %-14s = %d\n",
           xstr(100/SQ(2)), 100/SQ(2));

    /* ── (3) an argument carrying a sign ──────────────────────────── */
    printf("NEG_BAD(-3)  -> %-16s  <- a compile error (`--3`)\n", xstr(NEG_BAD(-3)));
    printf("NEG(-3)      -> %-16s = %d\n", xstr(NEG(-3)), NEG(-3));

    /* ── (4) what parentheses cannot stop — the argument is evaluated twice ─ */
    int i = 5, j = 3;
    int m = MAX(i++, j);
    printf("MAX(i++, j)  -> i has become %d (one increment was expected)\n", i);
    printf("             result m = %d\n", m);
    printf("             expansion: %s\n", xstr(MAX(i++, j)));
    return 0;
}

SQ_BAD(1+2)  -> 1+2 * 1+2        = 5   (9 was intended)
SQ_HALF(1+2) -> (1+2) * (1+2)    = 9
100/SQ_HALF(2) -> 100/(2) * (2)  = 100   (25 was intended)
100/SQ(2)      -> 100/((2) * (2)) = 25
NEG_BAD(-3)  -> --3               <- a compile error (`--3`)
NEG(-3)      -> (-(-3))          = 3
MAX(i++, j)  -> i has become 7 (one increment was expected)
             result m = 6
             expansion: ((i++) > (j) ? (i++) : (j))

① Without wrapping the argument, the operations inside it scatter.

#define SQ_BAD(x)  x * x
SQ_BAD(1+2)   →   1+2 * 1+2   →   1 + (2*1) + 2   =  5     /* 9 was expected */

1+2 does not go in bound into one value; three tokens are planted as they are, and over them the rule that multiplication is stronger than addition applies. Wrapping the argument in parentheses solves it — (1+2) * (1+2) = 9.

② Without wrapping the whole, the outer operator cuts in. Wrapping the argument alone is not enough.

#define SQ_HALF(x)  (x) * (x)
100 / SQ_HALF(2)   →   100 / (2) * (2)   =  100            /* 25 was expected */

Division and multiplication have the same precedence and bind from the left (appendix A), so 100/2 happens first and is then multiplied by 2. The whole body must be wrapped for 100 / ((2) * (2)) = 25. So the canonical form wraps both.

#define SQ(x)  ((x) * (x))

③ With a signed argument the tokens can even fuse.

#define NEG_BAD(x)  -x
NEG_BAD(-3)   →   --3      /* it becomes the decrement operator — a compile error */

It is why the example prints this expansion as text only — written for real it does not compile. Wrapped as (-(x)) it goes properly to (-(-3)) = 3.

④ What parentheses cannot block — the argument is evaluated twice.

#define MAX(a, b)  ((a) > (b) ? (a) : (b))
int i = 5, j = 3;
int m = MAX(i++, j);      →   ((i++) > (j) ? (i++) : (j))

The example’s output is that result — i jumped from 5 to 7, and m is 6 rather than the expected 5. Because i++ was planted in the body twice. Parentheses solve only the precedence problem, not this one. It is the place where chapter 33′s “one variable changed only once in one statement” is broken behind your back by a macro.

The last row is today’s answer. The reason for computing values with #define was to save the cost of a call, and today’s compilers inline small functions by themselves (chapter 13), so that reason has nearly vanished. The parenthesis rules are needed only where a macro must be used, and those places are what the rest of this chapter shows — assembling tokens, conditional compilation, capturing source location.

57.2 # — turning a token into a string (stringize)

Put # before a macro argument and that argument becomes a string literal of it literally. The key is that what becomes a string is not the value but the shape as written.

#include <stdio.h>

/* # : makes the argument into a string "letter for letter" (stringize) */
#define SHOW(expr)  printf(#expr " = %d\n", (expr))

/* ## : pastes two tokens into one (token paste) */
#define MAKE_NAME(prefix, n)  prefix##n

/* one layer does not expand the macro — the double indirection is the canonical way */
#define STR_RAW(x)  #x
#define STR(x)      STR_RAW(x)
#define WIDTH       80

int MAKE_NAME(value_, 1) = 11;   /* -> int value_1 = 11; */
int MAKE_NAME(value_, 2) = 22;

int main(void)
{
    SHOW(2 + 3 * 4);
    SHOW(value_1 + value_2);

    printf("STR_RAW(WIDTH) = %s\n", STR_RAW(WIDTH));  /* not expanded */
    printf("STR(WIDTH)     = %s\n", STR(WIDTH));      /* expanded first, then made a string */
    return 0;
}

2 + 3 * 4 = 14
value_1 + value_2 = 33
STR_RAW(WIDTH) = WIDTH
STR(WIDTH)     = 80

The demonstration’s SHOW(2 + 3 * 4) is that in the flesh — #expr becomes the string "2 + 3 * 4" while the (expr) beside it is calculated and becomes 14. One argument was used both as text and as a value. This pattern is the basic form of debugging and checking macros: printing the condition “x > 0” as it stands when assert(x > 0) fails works on exactly this principle (the inner workings of the assert met in chapter 51).

57.3 ## — joining two tokens into one (token paste)

## joins the tokens on either side into one new token. The demonstration’s MAKE_NAME(value_, 1) becoming the identifier value_1 is the example — not a string but a real name, so it can be used in a variable declaration.

Its use is code generation that assembles names. When functions or structs of the same shape must be stamped out per type (C having no generics), joining a prefix and a type name to produce names like stack_int_push and stack_double_push from one macro is the classic technique.

57.4 Double expansion — why one layer does not unfold

There is a rule you must meet when using # and ##. A macro argument that becomes the operand of # or ## is not expanded first. The demonstration’s last two lines are the contrast — STR_RAW(WIDTH) becomes "WIDTH", while STR(WIDTH) becomes "80".

57.4.1 The names first — four rules

What happens in this section is what the standard writes down in four clauses. Knowing the names makes it far easier to put the order together in your head.

The heart of it is the collision between the first row and the middle two. Normally an argument is expanded first (argument substitution), but an argument that becomes the operand of # or ## is the exception and the unexpanded original token is used. So when the value is wanted, one more layer is added so that the argument is expanded before it meets # or ## — this is the idiom called double expansion, or an indirection macro.

One more rule is needed. Its name is not the standard’s but people’s — the blue paint rule. If a macro’s own name appears again while it is being expanded, that name is not expanded again but left as it is (the metaphor is that it is painted blue to mark it). It is the device that prevents infinite recursion, and the LOW of the standard example below turns on exactly this rule.

57.4.2 Following the expansion by hand — a contrast of two lines

What makes this hard to simulate in the head is that “when to expand and when not” differs from place to place. Let us put two lines side by side and follow them step by step. WIDTH is defined as 80.

One layer — STR_RAW(WIDTH)

Two layers — STR(WIDTH)

The only place a difference arises is step 2. Thanks to the extra layer outside, the argument was expanded once before meeting #, and so the token # saw was not WIDTH but 80. ## is in the same circumstance, so keeping a PASTE/PASTE_RAW pair is idiom.

57.4.3 The standard’s own example — glue and xglue

This idiom is also one the standard document gives directly as an example (in C17 it is the EXAMPLE of §6.10.3.5, “Scope of macro definitions”; C23 pushed the number along by one as #embed came in at 6.10.3). We run that example as it stands.

/* Double expansion — the example the C standard gives, run as it stands.
   How one layer and two layers part ways is all here. */
#include <stdio.h>

/* ── the macros of the standard's example (C17 6.10.3.5) ──────── */
#define str(s)       # s              /* stringize */
#define xstr(s)      str(s)           /* the version wrapped one layer more */
#define INCFILE(n)   vers ## n        /* token pasting */
#define glue(a, b)   a ## b
#define xglue(a, b)  glue(a, b)       /* the version wrapped one layer more */
#define HIGHLOW      "hello"
#define LOW          LOW ", world"    /* it calls itself — the stage for the blue paint rule */

/* the assembled name becomes the header name: xstr(INCFILE(2).h) -> "vers2.h" */
#include xstr(INCFILE(2).h)

#define WIDTH 80

int main(void)
{
    /* ── (1) stringizing: one layer does not expand, two layers do ── */
    printf("str(WIDTH)   = %s\n", str(WIDTH));    /* the argument is the operand of # */
    printf("xstr(WIDTH)  = %s\n", xstr(WIDTH));   /* expanded first, then turned into a string */

    /* ── (2) pasting: those two lines of the standard's example ───── */
    printf("glue(HIGH, LOW)  = %s\n", glue(HIGH, LOW));
    printf("xglue(HIGH, LOW) = %s\n", xglue(HIGH, LOW));

    /* ── (3) did the header included under the assembled name arrive? ── */
    printf("VERS_TAG = %s\n", VERS_TAG);

    /* ── (4) the standard example's two stringizings ──────────────── */
    printf("%s\n", str(strncmp("abc\0d", "abc", '\4') == 0));
    return 0;
}

str(WIDTH)   = WIDTH
xstr(WIDTH)  = 80
glue(HIGH, LOW)  = hello
xglue(HIGH, LOW) = hello, world
VERS_TAG = vers2.h really was included
strncmp("abc\0d", "abc", '\4') == 0

The third and fourth lines of the output are this section’s conclusion. Given the same arguments, glue produced hello and xglue produced hello, world. We follow why, a step at a time. Three definitions are involved.

#define glue(a, b)   a ## b
#define xglue(a, b)  glue(a, b)
#define HIGHLOW      "hello"
#define LOW          LOW ", world"

glue(HIGH, LOW)

Notice that a macro called HIGH was never defined. The reason no error arises is that HIGH, before being used on its own, was pasted with LOW and became a different name, HIGHLOW. This is what ## does — it makes one name that exists out of two that do not.

xglue(HIGH, LOW)

Step 5 is the place that confuses most. ## is an operator that pastes two tokens, not “the whole of the right-hand argument”. So even though b has expanded into several tokens (LOW, ", world"), only the leading LOW is pasted and the rest follows after it as it stands.

And it is worth pointing out that step 7 is the business not of the preprocessor but of translation phase 6. What the preprocessor produced was, after all, the two string literals "hello" ", world", and their joining into one is the next stage.

#define CONCAT(a, b) a ## b
#define VERSION 2
int CONCAT(api_v, VERSION)(void);   /* api_vVERSION — not what was wanted */

#define CONCAT_RAW(a, b) a ## b
#define CONCAT(a, b)     CONCAT_RAW(a, b)
int CONCAT(api_v, VERSION)(void);   /* api_v2 */

It is the preprocessor trap that catches people most often, and also a problem solved in two lines once known.

57.5 The curious examples the standard gives

The standard document’s macro replacement clauses hold several examples that make one ask “was even this settled?”. They are not things to use often in practice, but they show what happens when the expansion rules are pushed to their limit, so they are worth reading once. We pick three and really run them.

/* The odd examples the standard itself gives — the expansions printed as text.
   Wrap a thing in xstr() and "what did it become, then" can be seen as a
   string (the double indirection). */
#include <stdio.h>

/* to stringize even something containing commas, take it as variadic arguments */
#define str(...)   # __VA_ARGS__
#define xstr(...)  str(__VA_ARGS__)

/* ── (1) placemarkers — pasting an empty argument with ## ──────── */
#define t(x, y, z)  x ## y ## z

int j[] = { t(1,2,3), t(,4,5), t(6,,7), t(8,9,),
            t(10,,), t(,11,), t(,,12), t(,,) };

/* ── (2) the expansion the standard pins down as "unspecified" ─── */
#define f(a)  a*g
#define g(a)  f(a)
int g = 1;                     /* g is a function-like macro, so used alone it is not expanded */

/* ── (3) __VA_OPT__ — when is an argument judged "empty"? ──────── */
#define LOG(...)          log(0 __VA_OPT__(,) __VA_ARGS__)
#define SDEF(name, ...)   S name __VA_OPT__(= { __VA_ARGS__ })
#define EMP

int main(void)
{
    printf("(1) placemarkers\n");
    printf("   j[] = {");
    for (size_t i = 0; i < sizeof j / sizeof *j; i++)
        printf(" %d", j[i]);
    printf(" }   (%zu elements)\n", sizeof j / sizeof *j);
    printf("   t(,,)  -> \"%s\"  (not one token is left)\n", xstr(t(,,)));
    printf("   t(6,,7)-> \"%s\"\n", xstr(t(6,,7)));

    printf("\n(2) unspecified expansion\n");
    printf("   f(2)(9) -> \"%s\"\n", xstr(f(2)(9)));
    printf("   (the standard does not settle which of \"2*9*g\" and \"2*9*f(9)\" it is)\n");

    printf("\n(3) __VA_OPT__\n");
    printf("   LOG(1,2)  -> \"%s\"\n", xstr(LOG(1,2)));
    printf("   LOG()     -> \"%s\"\n", xstr(LOG()));
    printf("   LOG(EMP)  -> \"%s\"   <- an argument was passed, yet there is no comma\n", xstr(LOG(EMP)));
    printf("   SDEF(foo)      -> \"%s\"\n", xstr(SDEF(foo)));
    printf("   SDEF(bar,1,2)  -> \"%s\"\n", xstr(SDEF(bar, 1, 2)));
    return 0;
}

(1) placemarkers
   j[] = { 123 45 67 89 10 11 12 }   (7 elements)
   t(,,)  -> ""  (not one token is left)
   t(6,,7)-> "67"

(2) unspecified expansion
   f(2)(9) -> "2*9*g"
   (the standard does not settle which of "2*9*g" and "2*9*f(9)" it is)

(3) __VA_OPT__
   LOG(1,2)  -> "log(0 , 1,2)"
   LOG()     -> "log(0 )"
   LOG(EMP)  -> "log(0 )"   <- an argument was passed, yet there is no comma
   SDEF(foo)      -> "S foo"
   SDEF(bar,1,2)  -> "S bar = { 1, 2 }"

The knack the example uses of printing the expansion as text is worth noticing too — wrap something in xstr(…) and “so what did it become” can be seen as a string. The double expansion just learned is, in effect, used as a tool for debugging the preprocessor.

57.5.1 ① Placemarkers — pasting an empty argument

#define t(x, y, z)  x ## y ## z
int j[] = { t(1,2,3), t(,4,5), t(6,,7), t(8,9,),
            t(10,,), t(,11,), t(,,12), t(,,) };

What happens if an argument is left empty and ## is used? There being nothing to paste, it looks like an error, but the standard settles that in an empty argument’s place there is an invisible token called a placemarker. This token leaves whatever it is pasted with as it is, and vanishes when the expansion ends.

So t(6,,7) pastes 6 and 7 into 67, and t(,,) leaves no token at all. It is why the example’s array ends with seven elements — the eighth t(,,) vanished whole and left only a trailing comma (that comma being legal thanks to chapter 46′s initialiser syntax).

Where this rule pays in practice is in a macro whose argument may or may not be there. A code-generating macro that chooses with one argument whether to attach a prefix leans on this property.

57.5.2 ② The expansion the standard declares “unspecified”

#define f(a)  a*g
#define g(a)  f(a)
f(2)(9)

These two lines call each other. f(2) becomes 2*g, and with (9) following it it could be read as g(9) — and expanding g gives f(9) again, whose f is already being expanded and so is caught by the blue paint rule.

Here the standard did not settle on one answer. It writes into the example itself that “the result is either 2*9*g or 2*9*f(9), and which it is is unspecified”. It is a textbook case of unspecified behaviour, one of the grey zones learned in chapter 52 — it is one of two, but which is settled by the implementation.

The GCC on the machine that ran this example chose 2*9*g. Another answer on another implementation would not be a bug. The practical lesson is one — do not make macros call each other.

57.5.3 ③ __VA_OPT__ — when is “empty” judged?

The __VA_OPT__ C23 brought in (chapter 82) puts something in “only when the variadic arguments are not empty”. But when it looks to see whether they are empty is subtle.

#define LOG(...)  log(0 __VA_OPT__(,) __VA_ARGS__)
#define EMP                       /* a macro that expands to nothing */

LOG(1,2)   →  log(0 , 1,2)
LOG()      →  log(0 )
LOG(EMP)   →  log(0 )             ← an argument was passed and yet there is no comma

LOG(EMP) is the heart of it. One argument was plainly passed, but EMP expands to no token at all. Because the judgement is made on the tokens after expansion rather than before, it counts as “empty” and no comma attaches.

The SDEF side shows this property’s practical use.

#define SDEF(name, ...)  S name __VA_OPT__(= { __VA_ARGS__ })

SDEF(foo)         →  S foo                 /* without initialisation */
SDEF(bar, 1, 2)   →  S bar = { 1, 2 }      /* with it */

That a macro whose very syntax changes with the presence of arguments can be written with standard syntax alone is C23′s contribution. Before it one had to lean on a GCC extension (, ##__VA_ARGS__).

57.6 The X macro — several sets of code from one list

C has no code generation. So the problem arises of “having to write the same list repeatedly in several forms” — take error codes: once in an enumeration, once in a table of name strings, once in a switch returning messages. Miss one place when adding an item and the tables are out of step from then on.

The X macro solves this by writing the list only once. The knack is two steps.

1. Write the list in the form of “calls to a macro X that is not yet defined.”
2. Define X differently each time and unfold the list. The same list comes out as different code every time.

#include <stdio.h>

/* One list produces the enumeration, the name table and the lookup code
   together. Change the list in one place and the rest follows. */
#define ERROR_LIST(X)                       \
    X(OK,        0, "no error")             \
    X(NOT_FOUND, 2, "resource not found")   \
    X(DENIED,    5, "permission denied")    \
    X(TIMEOUT,   9, "operation timed out")

/* (1) making the enumeration — only the name and the value are used */
#define AS_ENUM(name, code, text) ERR_##name = code,
enum error_code { ERROR_LIST(AS_ENUM) };
#undef AS_ENUM

/* (2) making the table of name strings — the name is turned into a string (#) */
#define AS_NAME(name, code, text) [code] = #name,
static const char *const error_name[] = { ERROR_LIST(AS_NAME) };
#undef AS_NAME

/* (3) making the function that returns the description — a whole switch is generated */
#define AS_CASE(name, code, text) case ERR_##name: return text;
static const char *error_text(enum error_code e)
{
    switch (e) {
        ERROR_LIST(AS_CASE)
        default: return "unknown error";
    }
}
#undef AS_CASE

/* (4) counting — the length of the list comes automatically too */
#define AS_COUNT(name, code, text) + 1
enum { ERROR_COUNT = 0 ERROR_LIST(AS_COUNT) };
#undef AS_COUNT

int main(void)
{
    printf("%d error codes defined\n", ERROR_COUNT);

#define AS_ROW(name, code, text) \
    printf("  %-10s code=%d name=%-9s text=%s\n", #name, (int)ERR_##name, \
           error_name[code], error_text(ERR_##name));
    ERROR_LIST(AS_ROW)
#undef AS_ROW

    return 0;
}

4 error codes defined
  OK         code=0 name=OK        text=no error
  NOT_FOUND  code=2 name=NOT_FOUND text=resource not found
  DENIED     code=5 name=DENIED    text=permission denied
  TIMEOUT    code=9 name=TIMEOUT   text=operation timed out

Four sets came out of one list. Taking them one at a time.

① The enumeration — AS_ENUM unfolds to ERR_##name = code,. ## joins ERR_ with the name and produces ERR_NOT_FOUND = 2,.

② The table of name strings — #name turns the name straight into a string. Moreover it uses the designated initialiser [code] = #name (chapter 46) so that the code value is itself the array index. Even with sparse values (0, 2, 5, 9) the table lines up by itself.

③ Generating the switch — case ERR_##name: return text; unfolds as many times as there are items. The whole body of the function was made from the list.

④ Counting — the knack of making it unfold to 0 + 1 + 1 + 1 + 1. Add items to the list and ERROR_COUNT follows by itself.

⑤ And in the body — inside main the same list is unfolded once more to print a table. Erasing the macro with #undef after use is the convention, because a lingering name collides with the next use.

/* when there are only three items and only one derived form */
#define COLOR_LIST(X) X(RED) X(GREEN) X(BLUE)

57.7 The translation phases — the eight steps the standard pins down

We knew preprocessing happens “first”, but exactly what happens in what order? The standard pins this down as eight steps under the name translation phases. A real compiler need not perform these steps physically separately, but must behave as if it had done them in this order (chapter 14′s “as long as it looks the same” applies here too).

This table unravels several riddles at once. Because comments become a space in phase 3, a/**/b is a b, not ab. Because #include is processed recursively in phase 4, headers within headers unfold naturally. Because string joining is in phase 6, the idiom of writing a long string over several lines works, and string fragments made by a macro (the result of the # operator) also join with a literal beside them. And that grammatical interpretation is phase 7 restates this chapter’s core — when the preprocessor works, C’s grammar does not yet exist.

We now know layer by layer the whole process of source becoming a program. The next chapter is the secret of how printf takes any number of arguments — the variadic function.