9 Characters and text — scars in the standard

9.1 A character is a number

There is only one way to store “A” in a machine — make an agreement that “A shall be number 65” and store the number 65. Such a table of agreements, the correspondence between letters and numbers, is called a character set or character code.

The problem is that there were several such tables. Every company and every country made its own. The same number meant a different letter in different tables, and when two machines with different tables exchanged text the letters came out tangled. The rest of this chapter is the history of that tangle, and every turn of that history left a trace in today’s C and computing.

9.2 ASCII and EBCDIC — two tables

Two branches of table established themselves in the 1960s. IBM’s mainframe world used a table called EBCDIC — grown out of punched-card codes, so its layout looks strange today (the alphabet is not in consecutive numbers, among other things), but in IBM’s world it was law.

On the other side, centred on the communications industry, came ASCII. Seven bits, that is a small table of 128 slots, holding the English upper and lower case, digits, punctuation, and the control characters that are invisible on screen. A control character is not a letter but an instruction to a device — “advance the paper one line”, “sound the bell” — and their story comes in the next chapter (streams). And sitting at slot 0 of this table is the NUL character whose face we learned in chapter 6. A character meaning “no character at all”, which C later adopted as the mark for the end of a string (chapter 42).

9.2.1 What “the alphabet is not consecutive” does to code

That phrase, “its layout looks strange”, must not be passed over — it decides the fate of a line of C. Everyone writes this line at least once.

if (c >= 'A' && c <= 'Z') { … }        /* is it an uppercase letter? */

On ASCII it works, because A is 0x41 and Z is 0x5A and the 26 slots between them are exactly the alphabet. But its being true is a fact about the table, not a rule of the language.

EBCDIC is different. This book measured the actual table (code page 037).

The alphabet coming in three pieces is a trace of its growth from punched-card codes. So the range from 'A' to 'Z' holds 41 slots, of which only 26 are letters. We counted what sits in the other 15.

★ That is, on EBCDIC the line above answers “yes, an uppercase letter” for a closing brace and a backslash. No compile error, no warning, no odd value. It is simply, quietly wrong.

9.2.2 Which is why there are libraries — what an abstraction does

Here a thesis this book will repeat first shows its face. Do not write the table’s circumstances into your code; ask.

if (isupper((unsigned char)c)) { … }   /* right, whatever the table */

That is exactly what the classification functions of <ctype.h> do. They do not test “at least 0x41 and at most 0x5A” — they know what counts as uppercase in the character set this implementation runs on. A C implementation on an EBCDIC machine ships a test that knows about the three pieces; one on an ASCII machine may use a simple range check. Either way the calling code does not change by a single character.

This is what an abstraction sells — not a feature but the confinement of what varies into one place.

★ And the story does not end at character classification. The same logic recurs throughout this book — counting a string’s length (chapter 42), deciding sort order (chapter 69′s strcoll), counting characters (chapter 72). All of them are the work of stripping out the assumption that the table I know is the whole world.

ASCII became the winning table — nearly every character code today carries ASCII inside it. But 128 slots were tight even for English and nowhere near enough for the world’s writing. Here the history of the scars begins.

9.3 ISO 646 — national variants and C’s trigraphs

The first stopgap was ISO 646, the international edition of ASCII. The idea went: “keep one table, but allow each country to replace a few less important slots with its own letters.” The slots designated for replacement happened to include symbols such as [ ] { } \ | #.

The result was a disaster for programmers. Open C code on a Danish terminal and Danish letters appeared where the braces should be. Same number, different table, different letter printed.

C was standardised in the middle of this confusion (chapter 12). Since C had to be usable in countries where braces were invisible, C89 provided a grotesque detour — the trigraph. Write ??< and it was read as {; write ??/ and it was read as \; a three-character cipher. A scar of the character-code wars, carved into the grammar of the language.

9.4 A hundred schools of eight bits, and Hangul

Once the byte settled at 8 bits (chapter 4), the ISO 8859 family extended the table to 256 slots — the first 128 exactly ASCII, the second 128 holding the letters of one language region (Latin-1 for Western Europe, and so on). But since the upper 128 were used differently per region, the problem of tangled letters when you picked the wrong table remained.

And there were writing systems for which 256 slots were nowhere near enough. Hangul, Han characters, kana — East Asian scripts run to thousands and tens of thousands. The solution was multi-byte encodings using several bytes per character (Korea’s EUC-KR among them). With lengths varying per character and methods varying per country, a Korean document read with the wrong table breaking into a parade of meaningless symbols — the so-called “broken characters” — was daily life for Korean computer users of that era.

9.5 Unicode and UTF-8 — one table, a clever way of holding it

In the end there was only one fundamental solution. Put every letter in the world into one table. That is Unicode, from the 1990s. Each letter gets a unique number (a code point), written like U+AC00 (가). The table has more than a million slots.

The remaining problem was how to hold those large numbers in bytes. Spending 4 bytes on every letter is wasteful and, above all, incompatible with the mountains of existing ASCII text. The answer was the variable-length encoding UTF-8 — letters in the ASCII range stay 1 byte as they are, and everything else takes 2–4 bytes. Thanks to the exquisite design by which an existing ASCII file is already valid UTF-8, UTF-8 has become effectively the only standard for the web and for source code today. It is what this book’s examples and C23′s u8"" strings use as well (chapter 42).

9.6 The two character sets C keeps apart — source and execution

So far this has been the world’s story of character sets. Now for how C handles them. The core is one sentence — C treats the character set the source code is written in and the character set of the running program as two different things.

Start with the standard’s exact names (§5.2.1).

The standard does not say the two must share an encoding. Section 5.2.2 splits them explicitly — the execution character set may also contain multibyte characters, and those are not required to have the same encoding as for the source character set. And §5.2.1 drives in one more nail: the values of the members of the execution character set are implementation-defined.

9.6.1 Where the crossing happens — translation phases 1 and 5

The place the two sets meet is pinned down in the translation phases (chapter 57).

Phase 5′s second sentence matters most — where there is no corresponding member, it is converted, in an implementation-defined manner, to some member other than the null character. That is: it quietly becomes a different character. Not an error.

9.6.2 The more precise name C23 added — the literal encoding

C23 added §6.2.9, “Encodings”, tidying up one more name.

If the “execution character set” is which characters exist, the “literal encoding” is what byte values write them. The standard requires only that both map the whole basic execution character set, and leaves the rest to the implementation.

9.6.3 Prefixes that nail the encoding down

There is a way to fix an encoding regardless of the execution character set — the literal prefixes (§6.4.5).

Where the bytes must not move — protocols, file formats, a test’s expected value — write u8"…". Two characters that turn “the implementation decides” into “the standard decides.” The fuller story of the wide side returns in chapters 70 and 71.

9.7 Same letter, several representations — the security terrain hidden in text

Unicode unifying the table did not end the problems. If anything, as the table grew and the representations layered up, new traps appeared. It is terrain any program handling text meets eventually, so let us survey it as a guide. The root is single — if the same thing can be written in two or more ways, the side checking it and the side interpreting it can disagree.

① Normalization — Hangul’s NFC and NFD. The letter “각” can be written two ways in Unicode: as one composed syllable (NFC, the single code point U+AC01) and as a sequence of jamo (NFD, the three code points ᄀ+ᅡ+ᆨ). They look identical, but the byte sequences are entirely different — so a naive byte comparison answers “각 ≠ 각”. For Korean users this is not a textbook story: because macOS stored file names in the NFD family, it was common for Hangul file names crossing to another OS to appear with the jamo undone, or for names to go wrong inside archives. The solution is to normalise to one form at the boundary — and in a security context the order of that normalisation is decisive (see ④ below).

② Overlong encodings — the back door UTF-8 once left open. By the UTF-8 rules in the maths box above, / (U+002F) is the single byte 2F. But early implementations generously accepted the same character written “long”, in 2 bytes (C0 AF) or 3 — an overlong representation. From this came a classic attack: the security check looked only for 2F to filter out path traversal (../), while the file system interpreted the overlong C0 AF as / too. A disagreement between checking and interpreting — that was the identity of the directory-traversal vulnerability that swept IIS servers in 2001. Today’s standard therefore declares overlong forms illegal and requires decoders to reject them.

③ The ghost of a vanished encoding — UTF-7. UTF-7, made to send Unicode through 7-bit channels (early email), represents other characters using ordinary-looking ASCII letters (+ADw- becomes <). In the days when browsers guessed encodings automatically, attackers used this property to get through filters — a harmless-looking string at checking time that resurrects as a tag the moment the browser interprets it as UTF-7. UTF-7 was effectively retired because of these incidents, and today’s norm has hardened into do not guess the encoding; state it.

④ Traps in Unicode itself — homoglyphs and direction controls. A large table also means there are different letters that look identical. Latin a and Cyrillic а are indistinguishable on screen — using this to build a fake domain that looks exactly like the real one is a homograph attack (which is why browsers show suspicious mixed-script domains back in their original notation). More cunning are the directional control characters: Unicode has invisible characters that reverse the display order of text, making it possible for the order a human reads in the source and the order the compiler reads to differ — “Trojan Source”, published in 2021, used this technique to show that code looking perfectly fine to a reviewer could compile with different logic. Compilers began warning about such characters afterwards.

⑤ When layers stack — escaping and multiple interpretation. To display < as a letter on the web you write < (HTML escaping). Stack several such layers and accidents follow — decode once-escaped text in another layer and the original symbol resurrects; conversely, check but forget to escape and the input becomes code. The format-string vulnerability of chapter 22, SQL injection in databases and XSS on the web are all the same pattern — making data be mistaken for the frame (the code). C’s trigraphs were an ancestral case of the pattern: if ??/ happened to appear inside a string, the preprocessor turned it into a backslash, creating an escape the programmer never wrote (one of the reasons C23′s removal of trigraphs is welcome).

9.8 Sequences of letters — ways of holding a string

Now that we know how to hold one letter, a last question remains. When a sequence of letters — a string — goes into the lockers, how do we know “where does this string begin and end?” Chapter 5′s refrain returns: memory does not know the boundaries of a lump; what knows is the agreement of the reader. There are several such agreements, so let us look at the main branches.

Method 1 — write the length in front (length-prefixed). Attach a number meaning “this many characters follow” in front of the string. The Pascal family used it, so it is also called a Pascal string. Asking for the length needs no counting (read the number and you are done), and any byte at all — even character 0 — can be held as content. The price is that the size of the length field has to be fixed in advance — old Pascal held the length in one byte, so a string could not exceed 255 characters (chapter 4′s container story, again).

Method 2 — plant a marker at the end (NUL-terminated). Instead of writing the length, plant a special character at the end of the sequence — the NUL character of value 0 whose face we learned in chapter 6. This is the method C chose. Its virtue is extreme simplicity — you need only the starting point, and pointing anywhere in the middle of the sequence makes “from there to the NUL” a string. The price is threefold: to know the length you must count all the way to the NUL; you cannot hold a NUL in the content; and above all, forget to plant the marker and the reader runs on into other people’s land. That last price is the terrain of more accidents than any other in C’s history, and chapter 42 is the scene.

Method 3 — manage length and capacity together. The dynamic strings of modern languages usually manage three values as one bundle: [where the content is, the current length, the capacity of the container] — length queries are immediate, appending is free within the capacity, and when it runs out the content moves to a larger container. C++‘s std::string and Rust’s String have this structure, and proven’s string handling, which we meet later, thinks in the same family by treating length explicitly (Part XII).

The terrain is wider still — representations for editors that manage a large document as a tree of pieces (ropes), views that point at [start, length] without copying the original, and so on: each use has its own fitting representation. One thing to remember: the shape a string takes in memory is not one thing but a choice made by a language and a library, and that choice determines the character of its performance and safety. What character C’s choice (NUL termination) has, and how it must be handled, is faced head on in chapter 42.

The next chapter is the last rung on the ladder of representation. Now that we can hold letters, it is the story of letters flowing — why a computer’s input and output is shaped as “characters going past one line at a time”, an answer that lies in the age of paper cards and typewriters.