Proven C Book한국어 GitHub

71 Wide characters ② — the platforms, and wide I/O

What to know first

chapter 70, Wide characters ① · wchar_t’s contract and conversion
chapter 63, Streams in practice · streams and buffers

Looking back

Chapter 70 said the standard fixes neither the size nor the encoding of wchar_t. If it does not, the implementation picks — is that really such a problem?

A. That it split into two camps is the problem. Linux and macOS chose four bytes; Windows chose two. And that choice stood on one fact of the 1990s — “sixteen bits are enough for Unicode” — which broke in 1996.

Once Unicode went past U+FFFF, a two-byte wchar_t could no longer keep its original promise of holding one character in one unit. The patch was the surrogate pair, and most of the string traps in Windows programming today come from it.

This chapter is the exact shape of that split, and what each platform and toolkit chose to build on top of it.

The need for this chapter, and its context

Chapter 70 was what the standard settled; this is what the platforms actually did. That is why they are split: wchar_t is the word in this book where the distance between the standard’s promise and reality is greatest, and mixed together there is no telling contract from custom.

By the end of this chapter

What the same wchar_t became on each platform. The limit of UTF-16 and the arithmetic of surrogate pairs, Windows’s W functions and their exact encoding, glibc’s choice, how GTK and Qt hold strings, and the little-known rule of stream orientation.

The questions this chapter answers

  1. Why did Windows choose two bytes?
  2. So what should my program choose?

71.1 A wchar_t split into two camps

Platformsizeof(wchar_t)Encoding in practice__STDC_ISO_10646__
Linux (glibc), macOS4UTF-32 (UCS-4)Defined
Windows (MSVC, MinGW)2UTF-16Not defined
Some other Unixes (AIX, …)2 or 4VariesVaries

Table 72.1

Chapter 70′s demonstration gave this machine’s answer — four bytes, and __STDC_ISO_10646__ defined. One wchar_t is one code point. On Windows that equation does not hold.

Q. Why did Windows choose two bytes?

A. The circumstances of the age. In the early 1990s, while Windows NT was being designed, Unicode was a fixed-width 16-bit character set. The design of the day was to hold every character in the world within 65,536, and on that premise “two bytes is one character” was a reasonable choice. Java’s char, JavaScript’s strings and Qt’s QChar all made the same choice at the same time.

Unicode 2.0 broke the premise in 1996. Chinese characters alone exceeded 65,536, and opening the range to U+10FFFF created characters that sixteen bits cannot hold. Systems already built on 16 bits could not change the type, so they changed the encoding instead — UTF-16, writing one character as two units.

So Windows’s wchar_t today is not “one character” but “one UTF-16 code unit.” Name and reality have been at odds for thirty years.

71.2 The limit of UTF-16 and the surrogate pair

Unicode’s code point space runs from U+0000 to U+10FFFF. The first part, U+0000~U+FFFF, is the Basic Multilingual Plane (BMP), and that is as far as sixteen bits reach.

To write characters above it in 16-bit units, Unicode set aside a region that is never used for characters inside the BMP itself.

RegionRangeWhat
High surrogatesU+D800~U+DBFFThe first unit of a pair (1024 of them)
Low surrogatesU+DC00~U+DFFFThe second unit of a pair (1024)
(The range they express together)U+10000~U+10FFFF1024 × 1024 = 1,048,576 characters

Table 72.2

examples-en/ch71/surrogate.c

/* The limit of UTF-16 — surrogate pairs, and why "length" has three answers. */
#include <locale.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <uchar.h>
#include <wchar.h>

/* Split one code point into UTF-16 units.
   The BMP (U+0000..U+FFFF, minus D800..DFFF) is one unit as it stands;
   above it (U+10000..U+10FFFF) it takes two — a surrogate pair. */
static int to_utf16(unsigned long cp, char16_t out[2])
{
    if (cp > 0x10FFFFUL) return 0;                 /* outside Unicode */
    if (cp >= 0xD800UL && cp <= 0xDFFFUL) return 0; /* surrogates are not characters */
    if (cp < 0x10000UL) { out[0] = (char16_t)cp; return 1; }

    unsigned long v = cp - 0x10000UL;              /* down to 20 bits */
    out[0] = (char16_t)(0xD800UL + (v >> 10));     /* top 10 bits -> high surrogate */
    out[1] = (char16_t)(0xDC00UL + (v & 0x3FFUL)); /* low 10 bits -> low surrogate */
    return 2;
}

/* The other way — a pair back into a code point */
static unsigned long from_pair(char16_t hi, char16_t lo)
{
    return 0x10000UL + (((unsigned long)hi - 0xD800UL) << 10)
                     +  ((unsigned long)lo - 0xDC00UL);
}

static void report(const char *name, unsigned long cp)
{
    char16_t u16[2] = { 0, 0 };
    int n = to_utf16(cp, u16);

    char cpname[16];
    snprintf(cpname, sizeof cpname, "U+%04lX", cp);
    printf("  %-9s -> ", cpname);
    if (n == 0)      printf("cannot be represented   ");
    else if (n == 1) printf("%04X        one unit        ", u16[0]);
    else             printf("%04X %04X   back to U+%04lX", u16[0], u16[1],
                             from_pair(u16[0], u16[1]));
    printf("  %s\n", name);
}

/* Measure one string three ways */
static void lengths(const char *label, const char *utf8)
{
    size_t bytes = strlen(utf8);

    /* Count code points and UTF-16 units by hand */
    size_t cps = 0, u16units = 0;
    mbstate_t st;
    memset(&st, 0, sizeof st);
    for (size_t pos = 0; pos < bytes; ) {
        wchar_t wc = 0;
        size_t r = mbrtowc(&wc, utf8 + pos, bytes - pos, &st);
        if (r == (size_t)-1 || r == (size_t)-2) break;
        if (r == 0) r = 1;
        pos += r;
        cps++;
        char16_t tmp[2];
        u16units += (size_t)to_utf16((unsigned long)wc, tmp);
    }
    printf("  UTF-8 %2zu bytes  %zu code point%s  %zu UTF-16 unit%s   %s\n",
           bytes, cps, cps == 1 ? " " : "s", u16units,
           u16units == 1 ? " " : "s", label);
}

int main(void)
{
    puts("[code point -> UTF-16]");
    report("'A'", 0x41);
    report("'한' (Hangul)", 0xD55C);
    report("U+FFFD replacement char", 0xFFFD);
    report("a surrogate itself, D800", 0xD800);
    report("emoji U+1F600", 0x1F600);
    report("CJK extension U+2A6B2", 0x2A6B2);
    report("the last code point", 0x10FFFF);
    report("beyond the range", 0x110000);

    puts("\nThe arithmetic, for U+1F600:");
    unsigned long v = 0x1F600UL - 0x10000UL;
    printf("  0x1F600 - 0x10000 = 0x%05lX (20 bits)\n", v);
    printf("  top 10 bits 0x%03lX + 0xD800 = 0x%04lX\n", v >> 10, 0xD800UL + (v >> 10));
    printf("  low 10 bits 0x%03lX + 0xDC00 = 0x%04lX\n",
           v & 0x3FF, 0xDC00UL + (v & 0x3FF));

    const char *loc = setlocale(LC_CTYPE, "C.UTF-8");
    if (!loc) loc = setlocale(LC_CTYPE, "en_US.UTF-8");
    if (!loc) { puts("\nno UTF-8 locale — skipping the length demonstration"); return 0; }

    puts("\n[three lengths of the same string]");
    lengths("\"Hi\"", "Hi");
    lengths("\"한글\" (two Hangul syllables)", "한글");
    lengths("one emoji", "\xF0\x9F\x98\x80");
    lengths("a mixture", "a한\xF0\x9F\x98\x80");
    printf("  wchar_t here: %zu bytes -> UTF-32. On Windows: 2 bytes -> UTF-16.\n",
           sizeof(wchar_t));
    return 0;
}

Output

[code point -> UTF-16]
  U+0041    -> 0041        one unit          'A'
  U+D55C    -> D55C        one unit          '한' (Hangul)
  U+FFFD    -> FFFD        one unit          U+FFFD replacement char
  U+D800    -> cannot be represented     a surrogate itself, D800
  U+1F600   -> D83D DE00   back to U+1F600  emoji U+1F600
  U+2A6B2   -> D869 DEB2   back to U+2A6B2  CJK extension U+2A6B2
  U+10FFFF  -> DBFF DFFF   back to U+10FFFF  the last code point
  U+110000  -> cannot be represented     beyond the range

The arithmetic, for U+1F600:
  0x1F600 - 0x10000 = 0x0F600 (20 bits)
  top 10 bits 0x03D + 0xD800 = 0xD83D
  low 10 bits 0x200 + 0xDC00 = 0xDE00

[three lengths of the same string]
  UTF-8  2 bytes  2 code points  2 UTF-16 units   "Hi"
  UTF-8  6 bytes  2 code points  2 UTF-16 units   "한글" (two Hangul syllables)
  UTF-8  4 bytes  1 code point   2 UTF-16 units   one emoji
  UTF-8  8 bytes  3 code points  4 UTF-16 units   a mixture
  wchar_t here: 4 bytes -> UTF-32. On Windows: 2 bytes -> UTF-16.

The demonstration shows the arithmetic itself. For U+1F600:

  1. Subtract 0x10000 from the code point → 0x0F600 (it fits in 20 bits)
  2. Add the top ten bits to 0xD800 → 0xD83D (high)
  3. Add the bottom ten bits to 0xDC00 → 0xDE00 (low)

Surrogate values are not characters. U+D800~U+DFFF are assigned to no letter, and appearing alone in UTF-8 or UTF-32 they are invalid data (we meet them again in chapter 72′s validation). That is why trying to hold U+D800 as UTF-16 printed “cannot be represented”.

A common misconception. “The length of a string is the number of characters”

Which layer that sentence is about decides between three answers. The latter part of the demonstration measures them — one emoji is 4 bytes in UTF-8, 1 code point, 2 UTF-16 units.

So “length” means different things in different languages. C’s strlen counts bytes; the length of Java, JavaScript and C# counts UTF-16 units; Python 3′s len counts code points. JavaScript’s famous surprise — putting in one emoji and getting a length of 2 — is exactly this place.

And none of the three is the number of characters a reader sees. That fourth layer is chapter 72.

71.3 Windows — the W functions and their exact encoding

The Windows API offers nearly every function that takes a string in two versions.

SuffixString typeEncoding
A (ANSI)char *The process’s active code page (CP949 on a Korean install)
W (Wide)wchar_t * (WCHAR)UTF-16LE

Table 72.3

MessageBoxA and MessageBoxW are the pair, and the name MessageBox is a macro — it expands to W if UNICODE is defined and to A otherwise. Names like TCHAR and _T() are relics of the same era.

Platform note. Three roads for strings on Windows

① Use the W functions and convert at the boundary (the classic road). Keep the program’s insides in UTF-8 and convert to UTF-16 only when calling the API, with MultiByteToWideChar(CP_UTF8, …), and back with WideCharToMultiByte. Both are two-call functions — ask the required size first (pass 0 for the length and it returns the count), then fill.

② Write the whole program in UTF-16. The old way for Windows-only programs. Every literal carries an L, and you use wcslen and the wprintf family. Portability is given up.

③ Make the active code page UTF-8 (the modern prescription). Since Windows 10 1903, putting <activeCodePage>UTF-8</activeCodePage> in the application manifest makes the A functions take UTF-8. Then UTF-8 code written for Linux runs almost unchanged. For a new program this is the shortest road.

The console is separate again. To get Unicode out of a console, either turn on wide mode with _setmode(_fileno(stdout), _O_U16TEXT) or change the code page with SetConsoleOutputCP(CP_UTF8). “Korean comes out as question marks” is usually this.

In practice. Unpaired surrogates — the shadow over Windows file names

A Windows file name is “an array of UTF-16 units”, not “a valid Unicode string”. Because nothing checks, a name holding a lone high surrogate can exist in the file system.

Converting such a name to UTF-8 causes trouble — the value cannot be represented in valid UTF-8. WideCharToMultiByte either fails or substitutes U+FFFD, and then the file can no longer be opened.

Because of this some programs use an extension called “WTF-8” internally (a non-standard encoding that writes unpaired surrogates in UTF-8 style). Rust’s OsString is implemented that way on Windows.

The lesson: do not assume that a string the platform hands you is valid Unicode. File names, command-line arguments and environment variables especially.

71.4 Linux and glibc — large, and unused

glibc’s wchar_t is four-byte UCS-4 and __STDC_ISO_10646__ is defined; it is the implementation closest to the picture the standard drew. And yet Linux code in practice hardly uses wchar_t. There are four reasons.

ReasonExplanation
It uses four times the memoryFor mostly-ASCII text, 4× is a large waste
You must convert at every boundaryFiles, sockets and APIs are all bytes
It is tied to the localeConversion fails unless LC_CTYPE is UTF-8 (chapter 70)
UTF-8 already does most of the workstrlen, strcmp and strstr still work

Table 72.4

The last line is decisive. UTF-8 is ASCII-compatible, self-synchronising, and its byte order matches code-point order, so the existing byte functions remain mostly useful (chapter 72). So the Linux camp took the road of “UTF-8 inside too.”

71.5 What the toolkits chose — GTK and Qt

The same split appears at the application layer.

ToolkitString typeEncodingCode point type
GTK / GLibgchar * (= char *)UTF-8gunichar (32-bit), gunichar2 (16-bit)
QtQStringUTF-16QChar (a 16-bit unit)
Windows APIWCHAR *UTF-16

Table 72.5

GLib set the rule “every string is UTF-8” and laid functions on top of it — g_utf8_strlen (characters), g_utf8_next_char, g_utf8_validate. It does not use wchar_t at all: a decision not to put a type that changes per platform into an API.

Qt went the other way. Its insides are fixed at UTF-16, and it crosses the boundary with QString::fromUtf8 and toUtf8. The gain is no conversion when meeting the Windows API; the cost is the trap that QString::size() counts UTF-16 units.

Q. So what should my program choose?

A. For new code, UTF-8 byte strings. The table above says why — smaller, at home with existing C functions, the same as the representation on files and networks, and less tied to the locale.

The places wchar_t is needed are narrow: calling the Windows API directly (and only at the boundary), and fitting an existing library that only takes wide characters.

If you genuinely need fixed-width code points, use char32_t rather than wchar_t. It is four bytes everywhere and its being UTF-32 is guaranteed by a macro — two things wchar_t cannot give you.

71.6 Streams have an orientation

Wide characters come with I/O functions of their own — wprintf, fputws, getwc, fgetws, and WEOF in place of EOF. But there is a rule that is not widely known.

A stream has an orientation, and once settled it cannot be changed.

StateMeaning
No orientationA freshly opened stream. Neither yet
Byte-orientedSettled by the first use of a byte function such as fputs or fprintf
Wide-orientedSettled by the first use of a wide function such as fputws or fwprintf

Table 72.6

Using the opposite family after it is settled is undefined behaviour. The fwide function asks (pass 0) or settles it in advance while there is none.

examples-en/ch71/wide_io.c

/* Streams have an orientation — byte or wide, and it cannot be undone. */
#include <locale.h>
#include <stdio.h>
#include <wchar.h>

/* fwide(stream, 0) only asks.
   Negative = byte-oriented, 0 = not yet decided, positive = wide-oriented. */
static const char *orientation(FILE *f)
{
    int w = fwide(f, 0);
    return w < 0 ? "byte" : w > 0 ? "wide" : "none yet";
}

int main(void)
{
    setlocale(LC_ALL, "");

    /* Open a file and watch the orientation settle */
    FILE *f = tmpfile();
    if (!f) { perror("tmpfile"); return 1; }

    printf("a freshly opened stream:   %s\n", orientation(f));

    /* fwide can settle it in advance — but only while there is none */
    fwide(f, 1);
    printf("after fwide(f, 1):         %s\n", orientation(f));

    fputws(L"a line written wide\n", f);
    printf("after fputws:              %s\n", orientation(f));

    /* Writing byte functions to a wide-oriented stream is undefined behaviour,
       so we do not try it — we only show that the orientation will not move. */
    fwide(f, -1);
    printf("fwide(f, -1) to undo it:   %s (it does not move)\n", orientation(f));

    rewind(f);
    wchar_t line[64];
    if (fgetws(line, 64, f)) printf("read back:                 %ls", line);
    fclose(f);

    /* A second stream, settled the other way */
    FILE *g = tmpfile();
    if (!g) { perror("tmpfile"); return 1; }
    printf("\na second stream:           %s\n", orientation(g));
    fputs("a line written byte-wise\n", g);
    printf("after fputs:               %s\n", orientation(g));
    fclose(g);

    /* The standard streams work the same way. This program has used printf… */
    printf("\nthe orientation of stdout: %s\n", orientation(stdout));
    puts("-> because this program started with printf.");
    puts("   Mixing wprintf in from here is undefined behaviour.");
    return 0;
}

Output

a freshly opened stream:   none yet
after fwide(f, 1):         wide
after fputws:              wide
fwide(f, -1) to undo it:   wide (it does not move)
read back:                 a line written wide

a second stream:           none yet
after fputs:               byte

the orientation of stdout: byte
-> because this program started with printf.
   Mixing wprintf in from here is undefined behaviour.

The demonstration shows the rule plainly. A freshly opened stream has no orientation; fwide(f, 1) can make it wide; and afterwards fwide(f, -1) does not move it back. And this program’s stdout is already byte-oriented because it began with printf — calling wprintf here would be outside the contract.

Counter-example. Mixing printf and wprintf

printf("name: ");
wprintf(L"%ls\n", name);      /* wide output on an already byte-oriented stdout */

At best the output interleaves; at worst nothing appears. On Windows, calling printf after turning on _O_U16TEXT can kill the program outright.

One discipline — one family per stream. If you decide on wide output, do it throughout the program; otherwise do not use it at all. This book recommends the latter: what wide functions gain is smaller than the portability they cost.

Recap

What to rememberThe point
The splitLinux 4-byte UTF-32, Windows 2-byte UTF-16
The causeThe 1990s premise “Unicode = 16 bits” broke in 1996
SurrogatesU+10000 and above as a D800~DBFF + DC00~DFFF pair. The values themselves are not characters
LengthBytes, code points and UTF-16 units all differ
WindowsW functions are UTF-16LE. New code: a UTF-8 code page via the manifest
File namesMay not be valid Unicode (unpaired surrogates)
ToolkitsGTK = UTF-8 char*, Qt = UTF-16 QString
If you need fixed widthchar32_t, not wchar_t
Stream orientationOnce settled it cannot change. Mixing is undefined behaviour

Table 72.7

We have seen the platforms as they are. The next chapter is the conclusion — how, in practice, to handle Unicode and multibyte encodings. The three layers of length, normalisation, UTF-8 validation, and the traps of the legacy two-byte encodings that are still with us.