Proven C Book한국어 GitHub

70 Wide characters ① — wchar_t and multibyte conversion

What to know first

chapter 9, Characters and text · code points and encodings
chapter 68, Locales ① · LC_CTYPE
chapter 67, Character classification · the limit of per-byte judgement

Looking back

Chapter 9 said one character in UTF-8 is one to four bytes. Then why not make a type that holds one whole character? Does C have one?

A. It does. It is wchar_t, and C95 brought it in on exactly that thought — hold one character in one unit. The trouble is that the thought did not survive the next thirty years.

The standard fixed neither the size nor the encoding. It said only “an integer type whose range of values can represent distinct codes for all members of the largest extended character set specified among the supported locales.” The result: 4-byte UTF-32 on Linux, 2-byte UTF-16 on Windows — and on the latter, “one character = one unit” does not hold.

This chapter takes the exact contract of the type, and the process by which bytes unfold into characters, one step at a time. The consequences of the mismatch are the next chapter.

The need for this chapter, and its context

The code points and encodings learned in chapter 9 finally come down into C’s types and functions. The sixty-chapter wait is because this subject leans entirely on locales (chapter 68). Reverse the order and there is no saying what mbrtowc depends on.

By the end of this chapter

What wchar_t really is. The standard’s definition and the macros an implementation uses to declare itself, the five kinds of character constant and string with their types, sizes and bytes, the difference between MB_CUR_MAX and MB_LEN_MAX, why mbstate_t has to exist, and a measured, step-by-step walk through mbrtowc eating bytes and producing characters.

The questions this chapter answers

  1. What bytes does writing "한" in the source produce?
  2. UTF-8 is at most four bytes; why is MB_CUR_MAX six?

70.1 What the standard fixes, and what it does not

wchar_t is an integer type defined in <stddef.h>. The standard’s definition, verbatim, is “an integer type whose range of values can represent distinct codes for all members of the largest extended character set specified among the supported locales.”

Look at what is missing from it.

The standard fixesIt does not fix
That it is an integer typeHow many bytes
That each character has a distinct valueWhether it is signed
That basic characters are non-negativeWhich encoding (not even that it is Unicode)

Table 71.1

So portable code cannot assume a wchar_t value is a Unicode code point. Instead there are macros by which an implementation declares the fact.

MacroIf it is defined
__STDC_ISO_10646__wchar_t values equal ISO/IEC 10646 (Unicode) code points. Its value has the form yyyymmL
__STDC_UTF_16__char16_t is UTF-16
__STDC_UTF_32__char32_t is UTF-32
__STDC_MB_MIGHT_NEQ_WC__Even for a basic character, 'x' and L'x' may differ in value

Table 71.2

examples-en/ch70/wide_lit.c

/* Five kinds of character constant and string — type, element size, bytes. */
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <uchar.h>
#include <wchar.h>

/* Lay out any array, one element at a time, in hex.
   char is signed on many implementations (chapter 27), so mask to the element
   width — otherwise 0xED shows up as FFFFFFED. */
#define DUMP(label, arr)                                                     \
    do {                                                                     \
        printf("  %-10s %zu-byte elements x %zu:", (label),                  \
               sizeof (arr)[0], sizeof (arr) / sizeof (arr)[0]);             \
        for (size_t i = 0; i < sizeof (arr) / sizeof (arr)[0]; i++)          \
            printf(" %0*llX", (int)(sizeof (arr)[0] * 2),                    \
                   (unsigned long long)(arr)[i]                              \
                   & ((1ULL << (8 * sizeof (arr)[0])) - 1));                 \
        putchar('\n');                                                       \
    } while (0)

int main(void)
{
    puts("[character constants] the same 'A' written five ways");
    printf("  %-14s type size %zu, value %d\n",  "'A'",   sizeof('A'), 'A');
    printf("  %-14s type size %zu, value %ld\n", "L'A'",  sizeof(L'A'), (long)L'A');
    printf("  %-14s type size %zu, value %u\n",  "u'A'",  sizeof(u'A'), (unsigned)u'A');
    printf("  %-14s type size %zu, value %u\n",  "U'A'",  sizeof(U'A'), (unsigned)U'A');
    printf("  %-14s type size %zu, value %u\n",  "u8'A'", sizeof(u8'A'), (unsigned)u8'A');
    puts("  ('A' is an int — in C a character constant is not a char)");

    puts("\n[strings] the single letter \"\" written five ways");
    static const char     s_plain[] =   "";
    static const char8_t  s_u8[]    = u8"";
    static const char16_t s_u16[]   =  u"";
    static const char32_t s_u32[]   =  U"";
    static const wchar_t  s_wide[]  =  L"";
    DUMP("char",     s_plain);
    DUMP("char8_t",  s_u8);
    DUMP("char16_t", s_u16);
    DUMP("char32_t", s_u32);
    DUMP("wchar_t",  s_wide);

    puts("\n[a character beyond the BMP] U+1F600 (grinning face)");
    static const char     e_plain[] =   "\U0001F600";
    static const char16_t e_u16[]   =  u"\U0001F600";
    static const char32_t e_u32[]   =  U"\U0001F600";
    static const wchar_t  e_wide[]  =  L"\U0001F600";
    DUMP("char",     e_plain);
    DUMP("char16_t", e_u16);   /* <- two elements: a surrogate pair */
    DUMP("char32_t", e_u32);
    DUMP("wchar_t",  e_wide);
    puts("  Only char16_t needs two — 16 bits cannot hold it, so it splits.");

    puts("\n[what the implementation declares]");
#ifdef __STDC_ISO_10646__
    printf("  __STDC_ISO_10646__ = %ldL — wchar_t values are Unicode code points\n",
           (long)__STDC_ISO_10646__);
#else
    puts("  no __STDC_ISO_10646__ — the wchar_t encoding is implementation-defined (Windows)");
#endif
#ifdef __STDC_UTF_16__
    puts("  __STDC_UTF_16__  = 1 — char16_t is UTF-16");
#endif
#ifdef __STDC_UTF_32__
    puts("  __STDC_UTF_32__  = 1 — char32_t is UTF-32");
#endif
    printf("  sizeof(wchar_t) = %zu, WCHAR_MIN = %ld, WCHAR_MAX = %lld\n",
           sizeof(wchar_t), (long)WCHAR_MIN, (long long)WCHAR_MAX);
    return 0;
}

Output

[character constants] the same 'A' written five ways
  'A'            type size 4, value 65
  L'A'           type size 4, value 65
  u'A'           type size 2, value 65
  U'A'           type size 4, value 65
  u8'A'          type size 1, value 65
  ('A' is an int — in C a character constant is not a char)

[strings] the single letter "한" written five ways
  char       1-byte elements x 4: ED 95 9C 00
  char8_t    1-byte elements x 4: ED 95 9C 00
  char16_t   2-byte elements x 2: D55C 0000
  char32_t   4-byte elements x 2: 0000D55C 00000000
  wchar_t    4-byte elements x 2: 0000D55C 00000000

[a character beyond the BMP] U+1F600 (grinning face)
  char       1-byte elements x 5: F0 9F 98 80 00
  char16_t   2-byte elements x 3: D83D DE00 0000
  char32_t   4-byte elements x 2: 0001F600 00000000
  wchar_t    4-byte elements x 2: 0001F600 00000000
  Only char16_t needs two — 16 bits cannot hold it, so it splits.

[what the implementation declares]
  __STDC_ISO_10646__ = 201706L — wchar_t values are Unicode code points
  __STDC_UTF_16__  = 1 — char16_t is UTF-16
  __STDC_UTF_32__  = 1 — char32_t is UTF-32
  sizeof(wchar_t) = 4, WCHAR_MIN = -2147483648, WCHAR_MAX = 2147483647

The last part of the demonstration is this machine’s answer — __STDC_ISO_10646__ is defined, and wchar_t is four bytes. On Windows that macro is not defined. Two bytes cannot hold every Unicode character as one distinct value, so the condition is not met. That one line is the seed of the whole of chapter 71.

70.2 Five kinds of character constant and string

C23 has five ways to write a character. Each has its own type and encoding.

NotationType of the constantString elementEncoding
'A'intcharThe execution character set (usually UTF-8 bytes)
u8'A' / u8"…"unsigned charchar8_tUTF-8
u'A' / u"…"char16_tchar16_tUTF-16
U'A' / U"…"char32_tchar32_tUTF-32
L'A' / L"…"wchar_twchar_tImplementation-defined

Table 71.3

The first part of the demonstration puts this table in the flesh. Three things stand out.

First, the type of 'A' is int, not char. In C a character constant is an int, so sizeof('A') is 4 (a place where C and C++ differ).

Second, the u8 prefix applies to character constants too in C23 — but only for characters representable as a single UTF-8 code unit, that is, ASCII.

Third, the same “한” becomes entirely different bytes in each notation. As char it is the three bytes ED 95 9C; as char16_t, the single unit D55C; as char32_t and wchar_t, the single unit 0000D55C.

And with a character beyond the BMP the decisive difference appears. Holding U+1F600, only char16_t needs two elementsD83D DE00. It does not fit in 16 bits, so it is split into a pair, and that pair is called a surrogate (chapter 71).

Q. What bytes does writing "한" in the source produce?

A. It passes through two stages — the source file’s encoding and the execution character set (chapter 17′s chain of encodings). The compiler decides in which encoding to read the source (GCC defaults to UTF-8; -finput-charset changes it), then converts to the execution character set to make the bytes of the literal (-fexec-charset).

If those two disagree, the code compiles and only the strings are broken. So code that must be portable writes non-ASCII characters as universal character names rather than directly in the source.

const char *s = "";        /* U+D55C into the execution character set */
const char32_t *t = U"\U0001F600";

\u takes four digits, \U eight. The demonstration writes its emoji that way — that is how this book’s examples produce the same bytes under any editor and any build setting.

70.3 MB_CUR_MAX and MB_LEN_MAX

Two macros with similar names mean quite different things.

MacroHeaderWhat
MB_CUR_MAX<stdlib.h>The maximum bytes per character in the current locale — it changes at run time
MB_LEN_MAX<limits.h>An upper bound over any supported locale — a compile-time constant

Table 71.4

Chapter 68′s demonstration showed MB_CUR_MAX changing with the locale: 1 (C), 2 (EUC-KR), 6 (UTF-8). Which to use when sizing a buffer splits here — size the array with MB_LEN_MAX, and judge inside the loop with MB_CUR_MAX.

Q. UTF-8 is at most four bytes; why is MB_CUR_MAX six?

A. It is a scar from history. UTF-8′s original definition (RFC 2279) allowed up to six bytes and could write codes of 31 bits. When Unicode stopped at U+10FFFF, RFC 3629 cut it to four — but glibc left MB_CUR_MAX at the roomier old value.

The practical conclusion does not change — size buffers by MB_CUR_MAX (or MB_LEN_MAX). Sizing by 4 because UTF-8 never exceeds it will overflow under a locale with another encoding.

70.4 Three layers of conversion function

Several functions move between byte strings and wide characters; they sort into three layers.

LayerFunctionsCharacter
The old layer (C89)mbtowc, wctomb, mbstowcs, wcstombsHide the conversion state inside the function — not reentrant
The state-exposed layer (C95)mbrtowc, wcrtomb, mbsrtowcs, wcsrtombsTake an mbstate_t *. The r is for restartable
The Unicode layer (C11)mbrtoc16, c16rtomb, mbrtoc32, c32rtomb<uchar.h>. Between the locale’s encoding and UTF-16/32

Table 71.5

New code uses the middle layer. The r in the name means restartable, and what makes restarting possible is mbstate_t.

70.4.1 Why mbstate_t has to exist

Why does conversion need state? Two reasons.

First, characters arrive cut in half. Reading from a network or a file puts buffer boundaries in the middle of characters. With two of “한”‘s three bytes in hand, the function must answer “not enough yet” rather than “wrong”, and must carry on when the next piece arrives.

Second, some encodings have state. Encodings such as ISO-2022-JP switch between “kanji from here” and “ASCII from here” with escape sequences. Without remembering which mode you are in, the same byte is a different character.

mbstate_t is the opaque object holding those two. Three rules: filled with zeroes it is the initial state, each conversion uses its own object, and you do not look inside.

mbstate_t st;
memset(&st, 0, sizeof st);      /* to the initial state */

70.5 mbrtowc — taken apart one step at a time

The key function of the chapter. The contract is dense, and reading the return value exactly is the whole of it.

size_t mbrtowc(wchar_t * restrict pwc, const char * restrict s,
               size_t n, mbstate_t * restrict ps);
Return valueMeaning
0A null character was completed (the result is the null wide character)
1~nThis many bytes were consumed and one character was completed
(size_t)-2All n bytes were examined and no character is complete yet — the state is updated
(size_t)-1An invalid sequence. errno holds EILSEQ and the state is unspecified

Table 71.6

examples-en/ch70/mbrtowc_step.c

/* mbrtowc — bytes turning into wchar_t, one step at a time. */
#include <errno.h>
#include <locale.h>
#include <stdlib.h>          /* MB_CUR_MAX */
#include <stdio.h>
#include <string.h>
#include <wchar.h>

/* The four kinds of return value, in words */
static const char *explain(size_t r)
{
    if (r == 0)           return "hit the null character (returns 0)";
    if (r == (size_t)-1)  return "invalid sequence (errno=EILSEQ)";
    if (r == (size_t)-2)  return "not a whole character yet — needs more";
    return "consumed this many bytes and produced one character";
}

static void walk(const char *label, const char *s, size_t len)
{
    mbstate_t st;
    memset(&st, 0, sizeof st);      /* the initial conversion state */

    printf("\n[%s] %zu bytes:", label, len);
    for (size_t i = 0; i < len; i++) printf(" %02X", (unsigned char)s[i]);
    puts("");

    size_t pos = 0;
    while (pos < len) {
        wchar_t wc = 0;
        errno = 0;
        size_t r = mbrtowc(&wc, s + pos, len - pos, &st);

        printf("  at %zu: mbrtowc -> ", pos);
        if (r == (size_t)-1)      printf("(size_t)-1   ");
        else if (r == (size_t)-2) printf("(size_t)-2   ");
        else                      printf("%-12zu", r);
        printf("%s", explain(r));

        if (r != (size_t)-1 && r != (size_t)-2)
            printf("  -> U+%04lX", (unsigned long)wc);
        puts("");

        if (r == (size_t)-1 || r == (size_t)-2) break;
        pos += (r == 0) ? 1 : r;
    }
}

int main(void)
{
    /* This one needs a UTF-8 locale — if there is none, say so and stop */
    const char *loc = setlocale(LC_CTYPE, "C.UTF-8");
    if (!loc) loc = setlocale(LC_CTYPE, "en_US.UTF-8");
    if (!loc) { puts("no UTF-8 locale — skipping this demonstration"); return 0; }
    printf("LC_CTYPE=%s, MB_CUR_MAX=%zu\n", loc, (size_t)MB_CUR_MAX);

    walk("two ASCII letters", "Hi", 2);
    walk("one Hangul syllable", "", 3);
    walk("an emoji beyond the BMP", "\xF0\x9F\x98\x80", 4);

    /* A truncated character: two of three bytes -> (size_t)-2 */
    walk("a truncated character", "\xED\x95", 2);

    /* An invalid sequence: starting on a trail byte -> (size_t)-1 */
    walk("an invalid sequence", "\x9C\x41", 2);

    /* The state carries over: feed the pieces in two calls and they join */
    puts("\n[arriving in pieces, as from a stream] \"\" fed as 2+1 bytes");
    mbstate_t st;
    memset(&st, 0, sizeof st);
    wchar_t wc = 0;
    size_t r1 = mbrtowc(&wc, "\xED\x95", 2, &st);
    printf("  first call (2 bytes): %s\n", explain(r1));
    size_t r2 = mbrtowc(&wc, "\x9C", 1, &st);
    printf("  second call (1 byte): %s -> U+%04lX\n", explain(r2), (unsigned long)wc);
    puts("  mbstate_t remembers how far it got, so the pieces join up.");
    return 0;
}

Output

LC_CTYPE=C.UTF-8, MB_CUR_MAX=6

[two ASCII letters] 2 bytes: 48 69
  at 0: mbrtowc -> 1           consumed this many bytes and produced one character  -> U+0048
  at 1: mbrtowc -> 1           consumed this many bytes and produced one character  -> U+0069

[one Hangul syllable] 3 bytes: ED 95 9C
  at 0: mbrtowc -> 3           consumed this many bytes and produced one character  -> U+D55C

[an emoji beyond the BMP] 4 bytes: F0 9F 98 80
  at 0: mbrtowc -> 4           consumed this many bytes and produced one character  -> U+1F600

[a truncated character] 2 bytes: ED 95
  at 0: mbrtowc -> (size_t)-2   not a whole character yet — needs more

[an invalid sequence] 2 bytes: 9C 41
  at 0: mbrtowc -> (size_t)-1   invalid sequence (errno=EILSEQ)

[arriving in pieces, as from a stream] "한" fed as 2+1 bytes
  first call (2 bytes): not a whole character yet — needs more
  second call (1 byte): consumed this many bytes and produced one character -> U+D55C
  mbstate_t remembers how far it got, so the pieces join up.

The demonstration shows those four in turn. ASCII goes one byte at a time, Hangul takes three at once, an emoji beyond the BMP takes four. A truncated piece gives (size_t)-2, and a piece starting on a trail byte gives (size_t)-1.

The final part shows exactly why mbstate_t exists. Feeding “한”‘s three bytes as 2+1, the first call returned (size_t)-2 and the second completed U+D55C. The state object was holding “I have seen two bytes so far.”

A common misconception. (size_t)-2 is an error”

It is not. It is a normal intermediate state. The only error is (size_t)-1.

Lump the two together and a stream processor throws away perfectly good characters from fragmented input. Code reading 4096 bytes at a time from a network meets (size_t)-2 at nearly every buffer’s end — and what to do then is to prepend the leftover bytes to the next buffer and read on, not to stop with an error.

That both values are large size_t numbers is a trap of its own. Received into an int and compared with < 0, the result is implementation-dependent nonsense. Always receive them in a size_t and compare directly with (size_t)-1 and (size_t)-2.

70.5.1 A whole string at once — mbsrtowcs

To move a whole string rather than one character at a time, use mbsrtowcs. Two knacks belong to its contract.

mbstate_t st;
memset(&st, 0, sizeof st);
const char *p = utf8;                       /* a pointer holding the position */
size_t need = mbsrtowcs(NULL, &p, 0, &st);  /* (1) ask the length first */
if (need == (size_t)-1) { /* EILSEQ */ }

wchar_t *buf = malloc((need + 1) * sizeof *buf);
p = utf8;                                   /* (2) reset pointer and state */
memset(&st, 0, sizeof st);
mbsrtowcs(buf, &p, need + 1, &st);          /* (3) and convert for real */

A null first argument counts only the length — so the buffer size can be known in advance. And the reason s is a const char ** is to hand back how far it read when the conversion stopped partway. The old mbstowcs lacked that, which is why it could not be used for stream processing.

Platform note. Conversion is tied to the locale

One fact easily missed here. mbrtowc understands UTF-8 only because the current locale is UTF-8. With LC_CTYPE set to "C", the same bytes are an invalid sequence.

So a program using wide characters must call setlocale(LC_ALL, "") at once, and must cope with the fact that the locale might not be UTF-8. That is why the demonstration explicitly looks for a UTF-8 locale and skips if there is none.

The constraint “these functions cannot choose the encoding” is a large part of why wide characters are avoided in practice. The premise that the program’s encoding and the user’s locale must agree is especially hard to keep in a server.

Recap

What to rememberThe point
wchar_tAn integer type. The standard fixes neither size nor encoding
Unicode guaranteeOnly if __STDC_ISO_10646__ is defined
Character constants'A' is an int. Four prefixes: u8, u, U, L
Beyond the BMPOnly char16_t takes two elements (a surrogate pair)
MB_CUR_MAXChanges with the locale. The bound is MB_LEN_MAX
ConversionUse the ones with r (mbrtowc)
mbstate_tZero it to start. For cut characters and stateful encodings
Return values-2 is not an error but “more needed”
The premiseLC_CTYPE must be that encoding

Table 71.7

We have seen wchar_t’s contract and the process of conversion. The next chapter is how that contract split across real platforms — Windows’s two bytes, Linux’s four, and what happened when UTF-16 could not hold all of Unicode.