93 The outside world — files, streams, time, random numbers

What to know first

chapter 63, Streams in reality · streams

chapter 25, Input · input

Looking back

Chapter 10′s design had a stream be one where “the program does not know what it is connected to”, and chapter 61 said fopen reports failure with null. Then what failure is most often missed in a file API?

A. The partial write. A failure to open is conspicuous, but the case of write returning without writing all that was requested is easy to forget — it really happens when the disk fills, when a signal cuts in, or when the other end is a pipe. So this library has two editions. proven_fs_write returns the number of bytes actually written, and proven_fs_write_all repeats until all is written and then reports only success or failure. What most code wants is the latter, and the former remains so as not to hide that fact.

The need for this chapter, and its context

Only after the self-contained layers are built does the library step outside. This order follows from the promise made in chapter 85: the price of portability changes the moment you touch an operating system, so the layer that stands without one is finished first. It also sets up chapter 94, where there is no OS at all.

By the end of this chapter

From here we touch the operating system. Opening, reading and writing files, buffered streams, reading and formatting time, and random numbers — those random numbers that become entirely different things according to the purpose. Chapter 10′s story of streams and chapter 25′s story of input are completed here as real APIs.

The questions this chapter answers

What becomes of random numbers for secrets if there is no operating system?

93.1 The life of one file

A file is the first resource in this part that touches the outside world. The discipline of making and giving back is the same as in the earlier chapters, but the kinds of failure are far more numerous.

examples-en/ch93/fslife.c

/* The life of one file — the open modes, partial writes, the position, and
   saving safely. How failure arrives as a value where the outside world is
   touched. */
#include <proven.h>

/* a helper for seeing the bytes read as text (to keep a comma out of a macro argument) */
static proven_u8str_view_t as_text(const proven_byte_t *p, proven_size_t n)
{
    return (proven_u8str_view_t){ .ptr = p, .size = n };
}

static const char *codename(proven_err_t e)
{
    switch (e) {
    case PROVEN_OK:               return "OK";
    case PROVEN_ERR_NOT_FOUND:    return "NOT_FOUND";
    case PROVEN_ERR_PERMISSION:   return "PERMISSION";
    case PROVEN_ERR_IO:           return "IO";
    case PROVEN_ERR_EOF:          return "EOF";
    case PROVEN_ERR_INVALID_ARG:  return "INVALID_ARG";
    default:                      return "(other)";
    }
}

int main(void)
{
    proven_allocator_t scratch = proven_heap_allocator();
    proven_u8str_view_t path = PROVEN_LIT("build/ch77-demo.txt");

    /* ── (1) opening — the mode weaves bit flags together ────────── */
    proven_result_file_t opened = proven_fs_open(
        scratch, path,
        PROVEN_FS_WRITE | PROVEN_FS_CREATE | PROVEN_FS_TRUNC);
    if (!proven_is_ok(opened.err)) {
        proven_println("open failed: {}", PROVEN_ARG(codename(opened.err)));
        return 1;
    }
    proven_file_t f = opened.value;
    proven_println("open            -> OK (write, create, truncate)");

    /* ── (2) the two versions of writing ─────────────────────────── */
    proven_u8str_view_t line = PROVEN_LIT("first line\nsecond line\n");

    /* _write returns "how much was really written" — a partial write can happen */
    proven_result_size_t w = proven_fs_write(f, proven_mem_view_from_u8(line));
    proven_println("write           -> err={} bytes written={} (requested {})",
                   PROVEN_ARG(codename(w.err)), PROVEN_ARG(w.value),
                   PROVEN_ARG(line.size));

    /* _write_all repeats until everything is written and then reports success or failure only */
    proven_err_t e = proven_fs_write_all(f, proven_mem_view_from_u8(PROVEN_LIT("third\n")));
    proven_println("write_all       -> {} (what most code wants)",
                   PROVEN_ARG(codename(e)));

    /* ── (3) nailing it to the disk — flush and sync differ ──────── */
    e = proven_fs_sync(f);
    proven_println("sync            -> {} (only this far does it survive a power cut)",
                   PROVEN_ARG(codename(e)));

    proven_result_size_t sz = proven_fs_size(f);
    proven_result_u64_t at = proven_fs_tell(f);
    proven_println("size={} tell={}", PROVEN_ARG(sz.value), PROVEN_ARG(at.val));
    (void)proven_fs_close(f);

    /* ── (4) reading — what is asked for and what is read differ ─── */
    proven_result_file_t ro = proven_fs_open(scratch, path, PROVEN_FS_READ);
    if (proven_is_ok(ro.err)) {
        proven_file_t r = ro.value;
        proven_byte_t buf[16];
        proven_result_size_t got = proven_fs_read(r, (proven_mem_mut_t){ buf, sizeof buf });
        proven_println("read(16)        -> bytes read={} \"{}\"",
                       PROVEN_ARG(got.value),
                       PROVEN_ARG(as_text(buf, got.value)));

        /* the position is rewound and it is read again */
        (void)proven_fs_seek(r, 0, PROVEN_FS_SEEK_SET);
        proven_result_size_t again = proven_fs_read(r, (proven_mem_mut_t){ buf, 5 });
        proven_println("seek(0)+read(5) -> {} bytes", PROVEN_ARG(again.value));
        (void)proven_fs_close(r);
    }

    /* ── (5) reading it all at once — when the size is unknown ───── */
    proven_result_u8str_t all = proven_fs_read_all_u8str(scratch, path);
    if (proven_is_ok(all.err)) {
        proven_u8str_t s = all.value;
        proven_println("read_all        -> {} bytes (the whole file at one go)",
                       PROVEN_ARG(proven_u8str_as_view(&s).size));
        proven_u8str_destroy(scratch, &s);
    }

    /* ── (6) a file that is not there — failure arrives as a value ─ */
    proven_result_file_t missing =
        proven_fs_open(scratch, PROVEN_LIT("build/no-such-file.txt"), PROVEN_FS_READ);
    proven_println("opening a missing file -> {} (check which code the platform layer maps it to)",
                   PROVEN_ARG(codename(missing.err)));

    /* ── (7) an atomic save — written to a temporary and swapped in ─ */
    e = proven_fs_write_file_atomic(scratch, PROVEN_LIT("build/ch77-atomic.txt"),
                                    proven_mem_view_from_u8(PROVEN_LIT("all or nothing\n")));
    proven_println("write_file_atomic -> {} (no half-written file is left behind)",
                   PROVEN_ARG(codename(e)));

    (void)proven_fs_remove(scratch, path);
    (void)proven_fs_remove(scratch, PROVEN_LIT("build/ch77-atomic.txt"));
    return 0;
}

Output

open            -> OK (write, create, truncate)
write           -> err=OK bytes written=23 (requested 23)
write_all       -> OK (what most code wants)
sync            -> OK (only this far does it survive a power cut)
size=29 tell=29
read(16)        -> bytes read=16 "first line
secon"
seek(0)+read(5) -> 5 bytes
read_all        -> 29 bytes (the whole file at one go)
opening a missing file -> IO (check which code the platform layer maps it to)
write_file_atomic -> OK (no half-written file is left behind)

The life cycle is four steps, and at each step failure comes as a value.

step	function	to know
opening	`proven_fs_open(scratch, path, mode)`	a scratch allocator is needed (for path conversion)
writing, reading	`_write`/`_write_all`, `_read`	amount requested ≠ amount handled
nailing it down	`proven_fs_sync(file)`	closing alone does not leave it on the disk
closing	`proven_fs_close(file)`	it has a return value — there is something to check

Table 94.1

The mode weaves bit flags. Instead of a string like the standard fopen’s "w+b", named values are joined with |.

flag	meaning
`PROVEN_FS_READ`	reading
`PROVEN_FS_WRITE`	writing
`PROVEN_FS_APPEND`	appending at the end
`PROVEN_FS_CREATE`	create it if absent
`PROVEN_FS_TRUNC`	empty it if present
`PROVEN_FS_CREATE_NEW`	★ fail if it already exists — creating anew without a race

Table 94.2

Two things are better than string modes. First, the combination is visible — there is no need to memorise what "a+" exactly is. Second, things absent from string modes, such as CREATE_NEW, can be expressed. When making a lock file or a temporary file, “fail if it already exists” is the only road that blocks a race condition (recall chapter 64′s tmpnam story).

Handles travel by value. That the functions take a proven_file_t by value rather than by pointer is the mark of it, because what is inside is about one integer descriptor. So the discipline of not using that value again after closing is still a human’s part.

93.1.1 The convenience functions that read and write in one go

In half the cases in practice the file is small and may be handled whole. Then opening, reading and closing need not be woven by hand.

function	what it does	caution
`proven_fs_read_all(alloc, path)`	the whole file as bytes	a large file eats memory
`proven_fs_read_all_u8str(alloc, path)`	the whole file as a string	the same. it does not check the encoding
`proven_fs_write_file(scratch, path, data)`	writing it whole	die in the middle and a half-written file is left
`proven_fs_write_file_atomic(...)`	write to a temporary and swap	★ no half-written file is left
`proven_fs_write_file_durable(...)`	atomic + `sync`	it survives a power cut. the slowest

Table 94.3

The difference between the last three rows matters in practice. Code that overwrites a configuration file or saved data, if it dies in the middle, leaves a file that is neither the original nor the new one, and the standard practice that prevents it is “write to a temporary file and rename” (renaming is atomic within the same file system). _atomic does that work for you, and _durable hangs a sync on it as well so that it survives a power cut.

examples-en/ch93/fileio.c

#include <proven.h>
#include <stdio.h>

int main(void)
{
    proven_allocator_t alloc = proven_heap_allocator();
    proven_u8str_view_t path = proven_u8str_view_from_cstr("proven_demo.txt");
    const char *text = "one\ntwo\nthree\n";

    /* writing: created if absent, emptied if present */
    proven_result_file_t opened = proven_fs_open(
        alloc, path, PROVEN_FS_WRITE | PROVEN_FS_CREATE | PROVEN_FS_TRUNC);
    if (!proven_is_ok(opened.err)) {
        printf("open for write failed (err=%d)\n", (int)opened.err);
        return 1;
    }
    proven_file_t f = opened.value;

    proven_mem_view_t src = {
        .ptr = (const proven_byte_t *)text,
        .size = proven_cstr_len(text)
    };
    proven_err_t e = proven_fs_write_all(f, src);   /* it repeats over partial writes for us */
    printf("write_all  : %s (%zu bytes)\n", proven_is_ok(e) ? "ok" : "failed", src.size);
    (void)proven_fs_close(f);

    /* reading */
    opened = proven_fs_open(alloc, path, PROVEN_FS_READ);
    if (!proven_is_ok(opened.err)) return 1;
    f = opened.value;

    proven_result_size_t sz = proven_fs_size(f);
    printf("size       : %zu bytes\n", sz.value);

    proven_byte_t buf[64];
    proven_result_size_t got = proven_fs_read(f, (proven_mem_mut_t){ .ptr = buf, .size = sizeof buf });
    printf("read       : %zu bytes\n", got.value);

    /* what was read is handled as a view — sliced line by line */
    proven_u8str_view_t all = { .ptr = buf, .size = got.value };
    proven_u8str_view_t nl = proven_u8str_view_from_cstr("\n");
    proven_size_t start = 0;
    int line = 0;
    while (start < all.size) {
        proven_size_t hit = proven_u8str_view_find(all, start, nl);
        proven_size_t end = (hit == PROVEN_INDEX_NOT_FOUND) ? all.size : hit;
        proven_u8str_view_t v = proven_u8str_view_slice(all, start, end - start);
        printf("  line %d   : %.*s\n", ++line, (int)v.size, (const char *)v.ptr);
        if (hit == PROVEN_INDEX_NOT_FOUND) break;
        start = hit + 1;
    }
    (void)proven_fs_close(f);

    /* open a file that is not there and the failure arrives as a value */
    proven_result_file_t missing = proven_fs_open(
        alloc, proven_u8str_view_from_cstr("no_such_file.txt"), PROVEN_FS_READ);
    printf("missing    : %s\n", proven_is_ok(missing.err) ? "opened" : "refused with an error code");

    (void)proven_fs_remove(alloc, path);
    return 0;
}

Output

write_all  : ok (14 bytes)
size       : 14 bytes
read       : 14 bytes
  line 1   : one
  line 2   : two
  line 3   : three
missing    : refused with an error code

Several things stand out.

The path is a view too. proven_u8str_view_from_cstr("...") — wherever the string came from, it is handled as a pointer and a length (chapter 90).

Opening needs an allocator. The signature’s first argument is a scratch allocator, because temporary memory may be needed to turn the path into the form the operating system requires. Chapter 89′s rule is honestly kept here too — if it can allocate, it takes an allocator.

What is read becomes a view. Bind the buffer and the number of bytes read and from then on all of chapter 90′s tools can be used. That is what the example used to divide the lines, and copying never happened once.

Counter-example. Trusting size and assuming that much was read

proven_result_size_t sz = proven_fs_size(f);
proven_byte_t *buf = malloc(sz.value);
(void)proven_fs_read(f, (proven_mem_mut_t){ buf, sz.value });
process(buf, sz.value);      /* was that much really read? */

A file’s size and the amount this read brought are different. From pipes, terminals and networks it comes a little at a time, and even a file may end in the middle. The number read returned must be used, and that is why this library returns the read result as a bundle. The fact chapter 25 stated — “input is not a keyboard but a stream” — becomes a practical rule here.

93.2 Streams — reading and writing with a buffer

System calls are expensive. Call write one byte at a time and that cost accumulates as it stands. So standard C’s FILE* kept a buffer (chapter 10′s story of line buffering), and proven puts a stream in the same place — differing in two ways.

The caller gives the buffer. The stream does not allocate in secret.

- The failure of a flush comes as a value. It is not quietly swallowed on closing.

These two aim at the same problem. In buffered writing the real failure shows itself not in write but in the flush, and missing that failure creates data that “was believed successful but is not on the disk”. It is also the place databases and file systems are most careful about (the reason proven_fs_sync exists separately).

93.3 Time — two different clocks

Two different things are mixed together in time.

Calendar time — a time a human reads, like “5 August 2026, 09:00”. It is used for showing to users and leaving in records. Time zones, leap seconds and summer time are entangled in it.
Monotonic time — a scale that does not go backwards. It is used for measuring elapsed time.

Mix the two and you get the famous bug. Measure elapsed time with a calendar clock and the moment the system adjusts the time or summer time changes, a negative elapsed time comes out. Let that value into a timeout calculation and it waits forever or expires at once.

Date formatting uses the format syntax seen in chapter 61 as it is — named placeholders with width and fill specified, as in "{year}-{month:0>2}-{day:0>2}". Unlike strftime’s %Y-%m-%d, the difference is that there is no need to memorise what symbol means what.

A common misconception. “Time is just a number, so adding and subtracting is fine”

Not in calendar time. A day is not always 86400 seconds (a summer-time transition day is 23 or 25 hours), the lengths of months differ, and time zones change by political decision. “A month later” is not arithmetic but a calendar rule. The difference of monotonic times, on the other hand, may be handled as a plain number — that is one more reason to use a monotonic clock for measuring elapsed time.

93.4 Random numbers — the purpose settles the thing

Few tools are as much “the same name, different demands” as random numbers. The library does not hide this but divides it into three.

examples-en/ch93/rng.c

#include <proven.h>
#include <stdio.h>

int main(void)
{
    /* Reproducible random numbers: the same seed = the same sequence. For tests and simulations */
    proven_xoshiro256ss_t g;
    proven_xoshiro256ss_seed(&g, 12345);
    proven_rng_t rng = proven_xoshiro256ss_rng(&g);

    printf("seeded run 1:");
    for (int i = 0; i < 5; i++) printf(" %llu", (unsigned long long)proven_rng_below(rng, 100));
    printf("\n");

    proven_xoshiro256ss_seed(&g, 12345);          /* rewound with the same seed */
    printf("seeded run 2:");
    for (int i = 0; i < 5; i++) printf(" %llu", (unsigned long long)proven_rng_below(rng, 100));
    printf("\n");

    /* a number in a range: the bounds are included */
    proven_xoshiro256ss_seed(&g, 7);
    printf("dice        :");
    for (int i = 0; i < 8; i++) printf(" %lld", (long long)proven_rng_range(rng, 1, 6));
    printf("\n");

    /* randomness for secrets is not taken from here — the OS has its own source */
    proven_byte_t key[16];
    bool ok = proven_random_bytes(key, sizeof key);
    printf("os entropy  : %s (%zu bytes requested)\n", ok ? "available" : "unavailable", sizeof key);
    return 0;
}

Output

seeded run 1: 74 13 96 4 55
seeded run 2: 74 13 96 4 55
dice        : 5 2 6 6 6 6 1 1
os entropy  : available (16 bytes requested)

Reproducible random numbers (proven_xoshiro256ss_t) are for simulation, games and testing. The same seed gives the same sequence — the very reason the example’s two lines are identical, and also the property that lets a failed test be reproduced. They are fast but predictable, so they are never used for secrets.

Random numbers for secrets (proven_random_bytes) come from the operating system’s cryptographic source of randomness. They are used for values an attacker must not guess — keys, tokens, session identifiers.

The third is the compromise between the two, proven_chacha_rng_t, which takes a seed once from the OS source of randomness and then continues a cryptographically secure sequence quickly.

In practice. The accidents predictable random numbers made

Accidents breaking this distinction have happened repeatedly. An online card game whose hands were predicted because it used the time as a seed, a case where session identifiers made with a fast random generator let one into somebody else’s account, and, representatively, the 2008 incident in which Debian’s OpenSSL patch¹ deleted the entropy-gathering code so that the number of generable keys shrank to a few tens of thousands. The last required every key already made to be discarded. The lesson is summed up in one line of the library’s documentation — random numbers for secrets come only from a cryptographic source of randomness.

Q. What becomes of random numbers for secrets if there is no operating system?

A. They cannot be obtained, so proven_random_bytes returns a falsehood — and this is an important design decision. Many libraries slip back to the time or an address value in this situation, and then you have the worst state of “believed safe but predictable”. This library does not fall back; it declares failure. If the board has a real source of entropy (a hardware random number generator) it can be registered and used. Rather than quietly give something bad, say there is none — another face of the principle met continually in this part.

93.5 Memory mapping

There is one more way of reading a file. Hanging the file whole in the address space and accessing it with a pointer — proven_mmap_* is that window. It is advantageous when reading a large file by roaming randomly over it, and it is used when several processes share the same file.

The price is clear too. If the file is truncated while the mapped region is being touched, the program can receive a signal and die, and portability is lower than the file API’s. So this tool is not “what is used by default” but “what is chosen when there is a reason”.

Recap

The outside world in summary.

what it does	API	to beware of
open and close	`proven_fs_open/close`	needs a scratch allocator
reading	`proven_fs_read`	amount requested ≠ amount read
writing	`proven_fs_write` / `_write_all`	partial writes
nailing it to the disk	`proven_fs_sync`	flush ≠ sync
buffered I/O	`proven_stream_*`	the caller gives the buffer
the current time	`proven_time_now_datetime`	not used for measuring elapsed time
date formatting	`proven_time_u8_fmt`	named placeholders such as `{year}`
reproducible random	`proven_xoshiro256ss_*`	not used for secrets
random for secrets	`proven_random_bytes`	a falsehood if absent — it does not fall back
memory mapping	`proven_mmap_*`	only when there is a reason

Table 94.4

What remains now is the boundaries — the way of running several things overlapped, and the place with no operating system at all. The last chapter closes this part.

Notes

DSA-1571-1 openssl — predictable random number generator (CVE-2008-0166). 2008. Debian Security Advisory, 2008-05-13. lists.debian.org/debian-security-announce/2008/msg00152.html ↩

← PrevContentsNext →