108 The outside world — files, streams, time, random numbers

What to know first

chapter 78, Streams in reality · streams

chapter 26, Input · input

Looking back

Chapter 9 described a stream as designed so that “the program does not know what it is connected to”, and chapter 76 said fopen reports failure with null. Then what failure is most often missed in a file API?

A. The partial write. A failure to open is conspicuous, but the case of write returning without writing all that was requested is easy to forget — it really happens when the disk fills, when a signal cuts in, or when the other end is a pipe. So this library has two variants. proven_fs_write returns the number of bytes actually written, and proven_fs_write_all repeats until all is written and then reports only success or failure. What most code wants is the latter, and the former remains so as not to hide that fact.

The need for this chapter, and its context

Only after the self-contained layers are built does the library step outside. This order follows from the promise made in chapter 100: the price of portability changes the moment you touch an operating system, so the layer that stands without one is finished first. It also sets up chapter 109, where there is no OS at all.

By the end of this chapter

From here we touch the operating system. Opening, reading and writing files, buffered streams, reading and formatting time, and random numbers — those random numbers that become entirely different things according to the purpose. Chapter 9′s story of streams and chapter 26′s story of input are completed here as real APIs.

The questions this chapter answers

What becomes of random numbers for secrets if there is no operating system?

108.1 The life of one file#

A file is the first resource in this part that touches the outside world. The discipline of making and giving back is the same as in the earlier chapters, but the kinds of failure are far more numerous.

examples-en/proven-world/fslife.c

/* The life of one file — the open modes, partial writes, the position, and
   saving safely. How failure arrives as a value where the outside world is
   touched. */
#include <proven.h>

/* a helper for seeing the bytes read as text (to keep a comma out of a macro argument) */
static proven_u8str_view_t as_text(const proven_byte_t *p, proven_size_t n)
{
    return (proven_u8str_view_t){ .ptr = p, .size = n };
}

static const char *codename(proven_err_t e)
{
    switch (e) {
    case PROVEN_OK:               return "OK";
    case PROVEN_ERR_NOT_FOUND:    return "NOT_FOUND";
    case PROVEN_ERR_PERMISSION:   return "PERMISSION";
    case PROVEN_ERR_IO:           return "IO";
    case PROVEN_ERR_EOF:          return "EOF";
    case PROVEN_ERR_INVALID_ARG:  return "INVALID_ARG";
    default:                      return "(other)";
    }
}

int main(void)
{
    proven_allocator_t scratch = proven_heap_allocator();
    proven_u8str_view_t path = PROVEN_LIT("build/ch77-demo.txt");

    /* ── (1) opening — the mode weaves bit flags together ────────── */
    proven_result_file_t opened = proven_fs_open(
        scratch, path,
        PROVEN_FS_WRITE | PROVEN_FS_CREATE | PROVEN_FS_TRUNC);
    if (!proven_is_ok(opened.err)) {
        proven_println("open failed: {}", PROVEN_ARG(codename(opened.err)));
        return 1;
    }
    proven_file_t f = opened.value;
    proven_println("open            -> OK (write, create, truncate)");

    /* ── (2) the two versions of writing ─────────────────────────── */
    proven_u8str_view_t line = PROVEN_LIT("first line\nsecond line\n");

    /* _write returns "how much was really written" — a partial write can happen */
    proven_result_size_t w = proven_fs_write(f, proven_mem_view_from_u8(line));
    proven_println("write           -> err={} bytes written={} (requested {})",
                   PROVEN_ARG(codename(w.err)), PROVEN_ARG(w.value),
                   PROVEN_ARG(line.size));

    /* _write_all repeats until everything is written and then reports success or failure only */
    proven_err_t e = proven_fs_write_all(f, proven_mem_view_from_u8(PROVEN_LIT("third\n")));
    proven_println("write_all       -> {} (what most code wants)",
                   PROVEN_ARG(codename(e)));

    /* ── (3) nailing it to the disk — flush and sync differ ──────── */
    e = proven_fs_sync(f);
    proven_println("sync            -> {} (only this far does it survive a power cut)",
                   PROVEN_ARG(codename(e)));

    proven_result_size_t sz = proven_fs_size(f);
    proven_result_u64_t at = proven_fs_tell(f);
    proven_println("size={} tell={}", PROVEN_ARG(sz.value), PROVEN_ARG(at.val));
    (void)proven_fs_close(f);

    /* ── (4) reading — what is asked for and what is read differ ─── */
    proven_result_file_t ro = proven_fs_open(scratch, path, PROVEN_FS_READ);
    if (proven_is_ok(ro.err)) {
        proven_file_t r = ro.value;
        proven_byte_t buf[16];
        proven_result_size_t got = proven_fs_read(r, (proven_mem_mut_t){ buf, sizeof buf });
        proven_println("read(16)        -> bytes read={} \"{}\"",
                       PROVEN_ARG(got.value),
                       PROVEN_ARG(as_text(buf, got.value)));

        /* the position is rewound and it is read again */
        (void)proven_fs_seek(r, 0, PROVEN_FS_SEEK_SET);
        proven_result_size_t again = proven_fs_read(r, (proven_mem_mut_t){ buf, 5 });
        proven_println("seek(0)+read(5) -> {} bytes", PROVEN_ARG(again.value));
        (void)proven_fs_close(r);
    }

    /* ── (5) reading it all at once — when the size is unknown ───── */
    proven_result_u8str_t all = proven_fs_read_all_u8str(scratch, path);
    if (proven_is_ok(all.err)) {
        proven_u8str_t s = all.value;
        proven_println("read_all        -> {} bytes (the whole file at one go)",
                       PROVEN_ARG(proven_u8str_as_view(&s).size));
        proven_u8str_destroy(scratch, &s);
    }

    /* ── (6) a file that is not there — failure arrives as a value ─ */
    proven_result_file_t missing =
        proven_fs_open(scratch, PROVEN_LIT("build/no-such-file.txt"), PROVEN_FS_READ);
    proven_println("opening a missing file -> {} (check which code the platform layer maps it to)",
                   PROVEN_ARG(codename(missing.err)));

    /* ── (7) an atomic save — written to a temporary and swapped in ─ */
    e = proven_fs_write_file_atomic(scratch, PROVEN_LIT("build/ch77-atomic.txt"),
                                    proven_mem_view_from_u8(PROVEN_LIT("all or nothing\n")));
    proven_println("write_file_atomic -> {} (no half-written file is left behind)",
                   PROVEN_ARG(codename(e)));

    (void)proven_fs_remove(scratch, path);
    (void)proven_fs_remove(scratch, PROVEN_LIT("build/ch77-atomic.txt"));
    return 0;
}

Output

open            -> OK (write, create, truncate)
write           -> err=OK bytes written=23 (requested 23)
write_all       -> OK (what most code wants)
sync            -> OK (only this far does it survive a power cut)
size=29 tell=29
read(16)        -> bytes read=16 "first line
secon"
seek(0)+read(5) -> 5 bytes
read_all        -> 29 bytes (the whole file at one go)
opening a missing file -> NOT_FOUND (check which code the platform layer maps it to)
write_file_atomic -> OK (no half-written file is left behind)

The life cycle is four steps, and at each step failure comes as a value.

step	function	to know
opening	`proven_fs_open(scratch, path, mode)`	a scratch allocator is needed (for path conversion)
writing, reading	`_write`/`_write_all`, `_read`	amount requested ≠ amount handled
nailing it down	`proven_fs_sync(file)`	closing alone does not guarantee it reaches the disk
closing	`proven_fs_close(file)`	it has a return value — there is something to check

Table 108.1 — The steps of handling a file

The mode weaves bit flags. Instead of a string like the standard fopen’s "w+b", named values are joined with |.

flag	meaning
`PROVEN_FS_READ`	reading
`PROVEN_FS_WRITE`	writing
`PROVEN_FS_APPEND`	appending at the end
`PROVEN_FS_CREATE`	create it if absent
`PROVEN_FS_TRUNC`	empty it if present
`PROVEN_FS_CREATE_NEW`	★ fail if it already exists (`PROVEN_ERR_EXISTS`) — creating anew without a race

Table 108.2 — What the file flags mean

Flags are better than string modes in two ways. First, the combination is visible — there is no need to memorise what "a+" exactly is. Second, things absent from string modes, such as CREATE_NEW, can be expressed. When making a lock file or a temporary file, “fail if it already exists” is the only road that blocks a race condition (recall chapter 79′s tmpnam story). The code that comes back then is PROVEN_ERR_EXISTS. It is a different code from the PROVEN_ERR_IO of a failing disk, so “choose another name” can be told from “give up”.

Handles travel by value. That the functions take a proven_file_t by value rather than by pointer is the mark of it, because what is inside is about one integer descriptor. So the discipline of not using that value again after closing is still a human’s part.

108.1.1 The convenience functions that read and write in one go#

In practice the file is often small enough to be handled whole. Then opening, reading and closing need not be woven by hand.

function	what it does	caution
`proven_fs_read_all(alloc, path)`	the whole file as bytes	no ceiling — only for a path you trust
`proven_fs_read_all_bounded(alloc, path, max)`	reads at most `max` bytes	past that, `ERR_OUT_OF_BOUNDS` and no buffer
`proven_fs_read_all_u8str(alloc, path)`	the whole file as a string	the same as `read_all`. it does not check the encoding
`proven_fs_write_file(scratch, path, data)`	writing it whole	die in the middle and a half-written file is left
`proven_fs_write_file_atomic(...)`	write to a temporary and swap	★ no half-written file is left
`proven_fs_write_file_durable(...)`	atomic + `sync`	it survives a power cut. the slowest

Table 108.3 — The whole-file read and write functions and what to watch

Start with the first two reading rows, read_all and read_all_bounded. proven_fs_read_all reads until the source ends. When the path names /dev/zero or a FIFO that never ends, the buffer grows until the allocator refuses. A path that came from outside the program is read with proven_fs_read_all_bounded — it refuses before allocating when the size is known in advance, and at the moment byte max + 1 arrives when it is not.

On the writing side, the difference between the last three rows matters in practice. Code that overwrites a configuration file or saved data, if it dies in the middle, leaves a file that is neither the original nor the new one, and the standard practice that prevents it is “write to a temporary file and rename” (renaming is atomic within the same file system). _atomic does that work for you, and _durable hangs a sync on it as well so that it survives a power cut.

The temporary file is named <path>.pvtmp followed by thirteen random characters. The name cannot be guessed in advance, so nobody can block the write by creating that name first. When the process dies in the middle of a write, the temporary file stays. The library does not remove it — it cannot tell an abandoned file from one another process is writing right now. Cleaning up is the program’s job, and proven_fs_is_staging_name answers “does this name have the shape of that temporary file”.

examples-en/proven-world/fileio.c

#include <proven.h>
#include <stdio.h>

int main(void)
{
    proven_allocator_t alloc = proven_heap_allocator();
    proven_u8str_view_t path = proven_u8str_view_from_cstr("proven_demo.txt");
    const char *text = "one\ntwo\nthree\n";

    /* writing: created if absent, emptied if present */
    proven_result_file_t opened = proven_fs_open(
        alloc, path, PROVEN_FS_WRITE | PROVEN_FS_CREATE | PROVEN_FS_TRUNC);
    if (!proven_is_ok(opened.err)) {
        printf("open for write failed (err=%d)\n", (int)opened.err);
        return 1;
    }
    proven_file_t f = opened.value;

    proven_mem_view_t src = {
        .ptr = (const proven_byte_t *)text,
        .size = proven_cstr_len(text)
    };
    proven_err_t e = proven_fs_write_all(f, src);   /* it repeats over partial writes for us */
    printf("write_all  : %s (%zu bytes)\n", proven_is_ok(e) ? "ok" : "failed", src.size);
    (void)proven_fs_close(f);

    /* reading */
    opened = proven_fs_open(alloc, path, PROVEN_FS_READ);
    if (!proven_is_ok(opened.err)) return 1;
    f = opened.value;

    proven_result_size_t sz = proven_fs_size(f);
    printf("size       : %zu bytes\n", sz.value);

    proven_byte_t buf[64];
    proven_result_size_t got = proven_fs_read(f, (proven_mem_mut_t){ .ptr = buf, .size = sizeof buf });
    printf("read       : %zu bytes\n", got.value);

    /* what was read is handled as a view — sliced line by line */
    proven_u8str_view_t all = { .ptr = buf, .size = got.value };
    proven_u8str_view_t nl = proven_u8str_view_from_cstr("\n");
    proven_size_t start = 0;
    int line = 0;
    while (start < all.size) {
        proven_size_t hit = proven_u8str_view_find(all, start, nl);
        proven_size_t end = (hit == PROVEN_INDEX_NOT_FOUND) ? all.size : hit;
        proven_u8str_view_t v = proven_u8str_view_slice(all, start, end - start);
        printf("  line %d   : %.*s\n", ++line, (int)v.size, (const char *)v.ptr);
        if (hit == PROVEN_INDEX_NOT_FOUND) break;
        start = hit + 1;
    }
    (void)proven_fs_close(f);

    /* open a file that is not there and the failure arrives as a value */
    proven_result_file_t missing = proven_fs_open(
        alloc, proven_u8str_view_from_cstr("no_such_file.txt"), PROVEN_FS_READ);
    printf("missing    : %s\n", proven_is_ok(missing.err) ? "opened" : "refused with an error code");

    (void)proven_fs_remove(alloc, path);
    return 0;
}

Output

write_all  : ok (14 bytes)
size       : 14 bytes
read       : 14 bytes
  line 1   : one
  line 2   : two
  line 3   : three
missing    : refused with an error code

Several things stand out.

The path is a view too. proven_u8str_view_from_cstr("...") — wherever the string came from, it is handled as a pointer and a length (chapter 105).

Opening needs an allocator. The signature’s first argument is a scratch allocator, because temporary memory may be needed to turn the path into the form the operating system requires. Chapter 104′s rule is honestly kept here too — if it can allocate, it takes an allocator.

What is read becomes a view. Bind the buffer and the number of bytes read and from then on all of chapter 105′s tools can be used. That is what the example used to divide the lines, and copying never happened once.

Counter-example. Trusting size and assuming that much was read

proven_result_size_t sz = proven_fs_size(f);
proven_byte_t *buf = malloc(sz.value);
(void)proven_fs_read(f, (proven_mem_mut_t){ buf, sz.value });
process(buf, sz.value);      /* was that much really read? */

A file’s size and the amount this read brought are different. From pipes, terminals and networks it comes a little at a time, and even a file may end in the middle. The number read returned must be used, and that is why this library returns the read result as a bundle. The fact chapter 26 stated — “input is not a keyboard but a stream” — becomes a practical rule here.

108.2 Streams — reading and writing with a buffer#

System calls are expensive. Call write one byte at a time and that cost accumulates as it stands. So standard C’s FILE* kept a buffer (chapter 9′s story of line buffering), and proven puts a stream in the same place — differing in two ways.

The caller gives the buffer. The stream does not allocate in secret.
The failure of a flush comes as a value. It is not quietly swallowed on closing.

These two aim at the same problem. In buffered writing the real failure shows itself not in write but in the flush, and missing that failure creates data that “was believed successful but is not on the disk”. It is also the place databases and file systems are most careful about (the reason proven_fs_sync exists separately).

108.3 Time — two different clocks#

Two different things are mixed together in what we call time.

Calendar time — a time a human reads, like “5 August 2026, 09:00”. It is what you show to users and write into records. Time zones, leap seconds and summer time are entangled in it.
Monotonic time — a scale that does not go backwards. It is used for measuring elapsed time.

Mix the two and you get a well-known bug. Measure elapsed time with a calendar clock and the moment the system sets the clock back — or, if you measure in local time, the moment summer time ends — a negative elapsed time comes out. Let that value into a timeout calculation and the wait either never ends or expires at once.

Date formatting uses the format syntax seen in chapter 106 unchanged — named placeholders with width and fill specified, as in "{year}-{month:0>2}-{day:0>2}". The difference from strftime’s %Y-%m-%d is that there is no need to memorise what symbol means what.

A common misconception. “Time is just a number, so adding and subtracting is fine”

Not in calendar time. A day is not always 86400 seconds (a summer-time transition day is 23 or 25 hours), the lengths of months differ, and time zones change by political decision. “A month later” is not arithmetic but a calendar rule. The difference between two monotonic times, on the other hand, may be handled as a plain number — that is one more reason to use a monotonic clock for measuring elapsed time.

108.4 Random numbers — the purpose decides the generator#

Few tools are as much “the same name, different demands” as random numbers. The library does not hide this but divides it into three.

examples-en/proven-world/rng.c

#include <proven.h>
#include <stdio.h>

int main(void)
{
    /* Reproducible random numbers: the same seed = the same sequence. For tests and simulations */
    proven_xoshiro256ss_t g;
    proven_xoshiro256ss_seed(&g, 12345);
    proven_rng_t rng = proven_xoshiro256ss_rng(&g);

    printf("seeded run 1:");
    for (int i = 0; i < 5; i++) printf(" %llu", (unsigned long long)proven_rng_below(rng, 100));
    printf("\n");

    proven_xoshiro256ss_seed(&g, 12345);          /* rewound with the same seed */
    printf("seeded run 2:");
    for (int i = 0; i < 5; i++) printf(" %llu", (unsigned long long)proven_rng_below(rng, 100));
    printf("\n");

    /* a number in a range: the bounds are included */
    proven_xoshiro256ss_seed(&g, 7);
    printf("dice        :");
    for (int i = 0; i < 8; i++) printf(" %lld", (long long)proven_rng_range(rng, 1, 6));
    printf("\n");

    /* randomness for secrets is not taken from here — the OS has its own source */
    proven_byte_t key[16];
    bool ok = proven_random_bytes(key, sizeof key);
    printf("os entropy  : %s (%zu bytes requested)\n", ok ? "available" : "unavailable", sizeof key);
    return 0;
}

Output

seeded run 1: 74 13 96 4 55
seeded run 2: 74 13 96 4 55
dice        : 5 2 6 6 6 6 1 1
os entropy  : available (16 bytes requested)

Reproducible random numbers (proven_xoshiro256ss_t) are for simulation, games and testing. The same seed gives the same sequence — the very reason the example’s two lines are identical, and also the property that lets a failed test be reproduced. They are fast but predictable, so they are never used for secrets.

Random numbers for secrets (proven_random_bytes) come from the operating system’s cryptographic source of randomness. They are used for values an attacker must not guess — keys, tokens, session identifiers. When one 64-bit value is needed, use proven_random_u64_checked. It returns false when there is no source of randomness. The similarly named proven_random_u64 returns 0 then, so it is not used for secrets.

The third is the compromise between the two, proven_chacha_rng_t, which takes a seed once from the OS source of randomness and then continues a cryptographically secure sequence quickly.

In practice. The accidents predictable random numbers made

Accidents breaking this distinction have happened repeatedly. Typical cases are an online card game whose hands were predicted because it used the time as a seed, a case where session identifiers made with a fast random generator let one into somebody else’s account, and, above all, the 2008 incident in which Debian’s OpenSSL patch¹ deleted the entropy-gathering code so that the number of generable keys shrank to a few tens of thousands. The last required every key already made to be discarded. The lesson is summed up in one line of the library’s documentation — random numbers for secrets come only from a cryptographic source of randomness.

Q. What becomes of random numbers for secrets if there is no operating system?

A. They cannot be obtained, so proven_random_bytes returns false — and this is an important design decision. Many libraries slip back to the time or an address value in this situation, and then you have the worst state of “believed safe but predictable”. This library does not fall back; it declares failure. If the board has a real source of entropy (a hardware random number generator) it can be registered and used. Rather than quietly give something bad, say there is none — another face of the principle met continually in this part.

108.5 Memory mapping#

There is one more way of reading a file. Hanging the file whole in the address space and accessing it with a pointer — proven_mmap_* is that window. It is advantageous when reading a large file by roaming randomly over it, and it is used when several processes share the same file.

The price is clear too. If the file is truncated while the mapped region is being touched, the program can receive a signal and die, and portability is lower than the file API’s. So this tool is not “what is used by default” but “what is chosen when there is a reason”.

Recap

The outside world in summary.

what it does	API	to beware of
open and close	`proven_fs_open/close`	needs a scratch allocator
reading	`proven_fs_read`	amount requested ≠ amount read
writing	`proven_fs_write` / `_write_all`	partial writes
nailing it to the disk	`proven_fs_sync`	flush ≠ sync
buffered I/O	`proven_stream_*`	the caller gives the buffer
the current time	`proven_time_now_datetime`	not used for measuring elapsed time
date formatting	`proven_time_u8_fmt`	named placeholders such as `{year}`
reproducible random	`proven_xoshiro256ss_*`	not used for secrets
reading whole	`proven_fs_read_all` / `_bounded`	give a ceiling to a path from outside
random for secrets	`proven_random_bytes` / `_u64_checked`	false if absent — it does not fall back
memory mapping	`proven_mmap_*`	only when there is a reason

Table 108.4 — The APIs of the outside world and what to watch

What remains now are the boundaries — running several things at once, and the place with no operating system at all. The next chapter covers those boundaries, and the one after it closes this part.

Notes

DSA-1571-1 openssl — predictable random number generator (CVE-2008-0166). 2008. Debian Security Advisory, 2008-05-13. lists.debian.org/debian-security-announce/2008/msg00152.html ↩

← PrevContentsNext →