Chapter 9: Networking#

Part V - Talking to the operating system. Prerequisites: Part II (1, 2, 3) and Chapter 5 section 4 (the two clocks) and its section on streams. After this chapter you can open a TCP or UDP socket, put a time limit on everything that waits, tell a refused connection from a lost one, serve many sockets from one thread, and write a protocol that does not care what carries it.

This chapter covers net.h. Like Chapter 5 it needs an operating system and is not part of a freestanding build; a hosted build can leave it out with PROVEN_NO_NET.

Table of contents#

1. Three rules
2. Addresses
3. Deadlines
4. TCP
5. UDP
6. Many sockets, one thread
7. Transports
8. A pair, and waking a loop
9. A selector: thousands of sockets
10. What the platforms do differently
11. What is not here

1. Three rules#

BSD sockets are forty years old and every one of their sharp edges has a name. net.h is a thin layer whose whole job is to take three of them away.

Every call that can wait takes a deadline. recv on a plain socket waits as long as the peer keeps it waiting. A program built that way works on a desk and stops in production the first time a client opens a connection and says nothing - and one such client is enough to stop a server that handles one connection at a time. Here there is no way to write that program: a read, a write, an accept and a connect each take a deadline argument, and you must say something.

A failure says which failure. "The call failed" is not enough to decide what to do next. Whether the other side said no, went away, or finished properly are three different situations with three different responses:

PROVEN_ERR_IO is what is left when none of these fits.

Nothing is hidden. No call in this header allocates. A socket is a small value you own and must close. A reader or writer over a socket uses state you declare. The one piece of process-wide state - starting Winsock on Windows - is done once, on first use, and is not yours to manage.

2. Addresses#

A proven_net_addr_t says where a socket is or where it should go. It is a plain value: copy it, compare it, store it in a table.

typedef struct {
    proven_net_family_t family;   /* IPV4, IPV6 or UNIX; NONE when unset */
    proven_u16 port;              /* host byte order: the number you would write down */
    proven_u32 scope_id;          /* IPv6 zone for a link-local address; 0 otherwise */
    proven_byte_t ip[16];         /* network order: 4 bytes for IPv4, 16 for IPv6 */
    proven_u8 path_len;           /* Unix-domain only */
    char path[104];               /* Unix-domain only */
} proven_net_addr_t;

Reference#

Two decisions in the parser are deliberate. An IPv4 literal is exactly four decimal numbers: 127.1 and 2130706433 are refused although older parsers accept both, and so is 010.0.0.1, which those parsers read as octal eight. And IPv6 has many spellings going in but one coming out

• lowercase, no leading zeros, the longest run of zero groups as :: (RFC 5952) - so two equal

addresses print as equal text and the text can be a map key or a log field you search for.

Cautions, and what goes wrong#

proven_net_resolve blocks and has no deadline. It is the system resolver, which offers none, and on a broken network it can take tens of seconds. It is the one call in this header that waits without a limit, and the header says so rather than pretend.

Wrong - resolving inside the loop that serves every other connection:

for (;;) {
    wait_for_events();
    if (client_wants_upstream)
        proven_net_resolve(name, 443, addrs, 4, &n);   /* wrong: every client waits on DNS */
}

Correct - resolve where waiting is acceptable: at startup, or on a job (Chapter 6), and hand the address to the loop.

Binding to "any" publishes the service. proven_net_addr_any listens on every interface the machine has, including the ones that face a network. A tool meant for your own machine binds proven_net_addr_loopback.

Compiled and run by the test suite:

/*
 * Addresses are plain values: build one, parse one from text, print it, compare two.
 * Nothing in this program opens a socket or asks the network anything.
 */

static bool prints_as(const proven_net_addr_t *addr, proven_u8str_view_t want) {
    proven_byte_t text[PROVEN_NET_ADDR_TEXT_MAX];
    proven_size_t n = 0;
    if (proven_net_addr_format(addr, (proven_mem_mut_t){ text, sizeof text }, &n) != PROVEN_OK) return false;
    return proven_u8str_view_eq((proven_u8str_view_t){ text, n }, want);
}

int main(void) {
    /* Built from parts. The port is the number you would write down - host byte order. */
    proven_net_addr_t web = proven_net_addr_ipv4(192, 0, 2, 10, 80);
    EXAMPLE_REQUIRE(prints_as(&web, PROVEN_LIT("192.0.2.10:80")), "an IPv4 address prints as address:port");

    /* The two fixed addresses a program needs most: this machine only, and every interface. */
    proven_net_addr_t local = proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 8080);
    proven_net_addr_t everywhere = proven_net_addr_any(PROVEN_NET_FAMILY_IPV6, 8080);
    EXAMPLE_REQUIRE(prints_as(&local, PROVEN_LIT("127.0.0.1:8080")), "loopback is 127.0.0.1");
    EXAMPLE_REQUIRE(prints_as(&everywhere, PROVEN_LIT("[::]:8080")), "any is ::, and IPv6 is bracketed before a port");

    /* Parsed from text. Many spellings of one IPv6 address parse to the same bytes, and there
     * is exactly one spelling on the way out, so the text can be compared and used as a key. */
    proven_net_addr_t a, b;
    EXAMPLE_REQUIRE(proven_net_addr_parse(PROVEN_LIT("2001:DB8:0:0:0:0:0:1"), 443, &a) == PROVEN_OK, "the long form parses");
    EXAMPLE_REQUIRE(proven_net_addr_parse(PROVEN_LIT("[2001:db8::1]"), 443, &b) == PROVEN_OK, "the short, bracketed form parses");
    EXAMPLE_REQUIRE(proven_net_addr_eq(&a, &b), "and they are the same address");
    EXAMPLE_REQUIRE(prints_as(&a, PROVEN_LIT("[2001:db8::1]:443")), "printed in the one canonical form");

    /* What is not an address is refused - including what other parsers quietly accept. */
    proven_net_addr_t bad;
    EXAMPLE_REQUIRE(proven_net_addr_parse(PROVEN_LIT("010.0.0.1"), 80, &bad) == PROVEN_ERR_INVALID_FORMAT,
                    "a leading zero is refused, not read as octal");
    EXAMPLE_REQUIRE(proven_net_addr_parse(PROVEN_LIT("example.com"), 80, &bad) == PROVEN_ERR_INVALID_FORMAT,
                    "a name is not a literal: that is proven_net_resolve's job");

    /* Sixteen raw bytes, in network order, when the address comes from a packet or a file. */
    proven_byte_t raw[16] = { 0xfe, 0x80, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1 };
    proven_net_addr_t link_local = proven_net_addr_ipv6(raw, 22, 3);
    EXAMPLE_REQUIRE(prints_as(&link_local, PROVEN_LIT("[fe80::1%3]:22")), "a link-local address carries its zone");

    /* A Unix-domain address is a filesystem path. */
    proven_net_addr_t sock;
    EXAMPLE_REQUIRE(proven_net_addr_unix(PROVEN_LIT("/run/app.sock"), &sock) == PROVEN_OK, "a path makes an address");
    EXAMPLE_REQUIRE(prints_as(&sock, PROVEN_LIT("/run/app.sock")), "and prints as the path");

    /* Resolving a literal is answered without a query, so this line works with no network.
     * A real name goes to the system resolver, blocks, and has no deadline: call it where
     * that is acceptable. */
    proven_net_addr_t found[4];
    proven_size_t count = 0;
    EXAMPLE_REQUIRE(proven_net_resolve(PROVEN_LIT("192.0.2.10"), 80, found, 4, &count) == PROVEN_OK && count == 1,
                    "a literal resolves to itself");
    EXAMPLE_REQUIRE(proven_net_addr_eq(&found[0], &web), "the same address as the one built by hand");

    return EXAMPLE_OK();
}

3. Deadlines#

A deadline is a moment, not a duration: a reading of the monotonic clock (Chapter 5 section 4) at which a call must give up.

It is absolute so that one deadline can bound a whole exchange. "This request gets five seconds" is one value passed to connect, to each write and to each read; with a per-call timeout the same five seconds would have to be recomputed after every step, and usually is not.

It is on the monotonic clock so that setting the system time does not move it.

And a deadline that passes breaks nothing. PROVEN_ERR_TIMEOUT means "not yet"; the socket is exactly as usable as it was. Whether to try again, give up, or close is yours to decide.

Wrong - a timeout treated as a dead connection:

proven_result_size_t r = proven_net_read(&conn, buf, proven_net_deadline_in(100));
if (r.err != PROVEN_OK) proven_net_close(&conn);   /* wrong: TIMEOUT is not a failure of the connection */

Correct - decide what silence means for this protocol:

if (r.err == PROVEN_ERR_TIMEOUT) { send_keepalive(&conn); continue; }
if (r.err == PROVEN_ERR_EOF)     { finish(&conn); break; }
if (r.err != PROVEN_OK)          { proven_net_close(&conn); break; }

4. TCP#

A TCP connection is a stream of bytes in each direction. It has no message boundaries: what one side sends in a single write the other may receive in three reads, or two writes in one read.

Reference#

proven_net_listener_t and proven_net_conn_t are small values you own. Close each one you opened, and do not copy one that is open: two copies are two owners of one socket.

A connect in two halves. proven_net_connect waits. A program that is waiting for many sockets at once cannot afford to, so the same connect is offered as two calls:

proven_err_t proven_net_connect_start(proven_net_addr_t to, proven_net_conn_t *out);
proven_err_t proven_net_connect_finish(proven_net_conn_t *conn);

proven_net_connect_start returns at once: PROVEN_OK when the connection is already made (on the loopback interface it often is), or PROVEN_ERR_AGAIN when it is under way. In the second case the socket can be watched and closed and nothing else: wait until it is reported writable - with proven_net_poll, a selector, or the loop of Chapter 15 - and ask proven_net_connect_finish how it ended. A failure, from either call, leaves nothing open. The event-driven client of Chapter 15 is built on this pair.

The same calls serve Unix-domain stream sockets - pass an address made by proven_net_addr_unix. Listening creates the path; closing the listener does not remove it, so a server removes a leftover path before it listens.

Cautions, and what goes wrong#

One read is not one message. A stream delivers bytes, in order, in whatever pieces the network made of them.

Wrong - assuming the request arrives in one read:

proven_result_size_t r = proven_net_read(&conn, buf, until);
handle_request(buf, r.value);     /* wrong: this may be half a request, or one and a half */

Correct - read until the protocol says the message is complete: a length you were told, a delimiter you were promised, or PROVEN_ERR_EOF. The worked example below reads to the end.

One write may not send everything. proven_net_write reports how much went. Ignoring that count silently truncates the message whenever the buffer happens to be full - which is to say, under load, in production. proven_net_write_all is the call that keeps going; and when it fails, its count is what tells you whether resending would duplicate data.

A reset is not an end. PROVEN_ERR_EOF means the peer said it was finished and everything it sent has been read. PROVEN_ERR_RESET means it went away without saying so, and some of what was in flight - in either direction - may be gone. A file transfer that ends in EOF is complete; one that ends in RESET is not, however many bytes arrived.

Closing is not the same as finishing. To be sure the peer received everything, say you are done with proven_net_shutdown_write, then read until EOF, then close. Closing a connection that still has unread data in it is how the other side gets PROVEN_ERR_RESET.

Compiled and run by the test suite:

/*
 * A TCP exchange, start to finish, with a deadline on everything that can wait.
 *
 * Both ends live in this one program, which works because the kernel completes a connection as
 * soon as the listener's backlog takes it - before accept is called. A real client and server
 * are these same calls in two processes.
 */

int main(void) {
    /* Port 0: the OS picks a free one, and `at` says which. Loopback: this machine only. */
    proven_net_listener_t listener;
    proven_net_addr_t at;
    proven_err_t err = proven_net_listen(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), 16, &listener, &at);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK && proven_net_listener_is_open(&listener), "the listener opens");
    EXAMPLE_REQUIRE(at.port != 0, "and reports the port it was given");

    /* One deadline for the whole exchange: five seconds from now, however many calls it takes. */
    proven_net_deadline_t until = proven_net_deadline_in(5000);

    proven_net_conn_t client, server;
    proven_net_addr_t peer;
    EXAMPLE_REQUIRE(proven_net_connect(at, until, &client) == PROVEN_OK, "the client connects");
    EXAMPLE_REQUIRE(proven_net_accept(&listener, until, &server, &peer) == PROVEN_OK, "the server accepts");
    EXAMPLE_REQUIRE(proven_net_conn_is_open(&client) && proven_net_conn_is_open(&server), "two open ends");

    /* Each end knows both addresses. The accepted peer is the client's own local address. */
    proven_net_addr_t client_local, client_peer, listening;
    EXAMPLE_REQUIRE(proven_net_conn_local_addr(&client, &client_local) == PROVEN_OK &&
                    proven_net_conn_peer_addr(&client, &client_peer) == PROVEN_OK, "the client's two addresses");
    EXAMPLE_REQUIRE(proven_net_addr_eq(&client_local, &peer), "the server saw the client's address");
    EXAMPLE_REQUIRE(proven_net_listener_addr(&listener, &listening) == PROVEN_OK &&
                    proven_net_addr_eq(&client_peer, &listening), "the client's peer is where the server listens");

    /* A request written in two small pieces: without this they may wait to be sent together. */
    EXAMPLE_REQUIRE(proven_net_conn_set_nodelay(&client, true) == PROVEN_OK, "small writes go at once");

    /* write_all sends everything or says how far it got. */
    proven_mem_view_t request = proven_mem_view_from_u8(PROVEN_LIT("what time is it?"));
    proven_result_size_t sent = proven_net_write_all(&client, request, until);
    EXAMPLE_REQUIRE(sent.err == PROVEN_OK && sent.value == request.size, "the whole request went");

    /* "I have nothing more to send." The server will read the request, then end of input. */
    EXAMPLE_REQUIRE(proven_net_shutdown_write(&client) == PROVEN_OK, "the client half-closes");

    /* The server reads until the end. A read returns what has arrived - a stream has no message
     * boundaries - so a loop is the only correct way to read "all of it". */
    proven_byte_t buf[64];
    proven_size_t got = 0;
    for (;;) {
        proven_result_size_t r = proven_net_read(&server, (proven_mem_mut_t){ buf + got, sizeof buf - got }, until);
        if (r.err == PROVEN_ERR_EOF) break;                  /* the client finished */
        EXAMPLE_REQUIRE(r.err == PROVEN_OK, "a read that is not the end succeeds");
        if (r.err != PROVEN_OK) break;
        got += r.value;
    }
    EXAMPLE_REQUIRE(got == request.size, "the server received the whole request");

    /* A single write may send only part; the count says how much. Here it all fits. */
    proven_mem_view_t reply = proven_mem_view_from_u8(PROVEN_LIT("time to close"));
    proven_result_size_t w = proven_net_write(&server, reply, until);
    EXAMPLE_REQUIRE(w.err == PROVEN_OK && w.value == reply.size, "a short reply goes in one write");

    proven_result_size_t r = proven_net_read(&client, (proven_mem_mut_t){ buf, sizeof buf }, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_OK && r.value == reply.size, "the client reads the reply after its half-close");

    /* Nothing more is coming, and the client will not wait for ever to find that out. A timeout
     * is not damage: the connection is exactly as usable as before. */
    r = proven_net_read(&client, (proven_mem_mut_t){ buf, sizeof buf }, proven_net_deadline_in(50));
    EXAMPLE_REQUIRE(r.err == PROVEN_ERR_TIMEOUT, "silence is a timeout, not a hang");

    /* Every socket is closed by its owner. */
    EXAMPLE_REQUIRE(proven_net_close(&server) == PROVEN_OK, "the server closes its end");
    r = proven_net_read(&client, (proven_mem_mut_t){ buf, sizeof buf }, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_ERR_EOF, "which the client reads as the end");
    EXAMPLE_REQUIRE(proven_net_close(&client) == PROVEN_OK, "the client closes");
    EXAMPLE_REQUIRE(proven_net_listener_close(&listener) == PROVEN_OK, "the listener closes");

    /* With the listener gone, the same address refuses - a different answer from a timeout. */
    proven_net_conn_t nobody;
    EXAMPLE_REQUIRE(proven_net_connect(at, proven_net_deadline_in(15000), &nobody) == PROVEN_ERR_REFUSED,
                    "nothing listening is PROVEN_ERR_REFUSED");

    return EXAMPLE_OK();
}

5. UDP#

UDP sends messages - datagrams - one at a time, each on its own. A datagram arrives whole or not at all; datagrams may arrive out of order, twice, or never, and nothing tells the sender which.

Reference#

Cautions, and what goes wrong#

"Sent" means it left, not that it arrived. A successful proven_net_udp_send_to has handed the datagram to the network. A protocol that needs an answer waits for one with a deadline and sends again when it does not come.

A zero-length datagram is a message. On a stream, zero bytes would mean the end, which is why proven_net_read never returns it. A datagram can really be empty, so proven_net_udp_recv_from returning success with .value 0 is not an end and not an error.

A buffer that is too small loses the rest. The part of a datagram that did not fit is not kept for the next receive - it is gone.

Wrong - a buffer sized by guess, and the error ignored:

proven_byte_t buf[64];
proven_result_size_t r = proven_net_udp_recv_from(&udp, buf_mem, &from, until);
parse(buf, r.value);      /* wrong: on OUT_OF_BOUNDS this is the first 64 bytes of something longer */

Correct - size the buffer for the largest message the protocol allows, and treat PROVEN_ERR_OUT_OF_BOUNDS as a malformed packet.

Compiled and run by the test suite:

/*
 * UDP: messages, not a stream. Each send is one datagram and each receive is one datagram,
 * with the sender's address attached - and no promise that it arrives.
 */

int main(void) {
    proven_net_udp_t server, client;
    proven_net_addr_t server_at, client_at;
    proven_err_t err = proven_net_udp_open(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &server, &server_at);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK && proven_net_udp_is_open(&server), "the server's socket opens");
    EXAMPLE_REQUIRE(proven_net_udp_open(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &client, NULL) == PROVEN_OK,
                    "and the client's");
    EXAMPLE_REQUIRE(proven_net_udp_addr(&client, &client_at) == PROVEN_OK && client_at.port != 0,
                    "a socket can be asked where it is bound");

    proven_net_deadline_t until = proven_net_deadline_in(2000);

    /* Two sends are two datagrams. Success means each left this machine. */
    EXAMPLE_REQUIRE(proven_net_udp_send_to(&client, server_at, proven_mem_view_from_u8(PROVEN_LIT("first")), until) == PROVEN_OK &&
                    proven_net_udp_send_to(&client, server_at, proven_mem_view_from_u8(PROVEN_LIT("second message")), until) == PROVEN_OK,
                    "two datagrams leave");

    /* Each receive is one whole datagram and says who sent it. */
    proven_byte_t buf[32];
    proven_net_addr_t from;
    proven_result_size_t r = proven_net_udp_recv_from(&server, (proven_mem_mut_t){ buf, sizeof buf }, &from, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_OK && r.value == 5, "the first datagram, alone: five bytes");
    EXAMPLE_REQUIRE(proven_net_addr_eq(&from, &client_at), "from the client's address");

    /* A buffer that is too small cuts the datagram and says so. The rest of it is gone - the
     * next receive is the next datagram - so size the buffer for the largest message. */
    proven_byte_t small[6];
    r = proven_net_udp_recv_from(&server, (proven_mem_mut_t){ small, sizeof small }, &from, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_ERR_OUT_OF_BOUNDS && r.value == sizeof small,
                    "fourteen bytes into six: PROVEN_ERR_OUT_OF_BOUNDS, six kept");

    /* The reply goes to the address the datagram came from. */
    EXAMPLE_REQUIRE(proven_net_udp_send_to(&server, from, proven_mem_view_from_u8(PROVEN_LIT("ack")), until) == PROVEN_OK,
                    "the server answers the sender");
    r = proven_net_udp_recv_from(&client, (proven_mem_mut_t){ buf, sizeof buf }, NULL, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_OK && r.value == 3, "and the client receives it");

    /* Nothing else is coming. A lost datagram looks exactly like this, which is why every
     * receive has a deadline and every UDP protocol has a retry. */
    r = proven_net_udp_recv_from(&client, (proven_mem_mut_t){ buf, sizeof buf }, NULL, proven_net_deadline_in(50));
    EXAMPLE_REQUIRE(r.err == PROVEN_ERR_TIMEOUT, "silence is a timeout");

    EXAMPLE_REQUIRE(proven_net_udp_close(&client) == PROVEN_OK && proven_net_udp_close(&server) == PROVEN_OK,
                    "both sockets are closed by their owner");
    return EXAMPLE_OK();
}

6. Many sockets, one thread#

A deadline bounds a wait on one socket. A server waits on all of its sockets at once and serves whichever needs it: that is proven_net_poll.

Reference#

typedef struct {
    proven_net_handle_t handle;
    proven_u8 want;   /* PROVEN_NET_READABLE and/or PROVEN_NET_WRITABLE */
    proven_u8 got;    /* filled in: zero, or what is ready - which may include PROVEN_NET_FAILED */
} proven_net_poll_item_t;

Ready means "will not wait", not "will succeed". A readable connection may hold data, or the end, or an error; a readable listener has a connection to accept. So the call that follows a ready report takes PROVEN_NET_DONT_WAIT, and its result is what you act on. An item reported PROVEN_NET_FAILED is in error or hung up; the next read or write on it says which.

Ask for writability only when you have something to write. An idle connection is almost always writable, so a loop that always asks is woken at once, every time, and spins a processor doing nothing.

Wrong:

items[i].want = PROVEN_NET_READABLE | PROVEN_NET_WRITABLE;   /* wrong when nothing is queued: never sleeps */

Correct - PROVEN_NET_WRITABLE only for a connection whose last write returned less than you asked or timed out, and only until that data has gone.

proven_net_poll_with exists because the operating system wants an array of its own and this library does not allocate one behind your back: you give it working memory, sized by proven_net_poll_scratch_size, and reuse it for the next call. Readiness here is poll on POSIX and WSAPoll on Windows - the right tool for dozens or hundreds of sockets. For thousands, of which few are busy at a time, use the selector of section 9.

Compiled and run by the test suite:

/*
 * One thread, many sockets: wait until any of them needs attention, then serve the ones
 * that do. This is the loop inside every single-threaded server.
 *
 * "Ready" means the next call of that kind will not wait - so each socket reported ready is
 * handled with PROVEN_NET_DONT_WAIT, and the loop never blocks anywhere but in the poll.
 */

int main(void) {
    proven_net_listener_t listener;
    proven_net_addr_t at;
    proven_err_t err = proven_net_listen(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), 16, &listener, &at);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK, "the listener opens");

    /* Two clients connect and each sends one message. */
    proven_net_conn_t clients[2];
    for (int i = 0; i < 2; ++i) {
        EXAMPLE_REQUIRE(proven_net_connect(at, proven_net_deadline_in(5000), &clients[i]) == PROVEN_OK, "a client connects");
    }
    EXAMPLE_REQUIRE(proven_net_write_all(&clients[0], proven_mem_view_from_u8(PROVEN_LIT("from the first")), proven_net_deadline_in(5000)).err == PROVEN_OK &&
                    proven_net_write_all(&clients[1], proven_mem_view_from_u8(PROVEN_LIT("from the second")), proven_net_deadline_in(5000)).err == PROVEN_OK,
                    "each sends a message");

    /* The server: a listener and up to two connections, in one list. */
    proven_net_conn_t served[2] = {0};
    proven_size_t served_count = 0;
    proven_size_t bytes_seen = 0;

    proven_net_deadline_t until = proven_net_deadline_in(5000);
    while (bytes_seen < 14 + 15) {
        proven_net_poll_item_t items[3];
        proven_size_t count = 0;
        items[count++] = (proven_net_poll_item_t){ .handle = proven_net_listener_handle(&listener), .want = PROVEN_NET_READABLE };
        for (proven_size_t i = 0; i < served_count; ++i) {
            items[count++] = (proven_net_poll_item_t){ .handle = proven_net_conn_handle(&served[i]), .want = PROVEN_NET_READABLE };
        }

        proven_size_t ready = 0;
        err = proven_net_poll(items, count, until, &ready);
        EXAMPLE_REQUIRE(err == PROVEN_OK && ready > 0, "something becomes ready before the deadline");
        if (err != PROVEN_OK) break;

        /* A readable listener has a connection waiting. */
        if (items[0].got & PROVEN_NET_READABLE) {
            EXAMPLE_REQUIRE(served_count < 2 &&
                            proven_net_accept(&listener, PROVEN_NET_DONT_WAIT, &served[served_count], NULL) == PROVEN_OK,
                            "the accept that poll predicted does not wait");
            served_count++;
        }
        /* A readable connection has data, or its end. `items[1 + i]` was built from `served[i]`. */
        for (proven_size_t i = 1; i < count; ++i) {
            if (!items[i].got) continue;
            proven_byte_t buf[64];
            proven_result_size_t r = proven_net_read(&served[i - 1], (proven_mem_mut_t){ buf, sizeof buf }, PROVEN_NET_DONT_WAIT);
            EXAMPLE_REQUIRE(r.err == PROVEN_OK, "the read that poll predicted does not wait");
            bytes_seen += r.value;
        }
    }
    EXAMPLE_REQUIRE(served_count == 2 && bytes_seen == 29, "both clients were accepted and both messages read, in one thread");

    /* proven_net_poll takes up to PROVEN_NET_POLL_INLINE_MAX sockets. A server with more passes
     * its own working memory: the OS wants an array, and this library does not allocate one. */
    proven_net_poll_item_t one = { .handle = proven_net_conn_handle(&served[0]), .want = PROVEN_NET_WRITABLE };
    proven_byte_t scratch[256];
    proven_size_t ready = 0;
    EXAMPLE_REQUIRE(proven_net_poll_scratch_size(1) <= sizeof scratch, "the scratch for one item is small");
    EXAMPLE_REQUIRE(proven_net_poll_with((proven_mem_mut_t){ scratch, sizeof scratch }, &one, 1, PROVEN_NET_DONT_WAIT, &ready) == PROVEN_OK &&
                    (one.got & PROVEN_NET_WRITABLE), "an idle connection is writable at once");

    /* A UDP socket waits in the same list as everything else. */
    proven_net_udp_t udp;
    EXAMPLE_REQUIRE(proven_net_udp_open(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &udp, NULL) == PROVEN_OK, "a UDP socket");
    proven_net_poll_item_t quiet = { .handle = proven_net_udp_handle(&udp), .want = PROVEN_NET_READABLE };
    EXAMPLE_REQUIRE(proven_net_poll(&quiet, 1, proven_net_deadline_in(30), &ready) == PROVEN_ERR_TIMEOUT && ready == 0,
                    "nothing arrives on it: PROVEN_ERR_TIMEOUT, the idle tick of an event loop");
    (void)proven_net_udp_close(&udp);

    for (int i = 0; i < 2; ++i) {
        (void)proven_net_close(&clients[i]);
        (void)proven_net_close(&served[i]);
    }
    (void)proven_net_listener_close(&listener);
    return EXAMPLE_OK();
}

7. Transports#

A protocol - a line-based command set, an HTTP exchange - should be written once and not know what carries its bytes. Today that is a TCP connection. Under TLS it is an encrypted session over one. In a test it is two memory buffers and no network at all. proven_transport_t is that seam.

typedef struct {
    void *ctx;
    proven_result_size_t (*read_fn)(void *ctx, proven_mem_mut_t dest, proven_net_deadline_t until);
    proven_result_size_t (*write_fn)(void *ctx, proven_mem_view_t src, proven_net_deadline_t until);
    proven_err_t (*shutdown_fn)(void *ctx);   /* may be NULL */
    proven_err_t (*close_fn)(void *ctx);      /* may be NULL */
} proven_transport_t;

It is a small vtable passed by value, the same shape as proven_allocator_t (Chapter 2) and proven_writer_t, and for the same reason: the caller decides, and nothing is hidden. To supply your own, fill in the struct. The contract is that of the TCP calls: a read never succeeds with zero bytes, and a write reports the bytes that went even when it then fails.

Reference#

The reader and writer adapters connect a socket to everything in Chapter 5's stream section. A reader has nowhere to pass a deadline, so each read or write through an adapter gets its own: timeout_ms from the moment it starts, or no deadline when that is 0. The proven_transport_stream_t state is caller-owned: it must outlive the reader or writer and must not move.

The writer sends each piece as it is given. Wrap it in proven_writer_buffered when a message is formatted in many small pieces, so that it leaves in few packets - and flush before you wait for the reply.

Compiled and run by the test suite:

/*
 * A protocol written once, against proven_transport_t, and run over a TCP connection.
 *
 * The function below does not know what carries its bytes. Today it is a socket; under TLS it
 * will be an encrypted session over a socket; in a test it can be two memory buffers. That is
 * what the interface is for.
 */

/* Send one request line and read one reply line, each with a five-second limit per step. */
static proven_err_t ask(proven_transport_t t, const char *question, proven_u8str_view_t expected) {
    if (!proven_transport_is_valid(t)) return PROVEN_ERR_INVALID_ARG;

    /* The formatter writes straight to the transport. */
    proven_transport_stream_t out_state;
    proven_writer_t out = proven_transport_writer(&out_state, t, 5000);
    proven_err_t err = proven_fprintln(out, "ASK {}", PROVEN_ARG(question)).err;
    if (err != PROVEN_OK) return err;

    /* The line reader reads from it, into a buffer this function owns. */
    proven_transport_stream_t in_state;
    proven_byte_t line_buf[128];
    proven_reader_buffered_t lines;
    (void)proven_reader_buffered(&lines, proven_transport_reader(&in_state, t, 5000),
                                 (proven_mem_mut_t){ line_buf, sizeof line_buf });
    proven_result_u8str_view_t reply = proven_reader_read_line(&lines);
    if (reply.err != PROVEN_OK) return reply.err;
    return proven_u8str_view_eq(reply.val, expected) ? PROVEN_OK : PROVEN_ERR_INVALID_FORMAT;
}

int main(void) {
    proven_net_listener_t listener;
    proven_net_addr_t at;
    proven_err_t err = proven_net_listen(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), 4, &listener, &at);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK, "the listener opens");

    proven_net_conn_t client_conn, server_conn;
    EXAMPLE_REQUIRE(proven_net_connect(at, proven_net_deadline_in(5000), &client_conn) == PROVEN_OK &&
                    proven_net_accept(&listener, proven_net_deadline_in(5000), &server_conn, NULL) == PROVEN_OK,
                    "a connected pair");

    /* A connection as a transport. From here on, neither side names a socket. */
    proven_transport_t client = proven_net_conn_transport(&client_conn);
    proven_transport_t server = proven_net_conn_transport(&server_conn);
    proven_net_deadline_t until = proven_net_deadline_in(5000);

    /* The server's half, done by hand: the reply is written before the client asks, which is
     * fine in one thread because the bytes simply wait in the connection. */
    proven_result_size_t w = proven_transport_write_all(server, proven_mem_view_from_u8(PROVEN_LIT("ANSWER 42\n")), until);
    EXAMPLE_REQUIRE(w.err == PROVEN_OK && w.value == 10, "the server's reply is on its way");

    EXAMPLE_REQUIRE(ask(client, "the question", PROVEN_LIT("ANSWER 42")) == PROVEN_OK,
                    "the protocol function sends its line and reads the reply");

    /* And the server reads what the client wrote, with the raw calls. */
    proven_byte_t buf[64];
    proven_result_size_t r = proven_transport_read(server, (proven_mem_mut_t){ buf, sizeof buf }, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_OK &&
                    proven_u8str_view_eq((proven_u8str_view_t){ buf, r.value }, PROVEN_LIT("ASK the question\n")),
                    "the request line arrived as written");

    /* A single write reports how much went; here, all of it. */
    w = proven_transport_write(server, proven_mem_view_from_u8(PROVEN_LIT("BYE\n")), until);
    EXAMPLE_REQUIRE(w.err == PROVEN_OK && w.value == 4, "one more line");

    /* Ending and closing go through the same interface. */
    EXAMPLE_REQUIRE(proven_transport_shutdown(server) == PROVEN_OK, "the server ends its sending side");
    r = proven_transport_read(client, (proven_mem_mut_t){ buf, sizeof buf }, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_OK && r.value == 4, "the client reads the last line");
    r = proven_transport_read(client, (proven_mem_mut_t){ buf, sizeof buf }, until);
    EXAMPLE_REQUIRE(r.err == PROVEN_ERR_EOF, "and then the end");

    EXAMPLE_REQUIRE(proven_transport_close(client) == PROVEN_OK && proven_transport_close(server) == PROVEN_OK,
                    "closing a transport closes what carries it");
    EXAMPLE_REQUIRE(!proven_net_conn_is_open(&client_conn), "the connection underneath is closed");
    (void)proven_net_listener_close(&listener);
    return EXAMPLE_OK();
}

8. A pair, and waking a loop#

A loop blocked in proven_net_poll sees sockets and nothing else. A thread that has finished some work for it, or wants it to stop, cannot reach it through a variable: the loop is asleep in the operating system. What it needs is a socket that becomes readable when another thread says so. That is a waker.

Reference#

Drain after you wake. A waker stays readable until it is drained. A loop that forgets polls, returns at once, polls, returns at once - and spins a processor.

Wakes merge. Ten calls to proven_net_waker_wake before one drain are one wake. A waker says "look", not how many times, so what the loop should look at lives somewhere else - a queue, a flag - and the loop checks it every time it wakes.

Set the flag, then wake - and look after draining, not before. In the other order a wake can land between the look and the drain and be drained away unseen.

Wrong:

if (work_pending()) do_work();        /* wrong: work queued after this line... */
proven_net_waker_drain(&waker);       /* ...has its wake drained here, and waits for the next poll to time out */

Correct:

proven_net_waker_drain(&waker);
if (work_pending()) do_work();

Close it last. proven_net_waker_close is not safe while another thread can still call proven_net_waker_wake: stop or join those threads first.

proven_net_pair is what a waker is built from, and is useful alone wherever a test or a program wants a connection without a network. It is a Unix-domain pair on POSIX and a loopback TCP connection on Windows, where nothing else is available on every supported version.

Compiled and run by the test suite:

/*
 * A loop blocked in proven_net_poll sees only sockets. To tell it something from another
 * thread - "there is work for you", "stop" - you need a socket that becomes readable when
 * you say so. That is a waker.
 *
 * proven_net_pair is what a waker is made of, and is useful by itself: two connected ends
 * with no address, for handing bytes between two parts of a program.
 */

static void finish_work(void *arg) {
    proven_net_waker_t *waker = arg;
    proven_time_sleep(20);              /* stands for some work */
    proven_net_waker_wake(waker);       /* any thread may call this */
}

int main(void) {
    /* A pair: what goes into one end comes out of the other. */
    proven_net_conn_t a, b;
    proven_err_t err = proven_net_pair(&a, &b);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK, "a pair opens");
    EXAMPLE_REQUIRE(proven_net_write_all(&a, proven_mem_view_from_u8(PROVEN_LIT("across")), proven_net_deadline_in(1000)).err == PROVEN_OK, "write to one end");
    proven_byte_t buf[16];
    proven_result_size_t got = proven_net_read(&b, (proven_mem_mut_t){ buf, sizeof buf }, proven_net_deadline_in(1000));
    EXAMPLE_REQUIRE(got.err == PROVEN_OK && got.value == 6, "read from the other");
    (void)proven_net_close(&a);
    (void)proven_net_close(&b);

    /* A waker in a poll list. Nothing is readable, so the poll would wait its full second... */
    proven_net_waker_t waker;
    EXAMPLE_REQUIRE(proven_net_waker_open(&waker) == PROVEN_OK, "a waker opens");
    proven_job_sys_t *jobs = NULL;
    EXAMPLE_REQUIRE(proven_job_system_init(proven_heap_allocator(), 1, 4, &jobs) == PROVEN_OK, "a worker thread");
    EXAMPLE_REQUIRE(proven_job_submit_ex(jobs, finish_work, &waker) == PROVEN_OK, "work is handed to it");

    proven_net_poll_item_t item = { .handle = proven_net_waker_handle(&waker), .want = PROVEN_NET_READABLE };
    proven_size_t ready = 0;
    proven_time_t start = proven_time_monotonic_now();
    err = proven_net_poll(&item, 1, proven_net_deadline_in(1000), &ready);
    proven_i64 waited_ms = (proven_time_monotonic_now() - start) / 1000000;
    /* ...but the worker woke it. */
    EXAMPLE_REQUIRE(err == PROVEN_OK && ready == 1 && waited_ms < 900, "the poll returns when the worker says so, not at its deadline");

    /* Drain it, or it stays readable and the next poll returns at once for nothing. */
    proven_net_waker_drain(&waker);
    EXAMPLE_REQUIRE(proven_net_poll(&item, 1, proven_net_deadline_in(20), &ready) == PROVEN_ERR_TIMEOUT, "drained: quiet again");

    /* Several wakes before a drain are one wake. */
    proven_net_waker_wake(&waker);
    proven_net_waker_wake(&waker);
    proven_net_waker_drain(&waker);
    EXAMPLE_REQUIRE(proven_net_poll(&item, 1, proven_net_deadline_in(20), &ready) == PROVEN_ERR_TIMEOUT, "two wakes, one drain: quiet");

    /* Close the waker only when no thread can still wake it: stop the workers first. */
    proven_job_system_close(jobs);
    proven_job_system_destroy(jobs);
    proven_net_waker_close(&waker);
    return EXAMPLE_OK();
}

9. A selector: thousands of sockets#

proven_net_poll is handed its whole list on every call, and the system examines every entry on every call. With a few dozen sockets that is nothing. With ten thousand connections of which five have something to say, it is ten thousand entries copied and examined to find five - on every turn of the loop.

A selector asks the same question the other way round. You register a socket once; the system remembers it; a wait returns the sockets that are ready and does not look at the others.

Reference#

typedef struct {
    void *tag;        /* what you registered with the socket */
    proven_u8 got;    /* PROVEN_NET_READABLE, PROVEN_NET_WRITABLE and/or PROVEN_NET_FAILED */
} proven_net_ready_t;

A socket of which nothing is asked is still told when its connection is lost - once. The system reports an error or a hang-up whether you asked about the socket or not, and goes on reporting it. Registered with want of 0 - a connection you have paused - the socket is reported with PROVEN_NET_FAILED (or as readable and writable, for a hang-up) one time, and then left alone until you change what is asked of it with proven_net_selector_modify; then it is reported again. Before v0.38.0 it was reported on every wait, so a loop with one such socket never slept. (So on Linux and with the portable kind. The kqueue kind registers nothing for a socket of which nothing is asked, and reports nothing for it until something is; that kind has not been run.)

What you get depends on the system, and the selector says which. On Linux it is epoll; on the BSDs and macOS it is kqueue. There a wait costs in proportion to the sockets that are ready. On Windows it is an I/O completion port (PROVEN_NET_SELECTOR_IOCP): a connection is watched for reading by a receive of zero bytes, and room to write, a connection waiting to be accepted, a datagram, or a connection lost while nothing is asked of it, by a notice of the system's thread pool - about 500 bytes and two handles per socket while it is asked, so that cost falls on listeners, datagram sockets, connections waiting to write and those of which nothing is asked, not on connections waiting to read. Each wait checks what was so reported with the call proven_net_poll uses, so it reports what the portable kind would. A connection it has watched for reading stays bound to its port while it is open. Anywhere else it is the portable kind: the same interface kept over the call proven_net_poll uses, which still examines every socket on every wait. It behaves the same and it does not scale the same. proven_net_selector_kind tells you; a server that means to hold tens of thousands of connections should look.

Remove a socket before you close it. A closed socket's number is given to the next socket opened. A selector that still holds the old registration then reports events for a socket that is not the one you registered - with the old tag - or, in the portable kind, reports a failure on every wait.

Wrong:

proven_net_close(&conn);                                  /* wrong: the selector still holds its number */
proven_net_selector_remove(selector, proven_net_conn_handle(&conn));   /* too late: the handle is no longer valid */

Correct - proven_net_selector_remove first, then proven_net_close.

Readiness is level-triggered. A socket is reported for as long as it is ready, exactly as with proven_net_poll: an event you do not act on comes back on the next wait. So the rules of section 6 hold unchanged - follow an event with a call that takes PROVEN_NET_DONT_WAIT, and ask for writability only while you have something to write.

The tag is yours, and so is its lifetime. The selector returns the pointer you gave it and never looks through it. If the record it points at is freed while events for it may already have been taken from the selector, the next loop iteration reads freed memory. The pattern that avoids this: when you close a connection in the middle of handling a batch of events, mark its record and free it after the batch.

One thread. A selector is not safe to use from several threads. To make a waiting loop act on something another thread did, register a waker (section 8) and wake it.

A socket ready both ways may be one event or two. kqueue reports readability and writability separately; handle each event for what it says rather than expecting one per socket.

Compiled and run by the test suite:

/*
 * A selector: register each socket once, and be told only about the ones that are ready.
 *
 * The loop is the one from the poll example - wait, then serve what is ready without waiting -
 * but nothing is rebuilt between waits, and a thousand idle connections cost a wait nothing.
 */

typedef struct {
    proven_net_conn_t conn;
    int number;
    bool open;
} ex_client_t;

int main(void) {
    proven_net_listener_t listener;
    proven_net_addr_t at;
    proven_err_t err = proven_net_listen(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), 16, &listener, &at);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK, "the listener opens");

    proven_net_selector_t *selector = NULL;
    EXAMPLE_REQUIRE(proven_net_selector_create(proven_heap_allocator(), &selector) == PROVEN_OK, "a selector");
    /* epoll on Linux, kqueue on the BSDs and macOS; elsewhere the same interface over poll. */
    proven_net_selector_kind_t kind = proven_net_selector_kind(selector);
    printf("selector kind: %s\n", kind == PROVEN_NET_SELECTOR_EPOLL ? "epoll" : kind == PROVEN_NET_SELECTOR_KQUEUE ? "kqueue" : kind == PROVEN_NET_SELECTOR_IOCP ? "iocp" : "poll");

    /* The tag is what comes back with an event: a pointer to your own record of the socket.
     * The listener's tag here is the listener itself. */
    EXAMPLE_REQUIRE(proven_net_selector_add(selector, proven_net_listener_handle(&listener), PROVEN_NET_READABLE, &listener) == PROVEN_OK,
                    "the listener is registered, once");

    /* Three clients connect; the second and third say something. */
    proven_net_conn_t far[3];
    for (int i = 0; i < 3; ++i) EXAMPLE_REQUIRE(proven_net_connect(at, proven_net_deadline_in(5000), &far[i]) == PROVEN_OK, "a client connects");
    EXAMPLE_REQUIRE(proven_net_write_all(&far[1], proven_mem_view_from_u8(PROVEN_LIT("from one")), proven_net_deadline_in(5000)).err == PROVEN_OK &&
                    proven_net_write_all(&far[2], proven_mem_view_from_u8(PROVEN_LIT("from two")), proven_net_deadline_in(5000)).err == PROVEN_OK, "two of them send");

    ex_client_t clients[3] = {0};
    int accepted = 0, messages = 0;
    for (int round = 0; round < 20 && messages < 2; ++round) {
        proven_net_ready_t ready[8];
        proven_size_t count = 0;
        err = proven_net_selector_wait(selector, ready, 8, proven_net_deadline_in(1000), &count);
        if (err == PROVEN_ERR_TIMEOUT) continue;        /* the idle tick of a loop */
        EXAMPLE_REQUIRE(err == PROVEN_OK, "something is ready");
        for (proven_size_t i = 0; i < count; ++i) {
            if (ready[i].tag == &listener) {
                /* Ready means "will not wait": take every pending connection, without waiting. */
                while (accepted < 3 && proven_net_accept(&listener, PROVEN_NET_DONT_WAIT, &clients[accepted].conn, NULL) == PROVEN_OK) {
                    ex_client_t *c = &clients[accepted];
                    c->number = accepted++;
                    c->open = true;
                    EXAMPLE_REQUIRE(proven_net_selector_add(selector, proven_net_conn_handle(&c->conn), PROVEN_NET_READABLE, c) == PROVEN_OK,
                                    "each accepted connection is registered, with its record as the tag");
                }
            } else {
                ex_client_t *c = ready[i].tag;
                proven_byte_t buf[32];
                proven_result_size_t got = proven_net_read(&c->conn, (proven_mem_mut_t){ buf, sizeof buf }, PROVEN_NET_DONT_WAIT);
                if (got.err == PROVEN_OK) messages++;
            }
        }
    }
    EXAMPLE_REQUIRE(accepted == 3 && messages == 2, "three connections accepted, two messages read");
    EXAMPLE_REQUIRE(proven_net_selector_count(selector) == 4, "four sockets registered: the listener and three connections");

    /* The first connection never spoke, so it was never reported: nothing to do for it. Now ask
     * a different question of it - may I write? - and it is reported at once. */
    proven_net_ready_t one[4];
    proven_size_t count = 0;
    EXAMPLE_REQUIRE(proven_net_selector_wait(selector, one, 4, proven_net_deadline_in(30), &count) == PROVEN_ERR_TIMEOUT, "all quiet: nothing is reported");
    EXAMPLE_REQUIRE(proven_net_selector_modify(selector, proven_net_conn_handle(&clients[0].conn), PROVEN_NET_WRITABLE, &clients[0]) == PROVEN_OK, "ask about writing instead");
    EXAMPLE_REQUIRE(proven_net_selector_wait(selector, one, 4, proven_net_deadline_in(1000), &count) == PROVEN_OK && count == 1 &&
                    one[0].tag == &clients[0] && (one[0].got & PROVEN_NET_WRITABLE), "an idle connection is writable");
    /* Ask for writability only while you have something to write, or every wait returns at once. */
    EXAMPLE_REQUIRE(proven_net_selector_modify(selector, proven_net_conn_handle(&clients[0].conn), PROVEN_NET_READABLE, &clients[0]) == PROVEN_OK, "back to reading");

    /* Remove a socket BEFORE closing it. */
    for (int i = 0; i < 3; ++i) {
        EXAMPLE_REQUIRE(proven_net_selector_remove(selector, proven_net_conn_handle(&clients[i].conn)) == PROVEN_OK, "removed");
        (void)proven_net_close(&clients[i].conn);
        (void)proven_net_close(&far[i]);
    }
    EXAMPLE_REQUIRE(proven_net_selector_remove(selector, proven_net_listener_handle(&listener)) == PROVEN_OK &&
                    proven_net_selector_count(selector) == 0, "and the listener: the selector is empty");
    (void)proven_net_listener_close(&listener);

    /* The portable kind can be asked for by name - to compare the two, or to run the code path
     * of a system that offers nothing better. */
    proven_net_selector_t *portable = NULL;
    EXAMPLE_REQUIRE(proven_net_selector_create_poll(proven_heap_allocator(), &portable) == PROVEN_OK &&
                    proven_net_selector_kind(portable) == PROVEN_NET_SELECTOR_POLL, "a selector of the poll kind");
    proven_net_selector_destroy(portable);
    proven_net_selector_destroy(selector);
    return EXAMPLE_OK();
}

10. What the platforms do differently#

The calls behave the same on Linux, the BSDs, macOS and Windows. These are the differences the library absorbs, written down because they are the ones that bite when sockets are used directly:

Two things differ and stay different. Windows waits about two seconds before it reports a refused loopback connection, where POSIX answers at once - allow for it in the deadline. And Unix-domain sockets need Windows 10 version 1803 or later; where the family is missing, proven_net_listen and proven_net_connect return PROVEN_ERR_UNSUPPORTED.

11. What is not here#

• Encryption. A connection made here is not encrypted. TLS is a layer over the transport interface: Chapter 14.
• A deadline on name resolution. See section 2.
• Windows at scale, beyond one machine. The selector and the event loop of Chapter 15 are built on I/O completion ports there. Measured once on a Windows 11 virtual machine, over loopback: with 10,000 connections held and ten making requests, the blocking server of Chapter 11 (on this selector) answered about 52,000 requests a second with a median round trip of 0.15 ms - as with 1,000 held; the event loop's figures are in Chapter 15, section 14. Not measured there: a client on another machine, datagram sockets on the selector, and more than about 16,000 connections (the machine's ports ran out with both ends on it).
• Unix-domain datagrams, raw sockets, multicast, socket options beyond TCP_NODELAY.
• HTTP. The message codec is Chapter 10 the client and the server that drive it over these sockets are Chapter 11.