Chapter 15: The event loop and the event-driven server#

Part V - Talking to the operating system. Prerequisites: Chapter 9 (sockets, deadlines, the selector); Chapter 11 (the HTTP server this one is the other form of); Chapter 14 if it is to speak TLS. After this chapter you can run a loop with timers and work that comes back from other threads, serve HTTP from it without a call that waits, answer a request later, stream a response to a client slower than your program, keep WebSocket connections open without a thread each, make many HTTP requests at once from one thread, spread a server over several processor cores, and say what a connection costs while it does nothing - and what was measured.

This chapter covers loop.h, http_event.h, ws_event.h and http_event_client.h. All are hosted-only, and PROVEN_NO_NET leaves them out.

Table of contents#

1. Two ways to serve
2. A loop
3. Timers
4. Work that is done elsewhere
5. A socket of your own
6. The event-driven server
7. Answering
8. Request bodies
9. Limits, time and refusals
10. Threads, TLS and shutting down
11. WebSocket on the loop
12. The event-driven client
13. Several loops
14. What a connection holds, what was measured, and what is not here

1. Two ways to serve#

The server of Chapter 11 hands your handler a request and lets it read and write with calls that wait. Nothing is simpler to write: the handler is a straight line from the request to the response. The price is that a request being handled occupies a thread for as long as it takes - including all the time its client is slow.

The server of this chapter turns that around. Nothing in it waits. It sits on an event loop, one thread that asks the system which sockets have something to say, deals with those, and asks again. Your functions are told what happened - a request arrived, a piece of its body arrived, there is room to write more - and return at once. A connection that is doing nothing is then a small struct and nothing else: no thread, no stack, no buffer.

What you take on in exchange is real, and it is the whole of this chapter:

• Your functions must not wait - not on a socket, a file, or a lock someone may hold for long. Work that takes time goes to another thread and its result is posted back (section 4).
• Writing can be refused. When the client is not taking what was sent, the server stops accepting more from you and says when to continue (section 7).
• What you are shown is short-lived. The request head and the pieces of a body are views good only until your function returns. Copy what you keep.

Which to choose: the blocking server while the number of connections open at once is something a pool of threads carries comfortably and the handlers are easier to write as straight lines; this one when most connections are idle most of the time and there are many of them - long polling, event streams, slow clients, a great many kept connections. Both parse HTTP with the same code of Chapter 10 and refuse the same malformed requests.

2. A loop#

proven_err_t proven_loop_create(proven_allocator_t alloc, proven_loop_t **out);
proven_err_t proven_loop_create_ex(proven_allocator_t alloc, const proven_loop_options_t *options, proven_loop_t **out);
proven_loop_backend_t proven_loop_backend(const proven_loop_t *loop);
void proven_loop_destroy(proven_loop_t *loop);
proven_err_t proven_loop_run(proven_loop_t *loop);
proven_err_t proven_loop_poll(proven_loop_t *loop, proven_net_deadline_t until);
void proven_loop_stop(proven_loop_t *loop);

A proven_loop_t is a set of operations on sockets that complete (section 5), a set of timers, a queue that other threads can post to, and a scratch buffer.

proven_loop_run is for a thread that does nothing else. proven_loop_poll is for a program that already has a main loop of its own and gives this one a turn in it.

Backends. How a loop does its work is chosen when it is made, and nothing else about it changes with the choice: the same calls, the same answers in the same order rules.

io_uring is the default on Linux 6.0 or later since v0.94.0, where the system lets the process use it; elsewhere on Linux, and where it is refused, AUTO is epoll, as it was before. In an emulated machine an HTTP/1.1 server on it made 0.071 system calls a request against epoll's 2.07, and answered 5 to 7 % more requests a second; no real machine has measured it here yet. PROVEN_LOOP_BACKEND_SELECTOR asks for epoll whatever the kernel. A loop made with io_uring frees what the kernel still holds the way Windows does - cancelled and waited for, up to two seconds, when the loop is destroyed. On Linux 6.0 or later a connection read with proven_loop_read (section 5) is received for by the kernel into a pool of the loop's - by default 64 buffers of 16 KiB, made at the first such read; ring_buffers and ring_buffer_bytes in proven_loop_options_t change it - so that the read after a receive of zero bytes is a copy, not a system call. A buffer is taken only when data arrives: an idle connection still holds none. A connection whose data nobody reads holds four at most; then the kernel's own window holds the peer, as on the other backends. Sends too go through the ring: a send is copied and handed to the kernel in the loop's next round, together with the receives, so that the reads and sends of every connection share the loop's one entry into the kernel; proven_loop_send_queued counts the bytes until the kernel has taken them, and a close writes to the socket what had not been handed over yet.

The thread rule. Everything in this chapter except proven_loop_stop and proven_loop_post must be called on the loop's own thread - the one inside proven_loop_run or proven_loop_poll - or before the loop starts. That covers the server and every stream function of the later sections. Nothing checks it for you.

3. Timers#

void proven_loop_timer_set(proven_loop_t *loop, proven_loop_timer_t *timer, proven_u32 ms, proven_loop_fn fn, void *ctx);
void proven_loop_timer_cancel(proven_loop_t *loop, proven_loop_timer_t *timer);
bool proven_loop_timer_is_set(const proven_loop_timer_t *timer);

A proven_loop_timer_t is yours: put it inside the struct it belongs to, zero-initialised, and do not copy or move it while it is set. Setting one allocates nothing.

• A timer fires once, ms milliseconds from now. It does not fire early, and on POSIX it fires within about a millisecond of that time; on Windows the wait ends at the system timer's interval, about 16 ms (until v0.70.0 every timer was rounded up to the loop's 16 ms tick). One that should repeat sets itself again in its own function.
• Setting a timer that is already set moves it.
• proven_loop_timer_cancel is safe inside any callback, another timer's included, and harmless when the timer is not set.
• Setting, cancelling and firing are constant-time, and a timer that is waiting is not looked at until its time. That is what lets every connection of a server have a timeout of its own.

Timers measure the monotonic clock of Chapter 5 setting the wall clock does not move them.

4. Work that is done elsewhere#

proven_err_t proven_loop_post(proven_loop_t *loop, proven_loop_fn fn, void *ctx);

proven_loop_post asks the loop to call fn(ctx) on its own thread, soon. It is safe from any thread, and it is how a result comes back: the callback hands the slow part to a worker (the job system of Chapter 6), returns, and the worker posts a function that finishes the work where the loop's data may be touched.

• Functions are called in the order they were posted.
• The queue entry comes from the loop's allocator, so when other threads post, that allocator must be one that may be called from several threads. The heap allocator is.
• It returns PROVEN_ERR_NOMEM when that allocation fails, and PROVEN_ERR_INVALID_ARG.

The program below is a loop with nothing on it but timers and one job. The test suite compiles and runs it:

#include <stdatomic.h>

/*
 * An event loop by itself: timers, and work that is done on another thread and comes back.
 *
 * A loop's thread never waits for anything but "what is next". Whatever takes time - here a
 * worker that sleeps, standing in for a file or a database - happens elsewhere, and its result
 * is posted to the loop, where it runs between other events.
 */

typedef struct {
    proven_loop_t *loop;
    proven_loop_timer_t tick, late;
    int ticks;
    int order[8];
    int count;
    atomic_int from_worker;
} ex_app_t;

static void note(ex_app_t *app, int what) { if (app->count < 8) app->order[app->count++] = what; }

/* A timer fires once. One that should repeat sets itself again. */
static void on_tick(void *ctx) {
    ex_app_t *app = ctx;
    note(app, 1);
    if (++app->ticks < 3) proven_loop_timer_set(app->loop, &app->tick, 20, on_tick, app);
}

static void on_late(void *ctx) { note(ctx, 9); }

/* This runs on the loop's thread, although a worker thread asked for it. */
static void on_result(void *ctx) {
    ex_app_t *app = ctx;
    note(app, 5);
    proven_loop_timer_cancel(app->loop, &app->late);      /* it would have fired in five seconds; now it never will */
    proven_loop_stop(app->loop);
}

/* This runs on a worker thread. It may take as long as it likes: the loop is not waiting. */
static void slow_work(void *ctx) {
    ex_app_t *app = ctx;
    proven_time_sleep(400);
    atomic_store(&app->from_worker, 1);
    if (proven_loop_post(app->loop, on_result, app) != PROVEN_OK) proven_loop_stop(app->loop);   /* only a failed allocation refuses a post; then just end the run */
}

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    ex_app_t app = { 0 };

    // ---- a loop -------------------------------------------------------------------
    /* A loop is operations on sockets that complete, a set of timers, and a queue other threads can post to.
     * The options choose how it does its work; AUTO is what proven_loop_create gives. */
    EXAMPLE_REQUIRE(proven_loop_create_ex(heap, &(proven_loop_options_t){ .backend = PROVEN_LOOP_BACKEND_AUTO }, &app.loop) == PROVEN_OK, "a loop");
    EXAMPLE_REQUIRE(proven_loop_backend(app.loop) != PROVEN_LOOP_BACKEND_AUTO, "it says which backend does the work");

    // ---- timers --------------------------------------------------------------------
    /* The timer structs are yours: they live in your own state, and setting one allocates
     * nothing. A timer waiting to fire costs nothing either - it is not looked at until its time. */
    proven_loop_timer_set(app.loop, &app.tick, 20, on_tick, &app);
    proven_loop_timer_set(app.loop, &app.late, 5000, on_late, &app);
    EXAMPLE_REQUIRE(proven_loop_timer_is_set(&app.tick) && proven_loop_timer_is_set(&app.late), "two timers are set");

    // ---- work elsewhere ------------------------------------------------------------
    /* The job system runs slow_work on a worker thread. When it is done it posts on_result back. */
    proven_job_sys_t *workers = NULL;
    EXAMPLE_REQUIRE(proven_job_system_init(heap, 1, 4, &workers) == PROVEN_OK && proven_job_submit(workers, slow_work, &app), "a worker has the slow job");

    // ---- run -----------------------------------------------------------------------
    /* proven_loop_run returns when something calls proven_loop_stop - here on_result does. */
    proven_time_t began = proven_time_monotonic_now();
    EXAMPLE_REQUIRE(proven_loop_run(app.loop) == PROVEN_OK, "the loop ran and was stopped");
    proven_time_t took = proven_time_monotonic_now() - began;

    EXAMPLE_REQUIRE(app.ticks == 3 && app.count == 4 && app.order[0] == 1 && app.order[1] == 1 && app.order[2] == 1 && app.order[3] == 5,
                    "three ticks, then the worker's result, in that order");
    EXAMPLE_REQUIRE(atomic_load(&app.from_worker) == 1 && took < 2000000000, "the result came from the worker, and long before the five-second timer");
    EXAMPLE_REQUIRE(!proven_loop_timer_is_set(&app.late), "a cancelled timer is no longer set");

    proven_job_system_close(workers);
    proven_job_system_destroy(workers);
    proven_loop_destroy(app.loop);
    return EXAMPLE_OK();
}

5. A socket of your own#

The server of the next section does this for you. It is here for a protocol that is not HTTP.

The loop does not tell you that a socket is ready and leave the rest to you. You start an operation - a receive, an accept, a connect - and its function is called when it has finished; you give the loop bytes to send, and it sends them. On Windows these are the I/O completion port's operations; on Linux, the BSDs and macOS the loop does the same over the selector of Chapter 9 (epoll, kqueue). Your program sees the same calls, answered in the same order, everywhere.

proven_err_t proven_loop_attach(proven_loop_t *loop, proven_loop_sock_t *sock, const proven_net_conn_t *conn,
                                const proven_loop_sock_fns_t *fns, void *ctx);
proven_err_t proven_loop_connect(proven_loop_t *loop, proven_loop_sock_t *sock, proven_net_addr_t to, proven_net_conn_t *out,
                                 const proven_loop_sock_fns_t *fns, void *ctx);
proven_err_t proven_loop_recv(proven_loop_t *loop, proven_loop_sock_t *sock, proven_mem_mut_t buf);
void proven_loop_recv_cancel(proven_loop_t *loop, proven_loop_sock_t *sock);
proven_result_size_t proven_loop_read(proven_loop_t *loop, proven_loop_sock_t *sock, proven_mem_mut_t buf);
proven_result_size_t proven_loop_send(proven_loop_t *loop, proven_loop_sock_t *sock, proven_mem_view_t bytes);
proven_size_t proven_loop_send_queued(const proven_loop_sock_t *sock);
proven_err_t proven_loop_send_wait(proven_loop_t *loop, proven_loop_sock_t *sock, proven_size_t at_most);
void proven_loop_close(proven_loop_t *loop, proven_loop_sock_t *sock);
proven_err_t proven_loop_accept(proven_loop_t *loop, proven_loop_accept_t *acc, proven_net_listener_t *listener,
                                proven_loop_accept_fn fn, void *ctx);
void proven_loop_accept_cancel(proven_loop_t *loop, proven_loop_accept_t *acc);
proven_mem_mut_t proven_loop_scratch(proven_loop_t *loop);
proven_allocator_t proven_loop_allocator(const proven_loop_t *loop);

A connection given to the loop is a proven_loop_sock_t - yours, inside your connection struct, not moved from proven_loop_attach until its closed - and a table of four functions, proven_loop_sock_fns_t, usually one static const table for every connection of a kind:

The rules that hold it together:

• Every operation started is answered exactly once, by its function, on the loop's thread - and never from inside the call that started it, even when it could finish at once. Success, failure and cancellation all come this way, so a function of yours never runs inside your own call to the loop.
• One receive at a time per socket, one accept at a time per acceptor. Sends are not operations of yours: the loop takes the bytes.
• The end is proven_loop_close, then closed. Until closed the loop may still use the socket's memory; afterwards it never touches it. Close the connection itself in closed. For a graceful end - everything sent first - wait with proven_loop_send_wait(loop, &sock, 0) and close in writable.
• A receive of zero bytes is how ten thousand idle connections cost no read buffers. When it is answered, read with proven_loop_read into the scratch buffer, use what arrived, and keep only what cannot be used yet. Its contents mean nothing once your function returns, or calls anything that may use it too.
• Keep what the loop holds for a slow peer bounded. When proven_loop_send_queued says much is waiting, stop reading and ask proven_loop_send_wait to say when to go on - the example does.
• Once a connection is attached, read it with proven_loop_read and send through proven_loop_send, so that bytes keep their order: on io_uring the loop's pool may hold bytes the socket no longer has. A connection is given to one loop, once.

The program below is an echo server and its client on one loop. The test suite compiles and runs it:

#include <string.h>

/*
 * An echo server and its client on one loop, with nothing but the loop's operations.
 *
 * The server takes connections with an accept it arms again from inside its own function, and
 * waits for each connection's data with a receive of zero bytes: an idle connection holds no
 * read buffer. When data has arrived it reads it into the loop's scratch buffer and sends it
 * back; what the socket does not take at once, the loop keeps. The client connects through the
 * loop too, and reads with a receive into a buffer of its own, under a timer.
 */

#define MESSAGE "every operation is answered exactly once"
#define ROUNDS 3

typedef struct ex_app ex_app_t;

typedef struct {                /* one connection of the server */
    ex_app_t *app;
    proven_loop_sock_t sock;
    proven_net_conn_t conn;
    bool ending, used;
} ex_peer_t;

struct ex_app {
    proven_loop_t *loop;
    proven_net_listener_t listener;
    proven_loop_accept_t acc;
    ex_peer_t peers[4];
    int peers_closed;
    /* the client */
    proven_loop_sock_t sock;
    proven_net_conn_t conn;
    proven_loop_timer_t deadline;
    proven_byte_t buf[64];
    proven_size_t have;
    int rounds;
    bool echoed, client_closed, accepting_ended;
};

// ---- the server's connections ------------------------------------------------------

static void peer_end(ex_peer_t *p) {
    /* The client has said all it will: close once everything owed to it has left. */
    p->ending = true;
    if (proven_loop_send_wait(p->app->loop, &p->sock, 0) != PROVEN_OK) proven_loop_close(p->app->loop, &p->sock);
}

/* Read what has arrived and send it back. A client that sends faster than it reads would make
 * the loop hold more and more for it, so past 32 KiB waiting the server stops reading until it
 * has caught up: a send wait, and reading resumes in `writable`. */
static void peer_read(ex_peer_t *p) {
    proven_loop_t *loop = p->app->loop;
    proven_mem_mut_t scratch = proven_loop_scratch(loop);
    scratch.size = 16384;
    for (;;) {
        if (proven_loop_send_queued(&p->sock) > 32768) {
            if (proven_loop_send_wait(loop, &p->sock, 0) != PROVEN_OK) proven_loop_close(loop, &p->sock);
            return;
        }
        proven_result_size_t r = proven_loop_read(loop, &p->sock, scratch);
        if (r.err == PROVEN_ERR_TIMEOUT) break;                 /* all of it read: wait for more */
        if (r.err != PROVEN_OK) { peer_end(p); return; }        /* PROVEN_ERR_EOF: the client is done */
        proven_result_size_t s = proven_loop_send(loop, &p->sock, (proven_mem_view_t){ scratch.ptr, r.value });
        if (s.err != PROVEN_OK) { proven_loop_close(loop, &p->sock); return; }
    }
    /* Nothing is held while this connection waits: a receive of zero bytes takes no buffer. */
    if (proven_loop_recv(loop, &p->sock, (proven_mem_mut_t){ 0 }) != PROVEN_OK) proven_loop_close(loop, &p->sock);
}

static void peer_received(void *ctx, proven_err_t err, proven_size_t n) {
    ex_peer_t *p = ctx;
    (void)n;                                                    /* a receive of zero bytes takes nothing */
    if (err != PROVEN_OK) return;                               /* PROVEN_ERR_CANCELLED: it is being closed */
    peer_read(p);
}

static void peer_writable(void *ctx, proven_err_t err) {
    ex_peer_t *p = ctx;
    if (err == PROVEN_ERR_CANCELLED) return;
    if (err != PROVEN_OK || p->ending) proven_loop_close(p->app->loop, &p->sock);
    else peer_read(p);
}

static void peer_closed(void *ctx) {
    ex_peer_t *p = ctx;
    (void)proven_net_close(&p->conn);                           /* the connection itself is closed here, last */
    p->used = false;
    /* The one client of this program has gone: stop taking connections. */
    if (++p->app->peers_closed == 1) proven_loop_accept_cancel(p->app->loop, &p->app->acc);
}

static const proven_loop_sock_fns_t peer_fns = { .received = peer_received, .writable = peer_writable, .closed = peer_closed };

static void on_accept(void *ctx, proven_err_t err, proven_net_conn_t *conn) {
    ex_app_t *app = ctx;
    if (err != PROVEN_OK) {                                     /* PROVEN_ERR_CANCELLED: the listener may go now */
        (void)proven_net_listener_close(&app->listener);
        app->accepting_ended = true;
        return;
    }
    ex_peer_t *p = NULL;
    for (int i = 0; i < 4 && !p; ++i) if (!app->peers[i].used) p = &app->peers[i];
    if (!p) { (void)proven_net_close(conn); }
    else {
        *p = (ex_peer_t){ .app = app, .conn = *conn, .used = true };    /* the connection is ours now */
        if (proven_loop_attach(app->loop, &p->sock, &p->conn, &peer_fns, p) != PROVEN_OK ||
            proven_loop_recv(app->loop, &p->sock, (proven_mem_mut_t){ 0 }) != PROVEN_OK) { (void)proven_net_close(&p->conn); p->used = false; }
    }
    /* Armed again from inside: a connection already waiting is taken without another round. */
    if (proven_loop_accept(app->loop, &app->acc, &app->listener, on_accept, app) != PROVEN_OK) app->accepting_ended = true;
}

// ---- the client --------------------------------------------------------------------

static void client_deadline(void *ctx) {
    ex_app_t *app = ctx;
    proven_loop_recv_cancel(app->loop, &app->sock);             /* its `received` comes with PROVEN_ERR_CANCELLED */
}

static void client_send(ex_app_t *app) {
    app->have = 0;
    proven_result_size_t s = proven_loop_send(app->loop, &app->sock, (proven_mem_view_t){ (const proven_byte_t *)MESSAGE, strlen(MESSAGE) });
    /* A receive into the client's own buffer, which is the loop's until `received`. */
    if (s.err != PROVEN_OK || proven_loop_recv(app->loop, &app->sock, (proven_mem_mut_t){ app->buf, sizeof app->buf }) != PROVEN_OK) {
        proven_loop_close(app->loop, &app->sock);
        return;
    }
    proven_loop_timer_set(app->loop, &app->deadline, 5000, client_deadline, app);
}

static void client_received(void *ctx, proven_err_t err, proven_size_t n) {
    ex_app_t *app = ctx;
    if (err != PROVEN_OK) { proven_loop_close(app->loop, &app->sock); return; }    /* the deadline passed, or the server went */
    app->have += n;
    if (app->have < strlen(MESSAGE)) {                          /* a stream: the echo may come in pieces */
        if (proven_loop_recv(app->loop, &app->sock, (proven_mem_mut_t){ app->buf + app->have, sizeof app->buf - app->have }) != PROVEN_OK)
            proven_loop_close(app->loop, &app->sock);
        return;
    }
    proven_loop_timer_cancel(app->loop, &app->deadline);
    if (memcmp(app->buf, MESSAGE, strlen(MESSAGE)) != 0) { proven_loop_close(app->loop, &app->sock); return; }
    if (++app->rounds < ROUNDS) { client_send(app); return; }
    app->echoed = true;
    proven_loop_close(app->loop, &app->sock);                   /* the server reads the end, and closes its side */
}

static void client_connected(void *ctx, proven_err_t err) {
    ex_app_t *app = ctx;
    if (err != PROVEN_OK) { proven_loop_close(app->loop, &app->sock); return; }
    client_send(app);
}

static void client_closed(void *ctx) {
    ex_app_t *app = ctx;
    (void)proven_net_close(&app->conn);
    app->client_closed = true;
}

static const proven_loop_sock_fns_t client_fns = {
    .received = client_received, .connected = client_connected, .closed = client_closed
};

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    static ex_app_t app;
    EXAMPLE_REQUIRE(proven_loop_create(heap, &app.loop) == PROVEN_OK, "a loop");

    // ---- the server ----------------------------------------------------------------
    proven_net_addr_t bound;
    EXAMPLE_REQUIRE(proven_net_listen(proven_net_addr_ipv4(127, 0, 0, 1, 0), 16, &app.listener, &bound) == PROVEN_OK, "a listener on a free port");
    EXAMPLE_REQUIRE(proven_loop_accept(app.loop, &app.acc, &app.listener, on_accept, &app) == PROVEN_OK, "accepting");

    // ---- the client ----------------------------------------------------------------
    EXAMPLE_REQUIRE(proven_loop_connect(app.loop, &app.sock, bound, &app.conn, &client_fns, &app) == PROVEN_OK, "a connect begun");

    // ---- run until everything has ended ----------------------------------------------
    proven_net_deadline_t end = proven_net_deadline_in(10000);
    while (!(app.client_closed && app.accepting_ended) && proven_time_monotonic_now() < end) {
        EXAMPLE_REQUIRE(proven_loop_poll(app.loop, proven_net_deadline_in(100)) == PROVEN_OK, "a round");
    }
    EXAMPLE_REQUIRE(app.echoed && app.rounds == ROUNDS, "the message came back three times");
    EXAMPLE_REQUIRE(app.peers_closed == 1 && app.accepting_ended && !proven_net_listener_is_open(&app.listener),
                    "the server closed its side after the client's, and stopped accepting");
    proven_loop_destroy(app.loop);
    return EXAMPLE_OK();
}

6. The event-driven server#

proven_err_t proven_http_event_server_create(proven_loop_t *loop, const proven_http_event_server_config_t *config,
                                             proven_http_event_server_t **out);
proven_err_t proven_http_event_server_listen(proven_http_event_server_t *server, proven_net_addr_t at, proven_net_addr_t *bound);
void proven_http_event_server_destroy(proven_http_event_server_t *server);
proven_size_t proven_http_event_server_connections(const proven_http_event_server_t *server);
proven_err_t proven_http_event_server_set_max_streams(proven_http_event_server_t *server, proven_size_t n);
proven_err_t proven_http_event_server_set_window(proven_http_event_server_t *server, proven_size_t n);
void proven_http_event_server_stop_listening(proven_http_event_server_t *server);
typedef void (*proven_http_event_drained_fn)(void *ctx, proven_err_t why);
proven_err_t proven_http_event_server_shutdown(proven_http_event_server_t *server, proven_u32 grace_ms,
                                               proven_http_event_drained_fn on_drained, void *ctx);

A server is made on a loop, told where to listen, and does nothing until the loop runs.

Four functions. The configuration carries them in on, and ctx is passed to each:

A stream is one request and its response. A proven_http_stream_t * is valid from on_request until on_done returns, and not a moment longer. on_done is therefore the one place to free what you attached to it, and to cancel a timer or forget a job that still holds it: a function that arrives later with a stream that is over has nothing valid to call.

void proven_http_stream_set_user(proven_http_stream_t *stream, void *user);
void *proven_http_stream_user(const proven_http_stream_t *stream);
proven_net_addr_t proven_http_stream_peer(const proven_http_stream_t *stream);

proven_http_stream_set_user attaches a pointer of yours to the stream and proven_http_stream_user gives it back - a stream that was given none returns null, which an on_done that frees must allow for. proven_http_stream_peer is the client's address.

7. Answering#

proven_err_t proven_http_stream_respond(proven_http_stream_t *stream, proven_u16 status,
                                        const proven_http_header_t *headers, proven_size_t header_count,
                                        proven_mem_view_t body);
proven_err_t proven_http_stream_begin(proven_http_stream_t *stream, proven_u16 status,
                                      const proven_http_header_t *headers, proven_size_t header_count,
                                      proven_u64 content_length);
proven_result_size_t proven_http_stream_write(proven_http_stream_t *stream, proven_mem_view_t data);
proven_err_t proven_http_stream_end(proven_http_stream_t *stream);
void proven_http_stream_abort(proven_http_stream_t *stream);
proven_size_t proven_http_stream_buffered(const proven_http_stream_t *stream);

All of it at once. proven_http_stream_respond sends a status, your headers and a body held in memory. The body is copied whatever its size, so it is for bodies that are small. Content-Length, Date and Connection are written for you and may not be among headers. It may be called inside on_request or at any later time on the loop's thread, while the stream is valid.

In pieces. proven_http_stream_begin sends the head and promises a body of content_length bytes, or - given PROVEN_HTTP_EVENT_LENGTH_UNKNOWN - a body sent chunked, whose length nobody knows yet. Then proven_http_stream_write offers bytes, and proven_http_stream_end finishes.

proven_http_stream_write may take fewer bytes than you offer - as many as fit under max_buffered_output, possibly none - and returns how many it took. That is not an error. It means the client has not taken what was already sent, and the rule is: keep the rest, return, and offer it again when on_writable is called. A producer written that way goes exactly as fast as its slowest client, and the server's memory for that connection stays under the limit whatever the size of the response.

Three details that follow from the protocol and need nothing from you: a response to HEAD, and one with status 204 or 304, is sent without a body whatever you pass; a request that said Expect: 100-continue is told to go on when its body is first wanted; and after the response to a request that asked for it, or that could not be read to its end, the connection is closed rather than kept.

on_done may be called inside proven_http_stream_respond or proven_http_stream_end, when everything fits in the socket at once. After either returns, do not touch the stream or what on_done freed.

The program below answers in four ways - at once, later from a timer, streamed with a pump that gives way when it is refused, and after a body received in pieces. The test suite compiles and runs it:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*
 * An HTTP server on an event loop: one thread, and no call in it that waits.
 *
 * The server tells these functions what happened, and they return at once. Four kinds of
 * answer are shown: at once; later, from a timer; streamed, giving way when the client is
 * slow; and after a request body has arrived piece by piece.
 *
 * The client is the blocking one of chapter 11, on the example's main thread.
 */

typedef struct {
    proven_loop_t *loop;
    int served;
    proven_http_event_server_t *server;
    proven_err_t ended;                /* what on_drained was told */
} ex_app_t;

/* What this program keeps for one request while it is being answered. */
typedef struct {
    ex_app_t *app;
    proven_http_stream_t *stream;
    proven_loop_timer_t timer;
    proven_size_t sent, total;         /* /count: how far the body has got */
    proven_size_t part;                /* /count: how much of the current line has gone */
    proven_size_t received;            /* /size: body bytes seen */
} ex_exchange_t;

/* Write lines until the server says "no more for now". It says so when the client has not
 * taken what was already sent; the rest waits for on_writable. */
static void count_pump(ex_exchange_t *x) {
    while (x->sent < x->total) {
        char line[32];
        int n = snprintf(line, sizeof line, "%u\n", (unsigned)x->sent);
        proven_result_size_t w = proven_http_stream_write(x->stream, (proven_mem_view_t){ (const proven_byte_t *)line + x->part, (proven_size_t)n - x->part });
        if (w.err != PROVEN_OK) return;                   /* the client left */
        x->part += w.value;                              /* a write may take part of what it is given */
        if (x->part < (proven_size_t)n) return;           /* held back: on_writable will say when */
        x->part = 0;
        x->sent++;
    }
    (void)proven_http_stream_end(x->stream);
}

static void answer_later(void *ctx) {
    ex_exchange_t *x = ctx;
    (void)proven_http_stream_respond(x->stream, 200, NULL, 0, proven_mem_view_from_u8(PROVEN_LIT("worth the wait\n")));
}

static void on_request(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    ex_app_t *app = ctx;
    ex_exchange_t *x = calloc(1, sizeof *x);
    if (!x) { proven_http_stream_abort(stream); return; }
    x->app = app;
    x->stream = stream;
    proven_http_stream_set_user(stream, x);

    /* `head` and everything in it are good only until this function returns. */
    if (proven_u8str_view_eq(head->target, PROVEN_LIT("/"))) {
        /* The simplest answer: all of it, now. */
        (void)proven_http_stream_respond(stream, 200, NULL, 0, proven_mem_view_from_u8(PROVEN_LIT("hello\n")));
    } else if (proven_u8str_view_eq(head->target, PROVEN_LIT("/slow"))) {
        /* Not now: remember the stream and answer from a timer. The loop goes on meanwhile. */
        proven_loop_timer_set(app->loop, &x->timer, 50, answer_later, x);
    } else if (proven_u8str_view_eq(head->target, PROVEN_LIT("/count"))) {
        /* A body made as it goes, sent chunked because its length is not known first. */
        x->total = 20000;
        if (proven_http_stream_begin(stream, 200, NULL, 0, PROVEN_HTTP_EVENT_LENGTH_UNKNOWN) == PROVEN_OK) count_pump(x);
    } else if (proven_u8str_view_eq(head->target, PROVEN_LIT("/size"))) {
        /* Nothing yet: the body is on its way, and on_body answers when it has all come. */
    } else {
        (void)proven_http_stream_respond(stream, 404, NULL, 0, proven_mem_view_from_u8(PROVEN_LIT("not here\n")));
    }
}

static void on_body(void *ctx, proven_http_stream_t *stream, proven_mem_view_t piece, bool last) {
    (void)ctx;
    ex_exchange_t *x = proven_http_stream_user(stream);
    x->received += piece.size;
    if (last) {
        char text[32];
        int n = snprintf(text, sizeof text, "%u bytes\n", (unsigned)x->received);
        (void)proven_http_stream_respond(stream, 200, NULL, 0, (proven_mem_view_t){ (const proven_byte_t *)text, (proven_size_t)n });
    }
}

/* The output that was held back has gone: carry on from where the writing stopped. */
static void on_writable(void *ctx, proven_http_stream_t *stream) {
    (void)ctx;
    ex_exchange_t *x = proven_http_stream_user(stream);
    if (x->total > 0) count_pump(x);
}

/* Called once for every on_request, whatever happened. The place to free what was attached. */
static void on_done(void *ctx, proven_http_stream_t *stream, proven_err_t why) {
    ex_app_t *app = ctx;
    ex_exchange_t *x = proven_http_stream_user(stream);
    if (!x) return;
    proven_loop_timer_cancel(app->loop, &x->timer);
    free(x);
    if (why == PROVEN_OK) app->served++;
}

/* The end, on the loop's thread: what is open finishes, and then the loop is stopped. */
static void on_drained(void *ctx, proven_err_t why) {
    ex_app_t *app = ctx;
    app->ended = why;
    proven_loop_stop(app->loop);
}
static void begin_end(void *ctx) {
    ex_app_t *app = ctx;
    if (proven_http_event_server_shutdown(app->server, 2000, on_drained, app) != PROVEN_OK) proven_loop_stop(app->loop);
}

static void run(void *arg) { (void)proven_loop_run(((ex_app_t *)arg)->loop); }

static char g_url[64];
static proven_u8str_view_t url_for(proven_u16 port, const char *path) {
    int n = snprintf(g_url, sizeof g_url, "http://127.0.0.1:%u%s", (unsigned)port, path);
    return (proven_u8str_view_t){ (const proven_byte_t *)g_url, (proven_size_t)n };
}

static bool body_equals(proven_http_client_response_t *resp, proven_allocator_t alloc, const char *want) {
    proven_u8str_t body = { 0 };
    bool ok = proven_http_client_read_all(resp, alloc, &body, 1024 * 1024) == PROVEN_OK &&
              proven_u8str_view_eq(proven_u8str_as_view(&body), (proven_u8str_view_t){ (const proven_byte_t *)want, strlen(want) });
    proven_u8str_destroy(alloc, &body);
    return ok;
}

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    ex_app_t app = { 0 };

    // ---- the server ----------------------------------------------------------------
    /* A server is made on a loop and listens; it does nothing until the loop runs. */
    EXAMPLE_REQUIRE(proven_loop_create(heap, &app.loop) == PROVEN_OK, "a loop");
    proven_http_event_server_config_t config = {
        .on = { on_request, on_body, on_writable, on_done },
        .ctx = &app,
        .max_buffered_output = 16 * 1024,        /* hold at most this much per connection that the client has not taken */
    };
    proven_http_event_server_t *server = NULL;
    proven_net_addr_t at;
    EXAMPLE_REQUIRE(proven_http_event_server_create(app.loop, &config, &server) == PROVEN_OK &&
                    proven_http_event_server_listen(server, proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &at) == PROVEN_OK, "an event-driven server on a free port");
    app.server = server;

    /* This example gives the loop a thread of its own, so that its main thread can be a client. */
    proven_job_sys_t *threads = NULL;
    proven_job_group_t running;
    proven_job_group_init(&running);
    EXAMPLE_REQUIRE(proven_job_system_init(heap, 1, 4, &threads) == PROVEN_OK &&
                    proven_job_group_submit(threads, &running, run, &app) == PROVEN_OK, "the loop runs on another thread");

    // ---- four requests -------------------------------------------------------------
    proven_http_client_config_t client_config = { .alloc = heap, .max_idle_connections = 1 };
    proven_http_client_t *client = NULL;
    EXAMPLE_REQUIRE(proven_http_client_create(&client_config, &client) == PROVEN_OK, "a client");
    proven_http_client_response_t resp;

    EXAMPLE_REQUIRE(proven_http_client_get(client, url_for(at.port, "/"), &resp) == PROVEN_OK && body_equals(&resp, heap, "hello\n"), "answered at once");
    proven_http_client_finish(&resp);
    EXAMPLE_REQUIRE(proven_http_client_get(client, url_for(at.port, "/slow"), &resp) == PROVEN_OK && body_equals(&resp, heap, "worth the wait\n"), "answered 50 ms later, from a timer");
    proven_http_client_finish(&resp);

    /* Twenty thousand lines. Whenever the server holds back, the handler stops and on_writable resumes it. */
    EXAMPLE_REQUIRE(proven_http_client_get(client, url_for(at.port, "/count"), &resp) == PROVEN_OK && resp.status == 200, "a streamed response begins");
    proven_u8str_t counted = { 0 };
    EXAMPLE_REQUIRE(proven_http_client_read_all(&resp, heap, &counted, 1024 * 1024) == PROVEN_OK, "and is read to its end");
    proven_u8str_view_t all = proven_u8str_as_view(&counted);
    EXAMPLE_REQUIRE(all.size == 108890 && memcmp(all.ptr, "0\n1\n2\n", 6) == 0 && memcmp(all.ptr + all.size - 6, "19999\n", 6) == 0, "every line arrived, in order");
    proven_u8str_destroy(heap, &counted);
    proven_http_client_finish(&resp);

    static proven_byte_t upload[50000];
    proven_http_client_request_t post = { .method = PROVEN_LIT("POST"), .url = url_for(at.port, "/size"), .body = { upload, sizeof upload } };
    EXAMPLE_REQUIRE(proven_http_client_send(client, &post, &resp) == PROVEN_OK && body_equals(&resp, heap, "50000 bytes\n"), "a body counted piece by piece");
    proven_http_client_finish(&resp);
    proven_http_client_destroy(client);

    // ---- done ----------------------------------------------------------------------
    /* End gracefully, on the loop's thread; on_drained stops the loop. Then take the server
     * apart: nothing else is touching it by then. */
    app.ended = PROVEN_ERR_INVALID_STATE;
    EXAMPLE_REQUIRE(proven_loop_post(app.loop, begin_end, &app) == PROVEN_OK, "the end is asked for");
    proven_job_group_wait(threads, &running);
    EXAMPLE_REQUIRE(app.ended == PROVEN_OK && app.served == 4, "it ended with nothing cut, and four exchanges ended with on_done(PROVEN_OK)");
    proven_http_event_server_destroy(server);
    proven_loop_destroy(app.loop);
    proven_job_system_close(threads);
    proven_job_system_destroy(threads);
    return EXAMPLE_OK();
}

Look at count_pump: it is the shape every streamed response has. It is called from on_request to start and from on_writable to go on, it keeps its position in the exchange's own state, and it returns the moment a write takes less than was offered. Whether a given run is ever refused depends on how fast the client reads; the function is correct either way.

8. Request bodies#

void proven_http_stream_pause(proven_http_stream_t *stream);
void proven_http_stream_resume(proven_http_stream_t *stream);

A body arrives through on_body in pieces of whatever size the network delivered, with the framing - a Content-Length or chunks - already taken off. A request with no body gets no on_body call at all; one with a body gets a last call with last true.

A handler may answer before the body has arrived, or without reading it. The rest of the body is then read and thrown away if that is cheap, and the connection is closed after the response if it is not.

proven_http_stream_pause is backpressure in the other direction: no more pieces are delivered, and the connection is not read, until proven_http_stream_resume. A handler that writes what it receives to something slower than the network uses the pair so that the client is slowed instead of the body being piled up in memory. Pieces that were waiting may arrive inside the call to proven_http_stream_resume.

Trailers (since v0.107.0) are fields a sender puts after a body - a checksum, a status it knows only at the end. Over HTTP/1.1 they follow a chunked body; over HTTP/2 and HTTP/3 they are a HEADERS frame that ends the stream.

A received field that may not be a trailer (Content-Length, Set-Cookie…) is left out. Over HTTP/1.1 a chunked request body holds a buffer of PROVEN_HTTP_MAX_TRAILER_BYTES (8 KiB) for its trailer section while it arrives; an idle connection holds none. The client's side is in section 12.

9. Limits, time and refusals#

Zero-initialise a proven_http_event_server_config_t, set on.on_request, and set what you need. Every zero field has a default:

A request the server will not serve never reaches on_request. It is answered and the connection closed:

When one of these happens after on_request - a body that turns out too long, or malformed part-way - the exchange ends with on_done and the reason, and the client gets the status if a response had not already begun. A connection that passes body_timeout_ms, write_timeout_ms or idle_timeout_ms is closed; an exchange in progress on it ends with on_done(PROVEN_ERR_TIMEOUT). A client that goes away ends its exchange with PROVEN_ERR_RESET.

Requests sent one after another on a connection without waiting - pipelined - are served in order, one stream at a time.

10. Threads, TLS and shutting down#

Threads. One loop is one thread, and the server belongs to it. A handler that must compute or wait for something submits a job, returns, and the job posts the answer back:

1. on_request copies what it needs from the head, remembers the stream, and submits the job.
2. The job works on another thread, touching nothing of the server.
3. It calls proven_loop_post with a function that, on the loop's thread, calls proven_http_stream_respond.
4. If on_done came first - the client left - that function must find out and do nothing. The usual way is for on_done to mark the exchange's own state as finished and leave freeing it to whichever of the two runs last.

TLS. Give the configuration a proven_tls_config_t with a certificate (Chapter 14, section 2) and every connection is TLS; nothing else in your code changes. The handshake is carried by the loop like any other reading and writing, and is subject to head_timeout_ms. The configuration must outlive the server. A handshake's public-key work is done on the loop's thread, and while it runs nothing else is served: Chapter 14, section 9, gives what one costs.

Shutting down: proven_http_event_server_shutdown(server, grace_ms, on_drained, ctx). On the loop's thread. From the call nothing new is taken; an HTTP/1.1 connection with no request on it is closed; one where a request has begun to arrive, or is being answered, finishes it with Connection: close; an HTTP/2 connection says goodbye (GOAWAY) and finishes its open streams; a WebSocket is closed with 1001, "going away", and its peer's answer waited for. When grace_ms has passed, what is left is ended - an exchange gets on_done(PROVEN_ERR_RESET), a WebSocket on_closed. on_drained(ctx, why) is called once, from the loop, when the last connection has gone: why is PROVEN_OK, or PROVEN_ERR_TIMEOUT when the deadline cut something. It may call proven_loop_stop; then destroy the server and the loop after the loop has returned. A shutdown is final - the server listens and adopts no more. The deadline bounds connections, not handlers: work a handler sent to another thread comes back to a stream that is gone.

proven_http_event_server_stop_listening alone is the first step of that, without the rest: nobody new, HTTP/1.1 responses say Connection: close, HTTP/2 says goodbye - and connections end when their clients let them, or at idle_timeout_ms. Destroy the server on the loop's thread or after the loop has stopped - not while another thread is inside proven_loop_run. The example of section 7 ends with a shutdown.

11. WebSocket on the loop#

Chapter 12's WebSocket connection has calls that wait, and so occupies a thread for as long as it is open. ws_event.h is the other form: a request to the server of this chapter is turned into a WebSocket inside on_request, and from then on the loop tells your functions what arrived. A connection that is silent holds its state and no buffer.

proven_err_t proven_ws_event_accept(proven_http_stream_t *stream, const proven_http_request_t *head,
                                    const proven_ws_event_config_t *config, proven_ws_stream_t **out);
proven_err_t proven_ws_stream_send(proven_ws_stream_t *ws, bool text, proven_mem_view_t data);
proven_err_t proven_ws_stream_send_piece(proven_ws_stream_t *ws, bool text, proven_mem_view_t data, bool first, bool last);
proven_err_t proven_ws_stream_ping(proven_ws_stream_t *ws, proven_mem_view_t data);
void proven_ws_stream_close(proven_ws_stream_t *ws, proven_u16 code, proven_u8str_view_t reason);
void proven_ws_stream_abort(proven_ws_stream_t *ws);
void proven_ws_stream_pause(proven_ws_stream_t *ws);
void proven_ws_stream_resume(proven_ws_stream_t *ws);
void proven_ws_stream_set_user(proven_ws_stream_t *ws, void *user);
void *proven_ws_stream_user(const proven_ws_stream_t *ws);
proven_net_addr_t proven_ws_stream_peer(const proven_ws_stream_t *ws);
proven_size_t proven_ws_stream_buffered(const proven_ws_stream_t *ws);

Accepting. proven_ws_event_accept is called inside on_request, while head is valid. It checks the request, answers 101, and turns the connection into a WebSocket. The HTTP exchange ends inside that call: on_done(PROVEN_OK) is called for the stream, which is then over, and *out is what you hold until on_closed returns.

A proven_ws_event_config_t is zero-initialised, given on.on_message, and whatever else is needed:

Messages arrive in pieces. on_message is given each message in the pieces the network delivered, with first and last marking its ends: a message of any size passes through without being assembled or held, and piece is good only until the function returns. Text is checked to be UTF-8 over the whole message, and a piece may end in the middle of a character. A handler that wants whole messages collects the pieces itself, up to a limit of its own choosing - the program below does. max_message_bytes is the server's limit, not a buffer: a message past it is refused with close code 1009.

A send is taken whole or not at all. proven_ws_stream_send queues one message when less than max_buffered_output is held, and otherwise takes nothing and returns PROVEN_ERR_AGAIN: keep the message, and send it again when on_writable is called. That is the difference from the HTTP body of section 7, whose write says how many bytes it took - a frame cannot be half-sent without the library choosing where to cut your message. The most a connection holds is therefore the limit and one message of your choosing; for a large message produced as it goes, proven_ws_stream_send_piece sends it a fragment at a time under the same rule. A handler that only echoes can stop reading while it waits, with proven_ws_stream_pause, so that a client that sends faster than it reads is slowed rather than buffered.

How it ends. on_closed is called exactly once, and what it is given says how:

Pings from the client are answered for you, and its pongs need nothing from you. Liveness is judged by what is received: a client that is sent data and never reads it is not detected by that alone - but nothing more is queued for it than the output limit, and once it stops answering pings it is dropped.

The test suite compiles and runs this program:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*
 * WebSocket on the event-driven server: connections that cost no thread.
 *
 * A request becomes a WebSocket inside on_request. From then on the loop tells these functions
 * what arrived, and they return at once. This server shouts back: every text message is
 * answered in capitals.
 *
 * The client is the blocking one of chapter 12, on the example's main thread.
 */

typedef struct {
    proven_loop_t *loop;
    int opened, closed;
    proven_u16 last_code;
} ex_app_t;

/* What this program keeps for one connection: the message being collected. */
typedef struct {
    ex_app_t *app;
    char text[4096];
    proven_size_t len;
    bool waiting;                 /* a reply the server would not take yet */
} ex_peer_t;

/* Send the reply. If the client is not taking what was sent before, the server refuses it:
 * keep it, stop reading from this client, and try again when on_writable says so. */
static void reply(proven_ws_stream_t *ws, ex_peer_t *peer) {
    proven_err_t err = proven_ws_stream_send(ws, true, (proven_mem_view_t){ (const proven_byte_t *)peer->text, peer->len });
    if (err == PROVEN_ERR_AGAIN) {
        peer->waiting = true;
        proven_ws_stream_pause(ws);
        return;
    }
    peer->waiting = false;
    peer->len = 0;
}

/* A message arrives in the pieces the network delivered. This handler wants whole messages,
 * so it collects them - up to a limit of its own, which is the point of doing it by hand. */
static void on_message(void *ctx, proven_ws_stream_t *ws, bool text, proven_mem_view_t piece, bool first, bool last) {
    (void)ctx;
    ex_peer_t *peer = proven_ws_stream_user(ws);
    if (first) peer->len = 0;
    if (!text || peer->len + piece.size > sizeof peer->text) {
        proven_ws_stream_close(ws, 1009, PROVEN_LIT("short text only"));       /* 1009: too big */
        return;
    }
    for (proven_size_t i = 0; i < piece.size; ++i) {
        proven_byte_t ch = piece.ptr[i];
        peer->text[peer->len++] = (char)(ch >= 'a' && ch <= 'z' ? ch - 32 : ch);
    }
    if (last) reply(ws, peer);
}

/* There is room again. */
static void on_writable(void *ctx, proven_ws_stream_t *ws) {
    (void)ctx;
    ex_peer_t *peer = proven_ws_stream_user(ws);
    if (!peer->waiting) return;
    reply(ws, peer);
    if (!peer->waiting) proven_ws_stream_resume(ws);
}

/* Called once, however the connection ended. The place to free what was attached. */
static void on_closed(void *ctx, proven_ws_stream_t *ws, proven_u16 code, proven_err_t why) {
    ex_app_t *app = ctx;
    (void)why;
    free(proven_ws_stream_user(ws));
    app->last_code = code;
    app->closed++;
}

static void on_request(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    ex_app_t *app = ctx;
    ex_peer_t *peer = calloc(1, sizeof *peer);
    if (!peer) { proven_http_stream_abort(stream); return; }
    peer->app = app;

    proven_ws_event_config_t config = {
        .on = { on_message, on_writable, on_closed },
        .ctx = app,
        .max_message_bytes = sizeof peer->text,
    };
    proven_ws_stream_t *ws = NULL;
    /* Check the request, answer 101, and turn the connection into a WebSocket. The HTTP
     * exchange ends inside this call (on_done is called, if there is one). */
    proven_err_t err = proven_ws_event_accept(stream, head, &config, &ws);
    if (err != PROVEN_OK) {
        free(peer);
        /* Nothing was sent: the request is still an ordinary one, and is answered as such. */
        if (err == PROVEN_ERR_UNSUPPORTED) {
            proven_http_header_t version = { PROVEN_LIT("Sec-WebSocket-Version"), PROVEN_LIT("13") };
            (void)proven_http_stream_respond(stream, 426, &version, 1, (proven_mem_view_t){ 0 });
        } else if (err != PROVEN_ERR_RESET) {
            (void)proven_http_stream_respond(stream, err == PROVEN_ERR_NOT_FOUND ? 404 : 400, NULL, 0, (proven_mem_view_t){ 0 });
        }
        return;
    }
    /* `stream` is over now. `ws` is valid until on_closed returns. */
    proven_ws_stream_set_user(ws, peer);
    app->opened++;
}

static void run(void *arg) { (void)proven_loop_run(((ex_app_t *)arg)->loop); }

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    ex_app_t app = { 0 };

    // ---- a server on a loop, on its own thread -------------------------------------
    EXAMPLE_REQUIRE(proven_loop_create(heap, &app.loop) == PROVEN_OK, "a loop");
    /* HTTP/2 too, and WebSocket on it (RFC 8441): a client that speaks HTTP/2 opens its
     * WebSocket as a stream of that connection. on_request above is the same for both - it is
     * this program's client, over HTTP/1.1, that the run below uses. */
    proven_http_event_server_config_t config = { .on = { .on_request = on_request }, .ctx = &app, .http2 = true, .http2_websocket = true };
    proven_http_event_server_t *server = NULL;
    proven_net_addr_t at;
    EXAMPLE_REQUIRE(proven_http_event_server_create(app.loop, &config, &server) == PROVEN_OK &&
                    proven_http_event_server_listen(server, proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &at) == PROVEN_OK, "an event-driven server on a free port");
    proven_job_sys_t *threads = NULL;
    proven_job_group_t running;
    proven_job_group_init(&running);
    EXAMPLE_REQUIRE(proven_job_system_init(heap, 1, 4, &threads) == PROVEN_OK &&
                    proven_job_group_submit(threads, &running, run, &app) == PROVEN_OK, "the loop runs on another thread");

    // ---- a client ------------------------------------------------------------------
    /* A blocking client is the simplest way to show the other end. */
    proven_http_client_config_t client_config = { .alloc = heap };
    proven_http_client_t *client = NULL;
    EXAMPLE_REQUIRE(proven_http_client_create(&client_config, &client) == PROVEN_OK, "an HTTP client");
    char url[64];
    int n = snprintf(url, sizeof url, "ws://127.0.0.1:%u/shout", (unsigned)at.port);
    proven_ws_conn_config_t ws_config = { .alloc = heap };
    proven_ws_conn_t *conn = NULL;
    EXAMPLE_REQUIRE(proven_ws_conn_connect(client, (proven_u8str_view_t){ (const proven_byte_t *)url, (proven_size_t)n }, NULL, 0, PROVEN_LIT(""),
                                           &ws_config, &conn, NULL) == PROVEN_OK, "the server accepts a WebSocket");

    proven_ws_message_t msg;
    EXAMPLE_REQUIRE(proven_ws_conn_send_text(conn, PROVEN_LIT("hello, loop")) == PROVEN_OK &&
                    proven_ws_conn_receive(conn, proven_net_deadline_in(5000), &msg) == PROVEN_OK &&
                    msg.text && msg.data.size == 11 && memcmp(msg.data.ptr, "HELLO, LOOP", 11) == 0, "a message comes back in capitals");
    /* However the client cuts a message up, the handler sees one message. */
    EXAMPLE_REQUIRE(proven_ws_conn_send_part(conn, true, proven_mem_view_from_u8(PROVEN_LIT("in three ")), false) == PROVEN_OK &&
                    proven_ws_conn_send_part(conn, true, proven_mem_view_from_u8(PROVEN_LIT("pieces, ")), false) == PROVEN_OK &&
                    proven_ws_conn_send_part(conn, true, proven_mem_view_from_u8(PROVEN_LIT("one answer")), true) == PROVEN_OK &&
                    proven_ws_conn_receive(conn, proven_net_deadline_in(5000), &msg) == PROVEN_OK &&
                    msg.data.size == 27 && memcmp(msg.data.ptr, "IN THREE PIECES, ONE ANSWER", 27) == 0, "three fragments are one message to the handler");

    // ---- closing -------------------------------------------------------------------
    EXAMPLE_REQUIRE(proven_ws_conn_close(conn, 1000, PROVEN_LIT("done")) == PROVEN_OK, "the client's close is answered");
    proven_ws_conn_destroy(conn);
    proven_http_client_destroy(client);

    /* Stop the loop before looking at what its thread wrote. */
    proven_loop_stop(app.loop);
    proven_job_group_wait(threads, &running);
    EXAMPLE_REQUIRE(app.opened == 1 && app.closed == 1 && app.last_code == 1000, "one connection was opened, and closed once, with the client's code");
    proven_http_event_server_destroy(server);
    proven_loop_destroy(app.loop);
    proven_job_system_close(threads);
    proven_job_system_destroy(threads);
    return EXAMPLE_OK();
}

Over HTTP/2 (RFC 8441)#

With http2_websocket set (and http2), the server says so in its first HTTP/2 settings, and a client that knows RFC 8441 opens its WebSocket as a stream of the HTTP/2 connection it already has, instead of a connection of its own. The handler is the same: the request reaches on_request as CONNECT, with its path as the target and a :protocol field first, proven_ws_check_request recognises it (with an empty key - there is none), and proven_ws_event_accept answers 200 instead of 101. The program above enables it with one line; its own client is HTTP/1.1, which the same server goes on serving.

• Each such WebSocket holds a stream - one of max_concurrent_streams (100 by default) - for as long as it is open. A connection full of WebSockets refuses new requests.
• Its frames travel in DATA frames, so HTTP/2's flow control is under the WebSocket's own: http2_window_bytes is what a client may send before the WebSocket has taken it (with 0, a window that grows while the WebSocket reads, as for any body), and on_writable comes when the stream's output has moved.
• Its end is END_STREAM from each side after the close handshake - until the client's arrives the stream keeps its place; a peer that resets the stream is a WebSocket closed with 1006 and PROVEN_ERR_RESET. A shutdown closes it with 1001. A handler that refuses the request (any answer but 2xx) ends the stream as it would a request's.
• Without http2_websocket a client must not try (the setting says so), and a stream that does is refused with PROTOCOL_ERROR.
• Not here: this library's own WebSocket client over HTTP/2 - proven_ws_event_connect makes a connection of its own, as before. Tested against Node's HTTP/2 client; no browser was tried.

A connection this side makes#

proven_err_t proven_ws_event_connect(proven_loop_t *loop, const proven_ws_event_connect_options_t *options, proven_ws_stream_t **out);

A program on a loop is as often the one that asks. proven_ws_event_connect begins a connection to a server and returns at once; what it gives back is a proven_ws_stream_t, the same type a server's connections are. Every call of this section works on it as it does on one of those, and its messages arrive through the same three callbacks: a handler written for one side serves the other.

What differs is the beginning.

Nothing may be sent until the server has accepted: the sending calls answer PROVEN_ERR_INVALID_STATE, and on_open says when it has (sending from inside it is fine). proven_ws_stream_abort ends an attempt that is still under way, and so does proven_ws_stream_close.

on_closed is called exactly once for every connection that was begun - also for one that never became a WebSocket. Its code is then 0, and why says what happened:

A client's frames are masked, as the protocol requires of them; the keys come from a generator seeded from the system when the connection is begun. Not here: a proxy, reconnection (begin again), and WebSocket over HTTP/2.

This program has both ends on one loop. It is compiled and run by the test suite:

#include <stdio.h>
#include <string.h>

/*
 * A WebSocket client on the loop: a connection that costs no thread on this side either.
 *
 * proven_ws_event_connect begins a connection and returns at once. What it gives back is a
 * proven_ws_stream_t - the type a server's connections are - so the calls and the callbacks
 * are the ones the server above uses. What is new is the beginning: on_open says when the
 * server has accepted, and until then nothing may be sent.
 *
 * One loop carries both ends here: the server that is asked, and the client that asks.
 */

typedef struct {
    proven_loop_t *loop;
    int opened, answers, closed, refused;
    proven_u16 code;
    char last[64];
} ex_app_t;

// ---- the server: it says what it was sent ----

static void served(void *ctx, proven_ws_stream_t *ws, bool text, proven_mem_view_t piece, bool first, bool last) {
    (void)ctx; (void)first;
    if (!last) return;                      /* this example's messages are small: one piece each */
    char reply[48];
    int n = snprintf(reply, sizeof reply, "%s, %u bytes", text ? "text" : "binary", (unsigned)piece.size);
    (void)proven_ws_stream_send(ws, true, (proven_mem_view_t){ (const proven_byte_t *)reply, (proven_size_t)n });
}

static void on_request(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    (void)ctx;
    proven_ws_event_config_t config = { .on = { .on_message = served } };
    proven_ws_stream_t *ws = NULL;
    if (proven_ws_event_accept(stream, head, &config, &ws) != PROVEN_OK) (void)proven_http_stream_respond(stream, 400, NULL, 0, (proven_mem_view_t){ 0 });
}

// ---- the client ----

/* The server accepted. From here on - and from inside this call - messages may be sent. */
static void on_open(void *ctx, proven_ws_stream_t *ws, proven_u8str_view_t protocol) {
    ex_app_t *app = ctx;
    (void)protocol;
    app->opened++;
    (void)proven_ws_stream_send(ws, true, proven_mem_view_from_u8(PROVEN_LIT("hello")));
}

/* The same callback a server has. This side asks twice and then says goodbye. */
static void on_message(void *ctx, proven_ws_stream_t *ws, bool text, proven_mem_view_t piece, bool first, bool last) {
    ex_app_t *app = ctx;
    (void)text; (void)first;
    if (!last) return;
    proven_size_t n = piece.size < sizeof app->last - 1 ? piece.size : sizeof app->last - 1;
    memcpy(app->last, piece.ptr, n);
    app->last[n] = '\0';
    printf("the server says: %s\n", app->last);
    if (++app->answers == 1) {
        static const proven_byte_t bytes[300] = { 1, 2, 3 };
        (void)proven_ws_stream_send(ws, false, (proven_mem_view_t){ bytes, sizeof bytes });
    } else {
        proven_ws_stream_close(ws, 1000, PROVEN_LIT("thank you"));
    }
}

/* Called exactly once for every connection that was begun - also for one that never opened:
 * its code is then 0, and `why` says what went wrong. */
static void on_closed(void *ctx, proven_ws_stream_t *ws, proven_u16 code, proven_err_t why) {
    ex_app_t *app = ctx;
    (void)ws;
    if (code == 0) {
        /* Never a WebSocket: nothing listened, the server answered something else, or time ran out. */
        printf("no connection: error %d\n", (int)why);
        app->refused++;
    } else {
        app->code = code;
        app->closed++;
    }
    /* Both connections of this program have ended: there is nothing left for the loop to do. */
    if (app->closed == 1 && app->refused == 1) proven_loop_stop(app->loop);
}

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    ex_app_t app = { 0 };
    EXAMPLE_REQUIRE(proven_loop_create(heap, &app.loop) == PROVEN_OK, "a loop");
    proven_http_event_server_config_t server_config = { .on = { .on_request = on_request } };
    proven_http_event_server_t *server = NULL;
    proven_net_addr_t at;
    EXAMPLE_REQUIRE(proven_http_event_server_create(app.loop, &server_config, &server) == PROVEN_OK, "a server");
    proven_err_t err = proven_http_event_server_listen(server, proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &at);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        proven_http_event_server_destroy(server);
        proven_loop_destroy(app.loop);
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK, "it listens");

    /* The host is an address, so none has to be given: names are not resolved on a loop. */
    char url[64];
    int n = snprintf(url, sizeof url, "ws://127.0.0.1:%u/chat", (unsigned)at.port);
    proven_ws_event_connect_options_t options = {
        .url = { (const proven_byte_t *)url, (proven_size_t)n },
        .on_open = on_open,
        .ws = { .on = { .on_message = on_message, .on_closed = on_closed }, .ctx = &app },
    };
    proven_ws_stream_t *ws = NULL;
    EXAMPLE_REQUIRE(proven_ws_event_connect(app.loop, &options, &ws) == PROVEN_OK && ws != NULL && app.opened == 0, "a connection is begun; nothing has happened yet");
    /* The call has returned and the connection is being made: this is too early to speak. */
    EXAMPLE_REQUIRE(proven_ws_stream_send(ws, true, proven_mem_view_from_u8(PROVEN_LIT("too early"))) == PROVEN_ERR_INVALID_STATE, "nothing can be sent before the server has accepted");

    /* And one that will fail, to see how that is told: a port that was just closed. */
    proven_net_listener_t unused;
    proven_net_addr_t nobody;
    EXAMPLE_REQUIRE(proven_net_listen(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), 1, &unused, &nobody) == PROVEN_OK && proven_net_listener_close(&unused) == PROVEN_OK, "a port nobody listens on");
    n = snprintf(url, sizeof url, "ws://127.0.0.1:%u/chat", (unsigned)nobody.port);
    options.url.size = (proven_size_t)n;
    proven_ws_stream_t *lost = NULL;
    err = proven_ws_event_connect(app.loop, &options, &lost);
    EXAMPLE_REQUIRE(err == PROVEN_OK || err == PROVEN_ERR_REFUSED, "an attempt that cannot succeed: refused now, or told later");
    if (err != PROVEN_OK) app.refused++;

    /* Everything else happens here. */
    if (!(app.closed == 1 && app.refused == 1)) EXAMPLE_REQUIRE(proven_loop_run(app.loop) == PROVEN_OK, "the loop runs until both have ended");
    EXAMPLE_REQUIRE(app.opened == 1 && app.answers == 2 && strcmp(app.last, "binary, 300 bytes") == 0, "it opened once, and two messages were answered");
    EXAMPLE_REQUIRE(app.closed == 1 && app.code == 1000 && app.refused == 1, "it closed once, with this side's code - and the other attempt was reported once");

    proven_http_event_server_destroy(server);
    proven_loop_destroy(app.loop);
    return EXAMPLE_OK();
}

Two threads, one connection. A program in which one thread waits for messages and another decides what to send - a chat client, a feed with a control channel - cannot do that on a blocking connection (Chapter 12 says why). It can on this one: the loop waits for messages without a thread of its own, and the other thread hands it what to send with proven_loop_post. The send runs on the loop's thread, the only one that touches the connection; what crosses is a copy of the message, in memory the posted call frees. This program is compiled and run by the test suite:

#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*
 * One thread waits for messages while another decides what to send - a chat client's shape.
 *
 * A blocking connection (Chapter 12) cannot do it: it is one thread's, and a lock around
 * proven_ws_conn_receive would hold every send for as long as the receive waits. The loop's
 * connection can. It waits for messages without a thread of its own, and another thread hands
 * it what to send with proven_loop_post: the send itself happens on the loop's thread, the
 * only one that touches the connection.
 *
 * One loop carries both ends here: a server that answers each message, and the client.
 */

#define LINES 3

typedef struct {
    proven_loop_t *loop;
    proven_ws_stream_t *ws;           /* the client's connection: the loop's thread only */
    atomic_bool open;                 /* what the typing thread may read */
    int answers, closed;
} ex_app_t;

// ---- the server: it answers each message ------------------------------------

static void served(void *ctx, proven_ws_stream_t *ws, bool text, proven_mem_view_t piece, bool first, bool last) {
    (void)ctx; (void)text; (void)first;
    if (!last) return;                      /* this example's messages are small: one piece each */
    char reply[80];
    int n = snprintf(reply, sizeof reply, "got: %.*s", (int)piece.size, (const char *)piece.ptr);
    (void)proven_ws_stream_send(ws, true, (proven_mem_view_t){ (const proven_byte_t *)reply, (proven_size_t)n });
}

static void on_request(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    (void)ctx;
    proven_ws_event_config_t config = { .on = { .on_message = served } };
    proven_ws_stream_t *ws = NULL;
    if (proven_ws_event_accept(stream, head, &config, &ws) != PROVEN_OK) (void)proven_http_stream_respond(stream, 400, NULL, 0, (proven_mem_view_t){ 0 });
}

// ---- the client, on the loop's thread -----------------------------------------

static void on_open(void *ctx, proven_ws_stream_t *ws, proven_u8str_view_t protocol) {
    ex_app_t *app = ctx;
    (void)protocol;
    app->ws = ws;
    atomic_store(&app->open, true);
}

static void on_message(void *ctx, proven_ws_stream_t *ws, bool text, proven_mem_view_t piece, bool first, bool last) {
    ex_app_t *app = ctx;
    (void)text; (void)first;
    if (!last) return;
    printf("the server says: %.*s\n", (int)piece.size, (const char *)piece.ptr);
    if (++app->answers == LINES) proven_ws_stream_close(ws, 1000, PROVEN_LIT("bye"));
}

static void on_closed(void *ctx, proven_ws_stream_t *ws, proven_u16 code, proven_err_t why) {
    ex_app_t *app = ctx;
    (void)ws; (void)code; (void)why;
    app->ws = NULL;
    app->closed++;
    proven_loop_stop(app->loop);
}

// ---- what crosses from the typing thread ---------------------------------------

/* A line to send, in memory of its own: the typing thread has moved on by the time it is sent. */
typedef struct { ex_app_t *app; proven_size_t len; char text[64]; } ex_line_t;

/* On the loop's thread. */
static void send_here(void *ctx) {
    ex_line_t *line = ctx;
    if (line->app->ws) (void)proven_ws_stream_send(line->app->ws, true, (proven_mem_view_t){ (const proven_byte_t *)line->text, line->len });
    free(line);
}

/* The typing thread. It never touches the connection: it posts. */
static void typist(void *arg) {
    ex_app_t *app = arg;
    static const char *const lines[LINES] = { "hello", "how are you", "goodbye" };
    while (!atomic_load(&app->open)) proven_time_sleep(1);
    for (int i = 0; i < LINES; ++i) {
        ex_line_t *line = malloc(sizeof *line);
        if (!line) return;
        line->app = app;
        line->len = strlen(lines[i]);
        memcpy(line->text, lines[i], line->len);
        if (proven_loop_post(app->loop, send_here, line) != PROVEN_OK) free(line);
        proven_time_sleep(5);               /* someone typing */
    }
}

static void run(void *arg) { (void)proven_loop_run(((ex_app_t *)arg)->loop); }

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    static ex_app_t app;
    atomic_init(&app.open, false);
    EXAMPLE_REQUIRE(proven_loop_create(heap, &app.loop) == PROVEN_OK, "a loop");
    proven_http_event_server_config_t server_config = { .on = { .on_request = on_request } };
    proven_http_event_server_t *server = NULL;
    proven_net_addr_t at;
    EXAMPLE_REQUIRE(proven_http_event_server_create(app.loop, &server_config, &server) == PROVEN_OK, "a server");
    proven_err_t err = proven_http_event_server_listen(server, proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &at);
    if (err == PROVEN_ERR_PERMISSION || err == PROVEN_ERR_UNSUPPORTED) {
        printf("no sockets in this environment; nothing to show\n");
        proven_http_event_server_destroy(server);
        proven_loop_destroy(app.loop);
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK, "it listens");

    char url[64];
    int n = snprintf(url, sizeof url, "ws://127.0.0.1:%u/chat", (unsigned)at.port);
    proven_ws_event_connect_options_t options = {
        .url = { (const proven_byte_t *)url, (proven_size_t)n },
        .on_open = on_open,
        .ws = { .on = { .on_message = on_message, .on_closed = on_closed }, .ctx = &app },
    };
    proven_ws_stream_t *ws = NULL;
    EXAMPLE_REQUIRE(proven_ws_event_connect(app.loop, &options, &ws) == PROVEN_OK, "a connection is begun");

    // ---- two threads: the loop, and someone typing --------------------------------
    proven_job_sys_t *threads = NULL;
    proven_job_group_t looping, typing;
    proven_job_group_init(&looping);
    proven_job_group_init(&typing);
    EXAMPLE_REQUIRE(proven_job_system_init(heap, 2, 4, &threads) == PROVEN_OK &&
                    proven_job_group_submit(threads, &looping, run, &app) == PROVEN_OK &&
                    proven_job_group_submit(threads, &typing, typist, &app) == PROVEN_OK, "the loop on one thread, the typist on another");

    /* The loop stops when the connection has closed, after the third answer. */
    proven_job_group_wait(threads, &typing);
    proven_job_group_wait(threads, &looping);
    EXAMPLE_REQUIRE(app.answers == LINES && app.closed == 1, "three lines sent from the typing thread, three answers received on the loop's");

    /* Anything posted and not yet run is run here, before the loop goes. */
    EXAMPLE_REQUIRE(proven_loop_poll(app.loop, PROVEN_NET_DONT_WAIT) == PROVEN_OK, "a last round");
    proven_job_system_close(threads);
    proven_job_system_destroy(threads);
    proven_http_event_server_destroy(server);
    proven_loop_destroy(app.loop);
    return EXAMPLE_OK();
}

12. The event-driven client#

proven_err_t proven_http_event_client_create(proven_loop_t *loop, const proven_http_event_client_config_t *config,
                                             proven_http_event_client_t **out);
void proven_http_event_client_destroy(proven_http_event_client_t *client);
proven_size_t proven_http_event_client_requests(const proven_http_event_client_t *client);
proven_err_t proven_http_event_client_start(proven_http_event_client_t *client, const proven_http_event_request_options_t *options,
                                            proven_http_event_request_t **out);
proven_result_size_t proven_http_event_request_write(proven_http_event_request_t *request, proven_mem_view_t data);
proven_err_t proven_http_event_request_end(proven_http_event_request_t *request);
void proven_http_event_request_abort(proven_http_event_request_t *request);
void proven_http_event_request_pause(proven_http_event_request_t *request);
void proven_http_event_request_resume(proven_http_event_request_t *request);
void proven_http_event_request_set_user(proven_http_event_request_t *request, void *user);
void *proven_http_event_request_user(const proven_http_event_request_t *request);

Chapter 11's client sends a request and waits for the answer. This one starts a request and returns; the response head, the pieces of its body and the end arrive through callbacks on the loop's thread, and any number of requests are in flight at once.

It is deliberately narrow, and the two limits are worth knowing before anything else.

• A request is one exchange and nothing around it. There are no redirects, answers to challenges, cookies or proxies. Chapter 11's client has all of them; use it wherever a thread per request in flight is affordable.
• Names are not resolved. Looking a name up can take seconds, and nothing on a loop may wait. A request says where to connect with address; only a URL whose host is an IP address needs none. For a name: call proven_net_resolve on a job, post the result to the loop, and start the request there, as the program at the end of this section does. The URL keeps the name - it is what goes into Host, and over TLS it is the name the server's certificate must be for.

A request is described by a proven_http_event_request_options_t. Everything it points to is copied or used before proven_http_event_client_start returns:

Connections are kept, and counted. Since v0.36.0 an HTTP/1.1 request does not make a connection of its own and close it. The client keeps the connections of each origin - the URL's scheme, host and port, and the address connected to - and a request takes one of them:

• one that is idle, if its origin has one. A connection is kept when an exchange on it ended cleanly: the response was HTTP/1.1 without Connection: close, its body had an end of its own (a length, or chunks) and that end was reached, the request had been sent whole, and nothing came after the response. At most max_kept_connections (8) are kept in the whole client, the one idle longest making room for another, and none for longer than idle_timeout_ms (60 s);
• or a new one, while the origin has fewer than max_connections_per_origin (6), busy and idle together;
• or it waits, in the order of starting, for the next connection of its origin that comes free. This is the part to know: the seventh request in flight to one origin is not sent until one of the six has ended. A program that keeps more than six requests open to one server on purpose - long polls, streams that do not end - must raise max_connections_per_origin, or its seventh request waits and, after connect_timeout_ms, ends with PROVEN_ERR_TIMEOUT: waiting is on the connecting clock, which starts with the request. At most max_waiting_requests (1,000) wait in a client; the next start is PROVEN_ERR_BUSY. A waiting request holds its head and its body, can be written to (up to max_buffered_output), and can be aborted.

A kept connection may be dead by the time it is used: servers close idle connections without asking, and the close can cross the next request on the wire. So when a request is lost on a connection that had carried one before, and nothing at all had come back on it, the client takes it that the server never saw the request, and sends it once more, on a new connection, without telling you - one on_done, for whichever sending ends it. A request whose body is written in pieces cannot be sent again, since the client does not have the pieces; it ends with PROVEN_ERR_AGAIN, which means what it means for HTTP/2: not acted on, and yours to start again. Once any byte of an answer has arrived, or on a connection made for the request, a loss is PROVEN_ERR_RESET as it always was. (The reading "never saw it" can be wrong - a server may take a request and die before it answers. It is the reading every HTTP client that reuses connections makes, the blocking client of Chapter 11 among them; no_connection_reuse is for a program that will not have it.)

retry_unsent - a number, zero by default - has the client begin a request again by itself when the server cannot have acted on it. Over HTTP/1.1 that is one case, and a narrow one: a connection that was made and then lost before any of the request had been given to it, which in practice is a server that goes away while TLS is being agreed. Such a request ends with PROVEN_ERR_RESET without the setting, and with it is begun on a new connection, so many times, each with its own connect_timeout_ms. A request of which anything was sent is not begun again, and neither is one whose body is written in pieces. (The sending-again on a kept connection, above, needs no setting and is not counted. Over HTTP/2 the setting covers more: Chapter 17.)

retry_delay_ms is how long the client waits before each such try, in either protocol: zero by default, and the same for every try unless retry_delay_max_ms is set - then the wait doubles with each try of a request, from retry_delay_ms up to that bound. Nothing is added at random: requests that were refused together are tried again together. While it waits the request has no connection - the one it lost is closed, and its place may go to a request that was waiting - and connect_timeout_ms of the try begins when the wait ends. Aborting the request during the wait ends it as any abort does. The sending-again on a kept connection is not delayed. (With max_connections, below: when the wait is over the request takes a place as a new one would, waiting for one if there is none; if as many wait already as may, it ends with PROVEN_ERR_BUSY - the one case in which on_done says so, and not proven_http_event_client_start.)

max_connections bounds the connections of the whole client, of both protocols, busy and idle; zero, the default, is no bound. A request that would make one more first has an idle connection closed for it - an HTTP/1.1 one, the one idle longest, before an HTTP/2 one with nothing on it. With none idle the request waits, whichever its protocol. An HTTP/1.1 request waits in the queue described above, and has the next place that is freed (a connection that closes serves a request to its own origin first, then the one that has waited longest and whose origin has room). An HTTP/2 request whose origin has no connection waits on no connection at all, in the order of starting, and is put on one when a place can be had - or on its origin's, if another request has made one meanwhile. Either wait is the wait for a connection: connect_timeout_ms ends it with PROVEN_ERR_TIMEOUT, and no more requests wait than max_waiting_requests (the next start is PROVEN_ERR_BUSY). When both protocols wait, a place goes to an HTTP/1.1 request first. Setting max_kept_connections above max_connections is PROVEN_ERR_INVALID_ARG.

no_connection_reuse gives back the old behaviour for connections - Connection: close in every request, every connection closed when its response has ended - but not the old absence of a limit: six at a time to an origin still.

Starting. When proven_http_event_client_start returns an error, the request was not started and no callback is or will be made. When it returns PROVEN_OK, on_done will be called exactly once - never inside that call, so code that starts a request can finish what it was doing before anything happens to it. A request is valid from the start until on_done returns.

How it ends. on_done says:

A request body written as it goes follows the rule of section 7. proven_http_event_request_write takes as many bytes as fit under max_buffered_output, possibly none, and says how many; the rest is offered again when on_writable is called. It may be called at once after start, before the connection exists: what is taken waits. proven_http_event_request_end finishes the body; ending short of a promised length ends the request with PROVEN_ERR_INVALID_FORMAT.

The configuration's fields, each with a default when zero:

The test suite compiles and runs this program:

#include <stdio.h>
#include <string.h>

/*
 * The event-driven HTTP client: several requests in flight on one thread.
 *
 * A request is started and the call returns at once; what happens to it arrives through
 * callbacks, on the loop's thread. The server here is the event-driven one of this chapter, on
 * the same loop - this whole program is one thread and one worker for the one thing that waits.
 *
 * That one thing is looking a name up. The client does not resolve names: a lookup can take
 * seconds, and nothing on a loop may wait. So the third request shows the pattern - resolve on
 * a worker, post the answer back, start the request.
 */

typedef struct app ex_app_t;

/* What this program keeps for one request. */
typedef struct {
    ex_app_t *app;
    const char *path;
    int status;
    char body[64];
    proven_size_t len;
    proven_err_t why;
    bool done;
} ex_fetch_t;

struct app {
    proven_loop_t *loop;
    proven_http_event_client_t *client;
    proven_job_sys_t *workers;
    proven_u16 port;
    ex_fetch_t fetch[3];
    int finished;
    /* written by the worker, read on the loop after it has been posted */
    proven_net_addr_t found;
    proven_err_t lookup;
};

// ---- the server the requests go to -----------------------------------------

static void serve(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    (void)ctx;
    if (proven_u8str_view_eq(head->target, PROVEN_LIT("/a"))) (void)proven_http_stream_respond(stream, 200, NULL, 0, proven_mem_view_from_u8(PROVEN_LIT("first")));
    else if (proven_u8str_view_eq(head->target, PROVEN_LIT("/b"))) (void)proven_http_stream_respond(stream, 200, NULL, 0, proven_mem_view_from_u8(PROVEN_LIT("second")));
    else (void)proven_http_stream_respond(stream, 404, NULL, 0, proven_mem_view_from_u8(PROVEN_LIT("nothing here")));
}

// ---- the client's callbacks -------------------------------------------------

/* The head: good only until this function returns, so take what is wanted from it. */
static void on_response(void *ctx, proven_http_event_request_t *request, const proven_http_response_t *head) {
    (void)request;
    ((ex_fetch_t *)ctx)->status = head->status;
}

/* The body, a piece at a time. Nothing is collected for you; collect what you want. */
static void on_body(void *ctx, proven_http_event_request_t *request, proven_mem_view_t piece, bool last) {
    (void)request; (void)last;
    ex_fetch_t *f = ctx;
    for (proven_size_t i = 0; i < piece.size && f->len < sizeof f->body - 1; ++i) f->body[f->len++] = (char)piece.ptr[i];
}

/* Once for every request that was started, whatever happened to it. */
static void on_done(void *ctx, proven_http_event_request_t *request, proven_err_t why) {
    (void)request;
    ex_fetch_t *f = ctx;
    f->why = why;
    f->done = true;
    if (++f->app->finished == 3) proven_loop_stop(f->app->loop);
}

static proven_err_t fetch(ex_app_t *app, ex_fetch_t *f, const char *path, const proven_net_addr_t *address, const char *host) {
    char url[96];
    int n = snprintf(url, sizeof url, "http://%s:%u%s", host, (unsigned)app->port, path);
    f->app = app;
    f->path = path;
    proven_http_event_request_options_t options = {
        .url = { (const proven_byte_t *)url, (proven_size_t)n },
        .address = address,                               /* null is allowed only when the URL's host is an IP address */
        .on = { on_response, on_body, NULL, on_done },
        .ctx = f,
    };
    return proven_http_event_client_start(app->client, &options, NULL);     /* returns at once */
}

// ---- a name, resolved elsewhere ---------------------------------------------

/* On the loop's thread again: now there is an address, and the request can start.
 * The URL keeps the name - it is what goes into Host - and the address says where to connect. */
static void resolved(void *ctx) {
    ex_app_t *app = ctx;
    if (app->lookup != PROVEN_OK || fetch(app, &app->fetch[2], "/missing", &app->found, "localhost") != PROVEN_OK) {
        app->fetch[2].why = PROVEN_ERR_NOT_FOUND;
        if (++app->finished == 3) proven_loop_stop(app->loop);
    }
}

/* On a worker thread: this call may wait as long as the resolver takes. */
static void resolve(void *ctx) {
    ex_app_t *app = ctx;
    proven_net_addr_t all[8];
    proven_size_t count = 0;
    app->lookup = proven_net_resolve(PROVEN_LIT("localhost"), app->port, all, 8, &count);
    /* The server here listens on IPv4 only; a real program would try each address in turn. */
    bool have = false;
    for (proven_size_t i = 0; i < count && !have; ++i) {
        if (all[i].family == PROVEN_NET_FAMILY_IPV4) { app->found = all[i]; have = true; }
    }
    if (app->lookup == PROVEN_OK && !have) app->lookup = PROVEN_ERR_NOT_FOUND;
    if (proven_loop_post(app->loop, resolved, app) != PROVEN_OK) proven_loop_stop(app->loop);
}

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    ex_app_t app = { 0 };

    // ---- one loop, a server and a client on it ------------------------------------
    /* Nothing runs yet: a loop does nothing until it is run. */
    EXAMPLE_REQUIRE(proven_loop_create(heap, &app.loop) == PROVEN_OK, "a loop");
    proven_http_event_server_config_t server_config = { .on = { .on_request = serve } };
    proven_http_event_server_t *server = NULL;
    proven_net_addr_t at;
    EXAMPLE_REQUIRE(proven_http_event_server_create(app.loop, &server_config, &server) == PROVEN_OK &&
                    proven_http_event_server_listen(server, proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &at) == PROVEN_OK, "a server to ask");
    app.port = at.port;
    EXAMPLE_REQUIRE(proven_http_event_client_create(app.loop, NULL, &app.client) == PROVEN_OK, "an event-driven client");
    EXAMPLE_REQUIRE(proven_job_system_init(heap, 1, 4, &app.workers) == PROVEN_OK, "one worker, for the lookup");

    // ---- three requests ------------------------------------------------------------
    /* Two by address. Both are started before either has been sent. */
    EXAMPLE_REQUIRE(fetch(&app, &app.fetch[0], "/a", NULL, "127.0.0.1") == PROVEN_OK &&
                    fetch(&app, &app.fetch[1], "/b", NULL, "127.0.0.1") == PROVEN_OK, "two requests started");
    EXAMPLE_REQUIRE(proven_http_event_client_requests(app.client) == 2 && !app.fetch[0].done && !app.fetch[1].done, "both in flight, neither answered: nothing has run yet");

    /* One by name. Without an address it is refused - so resolve first, on the worker. */
    ex_fetch_t unresolved = { 0 };
    EXAMPLE_REQUIRE(fetch(&app, &unresolved, "/missing", NULL, "localhost") == PROVEN_ERR_INVALID_ARG, "a name with no address is refused");
    EXAMPLE_REQUIRE(proven_job_submit(app.workers, resolve, &app), "the lookup is handed to a worker");

    // ---- run -----------------------------------------------------------------------
    /* on_done stops the loop when the third request has ended. */
    EXAMPLE_REQUIRE(proven_loop_run(app.loop) == PROVEN_OK, "the loop ran until all three were done");

    EXAMPLE_REQUIRE(app.fetch[0].why == PROVEN_OK && app.fetch[0].status == 200 && strcmp(app.fetch[0].body, "first") == 0, "the first answer");
    EXAMPLE_REQUIRE(app.fetch[1].why == PROVEN_OK && app.fetch[1].status == 200 && strcmp(app.fetch[1].body, "second") == 0, "the second answer");
    EXAMPLE_REQUIRE(app.fetch[2].why == PROVEN_OK && app.fetch[2].status == 404 && strcmp(app.fetch[2].body, "nothing here") == 0, "the third, by name: a 404 is a response, not an error");

    proven_job_system_close(app.workers);
    proven_job_system_destroy(app.workers);
    proven_http_event_client_destroy(app.client);
    proven_http_event_server_destroy(server);
    proven_loop_destroy(app.loop);
    return EXAMPLE_OK();
}

Trailers#

A request whose body is written with body_length PROVEN_HTTP_EVENT_LENGTH_UNKNOWN can end with trailers, and a response's trailers can be read after its body (since v0.107.0), over every protocol the client speaks:

As on the server, a received field that may not be a trailer is left out, and over HTTP/1.1 a chunked response holds an 8 KiB buffer for its trailer section. Compiled and run by the test suite:

#include <stdio.h>
#include <string.h>

/*
 * Trailers on the loop: fields after a body. A client uploads with a checksum it knows only at
 * the end; the server checks it after the body and answers with a trailer of its own. The same
 * calls work over HTTP/2 and HTTP/3 - there the trailers are a HEADERS frame that ends the stream.
 */

static unsigned sum_of(const proven_byte_t *p, proven_size_t n, unsigned s) { for (proven_size_t i = 0; i < n; ++i) s = s * 31 + p[i]; return s; }

// ---- the server -------------------------------------------------------------

typedef struct { unsigned sum; } ex_upload_t;
static ex_upload_t g_upload;                    /* one request here; a real server keeps one per stream (proven_http_stream_set_user) */

static void serve(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    (void)ctx; (void)stream; (void)head;
    g_upload.sum = 0;
}

static void serve_body(void *ctx, proven_http_stream_t *stream, proven_mem_view_t piece, bool last) {
    (void)ctx;
    g_upload.sum = sum_of(piece.ptr, piece.size, g_upload.sum);
    if (!last) return;
    /* The body is over: its trailers are here, until on_done returns. */
    proven_http_header_t trailers[4];
    proven_size_t count = 0;
    proven_u8str_view_t claimed = { 0 };
    char mine[16];
    const int ml = snprintf(mine, sizeof mine, "%u", g_upload.sum);
    const bool good = proven_http_stream_trailers(stream, trailers, 4, &count) == PROVEN_OK &&
                      proven_http_header_find(trailers, count, PROVEN_LIT("X-Checksum"), &claimed) &&
                      proven_u8str_view_eq(claimed, (proven_u8str_view_t){ (const proven_byte_t *)mine, (proven_size_t)ml });
    /* Trailers follow a body of no stated length. */
    if (proven_http_stream_begin(stream, 200, NULL, 0, PROVEN_HTTP_EVENT_LENGTH_UNKNOWN) != PROVEN_OK) return;
    (void)proven_http_stream_write(stream, proven_mem_view_from_u8(PROVEN_LIT("stored\n")));
    const proven_http_header_t status = { PROVEN_LIT("X-Upload"), good ? PROVEN_LIT("verified") : PROVEN_LIT("corrupt") };
    (void)proven_http_stream_end_trailers(stream, &status, 1);
}

// ---- the client -------------------------------------------------------------

typedef struct {
    proven_loop_t *loop;
    proven_http_event_request_t *request;
    proven_mem_view_t data;
    proven_size_t sent;
    bool ended;
    char verdict[16];
    proven_err_t why;
} ex_put_t;

/* Write what the client takes; when it is all written, end with the checksum. */
static void put_more(ex_put_t *p) {
    while (p->sent < p->data.size) {
        proven_result_size_t w = proven_http_event_request_write(p->request, (proven_mem_view_t){ p->data.ptr + p->sent, p->data.size - p->sent });
        if (w.err != PROVEN_OK || w.value == 0) return;            /* on_writable says when to go on */
        p->sent += w.value;
    }
    if (p->ended) return;
    char sum[16];
    const int sl = snprintf(sum, sizeof sum, "%u", sum_of(p->data.ptr, p->data.size, 0));
    const proven_http_header_t trailer = { PROVEN_LIT("X-Checksum"), { (const proven_byte_t *)sum, (proven_size_t)sl } };
    p->ended = proven_http_event_request_end_trailers(p->request, &trailer, 1) == PROVEN_OK;    /* the fields are copied */
}

static void on_writable(void *ctx, proven_http_event_request_t *request) { (void)request; put_more(ctx); }

static void on_body(void *ctx, proven_http_event_request_t *request, proven_mem_view_t piece, bool last) {
    (void)piece;
    ex_put_t *p = ctx;
    if (!last) return;
    proven_http_header_t trailers[4];
    proven_size_t count = 0;
    proven_u8str_view_t v = { 0 };
    if (proven_http_event_request_trailers(request, trailers, 4, &count) == PROVEN_OK && proven_http_header_find(trailers, count, PROVEN_LIT("X-Upload"), &v))
        memcpy(p->verdict, v.ptr, v.size < sizeof p->verdict - 1 ? v.size : sizeof p->verdict - 1);
}

static void on_done(void *ctx, proven_http_event_request_t *request, proven_err_t why) {
    (void)request;
    ex_put_t *p = ctx;
    p->why = why;
    proven_loop_stop(p->loop);
}

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();
    proven_loop_t *loop = NULL;
    EXAMPLE_REQUIRE(proven_loop_create(heap, &loop) == PROVEN_OK, "a loop");
    proven_http_event_server_config_t server_config = { .on = { .on_request = serve, .on_body = serve_body } };
    proven_http_event_server_t *server = NULL;
    proven_net_addr_t at;
    EXAMPLE_REQUIRE(proven_http_event_server_create(loop, &server_config, &server) == PROVEN_OK &&
                    proven_http_event_server_listen(server, proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), &at) == PROVEN_OK, "a server");
    proven_http_event_client_t *client = NULL;
    EXAMPLE_REQUIRE(proven_http_event_client_create(loop, NULL, &client) == PROVEN_OK, "a client");

    /* A field that frames the message may not be a trailer; a checksum may. */
    EXAMPLE_REQUIRE(!proven_http_trailer_allowed(PROVEN_LIT("Content-Length")) && proven_http_trailer_allowed(PROVEN_LIT("X-Checksum")), "what may be a trailer");

    char url[64];
    const int n = snprintf(url, sizeof url, "http://127.0.0.1:%u/upload", (unsigned)at.port);
    ex_put_t put = { .loop = loop, .data = proven_mem_view_from_u8(PROVEN_LIT("the bytes of a file, sent as they are read")) };
    proven_http_event_request_options_t options = {
        .method = PROVEN_LIT("PUT"), .url = { (const proven_byte_t *)url, (proven_size_t)n },
        .body_length = PROVEN_HTTP_EVENT_LENGTH_UNKNOWN,         /* written afterwards, with no length: so trailers can follow */
        .on = { NULL, on_body, on_writable, on_done }, .ctx = &put,
    };
    EXAMPLE_REQUIRE(proven_http_event_client_start(client, &options, &put.request) == PROVEN_OK, "the upload started");
    put_more(&put);
    EXAMPLE_REQUIRE(proven_loop_run(loop) == PROVEN_OK, "the loop ran until the upload was done");
    EXAMPLE_REQUIRE(put.why == PROVEN_OK && put.ended && strcmp(put.verdict, "verified") == 0, "the server says, after its body, that the checksum matched");

    proven_http_event_client_destroy(client);
    proven_http_event_server_destroy(server);
    proven_loop_destroy(loop);
    return EXAMPLE_OK();
}

13. Several loops#

One loop is one thread, and a connection belongs to one loop for its whole life: nothing about it is ever touched by another thread, which is why nothing in this chapter takes a lock. A server that needs more than one processor core is therefore several loops, each with a server of its own - and something to share the incoming connections among them.

proven_err_t proven_http_event_server_adopt(proven_http_event_server_t *server, proven_net_conn_t *conn);

proven_http_event_server_adopt gives a server a connection that was accepted somewhere else. It becomes one of the server's exactly as if the server had accepted it; with TLS, the handshake begins. So one thread listens and accepts, and deals each connection to the servers in turn. Three rules make that correct:

• adopt is called on the server's loop thread, like everything else. The accepting thread does not call it: it posts a function to that loop (proven_loop_post), and the function calls it.
• The connection travels in memory of its own. A proven_net_conn_t is not to be copied while it is open, so what is posted is a small block holding it, freed by the function that adopts.
• Taken apart in order: the accepting thread first, so that nothing more is posted; then the loops; then - after one last round of each loop, for anything posted and not yet run - the servers.

The other way to the same end needs no call at all: give each loop's server a listener of its own on a different address or port, and let whatever is in front - a load balancer, or DNS - spread the clients. The load measurements of the next section were made that way.

Whether more loops help depends on what the time goes on. They multiply what handlers can compute; they do nothing for a handler that waits, which should not be on a loop at all (section 10).

The test suite compiles and runs this program:

#include <stdatomic.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/*
 * Several loops behind one port: one thread accepts, and deals connections to servers on
 * loops of their own.
 *
 * A connection belongs to one loop, and nothing about it is ever touched by another thread.
 * So scaling across processors is several loops, each with its own server - and something has
 * to share the incoming connections among them. Here that is a thread that accepts and deals
 * them out in turn, which works wherever the library does.
 */

enum { LOOPS = 2 };

/* One loop, its server, and what its handler counts. Only that loop's thread touches it. */
typedef struct {
    int index;
    proven_loop_t *loop;
    proven_http_event_server_t *server;
    int served;                       /* read by the main thread only after the loop has stopped */
} ex_worker_t;

static ex_worker_t g_workers[LOOPS];
static proven_net_listener_t g_listener;
static atomic_bool g_stop;

static void on_request(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    ex_worker_t *me = ctx;
    (void)head;
    char text[16];
    int n = snprintf(text, sizeof text, "loop %d", me->index);
    me->served++;
    (void)proven_http_stream_respond(stream, 200, NULL, 0, (proven_mem_view_t){ (const proven_byte_t *)text, (proven_size_t)n });
}

/* What crosses from the accepting thread to a loop. An open connection is not to be copied,
 * so it travels in memory of its own. */
typedef struct { ex_worker_t *to; proven_net_conn_t conn; } ex_handoff_t;

/* On the loop's thread - the only place a server may be given a connection. */
static void adopt_here(void *ctx) {
    ex_handoff_t *h = ctx;
    if (proven_http_event_server_adopt(h->to->server, &h->conn) != PROVEN_OK) (void)proven_net_close(&h->conn);   /* refused: still ours to close */
    free(h);
}

/* The accepting thread. It touches no server: it accepts, chooses, and posts. */
static void acceptor(void *arg) {
    (void)arg;
    int next = 0;
    while (!atomic_load(&g_stop)) {
        ex_handoff_t *h = malloc(sizeof *h);
        if (!h) break;
        /* A short wait at a time, so that the thread notices when it is told to stop. */
        if (proven_net_accept(&g_listener, proven_net_deadline_in(20), &h->conn, NULL) != PROVEN_OK) { free(h); continue; }
        h->to = &g_workers[next];
        next = (next + 1) % LOOPS;
        if (proven_loop_post(h->to->loop, adopt_here, h) != PROVEN_OK) { (void)proven_net_close(&h->conn); free(h); }
    }
}

static void run(void *arg) { (void)proven_loop_run(((ex_worker_t *)arg)->loop); }

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();

    // ---- two loops, each with a server that has no listener -----------------------
    /* A server with no listener serves only what it is given. */
    proven_job_sys_t *threads = NULL;
    proven_job_group_t loops, accepting;
    proven_job_group_init(&loops);
    proven_job_group_init(&accepting);
    EXAMPLE_REQUIRE(proven_job_system_init(heap, LOOPS + 1, 8, &threads) == PROVEN_OK, "three threads");
    for (int i = 0; i < LOOPS; ++i) {
        ex_worker_t *w = &g_workers[i];
        w->index = i;
        proven_http_event_server_config_t config = { .on = { .on_request = on_request }, .ctx = w };
        EXAMPLE_REQUIRE(proven_loop_create(heap, &w->loop) == PROVEN_OK &&
                        proven_http_event_server_create(w->loop, &config, &w->server) == PROVEN_OK &&
                        proven_job_group_submit(threads, &loops, run, w) == PROVEN_OK, "a loop on its own thread, with a server");
    }

    // ---- one listener, and the thread that deals ----------------------------------
    proven_net_addr_t at;
    EXAMPLE_REQUIRE(proven_net_listen(proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), 64, &g_listener, &at) == PROVEN_OK &&
                    proven_job_group_submit(threads, &accepting, acceptor, NULL) == PROVEN_OK, "a listener and its accepting thread");

    // ---- eight connections ---------------------------------------------------------
    /* A client that keeps no connection open: eight requests are eight connections. */
    proven_http_client_config_t client_config = { .alloc = heap, .max_idle_connections = 0 };
    proven_http_client_t *client = NULL;
    EXAMPLE_REQUIRE(proven_http_client_create(&client_config, &client) == PROVEN_OK, "a client");
    char url[64];
    int n = snprintf(url, sizeof url, "http://127.0.0.1:%u/", (unsigned)at.port);
    int answers[LOOPS] = { 0 };
    for (int i = 0; i < 8; ++i) {
        proven_http_client_response_t response;
        proven_u8str_t body = { 0 };
        EXAMPLE_REQUIRE(proven_http_client_get(client, (proven_u8str_view_t){ (const proven_byte_t *)url, (proven_size_t)n }, &response) == PROVEN_OK &&
                        proven_http_client_read_all(&response, heap, &body, 64) == PROVEN_OK, "a request is answered");
        proven_u8str_view_t text = proven_u8str_as_view(&body);
        if (text.size == 6 && memcmp(text.ptr, "loop ", 5) == 0 && text.ptr[5] - '0' < LOOPS) answers[text.ptr[5] - '0']++;
        proven_u8str_destroy(heap, &body);
        proven_http_client_finish(&response);
    }
    proven_http_client_destroy(client);
    EXAMPLE_REQUIRE(answers[0] == 4 && answers[1] == 4, "dealt in turn: four connections were served by each loop");

    // ---- taking it apart, in this order --------------------------------------------
    /* The acceptor first, so that nothing more is posted. Then the loops. Then the servers. */
    atomic_store(&g_stop, true);
    proven_job_group_wait(threads, &accepting);
    (void)proven_net_listener_close(&g_listener);
    for (int i = 0; i < LOOPS; ++i) proven_loop_stop(g_workers[i].loop);
    proven_job_group_wait(threads, &loops);
    int served = 0;
    for (int i = 0; i < LOOPS; ++i) {
        /* Anything posted and not yet run is run here, before its server goes. */
        EXAMPLE_REQUIRE(proven_loop_poll(g_workers[i].loop, PROVEN_NET_DONT_WAIT) == PROVEN_OK, "a last round");
        served += g_workers[i].served;
        proven_http_event_server_destroy(g_workers[i].server);
        proven_loop_destroy(g_workers[i].loop);
    }
    EXAMPLE_REQUIRE(served == 8, "eight requests were served in all");
    proven_job_system_close(threads);
    proven_job_system_destroy(threads);
    return EXAMPLE_OK();
}

One port and no accepting thread (Linux)#

proven_err_t proven_http_event_port_open(proven_allocator_t alloc, proven_net_addr_t at, proven_size_t loops, bool http3,
                                         proven_http_event_port_t **out, proven_net_addr_t *bound);
proven_err_t proven_http_event_server_listen_port(proven_http_event_server_t *server, proven_http_event_port_t *port, proven_size_t index);
void proven_http_event_port_close(proven_http_event_port_t *port);

The accepting thread works wherever the library does, and for TCP only: adopt takes a connection, and a QUIC connection is not a socket of its own but datagrams on one shared socket. On Linux a port can instead be opened for several loops. proven_http_event_port_open opens, on the calling thread and all at once, a TCP listener for each loop - and with http3 a UDP socket for each - on one address and port (SO_REUSEPORT). The system spreads TCP connections over the listeners. A UDP datagram goes to the loop that owns its QUIC connection, picked by a small program the port gives the system from the connection ID: so a client whose address changes - a NAT that rebinds it - is still answered by the loop that has its connection.

• Each server takes its share once, with proven_http_event_server_listen_port, where listen would be called: before its loop runs, or on its loop's thread. A server that has listened already is refused, and with http3 it must match the port.
• Then the port is closed: the servers keep what they took.
• The servers are stopped together. A socket that closes while the others go on changes where the system sends datagrams, and connections of other loops can be lost.
• Elsewhere than Linux the port returns PROVEN_ERR_UNSUPPORTED: there it is the accepting thread for TCP, and one loop for each HTTP/3 port.

Two things are visible from outside. Any process of the same user can join a port opened this way. And a QUIC connection ID shows in its first byte which loop issued it - the same few bits for every connection of that loop - which the clients' privacy rules (RFC 9000 section 5.1) would rather it did not.

Measured on one machine of 8 cores (16 threads), this library at both ends over loopback, 32 clients each downloading 128 MiB over HTTP/3: one loop 6.1 to 6.8 Gbit/s at 100 % of a thread; two loops 10.1 to 11.2; four loops 15.3 to 18.2 - where HTTP/2 through the same port was 18.3, so four loops already outrun the clients that share the machine.

The test suite compiles and runs this program:

#include <stdio.h>
#include <string.h>

/*
 * Several loops on one port, with no thread that accepts (Linux): the port opens a listener for
 * each loop on the same address and port, and the system spreads connections over them.
 *
 * The same port with `http3` true opens a UDP socket for each loop too, and sends each QUIC
 * datagram to the loop that owns its connection - by connection ID, so a client whose address
 * changes is still answered by its loop. Elsewhere than Linux the port is not to be had: there,
 * the accepting thread of the previous program is the way, and HTTP/3 is one loop per port.
 */

enum { LOOPS = 2 };

typedef struct {
    int index;
    proven_loop_t *loop;
    proven_http_event_server_t *server;
    int served;                       /* read by the main thread only after the loop has stopped */
} ex_worker_t;

static ex_worker_t g_workers[LOOPS];

static void on_request(void *ctx, proven_http_stream_t *stream, const proven_http_request_t *head) {
    ex_worker_t *me = ctx;
    (void)head;
    me->served++;
    (void)proven_http_stream_respond(stream, 200, NULL, 0, (proven_mem_view_t){ (const proven_byte_t *)"ok", 2 });
}

static void run(void *arg) { (void)proven_loop_run(((ex_worker_t *)arg)->loop); }

int main(void) {
    proven_allocator_t heap = proven_heap_allocator();

    // ---- the port: every socket opened here, at once -------------------------------
    proven_http_event_port_t *port = NULL;
    proven_net_addr_t at;
    proven_err_t err = proven_http_event_port_open(heap, proven_net_addr_loopback(PROVEN_NET_FAMILY_IPV4, 0), LOOPS, false, &port, &at);
    if (err == PROVEN_ERR_UNSUPPORTED) {
        printf("no shared port on this system: one listener and an accepting thread instead\n");
        return EXAMPLE_OK();
    }
    EXAMPLE_REQUIRE(err == PROVEN_OK, "a port for two loops");
    if (err != PROVEN_OK) return EXAMPLE_OK();

    // ---- two loops, each server taking its share ------------------------------------
    /* listen_port is called where listen would be: before the loop runs, or on its thread. */
    proven_job_sys_t *threads = NULL;
    proven_job_group_t loops;
    proven_job_group_init(&loops);
    EXAMPLE_REQUIRE(proven_job_system_init(heap, LOOPS, 8, &threads) == PROVEN_OK, "two threads");
    for (int i = 0; i < LOOPS; ++i) {
        ex_worker_t *w = &g_workers[i];
        w->index = i;
        proven_http_event_server_config_t config = { .on = { .on_request = on_request }, .ctx = w };
        EXAMPLE_REQUIRE(proven_loop_create(heap, &w->loop) == PROVEN_OK &&
                        proven_http_event_server_create(w->loop, &config, &w->server) == PROVEN_OK &&
                        proven_http_event_server_listen_port(w->server, port, (proven_size_t)i) == PROVEN_OK, "a loop with a server on its share");
    }
    /* The servers hold what they took; the port is no longer needed. */
    proven_http_event_port_close(port);
    for (int i = 0; i < LOOPS; ++i)
        EXAMPLE_REQUIRE(proven_job_group_submit(threads, &loops, run, &g_workers[i]) == PROVEN_OK, "each loop on its own thread");

    // ---- sixteen connections --------------------------------------------------------
    proven_http_client_config_t client_config = { .alloc = heap, .max_idle_connections = 0 };
    proven_http_client_t *client = NULL;
    EXAMPLE_REQUIRE(proven_http_client_create(&client_config, &client) == PROVEN_OK, "a client");
    char url[64];
    int n = snprintf(url, sizeof url, "http://127.0.0.1:%u/", (unsigned)at.port);
    int answered = 0;
    for (int i = 0; i < 16; ++i) {
        proven_http_client_response_t response;
        if (proven_http_client_get(client, (proven_u8str_view_t){ (const proven_byte_t *)url, (proven_size_t)n }, &response) != PROVEN_OK) continue;
        if (response.status == 200) answered++;
        proven_http_client_finish(&response);
    }
    proven_http_client_destroy(client);
    EXAMPLE_REQUIRE(answered == 16, "every request is answered");

    // ---- taking it apart: the loops together, then the servers -----------------------
    for (int i = 0; i < LOOPS; ++i) proven_loop_stop(g_workers[i].loop);
    proven_job_group_wait(threads, &loops);
    int served = 0;
    for (int i = 0; i < LOOPS; ++i) {
        printf("loop %d served %d\n", i, g_workers[i].served);    /* the system's choice, not a turn */
        served += g_workers[i].served;
        proven_http_event_server_destroy(g_workers[i].server);
        proven_loop_destroy(g_workers[i].loop);
    }
    EXAMPLE_REQUIRE(served == 16, "sixteen requests were served in all");
    proven_job_system_close(threads);
    proven_job_system_destroy(threads);
    return EXAMPLE_OK();
}

14. What a connection holds, what was measured, and what is not here#

What a connection holds. A connection with no request on it holds its struct and no buffer: requests are read into the loop's scratch buffer, and bytes are copied aside only when a request is incomplete. (A compressor exists only while a compressed response is being written.) On x86-64 Linux the registered tests measure 480 bytes of heap for an idle plain HTTP connection, failing above 512, and 976 bytes for an idle plain WebSocket connection, failing above 1 KiB. A TLS connection adds the engine's state, about 1.25 KiB while idle (Chapter 14, section 9). After a quiet second between requests the server gives back the expanded form of its record keys (wiped) and keeps the keys they were made from; the next request remakes them, which takes about a tenth of a microsecond per direction on a processor with AES instructions. An idle TLS connection quiet for a second measures 1,088 bytes in all, and the test fails at 2 KiB - the target for TLS, where a plain connection's is 1 KiB. An HTTP/2 connection is another size: 2,360 bytes idle after one small request, failing above 3 KiB, and growing with the header fields it is made to remember to about 12.8 KiB when both of its compression tables are full (Chapter 17, section 6). The numbers below are HTTP/1.1's; HTTP/2's, under the same load, are in Chapter 17, section 8.

What was measured. One run of a load program that is not part of the test suite, on one machine: a 16-thread x86-64 desktop processor at 3.5 GHz with 62 GB, Linux 6.12, GCC at -O2. The server and the client are separate processes on that machine, talking over the loopback interface, plain HTTP. Every connection makes one request and is then left open; after that a stated share of them make requests back to back, each waiting for its answer, for ten seconds. The handler answers two bytes. Loopback is not a network - there is no loss and no slow peer here - and a handler that does real work will be slower than one that answers "ok".

One loop:

What the table says: connections that are doing nothing cost a busy one nothing - ten connections are served as fast among fifty thousand as among a thousand - and they cost memory in proportion, about 480 bytes each in the server. One loop does a fixed amount of work per second, so when more connections ask at once each waits longer: the round trip grows with the number asking, not with the number held. No connection was lost in any row. Opening 100,000 took 96 seconds, against 1.7 for 50,000. That appears to be the client's system searching for a free port as its range runs out, not the server: the blocking server took as long or longer for the same step.

The system's own cost for a socket is separate and larger: over the same runs the kernel's slab memory grew by about 7.5 KiB per connection, for both of its ends together.

Several loops, with four client processes holding 12,500 connections each and 1% of them making requests:

The blocking server of Chapter 11, for comparison, in the same program with its handler on its loop's thread: it held the same 100,000 connections and answered slightly faster - 186,000 requests per second at 50,000 held, against 171,000. What it costs is memory: 39 KiB allocated per connection, 10.9 KiB of it resident, against 480 bytes. And it is one thread whose handler waits; the reasons to choose this chapter's server are the ones of section 1, not raw speed for a trivial handler.

On Windows, until v0.80.0, the ceiling was low. There the loop waited with WSAPoll, which looks at every socket on every call. Both ends of the same program on one Windows 11 virtual machine, ten connections making requests:

The client was on the same loop implementation, so each figure carries the cost twice. Asked for 20,000 connections, that machine opened 16,369 before its ports ran out. Since v0.81.0 the loop is built on an I/O completion port there (section 5), which is told only of the operations that finish and does not look at idle sockets. The same program on the same machine, two runs in a row (the two figures per cell; the client on the completion port as well):

Ten connections are served as fast among sixteen thousand as among a thousand: at 10,000 held, about 350 times the requests of WSAPoll and a round trip four hundred times shorter. What the server's allocator holds is 536 bytes per idle connection in every row (the registered test's figure); the process's private memory, with the heap's own bookkeeping, is about 1 KB per connection. The machine is a virtual one on a shared host, so one row in a run can be a quarter slower than the other run (5,000 held); the ports still ran out at about 16,370. The blocking server of Chapter 11, whose selector is on the completion port too, measured 50,000 / 54,000 requests per second at 1,000 held and 53,000 / 52,000 at 10,000, a median of 0.15 ms, with 40 KB of private memory per connection.

What is not here.

• A load test worth the name. The figures above are one machine talking to itself: no network, no slow or hostile clients, no TLS under load, one handler that does nothing.
• Upgrades other than WebSocket on this server; a WebSocket client through a proxy or over HTTP/2.
• In the event-driven client: redirects, authentication, cookies, proxies, and name resolution (section 12 says what to do instead); pipelining, and any order among waiting requests but the order of starting.
• TLS handshakes off the loop's thread. Each costs the loop about 2 ms during which it serves nothing else (Chapter 14, section 9).
• Sending a file without copying it, HTTP/2. (Compressed responses are in Chapter 16, section 6: proven_http_stream_compress in the server - with compress_zstd in its configuration for Zstandard - and decompress - and decompress_zstd for Zstandard, decompress_brotli for Brotli - in the client's.)
• Watching files, signals or child processes with the loop. It watches sockets.

How this was tested. The registered tests drive the loop's timers and posts from one thread and from several, including cancellation from inside the callbacks they concern; drive every operation of section 5 on the system's selector and again on the poll kind - each answered once, never inside the call that started it, through cancellation, close with everything in flight, a send that fails, and three MiB to a slow reader; and run the server over loopback, plain and over TLS, through every row of the tables in sections 7 and 9: answers at once, from a timer and from another thread, uploads with pause and resume, a 16 MB download to a client that does not read until a write has been refused, pipelining, each refusal, each timeout, clients that disappear, and a server destroyed under an unanswered request - checking that on_done was called once for every on_request. The WebSocket is run against the blocking client of Chapter 12 and against raw sockets, through the rows of the tables of section 11 - all but PROVEN_ERR_NOMEM, and PROVEN_ERR_RESET from proven_ws_event_accept and from the send calls, which are not provoked. The client is run against this chapter's server on the same loop and against answers the test scripts itself, through the rows of the tables of section 12 - all but PROVEN_ERR_UNREACHABLE, PROVEN_ERR_NOMEM, an expired certificate, and three of the four time limits: only the response timeout is made to pass. All of them were run several hundred times under address and undefined-behaviour checking with the machine loaded, and under thread checking. Several loops are run as three servers on three threads behind one accepting thread, plain and over TLS, under thread checking. Timers further away than one turn of the loop's wheel (16.4 seconds) are tested with a clock the test moves. Not done: no run against a hostile peer beyond the cases listed, no run of a published WebSocket conformance suite against this server, and no load beyond the single run described above.