35 Storage and handles — pool, shard, budget, wire

35.1 Generational handles#

module blocks .
rem run: main

use pool .

newtype objects u8 .

proc main input al cap allocator . output u8 . effects alloc .
do
  let gm option mut slice u8 be alloc_bytes al capacity 64 .
  let gg option mut slice u8 be alloc_bytes al capacity 32 .
  guard is_some gm . else return 255 .
  guard is_some gg . else return 255 .
  var gens mut slice u64 be view_array u64 (some_value gg) .
  let po option (pool.block_pool objects) be pool.init objects (some_value gm) gens 16 .
  guard is_some po . else return 254 .
  var p pool.block_pool objects be some_value po .
  let ho option (pool.handle objects) be pool.take objects p .
  guard is_some ho . else return 253 .
  let h pool.handle objects be some_value ho .
  let before bool be pool.alive objects p h .
  let first bool be pool.release objects p h .
  let after bool be pool.alive objects p h .
  let again bool be pool.release objects p h .
  var code u8 be 0 .
  if before . do set code (add code 1) . end
  if first . do set code (add code 2) . end
  if after . do set code (add code 4) . end
  if again . do set code (add code 8) . end
  return code .
end

$ lowentc --run main blocks.low
main() = 3

• pool.init objects mem gens 16 divides the bytes mem handed over by the caller into 16-byte blocks and keeps a generation count per block in gens. The pool seals mem and gens inside, so later ops do not take them separately.
• pool.take lends one block and gives a handle. The handle holds the block number and the generation count at that moment.
• pool.release takes the block back and increments that block’s generation count.
• An old handle after release has a different generation, so alive is false, and a second release is rejected too.

The result 3 means “alive before release (1), and the first release succeeded (2)”, and that after release it was neither alive (4) nor did a second release succeed (8). Use-after-free and double free come out as values at run time.

35.2 Brands — no mixing pools#

Pools and handles carry a brand as a translation-time type parameter. One newtype objects u8 . declaration is one store.

module mixed .
rem expect: E-TYPE-INSTANCE

use pool .

newtype meshes u8 .
newtype sounds u8 .

proc wrong input m mut pool.block_pool meshes . input h pool.handle sounds . output bool . effects none .
do
  return pool.release meshes m h .
end

$ lowentc --check mixed.low
mixed.low:11:0 E-TYPE-INSTANCE: two instances of the same generic template are DIFFERENT concrete types — a container opened for one element type cannot be bound to a variable declared for another. Monomorphization makes each instance a distinct struct (visible as `name#arg` in `--ir`), and their layouts differ; binding across them reads the same storage at the wrong element width and yields a wrong value with no error at all. Declare the variable with the SAME instance, or convert explicitly (RFC-0021 · RFC-0084)

Trying to return a handle borrowed from the sounds pool to the meshes pool is rejected, because the two instances are different concrete types. A brand is a type, not a value, so the handle’s fields do not grow. The cost is zero. Opening a store twice with the same brand is rejected too (E-BRAND-REUSED) — then one name would point at two, and the confusion the brand was meant to prevent would return.

35.3 Non-overlapping pieces — shard#

shard divides one block of storage into non-overlapping pieces and gives a token per piece. A token is the right “this range is mine” and is an owned value.

module split_tokens .
rem expect: E-OWN-MOVED

use shard .

newtype grid u8 .

fn halves_then_root output u64 .
do
  var r owned shard.token grid be shard.open grid 8 .
  var h shard.halves grid be shard.split_at grid r 4 .
  return shard.width grid r .
end

$ lowentc --check split.low
split.low:12:0 E-OWN-MOVED: this `owned` value was already MOVED (consumed) — using it again is use-after-move, which SPEC-004 §4.8 has always called a compile error and which nothing enforced. To keep using it, either CONSUME AND PUT IT BACK (`set <name> <new value>` re-initialises the place — that is how a handle threads through a loop), or borrow it LOCALLY with `ref h`. ☞ borrowing across an OP BOUNDARY is not lowered yet (E-IR-UNSUP says so at the call site), so `f (ref h)` is not a way out today

shard.split_at grid r 4 consumes the whole token r and splits it in two. Asking the width with r afterwards is E-OWN-MOVED. What stops this mistake is not the shard module but the language. The token is owned, so it leaves your hands the moment it is handed over. The idea of an access unit is not a new rule but the consequence of an existing one. Whether the pieces ride several flows or run sequentially, the fact that they do not overlap is itself the value.

The same shape is in pagecache. Pinning a page (acquire) consumes the cache token, so while the pin lives there is no token in hand to pass to eviction (evict). The defect of evicting a pinned page is stopped at translation.

35.4 Bit budgets — budget#

To put a generational handle in one word you must decide “how many bits go to slot number, shard number and generation count”. budget writes those numbers in the source and has the tool keep them.

module budget_bad .
rem expect: E-CONTRACT-IMPOSSIBLE

use budget .

fn too_wide output u64 .
do
  let ho option u64 be budget.pack 40 16 16 7 1 0 .
  guard is_some ho . else return 0 .
  return some_value ho .
end

$ lowentc --check budget_bad.low
budget_bad.low:8:0 E-CONTRACT-IMPOSSIBLE: this call breaks the callee's `requires`, and BOTH SIDES ARE CONSTANTS — so it can be decided here, now. It used to compile green and trap at run time (E-VM-CONTRACT): a bit budget that does not fit (`slot + shard + generation > word`) shipped as a runnable program. A contract that can be decided at compile time IS decided at compile time (RFC-0104 §8-3)

budget.pack 40 16 16 … sums to 72 bits, which does not fit in a 64-bit word. All arguments are constants, so the contract violation is decided at translation (chapter 14). As the diagnostic’s story says, a program with a bit budget that did not fit was once shipped in runnable form. This module is the first place to use the check born of that. When a generation count fills up, the slot retires (retired). An old handle never comes back to life by secretly wrapping to 0.

35.5 Bit fields — wire#

module fields .
rem run: mode_of 1234
rem run: with_mode 1234 5

use wire .

fn mode_of input w u64 . output u64 .
do
  return wire.pick 3840 w .
end

fn with_mode input w u64 . input m u64 .
  output u64 .
  requires le m 15 .
do
  return wire.merge 3840 w m .
end

$ lowentc --run mode_of fields.low 1234
mode_of(1234) = 4
$ lowentc --run with_mode fields.low 1234 5
with_mode(1234, 5) = 1490

When several values sit side by side in one integer, wire handles a field with a single mask. The mask 0xF00 (3840) states both the position and width, “4 bits starting at bit 8”. pick extracts the field’s value, and merge swaps just that field. 1234 is 0x4D2, so the field’s value is 4, and swapping in 5 gives 0x5D2 (1490). The two numbers “position” and “width”, which always diverged when written by hand, come from one mask and so cannot diverge. Packing named on/off settings into one word is the job of flags.

35.6 Other storage modules#

35.7 Common mistakes#

module mistake_staleuse .
rem trap: main

use pool .

newtype objects u8 .

proc main input al cap allocator . output u8 . effects alloc .
do
  let gm option mut slice u8 be alloc_bytes al capacity 64 .
  let gg option mut slice u8 be alloc_bytes al capacity 32 .
  guard is_some gm . else return 255 .
  guard is_some gg . else return 255 .
  var gens mut slice u64 be view_array u64 (some_value gg) .
  let po option (pool.block_pool objects) be pool.init objects (some_value gm) gens 16 .
  guard is_some po . else return 254 .
  var p pool.block_pool objects be some_value po .
  let ho option (pool.handle objects) be pool.take objects p .
  guard is_some ho . else return 253 .
  let h pool.handle objects be some_value ho .
  let done bool be pool.release objects p h .
  rem ✘ takes out the bytes of a returned block without asking --- the generation differs, so it is `none`
  var b mut slice u8 be some_value (pool.bytes objects p h) .
  set (index b 0) 7 .
  return index b 0 .
end

$ lowentc --run main mistake_staleuse.low
== ir diagnostics (1) ==
0:0 E-VM-NONE: some_value of none (panic)

module mistake_brandreuse .
rem expect: E-BRAND-REUSED

use pool .

newtype objects u8 .

proc two_pools input al cap allocator . output u8 . effects alloc .
do
  let g1 option mut slice u8 be alloc_bytes al capacity 64 .
  let g2 option mut slice u8 be alloc_bytes al capacity 64 .
  let e1 option mut slice u8 be alloc_bytes al capacity 32 .
  let e2 option mut slice u8 be alloc_bytes al capacity 32 .
  guard is_some g1 . else return 255 .
  guard is_some g2 . else return 255 .
  guard is_some e1 . else return 255 .
  guard is_some e2 . else return 255 .
  var gens1 mut slice u64 be view_array u64 (some_value e1) .
  var gens2 mut slice u64 be view_array u64 (some_value e2) .
  let p1 option (pool.block_pool objects) be pool.init objects (some_value g1) gens1 16 .
  rem ✘ opens a second store under the same brand `objects` --- the types could no longer stop handles of the two pools mixing
  let p2 option (pool.block_pool objects) be pool.init objects (some_value g2) gens2 16 .
  return 0 .
end

$ lowentc --check mistake_brandreuse.low
22:0 E-BRAND-REUSED: this brand already opened a storage somewhere else — a brand names ONE storage instance, and a second creation site makes it name two. Then two different storages share a type, and handles from one type-check against the other: the very confusion the brand exists to refuse. Declare a second `newtype` and use it here (a brand carries no data, so an extra one costs nothing at run time) (RFC-0104 §5.11 · §8-2)

module wire_truncates .
rem run: put_mode 5
rem run: put_mode 20
rem run: fits_mode 20

use wire .

rem the field is four bits (0 … 15) --- 20 is cut down and 4 goes in
fn put_mode input m u64 . output u64 .
do
  return wire.put 3840 m .
end

rem if truncation is unwanted, ask first
fn fits_mode input m u64 . output bool .
do
  return wire.fits 3840 m .
end

$ lowentc --run put_mode wire_truncates.low 5
put_mode(5) = 1280
$ lowentc --run put_mode wire_truncates.low 20
put_mode(20) = 1024
$ lowentc --run fits_mode wire_truncates.low 20
fits_mode(20) = 0

35.8 This chapter’s syntax at a glance#