16 Standard modules

(1) The standard modules come with the processor. They are opened with use MODULE and written MODULE.WORD (13.1). Words of standard modules cannot reach outside without the host — they use no files, clocks or environment.

(2) Words of standard modules make no new storage. Their results are one of:

a view of the bytes they were given; nothing is copied;
a view of new bytes written in the region (12.2);
bytes appended to a buffer the caller gave. Such words end in _into or take a buffer, and fail with err.full when there is not enough room.

16.1 `str` — strings

(1) The module str works on strings as bytes. Byte positions count from 0. Only the words that work on characters (chars char_count is_utf8) read UTF-8. Changing case (upper lower) changes ASCII only.

(2) Where a separator or a part to look for is taken (sep, part), a str or a single character (int, 6.5) may be given.

16.1.1 Looking and cutting

(3) All results are views of the string given.

Word	Signature	Meaning
`len` `is_empty`	`(str gives int)` `(str gives bool)`	the number of bytes; is it empty
`take` `skip`	`(s str, n int gives str)`	the first `n` bytes; what follows them; when short, what there is
`take_last` `skip_last`	`(s str, n int gives str)`	the last `n` bytes; what comes before them
`slice`	`(s str, start int, n int gives str)`	`n` bytes from `start`; out of range is the fault `range`
`trim` `trim_left` `trim_right`	`(str gives str)`	the view without white space (space, tab, CR, LF) at the ends
`cut`	`(s str, sep gives str, str)`	before and after the first `sep`; without `sep`, `s` and the empty text
`try_cut`	`(s str, sep gives str, str or fail)`	like `cut`, but `err.not_found` without `sep`
`span`	`(s str, class int gives str, str)`	the leading bytes of that class, and the rest

(4) The classes of span are the constants str.digit str.alpha str.alnum str.space str.upper_case str.lower_case str.hex_digit str.punct (ASCII).

16-str-views.pnx

use io
use str
word show (label str, s str) cat (label, " [", s, "]") io.println end
word demo ()
  "  key = value  " to line
  "trim" line str.trim show
  "trim_left" line str.trim_left show
  "take 5" "hello, world" 5 str.take show
  "skip 7" "hello, world" 7 str.skip show
  "take_last 5" "hello, world" 5 str.take_last show
  "skip_last 7" "hello, world" 7 str.skip_last show
  "take 99" "short" 99 str.take show                 rem n beyond the end: what is there
  "slice 7 3" "hello, world" 7 3 str.slice show
  "key = value" " = " str.cut to (k, v)
  "cut, before" k show
  "cut, after" v show
  "a=b=c" '=' str.cut to (k2, v2)                    rem the separator may be a character
  "cut at a char" v2 show
  "2026-10-15" str.digit str.span to (year, rest)
  "span digits" year show
  "span rest" rest show
end
demo

Output

$ pnx 16-str-views.pnx
trim [key = value]
trim_left [key = value  ]
take 5 [hello]
skip 7 [world]
take_last 5 [world]
skip_last 7 [hello]
take 99 [short]
slice 7 3 [wor]
cut, before [key]
cut, after [value]
cut at a char [b=c]
span digits [2026]
span rest [-10-15]

16.1.2 Searching and comparing

(5)

Word	Signature	Meaning
`starts_with` `ends_with` `contains`	`(s str, part gives bool)`
`find` `find_last`	`(s str, part gives int or fail)`	the first, last position; `err.not_found` when absent
`compare`	`(str, str gives int)`	negative, zero or positive, comparing in byte order
`byte_at`	`(s str, i int gives u8)`	byte `i`; out of range is the fault `range`
`char_count`	`(str gives int)`	the number of UTF-8 characters
`is_utf8`	`(str gives bool)`	is it valid UTF-8

The comparison words eq ne lt and the rest also compare strings in byte order (15.2).

16-str-search.pnx

use io
use str
word yes (b bool gives str) b if "yes" else "no" end end
"hello.pnx" ".pnx" str.ends_with yes io.println
"hello.pnx" "he" str.starts_with yes io.println
"hello.pnx" "lo." str.contains yes io.println
"a,b,c,b" ",b" str.find or_else (e) -1 end str.to_str io.println
"a,b,c,b" ",b" str.find_last or_else (e) -1 end str.to_str io.println
"a,b,c,b" "x" str.find or_else (e) -1 end str.to_str io.println
"abc" "abd" str.compare str.to_str io.println
"hello" 1 str.byte_at str.to_str io.println
"hello" str.len str.to_str io.println
"" str.is_empty yes io.println

Output

$ pnx 16-str-search.pnx
yes
yes
yes
1
5
-1
-1
101
5
yes

16.1.3 Making new text

(6) The results are new bytes written in the region; when the region is used up it is the fault full (12.2). The words that read numbers can fail.

Word	Signature	Meaning
`upper` `lower`	`(str gives str)`	ASCII case changed
`repeat`	`(s str, n int gives str)`	`n` copies joined; a negative `n` is the fault `range`
`replace`	`(s str, old str, new str gives str)`	every `old` replaced by `new`
`pad_left` `pad_right`	`(s str, width int gives str)`	filled with spaces to `width` bytes; as it is when longer; a `width` below 0 or from 2³² up is the fault `range`
`to_str`	`(int gives str)`	decimal text; a `dec(N)` with its `N` digits after the point (7.5)
`to_dec`	`(s str, N gives dec(N) or fail)`	read a decimal: a sign, digits, and a point with at most `N` digits after it (`N` a constant); more digits, or anything else, `err.format`
`to_int`	`(str gives int or fail)`	read as decimal: one sign and digits only (no spaces); otherwise, or beyond 64 bits, `err.format`
`to_int_base`	`(s str, base int gives int or fail)`	read in base 2–36; another `base` is the fault `range`

16-str-make.pnx

use io
use str
"Hello, World" str.upper io.println
"Hello, World" str.lower io.println
"ab" 3 str.repeat io.println
"one two one" "one" "1" str.replace io.println
cat ("[", "7" 4 str.pad_left, "]") io.println
cat ("[", "7" 4 str.pad_right, "]") io.println
-42 str.to_str io.println
"ff" 16 str.to_int_base or_else (e) -1 end str.to_str io.println
"  12" str.to_int or_else (e) -1 end str.to_str io.println
"12x" str.to_int or_else (e) -1 end str.to_str io.println
"한글" str.len str.to_str io.println
"한글" str.char_count str.to_str io.println
"\xff" str.is_utf8 if "utf-8" else "not utf-8" end io.println

Output

$ pnx 16-str-make.pnx
HELLO, WORLD
hello, world
ababab
1 two 1
[   7]
[7   ]
-42
255
-1
-1
6
2
not utf-8

16.1.4 Stems and writing into buffers

(7) Words that make stems (10.6):

s sep str.split — the pieces between seps, empty ones included; sep is a str or one byte (a u8 constant, usually a character literal), as for cut and try_cut;
s str.words — the pieces between white space, no empty ones;
s str.lines — the lines, without their LF or CR LF;
s str.bytes — the bytes (u8);
s str.chars — one view (str) for each UTF-8 character.

(8) Words that write into a buffer:

sep buf STEM str.join_into — appends the pieces of the stem joined by sep (or fail). The stem goes last, since the word right after a stem shall take it;
s old new buf str.replace_into — appends the text with replacements and gives their number (gives n int or fail).

16-str-stems.pnx

use io
use str
"one two  three" str.words do (w str) cat ("<", w, ">") io.print end each
"" io.println
"a,b,,c" ',' str.split do (w str) cat ("<", w, ">") io.print end each
"" io.println
text END
first line
second line
END
str.lines do (l str) cat ("line: ", l) io.println end each
"hi" str.bytes do (b u8) b str.to_str io.print " " io.print end each
"" io.println
"가a!" str.chars do (c str) cat ("<", c, " ", c str.len, ">") io.print end each
"" io.println
buffer out 64
", " out "x y z" str.words str.join_into or_else (e) end
out contents io.println
out clear
word swap_dashes () "a-b-c" "-" "+" out str.replace_into or_else (e) 0 end to n  cat (out contents, " (", n, " replaced)") io.println end
swap_dashes

Output

$ pnx 16-str-stems.pnx
<one><two><three>
<a><b><><c>
line: first line
line: second line
104 105 
<가 3><a 1><! 1>
x, y, z
a+b+c (2 replaced)

Constraints

(9) A value that is not a str given to a word of str is refused.

16-str-type.pnx

use str
42 str.upper drop

Refused

$ pnx 16-str-type.pnx
16-str-type.pnx:2:4: this word takes a str

Faults

(10) An out-of-range position for slice or byte_at is a fault. For cuts that may run short, use take and skip.

16-str-slice-range.pnx

use io
use str
"hello" 3 5 str.slice io.println

Fault

$ pnx 16-str-slice-range.pnx
16-str-slice-range.pnx:3: index out of range

16-str-byte-range.pnx

use str
"hello" 5 str.byte_at drop

Fault

$ pnx 16-str-byte-range.pnx
16-str-byte-range.pnx:2: index out of range

16.2 `fmt` — numbers as text

(1) cat puts an int in decimal (9.8). Other forms are made into a str by the words of the module fmt. There is no small language of formats (such as %08x). Results are written in the region.

Word	Signature	Meaning
`hex` `bin`	`(n gives str)`	hexadecimal (lower case), binary; `n` an `int` or a narrow integer (7.6): a negative `int` as 64-bit two’s complement, a narrow integer as the bits of its width
`hex_pad`	`(n, width int gives str)`	hexadecimal filled with leading zeros to `width` digits; `n` as for `hex`
`zero_pad`	`(n, width int gives str)`	decimal filled with leading zeros to `width`; the sign takes a place; `n` an `int` or a narrow integer (a `u64` unsigned)
`append_hex`	`(n, buf or fail)`	appends the hexadecimal to a buffer; `n` as for `hex`

Decimal is str.to_str; decimal appended to a buffer is append_int (15.3). A width of hex_pad or zero_pad below 0 or from 2³² up is the fault range; when the region is used up it is the fault full.

16-fmt.pnx

use io
use fmt
255 fmt.hex io.println
5 fmt.bin io.println
255 8 fmt.hex_pad io.println
7 3 fmt.zero_pad io.println
-7 3 fmt.zero_pad io.println
buffer out 32
"0x" out append or_else (e) end
48879 out fmt.append_hex or_else (e) end
out contents io.println

Output

$ pnx 16-fmt.pnx
ff
101
000000ff
007
-07
0xbeef

Constraints

(2) The words of fmt take an int. A value of another type is refused. A dec(N) is written by str.to_str and cat (7.5).

16-fmt-type.pnx

use fmt
use io
"abc" fmt.hex io.println

Refused

$ pnx 16-fmt-type.pnx
16-fmt-type.pnx:3:7: this word takes an int or a narrow integer

Faults

(3) A width of hex_pad or zero_pad below 0 or from 2³² up is the fault range.

16-fmt-width.pnx

use fmt
use io
255 -1 fmt.hex_pad io.println

Fault

$ pnx 16-fmt-width.pnx
16-fmt-width.pnx:3: index out of range

(4) A result that does not fit in the region is the fault full. The width below is allowed, but larger than the region.

16-fmt-full.pnx

use fmt
use str
use io
7 4294967295 fmt.zero_pad str.len str.to_str io.println

Fault

$ pnx 16-fmt-full.pnx
16-fmt-full.pnx:4: a declared capacity is used up

Diagnostics

Condition	Message
a value that is not an `int`	`this word takes an int`

16.3 `hash` — hashes

(1) The module hash gives hashes of bytes as an int. There are no cryptographic hashes; a host gives those as host words.

Word	Signature	Meaning
`crc32`	`(str gives u32)`	CRC-32 (IEEE 802.3, as zlib)
`fnv1a`	`(str gives u64)`	64-bit FNV-1a
`hash64`	`(str gives u64)`	SipHash-2-4 with the key 0

16-hash.pnx

use io
use hash
use fmt
"hello" hash.crc32 fmt.hex io.println
"hello" hash.fnv1a fmt.hex io.println
"hello" hash.hash64 fmt.hex io.println

Output

$ pnx 16-hash.pnx
3610a686
a430d84680aabd0b
8cc15d5db2f752b9

(1a) Check values: the crc32 of the empty string is 0, and that of "123456789" is cbf43926 (the standard check value of CRC-32). The fnv1a of the empty string is FNV’s offset basis cbf29ce484222325. Another processor shall give the same values.

16-hash-vectors.pnx

use io
use str
use fmt
use hash
"" hash.crc32 str.to_str io.println
"123456789" hash.crc32 fmt.hex io.println
"" hash.fnv1a fmt.hex io.println
"" hash.hash64 fmt.hex io.println

Output

$ pnx 16-hash-vectors.pnx
0
cbf43926
cbf29ce484222325
1e924b9d737700d7

Constraints

(2) The hash of a value that is not a str is refused. To hash a number, make bytes of it first (16.5).

16-hash-type.pnx

use hash
42 hash.crc32 drop

Refused

$ pnx 16-hash-type.pnx
16-hash-type.pnx:2:4: this word takes a str

16.4 `encode` — bytes as text

(1) The words of the module encode are all (s str, buf or fail). They append the result to the buffer and fail with err.full when there is not enough room, or err.format when the input has the wrong form.

Word	Meaning
`base64_encode` `base64_decode`	Base64 of RFC 4648; when decoding, the closing `=` may be left out
`hex_encode` `hex_decode`	two hexadecimal digits per byte; lower case when writing, either case when reading
`url_encode` `url_decode`	percent-encoding of RFC 3986: bytes other than letters, digits and `-` `.` `_` `~` as `%XX`

16-encode.pnx

use io
use encode
buffer out 64
word show (label str) cat (label, " ", out contents) io.println  out clear end
word demo (or fail)
  "Pinax!" out encode.base64_encode or_fail  "base64" show
  "UGluYXgh" out encode.base64_decode or_fail  "decoded" show
  "\x00\xff" out encode.hex_encode or_fail  "hex" show
  "a b&c=d/é" out encode.url_encode or_fail  "url" show
  "a%20b" out encode.url_decode or_fail  "url decoded" show
end
demo or_else (e) cat ("failed: ", e err_text) io.println end

Output

$ pnx 16-encode.pnx
base64 UGluYXgh
decoded Pinax!
hex 00ff
url a%20b%26c%3Dd%2F%C3%A9
url decoded a b

16-encode-bad.pnx

use io
use encode
buffer out 16
"zz" out encode.hex_decode or_else (e) cat ("hex_decode: ", e err_text) io.println end
"ab*c" out encode.base64_decode or_else (e) cat ("base64_decode: ", e err_text) io.println end

Output

$ pnx 16-encode-bad.pnx
hex_decode: format
base64_decode: format

Constraints

(2) A value that is not a str and a buffer is refused. To encode a number, make text or bytes of it first (16.1, 16.5).

16-encode-type.pnx

use encode
buffer out 8
5 out encode.hex_encode or_else (e) end

Refused

$ pnx 16-encode-type.pnx
16-encode-type.pnx:3:7: this word takes a str and a buffer, as: s buf encode.base64_encode

Diagnostics

Condition	Message
a value that is not a `str` and a buffer	`this word takes a str and a buffer, as: s buf encode.base64_encode`

16.5 `bytes` — binary data

(1) The module bytes writes numbers and text into a buffer as fixed byte forms, and reads them from bytes. The writing words are (value, buf or fail) (err.full when there is not enough room; nothing is written). The reading words are (s str gives value, rest str or fail) (err.format when the bytes run short or have the wrong form).

(2)

Word	Bytes
`put_u8` `put_u16` `put_u32` `put_u64` `put_i8` `put_i16` `put_i32` `put_i64`	a value of the type the name says (7.6; `put_i64` an `int`), least significant byte first
the same names with `_be`	most significant byte first
`get_u8` … `get_i64`, `get_u16_be` …	their pairs: a value of the type the name says (`get_i64` an `int`)
`put_varint` `get_varint`	LEB128: the value as an unsigned 64-bit number, 7 bits at a time (a negative takes 10 bytes)
`put_str` `get_str`	a length (varint), then the bytes

16-bytes.pnx

use io
use bytes
use encode
use str
buffer out 64
buffer hex 128
word dump () out contents hex encode.hex_encode or_else (e) end  hex contents io.println  hex clear end
word demo (or fail)
  258 u16 out bytes.put_u16 or_fail            rem little end first: 02 01
  258 u16 out bytes.put_u16_be or_fail         rem big end first: 01 02
  300 out bytes.put_varint or_fail         rem LEB128: ac 02
  "hi" out bytes.put_str or_fail           rem a varint length, then the bytes
  dump
  out contents to s
  s bytes.get_u16 or_fail to (a, s1)       rem the value and the rest
  s1 bytes.get_u16_be or_fail to (b, s2)
  s2 bytes.get_varint or_fail to (c, s3)
  s3 bytes.get_str or_fail to (d, s4)
  cat (a, " ", b, " ", c, " ", d, ", left ", s4 str.len) io.println
end
demo or_else (e) e err_text io.println end

Output

$ pnx 16-bytes.pnx
02010102ac02026869
258 258 300 hi, left 0

16.5.1 Records

(3) row buf bytes.pack appends one row of a record (8.8) to a buffer, field by field in order (or fail). s row bytes.unpack reads the fields from bytes into the row and gives the rest (gives str or fail). row bytes.packed_size is the number of bytes pack would write.

(4) The bytes of a field: an integer width as that width (most significant byte first when the record has big_endian, least significant first otherwise), int as 8 bytes, bool as 1 byte, varint as LEB128, str(N) as a length (varint) and the bytes, a field holding a row as the number it holds (varint).

(5) When pack fails the buffer goes back to the length it had before the call. unpack writes into the row only when every field can be read; when it fails the row is unchanged.

16-bytes-pack.pnx

use io
use bytes
use encode
use str
record item
  id u16
  name str(8)
  delta varint
end
record wire
  big_endian
  id u16
  count u32
end
array items item 2
array wires wire 1
buffer out 64
buffer hex 128
word dump () out contents hex encode.hex_encode or_else (e) end  hex contents io.println  hex clear end
word demo (or fail)
  0 items idx to a
  513 u16 a to item.id  "pinax" a to item.name  -3 a to item.delta
  a out bytes.pack or_fail
  dump
  cat ("packed_size ", a bytes.packed_size, ", written ", out contents str.len) io.println
  1 items idx to b
  out contents b bytes.unpack or_fail to rest
  cat (b item.id, " ", b item.name, " ", b item.delta, ", rest ", rest str.len) io.println
  "\x01\x02\x05" b bytes.unpack or_else (e) cat ("short input: ", e err_text) io.println "" end drop
  cat ("the row is unchanged: ", b item.name) io.println
  out clear
  0 wires idx to w
  258 u16 w to wire.id  7 u32 w to wire.count
  w out bytes.pack or_fail
  dump
end
demo or_else (e) e err_text io.println end

Output

$ pnx 16-bytes-pack.pnx
01020570696e6178fdffffffffffffffff01
packed_size 18, written 18
513 pinax -3, rest 0
short input: format
the row is unchanged: pinax
010200000007

Constraints

(6) pack, unpack and packed_size take only rows of records.

16-bytes-pack-int.pnx

use bytes
buffer b 8
5 b bytes.pack or_else (e) end

Refused

$ pnx 16-bytes-pack-int.pnx
16-bytes-pack-int.pnx:3:5: pack takes a row of a table of records and a buffer: row buf bytes.pack

Faults

(7) The value written has the type the word’s name says (u8 for put_u8, 7.6). An int is made into that type first (n u8), and a value outside its range is the fault overflow there; nothing is cut.

16-bytes-narrow.pnx

use bytes
buffer b 8
word store (n int) n u8 b bytes.put_u8 or_else (e) end end
300 store

Fault

$ pnx 16-bytes-narrow.pnx
16-bytes-narrow.pnx:3: integer overflow
  n int = 300

16.6 `ptn` — small patterns

(1) The module ptn finds and replaces text with patterns of the size of Lua’s. It has no alternation (|), no repetition of groups and no back references; rex (16.7) takes the larger grammar.

16.6.1 The text of a pattern

(2) A pattern is a sequence of the items below, matched byte by byte.

Item	Matches
an ordinary character	that byte
`.`	any byte
`%a` `%d` `%l` `%u` `%s` `%w` `%x` `%p` `%c`	letters, digits, lower case, upper case, white space, letters or digits, hexadecimal digits, punctuation, control characters (ASCII); in upper case, everything else
`%` and a character that is not a letter or digit	that byte itself (`%.` `%%` `%(`)
`[…]` `[^…]`	a set and its complement, with `a-z` ranges and `%` classes inside; a `]` right after `[` or `[^` is a member
`*` `+` `-` `?` after an item	zero or more (longest), one or more (longest), zero or more (shortest), zero or one
`( … )`	a capture; up to nine
`^` first, `$` last	anchored at the start, at the end of the text

16-ptn-classes.pnx

use io
use ptn
"a1 B2_c3!" "%a" "." ptn.replace io.println     rem letters
"a1 B2_c3!" "%d" "." ptn.replace io.println     rem digits
"a1 B2_c3!" "%W" "." ptn.replace io.println     rem not letter or digit
"a1 B2_c3!" "[%u_]" "." ptn.replace io.println  rem a set: upper case or _
"aaab" "a-b" "<%0>" ptn.replace io.println      rem - is the shortest repetition
"color colour" "colou?r" "#" ptn.replace io.println

Output

$ pnx 16-ptn-classes.pnx
.1 .2_.3!
a. B._c.!
a1.B2.c3.
a1 .2.c3!
<aaab>
# #

16.6.2 Words

(3)

Word	Signature	Meaning
`is_match`	`(s str, pat gives bool)`	is there a match anywhere
`match`	`(s str, pat gives str or fail)`	a view of the first match; `err.not_found` when none
`capture`	`(n int gives str)`	capture `n` (1–9) of the last `match`; 0 is the whole match
`matches`	stem	the matches in order, each search starting where the last ended
`replace`	`(s str, pat, rep str gives str)`	new text with every match replaced by `rep`; in `rep`, `%0`–`%9` are captures and `%%` is `%`

(4) A pattern that begins with ^ is tried at the start only, so matches and replace match at most once.

(5) The last match capture reads belongs to each run (task) separately; another match changes it. A run the host starts begins with no match: the match of an earlier run is not seen, since its captures are views of text that run matched, which may be gone.

16-ptn.pnx

use io
use ptn
const date "(%d+)-(%d+)-(%d+)"
word yes (b bool gives str) b if "true" else "false" end end
"abc 123" "%d+" ptn.is_match yes io.println
word demo ()
  "on 2026-10-15 ok" date ptn.match or_else (e) "?" end io.println    rem the whole match
  cat (1 ptn.capture, "/", 2 ptn.capture, "/", 3 ptn.capture) io.println
  "key = value" "^(%w+)%s*=%s*(%w+)$" ptn.match or_else (e) "?" end drop
  cat (2 ptn.capture, " <- ", 1 ptn.capture) io.println
end
demo
"x=1, y=22, z=333" "%a=%d+" ptn.matches do (m str) cat ("[", m, "]") io.print end each
"" io.println
"2026-10-15" date "%3.%2.%1 (%0)" ptn.replace io.println
"  trim me  " "^%s*(.-)%s*$" "%1" ptn.replace io.println
"aaa" "^a" "-" ptn.replace io.println              rem ^ anchors: one match, at the start

Output

$ pnx 16-ptn.pnx
true
2026-10-15
2026/10/15
value <- key
[x=1][y=22][z=333]
15.10.2026 (2026-10-15)
trim me
-aa

Constraints

(6) A pattern shall be a string literal or a const of one, so that it is checked at load. To search with text made while running, use rex.compile (16.7).

16-ptn-variable.pnx

use ptn
word f (s str, p str gives bool) s p ptn.is_match end

Refused

$ pnx 16-ptn-variable.pnx
16-ptn-variable.pnx:2:38: a pattern is a string literal or a const name, so that it is checked at load

(7) A malformed pattern (an unclosed set or capture, a final %, a ) that closes nothing, ten or more captures) is refused; so are %b, %f, the back reference %1 and the position capture ().

16-ptn-bad.pnx

use ptn
word f (s str gives bool) s "[a-z" ptn.is_match end

Refused

$ pnx 16-ptn-bad.pnx
16-ptn-bad.pnx:2:36: a set is not closed by ]

16-ptn-balance.pnx

use ptn
word f (s str gives bool) s "%b()" ptn.is_match end

Refused

$ pnx 16-ptn-balance.pnx
16-ptn-balance.pnx:2:36: %b, %f and back references are not supported

Faults

(8) A capture whose n the last match did not make, or with no match yet, is the fault range.

16-ptn-capture-none.pnx

use ptn
use io
word f () "ab" "(a)b" ptn.match or_else (e) "" end drop  2 ptn.capture io.println end
f

Fault

$ pnx 16-ptn-capture-none.pnx
16-ptn-capture-none.pnx:3: index out of range

(9) Matching goes back and tries again. A word that takes more than 2²⁴ steps in one call stops with the fault depth. Rewrite such a pattern with fewer shortest repetitions (-), or use rex, which does not go back.

16-ptn-steps.pnx

use ptn
use str
"a" 300 str.repeat ".-.-.-.-.-.-.-.-x" ptn.is_match drop

Fault

$ pnx 16-ptn-steps.pnx
16-ptn-steps.pnx:3: call depth limit

Diagnostics

Condition	Message
A pattern that is not a constant	`a pattern is a string literal or a const name, so that it is checked at load`
An unclosed set	`a set is not closed by ]`
`%b`, `%f`, back references	`%b, %f and back references are not supported`

16.7 `rex` — regular expressions

(1) The module rex finds and replaces with regular expressions. Its matching never goes back, so the time it takes is bounded by the length of the text times the size of the pattern. Write patterns as raw strings (6.5): in an ordinary string "\d" is refused as an unknown escape, and "\b" becomes a backspace.

(2) In a source that has use rex the type name regex is visible. A regex value is a pattern program that cannot be changed once made. It is used only in locals, arguments and results.

16.7.1 The text of a pattern

(3)

Form	Meaning
a character	its bytes; a multi-byte character, its byte sequence
`.`	one UTF-8 character; without `(?s)` not a line break
`[abc]` `[a-z]` `[^…]`	a set; ranges are ASCII; a complemented set takes one UTF-8 character
`\d` `\w` `\s` `\D` `\W` `\S`	digits, word characters, white space and their complements (ASCII)
`*` `+` `?` `{n}` `{n,}` `{n,m}`	repetition (longest); with `?` after it, shortest
`a\|b`	alternation; the left first
`( … )` `(?: … )`	a group that captures, one that does not
`^` `$`	the start and end of the text; with `(?m)`, of each line
`\b` `\B`	a word boundary and its complement (ASCII)
`\n` `\t` `\r` `\f` `\v` `\0` `\xNN`	that byte
`\` and any other character	that character (`\.` `\*` `\\`)
`\zs` `\ze`	the match is narrowed to start, end here; the whole pattern must still match
`(?i)` `(?m)` `(?s)`	at the very start of a pattern only: ignore case (ASCII), anchors per line, `.` also matches a line break

There are no back references, lookaround, Unicode properties or named groups.

(4) A pattern that repeats a body able to match the empty text (such as (a*)*) may give other captures than Perl or Python.

16.7.2 Words

(5) re is a regex value. “The match” is the range narrowed by \zs and \ze.

Word	Signature	Meaning
`rx`	`(pat gives regex)`	makes the program at load; `pat` is a literal or a `const`
`compile`	`(pat str gives regex or fail)`	makes it while running; a malformed pattern is `err.format`; the program lies in the region
`is_match`	`(s str, re gives bool)`	is there a match
`match`	`(s str, re gives str or fail)`	the first match; `err.not_found` when none
`cut_match`	`(s str, re gives str, str, str or fail)`	what is before the first match, the match, and what is after
`matches`	stem	the matches in order; the block takes the whole match and one `str` parameter per group
`split`	stem	the pieces between the matches
`with_match`	`(s str, re, block or fail)`	runs the block once with the first match; `err.not_found` when none
`replace`	`(s str, re, rep str gives str)`	new text with every match replaced; in `rep`, `\0`–`\9` are groups
`replace_into`	`(s str, re, rep str, buf gives int or fail)`	appends the replaced text to a buffer and gives the number of replacements
`replace_by`	`(s str, re, buf, block gives int or fail)`	appends to a buffer, each match replaced by the text the block gives

(6) When matches, split and replace search again from the next place, they still see the whole text, so ^ and \b know the bytes before that place.

16-rex.pnx

use io
use str
use rex
const pair r"(\w+)=(\d+)"
word yes (b bool gives str) b if "true" else "false" end end
word total (line str gives n int)               rem groups arrive as block parameters
  0 to n
  line pair rex.rx rex.matches do (whole str, key str, value str)
    n value str.to_int or_else 0 end add to n
  end each
  n
end
word demo ()
  cat ("sum ", "id=7, x=20; name=kim, n=3" total) io.println
  "abc 123" r"\d+" rex.rx rex.is_match yes io.println
  "on 2026-10-15 ok" r"(\d+)-(\d+)" rex.rx rex.match or_else (e) "?" end io.println
  "key: value" r":\s*" rex.rx rex.cut_match or_else (e) "" "" "" end to (before, m, after)
  cat ("[", before, "|", m, "|", after, "]") io.println
  "from 2026-10-13 to 2026-11-02" r"(\d{4})-(\d{2})-(\d{2})" rex.rx r"\3/\2/\1" rex.replace io.println
  "a , b;c ,d" r"\s*[,;]\s*" rex.rx rex.split do (part str) cat ("<", part, ">") io.print end each
  "" io.println
  "가나다" r"." rex.rx rex.matches do (ch str) cat ("<", ch, ">") io.print end each   rem . is one UTF-8 character
  "" io.println
  "foobar" r"foo\zsbar" rex.rx rex.match or_else (e) "?" end io.println              rem \zs: the match starts here
  "Hello" r"(?i)HELLO" rex.rx rex.is_match yes io.println
end
demo

Output

$ pnx 16-rex.pnx
sum 30
true
2026-10
[key|: |value]
from 13/10/2026 to 02/11/2026
<a><b><c><d>
<가><나><다>
bar
true

16-rex-blocks.pnx

use io
use str
use rex
buffer out 64
word demo ()
  "2026-10-15" r"(\d+)-(\d+)-(\d+)" rex.rx do (whole str, y str, m str, d str)
    cat (d, "/", m, "/", y) io.println
  end rex.with_match or_else (e) "no date" io.println end
  "none here" r"\d+" rex.rx do (whole str) whole io.println end rex.with_match or_else (e) "no date" io.println end
  "the quick brown fox" r"\b[a-z]" rex.rx out do (w str gives str) w str.upper end rex.replace_by or_else (e) -1 end to n
  cat (n, " replaced: ", out contents) io.println
end
demo

Output

$ pnx 16-rex-blocks.pnx
15/10/2026
no date
4 replaced: The Quick Brown Fox

16-rex-compile.pnx

use io
use rex
buffer out 64
word swap_with (p str, s str gives n int or fail)  rem a pattern made while running
  out clear
  p rex.compile or_fail to re
  s re "-" out rex.replace_into or_fail
end
word demo ()
  "X" "aXbXc" swap_with or_else (e) -1 end to n
  cat (n, " replaced: ", out contents) io.println
  "(" "aXbXc" swap_with or_else (e) cat ("bad pattern: ", e err_text) io.println -1 end drop
end
demo

Output

$ pnx 16-rex-compile.pnx
2 replaced: a-b-c
bad pattern: format

Constraints

(7) The pattern of rx shall be a string literal or a const. A pattern made while running goes through compile.

16-rex-variable.pnx

use rex
word f (s str, p str gives bool) s p rex.rx rex.is_match end

Refused

$ pnx 16-rex-variable.pnx
16-rex-variable.pnx:2:38: rx takes a string literal or a const name; a pattern made while running is rex.compile

(8) Malformed patterns and missing features (back references and the like) are refused.

16-rex-bad.pnx

use rex
word f (s str gives bool) s r"a{3,1}" rex.rx rex.is_match end

Refused

$ pnx 16-rex-bad.pnx
16-rex-bad.pnx:2:39: a counted repetition goes downwards

16-rex-backref.pnx

use rex
word f (s str gives bool) s r"(a)\1" rex.rx rex.is_match end

Refused

$ pnx 16-rex-backref.pnx
16-rex-backref.pnx:2:38: back references are not supported

(9) A str where a regex goes is refused. The group parameters of a block shall be str.

16-rex-str.pnx

use rex
word f (s str gives bool) s "a" rex.is_match end

Refused

$ pnx 16-rex-str.pnx
16-rex-str.pnx:2:33: this word takes a str and a regex: s re rex.match

16-rex-group-type.pnx

use rex
word f (s str) s r"(a)" rex.rx rex.matches do (m str, g int) end each end

Refused

$ pnx 16-rex-group-type.pnx
16-rex-group-type.pnx:2:44: the groups are named str parameters

Faults

(10) The work space of a search is borrowed from the end of the region on every call. A region too small is the fault full. The search below comes after the region was nearly filled.

16-rex-full.pnx

use rex
use str
use io
word found (gives bool)
  "x" 4194000 str.repeat to s
  s r"(a|b|c|d|e|f|g|h)+x" rex.rx rex.is_match
end
found if "yes" else "no" end io.println

Fault

$ pnx 16-rex-full.pnx
16-rex-full.pnx:6: a declared capacity is used up
  s str = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Diagnostics

Condition	Message
A pattern for `rx` that is not a constant	`rx takes a string literal or a const name; a pattern made while running is rex.compile`
A count that goes downwards	`a counted repetition goes downwards`
A back reference	`back references are not supported`
A `str` where a `regex` goes	`this word takes a str and a regex: s re rex.match`
A group parameter that is not a `str`	`the groups are named str parameters`

16.8 `json` — JSON

(1) The module json reads and writes JSON (RFC 8259) text in one direction, without building a tree: it reads one token at a time and writes straight into a buffer.

16.8.1 Reading

(2) s json.next reads the first token of s and gives (kind int, value str, rest str) (or fail). At the end it fails with err.not_found; on something that is not a token, with err.format. White space, commas and colons between tokens are passed over. The structure of the grammar — matching brackets, where commas go — is not checked.

(3) kind is one of the constants below. The value of a string is its text without the quotes; when it has escapes, they are decoded into text written in the region. The value of a number is its text as written.

Constant	Token
`json.object` `json.object_end`	`{` `}`
`json.array` `json.array_end`	`[` `]`
`json.key`	a string followed by a colon
`json.string`	any other string
`json.number`	a number
`json.boolean` `json.null`	`true` `false`, `null`

(4) s json.skip passes over one value (with everything inside, for an object or array) and gives the rest (gives str or fail).

16-json-read.pnx

use io
use json
word kind_name (k int gives str)                    rem the kinds are constants of json
  k json.key eq if "key" return end
  k json.string eq if "string" return end
  k json.number eq if "number" return end
  k json.boolean eq if "boolean" return end
  k json.null eq if "null" return end
  k json.object eq k json.array eq or if "open" return end
  "close"
end
word tokens (s str)
  s to rest
  loop
    rest json.next or_else (e) break end to (kind, value, rest)
    cat (kind kind_name, " ", value) io.println
  end
end
text END
{"name": "kim", "tags": [1, -2.5e3, true, null], "note": "café"}
END
tokens

Output

$ pnx 16-json-read.pnx
open {
key name
string kim
key tags
open [
number 1
number -2.5e3
boolean true
null null
close ]
key note
string café
close }

16.8.2 Writing

(5) The writing words all append to a buffer and fail with err.full when there is not enough room. They put the commas in themselves.

Word	Signature
`begin_object` `end_object` `begin_array` `end_array` `put_null`	`(buf or fail)`
`put_key` `put_str`	`(s str, buf or fail)`; the word does the escaping
`put_int`	`(n int, buf or fail)`
`put_dec`	`(d dec(N), buf or fail)`
`put_bool`	`(b bool, buf or fail)`

16-json-write.pnx

use io
use json
buffer out 128
word demo (or fail)
  out json.begin_object or_fail
  "name" out json.put_key or_fail  "kim \"k\"" out json.put_str or_fail
  "tags" out json.put_key or_fail
  out json.begin_array or_fail  1 out json.put_int or_fail  true out json.put_bool or_fail  out json.put_null or_fail
  out json.end_array or_fail
  out json.end_object or_fail
end
demo or_else (e) e err_text io.println end
out contents io.println

Output

$ pnx 16-json-write.pnx
{"name":"kim \"k\"","tags":[1,true,null]}

16.8.3 Records

(6) row buf json.pack appends a row of a record as an object whose keys are the field names. s row json.unpack reads an object into the fields of the same names and gives the rest after the object. Values of unknown keys are passed over. Numbers shall be integers that fit the field’s width; a dec(N) field takes a number with at most N digits after the point and is written with all N (7.5); null is not accepted. The row is written only when everything is right; when it fails (err.format) the row is unchanged.

16-json-record.pnx

use io
use json
record user
  name str(8)
  id u16
  admin bool
end
list users user 4
buffer out 128
word input (gives str)
  text END
{"id": 9, "extra": {"x": [1, 2]}, "name": "park", "admin": false} and more
END
end
word demo (or fail)
  users add_row or_fail to u
  "lee" u to user.name  7 u16 u to user.id  true u to user.admin
  u out json.pack or_fail
  out contents io.println
  users add_row or_fail to v
  input v json.unpack or_fail to rest                  rem unknown keys are skipped
  cat (v user.name, " ", v user.id, " ", v user.admin, " /", rest, "/") io.println
  "{\"id\": 70000}" v json.unpack or_else (e) cat ("70000 in u16: ", e err_text) io.println "" end drop
  cat ("the row is unchanged: ", v user.id) io.println
end
demo or_else (e) e err_text io.println end

Output

$ pnx 16-json-record.pnx
{"name":"lee","id":7,"admin":true}
park 9 false / and more/
70000 in u16: format
the row is unchanged: 9

16-dec-text.pnx

use io
use json
use csv
record item
  name str(8)
  cost dec(2)
end
array one item 1
buffer out 64
word demo (or fail)
  0 one idx to r
  "pen" r to item.name  1.5 1 rescale 2 rescale r to item.cost
  r out json.pack or_fail  out contents io.println
  out clear  r out csv.pack or_fail  out contents io.print
  "{\"cost\": 3.05}" r json.unpack or_fail drop  cat (r item.cost) io.println
  "{\"cost\": 3.055}" r json.unpack or_else (e) cat ("3.055: ", e err_text) io.println "" end drop
end
demo or_else (e) e err_text io.println end

Output

$ pnx 16-dec-text.pnx
{"name":"pen","cost":1.50}
pen,1.50
3.05
3.055: format

Constraints

(7) json.next and json.skip take only a str; pack and unpack only rows of records.

16-json-type.pnx

use json
word f () 5 json.next drop drop drop end

Refused

$ pnx 16-json-type.pnx
16-json-type.pnx:2:13: this word takes the JSON text as a str

Diagnostics

Condition	Message
JSON text that is not a `str`	`this word takes the JSON text as a str`

16.9 `csv` — CSV

(1) The module csv reads and writes CSV in the form of RFC 4180: fields are separated by commas; a field that holds a comma, a quote or a line break is quoted, and quotes inside it are doubled.

(2) s csv.fields is a stem (10.6) of the fields of one line. A quoted field is given without its quotes and with doubled quotes made single. An unclosed quote gives the rest of the text as the last field (a stem cannot report a failure).

(3) The writing words append to a buffer (or fail, err.full when there is not enough room). s buf csv.put, n buf csv.put_int, d buf csv.put_dec and b buf csv.put_bool append one field (with the comma it needs); buf csv.end_line ends the line. Booleans are written true and false.

(4) row buf csv.pack appends one row of a record as one line. s row csv.unpack reads one line into the fields in order and gives what follows the line. The row is written only when every field is right (err.format). A dec(N) field is read and written as in json (16.8).

16-csv.pnx

use io
use csv
record user
  name str(8)
  id u16
  admin bool
end
list users user 4
buffer out 128
"a,\"b,c\",\"d\"\"e\",,f" csv.fields do (x str) cat ("[", x, "]") io.print end each
"" io.println
word demo (or fail)
  users add_row or_fail to u
  "lee, jr" u to user.name  7 u16 u to user.id  true u to user.admin
  u out csv.pack or_fail                               rem a line, quoted where needed
  "x" out csv.put or_fail  5 out csv.put_int or_fail  false out csv.put_bool or_fail  out csv.end_line or_fail
  out contents io.print
  users add_row or_fail to v
  "zed,5,true\nrest" v csv.unpack or_fail to rest
  cat (v user.name, " ", v user.id, " ", v user.admin, " [", rest, "]") io.println
  "zed,5,maybe" v csv.unpack or_else (e) cat ("bad bool: ", e err_text) io.println "" end drop
end
demo or_else (e) e err_text io.println end

Output

$ pnx 16-csv.pnx
[a][b,c][d"e][][f]
"lee, jr",7,true
x,5,false
zed 5 true [rest]
bad bool: format

Constraints

(5) pack and unpack take only rows of records.

16-csv-pack-int.pnx

use csv
buffer b 8
word f () 5 b csv.pack or_else (e) end end

Refused

$ pnx 16-csv-pack-int.pnx
16-csv-pack-int.pnx:3:15: pack takes a row of a table of records and a buffer: row buf csv.pack

Diagnostics

Condition	Message
A value that is not a row of a record	`pack takes a row of a table of records and a buffer: row buf csv.pack`