12 Patterns

Builds on: Chapter 5 (views), chapter 4 (stems).

Questions this chapter answers

12.1 Small patterns: ptn

The patterns of ptn are the size of Lua’s: classes such as %d %a %s, […] sets, * + - ?, and captures (…). There is no alternation (|). After a match, n ptn.capture reads a capture.

12-ptn.pnx

use io
use ptn
use str
rem Pull fields out of log lines with a small pattern.
const entry "^(%d+):(%d+) (%u+) (.*)$"
word show (line str)
  line entry ptn.match or_else (e) cat ("no match: ", line) io.println return end drop
  cat (3 ptn.capture, " at ", 1 ptn.capture, "h", 2 ptn.capture, ": ", 4 ptn.capture) io.println
end
"09:15 INFO server started" show
"10:02 WARN disk 91% full" show
"garbage" show
"a1b22c333" "%d+" ptn.matches do (m str) cat ("[", m, "]") io.print end each
"" io.println
"hello world" "(%w+)" "<%1>" ptn.replace io.println

Output

$ pnx 12-ptn.pnx
INFO at 09h15: server started
WARN at 10h02: disk 91% full
no match: garbage
[1][22][333]
<hello> <world>

A pattern is a row of small pieces. The pieces used most are these (the full list is in standard 16.6).

PieceMatches
an ordinary character such as athat one byte
.any one byte
%d %a %s %wone digit, letter, space, letter or digit; in capitals (%D …), anything else
%. %%a point, a % itself
[ae] [a-z] [^0-9]one of those listed, one in the range, one not listed
+ * ? after a pieceonce or more, zero times or more, zero times or once
^ and $at the start of the pattern: only at the start of the text; at its end: only at the end
( … )captures what it matched; read it with n ptn.capture

Trying patterns one at a time shows what each piece means.

12-ptn-steps.pnx

use io
use ptn
word say (ok bool, what str) cat (ok if "yes " else "no  " end, what) io.println end
"order 66" "%d+" ptn.is_match "%d+: a digit, one or more" say
"order 66" "^%d+" ptn.is_match "^%d+: digits at the start, and it starts with a letter" say
"order 66" "%a+ %d+$" ptn.is_match "%a+ %d+$: letters, a space, digits at the end" say
"grey" "gr[ae]y" ptn.is_match "gr[ae]y on grey: [ae] is a or e" say
"colour" "colou?r" ptn.is_match "colou?r on colour: ? makes the u optional" say
"x=" "x=%d*" ptn.is_match "x=%d* on x=: * allows none at all" say
"3.14" "%d%.%d" ptn.is_match "%d%.%d on 3.14: %. is a real point" say

Output

$ pnx 12-ptn-steps.pnx
yes %d+: a digit, one or more
no  ^%d+: digits at the start, and it starts with a letter
yes %a+ %d+$: letters, a space, digits at the end
yes gr[ae]y on grey: [ae] is a or e
yes colou?r on colour: ? makes the u optional
yes x=%d* on x=: * allows none at all
yes %d%.%d on 3.14: %. is a real point

ptn.capture reads the last match of this run. Reading it before any match is the fault range. A match made in another run is not seen — its text may be gone by then.

12-capture-none.pnx

use io
use ptn
word first_number (gives str)
  1 ptn.capture                               rem no match in this run yet: there is nothing to read
end
first_number io.println

Fault

$ pnx 12-capture-none.pnx
12-capture-none.pnx:4: index out of range

12.2 Regular expressions: rex

rex has regular expressions with alternation, counts ({n,m}), word boundaries (\b) and flags ((?i) and so on). It matches without going back, so any pattern ends in time proportional to the length of the text. A pattern is made into a program with rex.rx, and the block of matches takes the groups as parameters.

12-rex.pnx

use io
use rex
use str
rem Dates in any of two forms, rewritten to one.
const iso r"(\d{4})-(\d{2})-(\d{2})"
word demo ()
  "due 2026-10-15, paid 2026-11-02" iso rex.rx r"\3.\2.\1" rex.replace io.println
  "id=7, x=20; name=kim, n=3" r"(\w+)=(\d+)" rex.rx rex.matches do (whole str, key str, value str)
    cat (key, " -> ", value) io.println
  end each
  "a , b;c ,d" r"\s*[,;]\s*" rex.rx rex.split do (p str) cat ("<", p, ">") io.print end each
  "" io.println
end
demo

Output

$ pnx 12-rex.pnx
due 15.10.2026, paid 02.11.2026
id -> 7
x -> 20
n -> 3
<a><b><c><d>

A block can also make the text that replaces each match.

12-replace-by.pnx

use io
use rex
use str
buffer out 128
word title (s str gives int or fail)          rem capitalize every word
  out clear
  s r"\b[a-z]" rex.rx out do (w str gives str) w str.upper end rex.replace_by or_fail
end
word show () "the quick brown fox" title or_else (e) -1 end to n  cat (out contents, " (", n, " letters)") io.println end
show

Output

$ pnx 12-replace-by.pnx
The Quick Brown Fox (4 letters)

Three more words search. rex.match gives the first match and fails when there is none. rex.matches is a stem that runs the block once per match; the block takes the whole match and one str per group. rex.split is the stem of the pieces between matches. ptn has the same stem, ptn.matches.

12-rex-words.pnx

use io
use str
use rex
word demo ()
  "order 66, aisle 7" r"\d+" rex.rx rex.match or_else (e) "none" end io.println
  "no digits here" r"\d+" rex.rx rex.match or_else (e) "none" end io.println
  "x=1, y=22" r"(\w)=(\d+)" rex.rx rex.matches do (whole str, name str, value str)
    cat (name, " is ", value) io.println
  end each
  "a;b;;c" r";" rex.rx rex.split do (p str) cat ("[", p, "]") io.print end each
  "" io.println
end
demo

Output

$ pnx 12-rex-words.pnx
66
none
x is 1
y is 22
[a][b][][c]

12-ptn-matches.pnx

use io
use ptn
word demo ()
  "x=1, y=22, z=333" "%a=%d+" ptn.matches do (m str) cat ("<", m, ">") io.print end each
  "" io.println
end
demo

Output

$ pnx 12-ptn-matches.pnx
<x=1><y=22><z=333>
TokenStack after itWhat happened
"order 66, aisle 7" r"\d+" rex.rx"order 66, aisle 7" reThe text and the pattern program; rex.rx builds it at load.
rex.match"66"A view of the first match. With none, it fails.
or_else (e) "none" end"66"Success: skipped.
io.println—Prints it.

Common mistake: Writing a regular expression in an ordinary string

"\d+" is refused, because in an ordinary string \d is an unknown escape. Worse is "\b": it becomes a backspace character and quietly means something else. Write patterns in raw strings r"…".

12-escape.pnx

use rex
word digits (s str gives bool) s "\d+" rex.rx rex.is_match end

Refused

$ pnx 12-escape.pnx
12-escape.pnx:2:35: unknown escape; a raw string r"..." keeps backslashes as written

12.3 Patterns are constants

The patterns of ptn and of rex.rx shall be literals or consts, so that the processor can check them at load and make them into programs. A malformed pattern is refused before anything runs.

12-pattern-variable.pnx

use rex
word find (s str, p str gives bool) s p rex.rx rex.is_match end

Refused

$ pnx 12-pattern-variable.pnx
12-pattern-variable.pnx:2:41: rx takes a string literal or a const name; a pattern made while running is rex.compile

When a pattern made while running is needed, use rex.compile. A malformed pattern comes back as a failure (format).

12-compile.pnx

use io
use rex
use env
word count_matches (pattern str, s str gives int or fail)
  pattern rex.compile or_fail to re
  0 to n
  s re rex.matches do (m str) n 1 add to n end each
  n
end
word main ()
  1 env.arg or_else (e) "x" end to p
  cat ("matches of ", p, ": ", p "a1 b22 c333" count_matches or_else (e) -1 end) io.println
  "(" "a1" count_matches or_else (e) cat ("bad pattern: ", e err_text) io.println -1 end drop
end
main

Output

$ pnx 12-compile.pnx [0-9]+
matches of [0-9]+: 3
bad pattern: format

Q&A

Q. Which do I use, ptn or rex?

A. For short patterns with a few captures ptn is lighter. When you need alternation or counts, when the pattern comes from outside, or when the text is very long, use rex. ptn goes back and tries again, so a bad pattern runs into its step limit and stops with a fault.

Q&A

Q. Is a match copied?

A. No. What rex.match gives, the groups and the pieces of split are all views of the original text (chapter 5). Only replaced text is new: replace writes it in the region, replace_into and replace_by into a buffer.

Summary