5 Strings

Builds on: Chapter 2 (types of values), chapter 4 (stems and blocks).

Questions this chapter answers

Why does cutting and splitting a string copy nothing?
How do str and str(N) differ, and when is each used?
Where do the bytes of new text go, and how are they given back?

5.1 Views: strings that do not copy

A str value is a view. It points at where some bytes are and how many; it does not own them. So cutting (str.take str.skip str.slice), trimming (str.trim) and splitting (str.cut str.split str.lines str.words) copy no bytes at all; they give new views of parts of the original text.

05-views.pnx

use io
use str
rem Parse "key = value" lines without copying a byte.
word parse (line str)
  line str.trim to t
  t str.is_empty t "#" str.starts_with or if return end      rem blank lines and remarks
  t "=" str.try_cut or_else (e) cat ("no '=' in: ", t) io.println return end to (key, value)
  cat ("[", key str.trim, "] = [", value str.trim, "]") io.println
end
text END
# settings
name = Pinax
  version= 1

broken line
END
str.lines do (l str) l parse end each

Output

$ pnx 05-views.pnx
[name] = [Pinax]
[version] = [1]
no '=' in: broken line

First a small example of what a view is. head, tail and name below all point at the bytes of the same literal. New bytes appear only when cat makes the text to print.

05-slices.pnx

use io
use str
word show ()
  "Hello, Pinax world" to s         rem s is a view of the 18 bytes of the literal
  s 5 str.take to head              rem the first 5 bytes: "Hello", no copy
  s 7 str.skip to tail              rem all but the first 7: "Pinax world", no copy
  tail 0 5 str.slice to name        rem bytes 0 up to 5 of tail: "Pinax"
  cat ("[", head, "] [", tail, "] [", name, "]") io.println   rem cat makes new bytes, in the region
  cat ("lengths: ", s str.len, " ", head str.len, " ", name str.len) io.println
end
show

Output

$ pnx 05-slices.pnx
[Hello] [Pinax world] [Pinax]
lengths: 18 5 5

View	Points at	Length
`s`	the literal, from byte 0	18
`head`	the same place, from byte 0	5
`tail`	the same literal, from byte 7	11
`name`	byte 0 of `tail`, that is byte 7 of the literal	5

05-views.pnx above splits settings text into lines, trims each line and cuts it in two at =. Bytes move only when they are printed. str.try_cut fails when there is no =, so it is taken with or_else. str.cut, which does not fail, gives an empty text after the separator when there is none.

5.2 Stored text

A view cannot live longer than the bytes it points at. So places that live long — globals, cells of arrays, fields of records — cannot hold views; they use stored text, str(N). A str(N) takes N bytes of room and copies the bytes put into it. Reading it gives a view of that room.

05-stored.pnx

use io
use str
global longest str(16)                     rem stored text: 16 bytes, copied in
word keep_longest (w str) w str.len longest str.len gt if w set longest end end
"the quick brown fox jumps over the lazy dog" str.words do (w str) w keep_longest end each
cat ("longest word: ", longest) io.println

Output

$ pnx 05-stored.pnx
longest word: quick

05-global-view.pnx

global name str

Refused

$ pnx 05-global-view.pnx
05-global-view.pnx:2:1: stored text is str(N); a str view lives only in locals, arguments and results

Which one to use depends on how long the text lives.

	`str` (view)	`str(N)` (stored text)
Where	locals, arguments, results	globals, cells of arrays and lists, fields of records
Bytes	none of its own: it points at bytes elsewhere	room for N bytes of its own
Storing	copies nothing	copies; longer than N is a fault, or with a guard the failure `range`
Lives	as long as the bytes it points at (one run, or until `release`)	as long as its place

A view in a field is refused at load, and the message says to write str(N).

05-field-view.pnx

record person
  name str                          rem a view cannot be kept in a field: write str(N)
end

Refused

$ pnx 05-field-view.pnx
05-field-view.pnx:3:1: a field holds str(N); a str view lives only in locals, arguments and results

Common mistake: Text longer than str(N) is cut to fit

It is not. Putting text longer than the room stops with a fault. To store part of it, cut it first with str.take.

05-too-long.pnx

use io
global name str(8)
"Pinax" set name
cat ("stored: ", name) io.println
"an embeddable language" set name

Fault

$ pnx 05-too-long.pnx
stored: Pinax
05-too-long.pnx:5: index out of range

Q&A

Q. The text comes from outside and may be too long. Must I compare its length first?

A. No. Put a guard after the store, as after any word that can fail. A text too long is then the failure range, and the place keeps what it held. Without the guard it stays a fault.

05-too-long-guard.pnx

use io
global name str(8)
word remember (v str)
  v set name or_else (e) cat ("too long, kept ", name) io.println return end
  cat ("stored: ", name) io.println
end
"Pinax" remember
"an embeddable language" remember

Output

$ pnx 05-too-long-guard.pnx
stored: Pinax
too long, kept Pinax

The guard comes after the store, so what the word did before it stays done. To refuse a text before anything else happens (a row taken, a count raised), ask the place first with fits: v fits entry.name takes the text and fails the same way, with range, and writes nothing. The width is the place’s own.

05-fits.pnx

use io
record entry
  name str(8)
end
list names entry 4
word enter (v str)
  v fits entry.name or_else (e) cat (v, ": too long, no row taken") io.println return end
  names add_row or_else (e) "full" io.println return end to r
  v r to entry.name
  cat ("added ", r entry.name, ", rows ", names len) io.println
end
"Pinax" enter
"an embeddable language" enter

Output

$ pnx 05-fits.pnx
added Pinax, rows 1
an embeddable language: too long, no row taken

5.3 Making new text, and buffers

cat and words that make new bytes, such as str.upper and str.repeat, write those bytes into the region and give a view of them. To build text piece by piece, declare a buffer and add to it with the append words. When the buffer is full, the word that adds fails.

05-build.pnx

use io
use str
use fmt
buffer out 64
word report (or fail)
  "id: " out append or_fail
  42 out append_int or_fail
  ", hex: 0x" out append or_fail
  255 out fmt.append_hex or_fail
  '!' out append_byte or_fail
end
report or_else (e) cat ("buffer full: ", e err_text) io.println end
out contents io.println
cat ("upper: ", "pinax" str.upper, ", repeat: ", "ab" 3 str.repeat, ", padded: [", "7" 4 str.pad_left, "]") io.println

Output

$ pnx 05-build.pnx
id: 42, hex: 0xff!
upper: PINAX, repeat: ababab, padded: [   7]

5.4 Giving the region back

The region has a fixed size for each run. A loop that keeps making text fills it up. mark remembers the current place and release goes back to it; the bytes made in between are used again.

05-region.pnx

use io
use str
word line (i int gives str) cat ("line ", i) end
word lots ()
  region_free to before
  1 10001 range do (i int)
    mark to m                               rem every turn gives its bytes back
    i line drop
    m release
  end each
  cat ("bytes still taken after 10000 lines: ", before region_free sub) io.println
end
lots

Output

$ pnx 05-region.pnx
bytes still taken after 10000 lines: 0

After the release, views made in between cannot be used. The processor checks this at load too.

05-release-view.pnx

use io
word f ()
  mark to m
  cat ("made ", "after the mark") to s
  m release
  s io.println
end

Refused

$ pnx 05-release-view.pnx
05-release-view.pnx:6:3: this str was made after a mark that was released since; the region may hold other bytes there now

Q&A

Q. How do I keep text that must outlive the release?

A. Put it into a str(N) global or cell. It is copied there, so it survives rewinding the region. 05-stored.pnx above does that.

Q&A

Q. A word inside each fails, under defer m release. Can I pass the failure on with or_fail?

A. Yes. On the way out of a failing word the release does not look at the text left on the stack: the word ends there and nothing reads it again. The stem stops at the first failure.

05-release-failing.pnx

use io
use str
word short (w str or fail)
  w str.len 5 gt if err.range fail end
end
word read_words (line str or fail)
  mark to m
  defer m release end
  line str.upper str.words do (w str) w short or_fail  w io.println end each
end
"one two three" read_words or_else (e) cat ("stopped: ", e err_text) io.println end
"one embeddable two" read_words or_else (e) cat ("stopped: ", e err_text) io.println end

Output

$ pnx 05-release-failing.pnx
ONE
TWO
THREE
ONE
stopped: range

5.5 Bytes and characters

Lengths and positions count bytes. A Hangul character is 3 bytes in UTF-8. When characters matter, use str.char_count and str.chars. str.chars yields a view of each character (not a code point number).

05-chars.pnx

use io
use str
word show ()
  "한글 ok" to s
  cat ("bytes ", s str.len, ", characters ", s str.char_count) io.println
  s str.chars do (c str) cat ("<", c, ">") io.print end each
  "" io.println
end
show

Output

$ pnx 05-chars.pnx
bytes 9, characters 5
<한><글>< ><o><k>

5.6 Character classes, and reading a decimal

To take “as many digits as there are” off the front of a text, use str.span. The classes are the constants str.digit str.alpha str.alnum str.space str.upper_case str.lower_case str.hex_digit str.punct (all ASCII). A number with a point is read as a dec(N) by str.to_dec (chapter 2); more digits after the point than N is a failure.

05-classes.pnx

use io
use str
rem span cuts off the run of one class of bytes at the front; to_dec reads a decimal.
word demo ()
  "42abc" str.digit str.span to (digits, rest)
  cat ("[", digits, "] [", rest, "]") io.println
  "   x" str.space str.span to (blanks, after)
  cat (blanks str.len, " blanks, then ", after) io.println
  "12.50" 2 str.to_dec or_else (e) 0.00 end to price
  cat ("price ", price) io.println
  "12.505" 2 str.to_dec or_else (e) cat ("not a dec(2): ", e err_text) io.println 0.00 end drop
end
demo

Output

$ pnx 05-classes.pnx
[42] [abc]
3 blanks, then x
price 12.50
not a dec(2): format

Token	Stack after it	What happened
`"42abc" str.digit`	`"42abc" 48`	The text and the class (a class is a small integer).
`str.span`	`"42" "abc"`	The digits at the front and the rest, both views of the same text.
`to (digits, rest)`	`—`	The right-hand name takes the top value.

Two words write joined or replaced text straight into a buffer: str.join_into joins the pieces of a stem with a separator, str.replace_into appends the replaced text and gives the number of replacements. Both fail when the buffer is full. join_into takes the stem last, because a stem is taken by the word right after it (chapter 4).

05-join.pnx

use io
use str
buffer joined 64
buffer fixed 64
word demo (or fail)
  ", " joined "alpha beta gamma" str.words str.join_into or_fail   rem the stem comes last
  joined contents io.println
  "hello world" "o" "0" fixed str.replace_into or_fail to n
  cat (n, " replaced: ", fixed contents) io.println
end
demo or_else (e) e err_text io.println end

Output

$ pnx 05-join.pnx
alpha, beta, gamma
2 replaced: hell0 w0rld

Q&A

Q. Do the character classes know Korean or other Unicode letters?

A. No. A class covers ASCII only; a Korean syllable is not in str.alpha. When characters rather than bytes matter, take them one by one with str.chars and compare them yourself.

Summary

str is a view; cutting and splitting copy nothing (standard 7.3, 16.1).
A view cannot be kept in a long-lived place (global, cell, field) (7.3).
Long-lived places use str(N); putting text there copies it, and text too long is a fault, or with a guard after the store the failure range (7.3).
New bytes go into the region; mark/release gives them back, and views made in between are refused after (12.2, 12.3).
A buffer is where text is built; it fails when full (8.7).
Lengths are bytes; for characters, char_count and chars (16.1).