11 Text and binary formats

Builds on: Chapter 5 (views, buffers), chapter 7 (records), chapter 8 (failures).

Questions this chapter answers

Pinax has no heap, so it does not read JSON into a tree of objects. Instead it reads in one direction: one token, one field at a time. It writes straight into a buffer. Fitting records is done by code made from the record’s declaration.

11.1 CSV and JSON

csv.unpack reads one line into the fields of a record in order, and json.pack writes a record as an object keyed by field names.

11-csv-to-json.pnx

use io
use csv
use json
use str
rem Read CSV rows into records, write them out as JSON.
record city
  name str(16)
  people i32
  capital bool
end
list cities city 8
buffer out 512
word read_rows (input str or fail)
  input str.lines do (line str)
    cities len to before
    cities add_row or_fail to c
    line c csv.unpack or_else (e) before cities truncate  cat ("bad row: ", line) io.println  "" end drop
  end each
end
word to_json (or fail)
  out json.begin_array or_fail
  cities do (c cities) c out json.pack or_else (e) end end each
  out json.end_array or_fail
end
text END
Seoul,9411000,true
Busan,3349000,false
Daejeon,lots,false
END
read_rows or_else (e) end
to_json or_else (e) e err_text io.println end
out contents io.println

Output

$ pnx 11-csv-to-json.pnx
bad row: Daejeon,lots,false
[{"name":"Seoul","people":9411000,"capital":true},{"name":"Busan","people":3349000,"capital":false}]

The row is added first; when unpack fails, before cities truncate takes it back by cutting the list to the length it had before.

Common mistake: Taking the last row of a list of records off with pop

Lists of records have no pop (only lists of values do). The last row is taken back with truncate. When an index or a field holds rows of that list, truncate is refused too; then check before adding.

11-pop-records.pnx

record city
  name str(16)
end
list cities city 4
word undo () cities pop or_else (e) cities none end drop end

Refused

$ pnx 11-pop-records.pnx
11-pop-records.pnx:5:21: pop takes from a list of values; a list of records has no pop

Reading JSON by tokens picks out just what is needed, at any depth. json.next gives the kind of the token, a view of its value, and the rest.

11-json-read.pnx

use io
use json
use str
rem Sum every number in a JSON text, at any depth, without building a tree.
word sum_numbers (s str gives int)
  0 to total
  s to rest
  loop
    rest json.next or_else (e) break end to (kind, value, rest)
    kind json.number eq if total value str.to_int or_else (e) 0 end add to total end
  end
  total
end
text END
{"a": 1, "b": [2, 3, {"c": 4}], "d": {"e": [5]}, "f": "6"}
END
sum_numbers str.to_str io.println

Output

$ pnx 11-json-read.pnx
15

Look at the three things json.next gives, one token at a time. s json.next gives (kind int, value str, rest str): the kind of the token, its text (a view), and the text left after it. Giving what is left to json.next again, over and over, walks the whole text once from the front. At the end of the text json.next fails, and that failure ends the loop.

11-json-tokens.pnx

use io
use json
rem The tokens of a JSON text, one at a time: what json.next gives for each.
word kind_name (k int gives str)       rem module constants are compared with eq, not case
  k json.object eq if "object" return end
  k json.object_end eq if "object end" return end
  k json.array eq if "array" return end
  k json.array_end eq if "array end" return end
  k json.key eq if "key" return end
  k json.string eq if "string" return end
  k json.number eq if "number" return end
  "other"
end
word show_tokens (s str)
  s to rest
  loop
    rest json.next or_else (e) break end to (kind, value, rest)   rem fails at the end: leave the loop
    cat (kind kind_name, " [", value, "]") io.println
  end
end
"{\"name\": \"Pinax\", \"sizes\": [8, 16]}" show_tokens

Output

$ pnx 11-json-tokens.pnx
object [{]
key [name]
string [Pinax]
key [sizes]
array [[]
number [8]
number [16]
array end []]
object end [}]
TokenStack after itWhat happened
rest json.nextjson.object "{" …Reads {: kind json.object; what is left is "name": ….
or_else (e) break endjson.object "{" …It did not fail, so the body is skipped.
to (kind, value, rest)—The three go into names; the rightmost, rest, takes the top value.

A kind is a module constant like json.object, so it is compared with eq, not case (chapter 4). Only the work for each kind has to be chosen, so the values needed can be picked out at any depth; 11-json-read.pnx added up the json.number tokens only.

A value of no interest is skipped in one go with json.skip an object or an array is skipped with everything inside it, so there is no depth to count. The kinds are the constants json.object json.object_end json.array json.array_end json.key json.string json.number json.boolean json.null. The other way round, one object fills a row of a record with json.unpack: the row is written only when every field fits, else it fails.

11-json-skip.pnx

use io
use str
use json
record point
  x int
  y int
end
array pts point 1
word demo (or fail)
  "{\"tags\": [1, [2, 3]], \"x\": 4}" json.next or_fail to (k, v, rest)   rem the {
  rest json.next or_fail to (k, v, rest)                                   rem the key tags
  rest json.skip or_fail to rest                                           rem the whole array, however deep
  rest json.next or_fail to (k, v, rest)                                   rem the key x
  rest json.next or_fail to (k, v, rest)                                   rem 4
  cat ("after the skip: ", v, ", a number? ", k json.number eq) io.println
  "{\"x\": 7, \"y\": 9}" 0 pts idx json.unpack or_fail drop
  cat ("unpacked ", 0 pts idx point.x, " ", 0 pts idx point.y) io.println
  "{\"x\": true}" 0 pts idx json.unpack or_else (e) cat ("unpack failed: ", e err_text) io.println "" end drop
end
demo or_else (e) e err_text io.println end

Output

$ pnx 11-json-skip.pnx
after the skip: 4, a number? true
unpacked 7 9
unpack failed: format

CSV is written one field at a time. csv.put appends text (quoted when it holds a comma, a quote or a line break), csv.put_int csv.put_dec csv.put_bool numbers and booleans, and csv.end_line the end of the line; the commas between fields are the words’ business. One row of a record as one line is csv.pack.

11-csv-write.pnx

use io
use csv
buffer out 64
word demo (or fail)
  "kim" out csv.put or_fail  3 out csv.put_int or_fail  true out csv.put_bool or_fail  out csv.end_line or_fail
  "a, b" out csv.put or_fail  2.50 out csv.put_dec or_fail  out csv.end_line or_fail
  out contents io.print
end
demo or_else (e) e err_text io.println end

Output

$ pnx 11-csv-write.pnx
kim,3,true
"a, b",2.50

11.2 Bytes

The module bytes writes and reads fixed widths in a given byte order. With big_endian in a record, bytes.pack writes the most significant byte first. A varint field writes small numbers in few bytes.

11-binary.pnx

use io
use bytes
use encode
use hash
use fmt
record header
  big_endian
  magic u32
  version u16
  length varint
end
array headers header 2
buffer wire 32
buffer hex 64
word demo (or fail)
  0 headers idx to h
  0x50494e58 u32 h to header.magic           rem "PINX"
  1 u16 h to header.version
  300 h to header.length
  h wire bytes.pack or_fail
  wire contents hex encode.hex_encode or_fail
  cat ("bytes: ", hex contents, ", crc32: ", wire contents hash.crc32 fmt.hex) io.println
  1 headers idx to back
  wire contents back bytes.unpack or_fail drop
  cat ("read back: version ", back header.version, ", length ", back header.length) io.println
end
demo or_else (e) e err_text io.println end

Output

$ pnx 11-binary.pnx
bytes: 50494e580001ac02, crc32: 31d80bed
read back: version 1, length 300

The writing words (pack, put_…, encode.…_encode) fail when the buffer is full, so a guard always follows them.

11-pack-unguarded.pnx

use json
record point
  x int
  y int
end
list points point 2
buffer out 64
word write (p points) p out json.pack end   rem pack fails when the buffer is full: no guard

Refused

$ pnx 11-pack-unguarded.pnx
11-pack-unguarded.pnx:8:39: a word that can fail is followed by or_fail, or_return or or_else

Q&A

Q. Does the record lie in memory in big-endian order too?

A. No. big_endian sets only the order of the bytes that pack and unpack use. The layout in memory is the processor’s, and a script cannot see it.

11.3 Other conversions

fmt turns numbers into hexadecimal, binary and zero-padded text. hash gives CRC-32, FNV-1a and SipHash; encode gives Base64, hexadecimal and URL percent-encoding. Words that write their results into a buffer fail when the buffer is full. The full lists are in clause 16 of the standard.

A hash takes text and gives an int. hash.crc32 is a checksum; hash.fnv1a and hash.hash64 mix keys. How many bytes one row of a record becomes under bytes.pack is told in advance by bytes.packed_size, which helps in sizing a buffer.

11-hash.pnx

use io
use fmt
use hash
use bytes
record pair
  a u16
  b int
end
array pairs pair 1
word demo ()
  cat ("crc32 ", "pinax" hash.crc32 fmt.hex, ", fnv1a ", "pinax" hash.fnv1a fmt.hex) io.println
  cat ("a pair packs into ", 0 pairs idx bytes.packed_size, " bytes") io.println
end
demo

Output

$ pnx 11-hash.pnx
crc32 19afea99, fnv1a 131b946615bc7b35
a pair packs into 10 bytes

Q&A

Q. Can the JSON writer indent, or add line breaks?

A. No. The writing words of json append one line. When a layout for people is needed, the receiving side makes it. The reading side accepts spaces and line breaks as they come.

Summary