6 Lexical elements

(1) This clause defines how a source is split into tokens. The syntax rules (from clause 7 on) read the sequence of tokens this clause produces.

6.1 Characters of a source

Syntax

letter      = "A" | ... | "Z" | "a" | ... | "z" | "_" ;
digit       = "0" | ... | "9" ;
space       = ? SPACE, TAB, CR, LF ? ;
newline     = ? LF, or CR LF ? ;
shebang     = "#!" { ? any byte but CR and LF ? } newline ;   (* the first line only *)
source-text = [ shebang ] { space | remark | token } ;

(1) A source is a sequence of bytes written in UTF-8. Characters outside ASCII appear only inside remarks (6.3) and character, string and text literals (6.5).

(2) White space only separates tokens; it has no other meaning. A line break closes nothing. The exceptions are line remarks, the line rules of note and text, and strings, which must close on their line.

(3) When the first line of a file begins with #!, the line is ignored, so that a script file can be run directly on Unix-like systems.

Constraints

(4) Outside literals and remarks only the tokens of 6.6 and white space may appear. Any other character — ; { + and the like, and any character outside ASCII — is refused. Names cannot contain characters outside ASCII either.

use io
"done" io.println;

$ pnx 06-semicolon.pnx
06-semicolon.pnx:2:18: this character is not allowed outside literals and remarks

global 합계 int

$ pnx 06-hangul-name.pnx
06-hangul-name.pnx:1:8: this character is not allowed outside literals and remarks

6.2 Runs and names

Syntax

run    = ( letter | digit ) { letter | digit } ;
NAME   = letter { letter | digit } ;             (* and not a reserved word *)
PATH   = NAME "." member { "." member } ;        (* no space around a dot *)
member = letter { letter | digit } ;              (* a reserved word too: json.array *)

(1) Letters, digits and underscores that follow each other are read whole, as one run, and only then classified. A run that is a reserved word (6.6) is a reserved word; one that begins with a digit is a number (6.4); anything else is a name. So remark is a name, not a remark, and 12ab is neither a name nor a number.

(2) A path is names joined by dots, with no space around a dot. Inside a run of text a dot appears only in paths and in numbers (2.5). A dot that no name, digit, dot or quote follows is a token of its own: it ends a statement (10.8), also when it is written against the run before it (7 sq.). A path names a word of a module (io.println), a field of a record (p.x) or a constant (str.digit).

(3) A run after a dot is part of the path even when it is a reserved word (json.array). The first run of a path cannot be a reserved word.

Constraints

(4) A dot inside a path shall be followed by a name, with no space.

use io
"x" io. println

$ pnx 06-dot-apart.pnx
06-dot-apart.pnx:2:5: this name is not declared before this place

(5) Names are case-sensitive: Total and total are different names. A name is at most 31 bytes long (annex C).

6.3 Remarks

Syntax

remark       = line-remark | block-remark ;
line-remark  = "rem" { ? any byte but CR and LF ? } newline ;   (* "rem" as a whole run *)
block-remark = "note" TAG { ? any byte but CR and LF ? } newline
               { ? a line that does not start with TAG at the first column ? newline }
               TAG ;                                             (* at the first column *)
TAG          = NAME ;

(1) The run rem makes a remark from there to the end of the line, but only when the whole run is rem: remark is a name. The end of the file stands in for the line break.

(2) The run note and a tag open a block remark. The remark is closed by a line that starts, in the first column, with that tag, not followed by a letter or digit. An indented tag, or a longer run that begins with the tag (DOCUMENT), does not close it.

use io
rem A line remark runs to the end of its line.
"one" io.println rem a remark may follow code
note DOC
A block remark runs to a line that starts with its tag.
  DOC indented does not close it, nor does DOCUMENT.
DOC
global remark int        rem "remark" is a name: rem must stand alone
global notes int
3 set remark
"two" io.println

$ pnx 06-remarks.pnx
one
two

(3) A tool may read the remark right before a declaration as its description. There is no separate mark for documentation remarks.

(4) Where a new name is due, the runs rem, note and text open neither a remark nor a text: they are words of the language there, and refused as names (9.5). A new name is due after a word that declares one (word, test, global, const, array, list, pool, queue, buffer, tasks, record, type, index), after to, set and as, and right after the opening parenthesis or a comma of a signature or of the names of to, loop, while, until, break and or_else. In the body of a record text is always refused; rem and note are remarks there, as everywhere else.

global note int

$ pnx 06-remark-name.pnx
06-remark-name.pnx:1:8: note is a word of the language; choose another name

6.4 Numbers

Syntax

sign        = "+" | "-" ;                    (* attached: no space before the digits *)
digits      = digit { [ "_" ] digit } ;
INT         = [ sign ] ( digits | "0x" hex-digits | "0b" bit-digits ) ;
DEC         = [ sign ] digits "." digits ;   (* a decimal, 7.5; no exponent *)
hex-digits  = hex { [ "_" ] hex } ;          (* hex: digit, A-F, a-f *)
bit-digits  = bit { [ "_" ] bit } ;          (* bit: 0, 1 *)

(1) The value of an integer literal is a 64-bit two’s complement integer (7.1). It is written in decimal, hexadecimal (0x) or binary (0b). There is no octal form: 0755 is decimal 755.

(2) A sign is part of a number when it is attached to the digits: -42 is one number. Addition and subtraction are the words add and sub; + and - are not characters of their own.

(3) One underscore may stand between two digits; it is not part of the value. 0x and 0b are lower case only; hexadecimal digits A–F may be either case.

use io
use str
1_000_000 str.to_str io.println     rem underscores between digits
0xFF str.to_str io.println          rem hexadecimal
0b1010 str.to_str io.println        rem binary
0755 str.to_str io.println          rem decimal: there is no octal form
-42 str.to_str io.println           rem the sign is part of the number
-9223372036854775808 str.to_str io.println
'A' str.to_str io.println           rem a character is a u8: one byte
U'가' str.to_str io.println          rem U'..' is one code point

$ pnx 06-numbers.pnx
1000000
255
10
755
-42
-9223372036854775808
65
44032

Constraints

(4) A number directly followed by a letter, an underscore or a dot is refused.

use io
12ab drop

$ pnx 06-number-letters.pnx
06-number-letters.pnx:2:3: a number must not run into letters, an underscore or a dot

(5) A space between a sign and its digits is refused.

use io
- 7 drop

$ pnx 06-sign-apart.pnx
06-sign-apart.pnx:2:1: a sign is attached to its number; arithmetic uses words (add, sub)

(6) A value outside the range of a 64-bit signed integer (−2⁶³ to 2⁶³−1) is refused.

9223372036854775808 drop

$ pnx 06-int-range.pnx
06-int-range.pnx:1:1: the integer does not fit in 64 bits

(7) An underscore stands only between two digits, one at a time: 1__000 _1 1_ 0x_F are refused.

1__000 drop

$ pnx 06-underscore.pnx
06-underscore.pnx:1:2: an underscore stands between two digits

(8) A number with a point and digits on both sides and no exponent is a decimal literal (12.50, 7.5). At most 18 digits follow the point, and the digits together, as an integer, shall fit in 64 bits. A number with an exponent (2e3, 2.5e0) or a hexadecimal one with p is a float literal; its form is in the grammar (annex A), but this edition has no float type, so it is refused (annex D).

6.5 Characters, strings, text

Syntax

CHAR      = [ "U" ] "'" ( char-byte | escape ) "'" ;
STRING    = '"' { string-byte | escape } '"' ;          (* closes on the same line *)
RAWSTRING = "r" '"' { raw-byte } '"' ;                  (* no escapes; same line *)
TEXT      = "text" TAG { blank } newline
            { ? a line that does not start with TAG at the first column ? newline }
            TAG ;                                       (* at the first column *)
escape    = "\\" | '\"' | "\'" | "\a" | "\b" | "\f" | "\n" | "\r" | "\t" | "\v" | "\0"
          | "\x" hex hex | "\u" hex hex hex hex | "\U" hex hex hex hex hex hex hex hex ;
BOOL      = "true" | "false" ;

(1) The value of a character literal is, without a prefix, the u8 of one byte, and with U the u32 of one code point (7.6; 'A' is the u8 65, U'가' the u32 44032).

(2) The value of a string literal is a str. The fourteen escapes above are the only ones, each of a fixed length. \u and \U put a Unicode code point (not a surrogate, at most U+10FFFF) as UTF-8; \x puts one byte.

(3) A raw string is the run r with " directly after it. Backslash is an ordinary character, and there is no way to put a " inside. It is for paths and regular expressions.

(4) A text literal writes several lines as they are. The text TAG line may hold only spaces after the tag; the value is the lines after it, up to the line of the closing tag. The line break right before the closing line is not part of the value. Escapes are not decoded. The type is str.

use io
"tab:\t| quote:\" | hex:\x41 | code point:\u00e9" io.println
r"C:\new\table" io.println          rem raw: backslashes stay
text END
  A text literal keeps its lines as written, "quotes" and \n too.
END
io.println

$ pnx 06-strings.pnx
tab:  | quote:" | hex:A | code point:é
C:\new\table
  A text literal keeps its lines as written, "quotes" and \n too.

(5) true and false are the two values of bool.

Constraints

(6) A string shall close on the line where it opens.

use io
"no closing quote
" io.println

$ pnx 06-string-open.pnx
06-string-open.pnx:2:1: the string is not closed on its line

(7) An escape not listed above is refused. To keep backslashes as written, use a raw string.

use io
"C:\path" io.println

$ pnx 06-escape-unknown.pnx
06-escape-unknown.pnx:2:4: unknown escape; a raw string r"..." keeps backslashes as written

(8) A \u or \U that writes a value that is not a code point (surrogates D800–DFFF, values above 10FFFF) is refused.

use io
"\uD800" io.println

$ pnx 06-escape-surrogate.pnx
06-escape-surrogate.pnx:2:2: the escape is not a code point

(9) The only prefix of a string is r: u"…" U"…" R"…" are refused.

use io
u"wide" io.println

$ pnx 06-string-prefix.pnx
06-string-prefix.pnx:2:1: the only prefix of a string is r (raw string)

(10) A character literal holds one character (or one escape).

'ab' drop

$ pnx 06-char-two.pnx
06-char-two.pnx:1:3: a character literal holds one character

6.6 Reserved words and punctuation

Syntax

punct    = "(" | ")" | "," | "." ;              (* a lone dot ends a statement, (2) *)
token    = reserved | NAME | PATH | INT | CHAR | STRING | RAWSTRING | TEXT | BOOL | punct ;
reserved = "rem" | "note" | "text" | "true" | "false" | "pinax"
         | "use" | "as" | "export"
         | "word" | "global" | "const" | "array" | "list" | "pool" | "queue" | "buffer"
         | "tasks" | "record" | "type" | "extern" | "from" | "forward" | "index" | "bound" | "shared"
         | "make" | "test" | "row" | "assign" | "big_endian" | "columns"
         | "gives" | "or" | "fail" | "owned"
         | "to" | "do" | "end" | "if" | "else" | "loop" | "while" | "until" | "break"
         | "case" | "of" | "otherwise" | "return" | "or_fail" | "or_return" | "or_else" | "cat"
         | "defer" | "set" | "fits" | "search" ;

(1) The list of reserved words is closed. A reserved word cannot be a name.

(2) or is a reserved word only inside a signature (or fail). In the body of a word or is a core word (logical or).

(3) The only punctuation is ( ) , and the dot that ends a statement (6.2 (2)). Each is a token of its own even when written against other text: str(32) is four tokens, 7 sq. three.

(4) The names of types (int str bool …) and of core words (add dup …) are not reserved words but names declared in advance. Declaring them again is refused by the rule against hiding names (9.5).

(5) make is reserved for a future feature and refused when used (annex D).

6.7 Format version

Syntax

source = [ "pinax" INT ] { item } ;

(1) Writing pinax and an integer as the first tokens of a source states the format version the source follows. This document defines version 2. Without it, a source is read as version 2.

pinax 2
use io
"written for format version 2" io.println

$ pnx 06-version.pnx
written for format version 2

Constraints

(2) A source that states a version the processor does not read is refused.

pinax 1
use io
"never read" io.println

$ pnx 06-version-other.pnx
06-version-other.pnx:1:7: this source is written for another format version; this build reads version 2

Diagnostics