#Lexer (internal/lexer)

A hand-written, single-pass scanner.

#Token types

def introduces a variable or constant binding (TOKEN_DEFINE); func introduces a method (TOKEN_FUNC). import (TOKEN_IMPORT) is for file imports (import "path.j";); use (TOKEN_USE) is for library imports (use io;). DOT (.) no longer appears in import syntax (paths are strings now) and is reserved for future expression use. Comparison tokens LE, GE, EQ, NEQ are two-character (<=, >=, ==, !=) and are recognized by a one-character lookahead from <, >, =, !. ! exists only as the lead of != (logical negation is the word not); a bare ! is a positioned lex error whose message points at both not and !=. RETURN is the keyword behind return [EXPR]; (see grammar.md).

VARREF carries the variable name without the leading $. STRING carries the value without surrounding quotes. A double-quoted literal is cooked - escape sequences are already processed: \n \r \t \\ \" \' \0 \{ \}, plus the Unicode escapes \uXXXX (exactly 4 hex digits, the BMP) and \UXXXXXXXX (exactly 8 hex digits, any plane). readUnicodeEscape rejects a surrogate, a value above U+10FFFF, and a short / non-hex digit run as a positioned lex error (matching convert.fromCodepoint). A single-quoted literal is raw - its content is verbatim (no escape processing, no interpolation) from the opening ' to the next '. The two delimiters are thus not interchangeable; readString branches on the quote into readRawString or readCookedString.

A cooked string also interpolates: an unescaped {expr} is a slot. readCookedString decomposes the literal into ordered literal chunks and expression slots. With no slots it returns a plain STRING (the decoded value); with one or more it returns STRING_INTERP, whose Parts []StringPart carry the literal chunks and the raw expression source of each slot plus that slot's absolute Line / Col (so the parser's sub-lex / sub-parse annotates references and reports errors at the real location - TokenizeAt seeds the position). scanInterpSlot brace-counts nested {} (a map literal in a slot balances) and copies a nested string literal verbatim (copyNestedString) so a brace or quote inside it never disturbs the count; \{ / \} are literal braces and a bare unescaped } is a positioned lex error. A raw '...' string is returned verbatim as a plain STRING, never interpolated.

#Whitespace handling

Spaces, tabs and newlines are discarded between tokens; they only ever advance Line / Col for position tracking. There is no indentation-significant mode and no off-side rule. The user-facing consequence is documented in user-guide/syntax.md > Tokens and whitespace; the rule is load-bearing for jennifer fmt, which trusts that re-emitting the token stream with canonical spacing produces a semantically identical program.

The only place whitespace is retained is inside string literals - readString reads byte-by-byte until the closing quote, so a literal space, tab, or even a raw \n between the quotes becomes part of the string value (in both the cooked "..." and the raw '...' form). In a raw single-quoted literal a backslash is just a byte, so '\n' is the two-character string \n, and a multi-line block needs no escapes at all; fmt re-emits every literal in its original delimiter and spelling (via Token.Raw).

Comments and blank lines are emitted as trivia tokens (TOKEN_COMMENT_LINE, TOKEN_COMMENT_BLOCK, TOKEN_COMMENT_SHEBANG, TOKEN_BLANK_LINE) so jennifer fmt can round-trip them. The preprocessor and parser strip these tokens at entry; the formatter walks the raw lexer stream. See Comments below.

#Position tracking

Every token records Line and Col (both 1-based) and File (the absolute path supplied to TokenizeWithFile, or "" for unattributed input). The advance() helper bumps line on \n and otherwise bumps col. File flows from the token to the AST node (every node embeds pos{File, Line, Col}), so errors raised inside an imported .j still point at the imported file - see Interpreter > Errors and positions.

#Keywords

The lexer's keyword map covers: def func as init const import use return if elseif else while for true false null and or not int float string bool. Anything else lexed as a word stays a TOKEN_IDENT. define is not a keyword and lexes as a plain identifier. div was removed when // took over floor division.

#Comments

# ... runs to end of line and emits TOKEN_COMMENT_LINE; the special case of #! on line 1 col 1 emits TOKEN_COMMENT_SHEBANG instead so the formatter can re-emit the shebang verbatim at the file head. /* ... */ emits TOKEN_COMMENT_BLOCK and nests via a depth counter (increment on /*, decrement on */, exit at depth 0). Unterminated nested comments error positionally at the outermost /* so the message points at where the user meant to start.

Each comment token's Lexeme carries the verbatim source text including the delimiters (# ..., /* ... */, #! ...) so the formatter round-trips byte-for-byte.

Runs of blank lines collapse to one TOKEN_BLANK_LINE per run - matching the style rule "never more than one consecutive blank line".

# was chosen (over the C/Java // style) so the floor-division operator // is unambiguous and a Jennifer file can begin with a Unix shebang (#!/usr/bin/env -S jennifer run).

A . peeks for a second .: two together lex as one DOTDOT (the half-open range operator a..b), a single . stays DOT. Number scanning already requires a digit after . for a fraction, so 1..5 lexes as INT DOTDOT INT (not 1. .5) with no special-casing.

A decimal number may carry a scientific-notation exponent, [eE][+-]? followed by one or more digits (6.022e23, 1.6e-19, 1e10). An exponent makes the literal a FLOAT even with no fractional part, so 1e10 is a float where a bare 1 is an int; the exponent takes no _ separators (the mantissa still does, 1_000.5e3), and a digit followed by e/E with no exponent digit (1e, 1e+) is a positioned lex error, since it has no other valid reading. Only the decimal path scans an exponent - 0x/0o/0b literals return earlier, so 0xe5's e stays a hex digit. The stored Lexeme keeps the exponent's source case (2.5E8 stays 2.5E8, which strconv.ParseFloat accepts) and strips only the mantissa's _; Raw keeps the exact source so fmt re-emits 6.022e23 / 1.6E-19 verbatim.

The parser's strconv.ParseFloat governs the magnitude boundaries. An overflowing exponent (1e400) is a positioned parse error, never an Infinity (the strict stance rejects non-finite results). An underflowing one (1e-400, below the smallest denormal ~5e-324) rounds to 0.0 with no error - 0.0 is a finite, correctly-rounded value, unlike the non-finite Infinity overflow would give, so it is the same value every IEEE-754 language yields there. The asymmetry is deliberate: the strict check bans the non-finite, not a finite zero.

#Identifier rule

Every identifier is letter-initial (a token that starts with a digit is always a number, which keeps lexing unambiguous). Variable, method, parameter and library names are [A-Za-z][A-Za-z0-9]{0,63} - a letter, then letters and digits, no underscores. Constants use a looser form: uppercase chunks separated by single _ characters, with in-chunk digits - [A-Z][A-Z0-9]*(_[A-Z][A-Z0-9]*)*. Each chunk starts with an uppercase letter (never a digit), and every _ must be immediately followed by that letter, so leading, trailing, consecutive, and underscore-then-digit forms are all rejected (SHA256 / SCRAM_SHA256 are legal, AES_256 is not - write AES256).

The lexer reflects this by accepting digits and _ as continuation characters for bare IDENT tokens (so sha256 and MAX_RETRIES are each a single token) but rejecting any identifier that ends with _. The full per-kind rule is then enforced by the parser at each def / use site - variables, methods, parameters, library names and call callees may not contain _; constants may, with the leading-_ case already excluded by isIdentStart. $var references go through a separate lexer path (readVarRef) whose isIdentPart allows the leading letter plus digits but not _, so $x2 lexes and $foo_bar lex-errors directly.

#Resource limit: the token budget

TokenizeWithFile caps how many tokens a single source file may produce at limits.MaxTokens (build-tag split, like the parser's nesting cap: 1<<22 on the default binary, 1<<18 on jennifer-tiny). Past the cap it returns a positioned, catchable LexError ("token budget exceeded"). This bounds the front of the pipeline against a token bomb: a small, dense file (a few MB of 1+1+1+1;) amplifies to millions of ~72-byte Token structs - roughly 100x the input byte count - which the parser then walks again. The over-long-identifier early-exit (a single lexeme is capped at 64 chars) bounds one token; this bounds the token stream.

It complements the preprocessor's spliced-token budget (maxSplicedTokens), which counts tokens across include splices but does not count the entry file's own tokens - so a giant root program with no includes would otherwise reach the parser unbounded. The cap is a package var defaulting to the const only so a test can lower it without generating millions of tokens; production code never writes it.