Module lexer 

Lexer for Bynk v0.

Token kinds correspond to the terminals defined in the grammar (spec §3 and §4). Whitespace is skipped; line comments are emitted as Comment tokens so the formatter can preserve them through round-trips (v1.1 LSP spec §3.5). Doc blocks (---) are emitted as DocBlock tokens, lexed outside of logos (see tokenize).

Structs

Token

A token plus its source span.

Enums

TokenKind

Token kinds. Discriminants without payload data; the lexeme is recovered from the source string via the token’s Span.

Functions

comment_body

Extract the body of a Comment trivia token: everything after the leading -- marker, preserving its inline whitespace verbatim. Used by the parser when attaching comments to declarations.

doc_block_body_range

The doc-block body as a byte range into source — leading/trailing --- marker lines stripped, no further processing (unlike doc_block_content, which additionally strips a common per-line indent — not offset-preserving). Callers that need to map a position in the body back to source (e.g. document-link spans) use this instead of re-deriving it from the string doc_block_content returns.

doc_block_content

Extract the body content of a doc-block token from its source span. Strips the leading and trailing --- marker lines and returns the body verbatim. If every non-empty content line begins with the same horizontal whitespace prefix (e.g., because the doc block sits inside a brace-form commons body), that common prefix is removed so the body reads naturally when emitted as JSDoc.

has_blank_line_between

Returns true if there is a blank line (a line containing only whitespace) in source strictly between byte offsets from (inclusive) and to (exclusive). Used by the parser to detect orphan doc blocks.

tokenize

Tokenise a source string with no real file identity — every span’s FileId defaults to FileId::UNKNOWN. See tokenize_in for the real-identity entry point production callers use.

tokenize_expanding_holes

Like tokenize, but with every interpolated-string token replaced by the tokens of its holes — each hole’s bytes re-lexed and its token spans rebased to absolute source positions (the same rebase crate::parser applies when parsing a hole), recursing through nested interpolation. Chunk (literal) text between holes yields no tokens.

tokenize_expanding_holes_in

Like tokenize_expanding_holes, but stamping every token’s span (including rebased hole tokens) with file.

tokenize_in

Tokenise a source string, stamping every token’s span with file. Returns the full token vector or the first lexical error.