Module lexer 

Lexer for Bynk v0.

Token kinds correspond to the terminals defined in the grammar (spec §3 and §4). Whitespace is skipped; line comments are emitted as Comment tokens so the formatter can preserve them through round-trips (v1.1 LSP spec §3.5). Doc blocks (---) are emitted as DocBlock tokens, lexed outside of logos (see tokenize).

Structs

Token

A token plus its source span.

Enums

TokenKind

Token kinds. Discriminants without payload data; the lexeme is recovered from the source string via the token’s Span.

Functions

comment_body

Extract the body of a Comment trivia token: everything after the leading -- marker, preserving its inline whitespace verbatim. Used by the parser when attaching comments to declarations.

doc_block_content

Extract the body content of a doc-block token from its source span. Strips the leading and trailing --- marker lines and returns the body verbatim. If every non-empty content line begins with the same horizontal whitespace prefix (e.g., because the doc block sits inside a brace-form commons body), that common prefix is removed so the body reads naturally when emitted as JSDoc.

has_blank_line_between

Returns true if there is a blank line (a line containing only whitespace) in source strictly between byte offsets from (inclusive) and to (exclusive). Used by the parser to detect orphan doc blocks.

tokenize

Tokenise a source string. Returns the full token vector or the first lexical error.