pub fn tokenize_in(
source: &str,
file: FileId,
) -> Result<Vec<Token>, CompileError>Expand description
Tokenise a source string, stamping every token’s span with file.
Returns the full token vector or the first lexical error.
Doc blocks (--- … ---) and line comments (-- ...) are recognised
outside the logos-generated lexer: we scan the source one segment at a
time, dispatching to logos for ordinary tokens between non-token spans.