pub fn tokenize(source: &str) -> Result<Vec<Token>, CompileError>Expand description
Tokenise a source string. Returns the full token vector or the first lexical error.
Doc blocks (--- … ---) and line comments (-- ...) are recognised
outside the logos-generated lexer: we scan the source one segment at a
time, dispatching to logos for ordinary tokens between non-token spans.