Function tokenize 

pub fn tokenize(source: &str) -> Result<Vec<Token>, CompileError>

Tokenise a source string. Returns the full token vector or the first lexical error.

Doc blocks (--- … ---) and line comments (-- ...) are recognised outside the logos-generated lexer: we scan the source one segment at a time, dispatching to logos for ordinary tokens between non-token spans.