pub fn tokenize(source: &str) -> Result<Vec<Token>, CompileError>Expand description
Tokenise a source string with no real file identity — every span’s
FileId defaults to FileId::UNKNOWN. See tokenize_in for the
real-identity entry point production callers use.