pub fn tokenize_expanding_holes(
source: &str,
) -> Result<Vec<Token>, CompileError>Expand description
Like tokenize, but with every interpolated-string token replaced by the
tokens of its holes — each hole’s bytes re-lexed and its token spans rebased
to absolute source positions (the same rebase crate::parser applies when
parsing a hole), recursing through nested interpolation. Chunk (literal) text
between holes yields no tokens.
An interpolated string lexes to a single opaque InterpStr token, so the
LSP’s token-based cursor resolution (hover, go-to-definition, references,
semantic tokens) is otherwise blind to identifiers inside "… \(name) …".
Expanding the holes makes those identifiers visible as ordinary Ident
tokens with their real spans. (Issue #473.)
On a malformed interpolation (an InterpStr whose holes don’t split, or a
hole whose bytes don’t re-lex) the offending token is kept opaque rather than
dropped, so resolution degrades to the pre-fix behaviour instead of losing
tokens.