tokenSlice
Function parameters
Parameters
Index into `tokens`, or null.
Types
- OptionalTokenIndex
- Index into `tokens`, or null.
- TokenOffset
- A relative token index.
- OptionalTokenOffset
- A relative token index, or null.
- full
- Fully assembled AST node information.
- ExtraIndex
- Index into `extra_data`.
Functions in this namespace
Functions
- parse
- Result should be freed with tree.deinit() when there are
- renderAlloc
- `gpa` is used for allocating the resulting formatted source code.
- errorOffset
- Returns an extra offset for column and byte offset of errors that
- legacyAsm
- To be deleted after 0.15.0 is tagged
Source
Implementation
pub fn tokenSlice(tree: Ast, token_index: TokenIndex) []const u8 {
const token_tag = tree.tokenTag(token_index);
// Many tokens can be determined entirely by their tag.
if (token_tag.lexeme()) |lexeme| {
return lexeme;
}
// For some tokens, re-tokenization is needed to find the end.
var tokenizer: std.zig.Tokenizer = .{
.buffer = tree.source,
.index = tree.tokenStart(token_index),
};
const token = tokenizer.next();
assert(token.tag == token_tag);
return tree.source[token.loc.start..token.loc.end];
}