#How to use `logos` to tokenize template literals?

4 messages · Page 1 of 1 (latest)

light birch
#

Hello, does anyone know if it's possible to use logos to tokenize template literals such as this:

`this {true} hey {false}`

[
  TemplateStart,
  TemplateInterp { start: curly_brace_offset_from_source },
  True, // this would be the regular tokenizers
  TemplateInterp { start: curly_brace_offset_from_source },
  False,
  TemplateEnd { start: offset_from_last_interpolation_close_brace },
]
woven knot
#

with logos it's easier to lex into

[
  Edge,
  Literal,
  Interp,
  Literal,
  Interp,
  Edge,
}
```Use `spanned` to get access to `(Token, Span)` pairs. If we include those, your lexer output will look something like this
```rs
[
  (Edge, 0..1),
  (Literal, 1..6),
  (Interp, 6..12),
  (Literal, 12..17),
  (Interp, 17..23),
  (Edge, 23..24),
]
```To get access to the actual string data, you can take slices of the source: `source[span]`
# 
#[derive(Logos)]
enum TokenKind {
  #[token("`")]
  TemplateEdge,
  #[regex(r"[^{}`]+")]
  TemplateLiteral,
  #[regex(r"{[^{}`]*}")]
  TemplateInterp,
}

struct Token {
  kind: TokenKind,
  span: Span,
}
light birch
#

ah, thank you