warm void
Hi, I've dived into the whole coding paradigm on Saturday, so today marks the end of my first week not only with Rust, but programming. 🎉
I've made a miniature lexical analyser with a few pointers from my school buddy (he's like a grandmaster at coding...).
If anyone could tell me what I did wrong, could do better, then it would be greatly appreciated.
Here is the repository file that I've worked on for a few days: https://github.com/daquav1s/sapphire/blob/main/src/lexer.rs
tidal sphinx
Read this https://matklad.github.io/2024/06/04/regular-recursive-restricted.html (That's the guy who build rust-analyzer, he also has other relevant things, search for them too)
warm void
Read this https://matklad.github.io/2024/06/04/regular-recursive-restricted.html...
What am I supposed to get out of it?
tidal sphinx
Take a look at (18 / 2) section - you're tokenizing it linearly as if brackets are operators. You want to tokenize into a tree, otherwise there is hardly a way to notate relationships.
Same thing with 132 + 3 * (18 / 2), you can very well break it down to a flat vec of components, but to reason about ordering of + and * operations you would still need to transpose it into a tree structure with 132 and 3 * (18 / 2) components being at the same level, while you split Times(3, (18/2)) down from there
Lexical tokenizer is an incredbily difficult thing to start with, props for attempting it
warm void
Take a look at `(18 / 2)` section - you're tokenizing it linearly as if brackets...
Yeah, maybe adding a few more types would be handy. What do you mean by tokenizing it into a tree? Like for the example 10 - 3 * (1 + 8) + 12 What would I try to get the output of tokenize() looking like? Thanks for taking the time to write this out btw, I really appreciate it.
tidal sphinx
I'm no expert in tokenization, knowledge caps with articles I read, but would imagine this normally ends up structured like this
+
/ \
- 12
/ \
10 *
/ \
3 ()
|
+
/ \
1 8
So you'd have (as ungrammar)
Number = 'value'
Expr = Number
| ExprParen
| ExprBinary
ExprParen = '(' Expr ')'
ExprBinary = Expr ('+' | '-' | '*' | '/') Expr
And then you just build a tree with it and are happy
kind storm
@tidal sphinx You're describing parsing, not lexing. A lexical analyzer / tokenizer specifically doesn't build a syntax tree; it produces a sequence of tokens which the parser can then turn into a tree.
tidal sphinx
oh well ok, I'm dumb. That would be logical next step though, right?
kind storm
Yes.