#Writing a parser, struggling with ownership

6 messages · Page 1 of 1 (latest)

surreal idol
#

the error

error[E0271]: expected `Rev<Iter<'_, Token>>` to be an iterator that yields `Token`, but it yields `&Token`
  --> src/commands/eval/parse.rs:55:40
   |
55 |         let (tree, _) = ParseTree::new(&mut tokens).expect("Failed to parse input");
   |                         -------------- ^^^^^^^^^^^ expected `Token`, found `&Token`
   |                         |
   |                         required by a bound introduced by this call
   |
note: required by a bound in `parse::ParseTree::new`
  --> src/commands/eval/parse.rs:29:28
   |
29 |     pub fn new<I: Iterator<Item = Token>>(tokens: &mut I) -> Result<(Self, usize), ParseError> {
   |                            ^^^^^^^^^^^^ required by this bound in `ParseTree::new`
#

the code

impl ParseTree {
    pub fn new<I: Iterator<Item = Token>>(tokens: &mut I) -> Result<Self, ParseError> {
        if let Some(token) = tokens.next() {
            match token {
                Token::Scalar(_) | Token::Identifier(_) => Ok(Self::Leaf(token)),
                Token::Operator(op) => {
                    let left = ParseTree::new(tokens)?;
                    let right = ParseTree::new(tokens)?;

                    Ok(Self::Branch(op, Box::new(left), Box::new(right)))
                }
            }
        } else {
            Err(ParseError::EmptyInput)
        }
    }
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn parse_tree() {
        let tokens = Token::tokenize("1 2 3 + *").expect("Failed to tokenize input");
        let mut tokens = tokens.iter().rev();

        let tree = ParseTree::new(&mut tokens).expect("Failed to parse input");

        println!("{tree:?}");
    }
}

// elsewhere

impl Token {
    /* ... */
    pub fn tokenize(s: &str) -> Result<Vec<Self>, Box<dyn Error>> {
        s.split_whitespace().map(Token::from_str).collect()
    }
}
#

so in my mind, there should be no reason for tokens to get passed as an Iterator with Items as &Tokens but for some reason it is. i want ParseTree::new to be able to eat the tokens it uses, so that way the function calling it doesnt have to worry about that, so thats the rationale behind using &mut I.

grave verge
#

you used tokens.iter() which (assuming tokens is a Vec or similar) returns references to the items

#

use tokens.into_iter() instead

wooden fiber
#

beginner question alert This may be way off, but, in that instance, could you possibly use drain so that you have the chance to use the same memory if there's an error and the parser has to exit but keep track of state?