Is it possible to create a slice from an iterator? | Rust Programming Language Community | Page 1

cobalt thicket May 13, 2024, 6:31 PM

#

this is not a part of the Iterator trait

#

you could make it a method on your iterator, like https://doc.rust-lang.org/std/slice/struct.Iter.html#method.as_slice does

Iter in std::slice - Rust

Immutable slice iterator

#

Or https://doc.rust-lang.org/std/str/struct.Chars.html#method.as_str in your case

Chars in std::str - Rust

An iterator over the chars of a string slice.

trail citrus May 13, 2024, 7:56 PM

#

Note that to do this you have to avoid using a dyn Iterator. You probably only ever use chars' iterator, so you could write the type as Peekable<Chars<'a>>.

Now of course, Peekable does not give you access to its wrapped iterator, including & access, so you can't calls Chars::as_str anyway. Fortunately, reimplementing Peekable is very easy:

struct Lexer<'a> {
  source: Chars<'a>,
  peeked: Option<Option<char>> 
  // The outer option represents whether we have a peeked item, the inner is whether that item is the end of the iterator
  // None => We have no currently peeked character
  // Some(None) => We have peeked that there is no next character
  // Some(Some(c)) => We have peeked that `c` is the next character
}
impl<'a> Lexer<'a> {
  //source code for both these methods copied and slightly adapted from std
  fn peek(&mut self) -> Option<char> { 
    *self.peeked.get_or_insert_with(|| self.source.next())
  }
  //this would normally be an iterator impl, I'm putting it here for simplicity
  fn next(&mut self) -> Option<char> {
    match self.peeked.take() {
        Some(v) => v,
        None => self.source.next(),
    }
  }
}

wraith thunder May 13, 2024, 8:53 PM

#

FYI, in general, adding Box, or adding dyn, will never solve a lifetime problem.
Replacing &'a mut dyn Iterator with Box<dyn Iterator> could, though.

final meteor May 13, 2024, 9:09 PM

#

also from what I can tell, writing lexers not using an iterator is more common

#

it of course depends on the language, but working directly with the underlying bytes is far easier

#

what I do at least is have a fast path for ascii, and then have a slow path for anything that's not ascii, which tends to only matter in identifiers

#

writing lexers is a very specialized thing that doesn't map that well over what std provides

#

forcing people to use code points is great, but not for this

#

because the common case is that it's mostly ascii

#

and you can usually isolate unicode's complexities in a small piece of code away from the main thing

clear crescent May 13, 2024, 11:34 PM

#

personal recommendation: consider sticking with a concrete type that you know will work, before figuring out what kind of trait you need to define.
For example, in your case, you could use a &mut &[char] or a &mut &str

#

or define your own trait as an abstraction on that, which you can extend as needed. Once you know more precisely which features you need of the abstraction, it becomes much easier to check the std library for relevant stuff

#Is it possible to create a slice from an iterator?