#Something like `par_iter(..).try_map`?

1 messages · Page 1 of 1 (latest)

wild ledge
#

I got some code that grabs an item which is them added to a vec.
The important part isn't that it adds them, but that i can gather all the items up in the end.

Because the item is a Result<Option<..>> i need to check that, as i only want the Ok(Some(..)).

Currently i have two ways i have tried that works.
The first is is slower but reacts to errors faster, the other is faster but doesn't check the states until afterwards.
The ideal would be to get the both of both worlds, but not sure if it's possible.
I have done it in a hacky way with match and filter_map chaining but it ends up slower, not surprising.

Here's basically how the code looks:

    // example 1 (gather and check)
    let mv = std::sync::Mutex::new(&mut folder.children);
    sub_paths
        .par_iter()
        .map(|f| folder.path.join(f))
        .try_for_each(|c| -> Result<()> {
            if let Some(f) = get_folder(c, supports_long_path)? {
                mv.lock()
                    .map_err(|e| {
                        custom_windows_error(
                            ERROR_LOCK_FAILED,
                            &format!("mutex lock failed: {}", e),
                        )
                    })?
                    .push(f);
            }
            Ok(())
        })?;

    // example 2 (gather in bulk, check in bulk )
    for l in sub_paths
        .par_iter()
        .map(|f| get_folder(folder.path.join(f), supports_long_path))
        .collect::<Vec<Result<Option<...>>>>()
    {
        if let Some(f) = l? {
            folder.children.push(f);
        }
    }

Or perhaps there's some other way to tackle this that i didn't think of as well.
Please give it a look and throw any suggestions at me:)

mortal verge
# wild ledge I got some code that grabs an item which is them added to a `vec`. The important...

try_map is fundamentally impossible on both Iterator and ParallelIterator. You'd need something like the fallible_iterator crate's FallibleIterator trait, which defines next to return Option<Result<Self::Item, Self::Error>>

Since it seems you want to discard None and propagate Err, you could do the following:

.par_iter()
.filter_map(|f| get_folder(...).transpose()) //transpose: Swap Result<Option<T>, E> with Option<Result<T, E>>
.collect::<Result<Vec<_>, _>()?; //an iterator of results (or options) can be collected into a result (or option) of a collection
wild ledge
#

does map not allow something to be thrown and that's the issue?
i will try that out, never used transpose before

#

oh transpose is awesome, was that i did with match and filter_map etc. Had no idea there was a function that swapped them so easily. Thanks:)

azure mauve
#

does map not allow something to be thrown and that's the issue?
no, it's that iterators are lazy
you can't know whether the entire iteration failed somewhere until you ask for the last item

#

(or for parallel iteration, until all the items have been processed — which makes a sequential bottleneck)

mortal verge
#

Or in other words, to know if something failed, first you need to process all the things

#

But map does not process all the things eagerly, so it can't know if anything failed.

#

(This is also why collect can do that)

mortal verge
#

@wild ledge

wild ledge
#

sorry for the late response, not sure how to actually keep track of my questions in discord unless i happen to have it open all the time, so i lost it a bit.

Ah is being lazy how Map is defined, compared to say ForEach (or perhaps that's just what an Iterator is, and ForEach isn't one?).
It makes sense that it doesn't work if that's how it behaves though, Results loses their meaning in htat regard and it would behave more or less like a Collect -> Try.

mortal verge
#

for_each just consumes the iterator, it doesn't return an adapter