whats more efficient?

[final iris]

Is it more efficient to have a for loop that loops 100 times creating a new stream of like 300-600 items each time. or is it more efficient to do some sort of flatmap to just create one very big stream (i think either way using a parallel stream is most efficient?) I need what runs the fastest.

[jagged mortar]

<@&987246399047479336> please have a look, thanks.

Here is an AI assisted attempt to answer your question 🤖. Maybe it helps! In any case, a human is on the way 👍. To continue talking to the AI, you can use </chatgpt:1108714622413963314>.

TJ-Bot

Is it more efficient to have a for loop that loops 100 times creating a new stream of like 300-600 items each time. or is it more efficient to do some sort of flatmap to just create one very big stream (i think either way using a parallel stream is most ef

It is generally more efficient to use flatMap to create one big stream instead of creating multiple smaller streams in a loop. This is because creating and combining multiple streams can be more resource-intensive compared to just creating one large stream. FlatMap also helps in maintaining code readability and simplicity.

Useful links:

• Java Stream: flatMap()
• When to use flatMap() and map() in Java 8
• The difference between map() and flatMap() in Java 8

- AI generated response

[final iris]

im not sure from what number of items parallel stream is better than a regular stream

[nocturne raven]

Make an Iterable

[white hamlet]

Uh wut?

I don’t see how that answers the question

[final iris]

iterable?

i wanted to use streams

which i think using parallel is just faster than a loop anyways

but its also easier to work with for this

though if a loop is much faster ill switch to that

[nocturne raven]

Really do a jmh to check the difference.

Anything else is just talk.

[final iris]

jmh?

[real tendon]

one big stream

I've done streams over a billion elements (those I did parallel), and it works just fine, and it was pretty fast as well

[white hamlet]

microbenchmarking library for java

but not sure if there is a need here

these numbers are still pretty small

[final iris]

60k is small?

[final iris]

ah

[white hamlet]

Well depends what kind of computations you are doing but generally yes

[nocturne raven]

60k...

dont even think about it.

[final iris]

oh so i shouldnt be using a parallel stream?

[nocturne raven]

Well, it depends, do you even see a difference?

[final iris]

probably not

havent tried with a normal stream

[rain path]

Parallel stream is probably significantly better above 400k-ish mark

[real tendon]

my parallelstream read multiple files in parallel while parsing them

So that's why I did it

    public void process(Consumer<Stream<String>> consumer) {
        int processors = Runtime.getRuntime().availableProcessors() / 4;
        ForkJoinPool forkJoinPool = new ForkJoinPool(processors);
        try {
            forkJoinPool.submit(() -> consumer
                .accept(sourceFiles.stream()
                    .parallel()
                    .flatMap(FileProcessor::readFile)
                )
            ).get();
        } catch (ExecutionException | InterruptedException e) {
            throw new RuntimeException(e);
        } finally {
            forkJoinPool.shutdown();
        }
    }

I also needed to limit the amount of cpus etc