the relevant part is in the client-side

[unique tree]

changing the send.go semaphore didn't seem to make any difference. Just to double check, do I need to only recompile the client when changing that? or does it also affect the server?

[steel ether]

I thought so but let me double check

Yeah for increasing the size of the send pipeline that's all you should need to change. What did you increase it to out of curiousity?

Other random thoughts:

1. There's technically another limit of only queueing up 128 paths to diff at a time https://github.com/sipsma/dagger/blob/c9df98d956f46faf835b82f692eac98a90933347/engine/filesync/diff.go#L50-L50, sorta doubt that's a bottleneck but maybe? That one is server side
   • another related limit here https://github.com/sipsma/dagger/blob/c9df98d956f46faf835b82f692eac98a90933347/engine/filesync/remotefs.go#L79-L79
2. grpc stream behavior is potentially relevant here. I don't remember what limits or behavior, if any, apply here. Might be worth a look
3. I guess worst case you can definitely figure out the bottleneck eventually by a combination of adding spans and maybe looking at pprof to see if there's any obvious bottlenecks in userspace code
   • if you run the engine w/ --debugaddr=0.0.0.0:6060 pprof is available on that port, these are all the endpoints https://github.com/sipsma/dagger/blob/c9df98d956f46faf835b82f692eac98a90933347/cmd/engine/debug.go#L34-L34

[unique tree]

Yeah for increasing the size of the send pipeline that's all you should need to change. What did you increase it to out of curiousity?

that 4 to 10 basically which is hardcoded in send.go

[steel ether]

does anything happen if you increase it to something way way higher like 1024?

for the "thousands of little file cases" going from 4->10 might not move the needle enough

[unique tree]

checking