empty topaz
This is the testing loop from my demo, it seems like that extra time per test might be clearer on v0.8.2. It looks like the same FS merge is repeated each time, despite syncing the image between that point and all of the micro-containers for tests
https://github.com/hofstadter-io/hof/blob/bug-fixes/test/dagger/main/hof.go#L89
Any help here optimizing would be great! My testing turn around time is painful 🤕
It also makes my machine go brrrrrrr and wooooosh, feels like a lot of overhead compared to running the same test suite outside of dagger
Here you can see 3 times:
I would expect that a single text file should take much less time
Here's where the tests are taken from the source, looped over, and added individually to the testing container: https://github.com/hofstadter-io/hof/blob/_dev/test/dagger/testscripts.go#L47
young coral
Is it any better if you use WithMountedFile instead of WithFile?
empty topaz
seems so!
weak mango
seems so!
Using WithMountedFile is always going to be a faster option when you don't need the files in the actual rootfs, so if that works then problem solved, but I am curious why the merge is taking so long. Provided it's not a progress output problem, merge-op is generally very fast since it just hardlinks files rather than copying them.
The only bottlneck is number of files/dirs, but in tests I ran a while back I merged 10k files in ~5 seconds, so I'm curious if that /work dir has a truly enormous number of files, like 30k+? If so, there's some low-hanging fruit for optimizing merge-op more for those cases.
Otherwise, another possibility would be that your dagger engine isn't able to overlayfs for some reason. Is this running against the default engine (the one that starts in docker)? Or is it one of the dagger-in-dagger engines you have been using?
empty topaz
it could very well be overlay related, standard engine, but there is a warning, let me go get it
(I think it's because cgroupv1 (I'm on a chromebook and then have been locking down the kernel more and more)
empty topaz
This is the error I'm thinking of (Docker Daemon as a ServiceContainer & Mount)
But I suspect that first line might point at my system not having the necessary dependencies for better performance.
I can try on a cloud vm later, see if it runs better or with fewer warnings there
woeful reef
But I suspect that first line might point at my system not having the necessary ...
seems like there's the culpirit