#Performance issue with ZFS?

1 messages · Page 1 of 1 (latest)

brittle wolf
#

Hi there,
Is there some known performance related to ZFS?
I'm using NixOS with ZFS, and every dagger call takes a long time, on first and subsequent runs:

❯ dagger call container-echo --string-arg foo stdout

✔ connect 0.7s
✔ initialize 0.9s
✔ prepare 0.0s
✔ testDagger: TestDagger! 0.0s
✔ TestDagger.containerEcho(stringArg: "foo"): Container! 8.2s
✔ Container.stdout: String! 0.0s

foo

Checking the trace, all that time is spend in the exec /runtime step
I tried with both the zfs and overlay2 storage drivers in docker but the result is the same (I think overlay2 might be faster, but not by much)
Now, in a VM, with both NixOS and Debian, when not using ZFS I the second call is always in the 0.xs range

Screenshot_20240827_181116.png
shut anvil
#

I wonder how other buildkit-based tool compare in terms of performance. Have you tested some docker build commands with ZFS and without by any chance?

#

Dagger relies on buildkit which is very disk intensive. In our pipelines the most limiting factor is usually disk. I haven't played with ZFS much, but maybe copy-on-write could hurt with buildkit? If thats the case then docker build would also be slower.

Maybe the ZFS storage driver is smart about this though, I'm really talking without knowing much, sorry 👼

brittle wolf
#

I've used Earthly to build some containers on this machine and it's been pretty fast, I'll try do do some comparisons inside the VM
Something else I can try is setup an zvol formatted with ext and use that as my docker storage, to rule out any other issue with the machine
As for docker build, performance seems identical inside and outside the VMs

brittle wolf
#

Update: using a zvol formatted with ext4 , performance is great now

twin hedge
#

maybe @vocal junco or @waxen flax have some deeper knowledge about the zfs performance in buildkit. Silently pinging them

waxen flax
#

hm, yes, there are known to be some buildkit weirdness things around zfs - usually in the form of weird difficult to debug errors, but potentially there could be some perf issues as well