Help: WithExec layers not being cached

[candid tide]

Hi daggernauts,

As usual, thanks for the amazing work.

Im using python SDK and since I updated to 0.11.4 my with_exec commands are not being cached.
Maybe I am missing some new config...

The command in question is:

pip install git+https://[e-mail]/spi3pl/lib/core-lib.git@0.15.0#egg=core

and the dagger call is:

        for dep in sorted_dependencies:
            if dep.dependency.connection_id:
                if dep.dependency.type == 'pip':
                    # add version to pip install command
                    dep.dependency.command[2] = \
                        f'{dep.dependency.command[2]}@{dep.dependency.version}#egg={dep.dependency.name}'
            container = container.with_exec(dep.dependency.command)

And also i realized im now a "classic" dagger user... things move fast in the daggerverse...

Thanks in advance!

It used to be cached beautifully by buildkit 💔

[opaque jewel]

Hey @candid tide! It should work, we are not aware of any caching issues for with exec calls.

Would it be possible to look at the values of dep.dependency.command for each with_exec call? Maybe due to the conditions or ordering the values are changing which is causing the cache to bust?

[candid tide]

I tried it several times, the command is being built correctly, otherwise it would fail, the constructed command is:

pip install git+https://[e-mail]/spi3pl/lib/core-lib.git@0.15.0#egg=core

the same is happening with commands like apk add git

its like its not caching anything anymore

[tiny temple]

@candid tide if you had the engine running for a long time it's possible that you're being bitten by the engine GC mechanism which doesn't have a very generous default value (https://github.com/dagger/dagger/issues/7392)

if you do docker logs $engine_cointainer and you see removed snapshot logs it's because the GC is probably removing things to prevent your disk to become full

[candid tide]

gracias marcos for following up, i see some remove snapshot entries, but im not sure if they are related to those execs

i found these logs for that specific command im after:

time="2024-05-21T19:51:10Z" level=debug msg="> creating w0bkzhpsjn3gxuogrb4jma2zn [pip install git+https://[e-mail]/spi3pl/lib/core-lib.git@0.15.0#egg=core]" span="exec pip install git+https://[e-mail]/spi3pl/lib/core-lib.git@0.15.0#egg=core"
time="2024-05-21T19:51:39Z" level=debug msg="engine metrics" cpu-count=16 cpu-idle=3478361 cpu-iowait=38531 cpu-irq=0 cpu-nice=0 cpu-softirq=3129 cpu-steal=0 cpu-system=61973 cpu-total=3754313 cpu-user=172319 dagger-server-count=1 disk-available-/=23223767040 disk-available-/var/lib/dagger=23223767040 disk-free-/=28823945216 disk-free-/var/lib/dagger=28823945216 disk-size-/=109584551936 disk-size-/var/lib/dagger=109584551936 goroutine-count=104 loadavg-1=2.84 loadavg-15=2.23 loadavg-5=2.64 mem-active=1047347200 mem-available=2546814976 mem-buffers=933236736 mem-cached=851570688 mem-committed=34323038208 mem-free=137793536 mem-inactive=1716948992 mem-mapped=182657024 mem-page-tables=37138432 mem-shmem=10297344 mem-slab=1043591168 mem-swap-cached=129269760 mem-swap-free=2748010496 mem-swap-total=3221221376 mem-total=4102463488 mem-vmalloc-used=6402048 proc-self-mem-anonymous=51683328 proc-self-mem-private-clean=30539776 proc-self-mem-private-dirty=51683328 proc-self-mem-pss=82223104 proc-self-mem-referenced=78823424 proc-self-mem-rss=82227200 proc-self-mem-shared-clean=4096 proc-self-mem-shared-dirty=0 proc-self-mem-swap=0 proc-self-mem-swap-pss=0 server-w4x0ctxaui45n0cyzkhawb7bt-client-count=1 uptime=39m49s
time="2024-05-21T19:52:11Z" level=debug msg="> container done w0bkzhpsjn3gxuogrb4jma2zn [pip install git+https://[e-mail]/spi3pl/lib/core-lib.git@0.15.0#egg=core]" span="exec pip install git+https://[e-mail]/spi3pl/lib/core-lib.git@0.15.0#egg=core"

but i dont see a removed snapshot for w0bkzhpsjn3gxuogrb4jma2zn

is there a config i can use to bring that behaviour (the caching) back?

[tiny temple]

@candid tide wanna jump into a quick #911305510882513037 chat to check what's happening?

[candid tide]

hey @tiny temple i got caught yesterday... im available today for a quick review, let me know whenever you are available 🙂 thanks a lot!

[tiny temple]

cc @opaque jewel have a few cycles? trapped in a call now 🪤

[candid tide]

hey! i figured it out i believe

i kept inspecting my code and i found that there is a file that has a randomly generated filename

that caused the layers being recreated for each run

i tested it against dagger versions 0.6.4, 0.93 and 0.11.4
the behaviour is consistent

You always learn something new 🙂

All layers must be the same for them to be cached

if one changes, the cache gets invalidated

do you know if thats something that can be configure? thanks for taking the time!

[tiny temple]

yes, that's how the layering model works