How to avoid garbage collection of cached files?

[clear moat]

Let's say I create a file on a cached volume attached to a Container in some Function.

How can I use that cached file in another Function without the cache getting garbage collected? I thought mounting the cache to the container it was created in before calling with_file would suffice but it seems not:

# backend_test_build populates the cache
backend = await self.backend_test_build(backend_dir) 
jar_path = f"/build/libs/{consts.MONOLITH_JAR}"
jar_file = 
backend.with_mounted_cache("/build", dag.cache_volume('build')).file(jar_path)

container.with_file("/build/libs/my.jar", jar_file)

I think the problem is that the Function is "cached" because its contents have not changed, so the cache eventually gets garbage collected.
When I terminal into the Container, forcing the Function to run, the problem goes away, like a heisenbug.

Do empty circles in the trace indicate skipped steps? In this trace you will see that several steps in the pipeline leading up to the failure are marked as such.

[haughty oracle]

I'm wondering how this even work since I was expecting the file(jar_path) call to fail since you can't retrieve files or directories from a cache volume directly.

I was expecting the "cannot retrieve path from cache" error that you have seen already

Currently AFK so I can't test it

[clear moat]

I'm not retrieving directly from the cache but from a container (backend) with the cache mounted. jar_path is a string. jar_file is a reference to a file in a container with a mounted cached.

[haughty oracle]

you should be getting the same error.

For example, this fails with "cannot retrieve path from cache" for me:

func (m *Cache) Test(ctx context.Context) *dagger.Container {
    vol := dag.CacheVolume("vol")

    // populate something in the cache
    dag.Container().From("alpine").WithMountedCache("/cache", vol).WithExec([]string{"sh", "-c", "echo hello > /cache/hello.txt"}).Sync(ctx)

    //get the file
    f := dag.Container().From("alpine").WithMountedCache("/cache", vol).File("/cache/hello.txt")

    return dag.Container().From("alpine").WithFile("/hello.txt", f)
}

that's why I'm asking for a more complete repro given that IIUC I'm doing the same thing you're doing in your example above

[clear moat]

Is this the wrong way? What is the right way then?

[haughty oracle]

a better approach that I use is to put the building artifact (in this case the MONOLITH_JAR) outside the cache volume so you can then return it with a .file($path)call and return that from your function

looking at the gist you've sent me, I'd keep the cache volumes as they are since they will speed up subsequent builds as all the .class and deps should be re-used. I'd only make gradle put the monolith jar outside the /build/libs/foo.jar path so it doesn't belong to the cache volume