#Mounted directory is missing files

1 messages · Page 1 of 1 (latest)

pale mason
#

I am mounting a host directory into my container using container.WithMountedDirectory(). The target directory is being used an intermediary artifact store so that I can pass outputs between actions without reusing the same container.

The mount itself works without any issues, but files added to the host directory using container.Directory().Export() from previous containers are missing when the directory is mounted in future containers.

Is there any potential for race conditions or other gotchas when mounting directories?

lament summit
#

if you need mounted dirs/files to be available in exports you need to move them outside the mount path

pale mason
#

I assumed something weird was happening with the same mount being exported, but I changed the procedure as such.

I am mounting the shared directory in the container at /artifacts and I am exporting the new artifacts from the container at /tmp/artifacts

#

should that be working?

lament summit
pale mason
#

@lament summit
I guess what I don't understand is that the exported files from the container are working, but they don't show up in the mount. Let me paste an example so I can see what I am doing wrong.

This is the second action being run after a prior action was run that exported a build directory into the artifacts path on the host.

I setup some test code to echo the contents of the host artifact directory that is mounted into the container at /artifacts. The host artifact directory itself is just a temporary directory that is created before I start the pipeline.

container = container.WithMountedDirectory("/artifacts", client.Host().Directory(project.Artifacts))
ent, err := os.ReadDir(project.Artifacts)
if err != nil {
    return nil, err
}
for _, e := range ent {
    container = container.WithExec([]string{"echo", e.Name()})
}
container = container.WithExec([]string{"ls", "-R", "/artifacts"})

The output of this is:

#18 mkdir -p /tmp/artifacts
#18 DONE 0.5s

#19 echo build
#19 0.535 build
#19 DONE 0.6s

#20 echo test
#20 0.974 test
#20 DONE 1.0s

#21 ls -R /artifacts
#21 0.598 /artifacts:
2023-02-08 11:04:09    #21 0.598 test
#

I added the test file to the artifacts directory on the host for test purposes. You can see that both the test file and build directory are echoed in the container from the entries loop of the directory read.

The ls -R /artifacts in the container only shows the test file in the mounted artifacts directory, but the build file should be there as well. Why is the mount excluding the build directory?

lament summit
#

This is the second action being run after a prior action was run that exported a build directory into the artifacts path on the host.

do you need to export things to the host to re-mount them later? Dagger allows you copy files between different containers without having to send them to the host again

pale mason
#

Yes, but I also need them mounted back to the host.

My use case is that there are certain actions that I am running that might fail. In one particular case the only way for me to debug the failure is to look at the resulting reports from the tool. I am trapping specific errors in the shell in order to fail gracefully so that I can export the artifacts out of the container rather that letting it brick from the non-zero exit status

lament summit
pale mason
#

Excellent.

lament summit
# 
package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        log.Println(err)
        return
    }

    defer client.Close()

    // this generates "./build" in my local folder
    client.Container().From("alpine").WithExec([]string{"sh", "-c", `
    mkdir -p /build/outputs

    touch /build/one /build/outputs/two
    `}).Directory("/build").Export(ctx, "./build")

    out, err := client.Container().From("alpine").WithMountedDirectory("/build", client.Host().Directory("./build")).WithExec([]string{"ls", "-lRa", "/build"}).Stdout(ctx)
    if err != nil {
        panic(err)
    }

    fmt.Println(out)
}

^ this prints this:


#2 
/build:
total 12
drwxr-xr-x    1 root     root          4096 Feb  8 17:25 .
drwxr-xr-x    1 root     root          4096 Feb  8 17:25 ..
-rw-r--r--    1 root     root             0 Feb  8 17:21 one
drwxr-xr-x    2 root     root          4096 Feb  8 17:23 outputs

/build/outputs:
total 8
drwxr-xr-x    2 root     root          4096 Feb  8 17:23 .
drwxr-xr-x    1 root     root          4096 Feb  8 17:25 ..
-rw-r--r--    1 root     root             0 Feb  8 17:21 two
#

and I also have the /build folder exported locally in my machine

#

is that what you're trying to do?

pale mason
#

That's exactly what I am doing, but it isn't working for me...

#

I have to some nuanced semantic error on my end that I am missing

lament summit
#

seems to be the case. Try running my example to make sure the basics work 🙏

pale mason
#

I know it will work because I have an integration test that does the same thing. I appreciate you confirming though

#

Back down the rabbit hole I go...

#

I will keep you posted if I need more assistance. Thanks for the help

pale mason
#

@lament summit Is the export method guaranteed to be synchronous?

lament summit
#

yes

pale mason
#

@lament summit I would appreciate that. I am going to grab a cup of coffee first if that works

lament summit
pale mason
#

be there shortly. Sorry I got pulled into a call

lament summit
#

np. Still have some more time