Export/Import directories with content across containers | Dagger | Page 1

jolly spindle Jan 15, 2024, 4:11 AM

#

Hey guys, I have the following scenario:
I'm running a certain job, let's stay a terraform init — this command runs in one container A, which runs successfully and I'm exporting two things to the host (this part works well): .terraform (folder with content, meaning files and subdirectories) and .terraform.lock.hcl file.
Then, there's (with the same dagger client connected) a second container that runs terraform plan — of course it's going to fail if I'm not passing it the generated .terraform directory and the .terraform.lock.hcl which is precisely what I'm doing with this function:

func (c *ContainerImporterImpl) AddDataToImportInContainer(container *dagger.Container, options *DataTransferToContainer) (*dagger.Container, error) {
    for _, file := range options.Files {
        daggerFile := c.td.DaggerBackend.Host().File(file.SourcePathInHostAbs)
        container = container.WithFile(file.DestinationPathInContainer, daggerFile)
    }

    for _, dir := range options.Dirs {
        daggerDir := c.td.DaggerBackend.Host().Directory(dir.SourcePathInHostAbs)
        container = container.WithMountedDirectory(dir.DestinationPathInContainer, daggerDir)
    }

    return container, nil
}

I'm getting an error that says basically that the declared plugins into the .terraform.lock.hcl file aren't consistent with the .terraform plugins — which means, I'm uncertain that I've uploaded the whole content of the generated .terraform directory. Would the WithMountedDirectory be enough to mount the .terraform directory from the host (which was exported from the container A) to the workDir target path in container B? What I'm missing here?

strong delta Jan 15, 2024, 7:10 AM

#

@jolly spindle I think this pertains to you

Synchronization of a local directory happens once per Dagger client instance (in user-facing terms, once per dagger.Connect call in the Dagger SDKs). This means that if you load the local directory, then make changes to it on the host, those changes will not be reloaded within the context of a single Dagger client. Furthermore, due to lazy executions, the loading happens the first time the directory is used in a non-lazy operation.
https://docs.dagger.io/421437/work-with-host-filesystem/

Work with the Host Filesystem | Dagger

Introduction

jolly spindle Jan 15, 2024, 7:29 AM

#

That's the part that it's a bit confusing to me. For these two containers, both (A and B) are instantiated from the same dagger instance.
What's happening in terms of the order of things in my pipeline is that:

Client is instantiated.
Container A mounts the directory where the source code lives, and run a command, that export these two folders.
Container B mounts the directory where the source code lives, and adds (WithDirectory) the exported folder (.terraform) of the first container (that was persisted in the host).

thinkies

#

Oh, and by the way, the directory that's originally mounted isn't the one that's inspected in the host where the exported directory and files from the first container are stored.

strong delta Jan 15, 2024, 7:50 AM

#

Can you just share the directory with Container B in Dagger without the intermediate Export to the Host?

Edit:
I exported the /tf directory to the host from Container B at the end for fun.

package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "time"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        log.Println(err)
        return
    }
    defer client.Close()

    var tf = client.Directory()

    containerA := client.Container().
        From("hashicorp/terraform").
        WithDirectory("/tf", tf).
        WithWorkdir("/tf").
        WithExec([]string{"init", "-from-module=terraform-aws-modules/vpc/aws"})

    listA, _ := containerA.
        WithEntrypoint(nil).
        WithEnvVariable("CACHEBUSTER", time.Now().String()).
        WithExec([]string{"sh", "-c", "ls -al"}).
        Stdout(ctx)    
    fmt.Println(listA)

    containerB := client.Container().
        From("alpine").
        WithDirectory("/tf", containerA.Directory("/tf"))

    listB, _ := containerB.
        WithEnvVariable("CACHEBUSTER", time.Now().String()).
        WithExec([]string{"sh", "-c", "ls -al /tf"}).
        Stdout(ctx)    
    fmt.Println(listB)
}

 successB, _ := containerB.
        Directory("/tf").
        Export(ctx, "out")
    fmt.Println(successB)

quartz temple Jan 15, 2024, 2:45 PM

#

strong delta Can you just share the directory with Container B in Dagger without the intermed...

I was about to mention this. You don't need to export things to the host and back to a container to pass files / folders between them. You can just reference them as Jeremy is doing here which should be the proper way to do to maximize performance.

jolly spindle Jan 16, 2024, 4:54 AM

#

Oh, that's great! I thought I'd need to export it back to the host to make some files/dirs between containers to talk to each other. @strong delta thanks so much!

fierce carbon Mar 5, 2024, 1:59 PM

#

Hi everyone, in my case i am building a gradle project to generate a build folder, but then i am building a Dockerfile that needs this build folder, how do i manage this?

quartz temple Mar 5, 2024, 2:12 PM

#

fierce carbon Hi everyone, in my case i am building a gradle project to generate a build folde...

hey there! you can call the DockerBuild method and you can then export the build folder that gets created after the Dockerfile finishes building

fierce carbon Mar 5, 2024, 2:57 PM

#

what i'm trying to do is something like this, but the last part fails because it does not find exported build folder

#

    # initialize Dagger client
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
        # get reference to source code directory
        source = (
            client.host()
            .directory(".")
        )

        build_command = "gradle build -PbuildName=test -PbuildNumber=1.0-SNAPSHOT"

        print(f"Running build command: {build_command}")

        await client.container() \
            .from_("gradle:7.6.4-jdk17") \
            .with_mounted_directory("/src", source) \
            .with_workdir("/src") \
            .with_exec(build_command.split(' ')) \
            .directory("/src") \
            .export("./build") \

        await client.container() \
            .build(
                context=source,
                dockerfile='./docker/context/Dockerfile',
                target='service',
                build_args=[dagger.BuildArg(name='VERSION',value='1.0-snapshot')]) \
            .publish(f'ttl.sh/test:test')

anyio.run(main)

quartz temple Mar 5, 2024, 4:28 PM

#

fierce carbon ```async def main(): # initialize Dagger client async with dagger.Connec...

hey, that's because .export actually exports the resulting directory to your local machine and on the subsequent build step

#

what you need to do is not calling the export step and then do something like this:

        src_with_build =  client.container() \
            .from_("gradle:7.6.4-jdk17") \
            .with_mounted_directory("/src", source) \
            .with_workdir("/src") \
            .with_exec(build_command.split(' ')) \
            .directory("/src")

        await client.container().with_mounted_directory("/src", src_with_build) \
            .build(
                context=source,
                dockerfile='./docker/context/Dockerfile',
                target='service',
                build_args=[dagger.BuildArg(name='VERSION',value='1.0-snapshot')]) \
            .publish(f'ttl.sh/test:test')

fierce carbon Mar 5, 2024, 4:32 PM

#

ok, and what about if i run 1st part as a @function am i able to export to host?

#

let's say i create a 1st function to build_and_export, i trigger it with dagger call build-and-export then i trigger the second one dagger call build-image, is that possible?

quartz temple Mar 5, 2024, 6:17 PM

#

fierce carbon let's say i create a 1st function to build_and_export, i trigger it with `dagger...

so, IIUC you want to export the build folder to the host AND use it in the dockerbuild pipeline afterwards, correct?

fierce carbon Mar 5, 2024, 7:06 PM

#

yes

quartz temple Mar 5, 2024, 8:51 PM

#

fierce carbon yes

ok, in you can still do both, you just need to re-assign the variables accordingly to do so. For example:


       src_with_build =  client.container() \
            .from_("gradle:7.6.4-jdk17") \
            .with_mounted_directory("/src", source) \
            .with_workdir("/src") \
            .with_exec(build_command.split(' ')) \
            .directory("/src")
        await src_with_buid.directory("build").export("build")  # export here

        await client.container().with_mounted_directory("/src", src_with_build) \ # re-use src_with_build here
            .build(
                context=source,
                dockerfile='./docker/context/Dockerfile',
                target='service',
                build_args=[dagger.BuildArg(name='VERSION',value='1.0-snapshot')]) \
            .publish(f'ttl.sh/test:test')

fierce carbon Mar 6, 2024, 7:57 AM

#

       src_with_build =  client.container() \
            .from_("gradle:7.6.4-jdk17") \
            .with_mounted_directory("/src", source) \
            .with_workdir("/src") \
            .with_exec(build_command.split(' '))

        await src_with_buid.directory("/src")  # export here

        await client.container() \ # re-use src_with_build here
            .build(
                context=src_with_build,
                dockerfile='./docker/context/Dockerfile',
                target='service',
                build_args=[dagger.BuildArg(name='VERSION',value='1.0-snapshot')]) \
            .publish(f'ttl.sh/test:test')

#

☝️ this worked

#

but what i was expecting was the following, not sure if it's possible...

#

@function
def build_and_export(dir):
   return await client.container() \
            .from_("gradle:7.6.4-jdk17") \
            .with_mounted_directory("/src", source) \
            .with_workdir("/src") \
            .with_exec(build_command.split(' '))
            .directory("/src") \
            .export('./build')

#

doing dagger call build-and-export --dir . and have the build folder in host, but it's not happening...

simple sierra Mar 6, 2024, 8:23 AM

#

I had the same issue today, I change my function to return directly a directory then this one expose an export function with path argument dagger call build-and-export export --path=./build

fierce carbon Mar 6, 2024, 8:29 AM

#

did your change make it work?

simple sierra Mar 6, 2024, 9:04 AM

#

yes

fierce carbon Mar 6, 2024, 9:05 AM

#

can you share an example?

simple sierra Mar 6, 2024, 9:30 AM

#

yes I will push my module soon I'm finalizing some parts I ping you when it's done

fierce carbon Mar 6, 2024, 9:34 AM

#

thx!

simple sierra Mar 6, 2024, 10:53 AM

#

@fierce carbon I push my yq module: https://github.com/Dudesons/daggerverse/tree/main/yq#export-to-host

GitHub

daggerverse/yq at main · Dudesons/daggerverse

Contribute to Dudesons/daggerverse development by creating an account on GitHub.

fierce carbon Mar 6, 2024, 1:51 PM

#

great! thanks a lot!

quartz temple Mar 6, 2024, 2:00 PM

#

🚀

#Export/Import directories with content across containers