#Hi everyone! I'm taking Dagger for a

1 messages · Page 1 of 1 (latest)

stark sage
#

For transferring files specifically, there's no need to use the host or cache as an intermediate, you can pass the files/directories directly like so https://docs.dagger.io/cookbook/#perform-multi-stage-build

so some rough pseudocode

// dotnet restore pipeline
restoreContainer := restore(client)

// build
g.Go(func() error {
  return build(client, restoreContainer.Directory("/src/foo")).Sync(ctx)
})
// test
g.Go(func() error {
  return test(client, restoreContainer.Directory("/src/foo")).Sync(ctx)
})
Cookbook | Dagger

Filesystem

wheat sphinx
#

Oh, I thought the *dagger.Container was not goroutine safe, only the *dagger.Client - which would force me to recreate the container in each goroutine in that case.

stark sage
#

Ah I didn't see that note before, I'm not actually sure what the implications are; cc @oblique glen

This can also be done without errgroups, like so (using the same psuedocode):

// dotnet restore pipeline
restoreContainer := restore(client)

// build
buildContainer := build(client, restoreContainer.Directory("/src/foo"))
// test
testContainer := test(client,restoreContainer.Directory("/src/foo"))

outputDirectory := client.Directory().WithDirectory("/", buildContainer.Directory("/src/foo/out")).WithDirectory("/testresults", testContainer.Directory("/src/foo/testoutput"))
err := outputDirectory.Export(ctx, ".")

In this case the lazy evaluation of the DAG happens when the final directroy Export is called, and since the build and test do not depend on eachother, they'll be executed by the engine concurrently

wheat sphinx
#

It feels like just using the *dagger.Container in the goroutine is not running concurrently, running 3 goroutines with .WithExec([]string{"sleep", duration}).Sync(ctx) now with duration 4s, 5s and 6s and it takes around 15s to execute

#

Which I guess makes sense

oblique glen
# wheat sphinx It feels like just using the *dagger.Container in the goroutine is not running c...

*dagger.Container is concurrently safe. Here's a quick snippet that validates it:

package main

import (
    "context"
    "fmt"
    "os"
    "sync"
    "time"

    "dagger.io/dagger"
)

func main() {
    if err := build(context.Background()); err != nil {
        fmt.Println(err)
    }
}

func build(ctx context.Context) error {
    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        return err
    }
    defer client.Close()

    ctr := client.Container().
        From("alpine:latest").WithEnvVariable("CACHE", time.Now().String())

    var wg sync.WaitGroup

    wg.Add(3)

    go func() {
        ctr.WithExec([]string{"sleep", "10"}).Sync(ctx)
        fmt.Println("Done 10")
        wg.Done()
    }()

    go func() {
        ctr.WithExec([]string{"sleep", "5"}).Sync(ctx)
        fmt.Println("Done 5")
        wg.Done()
    }()

    go func() {
        ctr.WithExec([]string{"sleep", "4"}).Sync(ctx)
        fmt.Println("Done 4")
        wg.Done()
    }()

    wg.Wait()

    return nil
}

If you run this, you'll see that Done 5 and Done 4 are printed before Done 10.

#

and the whole execution takes ~10s, not 19

wheat sphinx
#

Hmmm, let me check my code, I saw the opposite earlier

#

Yeah you are totally correct - I looked at the total runtime ticking in at around 17s and assumed that was because it did the sleeps sequentially but the setup takes some time as well so the extra 11s was there.

Then that simplifies a lot, thanks folks!