How to express a dependency of one container on another?

[little crypt]

I have a container that I use to start up as a service. Once that service is running, I need to continue using that container in order to initialize the service. I use WithServiceBinding in that container, and that all works fine.

I have another container which talks to that service, but I need to ensure that it doesn't run until after the service is fully initialized. So, WithServiceBinding isn't sufficient -- I need to express a dependency on the other container.

Here is some pseudo-Go code for what I am doing:

// Create the service container.
c1 := dag.Container().From("...")

// Start the service.  Error handling elided.
svc, err := c1.AsService().Start(ctx)

// Continue using the container to run a command to initialize the service.
// Error handling elided again.
output, err := c1.WithServiceBinding("svc", svc).WithExec(...).Stderr(ctx)

// Create the second container, which depends not only on the service running, but on c1 having run.
c2 := dag.Container().From("...").WithServiceBinding("svc", svc).WithExec(...)
output, err := c2.Stderr(ctx)

I'm still learning Dagger so I'm sure there's something I'm missing. WithServiceBinding expresses that c2 depends on svc, but is there a way to express that c2 also depends on c1 having run?

[ripe root]

@little crypt you did it correctly, c2 will only run after c1 runs.

oh wait, c1.asservice().start() is confusing me though

oh no that's not going to work 🙂 but i understand what you're trying to do now

@little crypt what issue are you seeing? what does your init command do specifically? does it connect to svc hostname to do its job?

[little crypt]

the init command connects to svc, yeah.

the issue I'm seeing is that it seems like sometimes that init command doesn't run, and my c2 command fails as a result. I'm not sure if it's cached or just an ordering problem

[ripe root]

might be caching. are you familiar with "cache busting" to disable caching of an operation?

[little crypt]

yeah, I am doing that, setting an env variable with time.Now()

[ripe root]

also that start() I think is not necessary.

[little crypt]

so i think it might be an ordering issue. it's a bit tough to see because it only happens in our CI environment from GHA... I haven't been able to replicate it locally

[ripe root]

do you have tracing setup in dagger cloud? will help you see what happens in the ci runs

[little crypt]

I don't think so. @restive needle do you know?

but just from a pure DAG perspective, I really want to describe that dependency 😄

Would Sync on c1 help here? I don't really understand what it does.

[oak perch]

Sync won't help because you're already calling Stderr which also forces the evaluation. Where in the pipeline do you have the cache busting?

[little crypt]

I have it on the c1 execution. So it's really more like: output, err := c1.WithServiceBinding("svc", svc).WithEnvVariable("CACHEBUSTER", time.Now().String()).WithExec(...).Stderr(ctx)

[restive needle]

We don't have dagger cloud setup (company policies I'm afraid...) 😦

[static rampart]

@little crypt IIUC the pseudo-code you shared the reality is that there doesn't seem to be a "strong" dependency between c1 and 2, right?

[little crypt]

@static rampart correct. there is an implicit dependency there, and I would like to make it explicit.

to be honest I don't know if that is the cause of my issue, but just from a correctness standpoint I would like to express it

[static rampart]

if you'd like to express that dependency, you could just get any file from c1 and pass it to c2. i.e:

f1 := c1.File("/some-file")

c2.WithFile("/tmp/file", f1)

That will make the dependency explicit as there's indeed a hard dependency between c1 and c2

^ the downside of doing that is that c2 won't run until c1 has finished, so you lose the ability to run both c1 and c2 in parallel beacuse of this explicit dependency.

[little crypt]

that's great. I don't want them to run in parallel 😄

that -- i think! -- might be the issue i'm running into

[oak perch]

Yeah I was going to show something similar

c1 = c1.WithServiceBinding("svc", svc).WithExec(..., dagger.ContainerWithExecOpts{ RedirectStderr: "/output.txt"})

// Create the second container, which depends not only on the service running, but on c1 having run.
c2 := dag.Container().From("...").
    WithFile("/c1_output.txt", c1.File("/output.txt")).
    WithServiceBinding("svc", svc).WithExec(...)
output, err := c2.Stderr(ctx)

with the implicit dependency today, they will run one after another though, so there's still something to fix

[little crypt]

I will do the file thing. It might be good to have another "artifact-less" way to express this

[static rampart]

SGTM. If you could open a brief issue that'd be much appreciated 🙏

[little crypt]

sure, will do. thanks all for the help

[ripe root]

There's something I don't understand, with the initial implementation (using c1.stderr), c2 should already be sequenced after c1 completes, no? Why the need for making the dependency explicit with a copied file?

[little crypt]

yeah, that was a property of stderr I wasn't aware of. so there's probably something else here happening that I don't understand.

[oak perch]

Yeah I think the core issue is still a caching issue. I'd try moving the cachebuster earlier in c1