state of services | Dagger | Page 1

deep oracle Mar 8, 2023, 11:11 PM

#

@rose kestrel docs are pending but you can find the current draft here: https://deploy-preview-4712--devel-docs-dagger-io.netlify.app/757394/use-services/ - we estimate it'l be published tomorrow

note that it currently doesn't support host-to-container networking; it's mainly focused on the sidecar use case (e.g. db for integration tests).

i'm hacking on host-to-container networking but I wouldn't expect a 1:1 replacement for Docker Compose, for example bind-mounts aren't really a thing at the moment so it's probably not well suited for a long-running live-updating server.

rose kestrel Mar 9, 2023, 1:13 AM

#

Ok. Will keep an eye out. Was trying to cut out docker compose files in my projects if I could but seems like a hybrid will be required, even if I can cut out the dockerfile I'll still need it for running local services.

deep oracle Mar 9, 2023, 1:44 AM

#

totally sympathetic to the cause! is it live-updating bind mounts that you need? i could see us slowly chipping away at this problem, so knowing any requirements you have will help

rose kestrel Mar 9, 2023, 2:34 AM

#

Mostly seeing if I want dagger and mage to be the single point of automation then I’d want to spin up a local DynamoDB container and run a background serve task etc. just replace dockerfile + compose. Mixing is fine just reduces the reliance on dagger sdk more so becomes more mix and match.

wild summit Mar 9, 2023, 10:40 AM

#

Maybe this is a dumb question, but why not tie the lifetime of the service to the lifetime of the object returned from Endpoint? A 10 second automatic timeout will likely work great for a small web server, but what about more complex things with long startup times that would get cycled a lot with a 10 second timeout or simply older hardware?

tropic saffron Mar 9, 2023, 11:26 AM

#

If I'm not wrong, I believe this is the behaviour: the service will be alive as long as one of dependent pipeline is still running. Once the pipeline is done, the service will shutdown automatically.

wild summit Mar 9, 2023, 11:48 AM

#

Suppose there are 4 containers S, A, B, and C defined in order. S provides a service to A and C and takes 10 minutes to start. Now suppose on this system Dagger (or the language async runtime) decides it can run two containers at once. So it dutifully starts S and A, and A runs to completion as A’s output was awaited. Next it runs B which takes more than 10 seconds to run to completion as its output was also awaited. As I understand Dagger will stop S at this point. Then it reaches to definition of C which uses S, what does Dagger do then?

If mental model of a container is that it lives as long as there is a reference to its endpoints makes this question simple. Here I think the question is more ambiguous.

#

This example is somewhat contrived, but I can see this kind of a pattern existing with larger numbers of containers where the interactions are more complex.

tropic saffron Mar 9, 2023, 12:48 PM

#

(again, I haven't played enough with it to be sure, but I could create a POC to show). I believe as long as you created/declared that both service A and C depend on S, S would keep running until both A and C are finished.

#

(like some kind of smart reference pointer)

tropic saffron Mar 9, 2023, 2:38 PM

#

package main

import (
    "context"
    _ "embed"
    "fmt"
    "os"
    "sync"

    "dagger.io/dagger"
)

func panicerr(err error) {
    if err != nil {
        panic(err)
    }
}

func main() {
    ctx := context.Background()
    c, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    panicerr(err)

    svc := c.Container().From("golang:1.19").
        WithNewFile("./web.go", dagger.ContainerWithNewFileOpts{
            Contents: webgo,
        }).
        WithExposedPort(9999).
        WithExec([]string{"go", "run", "web.go"})

    curler := c.Container().From("alpine:3.14.6").
        WithServiceBinding("web", svc).
        WithExec([]string{"apk", "add", "curl"})

    var wg sync.WaitGroup
    wg.Add(1)
    go func() {
        res, err := curler.
            WithExec([]string{"echo", "sleeping 3s"}).
            WithExec([]string{"sleep", "3s"}).
            WithExec([]string{"curl", "-s", "http://web:9999/A"}).
            Stdout(ctx)
        panicerr(err)
        fmt.Println(res)
        wg.Done()
    }()

    res, err := curler.
        WithExec([]string{"curl", "-s", "http://web:9999/C"}).
        Stdout(ctx)
    panicerr(err)
    fmt.Println(res)

    wg.Wait()

}

var webgo = `package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, r.URL.Path)
    })
    err := http.ListenAndServe(":9999", nil)
    if err != nil {
        panic(err)
    }
}`

#

I confirm it works, pipeline A sleeps 3 second in a go routine before curling the server. The service will wait for the A to be done to shutdown correctly

#

Dagger does the DAG dependencies solving.

deep oracle Mar 9, 2023, 3:36 PM

#

@wild summit you're right, that wouldn't behave very well. @tropic saffron the issue is when running B in their example, which takes longer than the 10 second timeout and runs before C can keep S alive.

@wild summit the current implementation is optimistic; we wanted to start with a "pure DAG-gy approach" and see who would come out of the woodwork with cases it can't account for. for example I considered adding a plain old 'start' method that would just keep the service running as long as the session is alive. but it turns out you can do that already with e.g. go svc.ExitCode(ctx) (or perhaps more clearly go svc.Run(ctx) in the future) - since they're just plain old containers, deduped like all the rest. other ideas could be making the timeout configurable, but I'd wager everyone is better off just explicitly starting the service instead; fudging with timeouts feels too toilsome.

wild summit Mar 9, 2023, 4:38 PM

#

Yeah timeouts are toilsome. My point is that you don’t need explicit lifetime management if it’s handled implicitly by object lifetime. Just food for thought.

deep oracle Mar 9, 2023, 4:45 PM

#

gotcha. i think we have the same goal, but we're getting lost in the framing. note that Endpoint just returns a string; once you have the string there's no conceptual object to hold a reference to, and the service hasn't actually even started yet. we already tie service lifecycle to objects (its bound containers) to avoid explicit lifecycle management, but in cases like this it seems like you do want to be more conscious of its lifecycle. so in that case you can just start it manually, and its lifecycle will be tied to your Dagger session. which seems equivalent to creating an object, having the service lifecycle dependent on it, and having the object go away when the session ends.

#state of services