#go

1 messages · Page 1 of 1 (latest)

violet cosmos
#

@astral lotus 👋 let's get you setup on Cloud so you can show it off at Gophercon 🙂

#

paging @cosmic wyvern 😇

quick snow
#

This is probably common knowledge or just assumed... but the functional style of the Go SDK chaining requests means that it is translated into a single GraphQL query... right? At first it seemed really strange and very different from most Go SDKs, but after some fumbiling it finally clicked this morning 🙂

violet cosmos
#

Yes, each "step" in the chain corresponds to one "step" in the graphql query. The SDK does some additional stitching, to overcome limitations of the graphql language

#

So one Go "chain" might map to multiple graphql queries, stitched together seamlessly (via saving and loading the pipeline ID)

cosmic wyvern
pine flame
#

Is it possible to just build an image with Dagger? I want to reproduce something like docker build -t foo . which would build foo:latest for me. Here's the code I have:

    _, err := client.Container().Build(
        client.Host().Directory("."),
        dagger.ContainerBuildOpts{
            Dockerfile: "Dockerfile",
        },
    ).Publish(context.Background(), "foo")

And I get:

failed building Docker image: input:1: container.build.publish failed to solve: failed to push f:latest: push access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
dusky chasm
pine flame
#

Yeah, but I don't want to push it, only build it locally like with docker build command

cosmic wyvern
#

so if I run the above with dagger run go run main.go and then run docker import /tmp/my-nginx.tar foo
and docker images I see:

REPOSITORY                  TAG                        IMAGE ID       CREATED              SIZE
foo                         latest                     e341a66a3dd0   28 seconds ago       16.8MB
dusky chasm
patent ermine
pine flame
#

Thanks for the help, folks, let me try this once I get in front of a computer

violet cosmos
crystal hull
#

Hey all, this may be a dagger in general question, but just coming back to dagger after awhile ago - was noticing something odd earlier:

    version := "1.21.0"
    golang := client.Container().
        From("golang:latest").
        WithExec([]string{"go", "install", "golang.org/dl/" + version + "@latest"}).
        WithExec([]string{"./bin/go1.21.0", "download"}).
        WithExec([]string{"./bin/go1.21.0", "get", "-u"}).

Using that chain of commands (now that 1.21 is out, I don't need this anymore, but....) - I kept finding that the call to go get seemed to no use the result of the go download:

│ ┣─┼─╮ pull docker.io/library/golang:latest
│ ┻ │ │
█◀──┼─╯ CACHED exec go install golang.org/dl/go1.21.0@latest
█   │ CACHED exec ./bin/go1.21.0 download
█◀──╯ [0.19s] ERROR exec ./bin/go1.21.0 get -u
┃     panic: fork/exec ./bin/go1.21.0: no such file or directory

It's like its cached the call to go download but not actually getting its output. The DAG kinda makes it look like it's running the install/download separately, rather than linearly.

Does Dagger have any good means of diagnosing this type of thing? Like dropping into a docker shell at a specific stage/step?

dusky chasm
#

Hey all this may be a dagger in general

icy zephyr
#

Is there a way to emit a status message into the Client output stream so that dagger run formats it nicely and consistently?

violet cosmos
# icy zephyr Is there a way to emit a status message into the Client output stream so that `d...

We had some discussion of a status API in this issue: https://github.com/dagger/dagger/issues/5156

I'm guessing you already tried just printing to stdout from your code?

GitHub
TUI: hide container stdout by default · Issue #5156 · dagger/dagger

Context Now that we have an awesome TUI merged, and soon released (yay!) I think we can finally make progress on an old topic: container output streams (stdout and stderr); pipeline output; and the...

#

Step 2: we add an API for steps to explicitly send status, error etc. Data from that API is displayed prominently, by default.

icy zephyr
violet cosmos
#

I don't know if there's a TUI hack that allows this today? @patent ermine is the expert on that

patent ermine
icy zephyr
# patent ermine what's the delta from today's behavior? do you want the command's output more pr...

I'm interested in the output being correctly sequenced within the stream of container operations. I'm exporting a container image into the local docker, and I print a message when I'm done. So I would like that message to be correctly ordered after the logs of all the steps that build the container. go run doesn't buffer output, so the ordering is fine, but IIUC dagger run buffers so my status message ends up being printed before any container operation messages.

patent ermine
#

Ah, yeah that's a somewhat fundamental characteristic of that UI, but you could try --progress=plain which will interleave (with a bit of buffering to prevent thrashing)

#

Which might be what's already happening if you just use go run now that I reread your message

sick arch
#

I want to test this code with different directories, but every time I run go run main.go the results are always cached.
How do I disable caching it here?

package main

import (
    "context"
    "fmt"
    "log"
    "os"
    "path/filepath"

    "dagger.io/dagger"
)

func main() {
    dir := os.TempDir()
    // create random files in a directory
    os.Mkdir(filepath.Join(dir, "subdir"), 0700)
    os.Mkdir(filepath.Join(dir, "subdir2"), 0700)
    os.WriteFile(filepath.Join(dir, "subdir/foo.txt"), []byte("1"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir/baz.rar"), []byte("3"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir/faz.out"), []byte("4"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir2/bar.txt"), []byte("2"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir2/for.rar"), []byte("4"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir2/foz.out"), []byte("4"), 0600)

    ctx := context.Background()

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr), dagger.WithWorkdir(dir))
    if err != nil {
        log.Println(err)
        return
    }
    defer client.Close()

    entries, err := client.Host().Directory(".", dagger.HostDirectoryOpts{
        Include: []string{"*/*.rar", "*/*.out"},
        Exclude: []string{"*/*.txt"},
    }).Entries(ctx)
    if err != nil {
        log.Println(err)
        return
    }

    fmt.Println(entries)
}
# 

5: copy ./subdir (exclude */*.txt) (include */*.rar, */*.out) CACHED
5: > in host.directory ./subdir
5: copy ./subdir (exclude */*.txt) (include */*.rar, */*.out) CACHED
restive hollow
# sick arch I want to test this code with different directories, but every time I run `go ru...

@sick arch , so this copy steps of the files are cached because you didn't change the filenames between the consecutive runs, nor their contents. Every time you run this test, the hash of the file remains the same, even though its location is a in different tempdir.

Example:

go run . # This is my second run
Creating new Engine session... OK!
Establishing connection to Engine... OK!
6: upload . DONE
6: > in host.directory .
6: upload . DONE

6: upload .
6: > in host.directory .
6: transferring eyJvd25lcl9jbGllbnRfaWQiOiJjbTg5a3hxcnQzZGU5OGM2bDIzcTIxY2dnIiwicGF0aCI6Ii4iLCJpbmNsdWRlX3BhdHRlcm5zIjpbIiovKi5yYXIiLCIqLyoub3V0Il0sImV4Y2x1ZGVfcGF0dGVybnMiOlsiKi8qLnR4dCJdLCJmb2xsb3dfcGF0aHMiOm51bGwsInJlYWRfc2luZ2xlX2ZpbGVfb25seSI6ZmFsc2UsIm1heF9maWxlX3NpemUiOjB9:
6: transferring eyJvd25lcl9jbGllbnRfaWQiOiJjbTg5a3hxcnQzZGU5OGM2bDIzcTIxY2dnIiwicGF0aCI6Ii4iLCJpbmNsdWRlX3BhdHRlcm5zIjpbIiovKi5yYXIiLCIqLyoub3V0Il0sImV4Y2x1ZGVfcGF0dGVybnMiOlsiKi8qLnR4dCJdLCJmb2xsb3dfcGF0aHMiOm51bGwsInJlYWRfc2luZ2xlX2ZpbGVfb25seSI6ZmFsc2UsIm1heF9maWxlX3NpemUiOjB9: 509B [0.26s]
6: upload . DONE

5: copy . (exclude */*.txt) (include */*.rar, */*.out) CACHED # I didn't change anything, this is cached
5: > in host.directory .
5: copy . (exclude */*.txt) (include */*.rar, */*.out) CACHED
[qemu-console2084740384 qemu-console3153803277 qemu-console4073904031 subdir subdir2]
#

I change the file names to :

➜  test vim main.go
 // create random files in a directory
    os.Mkdir(filepath.Join(dir, "subdir"), 0700)
    os.Mkdir(filepath.Join(dir, "subdir2"), 0700)
    os.WriteFile(filepath.Join(dir, "subdir/foo.txt"), []byte("1"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir/baz1.rar"), []byte("3"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir/faz1.out"), []byte("4"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir2/bar.txt"), []byte("2"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir2/for3.rar"), []byte("4"), 0600)
    os.WriteFile(filepath.Join(dir, "subdir2/foz1.out"), []byte("4"), 0600)

This is not cached (same content, other filename):

➜  test go run .
Creating new Engine session... OK!
Establishing connection to Engine... OK!
6: upload . DONE
6: > in host.directory .
6: upload . DONE

6: upload .
6: > in host.directory .
6: transferring eyJvd25lcl9jbGllbnRfaWQiOiJsaHJlaTByZ3VxcTV3enByNnZ3MWxpcGs1IiwicGF0aCI6Ii4iLCJpbmNsdWRlX3BhdHRlcm5zIjpbIiovKi5yYXIiLCIqLyoub3V0Il0sImV4Y2x1ZGVfcGF0dGVybnMiOlsiKi8qLnR4dCJdLCJmb2xsb3dfcGF0aHMiOm51bGwsInJlYWRfc2luZ2xlX2ZpbGVfb25seSI6ZmFsc2UsIm1heF9maWxlX3NpemUiOjB9:
6: transferring eyJvd25lcl9jbGllbnRfaWQiOiJsaHJlaTByZ3VxcTV3enByNnZ3MWxpcGs1IiwicGF0aCI6Ii4iLCJpbmNsdWRlX3BhdHRlcm5zIjpbIiovKi5yYXIiLCIqLyoub3V0Il0sImV4Y2x1ZGVfcGF0dGVybnMiOlsiKi8qLnR4dCJdLCJmb2xsb3dfcGF0aHMiOm51bGwsInJlYWRfc2luZ2xlX2ZpbGVfb25seSI6ZmFsc2UsIm1heF9maWxlX3NpemUiOjB9: 691B [0.27s]
6: upload . DONE

5: copy . (exclude */*.txt) (include */*.rar, */*.out) # Not cached
5: > in host.directory .
5: copy . (exclude */*.txt) (include */*.rar, */*.out) DONE
[qemu-console2084740384 qemu-console3153803277 qemu-console4073904031 subdir subdir2]
#

This is not cached: (same filename, different content):

go run .
Creating new Engine session... OK!
Establishing connection to Engine... OK!
6: upload .
6: > in host.directory .
6: transferring eyJvd25lcl9jbGllbnRfaWQiOiJpaW9lZWU3OHVqZnIyNGF4c3VjanlqOGR3IiwicGF0aCI6Ii4iLCJpbmNsdWRlX3BhdHRlcm5zIjpbIiovKi5yYXIiLCIqLyoub3V0Il0sImV4Y2x1ZGVfcGF0dGVybnMiOlsiKi8qLnR4dCJdLCJmb2xsb3dfcGF0aHMiOm51bGwsInJlYWRfc2luZ2xlX2ZpbGVfb25seSI6ZmFsc2UsIm1heF9maWxlX3NpemUiOjB9:
6: transferring eyJvd25lcl9jbGllbnRfaWQiOiJpaW9lZWU3OHVqZnIyNGF4c3VjanlqOGR3IiwicGF0aCI6Ii4iLCJpbmNsdWRlX3BhdHRlcm5zIjpbIiovKi5yYXIiLCIqLyoub3V0Il0sImV4Y2x1ZGVfcGF0dGVybnMiOlsiKi8qLnR4dCJdLCJmb2xsb3dfcGF0aHMiOm51bGwsInJlYWRfc2luZ2xlX2ZpbGVfb25seSI6ZmFsc2UsIm1heF9maWxlX3NpemUiOjB9: 691B [0.29s]
6: upload . DONE

5: copy . (exclude */*.txt) (include */*.rar, */*.out)
5: > in host.directory .
5: copy . (exclude */*.txt) (include */*.rar, */*.out) DONE
[qemu-console2084740384 qemu-console3153803277 qemu-console4073904031 subdir subdir2]
final fable
short yacht
delicate reef
#

GitHub - abtris/dagger-tutorial: Dagger ...

finite musk
#

Has anyone used Dagger for multi-module Go repositories?

astral lotus
#

Well, dagger itself is a multi module repository. There is the go.mod for the project itself, and the go.mod for the generated Go SDK.

modest nacelle
finite musk
modest nacelle
finite musk
modest nacelle
finite musk
violet cosmos
#

I know @boreal jungle way back had experimented with Dagger-native tools for managing Go repos

modest nacelle
modest nacelle
finite musk
#

Where sometimes the Go tooling does not always seem to cache when -race is enabled. On the topic of selectively deciding what to push, it can be tricky to decide which microservices to publish (broken up into multiple modules in the same repo) because if you have a larger repository with multiple microservices (all part of the same product) it can be hard to know or remember if some code you touched is used by x, y, z microservice. We thought it would be a nice experience to take that pressure off them.

finite musk
finite musk
finite musk
modest nacelle
modest igloo
#

hi team, could you please help me this case? I am trying to build docker image by using dagger, how I can export as docker image on my local machine, not pushing to registry?

violet cosmos
#

@modest igloo if you want to export the container image to a tar archive on your local filesystem filesystem (instead of loading into your docker engine), then you can use container.export

finite musk
violet cosmos
#

In the Go SDK, Container.WithDefaultArgs([]string{"foo"}) does not work, instead I have to call Container.WithDefaultArgs(ContainerWithDefaultArgsOpts{Args: []string{"foo"}}) which is confusing

sick arch
agile mural
#

Pro tip (or maybe not, time will tell): Use {} to encapsulate context and create pipelines:

var test *dagger.Container

    // Prepare
    {

        client := client.Pipeline("Prepare")

        {
            // ...
        }

        {
            // ...
        }
    }

    // Build
    {
        client := client.Pipeline("Build")

        var app *dagger.Container
        {
            client := client.Pipeline("App")
            host := client.Host()

            app = appContainer(client)
        }

        {
            client := client.Pipeline("Test container")

            test = testContainer(client).
                WithServiceBinding("app", app)
        }
    }

// use test container

See more here: https://github.com/portward/portward/pull/4/files#diff-46921c3055de31d26525577c40a40134eb58493dc89ae4e34a94fec194dcfc70

GitHub
test: add dagger based conformance tests by sagikazarmark · Pull Re...
livid quiver
#

yeah I like using scopes like this, shadow vars are nice

#

wish go had the same shadow capability rust does, with re-letting vars

prisma mesa
#

any tips on how you would debug a go program. So basically dagger run go run main.go but I want to debug from an IDE. I assume I can type dagger run dlv .... and attach, but I'm so lazy.

prisma mesa
violet cosmos
prisma mesa
# violet cosmos what does that look like?

if you attach the debugger at some point, I think it's when you detach the dagger process will move to the background. You have to type fg to bring it back and then dagger just dies at that point. So if you run a series of debugs sessions after each other you notice a bunch of stopped background processes. Also, I'm on linux and probably doesn't do the same thing on macOS.

cloud thorn
prisma mesa
cloud thorn
#

Yup! dagger run is optional.

#

Just gives a nicer experience when looking at logs and some other stuff, but wraps your process so can sometimes cause funkiness.

prisma mesa
hard whale
#

Hello,
I'm beginner dagger user
I have a question. With the following code:

func main() {
    ctx := context.Background()

    // initialize Dagger client
    client, err := dagger.Connect(ctx,
        dagger.WithLogOutput(os.Stdout),
        dagger.WithWorkdir("../../"),
    )
    if err != nil {
        panic(err)
    }
    defer client.Close()

    sqlInit := client.Host().File("init.sql")

    client.Container().From("postgres:15.4-bullseye").
        WithEnvVariable("POSTGRES_USER", "admin").
        WithEnvVariable("POSTGRES_PASSWORD", "admin").
        WithMountedFile("/docker-entrypoint-initdb.d", sqlInit)
}

I got this error:

 panic: cannot configure workdir for existing session (please use --workdir or host.directory with absolute paths instead)

If you have any idea to help me to fix it.
I have no idea why I cannot mount the file

violet cosmos
#

@hard whale are you by any chance calling your program with dagger run ?

#

Long story short, dagger.WithWorkdir does not work reliably and will be deprecated in the future. You can achieve the same result by simply changing your working directory before calling your program

#

one way is to use dagger run --workdir ../..

hard whale
#

I got the same error

violet cosmos
hard whale
#

it work when I remove dagger.WithWorkdir
But I have this following error:

Screenshot_2023-10-15_at_01.49.25.png
violet cosmos
#

i think you can ignore that error (should be a warning) what happens after?

hard whale
#

after It's good

#

Thank you 🙂

hard whale
#

Do you have any idea of why I cannot access to database port ?
The code continue to run indefinitly :/
https://github.com/Stratorys/database-crash-simulator/blob/main/ci/dagger.go

░ [11.67s] check k2roar0cnhck8.5lk52kjj73m24.dagger.local 5444/tcp
┃ polling for port k2roar0cnhck8.5lk52kjj73m24.dagger.local:5444
┃ port not ready: dial tcp 10.87.0.23:5444: connect: connection refused; elapsed: 1.008292ms
modest nacelle
hard whale
#

I already have a database which are running in port 5432, but I. think I must forward port

#

Thanks for the doc link + where to look 🙂

violet cosmos
hard whale
#

WithExposedPort only allow to expose the container port, not map

#

Postgres runs on port 5432 in its container, but I'd like to remap it to expose port 5444 externally

violet cosmos
hard whale
# violet cosmos I see, you want to connect to the postgres container from the host machine?

The idea is to set up a temporary database, run a piece of code twice to see the result, then delete the whole thing
https://github.com/Stratorys/database-crash-simulator/blob/main/ci/dagger.go

GitHub
database-crash-simulator/ci/dagger.go at main · Stratorys/database-...

Simulate crash of an golang api to check if databases connections become a zombie connections or will be auto clean - Stratorys/database-crash-simulator

livid quiver
#

the service container is so useful (we have a pretty complex build, and this cuts out a lot of drone-related stupidity)

modest nacelle
modest nacelle
modest nacelle
hard whale
modest nacelle
hard whale
#

Maybe it's not a good practice? What would be the best practice to launch a database that allows me to run tests on my prod machine and then if everything passes I build my docker image which will then be used by my prod. Right now, my little project is more of a playground for testing, but with things I'm going to need right afterwards.

hard whale
hard whale
modest nacelle
hard whale
modest nacelle
hard whale
hard whale
modest nacelle
#

DB_HOST != DB_HOSTNAME
DB_USER != DB_USERNAME

hard whale
#

I was bad, I was looking so hard elsewhere that I missed the basic

crystal hull
#

I must be doing something blind/stupidly simple wrong here - experimenting with switching my pipeline to using server containers in dagger, rather than using TestContainers inside my own code - when I uses client.Container().From("smx/smx3:latest"). to fire up my own local image, dagger fails with resolve image config for docker.io/smx/smx3:latest ERROR: pull access denied - any reason why dagger looks first/only at docker.io?

#

(I guess this isn't go specific actually)

unborn herald
#

@crystal hull docker.io is magically prefixed to the registry URL in docker if the 1st path is not a URL, it assumes that smx/ is a repository org underneath, you'd have to override it with: myinternalregistry.com/smx/smx3:latest

#

not a dagger thing

crystal hull
#

weird - since that works perfectly fine in from docker-compose, or just the command line.

#

Not sure how to get around that then - as I don't overly want to run a registry on my laptop

grim karma
crystal hull
#

it's built locally by another process yeh - building it from within the pipeline is not (currently) possible, or desirable - separate git repo / build process, and trying to limit dagger for now whilst getting something I can sell to org.

For now, I've switched to using our published :latest image from Azure ACR, but as part of this pipeline dev work, I'm also working on the server and wanting to run the tests etc. against a locally built - non-pushed image.

You may be right about dagger not being able to sett local images, so I may have to resort to using a local registry.

violet cosmos
crystal hull
#

mm, I guess I'm just confused as the image already exists in the local docker engine, which can be/is used by docker compose elsewhere, so I guess services can do "compose like" - probably better to move this to #1113696290577059851 than #go tho 🙂

Ideally, I don't want to rewrite/merge 3 projects into this project (its not just as simple as 'building the dockerfile' ) - nominally I'd be referring to the ACR container image(s), I just have a need to run against local images for test purposes before pushing code changes for review at times. Setting a local registry may be the cleanest approach - then I can just change the image name used via an ENV var or something as an override, altho I can make my main container build export the tgz, so could use that conditionally. ( I suspect this is a code organisation issue, mono-repo vs non - i have 100+ repos/modules that make up the container, usually only 1-2 are under dev/test tho - a much larger discussion).

Now to figure out why the tests now fail using the service tho, doesn't help my Export either isn't running, or just not doing anything. Resorting to using tee and redirecting dagger output for now 🙂

violet cosmos
#

Whether you choose to run a local registry, mount the local docker socket, or build in dagger, all your inputs and outputs will have to be explicitly laid out. In the happy path it can seem repetitive, but it makes everything more repeatable and maintainable.

crystal hull
#

so mounting the socket would work? I'd asked about that a week or so ago - if that works I can trash all these service image changes. Currently TestContainers works as I pass thru the DOCKER_HOST var to my test job, and my laptop allows docker network API calls - Azure, not so much.

violet cosmos
#

the main issue is passing image data through the docker/dagger gap

#

dagger natively supports pushing/pulling to a registry but not a local docker engine (because docker engine has no clean API to do that). But there are hacks to make that work too (see the guide linked above)

crystal hull
#

right - currently those containers appear alongside dagger, and I had funky issues refering to host.docker.... - which the service model in Dagger definitely made A LOT cleaner

livid quiver
#

I've been making base containers then layering them with support using WithContainer functions

#

i have like 3 main types of builds and I like to be able to only include features where I need them for a stage then throw that away since it's all cached for me anyway

#

*dagger.Container.With(dagger.WithContainerFunc) lets me splice in a loop a buncha extra steps after doing setup (adding extra db migrations, setting up a client for our auth, etc.)

quick snow
#

Is there a way to modify the output from the Go SDK? It's getting mangled in the output to Cloudwatch Logs

crystal hull
#

I guess you could write your own logger? or redirect to a file?

client, _ := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
``` - that implies to me one could connect some other form of logging mechanism?
cosmic wyvern
livid quiver
#

io.Writer

violet cosmos
#

The API for querying a container's metadata is quite frustrating. To get the container's environment (realistically, a few dozen lines of text stored in a small JSON file), I have to do this:

func containerEnv(ctx context.Context, ctr *Container) (string, error) {
  vars, err := ctr.EnvVariables(ctx)
  if err != nil {
    return "". err
  }
  env := make([]string, 0, len(vars))
  for _, var := range vars {
    k, err := var.Name(ctx)
    if err != nil {
      return "", err
    }
    v, err := var.Value(ctx)
    if err != nil {
      return "", err
    }
    env = append(env, k + "=" + v)
  }
  return env, nil
}

This is insanity.

I don't even want to look into how many graphql queries this actually sends, I'm too scared to learn the answer.

Here's the equivalent raw graphql query:

query containerEnv($ctr: ContainerID!) {
  container(id: $ctr) {
    envVariables {
      name
      value
    }
  }
}
#

Does the Go SDK support dropping to raw query builder? I would rather just query everything I need myself at this point

#

(I'm trying to introspect all the contents of a container, so there are a lot more fields I need to query, and the DX for doing that is better with grapqhl, because it supports doing it all with one batch query, and it's all strings anyway so I don't care about the type safety)

#

Maybe even the query builder is not low level enough - perhaps our query builder doesn't actually support selecting more than one field?

#

In that case I'll happily take a raw http transport where I can plug my own graphql client

violet cosmos
#

Scanning through the querybuilder source code, it appears that it does not in fact support multiple selections

livid quiver
#

my co-worker was trying to do some stuff with environment but in his case writing a .env file. (in go, the source was secrets in aws). any way, similar problem to yours, a lot of iterating.

#

for me go really needs ? or at least sum types

tawny carbon
violet cosmos
fading void
# 
┣─╮ 
│ ▽ init
│ █ [0.44s] connect
│ ┣ [0.41s] starting engine
│ ┣ [0.03s] starting session
│ ┃ OK!                                                                                                                                                                    
│ ┻ 
█ [0.11s] ERROR go run multibuild/main.go
┃ # dagger.io/dagger/internal/engineconn                                                                                                                                   
┃ ../../../../go/1.19.5/pkg/mod/dagger.io/dagger@v0.9.3/internal/engineconn/session.go:151:8: proc.Cancel undefined (type *exec.Cmd has no field or method Cancel)         
┃ ../../../../go/1.19.5/pkg/mod/dagger.io/dagger@v0.9.3/internal/engineconn/session.go:155:8: proc.WaitDelay undefined (type *exec.Cmd has no field or method WaitDelay)   
┃ note: module requires Go 1.20                                                                                                                                            
┻ 
• Engine: 29c192ab1fa8 (version v0.9.3)
⧗ 0.55s ✔ 3 ✘ 1
exit status 2
➜  hello git:(master) ✗ go version
go version go1.19.5 darwin/arm64
➜  hello git:(master) ✗
#

dagger v0.9.3 (registry.dagger.io/engine) darwin/arm64

#

Found the issue, goenv was mixing versions in my shell, some where showing 1.19.5 but additionally trying to use 1.21, removing 1.19.5 and setting global to 1.21.1 resolved it

icy zephyr
#

Thanks for adding the Commit API to GitRef. It works great, and is exactly what I needed! ❤️

violet cosmos
livid quiver
#

it could be a vararg of string

#

but what if you want to also specify opts?

agile mural
#

Is there a better way to get a File pointing to a container image tar.gz than this?

func dummyImage(ctx context.Context) (*File, error) {
    imagePath := filepath.Join(os.TempDir(), "image.tar.gz")
    _, err := dummyContainer().Export(ctx, imagePath)
    if err != nil {
        return nil, err
    }

    return dag.Host().File(imagePath), nil
}
violet cosmos
spiral inlet
#

Hello all, I was working through a pretty simple http server in Go, and in my pipeline it gets stuck at the end on tunnel.Stop(ctx)
This is the snippet:

```

package main

import (
"context"
"fmt"
"io"
"net/http"
"os"

"dagger.io/dagger"

)

func main() {
ctx := context.Background()

client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
if err != nil {
    panic(err)
}
defer client.Close()

svc := client.Container().
    From("golang:1.21.5").
    WithDirectory("/src", client.Host().Directory("."), dagger.ContainerWithDirectoryOpts{
        Exclude: []string{"ci"},
    }).WithWorkdir("/src").
    WithExec([]string{"go", "run", "main.go"}).
    WithExposedPort(4000).
    AsService()

tunnel, err := client.Host().Tunnel(svc, dagger.HostTunnelOpts{}).Start(ctx)
if err != nil {
    panic(err)
}

defer tunnel.Stop(ctx)

endpoint, err := tunnel.Endpoint(ctx)
if err != nil {
    panic(err)
}

res, err := http.Get("http://" + endpoint)
if err != nil {
    panic(err)
}
defer res.Body.Close()

body, err := io.ReadAll(res.Body)
if err != nil {
    panic(err)
}

fmt.Printf("Answer:\n%s", string(body))

}


Any clues what I'm doing wrong? I catch `os.Interrupt` and `syscall.SIGTERM` on my http server. 
Edit: I also tested in Linux and I got the same behavior. The only way to overcome the issue is using a second container with service binding and curl
cosmic wyvern
#

@spiral inlet it worked for me with a basic example by commenting out the

//    defer tunnel.Stop(ctx)
# 
.
├── ci
│   └── ci.go
├── go.mod
├── go.sum
└── main.go

I put your code 👆 in ci/ci.go with the // defer tunnel.Stop(ctx) modification.

#

===
Here's 👇 my main.go http server.
Slightly modified version of https://gobyexample.com/http-servers to use port 4000 and handle the / route:

package main

import (
    "fmt"
    "net/http"
)

func hello(w http.ResponseWriter, req *http.Request) {

    fmt.Fprintf(w, "hello\n")
}

func headers(w http.ResponseWriter, req *http.Request) {

    for name, headers := range req.Header {
        for _, h := range headers {
            fmt.Fprintf(w, "%v: %v\n", name, h)
        }
    }
}

func main() {

    http.HandleFunc("/", hello)
    http.HandleFunc("/headers", headers)

    http.ListenAndServe(":4000", nil)
}
spiral inlet
cosmic wyvern
#

tl;dr Seems buggy that calling Stop() on a Service returned by Host().Tunnel() seems to cause Dagger to not terminate as expected.
Even though there is an explicitStart() and then a Stop(). I wonder if there is a race conditions with the other deferred methods and termination of the main() method...?

spiral inlet
patent ermine
#

hmm that's an odd one. it obviously should work as-written, it's really weird that changing it from defer worked 🤔

patent ermine
spiral inlet
radiant leaf
#

Hi, When using OrbStack and Dagger Linux hosts inside OrbStack use the command orb docker instead of docker. Typically for the shell I just alias the command however dagger is not using the alias and appears to just use exec on the docker image. Is there a way using the Go API I can specify that dagger should be using the orbstack version of docker ?
Many Thanks

violet cosmos
bright hazel
#

hi, you know how to use dagger with golang on dedicated subdirectory ? I want to put all dagger code on ci/dagger (with go.mod on this folder). But I should to run dagger from root directory like dagger run go run ci/dagger/main.go. It doesn't work if I doesn't put go.mod on root directory...

vapid vortex
bright hazel
bright hazel
#

I can invoke client.Host().Directory(path) with the same path multiple times ?
I create first container with WithDirectory("/project", client.Host().Directory(path, dagger.HostDirectoryOpts{Exclude: []string{"ci"}}))
Then I modify some local file from go code.
Then I create new container with WithDirectory("/project", client.Host().Directory(path, dagger.HostDirectoryOpts{Exclude: []string{"ci"}}))
But it look the old version of file. Before I modify it. It's normal ?

regal seal
#

Within the context of a single dagger run, yup, this would be normal. Dagger is caching the contents of your local filesystem.

#

You should avoid writing to your local filesystem while doing dagger work - I'd recommend making the changes inside dagger pipelines using WithFile, etc.

bright hazel
#

I think a found a way with Sync() function

violet cosmos
bright hazel
#

I use ti like this.

chartFilePath := fmt.Sprintf("%s/%s", option.PathContext, "Chart.yaml")
    chartFile, err := client.Host().File(chartFilePath).Sync(ctx)
    if err != nil {
        return errors.Wrap(err, "Error when force sync Chart.taml")
    }
    container = container.WithFile(chartFilePath, chartFile)
violet cosmos
bright hazel
#

What do you mean by snapshot ? I should it update the file content on the "cache" when I invoke the Sync.

violet cosmos
#

Usually, when you need to change the contents of a file on the host while your Dagger pipeline is running, and are getting stuck, it's a sign that you are trying to create an intermediary file between two parts of your Dagger pipeline, and you don't need to write to the host filesystem for that.

bright hazel
#

ok thx for the explanation

violet cosmos
#

@bright hazel would you mind sharing the snippet of code where you use that intermediary file? I can help you modify it to not rely on host, which should fix your issue

bright hazel
#

Hum, it seems is more complicated that I first thinking.
On the same sessions, now I create container to update Chart version

_, err = getYQContainer(client, option.PathContext).
        WithExec(
            []string{"--inplace", fmt.Sprintf(".version = \"%s\"", option.Version), "Chart.yaml"},
            dagger.ContainerWithExecOpts{InsecureRootCapabilities: true},
        ).
        File("Chart.yaml").
        Export(ctx, "Chart.yaml")
    if err != nil {
        panic(err)
    }

Then I create other container to build and push helm chart, but it still the old chart version.

getHelmContainer(client, option.PathContext).
  WithSecretVariable("REGISTRY_USERNAME", registryUsername).
  WithSecretVariable("REGISTRY_PASSWORD", registryPassword).
  WithEntrypoint([]string{"/bin/sh", "-c"}).
  WithExec(helper.ForgeCommand("helm package -u .")).
  WithExec([]string{fmt.Sprintf("helm registry login -u $REGISTRY_USERNAME -p $REGISTRY_PASSWORD %s", option.RegistryUrl)}).
  Stdout(ctx)
signal mantle
bright hazel
#

hum ...

signal mantle
bright hazel
#

Ok, it now work like a charm. Thx

violet cosmos
#

Thanks @signal mantle

ionic meadow
#

Hi everyone! I'm taking Dagger for a spin here with the Go SDK and am trying to build a .NET Core project using it. I have a working prototype but I would like to speed it up a little by running some steps concurrently. Say for example running build and test concurrently (bad example, but just hypothetically). I then am faced with an issue on how to avoid having to re-restore the dependencies of the projects when spinning up the containers for build and test.

My current simple pipeline now consists of a single *dagger.Container that runs:

  • dotnet restore
  • dotnet build --no-restore
  • dotnet test --no-restore

If I implement this with an errgroup so that I first spawn a dotnet container that does "dotnet restore", then with an errgroup I spawn two dotnet containers that does build/test using the same *dagger.Client as outlined in https://docs.dagger.io/cookbook#organize-pipeline-code-into-modules--classes, then I need to transfer the dependencies from the initial container into the two new ones and I'm not sure what the best practice for that is and how to solve that cleanly in the Go SDK. I guess my options are:

  1. Spawn the build/test containers based on a the resulting image after running dotnet restore, presumably after committing the image. I don't see this supported in the Go SDK however.
  2. Copy the restored packages from the restore container into the host and back to the new containers, it works but feels a bit like extra work given how docker works; just using the restore-image as the from image would probably be more Docker-esque.
  3. Use cache, but that seems like the wrong solution as that cache can presumably be used by other builds too (unless the cache id is unique for each build)

You guys have any suggestions?

opaque mica
#

Chaining funcs with With()

ionic meadow
#

Hm, I'm trying to spawn a docker in docker service and not have it output a lot of noise, from previous discussions here (#general message) I see that I can set RedirectStdout and RedirectStderr on the .WithExec call on the service but this does not seem to have an effect. Take for example this service:

dag.Container().
From("docker:dind").
WithExec([]string{
"dockerd-entrypoint.sh",
"dockerd",
"--tls=false",
"--host=tcp://0.0.0.0:2375",
}, dagger.ContainerWithExecOpts{
SkipEntrypoint: true,
InsecureRootCapabilities: true,
RedirectStdout: "/tmp/stdout",
RedirectStderr: "/tmp/stderr",
}).
WithExposedPort(2375).
AsService()

This starts just fine but it logs all the output of dockerd to stdout causing a lot of noise in my build logs. Am I misunderstanding how to use this somehow? Example:

39: start dockerd-entrypoint.sh dockerd --tls=false --host=tcp://0.0.0.0:2375
39: [0.06s] iptables v1.8.10 (nf_tables)
39: [0.06s] [WARN tini (15)] Tini is not running as PID 1 and isn't registered as a child subreaper.
39: [0.06s] Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
39: [0.06s] To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1.
39: [0.08s] time="2024-01-10T10:41:13.143873697Z" level=info msg="Starting up"

lethal vigil
#

Hey everyone! Was wondering if anyone has created unit tests for the pipelines themselves and has some examples they can show?

modest nacelle
#

👋 the reason for this is because RedirectStdX is mostly used for the Stdout and Stderr dagger methods. I think that you can accomplish what you need by wrapping the dockerd in a sh -c dockerd.... 2>/dev/null that should make it work. Having said that, seems to be worth to open an issue as it's confusing how to handle this

ionic meadow
#

I'm seeing these sporadically in my pipelines:

2024/01/11 22:54:51 http: panic serving 127.0.0.1:42872: context canceled
goroutine 10881 [running]:
net/http.(*conn).serve.func1()
/usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0x106b640?, 0x1cc9b70?})
/usr/local/go/src/runtime/panic.go:920 +0x270
github.com/dagger/dagger/engine/client.(*Client).ServeHTTP(0xc0001c4480, {0x13f73c8, 0xc00058e700}, 0xc00003e600)
/app/engine/client/client.go:535 +0xbf2
net/http.serverHandler.ServeHTTP({0x13f3498?}, {0x13f73c8?, 0xc00058e700?}, 0x6?)
/usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc000ed15f0, {0x13fc420, 0xc00055f080})
/usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 153
/usr/local/go/src/net/http/server.go:3086 +0x5cb

From what I can tell it doesn't affect anything, pipeline completes just fine still. Any idea what this is? I can't see any other discussions on Discord about it, or on GitHub, but it happens so often that it sort of seem strange. Mayte not every run, but some times more than once every run.

modest nacelle
ionic meadow
modest nacelle
ionic meadow
coarse ruin
#

Hi Folks. I am looking for help authenticating with gcr via dagger. So far I get this output:

18: resolve image config for gcr.io/***/***/***:latest ERROR: failed to authorize: failed to fetch oauth token: unexpected status from POST request to https://gcr.io/v2/token: 404 Not Found

I am not sure what kind of setup I am meant to do for this to work, nor can I seem to find the correct docs. I played a little with WithRegistryAuth but don't know if I am using it correctly or if I am in the correct track. Any help would be appreciated !

violet cosmos
#

GCR auth

maiden flicker
teal summit
#

new version is out!

fallen root
#

I keep getting bit by go patch version issues, curious to hear about how other people work.

For example running go work init creates a file with go major.minor.patch but some of my submodules may have a lower patch version. Go gets very sad about this.

I tried setting the go.work file to just consider major.minor but it does not seem to resolve the problem.

ionic meadow
#

Any good way to read a file from a *dagger.Container that may not exist? I can just ignore the err in code but that causes a big red line yelling at me in Dagger Cloud even if I suppress the error, causing the build to be marked as failed there as well, so it feels like I'm doing it the wrong way. I could do WithExec().Stdout() and [ -f file ] cat file but that adds the entire file contents to the build output

signal mantle
#

Good question. Ideally we could add a Stat() function or something for that purpose. Right now you could use Directory.Entries() and check if it's there, assuming the directory itself is known to exist

ionic meadow
#

Ah, maybe that works yeah, the directory exists. Thanks!

violet cosmos
patent ermine
#

Yeah I've had a similar experience, one of my pipelines runs ldd against a binary and expects it to fail, checking that it's statically linked

#

Maybe the source of truth becomes the wrapping call? So sub-calls that fail can be OK if the wrapping call succeeds, and failures don't bubble up

violet cosmos
#

will zenith viz help? with encapsulation and only showing top level by default

violet cosmos
modest nacelle
# ionic meadow Any good way to read a file from a *dagger.Container that may not exist? I can j...

well.. there's two ways you can achieve this.

I'd assume you're probably doing:

_, err := c.File("may-not-exist").Size(ctx)
if err != nil {
    // error might be "file doesn't exist" or something else. Only way is by parsing the error message
}

c.WithExec.... // continue your pipeline here

other option simlar to the strategy about ignoring errors (https://docs.dagger.io/cookbook/#continue-using-container-after-command-execution-fails) is

file, err := c.WithExec([]string{"sh","-c", `ls may-not-exist || echo ""`}).Stdout(ctx)
if len(file) == 0 {
   // file effectively doesn't exit
}

cc @violet cosmos

Cookbook | Dagger

Filesystem

modest nacelle
patent ermine
modest nacelle
#

if someone forgets to parse the response message and just ignores the error, that could perfectly be a false negative

patent ermine
#

not sure what you mean; in the context of wrapping calls, it's the user's responsibility to propagate errors. if they don't propagate them, we have to assume that's intentional, otherwise there's no way to do this, no?

#

propagating errors in Go would just mean the usual if err != nil, in Python it'd be an exception, etc

modest nacelle
patent ermine
#

oh yea sure - we could have better APIs for that, and it probably does trend away from dealing with err return values in the first place

#

even the ldd case I mentioned would be better off as an ExitCode check or something

#

but in that case, for example, you still wouldn't want the failed ldd exec from Buildkit to bubble up

#

so I think we still need encapsulation, but also to not have that be our entire strategy for these cases

violet cosmos
#

So:

    1. solve the viz problem with better encapsulation (sub-call errors shouldn't be a big deal if not propagated),
    1. better DX for differentiating errors I want to propagate, from errors I don't

Right?

patent ermine
# violet cosmos So: - 1) solve the viz problem with better encapsulation (sub-call errors shoul...

for 2) I think @modest nacelle's point is more that it's safer to have explicit APIs rather than lean too hard on distinguishing error cases. like maybe in this case you'd have something like exists, err := c.File("may-not-exist").Exists(ctx).

but there's already precedent for proper error types (we have one for exec errors), so we could keep doing that, it's just a bit less safe since lazy/unaware devs will just do an err == nil check and move on

#

(FWIW the .File API already does a check that the file exists and is a file, but you need to .Sync it to find out, which isn't as explicit)

violet cosmos
modest nacelle
#

I don't like the current situation with error types fwiw. I just want WithExec(&ContainerWithExecOpt{IgnoreError: true}), and propagate all errors without having to disset them. So same pattern you're describing.
this. Thing is that we'll also need that flag for all the other leaf opreations? Size, Contents, etc?

violet cosmos
modest nacelle
opaque mica
#

Do people here use the dagger CLI to run pipelines or just “create” their own?
With Golang is easy to do that, but I wonder if there is any difference in the operation?

Also are there away example repos that have many “actions” (eg apt “package”, npm, etc)
Trying to find a good way to move code into packages and make it reusable, As I have many pipelines that are almost identical expect maybe the order and/or small changes or opt-in, opt-out of actions.

shrewd turtle
# opaque mica Do people here use the dagger CLI to run pipelines or just “create” their own? ...

Hey Stavros! Have you checked out #daggernauts and the new concept of dagger modules we are introducing? Dagger modules, among other things, were thought to provide better re-usability of pipelines or individual actions that form a pipeline and to create an ecosystem that is cross CI platform. With the use of dagger modules you could write your pipeline logic in one or more module, and then simply call those functions from each pipeline. You can check out the https://daggerverse.dev/ for a list of existing modules, maybe some of the logic you need is already implemented!

Daggerverse

Find modules built by the Dagger community, or publish your own.

violet cosmos
#

Fun fact: GOOS and GOARCH don't support the same values as uname -s and uname -m respectively, and Go doesn't provide a tool or library to make that translation. They never foresaw a scenario where you're 1) cross-compiling, 2) on a remote system, 3) with the current system as a target. Which is exactly what happens when I call this:

dagger call \
  -m github.com/kpenfound/golang --proj https://github.com/kpenfound/greetings-api \
  build --os $(uname -s) --arch $(uname -m) --args . \
  file ./greetings-api \
  export --path=./greetings-api
regal seal
#

Ah I think this is why Tonis made "xx"

signal mantle
#

Packaging for rpm vs deb has the exact same issues. Deb based systems use arch strings similar to Go, and Rpms look more like uname

regal seal
signal mantle
regal seal
#

okay so i actually thought about why this is actually maybe not a good idea

#

suppose i have a module that builds a binary - the caller should be the one to decide what architecture should be set, not the callee

#

you also get some really weird results.
let's say i have a strip function that takes a File and returns a binary stripped version of that file

#

now, if i take a binary i build in, strip it, what architecture should the binary be built in? it all depends on who's expected to run it, but we don't really know that

#

docker has the exact same issue and it's a bit of a pain
i'd suggest, we could allow an explicit --platform argument (that uses docker-style names, and maybe a shorthand for current), which any module could access by dag.Arch() or similar, but it defaults to the platform of the dagger server, not of the dagger client

signal mantle
#

Yeah I'm totally with you there, in my case I'm thinking more about creating defaults for things like --platform

#

And IMO it really matters most from the CLI, so maybe it's a CLI convenience feature request and not an API request

violet cosmos
#

My very short term conclusion, is that we should not depend on any of this for our quickstart example 🙂

regal seal
#

this is a very good point yes 😄

#

is there a way we can still neatly get the binary to build? could we have the greetings-api support uname-styles args?

#

actually actually

signal mantle
signal mantle
regal seal
#

i have a really cool idea, gimme a few moments 😄 i think we could have the cli handle dagger.Platform types, and we could have it take current as a value

#

if we can do that, it's super simple, no need for uname, and we have one consistent value we get to pass everywhere (we even have a special type dedicated for this today)

violet cosmos
#

@signal mantle the problem is that in the quickstart (last 2 parts) we have the user build a binary, export it, and execute it on their local machine.

signal mantle
regal seal
signal mantle
regal seal
#

of the server that is

signal mantle
#

yeah the rest of this might as well get wrapped up in the 'host capabilities' discussion

regal seal
#

for the host, i think explicitly passing > implicit magic, i can't quite put together an exact reason why, but the dockerfile mess of xx/TARGETPLATFORM stuff was always a source of confusion when i was working on that

signal mantle
#

makes sense. At least uname -s maps nicely for os afaik

patent ermine
#

TIL about context.WithoutCancel in Go 1.21, which we should start using everywhere instead of context.Background() since it keeps all the nice context values around (e.g. opentelemetry, progrock, slog, ...): https://pkg.go.dev/context#WithoutCancel
cc @pure flare

#

(gonna do that refactor now since I had to manually propagate span contexts in a ton of places)

exotic thicket
#

@patent ermine that's a great find, not just for Dagger

tawdry scarab
#

Hi, all! 👋 Does anyone know why the dagger.gen.go file has so many type re-exports, that mostly just seem to remove useful documentation when I'm hovering over types in VS Code? 😅 Eg.

// GolangBuildContainerOpts contains options for Golang.BuildContainer
type GolangBuildContainerOpts = dagger.GolangBuildContainerOpts
image.png
#

..whereas if I use the dagger.GolangBuildContainerOpts, I see much more useful docs:

image.png
#

Obviously I can just use the dagger.XYZ definitions, but as the re-exports are already in scope, it gets a bit confusing sometimes, and I'll realize it when I hover to check what params I can use 😂

tawny carbon
#

type re-exports, that mostly just seem to remove useful documentation when I'm hovering over types in VS Code

Yeah!

I think the re-exports are there for compatibility reasons, but our documentation doesn't reflect that. I suggest you import from the subpackage instead. \cc @regal seal?

regal seal
#

The subpackage was added so those types can be accessed from other packages, without that it wasn't possible to have util sub-packages at all that used those types which was quite annoying.
But when doing that change we didn't want to break any of the existing modules.

#

Maybe this is something we could use engine version (in dagger.json) for in the future? We could keep existing modules working, but ones using new engines would need to use the subpackage types

tawny carbon
#

We can change the default template (on init) and the docs, while still keeping this file for a while. That would be a good first step I think.

regal seal
#

Updating docs sounds like a good first step imo - curious what @patent ermine and @pure flare think (as other go SDK devs)

tawny carbon
#

We need support in the SDKs for raising deprecation warnings in the TUI. Then you can also put warnings in the re-exports file for a while too.

patent ermine
#

Low-conviction +1 to switching to the the dagger. types in the long run, I feel like the auto-complete UX is slightly better. It's hard to auto-complete when you have to think about the first letter. e.g. for ctr.WithExec(args, _ (_ = cursor) you have to know you're calling against a Container type, and type C, and then complete to ContainerWithExecOpts. With dagger. you just type dagger. and then your editor should be able to deduce ContainerWithExecOpts from there. More keystrokes, fewer brainstrokes. (Though maybe @pure flare can make a compelling argument that keystrokes matter more. :P)

#

(Haven't actually tested that it works out that way, either)

tawdry scarab
#

I've understood that there have been significant changes in Dagger recently, and I appreciate hearing about the reasoning! 😄 I've had an eye on Dagger for a while, but didn't really dive in until now!

I'll make sure to go with the dagger. types for now, anyway, and just ignore the redefined ones.

placid girder
#

Hi! I'm building a new project's pipeline and I was greeted by a pipeline run timeout of 2 hours alert today. I've managed to reproduce it locally. Turns out, Directory("/src/dist").Export(ctx, "./dist/") hangs if the ./dist/ directory exists. But if I remove the directory beforehand and rerun the pipeline, it runs and exits as expected, w/o any hanging. Is this expected?

regal seal
#

argh, this sounds a lot like https://github.com/dagger/dagger/issues/6355
we're tracking the issue there, it's partially complicated by the fact that it also affects an upstream dependency of ours (in buildkit)

i've been planning to take a look at this for ages, will definitely try and find some time for it ❤️ sorry, it's been an insane couple of weeks 😅

placid girder
#

interesting. I don't mention any relation to file size. Could it be that the issue you've referred to above is happening because the OS flushes files to the filesystem as soon as the buffer hits a certain size and THEN my issue kicks in (i.e. "the file is already in the filesystem, so let's freeze and do noting")? What do you think?

#

anyways, thanks for a reply. This tells me this is not an expected behaviour and I'll design my CI accordingly (hoping for a future Dagger updates fixing this issue)

#

Then, I have this other issue in my react-native Android build:

27: [16.1s] Starting a Gradle Daemon (subsequent builds will be faster)
27: [99.0s] > Task :gradle-plugin:pluginDescriptors
27: [99.0s] > Task :gradle-plugin:processResources
panic: returned error 502 Bad Gateway: http do: Post "http://dagger/query": rpc error: code = Unavailable desc = error reading from server: command [docker exec -i dagger-engine-a0d01c49542e764e buildctl dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=
goroutine 1 [running]:
main/task.BuildAndroid()
    /opt/atlassian/pipelines/agent/build/tools/dagger-go/task/buildAndroid.go:41 +0xe54
main.init.func3()
    /opt/atlassian/pipelines/agent/build/tools/dagger-go/ci.go:15 +0xf
main.main()
    /opt/atlassian/pipelines/agent/build/tools/dagger-go/ci.go:39 +0x30d
exit status 2

Any idea where the hell his http://dagger/query come from.....? I sure don't type it in manually anywhere
I understand that you don't have my pipeline nor my code. I'm asking for any possible assistance in the capacity you can provide (perhaps you know anything about this URL).

regal seal
#

hm, that's odd, i wonder if that's caused by an engine crash - could you share the logs from the dagger engine container?

placid girder
#

unfortunately, the job is ran in a CI runner that's out of my reach. I'll try to repro it locally though.

EDIT: Nope, can't reproduce it locally.

marsh lake
#

Hello 👋 After a demo in the Dagger booth at Kubecon, I've decided to play with it. I've started with a simple app in Go 1.22.1, but I'm facing issues with the following command:

dagger -m golang call --ctr golang:1.22.1-alpine3.19 golangci-lint --source ./app

The error is the following:

    ┃ level=error msg="Running error: context loading failed: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: go.mod requires go >= 1.22.1 (running go 1.21.3; GOTOOLCHAIN=local)\n"

What am I doing wrong? Thanks in advance.

placid girder
regal seal
# marsh lake Hello 👋 After a demo in the Dagger booth at Kubecon, I've decided to play with ...

oh hey @marsh lake, sorry i completely missed this yesterday ❤️ (glad to see you from kubecon!)
so today, the codegen runs in go 1.21 - and because of how we do the codegen, we can only parse go versions 1.21 (since old go refuses to parse new go)
this seems kind of related to https://github.com/dagger/dagger/issues/6706 - previously we just got lots of silent failures (fixed by https://github.com/dagger/dagger/pull/6737)
ideally we'd be able to handle this in a smarter way, but unfortunately there's not really an easy way to be able to do it - one possible thought we've had is to decouple the go sdk tooling more from the main tooling, so we can have separate go versions

#

we might be able to upgrade just the codegen to 1.22, i'll investigate that 😄

marsh lake
# regal seal oh hey <@769210171779973140>, sorry i completely missed this yesterday ❤️ (glad ...

The thing I don't understand is how my app version impacts this 🤔 If I decrease the Go version it works.

├── LICENSE
├── README.md
├── app
│   ├── app.go
│   └── go.mod
└── dagger.json

If I pass --ctr before linting function or using with-container (https://github.com/kpenfound/dagger-modules/blob/main/golang/main.go#L170), it gives the error.
I was assuming that by using a container with the correct Go version this would work. Isn't this the container parsing my app code?
I might be missing some dagger fundamentals here 😅

regal seal
#

so i think the error you're getting is from the module right? not a dagger internal error?
the problem is @signal mantle's module uses a hardcoded linter image - if the go.mod version is too recent, then the golangci linter image is going to fail out
i think there needs to be a way to make it configurable - not quite sure what that interface should look like though

#

i do think that this does raise some interesting questions around how modules should handle image names/versions - should they hardcode versions? or allow configurability? etc

marsh lake
# regal seal so i think the error you're getting is from the module right? not a dagger inter...

Doing a docker run --rm -v $(pwd):/app -w /app golangci/golangci-lint:v1.55.2 golangci-lint run -v works.
But the error indeed seems to be from the lint container:

Error: response from query: input: golang.golangciLint resolve: call function "GolangciLint": process "/runtime" did not complete successfully: exit code: 2

Stdout:
input: container.from.withMountedDirectory.withWorkdir.withExec.stdout resolve: process "golangci-lint run -v --timeout 5m" did not complete successfully: exit code: 3

Stderr:
level=info msg="[config_reader] Config search paths: [./ /src / /root]"
level=info msg="[lintersdb] Active 6 linters: [errcheck gosimple govet ineffassign staticcheck unused]"
level=info msg="[loader] Go packages loading at mode 575 (exports_file|files|types_sizes|compiled_files|imports|name|deps) took 5.098125ms"
level=error msg="Running error: context loading failed: failed to load packages: failed to load with go/packages: err: exit status 1: stderr: go: go.mod requires go >= 1.22.1 (running go 1.21.3; GOTOOLCHAIN=local)\n"
regal seal
#

Okay, so I'm trying to add a feature to the Go SDK and need some opinions - I keep reaching for With, but I want variations that return errors/take contexts - without that, lots of our own CI code is filled with panics which is annoying.
I think I know how to handle this in the SDK itself (we give each type an err property, and can propagate that until we get to a function that can return an err). But... I'm not quite sure how these new variants should exist. Some ideas I had:

  • WithCtx/WithErr/WithCtxErr to try and capture all the options - very verbose
  • With could always take a context and return an error, and we break the API
  • With could take an interface, and we break the API: ctr.With(Func(...)), ctr.With(FuncErr()), etc

It feels like we need something to handle this, since without this, it becomes necessary to keep breaking the chain when we don't need to (or we end up with panics). Or maybe With is just the wrong pattern for these kind of cases: https://github.com/dagger/dagger/blob/f2af48d764a65d23cf5f540d8276c92b3bba5f85/internal/mage/util/engine.go#L253-L255

#

cc @patent ermine this is what i was talking about previously when i wanted "monadic error handling" 😆

tawny carbon
#

The point of With is to not break the chain. It's only for lazy functions, not the right pattern if you need to return an error.

#

I say that if you panic in a With function, it's a code smell, i.e., using the wrong pattern.

marsh lake
lethal vigil
#

Is it possible to combine two directories and generate a ZIP from them that can be exported to the host without relying on zip being installed in the container and using Go's stdlib zip package?

regal seal
#

i think so

#

so what you can do is take the two directories and .Export them into the module container (at some path of your choice) - then you can use https://pkg.go.dev/archive/zip#Writer.AddFS to put the files from your filesystem into a zip package

#

no need to shell out

violet cosmos
#

Note that this will change how the zip operation is cached, because Dagger doesn't cache execution of the function itself (yet), but it does cache further system operations (exec of a container etc). Something to keep in mind if you want to do expensive computation inside the function.

lethal vigil
#

thanks @regal seal and @violet cosmos for the hints! got it working with:

    _, err := ctr.Directory("/mnt/src").Export(ctx, "./src")
    if err != nil {
        return nil, err
    }

    _, err = ctr.Directory("/mnt/deps").Export(ctx, "./deps")
    if err != nil {
        return nil, err
    }

    zf, err := os.Create("function.zip")
    if err != nil {
        return nil, err
    }
    defer zf.Close()

    zw := zip.NewWriter(zf)
    defer zw.Close()

    if err := zw.AddFS(os.DirFS("./src")); err != nil {
        return nil, err
    }

    if err := zw.AddFS(os.DirFS("./deps")); err != nil {
        return nil, err
    }

    return dag.CurrentModule().WorkdirFile("function.zip"), nil
#

I can see a lot of potential with this new modules and functions features, cheers!

#

quick question: can a dagger call get the results of a previous dagger call? for example:

dagger call build ...   # returns a BuildResult with build info
dagger call release ... # uses previous BuildResult to create a release

I know this could be achieved with chaining functions, but I was wonder if I could call them in separate dagger calls I could create a GitHub Workflow where the diagram would contain all the different steps, instead of a single one (not a big deal, but would be great for end-users to visualize what's going on). does this even make sense?

oak crescent
#

I am curious when you're using golang dagger functions does it basically call go run for each invokation? I'd guess there is no compiled binary that gets saved or cached?

violet cosmos
#

Note that the SDK itself is a regular dagger module, made of regular dagger functions. So you can make your own SDKs without changing the engine

oak crescent
violet cosmos
#

As for its own binary vs. go run, I don't know. Try finding the relevant source in sdk/go, but no longer familiar with that part of the codebase

tawny carbon
pure flare
dense wind
dense wind
violet cosmos
# dense wind I'm confused, I'm trying to understand dagger. The docs say "Note that Dagger re...

Yeah that paragraph is confusing, sorry. There are 2 distinct points:

  1. Dagger does not really need a container runtime to work. Really it only needs a Linux kernel that can run containers.

  2. The most common way to use Dagger is by installing the CLI only. Then the CLI can bootstrap the rest of the engine, by running it in a privileged container. By default that's Docker, but can be anything. Once the bootstrapping is done, no container runtime is needed (per point 1)

dense wind
violet cosmos
#

The reason you need a container runtime to bootstrap dagger, is that the engine has many moving parts, and it's easier to package it as an OCI image to make sure it runs properly on your end.

It's also easier to run it that way on CI machines, kubernetes, etc

dense wind
#

ah gotcha

steady bluff
#

Would I be safe in saying, if you are running the Dagger CLI in Linux, you don't need a Docker(-like) container runtime, as the CLI will boostrap what is needed and will run the Dagger Engine for you. In any other OS (Windows, MacOS), you'll need a Docker(-like) runtime to work with Dagger?

#

And would it be correct to suggest running the same Docker(-like) runtime in all environments (local/ CI, etc.), just to be certain the environment is in parity, no matter where Dagger is ran?

violet cosmos
#

the next step is to add a "compute driver" system that makes it very easy to customize how and where to run it

#

(for example on a remote VM for burst-to-cloud capabilities)

violet cosmos
#

one more improvement on the horizon is to distribute the CLI + engine in a single OCI image, so you can eg. deploy a self-contained dagger installation directly to kubernetes and use it directly

#

this would also work for Container-native linux distros where you install software on the host by running containers

steady bluff
#

better to use your system's existing docker-like tooling
Ok. So, better to suggest a Docker(-like) runtime be available. Got it. 🙂
And, it doesn't matter what Docker(-like) runtime is available, Dagger will run without any hiccups caused because different Docker(-like) runtimes were used. Got it. 👍🏻 (and correct me if I am still wrong 😊 )

one more improvement on the horizon is to distribute the CLI + engine in a single OCI image, so you can eg. deploy a self-contained dagger installation directly to kubernetes and use it directly

So, Dagger modules could be installed as a Deployment, instead of having the engine running as a daemon set? 🤔

violet cosmos
#

So, Dagger modules could be installed as a Deployment, instead of having the engine running as a daemon set? 🤔

That's the goal, yes. We need to unlock a few things in the engine before it can be done reliably. The good news is, that's now the number one priority, now that we've shipped Functions and Daggerverse

#

The most important requirement is to make the engine more stateless. The main reason for the daemonset, is to keep the engine close to the kubernetes node's persistent local storage.

steady bluff
#

Ok. But stateless means also being dependent on a remote cache or Dagger Cloud? The daemon set install will still be possible though, hopefully? Or, I believe you mentioned a self-hosted cache will be available too? Ooh. If that is possible, I'd be a first taker. 😄

violet cosmos
#

Yes, daemonset will still be possible. There will just be more options available in how to deploy it

#

And yes, self-hosted cache will be available as well. We are working on all the above.

#

The visualization and collaboration features in Dagger Cloud will not be open sourced. I just want to manage expectations that not everything we build will be free and open-source. It's a product design decision each time.

slow dune
#

Hello. Does the local cache have a size limitation by default? Is it configurable?

violet cosmos
slow dune
#

🙂

violet cosmos
#

cc @signal mantle @pure flare @final fable

slow dune
violet cosmos
slow dune
steady bluff
#

Is it possible to access a file higher in the directory tree than where my module is located? I'm trying to pull in a dockerfile for building a container, but I keep getting a "no such file or directory" error. Or do files the module should use need to be in sub-directories within the module directory?

wintry wadi
steady bluff
#

Noob ahoy!

slow dune
#

buildkit/cmd/buildkitd/config/gcpolicy.g...

steady bluff
#

How can I reference a built container (one in an Engine's cache), when I make a Dagger call to publish it (via a function)?

signal mantle
#

How can I reference a built container (

steady bluff
#

How can I regenerate the querybuilder code?

#

dagger develop isn't generating it, like I thought it would.

regal seal
#

Ah is at the top level? The old top level querybuilder package is replaced with internal/querybuilder

steady bluff
#

@regal seal - Thanks for replying. I had made a mistake. I'm trying to make the dagger module a sub-package and had renamed the folder and didn't see that dagger created a whole new folder. Once I changed the dagger.json to the new name, everything started working.

#

Well, at least it seems to be working. 😄 I can call up the functions I have now. What I need to figure out is how to call the functions programmatically (if that is even possible). I'm new to all this. 🙂

rough forge
flat minnow
#

Hi, I'm trying to build an image and parallelize binary download in order to put everything in a same container but I'm little loss on how work the copy of file to a container

#

An extract of my code:

# 
        Container().
        From("ubuntu:jammy-20240227").
        WithFiles(
            binInfraBoxPath,
            []*File{
                tfDownloader.File("/opt/bin/terraform"),
                tgDownloader.File("/opt/bin/terragrunt"),
                tfDocDownloader.File("/opt/bin/terraform-docs"),
                sopsDownloader.File("/opt/bin/sops"),
            },
            ContainerWithFilesOpts{Permissions: 0755},
        ).
        WithEnvVariable("PATH", "$PATH:"+binInfraBoxPath, ContainerWithEnvVariableOpts{Expand: true})```
#

In the output of my dagger call I have that:

# 
283: > in from ubuntu:jammy-20240227
283: pull docker.io/library/ubuntu:jammy-20240227 CACHED

315: copy /opt/bin/terraform-docs /opt/infrabox/opt/infrabox/bin/terraform-docs CACHED
315: copy /opt/bin/terraform-docs /opt/infrabox/opt/infrabox/bin/terraform-docs CACHED

322: copy /opt/bin/sops /opt/infrabox/opt/infrabox/bin/sops CACHED
322: copy /opt/bin/sops /opt/infrabox/opt/infrabox/bin/sops CACHED

319: copy /opt/bin/terraform /opt/infrabox/opt/infrabox/bin/terraform CACHED
319: copy /opt/bin/terraform /opt/infrabox/opt/infrabox/bin/terraform CACHED

317: copy /opt/bin/terragrunt /opt/infrabox/opt/infrabox/bin/terragrunt CACHED
317: copy /opt/bin/terragrunt /opt/infrabox/opt/infrabox/bin/terragrunt CACHED
#

but binInfraBoxPath is equal to /opt/infrabox/bin

#

someone has an idea why the destination path is not matching /opt/infrabox/bin/{BINARY_NAME}

shrewd turtle
#

Hey @flat minnow! I'm trying to repro that here but I'm unable to. This is what I have:

func (m *Go) CopyFile(ctx context.Context) (string, error) {
    c := dag.Container().
        WithNewFile("/bin/terragrunt", ContainerWithNewFileOpts{Contents: "terragrunt"}).
        WithNewFile("/bin/terraform-docs", ContainerWithNewFileOpts{Contents: "terraform docs"})

    binPath := "/opt/infrabox/bin"
    return dag.Container().
        From("alpine:latest").
        WithFiles(binPath, []*File{
            c.File("/bin/terragrunt"),
            c.File("/bin/terraform-docs"),
        }).
        WithExec([]string{"cat", "/opt/infrabox/bin/terragrunt"}).
        Stdout(ctx)
}

And when running it it works correctly and prints terragrunt:

image.png
#

Is it possible the variable is set to a different value than expected?

flat minnow
#

I try to change the WithFiles with multiple WithFile and it works so not sure to understand why

#

I think it's not possible the value is set in a const

# 
    binInfraBoxPath = "/opt/infrabox/bin"
)

func (m *Infrabox) Build(
    ctx context.Context,
) (*Container, error) {

    infraBoxCtr := dag.
        Container().
        From("ubuntu:jammy-20240227").
        WithFiles(
            binInfraBoxPath,
            []*File{
                tfDownloader.File("/opt/bin/terraform"),
                tgDownloader.File("/opt/bin/terragrunt"),
                tfDocDownloader.File("/opt/bin/terraform-docs"),
                sopsDownloader.File("/opt/bin/sops"),
            },
            ContainerWithFilesOpts{Permissions: 0755},
        ).
        WithEnvVariable("PATH", "$PATH:"+binInfraBoxPath, ContainerWithEnvVariableOpts{Expand: true})

    return infraBoxCtr, nil
}```
#

I'm running on version v0.10.2

shrewd turtle
#

Okay, I'm testing 0.11.0. Let me see if I can repro with 0.10.2

#

Indeed, same error happens with 0.10.2

image.png
#

It seems to copy it to an incorrect place. cat fails to show the contents because the file does not exist

flat minnow
#

ok I will try to upgrade to 0.10.3 because with 0.11.0 I have a bug

shrewd turtle
#

0.10.3 worked as expected here!

image.png
flat minnow
#

same on my side 👍

#

thank you for your help

astral finch
#

Hi, i'm trying to Export container file to Host without success.

func (m *Anotherpoll) Test(ctx context.Context) (bool, error) {
    build := dag.Container().From("alpine:latest").
        WithWorkdir("/tmp").
        WithExec([]string{"/bin/sh", "-c", "`echo \"salut srm\" > bouh-file`"})

    return dag.Directory().
        WithDirectory("/tmp", build.Directory("/tmp")).
        Export(ctx, "/tmp")
}```
Do you know why it doesn't work ?
flat minnow
gleaming steppe
#

Is it possible to debug the dagger go code with the go debugger? E.g. in vscode?

tender marsh
#

Is there a way to create custom OTPL spans in the Go SDK? I think that could be useful in the Module's runtimes to split part and simplify the debugging/observability

tawny carbon
# 
ctx, span := Tracer().Start(ctx, "span name")
defer span.End()

To get current active span from context:

import "go.opentelemetry.io/otel/trace"
span := trace.SpanFromContext(ctx)
tender marsh
tender marsh
#

@patent ermine Do we display span events on Dagger cloud?

patent ermine
#

not at the moment, there's an issue for it

tender marsh
#

Okay! ty

stable pasture
#

Hi yall - new dagger user here, a bit confused about how to enable dagger modules to build a container and populate it with a PRIVATE github repo as a mounted directory. The issue I am running in to is that the module is not able to call the underlying Host's unix socket APIs. This is a good design choice from a security perspective but I am a bit at a loss as to how to write such a module correctly given that design choice. Since i intend to run this in github actions, perhaps the correct implementation is to have the GHA runner clone the repo with its embedded credentials and then simply pass that entire "directory" into the container for further operations within the dagger container. This would mitigate the need to have dagger authenticate as an SSH user and subsequently clone the given private repo.

For reference heres a relevant error message:

invoke: input: host.unixSocket resolve: only the main client can access the host's unix sockets

Here's an excerpt from my module which is meant to build and return a container with the private github repo mounted. I would draw your attention to the dagger.GitOpts{} parameter that i am passing to dag.Git() - it essentially only allows me to pass in a path to a UNIX socket. Of course, this module can't actually interact with said socket..

func (g *Github) Container(sshSocketPath string) (*Container, error) {
    sshAgentPath := os.Getenv(sshSocketPath)

    repo := dag.Git(
        g.URL,
        dagger.GitOpts{
            SSHAuthSocket: dag.Host().UnixSocket(sshAgentPath),
        }).
        Branch(g.Branch).
        Tree()
    if repo == nil {
        return nil, fmt.Errorf("invalid Git repository or branch: %s/%s", g.URL, g.Branch)
    }

    cntr := dag.Container().
        From("alpine:latest").
        WithDirectory("/src", repo, dagger.ContainerWithDirectoryOpts{})

    g.Cntr = cntr

    return cntr, nil
}
regal seal
#

The workaround for the time being would be to pass the secret key as a *Secret (or even, use clone-over-HTTPS with the feature to be released later today :D)

#

The above issue is a bit more involved than it might first look, but we're making progress towards it!

stable pasture
#

oh interesting! thanks @regal seal for your response - i assumed it was a security related design choice - a feature, rather than a bug

regal seal
#

so the idea is that dag.Host shouldn't be accessible from modules really at all - but the problem is that not every feature in dag.Host has a corresponding way to transfer things at the moment

#

yet 😄

tawny carbon
#

@stable pasture, if you're able to pass the repo address as an argument (string in the CLI, Directory in code), the CLI will use your ssh agent for you. See #1224350244620468345 message:

func (g *Gitbub) Container(repo *Directory) *Container {
    cntr := dag.Container().
        From("alpine:latest").
        WithDirectory("/src", repo)

    g.Cntr = cntr

    return cntr, nil
}

Test with:

dagger call container --repo=git@private-repository.git directory --path=/src entries

One advantage of this approach is you can easily replace the repo with a local directory or somewhere else for testing/debugging, without having to change the code.

stable pasture
#

@tawny carbon oh thats actually really cool, i wonder if this would work in a github actions environment

#

kinda definitely feels like magic, is this behavior of WithDirectory() expected to be stable? thinking ahead, i would want to implement whichever solution is designed to fit this use case and thus be the most stable

#

that said i dont mean to offend by calling it magic, its just behavior that I wouldn't have expected, though pleasantly surprised

tawny carbon
stable pasture
#

got ya so as long as we operate the CLI in github actions, behavior should be consistent

tawny carbon
#

Yep

stable pasture
#

wonderful, well thank you @regal seal & @tawny carbon for your support on this issue

#

i am keen to learn more about the new release with clone-over-HTTPS, if yall can send more info when you get a chance

tawny carbon
desert plinth
tawny carbon
#

@desert plinth, since we're in the Go channel:

dagger init --sdk=go --source=test test

Replace main.go with:

package main

import (
    "dagger/test/internal/dagger"
)

type Test struct{}

func (m *Test) Repo(url string, token *dagger.Secret) *dagger.Container {
    return dag.Git(url).
        WithAuthToken(token).
        Head().
        Tree()
}

Test it with:

dagger call repo --url=https://github.com/my/private-repo.git --token=env:GITHUB_PAT entries

Change appropriately for the repo's address and the env var where you have your access token, it should list you the files in your repo.

desert plinth
#

you're the best man thanks so much

agile mural
#

Is it just me, or there really isn't no way to force dagger to create go.mod in a newly created module (or rather it's source directory).

dagger init updates go.mod in my root module which is 100% not what I want.

violet cosmos
agile mural
violet cosmos
agile mural
violet cosmos
tawny carbon
#

Yeah, that's what should happen. Just tested and it's what I get in v0.11.1:

dagger init --sdk=go --source=mark

Puts the sources, including go.mod in mark subdir, while leaving dagger.json in current dir and untouching current dir's go.mod.

#

If you want dagger.json in the subdir as well:

dagger init --sdk=go --source=mark mark

(removing the --source here, puts the sources under mark/dagger)

tawny carbon
agile mural
tawny carbon
#

Yeah, can't reproduce. That creates go.mod in dagger/ for me, without messing the root's go.mod.

pure flare
pure flare
tawny carbon
#

From the root of a git dir, actually. Specifically, my daggerverse repo. Made sure to have a go.mod first.

agile mural
tawny carbon
#

What Mark's saying could happen if you do dagger init --sdk go --source . in the root of the repo though.

#

@agile mural, can you check if you have a dagger.json in the root of that repo locally?

agile mural
#

I do not. This repo has no dagger stuff in it yet

tawny carbon
#

Ok, let me check.

#

@agile mural, I can reproduce it now. Thanks for the repo, that's helpful.

#

Happens, on v0.10.3 too. Curiously doesn't happen on my repo.

tawny carbon
#

Ok, the cause is having .go files at the root. In my repo I added go.mod et all, but no .go files at the root. Our repo also doesn't have .go files at the root.

#

Go SDK's codegen is considering that it's a module with existing sources.

soft sparrow
#

How does one call a function from another function in the same module?
This doesn't work because Function1 has optional arguments

func (m *Module) Function(
//+optional
arg1
) {}
func (m *Module) Function2() {
  m.Function()
}

And this

func (m *Module) Function() {}
func (m *Module) Function2() {
  dag.Module().Function()
}

doesn't work because...

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "golang" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1

Stderr:
# dagger/module
./main.go:162:19: dag.Module undefined (type *dagger.Client has no field or method Function)
agile mural
soft sparrow
violet cosmos
#

This is a rough edge that we plan on addressing, by adding support for "self calls". This will allow intra-module function calls via the dagger API, instead of directly within the same process. That will help make the DX more consistent (no more "forced to set all optional arguments" weirdness), and it will also register in the terminal UI and Dagger Traces for example.

(Was going to paste an issue, but I'm not sure we have one)

reef sapphire
#

Is there a known way to combine dagger directory mounts with e.g. a .dockerignore or .gitignore? I work in a monorepo that gets a ton of stuff built in target directories, downloaded to various node_modules, terraform providers downloaded from inits in various different stacks, etc. The total size when I first started trying to mount was north of 45GB, and that mount was slowwww. I since cleaned up all those aforementioned directories, although some unnecessary things are still hanging around here and there. They're all defined in .gitignore, so if it was possible for me to do a local mount following that ignore file, that would be ideal. If this is something that already exists, I haven't found it yet.

Edit: Found it -- I imagine this is exactly what https://pkg.go.dev/dagger.io/dagger#DirectoryWithDirectoryOpts.Exclude is for.

tender marsh
#

I spotted an issue if I try to use Dagger in a Go Project that pull code from private dependencies, for example buf
https://buf.build/

When I try to do a dagger init xxx, it tries to pull these dependencies but since the module isn't logged in at the init, it cannot.
Is there a way to workaround this?

Buf

Accelerate API development with Buf, the only end-to-end developer platform for Protocol Buffers and gRPC.

#

The best workaround would be to not use the parent go.mod during the init, since dagger should only need its own module dependencies

agile mural
steady bluff
#

So, I now have a client-based function running similar to this recipe, but I also must run it with dagger run go run as that is the only way to run the code in a "dagger session". How can I get a dagger session going programatically? Or, should I simply be calling CLI commands programmatically like in this Jenkins example? I believe I'm missing something as to why there is such a CLI-centric approach with working with Dagger or rather, someone should just say "Scott, you'll be better off just calling CLI commands via code to get Dagger to do its thing, than trying to get Dagger to do its thing 100% programmatically." 🤔

signal mantle
#

So, I now have a client-based function

agile mural
#

Anyone saw this before? (Happening on dagger init --sdk go --name ci)

go: asd/dagger imports
        github.com/sagikazarmark/demo-kcd-romania-2024/dagger/dagger/internal/dagger: git init --bare in /go/pkg/mod/cache/vcs/3878b39be72664b46127f191cf72ab6cf3e9d1db601741c4f80368ffd98c4d03: exec: "git": executable file not found in $PATH
go: asd/dagger/internal/dagger imports
        github.com/sagikazarmark/demo-kcd-romania-2024/dagger/dagger/internal/querybuilder: git init --bare in /go/pkg/mod/cache/vcs/3878b39be72664b46127f191cf72ab6cf3e9d1db601741c4f80368ffd98c4d03: exec: "git": executable file not found in $PATH
#

Happens in this repo when trying dagger init in a subdir: https://github.com/sagikazarmark/demo-kcd-romania-2024

Interestingly, the error is completely different in the root dir:

Error: generate code: template: alias.go.tmpl:74:3: executing "_dagger.gen.go/alias.go.tmpl" at <ModuleMainSrc>: error calling ModuleMainSrc: cannot find default codegen DaggerObject interface
undone ether
# agile mural Anyone saw this before? (Happening on `dagger init --sdk go --name ci`) ``` go:...

I'm encountering it too using dagger v0.11.1 -- dagger init ; dagger develop --sdk=go works fine in an empty directory but in a pre-existing Go project running it from the root (and a dagger subdir I made) it gives similar errors:


Stdout:
generating go module: query-analyzer
writing dagger.gen.go
creating directory internal
creating directory internal/dagger
writing internal/dagger/dagger.gen.go
creating directory internal/querybuilder
writing internal/querybuilder/marshal.go
writing internal/querybuilder/querybuilder.go
creating directory internal/telemetry
writing internal/telemetry/attrs.go
writing internal/telemetry/batch_processor.go
writing internal/telemetry/init.go
writing internal/telemetry/processor.go
writing internal/telemetry/proxy.go
writing internal/telemetry/span.go
writing main.go
running post-command: go mod tidy
post-command failed: exit status 1
Stderr:
go: downloading github.com/<private>/<repo> v1.7.2
go: downloading github.com/<private>/<repo> v1.28.0
go: github.com/<private>/<project> imports
        github.com/<private>/<project>/kvp: git init --bare in /go/pkg/mod/cache/vcs/376c9c60a03e84801bde36bb8a6bd95903f25ee51f4f9c43c9ec63643bd0f944: exec: "git": executable file not found in $PATH
#

I suspect it's because the alpine container used to run /usr/local/bin/codegen here https://github.com/dagger/dagger/blob/3266131d6d24f4fc3109d98351456e6365701e7b/ci/build/sdk.go#L127 doesn't have git baked into it (because alpine 💀 ) and in most cases this is fine until we hit go mod tidy further into the process (https://github.com/dagger/dagger/blob/main/cmd/codegen/generator/go/generator.go#L70) for private modules already in the go.mod I suppose?

GitHub
dagger/ci/build/sdk.go at 3266131d6d24f4fc3109d98351456e6365701e7b ...

Application Delivery as Code that Runs Anywhere. Contribute to dagger/dagger development by creating an account on GitHub.

GitHub
dagger/cmd/codegen/generator/go/generator.go at main · dagger/dagger

Application Delivery as Code that Runs Anywhere. Contribute to dagger/dagger development by creating an account on GitHub.

regal seal
#

i tried adding this a while back, but got insane issues when doing it - i'll try reviving this 😄

tawny carbon
#

Just yesterday I canceled an issue for adding git to all runtime containers 😄

regal seal
#

cancelled? 😄 any reason why?

#

i guess it ramps up space usage quite a bit 🤔 i wonder if we could do some funkiness where the engine adds git and similar tooling from the host engine into the runtime container

tawny carbon
#

Was just trying to clean up and seemed like something in limbo. My reasons were

  • Most people may not need it and it adds time to the build because you need to update the sources and install the dependency, but granted that it should be cached after first run.
  • One less requirement for sdks to keep up to date. While any project in any language may have requirements from git, that's actually very central to Go's dependency management so maybe it's more commonly needed in Go than others? Considering that other than forks, my usual need to install a dependency from git was for private repos and that's another problem.
  • Py and TS have the base to accept user configuration when building the module, which is missing from Go atm. You can specify your own base image in Python for example. If we make it simple or at least accessible to add system dependencies, then it's less necessary to always add git.
regal seal
#

mm, agreed that we probably shouldn't add it to all runtimes, it definitely is more central to go's dependency system - the proxy is really just a convenience, the "canonical" way to install is through git i think

#

i like the idea of maybe being able to configure runtimes, but it is quite nice how we've been able to not do that for this long (maybe it's needed as part of general go versioning stuff as well, which is nightmarishly annoying)

ebon urchin
#

Does anyone know what the value of _EXPERIMENTAL_DAGGER_RUNNER_HOST should be? In my scenario, I aim to execute the Dagger build code within a service against the Dagger engine.

or
how to define url in below code

client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
    if err != nil {
        panic(err)
    }
violet cosmos
#

If that's the case, then you just need to set the experimentalPrivilegedNesting argument to true in your withExec call (the one running the dagger client). Then everything will work out of the box.

ebon urchin
violet cosmos
ebon urchin
violet cosmos
#

Oh I see. Then the simplest way to deploy is to simply run your client tool, and the Dagger SDK will bootstrap the dagger engine from there. Make sure the pod is privileged.

viscid dock
#

Hi guys! I'm quite new to Go and programming in general. Does anyone have any examples of how you can use the Docker or Terraform modules to make your own functions that you later can use in a CI/CD pipeline.

placid girder
# viscid dock Hi guys! I'm quite new to Go and programming in general. Does anyone have any ex...

Welcome!

I'm not that proficient in Go myself. What helped me start with Dagger modules is:

  1. go to https://daggerverse.dev/
  2. pick any module, open its description
  3. click on the [Open Source Code] button on the right
  4. Look at how it's implemented in code level.

To create a new module -- dagger init --sdk=go my-module , open up the main.go file, edit and save it (if you need to use some other module in your module -- dagger install github.com/url/to/your/wanted/dependency-module and it'll become available at dag.DependencyModule). When you're done, either run your module locally (dagger -m ./path/to/your/module/dir/my-module call myFunction --arg 'hellooooo') or push it to github and load it the same way, except using the github URL instead of a local path (url w/o https://). Docs are quite helpful with writing function args.

Daggerverse

Discover and share modules full of Dagger functions encapsulating the community's devops knowledge

devout cypress
#

I'm trying to do a proof of concept with dagger and I want to show something similar to https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners where the image is built natively on different dagger hosts and then merged into a manifest later. I assume for setting up my client I'd need to connect to a separate dagger engine for each platform and start the builds on each and use Publish to push them up to my registry. As for the part about merging the digests into a single manifest, what's the best way to do that?

Docker Documentation
Multi-platform image with GitHub Actions

Build for multiple architectures with GitHub Actions using QEMU emulation or multiple native builders

cosmic wyvern
#

Multi-platform image with GitHub Actions

regal seal
#

@patent ermine wdyt about the idea of having a custom errgroup implementation that takes a string/similar for each eg.Go? That way, we could have that automatically generate spans, which would be pretty neat 😄 I think @tawny carbon did something similar for python

patent ermine
# regal seal <@108011715077091328> wdyt about the idea of having a custom errgroup implementa...

If there's overlap with self-calls, I think we should just do that, since it keeps coming up, and with self-calls you'd automatically get everything named for you by virtue of the function names. I'm not sure an eg.Go pattern generalizes to most cases, since e.g. for integration tests you'd want something integrated to t.Cleanup instead. I've got a slight opposition to helpers but mostly out of principle ("we're not an otel shop so we shouldn't have to maintain a ton of user-facing otel plumbing on top of the official otel stuff")

#

ideally folks would just use otel-contrib-go or something imo

regal seal
#

yeaaa, fair point 😄 i guess i just want self calls 😛

#

again, context for this comes back to https://github.com/dagger/dagger/pull/7272, where i'm trying to work out how to get something that's at all readable when you can't just refresh the entire terminal (making good progress though)

patent ermine
#

nice, yeah looking good! streaming all this is definitely challenging

#

re: the last comment, I wonder if you could get away with re-using the numeric indices as back-references. 4: 3 > upload ... instead of repeating the 3 header. Maybe a dumb idea, but maybe there's something there

regal seal
#

Yeah I thought about this, I like the idea

#

Is there any reason we need the numbers actually? I guess they're for tracking things over time, but they are a little tricksy by themselves

#

I'm undecided if it's worth exploring a design where they're just ... gone

patent ermine
#

They're pretty nice just for confirming that something that completed way later is indeed the same thing that started a while back

#

I always found them useful or at least comforting in Buildkit's console output 🤷‍♂️

#

but hey, anything's worth a try

regal seal
#

Yes I also find them comforting 😂

patent ermine
#

honestly I remember staring at this ages ago while I was adding console mode to Progrock, and I came away from it thinking they basically nailed it haha

#

couldn't really think of anything to take away, it works in a particular way, but almost every ounce of output serves a purpose

#

could maybe do without the repeated names, but even those are useful, again in that case where something finishes long after it started

regal seal
#

Yeah the problem now is, we added OTEL calls, and now we have more info to display

#

Like you can give things context now which is really nice

#

E.g. "why in the world is go mod tidy running?" is now actually answerable

patent ermine
#

we had some of that before, with the old pipeline/groups system, which amounted to putting "> in [group name]" beneath the vertex name in blue. but yea, it's all a bit different now

agile mural
slow dune
#

Parallel builds across chained actions

devout cypress
#

I'm working on a custom client and looking at examples like: https://archive.docs.dagger.io/0.9/cookbook#aws-cloud-development-kit it looks I basically need to call client.Container().From("my/base/image").WithExec([]string{"curl", "ifconfig.co"}) . Is there a way for me to execute actual go code in the engine or will I need to use the dagger cli to do that? Is this something that would be enabled when https://github.com/dagger/dagger/issues/5993 is done?

GitHub
Call Dagger Functions from an external client · Issue #5993 · dagge...

Right now if you want to call functions available in a module, the only reasonable ways are from another module or from the CLI. Otherwise you are on your own to load the module and construct graph...

Cookbook | Dagger

Filesystem

devout cypress
#

Authenticate to EKS from dagger function

delicate reef
#

Is use of go workspace in modules currently pretty discouraged right now due to limitations? I saw some rumblings to that extent in another channel I think

agile mural
#

I use workspace locally, when working on modules, so LSP is happy (though it isn't always), but I ignore workspace files everywhere else.

violet cosmos
#

@delicate reef there are rough edges but it's doable. I personally got annoyed at Go for forcing me to use workspaces just for my IDE auto-complete to work in a multi-module repository. So now switched my preference to single-module repositories just so I wouldn't have to deal with them

#

the main issue I kept running into, is that go workspace doesn't like several go modules being called main; but that's what dagger init generates in the boilerplate. So I had to manually change the go module name after the fact, which was a PITA

#

then I had to rename one of my modules, and that was the last straw for me

#

If anyone's interested, I have a dagger function that can take any git repo and "spin out" a subdirectory into its own git repo, with history preserved 😛

delicate reef
#

Thanks for the detailed answer! I am guessing that there are some circumstances where you do want dagger init to default to main, but perhaps that can be configurable or a flag eventually

violet cosmos
#

I think @regal seal gave me a detailed answer, which I forgot... He's in UK so will see this in his morning

#

(good to see you @delicate reef btw!)

agile mural
#

Technically, it could very well be a fully qualified module name (ie. github.com/owner/repo/…), but init can’t figure that out.

regal seal
#

go workspaces are a pain
they should, however, "work". the general setup is that if you have a go.work at the top-level of your daggerverse repo/etc, then when you init new modules, dagger will update the go.work to include your new module/etc.

#

there's some real funkyness with dependencies though. go.work tends to like having similar dependencies in each of the modules.
this can be an issue, if the dagger module needs a different version of a dependency than another module in the workspace (e.g. otel is a particularly annoying candidate here).

#

i have no great answers for how to solve this sadly:

  • we can rework our otel usage a bit to "try" and be compatible as much as possible (increased risk of silent failures, and also because of otel tooling this is fiddly)
  • we could fork/vendor the otel stack for go modules, and then there's no risk of clashing (this feels like suffering and pain)
#

the issue to me seems that go workspaces serve two very different purposes: ide support for multiple modules together, and unifying dependencies - it does both of these things at the same time. the problem is, we really only want the first, and the dependencies should be mostly handled by dagger alone.

#

sorry, long and rambly, but more specifically: if you hit a specific error or issue with using go.work in your dagger module, this is definitely in-scope - i shall do my best salute

shrewd turtle
violet cosmos
modest nacelle
violet cosmos
#

couldn't find it in the browser

modest nacelle
#

if you click on the function name the URL changes

modest nacelle
violet cosmos
#

Ah thanks. Yeah probably needs a visual cue that it's a link

#

doesn't seem to work on mobile

#

anyway @shrewd turtle :

dagger call -m github.com/shykes/git  \
 clone --url github.com/shykes/daggerverse \
 filter-subdirectory --path hello \
 directory \
 export --path=shykes-hello
restive hollow
modest nacelle
#

doesn't seem to work on mobile

yes, that's because it's only possible via the sidebar index. Will ship a quick fix in a minute 🙏

modest nacelle
patent ermine
modest nacelle
patent ermine
#

oh cool lol

#

we need a :fnf:

#

best i can do for now

brave vine
#

Is there a reason that the generated .gitignore doesn't ignore internal/telemetry?

regal seal
#

what version of dagger are you using?

#

there was a bug about this a while back

ornate seal
#

Is there a way to limit the output to stdout/stderr from services? Either write it to a file or only write it when encountering an error(if that's even possible)?

tender marsh
regal seal
ornate seal
#

awesome, I'll have a look. Thanks

brave vine
#

How are folks testing their functions? Should I shell out to dagger call?

violet cosmos
lethal vigil
#

Is there a workaround to support interfaces in our modules' type? for example:

type Builder interface {
    DaggerObject

    Build(context.Context, *Directory) (*Container, error)
}

type MyBuilder struct {
    DaggerObject
}

func (m *MyBuilder) Build(context.Context, *Directory) (*Container, error) {
    return nil, nil
}

type MyModule struct {
    Builders []Builder
}

func New() *MyModule {
    return &MyModule{Builders: []Builder{MyBuilder{}}}
}

func (m *MyModule) Build(ctx context.Context, source *Directory) error {
    for _, b := range builders {
        b.Build(ctx, source)
    }
}

With something like the above it errors when dagger call is called:

Error: response from query: input: mymodule resolve: call constructor: process "/runtime" did not complete successfully: exit code: 2

Stderr:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x52f768]

For more context, here's the place where the panic is happening (in generated code):

func (r MyModule) MarshalJSON() ([]byte, error) {
    var concrete struct {
        Builders any
    }
    concrete.Builders = r.Builders
    return json.Marshal(&concrete)  // <--- panics
}
violet cosmos
#

@lethal vigil I believe we do support interfaces in the Go SDK, but I haven't found how to use it either 😅

lethal vigil
#

There's this piece of documentation: https://docs.dagger.io/manuals/developer/interfaces but from my understanding and trials it's use case is to call other modules (deps) and being able to pass arbitrary arguments, as long as they implement the interface.

Not sure what's the difference implementation-wise tbh, still need to deep-dive more into Dagger internals/codegen (shame on me). Something related to marshalling/unmarshalling from what I can see.

lethal vigil
#

Well... I can just read the configuration in each phase/func (i.e. build, test) as a workaround, instead of passing it through the module's struct 😅

#

Just adds a bit of overhead if I'm chaining like dagger call build test archive, but shouldn't be a big issue.

violet cosmos
#

@tawny swallow does MarshalJSON play a special role here?

frozen raft
#

hey folks, I'm using the client like

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))

and was wondering if possible to only get this kind of "progress" logs?

✔ connect 1.4s
✔ Directory.asModule: Module! 1.9s
✔ Host.directory(path: "/some/path"): Directory! 0.7s
  ✔ upload /some/path from AA123230 (client id: h1xmcqimzdpf6bmh12vy7i0rr) 0.7s
✔ Ci.buildImage(
    actor: "actor"
    source: ✔ Host.directory(path: "/some/path"): Directory! 0.0s
    token: ✔ setSecret(name: "GITHUB_TOKEN"): Secret! 0.0s
  ): Container! 5.9s
  ✔ Gradle.fromVersion(version: "8.5"): Gradle! 0.4s
  ✔ Gradle.withDirectory(
      src: ✔ Host.directory(path: "/some/path"): Directory! 0.0s
    ): Gradle! 0.3s
  ✔ Gradle.container: Container! 0.7s
    ✔ Container.from(address: "gradle:8.5"): Container! 0.4s
      ✔ remotes.docker.resolver.HTTPRequest 0.4s
        ✔ HTTP HEAD 0.4s
  ✔ Testcontainers.setup(
      ctr: ✔ Gradle.container: Container! 0.0s
    ): Container! 1.0s
    ✔ Docker.daemon: DockerDaemon! 0.2s
    ✔ DockerDaemon.service: Service! 0.5s
      ✔ Container.from(address: "docker:dind"): Container! 0.1s
        ✔ remotes.docker.resolver.HTTPRequest 0.1s
          ✔ HTTP HEAD 0.1s
  ✔ Container.file(path: "./some/path/file.jar"): File! 2.2s
    ✔ start dockerd-entrypoint.sh dockerd --tls=false --host=tcp://0.0.0.0:2375 3.3s
#

Because currently Im getting a lot more like

Creating new Engine session... OK!
Establishing connection to Engine... 1: in exec docker start dagger-engine-835668aa455cbb8f
1: dagger-engine-835668aa455cbb8f

Connected to engine 82697476ca5b (version v0.11.4)
OK!
2: in starting session
2: OK!

3: in Query.checkVersionCompatibility
3: 03:03:12 DBG Using development engine; skipping version compatibility check.

4: in starting session
4: OK!

5: in starting session
5: OK!

6: in starting session
6: OK!

7: in starting session
7: OK!

8: in starting session
8: OK!

9: in starting session
9: OK!

10: in starting session
10: OK!

11: in starting session
11: OK!

12: in check i454t3k1eg5q4.i2l6ceg5qo6qi.dagger.local 2375/tcp
12: polling for port i454t3k1eg5q4.i2l6ceg5qo6qi.dagger.local:2375
12: port not ready: dial tcp 10.87.0.75:2375: connect: connection refused; elapsed: 612.791µs
12: port not ready: dial tcp 10.87.0.75:2375: connect: connection refused; elapsed: 68.800833ms
12: port not ready: dial tcp 10.87.0.75:2375: connect: connection refused; elapsed: 170.900208ms
12: port not ready: dial tcp 10.87.0.75:2375: connect: connection refused; elapsed: 334.972958ms

13: in start dockerd-entrypoint.sh dockerd --tls=false --host=tcp://0.0.0.0:2375
13: iptables v1.8.10 (nf_tables)
13: iptables v1.8.10 (nf_tables)
13: time="2024-05-29T03:03:17.833074167Z" level=info msg="Starting up"
13: time="2024-05-29T03:03:17.835422834Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:2375"
13: time="2024-05-29T03:03:17.835437084Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:2375"
...
tender marsh
regal seal
#

hm, @frozen raft you might be interested in upgrading to v0.11.5? I think the new plain progress logs should probably solve the issue you're having.

frozen raft
frozen raft
tawny swallow
violet cosmos
#

sorry about that!

tawny swallow
# violet cosmos sorry about that!

not the first time I ended up being the wrong Kyle... it was super funny to see when somebody at KubeCon EU ended up being sent to me instead of the right Kyle 🙂

rough forge
#

dagger-engine@v0.11.5 for linux/arm64 (prue host without qemu) may pull wrong arch then cause exec format error
dagger-engine@v0.11.4 works well.

seems something not respect the request platform.

signal mantle
rough forge
# signal mantle Are you able to see the sha of the image you pulled? I have 0.11.5 arm64 working

pull seems work well.
it break when in RUN

error logs after creating q4djs8qyoy48so0ew092y10uo [/.init /bin/sh -c if 

level=warning msg="error resolving container(platform: \"linux/arm64\").from(address: \"docker.io/library/debian:bookworm-slim\").withEntrypoint(args: [\"/bin/sh\"]).withEnvVariable(name: \"LINUX_MIRROR\", value: \"http://repo.huaweicloud.com\").withWorkdir(path: \"/\").withUser(name: \"root:root\").withExec(args: [\"-c\",sha256:b8d2076367121fe1e7f0b39aaa3de4ad9d6360af7505e8eccc12552994a89180:553:\"if [ \\\"${LINUX_MIRROR}\\\" != \\\"\\\" ]; then\\n\\t\\ti...tc/apt/sources.list.d/debian.sources\\n\\t\\tfi\\nfi\"]).sync: ContainerID!" error="map[error:process \"/bin/sh -c if [ \\\"${LINUX_MIRROR}\\\" != \\\"\\\" ]; then\\n\\t\\tif [ -f \\\"/etc/apt/sources.list\\\" ]; then\\n\\t\\t\\t\\tsed -i \\\"s@http://deb.debian.org@${LINUX_MIRROR}@g\\\" /etc/apt/sources.list\\n\\t\\t\\t\\tsed -i \\\"s@http://security.debian.org@${LINUX_MIRROR}@g\\\" /etc/apt/sources.list\\n\\t\\tfi\\n\\t\\tif [ -f \\\"/etc/apt/sources.list.d/debian.sources\\\" ]; then\\n\\t\\t\\t\\tsed -i \\\"s@http://deb.debian.org@${LINUX_MIRROR}@g\\\" /etc/apt/sources.list.d/debian.sources\\n\\t\\t\\t\\tsed -i \\\"s@http://security.debian.org@${LINUX_MIRROR}@g\\\" /etc/apt/sources.list.d/debian.sources\\n\\t\\tfi\\nfi\" did not complete successfully: exit code: 1 kind:*buildkit.ExecError stack:<nil>]"

What this /.init ?

I haven't found it in v0.11.4

I think it not aarch64 format, which cause the issue

Btw i always remove qemu in may arm64 host to make sure it is prune.
docker.io/tonistiigi/binfmt:latest --uninstall qemu-*

buildctl debug workers
ID                              PLATFORMS
ykmli5r2z1l0z7on6t1atfi20       linux/arm64
#

It is, i copy /usr/local/bin/dumb-init from ghcr.io/dagger/engine:v0.11.5 linux/arm64.

file dumb-init
dumb-init: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped

which is x86-64

arctic niche
rough forge
violet cosmos
turbid nacelle
agile mural
# 
mkdir dagger
cd dagger
go mod init dagger
dagger init --sdk go --source .
#

I think this should do it

quiet hinge
#

how can I call a module from a PR? something like #pull/73/head?

agile mural
#

call as in from the CLI? or install it as a dependency?

#

CLI: dagger call -m github.com/you/module@REF

quiet hinge
#

call as in from the CLI

#

@ref could a PR link?

#

smth like #pull/id/head

agile mural
#

any valid ref that git pulls by default

#

refs/pull/126/merge

quiet hinge
quiet hinge
agile mural
#

It should be dagger call -m github.com/Excoriate/daggerverse/goreleaser@refs/pull/73/merge check --src=".",

quiet hinge
#

thanks it worked!!

frozen raft
#

hey folks, question;
is it possible to run a session "silent" with no logs at all? I've tried with no luck:

    client, err := dagger.Connect(ctx)

and

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(nil))
violet cosmos
#

@frozen raft how do you run your client? Is it wrapped with dagger run ?

violet cosmos
#

@signal mantle FYI I started messing with ci/std/go in the dagger/dagger repo. I know we talked about you merging your golang module into that at some point. Did you start yet? Want to make sure I don't cause painful conflicts

#

(Context for me is making Lint smarter, ie. have it return a queriable test report rather than just an error)

signal mantle
violet cosmos
#

I'm learning fun things about the go test report format

frozen raft
tulip matrix
#

segfault when chaining struct with custom type

violet cosmos
frozen raft
frozen raft
#

hey folks, is it possible to start the engine session not in debug mode?

5   :   exec docker run --name dagger-engine-32e0269fab8e9d98 -d --restart always -v /var/lib/dagger --privileged registry.dagger.io/engine:v0.11.6 --debug DONE [0.3s]
grave pelican
#

Hey guys, I can't find it anywhere but do we have some examples on how to create test and debug for my dagger module? I've checked some modules from the daggerverse and can't see any pattern on people are doing it as well.. thanks

violet cosmos
violet cosmos
# violet cosmos Hello! Here is a thread discussing testing patterns for modules: https://discord...

For debugging, a few techniques that work well for me:

  • Good old printf + looking at logs 🙂
  • The new Traces feature in Dagger Cloud is very useful here. I would say debugging is the killer feature of Traces. https://dagger.io/cloud. You can hook it up with dagger login (requires the latest dagger installed)
  • Often debugging involves inspecting the state of a container. For example, if a command fails, were the right files available at the right place in the container? Was the env variable set correctly? Etc. To debug this, I like to have an intermediary function that returns a container, and call it from the CLI, followed by a call to terminal. For example in our own CI, the function go().env() builds a container with our source code + ready-to-use go dev environment. Various pipelines in our CI run commands from that container. I can debug it at any time with: dagger call -m github.com/dagger/dagger --source https://github.com/dagger/dagger go env terminal
ornate seal
violet cosmos
patent ermine
violet cosmos
patent ermine
#

woops 😛

violet cosmos
turbid nacelle
#

wondering if anyone else is using a monorepo- im noticing that caching tends to perform pretty poorly when mounting a directory of reasonable size (1gb and up), so looking for some better patterns here. maybe creating a container for each testable unit?

grave pelican
violet cosmos
ornate seal
violet cosmos
brave vine
#

It's not entirely clear to me when Dagger will handle concurrency stuff and when I should use normal Go concurrency patterns. It seems like dagger will only do it for me if I'm calling builtin dagger stuff like myCtr := dag.Container().Sync() or whatever, but if I have some functions that are plain go code, I would need to implement the concurrency pieces myself? Is that accurate?

brave vine
#

It's not entirely clear to me when

frozen raft
regal seal
#

yes, you can configure this, but you need to manually start the engine using docker

#

then you can configure the cli to connect to it using the _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://name-of-your-container env var setting

#

atm it's a bit clunky - we are working on improving this in the future though

regal seal
#

@dusky chasm @shrewd turtle, moving from #releases to here

#

(can we make that channel announce only? it shouldn't be a general discussion channel)

#

i'm confused, the bump to go 1.22 is actually intentional

#

oh i see, but it changed for v0.11.8-rc2, but not for v0.11.7

#

i don't neccessarily think it's an issue though - v0.11.7 is a retracted release, we don't have to be bound to compatability with it

frozen raft
violet cosmos
turbid nacelle
#

i have kind of a dumb general docker/go question- what is the best practice for building a go binary for use inside a docker image? is there any advantage to building it outside and copying it into the image in the dockerfile? or is it always better to do the whole process inside a dockerfile

marble dragon
#

I'm not sure this is the correct place. But I wanted to share a pattern I'm using. In case it helps anyone. I'm using GHAS Dependabot to update Dockerfiles with upstream changes. Dependabot has no idea how to identify image tags in Go code. So I created a function to extract the base tag from a Dockerfile, which is in the sub-directory of a module:

func (m *Dockerfile) ExtractBaseTag(ctx context.Context, file *File) (string, error) {
    contents, err := file.Contents(ctx)
    if err != nil {
        fmt.Println("Error reading Dockerfile")
        return "", nil
    }

    scanner := bufio.NewScanner(strings.NewReader(contents))

    var tag string

    for scanner.Scan() {
        line := scanner.Text()
        if strings.HasPrefix(line, "FROM ") {
            tag = strings.TrimSpace(strings.TrimPrefix(line, "FROM"))
            break
        }
    }

    if tag == "" {
        fmt.Println("No FROM line found in Dockerfile")
        return "", nil
    }

    return tag, nil
}

And then use that function to read individual Dockerfile for each of my other modules:

func (m *Checkov) BaseContainer(ctx context.Context) *Container {
    checkovTag, err := dag.Dockerfile().ExtractBaseTag(ctx,
        dag.CurrentModule().Source().
            File("dockerfiles/checkov.Dockerfile"))
    if err != nil {
        fmt.Println("Error reading Dockerfile")
        return nil
    }

    return dag.
        Container().
        From(checkovTag)
}

Simple, and Dependabot can keep the Dockerfiles updated.

tawny carbon
#

That’s interesting. Why not just DockerBuild that Dockerfile though?

marble dragon
violet cosmos
#

we've all been there 😁

steady bluff
#

I don't know how often I've run into walls, because I was overthinking how the solution should be, when in fact, it could have been done so much simpler. It's a roundabout way, but it is still a good learning process, because I end up experimenting so much and failing.

brave vine
#

How do I import private modules? I see this issue https://github.com/dagger/dagger/issues/5521 but that's really not applicable in my situation. I'm trying to do something basic like

package main

import (
    "context"
    "github.com/myorg/myrepo/logging"
)

type Mymod struct{}

// Returns a container that echoes whatever string argument is provided
func (m *Mymod) ContainerEcho(stringArg string) *Container {
    return dag.Container().From("alpine:latest").WithExec([]string{"echo", stringArg})
}

// Returns lines that match a pattern in the files of the provided Directory
func (m *Mymod) GrepDir(ctx context.Context, directoryArg *Directory, pattern string) (string, error) {
    logging.Info("running grepdir")
    return dag.Container().
        From("alpine:latest").
        WithMountedDirectory("/mnt", directoryArg).
        WithWorkdir("/mnt").
        WithExec([]string{"grep", "-R", pattern, "."}).
        Stdout(ctx)
}

I can go get github.com/myorg/myrepo/logging just fine. But when I run dagger call or dagger develop it seems to not be honoring my gitconfig to use ssh. Do I need to use https instead?

GitHub
How to configure Go to use private module dependencies · Issue #552...

What is the issue? ref: https://discord.com/channels/707636530424053791/1134415015198330981 We have a cookbook that explains how to clone private repositories here https://docs.dagger.io/710884/pri...

violet cosmos
restive hollow
brave vine
#

Awesome!

restive hollow
# brave vine Awesome!

I'll make sure to ping you on the PR so that you can test in preview, and to collect feedback 👼 🙏

brave vine
#

I have a function with a signature like func (m *MyMod) BuildContainer() (BuiltContainer, error) {} but when I try to use it in another function the generated code only has a single return value, BuiltContainer. How should I check for an error? Should put an error field into my struct?

brave vine
#

Ah, putting an error into my struct doesn't seem to work. Causes dagger develop to panic.

tawny carbon
# brave vine I have a function with a signature like `func (m *MyMod) BuildContainer() (Built...

What you see in the module and what you see in generated clients, they are different things. One doesn't call the other directly.

The generated code is client bindings to make API calls. It's basically a query builder and not executing your function directly. Basically your module's code is used to extend the API schema, and the client bindings are generated based on the API schema. So, for the generated client, functions that return objects are chainable and lazy. These don't return an error. Only functions that return a scalar, like string, cause the query to be built and a request sent. That's when you handle possible errors on the client side.

Notice that these don't necessarily match. Even if BuildContainer returns an error, it'll only be caught on the client side when a request is made. So let´s say you have this chain: myMod > buildContainer -> container -> stdout. If BuildContainer returns an error server side, it'll show on stdout because that's when you have to pass ctx and handle an error, from making the API request. Any step in that chain could return an error, stopping the chain. In this case, the server side code for stdout wouldn't even have executed.

brave vine
tawny carbon
# brave vine That's helpful context thank you! In this situation what is the right way to che...

In that case it won't work because BuildContainer is lazy, so it'll never return nil as it's not making any request. You'd need to create a function that forces execution like a function that just returns an error and inside just calls sync on the built container to force it to build, or expose the underlying container and let the caller call sync. That's if you really need to check before continuing as sometimes it's not really needed to do it like this and can slow down your pipeline.

quiet hinge
#

created an example of showing how you can use apko&melange modules on a go project ^^

brave vine
#

If I have a function that accepts a []string . How do I pass that from the CLI? The docs say "Dagger Functions, just like regular functions, can accept arguments. In addition to basic types (string, boolean, integer, array...)," but its not clear how I'd pass an array/slice from the CLI

ornate seal
brave vine
#

Ahh I didn’t realize that I could do that thanks!

ornate seal
#

😄

shrewd turtle
regal seal
#

@tawny carbon I think we discussed removing the top-level aliases from go modules? Couldn't find a reference as to where it was, but ended up starting to implement it here: https://github.com/dagger/dagger/pull/7831

regal seal
regal seal
mortal rune
#

Is there a way to selectively include only the paths you need in a build, given I provide a --source .?

dusky chasm
violet cosmos
mortal rune
olive apex
unreal whale
#

what is the equivalent of using CMD instruction in a Dockerfile?

violet cosmos
lethal vigil
#

like:

func (hop *Hop) Build(ctx context.Context) error {
    cfg, err := config.Read(ctx, hop.Source)
    if err != nil {
        return err
    }

    src := hop.Source

    for _, b := range cfg.Builders {
        setOptions(b, hop)

        ctr, err := b.Build(ctx, src)
        if err != nil {
            return err
        }
        src = ctr.Directory(consts.Workdir)
    }

    return nil
}
olive apex
unreal whale
#

how can I convert this Docker RUN instruction to dagger? RUN --mount=type=cache,target=/home/node/.cache/yarn,sharing=locked,uid=1000,gid=1000 && yarn install. I've tried with Container.WithExec but doesn't seem to work.

vast knoll
#

I cant seem to find a way to get the string representation of a Directory type

#

something like Directory.String() that returns a string which is the (relative or absolute) path to the directory

#

well it turns out I didnt need that in the first place

violet cosmos
# vast knoll well it turns out I didnt need that in the first place

That's good news, because I don't think we have that 😅 (because of the nature of how Directory type works, each Directory is an immutable object that can be referenced at multiple locations within different parent directories, and exist standalone as its own root. So by definition there is not a single answer to "where does this directory live?"

violet cosmos
vast knoll
brave vine
#

Yeah it took me a bit to realize that. If you need directory to be at "a path" I think you can use Export within your function and it should show up there inside the container.

violet cosmos
#

Right, a path is always relative to a directory - either a Directory object, or to the client's local filesystem

#

This is fundamental to how Dagger works, and is comparable to how Git objects work, for example

#

It's what allows each Dagger object to be content-addressed, which in turns is what makes the aggressive caching possible

brave vine
#

Does anyone have renovate bot setup in a dagger repo? Mine is getting errors like:

Command failed: go get -d -t ./...
go: downloading go.opentelemetry.io/otel v1.27.0
go: downloading go.opentelemetry.io/otel/sdk v1.26.0
go: downloading go.opentelemetry.io/otel/trace v1.27.0
go: downloading golang.org/x/sys v0.20.0
go: downloading github.com/go-logr/logr v1.4.1
go: downloading go.opentelemetry.io/otel/metric v1.27.0
go: downloading github.com/go-logr/stdr v1.2.2
go: dagger/containers imports
    dagger/containers/internal/dagger: package dagger/containers/internal/dagger is not in std (/opt/containerbase/tools/golang/1.22.5/src/dagger/containers/internal/dagger)
go: dagger/containers imports
    dagger/containers/internal/telemetry: package dagger/containers/internal/telemetry is not in std (/opt/containerbase/tools/golang/1.22.5/src/dagger/containers/internal/telemetry)

I'm a renovate noob so not sure if there's something on the renovate side I can tweak.

vapid halo
#

Yeah, I just have renovate ignore that directory

brave vine
#

ah right that makes sense 😄

brave vine
vapid halo
brave vine
#

Oh gotcha. I may need to break out most of my code out of my dagger dir then so it can still get updates

violet cosmos
#

Hello Gophers, a breaking change in the Go SDK 0.12: the dagger. prefix for Dagger types is now mandatory.

Before: func build(src *Directory) *Container

After: func build(src *dagger.Directory) *dagger.Container

This change is backwards compatible (0.11 supports it) so I recommend making the change to your modules as soon as possible.

Thanks to compatibility mode you can still upgrade to 0.12 right away, but won't get the other 0.12 features while compat mode is active.

This is a bandaid we've meant to rip off for a while, it will make the Go SDK more consistent with the other SDKs, and will improve IDE support.

agile mural
unreal whale
#

I have a workflow where I build two images within a single build function. Will Dagger automatically build these images in parallel, or do I need to make separate calls for each image to achieve parallel builds?

agile mural
#

Depends on the function and if you added any manual synchronization points. Normally, Dagger would run everything in parallel (or rather anything in the DAG that don't depend on each other)

#

But if it's a single function, I'm guessing you either call Sync on both or do something similar. The function would then be blocked, until Dagger finished building the image and it wouldn't start building the other till it's finished.

agile mural
unreal whale
#

is there any guide on how to upgrade to a new version of dagger? including all dependencies to external modules?

agile mural
#

Here is what I did:

  • Upgrade Dagger binary
  • Run dagger develop
  • Change all core types to dagger.[Type]
  • Add command to WithExec where you relied on entrypoint before
violet cosmos
unreal whale
#

thanks 🙂

violet cosmos
#

Note this is specific to upgrading to 0.12

#

Also, with compat mode you have the option of upgrading the dagger binary now, and doing the rest later

flat minnow
#

Hi, I have an error on a module since I upgrade to 0.12.2. I'm not sure if it's a bug or a mistake on my side

# 

Stderr:
/go/pkg/mod/google.golang.org/grpc@v1.64.0/status/status.go:35:2: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/status in multiple modules:
        google.golang.org/genproto v0.0.0-20220503193339-ba3ae3f07e29 (/go/pkg/mod/google.golang.org/genproto@v0.0.0-20220503193339-ba3ae3f07e29/googleapis/rpc/status)
        google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 (/go/pkg/mod/google.golang.org/genproto/googleapis/rpc@v0.0.0-20240515191416-fc5f0ca64291/status)
/go/pkg/mod/github.com/grpc-ecosystem/grpc-gateway/v2@v2.20.0/runtime/handler.go:12:2: ambiguous import: found package google.golang.org/genproto/googleapis/api/httpbody in multiple modules:
        google.golang.org/genproto v0.0.0-20220503193339-ba3ae3f07e29 (/go/pkg/mod/google.golang.org/genproto@v0.0.0-20220503193339-ba3ae3f07e29/googleapis/api/httpbody)
        google.golang.org/genproto/googleapis/api v0.0.0-20240520151616-dc85e6b867a5 (/go/pkg/mod/google.golang.org/genproto/googleapis/api@v0.0.0-20240520151616-dc85e6b867a5/httpbody)
/go/pkg/mod/go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@v1.27.0/client.go:12:2: ambiguous import: found package google.golang.org/genproto/googleapis/rpc/errdetails in multiple modules:
        google.golang.org/genproto v0.0.0-20220503193339-ba3ae3f07e29 (/go/pkg/mod/google.golang.org/genproto@v0.0.0-20220503193339-ba3ae3f07e29/googleapis/rpc/errdetails)
        google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291 (/go/pkg/mod/google.golang.org/genproto/googleapis/rpc@v0.0.0-20240515191416-fc5f0ca64291/errdetails)```
#

not sure if I should post this message here or in #daggernauts

regal seal
#

huh well that's possibly one of the least helpful error messages i've seen the go sdk spit out in a while

#

have you run dagger develop on your module?

flat minnow
#

yes this one is ok but when I do a dagger call / dagger functions I have this error

#

I installed a module, build a module around this one and I didn't add modules in the go.mod

regal seal
#

Okay so dagger develop passes - if you manually run go mod tidy on the dagger module, what do you see?

#

Is your module public? If so, a link to it would definitely help

flat minnow
#

no this one is not public, but now I have a different error on all command

#

failed to checkout remote https://github.com/Dudesons/daggerverse: git error: exit status 128

#

but this repo is public normally

#

and I can pull from my terminal, I try to restart my docker engine

#

ok I try to reinstall my module and now I have the original error and I did the go mod tidy

regal seal
#

can you share the results of ls ./path/to/your/module?

flat minnow
# 
dagger.gen.go  go.mod  go.sum  image.go  internal  main.go  tf.go```
flat minnow
#

@regal seal I'm trying to create a new module without external dependencies, I just did the the dagger init then develop then functions without modifying the code I have the same error

# 
dagger.gen.go  go.mod  go.sum  internal  main.go```
#

what is strange is I have my personal repo in public with modules in this one there is no issue but in my private repo I have an issue

#

ok I solve my issue but I don't understand the root cause or what I miss. I delete my go workspace files then everything is ok

unreal whale
#

Is it possible for a module to have constructor that takes an interface as argument? then the main module will provide the implementation of that interface

livid quiver
#

so I've been using Sync() everywhere to make sure my shells run, but I feel like there should be a better way to do this. does it make sense to convert my workflow to Stdout()/Stderr()?

tawny carbon
livid quiver
#

it seems to still stream stderr/out :/

tawny carbon
brave vine
#

I'm thinking through logging for my functions. Is there a way for me to figure out what the verbosity is as set by the dagger CLI (-vvv) or do I need to pass in my own --log-level argument?

fallen root
#

I'm thinking through logging for my

slim radish
agile mural
#

A better example for code might be any of the module calls: dag.Go or dag.Helm.

Those will call the respective module constructors in my Go and Helm modules.

slim radish
agile mural
#

I have two feature requests, but I'm not sure there are open issues for them, so I thought I'd ask here first:

  • Allow specifying CLI names of functions and arguments (using numbers in function names is a bit annoying at the moment)
  • Allow specifying optional arguments in a struct (not an anonymus one; passing around a struct to subsequent functions is much easier than passing around all of the arguments again, see https://github.com/sagikazarmark/daggerverse/pull/143)

Are there any open issues for these? (I couldn't find any, but I recall seeing one for the first one)

tawny carbon
agile mural
agile mural
#

Trying to use an enum in Go, but it doesn't work:

// This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server.
type SSLMode string

const (
    // Only try a non-SSL connection.
    Disable SSLMode = "disable"

    // First try a non-SSL connection; if that fails, try an SSL connection.
    Allow SSLMode = "allow"

    // First try an SSL connection; if that fails, try a non-SSL connection.
    Prefer SSLMode = "prefer"

    // Only try an SSL connection. If a root CA file is present, verify the certificate in the same way as if verify-ca was specified.
    Require SSLMode = "require"

    // Only try an SSL connection, and verify that the server certificate is issued by a trusted certificate authority (CA).
    VerifyCA SSLMode = "verifyca"

    // Only try an SSL connection, verify that the server certificate is issued by a trusted CA and that the requested server host name matches that in the certificate.
    VerifyFull SSLMode = "verifyfull"
)

failed to install constructor: failed to create function: failed to find mod type for function "" arg "sslmode" type

#

(yes, I use it in a constructor as the error says)

regal seal
#

Do you have a link to a code snippet? Fine if not

agile mural
#

Not yet

tawny carbon
# agile mural

You have the type in the constructor to be string, when it should be sslmode SSLMode.

agile mural
agile mural
#

Changing to SSLMode yields the above error

tawny carbon
nova sequoia
#

I get a similar error as @agile mural (with regards to enums) when trying to use an interface in a constructor, but presume that's due to the nascent support for interfaces. Does that sound accurate and should forgo the use of interfaces in constructors for the time being?

Error: failed to get module SDK: input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get schema introspection json during module sdk codegen: failed to get schema for module "create": failed to install constructor: failed to create function: failed to find mod type for function "" arg "kc" type
tawny carbon
#

Possibly the same bug. Might solve both. Need to add the test cases. If you're using Go, you can use interfaces, if you can avoid putting it in the constructor until that bug is fixed.

nova sequoia
#

I'll create a PR with a test to reproduce if that will help

tawny carbon
#

That issue is still open because Python and TypeScript don't support defining interfaces yet.

livid quiver
#

what triggers stopping a service declared under a container with WithServiceBinding?

#

I have three containers that I need to hit up with the same service, but once one of my containers does its work, then the remaining two will have an error that the service isn't on the host, even though all three have the same binding.

#

I even tried calling Start() on them

violet cosmos
patent ermine
# livid quiver I even tried calling Start() on them

hm if you're calling Start() but it's still happening it sounds like something other than dependencies getting stopped incorrectly. it might be an error in userland/runtime - not sure "service isn't on the host" means. is something printing that? do you have a repro?

livid quiver
#

ok, so if it's all services, there's no pattern for me to say create a database, then run a ready check, then run a db migration, if the ready check will be done and then there's nothing bound and the service shuts down

#

I was originally making it work with withcontainerfunc calls that manipulate the host directly as part of its startup.

#

but that felt hacky

violet cosmos
patent ermine
violet cosmos
#

But other than that: that's the solution @livid quiver, as long as you explicitly call Start() on your service, it won't be auto-stopped, it's up to you to stop it (or just let it happen at the end of the pipeline)

patent ermine
violet cosmos
patent ermine
# violet cosmos The disabling of auto-stop

the current model is that:

  • a service starts at most once once per session, and is bound to different callers
  • starting via service binding and starting via API are the same thing (both start and bindings++)
  • however, service bindings detach 10 seconds after the binder is finished (bindings--)
  • after each detach, if bindings=0, the service is stopped automatically
  • but calling start via API never detaches, so that's how it stays running
  • calling stop stops the service, no matter how it started, no matter how many bindings there are

so, I don't see why you'd need a flag for it in this model, but maybe i'm missing something

livid quiver
#

If I run Start(), am I then on the hook for checking ready?

#

actually I take that question back since it's documented that in Start() it does the check.

livid quiver
#

What's presently happening:

#
  1. I'm starting my container manually
  2. I'm able to connect my first container to check the pg up state
  3. when I get to my third step, the host is gone, despite passing check.
violet cosmos
livid quiver
#

oh wait, there it is, I'm not running WithExec on the pg_isready, I'm running WithEntrypoint which obviously won't work. Now to figure out why the initial db container is failing

patent ermine
violet cosmos
patent ermine
#

i just want to be sure that's actually the root of the confusion, because from what I'm reading it doesn't sound like it, could use a pointer to that

#

(bleary-eyed and pre-coffee at the moment)

livid quiver
#

ah, pg_isready doesn't actually wait for a connection, had to put a hard sleep in front of it 😦

violet cosmos
#

@livid quiver but normally Dagger's health check will not run pg_isready until the db is listening on its port

#

so you shouldn't need to run sleep anywhere

livid quiver
#

right, because I ran it manually 😄

patent ermine
#

just confirming if you start it manually it should still wait for health checks (like any other start)

#

do you have any code you can share?

livid quiver
#

i do but it's incomprehenisble crap

patent ermine
#

better than nothing! 😁

livid quiver
#

that's the db code

#

i've gotten farther since delaying the ready check for a bit (what I do in drone as well)

#

I now have to go back and make sure the db gets setup with our seed data (it's not currently wired correctly)

#

I think postgres reserves the socket but doesn't respond on it until it's ready which is why it'd do the ping check fine, but not serve postgres protocol

patent ermine
#

yeah that might be the case. what might be nice is to have a readiness check that's just "look for this in the output"

#

since postgres prints database system is ready to accept connections when it's ready

livid quiver
#

oh, and one for the output censor: url parameters indicating password: postgres://postgres:password@events:5432/events?sslmode=disable

patent ermine
#

but, it won't be scrubbed if that value is being passed around in plaintext (like as a function arg)

#

design wise i'd say the password should be a separate secret, or the entire arg should be a secret (maybe that)

livid quiver
#

yeah, I can secret it into the shell as a container secret

#

oh weird. I put a Terminal() in one place, and it keeps popping up over and over again

#

oh nevermind, second database

#

the only reason we run two databases is because we use goose and it really doesn't understand two schemas co-existing on the same host lol

patent ermine
livid quiver
#

I got my database squared away today thanks to your help, vito and solomon!

frail plank
#

Hello guys! I would like to know is there a way to get the container execution output "line by line" as WithLogOutput does?

violet cosmos
frail plank
violet cosmos
#

Unfortunately no. That is structurally very difficult to do. But perhaps can be done in "userland" by writing the logic in your own functions? May I ask what your use case is?

frail plank
#

Unfortunately I do not know what the script can do and if that's not possible I may need to wrap it into another script that will stream the output if I get your "userland" idea correctly

violet cosmos
#

days, wow!

frail plank
#

thank you for the idea and your response

violet cosmos
#

the way you could do it:

  • implement a module with a "run" function, where you pass the script and base container as arguments
  • you wire up a Service that can stream logs (could be very basic since you only need to support the one log). your wrapper sends the stream to that service
  • you return the Service to the caller so they can connect to it and stream the logs. again that can be a very simple implementation
#

in the future we might add a Stream type which would make this even more convenient to consume from the CLI

#

for the "streaming service" implementation you might get away with just running socat and exposing 2 ports: one for writing the other for reading

frail plank
#

this is awesome, thank you very much! I hope you'll add this possibility in the future

violet cosmos
#

Also instead of a proxy service you could upload to an s3 bucket, and return the url. then have another function stream the contents of the bucket

livid quiver
#

not 100% compatible with dagger though, though I haven't found myself reaching for it lately. we mainly use it to run R scripts

#

mainly you can get out the stdout / err as a stream and react on a line-by-line expect basis

crystal hull
#

Hrm, I was updating some dependencies in one of my projects which pulled in some newer indirect OTEL deps:

go: upgraded go.opentelemetry.io/otel/sdk v1.28.0 => v1.29.0
go: upgraded go.opentelemetry.io/otel/sdk/log v0.4.0 => v0.5.0
go: upgraded go.opentelemetry.io/otel/trace v1.28.0 => v1.29.0
``` - which then resulted in ```
# dagger.io/dagger/telemetry
../../../go/pkg/mod/dagger.io/dagger@v0.12.5/telemetry/exporters.go:92:23: cannot use log (variable of type "go.opentelemetry.io/otel/sdk/log".Record) as *"go.opentelemetry.io/otel/sdk/log".Record value in argument to e.OnEmit

Breaking changes on minor releases? Shame on OTEL for that...

livid quiver
#

after having a golangci-lint minor break my build this week, yeah... shame

patent ermine
patent ermine
violet cosmos
#

figured it out, I had run go mod tidy because that's what go init told me to do. Turns out that's what installs the breaking version of otel (shame). Had to wipe go.mod and go.sum and start over

patent ermine
regal seal
violet cosmos
#

I'm using enums from another module, and noticed that enum types are not namespaced by module...

pale moat
#

is there a way to get logs from running .AsService().

context: I have a container that I am trying to run as service, but it fails. when I get into the terminal of the container and run the exact command as Entrypoint, it works.

agile mural
#

Try --progress plain. It should give you the logs from the service.

robust ridge
#

is there a way to influence the generated name of my function? i have a dagger function:

func (r *MyProjet) EndToEndK3S(

but that comes out as dagger call end-to-end-k-3-s. I didn't see anything, but i was hoping for a tag or // + something i could put in here to try to push the generation to a better name

violet cosmos
agile mural
tawny carbon
# violet cosmos Yeah that would be cool. Doesn't exist at the moment unfortunately. Try `EndToE...

Python and TypeScript do have this feature. The motivation was to get around limitations on using reserved keywords (like from, import, with....). It was requested to be removed from the docs however, with the argument that it was unnecessary, and to avoid more concepts to learn. So I personally didn't pursue consistency here, just seeing if more people request it for Go. Even so, it's still in the backlog as @agile mural kindly created an issue for it 🙂

tulip matrix
#

Debugging multiple running services

main trail
#

I have a repo with a CLI tool whose functionality is also exposed as a Go package in the same repo and I want to provide a dagger function that would just import the (local) package. However dagger init creates its own new Go module, so importing it from elsewhere in the repo is not as straight forward. (This is maybe a general "Go monorepo" issue). Is there any best practice from a Dagger perspective here ? Would you just host the dagger module in another repo always?

livid quiver
#

we keep our dagger stuff in our magefiles directory off our root in our monorepo

#

with its own go.mod, of course

jaunty salmon
violet cosmos
#

you can either 1) keep the separate go.mod and add a replace directive to it, so that it can import the "host" module (I do this) or 2) remove the separate go.mod and make sure you don't have conflicting dependencies.

Both are possible

#

We err on the side of more isolation by default, it makes for more predictable behavior OOTB. But you can fine tune to meet the needs of your project

main trail
#

add a replace directive to it
ok cool, I've only done this for local overrides when developing before, so wasn't sure if that's what people also do for monorepos and more permanent setups, great thanks 🙂

livid quiver
#

I use replace a lot in my mono repo but you have to make sure all go.mod have identical relative paths. So we have libs, services, lambdas, and apps at our top level trees and use "../../libs/libname" as the replace paths. Kinda wish that wasn't the case, could have gone flatter, I suppose

granite valve
#

Can I execute dagger pipelines directly in go or do I need to use the cli?

tawny carbon
livid quiver
#

I use mage to organize my dagger builds but you could just as easily call it from any program. It's so nice to use because of this feature!

#

I might use modules in the future, but this gives me paus

violet cosmos
#

yeah we plan on adding it. You should be able to load a module from your custom client (eg. magefile in your case). You actually already can, you just won't get generated bindings: need to craft the grapgql query yourself.

fair bridge
#

👋 I'm having issues getting Dagger working in a monorepo with a go.work file that references private repos. My workaround at the minute is to move go.work out the repo before each run, and then put it back 😄

It is failing on initialize, specifically the go build -o /runtime . step, and is trying to pull the private repos over HTTPS:


Stderr:
internal/querybuilder/marshal.go:11:2: github.com/**fastly/fst-go**@v1.8.0: reading github.com/fastly/fst-go/go.mod at revision v1.8.0: git ls-remote -q origin in /go/pkg/mod/cache/vcs/d5531381cb31f8d30953a9a9b61da88e7f81c6d12b1f111ef6d87bfc4215fcad: exit status 128:
        fatal: could not read Username for 'https://github.com': terminal prompts disabled
Confirm the import path was entered correctly.
If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.
internal/querybuilder/querybuilder.go:11:2: github.com/fastly/fst-go@v1.8.0: reading github.com/fastly/fst-go/go.mod at revision v1.8.0: git ls-remote -q origin in /go/pkg/mod/cache/vcs/d5531381cb31f8d30953a9a9b61da88e7f81c6d12b1f111ef6d87bfc4215fcad: exit status 128:

I'm not too sure where to start with this one. Any suggestions?!

agile mural
fair bridge
granite valve
#

How can i create a dagger.Directory other than passing it in as an arg?

regal seal
#

you can create an empty directory using dag.Directory() and then manually add files with WithNewFile and such

granite valve
#

How can I add a host file to that dir?

#

Do i need to read the contents first and then pass it in?

vapid vortex
#

Is there a more effcient way I could be running this?

// Lint will run yamllint on the given directory via dagger. The directory will
// be filted for .yaml and .yml files and the .yamllint file will be respected
func Lint(ctx context.Context, dag *dagger.Client, srcDir *dagger.Directory) (string, error) {
    fileList := []string{".yamllint"}

    yamlFiles, err := srcDir.Glob(ctx, "**/*.yaml")
    if err != nil {
        return "", err
    }
    fileList = append(fileList, yamlFiles...)

    ymlFiles, err := srcDir.Glob(ctx, "**/*.yml")
    if err != nil {
        return "", err
    }
    fileList = append(fileList, ymlFiles...)

    // Sort the fileList to try and keep cache hits
    sort.Strings(fileList)

    // Create a new emptry directory and add the files to it
    dir := dag.Directory()
    for _, filePath := range fileList {
        // Skip any node_modules or vendor directories
        if strings.Contains(filePath, "node_modules") || strings.Contains(filePath, "vendor") {
            continue
        }

        file := srcDir.File(filePath)
        dir = dir.WithFile(filePath, file)
    }

    return dag.Container().
        From("registry.gitlab.com/pipeline-components/yamllint:0.32.1").
        WithDirectory("/code", dir).
        WithWorkdir("/code").
        WithExec([]string{"yamllint", "-c", ".yamllint", "."}).
        Stdout(ctx)
}

My aim is to ensure as many cache hits as possible in the pipeline, and avoid running the lint if no yaml files have changed

I can do this in gitlab-ci quite easily, however this is this the only way I can think of doing this with dagger.

Any tips or pointers would be appreciated as it's a pattern I want to repeat with other, more time consuming pipelines

dire atlas
#

Do i need to read the contents first and

rocky karma
tawny carbon
#

This probably needs to be better documented and/or enforced, but enum values in the API don't have "names", they only have values. So when the client bindings are generated, that's all the SDK sees (values and their descriptions). In other words, when a custom enum is defined in a module, the names (e.g., MyType) are only available from within the language, it doesn't get reflected in the API.

The values, ideally, should be in "SCREAMING_SNAKE_CASE", while the names in the language's convention:

cons MyType MyProxyType = "MY_TYPE"

Which produces dagger.MyType (converted from the value in codegen).

Or:

cons Blah MyProxyType = "BLAH"

Which produces dagger.Blah.

tawny carbon
#

I’ve been discussing this with @regal seal, and I think we can make this better (i.e., more intuitive) since he added directives support in dagql recently. It would create what you expect but it’s a breaking change.

agile mural
#

What happense when there is an overlap between values in two different modules?

tawny carbon
#

The values would no longer matter, they'd just be strings. If you mean whether there's an overlap on enum names, then we've also discussed that the Go SDK needs to namespace the enum members, unlike the other SDKs, where they're scoped in their class already. So probably MyProxyTypeMyType, like in Python's MyProxyType.MY_TYPE.

agile mural
regal seal
vapid vortex
#

I have a quesiton about function chaining I was wondeirng if anyone could help with

I have the following module:

// GoTest will run the tests for a folder
func (Test) GoTest(ctx context.Context,
    // +defaultPath="/"
    srcDir *dagger.Directory,
) (string, error) {
    files, _ := srcDir.Entries(ctx)
    for _, file := range files {
        fmt.Println(file)
    }

    return "", nil
}

// FilterDirs will return a list of directories within the srcDir
func (Test) FilterDirs(ctx context.Context,
    srcDir *dagger.Directory) ([]*dagger.Directory, error) {
    filterDirsList := []*dagger.Directory{}

    files, err := srcDir.Entries(ctx)
    if err != nil {
        return filterDirsList, err
    }

    for _, file := range files {
        _, err := srcDir.Directory(file).Entries(ctx)
        if err != nil {
            fmt.Printf("Directory %s does not exist", file)
            continue
        }

        filterDirsList = append(filterDirsList, srcDir.Directory(file))
    }

    return filterDirsList, nil
}

That I want to chain together:

dagger -m ci/test call filter-dirs --src-dir images go-test
....

! unknown command "go-test" for "dagger call filter-dirs"

High level logic of what I want to do:

  1. Fetch dirs in the images dir
  2. Check if there is a go.mod in that dir (not implemented above, but will be in the GoTest metnod)
  3. run a go test in each image dir (not implemented yet)

Is this possible?

vapid vortex
#

Related to the above, when I was running and calling the SDK direct, I could call

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
for , pathName := range folderNameList {
  pipeline := client.Pipeline(pathName)
  pipeline.Container().Exec([]string{"go", "test"})
  ...

and have this then "grouped" in my TUI output.

Is there a to have this same grouping in a dagger module so that I can identify which part of my test suite has either failed or taken a long time to execute?

radiant torrent
#

Hey folks, I have a question about interacting with a Git repository inside of a container. I'm trying to building a workflow that needs to generate some commits to a repository. I am able to mount the repository tree at a certain ref using d := dag.Git("<URL>").Ref(<ref>).Tree() and then I use .WithDirectory("/path", d), but what I realized is that the directory only contains the contents of the tree and none of the .git folder

Is there some way that I can use dag.Git() and preserve git .git directory. Or is my only option to effectively peform a git clone within my container?

Thanks in advance for any pointers

radiant torrent
limpid mango
# radiant torrent Hey folks, I have a question about interacting with a Git repository inside of a...

There's an option that you can pass to preserve the .git directory:

// GitOpts contains options for Client.Git
type GitOpts struct {
    // Set to true to keep .git directory.
    KeepGitDir bool
    // A service which must be started before the repo is fetched.
    ExperimentalServiceHost *Service
    // Set SSH known hosts
    SSHKnownHosts string
    // Set SSH auth socket
    SSHAuthSocket *Socket
}

Ref: https://github.com/Excoriate/daggerverse/blob/57dee843943e817d72ca46f1ee3fc0ed2dea9ff1/module-template-light/content.go#L96

radiant torrent
rocky karma
livid quiver
#

This reporting is a little weird looking, is there something that can simplify this?

container.from.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withDirectory.withWorkdir.withServiceBinding.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.withExec.sync resolve: process "/bin/sh -c ..." did not complete successfully:

regal seal
#

huh

#

yeah that's not super easy to read is it 🤔

rocky karma
regal seal
#

^ @agile mural if you get a moment would appreciated any feedback you have on the ideas there, hopefully it should fix a lot of the issues with enums we have today!

fallen root
#

Has anyone ran into this one?

I am trying to use a basic bool argument and I get this error

Error: response from query: input: query:
query{ci{test(verbose:1){stdout}}}

error: parse selections: parse field "test": init arg "verbose" value as dagql.DynamicOptional (Boolean) using dagql.DynamicOptional: cannot create Boolean from int64

Heres the full function

// Run test suite
func (m *Ci) Test(
    ctx context.Context,
    // +defaultPath="/"
    dir *dagger.Directory,
    // +optional
    // +default=false
    verbose bool,
) *dagger.Container {
    ctr := m.base().
        WithDirectory("/src", dir).
        WithWorkdir("/src")

    if verbose != false {
        ctr = ctr.WithExec([]string{"go", "test", "-v", "./cmd/web"})
    } else {
        ctr = ctr.WithExec([]string{"go", "test", "./cmd/web"})
    }

    return ctr
}

Heres my command dagger call test --verbose stdout

fallen root
#

Thank to @restive hollow for helping me out - it turns out we reserve verbose - updating my variable name to verboseOutput did the trick

restive hollow
agile mural
#

One caveat I found recently with modules written in Go: if you want to call a function from within the same module, pointers may get in the way as they allow you to modify "global" (module scope) state even when you don't want to.

Here is an example:

// Run a Go command.
func (m *WithSource) WithExec(
  // Arguments to pass to the Go command.
  args []string,
  
  // TODO: add back the platform argument, but make sure it's not persisted across calls
  ) *WithSource {
  m.Source = m.Exec(args, dagger.Platform("")).Directory(workdir)
  
  return m
}

// Run a Go command.
func (m *WithSource) Exec(
    // Arguments to pass to the Go command.
    args []string,

    // Target platform in "[os]/[platform]/[version]" format (e.g., "darwin/arm64/v7", "windows/amd64", "linux/arm64").
    //
    // +optional
    platform dagger.Platform,
) *dagger.Container {
    if platform != "" {
        m = m.WithPlatform(platform)
    }

    return m.Container().WithExec(args)
}

// ...

// Set GOOS, GOARCH and GOARM environment variables.
func (m *WithSource) WithPlatform(
    // Target platform in "[os]/[platform]/[version]" format (e.g., "darwin/arm64/v7", "windows/amd64", "linux/arm64").
    platform dagger.Platform,
) *WithSource {
    m.Go = m.Go.WithPlatform(platform)

    return m
}
#

If you call Exec anywhere in the same module (eg. WithExec with a platform argument), it's going to change the platform permanently, because WithPlatform modifies the module's Go context.

I'm guessing this isn't a problem when calling it through GraphQL or any of the generated modules?

One potential solution is to stop using pointer receivers/return values. Haven't tried it though and I'm not sure if it works.

The reason why I raise this point here is:

  • The above behavior could potentially cause hidden or hard to debug errors
  • If dropping the pointer receiver is an option (ie. it works today and it indeed solves the problem), maybe defaulting to that would make sense?
  • If it doesn't work for some reason, is it an option?
regal seal
#

Dropping the pointer receiver works, but essentially only does a shallow copy.

#

It feels like dag.Self would work here - the idea is to codegen a module's own API and serve it to itself, so you could somehow call WithPlatform through the dagger api (though ofc even with this, you'd still be able to call the function directly, and avoid that)

agile mural
#

So WithPlatform would return a new WithSource with a new Go with a new Container.

#

That's exactly what I want

sonic idol
#

Background:

  • repo A I own, and my Dagger specs will live here.
  • repo B someone else owns and contains a Dockerfile

Is there a way to build the Dockerfile in B from a repo A? From what I'm seeing I can mount the repo into a container but it's not clear to me how to execute a DockerBuild on the dockerfile. It feels like that would be a Docker in Docker style build. Is that something that's supported? Is there another way I could approach it?

agile mural
#

Background:

brave vine
#

Can I bind a service to a container and expose it to the host?

rocky karma
brave vine
signal mantle
finite pendant
#

Hey folks, I’m new to dagger so apologies if this question has already been answered elsewhere. Just wondering where the bulk of my Go code should live? For example, if I’d like a couple of module functions to make different api calls, in a normal Go project I’d make a sub module called “api” or so to reuse the mechanism and expose the necessary functions and structs.

When I try that in my dagger module, I get a “cannot code generate for foreign type ExampleType” error.

Thanks for your help!

violet cosmos
# finite pendant Hey folks, I’m new to dagger so apologies if this question has already been answ...

Hi Matt! No apology needed.

The Dagger Go SDK lets you organize the code in your module the same way you would in a regular Go module, so there aren't too many constraints in that regard.

However, it seems that the SDK has constraints on where you can define the types exposed in your Dagger API. Specifically, those types can't be defined in a sub-package (at least that's my understanding of the error message you pasted).

So you can create sub-packages etc. You just can't expose their types in your Dagger Module's API. ie. you have to keep them a private implementation detail of the module.

#

Personally I always keep my module's code in a single flat directory. If it gets too big and I feel the need to break it up into more components.. That usually means it's time to create a new Dagger module, and use Dagger-level dependency between them 🙂

olive apex
#

Hey! Unsure how to search back for this. I'm going back and forth on how to best code functions that modify the state of the module struct (things like With<> or As<> functions).
Is there any difference between modifying the receiver variable pointer compared to creating a new object?
Here is a dumb example, is one pattern recommended for dagger?

package main

type Counter struct {
    N int
}

func (c *Counter) Add_A() *Counter {
    return &Counter{
        N: c.N + 1,
    }
}

func (c *Counter) Add_B() *Counter {
    c.N = c.N + 1
    return c
}
regal seal
#

in the first case, you're returning a new Counter - in the second, you're modifying the existing Counter - so if your calling code keeps the old one behind, and tries to use it again later, then you're gonna get strange results

#

one easy way of getting both worlds: you can change the receiver to be (c Counter), then you can safely modify this (since it's a copy of the value)

#

then just return &c

#

something like:

func (c Counter) Add_C() *Counter {
    c.N = c.N + 1
    return &c
}
olive apex
#

Good to know that there is no difference through the engine 🙂 Thanks!

finite pendant
flat minnow
#

Hi, I'm trying to add an optional boolean argument to a function with a default value to true but when I'm calling my function and give another value than the default, this one is ignored. If I try with a type string this one the default value can be overwritten.
Is it possible to do that with boolean or should I do something special ?

regal seal
#

if you call the function from the cli, it should work (from my memory)?
the issue comes when you call it from go

#

the problem is that go has no way of distinguishing between a missing argument and a false argument - while languages like python+typescript do

flat minnow
#

yes it's when I'm doing the call from go and later I will do from python for my end user which are python dev

#

there is a workaround for that in golang when the function is not called from the cli ?

violet cosmos
#

I believe there is no workaround. I just resigned myself to never have any boolean arg that defaults to true. It's pretty bad IMO. But I've learned to live with it

tawny carbon
#

To fix this we would have to change the DX for specifying Go options, like using the same chainable API we use for functions, instead of a single struct. That's quite a breaking change though, but I wonder if at some point it may be worth it or if users can live with the status quo.

flat minnow
#

Ok maybe a notice/warning on that in the documentation could save some time or if it's present I miss it

regal seal
#

okay, so i wrote this up into an issue, cause this also bothers me

#

I've written down all the solutions that I think are reasonable (feel free to propose more) - but I think the last one is actually a reasonable route forwards.

I think we can kind of change the DX as @tawny carbon suggests: we continue to support opts structs, but we also introduce support for opts funcs - so you can either do:
dag.Foo().Func(dagger.FooFuncOpts{XYZ: "opt goes here"}) or dag.Foo().Func(dagger.WithFooFuncXYZ{"opt goes here"))
this be completely backwards compatible, and the funcs approach allows you to very easily detect if a specific arg actually got set.

#

the big downside of this would be that now there are two ways of specifying opts - the upside is that both are pretty common in the go ecosystem, and users can choose which one they would want to use in a given scenario.

#

anyways, that's my proposal, lemme know 😛

frail plank
#

Hello guys. I have a question regarding traces and time spans. Could you please help me figure out what happens during "withExec" that takes so much time and is not mentioned in the traces? It takes a lot of time and I wish if it's possible to eliminate it

Screenshot_2024-10-29_at_17.03.52.png
violet cosmos
regal seal
#

welp

#

then i don't think it's solvable 🤷‍♀️

#

i think the other options are worse in terms of backwards-compat and usability

regal seal
# violet cosmos Yeah the opts func is a non-starter IMO. 1) complicated pattern to begin with 2)...

  1. complicated agreed - but i think it's relatively widespread in the go ecosystem? i've seen it everywhere from big codebases to little codebases - there's not a neat way of doing this in go natively, this is the option that folks tend to settle on.

  2. two patterns? do you mean between required+optional args? there are two patterns, but again, this is relatively common

  3. it doesn't have to be WithXXX, we could have it named differently - OptXXX, SetXXX, ArgXXX, OptionXXX

#

i think it's completely non ideal - but i think it's better than having to document that you just can't do this, and have modules in the daggerverse that cannot be consumed in go properly - i think that's what we need to weigh proposed solutions against

#

the idea is to support both opts structs and opts funcs - we would keep showing users the structs as a primary method of configuration, but have the funcs as another way, that allows us to neatly express this specific case

tawny carbon
#

Or, nudge towards the funcs in order to deprecate struct in the future and have only one way of doing this (at some point). 🙂

dagger.FooBarOpts().Arg1("hello").Arg2(false)
violet cosmos
regal seal
#

i definitely agree on the callee example

#

we should do that anyways

#

for callee goofy convention, i think this is almost exactly the same as what i'm suggesting - except i think we should keep the required args positional: doing otherwise is a huge departure from what we're doing in every sdk
we're also spending a lot of effort in the new "dagger shell" to avoid having named positional args, i don't think we should introduce them anywhere else

violet cosmos
# regal seal for callee goofy convention, i think this is almost exactly the same as what i'm...

Yeah it's definitely inspired by the other options you guys had in there.

I think "goofy" only makes sense as an opt-in, knowing the regular convention is still there. Otherwise it would be too many breaking changes.

BUT in the context of an opt-in, I think including all args is better, because it solves the middleware problem. It means it's always possible to handle your arguments as data. Which is what we need for middleware. Since the args type is generated, we can still make the API very tailored. Including a dynamic check for missing mandatory arguments.

Having a single argument in Goofy also keeps the DX clean and focused. It's more clear what each convention is about IMO.

regal seal
#

i think the problem of having entirely distinct ones means that if you want to call a module that has an optional bool set to true (the original problem to solve here), you have to switch to an entirely different calling convention

#

the func one covers more functionality, i don't think we should try and leave things in a state where we have two distinct conventions, with one that has more functionality than the other

i think we should either:

  • aim to have two conventions, deprecate the old one, and aim to switch everyone over to the new one
  • or have something inbetween, and have the "merged" convention i suggested

otherwise, i think we end up in a really confusing state - e.g. what do we do for docs where there are two different conventions? in the "merged" convention, it's an extension of the existing one, so it's easy to document as part of the go sdk, and easy to discover when using the ide

violet cosmos
#

We will have the same problem with "merged". Do we convert every doc and example to using opt funcs instead of structs? That seems like a big disruption, and qualifies as a "breaking change" for the end-user's code, even if technically the graphql API doesn't change.

regal seal
#

we're also doubling up the number of method - with two distinct calling conventions, every graphql function gets two go functions
one of them is going to have the main name "Foo", the other will have to get "CallFoo" (or similar). the names for opts structs will likely have to be different as well, we probably can't/shouldn't share them

what if a module declares two functions "Foo" and "CallFoo"? there's so many edge cases here around having new automatically generated names.

violet cosmos
#

Yes that is true. It's why I prefixed everything with Call in the examples. There are no edge cases, but it would definitely add more types and methods like you said.

#

It seemed like an option worth discussing for me, considering how painful our past attempts at changing call conventions in Go were. I don't really look forward to repeating that experience.

#

The main downside IMO is having to switch call convention on the caller side, for the one call that has a boolean that defaults to true

regal seal
#

another option could be to support two calling conventions - but each module opts into one-or-the-other

#

instead of each function call getting to pick and choose

regal seal
#

going to summarize thoughts on the github issue, just for the record (discord searchability is not my friend)

frail plank
regal seal
#

i can take a quick look

frail plank
#

I'm asking about the time span that goes after the "withExec" trace

#

and there is nothing specified what actually happens there

regal seal
#

i'm having difficulty making out the details of the screenshot, but from what i can tell it looks like the from is to a non-dockerhub registry

#

potentially, the image resolution in From is taking some amount of time?

#

but without a trace link or engine logs, i don't really have any ideas

frail plank
frail plank
regal seal
#

are there any details under the from span? also, i can't make out anything in the screenshot, all the text is blurry

frail plank
#

just open it in the browser

regal seal
#

are there any more details under the from span? there's a button to expand that.
also curious to see all the details under the withExec span, and the stdout span

frail plank
#

ok one moment

frail plank
sonic idol
#

main.go vs a module.

Is there any guidance on the tradeoffs and benefits of either approach? Where main.go is a direct call like client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

violet cosmos
# sonic idol `main.go` vs a module. Is there any guidance on the tradeoffs and benefits of e...

Modules are more powerful, better documented, and a better onramp into Dagger for your end users. The tradeoff is that you have to learn and embrace Dagger's programming model.

Even if you start with just main.go, if you get serious about your usage of Dagger, you'll write modules. Actually those two things are not necessarily mutually exclusive: a Dagger module lets you extend the Dagger API with your own types and functions; whereas main.go is a custom way to call the Dagger API. In theory there's nothing keeping you from writing a main.go that loads and calls modules - but in practice we haven't made that a very polished experience. So in practice people either do one or the other. We hope to change that in the future, and make it less of a radical "left or right" choice

#

The most common usage pattern is modules + CLI:

  • Implement your pipeline logic as custom functions and types in a Dagger module
  • Call pipelines (using your module) from the Dagger CLI. No need to write a custom client
#

IMO the killer feature of modules, is the cross-language composition model. Everything you implement is a reusable lego brick from day one. In the context of CI/CD, implement your pipeline logic as Go code is great, but not enough. You need interop with other parts of the CI/CD stack that are CLI-based, or Python-based, or Typescript-based etc. By starting from modules, you get that interop from day one. CLI interop in particular is huge. Any Dagger function you write in Go, is instantly available from the CLI. That makes that code way more likely to be useful outside of the controlled setting of people who don't mind reading your Go code and linking to it directly.

sonic idol
#

Yea I gathered that as being a major benefit from your interview on Go Time.

#

My initial use-case was adjacency to the code I'm working on and "standard go" workflow.

violet cosmos
#

For example, here's how to get a dev build of Dagger version 0.13.0 (CLI + engine) in a container ready to use, from the CLI:

dagger call -m github.com/dagger/dagger@v0.13.0 dev

You can call this command right now and it will work the same. So can anyone on our team 🙂 It happens to be a pipeline written in Go, but it's not "trapped" in the Go ecosystem

violet cosmos
sonic idol
# violet cosmos Would love to learn more, do you have specific constraints in your project that ...

Not particularly. More just thinking of evolution from "simple build scripts" to more evolved pipelines. The ability to use go run ci/build_docker.go is pretty easy to grok for anyone familiar with Go. Whereas dagger call requires a (very minor) degree of knowledge of getting dagger up and running locally. I'm just trying to build a mental model of how dagger is ideally used vs how I might be shoehorning it based on my priors.

#

I've got a few use-cases right now:

  1. Reusing "other repos" Dockerfiles and adding an additional "stage" so I can publish a lighter container (e.g. using distroless, cleaning up intermediate layers correctly, etc).
  2. Mirroring across container registries.
  3. Building and publishing images of code we own and eventually having a CD pipeline around them.
#

For the 3rd I'm still working through what "good" would look like after the docker image is published (e.g. promotion through multiple envs automatically and/or with manual gates as required).

Is the reference section here a good place to start? https://docs.dagger.io/integrations/kubernetes/#resources

Additional what that looks like with a monorepo where ideally we only publish what has changed rather than the entire repo.

Kubernetes | Dagger

This section covers different strategies for deploying Dagger on a Kubernetes cluster.

steady bluff
#

Sorry to butt in (again), but this is the one thing I'm still not getting under my belt. The "everyone on your team can do the same thing" bit is understandable, but what about automation? I'm of the opinion that anything Dagger does should be able to be done in an automated way (too). Otherwise, why have it? Right? So, does this mean whatever automation I come up with should be using CLI calls to run any work in Dagger (like a human would)?

Learning "the right way" to use Dagger modules or code using Dagger client for automation is highly important to me and why I'll go back to being a fly on the wall/lurker here. 😊 The replies will hopefully get my understanding where it needs to be.

rocky karma
steady bluff
violet cosmos
#

@steady bluff the Dagger API can be called from CLI or code, with 100% feature parity. Since Dagger Modules work by extending the Dagger API, they inherit this property also. So every function and type you define in your module, is instantly usable from the CLI or code.

#

There is no scenario where you have to use the CLI instead of code. And vice-versa 🙂

steady bluff
ruby reef
#

How do I run something like go fmt via dagger and get the results out ?

#

(without users having to know about export or having to export the entire repo back out)

#

I was expecting WithDirectoryMount to handle that, but it looks like any changes are written to the copy on write volume, not to the host.

#

AFAICT that's not possible though, w/o export --path=./ --wipe ... .. is that right ?

violet cosmos
#

there is a dagger call -o PATH which is a shorthand for export. But otherwise, that's corret @ruby reef , pipelines that involve generating / modifying local files are not 100% hidden from the end user. We want to improve that, without breaking the sandboxing that makes Dagger work so well

ruby reef
#

This command also errors .... dagger -m .moddir call --vault-token env:VAULT_TOKEN go-fmt export --path=./ --wipe
! unknown command "export" for "dagger call go-fmt"

violet cosmos
#

what type does go-fmt return?

ruby reef
#

oh right

violet cosmos
#

export works for types File and Directory

ruby reef
#

that makes sense

violet cosmos
#

By the way, we are experimenting for a new command-line UX, that might be more intuitive for end-users. It's not finished but you can try it with dagger shell

#

It's a bash-compatible interpreter for composing dagger pipelines

ruby reef
#

How do I return a container and also run something ?

violet cosmos
#

the command above would be come:

go-fmt | export --wipe
violet cosmos
ruby reef
#

oh right withExec will run the cmd and return the container

#

hmm. doesn't have any error return though. Will withExec panic if it errors?

violet cosmos
#

No error just means the operation is lazy. The engine itself will return an error if it encounters one, when it actually resolves the DAG

#

So technically withExec returns a promise of a future container

ruby reef
#

! failed to export: failed to copy to tar: rpc error: code = Unknown desc = destination "/Users/emuller/go/src/github.com/foo/bar" is a directory; must be a file path unless allowParentDirPath is set

#

oh right, need to return a *dagger.Directory

#

Hmm. This code base is > 300MB, so that won't work well. Probably not a good use case for dagger the way it is atm

violet cosmos
#

Are you worried of performance impact of upload 300MB into the engine on each run?

ruby reef
#

@violet cosmos downloading files again after the run

#

it's already been 7m

violet cosmos
#

Are all 7m actually needed, or is that mostly re-downloading files that haven't changed?

#

If it's the latter, you should be able to filter them out without too much trouble

granite valve
#

Any docs on how to use dagger directly in golang instead of using the cli?

ruby reef
#

@violet cosmos how would I determine the changed files ?

#

(also not 7 million, but 7 minutes and continuing ... I killed it)

#

I can't run git commands in the container because .git is excluded (it's ~1GB)

violet cosmos
ruby reef
#

Add adding Diff(<original directory>)

#

lol well that doesn't work well with --wipe
Just nuked my local checkout