general

[mystic mantle]

Dagger is not a CI platform like circle ci, github actions, travis ci, etc.
It is a CI toolkit that let you codify your CI in it, and call some subpart or all of it (we would love to integrate more with those systems) from those platforms. As such, we don't have some server that waits on some webhook to start pulling some code and building stuff. It might come in the future if the interest is there for the community.

[candid wave]

I do have my server, it's running woodpecker now instead of dagger

[sharp marsh]

you can still leverage your current woodpecker setup and run Dagger withing your Drone deployment. Good thing is that the CI UI and configuration won't change for you, so you don't need to spend time in changing that and you can focus primarily in your apps pipelines.

I don't think we have Drone specific examples, but happy to help you out if you want to try it out

[candid wave]

im not gonna play the nesting game, if there is no native server its not a tool i will use

[sharp marsh]

out of curiosity, how do you currently build docker/OCI images in woodpecker? Do you use the docker CLI? or you're relying on a different builder?

[mystic mantle]

I'm sorry if I pinged you unnecessarily, I was going through the channel after the holidays.
Having used Drone for years, I do understand you of not having to maintain too much CI stuff around. The drone file format migration were quite painful along the years. I still like it, though.

[candid wave]

i don't, i run basic images and and upload certain artifacts to pypi (i primarly need test running and deploy against pypi&co

for the occasional docker image build thats still necessary, podman build + a dockerfile do currently solve what i need

[sharp marsh]

gotcha. If we come up with a drone native dagger plugin similar to the docker one, would that be something that could justify trying out Dagger? Or you're looking into getting rid of the woodpecker setup entirely?

[candid wave]

a key part for me is good integration of the pipeline concepts in the ui (not just cli)
so there is command output logs, artifacts that are reports, artifacts that are releasable items (like python wheels, container images) and caches of multiple levels

[sharp marsh]

I see.. mind sharing which CI system in your opinion currently presents this information the best? Even though we're not focusing on that in this specific moment, it's good to keep an eye on good examples.

[candid wave]

from my pov github is currently the best, but that best is stil kinda horrible

[sharp marsh]

haha, yeah... I also find it difficult for some things. UI / UX particularly for this things is not easy; we'll try to do better

[candid wave]

its very easy to "sprinkle shit" all those information in a way that's a terror, its a painfully hard art to line them up in a way that makes it a joy to get when needed

[tender stump]

both approaches can work i think. Its interesting to evaluate both approach, in the last you mentioned the command runs inside a container?

[sharp marsh]

yes, if you run your command inside an action, in that case using dagger-cue --log-format plain will give you a log stream you can follow

[sharp marsh]

here's an example:

package hello

import (
    "dagger.io/dagger"
    "universe.dagger.io/bash"
    "universe.dagger.io/alpine"

)

dagger.#Plan & {
    actions: {
        _alpine: alpine.#Build & {
            packages: bash: _
        }

        // Hello world
        hello: bash.#Run & {
            input: _alpine.output
            script: contents: "while true; do echo hello; sleep 1; done"
            always: true
        }
    }
}

By running the command in the action, you'll see the outputs while its running.

[tender stump]

Thank you!

[lime path]

Hi, sorry if this information is readily available as I'm quite new to the whole js ecosystem. Is the Node dagger sdk pure esm package ? Meaning I cannot use this package in a commonjs project ?

[slender star]

@last pine @mystic mantle ☝️

[last pine]

Hi @lime path ! Yes, Dagger node SDK is pure esm package.
You'll need to add type=module in your project so AFAIK you'll not be able to use commonjs

Sorry @lime path I wanted to say not be able to use commonjs

[timber sphinx]

Azure devops as far as pipelines go so far is the best I’ve used even though I prefer to use github actions for immediate feedback when possible. The templating capability is just so far beyond most systems I’ve touched. That said maybe gitlab or others do it. The artifact and test view for azdos is also really clear. If any screenshots or examples would be useful maybe a post or github discussion and mention me? I’ll add anything that could be useful in your research.

[sharp marsh]

@last pine AFAIK there's a way to import dagger in a cjs app, correct?. ie:

(async function() {
  // initialize Dagger client
  let connect = (await import('@dagger.io/dagger')).connect;
  connect(async (client) => {
    // get Node image
    // get Node version
    const node = client.container().from("node:16").withExec(["node", "-v"])

    // execute
    const version = await node.stdout()

    // print output
    console.log("Hello from Dagger and Node " + version)
  }, { LogOutput: process.stdout });

})();

^ that can be used in a traditional index.js file without having module defined in the project

[lime path]

yup, thanks !

[sharp marsh]

@lime path we encourage the use of esm when possible since it's becoming the standard in the JS ecosystem. However, if you have a CJS project, you can use the snippet I've pasted above to make it work

[lime path]

Ah I see, I will play around with it, many thanks !

[last pine]

Oh.. you're right. I've never thought about importing Dagger this way. Good catch @sharp marsh 💪

[sharp marsh]

that's great feedback Sheldon! I guess a good start would be to bump this discussion @slender star started some time ago: https://github.com/dagger/dagger/discussions/3800

cc @candid wave

[last pine]

We should definitely document that. cc @heavy karma

[last pine]

I’ll create an issue

[fringe hedge]

Thank you for adding Reminder Bot!

To get started:
• Set your timezone with /timezone
• Set up permissions in Server Settings 🠚 Integrations 🠚 Reminder Bot (desktop only)
• Create your first reminder with /remind

Support
If you need any support, please come and ask us! Join our Discord.

Updates
To stay up to date on the latest features and fixes, join our Discord.

[lusty void]

Hi dagger team, may I know if WithMountedDirectory will copy the directory into the container or just mounted it? I have a relative large repository (due to build artifact exists inside the directory). When running dagger using WithMountedDirectory, dagger eat up all my disk space

[winter linden]

It only mounts. But if you use Host.Directory as a source of the mount, that does copy (upload) into the runner. You can use the exclude parameter to avoid uploading unnecessary & large subdirectories. I hope this helps.

[lean pawn]

Hi, Could anybody explain me how to catch up STDERR from CI container which returned with not zero exist code ?
I'am using Go SDK, example function:

func installCRD(ctx context.Context) error {
 fmt.Println("Install CRDs")

 client, err := dagger.Connect(ctx)
 if err != nil {
  return err
 }
 defer client.Close()

 wd := client.Host().Directory(".")
 kubeConfigFile := client.Host().Directory(path.Dir(localKubeConfigPath)).File(path.Base(localKubeConfigPath))
 ciContainer := client.Container().From(kubectlImage).
  WithEnvVariable("KUBECONFIG", kubeConfigPath).
  WithMountedFile(kubeConfigPath, kubeConfigFile).
  WithMountedDirectory("/workdir", wd).
  WithWorkdir("/workdir").WithExec([]string{"version"})

 stderr, err := ciContainer.Stderr(ctx)
 if err != nil {
  log.Println(err)
 }
 log.Printf("CONTAINER STDERR: %s\n", stderr)

 return nil
}

ciContainer.Stderr(ctx) return not nil error but stderr is always an empty string. What I do wrong?)

[winter linden]

Container stderr on error

[lusty void]

Hi there, I have several question on dagger cache mechanism.

1. Previously I see that Cue SDK have the options to export/pull CacheVolume from external registry. May I know how to do it in Go SDK?
2. may I know how to purge CacheVolume in case we dont need it anymore
3. Is it safe if we use the same cache key for multiple/concurrent build? In case we have multiple dagger engine using the same cache registry

[sharp marsh]

👋

1. Still not implemented for SDK's but will land soon: https://github.com/dagger/dagger/issues/3965
2. https://docs.dagger.io/1229/empty-buildkit-cache
3. Yes. Worst case scenario, some builds will pull old caches

[lusty void]

For (2), it is either purge all cache or just dont use any cache right? Can I purge just 1 layer using the cache volume key?

[sharp marsh]

correct. You can't purge only 1 layer

[lusty void]

If that is the case, we might need to allow for multiple remote cache source together in a run for storage efficiency

[sharp marsh]

not sure I understood

[solar bison]

Hi 🙂 I have a little question regarding the implications of CacheVolumes. Basically, (1) where are they stored, (2) can they theoretically be backed by something like Google Cloud Storage in order to persist them across different Engine hosts? Thank you 😄

[sharp marsh]

👋

1. they're managed by buildkit cache (https://github.com/moby/buildkit/blob/master/README.md#cache)
2. Buildkit has the ability to export cache to s3 and azure blob (experimental) currenty https://github.com/moby/buildkit/blob/master/README.md#s3-cache-experimental. Unfortunately, seems like GCS is still not available.

[solar bison]

Thank you 🙂 Will read up on those 😄

[turbid tulip]

Last I heard, CacheVolumes are not included in that exported cache. @pseudo stream would know more confidently.

[sharp marsh]

right, forgot to clarify that. here's the official issue about it with some suggestions on how to handle it (https://github.com/moby/buildkit/issues/1512). Thx Joel 🙏

[silver rivet]

Hello. I'm looking to replace gitLab-ci with runners for windows and linux so my tests can run on multiple platforms. Are there any ideas floating around for using dagger to solve that problem? If I've understood the approach running tests for windows stills requires another approach.

[winter linden]

👋 this is correct, although Dagger can run on Windows, it can only run Linux containers. As a result it cannot run your test suite on a Windows build (except with vine or other esoteric solutions). We may add this capability in the future but not in the short term, as it requires modifying buildkit, which is a core upstream component.

[solar bison]

Thanks @sharp marsh and @turbid tulip 🙂

[silver rivet]

Thank you.

[real kite]

So just to clarify from this response, is using remote cache currently not possible at all when using an SDK? Or is there a workaround currently if I just want to cache the results of my build using Github Actions' cache, for instance?

[winter linden]

Correct, not possible at all (feature regression from 0.2 to 0.3) but will be fixed very soon, possibly with an experimental/unstable flag first

[green leaf]

Is there an issue to follow for this? This is interesting to me too!

[sharp marsh]

yes. https://github.com/dagger/dagger/issues/3965

[lusty void]

Hi dagger team, I have a multi stage Dockerfile from https://github.com/FoundationDB/fdb-build-support/blob/main/docker/centos7/Dockerfile
May I know if it is possible to build only a specific target with Go SDK?
docker build --tag foundationdb/build:centos7-latest --target build . this is what native docker used

[swift inlet]

Hi, it doesn't seem possible as of now: https://pkg.go.dev/dagger.io/dagger#Container.Build && https://pkg.go.dev/dagger.io/dagger#ContainerBuildOpts. If it's something you are looking for, can you please make an issue out of it ? It will help prioritize the backlog 😇

[swift inlet]

@lusty void, I double checked the source code. It turns out that the latest engine (not released yet, but will be probably today) contains the buildArgs option: https://github.com/dagger/dagger/commit/b21f40664090944e5990788db11e32f8033b383c. You can either:

1. work today on a local version of the engine
2. wait for tomorrow 😇
   PS: no need to create the issue

[sharp marsh]

@swift inlet correct me if I'm wrong but buildArgs won't enable what @lusty void is needing, correct? Since he's trying to set the --target option with Dagger through a Dockerfile but we don't support that for the Build step

[swift inlet]

I thought that we could add this target option as a buildArgs. Let me check if possible 😇

[sharp marsh]

I don't think that's possible since target and buildArgs are different buildkit opts AFAIK.

[swift inlet]

Tested directly on Buildkit. Doesn't work, you're right: DOCKER_BUILDKIT=1 docker build --no-cache -f Dockerfile --build-arg "target=stage1" . @lusty void it won't work, my bad. Creating the corresponding issue and prioritizing it in our backlog 😇

[real kite]

Great, that would be awesome to see sooner rather than later! I'm actually starting to move some of my Python SDK stuff over into production and it's a sorely needed feature for that.

[swift inlet]

It was already here: https://github.com/dagger/dagger/issues/4244 😇

[frozen basin]

Hey yall - quick question re: multi-architecture builds... might be more related to k8s/registry than dagger, but I think this group will know whats up.

Say I've built a multiplatform image, and pushed it to a registry... Will k8s know which image to pull based on the node's architecture? It looks like I might be able to use a single tag with multiple supported architectures. Basically a little worried about what will happen if I flip a switch and start provisioning ARM systems, hoping kubelet is smart enough to pull the right image

[sharp marsh]

Yes, k8s will try to resolve the image automatically based on the node architecture

That's supported automatically by containerd

[frozen basin]

dope. ty

[swift barn]

are they future plans for dagger to support multiple backends like k8s, or maybe even other ci's like github actions?

[winter linden]

multiple backends?

[neon warren]

Is there a way to specify a remote buildkit address with the Go SDK?

[rigid epoch]

I know that at some point there was talk about dagger "services" for e.g. development -- has there been any movement on that front? Probably also relevant to the dagger-on-k8s question

[winter linden]

Lots of movement in the design discussion. I think we're converging towards a design everyone likes, on the networking front (which was the bottleneck). We have a bandwidth problem (so many features to build, so little time), so it got a little delayed, but my guess is that we can ship it in February or March. Others may disagree 🙂

It would be sooner, but we prioritized solving the "dagger in your CI" operational story first

OX over DX on that one

[rigid epoch]

yeah that totally makes sense to me

just leaped back into my head as I was thinking about decomposing my frontend into libraries -- raises interesting questions about how you might stitch together "build pipelines" for development-specific build flows -- for things where there's good tooling for incremental builds and you want to run an ongoing server

[elfin frigate]

here's the PR with the most recent discussion around networking/services: https://github.com/dagger/dagger/pull/4163

and yeah, for me CI and networking are 1a/1b priorities, hope we can get to it soon 🙂

[winter linden]

speaking of OX and CI, @rigid epoch we will have a demo for you very soon 😁

[swift barn]

Hi! Quick question, do you know how to output to stdout with dagger-cue? I have this in my plan:

version: bash.#Run & {
  input: run.output  # python3 image
  workdir: "/app"
  script: contents: "python3 --version"
}

and this is the output I am getting:
{"level":"info","task":"actions.version._exec","time":"2023-01-11T10:52:09Z","caller":"/home/runner/work/dagger/dagger/client/client.go:401","message":"#9 0.114 Python 3.9.7"}
But I would like Python 3.9.7 instead

[rain oriole]

Sure, see here: https://docs.dagger.io/sdk/cue/490715/handling-outputs

Note --output-format as well.

[swift barn]

thank you!

[swift inlet]

Hi @lusty void 👼,
A PR enabling you to build a Dockerfile to a specific target just got merged: https://github.com/dagger/dagger/pull/4362. It is currently only available on the main branch, but should be usable right away 😇 Do you know how to work with the dev version of Dagger? Otherwise, it will be available in the next Dagger release

[chilly arch]

New Tutorial Alert - Replacing your Dockerfile with Go code - thanks to @heavy karma ! https://dagger.io/blog/replace-dockerfile-with-go

[rare night]

Quick question, is the Cue version of Dagger is not the preferable way to go?

[slender star]

Hi @rare night 👋 The new multi-language SDKs are the ones based on a common GraphQL API and are likely the best place to start if you're beginning your Dagger journey. Would one of those SDKs be a fit for you/your team?
Go, Python, Node.js
https://docs.dagger.io/#which-sdk-should-i-use

[lusty void]

I assume we just go get the latest commit right?

[slender star]

Not 100%, but guessing with a clone of dagger/dagger with latest and your Dagger code in pwd, you could run <path to dagger repo clone>/hack/dev go run main.go for example.

https://github.com/dagger/dagger/blob/main/hack/README.md#hackdev

Yep. Worked for me.

hello-go ➤ tree
.
├── go.mod
├── go.sum
└── main.go

0 directories, 3 files
hello-go ➤ ../dagger-fork/hack/dev go run ./main.go

[winter linden]

Hey @rigid epoch I'm playing with your monorepo scanner: https://github.com/davidwallacejackson/dagger-monorepo-dep-crawler . I was wondering if this could be packaged as a standalone command-line tool, usable as-is without having to know about dagger at all. wdyt?

[winter linden]

Kudos to @vestal creek for introducing @past ivy to Dagger, who's showing us an awesome demo right now 🙂

[vestal creek]

Link? It's not in my calendar 😢

Nevermind, I'm in

Ha. I caught the last 3 minutes

Sorry, I didn't know we had to sign up via the Zoom thing. See y'all at the next one

[winter linden]

No worries, it's all recorded. You got multiple shoutouts. 🙂

[past ivy]

@winter linden to expand a bit on the CI bit: one short-term option I've been considering is this kind of approach:

1. Generate a gitlab CI with one job per service
2. Have our dagger code send requests to gitlab to update pipeline/release status

This is still a bit messy, but I think we might be able to still run it in gitlab, but simultaneously send gitlab a request to create an external pipeline execution, which (might?) let me display it like a multi-stage thing while running it as one dagger execution, possibly, maybe

[winter linden]

Yeah we should really show you a demo very soon 🙂 cc @slender star @chilly arch

The short version is that yes it's a viable approach, and we're considering productizing it. Would love your early feedback.

[warm temple]

I forgot to link my demo repo in the zoom chat, here it is: https://github.com/kpenfound/hello-nomad

[vestal creek]

@chilly arch When the Zoom recording is available, can you drop the link to me, please?

[winter linden]

One thing I'm curious about @past ivy is the VM parallelization requirement. That's one sticking point for us at the moment, that prevents shrinking the yaml "glue" to a single file that never changes (one mega-job, everything dispatched by Dagger).

We're exploring ways to hook VM parallelization into Dagger, but that will take some time and effort (will happen eventually though).

In the meantime, I'm curious how you go about determining the right level of parallelization for your workloads, how much you utilize the VMs etc. Did you go as far as benchmarking different infrastructure profiles?

One theory we have is that, for some workloads (perhaps most?) the efficiency gain of switching to Dagger is so great, that even with less VM parallelization, the actual pipeline run time is faster anyway. We're still gathering data to test this theory - but curious to hear your thoughts.

[past ivy]

For now, it's done based on visualisation and retry needs; image builds are often one of the slower pieces, and so if it fails (e.g runner disconnects, times out, whatever else), it's useful to be able to retry from a point (we talked about "checkpoints" previously). The other part is that it just feels nice to see the multiple steps, and it makes it fast for developers to find what failed.

The performance aspect only really becomes relevant (at least this is my guess) for running pipelines for multiple services

[winter linden]

The performance aspect only really becomes relevant (at least this is my guess) for running pipelines for multiple services

Ah I see. In your case that would be multiple sub-directories in your monorepo? Each a distinct dagger "project" so to speak?

[past ivy]

The other thing to consider of course is that we specify a certain amount of resources for each gitlab job that is created by our runners, and it's very valuable for us to be able to spin down servers based on need.

[past ivy]

We have one root build/dagger/index.ts that we execute, which will read the ci/index.mts from a given service

so you might run ./build/dagger/index.ts build {service}

which will then import the dagger setup from services/{service}/ci/index.mts, and run it

[sharp marsh]

This is what I've usually seen in different projects. The downside of this is that it adds a bit of performance overhead since parallel jobs don't have a direct way to share data one with another, so you need to rely on the github cache for that and/or upload/download artifacts across different steps. @winter linden this is one of the pipelines that I helped migrating for the CI K8s book where you'll be able to see this:

https://github.com/salaboy/fmtok8s-frontend/blob/main/.github/workflows/ci_workflow.yml#L52 > original pipeline
https://github.com/salaboy/fmtok8s-frontend/blob/main/.github/workflows/dagger_workflow.yml > dagger workflow

You'll notice the original pipeline doesn't really run in parallel since it has a bunch of dependent needs between tasks as well as performing upload/downloads of artifacts to make it work. IMHO even though it works, seems like a hack that people have to do this to actually have better visualization and retry capabilities.

[past ivy]

Yep. I think we can get the visualisation aspect down easier tbh, by reporting as an external CI to the vcs provider, but retries are a harder bit (especially if you're still running within your regular CI's runners with docker:dind like we do, meaning you lose caching between runs without a bunch of work to setup shared cache)

There's a couple other things that make me want to split up pipelines, particularly around conditional execution. We rely on the changes keyword in Gitlab CI, to only run pipelines for services that have changed. I believe we could achieve something similar in dagger, if we'd e.g get a list of changed files, but that would still require us to start the job for each service (or run one massive job for everything)

[pseudo stream]

Just to double check my understanding, you lose the caching between runs in this case because it's not guaranteed that when you retry you will end up hitting the same node that has local cache from your previous attempt, correct?

[past ivy]

Correct. Note that I believe this can be solved with shared external cache which is supported.

[sharp marsh]

If you're using Gitlab's hosted runners then the cache will be gone for each run. @past ivy do you have self-hosted runners? or you're using Gitlab's?

[past ivy]

self-hosted runners (we run a lot of jobs)

self-hosted runners with the k8s executor and node autoscaling to be precise

[pseudo stream]

Cool, yes external cache is one possibility, another might be use of hashing to map the same builds to the same nodes. Setting that up might be a pain, but maybe if we were able to do that on behalf of users somehow it could be a viable route too. Has less tradeoffs than managing external cache. Just kind of thinking out loud here

[sharp marsh]

AFAIK this is resolved automatically by Dagger as long as you can share the cache between runs. Since buildkit automatically understands which files were modified and will only trigger the parts of the DAG that should recompute. Is that correct @pseudo stream ?

[past ivy]

still beneficial to use external cache imo as the builds for services would be largely identical aside from the compilation step

[pseudo stream]

Yes that works if your job always goes to the same node with the local cache from the previous attempt, but I think the problem is when you have multiple self-hosted runners, you can resubmit a job but it's not guaranteed to end up on the same node

[pseudo stream]

Yes absolutely, hashing wouldn't be mutually exclusive w/ external caching. External caching is best but takes a lot of tuning and fancy optimizations to get working really "magically". Certainly something we want to do though

[sharp marsh]

startup idea: use dev machines as self-hosted runners using tailscale and you'll have a fairly updated build cache most of the times 😉

[sharp marsh]

I guess it'd be great to have some sort of cache background replication / snapshotting along with a sumdb so you can mount a volume in the node and resume from there. If pipelines are created using the best practices, a big part of the cache shouldn't change that often

[pseudo stream]

Right, that's the dream without a doubt

[winter linden]

I wonder if the sweet spot may end up being:

1. Spawn a new machine per change, per project
2. Run the entire suite of pipelines for that project change on the one machine - no further parallelization.

That way you get the full benefit of elastic compute resources (compute usage scales with change frequency, and number of projects in your monorepo). But you also get the full benefit of running your project's DAG in a single dagger session: maximum cache reuse, optimal concurrency. Not to mention less yaml. Also less moving parts in orchestrating cache and machine replication, and all the other fancy things being discussed here: those moving parts have a cost, both in reliability, maintenance overhead, etc

Then you simply grow the allocated machine configuration to match the demands of an individual project change (which you can figure our by monitoring)

[past ivy]

I might experiment a bit with the visualisation bit over the weekend if I get a chance, I think there's some pretty easy things that can be built to enable it

[past ivy]

Love the idea – our experience is that spawning machines + time to connect to cluster is a significant portion of the total time of an execution

[winter linden]

https://tenor.com/view/i-rest-my-case-vanellope-ralph-breaks-the-internet-gif-15441859

[past ivy]

Oh wait I see what you're saying, spawn one node for each change, and run the whole DAG

I misread that from our context of multiple jobs per svc, but yeah, if we could run the whole DAG, that'd be more tolerable. Downside of that is that for pull requests/non-main commits, we often just run test + lint. Those are very fast. Build is what takes time.

[winter linden]

right. then scale that machine vertically. possibly all the way to a dedicated bare metal node per change. that’s a lot of room for growth

[winter linden]

possible optimization might be: control plane inspects the change in that commit; magically infers what the best machine size is based on past runtime telemetry 😁

https://tenor.com/view/dear-santa-saturday-night-live-message-for-santa-to-santa-letter-for-santa-gif-19567806

[past ivy]

I think we today have the ability to do per-service DAG which gets close. My dream is one DAG for the entire monorepo, only running pipelines for changes (= not even tagging a new version of a service, just completely skip that whole node)

re running on multiple nodes: is there some possibility to build a system where you have one (or a set of) dagger agent(s) to build DAGs, and then be able to distribute the execution of that DAG onto one of a set of nodes?

[winter linden]

We will have that eventually. But it will take a while.

[past ivy]

So the gitlab executor has a couple agents in our cluster, and those are responsible for spawning runner pods. Curious if we could have centralised dagger agents building DAGs, and then spawning executors separately (e.g something like dagger execute --dag=http://dagger-agent/dags/30badde3-6c6d-47f5-b59d-00966894b400)

[winter linden]

There are massive low hanging fruits to pick before then, too

[chilly arch]

Here is the recording of today's community call.

Thank you to @past ivy , @mystic mantle , @heavy karma and @warm temple for taking the time to show off their demos! https://dagger-io.zoom.us/rec/share/jtuWEoBkfO4OBvh5wnqq5UTgBShFTQnlluycAb81s74M-QRbZ-QCf6j_JpwWYWRb.urt7ip9J7ozjdQAV

If you ever need to get any of the older demo recordings, you can find all of them here too -https://dagger.io/events

[lilac sphinx]

Hi all, looking for some advice. We are about to PoC using dagger in our company which has about 40 git repos and some of our CI pipelines are on the complex side of things. We use gitlab-ci (bash + too many shell scripts) right now and we are exceeding their API limits with few options remaining. 90% of the code written at the company is golang. We are in the process of moving to ArgoCD for deploys.

What would you all recommend we use for picking up dagger? Go SDK? CLI? CUE?

Also, we are fully in production, should we hold back and wait for more stability before adopting dagger?

Thanks in advanced

[slender star]

@lilac sphinx Short answer is that the Go SDK is your best bet given your company already uses it heavily. We'd love to chat more about the nuances of setting things up in GitLab if you're open to it 🙂

[lilac sphinx]

Cool!

Yeah, im currently maintaining all the gitlab stuff right now, so im interested in the details

[slender star]

Awesome! I just sent you a friend request on discord so we can coordinate a time 🙌

[past ivy]

We're currently setting up a prod system (have one service there currently) with Gitlab + dagger + argo, happy to chat as well

[fossil pine]

I started building a high-level Go SDK for Go projects using Dagger.

First draft: https://github.com/sagikazarmark/dagger-go-library/pull/6

(magic happens in the ci directory)

It doesn't aim to be perfect, merely a friendly interface over Dagger over Go commands. It should make using Dagger for simple libraries and applications easier.

Feedback welcome!

[warm temple]

Hi all looking for some advice We are

[timber sphinx]

You made me think azure DevOps agent scalesets with this. Ephemeral vms ondemand for self hosted agent runs. Cool stuff

[atomic plank]

Hey, can someone please help me understand what is actually going on when I'm running a local engine with Docker?
I mean, literally, where is the underlying container running?
If I'm running

const node = client.container().from("node:18").withExec(["node", "-v"])

Everything works fine, but I don't understand what is happening behind the scenes. Thanks!

[fresh depot]

Hey ! It spawn a docker container with the dagger engine and also a dagger session(graphql api) with connect(async (client: Client) => {
And then everything is run inside the dagger container. But your line alone does not do anything you need const version = await node.stdout() to actually execute thing

[solar bison]

You need to indicate to the engine that you actually want to know what happened in the container for it to also execute 🙂 Like retrieving the output or status code of the last command or exporting a file from the container 🙂

[mild obsidian]

If I use this sample, can you describe for me what happens?

#!/bin/bash

content=$(dagger query <<EOF | jq -r .container.from.withExec.stdout
{
  container {
    from(address:"alpine:latest") {
      withExec(args:["ls"]) {
        stdout
      }
    }
  }
}
EOF
)

echo $content

In other words, where does the alpine image runs? On my system it doesn't show being run. I only see the dagger/engine:v0.3.9 being run.

[ I am using the CLI version here but I assume the same concepts would apply to all other SDKs ]

[sharp marsh]

👋 the alpine container is executed by the dagger/engine container that's running in your machine which which uses buildkit for that.
buildkit doesn't use docker to run containers, that's why you won't see your container with docker ps. The containers buildkit starts are managed by runc.

[sharp marsh]

here's the process tree you'd usually find for a container executed through dagger in a typical systemd managed OS: systemd(1)───containerd-shim(3169260)───buildkitd(3169281)───dagger-shim(3181999)───buildkit-runc(3182009)+++

[atomic plank]

Thanks! So just to clarify, this happens LOCALLY, right? Meaning that if I want to test this without internet connection at all, it should work.
I'm a bit confused because when I'm spawning a docker container from within dagger, I cannot investigate it with the docker CLI.

[mild obsidian]

here s the process tree you d usually

Hi there. I am new to (hands-on) Dagger (albeit I have been following it for a while). I went through the docs / blogs and started to play around a bit. I see there is a lot that is centered around the CI / tests parts of the pipeline but there is not much I can see as example for the deployment parts of the pipeline. I do remember Solomon presenting (early on) about examples of deploying the same application across ECS, K8s and Vercel (if my memory serves). Is there any such examples available? As I said they don't seem to be in the docs available. Thanks!

[winter linden]

Hi there! Welcome and thanks for your interest in Dagger.

Yes you're right, recently our guides have skewed towards build and test. Deployment remains a great use case for Dagger, I'm sure we can find resources for you on that topic. And we'll make sure to add more, and make it all easier to find.

A few pointers:

• @warm temple wrote quite a few demos and guides, some of which involve Terraform.
• Just yesterday @heavy karma showed a demo of a pipeline deploying to Google Cloud. The recording should be available soon (cc @chilly arch )
• I know @vestal creek has been playing with Pulumi + Dagger

[magic mauve]

👋

@pseudo stream I was wondering if dagger has end to end encryption down into the buildkit layer? That is, is the grpc encrypted by default?

[pseudo stream]

No we don't currently set up any encryption on that layer. For the default path that's okay because everything happens locally (localhost + local docker socket), but it's something to be aware of if you're trying any of the unofficial experimental modes that use a remote engine. You'll want to make sure that the connection type layers encryption itself or that you're okay with it being plaintext.

[magic mauve]

Thanks.

[winter linden]

How does Dagger run containers

[heavy karma]

@mild obsidian here is a guide on using Dagger together with GitHub Actions and the Google Cloud SDK to deploy to Google Cloud Run: https://docs.dagger.io/sdk/nodejs/620941/github-google-cloud

[winter linden]

If you’re paying for CI, I have a question for you 😁 👉 https://twitter.com/solomonstre/status/1614050547375702017

Solomon Hykes / @[e-mail] (@solomonstre)

People who pay for CI: what cost model do you prefer? Usage-based (fixed performance, variable cost) or flat (fixed cost, variable performance)?

Twitter · Jan 14, 2023, 12:03 AM

[stuck wyvern]

Have to say, I'm just now starting to use the new SDK stuff... I really that instead of relying on an extra CLI (dagger) I can just write what I need to write and the SDK takes care of setting up the engine and everything. Truly is just "do you have a docker socket available? You're good"

[stuck wyvern]

And there's not really any trade-offs from using something like buildx and raw dockerfiles.

[stuck wyvern]

That said I'd really like to be able to choose what images are used for buildkitd and even the engine so I can plug that into my own supply chain.

[winter linden]

running your own engine image is coming soon; I think you can already do it now if you’re ok with experimental flags. Buildkit version is “soldered in”: swapping it out at runtime is not supported. But of course you can fork & build a modified engine image from source, with a different buildkit. Voids the warranty though 🙂

Also, glad you are enjoying the new SDKs @stuck wyvern 🙂

[stuck wyvern]

Really it changes a lot.

[stuck wyvern]

Whatever happened to extensions you were talking about during the pre-release phase?

[winter linden]

Still in development. We decided to launch the rest first, for focus. We worried we were stretched too thin to ship something good, and it was a lot to take in all at once for the market. Extensions are still planned, in the form you saw them.

It was the next big item on the backlog, but just got pushed down to second, after “fix the operator experience (OX)”, which includes things like your request for running a custom engine image 🙂

[stuck wyvern]

Nice, glad the change happened even without it.

[empty jay]

👋 hey, I wonder is there anyone tried to create kind cluster using dagger and docker socket? I'm having issue creating one and didn't able to progress well. Maybe a seeing a working example could help 🤔

[fresh depot]

Kind cluster with docker socket

[stuck wyvern]

Is there a way to get a stream of a built image without having to write it to disk first?

[elfin frigate]

@stuck wyvern unfortunately not, I wanted to add that but it's unclear how to model it with our GraphQL API. feel free to open an issue though

[past ivy]

I vaguely recall having a conversation about running containers with Dagger. The use case at the time was for feature testing, running another container on the side for the test to interact with. Are there any plans/some roadmap for this?

[swift inlet]

Hi 👋 ,
There is one, you're right: https://github.com/dagger/dagger/issues/3037 😇. However, this API is not the biggest blocker atm. What we are lacking is a nice DX for container networking: https://github.com/dagger/dagger/issues/4080 (which on top of our priority list). Once the container networking is implemented, then you'll be able to workaround the "service API" in your preferred SDK using subprocess and client side healthcheck implementations

[past ivy]

Awesome! Yep, networking is absolutely the more important one. My reason for asking now is that I'm considering a move away from docker-compose for development setups. We've discussed a bit about how we want to allow developers to control their service dependencies, while still ensuring that all services are up-to-date with our actual infrastructure.

The thinking then went that it'd make sense to expose, for example, a Database construct owned by platform, which would be updated whenever a change is made in our infrastructure (e.g if we were replacing mysql with mariadb, or changing over to another mysql version). That way, developers could declare a dev.mts (or dependencies or something else), with a list of services

This would also enable much better composability of dependencies, and not require us to build out a custom yaml format

[loud briar]

I was just about to hit submit on a help post in #1030538312508776540 , offhandedly tried **/.terraform/* in an exclude parameter and low and behold, it did exactly what I wanted 🤌🏻

[real kite]

I'd be curious what your terraform pipeline looks like in dagger. I just wrote one that is basically a more powerful form of terragrunt. I have reusable terraform modules such as /network, /project, /kms, /serverless and then my environments are a .env file + a python file with dagger code that calls a generic terraform.py over and over that runs a terraform module. Works quite well but is kinda slow because you're calling terraform init a ton of times. Caching the dependencies helps but not a ton.

[loud briar]

I'm not quite as far as that, just experimenting with service integration at the moment - terraform commands are working fine, getting vault authentication working next

[real kite]

nice, for services I ended up using docker compose and spinning it up with dind... it's... flaky and slow. Looking forward to when it's properly supported in dagger

[loud briar]

Vault is trickier since by default it places the token in a file in the home dir rather than an environment variable like most binaries

[real kite]

that is tricky. I will say that (may not be an option) if you're lucky enough to have control over how you handle secrets, I had really good luck getting sops to work well with the dagger way

it really made it easy for me to hand a pipeline to developers to run locally on their machines - as long as they are authed to gcloud, where kms lives, they can run the entire pipeline without caring about anything else

[loud briar]

If I had my way I'd not be using vault at all, but alas nope, I don't get a say in it

[warm temple]

Here's an example querying a specific value from vault 🙂 https://github.com/kpenfound/hello-nomad/blob/main/ci/utility/secrets.go#L33-L47

[loud briar]

That assumes the vault token is already available as an env var I believe

[warm temple]

Correct. What's your auth flow look like?

[loud briar]

Unfortunately it's vault login with the userpass method. I've found a way to do it simply enough, a user logs into vault as normal and I grab the file with client host directory and only include the vault token file

[warm temple]

Ah interesting. I wonder if you could vault login in dagger and grab the token for subsequent commands. I'm not sure if that's better though

[real kite]

I have a similar workflow for authing to gcloud currently, basically what I do is look in the user's home directory for ~/.config/gcloud/application_default_credentials and if it exists i mount it into the container, otherwise I echo out a service account key from an environment variable (this is the only secret that lives in CI)

[loud briar]

That's basically what I'm doing here. Use os.path.expanduser to get the home dir, stick .vault-token on the end, checkit exists and if so I grab it in dagger with client.host.directory(home_dir, include=[".vault-token"]).file(". vault-token").secret

It's not perfect but it works and it's not too scrappy

[real kite]

yeah, vault sounds like gcloud in that the auth flow is really meant to be done manually by a human. They seriously discourage usage of automation with service account keys really being the only way, so it's difficult to do a login flow in dagger and you kind of have to rely on the auth flow having already happened on the client machine if you want to be secure

but asking end users to do something like gcloud auth application default login one time is not really a big deal

[timber sphinx]

Do dagger workflows support interaction yet? Nothing here on that yet https://docs.dagger.io/162770/faq

Clarification, interactions in the workflow. I know I can use mage to invoke or prompt, but I'm assuming any dagger activity wouldn't surface a wait for user input, correct?

[sharp marsh]

not sure I follow sheldon. you're asking if there's a way to make dagger interactively ask the user for prompts?

[timber sphinx]

yes. I know how mage works, but if lets say I use dagger to invoke a shell script and that shell script asks for an input, does it allow that response or would I need to ensure all inputs asked for up front and then passed in?

[sharp marsh]

you'll need all the inputs to be passed-on in advance since Dagger can't know or be aware if any of the things that run actually require an input

and everything runs in containers similarly to docker build where interactive tty inputs are not part of the workflow

[stuck wyvern]

I'm trying to use WithUnixSocket but it's not showing up in the container.
Any ideas?

WithUnixSocket("/run/docker.sock", client.Host().UnixSocket(sockPath))

[slender star]

Take a look at this thread 🙏 #1064500691638440000 message

[stuck wyvern]

Good to know about the exec issue since I'd planned to do kind with this as well 🙂
Unfortunately mine isn't even getting the socket inside the container for some reason.

[empty jay]

I m trying to use WithUnixSocket but

[timber sphinx]

I keep trying to join dagger zoom call but it says waiting for host to start meeting. I just reregistered too and used that link and same result. Can someone share a link that worked. Maybe I did something wrong 🤷‍♂️ Edit: got it. Was email conflict with user. Didn’t tell me just timed out instead of error. Edit: never mind it did the same thing 🙁 Edit. This was older series and not scheduled for today. My bad!!’

[chilly arch]

Thanks for the update, Sheldon! For anyone who is looking for upcoming community calls, you can always see the next events or the recordings of previous events here. - https://dagger.io/events

[winter linden]

Calling for comments 😁 https://github.com/dagger/dagger/issues/4414

[timber sphinx]

Some great points. I’ll pin this for some response today or tomorrow. I’ve been mulling over where dagger fits in my current setup at my company and definitely recognize some of the challenges you’ve mentioned. 👍

[timber crater]

Are there any caveats, known issues, etc running dagger on podman/containerd ?
I did not see anything in the docs about podman.

Underrated thread: https://discord.com/channels/707636530424053791/1049373535849693225

[timber crater]

I know I'm relatively new here and this may seem like a strange suggestion.
I'm in a position to build a "proper" infrastructure hosting business as a coop and I'm looking for a project to host. This seems to fit my criteria in terms of vendor agnosticism and high standards.
I'm happy to provide more details.

("proper" includes things like stratum 1 time servers, private certificate authorities, SPIFFE, BGP and IPV6 support, payment service integration)

[winter linden]

I think other than the lack of docs, there are no known technical issues. In the past there was a glitch with podman because its "docker alias" feature doesn't handle image names exactly in the same way. But I believe that has been fixed.

[chilly arch]

Your CI needs CI: A New Way to Build CI Pipelines by @warm temple. Great read 🙂 https://dev.to/kpenfound/your-ci-needs-ci-a-new-way-to-build-ci-pipelines-2gbi

[past ivy]

just following up on this: is there a timeline for container networking yet?

[swift inlet]

Hi, I'll let the lead answer this question 😇🙏

[marble tendon]

So, new here, literally started reading about dagger maybe an hour ago. But between the official docs, https://github.com/dagger/dagger/issues/4414, and some threads here, I'm having trouble wrapping my head around what dagger actually...is? My CI/CD currently builds a couple images, then executes around 20 tasks with those images, some in parallel some serially, some retrying and some not, on ephemeral worker instances...am I dumb or does Dagger not really let me replace that pipeline with code (for the time being anyways)?

I started a reddit thread complaining about having to do my pipelines in vendor-specific yaml instead of the language the rest of the project uses, which is how I got here

[inland flower]

Coming from the same reddit post and also wondering if I actually could replace my gitlab ci yaml mess with python or whatever supported code 🤷🏼‍♂️

[sharp marsh]

yes, that's correct. Dagger enables you to replace "almost" all that pipeline glue with code. I'm saying "almost" because Dagger itself is not an E2E CI/CD platform (yet). We currently provide the engine which enables what you're correctly assuming but you then need to kick the Dagger pipelines from your current CI runner (github, gitlab, jenkins, etc). Does that make some sense?

We have some snippets here that shows how to configure Dagger with generally the most common CI runners out there: https://docs.dagger.io/768421/go-ci

[sharp marsh]

👋 welcome! mind asking which reddit post?

To answer your question:

yes, that's the idea. You can replace "almost" all your gitlab yaml pipelines with Dagger code. You just only need to have the minimum as shown here (https://docs.dagger.io/768421/go-ci) to make Gitlab trigger your Dagger pipeline

[marble tendon]

https://www.reddit.com/r/devops/comments/10gzdqg/but_really_why_is_all_cicd_pipelines/

[marble tendon]

Ok, so Dagger currently really has no notion of 'run X on 10 workers, while running Z on 3, if Z fails retry once but on a different worker' kind of thing?

[sharp marsh]

Not out the box, since as I said before, Dagger doesn't provide any infra yet to run your pipelines. Having said that, that doesn't mean that you can still use Dagger with your current CI runner and code that logic in-code instead of YAML. @marble tendon mind me asking what CI platform you're using currently to orchestrate that kind of logic? How are you currently defining those rules?

[marble tendon]

Sure! This is the main development pipeline for the project I'm working on

We're using buildkite with their AWS cloudformation stack. Each of those steps is defined via YAML that's uploaded at runtime (that's the first step).

After the two initial build steps run in parallel, as seen above, we kick off what can be kicked off:

a majority of the subsequent steps leverage BK's docker-compose plugin to run one of the built images (test or prod) plus required dependencies

most of the steps are just calls to shell scripts inside the running compose environment

[sharp marsh]

is all that "run X on 10 workers" and "Z on 3 and if fails try a different worker" logic encoded in buildkite?

[marble tendon]

scared me before that edit 😆 yes, we're reliant on their YAML syntax

the ones in yellow ^ are marked as depending on one of the two green ones. The ones still waiting are marked as depending on yellow ones

then we've got some logic around exit codes and retries https://buildkite.com/docs/pipelines/command-step#retry-attributes

we can't actually dynamically re-target agents ATM

[sharp marsh]

ok, gotcha. So Dagger can definitely help here but it won't replace all your current stack.

The idea is that you can port a big chunk of those pipelines to code so you can run them in your local machine and in buildkite, but all the buildkite specific logic to handle retries and set some parallelization will not likely go away. Since Dagger currently doesn't manage build infrastructure. It gives you the core engine so you can encode all your pipeline steps in code and run them in containers very easily

[marble tendon]

Please don't take this the wrong way, but that sounds like it lets me write my existing shell scripts in go or typescript 😆

like, my steps are:

- name: build-prod
  command: docker/build-prod
  (lots of yaml around retries and parellelization and conditionals)
- name: build-test
  command: docker/build-test
  (lots of yaml around retries and parallelization and conditionals)

it sounds like I get to s/docker\/build-test/go run build-test.go/

[winter linden]

@marble tendon what happens in practice, is that you end up encoding much of your DAG in Dagger, and you usually get major gains in caching in concurrency, out of the box. So even without the usual CI knobs, and even without parallelism, you cut a lot of waste

It's not apples-to-apples comparable to your current CI configuration, for the reasons @sharp marsh explained, But it's definitely not comparable to your current shell scripts either

(retry logic is a separate topic. I would be interested to learn more about your particular use of retries, but my guess is that it can be advantageously replaced by code.)

[marble tendon]

well, for instance, we use spot instances for our AWS stack - they can be preempted. So one reason we use shell scripts is to normalize the exit codes. If the worker dies for extraneous reasons, we'll retry a few times, it's not the code's fault. But if the failure happens inside the shell script, no retry so no waste

our build steps come first because we're pushing up cache to ECR - subsequent steps don't rebuild, and if there's a local cache there's a local cache 🤷‍♂️

[sharp marsh]

you can also make Dagger pipelines exit with the same code your scripts are currently exiting. As Solomon said, the advantage is that your pipeline DAG is encoded in-code in Dagger instead of being a very difficult to maintain YAML platform specific code. This allows us to perform some optimizations but also allows your teams to re-use pipeline logic more efficiently since you can start encoding the common pieces in libraries you can test and re-use instead of sourcing or calling scripts here and there

[marble tendon]

What part of the dag, in my scenario, would get encoded in Dagger?

[winter linden]

Hard to tell for sure without an example of your CI config... Also let me start a thread so we don't drown this channel too much 🙂

[marble tendon]

sure

[winter linden]

Dagger vs traditional CI

[wise prism]

Hello guys 🙂
Early adopter of dagger, I'm trying to use it to introduce in our dotnet team, and I'm trying to build a first sample using the go sdk.

I've couple of questions.

How do I disable the cache for the local testing? I'm running withexec ls to test and it's obv always cached and it's a little annoying.
How do I make a graph? Tasks that are dependent on tasks? are there any examples there?

Thanks ❤️

[wise prism]

Just noticed a help chat, I'll move there

[swift barn]

Hey! Our team is starting to refactor our CI-pipelines that have a lot of business logic in them. Our goal is to have proper unit and integration tests for our pipelines. Naturally Dagger is one our first option for this. Do you have any resources related to writing tests for pipelines? We are planning to use Go SDK as our service is also written in Go.

Another question is that how can I access Gitlab CI-variables in our code or are those just normal env-variables? Also is there any "Dagger-native" way to trigger a pipeline in another repo or should we just use Gitlab's API for this?

We are doing little PoC with Dagger this week so probably more questions will arise.

[past ivy]

Integration tests are a bit tricky as Dagger does not currently support running containers or networking with running containers (though that is currently high-prio work, see https://github.com/dagger/dagger/issues/4080).

What do your tests look like?

As for Gitlab CI variables, yes, they're just regular env-variables, so you can do things like this (our pipelines are TS but this should map fairly well)

client.container()
  .from('golang')
  .withMountedDirectory('/app', client.host().directory('.'))
  .withEnvVariable('SOME_VAR', process.env.SOME_VAR)
  .withExec(['go', 'test', './...'])

[swift barn]

Thanks! Currently we don't have any tests for our pipelines and that has been a pain point for us.

[past ivy]

oh wait I just re-read your question, so you want to write tests for the pipelines themselves? I suppose that'd be doable. Personally I'd probably write the business logic in normal go, then just test that module. Is that what you're looking to do? Or are you looking to do an integration test where your test runs a dagger pipeline and confirms some output?

[marble tendon]

I'd second ^... if you have logic you can't move from the pipeline into a go module (like logic about steps running if others fail where these steps MUST run in different VMs) dagger won't actually let you test this either

[sharp marsh]

I guess what you generally want to test in a pipeline is that certain things are actually executed in specific order. The good thing about having the DAG encoded in code is that you can generate marks in the DAG of some steps being executed and you can then assert on top of that. i.e:

    alpine := c.Container().From("alpine").
        WithExec([]string{"sh", "-c", "echo \"tests ran\" > /tests_ran"})

    t, err := alpine.Directory("/").File("/tests_ran").Contents(ctx)
    if err != nil {
        panic(err)
    }

By encoding conventions like these in your CI/CD pipelines, you can start enforcing some tests to verify that your pipelines always execute certain steps that you don't want to miss. It'd be awesome though to have a way to assert some things "before" the pipeline gets executed. I don't think we currently have a way to "inspect" the status of the dag with code currently. Is that correct @cloud canyon @pseudo stream ?

[sharp marsh]

I'm thinking that by using the Pipeline API it'd be nice to be able to do things like:

// Define pipeline
p := c.Pipeline("my-app")
p = app.Pipeline("lint").From("img")....
p = app.Pipeline("generate").From("img")....
p = app.Pipeline("test").From("img")......

// Test pipeline
assert.Exist("lint", p)
assert.Exist("generate", p)
assert.Exist("test", p)
assert.Order(["lint", "generate", "test"], p)

// Execute the pipeline
p.ExitCode(ctx)

[swift barn]

Thanks for the response. Yes, we are wanting to test the actual pipelines 🙂 We have same pipelines shared by multiple repositories that have customer configs in them. We need to compile some DSL, install correct dependencies etc. Btw I can't see any references to Pipeline API in the docs of Go SDK.

[sharp marsh]

code is merged in main but SDK hasn't been released yet (https://github.com/dagger/dagger/pull/4248)

[swift barn]

So ideally we'd like to define some different scenarios (ie. tag is released, commit to main, normal commit etc.) and define intended outputs and make sure certain jobs ran in specific order.

[swift barn]

How long it usually takes for the SDK to be updated after merge to main? 🙂

[sharp marsh]

not much. We should have a release early this week 🙏

[real kite]

Curious what patterns people are using when they want to farm out parallelization with dagger on multiple CI machines.

For instance, in Github Actions YML you would do this by declaring multiple "jobs" in a single "workflow" which would run on seperate machines. Are people using remote buildkit for this? Or calling some api from dagger that arbitrarily spawns up more machines? Or just using a single super beefy machine?

[swift barn]

Hey, so far Dagger is looking good. Next step is to make it run on Gitlab CI. I found this document (https://docs.dagger.io/1201/ci-environment/) but it says "This documentation is for an older version of Dagger" but from the newest docs I can't find any hints how to make this run on Gitlab. Any hints?

[slender star]

Welcome @swift barn 👋

Have you decided on an SDK? Go, Python, Node.js (Javascript or Typescript)?

[swift barn]

Yes, we are using Go

[slender star]

In the Guides for Go (and for the other SDKS) in the docs, there is a guide for using Dagger in CI.
https://docs.dagger.io/sdk/go/275922/guides
https://docs.dagger.io/768421/go-ci

Hope that helps! 🙂

[swift barn]

Ah, how could I miss that 🙈 Thanks!

[slender star]

NP! We're working on making that better. Stay tuned! cc @heavy karma

[fallen moat]

Thanks for your answer 🙏 Yes for the second one. A gitlab ci configuration with multiple steps (e.g. "build", "test", "deploy") and I want certain steps to be triggered manually. Sorry for my late answer, I rarely go on Discord and missed the answer

[mystic mantle]

GitLab

[scarlet anvil]

how long normally dagger.Connect complete? it seems stuck, and where is it connect to?

oh nevermind it's already running in 2s now '__') previously i was waiting 30s and nothing happened

[winter linden]

@scarlet anvil that is highly unusual, normally opening a dagger session should be nearly instant, definitely less than 1s

[scarlet anvil]

so it was stuck downloading docker image

3rd world country problem

and nothing shown on terminal so i dont know if its doing something at all

[winter linden]

Sorry about that... Normally it should show a message when it starts downloading the engine for auto-install. You're right that when auto-install is triggered, end-to-end it will take more time.

[loud briar]

Quick one, how do I add run arguments? E.g. docker run --cap-add=<> <the rest>. I've skipped through the few guides but none of them use any run arguments?

with_default_args did the job

[slender star]

Awesome! Out of curiosity, which capability or capabilities did you need? What for? 🙂

[loud briar]

Hashicorp Vault, a tool I really rather not use at all but have to

[slender star]

ah --cap-add=IPC_LOCK

[loud briar]

Indeed

Now onto extracting vault secrets from that container and placing them in a second container. Once I figure this out I'm onto something that I think will go down well internally

[slender star]

Wondering did you try without the cap-add and get errors, or just figured you'd add it proactively?

Here's another example of using hashicorp/vault:latest to get secrets out. It's in go, but should be super easy to translate to the Python SDK.
https://github.com/kpenfound/hello-nomad/blob/main/ci/utility/secrets.go#L26-L43

[loud briar]

I tried without and it does return secrets but with a warning

Adding the run argument does remove that warning

That is a very useful example indeed, thanks for linking that

[chilly arch]

FOSDEM - Dagger Community Participation - https://fosdem.org/2023/

[timber sphinx]

Went through tutorial and first time I ran a fully successful hugo build with dagger. Very nice work on the tutorial for making it easy to get up and running!

[winter linden]

FYI, we opened an issue to track this: https://github.com/dagger/dagger/issues/4456

[vestal creek]

Lost internet connection on computer 😞

[winter linden]

FYI since there was much interest in the "project entrypoint" issue, I invested some effort to improve and expand the introduction: https://github.com/dagger/dagger/issues/4414

[chilly arch]

Thank you for trying it out! cc @heavy karma

Happy Thursday, everyone! If you missed the community call today, you can find the recording here: https://dagger-io.zoom.us/rec/play/kuiW8ZQUfvWXikjFbhwG3jIZi_BRUostX9HyNxun7VLSUiJEde7yMgOy92m0YtCDN4at2y7JSLnjeXnj.Wfb2MmQ-eiwCfNUi?continueMode=true

[timber sphinx]

finally got round to it. Hopefully it's somewhat useful. That said, quick clarification.

I searched for a bit in github discussions as it seemed an open ended discussion. It was an issue, so missed it initially. Do you have anything written down on what goes to discussions vs issues so I know where best to post/look?

[winter linden]

@timber sphinx we're pretty flexible (you can convert one to the other later). General rule of thumb is: if it can possibly be closed in the future (solving a bug; solving a feature; making a decision) then it's an issue. If not, maybe a discussion

[winter linden]

In the case of entrypoint: I firmly intend for us to ship something, sooner rather than later, so I made it a design issue to be closed when we have a decision on a spec

Speaking of which @timber sphinx : I have not yet written it out in the issue (hope to, tomorrow) but we have 3 rough candidates for a design. One of them is @cloud canyon ‘s favorite and happens to be what we call “mage style” where we roughly copy the interface defined by mage, and transpose it to the nearest equivalent in other languages. Since you’re a mage fan I thought I’d let you know 🙂

[timber sphinx]

YES. Mention me in the issue when you create it. I'm intrigued.
Mage is on nearly feature freeze due to obligations of the original author and there are some definite things I'd change on behavior. Knowing there's a possible improvement to integrate mage, discover the tasks, and write Go code is exciting. 🌮🌮🌮🌮

[winter linden]

In practice it might simply mean: your existing mage files can double as your dagger entrypoint, with no or minimal adaptation. And ideally also continue to work with mage in parallel (that may be wishful thinking , tbd)

One follow-up question @timber sphinx : are you aware of mage-equivalent tools in other languages, and which did you enjoy using, if any. Any tool that exposes native code as tasks, as opposed to (probably) shell commands

[timber sphinx]

I've explored a range of frameworks but not primarily in native general purpose languages. For things I've tried InvokeBuild is pretty great (powershell cross platform). Taskfile is also nice but very restrictive beyond basic cli commands. There are others I looked at like Variant 2 that use hcl2 but at the end of the day a general purpose languages like powershell or Go ended up being the best. If you want more let me know and I can write up a blog post when I get a chance.

[swift barn]

can dagger detect dependency in git sub module way? like if we drive monorepo with git submodule then dagger will be able to detect dependencies easily?

[sharp marsh]

not automatically since Dagger is unaware of git submodules. Having said that, it's not difficult to create a Dagger pipeline that can run some logic around git submodules and buid your apps according to some custom logic. Happy to help out from the Dagger side if you're willing to give it a try.

By any chance, do you currently know of a CI tool that's git submodule aware?

[chilly arch]

tutorial clarification

[fresh depot]

👀
https://twitter.com/AupseLaurent/status/1618988507468550144

Laurent Aupse 🐘 (@AupseLaurent)

Je vous souhaite de passer une bonne fin de semaine 🤗
Parce que la semaine prochaine on décolle pour une prochaine vidéo sur https://t.co/wThKs6c7qH 🚀

▶ Play video

Twitter · Jan 27, 2023, 3:05 PM

[main spear]

Heyo, just wanted to share that I am making good progress on a rust sdk (https://github.com/kjuulh/dagger-rs). I am doing this totally for fun and for my own use.

I currently only have dagger cli download done, as well as booting it up and getting introspection results.

I am currently doing the codegen parts.

Anyways, just wanted to share it, if it is of any interest 😄

[winter linden]

That’s amazing! I think you will find plenty of people interested, and if we can help you in any way, let us know. The codegen tooling still has sone rough edges as you may have noticed 😁

By the way: https://github.com/dagger/dagger/issues/4447

@main spear are you re-using the existing codegen tooling from other SDKs, or developing your own in rust? Any questions or blockers so far? cc @cloud canyon

[main spear]

Hermansen 4325 are you re using the

[loud briar]

A Rust SDK is going to be good for Dagger - Rust is growing remarkably quickly and for good reason

[main spear]

Made some good progress today. I've added all types, args, descriptions etc. I now basically need to hook up the types to the dagger core, and I can get the mvp done 😄

https://github.com/kjuulh/dagger-rs/blob/main/crates/dagger-sdk/src/gen.rs

[stuck wyvern]

Just me?

EOF: Error: resolving image digest: GET https://ghcr.io/v2/: UNKNOWN: unknown error

[swift inlet]

What command are you running ?

[stuck wyvern]

Probably dagger.Connect?

Looks like probably a ghcr issue:

docker pull ghcr.io/dagger/engine:v0.3.10
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

[swift inlet]

Getting the same error, was actually testing this very command on the side. Sounds weird: https://www.githubstatus.com/, let's see in a few minutes 😇

[stuck wyvern]

status page says pages is down now.

[pseudo stream]

Just started getting the same thing. We should really make our code more resilient to this, e.g. if you get an error like this just try to use whatever engine exists locally if anything. Opened an issue here: https://github.com/dagger/dagger/issues/4484

In the meantime if you already have a dagger engine running locally from previous runs you can temporarily tell dagger to use that by running with an env var like _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://<container name>

So for example if you have upgraded to the latest engine then that would be _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine-8d8bdc86d448fd7b

[stuck wyvern]

And now github seems to be aware.

[elfin frigate]

it's upgraded to "Major Outage" (red color) now 😮

[swift inlet]

Seems to be back 😇 (works for me)

[wispy tapir]

Still red on the website

[elfin frigate]

green check now

[fresh depot]

My dagger youtube video is out !
It's french but if you are not allergic to it there is english subtitles
https://www.youtube.com/watch?v=m5mauLiCp3Y

[winter linden]

That video is so good! And hilarious 😂

[fresh depot]

I'm glad my sense of humour works on other 😂

[swift inlet]

Yes it's awesome ! 😍

[winter linden]

I liked your video so much @fresh depot that I engaged in some light hearted trolling to get it more views 😇 https://twitter.com/solomonstre/status/1620442514644955136

Solomon Hykes / @[e-mail] (@solomonstre)

It’s amazing to me that, in the end, Docker will probably outlive Kubernetes, by a lot.

Dockerfiles are low-tech and weird… But people will continue writing them long after the last kub node is retired, because they solve a more important problem in a harder to replace way.

Twitter · Jan 31, 2023, 3:22 PM

[fresh depot]

I saw that, was a little confused by the tweet but that explains it haha. Thx a lot !

[timber sphinx]

Does tagging and calculating semver of the repo fall outside normal Dagger code because it’s impacting the environment or does it fit more in y’all’s mind with the outside calling tasks?

[frozen spindle]

Where's the right place to get help understanding what needs to be done to bring the cue sdk up to date so that can follow the other sdks?

[timber sphinx]

Another question. If I’m using mage and Go SDK but sharing with some others for Dagger pipeline like angular build etc that are NOT Go devs, would you compile and just commit the compiled binary (yuck but ok) or do it another way. They primarily use dotnet so asking them to install Go is possible but not ideal. Any other ideas?

[winter linden]

I think once we standardize an interface for project entrypoint (https://github.com/dagger/dagger/issues/4414), they will have the option to run your pipelines via the dagger CLI (possibly dagger do like in 0.2 🙂

Later, we will introduce cross-language extensions and that will allow them not just to invoke your pipelines, but compose them into their own pipelines in their own language

[chilly arch]

Hi all! The team has been working hard on a generalized quickstart within our documentation - if you have some time, we'd love the community to try it out and give us feedback.

If you haven't had a chance to try Dagger yet, this quick tart might be the perfect start for you! You can find the latest version here: https://devel.docs.dagger.io/648215/quickstart

[chilly arch]

Dagger Quick Start - Looking for feedback

[rocky grove]

Hi all - I am working though an issue with a WithExec right now. When I run docker run container MY_COMMAND i get the output of the command and it exits, but when in dagger, i get the output of the command and it hangs (doesnt move on the the next go instruction). This particular container does have a entrypoint.sh entrypoint thats running the command via exec "$@".....and guidance on how to not have the WithExec hang?

[lusty void]

Hi all! Just curious with the Connect api supported remote dagger engine in the future, does it mean that we will need to run dedicated dagger engine in order to utilize buildkit caching? Will it be possible that dagger engine will pull docker layer from some sources like S3 to avoid the need of dedicated server?

[past ivy]

I miss our dagger-based project already (working on something older today) 😢

[fresh depot]

Don't smash your keyboard 😏

[timber sphinx]

Just did a demo for my org and it included dagger angular and nginx builds. (Done in one day... Yes I like to lived on the edge 😆) Didn't get to highlight Dagger specifically yet but starting to see a great example of where it fits by relying less on yaml templating (azure pipelines had incredible templating features). It's great when you write templates but consuming what others built is much harder. Really interested in how extensions will work vs mage packages etc. Will keep doing my best to experiment 💪

[fallen temple]

@heavy karma, I ran into an error when I tried to follow one step of the Google Cloud example: https://docs.dagger.io/sdk/nodejs/620941/github-google-cloud#appendix-a-create-a-github-repository-with-an-example-express-application. I had to add the -e flag before the expression so that sed would take the command. 🙂 I thought about filing an issue, but it's so small that I thought a message here might be easier. Updated command: sed -i -e 's/Express/Dagger/g' routes/index.js

[heavy karma]

Thanks for letting me know, I'll have a look today and fix it

[short crypt]

Not sure if I should go straight to creating a post in help, but having an issue going through the Python SDK tutorial. I'm at Step #3, "Test against a single Python version" and when I run python test.py, I get the following error which unfortunately doesn't give me a lot of info to go on:

#2 ERROR: process "pytest tests" did not complete successfully: exit code: 1
------
 > :
------
Traceback (most recent call last):
  File "/Users/ln/dev/python/dagger-project/tests/test.py", line 33, in <module>
    anyio.run(test)
  File "/Users/ln/dev/python/dagger-project/.venv/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
...ELIDED...
  File "/Users/ln/dev/python/dagger-project/.venv/lib/python3.10/site-packages/dagger/api/gen.py", line 222, in exit_code
    return await _ctx.execute(Optional[int])
  File "/Users/ln/dev/python/dagger-project/.venv/lib/python3.10/site-packages/dagger/api/base.py", line 87, in execute
    result = await self.session.execute(query, get_execution_result=True)
  File "/Users/ln/dev/python/dagger-project/.venv/lib/python3.10/site-packages/gql/client.py", line 1231, in execute
    raise TransportQueryError(
gql.transport.exceptions.TransportQueryError: {'message': 'process "pytest tests" did not complete successfully: exit code: 1\nStdout:\n\nStderr:\n', 'locations': [{'line': 11, 'column': 15}], 'path': ['container', 'from', 'withMountedDirectory', 'withWorkdir', 'withExec', 'withExec', 'exitCode']}

Any ideas what I might be doing wrong here?

One thing maybe worth noting is that my project is structured like this:

├── README.md
├── dagger_project
│   ├── __init__.py
│   └── app.py
├── poetry.lock
├── pyproject.toml
└── tests
    ├── __init__.py
    └── test.py

which is a bit different than the example in the docs because test.py is not in the project root, but in tests/test.py, but not sure if this is an issue...

[heavy karma]

@rain oriole can you help with above?

[indigo pawn]

does I understand it right that tests/test.py is the file containing the dagger code?

[indigo pawn]

Okay... I have played around with my setup. Might be a dumb question from me but are you sure that pytest is installed in the container (I mean is it inside the pyproject.toml in the right section)?

[mystic mantle]

I see you master Twitter engagement techniques

Unrelated, what's with the new Google Artifact Registry that is supposed to replace GCR.
Has any of you some experience with it? Is it fully compatible? Just use it instead of gcr?

[rain oriole]

One thing maybe worth noting is that my

[fresh depot]

I tried, the docs wasn't clear enough so i put it aside 😅

[real kite]

I've been using GAR - it's basically a superset of GCR functionality in that you can also push stuff like python and java packages to it. It's pretty good, just have to get permissions set up. Haven't really had much issue with it

It's intended to fully replace GCR is my understanding

[short crypt]

Correct

[short crypt]

No, I'm not sure 🙂 I didn't see that in the docs. If it's there, I missed it.

It's definitely not in my pyproject.toml. Lemme try again after adding it.

@indigo pawn added pytest to pyproject.toml and still had the same issue. Here's my pyproject.toml:

[tool.poetry]
name = "dagger-project"
version = "0.1.0"
description = ""
readme = "README.md"
packages = [{include = "dagger_project"}]

[tool.poetry.dependencies]
python = "^3.10"
pytest = "^7.2.1"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Anything look wrong there?

[rigid epoch]

It's been a drop-in replacement for me, so far. It adds the ability to do image streaming in GKE clusters, which is really, really cool -- basically lets you start containers before the image is entirely downloaded, with the filesystem being incrementally loaded in the background (attempts to access files that aren't loaded yet will just block). GAR also gives you much better permissions options than GCR: in GCR you needed to grant users read/write access to a generated Cloud Storage bucket that contained all your images, and there was no way to break permissions down any further than that. With GAR there's no bucket to manage access to, and you can create (and permission) multiple separate repositories within a project.

[heavy karma]

https://github.com/dagger/dagger/pull/4525

[mystic mantle]

The image streaming sounds really cool! And more fine grained permission as well.
Nice! Thanks for the heads up on GAR everybody!

[tender goblet]

Next Monday, I'm doing a talk on CfgMgmtCamp about the useless discussion between imperative & declarative in tools. (https://cfp.cfgmgmtcamp.org/2023/talk/HTDHAK/) Because of its DAG, I'm listing Dagger as a tool which integrates both aspects for best-of-breed integration.
To the team: request for permission to use the graphic from the quickstarts page (https://docs.dagger.io/319191/quickstart-basics) in my slides?

[fossil pine]

There is a better version of that on the website (in case you like the Dagger UI: https://dagger.io/img/197578121-ab95cb56-6dae-44c3-ad51-50b17ab3ff96.png).

[empty jay]

Error: Post "http://dagger/query": EOF while executing dagger pipeline with golang 1.20

[sharp marsh]

sure, that's ok if you use that image

[timber sphinx]

Fyi my coworker in Ukraine ran into this and was bummed. 😬. Would love to see something on this soon if possible. Also what's your pitch to someone who says why don't I wrap up docker commands in Go myself? Would be a cool faq as he's open minded to it but not immediately groking the value. Could be a good section to add to the FAQ. https://github.com/dagger/dagger/issues/3994

[sharp marsh]

re issue: we're fixing that ASAP. Solomon commented a few minutes ago that should be fixed
re docker wrap: communication needs refinement, definitely. Having said that, there's always going to be someone that says: "OH, I can do that with two lines of bash, why do I need this?". ref: https://github.com/p8952/bocker

[winter linden]

Right. I think your friend should go ahead and write that docker wrapper @timber sphinx . When they need help with it, we will be happy to support them 🙂

Some things you just need to experience yourself

[frozen basin]

Hey yall - trying to dig through the discord here - wondering about debugging containers in intermediate steps. Is this possible?

I think it had been discussed a while back. I've been pinned on an older version of dagger for a little bit, curious about developments since then

edit: Just found https://github.com/dagger/dagger/issues/4463 - I'll follow for updates there.

[chilly arch]

That is great to hear! And good luck at your talk. Once you have the recording, please let me know and I'll be happy to help promote it. Yes, as Marcos mentioned, you can use that graphic

[fossil pine]

Huh, all FOSDEM talks delivered. Lot's of interesting questions from the audience.

I completely botched the first demo. It didn't work at all (network was poor).

Second demo: aced it.

Thanks for the support @mystic mantle It was great to finally meet you.

Queue before my presentation in the CI/CD devroom.

[winter linden]

Nicely done Márk! Thanks for sharing the Dagger love

[olive tree]

I was there for the second talk, was great! I always find it hard to convince people to do cicd as code instead of yamls but now I have some more ammo 🤣

[winter linden]

Did you get any questions @fossil pine ?

[fossil pine]

I did. I'll write a summary or blog post or something.

[rigid epoch]

philosophical question that's on my mind as I think about introducing Dagger to my organization: who is the Dagger SDK "for?" I think different answers to this have been floated and I'm curious how the team thinks about it...

[fossil pine]

Dagger questions and feedback from FOSDEM

[covert jungle]

hello, can someone point me to the elevator pitch of dagger? Im kind of lost

[sharp marsh]

hey @covert jungle! as the institutional website (https://dagger.io/) mentions, Dagger is a programmable CI/CD engine that runs your pipelines in containers. You can develop pipelines in your favorite language (currently Node, Go and Python, but more to come) and run them anywhere.

[pine sage]

We're trying to embed the dagger cuelang SDK in our tool (to run dagger plans that users write in CUE inside other CUE structures), but it seems the "go.dagger.io/dagger" package is not available anymore.

Can anyone help?

https://github.com/devopzilla/guku-devx

[covert jungle]

Thanks, I've gone through the page(and related hackernews discussions) but I still have issues to grasp on how this brings me.

[slender star]

Feel free to ask more questions 😁

You might find this fun video from a community member helpful too. The beginning decribes some of the "why" of Dagger, the middle shows how it works at the GraphQL API layer, and the ending (about 10:00 mark) shows how you use an SDK in Go, Node.js, or Python to write pipelines as code (that call the API).

https://www.youtube.com/watch?v=m5mauLiCp3Y

I'd love to know if you identify with any of the pain shown at the beginning of the video.

[covert jungle]

thanks, i am getting a feeling that this is more a containized makefile system than a ci/cd system itself

[sharp marsh]

Yes, we've had those confusions in the past. Thing is that some users are used to viewing CI/CD as the "traditional" solutions out there like Github Actions, Gitlab CI, Circle, etc where they basically provide a service where they give users some compute, and an imperative DSL to define some sort of DAG that they will run for you.

We're taking a different approach here where dagger gives users the engine and the SDK's so they own those DAGs/pipelines themselves and they can run them wherever they prefer. Locally, in a machine in the cloud, or any of the existing CI/CD runners.

So in a sense, it's a CI/CD solution. The main difference is that we don't host the compute (at least yet) as we're aware that it's a space where there are lots of alternatives which gives us the opportunity to focus on the other aspects where we see users in pain currently.

[covert jungle]

thanks for the clarification. it's pretty hard to really know what's that about

I supopse this is like pulumi or AWS CDK to IaCs : why invent a DSL which is close to tuning complete when you can just use a real language

[round lark]

Hello - Dagger n00b here; I have a naive question... is the Dagger Engine always intended to be executed on the local (host) machine or could there be a situation where the Dagger Engine was running in the cloud and client SDK pipelines were run "locally" but agains the remote engine?

[winter linden]

@covert jungle yes that’s part of it. You mention “containerized makefile system” vs “ci/cd system”: we believe those should be the same. Your pipeline ogic should be written in code that you can run anywhere. That goes for build, test, deploy, lint, etc. Anything that can be represented as a DAG. Traditional CI runners should just be “dumb” infrastructure.

[winter linden]

You can run engines locally against remote engines, although the interface to configure that is still experimental. It is not a very common scenario but it is certainly possible. Do you have a specific use case in mind?

[covert jungle]

thanks for the clarification

[round lark]

Not entirely specific, but we have some regulatory requirements that tend to end up forcing us to "centralize" things. It's a pain, because (for example) right now our CI platform is a bunch of self-hosted Gitlab runners (one per cloud account). If I swap those out for Dagger engines, then I imagine our engineering teams get slightly faster feedback without me having to go through the "please can we whitelist access to this new thing" and "please can we let all the engineers install this other thing". I'm imaging a scenario where a central repository of pipelines exists that each team can pull and use in their own projects; that allows the engineers to essentially execute a "local" pipeline, but it runs against a cloud Engine. An Engine that we can maintain, audit, secure etc. Kind of a half-way house between self-service and "we own all the things". Am I making any sense ?

[sharp marsh]

It kind of does to me. Specially since there are some environments where installing docker locally is not possible. @winter linden this would be similar the the satellites use-case I'd assume with the difference that companies decide to maintain their engines instead of a third-party provider

[swift barn]

In my customers organisation the DinD setup is (almost) forbidden on Gitlab runners so we are looking for ways to run Dagger without that. We are currently rolling out own runners for our team but maintaining them isn's something we are too keen about but we like Dagger that much that we are biting the bullet 🙂 Hopefully soon there are other options.

[winter linden]

Custom Gitlab runners

[stuck wyvern]

When I set ExperimentalPrivilegedNesting the inner dagger (just a direct exec of a go bin) is trying to connect to docker. Is this always needed?

[covert jungle]

I dunno, the syntax felt a bit verbose than say, Dockerfile

take example of the intro page

 package main

  import (
      "context"
      "os"
      "fmt"
      "dagger.io/dagger"
  )

  func main() {
      ctx := context.Background()

      // initialize Dagger client
      client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
      if err != nil {
          panic(err)
      }
      defer client.Close()

      hostSourceDir := client.Host().Directory(".", dagger.HostDirectoryOpts{
          Exclude: []string{"node_modules/", "ci/"},
      })

      source := client.Container().
          From("node:16").
          WithMountedDirectory("/src", hostSourceDir)

      runner := source.WithWorkdir("/src").     
          WithExec([]string{"npm", "install"})

      out, err := runner.WithExec([]string{"npm", "test", "--", "--watchAll=false"}).
          Stderr(ctx)
      if err != nil {
          panic(err)
            }
      fmt.Println(out)
}

that's almost 40 lines for something like 5 lines in Dockerfile

Don't get me wrong , I like the idea, but I think the DX is a bit more verbose than what a scrub engineer like me wanted

[sharp marsh]

I guess that if you're seeing Dagger mostly as a "single" Dockerfile replacement, you're somehow technically correct. The power comes from all the gains that you get by generalizing this to a programming language and en engine which you can't with Dockerfiles like:

• Unit test your code
• Refactor logic into functions or libraries that you can share and re-use
• Re-use your language tooling like LSPs, linters, formatters, etc to enforce standards
• Mix imperative and declarative code in a single pipeline execution

I'm probably missing some other things, and you also get the addition of better visibility and automagical caching.

It ultimately depends on what you're trying to accomplish. If you just have a single app and you're super small team, sure Dockerfiles and some simple CI YAML's should be more than enough. If your project or shop starts to grow, you'll soon realize that you'll eventually find yourself with a spaghetti madness of bash, yaml and Dockerfiles glued altogether. That's where Dagger shines and the main problem we're tackling

I can also give you a very simple example which is also a bit annoying to do just with Dockerfiles.

Try building a very simple app to support multi-architectures like arm64 and amd64 and setup a CI / CD pipeline to deploy it to google cloud run. That single task only will require a considerable amount of Dockerfiles + Bash + YAML were in Dagger you can use your language of choice to handle it.

IDK how you fell, but I personally like it much better to be code rather than anything else.

[covert jungle]

of course, maybe it's just me but comapred to pulumi api sometimes i felt lost
https://github.com/pulumi/examples/blob/master/aws-apigateway-go-routes/main.go

thanks for the detailed response tho

[sharp marsh]

that makes sense, it takes time to get used to the API since it's not something that most people (specially Go devs) are used to. Having said that, if you ever experimented with reactive programming, that would be a bit more similar to the Dagger API

[olive tree]

Hey just wondering if you guys will be at KubeCon and if there are any talks in Sched you could link? 🤞

[cunning jolt]

@chilly arch could you please enable the Q/A feature of webinars for the dagger community call? Chat is blocked within my company 😦

[chilly arch]

Hi! I am sorry about that blocker. Yes, I can enable it for the next one!

[chilly arch]

During the community call (right now), we've heard that some people are experiencing slow Go builds in Dagger - If you are experiencing this, please add some comments here - https://github.com/dagger/dagger/issues/4567 cc @cunning jolt @blazing sluice

[frozen basin]

yo this is 🔥. Seems like the same thing could be applied to GH actions too

https://github.com/kpenfound/encircle/blob/main/README.md

[winter linden]

That's the idea 😇

[frozen basin]

brilliant. I have a buddy who was just complaining about lack of circleci local workflow the other day, I sent this his way

[winter linden]

The beauty of it, is that it runs on the same dagger engine as all your other pipelines. So the cache is shared; telemetry is shared; future integrations with your logging, monitoring, access control is shared

[frozen basin]

I don't use circle so couldn't test myself, but it's incredible that it could be implemented so easily. Like that codebase isn't huge.

Really cool stuff fellas 👍

[winter linden]

Yeah, there's a lot you can build on the Dagger API trivially, we haven't dont a great job at showing that - just covering the basics and explaining what the hell it is, has kept us busy 🙂 Showcasing cool projects like Encircle is a perfect way to demonstrate the potential.

[warm temple]

Thanks ❤️ Tell your friend to open issues if they run into any blockers! There are probably a lot, but at least we can keep track of them 🙂

[winter linden]

@frozen basin do you remember the "fake docker engine" proof of concept? That was a fun one too 😄

[frozen basin]

not sure I saw this hah

[winter linden]

A fake docker engine, backed by dagger under the hood. You type docker run, it works, but there's no docker engine: it's all translated to dagger pipelines on the fly

[swift inlet]

Do you still have this code ? 😇 🙏 Were you wrapping inside a build, or transpiling ?

[frozen basin]

interesting

[winter linden]

yes I have it somewhere, will open the repo. Very broken and experimental... It's just a daemon, that speaks the Docker API, but with all the actual docker implementation removed, and replaced with dagger client calls. Uses the Dagger Go SDK.

Obviously with services support, it would be able to implement a larger chunk of the API 🙂

[swift inlet]

Yes please 🙏 Very very interested 👼

[chilly arch]

Thank you to @heavy gazelle @blazing sluice and @warm temple for the demos at the community call today. If you missed the call, you can watch the recording here -https://dagger-io.zoom.us/rec/share/sSFUOJOT1qY7b8ihKt51LKeG9rh--PJHADa7uZXKvnbuACmVl37EHkegK9VXrjoa.XBfAm67P0t1m2-fz

[wheat agate]

Has anyone successfully used dagger with a rootless buildkit setup? The tutorials went fine, but building something more complicated caused to hang indefinitely on file export. Running builtkitd as root worked as expected

[sharp marsh]

@winter linden was this to test the buildpacks workflow that used the Docker API? I was coincidentally taking a look at this yesterday for a different thing and would be nice to resume that work if it's still interesting to pursue

[cunning jolt]

Thank you for following up! I have add details about my observations 🙂

[winter linden]

Yes that was the original motivation for trying. And yes still interesting to pursue - I think in general building more interesting tools and in particular compat layers on top of Dagger (like Encircle, or docker compat, or buildpack compat) is very interesting and sorely needed!

[clever mesa]

May qemu issue?

Which platform will be used default, when Container() without any opts?

[swift inlet]

The one on which the engine runs

[sharp marsh]

👍 if you have the repo around I'd like to peek

[stuck wyvern]

Is there some way to export the cache ala buildx's --cache-to and --cache-from from the go SDK?

[sharp marsh]

it's been added very recently in the engine https://github.com/dagger/dagger/pull/4543. Not officially released yet

[potent plover]

Is there a good example of a more complex Python or GraphQL pipeline somewhere? I wanted to see what the structure would look like doing multiple withExec in a container, accessing multiple leaf nodes?
Bit unclear on the limitations and how some things are intended to be structured. Or is the graph only splittable across multiple queries?

As in, by returning an ID and forking off of that.

[potent plover]

Also hello. You can blame @heavy gazelle for my presence 🙂

[winter linden]

Here’s a guide using the Python SDK if that helps. https://docs.dagger.io/sdk/python/648384/multi-builds

And you are right that not everything can be expressed in a single graphql query. Any operation that requires more than one pipeline as input (for example mount, copy…) will need exactly what you described: “stitching” of pipelines by retrieving and referencing IDs (which are really the pipeline’s “code”). SDKs take care of the ID lookup for you, for a more natural DX. In pure graphql you have to do the stitching yourself - slightly more verbose but it works exactly the same in the end.

One rough spot us that the graphql api playground is not as useful as it could be for more complex queries (you need to copy paste IDs). But we have ideas for fixing that, although they are low priority.

Hope this helps.

And hello, welcome! 🙂

[potent plover]

Thanks, that is helpful 🙂

I'm looking a potential SDK build so good to know what they need to wrangle:)

[winter linden]

Note that all SDKs are generated from the GraphQL schema. We’re happy to show you the ugliness behind the scenes if you’re thinking about contributing a SDK?

[slender star]

Elixir SDK, maybe? 🙂

[potent plover]

I couldn't possibly comment 😄

[potent plover]

from("bla") {
  id
  withExec (["bar]) {
    id
  }
}

This type of thing works in the playground.
From what I've seen of the Python SDK I don't think it can express that.

q.from("bla").with_exec(["bla"]).id

Or is it?

q.from("bla").id().with_exec(["bla"]).id

Might be intentional. Not sure how often those parallel fields come in handy. I guess they fork the DAG and you can't really merge it again right?

Trying to get a sense of intended use before I have a chance to play with it.

[potent plover]

The latter Python hypothetical doesn't make much sense I think..

[loud briar]

Separate them?

container = q.from("bla")
container_id = q.id()
bar = container.with_exec(["bar"])

[sharp marsh]

@potent plover hopefully this sheds some light: https://play.dagger.cloud/playground/724m7z2YbBl

this is the equivalent python code:

"""Execute a command."""

import sys

import anyio

import dagger


async def test():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:

        gitDir = client.git("github.com/dagger/dagger").branch("main").tree()

        build = (
            client.container()
            # pull container
            .from_("alpine")
            # get Python version
            .with_mounted_directory("/app", gitDir)
            .directory("/app")
        )

        # execute
        version = await build.entries()

    print(f"Hello from Dagger and {version}")


if __name__ == "__main__":
    anyio.run(test)

the SDK is using id's internally and it's "stiching" those queries without the user knowing about it

[potent plover]

Alright. Then the SDK works pretty much as I thought 🙂

[potent plover]

That is helpful. Thanks 🙏

[winter linden]

@potent plover you are correct that this particular feature of graphql (querying multiple scalars at multiple locations in the query) is not supported by our SDKs as a single chain. We could support it but it would require giving up something in the DX. For example we could achieve it by using callbacks instead of return values when querying a scalar field. It would make your example query seamless to replicate in Go/Python, but it would also make simpler queries perhaps less approachable: return values are “simpler”. This is of course debatable.

You could also approach it differently with an unofficial SDK since the source of truth is GraphQL

[potent plover]

Alright, that lines up with my understanding and I think it could very well be the right tradeoff.

[timber sphinx]

Can someone point me to the latest PR/discussion/issue on improved logging (even just similar to buildkit collapsed output, or leveled color logs). Digging in github to see where it stands and missing it.

[warm temple]

I think this is the PR you want to follow: https://github.com/dagger/dagger/pull/4522, and the high level issue for discussion here: https://github.com/dagger/dagger/issues/3587

[stuck wyvern]

Started having an issue with the go SDK getting stuck in a round trip with dagger.
Tried looking at traces but it's not even showing any in-flight traces.

Strangely the code was working fine yesterday, not so much today, and on two entirely different machines.

[hushed breach]

Hello, I'm running into a problem that dagger-cue takes lots of memory when more than 20 actions run in parallel. e.g.
execute following action takes more than 1g mem. Is there anything I can do to reduce the memory usage?

dagger.#Plan & {

        actions: test: {
                for k, v in list.Range(0,20,1) {
                    "op_\(k)": bash.#RunSimple & {
                          script: contents: "echo hello \(k)"
                    }
                }
        }

}

[rain oriole]

Hey. There's a known CUE issue regarding high memory usage (https://github.com/cue-lang/cue/issues/1795). You may get more information from there in case there's some optimization you can do in your use case, but as I remember it there was no fix. Not sure where it stands now but that issue is still open.

[rain oriole]

Has anyone come across a pipeline originating multiple API errors (same query)?

[vast gust]

Hello. I just started learning Dagger. I would like to use the Go SDK and I followed the quick start guide (very nice BTW).
If I want to install tools inside the container how can I do that. i.e. I want specific node version or go version or terraform version.

[winter linden]

Welcome! It’s very similar to a Dockerfile but in Go. You can execute commands in the container to install packages, or use a preconfigured image from a registry, or copy binary distributions into the container… Then keep going and execute the tool you just installed: go, terraform etc

[vast gust]

Thank you. I have figure it out.

[vast gust]

is there a way to run the dagger inside the container instead of using the bash/sh commands like: WithExec([]string{"echo", "kokos"}).
What if I wat to have some if/else logic there based on the output?

[warm temple]

You have a few options there 🙂 You can execute an entire bash script inside a container which has that conditional logic, like WithExec([]string{"sh", "-c", "./script.sh"}), or you can handle the conditional in your pipeline code, like:

output, err := foo.WithExec([]string{"echo", "bar"}).Stdout(ctx)
if output == "bar" {
 // do something
}

[vast gust]

It looks like every time that I'm running the WithExec it spins up a new container. Can I instruct the tool use the same session
Here's my sample code:

    output, err := golang.WithExec([]string{"terraform", "workspace", "select", "kokos"}).Stdout(ctx)
    output2 := strings.TrimSpace(output)

    if err.Error() != "exit status 1" {
        output, err := golang.WithExec([]string{"terraform", "workspace", "new", "kokos"}).Stdout(ctx)
        fmt.Println(output)
        if err != nil {
            fmt.Printf("Error: %s", err)
            os.Exit(1)
        }
    } else {
        fmt.Println(output2)
    }

    output, err = golang.WithExec([]string{"terraform", "workspace", "list"}).Stdout(ctx)
    fmt.Println(output)

[winter linden]

You can chain exec operations:

newThenList := golang
  .WithExec([]string{"terraform", "workspace", "new"})
  .WithExec([]string{"terraform", "workspace", "list"})

output, err := newThenList.Stdout(ctx)

But in your case, might be simpler to execute your logic in a shell script

[vast gust]

Thanks. Thanks but here, I want to run command 1 and based on it's output run command 2 or command3

[winter linden]

And you want to run command2 or command3 in the container modified by command1?

[vast gust]

Yes.
I think I need to set the export directory from the container.
It looks like the files that the command is generating they do not end up on the host.

[winter linden]

You don't need to export, you can still use chaining, just by storing the intermediary pipeline state in a variable

In your case, though, there's a separate problem which is that your Dagger pipeline fail if one command fails, so you won't be able to retrieve the exit code and use it in your conditional. This is a bug on our end: https://github.com/dagger/dagger/issues/3192

[vast gust]

Then something is wrong when I run the commands from dagger. It looks like the files are missing.
When I run the same command on my host
Init state:

tree -a .
.
└── main.tf

terraform init

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/local...
- Installing hashicorp/local v2.3.0...
- Installed hashicorp/local v2.3.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
➜  terraform git:(main) ✗ tree -a .
.
├── .terraform
│   └── providers
│       └── registry.terraform.io
│           └── hashicorp
│               └── local
│                   └── 2.3.0
│                       └── darwin_arm64
│                           └── terraform-provider-local_v2.3.0_x5
├── .terraform.lock.hcl
└── main.tf

7 directories, 3 files

The above structure is missing when I run the same command from dagger

[winter linden]

@vast gust do you mind if we continue here: #1075891040403329075 ?

[neon ore]

Hi everyone

[neon ore]

Are you using dagger pipelines to trigger terraform? Thats kind of interesting and was inline with what I was hoping to test out

[wise prism]

if somebody is looking to do stuffs with dotnet, I've just pushed a demo in the demo channel... it supports restore, build, tests, publishing test results, dockerize and patching gitops.

[loud briar]

Before I open a help ticket there might be a simple answer to this one. DOCKER_HOST in Gitlab runner is correctly set as tcp://docker:2376 (and the other vars are set - TLS verify, cert path, region etc). Docker commands in the Gitlab jobs work fine, but docker commands in the python script called by a Gitlab job fails with Error during connect: Get http://docker:2375/_ping: dial TCP: lookup docker on a.b.c.d:53: no such host... Which is correct, it's the wrong port. Anyone seen this before?

[sharp marsh]

Is this related to dagger? Just wondering

[loud briar]

I'm getting this while trying to run a python script that uses Dagger, if that counts. This is the first time I've had to use Dagger with Gitlab + Docker:dind and it has been a horror story so far!

It's specifically failing on the first with_exec that calls a docker command, returning the above error, but the container the script is running in has no issue running docker commands. Something is misconfigured at or after the point I call poetry run python3 build.py where build.py contains the Dagger code

[sharp marsh]

Ok I see now, you're trying to calla docker command within Dagger, I was missing that part.
This happens because your with_exex is unaware of the DOCKER variables your gitlab runner has. You need to use with_env_variable and forward those DOCKER_* vars to your Dagger pipeline

[loud briar]

I've tried that - wasn't successful.

[sharp marsh]

Regarding Gitlab + sind, yeah.. it's tricky since Gitlab for some reason doesn't allow using the docker socket directly.

[sharp marsh]

Mind sharing a snipper about you were doing this?

[loud briar]

To check my work, I added docker host and driver to the pipeline, as well as TLS verify

Do I need to add anything else?

[sharp marsh]

I think you do, you also need to mount the CERTS into your Dagger pipeline

Apologize for the caps, I'm on mobile

[loud briar]

So add the env vars again, mount the certs dir?

[sharp marsh]

Yep, that should be it

Hmm there's something else I believe...

Seems like dagger is not aware about what docker resolves to.. which makes sense.. wondering how we can hook that

[loud briar]

I'll try again with the vars and dirs added

[sharp marsh]

I don't think that will be enough since the error that you posted before shows that dagger can't resolve docker

As a hostname

[loud briar]

I believe that'll be the case. Will find out in a few minutes.

Would like to figure this out, I have Dagger pipelines replacing some gnarly bash/yaml locally, and using a mounted docker socket. Just need to figure out the dind config for teams that don't have control over their runner config files

(Which is most of them)

Also worth knowing to update the recent SDK documentation pages - I expect other Gitlab CI users will hit the same problems

[sharp marsh]

We'll figure it out. Just thinking about the easiest solution

[sharp marsh]

cc @heavy karma

[loud briar]

Yep, still fails to resolve https://docker:2376/_ping with lookup docker a.b.c.d:53: no such host

I'll be able to work on this all week if there's ideas/workarounds/whatever to get it running, but I'm away next week.

[sharp marsh]

np. I'm checking rn

can you share with me your .gitlab.yml file please?

[loud briar]

Will do, 5ish mins

[sharp marsh]

@loud briar found a "good enough" solution: https://gitlab.com/marcosnils/dagger-ci-test/-/blob/main/.gitlab-ci.yml

[heavy karma]

Which page are you referring to @loud briar ? What was outdated?

[loud briar]

It's in a help thread: #1077289195254595786 message

Not outdated as such, but there's a way to improve that page that I think will prevent users hitting the same problem I did

[loud briar]

Does the Container from_ method support any of the docker pull options? Quick glance through the sdk references suggests it doesn't?

[sharp marsh]

What are you looking for particularly?

[loud briar]

-a - all tags. Nothing critical, just a thought/experiment

[trail delta]

Hey
I'm trying to use a locally built image, but I don't get how I can made that with the Python SDK?

[slender star]

I made a help topic here to discuss 🙂 #1077626911485853826 message

[sharp marsh]

yeah.. definitely not there. Would be great to know what your experiment is about to see if/how adding that flag to Dagger pull operations works

[swift barn]

Hey, how can I empty the Dagger-cache when developing locally? I tried killing the Dagger-engine container but now getting panic: failed to connect to dagger engine 😦

I don't quite understand how the Dagger-engine container gets created. I'm using Go SDK

After restarting my Docker-daemon I can connect again but the cache is still there.

[sharp marsh]

you can do: docker exec $(docker ps --filter name=dagger-engine -q) buildctl prune

[swift barn]

Thanks! Is this documented somewhere?
I also tried to bypass the cache with WithEnvVariable("BURST_CACHE", time.Now().String()). but for some reason it was still using the cache with the steps that followed.

[sharp marsh]

it was documented before but since we're going through a docs clean up it's not there now.

The BURST_CACHE should have worked. I can help troubleshoot if you want

[swift barn]

yeah, it used to work in our other job but for some reason now it didn't

Help appreciated 🙂

I'll see if I can reproduce that after I have ran my pipeline with emptied cache

[sharp marsh]

🙏 . If you could share are repro snippet about the BURST not working that'd be awesome

[swift barn]

It was most likely me misreading the logs 🙂

Is there a way to add logging to the engine side somehow btw?

[sharp marsh]

which SDK are you using?

[swift barn]

go

[wheat agate]

Using the go sdk, has anyone encountered very long runtimes resulting from many chained WithExec calls? Performance seems to get worse the more I chain together.

[timber crater]

Some cool demos in the community call today!!!

[wraith niche]

I laughed at your last comment on Zoom 😆 "friends don't let friends use cloudformation templates"

[main spear]

Ain't that the truth

[wraith niche]

100%

IMO the only advantage of using CFN (rather than terraform for example) is to keep the infra state on AWS, but jeez, templates are awful to deal with. That's why I'd go with the CDK, it's a good improvement if you want to rely on CFN as a low-level piece.

[verbal canopy]

Has anyone tested out Dagger with https://depot.dev/?

[elfin frigate]

this might be related: https://github.com/dagger/dagger/issues/4620

[wheat agate]

@elfin frigate Thanks! I was unaware this issue had been opened. It's the same issue for sure. I'll try to put together another repro

[turbid tulip]

We've customized our buildkit config and its runtime... so I'm pretty sure depot.dev won't work well with Dagger. I remember @pseudo stream speaking to that recently.

[wheat agate]

I"m certain it's the same issue because It's literally the same project as @stuck wyvern 🙂

[sharp marsh]

if that's the case, have you tried adding dagger.WithLogOutput in your Connect function? i.e: client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout)). That should show buildkitlogs in your pipelines

[pseudo stream]

Yeah I don't believe it will work in any reasonable way. Maybe it's possible to start up the dagger engine as a privileged buildkit exec in depot if you really wanted to, but then you are doing container-in-container-in-container, so not exactly ideal 🙂

[sharp marsh]

https://giphy.com/gifs/animation-selfie-iphone-3o7WTQqsDe6gyLv0hG

[swift barn]

Yes we have that. I was more thinking can I somehow logging that will be shown in the engines logs. Something along client.Log.Debug()

[sharp marsh]

hmm there's currently no way to print a line in the engine logs. What's the use-case if I may ask?

[chilly arch]

Recording of yesterday's community demo can be found here: https://dagger-io.zoom.us/rec/share/DMYV5G2eO_JlFJiF74cknSifEPthJiRZkhJOBiZqFeIPxfS2P9YHu0UBRZc6CTSw.j65n8ifiL7LjSlzI

[vestal creek]

Anyone seen Moon?

https://moonrepo.dev/docs/guides/ci

[sharp marsh]

seems like a turborepo competitor? Looks like it's mostly designed for the web ecosystem

[vestal creek]

Very much seems multi lang and Rust, Go, Bash, and others are supported

https://moonrepo.dev/docs

[sharp marsh]

I see.. looks interesting. Looking forward to see a Dagger/Moon extension 😛

[stuck wyvern]

Feeling a little silly right now... trying to make dagger build a dockerfile: _, err = client.Container().Build(client.Host().Directory("."), dagger.ContainerBuildOpts{Target: "final"}).ExitCode(ctx)

It definitely reads it but doesn't actually run any of the steps, it just loads the image metadata.

[elfin frigate]

@stuck wyvern I think I ran into this gotcha too - try adding WithExec(nil) before ExitCode(ctx)

[stuck wyvern]

Yep, came to that conclusion just as you were saying it.

[slender star]

Same issue if you use client.Host().Directory(".").DockerBuild().ExitCode(ctx) ?

[winter linden]

That seems to be a common source of confusion - the extreme laziness of all Dagger operations by default. Should we consider adding an optional argument like dagger.ContainerBuildOpts{AndDoItNow: true} ? 🙂

[stuck wyvern]

That's why I'm running with .ExitCode

[main spear]

I guess a problem with running it with the option to run it now, is that you need to either store return response, or do it immediedly. Which would break some of the niceness, another option could be to add an extra method to the chain instead that just panics on error.

I.e. ```rust
client.container().from("").try().await

potentially even with a function
```rust
client.container().from("").validate(|e| myCustomErrorHandler(e)).await

the api would be kind of the same for go. Though rust would in this case could be made even more concise

[rain oriole]

Hey, you can add your thoughts to https://github.com/dagger/dagger/issues/4205 🙂

[main spear]

Thanks. Difficult problem to solve, you want a bit more handling, but doesn't necessarily want to add a custom DSL on top of the package.

I really like the idea of using helper packages (like daggers own version of errgroup). The last workaround not as much, I feel like it is generally against best practice to run new functions in cleanup code (which defer and drop generally is used for). I will pitch in with my own ideas in the thread 😄

[trail delta]

Hello, I’m currently migrating one of our CI pipeline which is in team city in shell to dagger. I really like the dagger code in the end but the devs are missing the différents stages state from the CI view. My first though about this is to call in the différents stages a call to the same script but a different entrypoint (python ci.py build or python ci.py test) but I don’t know your point of view on this

[sharp marsh]

this is exactly what we do in the meantime as an alternative until we have native Dagger check integration with different CI SaaS solutions.

@trail delta which VCS are you currently using with your CI runners? Github? Gitlab?

[trail delta]

Nice. So do you create 1 client per sub step or a global one ? And what do you mean by check integration ?

I’m both on team city and Jenkins

[sharp marsh]

and where is your code stored? github, gitlab?

[trail delta]

Currently is bitbucket and svn 🙂

So the current way I’m doing it is using the build agent checkout step, then executing both python function in each different stage to have the result in the CI UI

[sharp marsh]

By checks I mean something like this

Eventually we want Dagger to add to your user's PR's the different "steps" in your pipeline automatically, so you don't have to do it manually in your pipeline

[trail delta]

Ok so yes Checks/Stages the same 🙂

Are you using go sdk or python ?

[sharp marsh]

golang is my strongest SDK rn

[slender star]

But the SDKs all have equivalent functionality, so you should pick the one that works best for you! (perhaps Python)

[rain oriole]

Hey. In Python I prefer using Typer (installed with dagger python sdk by the way) to create subcommands that map to different jobs (lint, build, test, etc). So it's basically what you suggest with python ci.py build, just a reminder that you can leverage tools to parse the arguments instead of doing it yourself. You can also turn it into an executable if you install your pipeline code with (e.g., myci build).

[trail delta]

Thanks for that. I’m more the go ecosystem usually but the current company ask for python. I will have a look on the typer

[winter linden]

@trail delta here is the issue discussing that topic: https://github.com/dagger/dagger/issues/4558

[wooden cape]

I just found out about dagger ci today and this was my first thought.

I also have a question. my ci jobs are not entirely containerized, most are but a few must be ran on a host the underlying host.

i understand that containerization is the strength of dagger.io but can it be easily leveraged for host operations as well?

[sharp marsh]

You can't currently encode host operations in the Dagger DAG. Having said that, nothing prevents you to have the best of both worlds. You can pick any native based building tool for your language like Mage in Go or Typer in python and mix both behaviors. Here's an idea of how that would look using Mage (https://magefile.org/) & Dagger:

package main

import (
    "github.com/magefile/mage/sh"
    "dagger.io/dagger"
)

// Runs a host operaiton & dagger pipeline
func Build() error {
    if err := sh.Run("my_host_program"); err != nil {
        return err
    }
    c, _ := dagger.Connect(ctx)
    // my dagge pipeline down here
}

[wooden cape]

holy crap the world of possibilities you have opened for me

currently devops for a 2 year old project with zero make files (because no one know how to make them)

[solemn owl]

Hey all, I am experimenting with Dagger for the first time today, via the Python SDK. I have a noob question. I am trying to run a couple of things in parallel and then exit based on all succeeding. I am finding that Container.exit_code() throws an exception on a non-zero exit, which rather seems to defeat the point of the method to me. Am I missing something?

[winter linden]

I heard a few requests for GPU access. Does anyone else have a use case that requires it? If so, could you share it here? https://github.com/dagger/dagger/issues/4675

[sharp marsh]

Just tested this and it also doesn't work @slender star

package main

import (
    "context"
    "os"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()
    c, _ := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

    _, err := c.Container().WithNewFile("Dockerfile", dagger.ContainerWithNewFileOpts{
        Contents: `FROM alpine as final
        RUN echo hello`,
    }).Directory(".").DockerBuild().ExitCode(ctx)
    if err != nil {
        panic(err)
    }
}

[simple idol]

Hey, I’ve been following dagger for a while, and I want to integrate it in to my team’s project. Initially it was only possible to write the pipeline logic in CUE, I understand that you’ve moved to a graphql engine, but assuming there is motivation to learn CUE, is it recommended over the other SDK’s? It seems that internally you’re using go, why?

[winter linden]

At this point I would not recommend starting with the CUE SDK. It is still using the older dagger engine (0.2), we got pretty far porting it to 0.3, but that effort is now stalled and there isn’t enough traction on to prioritize it over the other SDKs. So unclear when it will catch up, if ever.

All other SDKs have full feature parity.

[slender star]

Just tested this and it also doesn t

[winter linden]

For our own CI, we use Go simply because we are Go developers, and we like writing our pipelines in the same language as the rest of the project. Python or NodeJS SDK would work just as well, it’s purely a matter of familiarity.

[simple idol]

@winter linden I see. Did you generally have a good experience with CUE as a configuration language, and move to go due to lack of traction? Or were there specific issues you encountered with CUE?

Also, what is the complexity in creating the graphql queries using CUE in the new engine? Is that a viable strategy?

[wicked thorn]

Hey guys, I wanted to try and get all files and directories in my git repo with dagger, calling Entries on *dagger.Directory, and it doesn't return all files and directories recursively. Which is not surprising, just wanted to ask if anyone thought about how to get all entries recursively? I only get the current directory entries

[winter linden]

Yes, CUE as a configuration language is in my opinion excellent. There are some rough edges but the fundamentals are strong and the potential is huge.

We moved to a multi-language approach (backed by GraphQL) because ultimately a declarative configuration language is not the right fit for our use case. People want to write their pipeline logic in the language they’re already familiar with.

[winter linden]

That could potentially work, and is something we discussed. It would require someone in the CUE community to be interested enough to try it. If that were to happen, it could hypothetically lead to a new community-contributed CUE SDK. However that would probably not be backwards-compatible with 0.2 CUE configurations

[sharp marsh]

👋 doesn't seam that it's a feature buildkit exposes. You could call find <path> to get all files and directories?

[clever mesa]

I did it 🤣 . https://github.com/octohelm/wagon
I used dagger-cue in lots of projects.
To avoid breaking changes to migrate with new engine, I wrote this to proxy cue flow task to dagger engine.

[winter linden]

What! That’s amazing! @chilly arch 👆

[clever mesa]

Most flow tasks works https://github.com/octohelm/wagon/tree/main/pkg/engine/plan/task
I don't implement network, which i have not used.

Here a back ports to dagger cue-core https://github.com/octohelm/wagon/tree/main/cuepkg
With https://github.com/octohelm/cuemod help,
extensions could be in other repo like https://github.com/innoai-tech/runtime

[teal wraith]

Hi all, I'm evaluating porting my company's cicd to dagger. Our cicd involves e2e tests against our application running in KIND. Is KIND or any form of kubernetes supported by dagger currently? I can see there's some references to it in the 0.1 docs but sounds like you guys are on v0.3 now?

[mystic mantle]

I personally wanted some access to /dev/ttyUSBX to be able to flash some device with my dagger pipeline, so I don't need to install every flashing tool under the sun on my machine. I don't know if it's related. I'm gonna have a look at it now.

[mystic mantle]

I haven't tried and I'm no k8s expert, but if I get it right, you would like to access some kubernetes service on a pod inside some docker container?
If so, you could use the same docker network between your kind cluster and your dagger-engine and they could talk to each other (in that case, I guess it would be some dagger step container trying to connect to some kind service?)

So I guess it's possible, but I don't know how much you could do directly from a dagger pipeline, and how much is external configuration.
But also, since dagger is just a SDK, you could setup your kind cluster programatically in your CI script and then run your dagger pipeline in the next function call.

[wooden cape]

https://discord.com/channels/707636530424053791/1080938359226118214 cross posting for a teeny bit of vis

[stuck sierra]

If I .Publish() a container that I had a mounted a directory into when creating it, does the mounted directory end up in the image that gets published?

Trying to accomplish the same as a Dockerfile's COPY . . operation.

EDIT: My bad, I should be using .WithWorkdir and not .WithMountedDirectory ?

Wasn't immediately clear to me when we have WithDirectory, WithMountedDirectory, and WithWorkingDir.

[elfin frigate]

mounts + publish

[main spear]

Generally WithMountedDirectory are for adding files from the host into the container, while WithDirectory or WithFile are from pulling files from another container into the current one.

WithWorkingDir is differnt that is the equivalent of WORKDIR in a dockerfile

it is more of how dagger actually works internally as it you can't mount a file, from another container, hence the distinction
Edit: this last line may actually be wrong now that I think about it, I haven't test it, as it hasn't been useful to me yet, but I guess it should be possible to mount the file, and publish it without =D. As such like vito mentioned in the thread, WithMounted* for emphemeral, and With* for persistent files

[stuck sierra]

Thanks. What's the Dagger team's recommendation when it comes to replicating Dockerfile's COPY ., would one use WithDirectory or ~~ WithWorkingdir ~~ WithMountedDirectory?

[main spear]

I will defer to their expertise 😄

[sharp marsh]

it really depends on what you're trying to do. Do you really need to copy the whole context into your container? or mounting is sufficient to perform an operation and fetch the artifacts generated from it. Can you show us a more concrete example of what you're currently doing with COPY?

[stuck sierra]

@sharp marsh , I'm pulling down a standard Ubuntu image, adding a binary to it, and re-publishing the image locally.
Other pipelines, elsewhere, pull down this locally published image and make use of the binary from the image on their own pipeline execution runs.
If this is an odd way to go about things, let me know too - learning as I go along.

[sharp marsh]

yep, in this case using WithFile is what you need. Here's a Go one-liner to do it:

client.Container().From("ubuntu").WithFile("/app/appfile",client.Host().Directory(".").File("my-file")).Publish(ctx, "myregistry.com/myimage")

^ that copies my-file in your host machine into an ubuntu container to /app/appfile and publishes the image

[stuck sierra]

Good stuff, thanks!

[winter linden]

Anybody here planning to be at the KCD event in Paris next tuesday?

[rancid canyon]

Hey folks, what is your pretty printing logger of choice for Go?

[rancid canyon]

Hey folks what is your pretty printing

[fresh depot]

I wanted to but all the ticket are gone 😭

[muted quest]

Sorry if this is obvious,..

• How do we discover what's in the universe?
• Are there multiple universes (common places to find dagger packages/modules/reusable-components)
  e.g. I saw netlify examples for deployment, are there similar things for GCP Cloud Run, or vercel, etc..

[wild bluff]

Hey all, is it possible to build an e.g. Go binary that wraps the Dagger pipeline up so that I can ship it as part of e.g. a GitHub Action? My aim here is to remove any kind of pipeline logic in service repos and instead house all the pipeline definitions in a separate platform-maintained repository.

[winter linden]

Yes 🙂

[wild bluff]

Any guides or docs on doing this?

[winter linden]

Pretty much any guides using the Go SDK should help you get started. The pattern you describe (wrapping the pipeline in a Go binary) is the standard one we use in our guides.

As for shipping the binary in a Github action specifically, I don't know if there's a guide about that. But there shouldn't be anything dagger-specific in that part: just do what you would do for any other go binary, and everything should work as expected

--> https://docs.dagger.io/278912/guides?tags=go

[teal wraith]

i'm trying to do a dagger PoC and am running the first sample code in the typescript docs. I'm confused by the output though (screenshot attached)

1. it seems surprisingly slow especially on rerun (8 seconds to have a docker container log a single message?)
2. it doesn't seem to log any useful information about the build pipeline (is there a way i can enable this?)

running on a 32 core threadripper with popos

[winter linden]

Welcome! The slowness might be called by the npm cache being wiped each time. Of you add a cache mount in the right place, it might solve it. Would you mind sharing your code?

EDIT: just realized you’re following a guide from the docs. Could you share the link?

[simple idol]

Thank you!

[winter linden]

@wild bluff let me know if you need more help on this, or have follow-up questions. I now remember that @warm temple actually prototyped exactly what you wanted to do: a Github action powered by Dagger under the hood 🙂

[wild bluff]

Got a link to the prototype, by chance?

[winter linden]

That’s why I’m pinging Kyle 😁 Hopefully he sees this monday and shares the link

lots of gems in his account if you feel like foraging: https://github.com/kpenfound

[wild bluff]

Normally I would, but I'm in the middle of a bathroom reno. 😄

https://github.com/kpenfound/multiplatform-action <-- Ah, this looks like it

[frozen minnow]

Does dagger have/plans to support non-container execution like connect via ssh for compute that cannot run inside of a container?

Based on the documentation ive read I think the answer is no but wanted to confirm

[wild bluff]

Any reason you couldn't just run Ansible or something from the container?

[frozen minnow]

No reason never thought of that. Thanks for the suggestion

[wild bluff]

👍

[wild bluff]

Is there any way to get streaming output from Container.WithExec()?

[slender star]

Hi @wild bluff, do you get what you need from specifiying a log output like this?

    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))

[wild bluff]

Yes. Thanks!

[loud briar]

There's been no update on https://github.com/dagger/dagger/issues/4289 for a while. Any idea on prioritising this?

[sharp marsh]

cc @chilly arch @fallen temple @wraith niche

[fallen temple]

Miranda and I will talk about it today @loud briar and update this thread.

[loud briar]

That'd be great 👍

[warm temple]

Yes that's the one 🙂 It uses a bash script to download and run a precompiled dagger pipeline from it's own github releases. You could also write the action in typescript and have the action execute that directly, which would probably be easier to maintain if you're just writing an action. This project does the precompiled binary thing in order to support circleci orbs as well, which can only execute bash

[wild bluff]

Yeah, I'm planning to package the GitHub Action as a container so Go will be easier since I can package it FROM scratch

What's the easiest way to recurse through a Directory? I'm trying to search for all Markdown files a la find . -type f -name '*.md'

[winter linden]

@fresh depot in the house!

[trail delta]

Hey
Is it possiblle to link dagger containers to existing network, to be able to use services already created on the machine.
in the other hand, I swa some info that dagger will support soon servies, and I would like to know if there is any release date for this?

[sharp marsh]

Hi! No, dagger services are not docker network aware.

Starting from v0.3.13 dagger supports services. We're in the process of writing official docs about it. In the meantime you can check the PR here: https://github.com/dagger/dagger/pull/4505

[trail delta]

I Will Check this

[marsh ether]

Oh I hadn't noticed that there was a release with this already. Neat!

[sharp marsh]

@trail delta @marsh ether hopefully you can make it to this week community call. @elfin frigate will be making an intro about this feature and showing some more cool advanced use cases 😉

[fallen temple]

@loud briar , following up! Does the workaround in the issue solve your problem for now, or are you actively blocked? https://github.com/dagger/dagger/issues/4289

`The current way to achieve this is by leveraging build_args and modifying the Dockerfile as follows:

FROM alpine

ARG SECRET

RUN --mount=type=secret,id=$SECRET,target=/secrets/muysecret
cat /secrets/muysecret`

[loud briar]

Yeah it worked, but I'm not currently using it. The 'how do we improve our handling of secrets' question is still up in the air so I'm not in any real rush

I'm ultimately leaning towards something a bit like https://github.com/kpenfound/hello-nomad/blob/main/ci/utility/secrets.go#L28, but there's work outside of the ci/cd processes on how we store/secure images to be done as well

[fallen temple]

Oh! I see. You're working through how to do secrets in general for your team?

[loud briar]

Kind of. The current approach is several bash scripts of horror (with copies across multiple repos, many of which have diverged from the original) and a vault binary in images that don't need it. Testing/writing new options doesn't take that long; sharing those options, answering questions, making changes to runner bootstraps etc slows it down 🙁

[real kite]

We are spiking out a dagger POC using python SDK and were wondering from the community: does anyone have examples of how they implement unit tests for their dagger pipelines?

we have a couple of patterns in mind but are curious what other people have come up with

[timber sphinx]

Just jumping in for quick thing. I wrote this article for our team as I've been dogfooding using the cli in my $HOME/.envrc and then loading this with all my project using direnv. Maybe that will inspire you. I have other approaches/stuff but this was a quick one that might be semi useful whatever tool you end up using. https://docs.delinea.com/dsv/current/tutorials/use-with-direnv.md

There's other stuff out there beyond this for process substitution but I haven't tackled that right now with the time I have.

[wary peak]

Has dagger been tested much on ARM macs / is there anything specific needed to get some of these to work? like the --platform docker arg? Had a pipeline fail for a coworker on an M1 mac fail (or rather, stall) with some panics from moby/buildkit when the node container in the dagger pipeline started. Logs in 🧵

[timber sphinx]

Current state of services? Ie can I run Hugo serve/mkdocs serve now and leave it running in the background like docker compose commands? Been watching and think I read it’s there just not documented yet. Not sure though of the current limitations.

[elfin frigate]

state of services

[winter linden]

I’m on the plane and missed the community call 😭😭 the demos looked amazing

@elfin frigate did you really run a docker compose project on Dagger?!

[elfin frigate]

yup! 😄

this one: https://github.com/vito/dagger-compose/blob/main/wordpress.yml (and redis.yml)

[winter linden]

Can’t believe I missed that moment

We’ve been talking about about that for so long

[elfin frigate]

there are definitely caveats (e.g. no live-syncing bind mounts) but it was surprisingly easy to get the basics working

[winter linden]

One day we can add core support for livestreaming a directory

We chose a wire protocol; you get a service object (hint hint) connect to it to do your thing

or maybe it’s a magical mount (less plumbing to worry about for the devs)

[slender star]

Was so amazing to see how easily @elfin frigate handled the compose yaml "parsing" into Dagger. No yaml format is safe 😆

https://tenor.com/view/any-of-us-could-be-next-vulnerable-danger-were-not-safe-melissa-roxburgh-gif-14212594

[winter linden]

I heard there’s another “Github Actions running on Dagger” compat layer in the works in the community.

[chilly arch]

@here Community call recording from this morning - Thank you to @elfin frigate , @sharp marsh , and @heavy gazelle !

https://dagger-io.zoom.us/rec/share/n6WIx_qMSAU8KiVjm0J-94gUrycpxQWjTwjf6AGXWcvyb_6ztjvfRT_avZoVfduH.KGwTykoBtEreb2oz?startTime=1678381187000

Demo snippets in the demo forum coming soon.

[sturdy bluff]

hi folks, does anyone know if there is a way to fetch tags when using this mechanism to clone
client.Git(url, dagger.GitOpts{KeepGitDir: true}).Branch(m.Branch).Tree()

our use case is a CI pipeline which uses tags to know the previous version so it can calculate the next version (along with semver commits)

[elfin frigate]

listing git tags

[covert jungle]

Hello, is all 3 apis(go/python/node) equalvalent or there's some defect on some sdks?

[sharp marsh]

hi! they're equivalent

all the SDK's are generated at the same time with the same features

[covert jungle]

excellent, thank you.

[main spear]

The rust sdk I've built is a few days behind, as I don't build it until the actual dagger engine version is out. It may also lack some features.

I believe this version introduced a cleanup mechanism for erroring if orphaned pipelines aren't executed, I don't explicitly support this for now. The same goes for explicit .id() calls on files, directories and so on. All of it planned, just haven't gotten around to it yet.

Some of them aren't included because I don't use them myself atm. However, if there is interest I don't mind adding them

the v.0.4.0 release build is running as I write this 😄

edit: it is up now as well: https://github.com/kjuulh/dagger-rs/releases/tag/dagger-sdk-v0.2.16

[covert jungle]

Thanks

[tender goblet]

I truly hope Dagger (the company) isn't (too much) impacted by the SVB going down! 🙏🏼

[covert jungle]

hmm, dagger doesn't seem to honor .dockerignore when Dockerfile involves copy . ./

sent my issue to #1084100834830331916 message

[slender star]

If anyone is at SCaLE today or tomorrow in Pasadena hit me up!! Would love to do a hack session 🙂
https://www.socallinuxexpo.org/scale/20x

[main spear]

Hi, I am currently working on a mage like tool for our org, which is supposed to enable us to use dagger at scale. Would this be something you would be interested in as a demo.

i.e. the value proposition is that teams and services can share common tools and pipelines through a simple interface. And we as platform engineers can build currated paths for application development.

The workflow works as such:

• shuttle run build
• shuttletask builds a golang binary containing a Build function which contains a dagger pipeline and other useful tools.
• the binary is executed

The thing about this is that the golang code is pulled from a central place for each service: Money service -> golang service shuttle plan.

All of these tools are open-source btw 😄

[mystic mantle]

I think it seems a great idea for our next community call 😄

[faint sleet]

So, can you do exactly the same with the Go sdk, that you can with the dagger-cue command?

[sharp marsh]

Not currently, sorry. I wasn't thinking of CUE, I was mostly referring to the programming language SDK's.

More context of the CUE SDK future here: #general message

[trail delta]

Hey guys, is it possible to have capabilities enabled like SYS_RESOURCE
our curent build system is using it adn we cannot get rid of them easily

[elfin frigate]

you can set InsecureRootCapabilities in the exec options: https://docs.dagger.io/current/sdk/nodejs/reference/modules/api_client_gen/#type-declaration-15

[trail delta]

thanks I will check this

[fossil pine]

Huh, just saw the new service container feature. Looking forward to try it!

[fossil pine]

I'd love to see this in action!

[main spear]

Aight, once I get a bit further with the project I will send in a proposal 😄

[wild bluff]

Hey all, does the inclusion/exclusion in HostDirectoryOpts support path globs e.g. **/*.md?

[elfin frigate]

yep

[wild bluff]

So I have a pipeline that I'm building that does some processing on every Markdown file in a repository. For sake of argument, let's assume there are no ordering or other dependencies between the files.

If I use dagger.Directory.Entries() to get a list of all Markdown files, how do I "fan out" to N number of container executions, where N is the number of entries, and execute them all concurrently?

[winter linden]

The dagger engine can process nodes in your dag in parallel transparently. You just need to make sure the client (your controller code) makes the queries in parallel, using your language’s normal features. For example goroutines for Go, etc.

[winter linden]

We should make this topic a guide & blog post @chilly arch 😁

[wild bluff]

Also any way to recursively walk a dagger.Directory?

Directory.Entries only returns the top level

[winter linden]

Sadly no, you have to implement it yourself, see this thread: #1080723370330570802 message

[wild bluff]

That's fine, for my purposes I just used filepath.Walk.

Any way that I can interleave stderr and stdout from Container.WithExec()?

[rain oriole]

Yeah, I've been working on that → https://github.com/dagger/dagger/issues/4766

[stray jolt]

👋 is there an accepted pattern for operating on another image that I've built? let's say I want to do image scanning as part of my pipeline with trivy - what's the right way to go about that?

[sharp marsh]

👋 In the past community call I showcased a demo where I use grype to scan images in my pipeline. Here's the link for the demo: #1084673886299369493 message and the relevant code: https://github.com/marcosnils/dagger-knative/blob/dagger/ci/scan.go

[wild bluff]

Wanted to bump this: #general message

Trying to get a complete output stream that includes both stderr and stdout.

[sharp marsh]

there's currently no way to achieve that. The only way I'd think it's possible given the current architecture would be through GraphQL subscriptions, but I think that conversation has not even started. Maybe @pseudo stream has some opinions about this, but it's definitely not possible now.

[pseudo stream]

If the goal is just to get both returned as an interleaved string, that doesn't currently exist but there's an issue open for adding support here: https://github.com/dagger/dagger/issues/3496

There's a tiny bit of trickiness to it since we'd want to support both the interleaved output and the existing separate stdout/stderr, but it's doable without any huge new features

If, on the other hand, you actually want a streaming interface for retrieving the logs (as opposed to just getting a single string), that is indeed going to involve a lot more new underlying machinery like @sharp marsh mentioned. It's absolutely something we can and should do in the fullness of time though

[wild bluff]

Interleaved output would be my primary concern, but over time yes streaming output would be nicer.

[stuck wyvern]

One thing I wish, and I wish buildkit made this easier as well, is to just run the dag executor locally in process.
Sort of like how the containerd client can be configured with in-process versions of services (or anything that implements the service interfaces).
In a CI environment the daemon isn't really buying me anything.

Of course that's really only feasible in Go

[pseudo stream]

It's definitely possible technically; I had a side-project a while back that imported buildkit+libcontainer as libraries into a single binary that would bootstrap itself into a rootless user namespace and run builds, which is sort of taking the idea to the extreme. And even for non-go SDKs, we still always use a helper binary to communicate with the server (that's how python/nodejs support e.g. local dir sync, which is implemented only in go), so it would be possible for them too.

In a CI environment the daemon isn't really buying me anything.
I agree but I guess my question is whether it's hurting much too? We've tried to hide the need for awareness of it by bootstrapping the server in docker if a pre-provisioned one isn't explicitly specified. And while it certainly must add performance overhead I'm not sure if it's a bottleneck yet since everything will just go over local pipes/socks in the default case. However, very interested to hear if it's causing problems for you in practice.

The downside of optimizing this would be more complexity, so I'd just want to first make sure the upside is significant enough.

[stuck wyvern]

I think it's mostly just being able to hand someone a go module and just have it work without depending on other things.

[winter linden]

In CI you’ll want the engine pre-baked in the runner infra anyway

[pseudo stream]

I see, yeah that's totally fair. Honestly I wonder if you could accomplish this today not by updating the Go SDK with it but instead creating some go package that depends on the go sdk and the engine (our customized buildkitd, currently in cmd/engine in our repo). Then the package would first start the engine, probably have it listen on a local unix socket, set _EXPERIMENTAL_DAGGER_RUNNER_HOST=<that sock>, and then do a bunch of other shenanigans to deal with some of the other binary dependencies (the helper I mentioned before and runc) so they all point to your own binary.

You'd still need sudo (unless you make a rootless userns or are already root), but in theory it might be possible already... If I had more time I'd try it 🙂

[winter linden]

as we’re discovering

Auto-install of the engine from the SDK will not be how typical production workflows are run. It’s more of a convenience for development and onboarding

Of course for that prediction to become reality, we’ll need to provide a stable polished UX to use a pre-existing engine. _EXPERIMENTAL_xyz of course will not cut it for mainstream adoption 😁

[wild bluff]

TIL there's an option to use an outboard engine. 😄

[pseudo stream]

There's some docs in our repo here on it: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#connection-interface

Like Solomon said, we want to polish it all up first, hence the lack of discoverability for the feature. "Power-users" should feel free to try it out though!

[wild bluff]

So if I'm distributing my CI pipelines embedded into an internal Go CLI tool, users still need dagger present on their machine?

[winter linden]

nope 🙂 magic

[pseudo stream]

Yeah the default behavior is for the SDK to download one if not explicitly set (via _EXPERIMENTAL_DAGGER_CLI_BIN). We do hash checks and such to minimize the downsides, but it's very much just the default behavior at the moment in order to make onboarding trivial. For production use cases it's gonna be better to pre-install

And I think we can in time make the pre-install use case so trivial that we could probably consider switching the default behavior to not be the magic auto download (just an idea based on past discussions with @winter linden, nothing set in stone)

[wild bluff]

So if I'm defining a pipeline inside of a Go codebase, what's the "right way" to do that? I see that there are Project and Pipeline types in the SDK

My current approach has been very FP-ish in that I just sequentially execute functions that call e.g. Container.WithExec()

[stuck wyvern]

Oh that's a clever way to go about it.

[pseudo stream]

It'll depend on what your end goal is. I wouldn't worry about Project right now, that's related to something we've been working on called extensions, but they are not fully baked yet. Pipeline is meant for annotating the steps of your pipeline with human readable labels, which we are working using to improve the progress output and more advanced forms of visibility/observability/etc.

My current approach has been very FP-ish in that I just sequentially execute functions that call e.g. Container.WithExec()
That in general sounds like exactly the right idea. Is your question around structuring the overall codebase? Making reusable functions? Something else?

[wild bluff]

I don't know?

In general I'm just wrapping things up into a struct that includes methods for each "phase" of our pipelines, then using a simple factory to introspect the project, select the right pipeline, and then run it.

Felt at least reasonably intuitive, but my nagging feeling is that I'm losing the ability to have Dagger generate the dependency graph and do concurrent execution.

[pseudo stream]

Totally, we've tried to minimize the need to ever think about the need for optimizing those sorts of details and optimize everything in the background, but it still isn't immediately obvious by any means.

The general idea is that all of the operations you specify with the pipeline are lazy, so nothing will actually happen until you call a method that returns data that required execution. So if you do c.Container().From("alpine").WithExec("foo").WithExec("bar") nothing will happen synchronously. It's only when you call .Stdout() on that pipeline that foo and bar will actually execute. The "rule of thumb" in the go sdk is that if the method has an error return then it's doing some synchronous work. If it doesn't, it's 100% lazy.

On top of that buildkit's caching means that step execution is deduplicated, so if you submit two pipelines that have overlapping steps, there won't be duplicated execution of the same ones.

There are still cases where you may need to do some parallelism on your own, e.g. if you want to get the Stdout for two different pipelines in parallel. For that we currently just have users rely on the underlying language's built-in parallelism mechanisms. So for go errgroup is a common option.

[wild bluff]

Ah alright. That makes more sense.

[clever mesa]

Container().Build() failed with github.com/dagger/dagger v0.4.0 (dagger.io/dagger v0.5.0)

pipeline.container.withRegistryAuth.build.exitCode no command has been executed

Revert to github.com/dagger/dagger v0.3.13 works well ( dagger.io/dagger v4.6.0 & v0.5.0 )

[elfin frigate]

@clever mesa if you're calling ExitCode() on the result, this was a bug before (silently succeeding when actually running nothing). you might need to do Container().Build().WithExec(nil).ExitCode(ctx) instead

[clever mesa]

But when I switch Container().Build(dir, opts).WithExec(nil).ExitCode(ctx), it will hang by entrypoint of image.

[elfin frigate]

oh, are you trying to just build an image and discard the result to check if the build succeeds? maybe the dockerfile runs tests or something?

[clever mesa]

Just do a wrap for old projects which use Dockerfile

Without ExitCode(ctx), I don't known is this Dockerfile built or not.

[elfin frigate]

makes sense. I don't think we considered that use case with this change. the goal was to catch accidental no-ops, like running Container().From("postgres") as a service (which would just exit immediately)

you could try other methods on Container for now, like .Rootfs().Entries(ctx)

[clever mesa]

It works. but the log just start docker build without pipeline prefix.

I have to use the hack to define custom logger

if v.ProgressGroup != nil {
    pp := pipeline.Path{}
    // v.ProgressGroup.Name should always use Pipeline name if exists
    if json.Unmarshal([]byte(v.ProgressGroup.Id), &pp) == nil {
        for _, p := range pp {
            if strings.HasPrefix(p.Name, PipelinePrefix) {
                v.ProgressGroup.Name = p.Name
                break
            }
        }
    }
}

[mystic mantle]

My attempt with Snyk: https://github.com/dolanor/demos/blob/main/gosnyk/ci/ci.go

[swift barn]

Hello, how should I approach the need for human confirmation in the middle of a pipeline?

Any pointers in the right daggerio-ish way

Something that could work when running locally and also when in github (or similar).

[sharp marsh]

how would that go in a remote CI/CD system like github? I'm not aware of Github allowing human interaction in their pipelines.

[swift barn]

you are probably right.

[sharp marsh]

you can build something where your pipeline has to wait for a webhook to be triggered in order to continue or something around those lines, but that's outside of the scope of Dagger or Github actions. It's something you'll have to build internally

[swift barn]

thanks. I'll look in to it.

I'm running terraform plan but it is not running the second time I run the pipeline. I may be misunderstanding something about caching and the likes.

Some thing like:

plan = client.container().from_("docker.io/hashicorp/terraform:1.3.9").with_exec(["plan"])

Only runs once.

The second time I run the pipeline it's showing me chached output of plan. I'm shure because the resources are un reacheable and it cannot actually check the state of the infrastructure.

Im testing running dagger locally. If I delete del dagger-engine container it runs all the steps againg, including the terraform plan.

maybe I shuold be signaling somehow that some steps must be executed every time?

[rain oriole]

You can add an environment variable with the current timestamp.

[swift barn]

Thanks. Is this an acceptable way to proceed or am I taking the wrong approach entirely?

I meant, is common to do this sort of "signaling" ?

[rain oriole]

Yes, that’s acceptable. Sometimes it’s needed.

[swift barn]

Thanks for clearing that up for me.

[wild bluff]

btw, the way that you would normally do this is to save the plan file as an artifact and then wait for the PR to be approved (or merged) before running a separate pipeline with the apply step.

You can replicate that setup with Dagger.

[cunning jolt]

Is there an example somewhere of using a remote S3 cache? I'm interested in using that with our ephemeral Jenkins agents so I can maintain a cache without local storage.

[stuck wyvern]

After this suggestion I went and changed a project to manage its own dagger instance so it doesn't leave things behind. This works pretty well for what I was thinking. It's not totally stand-alone since it needs docker but it's pretty good.

Totally separate item:

I've been working with a colleague to change our build pipelines (we build all the container-ecosystem packages for all of MSFT+GitHub+all_the_things for all the distros) to use dagger.
Going from Turing-complete YAML in Azure DevOps + hodgepodge of dpkg-build and rpmbuild stuff + nearly Turing-complete HCL (buildx bake) to everything in go with tests and type safety and and and... it's been really nice.
Hoping to open source this soon so this is transparent for anyone using our packaging.

I started this rewrite out in CUE and bumped into some difficulties that made me back-burner it but things are going well with the go-sdk.

[fallen temple]

I hadn't worked with Go before learning Dagger. Honestly, I like it a lot.

[stuck wyvern]

Go is an amazing language.

[fresh depot]

Hi, i saw that you ship experimental telemetry in last release ? Is it a premise of seeing dagger run in the cloud ? If yes is it usable right now, i saw that you need a token for that ?

[modern merlin]

Is there any sort of test suite for dagger pipelines? (maybe something similar to this https://github.com/dagger/dagger/issues/4199) I can only find resources in the docs on how to run unitary tests from the applications but not the pipeline itself

[winter linden]

coming soon 😁

[stuck wyvern]

Last night I built this thing to map a go io.FS into a dagger.Directory.
It's a bit heavy since it has to walk the whole fs tree and read file contents into memory in order to create the dagger.Directory.
I'm not familiar with how client.Host().Directory() works, but I'm wondering if the client could have some facility to map this in the same way as the host fs is (assuming files are only read/transferred when needed).

https://gist.github.com/cpuguy83/7fc6349ed2631783ed5bc58433a25d0d

[slender star]

Last night I built this thing to map a

[wild bluff]

Does anyone have good strategies for mocking out Dagger calls?

[real kite]

Does anyone have good strategies for

[main spear]

I have been thinking about how to share pipelines between projects and languages.

The gist of it is some kind of sharing mechanism through pure graphql. The authoring experience could be normal sdk level of abstraction.

Then through regular sdk you could download the graphql and possibly other assets. The through either codegen or a specif function you could invoke said plugin.

You may have other plans already and this is just my 2 cents

Edit: written on mobile, I'd be happy to explore the idea further if it is of value

[past ivy]

We had a brief chat about this in #maintainers, and this seems to be the plan for the upcoming "extensions" #maintainers message

[main spear]

seems to be pretty much the same thing 😄

[winter linden]

Indeed 🙂 And to answer your earlier question @past ivy , I don’t think we have a main issue for extensions yet… A mistake we are going to fix very soon. in the meantime feel free to write down your own thoughts and requirements, we can reference them later from the primary issue.

We were just discussing the topic with @pseudo stream last week

[past ivy]

no particular thoughts and requirements beyond what I shared in #maintainers 🙂 for me it's mostly a way to be able to open-source what I think might be common tooling

[slender star]

👋 @boreal root

cc @winter linden @chilly arch @fallen temple

[boreal root]

https://giphy.com/gifs/spongebob-cnuS67F8IoVTYRvJXE

[winter linden]

welcome 🙂

[pseudo stream]

@past ivy @main spear @winter linden opened a tracking issue here: https://github.com/dagger/dagger/issues/4796

We didn't have one before this because when we started on Dagger v0.3 extensions were such a fundamental part of the idea that we didn't even consider them to be a "separate feature" that we'd track in an issue. Since then we decided to start simpler and just release the core API at first, but I'm glad that others are catching onto the idea again anyways, that's a good sign I think 🙂

[timber sphinx]

With the built in caching, do you think you can get a dagger build running via mage to get close to the same duration as a native go build command? I'm talking about after the first run. I'm curious if any tests have been run to see what duration the remaining process adds.

[pseudo stream]

Yeah, you need to use cache mounts in the right places (go mod cache and go build cache) and you need the base golang image already cached, but once you have those in place the inherent overhead should be quite minimal. It amounts to some RPC and container startup, which in most cases shouldn't be noticeable. I don't have any hard numbers to back up that assertion yet, but a lot of the observability improvements we're working on should make it much much easier to get those sorts of statistics soon.

I suppose the other factor is if you want the binaries you built in your local dir, which will require a local export. Provided the binaries aren't hundreds of MB I wouldn't expect this to be much, but that may actually be the biggest source of potential overhead. There are various improvements we could make to get the local export to be more "rsync-like" by breaking files down into blocks too.

[timber sphinx]

Good info. So basically with something like chainguard image I'd want to create a cache volume mapped to something like /home/nonroot/.cache/go-cache or equivalent style mapping as well as probably just handling /home/nonroot/go/ mounted anyway so all go mod bin/pkg/src are all saved in that mounted volume for reuse. Sound about right?

[pseudo stream]

Yep that sounds right, and you can override the values of those directories by setting GOMODCACHE and GOCACHE env vars in case you want to put them somewhere besides their default locations

[winter linden]

By the way @timber sphinx, you could probably have Dagger built those chainguard images too, either using their own tooling, or a trivial reimplementation of them using diff/merge operations.

[winter linden]

The trick of leveraging distro-native packaging tools to produce container images in a more efficient way than a "dumb" Dockerfile, is often presented as "build docker images without docker". But in fact it's not mutually exclusive with containerizing your build. You can get the best of both worlds (thanks to diff/merge)

[timber sphinx]

Not sure I follow. I'm using the chainguard go image builder as it's kept up to date and minimal and then using the chainguard static image for local/prod runs as it's also minimized with a prebuilt nonroot user, CA , etc, just some chore stuff. I just use that as it's like 10 mb or so as my running image.

Are you talking about using their starting point and replicating that myself? If so why would you add that if they are maintaining those already?

[slender star]

addressed in #help here: #1086695249864310824 message

[trail delta]

is it possible to have the dagger exec logs more verbose?
I got this line #2 [internal] load metadata for docker.io/library/debian:bullseye-slim that take a lot of time

[sharp marsh]

you can also check at the engine logs to see if they show anything

that should be docker logs <engine_container_id>

[trail delta]

The engine log is pretty silent

[sharp marsh]

does it happen every time you run the build?

[trail delta]

yes I htink it's the build itself, but with no logs shown

[sharp marsh]

hmm that's strange.. build outputs generally can be seen in the engine.

[trail delta]

rhis is the only logs that I have in the engine

[winter linden]

is anyone interested in giving a talk about Dagger’s use of GraphQL? if so, dm me

[frozen basin]

solomon saw your tweet re: docs... might want to check out lanchain vector db QA - I've used it to great success. It has some "map reduce" style chains to help work around context size

[humble sage]

Hi, I am trying dagger python sdk for the first time. As awaited, the first run of a pipeline are slowed by the download of the images. Next run are much quicker, which is great work by Dagger. Nevertheless, I see 5 seconds spent each time I run a pipeline. I guess this is the time needed to launch dagger engine. Is this a correct guess ? If correct, is there a way to have dagger running in the background (I did not manage to find that in the documentation).

[rapid steppe]

I run the dagger engine on my local env using a feature flag, that is, running dagger in a podman container, but you can run it in a Docker conainer aswell.

podman run -dt --name dagger --privileged ghcr.io/dagger/engine:v0.4.2
export _EXPERIMENTAL_DAGGER_RUNNER_HOST=podman-container://dagger

With Docker it would be something like this (untested)

docker run -dt --name dagger --privileged ghcr.io/dagger/engine:v0.4.2
export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger

[humble sage]

Thanks for the details. Is it documented somewhere that dagger engine runs as a container which I just found via docker ps thanks to your answer

[tardy mauve]

I've out of the loop on this one: where is self-signed support at? Do we still need to use _EXPERIMENTAL_DAGGER_RUNNER_HOST with a custom dagger engine? Any docs I can read?

[rapid steppe]

I stumbled upon the env var on the #1030538312508776540 channel and mixed it with these deprecated docs https://docs.dagger.io/1013/operator-manual/

[rapid steppe]

BUILDKIT_HOST (in the docs above) is not needed anymore since dagger seems to spawn it internally (correct me some pls if I am wrong)

[humble sage]

One of the things I am looking for in CICD tools is testability of my setup. Is there any information about testing the pipelines built with dagger ?

[tardy mauve]

BUILDKIT HOST in the docs above is not

[trail delta]

is it possible to always run a execution (like a output check or such)?

[rapid steppe]

I am using the following to trigger a build without exporting the output or pushing to a registry

// ...
img := client.Container().From("some-image").withExec(..)
fmt.Print(img.Stdout(ctx))

[trail delta]

but next time it is executed, it used the cache, then no execution if no change

[trail delta]

Is it possible to parallelize execution (like multipl test in parallel) ?

[winter linden]

Yes, all steps will be executed in parallel by dagger, as soon as its inputs are available. This happens automatically for you. On your side (client/controller code) you simply have to make sure that your client code makes calls to the dagger API in parallel when necessary.

In other words, the engine can run all your tests in parallel, but only if it knows about them 🙂

note, @rain oriole is writing a guide about exactly this topic

[trail delta]

eager to read this

[sharp marsh]

hi, it's not necessary to spawn the Dagger engine yourself. The first time Dagger runs in your machine, it starts the engine and it's not stopped after your pipeline finishes so your consequent runs should already use the engine that was started the first time

[sharp marsh]

Spawning the engine manually should be done in very particular cases like you need to target an engine deployed somewhere else from your CI runner for example

[rapid steppe]

hi it s not necessary to spawn the

[chilly arch]

30 minutes until the Dagger Community call - See you there! https://dagger-io.zoom.us/webinar/register/4416686668409/WN_USQjVBGXT0SWhNMvqYVvCA

[glad jolt]

It's not in the production docs, but the internal techincal docs in the repo. I think this resource will give you more context on that subject:
https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#runner

[old pollen]

Hi, is the second presentation code of today on some public repository?

[sharp marsh]

@stuck wyvern @wind violet thread about rust compilation caching in Dagger

[chilly arch]

@wind violet can you share the link here? I will make sure to include it in the demo forum post as well.

[wind violet]

I will prepare the code and build out the company relevant information. Ok?

[covert jungle]

Quick quiz: if I have a service container that take quite a while to start how do I specify the health check so my depending container don't start before it's ready?

[sharp marsh]

@covert jungle services containers automatically honor the healthcheck defined in the OCI image (https://docs.docker.com/engine/reference/builder/#healthcheck). Having said that, I'm not sure if it's possible to set up custom health checks before binding the service. cc @elfin frigate

[covert jungle]

sorry nevermind, I found the answer here

#maintainers message

I forgot to expose ports

apologies for the ruckus

tangential thoughts here, if I have 2 with_exposed_port in one container will it honor both?

[elfin frigate]

yep - it'll wait for both to be listening

[covert jungle]

fantastic

[elfin frigate]

no support for custom healthchecks yet, I think we'll need that at some point though

[sharp marsh]

@elfin frigate speaking from ignorance here. Aren't healthchecks just labels of the OCI image?

[covert jungle]

about the guide with service, here
https://github.com/dagger/dagger/blob/main/docs/current/guides/757394-use-service-containers.md?plain=1#L238
I noticed naming is redisSrc witch is fine for both node and go but this is inconsistent with python code, is it possible to change the doc for python only?

case in point

changing the python sample makes code breaks python convention(no camelCase), but I guess there's no easy way to change the doc?

[elfin frigate]

i looked into it (https://docs.docker.com/engine/reference/builder/#healthcheck, https://github.com/opencontainers/image-spec/blob/dd7fd714f5406d39db5fd0602a0e6090929dc85e/config.md?plain=1#L206-L208) but opted not to use it for now. I personally have never seen it used, and it's a pretty different mechanism. could potentially be what we build on in the future though, since it's already there

not sure if e.g. runc runs it for you

[sharp marsh]

cc @heavy karma

[elfin frigate]

this would probably take some magical integration between the docs and the language switcher. would be cool to see, but personally not sure it's worth the effort (even Stripe's docs don't seem to care about that)

[covert jungle]

ah, I suppose breaking python convention is better way to go ATM?

gonna trigger a few die hard PEP8 pythonist but at least it's consistent

[sharp marsh]

we have the ability to do that, if you check the quicksart (https://docs.dagger.io/593914/quickstart-hello), the code ref docs change when you switch languages

it's very likely that's a miss from our side

[elfin frigate]

[covert jungle]

ah, shoot I need to sign the commit with my real name before doing pull request

[timber sphinx]

Do y’all run unit tests locally or containerized? Guessing integration gets Dagger and unit runs in your ide/go test during development? Been playing with a tilt setup and running my go build with mage/dagger takes 12 seconds or so and go build and run caches fully and takes roughly 2 seconds. Have done some basic optimization with module and build cache. Curious as that’s too slow to replace running go test locally for me but I don’t like having multiple build methods in my project. Like to keep it easy. 😆

Signing commits

[sharp marsh]

I generally prefer to run them containerized since sometimes go build doesn't do all the necessary lifting for unit tests to pass. Having said that, if caching is set correctly, the difference between doing go build locally and running it with Dagger should be negligent when the engine is already running and cache is properly configured in your pipeline

[timber sphinx]

Go testing in Dagger

[chilly arch]

Hi all! If you missed the community call today, you can watch the full recording here:
https://dagger-io.zoom.us/rec/play/MprQZX6_u4mT5TE8Y8adcFUG8jlwt303Y5WSEviL-R3nB9zjsAlfnGDWzGnj43r7zc2T5-YKjWxTQMSs.UXDma1xt4sfO72SF?continueMode=true

Check out the demo forum posts to ask the speakers any questions. You can find the forum links below:

Building and Deploying a Rust Lambda Function with Cross-Compiling and AWS CDK- https://discord.com/channels/707636530424053791/1088575225026846822

Running Postgres for integration tests inside Dagger
https://discord.com/channels/707636530424053791/1088570917359194222

[oak quest]

Hey folks! Is anyone running their own buildkitd? How do you get it to listen on tcp?

[humble sage]

Any taker for this question ?

[pseudo stream]

Hey folks Is anyone running their own

[sharp marsh]

here's a recent thread about the topic: https://discord.com/channels/707636530424053791/1086352138172641330

[rain oriole]

You can follow https://github.com/dagger/dagger/issues/4766 for updates.

[rapid steppe]

Do I understand this correctly that all dagger snapshot are stored at /var/lib/dagger mount on the host?

Any chance that the defaultStateDir could be made configurable?

https://github.com/dagger/dagger/blob/4caeaf16513e2b7de3145b85f23a62fe6b562738/internal/engine/docker.go#L20
https://github.com/dagger/dagger/blob/4caeaf16513e2b7de3145b85f23a62fe6b562738/internal/engine/docker.go#L97

[rapid steppe]

dagger/docker.go at 4caeaf16513e2b7de314...

[fluid hound]

Hey y'all!

Been excited about dagger for a while and want to get started by making a little Go library of common CI/CD pipelines to use across my other Go projects.

Anyone here working on something similar?

[sharp marsh]

👋 @fluid hound I started something with this idea but haven't found the time to actively pursue it. https://github.com/marcosnils/dagger-libs

Feel free to fork / contribute if you're looking forward to improve this. I can help with some reviewing / ideas but don't have the bandwidth to actively contribute now 🙏

[timber sphinx]

Go task library

[sacred cypress]

Gave Dagger a try today (Python) and I like that it manages automating local docker containers. Is this a company intending to sell a hosted CI/CD platform any time?

[winter linden]

We considered selling CI/CD runners, but decided not to, because there is plenty of choice in the market already. Many platform teams have already invested the effort to self-host their CI runners, we prefer to meet them where they are.

However we can sell a great CI/CD management platform: visibility into your pipelines, and perhaps caching optimizations.

One benefit of not making money from the hosting, is that interests are more aligned: everyone overpays for their CI runners. We can help you reduce your CI hosting cost, without hurting our own business.

[sacred cypress]

Where can I find details on the mgmt. platform? I don't see it on the website

[winter linden]

It's still in development, we have a design partnership program for early access and input into roadmap & pricing. If you're interested, DM me 🙂

[modern merlin]

Is there a way of specifying the default cmd when building an image with dagger? I dont see anything related in the sdk docs

[elfin frigate]

@modern merlin you want WithDefaultArgs - here are the nodejs docs: https://docs.dagger.io/current/sdk/nodejs/reference/classes/api_client_gen.container/#withdefaultargs

[modern merlin]

Ty ❤️

[wary peak]

We considered selling CI CD runners but

[chilly arch]

If anyone is interested, @warm temple is doing a demo and hands-on lab today at 5 pm PST. He will be starting with the basics, so this is a great way to try out Dagger if you haven't already! https://www.eventbrite.com/e/intro-to-cicd-with-dagger-and-yna-squad-tickets-598090874907

[young scaffold]

Thanks for the fantastic demo @warm temple

[warm temple]

Thanks for having me! ❤️

[mystic mantle]

Hi @slow forum, following up on the secret topic.
https://github.com/dagger/dagger/pull/4667 has been merged, which means you can use any external secret manager without issues. Tell us if it helps you in your process.
We're also creating a guide about that you can check: https://github.com/dagger/dagger/pull/4822

[chilly arch]

New Secrets Guide: Thanks @mystic mantle and @heavy karma! https://docs.dagger.io/723462/use-secrets/

[sharp marsh]

https://matt-rickard.com/reflections-on-10-000-hours-of-devops

• The value of a CI/CD Pipeline is inversely proportional to how long the pipeline takes to run.
• Code is better than YAML.

[winter linden]

🎉

[real kite]

Reflections on 10,000 Hours of DevOps

[fluid hound]

Quick thought. Is there an easy way to debug paths within containers via a debugger with the Go SDK? Just got a basic container running tests working and had to use golang = golang.WithExec([]string{"ls"}) to figure things out

[sharp marsh]

You can use .Directory().Entries() for that

[fluid hound]

working on example tests for my first dagger based package I'm releasing and running into an issue with example tests where dagger related log output gets mixed in with go version output causing the test to fail.

Is there a way to silence all non-command related logs so that it only logs the version and nothing else?

[elfin frigate]

you could do dagger.WithLogOutput(os.Stderr) instead, or just remove it if you don't want logs

[fluid hound]

of course that makes sense and I should have done that first! Thank you.

[potent stratus]

Awesome work on Dagger; I have a question; we are trying to use dagger as a CI runner for projects; we need to able to package our apps with CloudNative build packs. Packs need Docker to build the image; we have a docker client in the image, but we don't know how to run this in privileged mode. Any idea on how to make this work would be appreciated.

[slender star]

👋 @potent stratus Yes, you can use https://pkg.go.dev/dagger.io/dagger#ContainerWithExecOpts with your WithExec() and set
InsecureRootCapabilities to true.

Example: https://github.com/dagger/dagger/blob/main/core/integration/container_test.go#L2948-L2950

[sharp marsh]

Hey there! Not so long ago we did a community call specifically about using Dagger and Knative / buildpacks. You can check the recording here: https://discord.com/channels/707636530424053791/1084673886299369493

Let us know in case you have any questions!

[potent stratus]

Awesome folks, this was the solution

[potent stratus]

The video shows that we need to pass .Secret() for File with Secrets like docker config.json But when I implemented it, I can see this has been deprecated and the blog recommends the same: https://dagger.io/blog/dagger-engine-0-4-0

please what is the new way if Secret() has been deprecated for files?

[loud briar]

https://pkg.go.dev/dagger.io/dagger#Client.SetSecret

For files specifically there's an example guide @potent stratus: https://docs.dagger.io/723462/use-secrets#use-secrets-from-the-host-filesystem

[potent stratus]

thank you

[potent stratus]

I have more questions; Please, Won't a golang/node application fail inside something like ECS(since it uses docker under the hood and you mount dockerd) if the app is containerized? My assumption is want to run the application; you will need to containerize the application with something like alpine, scratch, or distroless,

My issue is the app should fail since dagger might pull/use an executable like docker that is not in the image you used to containerize your node/golang app(I might be dumb)

For example,

 failed to list containers: exec: "docker": executable file not found in $PATH
noxe  | Error: failed to run container: : exec: "docker": executable file not found in $PATH
noxe  | 2023/04/02 21:24:03 http: panic serving 172.21.0.1:60870: EOF: failed to list containers: exec: "docker": executable file not found in $PATH
noxe  | Error: failed to run container: : exec: "docker": executable file not found in $PATH
noxe  |
noxe  | Please visit https://dagger.io/help#go for troubleshooting guidance.

[loud briar]

Not sure I fully understand what you're asking? Dagger doesn't require you change your application/dockerfiles in any way

[potent stratus]

I am trying to containerize a golang app with alpine; this app needs to be containerized and deployed to a docker daemon host.

Inside this app, we use the dagger to pull an image that uses the docker client, but it crashes when we fire the API with the error log I posted above.

I am trying to containerize a golang app

[swift barn]

I asked yesterday about cloning repos with submodules in Help-channel but my thread is gone now. Why is that?

[rain oriole]

That's strange 🤔 Can you post it again?

[rain oriole]

Wait, is this it? 👉 #1092324707262550056 message

It's a bit further down, under "Older posts"

Don't know why it's there when there's older posts up top though. But that's Discord.

[swift barn]

Ah, I scrolled a bit and saw some way older posts on top of it and thought it got deleted or somethin 🙂

[rain oriole]

Oh, I believe your post was "closed" as in done. That's why.

[wild bluff]

Hey all, how can I get Dagger to honour the HTTP_PROXY variable?

[wild bluff]

Actually, better question: How can I run my own Dagger engine explicitly?

[elfin frigate]

@wild bluff this might help: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md

[wild bluff]

@elfin frigate Thanks! How can I get Dagger to run wrong-arch stuff?

I’m on an M1 Mac and want to be able to build with/from amd64.

[elfin frigate]

I think it'll automatically use qemu for you when you specify a non-native architecture, but I haven't tried

[wild bluff]

It complains about running an amd64 image using client.Container(), but the VM certainly supports cross arch (we’re running Colima under Rosetta)

Any thoughts on how to debug this further? Running a wrong-arch image directly via docker run works as expected.

[slender star]

Anything like this helpful?
#1091705777317761104 message

[warm temple]

Was it failing because of the arch or just printing a warning? I think I've seen a warning in the logs before but never a failure

[wild bluff]

No, this would only be applicable if I couldn't run cross arch. I can, so this step isn't necessary.

[wild bluff]

Exiting non-zero. I'll see if I can repro, but we stumbled across it as part of some work we were doing and we're under a time crunch.

I expect it has to do with buildkit

[real kite]

Just read through the sweet new secrets guide: https://docs.dagger.io/723462/use-secrets/

If you're using a workflow with secrets encrypted in files like with SOPS, what's the recommended approach? Do you decrypt first outside of the container, and mount the decrypted files? Or do you decrypt inside of the container and load secret variables inside the container?

[winter linden]

At the moment, we recommend decrypting in the controller code outside the container. You can also do it inside a container (orchestrated by dagger) but you'll need to be careful to avoid leaking the secret into the cache. That should be as simple as forcing the relevant execs to never be cached (with a random cache buster etc), but there's always a chance of a footgun. In doubt I would start outside, and iterate from there. Over time we will define a safe / supported path to produce secrets inside a container - probably as part of the extensions feature.

[wary peak]

(Follow up from the community call) I thought this was pretty telling.

The commit where I added a dagger client: https://github.com/grafana/scribe/commit/9ce4a9873df68541fa61f15920ff7e56b55c4032#diff-756bee460f7da441b89f6d023542c47f28adc934dd115642773d0ea3f916543e

vs the commit where I removed the Docker one: https://github.com/grafana/scribe/commit/a98bc694f2ad8be9752f3d9369329f8472c31209#diff-c4bedf277ac0df8d1eb6e77a08c698b6b8032fe1c763c4cfeacc90223916d7de

Dagger did so much for me with so much less code!

oh weird, I thought discord would support markdown! fixed 🙈

[winter linden]

sadly no markdown support...

[chilly arch]

If you missed the community call, check out the recording here. Thank you @wary peak and @mystic mantle for your demos! The demo snippets will be added to the demo forum shortly.
https://dagger-io.zoom.us/rec/share/f1_p5xv5Jf4pRndj4eFQtlYxYea7nE4nqoK_HXFKp1fnN0amJkV-uw0JhLqz5eRH.SHJ7jhgZzXAm76FS

[main spear]

chatgpt is absolutely wild.

I have a pipeline in dagger_sdk (rust) that I wanted to reuse in go.

I just asked it to transform it.

it ran out of characters at the end. And there was some wonky stuff, but it is nearly perfect.

EDIT: Also for the following part, I just showed it what I'd changed and it corrected itself to the API from that point on, absolutely insane.

[main spear]

also keep in mind that the cutoff was in september 2021 (chatgpt 3.5), it hasn't seen any dagger like syntax before

[sharp marsh]

^ this. I find it very surprising that's correctly guessing how the code would end up being like. One thing that it'd bet it won't catch is the ctx handling in Go code since that's directly transferable given that you need to know beforehand the Go SDK structure for that. Same applies for custom dagger.OpStructOpts arguments .

Having said that, it's still quite impressive.

[winter linden]

I think the cutoff part is not so simple, for example it can tell you what dagger is, in a way that indicates some training data is more recent than our launch.

[main spear]

I guess. However it may have scanned cue dagger. It is basically the same mission executed differently.

There are some libraries where you can clearly tell it is running on an older version clap for one. Spf13/cobra equivalent for rust.

It may be partially trained on feedback but I am unsure how that would work.

@sharp marsh there was definitely some cleanup and corrections but having the possibility to reason with it in a basic way about the api is mind blowing. I told it to get rid of .ID because they aren't needed in the go sdk. Same thing for await -> ctx

[fresh depot]

Love that 🤩

Should i show that at devoxx ?

[main spear]

@fresh depot out of curiosity what is that visualization tool called?

[fresh depot]

Its the tui integrated with the last version of dagger

You can activate it with an expérimental env Var and the wrap your pipeline with dagger run

[mystic mantle]

I think it would be great at Devoxx, but with a disclaimer this is experimental for now 🙂

[wary peak]

That's awesome.

[fresh depot]

Actually sometimes it hangs forever without an obvious reason. So, difficult to show.

[wheat agate]

Awesome, but is there a way to run a container's entrypoint as --privileged? It looks like WithEntryPoint only accepts string arguments without any options

[heavy karma]

New Dagger+PHP guide incoming, looking for reviewers: https://github.com/dagger/dagger/pull/4913 - TIA!

[meager sentinel]

Hi guys, i'm new to Dagger, trying out the tutorial atm, i have a question about caching. if you create a cache, where is the data stored? is it stored in the container dagger engine container?

Is it also possible to add a cache on existing Dockerfiles?

[stoic forge]

Hi, the whole new "SDK in your preferred language" thing is new since the last time I looked at dagger, and I'm super excited to try it out. Since the SDKs seem like thin clients, why is there a python3.10 requirement?

[sharp marsh]

yes, you can add a WithExec(nil, dagger.ContainerWithExecOpts{InsecureRootCapabilities: true}) and that should work just fine. e.g: https://play.dagger.cloud/playground/jkcS1le5L8S.

the playground doesn't allow running privileged workloads, but that should work in your environment

[sharp marsh]

it's currently stored in a docker volume that's automatically created for the engine container. This one liner should give you the volume: docker inspect $(docker inspect $(docker ps --filter name=dagger-engine- -q) -f '{{(index .Mounts 0).Name}}')

[sharp marsh]

There's only a python requirement if you're using the python SDK? Not sure if I fully understand the question 🤔

[stoic forge]

Yes, I should have been specific. Because the python that runs the sdk has a minimum version, that constrains my ability to bootstrap a dagger/python project

[sharp marsh]

I see... not a python expert but I'd assume it's because the SDK might be using some python API's that are available from 3.10 onwards. Deferring this to@rain oriole since he's the 🐍 guru here.

[rain oriole]

Hi! Any language that can make HTTP requests, can use the dagger API, even older Python versions. Our SDKs just add conveniences to make the experience much better. As for the min version in the Python SDK, it's harder for maintainability to support older versions that don't have certain features. I tried to probe the need to support older versions in https://github.com/dagger/dagger/issues/3700, but you're the first one to say anything about it. 🙂 Can you please comment on that issue with your requirements? 🙏

[fossil pine]

Has anyone tried running Kubernetes with Dagger in service containers (for e2e tests)?

I'm looking at Kind and its provider based setup: https://github.com/kubernetes-sigs/kind/blob/main/pkg/cluster/internal/providers/docker/provider.go

[fresh depot]

I tried kind with dagger a while ago. The problem here is that kind is not a service itself. It cli that starts other docker container that mock kube node.
You can run kind command in dagger if you mount the docker sock.
But you won't be able to service bind the spanw docker container.
To make this work, one could connect dagger engine and the kube container to the same docker network

[fossil pine]

Yeah, that's why I'm thinking about not using the kind cli directly, rather the "library" part of it.

[winter linden]

It just discovered that the D2 diagram project has a test to represent the Dagger stack 😁 https://github.com/terrastruct/d2/blob/master/e2etests/testdata/files/dagger_grid.d2

[sharp marsh]

nice! maybe they saw your tweet and my comment about it? 😕

[winter linden]

I think that’s exactly what happened 😎

[pseudo stream]

I have run KinD inside of dagger successfully, but I did it by just running dockerd as a dagger service with InsecureRootCapabilities set. Then I was able to use the kind library directly against that. It worked but obviously is kind of crazy in term of the amount of nesting you end up with. It would be really cool to have a dagger implementation of their provider interface so you didn't have to run dockerd inside of dagger and each pod ended up as a dagger service in the backend: https://github.com/kubernetes-sigs/kind/tree/main/pkg/cluster/internal/providers

This was part of something larger but I can extract the relevant snippets of code in case they're helpful

[fossil pine]

That's exactly what I was thinking.

[pseudo stream]

kind in dagger

[swift barn]

Hi, do you think dagger should maybe use the oldest supported go version instead of the newest? I.e. currently its using 1.20 but I think many people havent upgraded yet, and due to the way go.mod works, its difficult to put the ci code in the same project.

[stoic forge]

It seems this page has no corresponding guide for the latest dagger versions (since the switch to SDKs). What is the current state of the "run anywhere" capabilities of dagger? (PS: I have spinnaker for CI, so that's where I want to run dagger)
https://docs.dagger.io/1201/ci-environment/

[winter linden]

There are equivalents, but currently they are split by SDK… not ideal

• https://docs.dagger.io/114934/nodejs-ci

• https://docs.dagger.io/454108/python-ci

• https://docs.dagger.io/768421/go-ci

[mystic mantle]

We actually started using some Go 1.20 features even in the SDK. So retrograding would give us more work to do it.
It might be less ideal, but having a dedicated go.mod for your CI shouldn't cause too much problem, I believe.
Can you write an issue with more details if it is really bugging you?

[main spear]

Just FYI i just wrote a blog post on our approach for using shuttle and developing with dagger: or rather how we've organized our developer tooling https://blog.kasperhermansen.com/posts/distributing-continuous-integration/

When I mention plugins and libraries think golang libs wrapping dagger. I know the title is a bit long

I will do a followup at some point on shuttle golang and dagger specifically

[warm temple]

For those who prefer to learn by reading code, we just merged a glow-up for the examples repo with links organized by "dagger core concepts" so that you can easily find working examples of specific dagger concepts in each langauge: https://github.com/dagger/examples#core-concepts

[bold canopy]

is dagger go sdk able to run docker-compose files in source directory. Since Jenkins is used in many organisations, there should be some conversion tool for jenkinsfile to dagger sdk initially typescript or golang

[slender star]

Hi 👋 @bold canopy. There was a demo at a recent community call on that topic. Incomplete and experimental at this point, but could be made more full-featured. Love to hear your impressions and needs 🙂

Here's a link to that demo discussion thread. You'll find GitHub repo and video link there cued to the correct spot: #1084672593329668187 message

Jenkinsfile to Dagger

[swift barn]

can rust workspace be a alternative of dagger?

[main spear]

@swift barn you can use the crate dagger-sdk if you wish to. It isn't a first party sdk though so it may have a few rough edges

In its github repo you can seebhow to get started with it

[loud briar]

Have been starting to experiment with services and seeing some undesirable behaviour - services that wait for requests need a with_exec else they fail the initial healthcheck, even if an entrypoint exists or is provided, but when a with_exec is included they start up and wait indefinitely for incoming requests, which blocks the non-service container from starting. Is these expected outcomes?

[mystic mantle]

It is the expected behavior based on the current implementation, but it is clearly not a great UX. We want to improve it because it is confusing: https://github.com/dagger/dagger/issues/4833

[heavy karma]

I have just opened a PR to merge these: https://github.com/dagger/dagger/pull/4937
Ongoing task for single-track docs: https://github.com/dagger/dagger/issues/4574

[fresh depot]

Soon 👀

[winter linden]

awesome ❤️

[solar raptor]

👋 hello @chilly arch @slender star @winter linden 😄 (this is Luke Marsden, I forgot I used my old IRC nick for discord lol)

[chilly arch]

Thanks for joining! We look forward to seeing how Bacalhau can integrate Dagger 🙂