general

[heavy gazelle]

Care to submit your conclusion @harsh ore on Dagger vs Waypoint as a PR against https://github.com/dagger/dagger/blob/main/docs/core-concepts/1220-vs.md ?

[sharp marsh]

sure. Will send something today 🙏

[heavy gazelle]

I was thinking that @harsh ore might want to capture his conclusion in our docs, but that will also work 💪🏻

[sharp marsh]

oh! that'd be even better 🙌

[swift barn]

somehow the dagger lsp command is not available anymore in 0.2.27

It was there in 0.2.26

its now cuelsp

[winter linden]

Yes we clarified naming and open-sourced https://github.com/dagger/cuelsp

You can call cuelsp directly, it is embedded in dagger only as a convenience

We want cuelsp to be useful to all cue developers, whether they use dagger or not. This naming makes that clearer.

[swift barn]

cool. thanks. The vscode extension seems to work also now. 🚀

[dreamy geode]

Hey! First of all thanks for your awesome software! Is it possible to push multiple tags with the docker push action or do I have to push using multiple actions?

[swift inlet]

Hi, thanks. Glad that you like it 😇 . I believe it's related to: https://github.com/dagger/dagger/issues/2774. The issue contains a workaround until we find a nice design pattern (your opinion is welcomed on the issue btw)

[astral shale]

Nice work! got the dagger extension and lsp running in vscode.

already a huge improvement to DX 🙂

[chilly arch]

@hereHi everyone, today we’re happy to introduce Dagger for VS Code, with native CUE LSP support. We’re also open-sourcing our CUE LSP implementation, to make CUE development easier from any text editor or IDE.

Full blog post: https://dagger.io/blog/dagger-vscode
VS Code extension: https://marketplace.visualstudio.com/items?itemName=Dagger.dagger
CUE LSP implementation: https://github.com/dagger/cuelsp

Special shout-out to @wispy tapir and @swift inlet for all their work on this!

[rigid epoch]

I look forward to trying this -- I gave up on Dagger while trying to do a POC because I had a hard time getting things working, I think this is a huge step in the right direction

[hardy sparrow]

the lsp is a huge win

[chilly arch]

@rigid epoch Thanks for giving us another shot! We'd love to get your feedback, so please let us know how it goes when you try it out. If you get stuck, our team will be happy to help you.

[rigid epoch]

will do! I might wait a little while yet -- I know that docs were high on your list of priorities before (I spoke with Erik last time), have you made big changes there?

[swift inlet]

Have you seen the new dagger.#FS docs (part of the core-concepts) https://docs.dagger.io/1247/dagger-fs ? It might interest you 😇

[rigid epoch]

oh, this looks great!

[chilly arch]

Our team is working on a new design for the doc experience and writing some crucial guides right now. We are hoping to release the new version very soon! I'll make a note to DM you when we do @rigid epoch

[rigid epoch]

ooh, thanks!

[arctic edge]

Any Dockerized Django with database(MongoDB) app examples here?

[sharp marsh]

@hereHappy Friday community! And what best way to celebrate but to ship a new feature in Dagger Cloud. For those users that have their engine connected and sending telemetry to Dagger Cloud (https://docs.dagger.io/1241/dagger-cloud/), we've just released a feature that will now display the caching status for your actions. This will help you understand better where and why some actions in your CI / CD process could be taking more time than expected.

As always, if you have any feedback or suggestions feel free to share them with us here https://github.com/dagger/dagger/discussions/2783#discussioncomment-3262864 so we can improve dagger to make your work easier.

[drowsy glacier]

大家好

[coarse jolt]

let's see if dagger can be a fit for Prometheus: https://github.com/prometheus/prometheus/pull/11107

with the dagger lib here: https://github.com/prometheus/promci

[stuck wyvern]

Great work on the cue LSP!
Really nice to be able to jump to definitions easily.

[chilly arch]

@here I'm happy to announce that our team has revamped the Dagger docs experience, making it easier for new users to get started. Please take a look and let us know what you think!
Dagger Docs - https://docs.dagger.io/

You'll find updates like:

• Improved search, so you can find what you need faster
• New templates to see Dagger in action with a particular stack like Node.js or Go -https://docs.dagger.io/install#explore-our-templates
• New and improved content such as https://docs.dagger.io/1247/dagger-fs, https://docs.dagger.io/1201/ci-environment, and more!

Thank you to @slender star @glad jolt @mint axle and @swift inlet for all their work on these updates! We look forward to bringing y'all even more new content soon!

[verbal canopy]

https://docs.dagger.io/1247/dagger-fs/ is really good!

I've worked with Bake, Earthly, and Dagger, and this is miles ahead of any documentation I've read on BuildKit.

[chilly arch]

Thank you for the kind words! This helps keep us motivated. @swift inlet has been doing an amazing job documenting our core concepts. If you have any other ideas or needs for specific topics, please let us know.

[stuck wyvern]

Was about to upstream an action for inject go into an arbitrary image (in the go package) but naming is hard.
I was calling it #Install, but as I'm working to add it I realize "oh there's already actions for test, build, <etc>"... go install is a thing already and a #Install action that is not go install is confusing.

[sharp marsh]

go.#Download? (following the convention mentioned here https://go.dev/doc/manage-install). I'm not sure I'm following exactly what injecting go into an arbitrary image actually is. Does in mean installing go there? If that's the case, is it really possible to do it on any image? What happens if you get a scracth one? Will the action also provide the tooling to install go?

[stuck wyvern]

It just takes either an image ref (assuming go is at /usr/local/go) or a dagger.#FS with go in it. Copy that to the input image and add GOPATH, and go bins to $PATH.

[sharp marsh]

Ohh I see. Maybe something like go.#Configure might be more appropriate? As you said, naming is hard 🤪

[fallen moat]

Hi, are there any examples of multi-stage Dagger pipelines on Gitlab? I want to be able to trigger some jobs of the pipeline using the Gitlab UI

[mystic mantle]

What do you mean by multi stage Dagger pipelines?

• a dagger pipeline with many actions that is run via a gitlab ci plugin?
• a gitlab ci configuration with multiple steps running a single dagger action at each step?

[willow rune]

Hello, Dagger has portable version on Windows ? I cant install anything new on my corporate laptop but I would like to check this Dagger

[verbal canopy]

Yes, it's a single binary.

But you do need access to a host running Docker.

[willow rune]

but the Windows install says use choco or ... other ways to install

Docker is allowed to run our corp laptop

where can I download the binary exe for Windows ?

[verbal canopy]

https://github.com/dagger/dagger/releases/download/latest/dagger_v0.2.30_windows_amd64.zip

[willow rune]

many thanks

!

I also thought that running CI\CD locally can be keypoint because too many problems can occur with remote hosted CI\CD

and an developer not interested in why a software can not be build and released on the hosted CI\CD

too many problems ok guys I made a specific version building locally and take it I dont interested in why the corp CI\CD not enough flexible

@verbal canopy link is broken

[sharp marsh]

Here you can find all the release binary assets: https://github.com/dagger/dagger/releases

[willow rune]

@sharp marsh many thanks

[still gull]

Hello everyone! There' a new package available in the alpha universe: JReleaser 🚀

https://github.com/dagger/dagger/tree/main/pkg/universe.dagger.io/alpha/jreleaser

A big Thank You to @slender star for reviewing & merging 😁 🎉

[safe storm]

Hi, I am still confused on how to integrate dagger with Jenkins, in case for dagger to talk to on-prem docker registry with self-signed certificate. Here is the doc url: https://docs.dagger.io/1224/self-signed-certificates/. How to create some Jenkins stage task to run this command only once on Jenkins agent host? Need to figure out how to only run this command in some Jenkins stage shell task only once on each Jenkins agent host? ```docker run --net=host -d --restart always -v $PWD/my-cert.pem:/etc/ssl/certs/my-cert.pem --name dagger-buildkitd --privileged moby/buildkit:latest

[winter linden]

In case anyone is at vmware explore tomorrow in SF: https://twitter.com/solomonstre/status/1564855297855741953

Solomon Hykes (@solomonstre)

Tomorrow I will be presenting at #VMwareExplore DevOps Loop. See you at the pop up theater, 11am. There will be live demos of unfinished code lovingly prepared by the @dagger_io team. Rumour says the one and only @BettyJunod may even be attending 🤞

Feels like old times!

Twitter · Aug 31, 2022, 5:59 AM

[grim dawn]

see you there!

[winter linden]

Glad you can make it 🙂

[winter linden]

Our demo will be streamed live at 11am pacific: https://m.twitch.tv/vmwaretanzu

[wet fox]

Running Dagger with self-signed certific...

[cosmic maple]

VMwareTanzu - Twitch

[lilac thicket]

Well done @winter linden 🙂

[chilly arch]

@here The team has been working hard on some new capabilities for Dagger. @winter linden is at VMware Explore right now, and just did a demo to showcase the multi-language support that we intend to bring to Dagger.

You can check out the demo and interview here:https://www.twitch.tv/videos/1578035654

The project is currently in private beta, but you can sign up for access through the private beta form at https://dagger.io/cloak.

NOTE: he had some technical difficulties at the start, but the VMware events team fixed it at the 16:40 mark 🙂

[stuck wyvern]

Great demo.

[tender goblet]

Indeed! Really excited to see how Dagger is evolving. Where Pulumi, my employer, is adding programmability to cloud infrastructure leveraging regular programming languages, Dagger is taking programmability to delivery flows. 👏🏼

[supple hornet]

Hey @chilly arch is the community call later today going to include the proper demo that was intended for VMware Explore? I was quite excited by the little bits we got to see 😄

[chilly arch]

@supple hornet we most likely won't do the full demo, but @winter linden will be there to dive into any parts that you'd like to see again and our engineering leads will be there as well to answer any questions!

[supple hornet]

Thanks!

[tulip rune]

is the new docs site using docasaurus?

[glad jolt]

Yep! You can read the source code here
https://github.com/dagger/dagger/tree/main/website

[stuck wyvern]

Just used core.Source for the first time... that's a nice little thing to have.

[winter linden]

If anyone here signed up for early access to Project Cloak and didn't get in yet, please DM me or @slender star and we'll give you access. If you're active enough to have joined our discord, you get priority 🙂

[wet fox]

I just realized that I lost the today's cloak zoom call 🤦‍♂️ . Is there any chance I can see the recording of it (if there is any)?

[winter linden]

Yes we switched to zoom specifically so we could record 🙂 Will share soon

[wet fox]

Thank you!

[stuck wyvern]

@winter linden I'm curious how dagger.io and universe.dagger.io are hosted. I see they seem to be on netlify but redirect to github.

[stuck wyvern]

Oh I see it's like go vanity URL's.

[sharp marsh]

yep, this. We have some netlify redirect rules so you can easily browse current actions 🙌

[modest nova]

I'm definitely curious to check this out 🙂 only requested access yesterday but eager to try!

[stuck wyvern]

What other magic is there? I see the redirect page for universe. Trying to replicate this but it's not working and I don't see any docs for this for cue.

[winter linden]

Some of it is hardcoded in the dagger source code

[gilded vortex]

Thoughts on having a .cue file for frontend and backend repos? I'm working with a mono-repo and initialized the dagger project at the root. Squeezing in a ton of commands for both backend/frontend into a cue file seems to be getting a bit messy

[stuck wyvern]

For multiple dagger.#Plans?

[winter linden]

@stuck wyvern it's all here: https://github.com/dagger/dagger/blob/main/pkg/pkg.go

At the moment the actual contents of universe are bundled in the dagger binary

But the UI assumes that in the future it will be moved out and downloaded as needed

This is where the embedding happens when you build the binary: https://github.com/dagger/dagger/blob/main/pkg/pkg.go#L23

[stuck wyvern]

Hmm, I see. So it's never actually fetching universe.

[winter linden]

Well it's "fetching" it from memory 🙂

The plan is that it will eventually fetch it from a repo instead - or that it will hand off to native cue package management to do so, whichever comes first

[gilded vortex]

yep

[stuck wyvern]

lol, I have so much indirection or something in this build I'm working on, it's taking forever for dagger to figure out what to do.

[slender star]

Interest piqued! Do you have a Dagger Cloud link?

[stuck wyvern]

Let me bring back the abomination and push it to a branch

[stuck wyvern]

I don't know if dagger cloud will work because it wants to call back to the client but the client is not local for me.
Here's the plan: https://github.com/cpuguy83/buildinfra/blob/abominiation/runc.cue

[raven quest]

Has anyone explored running macos builds? We have some CI steps that need to run on OS X, but it doesn't seem possible to do that with Dagger.

[sharp marsh]

Has anyone explored running macos builds

[stuck wyvern]

The vscode cue/dagger introspection has been really off the last couple of days, I'm guessing there was some update that broke something.
Sometimes I'll try to inspect some type and it just gives nothing at all (for dagger/core this seems to be 100% broken for me). Other times it takes me to the wrong type.

[swift inlet]

lsp broken

[verbal canopy]

Anyone looked at this yet? https://github.com/ktock/buildg

It would be awesome to have this functionality with Dagger

[sharp marsh]

yep, I saw that. It's a nice idea indeed. We have some initiatives that might help with debuggability considerably; stay tuned! 🙂

[frozen basin]

Anywhere yall are talking about cloak? Or is that only after EA invites go out? 🙂

[winter linden]

@frozen basin there's a separate discord channel so as to not confuse the hell out of everyone 🙂 Did you sign up for early access? We're sending another batch later today

[frozen basin]

Yup just signed up!

[winter linden]

@slender star I'm guessing you're already on it? 🙂

[slender star]

yep!

[winter linden]

If anyone reading this has signed up, and doesn't get access within 1-2 days, let us know

We're prioritizing people already active in discord or github

[sullen thicket]

Hey, I've been experimenting with Dagger recently. Wondering if there's any thoughts on how to structure a CI/CD pipeline using Dagger in GitLab - should I make a single Dagger action that does all the steps (e.g. test, lint, build, deploy), or multiple actions and therefore multiple GitLab steps? I can see pros and cons either way; a single action for all those things would probably require explicit chaining which Dagger doesn't natively support yet (aside from the workarounds), but multiple actions would require more GitLab CI to be written and so it feels like it loses some of the advantages of the decoupling/'run anywhere' philosophy. As far as I can tell, single action would also mean the whole pipeline is just represented as one step in GitLab CI which feels suboptimal.

[swift barn]

are there any strategies for working with local things? Like using a tool in a container to generate some output you want to use locally. Similar to something like

docker run -v $PWD:/work my-tool generate

I feel like this capability would be nice. Currently I am still using makefiles for this kind of stuff. But I wish I could drop the makefiles and centralize everything in a dagger plan.

[slender star]

are there any strategies for working

[swift barn]

Hi, is there a way to know what action was invoked by dagger do ?. I want to do this to do some conditionals based on the choosen action. The reason is that there is this issue that the pipeline fails if not all env variables are set as soon as one action touches the client env. So the idea is to declare the variables conditionally based on the chosen action.

[swift inlet]

knowing action invoked

[mystic mantle]

Dagger Project structure in GitLab

[frosty oak]

Hi, I am trying to with dagger to replace bash + makefile and have a small problem that need help. There are a few tasks that would want to see the action stdout output, ie:
output of terraform plan to decide if I should apply it or not, task that reports list of vulnerable packages ... I found out about action output https://docs.dagger.io/1228/handling-outputs#controlling-the-output but it is not suitable for terraform plan output, or multiline string in general. Log format doesn't help much as well.

I can work around this by writing output to a file, copy it back to client and ... use cat (bash script) to show it out but that sounds way too complicated and kinda defeat the purpose of using dagger in the first place.

So could u point me to any reference where I can see stdout of an action.

[swift inlet]

Hi, to see the stdout of an action, you currently need to dagger do --log-format plain

[gritty star]

Dagger Presentation at CloudNativeDay Bern 🌄⛰️ with @heavy gazelle
https://cloudnativeday.ch/an-electric-automation-engine/

[chilly arch]

@hereHi Dagger Community!

You may have heard the buzz around a mysterious “Project Cloak”: an experimental project to add multi-language support to Dagger. The experiment started in a separate repository, but we are now migrating it into the main Dagger repository. “Cloak” is the codename of our next major release of Dagger, and going forward we will be developing it in the open.

You can watch Solomon’s demo to learn more: https://www.twitch.tv/videos/1578035654

The stable Cloak release is still several months away. You can continue using Dagger as usual, and we will offer backwards compatibility in the future.

Here are a few ways to get involved:

• Check out Cloak in the Dagger Repo
• Create issues with your feedback using the cloak label
• Join the conversations on the #maintainers channel in Discord
• Join the weekly Community calls on Thursdays at 10 am PDT. You will see event reminders on the event section on Discord. If you’d like to get an invite directly to your email, please DM me.
• Share your use case for a multi-language Dagger! We love to talk to our users, so please feel free to reach out to me, so we can schedule some time to talk!

**Please remember: **

Project Cloak is still under active development, and you will certainly encounter bugs, confusing behavior, and incomplete documentation. Please tell us everything!

Thank you again for being an active part of the Dagger Community, and we look forward to building this new release with you!

[marble vapor]

Glad this is coming so quickly! Been looking forward to it

[winter linden]

Since we're developing multi-language support for Dagger, I'm curious: if you could choose any language to develop your pipelines with the Dagger engine, what would it be? (vote in the thread)

[swift barn]

would it be possible to redact strings (from error messages) that are wrongly passed to Secret fields? Because if the user did something wrong and tries to add a string from something like a client command stdout to a secret field, cue will throw an error and say that value "supersecret" doesnt match the type. And then it goes off into dagger cloud.

[native hinge]

cloak & dagger, har har

(j.k - look forward to it!)

[winter linden]

We do have secrets redaction, to avoid exactly that problem. You may have found a bug in that feature

[swift barn]

If everything is done correct, secrets are redacted. But if you try to pass a string to a field that is supposed to be a secret. You get an error, and then its not redacted. https://dagger.cloud/runs/3558af9d-84bd-4847-b6fb-50e2f5a475f4

string may come from a client command that wasnt configured correctly.

[slender star]

If everything is done correct secrets

[boreal tide]

is it possible to pass on all env variables from the client environment that are of certain wild cards say abc_* to dagger workflow actions? I don't want to add each and individual env

[winter linden]

Not an ideal way, but you could execute the command env on the client, then split the result by line (with client: commands)

[karmic sand]

@winter linden check this out https://www.the-guild.dev/graphql/codegen/plugins/typescript/typescript-oclif

[alpine wyvern]

Hey !
I'm the product owner for a small project in a large Data Science consultancy, and I often have to setup CI/CD for our clients. I was very eager to try out Dagger as an improvement over rewriting CI pipelines every times. However I have a few questions I could not find the answer to (I hope you don't mind!) :

• How do I manage concurrency in CI with Dagger ? For example, I'm using Circle CI with separate jobs for linting, building the docs and testing the code. They all get executed in parallel over different containers with their own resource pool. How would that fare if I invite dagger in ? Could I still benefit from multiple resource pools ?
• How do I cache stuff (like deps, venv, etc) ? Do I still have to use the CI provider's caching mechanism ? To be honest that's 1/3 of all I have in the workflow (alongside fetching code, secrets, and recording artefacts)
• If I disregard the CI completely, would dagger be a better fit to replace Airflow and other schedulers ?
• I was about to say that the cue lang was really putting me off, but I read that cloak could make it available in other languages; so, everything's fine 🙂

[chilly arch]

@here Hi everyone! The Dagger team is currently having a great time at our team retreat in SF. As some of you may know, we don't have the community call this week since we are at the retreat. We look forward to seeing all of you next week! Use the link below to easily add next week's call to your calendar: https://discord.gg/aUJws773?event=1022258709260410982

If you would like to add a demo, topic, or question to the agenda, please add it here: https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit?usp=sharing

[young stag]

Hi. I’ve a quite generic question; does dagger offers a (RESTful) API or smth like that? I‘d love to make an UI so jobs are visualized :)

[carmine sandal]

anybody has any good advice/resources/tutorial on how to setup Dagger that requires interaction with other Docker containers for e.g. https://localstack.cloud

[wraith niche]

web api

docker containers

[chilly arch]

Sorry we missed your note! Welcome to the Dagger Community! Our team was at an off-site. @winter linden or @wraith niche do you mind answering their questions since the team is traveling? @alpine wyvern I'll DM you shortly to set up some time to learn more about your use case 🙂

[winter linden]

Data science use case

[swift barn]

Hello, Dagger newbie.
Although I understand that any OS that supports DevKit can be used to run a dagger pipeline, what is unclear to me is how to build binaries for macOS as a target (with macOS as a build host, I don't even need cross-compilation). Do I need to use QEMU or is there a more direct, more performant way? A quick search of the documentation didn't help, maybe I missed something? Thanks 🙂

[pseudo stream]

Hello Dagger newbie

[tall gulch]

Hi, I'm looking for a workaround for explicit dependencies with client commands https://github.com/dagger/dagger/issues/940, or other approach to achieve dependencies between actions which exec into containers. The containers are managed outside of dagger.

[oak quest]

Tonight, Europa will be visible with binoculars!

https://twitter.com/astro_jaz/status/1574186492104445952

Jasmine 🌌🔭 (@astro_jaz)

Tomorrow, an incredibly rare, once-in-a-lifetime astronomical event is happening! Jupiter will be its closest to Earth since 1963! Along with opposition, it’ll be SO bright, you’ll be able to see its bands and some moons just with binoculars!
This won’t happen again until 2129!

Likes

152197

Retweets

25594

Twitter · Sep 25, 2022, 11:58 PM

[winter linden]

Good to see you again here @oak quest 🙂

[oak quest]

and you @winter linden ! I have a real world need for Dagger atm

did I inadvertently leave #dev or was that channel removed?

[winter linden]

Removed and replaced with #maintainers

[oak quest]

ah

[winter linden]

It was turning into a catch-all

[chilly arch]

@oak quest nice to meet you! @slender star and I would love to learn more about your use case. I'll DM you to find a time to talk! We have a community call coming up on Thursday too, so it'd be great to see you there -https://discord.gg/6R866YsK?event=1022258709260410982

[oak quest]

@chilly arch nice to meet you as well . I was the first engineer hired, but I think I left before you got there.

[rain oriole]

Yes, very glad to see you @oak quest!

[oak quest]

and you! and @cloud canyon , and @wraith niche 👋 wish I could have made it out for social hour last week, but I started a new gig the week before 🙂

[wraith niche]

glad you found something that suits you!

[chilly arch]

@here Our first official community call with the entire Dagger Community is on Thursday. If you would like to add a question for the group, a topic, or even a demo, please feel free to add it here:https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#heading=h.amkecn1w3n3t

These calls are the evolution of the Cloak Community Calls that some of you have already been attending. You can easily add this week's call to your calendar by using the link here: https://discord.gg/6R866YsK?event=1022258709260410982
(if you would like to have the reoccurring event added to your calendar through a google invite, please DM me your email, so I can add you)

[rain oriole]

So cool... I was able to see 2 small dots next to Jupiter with the binoculars 🔭

[winter linden]

Man... here in San Francisco the night was too cloudy. I didn't see anything 😦

[oak quest]

I got wrapped up in Rings of Power and forgot 🤦‍♂️ . will try again tonight

[slender star]

We could see 4 moons in Portland, OR last night with big binoculars

[winter linden]

https://tenor.com/view/zach-galifinakis-hangover-hangover-alan-alan-hairflip-gif-13142173

[winter linden]

Hoping you’re safe and have a full pantry @stuck wyvern and everyone else in the path of Ian..

[stuck wyvern]

❤️ Thankfully I moved to Seattle several years ago 🙂

[winter linden]

oops, sorry 😅 parasitic memories

[oak quest]

What's the current state of Dagger CUE support? I'm needing to deploy AWS resources which may lead to contributions to universe. Is that a worthwhile endeavor at this point?

well well well what do we have here? 🙂

I see a few different extensions for Cue, some more recently updated that others. Anyone have a particular favorite?

[stuck wyvern]

I hear dagger is pretty good.

[slender star]

Only have used the Dagger one with CUE support 🙂

[chilly arch]

@here the community call is going on right now, so feel free to join! @pseudo stream is doing a demo on multiplatform support right now.

Join Zoom Meeting
https://dagger-io.zoom.us/j/89360226840?pwd=aEZqU2Zsai9ScXhCcUwyZlp4RUpvQT09

Meeting ID: 893 6022 6840
Passcode: 652511

[timber sphinx]

Is this recorded? I'd like to catchup and listen to a few of these but have a work call that's starting and conflicts.

[chilly arch]

yup! I will share the recording as soon as it is done processing.

I will always share the recordings after the calls on this Discord channel, but you can find the previous ones in our community call notes. -https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit?usp=sharing

[swift barn]

Hi, if you have a multi stage docker file, and build one stage first and then later the full image, does the caching work as in normal docker, meaning it doesnt need to do the things from the first stage anymore?

[winter linden]

Yes I believe caching will work as expected

[sharp marsh]

yep, this is correct, I was doing some work with docker.#Build today and effectively works like that

[swift barn]

ok thanks.

its strange because I feel like dependecies are pulled twice.

[sharp marsh]

maybe something's invalidating the cache. Happy to 👀 if you can share

[swift barn]

sure, let me login to cloud and share a run

so here is is the first run installing the dependencies https://dagger.cloud/runs/2d78169a-4e7e-41db-95b1-e0631a33be1f and using that stage for testing. And here is the second run building the final image and it pulls the depencies again. https://dagger.cloud/runs/f171688f-d6d3-4e84-8e45-923ee3c40892

its also suspicous that vet and test take 41 seconds. They take less than a second locally.

[sharp marsh]

that's probably because locally you're hitting the go test cache? which doesn't happen here since you're not using a mount cache for that

[sharp marsh]

it'd otherwise say cached here

[sharp marsh]

looking into the Dockerfile thing now. Starting a thread for that

[sharp marsh]

looking into the Dockerfile thing now

[chilly arch]

@here Thanks to everyone who joined the community call live today! If you weren't able to make it, you can check out the recording here 🙂
https://dagger-io.zoom.us/rec/share/-U0ei4-0DkwRL46pvPRgTJUTgDUExwES7MynXp89AvF-WaSXgMq5sQTw-drL9FvV.rCzcyljP8wF0D9Pr?startTime=1664470683000
Passcode: e8CN2dY&

Add next week's community call to your calendar with the link below: https://discord.gg/FjdxRdyY?event=1025170650752630854

[stuck wyvern]

Phew, gonna have to get a bot to clear my inbox from the dagger community call invites.

[chilly arch]

Sorry, was updating the invite to change the wording from cloak preview to community calls. I clicked "don't send" for sending updates to the invite list, but go to know that button doesn't actually do as it says 😂

[rigid epoch]

Does Cloak support storing non-image outputs as OCI artifacts?

[winter linden]

Sure

except for custom content type

so registry will gladly store it but think it’s a regular container (probably better that way for compat)

[pseudo stream]

Do you mean artifacts as in: https://github.com/opencontainers/artifacts ?

[rigid epoch]

I think that's what I was thinking of, but not sure 😅 I've only just begun reading about using container registries to store non-container content and I'm thinking about ways my team might want to do that

And I was curious how Dagger handled that / will handle it in the future

[pseudo stream]

Haha yeah it sounds like that's what you're describing (helm charts are a common example I think, but now also SBOMs and some other exotic things).

We don't have built-in support for any of that (pretty much just because buildkit doesn't have built-in support for exporting generic oci artifacts). But I do suspect it would be straightforward to implement this in a user extension; you could just import an oci registry library with artifacts support (in whatever language) and then append the artifacts to whatever you want.

That's very theoretical because I have not tried it before though, so take with salt. If you have some specific examples of artifacts your team might want to utilize that would be helpful to get a better idea

[rigid epoch]

I was thinking of binaries, frontend assets, that sort of thing. Really anything that isn't an image, but not any specific standardized type

[winter linden]

Honestly you can do almost all of that without messing with the content type

So my guess is that if you control the tooling on both sides, you can already fo this with dagger.

If you don’t control the tooling and need to use tools that rely on recent extensions of the OCI format (like mew content types) then :

1. like @pseudo stream said it’s not supported natively but probably could be soon
2. you can just wrap the particular tool you want to use in a dagger pipeline, and let that tool deal with the OCI extension stuff

TLDR:

• You can probably do it now with some limitations
• Eventually there will be no limitations

Actually reading that “artifacts” repo… I’m really not conviced you need that spec at all to achieve your goals

It becomes useful (I think) when multiple vendors need to agree on content types for interop, to reuse each other’s artifacts and metadata

I say shove your artifact in a dagger directory; add a json file with your custom metadata if needed; push it as a 100% vanilla container image with your artifact as rootfs; done.

if in the future your tool doing this gets millions of downloads, then OCI will need to adapt the spec to be compatible with your thing. Kind of the true purpose of OCI in the first place 😁

[slender star]

Earlier today we were working with a user that was pushing Helm charts from Dagger using helm cli inside a container like this:

helm push helm-test-chart-0.1.0.tgz oci://aws_account_id.dkr.ecr.region.amazonaws.com/

https://docs.aws.amazon.com/AmazonECR/latest/userguide/push-oci-artifact.html

[chilly arch]

@here time to add your topics, demos, and/or questions to the Dagger Community call agenda 🙂 Feel free to add them to the doc below: https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#heading=h.9t6gqibgvt7n

See you at the community call on Thursday at 10 am PDT! You can add it to your calendar using this link: https://discord.gg/jEbTGnbt?event=1025170650752630854

[timber sphinx]

I wanted to restart a run through of dagger for a project. Should I do it from branch with cloak or normal install is fine? I couldn’t find the Hugo demo someone had done a while ago so post a link if it’s still around and relevant. Cheers

[winter linden]

It depends on the timing of your project, how you feel about using Cue, and how excited you are about a native SDK in another language

[timber sphinx]

Using a Go sdk would be cool if that’s included. I’m just looking to catch up as things have changed since I last experimented.

[winter linden]

I recommend joining the community call tomorrow and checking out the Go SDK demo 🙂 If you have a test project in mind, we’d love to discuss it on the call too

[mystic mantle]

Hi, I was the one doing the Hugo demo. I need to make sure it still works (we're moving a lot of stuff around). If it is, I'll ping you once it's published.

[chilly arch]

@here reminder that the Dagger Community call starts in 10 minutes. See you there! https://discord.gg/jEbTGnbt?event=1025170650752630854

[timber sphinx]

Best place to post ideas for consideration? Github discussions?

[winter linden]

Github discussion is excellent. You can also start here in #general or in #maintainers if it's specific to the future cloak release. Whatever you're comfortable with

Github discussion thread is more durable and easier to find for sure

[swift barn]

Hi, I am still looking for a solution to integrate with docker compose. This is a real showstopper for my team since they want to run integration tests utilizing their development compose file. I cant promote dagger without having a solution for that :/

[sharp marsh]

@swift barn docker-compose you mean the ability to start support services before running the CI pipeline?

[swift barn]

yes, It wouldnt be too bad, if dagger would not run with network mode host by default without option to change that behavior. Because then you could start compose outside of dagger and join the dagger container to the same network.

Although we use buildkit directly on our agents, I am not sure how this would play out in that case.

[sharp marsh]

yeah.. I see what you mean.. I think there's an issue somewhere to focus on the networking/connectivity aspects. We know we want to tackle it soon, thx for raising the concern again 🙏

[slender star]

@pseudo stream isn't that host network behavior different between Dagger 0.2 and Dagger next (cloak)?

[chilly arch]

@here Here is the recording from this week's community call:
https://dagger-io.zoom.us/rec/share/LsqMc7BtFVM-SzhLqeRJzMytpUte6giTUCbDpT4JkY5gstuSYI-SmoZ8iIWrGT9d.EcH3TSQwdGX-U9gR
Passcode: ?=.^q72D

Add next week's meeting to your calendar here: https://discord.gg/jEbTGnbt?event=1027696167192117259

[chilly arch]

During the call, there were a few key issues/prs/discussion posts that were mentioned and put in the zoom chats, you can find them noted here:

• https://github.com/dagger/dagger/issues/3065#issuecomment-1259699645
• https://github.com/dagger/dagger/discussions/3276?converting=1
• https://github.com/dagger/dagger/pull/3191
• https://github.com/dagger/dagger/pull/3123

[pseudo stream]

Yeah we don't run buildkitd in the host network namespace in cloak right now, no intention to change that unless there's a compelling need that arises

[mystic mantle]

Has anyone here some experience with scaleway elastic metal?

[sharp marsh]

what's the default/current model? veth pair bridged network?

[sharp marsh]

I though we used host since otherwise we'd have to use CNI mode?

[pseudo stream]

To be clear I was referring to the namespace buildkit itself runs in when we autostart it via docker run. So I don't remember docker's default but I would have guessed veth.

The network namespace that buildkit runs its containers in is a different issue

[pseudo stream]

Yes the buildkit docker image does not come with CNI configuration so all its containers are in the "host" ns, which is actually the docker container's netns 😵‍💫

[sharp marsh]

I see, got it. So yes, docker will run buildkitd in the default bridge network basically

[pseudo stream]

Ah yeah bridge

[swift barn]

Would be nice to have something like vscode devcontainer where you can point it to a compose file and tell it which service of the compose thing is the target. And then it starts the whole stack and uses the target, in this case some builkit image. That way you could have dagger run and use tests commands, in your actions, against sidecars in the network(s) configured by compose.

[faint sail]

Is there a design document for cloak (direction, reasons etc)?

[winter linden]

There’s a readme & new docs in development in the cloak branch: https://github.com/dagger/dagger/tree/cloak

The short version of “why” is that our number one user request, by far, is “support more languages”

[faint sail]

Config languages or target languages?

I guess that means even tighter integration

[winter linden]

“I want to develop my CI/CD pipeline in Go/Python/Javascript/Bash”

[tall gulch]

https://learn.acloud.guru/series/serverlessconf-nyc-2019/view/yaml-better at 14m22s: `But I wanna always code in my favorite language!" : "So grow up! Here is the thing, your comfort zone is not the point!"

[faint sail]

I'll check it out. Looking at the examples and core extentions for cloak (even after code first) being so language agnostic makes it feel not very specific language friendly.

The go code feels more graphql than go. Then there is the idea of compiled code masquerading as a scripting language.

All of this made me wonder more about the underlying decisions and hoping there was a design doc that stepping through the idea of graphql, etc.

So the idea of writing in any language makes sense. The implementation and real world use case seem more ambiguous.

[pseudo stream]

The exposure of graphql concepts everywhere is a temporary artifact of implementation; we are actively working on making improvements to all the SDKs to make them feel much more natural. For Go specifically, most bits of graphql should be abstracted away with this PR: https://github.com/dagger/dagger/pull/3125

[faint sail]

Nice.

[winter linden]

That is about infrastructure management, not CI/CD pipelines. They are related but not the same.

[timber sphinx]

I listened to the talk, but I’m not sure how that applies to dagger. This talk was specific to where in the abstraction for infrastructure should tools sit. While dagger seems to have some declarative aspects it seems more imperative sets of packaged instructions and less focused on desired state. Maybe create a thread of that doesn’t make sense and want to talk more about it? It’s an interesting topic. I’ve debated this in using Pulumi and other tools before and arrived at my happy compromises.

[tall gulch]

My link to the talk was more to suggest that users may not always know what is best for them. This is against a certain interpretation of the "Lean Startup", but I saw too often large features implemented because users were sure they needed them, with the end result that nobody used those.

[winter linden]

@tall gulch what’s your preferred environment for developing and running CI/CD pipelines? Have you used Dagger with CUE and if so, do you have any requests for improvements?

[tall gulch]

I think my favorite CI/CD environments were jenkinsfile with groovy and buildbot with python. Both allowed me to organize the pipelines into functions without yaml boilerplate, but the infrastructure was not reliable (jenkins plugins were fragile to updates). I tried dagger with cue, but found cue initially unreadable. I'll invest time in cue only if it solves my problem. My goal is to reduce the time spent on debugging the pipelines. In order to do so I imagine being able to selectively restart dagger actions using the original or modified (for debugging) pipeline code. I'm hoping that most, if not all traditional yaml pipeline constructs, even the more advanced ones (like https://github.com/dagger/dagger/issues/3149) can be represented in dagger somehow, and cloud identities propagated to dagger so I don't need to carry secrets around when running on a cloud (with an approach like https://github.com/kubernetes-client/python/issues/1005).

[timber sphinx]

So the last time I touched dagger was early last year at least. I asked last week about what version too use.
Hugo examples seemed to be in process of being updated, so instead...

Here's a recent enhancement I did for some of my $work public open source repos. (granted I only started using github actions with composite and such in the last few months, so maybe better way to do some of that).

release-composite: https://github.com/DelineaXPM/dsv-github-action/blob/main/.github/workflows/release-composite.yml

Now I'd like to replicate some steps, and I'm fine with whatever is stable and not subject to immediate breaking changes. If that's dagger classic with cue I'm fine trying that out. Most of the tasks are Mage driven. If that's better in cloak and the sdk is "reasonably stable" in that I could use it for a few months without worrying about entire sets of functionality being gone ,then I'll try cloak.

I also am going to look at putting trunk cli in this process as a container.

What route/branch etc should I give a shot and if there's no init to setup a starter plan file where do I grab the correct one?

No rush just want to give it a shot again!

---

Seperate idea/thought:

Might consider looking at trunk.io for perhaps a nice way to feature a linting toolchain/partnering or mentioning. Their community /slack group is very responsive and I've been rolling out trunk for formatting, linting, and triggering tests before push, removing pre-commit entirely. It's such a good experience compared to self-built toolchains of this stuff. Just thought trunk + dagger might be an interesting mix for the "chore" style lint/formatting/precommit style hooks.

[chilly arch]

@hereHope everyone is having a great week! Just a reminder that our community call is tomorrow at 10 am PDT.
https://discord.gg/VasygkaV?event=1027696167192117259

You can add any topics, questions, demos, etc. to our agenda here -
https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#

[strange jungle]

Hello everyone. I'm working at Cycloid, and we have built a product, a devops platform, that is currently using Concourse (choice made in 2017) for providing all the automation.

I'm currently happy to see that @elfin frigate joined Dagger project, but a bit sad also as that means Concourse lost his captain/father.
So I'm currently working on preparing the future, and as few customer told us they would like to use their CI/CD engine and not switch on Concourse, I think Dagger totally make sense for us to have an easy integration by having this engine abstraction.

Few questions come to my mind:

• what is the current plan with Dagger cloud (business model I mean)? I wouldn't like to start something on top of dagger with an UI etc, but finally compete with you and you'll close API/Engine etc ...
• do you think it would be possible to have a way to stream logs/plan of actions etc ? Maybe with an agent/wrapper 🤷 ? Is it something you plan to do with Dagger cloud ?
• @elfin frigate I saw you did an image vito/dagger-concourse, do you have a PoC working between concourse and dagger ? (that would be a first step to integrate dagger in our current ecosystem

Thanks 🙇‍♂️

[winter linden]

Hi Seraf! Thanks for saying hi. That’s a very interesting use case that I think could be a great fit for Dagger.

I will let Alex answer the questions addressed to him. As for the others it is quite late where I live, but I will share my perspective tomorrow. It is an optimistic one 🙂

Talk to you tomorrow

[elfin frigate]

hiya! I don't remember making a vito/dagger-concourse and can't find it now, but I am good at hiding things from myself. 😅

I have thought about Dagger as a potential landing spot for Concourse users, but don't have any work in progress. Concoure resources had a really simple container JSON stdio interface that can definitely run on Dagger, the only problem being they took arbitrary JSON payloads on stdin which a GraphQL can't represent in a nice way (there's no "arbitrary params" type) so using them directly with the Dagger SDK wouldn't be very ergonomic even if an extension were to be built for it. Concourse tasks would be pretty easy to run with Dagger though, and I also have an in-memory streaming implementation of Concourse's pipeline passed: constraints stashed away somewhere. Building on those primitives, it seems possible to implement a Concourse pipeline interpreter that runs on Dagger without a database or web UI. It'd be a decent chunk of work and I've got other distractions at the moment, but I'd like to see it happen someday and would be more than happy to help!

[slender star]

Thanks for the link to trunk, @timber sphinx 👋 I'll take a deeper look.

Looking forward to seeing you on the community call today. I think some things will become clearer. Excited to hear your feedback.

[chilly arch]

@here community call in 5 minutes - We have some updates to share on the Go SDK and our next release. See you there! https://discord.gg/VasygkaV?event=1027696167192117259

[rocky sage]

Hello to all here ! I am glad to join Dagger community

[sharp marsh]

Welcome Vadim!

[rocky sage]

🙂

[winter linden]

A few thoughts as promised @strange jungle ...

• Our business model is to sell subscriptions to Dagger Cloud - a collection of optional cloud services that improve the experience of using Dagger in a way that the open-source engine cannot.
• Our roadmap for Dagger Cloud is simple: identify the most painful obstacles encountered by the community when using Dagger, and look for the best solution. If it's an open-source engine feature, ship it. If it's one or more third-party products, integrate with them. If it's a cloud service run by Dagger: add it to Dagger Cloud. So it's completely driven by user pain, and identifying the best form factor.
• Yes, we plan on offering a way to stream telemetry from your Dagger engines to Dagger Cloud. This is a pre-requisite for several features that we already know we need to ship. In fact we already built this for Dagger 0.2: you can connect to Dagger Cloud and share a link to each run, for troubleshooting and support. We haven't ported that feature to cloak yet, but we will.
• We also want to support streaming telemetry from the engine to locations other than Dagger Cloud. It would be difficult for us to prevent it anyway, since the engine implementation and API are open source. Easier for us to embrace it: more integrations means a larger ecosystem, and we all win.
• There's nothing preventing you from forking the Dagger Engine and removing the link to Dagger Cloud entirely - but our goal is to make it more interesting for you not to do that. We rely on strict enforcement of our trademark (you can't use the TM on a modified fork) but not on restrictive licensing. The engine code and API will remain open.
• We expect many commercial products to integrate with Dagger Engine in ways that overlap with Dagger Cloud. That's just how the market works. We think that's compatible with a win-win relationship.
• The only scenario that would not be a win-win relationship would be if you made the entire strategy of your business to be "the enterprise Dagger company". This is because Dagger is not and will never be a vendor-neutral platform, like Kubernetes or Linux. We will never donate it to CNCF. It is very open, and very friendly to integrations and partnerships - but it is still owned by a company called Dagger. You need to be OK with that or you will be disappointed later.
• We will never close our API or open-source engine implementation - that would cause us to lose the trust of our community and ecosystem. FWIW we never did that at Docker either. We had non-open products, but they were all launched as such from day one. No bait and switch ever.

Hope this helps.

[strange jungle]

About Dagger Cloud

[timber sphinx]

Off topic - your developer environment and setup

[chilly arch]

@here Check out the community call recording from yesterday - The call includes a Go SDK demo and playground demo:
https://dagger-io.zoom.us/rec/share/guo7q5DobXhyXdKaBJc3RY7p55fIGV_HDWTpydlJy7v6H0dDXBPBWt9HiWD19tk.R15faI7hJGBs_CHS?startTime=1665680168000
Passcode: K?P#5.sL

Add next week's meeting to your calendar too. See you there!https://discord.gg/zS7QMzpm?event=1030545484428869672

[blazing sluice]

do you have another file available by any chance ? This is only a 35sec-long recording. Thanks.

[chilly arch]

ah yes, sorry about that! Updated the link above.

[ruby harness]

Hey Dagger Community,

I am Ganesh and I work as a full-time community person at Aviyel. I just discovered Dagger and thought I'd come around and be a part of the discussions. Meanwhile, if there's anything I could help with on the non-code side (I do have some ideas in mind) to spread the word, I'd be happy to volunteer.
Let me know your thoughts!

[winter linden]

Welcome Ganesh! Thanks for offering. I'm interesting in hearing your ideas 🙂

By the way I didn't know about Aviyel, I love the website. Very cute characters.

[hollow terrace]

Hi! Sorry I have been away for a while so missed the new development, is CUE going to be depreciated in the future and replaced with the new Go /Py/TS sdk?

[winter linden]

CUE will continue to work thanks to a compat layer. It will just no longer be the only language frontend available.

In other words, a CUE SDK will be available for the new engine, with the primary goal being compatibility with existing CUE configurations

[hollow terrace]

Thanks for clarifying!

[ruby harness]

Hey @winter linden thank you for your words
Happy to share the ideas with you

I have some ideas:

1. if you don't already have one, a contributor guide for anybody getting introduced to Dagger for the first time. This is the high-level idea: https://aviyel.com/post/3369. Here's what it would look like finally: https://bit.ly/contributor-roadmaps

2. An informal AMA session with our open source community https://www.youtube.com/watch?v=HQ9jYOFfCg0
   Let me know your thoughts on these ideas and would be great if you're available for a short call to discuss more.

[somber finch]

Hey guys, I just discovered Dagger and have a question around universe.dagger.io. Is there some place where I can see all the available Go modules?

[somber finch]

Is dagger do ci still active? I found this in the Azure Pipelines example https://docs.dagger.io/1201/ci-environment/

[swift inlet]

dagger do ci

[ruby harness]

Aviyel | Create a Contributor Roadmap fo...

[timber sphinx]

Pinging for an update in case it's up and running somewhere. Want to give it a shot for my hugo blog but need something to serve

[wary cave]

Are contributions to v0.2 still accepted? Seems a lot of things are changing in v0.3 in main

[winter linden]

Yes, but larger changes may take longer to be reviewed (and may not be accepted). Otherwise yes, contribute away 🙂

[chilly arch]

@here It's that time of the week again 🙂 Add your topics, questions, and/or demos to the Community Call agenda.

https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#heading=h.gp78qpx3dfxe

See you at the Community Call tomorrow at 10 am PDT! - https://discord.gg/3myZmR4w?event=1030545484428869672

[winter linden]

If anyone here is interested in developing CI/CD pipelines in Go: consider joining the community call in 5mn 🙂

[winter linden]

Sneak preview: https://devel.docs.dagger.io/8g34z/get-started-sdk-go/

[chilly arch]

If you missed the community call today, here is the recording. There is an API Playground demo and Go Getting Started demo 🙂
https://dagger-io.zoom.us/rec/share/O9XqOnUNjWF0ed4QfTzO0BlZsunHUZnkcaAAd1n-HjWltpOnj_cG9_QB3ZFv8yf_.yyyninEoOv1kApmN?startTime=1666285181000
Passcode: 6qk+yWM5

You can find the links to try out the demos in the community call notes here - https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#

[pulsar sigil]

hi, I am new here and have been looking into Dagger for a couple weeks. I was wrapping my brain around cue but now I see that 0.3 seems to be replacing this with more language specific APIs. I know that cue is supposed to be maintained via a compatibility layer, but is this still going to be considered a first class citizen moving forward or is there a desire to move away from cue and towards the Go SDK?

The other question I have is in terms of package management is the package management based on Git or will it be OCI images in the future? The reason I ask is I have some desire to use Dagger in an environment that may not have internet access and so the ability to proxy extensions would be very beneficial to me.

one of the biggest drawbacks I see to this solution is losing things like stage visualization when creating a pipeline in a tool such as GitLab CI or Jenkins. Is this something that the team is thinking about for the future? Perhaps through some sort of code generation to produce a Jenkins/GitLab CI file dynamically (not sure if something like this would be possible).

Anyway happy to be here, and I look forward to discussing with you all.

[winter linden]

Hi Donald, welcome!

You are correct that going forward, there will be SDKs for multiple languages. The CUE SDK will continue to exist - we are prioritizing compatibility with existing CUE configurations, at the expense of moving as fast as the newer SDKs which don't have this problem. Eventually though we expect all SDKs to keep evolving.

[winter linden]

It depends what you mean by "package management". Each SDK lets you use your language's native package manager (CUE, Go etc) just like any other software development project.

At the engine layer, we are developing an "extensions" feature that will be cross-language. There is still time to finalize the design. If you could share your requirements for using the Dagger Engine in an environment without internet access, perhaps in a github issue, that would be very helpful. One way or the other, we want to make sure that is supported.

[winter linden]

You are correct 🙂 We plan on doing two things:

1. Help you generate Jenkins / Gitlab / Github workflows from your Dagger project. Note that you can do this manually in the meantime - the generation is just a convenience.

2. Optionally hook up telemetry directly into the Dagger Engine, and give you a visualization service that works across all your CI runners, as well as development machines. Ultimately you will get more useful information from Dagger's telemetry, since it knows more about the logic of your pipelines.

Obviously we have neither of those features at the moment 🙂

[pulsar sigil]

thank you for the responses. How do you handle complexity when you are building extensions and pipelines in full blown language specific SDKs? I guess one thing that originally drew me to dagger was it seemed like a relatively lightweight CLI binary that I could integrate with my CI servers and have a common code base. Now it seems like we may need to add the Go SDK and other SDKs to make the pipelines work. Are you all at all worried about complexity bloat from that and how will that be managed?

I love the idea of the code generation, I think that will be helpful. I have some teams using Jenkins and others using GitLab so having the ability to have templates in a common code base is one of the things I am interested in Dagger for as well as being able to run locally

the reason I am concerned about things like the Go SDK is in general I have always thought my containers running in my CI should be as small as possible with minimal dependencies

[winter linden]

Well the complexity in today's tooling is not in the tool, it's in the spaghetti of configuration and scripts behind the tool... The simplicity comes from removing all of that spaghetti in favor of a small amount of code, written in the language that is most familiar to the team, and using the same tools they already know

Of course if the preferred language is a shell script rather than Go, then you should be able to write your pipeline logic in a shell script 🙂

But it should be one simple shell script, that includes all the logic. Not one of 12 scripts and yaml files to be orchestrated in an arcane custom way

[pulsar sigil]

I think I can agree with that. I was a bit nervous about the complexities of cuelang for example. Yaml is not much better and a lot of teams struggle with defining pipelines in general. Jenkins gave people a lot of issues in particular

I mostly want as much of the complexity hidden from my dev teams in favor of shared, reusable templates

so that everything is on rails for them because I do not want them thinking about the intricate details on tool integrations

[winter linden]

Right. In that scenario the choice of SDK might be guided by the language preference of the "platform team" building and maintaining the rails for everyone else. Which again, may be Bash.

[pulsar sigil]

sure that is reasonable

I like the idea of being able to use an SDK for extensions that makes sense. IE python if I need to integrate with a python specific library, and golang for other things and wiring it all together

[winter linden]

Ultimately, with API extensions, these "rails" can be delivered as an API. All developers get delivered an API describing all the build, test, deploy pipelines available to them in a domain-specific way. Each developer can invoke these pipelines in whatever way is most convenient to them: command-line, shell script, NodeJS script, Python etc. All of it funnels through calls to your API (powered by Dagger + your extensions)

[pulsar sigil]

very cool. so I guess the thing I would like there is the ability to "bake" the extensions to an OCI image that was fully self contained so that using the extension was just pulling that image. It would contain the runtime needed to run that particular extension

that to me would be a powerful thing

I appreciate the responses here, it really helps me understand the future of Dagger

one area I see is turning source into a container. There are a number of solutions for this such as S2I, Jib, cloud native buildpacks, and the tanzu build service. It seems like dagger could be used to develop a source to container solution possibly. not sure if that makes sense for the project, but it is an area where some of the current solutions are not really fully adequate

[winter linden]

Yeah that is a major strength of Dagger since it is built directly on top of buildkit.

I'm confident you could reimplement any of those tools on top of Dagger and remove 90% of the code

[pulsar sigil]

yeah definitely. And still be able to run the unit tests 😄

cloud native buildpacks have a major drawback in that you need a parallel setup to run your unit tests which to me made them not viable

[winter linden]

Effectively we are making source-to-container a commodity

Right, it's a continuum between building a container and doing other things with it (like testing, generating a deployment configuration for kubernetes, ecs etc)

[pulsar sigil]

yeah

one of the first projects I have in mind to prototype the use of Dagger is to build a set of pipelines for handling secure software supply chains

I want to be able to build some tooling to help me ingress open source software

IE security scans, license compliance, etc

a lot of the tooling there is stuff I can template fairly easily and wire together.

[winter linden]

Yeah that seems like an excellent fit

What would be your language of choice for this?

[pulsar sigil]

probably python mostly but golang wouldnt be a bad second option

a lot of the APIs have python bindings

[sharp marsh]

Happy to help out here in any way we can Donald. We have been speaking about eventually introducing software supply chain capabilities into Dagger and having someone understanding how this fit together seems like a perfect moment to collaborate together 💪

[winter linden]

@pulsar sigil Python SDK is pretty far along 🙂 /cc @rain oriole @wraith niche who can give you a sneak preview 🙂

[pulsar sigil]

sounds great!

awesome yeah a python SDK will be a great feature

I think on the commercial side you would have Blackduck which is popular but expensive. Having some libraries for things like Trivy, DefectDojo, LicenseFinder, and maybe like the microsoft SBOM tool would be a good starting point I think

maybe some extensions for things like fossology as well

[rain oriole]

Yeah, early next week I’ll release codegen for Python. Then I’ll just iterate on more testing, docs and better demos.

[pulsar sigil]

awesome!

[rain oriole]

By the way @pulsar sigil, do you have a max Python version requirement?

[pulsar sigil]

probably targeting python 3.6 if possible would simplify things. I got some servers that run RHEL8

not sure if that is possible or not

if not then I will get whatever version is needed

3.6 is pretty old so I would bet it needs a newer version

[rain oriole]

Yeah, that’s pretty old 🙂

[pulsar sigil]

hey at least it is not 2.7 XD

I had a team using CVS until last year soo

😛

[wispy wave]

Hello folk,
I have a writing problem with the client. But I think it's related to my environment.
I use a devcontainer for my projects, which creates a mount between my host and my machine in /workspace/my-project.
When I use dagger from my host, it works perfectly. But when I try from my dev container, I have an error:

[✗] client.filesystem."./public".write 0.2s
2:36PM FATAL failed to execute plan: task failed: client.filesystem."./public".write: error from receiver: error setting metadata for /workspaces/my-project/public/.tmp.087960590: lchown /workspaces/my-project/public/.tmp.087960590: permission denied
I think it's a problem with docker in docker when I try to write in a mounted volume.
Have you ever had this problem? If so, do you know how I can get around it?

[slender star]

Hi @wispy wave 👋 I'm going to move your question to the help section, if that's okay 🙂

[wispy wave]

yes of course

[slender star]

#1033842496251375647 message

[slender star]

Hi Dagger Community!! @warm temple and I are at Kubecon NA in Detroit this week. If you're around right now, we're in a 2nd floor lounge across from the Grand River Ballroom on the way to the Day 0 sessions. Come say hi! 👋

[winter linden]

With the imminent release of our Go SDK, it's time to start voting for the next candidates 🙂 https://github.com/dagger/dagger/discussions/3489

[narrow bane]

Can't believe I had to choose between my professional interests (Node) and my personal interests (Rust) 😢

[runic heron]

I'd been keen on seeing what could be done for a Java API ( have actually been looking at GraphQL java libs recently - or Kotlin - tho pure Java would be nice, either via JBang or the fact you can just java Foo.java and run that as a "script"

[swift inlet]

A Rust SDK will definitely pop up one day 😇 The community loves creating tools

[runic heron]

(JBang brings support for dependency management - which the simple java one doesn't)

[swift inlet]

very interesting

[runic heron]

https://www.jbang.dev for reference.

[winter linden]

@runic heron well now you've got to vote for Java in that poll 🙂

[runic heron]

@winter linden already did 🙂

For these non-go APIs, how is the "engine" started? Run it as a daemon somewhere, or System.exec("go-dagger-go") type process invocation? Liking the direction cloak/dagger is doing here -

[winter linden]

Yeah it's a system-exec. Eventually we'll document the full underlying API to the point that you can reimplement it all natively and get rid of that crutch. But it will be a while... And this way works fine in practice, so no rush

There's a bit more nuance than that if you zoom in... Starting with the fact that, currently the "engine" is made of two parts: an api router on the client machine, and a runner on a worker machine. So really your SDK execs the former, and (for now) auto-runs the latter in a container. There is some ongoing work to move the entire engine server-side, leaving just a "smart proxy" to system-exec on the client machine. But the result is the same for the SDK

[runic heron]

Right - I was gonna say, rewriting the entire engine seems crazy, but that router sounds like a proxy/bridge

[winter linden]

Yes that's the idea. Right now it's a pretty "fat" proxy... But it will get thinner 🙂

[runic heron]

We all do..... We all do 🙂

So my doc keeps telling me.

[swift barn]

Hello, I just skimmed through the docs and cannot seem to find a deployment diagram. This comment above helped a bit:

There's a bit more nuance than that if you zoom in... Starting with the fact that, currently the "engine" is made of two parts: an api router on the client machine, and a runner on a worker machine.
But I'm wondering - how is the runner executing tasks? Is there an extension API similar to Jeknins where one can run jobs localy, on VMs, launch VMs on demand, use a cloud provider etc. ?

[solar bison]

If I'm not mistaken, that's all the by buildkit within the Docker Engine but I'm completely new to dagger and just stumbled upon various documentation bits.

[slender star]

@swift barn @solar bison Good point! Today you don't need to do anything special to deploy the Dagger Engine. When you use one of the Dagger SDKS, a Dagger Engine will get provisioned on the fly for you. Like @solar bison said, there is Buildkit magic inside that allows all of your build steps to execute in containers 🙂 In the near future, there will be other ways to deploy the Dagger Engine. A deployment diagram is a great idea.

Which SDK would you use? Or if your preferred language isn't there yet, which one would you vote for? https://blocklayer.typeform.com/to/a6m5gKSS

[swift barn]

Regarding SDKs, I would prefer a configuration language. Will CUE be supported in the future? Regarding the distributed Buildkit, I found Buildx drivers - https://github.com/docker/buildx/blob/master/docs/guides/drivers/index.md . How would one use e.g. AWS EC2 or vSphere to spin a certain amount of VMs?

[pulsar sigil]

I put my vote for Python. Rust is cool and so is Java but I need Python sooner so I can do the real work I got 🤣

Python is used heavily by my shop

[wraith niche]

Python is currently work in progress, you can already try it. Ask @rain oriole in case you have specific question, he’s the expert!

[wraith niche]

CUE is the first SDK we released, it’s documented. We’ll support cue in the future, curious to hear about your use case.

[hoary spoke]

I think it refers to the current state of the cue sdk which is not a cloak native sdk; looking into the repositories/docs i am not able to understand how to use extension, how it works and the relationship with Europa universe.

[winter linden]

Extensions are currently not launched, so not available to anyone yet.

So for now, each SDK has its own separate ecosystem of packages (including CUE “universe” packages which are CUE-only).

We will make this more clear in the docs.

[rain oriole]

I put my vote for Python

[chilly arch]

@here** Today we are proud to introduce the Dagger Go SDK: a new way to develop your CI/CD pipelines as code and run them anywhere. Check out the links below to get started or learn more: **

• Go SDK Get Started Guide - https://docs.dagger.io/sdk/go
• Blog Post - https://dagger.io/blog/go-sdk
• Looking for a different SDK? Let us know here, and we will update you when it is ready - https://blocklayer.typeform.com/to/a6m5gKSS
• Stuck or have questions on how to get started? Use our #help forum or open a github discussion

Don’t forget to join our community call this week at Thursday at 10 am PDT for live demos, https://discord.gg/KprBdzKn?event=1034224930943930490

[winter linden]

We just launched the Dagger Go SDK 🙂 More information and discussion here: https://twitter.com/dagger_io/status/1584945060608888834

To get started: https://docs.dagger.io/sdk/go

dagger (@dagger_io)

Introducing the Dagger Go SDK: a new way to develop your CI/CD pipelines as code, and run them in containers anywhere. https://t.co/7riHAwUSMO

Twitter · Oct 25, 2022, 4:28 PM

[blazing sluice]

Congrats! It's a nice API, and pretty easy to use. The Cubzh team is already setting-up a CI/CD with Dagger Go SDK on https://github.com/cubzh/cubzh ! ✨

[runic heron]

Does the GoSDK have a nice pretty-printed progress output like the old dagger-cue - or that polish yet to come? Have a super simple lame little test project building with the GoSDK now - sure, it's little more than just the tutorial - but working is a starting point. Time to try something more complex

[winter linden]

Yes there is an option you can pass to dagger.Connect

oh sorry you mean the really pretty one 😅 I don’t think we have that polish yet

but can add it easily. Would you mind opening an issue?

[runic heron]

Will raise it in a bit - almost time for dinner and I need an escape away from that computer for a bit - Nice to see the underlying docker output when debugging tho 🙂

[winter linden]

To clarify, there is an option to get raw log output. But it's not pretty-printed.

[runic heron]

Raised as https://github.com/dagger/dagger/issues/3553

[runic heron]

Hrm - have I done something stupidly simple wrong - clean project:

go run main.go
# github.com/moby/buildkit/session/filesync
../../../golang/pkg/mod/github.com/moby/buildkit@v0.10.5/session/filesync/filesync.go:112:20: cannot use dir.Map (variable of type func(string, *types.Stat) bool) as type fsutil.MapFunc in struct literal

have I missed some override? I did the mod replace

mmm, I think I'd broken something in my go.mod

[swift inlet]

Did you manage to fix it ? 😇

[runic heron]

Yup! Now looking at how to add my github auth for private repo cloning. Altho its 10pm and I'm getting tired.

Noticing the SDK docs often lead to 404s 😦 like https://github.com/dagger/dagger/tree/main/sdk/go/blob/v0.3.1/api.gen.go#L783

[swift inlet]

Will take a look, thanks for sharing 🙏

[swift inlet]

Ok, found it https://pkg.go.dev/dagger.io/dagger#section-sourcefiles

[runic heron]

It's that "Plaintext" link that's 404ing - navigating around manually seems to work, just found the WithMountedCache which solves a bug bear I've had with a lot of docker/buildkit style pipelines - caching my build cache 🙂

altho - that MountedCache doesn't seem to be a docker volume (maybe that I was a bad assumption there). lets see if this still does want I'm expecting

[sharp oracle]

Dumb q about something I possibly misunderstood — does Cloak mean CUE is going away, or it just means it will be kept supported alongside the new SDKs?

[runic heron]

@sharp oracle My understanding is cue remains, but isn't going forward in the future. Cue was.... interesting - but also felt kinda awkward.

[swift inlet]

Yes, it's not a volume as in Docker, you're right. Here is more info on that (explained using the cue sdk) https://docs.dagger.io/sdk/cue/667572/dagger-fs#difference-between-a-docker-bind-mount-and-dagger-cue-sdk-mounts

[swift inlet]

Cloak is the name we introduced to work on this multilanguage feature. Now that we released it, Dagger keeps living with the same name. @runic heron is right, it is going to be kept supported alongside the new SDKs 😇
PS: it's a legitimate question, thanks for asking

[runic heron]

with all this Cloak and Dagger malarkey - are those who love Dagger Paladins, or Mercenaries?

[fossil pine]

Most examples seem to check out repositories, not use the local copy to run builds/tests. Is that to make examples easier/more self-contained?

[rain oriole]

@sharp oracle The future of the CUE SDK will depend on demand. The one that was launched this week is just a rebranding of Dagger v0.2 which only works with CUE. It doesn't use the underlying 0.3 engine. It's possible that we'll launch a future CUE SDK that is a binding to the 0.3 API, meaning it brings breaking changes to 0.2 plans but also feature parity with the other 0.3 SDKs. But that'll depend on demand/interest from the community.

[runic heron]

@fossil pine From the pipeline perspective I'd probably think "that depends on the pipeline", if you're pulling in 10-15 repos of code to form an integration test pipeline and build up a big distribution- having that pull external repos makes a bit of sense.

Currently from what I've seen of the (fairly simple) examples - by pulling from an external repo - you can quite easily change that and pull YOUR repo in the build and play with - without actually touching your repo and putting dagger code in there.

Right - 11:50PM here - I should sleep! More dagger playing tomorrow...

[rain oriole]

By the way, there is work currently to port the engine on the current CUE SDK to 0.3: https://github.com/dagger/dagger/issues/3121, but still keeping the same definitions for compatibility. So this'll be the next version for CUE and it's already underway. What I said above was yet another version that more closely matches the underlying Dagger API that the other SDKs are using, i.e., lighter bindings, codegen'd.

[sharp oracle]

So basically what this mean is that CUE is soft-deprecated and you will gauge people reaction to decide whether to bring it up to date to the way new SDKs work or hard-deprecate it?

[solar bison]

Is there already some kind of best-practice guide or a list or projects that are already using the Go SDK somewhere? I'm particularly looking for inspiration regarding if I should put dagger directly as a dependency to my whole project or create a subpackage with its own go.mod file 😅

[fossil pine]

@runic heron I guess it's fine for examples, but in a real world scenario I'd want to run the pipeline on my local copy.

[rain oriole]

See if this helps: https://github.com/kpenfound/dagger-demos

[solar bison]

It does, thank you 😄

[warm temple]

https://github.com/dagger/examples has some up-to-date examples as well 🙂

[hoary spoke]

Great, is there an example writing string content to container file ?

[rain oriole]

Do we have an example on building a Dockerfile?

[wraith niche]

core/integration/container_test.go:73 is what I used recently (sorry on my phone).

[hoary spoke]

Do you think this can be considered a viable way ```
package file

import (
"context"

"path/filepath"

"dagger.io/dagger"

)

type WriteFileOpts struct {
Path string
Content string
}

func WriteFile(ctx context.Context, client *dagger.Client, ctr *dagger.Container, opts WriteFileOpts) (*dagger.Container, error) {

dirPath := filepath.Dir(opts.Path)
fileName := filepath.Base(opts.Path)

container := ctr.Exec(dagger.ContainerExecOpts{
    Args: []string{
        "mkdir", "-p", dirPath,
    },
})

_, err := container.ExitCode(ctx)

if err != nil {
    return nil, err
}

dir, err := container.Directory(dirPath).WithNewFile(fileName, dagger.DirectoryWithNewFileOpts{
    Contents: opts.Content,
}).ID(ctx)

if err != nil {
    return nil, err
}

container = container.WithMountedDirectory(dirPath, dir)

return container, nil

}

[swift inlet]

Hi @hoary spoke, glad to see you play with our Go SDK 😇 Really interested to have your feedback on the subject

[swift inlet]

You'll need to get review from core team, to see how it fits with the current API. But at first sight, it looks nice. Wdyt @elfin frigate @pseudo stream

[elfin frigate]

I think you can skip the mkdir -p, it'll make it for ya

[river swift]

anything obviously wrong for building a Dockerfile?

    app, err := workdir.Read().Directory("app/").ID(ctx)
    if err != nil {
        return err
    }
    runtime := client.Container().Build(app)

getting input:1: container.build failed to read dockerfile: open /tmp/buildkit-mount3832100689/Dockerfile: no such file or directory ... the directory does contain a Dockerfile locally

[royal oak]

hey all. this is perhaps one of those obvious things … is there a list of cloud providers that I can publish the artifact to, via the sdks? or how do I handle the publishing/delivery step if I am building for a managed service like Azure Functions?

[winter linden]

Do you mean specifically an OCI artifact, or other kind?

[royal oak]

others, for example to deploy to Azure Webapps you package a specific zip, then trigger the deployment to a slot, test it, then swap the slot with production to deploy.

specifically I am looking to streamline our azure devops pipelines. they handle our dotnetcore services, nextjs front end, and integration with some SAST tooling as part of the CI pipeline. we then publish the artifact (a zip file), which triggers the deployment pipeline that publishes to azure.

I’m not sure if this is the intended use case for dagger, to simplify all the yaml required so I can do local tests of the pipelines.

[chilly arch]

@hereJoin our Community Call tomorrow at 10 am PDT. @warm temple will give a demo that will be using the new Go SDK, and I’m looking forward to discussing and learning more about all of the use cases that I’ve seen pop up in this general channel! Save the meeting to your calendar with the link below:

https://discord.gg/E9cZRraj?event=1034224930943930490

**If you’d like to add a demo, topic or question to the agenda, please add it here. See you there! https://docs.google.com/document/d/1-6RSWHwFoZr588kftPLVvT1RCHUrOrnmURldBRrP1Ak/edit#
**

[runic heron]

oooph - thats 6am my time 🙂

[chilly arch]

Darn, I was excited to hear more about the tests you've been doing 😦 I'm hoping that we can add more calls to accommodate time zones soon. For now though, you can watch the recordings after each call. I will post the recordings on our events page, and will notify this channel when the recording is ready -https://dagger.io/events

[runic heron]

Hopefully I'll drag myself out of bed in time 🙂 I'll try record a screen cast demo of something tonight tho.

[chilly arch]

@warm temple @slender star should be able to help you, @royal oak. They are at KubeCon right now, so just a heads up that there might be a delay. I did create an issue to create a guide for your use case though! https://github.com/dagger/dagger/issues/3570

[teal oar]

hello all! Found this today and it seems super cool! I was curious around how it’s expected to run. IE say I have a CI pipeline using Concourse or GitHub Actions, generally these are already ran using containers and usually to kick off the job they clone the repo they’re watching. I noticed the docs have you cloning the repo right off the bat and then booting up a container to do work. Is there any guidance around the things I mentioned above?

[royal oak]

I think what you are asking is, how to run the dagger pipelines from a github trigger? Not sure I understood your question - and I think I may have the same one, so just want to see if we are both talking about the same thing.

[runic heron]

In case I don't make the zoom call in the morning - I put up a quick screen cast of my experiments with GoSDK - only.... I forgot I switched to a 4k screen this week and erm, it's a bit small fonty (I did zoom in post-prod but may re-record eventually ) - does https://share.descript.com/view/1SXwOgfaC1M work for people?

(of course - I also forgot to RUN the pipeline in the demo, but since I was zooming in you'd probably not see that anyway ).

[slender star]

Thanks @runic heron! There are about leading 3 chars of your code cut off when I watch the vid, but it's still totally watchable/understandable, and super valuable to see/hear you walk through it.

[runic heron]

Cheers!

[placid briar]

For what little it's worth, I hope the cue interface is maintained and put forward; I rather like the idea of describing the workflows as DAGs in "just data", and once my brain got wrapped around cue, I'm fairly sold on it as a configuration language. Mixing that with code doesn't seem as elegant to me somehow.

Not to mention I now have invested a decent amount in building some configurations and I'd prefer they not rot!

[teal oar]

Yep! Generally any pipeline system really.

[smoky bobcat]

so... I want to build Docker images using nix inside a container, which results in a docker loadable tarball inside a container when used in docker.#Build, how can i 1) extract that file as an Image 2) build it directly in a specific container?

[chilly arch]

Thank you for this @runic heron ! I will share it during the community call today.

[smoky bobcat]

FYI: this works https://github.com/nazarewk-iac/cookiecutter-python/commit/7b227f91

[runic heron]

https://comby.dev/docs/basic-usage what the tool I mentioned just at the end of the call - for structural code replace ( thats language agnostic )- the docs even show using Go code tho.

[rain oriole]

Python now has structural pattern matching (https://peps.python.org/pep-0636/), curious if that would work natively here. But I'm hoping more for getting the introspection result in the right order based on the schema right off the bat.

[runic heron]

There is some deep fu that comby can handle with sub expressions/patterns, which are more useful when using the recipe files which offer a much larger set of rewrites - I did just spot https://pypi.org/project/comby/ which is a python wrapper for it too - such as (sorry for the Java):

[change-dataprovider-definition-to-junit]
match='''
@DataProvider
public Object[][] :[name] {
  return new Object[][] {:[args]};
}
'''
rewrite='''
public static Stream<Arguments> :[name] {
  return Stream.of(:[args]);
}
'''
rule='''
where
rewrite :[args] { "{:[v]}" -> "arguments(:[v])" }
'''

That has a basic top level match, but then an inner "rule" to also rewrite the argument structure. Google has something similar with their refaster templates and tooling for C and Java - I wonder if they already have something similar for Go somewhere.

[pulsar sigil]

I agree, and I second this. I think being able to define pipelines in a programming language is not a bad thing, but it is not what made me look at dagger in the first place.

[winter linden]

I understand and we intend to keep maintaining it.

Note however that the new SDKs are all built on an underlying API which is declarative. When you are querying the dagger engine you are actually sending a graphql query describing the data you want to see. Technically you query is not executed it is resolved. The result of each operation could be generated if needed, or retrieved from cache.

the breakthrough that makes these new SDKs possible is that we found a way to decouple this declarative DAG model from Cue and move it into a language-agnostic API.

So if anything, the DAG model that drew you to Dagger is only getting stronger: it can now “infect” an imperative language with a declarative parasite 😁

cc @placid briar

[teal oar]

Are there any docs on actually running a dagger “engine”? I’d like to look into using dagger but the lack of documentation leaves me hesitant.

[swift inlet]

Hi, just to be sure we're on the same page, can you please explain in more detail what you mean by "running a dagger engine" ? 😇

[teal oar]

I have no idea, hence the confusion. You see references in the Go SDK docs "querying an existing dagger engine" but no reference to what that means. So that's also my question, what is the "dagger engine" how do I run it? Is it in the SDK?

Ignore that sticker, that is def not what I wanted lol.

[runic heron]

I guess he means if writing a non-go SDK pipeline, I've seen early examples using bash etc. running the GraphQL commands directly - I believe the start of that is spawning a dagger process to run API calls against.

[teal oar]

"Using the SDK, your program opens a new session to a Dagger Engine: either by connecting to an existing engine, or by provisioning one on-the-fly."

https://docs.dagger.io/

And then no where is it defined what the "engine" is. You get pictures of where it's at in the flow (attached). This makes me think something is running constantly listening for calls which might answer my previous question of how to integrate this into actual pipelines.

[runic heron]

Yep - theres a GraphQL "server" that runs that accpets the API calls that do the actual heavy lifting work. My (assumption) is when you call dagger.Connect() in the GoSDK - that pulls that engine container and runs it up if needed, so that subsequent calls run.

[teal oar]

So is it possible to just run a server for dagger that only handles calls?

[runic heron]

Not sure on that - afaik there's still the dagger CLI tool - so I guess that may take the place of that server... I'm just a pleb user my knowledge is limited as well

[winter linden]

@teal oar we are intentionally vague because the engine is not yet consumable as a single “thing” that you can run and connect to. But that will change soon. In the meantime there are individual pieces that you can look at and run, but I would only recommend doing that if you specifically want to dive into the internals

[winter linden]

The current architecture is that the engine is made of 2 parts: an API router + a runner

• The router serves API queries and dispatches individual operations to the runner.
• The runner talks to your OCI runtime to execute actual operations. This is basically a buildkit daemon + some glue

[winter linden]

One tricky aspect is that the router curreny runs on the client machine, whereas the runner is on a worker machine that will run the containers. This could be the same machine but typically isn’t.

The SDK takes care of running both router and runner - each in a different way because they run in different machines and have different lifecycles.

[winter linden]

Eventually we will move the router to a server-side component, tightly coupled and co-located with the runner. This will be shipped as an OCI image which you will be able to provision, administer and upgrade yourself to your heart’s content.

In the meantime we have this 2-part architecture that was simpler to ship, but is harder to explain 🙂

[swift inlet]

Couldn't have explained it better 🙏

[winter linden]

In conclusion of my longer explanation above:

• Currently no
• But eventually yes
• In the meantime you can kind of do it with undocumented methods 🙂

[teal oar]

So mainly right now it's meant to be ran locally which I still have problems understanding how you'd use it in a system such as GithubActions, Concourse, or other CI engines that use containers by default other than doing Containers in Containers.

[swift inlet]

Currently, the buildkit daemon resides inside a docker image. If your CI can run container images, it will work. For example, GHA can run docker containers. We temporarly rely on that to run the buildkit image. You're right, It is, container in container, in a sense

[winter linden]

github actions doesn’t use containers by default, you get a VM with docker installed, dagger bootstraps from there

[swift inlet]

my bad; Thanks for the clarification

[teal oar]

Gotcha 👍, last but not least. Since most build scripts are located with the actual code I'm guessing a common pattern is to just do something like go run dagger.go (or w/e you called your script)?

[swift inlet]

Yes. Here is a repo with some examples: https://github.com/kpenfound/dagger-demos/tree/main/go. One nice pattern that seems to work well is to rely on mage

[winter linden]

what you said is correct 🙂

[teal oar]

I've never heard of mage but it looks pretty neat thanks!

[runic heron]

Anyone got any examples of using private github repos? Simply adding the env var with my token to the container didn't work - as I guess the Git call/declaration runs separate from that.

[swift inlet]

With the Cue SDK or Go SDK ?

[runic heron]

0.3 gosdk

[swift inlet]

I am pretty sure that we are passing your ssh-agent key to the operation. Run locally: ssh-add, then run your go code.
(currently testing in parallel)

[swift inlet]

Yes, that's it:

➜  ssh-add -D
All identities removed.
➜  go run .                                                                                                                  
panic: input:1: git.branch.tree.file.contents failed to load cache key: failed to fetch remote [e-mail]:grouville/privateExample.git: exit status 128

Please visit https://dagger.io/help#go for troubleshooting guidance.

goroutine 1 [running]:
main.main()
        /private/tmp/git/main.go:20 +0x61c
exit status 2

After:

➜ ssh-add
Identity added: xxxxx
➜  go run .  
readme #

=> Adding it tomorrow to the FAQ 🙏

[runic heron]

Ahh! Why didn't I think of ssh agent ( i rarely actually use it ) as I'm usually just in my direct shell space.

[turbid tulip]

I could be wrong, but I think that behavior is currently undocumented... so not too surprising. 🙂

[chilly arch]

@heavy karma In your opinion, can you review this to see if this info should live in any of the guides along with the FAQ?

[swift inlet]

On my todo for the day, but I'm stuck on a dumb go issue 😇

[heavy karma]

The FAQ is currently top-level ie. not specific to Go. We can create a Go FAQ but we don't have too much content for it yet, so it may be better to turn this into a short guide instead. Something like the guides at https://docs.dagger.io/sdk/cue/232322/guides

[chilly arch]

as a user, is there a specific place in the Go Docs where you'd expect to see this?

[heavy karma]

If making it a guide, I suggest creating a node named Guides in the Go SDK category nav tree, similar to what we have for CUE already, and using that as the default location for short guides for the moment.

[chilly arch]

@here you can now access the community call recordings from our Dagger events page. I have updated the Oct 27th tile with the recording from yesterday. Take a look!

We had great demos from @warm temple and @runic heron ! https://dagger.io/events

[earnest osprey]

thanks for uploading the recording, i was a little lost first as i couldn't find the code - but found it in the get param of your url a little later, automatically setting the passcode to access the recording doesn't seem to work with the current link (code was 5wc4a+Ra in case anyone else is looking)

[runic heron]

Probably just a link to guides for authentication / repositories - I saw someone a ticket elsewhere that also came down to SSH Agent stuff so it's not specific to Git.

[chilly arch]

@earnest osprey Fixed! Now you don't need a password to access, so you shouldn't run into issues anymore

[frozen basin]

Hey yall - curious if tracing still works? I thought at some point early on I was able to see build step, duration, etc in jaeger, but I was tinkering with tracing a couple times last week, didn't see much in jaeger.

[turbid tulip]

We did have tracing working in v0.2. It appears that @cloud canyon has done some work around tracing in 0.3: https://github.com/dagger/dagger/blob/main/tracing/tracing.go

I'm not sure what the current status of that is.

It looks like setting the environment variable OTEL_EXPORTER_JAEGER_ENDPOINT to a Jaeger endpoint should send traces to it.

[frozen basin]

fasho ty!

[chilly arch]

@here just a reminder that our community call is this Thursday at 10 am PDT. You can add it to your calendar with the link below. If you have any demos, topics, or questions that you want to add to the agenda, please let me know.

See you there! https://discord.gg/ZJdxVN5w?event=1035644790827647067

[chilly arch]

If you missed the community call last week, you'll want to check out this monorepo demo that @warm temple did. You can see the specific clip here -https://youtu.be/vJfoyN77fo0

[fossil pine]

Thinking about submitting a talk about Dagger to KubeCon EU. Also, there is a Nix vs Dagger comparison in my blog post queue. Just FYI 😄

[wraith dove]

Hello my name is Angel Rivera and I'm very interested in learning more about dagger and que projects. I'm currently a Developer Advocate for CircleCI and interested in learning about and assisting in how to better integrate/leverage dagger within the CircleCI and other continuous delivery platforms. I look forward to learning and contributing where ever possible. If there are any questions regarding CircleCI please feel free to ping me. Thank you

[slender star]

Hello my name is Angel Rivera and I m

[chilly arch]

Great, thanks @fossil pine ! Ping me when you have the blog post, talk accepted, etc. since I can get it added to the new Dagger event page or help promote on our social networks.

[chilly arch]

@hereCalling all Python users! We are looking for testers willing to try an early version and give us direct feedback before officially launching. If you would like to test and provide feedback this week, please comment on the thread below and I will reach out to you directly.

Python Early Access Requests

[chilly arch]

@here Shout out to the Python users! Tomorrow's community call will have a demo from @rain oriole to show the latest Python SDK developments. You can find the details for the community call here. See you there! https://discord.gg/ZJdxVN5w?event=1035644790827647067

If you want to get early access to help us test before launch, please let me know in the thread above.

[slender star]

@balmy orbit ☝️

[frozen basin]

yeah I'm not sure if I'm misconfigurationg things..

docker run -d --name dagger-buildkitd -e JAEGER_TRACE=192.168.1.16:6831 --privileged moby/buildkit:v0.10.5

Only seeing traces for workdir. Guess I could move to #1030538312508776540 if that's more appropriate

[turbid tulip]

Yeah, create a thread, and we can get some more feedback from the team there!

[rain oriole]

There's already a dagger project in PyPI: https://pythondagger.sourceforge.net

[winter linden]

copyright 2012

damn

last update 2012

maybe we can get it, but not next week

https://sourceforge.net/u/rsz/profile/

[elder nimbus]

Unable to install dagger sdk with go install dagger.io/dagger@latest on mac. getting error: go.mod file found replaceable directives.

[winter linden]

did you run the `go mod edit`` incantation from the install docs?

it is annoying but necessary, for now..

[elder nimbus]

ok let me try

[elder nimbus]

worked adding go mod edit first and then go install.

[winter linden]

that’s great! Sorry about the complication. We are working on removing it soon. Then it will just be go get and the balance of the universe will be restored

[elder nimbus]

no problem another issue is with cue sdk, the documentation doesn't indicate how to get it?

[winter linden]

oh really? there should be instructions for installing the dagger-cue cli

https://docs.dagger.io/sdk/cue/526369/install

[winter linden]

something weird with that embed no? @heavy karma 👆

[heavy karma]

Checking

[heavy karma]

https://docs.dagger.io/sdk/cue/526369/install/ renders fine on the live site, so this seems to be an issue when you use the link in a social post/message. Or maybe just Discord, since it looks correct when I post it on LinkedIn.

[heavy karma]

When I check the live site, I see this

[elder nimbus]

works on chrome ! on safari doesn't show up

[heavy karma]

Do you not see the tabs @elder nimbus?

I've checked on firefox/linux, firefox/windows, chrome/windows and chrome/android and the tabs show up in all those. I don't have safari so can't check there, sorry!

[winter linden]

looks fine on safari (Mac OS) for me

[elder nimbus]

No not on Safari however on chrome they are visible

Anyways got cue installed.

[heavy karma]

So two issues:

• social link sharing doesn't show the correct page snippet (might be only in Discord though)
• tabs don't show up in safari in some cases

I'll open issues for these and @elder nimbus I will ping you to collect some info on your safari and OS version in the ticket so I can check up on this further.

[heavy karma]

@elder nimbus I opened an issue to investigate this at https://github.com/dagger/dagger/issues/3683. Would you mind adding some more information there, such as your OS, Safari version, any browser plugins installed etc and also the screenshot you posted above? Does the issue also occur if you use Safari in safe mode ie. without any plugins or extensions loaded? Thank you!

[chilly arch]

Just a heads up that zoom says that the community call recording from yesterday is still processing. I submitted a zoom support ticket. Hopefully, I will have the recording for y'all soon! cc @wraith niche

[wary peak]

Hi all! I'm looking at dagger and loving the prospect of replacing error prone ci scripts with Go.

I'm trying to understand where exactly it fits, though. If i were to use the Go SDK, is the expectation that Circle CI, for example, will only have 1 step that runs my dagger program and that's all?

Before the Go SDK released my assumption was that Dagger is supposed to generate that config for me, but now that seems like a non-goal, is that accurate?

Thanks!

[winter linden]

Did anyone watch Nic Jackson's live stream yesterday? I'm watching the recording now 🙂

https://www.youtube.com/watch?v=PD8e1rDyUMo&t=1s

[winter linden]

Hi everyone, we released the Dagger Go SDK 0.4.0 today. Among other things it adds support for multi-platform pipelines, and an improved filesystem API. Note, it includes a breaking change. https://dagger.io/blog/go-sdk-0.4

[quaint jasper]

Hey all! Really excited to use Dagger when the Python SDK comes out, it ticks all of the right boxes for me!

Just one quick question that could perhaps be elaborated a bit in the documentation: Dagger has a client-server architecture, but I am not sure that I understand what the implications of this are. Is it possible, for example, to have a build agent running remotely somewhere that you can then run stuff on with a client? When is the server started?

Please excuse my ignorance. As I said, the tool looks absolutely amazing. Can't wait to make this a core part of my CI/CD pipelines.

[rancid canyon]

Hi folks I had a few problems with the live recording of part2, but I recorded it again last night and pushed up a new video

https://allthings.tube/dagger2

I am loving the updates to dagger 0.4, the directory updates make things super easy to use

One question I have for you all is do you plan to somehow abstract the concept of cache so that I have a way to store it and later retrieve it when the machine might be different

I was thinking that maybe cache could be stored in an image then it could be pushed and pulled from a registry

Being able to push the CacheVolumes to a remote place could be cool

That would make working with things like Terraform way easier, I could manage the state in a pure dagger way.

[jagged flicker]

Hi 👋
Just quick intro first, I’m a senior software engineer at Docker, working on Hub services for publishers (official images, verified publishers, open source program).
I played a bit with dagger and wanted to share with you what I tried. Especially I have not idea if I’m doing it ‘the right” way, or maybe there’s easier ways to do it. Let me know 🙂 (details in 🧵 )

[vestal creek]

Looking forward to the NodeJS SDK and using this, much like mage 😄

https://github.com/google/zx

[chilly arch]

The zoom recording of the Nov 3rd Community Call finally finished processing. You can see the recording below or on https://dagger.io/events. In this recording, you can see the latest demo of the Python SDK. If you are interested in trying it out before we officially launch it, please let me know!
recording - https://dagger-io.zoom.us/rec/play/0Tg_BdAPeBFlxOL-ICqStDVgXYE0CtoH0MzwPzTvlBa6luEGQ4AneViJWrXmyTgI8YyWoET0GpYO10jr.QUClEurUynq4CJdV?continueMode=true

[winter linden]

Cache management

[winter linden]

Hi everyone, Jeremy and Kyle from the Dagger team are live on the Containers on the Couch show : https://www.youtube.com/watch?app=desktop&v=z6pYfz_q9tc

[frozen basin]

Hey yall - curious if kotlin/java sdk is on the roadmap at all?

I'm imagining a gradle plugin similar to jib but backed by dagger could be quite powerful

[chrome vessel]

Is it possible to develop actions in go and the pipelines in cue?

[chrome vessel]

Any plans on making a dagger buildkit frontend? I would love to have the option to use dagger instead of a Dockerfile and not break local development workflows that build images with docker-compose and docker build.

[chilly arch]

Hi all! Check out @warm temple's latest Dagger Go SDK 0.4 "Getting Started" Video - He highlights the big features of the latest release in a quick tutorial -https://www.youtube.com/watch?v=0fSzOOZ2CO8

[chilly arch]

@rancid canyon has done an amazing job at showing how to use Dagger in real-world scenarios. Don't forget to click "notify me" for his next livestream, which is scheduled for Nov 13th at 11:00 am PDT. https://www.youtube.com/watch?v=eCKes2AKzK8

[winter linden]

Buildkit frontend

[chrome vessel]

What is the golang equivalent to core.#Start and core.#Stop?

[jolly needle]

@slender star 👋 thanks a lot for your Jenkins and Dagger talk at DevOps World!

[swift inlet]

core.Start

[jolly needle]

I actually wanted to join the community call and to share some of my experiments and ideas about Dagger observability and Dagger&Jenkins&Keptn integrations, but I've been unable to do so due to the OpenSLO meeting conflicts. Will try to join or to share asynchronously in this chat later

[slender star]

Repeat for the Americas at 1pm PT/4pm ET: https://www.techstrongevents.com/devops-world-2022/v/s-1132280?i=wtuh4-Nzu-GRDeBNP-W5T5RRDviKidoT

[timber sphinx]

@heavy gazelle just got quoted live on GitHub actions universe 🎉

[chilly arch]

Hi Dagger Community! We have some great demos planned for tomorrow's community call, so don't miss it. We'll showcase the Python SDK, preview the NodeJS SDK, and get some insight into how @rigid epoch's Dagger hackaton for work went - https://discord.gg/UhXqKz7SRM?event=1035645348372283472

[slender star]

Learned about this project from @jolly needle today during talk. Very cool! Imagine I might use to test running Dagger pipelines run from a Jenkinsfile. https://github.com/jenkinsci/jenkinsfile-runner

[swift barn]

what was the reason that dagger has been renamed to dagger-cue?

[winter linden]

That binary is highly CUE-specific. It’s now part of the CUE SDK.

We will release a new version of dagger soon that is language-agnostic

(release planned next week)

[sage prism]

What is the future vision of CUE in the Dagger world?

[winter linden]

Future of CUE & Dagger

[chilly arch]

@here Introducing the Dagger Python SDK: develop your CI/CD pipelines in Python, run them in containers anywhere. https://dagger.io/blog/python-sdk

[chrome vessel]

Is there a way to load an image into docker from the go sdk? Like the docker buildx --load flag?

[pseudo stream]

Yeah we actually have to do that in some of our internal tooling, so there's an example here: https://github.com/dagger/dagger/blob/c432581c36a481bbc86ab7f340a9365e2c6a26b0/internal/mage/util/util.go#L142-L175

The idea is basically to do an export of the image as a tarball to the local filesystem and then shell out to docker to load it. So not the most elegant thing in the world but gets the job done.

[chrome vessel]

thanks!

Are there any docs on cross sdk actions? I see dagger.Serve in the examples and I'm curious if thats is for cross sdk actions, how that works, and how to consume it.

[chilly arch]

@pulsar sigil as requested, the Python SDK is out. Would love your feedback!

[pulsar sigil]

Awesome! I will take a look!

[chilly arch]

We had a fantastic community call today! If you missed it, you can watch the recording below. We had a Python SDK demo to showcase the release from this morning, a quick preview of the NodeJS SDK (if you want to test it out let me know!), and a demo from one of our community members @rigid epoch. David showcased what he built with Dagger during his latest Hackathon. https://dagger-io.zoom.us/rec/share/LhJz5EY6scaxW2WexnPN_Tij39I4OOmxP_nn8zRdtmW12IDiJjdMb7TFpuoVPuUc.O9o9CuT0ujnD5fdC

[rigid epoch]

apropos of nothing in particular, whoever's making the art for your blog posts is doing a great job, love these sneks

[turbid tulip]

That'd be @mint axle!

[silver bay]

Hey guys! Saw you on HN, happy to do what I can to help out with this feature: https://news.ycombinator.com/item?id=33549919#33555383

yurisagalov

Dagger Python SDK: Develop Your CI/CD Pipelines as Code

Hacker News · Nov 10, 2022, 5:23 PM

[winter linden]

Welcome! Glad you came by to visit 🙂

[silver bay]

Thanks @winter linden . What should I do? Raise an issue on Github?

[winter linden]

An issue is great, or we can start a thread here. Whatever is more convenient for you.

[mint axle]

Thank you!!

[tardy mauve]

We had a fantastic community call today

[stable oar]

Hey - congrats on the python release - spent 20m kicking the tyres yesterday and it worked really nicely! Im not sure if I missed it in the docs but I wanted to get an idea of how dagger deals with docker in docker. Is there somewhere I can read up on that?

e.g. some of our Github actions use docker-compose to run end to end tests. Im assuming this would not work in Dagger? If so, what's the suggested way of doing this?

[dark wave]

So one thing that appealed to me about dagger earlier on was the composition of workflows and the ability to provide inputs at various places.

ie. https://github.com/dagger/dagger/blob/v0.1.0/docs/learn/1004-first-env.md#configure-user-inputs

Looking at the pivot to typed SDKs - is that ability gone and I need to program it manually?

[real kite]

Been playing with the python sdk a bit... Just wanted to check, because it seems that way, multi-stage builds that are composed in code like with cue (https://docs.dagger.io/1205/container-images/#multi-stage-build) are currently not supported in the Python SDK, correct? I would need to use Dockerfile instead?

[sharp marsh]

👋 welcome conor! having "multi-stage" like builds in dagger is definitely possible. Here's an example with the Go SDK that does exactly this. https://github.com/kpenfound/dagger-demos/blob/main/go/getting-started/ci/main.go#L107-L141

You'll notice how the binary gets built with the golangImage and then copied to a baseImage container that gets published into a registry. Since the python API has the same form as the Go SDK, translating between the two should be strightforward. Let us know if you have any other questions!

[real kite]

That helps, thanks!

[sterile mason]

Hello everyone, I was exploring dagger along with go SDK in general, is there any UI to visualise the pipeline as of now?

[slender star]

Hi @sterile mason 👋 Not yet, but planned! Could you help kick off this discussion with what you'd like to see? 🙏 https://github.com/dagger/dagger/discussions/3800

[slender star]

dagger1004-first-env.md at v0.1.0 · dag...

Hey congrats on the python release spent

[slender star]

@rancid canyon Excited for your live coding today in a little over an hour: https://www.youtube.com/watch?v=eCKes2AKzK8

[rancid canyon]

Thanks, I am actually just trying to debug a weird thing, seems like clients file system is snapshot on connect

that is stopping me do a build with an asset from another step

[slender star]

Ah, interesting. Can you make a new call for the client host workdir?

[rancid canyon]

    golang := client.Container().From("golang:latest")
    golang = golang.WithMountedDirectory("/src", src).
        WithWorkdir("/src").
        WithEnvVariable("CGO_ENABLED", "0")

    golang = golang.Exec(
        dagger.ContainerExecOpts{
            Args: []string{"go", "build", "-o", "build/"},
        },
    )

    path := "build/"
    err = os.MkdirAll(filepath.Join(".", path), os.ModePerm)
    if err != nil {
        fmt.Printf("Error creating output folder: %s", err)
        os.Exit(1)
    }

    build := golang.Directory(path)

    _, err = build.Export(ctx, path)
    if err != nil {
        fmt.Printf("Error writing directory: %s", err)
        os.Exit(1)
    }

  client.Close()
    client, _ = dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))

  buildDir := client.Host().Workdir() 

  _, err = client.Container().
    Build(buildDir).
        Publish(ctx, "nicholasjackson/dagger-example:latest")

    if err != nil {
        fmt.Printf("Error creating and pushing container: %s", err)
        os.Exit(1)
    }

The only way I can get the build directory that I export is to close and re init the client

[slender star]

@sharp marsh ☝️

[rancid canyon]

I probably did not spot this before as I already had that folder but it appeared on ci

[sharp marsh]

@rancid canyon this is to reproduce something like multi-stage build, correct?

IIUC

[rancid canyon]

No just plain single arch

https://github.com/nicholasjackson/dagger-example/tree/dev

[sharp marsh]

ack. You don't need to Export to bring the output file to new Container. You can do something like this: https://github.com/kpenfound/dagger-demos/blob/main/go/getting-started/ci/main.go#L127-L141

[rancid canyon]

Oh I wanted that file for testing

[sharp marsh]

basically use WithMountedFile or WithMountedDirectory and reference another from a different pipeline

[rancid canyon]

Also build I think needs to run from the context

I tried the publish approach like you have it too, I was getting no joy with that

let me try that again

[sharp marsh]

You should be able to Export to have the built file locally and use WithMountedDirectory like here https://github.com/kpenfound/dagger-demos/blob/main/go/getting-started/ci/main.go#L130 to mount the build directory into your dagger-example container

[rancid canyon]

That works if I just do a publish not build

that is actually neater, weirdly it seem the position of WithEntrypoint matters

[sharp marsh]

👍 . Does it unblock you for the stream?

[rancid canyon]

Yes thank you

[sharp marsh]

❤️

awesome. Feel free to ping if you run into any other weirdness

We'll follow up on the entrypoint thing

[rancid canyon]

I really like the publish from container and no dockerfile, had not figured out how to do that so this is cool

[sharp marsh]

🚀 new content for the stream

[rancid canyon]

yes, ty

And everything works https://github.com/nicholasjackson/dagger-example/actions/runs/3456417072/jobs/5769136162

Now I need to feed the dog and get ready

[sharp marsh]

same 😄

[rancid canyon]

I normally don't leave these things so late but I have had to work all day

[pliant owl]

Hello wonder if anyone could help me - I was taking a look at the Dagger Go SDK and have also looked at CLI. How do these two fit together as the CLI uses CUE? Do you use one or both somewhat confused 🙂

[winter linden]

the cue-based CLI is deprecated, a language-agnostic version will be released next week

[pliant owl]

Awesome thanks for clarifying - so I should be able to use dagger CLI in the same way but with a Go based project instead?

[winter linden]

not exactly, in Go the CLI is optional: you write a regular go program, using dagger as a library, and run that

You can basically ignore the CLI for now

[pliant owl]

Thanks makes sense 🙂

[wispy wave]

this weekend, I started to migrate some recipes written in CUE to the GO SDK.it's a quick thing, to see what it might look like. I migrated only 2 recipes. but I'm a feedback taker. for to know if people might be interested in having this level of abstraction?
https://github.com/guiyomh/dagger-sheath-go

[shrewd wharf]

Hi, is there any plan to support python in the github action? https://github.com/dagger/dagger-for-github

[wary peak]

Is there a Go example somewhere of a pipeline that uses multiple containers and shares files / directories between them? The way I'm writing it right now I lean towards exporting files onto the host and then mounting them when needed.

But I was hoping I could reuse the *dagger.Directory type

[tardy mauve]

Before I go down this 🐰 🕳️ further, is there a known issue where mounting /var/run/docker.sock into a container does not work with the Go SDK? It returns an ERROR: error from sender: open /run/containerd: permission denied but it works fine with the docker cli e.g. docker run -v /var/run/docker.sock:/var/run/docker.sock docker docker run nginx sort of thing. This also worked when I used the Cue option, but that was a long time ago.

[tardy mauve]

Before I go down this 🐰 🕳 further is

[rain oriole]

Hey 👋 Just a ping that it's out now if you didn't see it already 🙂

[pulsar sigil]

Yeah! I'm playing with it a bit

Its really cool

I've got some fires I'm fighting but I intend to give this a proper shake down very soon

[rain oriole]

I'm currently working on hiding IDs (https://github.com/dagger/dagger/issues/3558) so DX will be even better 🙂

[chilly arch]

Hi Dagger Community! if you'd like to be a Node.js SDK tester, please DM me. We just completed the Getting Started Guide, so we are looking for users who have some time to test it out today and/or tomorrow.

[timber sphinx]

Could we get a releases channel for just release github/general release notification? I’be been swamped and it’s hard to sift through all the significant changes. I’ll probably dig back through github release notes too but thought it might be useful 🫡

[winter linden]

Great idea. How about an #announcements channel?

[sharp marsh]

+1. We've spoken about doing this pre-cloak. Given the amount of different releases we have today, it might be a good moment to implement it.

[timber sphinx]

I did change log driven releases for my team lately (using changie) and it’s really helped as only release notifications get piped through. An announcement channel would be great as long as it’s not dev level commits but release/tag driven

[chilly arch]

@wraith niche should I make an issue for this, or is it something that you can help set up without one?

[chilly arch]

@here Tomorrow's community call is packed with demos, so you won't want to miss it! Add it to your calendar with the link below. https://discord.gg/ZvQWgtyJ?event=1035645751797235723

[wraith niche]

+1 on creating an issue 🙂

[chilly arch]

done, thanks for the request @timber sphinx - https://github.com/dagger/dagger/issues/3883

[rich cairn]

Hello Dagger! We have a Java server app and our integration tests spawn docker containers (to test LDAP connectors). Can we still use Dagger? I mean, would it mean we'll have a container inception?

[sharp marsh]

👋 there's currently 3 ways to achieve this:

1 - You can start your containers before calling your dagger pipelines
2 - You can mount the dagger docker socket and make dagger call your docker daemon to start your containers
3 - Use the dagger services to handle service initialization in Dagger

Only 1 is possible at this moment since 2 and 3 are in the works and should be ready soon

[trim canyon]

Is there somewhere on the doc that tells me how to trigger a dagger script on a git push?

[sharp marsh]

👋 welcome! Dagger doesn't currently provide this functionality. Ideally this should be provided by your current CI platform

[trim canyon]

Thanks!

And yeah, sorry, Hi all!

[sharp marsh]

no worries! Let us know if we can help with anything else!

[trim canyon]

Yup! mostly i'm evaluating some build platforms for a project i'm doing and I like dagger cause I don't like doing my pipeline in yaml or groovy but the triggering part is... annoying

having to use another tool for that

is there a roadmap somewhere? Are triggers something that's in the scope of the tool?

[slender star]

Hi 👋 could you describe more how you'd like this to work? From your local dev environment, from Git server, from CI tool. Which CI tools do you use today?

Guessing Jenkins, since Groovy is in the mix.

[trim canyon]

nah, right now I'm on gitlab using its CI to build a docker image and to store it in a registry

id like to be able to have a git server on which I push my files with my "dagger.py" script.
On every push, dagger would run the script

[sharp marsh]

if you have your own git server, you can configure git hooks to call your dagger pipelines, correct? Which git server are you looking forward to use?

[chilly arch]

@here Introducing the Dagger Node.js SDK: develop your CI/CD pipelines in Typescript or Javascript, run them in containers anywhere. https://dagger.io/blog/nodejs-sdk

[half relic]

@chilly arch do we know if a c# sdk is on the roadmap?

[winter linden]

All SDKs are on the roadmap 🙂 We're still prioritizing though. Make sure to vote for your preference!

[swift barn]

hi guys, is it possible to avoid having an env var exposed within the dagger project codebase (e.g. container.WithEnvVariable in main.go) somehow by inserting it as a docker run -e parameter for example?

[winter linden]

Avoiding exposing env variables

[winter linden]

@half relic we just got a demo by @verbal canopy of a Powershell client, which is (I think?) a first incremental step towards an eventual c# SDK given the common .net DNA

[half relic]

Powershell would be great as well!

[winter linden]

https://github.com/cdhunt/dagger/tree/sdk/powershell/sdk/powershell 🙂

[chilly arch]

As mentioned during the community call today, we are making it easier to sign-up, get reminders, and attend the community calls. We are moving the community calls to Wednesdays at 9 am PST starting on November 30th.

**To sign up for the future community calls, please use this link. **- https://dagger-io.zoom.us/webinar/register/9716685521713/WN_USQjVBGXT0SWhNMvqYVvCA

As normal, you can always find the sign-up link or the recordings of the calls here. Note that recordings are added as soon as zoom finishes processing it. - https://dagger.io/events

[verbal canopy]

I'm messing around. I don't know what I'm doing, but I did get Dagger to do stuff with PowerShell.

[winter linden]

By the way @verbal canopy , is there something we should do for the dagger CLI to work well with powershell?

That CLI works great for bash scripts - would make sense to make it powershell-friendly too?

[verbal canopy]

Which CLI?

[winter linden]

https://github.com/dagger/dagger/issues/3347

Merged but not yet released 🙂

[verbal canopy]

I'll take a look but 0.2 worked fine

[winter linden]

well 0.2 was cue specific 🙂

the graphql part changes things

[verbal canopy]

This syntax won't work.

dagger query <<EOF
{
  container {
    from(address:"hello-world") {
      exec(args:["/hello"]) {
        stdout {
          contents
        }
      }
    }
  }
}
EOF

But I assume a powershell here-string will work fine.

Writing non-errors to stderr is not ideal

[pseudo stream]

In reference to the buildkit log outputs? I think we should be able to move them to stdout too if that works better.

[pseudo stream]

Just curious, is this a windows convention specifically? On Linux there's strong opinions on both sides whether stderr should be used for logging (see this unfortunate flamewar in buildkit: https://github.com/moby/buildkit/issues/1186)

[verbal canopy]

Powershell has 6 or 7 output streams. By default things written to stderr generate bright red messages usually with metadata about the source of the error. It's just very difficult to read on the console.

[pseudo stream]

Ah cool, very good to know! That makes the situation much less ambiguous then

[sharp marsh]

good catch. Wasn't thinking about Windows for this example. I'll double check this

[pseudo stream]

https://github.com/dagger/dagger/pull/3906

[sharp marsh]

@pseudo stream for listen that makes sense, but dagger exec also uses that function to setup the server and in that case we don't want logs to be in stdout since you usually care about the output in exec mode

[pseudo stream]

(👍 responding on the pr)

[trim canyon]

Sorry! I realized I never anwsered.

No, I don't plan on having a specific git server, just the basc scm one

[sharp marsh]

ok, so IIUC you'll keep using gitlab in your case, just not gitlab CI. Is that correct?

[trim canyon]

yeah, that's it

[sharp marsh]

go it. And where do you plan to run dagger? In your own servers?

[trim canyon]

yup

well, a VPC but yeah

[sharp marsh]

perfect. Well, yeah.. we don't have "webhooks" integrations natively since dagger is not currently designed to run as a long living engine ATM. Having said that, what you can do is to deploy a sidecar webhook receiver (https://docs.gitlab.com/ee/user/project/integrations/webhooks.html#create-an-example-webhook-receiver) along side your dagger server so each time gitlab sends you this webhook, you can do what you need there.

I know it's not the most amazing experience, but it shouldn't be complex to setup

[trim canyon]

yeah, that's about what I was planning! Shame dagger doesn't do it itself

is it out of scope for the tool or can we expect it at some point in the future?

[winter linden]

I think @cloud canyon has some ideas incubating along these lines 🙂 Native handling of events etc.

[trim canyon]

ooh, nice!

[winter linden]

It's a natural progression from "everything is platform-specific" to gradually shrinking the platform-specific parts, and growing the cross-platform part (in this case using Dagger)

[cloud canyon]

@trim canyon Yes! But it's still very early, but definitely something we're looking at. [@]vito (not tagging since he's on vacation) implemented something very similar and has lots of ideas

[chilly arch]

Recording from Community Call - Thank you @mystic mantle @rancid canyon @rain oriole @verbal canopy @winter linden for your demos today - https://dagger-io.zoom.us/rec/play/2YpjclVtXT11A91miZwbh-0A-MuTIMeswIlIp_rBcjO6ixzw6z9DvtGCnNeat86Ucgvdia6-fUM0zctC.NnTNdpktwedUf68-?continueMode=true

[trim canyon]

Excited to see what it could be

[ocean pelican]

Hi! dagger looks very interesting. Where can I find a roadmap?
Do you have plan to add C#\dotnet SDK?

[winter linden]

👋 no public roadmap yet, just lots of issues for feature requests, bug reports etc.

We do plan on supporting dotnet but no release date yet

We will soon release an API which you can use to target Dagger from any language, without having to wait for an official SDK

That’s a matter of weeks. Then you’ll be able to program Dagger with dot net, just will be a less polished experience until the full SDK arrives. Cc @verbal canopy who is doing exactly that with powershell at the moment

[swift barn]

Rust sdk is on the way?

[winter linden]

Not at the moment, but that could change quickly if someone starts prototyping one 🙂

[verbal canopy]

Is there a discussion or PR I could follow to inform my work-in-progress?

[formal token]

How should I be setting up the buildkit daemon to support custom CAs for image pulls? I notice the FAQ docs on dagger docs have vanished since the great sdkification

[chrome vessel]

Are there any docs on cross SDK actions? I see dagger.Serve in the examples and I'm curious if thats is for cross SDK actions, how that works, and how to consume it.

[winter linden]

Not yet 🙂 Extensions are still in development. What examples are you referring to?

[winter linden]

It’s mostly all merged already. Mostly documenting what you already figured out the hard way 🙂

Let me find the docs PR for you: https://github.com/dagger/dagger/pull/3917

[chrome vessel]

https://github.com/dagger/dagger/tree/main/examples/netlify/go

[rigid epoch]

apropos of nothing in particular: I've been learning about Tilt this week, and I feel like it (or maybe something like it with a slightly different API) would pair really well with Dagger. Specifically in the sense that Tilt revolves around tracking k8s resources and continually updating them with the output of a quick build process: that mental model seems like it would be extremely powerful in the context of a framework where lazily-built containers and directories are first-class objects.

I've had this sense generally that Dagger could have a powerful role in development-time builds, but seeing Tilt gave me a much better idea of what DX I'd want there.

[winter linden]

That makes sense @rigid epoch , especially since (I believe) tilt uses buildkit, so it could even be ported to actually use dagger 🙂

There’s also okteto that I believe is a direct alternative to tilt

[tall gulch]

When browsing docs.dagger.io I don't see the documentation is versioned, and instead I see numbers in the links, like 1221 in https://docs.dagger.io/1221/action. What is the purpose of 1221? I would like to have stable links that point to versions, is this planned?

[winter linden]

Each article has a unique random ID precisely to avoid breaking URLs. I agree we should add warnings when an article is no longer correct in more recent versions of dagger, like this particular article. We’re working on that.

Versioning the entire docs introduces its own problems, doing it article by article gives is more flexibility

[timber sail]

I was looking for a place to say this, and still not quite sure, but I'd like to thank the dagger.io Go SDK dev team members because this allowed us to build a completely testable CI pipeline (our previous CI pipeline was using GitHub actions, but as our CI implementation gets bigger, testing it before releasing it was practically becoming impossible). Thank you, team!

[rose ermine]

Heyo, are there any public examples of using the nodejs sdk and github actions?

[warm temple]

Hey 👋 I just pushed up my local version of the js getting started guide with a github actions workflow added here:
https://github.com/kpenfound/js-getting-started/blob/main/.github/workflows/test.yml
npm run dagger defined here: https://github.com/kpenfound/js-getting-started/blob/main/package.json#L25

[rose ermine]

thank you 🙇

[sharp marsh]

I'd like to see how testable pipeline looks like or the approach that your team followed to make it testable. cc @rancid canyon was looking forward to start testing his pipelines in his upcoming streams

[rancid canyon]

Yes, I was going to check that out this week but, I was pretty beat after travelling. I think I will do a stream on Sunday

[heavy karma]

another one - includes building and publishing an image to a registry - https://github.com/vikram-dagger/node/blob/main/.github/workflows/main.yml.keep

[heavy karma]

Need some help understanding why this nodejs sdk code doesn't work

  const source = await client.host().workdir(["node_modules/", "ci/"]).id()
  const node = await client.container().from("node:16").id()
  const c = await client
    .container(node.id)
    .withMountedDirectory("/src", source.id)
    .withWorkdir("/src")
    .exec(["npm", "install"])
    .withEntrypoint(["npm", "start"])
const dh = await c
    .publish("vikramatdagger/node")

when published, the container doesn't include the source or the deps

$ docker run -it --entrypoint /bin/sh vikramatdagger/node
# ls /src

Question: Do I need to manually copy the files from the mount point to a location in the container?

[warm temple]

Yes, they need to be copied to the container's rootfs as mounts aren't included in the publish
related: https://github.com/dagger/dagger/issues/3673
In Go that looks like

.WithFS(
    c.FS().WithDirectory("/src",
        source),
    )).

[sharp oracle]

Are there any sort of deploy docs available? I vaguely remember some parts of 0.2 docs talk about that, but I can't find anything in 0.3's. What I'm specifically interested in, if it can use a remote BuildKit cluster (probably can, given 0.2 did) and how does that affect functionality? For example I would imagine having just BuildKit means dagger won't be able to just run networked containers, right?

[slender star]

@sharp oracle things are still in progress. Conversations happening in issues like: https://github.com/dagger/dagger/issues/3939

[river rock]

Are there any sort of deploy docs

[tough elm]

Does anybody try the java project? It's stuck. But in GitLab CI runner, it works.

Can't get any valuable info for the log

It worked in GitLab ci

[slender star]

@tough elm can we make this a #help post? Can you give us a some code to reproduce this locally?

[tough elm]

of course

[tough elm]

Thanks for the reply, it may be because of my maven image, after changing to the official image, it works.

[slender star]

Great to hear!

[marsh ether]

small question -- the golang SDK looks great, but what I don't understand is how/when I'd care for the graphql API...? 😕

[elfin frigate]

@marsh ether (hi!) there's a good chance you would never have to care about it, since it's just the api being used under the hood. you might care about it if you're building an entire SDK or something though

[winter linden]

There are 3 reasons you may be interested:

1. If you want to program Dagger in a language that doesn’t have an official SDK yet
2. If you use an official SDK & want to dig deeper, understand the internals, use the API playground to experiment or debug a query
3. If you want to run pipelines from the terminal or a shell script

Like @elfin frigate said, if you’re in the “happy path” using an SDK and don’t fit in the categories above, then you can ignore it.

[marsh ether]

@winter linden @elfin frigate thank you two, that helped ✔️

[swift barn]

How we can use dagger for monorepo? can it replace bazel?

[slender star]

Short answer is probably. Lots of folks here interested in taming their monorepo CI. I'll tag you into a #help topic with some links.

[swift barn]

can anyone explain need of https://dagger.cloud/ and what problems will be solved by dagger cloud?

[slender star]

Before the multi-language SDK launches (Go, Python, Node.js), we supported only Dagger with CUE language. Dagger Cloud initially was a way for folks to share their Dagger CUE runs so we could help with debugging, etc.

We'll likely bring back that use case and others like visualizing your pipelines and many others 🙂

[swift barn]

ok thanks

can we use github pipeline visualisation feature with dagger? https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/using-the-visualization-graph

[mystic mantle]

Not automatically yet. Our way for now is to have created some magefile tasks, and we call them from individual steps in GHA.

[mystic mantle]

But we would love some deeper integration with CI tools, for sure.

We are a monorepo btw, we create subrepos out of our monorepo for some packaging needs, but those clones are read only, and dagger deals with the whole monorepo.

[swift barn]

Do you mean sub repo as sub modules?

[mystic mantle]

Not exactly. It's specific for a few items. Most notably the SDKs.
https://github.com/dagger/dagger/tree/main/sdk/go is its own Go module. Due to some limitation in the Go vanity URL protocol, we had to extract this module in its own repository in https://github.com/dagger/dagger-go-sdk, so when go get-ting dagger.io/dagger, it would get this repo instead. But we don't submit code to this repo, it is just extracted by our CI (https://github.com/dagger/dagger/blob/main/internal/mage/sdk/go.go#L99)

[blazing obsidian]

Hello guys, may be there was a related question already and I missed that, but why don't you use Dagger in your own GItHub repo?

[swift inlet]

Hi,
We do use it, it's wrapped around this script: https://github.com/dagger/dagger/blob/main/.github/workflows/test.yml#L70

Basically, we use it here: https://github.com/dagger/dagger/tree/main/internal/mage

[blazing obsidian]

I see, thank you! Didn't check well enough)

[swift inlet]

No worries, we're here for that 😇

[dawn patrol]

Hi dagger team!

I like the approach of Dagger, being independent from any CI/CD solution. I'm trying to understand if it fits our use case:

We have a C++ based project that uses CI and CD by executing the following steps:

• Install deps (cmake, gcc/clang, git)
• Clone a git repo
• Compile via cmake
• Execute the result to run unit and integration tests
• Deploy result via FTP

I have not been able to find an example how to do this with dagger though. Is dagger only for Go/Node/Python? I thought about "coding" the CI in nodejs even though the project is based on C++. Is this possible?
I fear that I didn't understand the concept yet. Lets say I have docker installed on Windows, I have the CI and want to create a windows build. How do I "spawn" a Windows image that has a compiler installed and such?

Help is greatly appreciated

[swift inlet]

dagger with c++

[vapid stag]

Its interesting to see magefile pop up in many places in/adjacent to the dagger ecosystem. Do we see magefile or something like it occupying a special place in the dagger golang sdk ecosystem down the road? Or is it more just a case of right tool for the job?

[hardy sparrow]

Is there any way to get the nifty dagger-cue buildx like console output when using the go-sdk?

[swift barn]

Very personal opinion: I do not think Dagger should be opinionated. Integrating with mage is very simple... what kind of integration scenarios are you thinking about?

[vapid stag]

For example mage gives a little more structure / guidance around how to organize commands, and provides a more explicit way of modelling dependencies than the golang sdk (albeit not connected with the buildkit deps mechanism which is at the core of dagger). I think with cue, when you look at a pipeline, its like ok yeah here are the steps/stages. With golang and i assume the other language sdks it can be a little less clear.

But im also just curious if there has been any thinking here. The fact that we see the two used in conjunction a fair bit is interesting to me.

[swift barn]

I agree, I started using Dagger as a replacement for Makefiles but I have found myself using it as a wrapper around Mage.

[past ivy]

I thought dagger was meant to save me from this 😂

[sharp marsh]

wondering the reason of so many failed runs since ideally is that you only need to do yarn / npm install and then run your script as you do on your local machine

[past ivy]

gotta run it with docker:dind so most of that is just getting that to work 😬

[sharp marsh]

I see, I thought docker:dind was pretty straightforward to setup in gitlab 🤪

[past ivy]

with the kubernetes executor on our own runners

[sharp marsh]

😄 that's another story

[past ivy]

And then the docker image doesn't have node installed (obviously), but because I was running it through zx it just... did nothing

so that took far too long to realise too

[heavy karma]

Yes, I too had a lot of issues getting dind working on a self-hosted Gitlab runner

@past ivy there are some samples here btw https://github.com/dagger/dagger/issues/3918

[past ivy]

not so many issues tbh, we do it elsewhere, just been a while since I last did it 😅

[winter linden]

@past ivy one approach we’re seeing is farming out pipeline execution to a remote dagger host, to remove the need for dind. The gitlab ci runner in that case becomes a dagger client. Is that something you would you consider, now or in the future?

[past ivy]

For sure. tbh the best would be to have all CI in dagger and have it act as an external CI tool (similar to how travis, jenkins, or anything else would work). But that'd require an API compatibility layer in between to register with Gitlab

[slender star]

Great to see @full geyser at AWS re:Invent in Las Vegas!!

[naive obsidian]

Hello Dagger community ! I'm starting to use Dagger and I have a few questions 😁

Since languages SDK (Go, Node & Python) are available, do you plan to continue supporting CUE SDK ?

What do you recommend for a team working with different languages ? In order to avoid having our CI conf in multiple languages, does CUE remains a good "standard" option ?

[sharp marsh]

👋 welcome Thomas! Yes, the CUE SDK will be supported like any other SDK.

re multiple languages: It really depends on how complex your use-case is and how many people need to interact with those pipelines. We do we have plans to make pipelines be interoperable across languages through what we call "extensions". Having said that, we generally recommend starting with a language you know since that will make the dagger onboarding easier.

[chilly arch]

@here Just a friendly reminder that our weekly community call time slot has changed, and we've made the registration for the calls much easier. The calls will be on Wednesdays at 9 am PST, so we will have one tomorrow. Use the link below to register and easily add it to your calendar. See you there! https://dagger-io.zoom.us/webinar/register/9716685521713/WN_USQjVBGXT0SWhNMvqYVvCA

[naive obsidian]

Hello @sharp marsh, thanks for your answers !

[sharp marsh]

sure! happy to help in any other questions you might have

[naive obsidian]

I definitely will ! As I've just started my Dagger exploration and I like what I see 🙌

[past ivy]

Ah great to see you here @naive obsidian! Enjoy dagger 😄

[naive obsidian]

Haha 😁 Thanks Leo ! Enjoy it too !

[sinful knot]

@naive obsidian, we can have a chat about it at the DevOps D-Day 😉

[naive obsidian]

Sure ! Are you going there today for the speaker night or tomorrow ?

[sinful knot]

I'll be there today

[naive obsidian]

Great, see you tonight then ! I look forward to discussing how you use Dagger

[chilly arch]

Community call starting now - see y'all there! https://dagger-io.zoom.us/webinar/register/9716685521713/WN_USQjVBGXT0SWhNMvqYVvCA

[wraith niche]

Also, first time we use the webinar setup. Seems that we had a glitch on the chat permissions at the beginning. If you have feedback about this (or if you prefer the previous call setting), please let us know. If you had questions you wanted to ask during the demos, please ask them here!

[chilly arch]

If you missed the community call, you can access the recording here -https://dagger-io.zoom.us/rec/share/jI1UV11gn7itNqJBUjLEwwuoSvfqxVg51UtWv8-71PE8wrFlysQuLUSV9brKUgBx.lDEeos_Yjd3OnHKb

[chilly arch]

Introducing the Dagger GraphQL API - https://dagger.io/blog/graphql

👆 As always, we love feedback, so let us know what you think!

[merry sentinel]

So i was looking at dagger awhile back then came back to it yesterday and noticed it seems to have been updated! Which is really cool and love the SDKs. I was wondering if the whole how to use dagger in certain CI/CD platforms is in the past with the SDKs? So like using it in github actions is the same as using it in CodeBuild?

[solid wraith]

Will the new cli load dagger.json file if one's in the work directory? https://docs.dagger.io/cli/389936/run-pipelines-cli

[winter linden]

dagger.json is not currently supported. It’s an experimental feature still in development (related to extensions primarily)

[glad jolt]

Hmm, no, it should be fixed with a rebase though.

Oh, thought we were waiting for input on the graphql query

I can merge as is after rebasing

Sounds good!

[winter linden]

Hi, are you asking if Dagger can still be run in different CI platforms (the answer is yes). Or are you asking if some of the old documentation for how to do this has changed? (also yes)

[slender star]

@rigid epoch I didn't realize how cool these WandB stickers were until I got home and found some sunlight!! Was great to stop in at your booth at re:Invent 🙂

[turbid tulip]

Those are pretty!

[warm temple]

Ok this is pretty silly, but I decided to try this year's advent of code https://adventofcode.com/ with dagger
day 1 and 2 are complete https://github.com/kpenfound/adventofcode/tree/main/2022
I'll keep up with it when I have time, so star/watch the repo if you want to see how it goes 😄

[turbid tulip]

Hey all! I've been working on the CUE SDK a good deal lately, and I'd love to hear from you all if you're interested in that and potential for the future of Dagger + CUE. I've got a GitHub discussion going here, feel free to join! https://github.com/dagger/dagger/discussions/4086

[swift barn]

Hi all! I'm wondering if it is possible to use dagger to run pipeline into existing containers (built with docker-compose for example)?
My use case: I have existing docker stacks with docker-compose, but I would like to write pipeline to execute tests suites with Dagger, inside those containers. Thanks for help

[wary peak]

I know that in the Go SDK the Container(...) function allows you to specify a container ID. I'm not sure if that ID can be the unique name of the container or something more specific but it's worth a shot.

[swift barn]

I tried this with the GraphQL API, but it wants a ContainerID, not the docker id of the container.

[wary peak]

yeah that makes sense.

[past ivy]

You can use the Build function to build from a dockerfile, does that do what you want?

[winter linden]

The term “ID” is misleading, it’s not an ID of a stateful container object like in docker, it’s really the serialized state of the pipeline producing the container. So it won’t help with your problem.

[past ivy]

Are there any ideas from the dagger side on how parallel execution and multiple builds should be handled? So in a monorepo of microservices, I'd have a build step for each service, which should be run when a service changes, or something it depends on changes (e.g a utility library). Are there any plans or ideas on having dagger help with this dependency graph resolution?

I'd also want to run the builds in parallel, preferably as distinct jobs so it's easy to see from our CI platform (now Gitlab, but hopefully a standalone for dagger in the future) which jobs ran etc

[past ivy]

After a brief walk to refresh my brain, I realised all that I really want at its core is:

1. Some way to retrieve the DAG of a pipeline execution
2. Support for re-running and stopping at a given point (this requires building some kind of persistence layer, likely shouldn't be part of Dagger core but some kind of plugin)

Those 2 would allow Dagger to entirely replace classic CI tooling like GH Actions or Gitlab CI/CD

[mystic mantle]

Hi, it's not ready, but it's clearly something we want (and many of our users as well).
There is an issue tracking this: https://github.com/dagger/dagger/issues/3037

[mystic dune]

Hi, I work in a air gapped environment and would need to download dagger directly from the s3 bucket referenced in the installation scripts but I am getting auth denied? the link is https://dagger-io.s3.amazonaws.com/dagger-cue/releases/dagger-cue_v0.2.232_windows_amd64.zip

[sharp marsh]

here's the correct URL: https://dagger-io.s3.amazonaws.com/dagger-cue/releases/0.2.232/dagger-cue_v0.2.232_windows_amd64.zip

[mystic dune]

ops... thanks

[mystic dune]

For the universe CUE packages, shall we mirror the github repositories or does the tool download the packages from somewhere else?

[sharp marsh]

https://docs.dagger.io/sdk/cue/809436/project-file-organization

apologies for being brief. I'm on mobile

[chilly arch]

If anyone would like to demo what you've been working on with Dagger during our community calls, please DM me! I can get you added to our schedule. You can check out what others have shared here - https://www.youtube.com/playlist?list=PLyHqb4A5ee1tEgcr7KsNFzQSPN-R3fPs2

[river rock]

Community Call Demos

[chilly arch]

Thanks to all of you for your support as we update the structure of our Community Calls. Many of you have already brought some awesome demos to the calls, and I can't wait to see what you bring to the next ones!

We are moving the Community calls from weekly to every other week. If you haven't already registered, you can do so at the link below. We look forward to seeing you at the next call on Thursday, December 15th at 9 am PST. If you would like to be added to the agenda, please DM me. Register here - https://dagger-io.zoom.us/webinar/register/9716685521713/WN_USQjVBGXT0SWhNMvqYVvCA

[cunning jolt]

Hi @chilly arch I assume there's no call tomorrow?

[chilly arch]

Correct, no call tomorrow. We will kick off the series on December 15th. We have an awesome lineup of community demos ready for y'all!

[rose ermine]

heyo 👋 is there a way to publish a directory to an OCI container registry (as an artifact) using the dagger sdk? I'm specifically using the node.js sdk atm, and I can see that there's a publish method on the Container type in the docs, but can't figure out how I would do the same for a directory.

[mystic mantle]

I guess you could create a container from scratch and add the directory you want from your build step

const buildDir = client.container()/* … whatever you do here */.directory("/your/directory")
const scratchContainer = client.container().from(/* put nothing to get a scratch container */).withDirectory("/", buildDir, {exclude: [".git" ]})
scratchContainer.publish("docker.io/me/myimage:latest")

(the API for this is based on the NodeJS SDK v0.3.0 which should get out today (notable the exclude part)

[rose ermine]

Oh cool, i had no idea it was possible to create a "scratch" container. I'll give that a try and see if Flux likes it 😅

[winter linden]

This is generally correct. You don’t need the from part, and you can use withRootFS

@rose ermine As a graphql query:

{
 query publishDir($dir: DirectoryID!, $address: String!) {
  container {
   withRootFS(fs: $dir) {
    publish(address: $address)
     ref
    }
   }
  }
 }
}

(field names may be wrong)

[errant mauve]

Hi - new to Dagger, I'm experimenting with it in a hackday today at work. I'd like to run the multiarch example from https://github.com/dagger/examples/tree/main/go/multiarch-build in our Gitlab instance, but, well, there's no docs for running pipelines in actual CI for the current version.

Is this not supported?

[swift inlet]

Hi, I guess you can base yourself on this docs, which was created for dagger-cue. You might need to tweak it to make it work with the Go SDK: https://docs.dagger.io/sdk/cue/470907/get-started#step-3-build-run-and-test-in-a-remote-ci-environment

[errant mauve]

@swift inlet thanks - will read 🙂

First hurdle, no dagger-go install...

curl -I https://dl.dagger.io/dagger-go/install.sh
HTTP/2 403
content-type: application/xml
date: Thu, 08 Dec 2022 10:42:12 GMT
server: AmazonS3
x-cache: Error from cloudfront
via: 1.1 7b5cd9167634df8189bb5a88ba570ee0.cloudfront.net (CloudFront)
x-amz-cf-pop: LHR61-P5
x-amz-cf-id: MQCFAQAZnwpbe0Ao9G-LO7i79DsbpPF1qTDH5S4RVeFv3jwvly2LCw==

So... as I have just 1 day, could learn CUE instead

[swift inlet]

moved here -> #1050357740528205824 message

[frozen basin]

Hey yall - been away from the project for a little, got a quick question re: Magefiles etc.

Will dagger eventually support running tasks natively similar to mage? I think there was some discussion about this a while back. Right now I just use cobra from my internal cli tooling.

The behavior I'm looking for would look something like this

# list tasks in a project
dagger list
# Run a task
dagger run foo

[winter linden]

Short answer: yes, but when we do it it must be cross-language (as opposed to eg. mage that is go-specific). This requires us building the technical foundation for it, specifically: extensions

[frozen basin]

Got it. Thanks!

[timber sphinx]

I just finished a big final push on migrating a project to a more modern release automation process with Mage and moving it from another org to my companies newer GitHub org.

As I wrapped up last night with a "live tweet" that compromised many memes and text like:

• here we go
• watching paint dry
• no deploy for you. forgot your env var didn't you!
• no deploy for you! helps to have an aws bucket listed
• No like CGO and cross compilation
• etc.... for at least a few hours....

It made me get excited all over again to jump back towards Dagger + Mage and see what magic has been happening. Been so swamped haven't gotten to try it but my goal is Jan is Dagger Month (maybe sooner if I can swing it 😀

[timber sphinx]

I'd be interested if any of your repos are public to see how you use it for task automation (I've used Mage for a while now) and would like to see other styles to be familiar with what folks are doing

[empty jay]

I'm working on https://github.com/mesosphere/daggers to create catalog and wrapper for the our org with mage and daggers. However, this is still in active development and I'm trying to figure out best practices and most useful UX around it.

I'm not really happy with current UX after using couple of internal repositories. I'm still thinking ways to improve it. I'm open to suggestion 😄

[empty jay]

is it a public project or an internal one 🙄 ?

[timber sphinx]

https://github.com/DelineaXPM/dsv-cli/releases/tag/v1.39.1

That was the big one I finished lately.

[vestal creek]

What's happening with cuelsp? Is it being donated to cue-lang org, or will it be deprecated?

https://github.com/dagger/cuelsp

[swift inlet]

Hi, it's a subject that we postponed with the launches, but we'll definitely have to discuss about it 😇 @slender star

[maiden cloud]

Hi guys. Long time I am not joining here... I am stunned by all the updates/changes that are going on.
I started playing with Go SDK and I was wondering if there is a final goal to ever reach Dagger Cue SDK (formerly Dagger CLI), if it makes sense.
I mean Dagger Cue SDK is really able to create portable Plans and to make them available throughout the various/major CI/CD providers. Will be something like that possible through Go SDK in the future or they are intended to remain separate entities with a different purpose?

[winter linden]

Welcome back! Yes, the goal remains the same as before. The major difference is that you can achieve it without having to learn a new language.

Right now you can already develop a custom CI/CD tool or script in the language of your choice, and share it in a portable way on any CI/CD provider. So, it's already quite similar.

The missing piece is what we call API Extensions: this will allow you to package your own pipelines into cross-language extensions, which can be called directly as GraphQL API calls. In other words: Dagger clients written in any language, will be able to invoke pipelines written in any other language.

In the end the resulting GraphQL schema will replace the CUE plan as the universal entry point for your CI/CD configuration.

[maiden cloud]

@winter linden thanks for clarifying. So how can I make a comparison between what I currently can achieve with Dagger CUE plan + Dagger Cue CLI installed on CI/CD provider (e.g. Github Actions) and what I can write down using an SDK (e.g. GO)?
Because right now I was able to replicate my Plan in the SDK but I am struggling understanding what I can do with that, compared to how powerful is porting my plan to any CI/CD provider by just installing the Dagger CUE client into.
I hope it make sense what I am writing, Maybe I am mixing up concepts and just doing confusion and noise 😵‍💫

[chrome vessel]

Is there a timeline on the API Extensions and is the work being tracked somewhere that I can follow?

[winter linden]

Yes, the missing piece at the moment is a standardized "loader".

• Before: only one way to load your pipelines, provided by Dagger: dagger do which reads a CUE plan
• After: many ways to load your pipelines: write your own tool or script in any language.

The benefit is flexibility and control. The downside (for now) is lack of a standardized entrypoint from the command-line.

For example, for our own CI, we use the Dagger Go SDK. The result is a simple Go program that we can run with go run. Our CI is pretty complex and modular (we need to build and test the core engine, the CLI, and each SDK). So we greatly benefit from organizing the code of our CI in Go. Here is the source code: https://github.com/dagger/dagger/tree/main/internal/mage

[winter linden]

@maiden cloud our discussion prompted me to track this problem in an issue: https://github.com/dagger/dagger/issues/4189

[maiden cloud]

@winter linden that is exactly my feeling. I felt a little bit lost, trying to connect the dots.

[winter linden]

Totally understandable. In a perfect world we would have shipped everything all at once. But re-engineering GraphQL + shipping SDKs was such a massive undertaking, we decided to ship it in pieces to get something to users faster. Now we are dealing with the tradeoff..

[maiden cloud]

But I am pretty sure that last weekend while I was digging the SDK guide, I cannot find that !
https://docs.dagger.io/768421/go-ci

[winter linden]

This was merged yesterday I believe 🙂 A nice contribution from @slender star

[maiden cloud]

That is much more clear to me

[frozen basin]

@timber sphinx - I put this little walkthrough together, not full code samples but the concepts are in there I think. If anyone reads and has thoughts I'd love to hear it!

the tl;dr is that I'm planning to just use cobra to kick off a pipeline, and try to use standardized interfaces to execute builds based on language/type.

https://blog.stobias.dev/post/2022-12-12-dagger/

[maiden cloud]

We are digging into Dagger GO SDK... We are debugging the point where the dagger client is calling.
There is a dagger-engine container spawn which is running, that is fair enough.
However the library is using an HTTP client to call an endoint http://dagger
Where this hostname comes from? Meaning how it is resolved and where to?

[frozen basin]

I think the http call is literally to that dagger-engine container. the GO SDK just fires off GQL queries to the dagger server

[maiden cloud]

meaning for dagger server the dagger-engine container?

[timber sphinx]

I love that y’all have embraced mage. It’s been such a godsend in my automation efforts. I’m assuming based on your post that dagger do isn’t the primary invocation approach right now and mage is?

[winter linden]

There is no primary invocation approach. Mage is just one among many ways you could load and run dagger pipelines from the CLI. That is causing some confusion at the moment

Which I documented here: https://github.com/dagger/dagger/issues/4189

[chilly arch]

Thanks for this write up @frozen basin ! Since you mentioned logging towards the end, @cloud canyon is going to do a demo of the latest logging improvements on tomorrow's community call. Would love to get your thoughts after his demo!

[timber sphinx]

added to my newsletter and subscribed. Keep up the good writing! I love stuff like this.

[thorn axle]

Howdy everyone! 🙂 Quick question, will the community call be recorded and uploaded somewhere?

[winter linden]

Yes! We share each presentation on our youtube station, and the whole recording in this channel

[thorn axle]

Thank you!

[mystic mantle]

Community Call started: #announcements message

[chilly arch]

Yes, you will always be able to see the recordings of the community calls on our events page under previous events. I update the link for the call with the recording within 24 hours after the event -https://dagger.io/events

[small obsidian]

Is there a way to create and access DAG directly? In my project I want to wrap cue and dagger to execute some tasks on the host before and after execution of multiple pipelines in containers. These steps have dependencies among them (as well as dependencies between pipelines). The plan is to define dependencies using something similar to cue flow DSL. Then extract DAG from cue and feed it somehow to dagger to get correct execution order. Based on prior discussions in this channel I've got an impression that cue is getting replaced by GraphQL in dagger. Any ideas on: 1) a way to convert cue definitions into a GraphQL query of the right shape? 2) How to express external steps as nodes in the DAG?

basically some parts of the project would need to be build on the host and some other parts in containers.

[winter linden]

There is ongoing work to port the CUE SDK to the new graphql API. I think that achieves what you’re looking for. Are you looking specifically for compatibility with the Dagger CUE configuration, or looking to follow similar patterns with your own cue dsl?

[small obsidian]

Own cue DSL

[winter linden]

Great, so you can look at @turbid tulip ‘s work for inspiration. He got 80%+ of it working. We were actually considering spinning it out into a separate community project in case you find that useful

[turbid tulip]

https://github.com/jlongtine/dagger/tree/cue-sdk

[winter linden]

@turbid tulip did you settle on the ideal codename? 🙂

[turbid tulip]

I kinda like Amalthea

[chilly arch]

12/15 Community Call Recording - If you missed it, we had some great demos from @sharp marsh, @fossil pine , @river rock , and @cloud canyon - https://dagger-io.zoom.us/rec/share/nw_UfKVh1lyJMP6p49eR-s-fxOguUH0dHpAbf03Ygksr4mznfookKKUOJqsV5qtI.b8y-28mJMCLfQhfX

[chilly arch]

If you'd like to see the demos as snippets, you can check out our Community call demo playlist here - https://www.youtube.com/watch?v=oKO8WjyxRKY&list=PLyHqb4A5ee1tEgcr7KsNFzQSPN-R3fPs2

[real kite]

Is there an existing pattern for running docker-compose like stuff in dagger? For instance, I have a Supabase app that I want to build and test with Dagger. The easiest (and most production-like way) to do this is by using Supabase CLI, which spins up a local docker-compose instance and exposes some ports. It would be nice if I could spin one of those up with Dagger during my CI build to run tests with

[sharp marsh]

👋 conor. Native "services" support is still under design and something that we'd definitely like to add soon. In the meantime, we're recommending users to spin up their dependencies using compose before calling dagger. We know it's not ideal but it's a workaround until we can ship an integrated solution for this use-case.

So basically:

docker-compose up -d <my_services>...
go run my_dagger_pipeline.go

[real kite]

Thanks for the answer!

[vernal sage]

Hi. I'm really new to dagger. I don't know if there is a way to execute an xcode build command (to build an iphone app) on dagger ? Using a macos image?? Thanks for the answer

[slender star]

Hi @vernal sage 👋 Welcome! Do you use any MacOS images today? I've not tried any. I'm seeing this in Docker Hub for the first time: https://hub.docker.com/r/sickcodes/docker-osx

Would that sort of image work for your use case?

[celest crane]

Hey guys 👋 I did an introductory conference talk on dagger in november and did a significant part about cuelang (because I like the declarative approach) and only briefly touched the new SDKs. I want to rework the talk for 2023 now and after stumbling upon this note in documentation „a fan of the CUE language, and don't mind using the old Dagger engine“ I‘m wondering if cuelang is already deprecated? Did I miss something? Thx for clarification

[vernal sage]

I'm not sure, today I'm using a GitHub action running on a macos job. I thought it was a docker image. But it is more probably a VM. I've also seen the image you mention today. Maybe I should give it a try

[winter linden]

CUE

[slender star]

Yep, it's a VM runner. https://github.com/actions/runner-images/blob/main/images/macos/macos-12-Readme.md
If you want to try the above Docker image, you could run your GitHub Actions workflow on an Ubuntu runner which has Docker installed and then use your favorite Dagger SDK 🙂 I'm eager to try that myself when I'm back on a fast internet connection in a bit.

[timber sphinx]

To be clear if I start up a new dagger piepline with Go sdk the interface to Dagger right now is either go run or mage etc. Dagger do and such discovery per prior posts is not working in new SDK at this time right? So dagger is providing me the sdk but not the discovery/execution cli right now correct? Been swamped and not caught up so sorry if this is already answered.

[winter linden]

Yes that’s correct, that’s a hot topic right now 🙂 See https://github.com/dagger/dagger/issues/4189

You can dagger run go run or dagger run mage as an optional convenience

[sinful knot]

https://tenor.com/view/flintstones-yabba-dabba-doo-gif-11422371

how about "yabadabadoo !" 😆

[vernal sage]

I've tested the docker osx image on my computer. It is impressive. Just run the command and you have a Mac running on Linux. But it's not really a Mac container its just a Linux container that configure a Mac VM for you using KVM. The host KVM device must be mounted inside the container, and you must proceed to a full macos installation the first launch. And the resulting volume might be quite large (I've not terminated the install to test)

[wary cave]

Does anyone have a sample python build.py they can share quickly that does a simple dockerfile build + push ?

Looked through the docs, but it's mostly about building, not much about pushing or using dockerfile.

Also wondering if it's more idoimatic dagger to use DIND to do dockerfile builds ?

[sharp marsh]

👋 this should help out:

import sys

import anyio

import dagger


async def test():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
        python = (
            client.container()
            # build container
            .build(client.host().directory("."))
        )

        # push
        image = await python.publish("ttl.sh/footestdagger:1h")

    print(f"Image pushed {image}")


if __name__ == "__main__":
    anyio.run(test)

[sharp marsh]

basically build is the operation that you need https://dagger-io.readthedocs.io/en/sdk-python-v0.2.1/api.html#dagger.api.gen.Container.build

[sharp marsh]

not sure what you mean with this. Dagger currently requires docker to provision a buildkit engine for builds.

[wary cave]

@sharp marsh perfect, thanks! I tried a few commands but not exactly sure how they all relate. got it working now

[sharp marsh]

would be good to know what we could improve so this is more clear for future newcomers 🤗

[wary cave]

Yeah I think having an examples for some of the most common basic usecases would be nice. (Dockerfile builds, multi-stage builds, monorepo builds, docker login and pushes etc.)

The old cue docs have it but couldn't find them on the Python ones.

[sharp marsh]

👍 noted. We're improving our docs with the objective to provide basic multi-lang examples for the basic operations 🙏

[loud briar]

I'd second this. I've pretty much figured it out from various language SDK docs but the really basic stuff like building an image and pushing to a repository then updating a service is so useful. I found a good video by Containers By The Couch on YouTube that with enough pausing was enough to figure it out, but putting that on the dagger docs would be a win.

[celest crane]

Hi, i have a question about terminology: coming from the cue sdk i'm used to the concepts "plan", "action", "input" and "output". Will these concepts be brought to the SDKs, or is the wording now more aligned to the general "container" terminology?

[winter linden]

The terminology is different in the new SDKs to reflect the fact that there is no unique declarative configuration. Instead of writing a plan in cue, you write a regular program in the language of your choice, that then executes pipelines.

[winter linden]

Inputs and outputs as used in the CUE SDK, are also CUE-specific concepts, to compensate for the absence of functions with arguments and return values. Obviously other languages have that already, hence no need for a new word.

[celest crane]

That sounds reasonable. I was already thinking that it would not be a good idea to kind of force a common wording onto so many SDKs for different languages with different concepts. The result would be the worst of all worlds.

[fossil pine]

I've submitted two Dagger talks to FOSDEM and both have been accepted.

Here is the one for the Go devroom: https://fosdem.org/2023/schedule/event/gocidagger/

The CI devroom schedule is not public yet.

[sharp marsh]

that's great news Mark! let us know whatever we can help with if you need anything from our side for your presentation 💪

[tardy mauve]

👋 I don't think this is possible (yet?? 😄), but does Dagger supports pruning its cache? It's pretty useful if you need to re-run steps when developing your pipelines.

Earthly has this feature by dong earthly prune

[sharp marsh]

we have dagger-in-dagger support in the playground 😉 https://play.dagger.cloud/playground/XiV8l-ElLIi. Now we can use the playground to test and ANY SDK with minimal effort ;). cc @winter linden @cloud canyon @pseudo stream @glad jolt @slender star @warm temple @rain oriole @mystic mantle @last pine

You'll also noticed something in that link which is very cool

[sharp marsh]

we have dagger in dagger support in

[mystic mantle]

That's amazing! Well done!

I hope I can go this year

[winter linden]

The embedded go function is hilarious @sharp marsh 👍❤️

[winter linden]

I’m giving Mastodon a try today… Who else is there? My profile: https://hachyderm.io/@shykes

[last pine]

I'm in : https://hachyderm.io/@jffarge

nothing posted yet

[swift barn]

@[e-mail] 🙂

[fossil pine]

I'm looking at @warm temple 's nomad example, specifically the secret mgmt part: https://github.com/kpenfound/hello-nomad/blob/main/ci/utility/secrets.go#L46

It seems to me that the secret is cached on the runner machine (unencrypted).

Am I right in my assumption?

[mystic mantle]

Didn't have time to play with the self hosting the mastodon instance, so I joined hachyderm as well.
https://hachyderm.io/@dolanor

[turbid tulip]

Yes, that's correct. I'm working hard to try to get some clarity around how Secrets can/should work: https://github.com/dagger/dagger/issues/3442

[fossil pine]

Awesome!

Thanks for the link!

[real kite]

Is it expected behavior for client.directory() to return anything other than an empty directory? Docs say that it should return an empty directory if no id is supplied, but when I call it in my code I get the directory contents from a previous run. Is that expected behavior?

[carmine sandal]

the new dagger engine container doesn't work with docker compose?

the container can't access my other services in host

nvm i think its relevant to https://discord.com/channels/707636530424053791/1055609569356820570

[winter linden]

That doesn’t seem to be the expected behavior. Would you mind opening a github issue please?

[carmine sandal]

does Dagger works with remote buildkit?

[sharp marsh]

yes, but there are some considerations to take into account. Check this thread: https://discord.com/channels/707636530424053791/1042793428775350273

[lusty void]

Hi, I'm really like dagger's idea of implementing the whole pipeline as code. Correct me if I'm wrong, currently dagger is limited to single machine execution. Meaning that the whole pipeline will always run in the same machine. Is there any plan to integrate dagger with orchestration platform like k8s? I'm thinking of more like an workflow engine like tekton/argo workflows but as code

[swift inlet]

Hi @lusty void ,
You're right regarding the current state of Dagger 😇
Regarding the second part of your question, it is something under discussion / design. The idea is indeed to be able, at some point, to cluster the engines together. As it is something quite tricky, the first step, on our side, is to enable the use of custom provisioned engine image, more context on this issue https://github.com/dagger/dagger/issues/4105

[rain quiver]

Hi,
I discover Dagger and found it very interesting, I'm currently in my last year of DevOps Engineering and I'm searching for an Internship to validate my diploma. I was wondering if you’d have any opportunity at Dagger.
PS: I hope the place isn’t inappropriate for this question

[candid wave]

im kinda missing docs on how one would integrate dagger with forges like gitea, gitlab or github to run ones pipelines against prs /eventsof a repo - is there any quickstart for that?

[candid wave]

More specifically, it seems like there is no native dagger integration, only indirect kickstarters

[swift inlet]

Hi @candid wave,
Here is the available docs on how to integrate Dagger on most forges: https://docs.dagger.io/768421/go-ci/. It can be easily tweaked to all SDKs.

If some are missing, we're really open to contributions 😇 Regarding the native dagger integration, we currently don't have "apps" available to integrate Dagger on the pipelines. However, it's under discussion. We used to have a Github Apps, but it hasn't been updated to fit with latest Dagger engine.

There's just one thing I am not sure to understand: to run ones pipelines against prs /eventsof a repo. Currently, the context on which a Dagger pipeline runs lies inside the forges (to run a pipeline against a PR, specify that in your workflow). Does it answer your questions 😇

[swift inlet]

Hi @rain quiver,
We're always looking for talents 😇 . You can apply (even spontaneously) here: https://dagger.io/careers. You will find most of the context there. Please apply to the software engineer field, and mention that you are looking for an internship

[candid wave]

this practically means that dagger is not a standalone system, its always a "piggyback" system - which makes it far less attractive, as i'd have to deploy a ci system against my forge anyway (gitea in my case)

[swift inlet]

I understand your point of view 😇. As we you can use Dagger with code, there's no limit per-se, you could totally code that internal logic in your preferred language. By the way, the cache management of Dagger might help you implementing it.

The truth is that we had experiments to do exactly that. However, the investment was getting too big for the stage of the company.

Considering that most of forges nowadays include a ci system (Github Action, Gitlab ...), or that most of the users here alternatively rely on a CI system, it seems to be an acceptable tradeoff. Especially because most of CI systems accept containers, and integrate perfectly with Dagger.

Even though it would be nice to have that in Dagger, you're 100% right, we first prioritize the stabilization of the engine, and we believe that the fastest way to solve the run your pipeline locally and easily need is to focus on the engine for now. We hope that our solution is such a better developer experience that users (and you) would be ok to rely on a CI system as a tradeoff 😇

PS: nothing is set in stone + it's a personal opinion 👼

[candid wave]

i can totally get that part - i would recommend highlighting that mode of operation , currently i'm not in a position to invent a ci integration for my forge of choice and i want to avoid nested systems as of now (thy caused me headaches before)

[swift inlet]

Could you please make an issue, so that we don't lose track of that recommendation 🙏

[candid wave]

https://github.com/dagger/dagger/issues/4267

[rain quiver]

Thanks for answer

[thorn axle]

Just got the go ahead from upper management to automate our aws lambda deployments with dagger! We currently don't have any CI/CD in place for our deployments, so this will be interesting. I will be using CircleCI to run the dagger pipelines in our hosted runner. Super excited to deep dive into dagger and will post any interesting finds I come across

[swift inlet]

Such great news 👼 Do not hesitate to ping us if you have any question along the way

[winter linden]

AWS lambda deployment

[rough wind]

Is it possible to run health check on a MySQL DB with the graphql API?

[winter linden]

If you mean a mysql DB running externally to Dagger, then yes you can orchestrate running a mysql client in a container(or raw tcp port checker depending on the kind of health checklist logic you want).

If you mean for a mysql db run by dagger, that will require the services API , coming soon, and container networking which is also in development and includes a proposal for builtin TCP port checks. See https://github.com/dagger/dagger/issues/4080

[rough wind]

Cool, I'll wait my use case requires spinning up MySQL and another cli and let them communicate

[winter linden]

A perfect use case for services 😁

@rough wind would you mind adding a comment to that container networking issue, to mention that your use case requires it? It helps us prioritize, and keep you informed of our progress. Even a few words is fine.

[rough wind]

Done!

[rigid epoch]

I just had a kind of wild idea: how practical would it be for Dagger to upload cache in parallel with the execution of other steps? Maybe this is more of a buildkit question...

The reason I'm asking is because I was just pondering the idea of using spot instances for my integration tests. Because they're already chunked out into convenient units of work and highly parallel, they should be easy to preempt. And then it occurred to me that buildkit execution steps have this same property -- at least, if their cache has been saved.

This probably isn't suitable for all workloads, but just thinking about how much of ci is running on eg in development feature branches, I think there's a lot of cases where 5% interruption chance would be worth huge savings on compute. And that's something dagger could facilitate, on top of what it already does for caching.

[raw juniper]

Hi Every one, so i already have a react project which i wanto start using dagger with typescript/nodejs however if i change my project to type module it breaks.

[sharp marsh]

cc @last pine @wispy tapir

[last pine]

Hey @raw juniper 👋 What happens exactly ?

[raw juniper]

This is with type=module :
(node:52313) ExperimentalWarning: Custom ESM Loaders is an experimental feature. This feature could change at any time
(Use node --trace-warnings ... to show where the warning was created)
ReferenceError: exports is not defined in ES module scope
at file:///Users/mahmoudnashef/Work/workcanvas/build.ts:2:23
at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
at async Promise.all (index 0)
at async ESMLoader.import (node:internal/modules/esm/loader:541:24)
at async loadESM (node:internal/process/esm_loader:91:5)
at async handleMainPromise (node:internal/modules/run_main:65:12)

[raw juniper]

but type module breaks my app. so this is without type module:

(node:52554) ExperimentalWarning: Custom ESM Loaders is an experimental feature. This feature could change at any time
(Use node --trace-warnings ... to show where the warning was created)
/Users/mahmoudnashef/Work/workcanvas/node_modules/ts-node/dist/index.js:851
return old(m, filename);
^
Error [ERR_REQUIRE_ESM]: require() of ES Module /Users/mahmoudnashef/Work/workcanvas/node_modules/@dagger.io/dagger/dist/index.js from /Users/mahmoudnashef/Work/workcanvas/build.ts not supported.
Instead change the require of index.js in /Users/mahmoudnashef/Work/workcanvas/build.ts to a dynamic import() which is available in all CommonJS modules.
at require.extensions.<computed> [as .js] (/Users/mahmoudnashef/Work/workcanvas/node_modules/ts-node/dist/index.js:851:20)
at Object.<anonymous> (/Users/mahmoudnashef/Work/workcanvas/build.ts:3:18)
at m._compile (/Users/mahmoudnashef/Work/workcanvas/node_modules/ts-node/dist/index.js:857:29)
at require.extensions.<computed> [as .ts] (/Users/mahmoudnashef/Work/workcanvas/node_modules/ts-node/dist/index.js:859:16)
at async Promise.all (index 0) {
code: 'ERR_REQUIRE_ESM'

[last pine]

Could you show me your tsconfig please ?

[raw juniper]

shared privately

[sharp marsh]

hey David, good idea. I'm not sure if buildkit allows exporting the cache while executing other steps. IIUC exporting the buildkit cache is generally the last step after the build has finished using buildctl. Having said that, by using the buildkit API directly there might be a way to achieve what you're thinking which I also believe it's very valuable.

i.e:

c.Container().
From("maven").
WithExec([]string{"mvn", "install"}).
Export(). // find the way to do this async while starting the next exec?
WithExec([]string{"mvn", "package"}).
ExitCode()

^ is something like this what you had in mind?

cc @pseudo stream @elfin frigate

[rigid epoch]

Yes! But ideally this would be done automatically, to give the ability to resume at any step

[sharp marsh]

I'd guess automatically would be a bit inefficient since there are some layers that don't make a lot of sense to cache. For example, a go build layer doesn't make a lot of sense to export as cache since that's something that you'll very likely want to compute each time.

[rigid epoch]

I'd think that's a great example of something you'd want to cache, actually -- if the layers leading up to the build turn out to be the same, you'd want to be able to skip the build

In a monorepo that sort of thing happens a lot (in my experience)

[sharp marsh]

well.. it really depends. Imagine the usual common scenario of:

• Pull base image
• Add go.mod and go.sum
• Pull dependencies
• Add source code
• Build application

Generally you wouldn't want to cache the "add source code" and "build application" steps since those are the ones that usually change and you need to re-run. Having said that, dagger/buildkit already caches everything automatically as long as you're running the pipelines using the same buildkit underlying volume. I am assuming that in your case you're referring to export/import cache since you want that to be ephemeral across spot instace runs

[rigid epoch]

I am thinking about export and import, yeah (although obviously a volume is better if you have one). And that makes sense about the copy steps, I'm so used to thinking of cache as an all other nothing proposition (just set inline cache to true and forget about it)

[winter linden]

@rigid epoch may I suggest an issue for continuity? 🙂

[rigid epoch]

happy to! I wasn't sure if this was concrete enough to issue-ize. Is the issue "support for pre-emption," or "parallel cache export" or something else? 🤔

[winter linden]

In doubt I would recommend focusing on the problem you want to solve, that way it leaves room for exploring solutions together

[rigid epoch]

Sounds good -- I'll write something up!

[real kite]

For environment variables on the host, I noticed that if you do this:

 initialize the dagger client -> do something to load some new environment variables onto the host -> attempt to call client.host().env_variable("MY_NEWLY_LOADED_ENV_VAR")

it won't find the environment variable. The env var appears to have to exist before the client is initialized. This is in the Python SDK

1. Is this expected behavior?
2. If 1, is loading environment variables onto the host after client is initialized unsupported?

You can reproduce with this simple slice of code:

import os
import sys
import anyio
import dagger

async def run_workflow():

    config = dagger.Config(log_output=sys.stderr, execute_timeout=3000)
    os.environ["MY_VAR"] = "test"
    async with dagger.Connection(config) as client: 
        # os.environ["MY_VAR"] = "test"
        my_var = await client.host().env_variable("MY_VAR").value()
        my_container = (
            client.container()
            .from_("alpine")
            .with_env_variable("MY_VAR_INSIDE_CONTAINER", my_var)
            .exec(["/bin/sh", "-c", "echo $MY_VAR_INSIDE_CONTAINER >> /var.txt"])
            .file("/var.txt")
            .export("/tmp/var.txt")
        )
        await my_container
        with open('/tmp/var.txt', 'r') as f:
            print(f.read())



if __name__ == "__main__":
    anyio.run(run_workflow)

The above prints test as expected.

But if you comment out the first os.environ["MY_VAR"] = "test" and uncomment the second, it prints nothing

[rigid epoch]

https://github.com/dagger/dagger/issues/4279 !

[tender stump]

its is possible to run other CLI or custom script and follow the output? (something like this https://docs.dagger.io/sdk/cue/006395/client#running-commands but "with tail" so i can check possible error/warning from the command/cli while running)

[woven tide]

I think dagger-cue --log-format plain does what you want

[tender stump]

i tried but the final outcome of the command comes at the end

for now I'm using core.#Nop (because // using #Nop because we need an action for the outputs ) but I'm not sure if its the right approach

[woven tide]

Mhm yes I see

[sharp marsh]

does it need to be a local command or can you run it inside an action?