general

[fossil halo]

is there some trick to having “local” steps in pipelines? — certain things can be quite heavyweight to execute in a container, but it would still be great to express as part of the dagger pipeline to get the visualization and in general “fit in” the execution flow — any thoughts on that?

[winter linden]

At the moment you can't do that, but depending on your use case there might be workarounds. What heavyweight tasks do you have in mind?

[fossil halo]

an example would be something that is already doable with the binary that is built by dagger — but doesn’t need to be containerized — say just a quick processing function for some input or something — does that make sense?

[chilly arch]

Thank you to @real kite , @scenic ridge and the whole Airbyte team for sharing their Dagger story with us! https://dagger.io/blog/airbyte-use-case

[swift barn]

Any thoughts on this one? https://github.com/dagger/dagger/issues/5981 Not quite sure if I framed my problem understandable

[fallen marlin]

Hi, I'm creating a pipeline with dagger from scratch so actually we have all the test and build part in one dagger script. Now we would like to implement our deployments steps in this pipeline, but I'm asking how people are handling deployment to multiple environments sequentially

In my case we want to do a first deployment in staging then in our production environments
For beginning the deployment in staging is done when there is a merge in main and for production it will be triggered manually from github action
How do you recommend to handle that ? (multiple script / env vars / ...)

[timber sphinx]

Gonna be in Paris in November for the first time. Any get dagger magic planned? While I don't speak French I'd be happy to enjoy the musical language and pretend I follow with some profound mumbling and note taking. 😆 🇫🇷

[sand gorge]

It is done. TestCon in lithuania was much larger conference than I expected.

Someone in the community sent me this 😀

It concludes my tour. Home to Scotland next week to re-charge and then we move forward with our plan 😎

[sharp marsh]

✨ AllowToFail in ContainerWithExecOpts ·...

[fossil pine]

404 issues not found 😄

[cunning jolt]

sweet! My issue made it 404 🤣

[winter linden]

Docs feedback

[slender charm]

Hey! Could you move this question to #1030538312508776540 so that it does not get lost in the mix here.

[fallen marlin]

Hi ok thank you I just back port my message

[runic heron]

Is it possible to name service containers so that the display shows something more informative that random sha like name? I couldn't seem to spot anything

[sharp marsh]

If you add a Pipeline call to the service does it get displayed any different?

[runic heron]

Nah - I tried that.

could be a nice addition - it's small and subtle, but...

[sharp marsh]

Agree. Could you open an issue and tag @elfin frigate please?

[cunning jolt]

This issue also could be bunched up with that - https://github.com/dagger/dagger/issues/5994

[graceful pawn]

Reading the docs, I'm a little perplexed by the meaning of:

In addition to cache volumes, Dagger has a separate cache for pipeline operations.
Am I correct in saying that each .WithExec results in a BuildKit step that is pretty much equivalent to a RUN ... inside of a Dockerfile, thus taking advantage of Docker/BuildKit's caching? Which is maybe also the reason why users are required to specify which files they need for a job using .WithDirectory

[slender star]

Hi @graceful pawn 👋 . Yes, Dagger does both layer caching and has cache volumes for directories full of files (good for build dependencies, for example).

[graceful pawn]

Cool! So executing a dagger pipeline is more or less like building a Docker image by stacking layers, and then discarding the actual built image

Also this means that the cache must be stored somewhere in the cloud. But it’s storing not just the hashes of the files in the repository but the entire Docker filesystem. So if I run a “npm install” step for instance, I’m putting the file system layer containing node_modules into my GitHub Caches right?

[slender star]

I see the reference here: https://docs.dagger.io/quickstart/635927/caching/. You're correct that the node_modules would be stored in a layer, but if anything changed upstream of that step in the course of the build, that downstream layer in the cache would be invalidated and npm install would be re-run. That makes the node_modules cache volume useful as a cache outside of the layers scheme.

The Dagger caches work locally on your laptop automatically (thus enabling fast local iteration) because you have local storage for a long-running local engine. In CI, there is often an ephemeral CI runner and Dagger Engine running there, so having the Dagger Cloud cache keeps the layers and volumes persisted between runs. It's also great for an organization since everyone's activity feeds the cache.

[graceful pawn]

That explains it, thank you very much!

[runic heron]

Raised as https://github.com/dagger/dagger/issues/6013, and linked to 5994 as @cunning jolt suggested.

[runic heron]

Speaking of caches, and cache-volumes specifically - when does the cache volume get "updated", have been experimenting with forking sub-piplines on a Container that uses a cached volume, but changes in one fork don't seem to be visible to another ( 3 forked pipelines write to the cached "artifact repository", then all .Sync()'d - but the main build doesn't seem to see those cached items.

I'm assuming the cache only gets flushed at the end of the entire pipeline - unless I can trigger a flush or something?

Weird - I don't know why - but Apache Maven running directly on our AZDO agent was downloading from nexus at like 20 kB/s - and slooooooow as hell when it's cached was empty. Running under Dagger - I'm getting 2 MB/s - so a build that was taking almost 30-40mins to just download dependencies, let alone the build - now runs ENTIRELY in 13 minutes. And that's without any distributed caching.

[sand gorge]

Random question: Many devs already have docker files.. to give them their env.

I know from the SDK we do from(image here)

What is the approach now for people using Dagger ..

How would someone call "docker build" from a dagger file ?

I have to advise people to "granularly" transition .. so stopping them from using docker build is not feasible

Thank you

[narrow nymph]

You can build docker images from Dockerfiles using Dagger natively: https://docs.dagger.io/cookbook/#build-image-from-dockerfile
Since dagger is built on buildkit, it's using exactly the same mechanisms as calling docker build 🙂

[sand gorge]

Yes, exactly. I thought/knew this but wanted validation. Thanks for the link

[stuck wyvern]

Is it not possible to add a tag annotation to an exported image?

[next fulcrum]

I'm glad to see Dagger directly support basic service management in 0.9! Has the work done by Aurae on figuring out a minimal API for remote management of isolated processes/containers/VMs been considered while designing Dagger's service management API?
https://youtu.be/5a277u4j6fU (https://archive.fosdem.org/2023/schedule/event/rust_aurae_a_new_pid_1_for_distributed_systems/)
https://aurae.io/

Note that Dagger wants to provide higher order scheduling mechanisms that Aurae does not—

Principle of Least Awareness: A single Aurae instance has no awareness of higher order scheduling mechanisms such as the Kubernetes control plane.
—but I believe that the design of Aurae's API can still transfer to Dagger, mapping to the core of process and isolate management in Dagger's larger API.

[runic heron]

It's pretty what a nice terminal font can do for your graphs 🙂 Ops guy has suggested I demo my dagger foo at our tech-meeting this week which is cool, other dev has just looked at the "horrible evil programmable build go code infecting my codebase" - he's gonna be the hard sell - mostly on the programmable build side of things.

(PragmataPro Liga font BTW)

[tribal burrow]

I have a teammate who "doesn't get" the use case. Why do we build this way?Isn't drone fine for us (no)

[green leaf]

The reasons why I avoid drone is 1) for a long time there license was kind of sketchy with what was open source and what was proprietary, and they only distributed the closed source version so it was easy to accidentally do the wrong thing 2) it uses yaml every where which works great until you want to specify a large matrix of versions to build and there are some exceptions to exclude.

Dagger on the other hand has been pretty clear upfront what they proprietary part is, and uses traditional programming languages where looping and conditionals are much easier.

The downside is that in the short term, you’ll have to write more modules yourself, but things like Gale (GitHub actions runner) and dagger verse will make that better, and I expect that it will catch up quickly

I’m also not sure if Drone had something like service containers or anything like ‘dagger shell’. Which are pretty cool

[sharp marsh]

I recall watching some of Kris's (RIP) twitch streams about Aurae. I don't think Dagger design has been influenced by it since Aurae's initial scope is/was mostly to target some of the opportunities on in the distributed runtime ecosystem (k8s mostly). Dagger is a tool with a different approach where most of the design is inspired by local development tools and workflows like Docker, compose and some dev first oriented tooling

[sharp marsh]

@runic heron you also have the not so popular _EXPERIMENTAL_DAGGER_INTERACTIVE_TUI as well which I personally like it more than the default one

[tribal burrow]

so, my container moved up to 0.9.1 but I didn't upgrade my SDK. is it perchance pulling "latest"?

I mean thanks for the warning message, but I wasn't planning on upgrading in the middle of a product crunch 😦

[winter linden]

0.9.1 upgrade

[tribal burrow]

btw, one of my co-workers mentioned something that sorta bothered him, and I sorta agree. The go SDK's use of 'With' in function names starts to mess with your head after a while, due to semantic satiation.

[runic heron]

I do? I've not turned that on specifically, and er, have no idea where that's set? I have no env var for that...

[sharp marsh]

I'm saying that if you set that, you'll see a different TUI option 😉

[runic heron]

Oh right 🙂 Ooo thats pretty 🙂

A lot of screen flicker there tho (update: that appears to be Ghostty redrawing - not dagger)

[runic heron]

Interesting - Mitchel was asking if that TUI was using Synchronised Output ( https://gist.github.com/christianparpart/d8a62cc1ab659194337d73e399004036 ) and it looks like it's not, according to the terminal inspector in Ghostty, @sharp marsh any idea if that TUI is using SO, or what it would take to add that? (knowing full well this is an experimental TUI). This video is comparing Ghostty (top) with WezTerm (bottom).

[main spear]

Crosspost of our open source (internal) deployment strategy tool now supporting golang actions (which are supposed to use dagger themselves) without having golang installed locally (magic)

https://github.com/lunarway/shuttle/releases/tag/v0.20.0

The idea behind this release, is that if golang isn't available locally, we build our golang binaries in a dagger pipeline and export them to the user (in a future pr we will pull them from github releases themselves, but we haven't gotten that far yet.) This was the last major bottleneck for going GA inside of Lunar with dagger 😄

[runic heron]

that looks like..... spam.

[sharp marsh]

no idea @runic heron. We're leveraging bubble tea's TUI library (https://github.com/charmbracelet/bubbletea) here which does almost all the heavy lifting. Tried searching for "synchronized output" in their repo but nothing came up

[slender star]

Thanks to @runic heron and @halcyon harness for flagging the spam that I deleted 🙏

[heavy karma]

@stuck wyvern I'm not sure if this was already answered, but perhaps this recipe will help? https://docs.dagger.io/cookbook#add-oci-annotations-to-image

[runic heron]

Cool - might go hunting down there and maybe raise some issues

[runic heron]

I just raised https://github.com/charmbracelet/bubbletea/issues/850 asking for/about support for Synchronised Output, so hopefully that may kick off some improvements.

[patent slate]

Hi. I just ran into Dagger and am a little confused. What parts of it are OSS and what aren't and can I run Dagger without Dagger Cloud? We are just starting with our ideas and in POC/ investigation mode for tech and I'm not reading in the docs about where OSS starts and where it ends or rather what we can use self-hosted (in k8s clusters) and what not. Can someone clarify this for me please?

[topaz blade]

Can answer one for sure

can I run Dagger without Dagger Cloud?
Yes, and its awesome!

What parts of it are OSS
As from the talks I listened, the core product i.e CI/CD as code is OSS, while layer caching mechanism across multiple regions and nodes is proprietary.
Which I guess is only available in the Dagger Cloud

|| correct me if I am wrong team ||

[winter linden]

Hi Scott! The Dagger Engine (decentralized, runs on your machines) is open-source, and Dagger Cloud (centralized, provides add-on services) is not.

They are complementary: you can run Engine without Cloud. And when you use Cloud, it augments the same open-source engines running on your same machines. Specifically Dagger Cloud adds distributed caching, pipeline visualization and operational insights.

Here’s a blog post explaining how Airbyte combines the two in production: https://dagger.io/blog/airbyte-use-case

[patent slate]

@winter linden @topaz blade - Thanks for your replies. I'll keep investigating and reading and I'm very interested in the Argo-Workflows guide. Thanks again.

[winter linden]

Sorry I missed your answer @topaz blade , that’s all correct, thank you! 👍

[warm temple]

Awesome! Let me know if you have any feedback or further questions on the guide 🙂

[runic heron]

Support for Synchronized Output · Issue ...

[chilly arch]

@patent slate since you mentioned Argo, you might like this demo forum post with @warm temple live demo. There is a link to the guide there too - https://discord.com/channels/707636530424053791/1155142448985669642

Looking forward to seeing all of you at the Dagger Community call tomorrow! We have some great updates and demos lined up from @rain oriole , @warm temple and @winter linden 🚀

Just a reminder that we've moved from Zoom to Discord for these events.

You can mark yourself as interested here: https://discord.gg/sdtkm6t98a?event=1164767240953024542

[sly copper]

Is the dotnet sdk maintained at all? Considering trying it but looks like it might not be up to date.

[runic heron]

5am - lets see if I wake up 🙂 I actually joined LAST friday then went back to bed and grumbled ;p

[wild bluff]

Hey all, trying to figure out how to get Engine to work with GHA caching. I'm passing _EXPERIMENTAL_DAGGER_CACHE_CONFIG=type=gha,mode=max as an env to the Engine container (as well as setting the url and token envs) that we start, but I'm not seeing a Cache created in the GitHub Actions UI. Do I need to pass the url and token as parameters in the cache config?

[winter linden]

That should work (ish). However we now recommend against using this cache export method, in part because of too much support whack-a-mole, and generally poor performance and scalability

Instead wee recommend either 1) using Dagger Cloud for distributed caching, or 2) running your CI on a machine with persistent local storage, to avoid the need for distributed caching altogether

[wild bluff]

Neither of those options are viable for me and my team.

Is the registry caching option more reliable?

[winter linden]

Not really, the issue is with buildkit caching export as a whole.

It does work, and we can help you get it to work, if really no other options are viable. But you should consider it a stopgap.

[chilly arch]

Any Daggernauts in Tokyo?

We have a team member who will be there from Nov 10th - Nov 19th. If you are interested in meeting to discuss your Dagger use case or would like to hack on something together, please DM me and I'll be happy to coordinate. 🙂

[woeful lynx]

I have an architecture / design question; trying to find out if Dagger would be a good match. I am building a video transcription pipeline where the core work is done by a openai/whisper container. The videos and transcriptions are stored in S3.

I imagine the pipeline looking like this:

1. Discover a video without transcription in S3 and download it to a local file
2. Transcribe the video
   • Video file is mounted to the container
   • Transcript is exported from the container to a local file
3. Upload transcript to S3 (reading the local file)

In Concourse I would have used the S3 resource to discover and get the video in step 1 and put the translation back in step 3. Step 2 would have been a task based on the openai/whisper image.

Step 2 is a pretty obvious candidate for running a whisper container with Dagger. But I am unsure whether I should write steps 1 and 3 as native Go code (e.g. using the minio client SDK), or use a container (e.g. minio-client) to perform the discovery, download and upload to S3?

What are people's thoughts and opinions on when to encapsulate a step of the pipeline in a container, and when to write it in the language that's being used to drive Dagger anyway?

[narrow nymph]

Personally, I'd say you probably want to use minio-client in a container - the code you run in the SDK isn't cached, so you'd need to implement custom logic to avoid uploading something twice (though maybe that's ok for you?)

The work we're doing with modules sounds like a really good candidate for this, depending how on the bleeding edge you're willing to live. The idea is to allow distributing "steps" in your pipeline like this as modules (on https://daggerverse.dev/), so they can be re-used.

You could write a module (see https://docs.dagger.io/labs/project-zenith/) that has S3 upload/download functions, and then your pipeline can just use those modules. Under the hood, your modules run in a separate container, but that's kind of an invisible implementation detail. Additionally, the idea is that we will be able to cache these inputs/outputs automagically (but tbf, that's not quite implemented yet).

[sharp marsh]

👋 as Justin mentioned it, using a CLI to perform the download/upload will be better in terms of caching and de-duplication. Just wondering if there's any reason to use minio instead of the official aws s3 client?

[winter linden]

Virtual meetup is starting! https://discord.com/channels/707636530424053791/1154104516225015880

[woeful lynx]

No hard dependency on minio, I was just using it as an example for a generic S3 client and library.

[loud briar]

Are the plans for the Daggerverse available anywhere? Bunch of things it could do with over time: READMEs displayed (so content like examples, how-to-use etc is front and center), filtering options (tag-based, not text-based 🙏 ), community ratings (stars? Helps separate useful modules from... less useful modules!). I realise it's super early days, just a few things that popped into my head when looking at it 🙂

[winter linden]

No formal public roadmap, yet, but I encourage you to open issues for the things you care most about! We've been holding off on opening too many issues for something that is not released yet (don't want to confuse people), but maybe it's time!

[winter linden]

READMEs displayed

I believe that is already shipped! Are you not seeing your README rendered on your module's page?

filtering

Agreed!

community ratings

We'll get there 🙂

We appreciate all your ideas and suggestions!

[loud briar]

I was just clicking modules that have little/no README! Clicked a few more and some have more content 👍

[tender goblet]

Quick question: I'm wrapping a make/gcc based build of an existing codebase in a Dagger pipeline. The build output created by the build target, e.g. make bin-x86_64-efi/ipxe.efi results in the file with the same name, but folder bin-x86_64-efi also contains all intermediate build artefacts like object files. I have the names of the 4 build targets in a string array, and append these to the base make command. How can I export only these 4 final build results back to my host FS, without the intermediate build results?

[sharp marsh]

You can call Export on each file independently.

ctr.Directory(outDir).File(file).Export

So you can just iterate the array and call .File.Export on each filename

[tender goblet]

So obvious...

[sullen ridge]

I've got a bit of an unusual use-case that I think Dagger and the folks working on it seem uniquely positioned to address.

The outcome that I'm looking for is to have a docker-in-docker container that can run sub-containers based on a single rootfs (mounted read-only) that I have at the time of building the dind container image. Only one image will ever be used in the dind docker.

In pure runc this would probably be trivial but the docker abstraction and its graphdrivers / snapshotter plugins really feel like they're getting in the way.

I'm aware of approaches where I can docker load a tarball on daemon bootstrap but that introduces latency and risk at start-up.

So my latest thinking is to figure out if I can leverage Dagger to build a dind image w/ VFS layers and metadata pre-populated. I haven't been able to run docker during a docker build to get the on-disk structure figured out but maybe Dagger can let me do builds in user-namespaces or something.

Any ideas?

[regal helm]

👋 i am looking for host network equivalent in dagger. docker run/build --network host using Go SDK. anyone know how?

[fossil pine]

You can watch @slender star and myself talking about introducing Dagger to an existing project here (at Rejekts): https://www.youtube.com/live/tWWBzsZLrIw?si=nJpqDbPlqVIxA9S3&t=16364

[slender star]

Special thanks to the #daggernauts devs and module authors who made our talk possible! 🚀 Yes, these were a 100% Zenith dev workflow demos. 🙌

[unborn topaz]

@kubecon

[chilly arch]

Hey! I am probably sitting right across the room from you. Let's meet up, so I can give you some Dagger stickers 🙂

" When you think about CI, what emoji comes to mind? " @warm temple doing a Dagger demo at AppDeveloperCon before KubeCon tomorrow 🙂

[chilly arch]

KubeCon hangouts - thread to meetup

[runic heron]

When using client.Git for equiv. in non-go SDKS, how often/when does Dagger check if that repo has changed SHA1's to determine if it's cached or not? Is that every run, or daily, or something else?

or is doing the clone each time somewhere inside dagger/buildkit, then using normal source comparison/cache checking?

[narrow nymph]

Dagger does a check with the remote every run - the clone happens inside buildkit.

[tired moth]

Please help here https://discord.com/channels/707636530424053791/1171442034540810292

[sullen ridge]

So I've gotten a bit of a start on this. I've found that the .asService() abstraction is a really great way to run dind (with insecureRootCapabilities).

I've got another container that mounts the service and seems to have everything needed to get the dind container seeded with the images / files I want.

Now I'm blocked on getting a snapshot of the dind container pushed. Will take it to #1030538312508776540.

[runic heron]

anyone know if its possible to redirect Service container logs to a file? would make grepping/following service logs easier when debugging

[runic heron]

Just running go run main.go | tee debug.log helped a lot - see the output and also funnel to something I can grep, but would be nice to split that to services - mmm.

[sullen ridge]

In my exploration of the weird docker image I'd like to build, I feel like I want to be able to reference the filesystem of a Service. Are there technical reasons why traversing that isn't possible? The reason I ask is that it seems like calling .asService() kind of forks the chain so that all previous steps get re-run upon referencing (such as the network listener that becomes the service).

[runic heron]

Ive not seen services rerun yet in my usage, it forks the chain as it returns a layer snapshop as a lazy started container.

I suspect, if you have multiple services sharing base layers - it may look like rerunning if the logging is mentioned by both.

Could be nice to have some way of exploring all the buildkit layers/caches but I don't know if thats possible?

[sullen ridge]

I'm probably misinterpreting or doing it wrong. What I was seeing was the build getting stalled indefinitely and logs made me think the service's daemon was being re-run. Any ideas how to dig deeper?

[sullen ridge]

I don't think I'm understanding the client.directory() paradigm and how to leverage it. My assumption was that:

1. I can create a directory.
2. I can mount it onto a container (that will be exposed as a service).
3. I interact with the service, thereby mutating the mounted directory as a side-effect.
4. I can then extract the side-effects reflected in that directory.

But in practice, I'm struggling to get that to behave as expected. Is my understanding of the model off?

Here's a pretty minimal repro of the mounted dir confusion. I think the container vs service forking is a separate thing though. ```js
import { NetworkProtocol, connect } from "@dagger.io/dagger";

connect(
async (client) => {
const dir = client.directory();
const container = await client
.container()
.from("alpine:3.18")
// Mount our ephemeral directory to /tmp/test. The expectation is
// that this dir will be stateful for the life of this daggern run.
.withMountedDirectory("/tmp/test", dir)
.withExec(["apk", "add", "--no-cache", "curl"])
.withExec(["curl", "-o", "/tmp/test/out.log", "https://google.com"])
// Show that the container sees that it has written to /tmp/test
.withExec(["cat", "/tmp/test/out.log"])
.sync();

console.log(await dir.entries()); // []
console.log(await container.directory("/tmp/test").entries()); // ['out.log']

},
{ LogOutput: process.stderr }
);

[chilly arch]

Congrats to @frail mulch on his "Platform Engineering on Kubernetes" book release and thanks for the mention of Dagger!

[runic heron]

Woot - @frail mulch got any preferred purchase links? Amazon or elsewhere?

[fossil pine]

Wow, someone in Hungary wrote this about Dagger: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwitwOGDybaCAxW70AIHHRo2DEI4ChAWegQICBAB&url=https%3A%2F%2Fojs.uni-miskolc.hu%2Findex.php%2Fpsaie%2Farticle%2Fdownload%2F1999%2F1535&usg=AOvVaw12DVPhgkX0RxpYjcGh79XZ&opi=89978449

[prisma jolt]

Hi there! Is there a page or other resource that talks about Dagger Cloud security?

[frail mulch]

Yeah! The link to the manning page is here https://salaboy.com/book/

[winter linden]

I don’t think there’s a dedicated section yet, cc @fallen temple @heavy karma

[prisma jolt]

Thanks for the response! I've been really enjoying Dagger in my trials, but I'd like to try Cloud before recommending it to teammates. It would be super helpful to have somewhere I can point to as I'm sure they'll have questions about it.

[sand gorge]

Fellow speakers from TestCon are now sharing the Dagger knowledge that I was sharing.

It's working 🙂

[rain oriole]

Bind mount

[fallen temple]

We don't have a dedicated security page aside from our terms of service and privacy policy today. May I reach out to ask you a few questions about what you'd like to see? There are a few approaches I've taken in the past to creating these types of resources and understanding your perspective would help a lot.

[prisma jolt]

Absolutely. I’d be happy to help as much as I can.

[lime cipher]

Hey! I was wondering how I could use dagger to get an artifact from a run pipeline in code?

[sharp marsh]

hey! welcome! do you mind adding some clarification to the question? Not sure what you mean by getting an artifact from a run pipeline

[lime cipher]

If you have a log file within the run of a pipeline, and you want to get that file from a pipeline. How would I go about accessing that? Would I need to use S3?

(Sorry for the bad example. Ideally, I’d want to interface with targeted files/outputs that in the same runtime I launched the pipeline)

[sharp marsh]

oh, sure np! here's an example on how you can export directories and/or files from a Dagger pipeline: https://docs.dagger.io/421437/work-with-host-filesystem/#export-a-directory-from-a-container-to-the-host

[lime cipher]

Thanks!

[hot yarrow]

Thanks Dagger community for the gift 🎁🤩🤩🤩

[somber bluff]

I second that, sock game always helps

[winter linden]

Love those socks

[heady atlas]

hey folks! trying dagger for the first time this weekend (great stuff, kudos to the team! also, hi @elfin frigate & @wraith niche!), and got a few questions

[slender charm]

Wow, someone in Hungary wrote this about

[main spear]

I will be attending KCD Denmark tomorrow, so if anyone is attending feel free to reach out if you'd like a converstation on developer experience and daggers role in that at Lunar

Cheers

[main spear]

Just finished up. Kcd denmark
There was a lot of excitement around dagger. I had a lot of people come up and ask me about it etc.

Some feedback I got:

• how and when is dagger useful. People struggle with when to adopt dagger and what concrete benefits it gives dem to deliver a better developer experience.
  • my answer is pretty much that dagger allows software engineering principles comparable to either library or microservices development. As well as collecting concrete stats on their ci and developer tools.
  • and adopting it if the company has a dedicated platform team. If you do it earlier you may not need the engineering for the scale that dagger so well help maintain
• whom should use dagger. Is it developers, platform engineers etc.
  • my answer: platform engineers unless the developers are very close to platform engineering it should be delivered as either a product or as components ready to br used without much configuration.

I saw a lot of excitement and a lot of companies evaluating dagger, and in the upstart phase of using it.

[winter linden]

Thank you for the summary! And for advocating for Dagger 🙏

We were just at Kubecon and had a similar experience. Lots of interest, and lots of questions 😁

[main spear]

Dagger also seem applicable to nearly everyone. From banks, to Lego, to trading, startups etc. Enterprise certainly seems interested in solving their ci needs as they're currently in the process of moving to the cloud and containerizing their apps

[swift barn]

My Dagger wishlist for 2024:

1. Interactive debugger (https://github.com/dagger/dagger/issues/4463) Even if you know what you do, sometimes things magically fail 🦄
2. Java SDK (with full module support... not for me... but my co-workers would love it)
3. More documentation and clearer examples (e.g. best practice on how to write dagger modules, clearer instructions how to chain functions with dagger call, cross-language module examples, how to write tests for dagger modules)
4. More stable setup (Non Go SDK users sometimes had a rough start. Things were failing early, examples did not work properly for every user due to different reasons and bugs)
5. Attestation / SLSA / Compliance with Dagger

[chilly arch]

Hi everyone! I have an extra spot on the Dagger Community call this week.

If you have a Dagger project or module that you'd like to demo, please DM me 🙂 I'd be happy to add you to the schedule.

Looking forward to seeing you there! https://discord.gg/PGHVUMjbkk?event=1164768399210065970

[chilly arch]

See you all tomorrow at 9 am PT! We'll dive into some of the latest modules that were built and will have time for any questions that you might have about building your own Dagger module. https://discord.gg/E639DfFg3n?event=1164768399210065970

[winter linden]

Bummed that I’m missing this one 😭 The FOMO is strong.

[analog raven]

Solomon is on a live podcast today with Bret Fisher. Here's the link if anyone wants to watch! It starts at 10:00am PST

https://www.youtube.com/watch?v=RgE7Y2Lvxro

[winter linden]

In case you missed it, the answer was: yes, we need Dagger. 😛

[slender charm]

The pop up appearance was epic 😂

[grand topaz]

Thanks for being on the show @winter linden ! We had a fun time and plenty of topics to cover in the future ! @rustic brook

[rustic brook]

It. Was. Great.

[timber crater]

I dropped into an impromptu troubleshooting session with @sharp marsh and @carmine island . It was a lot of fun and reminded me that sometimes there are others in this group who can fill in our gaps in knowledge.
#kubernetes message

Since I am preparing for my CKA exam, I want to encourage any of you (except @heavy gazelle ) to feel free to contact me for help troubleshooting kubernetes. You do me a service by providing me with practice.
Seriously. IUf you don't see me online, you can slide into my Matrix DM at blaise1:pio-p.io

---

• If Gerhard has a problem, it's likely to be way over my head, but I would be happy to watch.

[radiant talon]

Just swinging by to say, wrote some code using services for the first time this week and it feels like magic. Spinning up pytest, grafana and a lambda image with the runtime emulator in <200 lines of Python and a well-honed Dockerfile. Much smoother than chaining make / compose / hatch together 🙂

[chilly arch]

I dropped into an impromptu

[soft horizon]

Generic question that probably doesn't have a single answer, but if I'm just starting to look at Dagger (no implementation for several months at the earliest) does it make more sense to be looking at the new zenith stuff?

[warm temple]

Hi Spencer 👋 Yes I'd say Zenith is a great place to start if you're just getting started 🙂 We have some guides here: https://docs.dagger.io/zenith/developer and more discussion over in #daggernauts

[soft horizon]

That's what I expected - I'll do some poking before my week vacation 😄 last time I looked at dagger I feel like it was CUE only. Thanks @warm temple

[warm temple]

It's been a while then 😅 Feel free to ping me or drop in #1030538312508776540 if you have any questions along the way!

[patent slate]

Does the word Dagger come from DAG as in Directed Acyclic Graph?

[winter linden]

Yes 😁

[patent slate]

If I already have Argo Workflows implemented, where would Dagger fit in? Seems like the one would replace the other? (correct me, if I am wrong).

[winter linden]

They are complementary. This might have some answers for you. https://m.youtube.com/watch?v=ysI86z0izHE

[patent slate]

Thanks for that. I think I'm getting it. 🙂

[woeful lynx]

What is the common thinking with regard to triggering a pipeline?

Following my earlier example⁰ about a video transcription pipeline, I'd like to start the pipeline whenever a new video appears in the S3 bucket.

[0] #general message

[acoustic echo]

Im going through the docs: https://docs.dagger.io/sdk/go/959738/get-started/#step-1-add-the-dagger-go-sdk-to-the-project

im also new to golang so maybe im missing something but it looks like I need a go.work file for this to work?

my question is adding a go.work file with use ./build_pipeline fixes this but apparently you aren't supposed to check in work files? so will everyone that uses my repo need to create their own?

[sharp marsh]

my question is adding a go.work file

[runic heron]

Odd question I've often wondered - when a build set gets run, and isn't cached - is there any way of logging why it wasn't cached? could be useful to see "this file.txt was touched" when it maybe shouldn't be...

[winter linden]

We just started adding exactly that feature to Dagger Cloud 🙂 Still rough, and doesn't have file granularity yet.. But we'll keep iterating until you get a reliable, detailed answer to the question "why isn't this cached?" for every run

(If you're already using Dagger Cloud: you can get this view by clicking on an individual operation in the run view)

[runic heron]

Not yet - hitting resistance to using Dagger still, and only have it on one project. So haven't even raised the idea of Dagger Cloud yet.

General resistance is more over introducing a new, different-than-the-rest-of-the-org pipeline model, that still needs Azdo yaml, combined with a dislike of "programmable builds" where no one else wants to write any go code (all valid concerns, and we do have larger projects on currently as well)

[winter linden]

Totally understandable for non-Go teams to be reluctant to writing Go pipelines. Cross-language modules should help with that no?

[runic heron]

Should do - hopefully the Java SDK will be generally usable soon - AFAIK I don't think that's deployed to maven central yet (I could be wrong there, need to recheck that - but I'm also not sure I like the setup for that, tho possible can work around that with jbang or yet more tooling. zenith may help there tho. Hopefully we'll be able to use private modules soon

[runic heron]

separate topic - is there something special one should be doing when exporting container paths back to the host OS - I have:

build := ....dagger stuff that compiles, launches services and run tests
testPaths := []string{"main", "reporting", "authorisation"}
for _, path := range testPaths {
    sourcePath := "/src/target/integration-" + path + "/target"
    targetPath := "target/integration-" + path
    os.MkdirAll(targetPath, 0755)
    build.Directory(sourcePath).Export(ctx, targetPath)
}
``` which, seems to work - but doesn't always contain all the content I'm assuming should be there - I see theres a `Directory.Sync` function, should I be calling that before the `Export` to ensure filesystems are flushed or something?  Maybe I should convert that to a module and exploit `dagger shell`.

[rain oriole]

No need to Sync before Export. dagger shell is very helpful for debugging, yes. As a tip on performance, since you're exporting to the same parent dir, I'd use the for loop to lazily add build.Directory(sourcePath) to an (initially) empty Directory and then have only one Export(ctx, "target") in the end. Also easier to debug just one directory with recursive dir.glob() when not using a module.

[runic heron]

Cheers - will give that a shot. The next part of that code created a zip file manually in go - to attach to the adzo job. What would be awesome if there was ExportZip like native dagger call but couldn't see anything like that.

[sharp marsh]

well. you could create the zip file withing the pipeline and just export that file? I know buildkit supports exporting tar files out of directories, but we're not exposing that API

[warm temple]

I need that module 🙂 I'll write it up

[arctic grove]

seems a missing quote was preventing Netlify docs to build successfully, added a quick PR here: https://github.com/dagger/dagger/pull/6140

[median ivy]

Hey Dagger Team!

It was great meeting everyone at KubeCon and chatting to you about Dagger.

I've been spending the day today getting my old hacked together multi-container-buildkit env upgraded to the 0.9.3 dagger engine deployed to k8s.

Long ~story~ day short, I've got everything working, Talos Backed Civo Cluster, Gitlab Runner pods talking to the dagger socket locally, Civo Object Store for caching ... and it's working a treat!

now ... just to rewrite all of my pipelines to take advantage of everything I've learnt at KubeCon!

Thanks again for building this

[winter linden]

Post-Kubecon love

[fierce condor]

Hey ho Dagger-Team!

I've intoduced Dagger to my team last tuesday and it went quite well. Especially the caching (combined with the local execution) was enjoyed.
We worked here for the last 6 years with Jenkins and the shared library "pipeline" feature of Jenkins. This however lead to one big "God"-Pipeline with one big "God"-Config. Now we want to start on a next-gen Pipeline. For me personally, this is dagger. However, we need to figure out what our "customers" in this case our own developers really need/want. To achieve this, we are preparing a questionaire, and I wonder, if you have some tips/questions which you might sshare with us.

E.g. we are not 100% sure if everyone in the company sees the need/usefulness of an pipeline. And/Or if they want to create/adapt a pipeline by themself.

Perhaps you had something similar in the past, and might be able to share your experience 🙂 - of course alsways nudging into the dagger direction 😉

[main spear]

Hey ho Dagger-Team!

[winter linden]

Job search

[chilly arch]

Hope everyone is having a great week! Looking forward to seeing y'all at the community call tomorrow. Shout out to our presenters @elfin frigate @swift inlet @solar raptor @heavy gazelle 🎉

https://discord.com/events/707636530424053791/1164768773526528051

[winter linden]

See you in the amphitheater 😁

[azure hatch]

wondering what the current pattern is for sharing a database service between containers; i have 3 services, a db service that starts, a migration service that runs migrations against it, and a app service that starts to run tests. it seems that when the app service starts, the db service has not yet run it's migrations. wondering if the db service is detatching after the migration service runs

(and is starting a new instance when the app service starts)

[sharp marsh]

Mind sharing an example how you're connecting things together in #1030538312508776540 please? cc @elfin frigate

[azure hatch]

sure

[blissful bone]

Hi there 🖐️
Hi @winter linden

So many times (i found my last message on that discord) :
#999688146616852540 message
😄

I followed dagger long time ago (early 2022) but my teams not ready for that last year
So many changes since, and got a new challenge and dagger is on the possible solutions

One of my team want to shift left integration test but were unable to get local docker instance
So have to find a solution for them to launch these tests in remote envs and i study few solutions like odo (remote containers on k8s) or eclipse che (totally remote env with web IDE), and dagger is in short list.

I made some tests last few days and don't know if they are ready for that, but there is a chance !

[sharp marsh]

Seems like the perfect use-case for Dagger. Let us know how we can help!

[blissful bone]

Woh, just discovered the (new ?) dagger modules feature (aka zenith ?)
We already got a lot a functions for our CI/CD jobs, packaged in many oci images on our registry
I was thinking of using dagger to use these already existing functions, but know i want to build dagger modules for that 😄

[winter linden]

We would love to help 😁

[runic heron]

That sounds very similar to my currnet dagger IT test usecase - I kick off a PG container, an LDAP server container, run DB migrations, then start a third service container with the actual application depending on the previous ones - then run my tests.

[compact flame]

Hello 🙂 I'm creating a postgres service, from a postgres container, where I'd like to import a schema in it, naively I did that:

    if schema != nil {
        c = c.WithFile("/schema.sql", schema).
            WithExec([]string{"psql", "-f", "/schema.sql", "secretmanager"})
    }

but the exec is done without waiting for postgres to start, do you have any tips?

[sharp marsh]

Are you binding the Postgres service to that container? Can you share a larger snippet?

[azure hatch]

yeah; that's essentially what i'm doing here

[cunning jolt]

When will the demos from the last community call go live? I am traveling on vacation and missed the call so I would love to catch up ☺️. If there is a full recording of the call that would suffice too.

[chilly arch]

Hi! The recording had some technical difficulties, so we need to re-record the demos from last week. I am working with the speakers this week to re-record, so we will have them up soon!

[timber crater]

In case you missed it, The KubeCon episode of ChangeLog has a number of really cool appearances with @winter linden and @heavy gazelle
https://changelog.fm/568
as a result, I'm excited about Talos #kubernetes

[undone gale]

Hi Everyone
@slender star & @warm temple are presenting on the Wolfi community call now 🙂 Tune in here: https://calendar.google.com/calendar/u/0/r/eventedit/copy/aWpydnZxc2VhaHBlMzY5OWIzN2xvdGJmYXZfMjAyMzEyMDZUMTcwMDAwWiBjXzdlYzYwZjQ4NTkzMWY5MDU2MDQwYTNlMjQyNzM0MDBkZTQxYTE0M2VjNjA3MDNiNDExZDc3YjFmNTM0ZWMxNWZAZw

[slender star]

Thanks @undone gale ! That was a super fun call! Looking forward to the recording.
Would love to know what you use to record your community calls on Google Meet 🙂 OBS?

[timber sphinx]

Current state cliff notes

[slender charm]

This was very cute 🙂 https://x.com/flipt_io/status/1732770350872567811?s=20

[undone gale]

We are low production right now, we literally record the Google Meet - but that may change 😄

[ocean anchor]

Hey, I'm curious if Dagger makes any sense in CI workloads that require that state be kept between CI invocations?
My case involves Bazel in a monorepo. If you are not familiar with bazel then it's a polyglot build tool which has an client/server architecture.

The server keeps track of what has changes between invocations and invalidates targets and rebuilds them if needed. The target invalidation in Bazel is very robust and it's not considered and anti-pattern to reuse CI workers between CI runs to reuse the state that the Bazel server has already built up. This is what we do today at my company. We have a pool of warm workers that CI jobs get scheduled to and once the run is finished the worker is returned to the pool.

I've been looking a bit into Dagger to create a CI pipeline that can be run and tested locally since currently our CI pipeline is just a bunch of Github Actions actions/workflows and it can be a PITA to do experiments on the pipelines. It looks to me that Dagger runs all steps in containers and then once the step is done the container is shut down and thus all state is lost. This would be a huge slowdown for us since there is significant cold start cost when building a large monorepo with Bazel. Are there any ways to work around this in Dagger?

[timber crater]

What are the cool kids using these days as a docker runtime on Mac OS X (Intel)?
Asking for me.

[slender star]

What are the cool kids using these days

[deep temple]

Hi folks
https://www.youtube.com/watch?v=L4hvYWknBfw
I just watched this ☝️ cool demo of dagger is there a repo with the sample somewhere ?

[slender charm]

Hi folks

[ocean anchor]

Does Dagger have a way to allow me to use the host in the same way as containers? E.g. I want to run some things directly on the CI host instead of in a container but I want the actions run on the host to be part of the build graph.

[deep temple]

I am about to embark on building small and secure go based images using https://ko.build/ is integration with ko and cosign on the roadmap ? or this is left as an addon someone can build on top of dagger ?

[winter linden]

Ko & Cosign integration

[cunning idol]

Hi! I've got an environment where we host repos on GitHub but we do not have access to GitHub actions, so I'm looking for an alternative that can run CI build jobs on PRs for us. Is Dagger the right tool for that, or is Dagger just a pipeline definition and no engine/trigger system to run the pipelines?

[ocean anchor]

Do I get any control over where caches are stored on the host? E.g. if I have an CI host that has both network storage and fast ephemeral storage and I want the caches to be written to the fast ephemeral storage.

[sharp marsh]

yes, that's depending how you're running the Dagger engine. If you're running it via docker, then you'll have to make sure to map the right docker volume to the Dagger engine when running the engine. i.e: docker run -v $my_fast_drive:/var/lib/dagger ...... $dagger_engine_image

[ocean anchor]

I'm running it via the CLI at the moment which starts it for me, do I have any options then?

[deep temple]

I was researching this the other day and i ended up with about 3 options

1. use github runners locally (its free and you can call dagger from github runner running locally) so you get the best (and cheapest)
2. use git pre commit hooks. So that you can start a build locally automatically after each commit to a local branch.
3. call dagger based builder you have directly when you want to do a full release and dont automate it

I opted for option 3. in my case i only need to check if the build will pass when doing a full release build. its not very often and it saves me from having to commit and push to gh. YMMV

[winter linden]

There is also an option 4: roll your own webhook server to process Github Events directly, and call Dagger Functions as triggers. I think this will become an increasingly common use case, and off-the-shelf tools will start appearing that make this very easy (yes, this is a call for project 🙂 but for now you would have to roll your own, not a huge engineering project, but not trivial either. Probably a few hundred lines of code, fully loaded.

[winter linden]

This looks like a good starting point for option 4, if someone is looking for a week-end project: https://martinheinz.dev/blog/65

[slender star]

If you are able to use the GitHub APIs, you could write a GitHub App that receives the webhook events uses the Checks API for feedback.
I wouldn't be surprised if there is a GitHub App project out there that does what you want. It turns out that GitHub Actions itself is just a GitHub App (built-in)
(I swear @winter linden added that 👆 when I wasn't looking 😂 )

[winter linden]

I think that's what I call option 4 🙂

[sharp marsh]

yes, you can still start the engine yourself and tell the CLI to use that engine via the _EXPERIMENTAL_DAGGER_RUNNER_HOST env variable. More info about that here: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#connection-interface

[winter linden]

OK now I really want this...

https://tenor.com/view/dear-santa-saturday-night-live-message-for-santa-to-santa-letter-for-santa-gif-19567806

[slender star]

https://partner.github.com/assets/pdfs/GitHub.Platform.Integration.101.pdf

Gives some nice GitHub App internals 👆

[sharp marsh]

there are some projects and libs that already provide a super easy way to integrate with this:
i.e:
https://github.com/go-playground/webhooks
https://github.com/octokit/webhooks.js

[winter linden]

@cunning idol to summarize the various answers: if you simplify CI down to 2 parts: triggers, and functions to run on those triggers; then Dagger only focuses on defining and running the functions, and you have to "bring your own triggers". So Dagger on its own won't replace a complete CI platform. But, it gives you half of it, and opens your options for choosing the other half. Basically, it unbundles CI.

The two most common triggers for running Dagger functions today are 1) shell commands in an interactive terminal, and 2) CI runners (Github Actions or alternative).

But those are only the most common - you can get creative with your triggers, and over time the community will support more and more exotic triggers for running Dagger functions. Which approach works for you, depends on the specifics of your project.

[sharp marsh]

@winter linden next community call demo? https://github.com/adnanh/webhook#configuration. Dagger module to process webhooks and does Dagger-in-dagger? 😄

[cunning idol]

Much appreciated! Thanks for all the answers!

[chilly arch]

@wary cave will be doing a Dagger workshop tomorrow and a presentation on Saturday at PyCon Thailand. If you are in the area, you should check it out! https://th.pycon.org/schedule

Thanks @wary cave for helping spread the word about Dagger!

[chilly arch]

Our last community call for 2023 happens tomorrow at 9 am PT. I have moved the location from our Discord stage to dev-audio. It will give everyone a chance to see each other and unmute to ask questions if you have them!
https://discord.com/channels/707636530424053791/911305510882513037

Since we will all be able to see each other, feel free to wear something festive to make the call extra fun 🙂 We have two demos lined up from @solar raptor and @wispy tapir. https://discord.com/events/707636530424053791/1164769039130837062

[ocean anchor]

Are there any good docs on the cache volume modes? I've got both caches that can only have one writer and then some that can have multiple concurrent writers and I would guess that SHARED is for concurrent writes and PRIVATE is for single writes?

[patent slate]

Sounds like a fairly easy job for something like Argo Workflows. 😉

[modern rain]

Saw this article this morning for garden (which was new to me). But as I was reading through I kept saying "dagger?"after each paragraph. And by the end I'm like "Dagger! Dagger could do that!"
https://thenewstack.io/imagine-a-smarter-ci-pipeline/

[slender charm]

Imagine a Smarter CI Pipeline

[chilly arch]

hey everyone! We are in Dev-audio now and about to start the community call. See you there!

[winter linden]

---> https://discord.com/channels/707636530424053791/911305510882513037

https://tenor.com/view/simpsons-homer-bart-lisa-join-us-gif-13066208

[chilly arch]

Looks like Discord is limiting people from joining the dev audio since there are already 25 people 😦 It is supposed to let up to 100 people join, so I will look into this.

[elfin frigate]

pictured: the chosen 25

[winter linden]

Oh no!

[chilly arch]

Since Dev audio has a 25 person limit (that was supposed to be 100), I will move the next community calls back to Discord stages. sorry all! Was hoping to use the audio channel to give people the ability to engage easier.

[chilly arch]

**Any Daggernauts in Singapore? **

We have a team member who will be there from Dec 24th - Jan 5th. If you are interested in meeting to discuss your Dagger use case or would like to hack on something together, please DM me and I'll be happy to coordinate!

He was able to meet up with a few people in Tokyo last time, so hoping we can make a few Daggernaut connections again 🙂

[slender charm]

Woh, just discovered the (new ?) dagger

[chilly arch]

We had a great time meeting many of you at KubeCon NA 2023. If you are going to KubeCon EU 2024, send me a DM! We'd love to meet up with you.

If you missed our talks or want a recap, check out this blog post https://dagger.io/blog/kubecon-na-2023-recap

[timber crater]

IDK if this is the same in discord stages, but I found a trick to suppress the echo we were getting during screen share. It's possible to "mute" the stream channel, which might help if the presenter is also an audio participant:

[winter linden]

I think that was it. Thanks Blaise!

[slender star]

**Any Daggernauts in Singapore? **

[chilly arch]

Shout out to @tired moth for helping promote Dagger at the GDG DevFest Ahmedabad 2023 over the weekend. He shared over 500 cards to help promote the Dagger Ahmedabad meetup and help spread the word about Dagger. This community is awesome 🚀

[tired moth]

Thank you for sharing and appreciation.

[chilly arch]

If any of you are in Singapore, @heavy karma and another Daggernaut will be meeting at PS. Cafe at Great World City Mall at 9.30 am on December 24th. https://g.co/kgs/dCttjJ

Anyone is welcome to join! Can't wait to see pics 🙂

[deep rampart]

Sorry for the "necro", Did you find anything?
Thanks

[winter linden]

Dagger Cloud integrates with the Github Checks API 😁 https://dagger.io/cloud

[timber crater]

A shoutout to @chilly arch for curating the demos channel!
I'm prepping a demo for next month and it's so helpful to review what has already been covered.

[timber crater]

Speaking of demos, Miranda suggested that I could talk about the efforts to build OKD (OpenShift K8s Distro) with Dagger.
but I have two three other ideas that might be more broadly appealing:

• Query SQLite files using GraphQL (https://github.com/simonw/datasette-graphql) ( @winter linden mentioned this once)
• module for building APKs using Melange (https://github.com/chainguard-dev/melange) (Wolfi crowd would like this)
• Show benchmarks of some massively parallel jobs. For example, my homelab has 40 threads so we could run a bunch of parallel tests and watch btop light up.

[timber crater]

I think everyone will enjoy reading this doc https://beartype.readthedocs.io/en/latest/install/#install
because of their witty content, but I think that it will really interest the pythonistas ( @rain oriole , @sharp marsh , etc )

[rain oriole]

Yeah I know it well. It’s a dependency of the Python SDK 😄

[runic heron]

Merry Christmas daggernauts 🙂

[candid crag]

hi @chilly arch how can I make a registration for creating a new community group for a new city? I'd like create a community group for Istanbul.

[deep rampart]

Most of my pipelines have to git clone some big repositories, and so far in Jenkins I use a git reference repository (this repo is a bare clone that gets updated every couple of hours)
So the clone is super fast as everything is local.

I was going through go docs and I can’t find something similar in dagger. Do I miss it or it’s not there?

My plan is to mount the ref repo as a read only directory, and git clone using that (I will run multiple pipelines in parallel)

I suppose its not supported (unless dagger uses a different git client)
https://github.com/go-git/go-git/issues/672

[sharp marsh]

Most of my pipelines have to git clone

[wide lava]

I'm relatively new to dagger, have mostly build pipelines with Jenkinsfiles. I'm eager to migrate to dagger but have one real concern. I see that the dagger engine is running as a privileged container. Is there a way to run it as non-privileged or eliminate the security concerns that comes with privileged containers?

[opal mist]

i love this tool already trying to convert some pretty complicated dockerfile that is used for deploying backstage in openshift

with 3 build stages, manages to put testing and linting in between, so that i dont have to do multiple installations

[narrow nymph]

Heya, welcome! 👋

So to run without privileged, dagger would need to be able to run as a rootless container - this currently isn't supported, see https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#can-i-run-the-dagger-engine-as-a-rootless-container. I am personally interested in trying to add support for this someday, but it's a massive project, that we don't really have the time for right now 😢

We do have an issue tracking the docs around this, see https://github.com/dagger/dagger/issues/6308

[narrow nymph]

eyy nice ❤️ happy to hear you're having a good time with it, feel free to shout if you have any questions!

[winter linden]

One quick note @wide lava , the Dagger Engine is packaged as an OCI image for convenience, but it’s basically software to install on your host, like Jenkins itself. That’s just the nature of Dagger’s function: orchestrate containers on your machine to run pipelines efficiently.

[wide lava]

Thanks @narrow nymph / @winter linden !

[wide lava]

I hear you, but the difference is that I can run Jenkins as a non-root, unprivileged containers to isolate my host machine. I can't do the same with the dagger engine. What's stopping malicious code execution from accessing the host machine? (I personally think it's not a big deal but I've have security teams complain when they see privileged, would love to have a good answer for them)

[loud briar]

There was a brief discussion about sysbox (https://github.com/nestybox/sysbox) in #gitlab message but as of a recent version of Dagger it no longer fully works. There's an issue here: https://github.com/dagger/dagger/issues/6255 that you might want to put some info in, the more people who need to consider rootless Dagger the more likely we find a solution

[cunning jolt]

This is something I've been keeping an eye on too. I think running privileged will be a deal breaker for certain companies in highly regulated industries. I know I will most definitely run into that when we go to production with Dagger.

[opal mist]

is there any way i can do a directory layour of my workdir inside my container and print it to stdout?

[cunning jolt]

You mean like list the files in a directory? Here's an example - https://docs.dagger.io/cookbook#list-host-directory-contents. Although that is for host directory, same applies to a directory within the container. Anywhere you have a directory you can call Entries on it - https://pkg.go.dev/dagger.io/dagger#Directory.Entries

[opal mist]

So i manage to convert my dockerfile multistage build to a dagger pipeline, but for some reason my final stage is a lot bigger then when i just use my docker file, is there a chance that i can do a hybrid setup, so i can still use the docker last stage, but those intimidate stages before in dagger.io so i can run my linter and test there?

[sharp marsh]

Yes, totally. Having said that, there's no reason why your final stage should be bigger while using Dagger. Could you share some snippets so we can double check?

[sharp marsh]

Happy new year Dagger community ❤️🎉

[chilly arch]

Istanbul Meetup

[main spear]

What is the recommended way of doing container scanning in dagger.

Normally with snyk, you can use an image tag and scan that. I.e. sha@abc... or my/image:vX.Y.Z with a mounted docker sock.

However, I guess for dagger you would either:

1. Export the tar ball, import it into the vanilla docker context.
2. Publish the image and tag on that.

Is there a better way, if not I will probably just go with 2.

[narrow nymph]

Container scanning in dagger

[plain lichen]

How can I get Dagger to work with SonarQube🤔

[sharp marsh]

👋 welcome! here's a Dagger Sonar module made by @cosmic moss (https://daggerverse.dev/mod/github.com/fluent-ci-templates/sonar-pipeline@377f28f15af20643e5f6aa6cbb5d79d4d41ea4bc). That might be handy

[plain lichen]

Thank you

[mystic citrus]

Hey all,
Trying to figure out how dagger might run in devcontainers?
We plan to use Development containers (https://containers.dev/) to ease and standardize local development.

I wondered can we run Dagger in a container, I guess yes, but what are the limitations and what we should be aware of?

[sharp marsh]

hey there! there's no limitations initially that you should be aware of. Dagger should work in devcontainers pretty much the same way that it works locally. In any case, let us know if you run into a particular situtation or have any questions and we're happy to help!

[acoustic echo]

Is it possible to have my dagger pipeline build and cache (reuse) a local image? for example I need to install tools like gotestsum which take 30 secs plus. I don't want to have to create a separate build/push pipeline for my builder images if I don't need to. most CI engines also support a cache as well. Im wondering if I can have the first run of my dagger pipeline create a local image with all the tools installed to reuse on subsequent runs. Does that make sense and is it a common pattern? Thanks!

maybe dagger is already doing this caching and I just need logic like "if <my-tool> installed" to skip if its already there perhaps?

[sharp marsh]

Hey there! Welcome!

Similarly to a Dockerfile, Dagger already does this natively for you. So, if as part of your pipeline you install gotestsum, in subsequent runs this will be automatically cached unless any steps before that invalidates the cache somehow

[acoustic echo]

I see so then I probably need that logic then because go install seems to take the same amount of time regardless

question about the cache. Im new to both dagger and github actions, is there any config I need to do for github actions to reuse/cache images built by my dagger pipelines?

[sharp marsh]

you shouldn't have to make any logic. what language are you building your pipelines with? go?.

if you run this for example:

package main

import (
    "context"
    "os"
    "time"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()

    // initialize Dagger client
    client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
        panic(err)
    }
    defer client.Close()

    // crdb
    _, err = client.Container().
        From("golang").
        WithExec([]string{"go", "install", "gotest.tools/gotestsum@latest"}).
        WithExec([]string{"echo", time.Now().String()}).
        Sync(ctx)

    if err != nil {
        panic(err)
    }
}

you'll notice that only the first time that you run this it'll actually install gotestsum. For subsequent runs, you should see something like this in your terminal output

11: exec go install gotest.tools/gotestsum@latest CACHED
11: exec go install gotest.tools/gotestsum@latest CACHED

this indicats that the go install was effectively cached

[acoustic echo]

@sharp marsh what does sync do? Didn't see that in the quickstart docs: https://docs.dagger.io/sdk/go/959738/get-started/

[deep rampart]

I use it as Run this now!, not at the end of "building" all the steps
I'm not sure if thats the case tho.

I actually use it on every command, so I have logging at the correct time of executin

[sharp marsh]

It's pretty much this. Allows you to have a different control flow over your pipeline by executing whatever is designed up to that point

[acoustic echo]

Is it common to include a sort of environment setup stage of a dagger pipeline?

for example for my go project a stage that just installs all the tools needed like pkgsite, test reporters, etc

[chilly arch]

Happy New Year everyone!

We have room for one more demo for the upcoming community call this week. If you have any Dagger projects or topics that you'd like to demo, please DM me 🙂

Looking forward to seeing y'all on Thursday!

You can see previous community call demos here: https://www.youtube.com/playlist?list=PLyHqb4A5ee1tEgcr7KsNFzQSPN-R3fPs2

[slender charm]

for example for my go project a stage

[patent slate]

I'm researching another "cloud" tech called Radius (by Microsoft) and I can't help but see similarities in what is offered between them and also Dagger and even Argo Workflows.

Dagger could theoretically be opened to do more than just CI, like running Terraform to build infra. Right?

In the end, these systems are handling process flows, as I see them. but are being either very specific i.e. Dagger with CI, Radius with infra or very open, like Argo Workflows, which can do both (like with Dagger).

Is my take incorrect? Thoughts? 🤔

My question is aimed to help me understand better on why I'm struggling to take a direction/ set of tools for the process of ramp-up and tear-down of resources and to control dev workflows. Nothing fits perfectly for our needs. 🤔

I'm researching another "cloud" tech

[timber sphinx]

how is changie working for you so far?

[mystic citrus]

I recently asked the question whether Dagger works nicely with and in devcontainers: Now that I am digging deeper and deeper into it, I ask myself if Dagger is running all flows is container and can build my code. Would I actually benefit from DevContainers if all could be done from Dagger?

[slender charm]

I recently asked the question whether

[worthy willow]

haven't been around the last few months — what's the latest in dagger land??

[still vector]

is there an idiomatic way of handling parallel execution? like this works ok in node:

async function runParallel(containers: Container[]): Promise<string[]> {
  // maybe do some more prep
  return Promise.all(containers.map((c) => c.stdout()));
}

connect(async (client) => {
  const image = "busybox:latest";
  const parallelContainers = [
    client.container().from(image).withExec(["sleep", "2"]),
    client.container().from(image).withExec(["sleep", "10"]),
    client.container().from(image).withExec(["sleep", "5"]),
  ];

  const parallelResults = await runParallel(parallelContainers);
});

but given the concurrency primitives of the different language sdks, it could be helpful to suggest patterns or provide a helper function with each sdk. I didn't see anything in the guides...

[sharp marsh]

hey there! not really. It's difficult to be opinionated about this since everyone has a different preference on how to handle async code. Given that all Dagger structures are concurrently safe, it's up to each team to decide what the best approach to handle concurrency is. After all, as you're showing up in your example, for trivial use-cases it doesn't require a sheer amount of code by using the language primitives.

[slender charm]

is there an idiomatic way of handling

[chilly arch]

Can't wait to see everyone tomorrow. We have a great lineup of speakers ready for y'all! @timber crater @fallen temple @last pine @warm temple

https://discord.gg/yVyx2wWz5G?event=1164769302323413023

[timber crater]

I got about three minutes of cristal clear audio and then .... silence.... eventually the video feed died too. 😿

[hearty rain]

Hacker News discussion on GitHub actions today

https://news.ycombinator.com/item?id=38938470

petercooper

GitHub Actions as a time-sharing supercomputer

Hacker News · Jan 10, 2024, 10:11 AM

[midnight nexus]

Hey y'all! We use serverless framework to deploy our projects. I'm curious if anyone's been able to build projects using serverless framework with dagger.

[sharp marsh]

hey Welcome! I don't recall anyone using the serverless framework but putting a module together should be straightforward. Happy to provide assistance if you're willing to give it a try. Out of curiosity, what's your preferred language? How are you using serverless framework today in your CI/CD pipelines to deploy your functions?

[midnight nexus]

Thanks for the suggestion. Yeah we use serverless framework to deploy all our infra. I'm doing some experimentation at the moment. I am hoping to see if I can make something that does mock deploys. Looking to avoid permission and validation errors in our CI by running local mock deploys.

[sharp marsh]

that sounds great! if you're looking for ideas and/or need to chat about possible approaches, feel free to create a post in #1030538312508776540 and we can get the conversation started there.

[midnight nexus]

Will do!

[timber crater]

I will be in the Bay Area next week (or two) if anyone wants to connect f2f for coffee (or boba tea)

[slender charm]

I will be in the Bay Area next week (or

[orchid cedar]

Hey all. I've recently inherited a client's Cloudflare Pro account. Am in the process of sorting out a compliance document for my client's client and one of the requirements is having access to request logs. Welp this feature is only in the Enterprise plan. My client doesn't really need all the features of the Enterprise plan but they do need access to request logs.
I guess the best plan of action is to contact a Cloudflare "expert", or ideally a Sales expert and see what could be sorted
out ? Like a custom Plan that doesn't include all the features not needed by my client ? Sorry for the long message 😬

[winter linden]

I think you might be asking on the wrong Discord? 😛

[orchid cedar]

Derp

[quiet lodge]

Not a dagger specific question but I was wondering if are there any techniques to estimate CPU/Memory usage of containerized applications. The end objective is to calculate a reasonable amount of CPU/Memory for AWS ECS tasks.

The best that I can think of is to run the container with different --memory and --cpu params and benchmark the app perf.

[cosmic moss]

I have a small prototype example of Dagger in Gleam here if anyone is interested https://github.com/fluentci-io/daggerverse/blob/feat/gleam-sdk/gleam-sdk/sdk/src/gleam_dagger.gleam

[sharp marsh]

Not a dagger specific question but I was

[winter linden]

I believe there is a known issue that requires you to set one yourself with a poorly documented env variable. I can't find an issue at the moment. does it ring a bell @sharp marsh ?

[empty badger]

#go message

[sand gorge]

Hey Folks! That's me picking up Dagger stuff now, and also will be giving my first talk of 2024, in February at PHPUK conference (around 500 people)

It will be a Dagger talk.

I'm thinking of some talk titles that would be good, for me to showcase show to use Dagger for CI Pipelines

My best idea so far is:

Dueling with Continuous Integration <-- because you duel against something with swords, and a dagger is a small sword 🙂

Please give me inspiration, thank you 😄

[slender charm]

Another dagger pun -> "cutting YAML out your CI"

[rigid pebble]

I'm really bad at names, so I asked chatgpt for some suggestions. This one was kind of fun: "No More Yaml-Yammering: Streamlining CI/CD with Dagger" 😆

[sand gorge]

I'm getting somewhere, thinking about "Cutting Edge" something

[sudden ocean]

daggerverse/gleam-sdk/sdk/src/gleam_dagg...

[chilly arch]

Loving the name ideas! I'll add this to our event calendar on Dagger.io, so we can highlight it. Thanks for sharing!

[hexed depot]

Just an FYI, the Learn More link for Dagger Cloud on the homepage 404s. Looks like y'all changed the path from /cloud to /dagger-cloud
https://dagger.io/cloud

[slender charm]

Thanks so much for reporting this! I will get this fixed

[viral lark]

I'm trying to prevent caching so my e2e test is run...repeatedly.

when I do this dagger run go run ci/main.go my pipeline runs fine but when I run it again, caching causes all steps to be skipped.

i could workaround caching with echo >>package.json and re-running dagger run and that causes all steps in pipeline to be re-run since in my case package.json is copied to container and dagger detects this change properly.

instead, I want just my out, err = runner.WithExec([]string{"npx", "playwright", "test"}).Stderr(ctx) to be run.

[slender charm]

I'm trying to prevent caching so my e2e

[sand gorge]

@slender charm @chilly arch ..etc

So is Dagger actually meaning knifes/blades .. or does Dagger imply something else??

the logo is a rocket taking off, presumably, but just wanted to be sure, before I make implications to it 😄

[narrow nymph]

@mint axle is the designer of it, so maybe she can speak to it 😄

[sand gorge]

😄 cool

[sand gorge]

Dagger: Carving a Path to CI Excellence

Cutting-Edge Continuous Integration with Dagger

Entering The Future of Cutting-Edge Continuous Integration with Dagger

Opinions? 🙂

[loud briar]

I think it's a play on direct acyclic graphs - they're often referred to as "DAGs"

[sand gorge]

ah

I know DAGs

[loud briar]

The rocket being an implication of speed - https://docs.gitlab.com/ee/ci/directed_acyclic_graph/: "A directed acyclic graph can be used in the context of a CI/CD pipeline to build relationships between jobs such that execution is performed in the quickest possible manner, regardless how stages may be set up. "

[sand gorge]

making sense 🙂

[chilly arch]

@loud briar is correct 🙂 Dagger is a play on DAGs, so I think it is better to stay away from the knifes/blades.

Anyone in the Istanbul area!? @candid crag has offered to organize Dagger meetups in Istanbul, so we kicked off a Meetup group. Join the group, so you can get a notification when we add events there! https://www.meetup.com/dagger-istanbul/

[narrow nymph]

oh hey @candid crag 😄 fancy seeing you here 🎉

[candid crag]

hi @narrow nymph its nice to be here either

[runic heron]

anyone having issues pulling registry.dagger.io/engine:v0.9.7 currently? seems dead to me...

[slender charm]

anyone having issues pulling `registry.

[simple condor]

I had a question about IDs. You can query the ID of any object with sync, you can form an object from its ID. I suspect this is rarely used. Except, internally the SDKs use IDs whenever a Dagger object is the type of an operation parameter (e.g. Container.withDirectory). In the case of the Node SDK, at least, but I imagine the others are similar, it will run the Directory query, then use its result to run the Container query (assuming the withDirectory is followed by something like publish that also solves). This is all well and good… Except, isn't it technically a race with the BuildKit garbage collector?

[tardy mauve]

FYI that 0.9.7 of the Node SDK is broken, at least with plain JavaScript:

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'typescript' imported from node_modules/@dagger.io/dagger/dist/introspector/scanner/scan.js
    at packageResolve (node:internal/modules/esm/resolve:844:9)
    at moduleResolve (node:internal/modules/esm/resolve:901:20)
    at defaultResolve (node:internal/modules/esm/resolve:1121:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:396:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:365:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:240:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)

Works fine with 0.9.6. Has this been raised on GitHub?

[cosmic moss]

Hi, I've added a REPL to fluentci cli, the idea is to load functions (build, test, deploy ...) automatically into the Deno REPL, behind it is just dagger run deno repl --eval=import ...

[narrow nymph]

FYI that 0.9.7 of the Node SDK is broken

[chilly arch]

Happy Monday everyone! Our community call is on Thursday this week at 9 am PT. During the community calls, Daggernauts shared how they are using Dagger for their projects, show cool tips/tricks, and demonstrated how to use Dagger for real-life use cases.

If you have a Dagger project that you are working on or have a cool trick to share, we'd love to learn more and see a demo during the call! DM me, so we can add you to the schedule!

[timber crater]

On that note, I would be grateful for any comments on the outline of my demo:
https://github.com/blaisep/pytest-bdd-ng/blob/dagger/docs/refactor.rst
<cough> @heavy gazelle </cough>

[winter linden]

looks good, main comment is that best practice will change with Zenith release

so maybe hint at that at the end of the end?

[wide lava]

Is there an self-hosted solution for dagger cloud? I'm dealing with a highly regulated environment here, unfortunately.
Or just a UI that I can see the steps and results, etc

[winter linden]

Self-hosted Dagger Cloud

[heavy gazelle]

I really like it, especially the experiment approach!

Some things which I would improve:

1. What is the benefit to the audience? I understand why you want to do it via Why are we here?, but why is the audience there?
2. How much do you aim to reduce the cycle time by? Reducing it by 1 minute might not be note-worthy, but a 2x or 3x reduction is something else
3. Quantifying the other indicators would also be worth it
4. The local dependencies are: a) Dagger CLI b) Docker. Everything else (including the runtime) are now self-contained in Dagger as of Zenith. FTR: https://docs.dagger.io/zenith/developer/python/419481/quickstart\
5. In preparation for this, I would encourage browsing https://github.com/dagger/dagger/pull/6117, especially all the Python code that was added. cc @rain oriole

When & where are you giving the demo?

[fossil pine]

Where do you put your Dagger pipeline code for your software? Ideally, your software pipelines include CI/CD, but they may not be exclusively CI/CD. For example: they may include tools for local development.

Therefore:

• ci? That's what I use, but I like it less and less.
• cicd? That's still not accurate.
• build? Such an overloaded name...also, many build systems or just build scripts use it for build results
• dagger? Not very discoverable, even though I wouldn't mind if dagger became the new term for CI/CD.
• pipelines? workflow? hack? Too generic, may actually collide with your own software packages.
• scripts? I hate this one with all my heart. It's like misc or utils.
• dev? again, too generic
• make? This is the first one I like, but it's easy to confuse with Make.
• .build? Meh....
• swf (as in software workflows)? this may trigger fatal PTSD in a lot of people

What's the perfect answer? Anything less than perfect (combined with my OCD) and I get pulled back into this endless circle of trying to find the right name. Please help! 😄

[elfin frigate]

Where do you put your Dagger pipeline

[winter linden]

Anyone currently scaling Dagger in production: please share your thoughts on horizontal vs vertical scaling 🙏 https://github.com/dagger/dagger/issues/6486

[timber crater]

I have been wondering the same thing. I have been meaning to throw some big jobs at a server (128G / 40 cores) I have sitting around to watch it glow.

Let's give a warm welcome to OKD upstream

[timber crater]

Do we have an emoji to represent Dagger?
I want to include it in my error messages.

[winter linden]

[wary peak]

i have a question for people here. How many steps do most of your dagger pipelines have? 👀

[narrow nymph]

as a guess? probably somewhere in the range of 20~40 😄

i was actually wondering the other day - do you start to hit limits? i have a weird idea that at the 120 mark you might start hitting some funky issues (based on my knowledge about internals, not actually backed up by evidence)

[slender charm]

How are you defining step here? Is it some high level function, or each invocation of "With*"

[wary peak]

i'm thinking specifically each layer (I think), so basically each invocation of "With".

[wary peak]

I was just running one with over 4,000. so far I haven't run into limitations.

[narrow nymph]

woah

👀 what kind of steps are they? some steps don't actually add layers, some do. e.g. WithExec does, but WithEnvVariable and friends don't

[slender charm]

@narrow nymph I am curious how you would define it, I suppose I would define it as anything that creates a new node in our graph, and I assume every With* thing does that

[wary peak]

Yeah I'm just looking at the number next to the green check mark when I run dagger run

[narrow nymph]

agh, sorry, i think i'm thinking of the steps that we translate into buildkit commands, which then correspond to layers in the output

[winter linden]

@wary peak based on that experience running pipelines of ludicrous size, can I interest you in sharing your thoughts on scaling Dagger in production? 😁 https://github.com/dagger/dagger/issues/6486

[wary peak]

happy to!

[timber crater]

I do a lot of ensemble programming. I may have someone join me to work on:
https://github.com/blaisep/pytest-bdd-ng/issues/4
pythonistas here are welcome to join me.
https://jitsi.pio-p.io/daggerize-pytest

[timber crater]

Ok, that's all for now.

[chilly arch]

Can't wait to see everyone at the community call tomorrow! @timber crater , @cosmic moss , @rigid pebble and @sharp marsh will be presenting. If you'd like to present at a community call, please DM me and I can add you to the schedule!

See you at 9 am PT!

https://discord.gg/WSshZmhnyP?event=1187631407015657563

[timber crater]

@rigid pebble , pocketci is koh-ho-NEW-doh

[cosmic moss]

@rigid pebble I've also recently been working on a command fluentci agent, which start an agent and listens to webhooks and executes dagger which looks a bit like pocketci

[rigid pebble]

@matipan I've also recently been working

[winter linden]

https://tenor.com/view/so-it-begins-raining-rainy-day-théoden-lord-of-the-rings-gif-16521230

[viral lark]

is there api to query daggerverse? i'm learning modules and thought i'd download a bunch and study them

[slender star]

Hi all! 👋 Some of us are hanging out in ⁠dev-audio for the Project Zenith Community Call. Join us to chat about Dagger functions, moudules, Daggerverse, etc 🙂

[viral lark]

I'm a discord noob. Does discord save chat from yesterday's community call? I'm in search of re-watching matipan's demo that i missed from video.

[slender charm]

It does, if you go to the #1154104516225015880 there should be a button on the top right that shows you the chat history

[viral lark]

Indeed, I see it now, thank you!

[chilly arch]

Just announced - @winter linden will be doing a keynote at KubeCon EU. If you will be there, let us know, so we can find some time to meet up! https://sched.co/1YhKQ

[winter linden]

Great now I have to actually make the slides. I mean, yay I'm so excited! 😁

[fossil pine]

Make it a dagger module. Could be more fun...or wasted time, I don't know 😛

[chilly arch]

We will be getting the recordings into the demo forum soon!

[chilly arch]

Hey folks! Our next opening for the Dagger Community call is on Feb 8th. If you have a demo that you'd like to share with the group, please DM me 🙂

If you need inspiration, you can check out our community call playlist here: https://www.youtube.com/playlist?list=PLyHqb4A5ee1tEgcr7KsNFzQSPN-R3fPs2

[gilded flicker]

Hi 👋🏻 I'm new and I've just tested Dagger (the Getting started), I'd like to know if I'm the only one to have the following problem: when I run Dagger, my PowerShell won't let me scroll and prints characters (I have the same issue with VS Code [which also uses PowerShell I think]).

I also wanted to ask you if you might know of a place where I can see some of the pipelines created with Dagger (examples). 🙂

[fierce sandal]

Hi @gilded flicker I personally found some of these examples helpful:
https://www.techchapter.com/en/blog/use-dagger-io-to-streamline-your-pipelines/
https://dagger.io/blog/dockerfiles-with-dagger
https://blog.matiaspan.dev/posts/exploring-dagger-streamlining-ci-cd-pipelines-with-code/
https://blog.matiaspan.dev/posts/exploring-dagger-building-a-ci-cd-pipeline-for-iac/

[elfin frigate]

the PowerShell issue is a known issue, sorry about that: https://github.com/dagger/dagger/issues/6417

[stoic knot]

Are there any links for Node SDK to build an AMD image from an ARM system (mac m2)?

of note, I don't reach the publish stage, just trying to build the image right now

[slender star]

of note, I don't reach the publish stage

[hexed depot]

I'm trying to sell my engineering team on Dagger at work and an interesting concern came up that I wanted to run past the core team with the vision of Dagger. They said

My world view on CI/CD basically boils down to, all I want from the CI/CD system is the ability to watch triggers, start jobs in a container, and collect and post logs. Everything else should be done in automation logic. This is what get's the portability I think you're looking for, because that automation logic will need to take care of everything else, caching, test suppression if there aren't changes, multi-arch building, etc, etc. Dagger kind of sounds like it's in the world view, but also a bit of the literature sounds more like it also wants to be the CI side, and I might just not understand it. So my experience with this is writing the automation logic in go, using mage as a lightweight project to allow for build targets and dependencies to be managed easily, and then writing a golang library to make working with containers / caching much better.

If dagger is just that, then I'm for it, if it's some heavyweight CI thing then I'd probably nudge towards a lightweight approach.
I guess I have similar feelings as they do, but it depends more on where Dagger is going than where it is right now. It feels largely like a docker orchestrator that adds in language SDKs for executing anything you might need plus caching. And that's really all I need (though I'm excited about public modules).

But I'm curious where y'all feel dagger is heading towards. I haven't seen any roadmap or general vision documented anywhere. Apologies for the block of text, but I thought it was all relevant to discussion

[winter linden]

It's 100% aligned. We don't aim for Dagger to be a full-blown "CI thing". We want Dagger to work great as automation logic on your favorite CI (or outside of CI altogether)

We do think that there's an opportunity to reinvent and simplify CI, and we think Dagger can provide the primitives to do that. Ie, if your CI runs 100% Dagger functions, do you really need the complexity of a traditional CI? Probably not. That's when it becomes tempting to reimplement a simple Dagger-native event trigger system. We think over time this pattern will happily co-exist with mainstream CI.

[hexed depot]

Beautiful ❤️ . I definitely vibe with wanting certain native CI-like functionality (like being able to selectively run pipelines based on certain directories changing), but as long as it can be used as generic automation, I think it'll work really nicely for us. Everyone is really on excited about being able to test pipelines locally and just using Go instead of yaml

[fierce sandal]

I’m encountering an authorization error when triggering Dagger to build and push a Docker image to the Docker Hub registry using GitHub Actions. The process works locally, but fails when triggered via GitHub Actions, showing a GraphQLRequestError related to OAuth token fetching.

I have confirmed that the Docker Hub login credentials are correct, I tested building and pushing a dummy image using the same Github Workflow. I know I am successfully logging in to Docker, as a success message is shown indicating this. Here’s the error

GraphQL Request Error Details: GraphQLRequestError: resolve: host: directory: dockerBuild: from: withExec: withSecretVariable: withSecretVariable: withExec: publish: failed to export: failed to push ***/interviewsageai:6c190f8a51d03e19c34479ab121be6926ec3ea27-dev: failed to authorize: failed to fetch oauth token: unexpected status from POST request to https://auth.docker.io/token: 401 Unauthorized

Relevant code:
https://stackblitz.com/edit/js-dbjajm?file=index.js

Could anyone offer insights into why this error occurs with GitHub Actions but not locally?

[sharp marsh]

Js (forked) - StackBlitz

[empty jay]

Hey folks, anyone heading to FOSDEM this weekend? Would love to meet up with any Dagger community members there. Just ping me if you're going and fancy a catch-up!

[narrow nymph]

Yes!

i'll be hanging around the container devroom on sat/sun and around the unofficial beer event on the friday

[leaden patio]

Hello 👋 is there any tool available to visualize dagger workflows from code? It would be really helpful to have a rendering of all the different steps within a particular command

[sharp marsh]

Hello 👋 is there any tool available to

[tired moth]

let's say 10 diff developers are contributing to one repo
How can we spin up a separate dev container through dagger like github codecpases for each PR of that repo?
locally or remotely?

[rigid pebble]

let's say 10 diff developers are

[leaden patio]

Hello 👋 is there documentation about the query language of dagger? Also do you know how I could have breakpoint and connect to the container being run while I'm executing my code? Is there some documentation about how to have productive debugging workflows while building pipelines and getting used to how dagger is working?

[winter linden]

Hi Remy, here is the documentation for the Dagger GraphQL API https://docs.dagger.io/api

For getting familiar with the workflow, I would recommend following the quickstart guide: https://docs.dagger.io/quickstart

Then browsing other guides.

[leaden patio]

I've tried to read them but I have trouble understanding how dagger is working. I get that there is an engine running on the background but it seems very opaque at the moment. Especially I would like to debug which files are where in the container and see what is the state of a given container but I don't see much content about it. Is there a way to connect to a container managed by dagger while it is running?

Typically I put a break point in my code, and I would like to see the container currently being worked on by dagger, is that an option?

[viral lark]

If I do this:

docker ps --format {{.Names}} --filter name='^/dagger-engine-*' | xargs --no-run-if-empty -I"{}" docker rm --force {}
cd directory/to/dagger.json
code .

vscode intellisense doesn't work until dagger mod install completes and then it works instantly.

This is true when vscode dagger.dagger extension is or is not installed...it doesn't affect.

I'm curious how vscode suddenly detects this or what the interaction/notifications that makes this work? How does vscode all of a sudden know that it needs to update?

[narrow nymph]

There's a magic dagger.gen.go file that contains the actual implementations and missing structs that are highlighted in red in that screenshot.
Dagger mod install forces those files to be generated 🎉

[vapid current]

Hi I am new to Dagger and I am trying to build a Docker image from a Dockerfile . The cookbook shows how to build a container and push its image to the registry. How does one locally build a Docker image ?

[vapid current]

I am using the Python SDK if that helps.

[slender charm]

Hi I am new to Dagger and I am trying to

[leaden patio]

Hello 👋 is there a way to create quickly a file with a given content in dagger? in the context of a container. I need to dump a configuration file inside a container at runtime with a static content

[sharp marsh]

hey there! yes, you can use WithNewFile which allows you to create a file in the container with static content.

[chilly arch]

@everyone The Dagger Community Call is moving back to Zoom starting this week. We will use Zoom for the video, and this Discord channel for the chat. I look forward to seeing all of you this Thursday!

Don't forget to register here, so you can easily login on Thursday - https://dagger-io.zoom.us/webinar/register/8517071155537/WN_j_xRpP1ASb6xrM4AiRKRnA

[winter linden]

Context: https://twitter.com/solomonstre/status/1750567718141727073

Solomon Hykes / @[e-mail] (@solomonstre) on X

FYI @discord we are forced to move our community calls to Zoom because there are just too many issues with Discord video calls. We love Discord and tried so hard to do it all in one place. It would be the best experience for our community - if it worked reliably.

Twitter · Jan 25, 2024, 5:14 PM

[real kite]

Zoom for community calls

[viral lark]

Moving from Optional to // +optional from #1203102934901071882 message.

What is the thing that enables this to work? Is it a preprocessing step?

I'm not familiar with this: // +optional

[winter linden]

Hi all, today we changed the pricing of Dagger Cloud.

If you're a paying customer on the Team Early Access plan, there is no urgent action needed. Your are still on the original plan, at the original price. We will contact you directly with details on how to take advantage of the new pricing. Here's the change in a nutshell:

• Old pricing: metered, $0.05/minute for visualization + distributed caching.

• New pricing: flat fee, $50/month for visualization up to 10 users. Includes free access to distributed caching, now labeled as experimental.

This new pricing addresses several concerns we've heard from you:

• No more paying for distributed caching even when not using it.
• Incentives are better aligned: no more paying more when pipelines are slower, and less when they are faster.
• No more paying by the minute when running Dagger on your laptop.
• No more surprise bills at the end of the month

You can see the new pricing here: https://dagger.io/pricing. We are also working on a new plan, that will nicely complement the Team plan. More on that soon.

Thank you to everyone making an early bet on Dagger Cloud, we appreciate it!

[viral lark]

Discord - A New Way to Chat with Friends...

[fierce sandal]

Hi all, looking to take on an issue on your GitHub, wanted to get more context on this issue:
https://github.com/dagger/dagger/issues/6589

Before diving into this, I wanted to understand if this is an issue that needs to be resolved, alternatively something you will address in your documentation or just a misunderstanding of how WithoutDirectory works?

[chilly arch]

Can't wait to see everyone at tomorrow's Dagger Community Call at 9 am PT. Just a reminder that we are moving from Discord to Zoom.** Please register at the link below to join the call seamlessly tomorrow. **

https://dagger-io.zoom.us/webinar/register/8517071155537/WN_j_xRpP1ASb6xrM4AiRKRnA

We have a great demo lineup - @warm temple, @fallen temple , @fierce sandal , Anais (joining Discord soon), and even @winter linden will share a few updates as well!

See you there!

[leaden patio]

Hello 👋 how can I gather both the stdout and the stderr of a .WithExec() ?

[leaden patio]

Also is there a way to make stdout mute the errorexec showing up while using dagger run ?

[slender star]

@leaden patio 👋 Let's explore over here in #1205190095246852188 message

[chilly arch]

The Community call starts in 10 minutes. See you there! https://dagger-io.zoom.us/webinar/register/8517071155537/WN_j_xRpP1ASb6xrM4AiRKRnA

[slender star]

👋

[warm temple]

Community call chat here 👋

[sharp marsh]

👋

[winter linden]

Good morning from San Francisco 🌅

[fallen temple]

Hi! Joining from Durham, North Carolina

[slender star]

Hi from Portland, Oregon. On the Dagger community call!

[cunning jolt]

👋 Westlake, TX

[fierce sandal]

Hi! Joining from New York

[slender star]

Zoom / Discord combo!

[warm temple]

Michigan here 🥊

[stray steeple]

Hi from Iowa

[lapis slate]

Hello again, from Porto Portugal

[tired robin]

Hey 👋 Joining from Copenhagen 🌆

[last pine]

Hello from Lyon, France

[little breach]

Hii everyone 👋

[wraith niche]

Hello! From California 👋

[winter linden]

Lots of good content today!

[winter linden]

👆

[sharp marsh]

[fallen temple]

Daggernauts 🚀

[slender star]

#daggernauts

[cunning jolt]

is the planned official Zenith release v0.10.0?

[chilly arch]

Not yet, that is coming very soon!

[winter linden]

Yes it will be 0.10.

(which is not yet released as Miranda mentioned)

[winter linden]

Not too late to join, if you want to see Kyle's real-time Zenith demo 🙂

Still on 0.9.8, that's so yesterday Kyle

[fallen temple]

If you're building and running functions now, you might have noticed the visualization isn't great yet in Dagger Cloud.

We're working now on adding visualization.

If you're using functions, we'd love your input on making that visualization awesome.

[slender star]

this demo is pretty ruff

[winter linden]

Be careful with those puns, we don't want to ruffle any feathers

[fallen temple]

Crispy puns

[slender star]

Field guide: generally, know a module by its dagger.json (and colorful stripes)

[stray steeple]

Invoking modules with the dagger cli is a very nice low barrier to entry for trying g this out! Lots of magic happening:)

[winter linden]

Yes that was important to us in this design. You should be able to learn and use Dagger without having to choose an SDK and write code right away.

[viral lark]

Here's the repo we're talking about https://github.com/kpenfound/example-voting-app?tab=readme-ov-file#example-voting-app

[winter linden]

This demo app brings up memories of Dockercon keynotes...

[fallen temple]

Kyle's not wearing a green shirt today. Demo fail.

[sharp marsh]

what does 0.9.9 will bring to improve this? Just curious

[fallen temple]

The way of spinning up services now is amazing

[sharp marsh]

dev container

we could have a module for that

for sure

https://containers.dev/

[little breach]

like a portable dev environment?

Like GitPod?

(is that the name)

[winter linden]

Yes I think that's the use case

Note, you can also call Dagger Functions from your dev environment

[little breach]

they might have containers you could use in Dagger?

[winter linden]

Reimplementing your own dev envirnioment on top of Dagger is also possible, but more experimental (and more fun 😛 )

[sharp marsh]

I guess one of the challenges is bind-mounting the code in the devs machine. I'm sure @pseudo stream and @narrow nymph already have an idea on how to fix that 😄

[fallen temple]

📢 📢 📢 📢 📢

[pseudo stream]

Yeah the idea is to use something like mutagen as a replacement for lack of bind mounting https://mutagen.io

[sharp marsh]

lol, I was thinking exactly about that

[cunning jolt]

saw this the other day. I think they are using mutagen too. Would be really cool to see this in dagger https://www.docker.com/blog/announcing-synchronized-file-shares/

[slender star]

I've scanned some stuff, but I'm no expert

[sharp marsh]

I'd assume the new compose watch thing uses something around that

[fallen temple]

Security scanning issues blocking merge 😦

Security scanning issues found before you push 🙂

[narrow nymph]

exactly yes

[warm temple]

https://github.com/aquasecurity/trivy

scan all the things!

[stray steeple]

Expanding on the dev container use case…being able to manage an apps environment end to end from dev to ci with the same build specs/source of truth would be a game changer for #WorksOnMyMachine!

Once we daggerize an app or repo, we still need to work through the dev environment setup which so painful across platforms and team members

[chilly arch]

https://daggerverse.dev/mod/github.com/jpadams/daggerverse/trivy@21f9211e5727a0b6acce8a0ab6c35d01896dc642

[timber crater]

Hi Sam, I'm in California too at the moment!

[winter linden]

Recently I was asked "if Dagger turns your pipeline into software, doesn't that make your pipeline vulnerable to poor security practices in development, like leaking credentiaks?"

I thought that was an interesting question. I gave a specific answer, but curious what everyone here thinks about that.

[warm temple]

My totally unbiased answer - bad security practices will be bad with or without dagger 🙂

[timber crater]

Yes, this goes back to my daggerization talk because having the pipeline as code makes it easier IMHO to tie into your existing toolchain, so if you need to extend or modify your pipeline, it gets monitored by the same tools as everything else.

[cunning jolt]

same thought as Kyle. I don't think this is inherent to "coding".

[winter linden]

Feature requests for the Trivy scan module:

1. Take a Container type as argument instead of a ref, that way I can reuse the same functions to scan all my containers, whether I'm building them on the fly, pulling them from a registry, loading them from a tarball or docker engine, etc

[timber crater]

BTW, I have a background in "confidential computing" and that takes dagger to a whole new level, where the whole workload is run in ephermeral containers.

[fallen temple]

That's

[slender star]

@winter linden Yep! Looks like working well on dagger 0.9.7, but needs an update for 0.9.8 🙂
WithMountedFile(tar, dag.Host().File(tar)).

[winter linden]

2. Parse the scan output and expose it as structured data in custom Dagger types, so that I can use the Dagger API to query the data in the results. Maybe filter by criticality, pipe the results into a follow-up function (notifications?) etc

Anais I think your dog is just excited to start scanning containers in their Dagger pipeline

[slender star]

Also need to update the optional args

severity Optional[string],
exitCode Optional[int],
format Optional[string],
trivyImageTag Optional[string]) (string, error)

to new recommended format

[cunning jolt]

Feature request for basically any public module

To enable usage in enterprise environments, allow pre-pending a host to the From image. For example, instead of trivy:latest allow setting mycompanyhost/trivy:latest so that we can proxy that image via our company container registry.

Without the above it makes it tough to use public daggerverse modules in regulated enterprise environments.

[winter linden]

That is supported out of the box by the Dagger CLI for any Container argument

Thank you @little breach !

[little breach]

thank you for having me 🥳

[cunning jolt]

Is it? I don't think a lot of existing modules in daggerverse takes in a Container as the source image. The From is basically hard coded

[slender star]

Right, you want us to "not hard code" but take an optional override to a default container image, right?

[cunning jolt]

exactly

[winter linden]

Oh sorry I misunderstood.

[little breach]

you could use it as a standalone module and then pass the container in?

[cunning jolt]

You may not want to allow the user to override in certain cases. But you'd want to allow the image to be proxied. Basically provide a host like mycompany.com instead of docker.io which is used by default

[winter linden]

That error message is very very dense...

[warm temple]

Recently I was asked "if Dagger turns

[winter linden]

At least one way in which Dagger makes credentials security better 🙂

[little breach]

hahaha amazing

will tell people

[slender star]

yep, you can have an arg of type Container you could specify the image ref string on the CLI for that arg and dagger would resolve the container image ref and convert to Container type for you.

that's similar to how the --directory . worked. You give a directory string and it gets converted to a proper Dagger Directory type.

[little breach]

yes

I was curious what type it has to be to export the data

[winter linden]

Reaction to error handling example: @sharp marsh @rain oriole @elfin frigate yeah we need to give developers a PropagateExitError option to WithExec, so they don't have to jump through hoops

Strong meme game @fierce sandal

[timber crater]

very, very nice @fierce sandal The diagrams for the various stages are super helpful because they illusterate the distinctions .

[winter linden]

@fierce sandal what's next on the roadmap for your project?

[little breach]

I have been there -- issue in Dagger 😄 it will only get better

[slender star]

If you return a Container type, you can follow up in the cli with API funcs to further operate on it like stdout or publish or whatever 🙂

[fierce sandal]

YouTube: https://www.youtube.com/watch?v=VCLBkHA0Uns&feature=youtu.be
Slides: https://docs.google.com/presentation/d/13mSFffjtdvD1us7d7qkkV3GEns1TkTifU-iyXzY3-gQ/edit?usp=sharing

[timber crater]

I would love to see reporting and o11y , @fierce sandal

... after party in dev-audio ?

[chilly arch]

Thank you so much @little breach @fierce sandal @warm temple @fallen temple @winter linden for presenting today! I will share the zoom recording as soon as it is done rendering, and then the demo snippets will be on youtube soon.

[fierce sandal]

Thanks. It is not my presentation if it has not been 'memefied'

[fierce sandal]

Sounds good, let me know when

[winter linden]

Thanks everyone for joining today's community call. That was fun.

[timber crater]

trunk based development

[chilly arch]

The full Feb 8th Community Call recording can be found here. Demo snippets will be added to the demo forum soon - https://dagger-io.zoom.us/rec/share/QePhcZ_IenZpAvHjO01DypO5kvsJn4VDKisTcoK_FmnXUnHwIf_X7jmCwNmcloe8.QreZamERy0fRBLpp

[stoic knot]

I swapped out ts-node for tsx for my typescript pipelines and it is grrrrreat!

https://github.com/privatenumber/tsx

[stoic knot]

It looks like: dagger run yarn run tsx ./ci/dagger build

[sharp marsh]

Yep, tsx is great. We have a PR to bump our guides to use which we haven't merged yet https://github.com/dagger/dagger/issues/6143

[stoic knot]

nice, I'm working on combining dagger with yarn workspaces right now

The node_modules gets funky in a workspace setup, so I'm punting on the cache issue for now

[stoic knot]

Is it now possible to run and expose multiple containers to the host in Dagger?

• build & run webapp & rest api Dagger
• browse in chrome, curl API from terminal

a.k.a. docker-compose or... dagger-compose?

[sharp marsh]

are you using modules or just the the standard SDK?

[stoic knot]

just the standard sdk

[sharp marsh]

Via the standard SDK you can use the c2h interface and expose as many services to the host as you need. https://dagger.io/blog/dagger-0-9

[stoic knot]

I've got a snippet like this

export async function serve(cli: Command, client: Client, source: Directory) {
    const { web, api } = await images(cli, client, source)

    const tunnels = [
        client.host().tunnel(
            web.asService()
        ).start(),
        client.host().tunnel(
            api.asService()
        ).start(),
    ]

    await Promise.all(tunnels)
}

But I get [0.00s] ERROR tunnel

Is there a way to give a service a name, so that upstream can refer to it with a known value?

Maybe these options? https://docs.dagger.io/sdk/nodejs/reference/classes/api_client_gen.Service#constructor

Is there the equivalent of -p 8000:80?

if I want to leave them up...

1. is there a way to daemonize them? (I see up, start, stop params?)
2. do I have to start services&tunnels, wait for user SIGINT, finally exit func?

[winter linden]

Check out arguments to Host.tunnel (or simpler: service.up in latest versions)

[patent slate]

I just ran into this today. Looks like you all won over Viktor finally. 😄 https://www.youtube.com/watch?v=oosQ3z_9UEM

[stoic knot]

from code to staging, with e2e dagger https://ts.verdverm.com
yarn workspaces, 3-tier app, with local -> k8s, multi-arch

[wide lava]

Lol, he's a great pick-up

[patent slate]

I don't know, but this looks more like spam to me than a useful post. 🤔

[stoic knot]

the dagger team has been helping through getting this work yesterday/today

[patent slate]

Ah. Ok. Then I guess it is a thank you to them? Right? 🙂

[sharp marsh]

did you end up figuring this out?

[stoic knot]

not yet, I ended up punting on that for now and got the e2e ci done instead

although, now that I fixed a number of things while getting e2e working, it might be easier

[winter linden]

@everyone Attention Daggernauts! We just launched our new pricing, and it includes free stuff.

Dagger Cloud is now free for individuals, and $50/month flat rate for teams up to 10 users. No more metering by minute.

Our goal with this new free plan is to make Dagger Cloud a standard part of every Daggernaut's toolbox, whether you're using Dagger for work, for fun, or to learn (or all three!).

You can learn more about the new pricing here: https://dagger.io/blog/new-dagger-cloud-pricing

A big thank you for everyone who shared their feedback. We hope that this new pricing addresses it. We appreciate you!

[winter linden]

I'll quote from the blog post, to explain the reasoning behind a free plan (and why we didn't start with that):

When we built Dagger Cloud, we focused on the needs of one specific group: medium-to-large teams integrating Dagger into their CI. We didn’t expect the rest of our community - students, hobbyists, researchers, open-source maintainers - to need anything more than our open-source engine. Wrong again! It turns out that visualizing your pipelines in a web interface is always fun and useful, no matter what you use Dagger for.

Everything Dagger has achieved so far, we owe to our amazing community of Daggernauts. We know that many of you can’t justify the cost of a Team Plan for a personal project. So we decided to make pipeline visualization free for individuals, starting today.

[stray marlin]

Signing up for an account takes me to the Dashboard which instructs me to Connect with Dagger Cloud which I do and it takes me back to create your first organization page.

I've done this three times nows.

Org names:
chrisshort
chris-short
thechrisshort

[sand gorge]

Who has a list of all dagger talks/presentations that have been done? I will be unleaching Dagger to the PHP community, in 2 days time, and have a bunch of homework to do.

Please drop the URLs to me here, thank you

[stoic knot]

having an issue signing up for cloud, was able to auth with github, put in the org name, clicked next, and then it took me right back to that page, org was created (toast), and can no longer be used, but going to the cloud index page sends me back to step one of the form to enter org name

ok, seems I can now reach the org

[chilly arch]

Thanks for the message! We are working on a bug fix here - https://discord.com/channels/707636530424053791/1206657183845056532

[stoic knot]

Got sorted for now!

ok, question, just had the issue where I have to restart my local dagger engine, does connecting my local dev to the cloud... can that help diagnose what is happening in the long run?

[sand gorge]

@heavy karma @rain oriole @heavy gazelle are any of you guys online at the moment?

[fallen temple]

@sand gorge , they're all offline right now because of time zones. Do you need help?

[sand gorge]

@fallen temple I've sorted that blocker issue btw.

Found a bunch of installation issues btw, like DX problems .. when installing Dagger CLI

[winter linden]

Can you post about them in #1030538312508776540 ? We'll make sure to address them. thanks!

[sand gorge]

aye, will do

@winter linden got any "common examples" on the docs somewhere? like: WebApp stuff, which might needs Redis or sqlite and so on ..

Can be any language, like python or JS or go ..

Update: I found the "Cookbook" on the website .. is that a good place for it? or do we have something better

[sand gorge]

@stoic knot what visualiation tools is this ? looks awesome 🙂

[stoic knot]

It's Dagger Cloud, you can hook it up to your local dev environment by exporting the key

[sand gorge]

@stoic knot great, seen some bits about Cloud so far but not checked it out yet! Got a good link (page, or video) on some cool shiny things to show people? I'm prepping for a talk in 2 days time, and the shiny stuff that's FREE would be great to showcase 😄 🙂

[stoic knot]

Scroll up to the "Dagger for Everyone" post earlier today. I signed up for cloud and this was my first attempt. Will definitely be useful for your demo

[sand gorge]

Okay, noted, thanks for the reply

[fallen temple]

@sand gorge and @stoic knot , any feedback or support you need, let me know. I'm a PM working on Dagger and am always looking for more feedback. We have some new visualization features coming soon and are looking for input on those as well. 😄

[sand gorge]

@fallen temple feel free to show them to me, once you're ready!

[fallen temple]

Great! I'll DM you.

[wide lava]

Thank you @winter linden this is huge!

[sand gorge]

@winter linden I'm looking through the cookbooks - https://docs.dagger.io/cookbook#build-image-from-dockerfile

So my angle here is.
a) everyone is using github actions/jenkins for existing CI pipeline
b) I want to show them how they can keep most of what they already do, just with Dagger instead.

Right now everyone has a Dockerfile (image/container), with things specified in it such the FROM line and COPY commands, as part of the docker build phase.

a lot of the examples/approach on the cookbook are like this

From("alpine:latest").
WithDirectory("/src", project).
WithWorkdir("/src")

Whereas this stuff is usually baked into the Dockerfile.

What is your opinion on people no longer doing this stuff inside their Dockerfiles, but instead inside Dagger DSL/code?

My opinion is that stuff (FROM, COPY, RUN/EXEC) still needs to be in the Dockerfile, as that's what is going to go to production, so it'll now be duplicated in the Dagger file and in the Dockerfile.

Thanks 🙂

[deft wyvern]

What's the easiest way from the CLI to dump the graphql schema for a module, like github.com/vito/daggerverse/go@4809c149d0ed114dc199bc17cfec6b51e4f27858

[winter linden]

I don't know about raw graphql schema, but there's a full introspection api builtin

[winter linden]

dag.Git(URL).Branch("main").Tree().AsModule() then introspect from there

[winter linden]

https://pkg.go.dev/dagger.io/dagger#Module

so you can write a regular module that inspects other modules (or itself)

[sharp marsh]

Probably using dagger session, load a module and query the graphql introspection API will work. cc @deft wyvern

[winter linden]

Introspect modules

[stoic knot]

Being able to filter builds by user, branch, commit message, etc... within a search box, so I don't have to look around & click, especially when beyond the fold or paging. Autocompleting known values for a column during typing would be great as well.

[stoic knot]

Feedback on Cloud UI / DX

[sand gorge]

Is it possible to have a "checklist" of all features that a SDK supports. That would be phenomenal help for me, as I'm looking to evaluate what our current WIP PHP SDK already supports, and what I need to implement to take it out of experimental phase into ready-for-production phase?

Slightly different question is, is there a way we currently track feature readiness of each of the SDKs, to compare them?

I've been manually cross-checking things but tbh maybe someone's already done this work

Thanks.

[sand gorge]

I've just found some issues whereby the PHP dagger SDK is designed for dagger engine v0.9.10

I'm following the official installation page, which told me to install 0.9.7

and now I get critical failures when using the SDK as it needs the new "stuff" inside 0.9.10

I'm now passing this onto here to see what we do about it? as if the general public try this, tomorrow, then they'll hit the same problem as me, and likely give up on trying Dagger (which is my intention for them)

https://docs.dagger.io/cli/465058/install/

This is what the docs say:

curl -L https://dl.dagger.io/dagger/install.sh | DAGGER_VERSION=0.9.7 sh

Just a FYI for dagger team to discuss on next steps. Do we update the website to 0.9.10?

Cheers 👍

[narrow nymph]

Those instructions are just for installing a specific version - the instructions above it describe how to just install the latest version

It would be nice if the specific version instructions were always the latest version I suppose though

[sand gorge]

If we're telling people what commands to copy and paste, on the website, but that version is out of date compared to the SDK's we're shipping, then we have a mismatch, from what I can see.

My $0.02 on my experience, following the docs like everyone else ultimately will

Next, we have a permissions issue on linux.

As a normal user, as per installation documentation, it is recommending to install things to the user's /.local/ DIR

curl -L https://dl.dagger.io/dagger/install.sh | BIN_DIR=$HOME/.local/bin sh

This works fine, but the next output the user gets is BASH completion, but it breaks since it's giving root-level instructions to non-root users

mkdir -p /bash-completion/completions
dagger completion bash > /bash-completion/completions/dagger

This is hitting root dirs, when you're not root. I think this installer output needs fixed to

~/bash-completion/..

or wherever the right place is.

On the 2nd command of the installation guide I get a roadblock, basically.

1. run installer SH command <-- works
2. run the bash autocomplete command <-- doesn't work

This is also free feedback, but I also foresee many users who are about to pick this up, hitting the same roadblocks.

Cheers 👍

[sudden ocean]

I remember something similar to this but it’s in elixir sdk https://github.com/dagger/dagger/pull/5530#discussion_r1293742628. The idea is that modifying sdk:lang:bump to bump the version inside the doc to make it up to date. I believe this way would solve on php as well.

[sand gorge]

I'm reporting a signup bug on Dagger Cloud

I clicked 'authenticate with github', did that. Then it asked me to enter an organization name. I did that.

Then it redirected me back to the "enter your organization name" page, with the green success flash message saying "organisation successfully created".

I've entered the organisation name again and it says "it already exists"

Thus
a) I made an org
b) I got sent back a page to the "please make an org page"
c) I can't make the org as it already exists
d) infinite signup loop

Sorry to be the bearer of bad news 🙈 was excited to try out Dagger Cloud

Hope you get it fixed 🙂

[sharp marsh]

cc @fallen temple

[chilly arch]

Thanks for reporting @sand gorge! Let's chat here, so we can get this fixed for you #1206657183845056532 message

[stoic knot]

So it dawned on me that all my local build logs are going up to the same UI that my CI uses. That's pretty great, and really useful

We just need a way to differentiate them: who, title, key/val pairs, etc...

[glad jolt]

https://github.com/dagger/dagger.io/blob/6f85d744cfb61037bb1c1b82ae4bfe10e0bcd9f7/app/cloud/providers/OrgProvider.tsx#L43

[rain oriole]

That's different. He's talking about the CLI's install instructions.

[rain oriole]

Make sure you see the SDK Contribution Guide, in the sdk dir of our repo. It's a WIP, but it's the most comprehensive at the moment. There's several collapsed sections that you can expand, especially notes on todos to doc. Reach out to me in https://discord.com/channels/707636530424053791/1121837200712142878 to get clarification in anything from there. We've been focused on getting modules ready. There's no docs for adding support for modules in an SDK yet, but it'll come.

[sand gorge]

Traditional CI pipelines, using docker-compose (or docker) is to use a volume, to perform a task, and then the volume is there to persist the "results" onto the CI host HDD.

Like
a) nodejs container does something, and outputs stuff to a ./output/ dir
b) upload the ./upload/ DIR to production, or sync it to s3

Given dagger doesn't do volumes (things shared between the host) - what's the procedure here? How can I take stuff from the dagger pipeline, and and get it back out of there, to use later in the CI pipeline?

Thanks in advance 🙏

[sharp marsh]

has been fixed Paul. Thx for reporting 🙏

[sand gorge]

I keep calling my dog Docker, instead of Koder

These last few days 🤣

Daggerland has renewed my mind

[sand gorge]

Who can help me fin some cool graphics, for my presentation, showing a DAG (or a workflow) ..

I wanna show people what workflows look like, and/or what a DAG looks like .. and how it can be complex.

Then I can show them that Dagger just "makes it easy" for them..

Showing them a few graphics will be totally worth it, compared to me trying to explain it to them .. thanks in advance <#

[stoic knot]

Who can help me fin some cool graphics,

[azure moat]

I'm trying to move my builds over to Dagger and I seem to be hitting network issues in the container.

1. npm ci fails with a timeout on the container, but works fine outside of the container.

npm ERR! code ETIMEDOUT
┃ npm ERR! errno ETIMEDOUT
┃ npm ERR! network request to https://registry.npmjs.org/supports-color/-/supports-color-5.5.0.tgz failed, reason:
┃ npm ERR! network This is a problem related to network connectivity.
┃ npm ERR! network In most cases you are behind a proxy or have bad network settings.

npm := client.Container().From("node:20-alpine")
npm = npm.WithDirectory("/src", src).WithExec([]string{"npm", "ci"})

2. One of my tests uses docker-compose to set up db dependencies and it doesn't seem to work in the container. I am using the docker in docker image (https://hub.docker.com/_/docker) and and I get tcp bind errors calling docker-compose from a shell script that is kicked off by a golang test. This also works fine outside of the container.

I'm curious if these are known issues. I looked through old messages and didn't see anything directly related.
I'm using Dagger version 0.9.8.

[stoic knot]

BAM, just got permission to do a Dagger POC with my main devops client, to start immediately to unblock a larger effort, but mainly from a long-term vision to solve/remove some "hacks" we've created to deal with a monorepo. This should hopefully stress test Dagger as well

[cunning jolt]

very nice! Are you planning to use modules? Or vanilla dagger?

[stoic knot]

Just vanilla, it's a monorepo after all :]

There is only one go.mod

Think of this pattern between polyrepo and monolith

1. If you have one repo where everything has it's own version and publish cycle, you're not really doing monorepo
2. Uniform versioning means you get the benefits of shared code during dev/ci, but can still deploy as microservices to k8s
3. Building & testing the minimal path is challenging, especially when you want to build and test the code as it would look post PR, before the PR merge. This is where Dagger might really shine for us long-term.

[sharp marsh]

docker - Official Image | Docker Hub

[chilly arch]

Tomorrow at 8:00 am PT, @little breach and @warm temple are going to do a Dagger livestream! If you are new to Dagger or want to learn about the latest ways to use Project Zenith, I highly suggest attending this - https://www.youtube.com/live/Rc0NCpc2YSc?si=-NjnNoeGOCvdXtrO

[slender star]

Live right now! 👆

[stoic knot]

is there a dagger env var to turn off the colors / pretty in CI only?

[winter linden]

I think dagger --progress=plain is what you're looking for

[stoic knot]

I'm hoping for an env var so I don't have to add / condition a flag everywhere

or maybe there is a config file where I can set the default output?

Another question, is WithMountedCache concurrency safe?

1. I expect this mainly different invocations of dagger run on the same VM
2. what about within a single program? (probably the same answer?)

I'm using it for go get, go build, which at the host level are concurrency safe (they use a lock file to prevent race conditions). I'm wondering if this will hold trun within buildkit for a shared mounted cache

[elfin frigate]

it should respect NO_COLOR (https://no-color.org/)

[elfin frigate]

it has 3 types of cache control; 'shared' (default) allows concurrent r/w, 'locked' only allows one to access it at a time, 'private' does some kind of weird copy-on-write or last-write-wins thing that i've never fully understood, but doesn't lock, and doesn't share

[winter linden]

I actually finally read the docs of "private" the other day 🙂

• Each build session starts with its own empty directory
• The directory is persisted across execs of that session
• At the end of the session everything is wiped

I also thought there was COW involved, but actually not.

[elfin frigate]

ah interesting, I interpreted it as 'private copy' or something 🤷‍♂️

but really it's private in a "burn after reading" sense

[winter linden]

Yeah like "a private implementation detail of this build"

[elfin frigate]

have you tried running dagger init in your home dir yet? seems to try to upload everything for me

[winter linden]

Yes...

[chilly arch]

All of the demos from the community call last week are up! Check out the recordings in our demo forum. The speakers are tagged, so feel free to ask them any questions:

• https://discord.com/channels/707636530424053791/1207941423207292968 - this is a great one to watch if you want to see the latest release changes
• https://discord.com/channels/707636530424053791/1207947753825636402 - awesome real-world example
• https://discord.com/channels/707636530424053791/1207924803122761770 - If you are a Trivy fan, this demo is for you!

Thank you again @little breach @fierce sandal and @warm temple !

[stoic knot]

is a session a dagger.connect from the user's pov?

[winter linden]

Yes I believe so

[sand gorge]

Happy dagger day everyone

[sudden rain]

@sand gorge at PHP UK conference 🎉

[chilly arch]

Thanks for sharing the picture. Great work @sand gorge !

[vestal apex]

Anyone got an example with multiple functions?
Playing a bit with dagger call completion: https://asciinema.org/a/K3ZA9tEInsOvK7D63M6A5m2cr

[elfin frigate]

PSA: we're in https://discord.com/channels/707636530424053791/911305510882513037 for this week's Zenith office hour!

[sand gorge]

I've en encouraged people to come here and join #php channel too! Let's go 💪

Many have already joined

[chilly arch]

Thanks to the Daggernauts worldwide, we have a few upcoming Meetups secured!

**If you are in the Raleigh, NC area, check out this Dagger meetup on March 27th **- hosted by @earnest prawn @supple garden and @lofty star with special visit from @fallen temple 🙂 https://www.meetup.com/dagger-raleigh/events/299243690/

If you live near Paris, FR or will be in the area for KubeCon EU, check out this Dagger meetup on March 18th - hosted by @tawdry imp. Many folks from the Dagger team will be there too! https://www.meetup.com/dagger-paris/events/299243165/

[stoic knot]

Where is the best place to learn about / get hands on with modules right now?

a YouTube video?

[winter linden]

https://docs.dagger.io/zenith

[stoic knot]

Does a module require its own go.mod?

I saw this function: https://pkg.go.dev/dagger.io/dagger#Directory.AsModule

What I'm after is...

1. We already use CUE to create a software catalog and to organize our build tree
2. There are custom steps in Dockerfiles that I'd like to migrate to Dagger
3. I'm running up against the dynamic loading problem, because we're using the Go SDK (Go shop)... because Go's imports, where should I define the custom steps as Go code?
4. Hoping that Dagger can solve this, might just load the Dockerfile as a stopgap
5. Can the 1000s of lines and go.mod stuff be handled at runtime? I don't think that will be committed to git for desired for the 100+ places we'd need it

Where is dag defined? Do I have to have a red squiggle in my VSCode all the time?

[mystic citrus]

I want to replace the GitHub Action https://github.com/goreleaser/goreleaser-action
with Dagger.

**I see two options: **

1. Create a Dagger Module, makes only sense if I am the owner of goreleaser, however we are just users
2. call the goreleaser binary/container with the args,

For the second option, I am to new to Dagger to understanad the complexity and effort, maybe someone can mention the right dagger Terminaloty or even docs.

[rigid pebble]

Hi Vadim!!

Create a Dagger Module, makes only sense if I am the owner of goreleaser, however we are just users
You can create a module that wraps any existing tool, no need to be the owner of the tool! If you do go down this route, you could let them now that you want to create a module and you might get some feedback and help!

call the goreleaser binary/container with the args,
This would definitely be the quickest approach. To use any CLI from within a dagger module you commonly have two alternatives: i) start from a docker image with go releaser already installed; or ii) start from an alpine/ubuntu/etc image and install it yourself. From there it's only a matter of mounting the directories/files you need and running the commands. Quick example here starting from a goreleaser pre-built image:

package main

import (
    "context"
)

type Goreleaser struct{}

func (m *Goreleaser) Version(ctx context.Context) (string, error) {
    return dag.Container().
        From("goreleaser/goreleaser:v1.24.0").
        WithExec([]string{"help"}).
        Stdout(ctx)
}

In the code snippet we start a container using From pointing to a specific goreleaser image, then we execute help. Since the image's entrypoint is goreleaser we don't need to specify that command

[stoic knot]

Has anyone been using git-lfs from within Dagger to pull stuff in for their builds? Or more generally git work within containers? There is a gap in the Dagger API for the things I need to do (lfs, diff, merge). I assume I have to mount in the git secrets I need at a minimum

[narrow nymph]

no support in the dagger api yet, but definitely feel free to open an issue for it if you like - i'd definitely love to have support for this as a first-class citizen 😄
when you do mount git secrets, you should do so super carefully though - you shouldn't create any persistent files on disk, you can use env vars like this if you like: https://github.com/dagger/dagger/pull/6488/files/5b05e797de6c2ffa9422c3cc098000bd1c6e49a8#diff-42ce6e1e153a825195b3159046bc878851a2645c5965daaa5a3db9c173678c2d

[winter linden]

lfs support is one of the reasons I started experimenting with a userland git module, that doesn't rely on builtin git functions. https://daggerverse.dev/mod/github.com/shykes/daggerverse/supergit

[stoic knot]

Another thing we do is to merge the current commit with the latest passing commit from the branch it will eventually be merged into, so we have more correct build/test

My current plan is to mount the host directory to a container with git&lfs + secrets, do the work I need to, then use the updated directory for all subsequent steps

Actually, we really only want to do that on demand, when we have to build a service that needs a lfs pull

[tired moth]

can we have jsr.io package to make dagger compatible with any js runtime
so developers can use dagger with deno and bun and all...i think
do i need to create issue for that in dagger repo?

[narrow nymph]

@wispy tapir is actually looking into this with the bun runtime 🎉
See the discussion in https://github.com/dagger/dagger/issues/6591

[wispy tapir]

Still a long way to go but we're actively working on it!

[sand gorge]

Hey Folks!

My dagger talk (at PHPUK) is now online: https://www.youtube.com/watch?v=iwhwWtQ2UQI

Dagger team, if you're curious what the general public's questions are about Dagger, upon first encounter, you can skip to 47:00 minutes in, to see that perspective: https://youtu.be/iwhwWtQ2UQI?t=2822

If you want to skip all the intro's and contexts about dagger, and "why" and skip to the live demos and code sessions, you can skip to 29:43: https://youtu.be/iwhwWtQ2UQI?t=1783

Thanks 👍

[sand gorge]

Right after my talk, we had my friend David Flanagan, who's talking generically about Pushing the limits of PHP stuff (Pulumi, Dagger, WASM ..etc) and he spent lots of time covering Dagger in his talk, so it was a good natural extension of my talk. (it was by design why he was after me)

Here's the part of David's talk where Dagger starts:
https://youtu.be/gJ7vEACAYmE?t=696

[winter linden]

nicely done @sand gorge @vestal creek ! We appreciate you!

[stoic knot]

Is there a way I can remove the need to push B to a remote registry in the following? i.e. tag a container in the local buildkit engine to avoid the remote aspect

A   (external golang image)
B   (dagger sdk) -> (my-reg/my-golang)
C   (Dockerfile) (FROM my-reg/my-golang) -> (my-reg/final-image)

[chilly arch]

Thanks @sand gorge and @vestal creek ! I just added your talks to our community content resource page 🙂 https://dagger.io/resources

**Tomorrow is the Dagger Community Call! **Sign up for the call at the link below if you haven't already.

https://dagger-io.zoom.us/webinar/register/8517071155537/WN_j_xRpP1ASb6xrM4AiRKRnA

Don't miss this great lineup of speakers: @fossil pine @rain oriole @cloud canyon

[sharp marsh]

Well. you could do something like this:

    dag.Container().
        From("alpine").
        WithFile("Dockerfile", dag.Host().File("Dockerfile")).
        Rootfs().DockerBuild().Sync(ctx)

and then in your Dockerfike:

FROM scratch
COPY / /
RUN ls -l

^ the copy / / will merge all the contents of the alpine image into the Dockerfile build context

[stoic knot]

that sounds expensive for a large base image (multi-GB)

[stoic knot]

Is it possible to attach to a Dagger / BuildKit container? like -it for docker run? I was just thinking about doing dagger stuff from the python repl and thought it would be cool to drop into a container along the way

[sharp marsh]

well.. since the COPY is running withing the build context, I'm not sure if in this case it can leverage the merge op to make this efficient. Probably @pseudo stream will know

[winter linden]

dagger call <function that returns a container> terminal

[stoic knot]

we're not using modules or call, I was in a python repl

[winter linden]

should work from a custom client also. Call Container.Terminal() from your client, it might just take over the client's terminal and give you a shell. Never tried to be honest

[stoic knot]

will try later, about to show off my dagger work to the client

[winter linden]

Ah it will not. You'll get a Terminal type. From there you can get a websocket endpoint I think

[sharp marsh]

doesn't work unfortunately. Terminal returns a struct with a websocket endpoint that you need to connect via the session http transport

[sharp marsh]

more about this here: #1204482091610280058 message

[analog raven]

Hey everyone! Reminder that our community call starts soon. See you all there!

https://dagger-io.zoom.us/webinar/register/9117071155619/WN_j_xRpP1ASb6xrM4AiRKRnA#/registration

[slender charm]

Welcome! You are invited to join a webin...

[wraith niche]

@everyone Dagger Community Call starting now - 3 awesome demos on the way, feel free to join us! https://discord.gg/KGUUp7J3?event=1206710665557971065

[fallen temple]

RAH-lee (Raleigh)

[analog raven]

Thank you!!

[fallen temple]

@cloud canyon demos always require you put on a helmet

OpenMeter ❤️ great solution for usage-based billing

[viral lark]

https://github.com/openmeterio/openmeter?tab=readme-ov-file#----openmeter--

[wraith niche]

Looks like a lot of this code is reusable (not specific to a single app), did you publish some of those modules Mark?

[winter linden]

I have the same question 🙂 We want the modules Mark! The world needs them

[cunning jolt]

it won't work in a private repo yet

[wraith niche]

Working on it! 🙂

[winter linden]

https://tenor.com/view/we-love-mark-gif-24846471

[slender star]

https://tenor.com/view/roman-atwood-smile-more-mark-gif-23679822

[fossil pine]

Here is my daggerverse repo: https://github.com/sagikazarmark/daggerverse/

[wraith niche]

great demo, thanks for sharing those modules!

[winter linden]

https://tenor.com/view/jim-the-office-what-is-going-on-what-going-gif-25239368

[stray steeple]

Thanks for sharing the overall approach and organization Mark! I’m excited to review more and use in other projects!

[winter linden]

Everyone not on the community zoom call right now 👆

[fossil pine]

AFAIK we are supposed to use this channel for chat during the call, aren't we?

[winter linden]

It's not too late 🙂 https://dagger-io.zoom.us/webinar/register/9117071155619/WN_j_xRpP1ASb6xrM4AiRKRnA#/registration

[fossil pine]

And here is the project I demoed: https://github.com/openmeterio/openmeter/tree/main/ci

[winter linden]

yes that's the idea. We tried tucking it away in a thread, but then people wouldn't find it

[slender star]

🤯

[wraith niche]

Sharing context for everyone, since Helder's demo is a deep dive into Dagger internal: Dagger's codegen is the core code that allows Dagger to work cross language (via a GraphQL API in the middle). And it allows to introspect the modules internal, like Helder is showing now.

[rain oriole]

https://daggerverse.dev/mod/github.com/helderco/daggerverse/codegen@d14418fe52e5180b60bb96e78afc80172dac79f7#Codegen.introspect

[slender star]

prediction: the SDKs that users are asking for are going to be easier than ever to build. In the last two days alone at a show in Austin, people asked for a Dart SDK, Ruby SDK, Kotlin SDK, and C# SDK (which is brewing).

[winter linden]

@rain oriole I really want to play with this, but could really use an example module using it

[fallen temple]

https://tenor.com/view/helmet-on-nick-zetta-basically-homeless-im-ready-ready-to-go-gif-17532760

[wraith niche]

Thanks Helder, great demo!

[rain oriole]

I have a python module: https://github.com/helderco/daggerverse/tree/main/python-codegen
Also have a Go one, but it's local. I experimented with navigating the introspection in a template 🙂

[wraith niche]

Some context on Andrea's demo (it took me some time to understand the potential when I saw the demo the first time): it allows to turn your entire Github workflow.yml into a Go pipeline (or later Python, Javascript).

[slender star]

So cool. I want to send ❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️❤️ on Zoom, but could only send
~ ~ ~ ~ ~ ~❤️

😂

[wraith niche]

"The last YAML you will ever write" -> and you can even get it from Andrea so you don't even have to write the last one yourself...

[winter linden]

My favorite part is that the github event format is the same in Github Actions, and in regular Github webhooks... So once I switch to this, I have the option of ditching Github Actions altogether, and just running a webhook server that passes all events to the same code I already wrote

[winter linden]

And that in turn opens the possibility of fixing the biggest security issue in CI right now: pull requests that can change their own CI pipeline.

[sand gorge]

I'm here now! Zooming up

[rain oriole]

Another experiment (go template):

dagger -m github.com/helderco/daggerverse/go-python-codegen call object-names

[cunning jolt]

this could potentially replace something like prow

[sand gorge]

I'm on the zoom! 🍿

[stray steeple]

Portable gitops -love it!

[sand gorge]

Question: daggerverse (zenith modules) - do you have to always write them in Golang?

Yes/No

[winter linden]

No

[rain oriole]

You need an SDK that's supports it. PHP could be next 🙂

[sand gorge]

To build the PHP module, we'll need to write that in another language though, right? (chicken and egg scenario)

Sorry if this is a silly question

[rain oriole]

Just the runtime module. It's used to create the container that will run PHP modules. Here's an example from Python: https://github.com/dagger/dagger/blob/main/sdk/python/runtime/main.go

[sand gorge]

Yes, this is exactly what I'm talking about. The python and JS runtimes are written in golang (main.go)

My Q was, when we do this for the PHP runtime, do I have to write it in Golang? Right?

[rain oriole]

We can help with that. That's a very small part. All the module support will actually need to be in PHP itself. In the SDK. The runtime module is just the pipeline that creates the container for that PHP code to run in.

[sand gorge]

"runtime module" let's align terminology. Okay, well I think first next steps is to create the runtime module for PHP in Golang, right? We need to pair on this as PHP works quite quirky, and won't be as straight forward as Python to integrate.

[elfin frigate]

technically you could implement the runtime module in any language that supports Zenith modules (so Go, TS, Python)

[rain oriole]

I'm familiar with PHP, I can help you guys with that.

[sand gorge]

@rain oriole we should set aside some time to have a call, and go over all the bits and have a first stab at making the PHP runtime, in Golang.

There are quirks that I should be around to explain and understand, so we can make any "workarounds/tweaks" as we go along.

We could do it next week @rain oriole if you're up for that.?

[rain oriole]

Yep 👍

[sand gorge]

🤝

[rain oriole]

It's actually not a requirement to get started. You can go suprisingly far in supporting modules without it. The hardest part is introspecting the PHP code and 1) being able to report which functions are there; 2) being able to execute one of them having the name and inputs.

[sand gorge]

I'm sure once I have a deeper understanding of "what's going on" then I'll be able to easily suggest the best route to solve each situation.

Looking forward to it @rain oriole 🙂

---

I just spent the last 2 hours in C++ land .. I dunno if I like golang better than C++ at this point .. I can read golang but didn't write it yet.

Keen to get stuck into this runtime module dev, and it'll be my first hands-on experience with golang 😄 and then I can fall in love with it

[ancient escarp]

are there any architecture diagrams about on how a runtime module is supposed to work?

[winter linden]

We are missing a "SDK developer manual". Will get there eventually.

[sand gorge]

Meet Chris/Carnage - he's a specialist in the PHP domain and my colleague/friend ☝️

[winter linden]

I looked at the source code of both examples and still don't get it...

[frozen basin]

if I've got a general feature request, would I put that on GH issues?

[sharp marsh]

yep, that works. You can also drop it here as well to expand about it 🙏

[frozen basin]

Just want to be able to set the title of my runs in dagger cloud. I execute multiple pipelines per commit, so it's a little hard to find what I'm looking for based on commit title.
See: https://discord.com/channels/707636530424053791/1210658598288359456

Feature request: https://github.com/dagger/dagger/issues/6727

Thanks!

[sharp marsh]

thx for reporting 🙏 you're using Zenith for what I see, correct? We're working on a new UI for module calls and we'll make sure to add this input 🙏 cc @fallen temple

[frozen basin]

no not on zenith for anything yet, I'm calling through a golang cli

[sharp marsh]

oh, I see!. Do the changes view help any better for that? https://dagger.cloud/$org/changes. That view is partiularly designed for multiple runs that are related to the same commit. It's optimized if you're using different Pipelines in those runs though 🙌

[frozen basin]

Umm no there's no difference there - shows the same commit as title

[sharp marsh]

yes, the difference is that once you click on the commit you're looking for (you should get only one commit per row), you'll be taken to another page where you can see the dagger runs for that commit by the dagger Pipeline

[frozen basin]

oh interesting yeah that's better

[timber sphinx]

When do y'all think the new dagger call for replacing my other mage tooling will be ready for me to use in org and relatively stable? I'm excited And with mixed environments of Linux, windows, and macOS perfect time to start building some modules 😁

[winter linden]

You can start building on it. We're in the final stages of preparing the release.

All the commands are already available in 0.9.11, at this point we're adding polish, fixing bugs, preparing new docs

[stoic knot]

Is there a reason I cannot scroll my terminal while running the Dagger CLI? (also copy/paste)

[stoic knot]

Getting my first feedback about Dagger, seeing some major issues with FS, where there is 10G+ in the context. This is causing 20m+ upload / directory({ exclude: [...]}) times

[winter linden]

Directory upload

[chilly arch]

Thank you @rain oriole @fossil pine and @cloud canyon for demoing at last week's community call, and thank you @analog raven and @wraith niche for hosting while I was OOO!

If you missed the community call, you can check out the full recording here: https://dagger-io.zoom.us/rec/share/DY1jxT-GL791yiFJD9yfv9pJdSAoR4aPw-87S7OGeYYkye7zDsz7t6ay8NsvnDjc.2uqrm4cBne4KTRnz

I will add the demo snippets to youtube and the demo forum soon 🙂

[loud briar]

I may be misunderstanding capability here (how compatible are engine version-mismatched modules? Assuming it varies by version diff) but I think it'd be useful to be able to filter daggerverse.dev by engine version to ensure I'm picking up modules that work on my current engine. The cards (nice improvement over the rows btw!) have module version, but that's... Not that useful initially?

[sharp marsh]

good feedback @loud briar! Thx for sharing, cc @fallen temple

[fallen temple]

Hello! I recorded your feedback. 🙂 @fossil pine recommended showing the engine version on the description page.

Regarding filtering on engine versions, I see that as a possible stop gap because of modules being in experimental phase. The engine is rapidly changing. However, once zenith/modules are official, those changes should become infrequent.

That said, Mark brought up the point of enhancements in the future. @wraith niche what are your thoughts about this topic? I'm wary of causing version fragmentation, where people have to constantly check if their engine can use a module.

[fossil pine]

To be clear: I had the most issues with new features, not breaking changes in the engine (eg. new function added to dagger, new function used in module which obviously breaks on an older engine)

That's why I emphasized at least a minimum version check which can easily be done, now that the engine version is in dagger.json.

[stoic knot]

I'm going to be talking Dagger Cloud UI and what a paid account might look like with my main client a little later today.

Can anyone from the Dagger team hop on a quick call in the next couple of hours?
(so I can be current & accurate)

[sharp marsh]

cc @slender charm

[slender charm]

Thanks @sharp marsh @stoic knot ill dm you

[winter linden]

Hi @everyone! Dagger 0.10 is out, and it includes a major new feature: Dagger Functions. For those of you who know about "project zenith", this is it - we finally shipped it 🙂

Dagger Functions are a core feature of the Dagger Engine which we are expanding and making available to all developers.

Dagger already let you distill complex CI into clean code. Now you can encapsulate that code into containerized functions, which can be shared, reused, and composed into higher-level pipelines.

• Dagger Functions can call other Dagger Functions, across languages.
• Dagger Functions can be packaged into reusable modules, and shared using git (no other infrastructure needed)
• You can call Dagger Function directly from the CLI

This is a major improvement to the Dagger platform - our biggest since launching multi-language support.

Thank you to all the Daggernauts who spent so much time on #daggernauts, testing and sharing feedback.

You can read more here: https://dagger.io/blog/introducing-dagger-functions

[sand gorge]

Well done, team!

[inland coral]

And folks, we also just refreshed the Home page of Dagger.io We hope you like it, are always open to feedback, and promise to keep expanding and iterating our website and docs quickly based on your input! https://dagger.io

[winter linden]

@fossil pine include/exclude didn't make the cut, but is next up, we'll cut a 0.10.1 for it

[stuck wyvern]

Congrats!

Been heads down on another project recently (buildkit frontend), so haven't been following things as closely.

[winter linden]

Thanks!

The TLDR: port that buildkit frontend to a Dagger Function, and thank me later 😛

[shadow socket]

anyone seeing issues with 0.10.0 not liking certs when connecting to docker? I have a trivial python script just setting up a Python 3.11 slim container and it fails.

Here is the script:

import sys
import anyio
import dagger

async def test():
async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
python = (
client.container()
# pull container
.from_("python:3.11-slim-bullseye")
# get Python version
.with_exec(["python", "-V"])
)

    # execute
    version = await python.stdout()

print(f"Hello from Dagger and {version}")

anyio.run(test)

And it gives me the following output:

[1.79s] ERROR python main.py
gql.transport.exceptions.TransportQueryError: {'message': 'failed to do request: Head "https://registry-1.docker.io/v2┃ /library/python/manifests/3.11-slim-bullseye": tls: failed to verify certificate: x509: certificate signed by unknown ┃ authority', 'path': ['container', 'from']}

I can do a docker pull for that image and everything is fine.

see this on Windows and WSL

[swift inlet]

Tls

[stray steeple]

Love the changes to the site and the updated docs!! Dagger feels more magical and special every day but I gotta say the updated docs are really starting to connect inspiration to execution effectively 🚀 Well done!

[wispy steppe]

Some feedback on the site:

1. I'm not sure how Framer works, but is it possible to predefine the size of the video/container of the video? That way, there's no/less layout shift
2. These buttons lead to v0.9 of the documentation:

• Get Started, first section
• Try Dagger Engine

Otherwise it looks fantastic, great job!

[fallen temple]

@inland coral @chilly arch @mint axle ^^

[vestal apex]

So we can add comments for the flag descriptions.
Can we also add custom tags similar to +optional e.g. for the possible values?

func (m *Dag) GrepDir(ctx context.Context,
    // directory to grep
    directoryArg *Directory,
    // pattern to search for
    pattern string) (string, error) {

"Args": [
  {
    "Name": "directoryArg",
    "Description": "directory to grep",
    "TypeDef": {
      "Kind": "OBJECT_KIND",
      "Optional": false,
      "AsObject": {
        "Name": "Directory",
        "Functions": null,
        "Fields": null,
        "Constructor": null,
        "SourceModuleName": ""
      },
      "AsInterface": null,
      "AsInput": null,
      "AsList": null
    },
    "DefaultValue": ""
  },
  {
    "Name": "pattern",
    "Description": "pattern to search for",
    "TypeDef": {
      "Kind": "STRING_KIND",
      "Optional": false,
      "AsObject": null,
      "AsInterface": null,
      "AsInput": null,
      "AsList": null
    },
    "DefaultValue": ""
  }
]`

[rain oriole]

Yes, you can use +default: https://docs.dagger.io/developer-guide/go/264203/functions#default-arguments

[violet bolt]

congratulations on the 0.10 release 🥳

now, before i can actually give things a proper spin i tried to create an initial module, but i need private module access to actually make this worth while .. has that been resolved, or is this an ongoing thing?

https://github.com/dagger/dagger/issues/6113

I saw this issue as well https://github.com/dagger/dagger/issues/6762

are the flags for dagger / dagger call documented anywhere?

[vestal apex]

Functions | Dagger

[fresh depot]

Congrats on the launch of 0.10/zenith ! It is really impressive. It happens at the same time with me having to modernize some old ci soon, so i might bring dagger into it !

[hot yarrow]

It's great and so interesting, thanks the #Dagger team for this amazing work 🤩 🤩

[inland pivot]

Congrats on the release of Zenith!

[slender star]

✨ Zenith: Allow installing modules from ...

[sand gorge]

New homepage is lookin sexxxyyyyy

[winter linden]

Thank you! And thanks for the great Twitter shout out 🙂

[chilly arch]

Just over 4 months ago we had our first (internal) hackathon to test the brand new alpha implementation of Dagger Functions and modules. Check out this video to see how it went. We had some much fun building our own functions and modules, I think you will love building your own as well!

https://youtu.be/qdrkZbgY1hY

If you have a module that you'd like to show off, please DM me, and I'll be happy to add you to one of our upcoming community calls!

[winter linden]

@pseudo stream superstar 😎

That video is hilarious

[cunning jolt]

Just watched this it's awesome to see the team at work. Thank you for recording and sharing it!

[deep temple]

Hi folks. i think i missed out a lot. is there a demo of the new tui ? is the new tui built on top of progrock or its a new direction altogether ?

[glad jolt]

new TUI

[blazing pagoda]

Hello folks, congrats on the launch of 0.10.

As I started building CI pipelines with Dagger last week following the documentation, I’m a bit confused now, as the documentation does not mention the previous way of building a pipeline. Is the "run" command considered the old way now? Should I start migrating to the new functions/module approach?

[deep temple]

Is the dagger sdk (the old non zenith in go) going to be phased out now that zenith is released ? or it will still be continued ?

[winter linden]

No. It's the same SDK, just with more capabilities.

[neat valley]

sooo my first docker hub image is almost 11 years old now 😄

hoping to say the same thing about my first modular dagger function in 2035!

[slender charm]

love this 😄

[stuck wyvern]

Co-worker needed a pre-release build of our artifacts which we don't currently build.
Really nice to tell them go install <pkg> and run it with the repo/commit you want it to build.

Also really handy to have static assets compiled into the binary which we then convert to a dagger filesystem.

[candid crag]

(っ◔◡◔)っ  dagger -m github.com/shykes/daggerverse/hello@v0.1.2 call hello
✘ load call ERROR [2.41s]
├ [2.41s] loading module
┃ Error: failed to get module config: resolve git ref: input: resolve: git: commit: commit: failed to fetch remote https://github.com/shykes/daggerverse:
┃ t error: exit status 128
┃ stderr:
┃ fatal: couldn't find remote ref v0.1.2
┃
┃ input: resolve: git: commit: commit: failed to fetch remote https://github.com/shykes/daggerverse: git error: exit status 128
┃ stderr:
┃ fatal: couldn't find remote ref v0.1.2
✘ commit ERROR [2.40s]
▶ commit
  ✘ git://github.com/shykes/daggerverse#v0.1.2 ERROR [0.68s]
  ┃ Initialized empty Git repository in /var/lib/dagger/runc-overlayfs/snapshots/snapshots/1/fs/
  ┃ 3d608cb5e6b4b18036a471400bc4e6c753f229d7        refs/tags/hello/v0.1.2
  ┃ 3d608cb5e6b4b18036a471400bc4e6c753f229d7        refs/tags/wolfi/v0.1.2
  ┃ fatal: couldn't find remote ref v0.1.2
• Engine: b0f1a4d55cc0 (version v0.9.7)
⧗ 1m18.5s ✔ 7 ✘ 3

having this issue when I run dagger -m github.com/shykes/daggerverse/hello@v0.1.2 call hello

according to the doc here: https://docs.dagger.io/quickstart/562821/hello

[winter linden]

If you want a glimpse at the future of CI-as-code... I highly recommend watching this 😇 https://discord.com/channels/707636530424053791/1214729466920706088

[candid crag]

in this example here https://docs.dagger.io/user-guide/ci/734201-gitlab people is going to call module github.com/kpenfound/dagger-modules/golang@v0.1.5 but there is more recent version like v0.1.8, which is the latest at the time of writing this, so is there any plans for calling the latest versions of the modules? or are we the responsibilities of keeping track of the changes between modules?

[cosmic moss]

I've been asked several times if it's possible to build android & iOS apps in dagger. For android there's no problem, but for iOS I'm not sure if it's possible to use tart https://github.com/cirruslabs/tart or another solution with dagger, do you have an idea?

[candid crag]

@warm temple https://github.com/kpenfound/dagger-modules/blob/8d662e001caf8c16253226d0d456f2f0f374f009/golang/main.go#L77 there is no way to build go binary with CGO_ENABLED=0 in this module, I built my go app with this module, that's the result:

/ # file /usr/local/bin/simple-golang-app
/usr/local/bin/simple-golang-app: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, Go BuildID=QJ4IC8LxFNY9zbT9evwY/P8X58bCQJD67GIcvpbLT/Fhh6N4nsTEl0Ls3bbnmR/6SiDUgHuCneJCH0h1H8Z, with debug_info, not stripped
base := dag.Container().From("alpine:latest").WithExec([]string{"apk", "add", "libc6-compat"})

[green leaf]

Don’t you need to add a WithEnv call to set CGO_ENABLED=0?

[sharp marsh]

GitHub - cirruslabs/tart: macOS and Linu...

[warm temple]

I'll add some options 😄 as for the previous question, if you omit the version from the dagger call -m it'll use the latest, but for the guide 0.1.5 was the version tested

[candid crag]

File could be added to that list here: https://docs.dagger.io/quickstart/292472/arguments#pass-arguments-to-a-function

[candid crag]

how to create a *Directory from a string in Go SDK in functions or how to get the name of the directory?

[winter linden]

String <-> Directory

[chilly arch]

Instanbul Daggernauts - @candid crag is hosting a Dagger meetup tomorrow night with the Istanbul Coders group. You can RSVP here: https://kommunity.com/istanbulcoders/events/pipeline-as-code-with-dagger-1ae30596

[chilly arch]

Looking forward to seeing everyone at the Dagger Community call tomorrow! We have a great line-up of demos by
@rigid pebble @solar raptor @heavy gazelle @sage plank and @shy yacht

If you haven't already, you can register here:
https://dagger-io.zoom.us/webinar/register/8517071155537/WN_j_xRpP1ASb6xrM4AiRKRnA

[winter linden]

https://tenor.com/view/fatbot-futurama-oh-boy-gif-22905414

[patent slate]

That list of topics is impressive. 🙂 Looks like a mini-daggerCon. 😛

[blissful stratus]

Hey there. Will the 'old' SDKs used for dagger run remain supported? If I gut it correctly, all their documentations just vanished - in 0.10 its all about writing functions/modules now.

[clever mesa]

help !!!

Client panic when with Pipeline. (v0.10.1) v0.10.0 works well

the private client not pass to new one.

https://github.com/dagger/dagger/blob/v0.10.0/sdk/go/dagger.gen.go#L5745-L5763

https://github.com/dagger/dagger/issues/6850

[narrow nymph]

aha, thanks for the report 🎉 looks like we were missing a test case, working on a fix right now

[narrow nymph]

I'm guessing you're also using .GraphQLClient or manually calling .Do?

[eager skiff]

That's a good question. I've used Dagger embedded in an internal CLI so users don't really have to know about it. Is there any similar recommended pattern with Dagger Modules/Functions or are users supposed to always dagger call?

[winter linden]

Both models will co-exist @blissful stratus @eager skiff . The missing link is the ability to load modules from a custom CLI (right now you can only call core functions).

The idea is that you can can call any function (core or external) from 1) the dagger CLI, 2) other functions, or 3) a custom client.

Custom clients will be more niche, but still a valid, supported use case.

As for the 0.9 docs, we're porting them over as fast as we can (cc @heavy karma @wraith niche @heavy gazelle )

[clever mesa]

manually call .Do.

I use it to get full image info.

https://github.com/octohelm/wagon/blob/main/pkg/engine/plan/task/core/image.go#L114

[eager skiff]

Nice, that's the answer I was hoping for. Is it being worked already somewhere where we can follow/contribute? When you say "custom client" you mean through the Dagger SDK or just doing GraphQL calls to the engine?

[winter linden]

by "custom client" I mean any code you wrote using 0.9 SDKs or raw GraphQL. Ultimately that code ends up running in a tool you wrote.

[sage plank]

👋🏼 from sunny Seville, Spain 🙂

[chilly arch]

Community Call starting now! Join us here: https://dagger-io.zoom.us/webinar/register/8517071155537/WN_j_xRpP1ASb6xrM4AiRKRnA

[fallen temple]

Hello from Durham, North Carolina

[cunning jolt]

Hi from Fort Worth, TX

[patent slate]

Hello from Germany!

[last pine]

Hi from Lyon, France!

[fallen temple]

Luke, be sure to turn off noise cancellation in Zoom if you want the airhorn to work. 😉

[winter linden]

Good morning from San Francisco! Love the music

[solar raptor]

greetings from bristol 😄

[winter linden]

Be advised that can't airhorn your own demo though , that's against the rules

[sharp marsh]

👋 from 🇺🇾

[solar raptor]

rofl, noted 🫡

[stray steeple]

Hello from Iowa!

[slender star]

Hello from Portland, Oregon, USA!

[sweet locust]

hello from Vancouver, but i'm from argentina 🇦🇷

[fallen temple]

Vancouver, BC or Washington state? 😄

[sweet locust]

BC!

[eager skiff]

Hello from Portugal 👋

[chilly arch]

Hosting from Vancouver, WA 🙂

[slender charm]

This is awesome @heavy gazelle - are you using some library underneath or just using raw HTTP.

I am inspired to imagine a https://knock.app/ module 🙂

[shy yacht]

Hello from rainy Poland 🙂

[cunning jolt]

Suggestion: add MSTeams notifications 🙂

[slender charm]

Yeah that is the beauty of Knock is once you integrate once you can notify all the things

[heavy gazelle]

That moustache worked: I almost didn't recognise you. Almost. Great to see you here 👋

[slender charm]

Also https://github.com/novuhq/novu -- might be even better since its FOSS

[sage plank]

I love that

[slender charm]

"I've been writing dagger modules since before you were born"

[winter linden]

@rigid pebble could image-updater also have a function that implements a push model instead of pull, so that I can pass an arbitrary Container object, instead of watching a remote registry?

[heavy gazelle]

Thank you! Knock looks like a good paid alternative. Looking forward to the module 💪

[patent slate]

I believe pushing anything into k8s is an anti-pattern.

[winter linden]

I guess my question was going to a place that is maybe too Dagger-native for an existing Argo/Kube stack 😛 @rigid pebble

[rigid pebble]

That's fair, and IMO interesting to explore. In this scenario you would have a component deployed on the k8s cluster that uses Dagger's core API and performs the necessary publishing steps to make it available within the cluster, right?

[winter linden]

ie. the Argo/Kube model assumes the pipeline already pushed the right image to a registry. But my pipeline being Dagger, I consider that intermediary registry to be a "dumb pipe" for injecting images into the cluster. So I want to abstract it away.

Obviously under the hood, since Kub/Argo doesn't have a registry that you can push to (note: it really should...), probably my deployment function needs to supply one. But it could be anywhere, I don't really want to know about it - it's just a "push" adapter on top of the cluster's "pull" model

[winter linden]

Doesn't need to run on the cluster necessarily, could be anywhere, but yeah

Side note: the Kubernetes prod ecosystem is so complex and convoluted, it didn't need to be this way...

[cunning jolt]

Pushing to a registry is a requirement at most orgs for audit and compliance reasons.

[patent slate]

ArgoCD's only job is to make sure set config is a reality in the cluster via reconciliation. It does nothing else really AFAIK.

[winter linden]

Sure. Doesn't mean I need to micro-manage the details of pushing to it in my high-level pipeline logic.

And most importantly: the registry should be tied to the production cluster deploying from it. It's an appendix of the kub cluster.

[chilly arch]

For those on the call who may not be familiar with Dagger Functions and Modules yet (what Miguel is showing right now), check out this post - https://dagger.io/blog/introducing-dagger-functions

Here is the module that Miguel is demoing right now: https://github.com/chainloop-dev/chainloop/tree/00d64321852ace5804ff3cbf1e478acd0cf91460/extras/dagger

[winter linden]

@sage plank love the demo 🙂 I have an idea for a contribution to your module, I think you can return a custom Attestation type instead of a string ID, and make your module even more seamless to use. Also we can add a download function that returns the artifact (wraps chainloop download).

Reallyi cool

[sweet locust]

very cool demo

[inner tide]

Does this get data from BuildKit under the hood about commands that are run etc?

[slender charm]

Thanks for the great demo @sage plank !

[sage plank]

I want to know more, what kind of metadata do (do not) expect Chainloop to get?

[winter linden]

Oh wow @solar raptor, this includes all necessary infra provisioning? Zero-to-deployed?

[cunning jolt]

Replacement for Terraform? 🙂

[slender charm]

This is dope @solar raptor !

[winter linden]

I'm curious about the relationship to Terraform and Pulumi

@solar raptor seems very reasonable to me!

[cunning jolt]

You could use TF directly in dagger

[warm temple]

Or AWS CDK 😛

[sweet locust]

had to leave, have a nice day! it was great

[cunning jolt]

That''s basically AWS CDK

[chilly arch]

Thanks for joining!

[solar raptor]

should we wrap AWS CDK?

[cunning jolt]

possibly, and provide a better UX/DX

[solar raptor]

does gcloud have an equivalent?

[patent slate]

Thanks for the demos!! Interesting and exciting stuff for sure. 🙂

[winter linden]

My guess is that wrapping TF (or Pulumi) is a better starting point, you get more use cases per lines of code written 🙂

Plus we already have the base modules as @solar raptor observed

[slender charm]

Thanks to everyone who demoed and a huge thanks to @chilly arch for coordinating and MCing, this was one of the best ones yet.

[warm temple]

Terraform 😂

[winter linden]

Yeah this was great, and a lot of fun. The only bad part of my experience, is that my kids stole and lost my damn airhorn.

[sage plank]

Thank you for having us, Dagger rocks

[inner tide]

I was more thinking that BuildKit generates SLSA Provenance and wondering if chainloop can leverage that?

[solar raptor]

here's a jungle remix of University Challenge to make you feel better https://twitter.com/i/status/1745196642335088942

I believe it contains some airhorn

[sage plank]

@Miguel Martinez Trivino love the demo

[sage plank]

I see, no we don't support "importing" attestations from other sources just yet, but we have plans on changing that. So the Chainloop attestation process can bundle multiple, existing intoto files. Let me make sure we have an issue for it. thanks!

[chilly arch]

For those going to KubeCon, check out all the Dagger-related events here and make sure to RSVP to save your spot for some of the external ones: https://dagger.io/blog/dagger-heads-to-paris

Can't wait to see you there!

[chilly arch]

If you missed the community call from today, you can check out the recording here: https://dagger-io.zoom.us/rec/share/9IKWMmb0lenkdwrAv_vL5R1AS6ISqBKZLnfdSbjR7d2tLsOZBDPZxPWrl7yuB42F.rppqnKAjJSVKWw6c

Demo snippets will be added to youtube soon!

[wanton portal]

Any chance I could get access to the demo used in this video?
https://www.youtube.com/watch?v=mv5oKRodkrM

I looked at the repo but it doesn't have any dagger stuff inside.

Or if anyone has other good examples for running a ci with dagger modules (preferably Node SDK), that would be much appreciated 🙂
Thank you

[sharp marsh]

hey! sure! The code is currently in a branch. You can find it here: https://github.com/kpenfound/example-voting-app/tree/daggerize

[wanton portal]

Thank you 🙂

[wanton portal]

Is there such a thing in dagger, for fetching files from remote machines (like ansible's fetch module), or how would I go about doing the same using dagger?

[sharp marsh]

there's nothing native but that's actually a good idea for a module. If you're already using ansible, you can just wrap your current ansible config in Dagger and use that same ansible fetch module to retrieve what you need

[winter linden]

This is in 40 minutes! https://twitter.com/chainguard_dev/status/1765744376590807384

Chainguard ⛓️ (@chainguard_dev) on X

🌠Mark your calendars: @solomonstre will show
@lorenc_dan how to use @dagger_io to do things better than Spongebob & Patrick below 🥴
✨March 8th at 3:30pm ET
https://t.co/JMsnlb3KN5

Twitter · Mar 7, 2024, 2:20 PM

[chilly arch]

If you are new to Dagger or want to check out the new way to use Dagger with Dagger Functions, check out @warm temple 's latest tutorial videos.

**He'll give you a quick overview in 5 minutes or less! **

Go - https://youtu.be/_6Tazl8HSQs
Typescript - https://youtu.be/dFMY8ZAoAdg
Python - https://youtu.be/Noh9cHLV318

[slender charm]

If you are new to Dagger or want to

[winter linden]

I thought I'd share my little experiment of the day... A Dagger Function that generates terminal recording gifs for you. It's very fun 🙂

dagger call -m github.com/shykes/daggerverse/termcast@v0.2.0 \
 demo gif export --path=./demo.gif

Or play it live in your terminal:

dagger call -m github.com/shykes/daggerverse/termcast@v0.2.0 \
 demo play

[winter linden]

I added a function to imagine a session. You'll need an OpenAI token.

dagger call -m github.com/shykes/daggerverse/termcast@v0.2.1 \
 --key env:OPENAI_API_KEY \
 imagine --prompt 'a system administrator troubleshoots an old red hat server at 4am' \
 gif \
 export --path=4am.gif

[slender kernel]

Hey guys, I'm new to dagger so I have some questions.

My first one is that I was wondering if dagger runs all calls remotely or locally. I can't fathom it running locally while making grpc calls and the execution times are just too long to run locally.

So I was wondering do all of these run on daggers remote servers that are behind the scenes.

It could be connecting to my local docker network and that itself could be slow.

[sharp marsh]

My first one is that I was wondering if dagger runs all calls remotely or locally. I can't fathom it running locally while making grpc calls and the execution times are just too long to run locally.

All dagger calls happen locally in your own computer.

[slender kernel]

Guess it's my own docker network that's slow

Also all the dag.Containers are probably pulling containers each time.

Thank you

[winter linden]

Slow run

[timber crater]

I would like recommendations for tech podcasts in French (Canada, Africa, EU, ok).

I like have French in the background and I'm curious about the argot of dev ops people (like, when to use the English word and when to modify a French word).

[cunning jolt]

This is cool! Would be nice if it had some kind of terminal colors, or emulated your current terminal setup.

[winter linden]

I think colors are supported. Will try.
What do you have in mind with "current terminal setup"?

(I also want to add support for running real commands in a real container, and making it look like it's typed by a human (random pauses etc)

[winter linden]

@cunning jolt it works 🙂

dagger call -m github.com/shykes/daggerverse/termcast --key env:OPENAI_API_KEY \
 imagine --prompt 'A unix shell. I type a command that is about saying hi to someone called Nipuna. The output of the command is full of color, like a rainbow. It has ascii art, too.' \
 gif \
 export --path=./color.gif

There's GOT to be a way we can use this to generate cool animated examples in the Dagger docs, right? 😇 cc @heavy karma

[cunning jolt]

I dont' necessarily care too much about my current shell/terminal config (eg: my custom prompt etc.). However, I was thinking the use case of running a dagger function. I believe this will be a great way to demo daggerverse functions. Can you provide an example of how that would look like?

[slender kernel]

I hope I did you guys proud. This is an awesome tool! https://youtu.be/0I6-gdHiOJY

[chilly arch]

Thanks for sharing! Quick intros are always helpful for the community 🙏

[slender kernel]

I'll make a more in-depth video later, stay tuned!

[deep rampart]

Hello, assuming the same pipeline starts multiple times at the same time.

Eg
go run main.go &
go run main.go &
How does dagger engine handles

1. Git clone
2. Cache volumes(shared mode)
   Reason I’m asking is that I will firing multiple pipelines where only the last step changes on Jenkins nodes. So there are chances that some will be on the same.
   Do I have to worry about corruption, will I get the caching benefits when multple run on the same node?

[winter linden]

concurrency will work fine

as long as you don't mess with the internal configuration of the engine container, and in particular which internal state directory is mounted where.

One engine can handle multiple clients concurrently.

Multiple engines cannot share the same state directory.

[deep rampart]

Nah these will be left as is. Not need to be too advanced on that

If 2 pipelines start at the same time, cloning the same repo (on the same engine).
Will the second wait for the first clone to finish and use the cache?

[winter linden]

If you are using the same engine (which should happen by default) then yes. It's mostly buildkit making that work under the hood. Buildkit generally has good deduplication.

It's possible that if you're not using the same session dagger will not dedup things like looking up the remote tag. You could wrap your go runs into the same session with something like dagger run sh -c "go run a/main.go & go run b/main.go & wait"

But that's just an additional optimization, not required.

[deep rampart]

Okay that should be good too.
Because I'll be running ~40 runs at the same time, on 4 nodes. so each node will have ~10 runs each.
I was thinking of running 1 run on each so cache "hydrates" and then run the 9 left, so it can benefit from the cache
But if buildkit handles all that, I'd like to avoid doing "math" on jenkins to do the above

(each run is, in simplified steps, git clone, npm install, npm configure, npm build)
first 2 steps are identical, second steps change.

[loud briar]

Is there a known issue with the terminal method on dagger.Container in 0.10.1? I'm getting Error: response from query: input: container.from.withWorkdir.withDirectory.withExec.withExec.withWorkdir.withExec.terminal panic while resolving Container.terminal: runtime error: invalid memory address or nil pointer dereference. Tried a few terminals (warp, kitty, alacritty) since I first thought it was warp being silly but same error on all

[winter linden]

Why not combine them into a single run that calls out to the others, in code?

Yes you definitely don't need to worry about "caching math", just throw it all at dagger and it will figure it out

[deep rampart]

Like, having a go run main.go and inside there create multiple dagger clients to connect to all the different dagger engines on the different nodes?

[winter linden]

Your wish is my command 🫡

dagger call -m github.com/shykes/daggerverse/termcast \
 --height=48 --width=160 \
 print --data 'You can now mix simulated output, like this line....' \
 enter \
 exec --cmd 'echo "...and real ones like this:"' \
 exec --cmd 'uname -a' \
 exec --cmd 'ls -l' \
 exec --cmd 'echo "state is persisted between commands" > message.txt' \
 exec --cmd 'cat -e message.txt' \
 print --data 'AND you can run dagger commands out of the box...' enter \
 exec --delay=1500 --cmd 'dagger --progress=tty call -m github.com/shykes/daggerverse/hello hello --greeting=bonjour --name=Nipuna' \
 wait --ms=1000 \
 gif \
 export --path=tada.gif

You can customize the container with --container at the root of the module

Note: some artifacts in the TUI output when calling dagger-in-dagger... That's because I don't capture the stream as it's being written by the command, but just redirect it entirely, and grab it at the end. So timing information gets lost. Will add real-time capture on my next chunk of free time 🙂

[cunning jolt]

This is fantastic! Now you can use this to auto generate a module-dev provided dagger call simulation in the daggerverse readme 🙂

[winter linden]

That's what I'm hoping! This was just a weekend project but became actually useful surprisingly fast!

I think it bodes well for the future of the Daggerverse. I think we're about to see some seriously cool new functions appear in the near future

I would NEVER have bothered to try developing this without the Dagger Functions DX.. Too much work!

[cunning jolt]

I agree! possibilities are endless

[tired moth]

Do we need to write dagger code in ts or we can write in js if we are using dagger typescript SDK?

[sharp marsh]

if you're using modules, only typescript is supported for now. If you're not using modules and you're building your own tool on top of Dagger, you can use the Node SDK and write javascript directly. Ref: https://archive.docs.dagger.io/0.9/sdk/nodejs/783645/get-started

[winter linden]

Better to avoid JS altogether, if you can.

[tired moth]

why? due to type checking?

[tired moth]

does node sdk actively maintained?

[winter linden]

Yes it's actively maintained, but writing Dagger Functions requires a type system, which Typescript provides but not Javascript. So it's inevitable that vanilla JS will get less and less usage by the Dagger community. A good opportunity to make the switch.

All our SDKs are auto-generated from the GraphQL API schema, so there's no immediate risk of JS falling behind the others in client functionality.

[tired moth]

we dont use typescript in our org, due to it's compilation dependency

[tired moth]

ok does typescript sdk understands ts files directly or after compiling to js?

[tired moth]

then why can't we call dagger function from JS?

[tired moth]

if all sdk are auto generated, then we should add node js docs also, here https://docs.dagger.io/developer-guide

[tired moth]

and though this packages it can be used with deno also as such deno has first class compatibility with npm modules

[rain oriole]

Dagger Functions with JS

[mystic mantle]

Hi 👋

I checked the page for dagger cloud, but the slides were kind of buggy. Some didn't pass automatically, some didn't play. (I'm linux + firefox)

[tulip grove]

hey guys, which is the python channel? I'm having a hard time looking for it =/

[chilly arch]

Great question! Looks like we are missing that one! I'll create it now, and add you 🙂

[tulip grove]

ohhh, thank you