#github-feed

1 messages · Page 20 of 1

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12108: ✨Support for SBOMs/Provenance Attestations

Ok so, I had some ideas related to this :tada: This is my proposal for what we should do with attestations in dagger - thoughts welcome :heart:

[!NOTE]

I think we should pretty much follow the in-toto spec: https://github.com/in-toto/attestation/blob/main/spec/README.md.

Buildkit implements attestations that follow this pretty closely, however, today it only produces statements (that contain predicates). We'd also want to upstream/wait for signing to be done, so we could use...

#
sagikazarmark
[dagger] New discussion #12109: ✨ Add Merge function to Directory

What are you trying to do?

When working with build results and generating final build artifacts (packages), I often need to create directories and merge them.

In theory, this works using the following:

dag.Directory().WithDirectory("", dag.Directory())

I'm not entirely sure it always merges directories though.

In any case, a function called Merge would reflect better what it does (similarly to Diff)

Why is this important to you?

No response

How are ...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12110: Global install

It would be great if dagger install always worked, no matter the current working directory:

If... Then... Status
Workdir is inside a dagger module Install into the current module Implemented ✅
Workdir is NOT inside a dagger module Install into the home directory as if it were a module Not yet implemented ❌

I propose implementing the second row: when calling dagger install outside a module, use the home directory as a pseudo-module instead....

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger/dagger] New comment on discussion #12110: Global install

Rather than putting it in $HOME/dagger.json and rely on find up, I think it's better to look for it in a "global" place like:

~/.config/dagger/dagger.json

Other than not polluting the home dir, I'm thinking what happens if you add a "source": ".". This would upload the entire $HOME directory to the engine. We can also forbid sources for this module.

One other feature I'm dying to have in this global module is global excludes, like git!

#
shykes
[dagger/dagger] New comment on discussion #12110: Global install

I've been trying to work around this by dropping a dagger.json at the root of my home directory, but I'm having trouble:

  1. Error: failed to generate code: input: blob.diff.export resolve: failed to export: error from receiver: open /Users/shykes/.Trash: operation not permitted
  2. Contents of my dagger.json gets randomly wiped by the Dagger CLI

Is there a way to have a reliable workaround?

#
levlaz
[dagger] New discussion #12111: ✨ Create Dockerfile from Dagger

What are you trying to do?

Dagger is really good at reading and executing Dockerfiles, but it would be really cool if you could generate a Dockerfile from Dagger as well.

Why is this important to you?

A few users have mentioned that rewriting their Dockerfile in Dagger (which is a good thing) has some downstream consequences. Most critically it breaks existing docker-compose setups.

How are you currently working around this?

No great workaround.

vivid lintelBOT
ornate vigilBOT
#
wingyplus
[dagger/dagger] New comment on discussion #12111: ✨ Create Dockerfile from Dagger

I'm not sure how feasible this is. This would only really be possible for the most basic of dagger pipelines, but it's kind of unclear how this might work with modules/etc.

If we did decide to do this, the first step would be to remove all of our custom low-level buildkit parts, and either upstream these, or work out how to do without them. We have quite a lot of these, which allow us to do things that the buildkit runtime wouldn't normally let us do (custom sources, our own local ...

#
jedevc
[dagger/dagger] New comment on discussion #12111: ✨ Create Dockerfile from Dagger

I'm not sure how feasible this is. This would only really be possible for the most basic of dagger pipelines, but it's kind of unclear how this might work with modules/etc.

If we did decide to do this, the first step would be to remove all of our custom low-level buildkit parts, and either upstream these, or work out how to do without them. We have quite a lot of these, which allow us to do things that the buildkit runtime wouldn't normally let us do (custom sources, our own local direct...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
joeshaw
[dagger] New discussion #12114: ✨ Explicit dependencies between containers without an artifact

What are you trying to do?

I would like a way to express an explicit dependency in the DAG between two containers without necessarily passing an artifact between them. Today, containers can depend on services (WithServiceBinding) and they can depend on other containers with artifacts (create a file in container 1, then WithFile("/tmp/foo.txt", c1.File("/tmp/foo.txt")) in container 2) -- it'd be nice to not require the file.

Why is this important to you?

I have an integra...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
vikram-dagger
[dagger] New discussion #12116: Review the PHP example on the api/http page

What is the issue?

There is a language-native PHP example on https://docs.dagger.io/api/http#language-native-http-clients, which demonstrates how to interact with Dagger using a PHP HTTP client.

This example was created before the PHP SDK existed.

The question is whether to retain or remove it.

Argument for retain: it provides an example of communicating with Dagger using a raw GraphQL client. This could be useful for users who don't want to use the PHP SDK, who don't want to use mod...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12129: core: File.search()

Add a core function to search the contents of a file.

extend type File {
  "Search the contents of the file"
  search(
    "The pattern to match the contents. Follows the Extended Regular Expression format"
    pattern: String!,
    "Number of matches to include"
    n: Int,
  ): [Match!]!
}

type Match {
  line: Int!
  column: Int!
  contents: String!
}
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12130: Enable nesting by default

Dagger natively supports nesting (dagger in dagger). At the moment this is an opt-in feature (Container.withExec(experimentalPrivilegedNesting: true). This is as a precaution while we finalized the feature and made sure it's properly sandboxed.

Are we now able to say "mission accomplished" and turn it on by default? If not, what's the path to getting there?

#
sipsma
[dagger/dagger] New comment on discussion #12130: Enable nesting by default

Right now when an exec has nesting enabled the cache key for it includes the engine version, so it's cache invalidated when the engine version changes (i.e. an upgrade or a rebuild w/ changes when doing local dev on the engine).

IIRC the main motivation for having this caching behavior was our own integ tests (which make very heavy use of nested execs) and wanting the tests to be invalidated when we modified the code in the engine API. However, we could accomplish that for our own tests in a...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
eunomie
[dagger/dagger] New comment on discussion #12131: ✨ Publish java SDK to mvn central

@benkeil Just to let you know we are working on publishing the SDK to maven central.

Regarding Kotlin, this is working well (not the modules for now, just integration in Kotlin code).
Once libraries will be published we will also add some docs to use them with kotlin.

For instance this very basic code:

package com.mycompany.app

import io.dagger.client.Container
import io.dagger.client.Dagger.dag
import io.dagger.client.Directory
import io.dagger.client.ReturnType
import java.uti...
#
c33s
[dagger] New discussion #12132: ✨ allow to set the container runtime via config/env var

What are you trying to do?

according the guide to use podman you should symlink podman to docker.

it would be much better to allow to configure the "runtime" via config and/or environment variable. something like DAGGER_RUNTIME=/usr/bin/docker or DAGGER_RUNTIME=/usr/bin/podman

(btw the issue template is quite uncool for keyboard first persons if you have to jump between the three blocks "what are you trying to do", "why is this...

vivid lintelBOT
ornate vigilBOT
#
gerhard
[dagger/dagger] New comment on discussion #12132: ✨ allow to set the container runtime via config/env var

Thank your for all the context @c33s.

While I don't use Podman myself, I am wondering whether you could leverage DOCKER_HOST to point to e.g. unix:///run/user/1000/podman/podman.sock

With the help of direnv, the projects that need this combo could use e.g. export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock . Using this config, Dagger would still use the docker CLI default for managing the Engine, but the docker CLI would be connecting to Podman instead of Docker...

#
gjung56
[dagger] New discussion #12133: ✨ core function to get a directory or file as tarball

What are you trying to do?

It would be nice to have a core function to get a tarball from a directory or a file

Methods api examples :
Directory.AsTarball()
File.AsTarball()

And eventually, the reverse functions, to get a directory from a tarball.

Why is this important to you?

We use a lot tarballs in our ci/cd pipelines and that make a lot of the same boilerplate to get what we need.

How are you currently working around this?

I use a basic alpine container, mount t...

vivid lintelBOT
ornate vigilBOT
#
beingminimal
[dagger] New discussion #12134: ✨ Support for Distributed Builds to Leverage More Resources

[!NOTE]
Eliminating On-Prem Kubernetes Needs
This feature could potentially eliminate the need for Kubernetes (K8s) on-premises for companies looking to scale their DevOps engineering. By leveraging Dagger's distributed build system, teams could manage and scale their CI/CD pipelines more easily, reducing the overhead and complexity involved in maintaining an on-prem Kubernetes setup for build orchestration.

What are you trying to do?

We are looking to add support for distri...

vivid lintelBOT
ornate vigilBOT
#
cwlbraa
[dagger] New discussion #12135: Dagger Shell: does it become the documented default interface? how far does it go?

For context, I'm one of those devs, and I think there are many of us out there, that almost never uses interactive repls or shells outside of my highly-customized zsh and the occaisonal database query webgui frontend.

I see the argument that folks are gonna write wrapper scripts for call regardless of SDK programmability, but is the idea here to build a featureful enough dagger shell that folks will be writing and committing dagger shell scripts as wrappers? a featureful enough shell th...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12136: Session config file

This is something that has come up quite a bit recently, mostly in the context of https://github.com/dagger/dagger/pull/8800#issuecomment-2479499749, but also around https://github.com/dagger/dagger/issues/8766#issuecomment-2474813161.

Even when we have a way to configure engine settings, there are a collection of user configurations that would make sense to apply per client-session, without requiring a whole engine reload. For example:

  • Registry settings - today these are all only in ...
vivid lintelBOT
ornate vigilBOT
#
benkeil
[dagger/dagger] New comment on discussion #12136: Session config file

I would love to just use the Client.connect function to specify engine specific stuff.

Dagger.connect(
    workingDir = "../", 
    caCertificatePath = File("/path/to/my/ca-certificates"),
).use { client ->
  ...
}

or maybe even better

val caCertificates: List<File> = functionThatLoadsCertificates()

Dagger.connect(
    workingDir = "../", 
    caCertificates = caCertificates),
).use { client ->
  ...
}

So I can get my certificates from somewhere (or what eve...

#
jedevc
[dagger/dagger] New comment on discussion #12136: Session config file

No disagreement on where we want the architecture to be eventually - but while we're not there, having the config fused together will be confusing - but I do think we can resolve that in docs.

If that is what we want, one fused file, some settings requiring a manual engine restart, and some not, then should we rename the existing engine.json to something else before we release, to avoid needing to rename it later on?

#
shykes
[dagger/dagger] New comment on discussion #12136: Session config file

  1. Agree we need a config file for Dagger that is environment specific, in other words: not dagger.json, and not checked into the app repo. Somewhere in the home directory makes sense.

  2. Really not a fan of further splitting that into multiple files for multiple components, because: a) it's more concepts to learn for the user b) these concepts are mostly implementation details, c) the separation between components might actually move, eg. engine-wide config might shrink over time as we ...

#
jedevc
[dagger/dagger] New comment on discussion #12136: Session config file

I'm not sure I agree.

There are 3 very distinct "types" of config, written by different "personas":

  • Module config. These settings belong only to the module, potentially #6323. The module author probably writes these.
  • Engine config. These settings are tied to the lifetime of an engine, and aren't specific to a user (in the case of a shared engine). Things like garbage collection, or security configuration, etc) - when these are changed, the entire behavior of the engine can change (an...
#
shykes
[dagger/dagger] New comment on discussion #12136: Session config file

Module config

I'm torn on this part because it's got a foot in each world:

  • One the one hand it's directly linked to module schemas... So if it's kept in the home directory it might get out of sync with the modules it's targeting
  • At the same time, it's specifically meant to be envir...
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12138: shell: portable scripts

Problem

One benefit of dagger shell is that it could potentially match the convenience of bash scripts. 👍 One drawback is that it could potentially match the portability of bash scripts 👎

Since the shell interprets scripts in the CLI, it gets its own sandbox, separate from module runtimes. This is an opportunity to design the sandbox for the ideal balance of portability and convenience.

Solution

We are still searching for the ideal balance of portability and conveni...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger/dagger] New comment on discussion #12138: shell: portable scripts

@idlsoft I think the bash syntax is a good way to enforce that host-side is for the last mile, and if your scripts start growing too large, it's time to move them to code in a regular sandboxed function - which means that has to be a good experience, too.

The second you want your host-side script to be callable by another script (something which is implied with "host-side scripts" and a more structured language like starlark), then you're on the wrong track.

To address your specific use...

#
idlsoft
[dagger/dagger] New comment on discussion #12138: shell: portable scripts

host-side scripting is exactly what I'm missing in my setup.
I currently use bash scripts to prepend initialization functions to a dagger call, but this is very limited.
Ideally I'd like to be able to define "host-side" functions, that glue things together.
For example, a dev-generate function that drops generated sources to local directories, or a docker-build function that exports a docker container to the local engine.
It may be a bit much to do that with a bash-like shell though, ...

#
idlsoft
[dagger/dagger] New comment on discussion #12138: shell: portable scripts

The second you want your host-side script to be callable by another script (something which is implied with "host-side scripts" and a more structured language like starlark), then you're on the wrong track.

Not exactly what I had in mind, but I get that this may be well outside the scope here.

I have a bunch of projects, different languages, but all of them should be able to do build, test, lint.
If certain env variables are defined, also deploy.

A "host side" skylark scri...

#
dippydocus
[dagger] New discussion #12139: ✨ Provide a GetIPAddress method on a Service container

What are you trying to do?

I'm writing some tests that require the IP addresses of Services. Unfortunately I can't use hostnames as it's for a bespoke load balancing configuration that requires IP addresses.

If using testcontainers-go this is a simple call on a container which provides you with the IP address. Following this idea would be incredibly useful as it avoids the need to either manually manage the lifecycle of a Service or bind containers to it to obtain this information ...

vivid lintelBOT
ornate vigilBOT
#
sagikazarmark
[dagger] New discussion #12140: ✨ Function parameter validation

What are you trying to do?

Today, Dagger enforces "some" validation rules on function parameters. For example, there is a way to define enums and enforce that a parameter passed to a function is a valid enum.

However, there are way more cases where validating input values would make sense and better support built into the SDKs would make module code more concise and easier to read.

Some examples:

  • version number
  • enums that are hard to represent as enums right now (eg. a limi...
#
sagikazarmark
[dagger/dagger] New comment on discussion #12140: ✨ Function parameter validation

Personally, I like the validation interface better. The pragma may be more universal across different SDKs on the surface, but the accepted values there may not be that easy unify.

For example, some Go validation libraries support struct tag based validation (which is similar) and uses a language like this: notNull,mandatory,constraints:[StringNoControlCharacters{},StringLength{Minimum: 1, Maximum: 255}]

Supporting only regex can quickly become uncomfortable, implementing a more sophi...

vivid lintelBOT
ornate vigilBOT
#
jasonmccallister
[dagger/dagger] New comment on discussion #12140: ✨ Function parameter validation

Just going to add some comments here as I was going to post this feature request this morning as I was working on a module and needed validation.

        // version must be a valid PHP version
	switch version {
	case "8.2", "8.3", "8.4":
	default:
		return nil, errors.New("invalid PHP version")
	}

Having syntax that supported validation inline would be immensely useful for module developers.

I've used a few validation libraries and packages but the one syntax that I...

vivid lintelBOT
ornate vigilBOT
#
sagikazarmark
[dagger] New discussion #12142: ✨ Improve module debugging experience

What are you trying to do?

With the introduction of modules, developers can now reuse pieces of code for building their workflows, either built by them or other developers.

I mostly used my own modules so far, so adding new features or debugging module functionality wasn't particularly difficult.

When it comes to tinkering with third-party modules, especially for someone who isn't very familiar with how modules work, this may be more difficult.

This is how you do it today:

  • ...
#
mihaelmiklec
[dagger/dagger] New comment on discussion #12142: ✨ Improve module debugging experience

That would be quite handy! I’d like to add that vendor would probably be the best name for this command, as it's a commonly used term for including third-party dependencies directly in the codebase. Mimicking the replace directive from go.mod would be familiar to Go developers, but it would need to be adjusted for the JSON format.

The vendor command could pull in specified dependencies and automatically add a replacement field in dagger.json for each dependency, like so:

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12144: Allow passing "empty" values from Go SDK

Suppose the following scenario:

// module foo
func (f *Foo) DoFoo(
	// +optional
	// +default=true
	arg bool,
	// +optional
	arg2 string,
} error {
	// ...
}

// module bar
func (f *Bar) DoBar() error {
	return dag.Foo().DoFoo(dagger.FooDoFooOpts{Arg: false})
}

As described in https://github.com/dagger/dagger/issues/7777#issuecomment-2203387161, this false won't apply, and we'll get the default true value set for arg.

That's because of the ...

#
jedevc
[dagger/dagger] New comment on discussion #12144: Allow passing "empty" values from Go SDK

Yeah, this is kinda weird - pointers let the callee handle this, but there's no way for the caller to handle it.

I can't call DoFoo from another Go module to get a false arg (no matter how it's been declared) - at the moment at least.

Is this problem present with pointers as well?

Woops, there is another way to handle this with pointers, edited to add this to the list of potential solutions.

#
jedevc
[dagger/dagger] New comment on discussion #12144: Allow passing "empty" values from Go SDK

Ooh, I like the arg chaining a lot, definitely way less verbose, and we don't need to do the messing around with interfaces. If we want to keep backwards compat, we can't quite have those names, we'd have to have WithArg or similar.

I think the FooDoFooOpts structs would just look like this then:

type FooDoFooOpts struct {
	Arg bool
	Arg2 string

	// we can use these set booleans to check if it was explicitly set
	// otherwise we fallback to querybuilder.IsZeroValue
	...
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger/dagger] New comment on discussion #12144: Allow passing "empty" values from Go SDK

Can I suggest an option that may seem weird, but could have some potential? I call it "goofy/regular".

TLDR: introduce an alternative calling convention with a different DX and more features. The two conventions would co-exist, preventing any breaking changes.

  • Regular convention: same as today.
  • Goofy convention: encode the call in a single object, then make the call with Callxxx.

Benefits

  1. A clean DX for explicitly setting and getting optional arguments
  2. No breakin...
#
jedevc
[dagger/dagger] New comment on discussion #12144: Allow passing "empty" values from Go SDK

We discussed in discord, but summarizing my thoughts on the goofy calling convention (for the record):

  • I agree on the callee example - this would be very useful in lots of places, like our gha module. No notes, let's do it :tada:
  • For the caller example:
    • Setting required args through .Set methods has a few down-sides: it's now possible to accidental omit a required arg (the type checker can't catch this, and instead we get a runtime error), it's entirely different from every ...
#
frizzr
[dagger/dagger] New comment on discussion #12144: Allow passing "empty" values from Go SDK

Just a simple question from a newbie here.

So is there no current workaround for having a default value of true? I'm not meaning to be a smarty-pants, but given none of this is documented (you guys really need to document this) I don't think I have any other easy way to find this out without just asking this question.

There are definitely multiple people in our company that have been directly impacted by this (just in case you need more evidence that this is an important issue.)

#
shykes
[dagger] New discussion #12145: Single-file modules

Problem

As mentioned by @kjuulh and others, modules are powerful, but they have a steep learning curve. Going "from zero to first module" is major source of friction for adopting Dagger.

Solution

One possible solution: introduce the concept of single-file modules.

Before After
dagger init --sdk=foo creates a dagger.json + a complete boilerplate source directory dagger init --sdk=foo creates a dagger.json+ a single boilerplate source file
...
vivid lintelBOT
ornate vigilBOT
#
Mjb141
[dagger/dagger] New comment on discussion #12145: Single-file modules

An isolated comment on this bit:

I think in practice, that requires containerizing the dev environment, and managing it in Dagger. IDE integration would then involve a devcontainer bridge, or equivalent. That makes IDE integration harder to ship, but we will need to cross that bridge eventually: the tech industry is already gravitating towards containerized dev environments, it's kind of silly that one of the most advanced container-based platforms out there, has no plan for that - not on...

#
Vad1mo
[dagger/dagger] New comment on discussion #12145: Single-file modules

I agree, the learning curve for dagger adoption is a bit more than it looks at the surface.

I do have a comment in the direction of doing two things/having two things pursuing the same goal.

From my experience with other tools in that space, e.b. GH Action, GitLabCI, both offer different ways to construct a workflow file. While both files end up doing the same thing, one looks entirely different from the other.
One is definitely smaller and nicer. However, and here is my big pay...

#
helderco
[dagger/dagger] New comment on discussion #12145: Single-file modules

Speaking from the POV of Python modules here.

I think we should address the fact that our current IDE support is already not great, and we don't have a clear plan for addressing that. In fact, I don't think we even have a clear consensus on how we want to integrate with IDEs.

My approach has been to make modules normal Python packages, using conventions and standards as much as possible. I think this has the best chance of working as expected in several IDEs, without much effort...

#
jedevc
[dagger/dagger] New comment on discussion #12145: Single-file modules

Beyond any implementation details, could we consider dagger.json + <other file> - we could merge it all together:

// main.dagger.go
package main

// we could have a custom dagger block somewhere in the file, this could be language specific
dagger (
	name "my-module"
	engineVersion "v0.13.6"

	dependency (
		name "hello"
		source "github.com/shykes/hello"
	)
)

// the my module as normal

type MyModule struct {}

type (m *MyModule) Hello(ctx context.Context) (st...
#
jedevc
[dagger/dagger] New comment on discussion #12145: Single-file modules

Seamless IDE integration

I think that to make single-file modules a viable option as the default, while we don't need to be seamless, I think we should aim to be at near as good as a standard setup today. Otherwise, I suspect most users would quickly bail out. That said, if we had this, I would switch over to this instantly, I would love to be rid of the clutter we currently have.

From my memory, IDE integration these days tends to involve:

  • Some syntax highlighting work (using ...
#
shykes
[dagger/dagger] New comment on discussion #12145: Single-file modules

@jedevc would it be viable in the future, for each SDK to embed a custom language server? Then you could configure your IDE to call that. It would remove one load-bearing wall in this conversation: the need for checked-out files to match perfectly what a vanilla language server expects, always and without exception.

#
shykes
[dagger/dagger] New comment on discussion #12145: Single-file modules

My approach has been to make modules normal Python packages, using conventions and standards as much as possible. I think this has the best chance of working as expected in several IDEs, without much effort from us. At least the Python SDK doesn’t cater to any one IDE specifically. And it doesn’t need to.

Whatever the reason, our users often complain about issues between their modules and their IDE. I'm not judging whether it's the best it can be - I'm just saying it's objectively a ...

#
shykes
[dagger/dagger] New comment on discussion #12145: Single-file modules

would it be viable in the future, for each SDK to embed a custom language server? Then you could configure your IDE to call that.

Viable, yeah. Whether they're distributed as distinct components, or bundled into an SDK as a dagger.Service doesn't hugely matter IMO. Not to trivialize the distribution part of this, but it's definitely the easiest bit, the actual language server bits are gonna be the heavy bits.

I agree, I was thinking more of the "actual language server bits" ...

#
bdols
[dagger] New discussion #12146: ✨ optionally request to not dereference symlinks in File().Size()

What are you trying to do?

just wanted to calculate the size of the files in a Container().Directory() with a combination of Directory().Entries() and File().SIze(), but calling Size on every File that is a symlink returns the size of the linked file.
would like to return the actual size on the filesystem of the symlink entry instead.

Why is this important to you?

had an idea to find the largest directories/files embedded in an image, a la the dive utility but simpler

How a...

#
kpenfound
[dagger/dagger] New comment on discussion #12145: Single-file modules

Good thing we have Astral 😄 This is where the Python SDK strikes a balance. It only supports the uv package manager (uv.lock). uv is pretty great, we don’t need to support more. If you’re on a system without any Python tooling, all you need is to download this single binary once, and run uv run vim in your module to set it all up for you and get code completions.

This is a crutch. I shouldn't have to install anything on my machine to develop a Dagger module, other than Dagger.

+1....

#
jedevc
[dagger/dagger] New comment on discussion #12145: Single-file modules

Ah my bad, there's no sync in the language server protocol. I guess that rules out running it in an entirely separate container, some component of the LSP server will need to run on the host next to the editor.

But we can still do this: the language server just accesses the local files on the host, and can create "virtual files" in /tmp to jump to - all the generated files are still hidden from the user's current directory. This is sort of how go's LSP allows you to jump to definitions in th...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12147: CLI: pass modules as argument

Problem

I can't pass a module as argument to a function. This prevents a whole class of use cases that require one module dynamically loading another. For example, the "platform module":

  • 100 different applications each have a Dagger module. They all implement the same interface: Build(), Test()
  • 1 module defines an App interface which includes Build and Test; and it implements a reusable Deploy method which takes an array of the Apps.
  • App teams can call the platf...
vivid lintelBOT
ornate vigilBOT
#
M-Pixel
[dagger/dagger] New comment on discussion #12147: CLI: pass modules as argument

Is it possible to pass any Dagger object other than the special cases like Directory as arguments in the CLI right now? I've wanted to do

x = dagger core container from --address=something/distroless directory --path=/
dagger --interactive core container from --address=fedora with-directory --path=/inspect --directory=$x terminal

But couldn't find a way to do it mentioned in the documentation. And I tried using from-id but the id expires by the time the next command runs.

Maybe...

#
chrisjpalmer
[dagger/dagger] New comment on discussion #12147: CLI: pass modules as argument

This kind of thing is basically what we (nine) are doing right now using dagger shell. Our setup is we are constructing "dependencies" of our "main" dagger module in dagger shell, then passing those dependencies into the main module. The main module accepts the dependencies by interface.

An example of one of the dependencies we have is an AWS module that provides access to aws resources. We construct it in dagger shell to abstract away the differences per environment ( where our dagger func...

#
kpenfound
[dagger] New discussion #12148: ✨ `WithMountedWorkdir()`

What are you trying to do?

Almost every pipeline has WithMountedDirectory().WithWorkdir(). It would be nice to have some sugar to do this in one call with something like WithMountedWorkdir()

The name probably needs some more thought because it could cause confusion between WithDirectory/WithMountedDirectory vs WithWordir != WithMountedWorkdir

Why is this important to you?

No response

How are you currently working around this?

No response

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger/dagger] New comment on discussion #12148: ✨ `WithMountedWorkdir()`

I think instead of WithMountedDirectory("/foo").WithWorkdir("/foo") we should recommend WithWorkdir("/foo").WithMountedDirectory("."). It achieves the same result I believe (avoid awkward repetition of /foo) without further complicating our filesystem API. I think WithDirectory vs WithMountedDirectory is already too complex for most users, adding more complexity in that area is really best avoided unless absolutely necessary IMO.

#
marcosnils
[dagger/dagger] New comment on discussion #12148: ✨ `WithMountedWorkdir()`

we should recommend WithWorkdir("/foo").WithMountedDirectory("."). It achieves the same result I believe

yes, this also works.

adding more complexity in that area is really best avoided unless absolutely necessary IMO.

I wasn't necessarily looking at this from the complexity perspective but more like a shorthand since I've found myself writing WithMountedCache().WithWorkdir() pretty much every time. I guess it's the same reason I also value Golang's stdlib shorthands like `...

#
jpadams
[dagger] New discussion #12149: ✨ Should have ability to surface warnings/messages at top level of TUI, WUI, log output

What are you trying to do?

The TUI buries a lot of warnings and messages by collapsing at the end, sometimes you need to communicate something to the user that is durable after a pipeline ends.

Why is this important to you?

For example: you might be telling folks that they need to update their module versions or that there is a next step needed.

Could be implemented by priority on custom OTEL spans, perhaps.

How are you currently working around this?

cc @vmaffet-fast @...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12151: Should the SDKs prioritize DX or flexibility?

Problem

We don't clearly answer the question: should we optimize for the best DX, or for flexibility?

  • Best DX: one package manager, one runtime, one way to organize your files. Probably customizable, but not infinitely so. Probably easier to learn, easier to develop, and faster. But won't "blend in" with the surrounding project; and maybe harder to import code from app into the module or vice-versa

  • Flexibility: adapt to your existing package manager, runtime, and layout as much as p...

#
TomChv
[dagger/dagger] New comment on discussion #12151: Should the SDKs prioritize DX or flexibility?

@shykes

Best DX: one package manager, one runtime, one way to organize your files. Probably customizable, but not infinitely so. Probably easier to learn, easier to develop, and faster. But won't "blend in" with the surrounding project; and maybe harder to import code from app into the module or vice-versa

I personnally prefer this option since we can really focus on making the best out of our choice of runtime, configuration etc...
We can make our choice based on few points like sp...

vivid lintelBOT
ornate vigilBOT
#
smolinari
[dagger/dagger] New comment on discussion #12151: Should the SDKs prioritize DX or flexibility?

@TomChv -

This is valid if we choose to adopt Node as runtime, but what if we choose Bun? Or Deno?

"what if we dropped Node in favor of Bun since Bun is waaay faster than Node in our benchmark" (Bun initialization is always <10s while Node is >15 uncached).

What is being done at initialization? I mean, is transpilation and type checking happening? I guess I need to work with the TypeScript SDK. 😝

Totally agree with you on that, do you have a suggestion of what time per run...

#
TomChv
[dagger/dagger] New comment on discussion #12151: Should the SDKs prioritize DX or flexibility?

In terms of the package managers, it depends on how strict you want to be with where dagger modules live. If you go down the road of only supporting bun (for instance), you will pretty much requiring people to have a separate repo for their dagger modules, as it would likely clash with any monorepo setup.

But the dagger module is supposed to be in a different directory of your monorepo and ignore by the root tsconfig so it shouldn't clash.

Those are slightly different goals based on...

#
Xiot
[dagger/dagger] New comment on discussion #12151: Should the SDKs prioritize DX or flexibility?

I agree that the two options aren't quite mutually exclusive, but it is a bit of a balancing act.

I'm not sure being more flexible would impact the learning curve but it would definitely make the path less straight forward and harder to make it work. You'll have to check the docs to see how to make the runtime using your specific package manager, because the they're more possibilities, you can get lost for a while, hitting strange dependency issues that could be avoided...

Personall...

#
Vanuan
[dagger/dagger] New comment on discussion #12151: Should the SDKs prioritize DX or flexibility?

So let me pose a question: How often do you experience that value?

I'd say this value can be experienced only when the CI takes so long it makes sense to endure the pain of dagger setup and runtime to run a small set of failing pipeline locally or avoid initial setup cost. Because any CI tool will always be slower than just running without a tool.

So CI tool should balance laziness and impatience. Should be easier than setting up and maintaining a local environment. Should be fast enough ...

#
smolinari
[dagger/dagger] New comment on discussion #12151: Should the SDKs prioritize DX or flexibility?

I'm not seeing how the one should be exclusive to the other.

So, here is my 2 cents.

For sure the most used package managers should be supported. npm, yarn, pnpm. If only one has to be picked, it should be pnpm IMHO. I'm not seeing the reasons why choice of package manager can be problematic though either. 🤔

I wouldn't say customizability is necessary per se. But, extensibility is for sure. But, I understand it is hard for SDKs like Dagger offers. I'm also wondering what customiz...

#
Xiot
[dagger] New discussion #12152: ✨ Have sdk live as dependency

What are you trying to do?

Currently the typescript sdk installs itself as a folder within the project and has its own package.json and lock file.
This makes things a little tricky to work around since there are now two separate places that try to manage the dependencies. This is especially true since the dagger code references the sdk.

Why is this important to you?

The 'normal' way for these types of dependencies are for it to be installed via a package manager and live within...

vivid lintelBOT
ornate vigilBOT
#
TomChv
[dagger/dagger] New comment on discussion #12152: ✨ Have sdk live as dependency

There are two unique parts. One is the SDK itself, and the other is the code-gen part.
Those could be split up. Some libraries that rely heavily on code-gen, like prisma, do this in order to keep a separation between the 'library' and the 'generated code'

That's actually a good idea, we might be able to split codegen & the SDK library, we would need 2 different important path though like

import { Container } from "@dagger.io/sdk" // Generated code
import { object, fun...
#
Xiot
[dagger/dagger] New comment on discussion #12152: ✨ Have sdk live as dependency

The way that prisma does it (and there are other ways), is that it has its static library as @prisma/client. Internally it references .prisma/client which is the pre-generated code that lives in node_modules.

This allows the user to import @prisma/client and have everything typed and working properly with their generated code.

The gritty details look something like

@prisma/client
  exports: 
    .: require('.prisma/client')
    ./runtime: // static code

.prisma/clie...
#
TomChv
[dagger/dagger] New comment on discussion #12152: ✨ Have sdk live as dependency

Currently the typescript sdk installs itself as a folder within the project and has its own package.json and lock file.
This makes things a little tricky to work around since there are now two separate places that try to manage the dependencies. This is especially true since the dagger code references the sdk.

We installed the SDK as a local dependency because the SDK itself is bundled inside the Dagger binary.

We could do it differently but that means you cannot use the SDK as local...

#
Xiot
[dagger/dagger] New comment on discussion #12152: ✨ Have sdk live as dependency

(Take this all with a grain of salt, since I haven't looked too deeply on how things are constructed internally)

We installed the SDK as a local dependency because the SDK itself is bundled inside the
Dagger binary.

That is the main part that I'm wondering about

We could do it differently but that means you cannot use the SDK as local dep... or we would need 2 different runtime, 1 local and 1 remote

There are two unique parts. One is the SDK itself, and the other is the cod...

#
helderco
[dagger/dagger] New comment on discussion #12152: ✨ Have sdk live as dependency

I’ve considered something similar for Python actually, but not exactly as suggested.

Let’s look at what makes up an SDK client library:

  • The client bindings
  • The client code that lives between the bindings and the communications with the API server, including a very simple connection logic based on only two env variables (session token and port)
  • Code for supporting modules (introspection and registration, deserialization and serialization, validation and dynamic execution or dispatc...
#
Xiot
[dagger/dagger] New comment on discussion #12152: ✨ Have sdk live as dependency

First, thanks for the great write up. I'll go through this from top to bottom first, then have more freeform comments below. I also want to point out that I only have knowledge/experience of using the dagger cli with modules, so I can't really comment on cases that fall outside of that

Initial reactions / comments

Now you’re just left with the generated client bindings and the underlying code that knows how to talk to the API. I actually think this is all that should be published to...

#
helderco
[dagger/dagger] New comment on discussion #12152: ✨ Have sdk live as dependency

Now you’re just left with the generated client bindings and the underlying code that knows how to talk to the API. I actually think this is all that should be published to the public package registry.

What are the generated client bindings here? I read that as it is the stuff that is generated from my 'client' module, although I'm most likely mistaken on this.

See this:

Basically the “client bindings” is what codegen generat...

vivid lintelBOT
ornate vigilBOT
#
kpenfound
[dagger] New discussion #12154: ✨ Daggerverse: Index of a user's modules

What are you trying to do?

it would be great to view all of a user's modules from something like https://daggerverse.dev/mod/github.com/levlaz/daggerverse (i.e. removing the module name from the URL to get to an index, similar to how you can do this on github)

Further, a github user's path would be good too if the user doesn't use a daggerverse repo. So https://daggerverse.dev/mod/github.com/levlaz

Why is this important to you?

No response

How are you currently worki...

vivid lintelBOT
ornate vigilBOT
#
jpadams
[dagger/dagger] New comment on discussion #12154: ✨ Daggerverse: Index of a user's modules

Since a given author might have modules in many git repos in many git servers under different user/org names, it's an interesting question of "who is a user" (since we don't have a Daggerverse login today).
You can make a search today that might accidentally work like:

lev1 OR lev2
real example: https://daggerverse.dev/search?q=jpadams+OR+levlaz

It could make sense to allow search by git username/orgname through something like:

git-org:levlaz

#
valorl
[dagger] New discussion #12155: ✨ dagger modules - semver constraints and lock file

What are you trying to do?

Expanding on https://github.com/dagger/dagger/issues/6605 which addresses the case for a dagger update command.

However, currently dependencies in dagger.json work more as a lock file - any kind of dagger install will be stored as the resolved ref. This makes it a bit hard to reason about dependencies and update them safely, etc.

It would be nice to have a way to declaratively pin the dependencies to a human-friendly ref and/or ideally a semver ex...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12156: N-to-N service dependencies

Problem

The current services API on allows 1-to-1, one-way service dependencies. The following topologies are not allowed:

  1. Service A needs to connect to B, and vice-versa
  2. Service A, B, C all need to connect to each other. Similar to how Docker-compose networks work out of the box.

Solution

Expand the Dagger API to support a wider variety of network topologies, without losing the benefits of the current design.

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12157: Make `dagger.json` include/exclude patterns consistent with `defaultPath`

Summary

Make include/exclude patterns dagger.json relative to the context directory if they start with a slash (/).

Motivation

The include and exclude patterns in a dagger.json are always relative to the dagger.json file.

But with the recent Context directory feature, module authors are able to specify a path that is either relative to the root directory, or absolute to the context directory.

Example for a `f...

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12158: Python: Improve DX on custom enum descriptions

Summary

Create custom enumerations the same way they’re generated by codegen.

Motivation

Python enumerations are generated like this:

from dagger.client.base import Enum

class Severity(Enum):
    """Vulnerability severity levels"""

    UNKNOWN = "UNKNOWN"
    """Undetermined risk; analyze further"""

    LOW = "LOW"
    """Minimal risk; routine fix"""

    MEDIUM = "MEDIUM"
    """Moderate risk; timely fix"""

    HIGH = "HIGH"
    """Serious risk...
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12159: Return or receive another module's type

Problem

A module cannot return another module's type, or receive it as argument. This makes it harder to break down larger projects into several modules.

Solution

I'm not sure what the solution is.

  • On the one hand, the lack of this feature is becoming quite painful
  • On the other hand, I know that this has been considered, and the conclusion was: it would create intractable "dependency hell" problems.

I wonder if the dependency hell problems could be solved with name mangling?

F...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12159: Return or receive another module's type

The original logic for this is illustrated in https://github.com/dagger/dagger/pull/6102#issuecomment-1813209934.

There was some discussion recently about this in #1285289216556925002 message.

  • In other words, GoBuilder as imported by App would get its own namespace. I think this might make the dependency hell problem go away.

I don't think this works - if you return a AppGoBuilder from a module that's not `A...

#
jedevc
[dagger/dagger] New comment on discussion #12159: Return or receive another module's type

I think this potentially solves this, but at the cost of making it very very difficult to use - the App name now has to propagate through the entire system - for deeply nested modules, this produces names that would be fairly unreadable.

It also means we'll struggle with caching - with different names in the graphql schema depending on the caller, different callers won't cache. Maybe we could inject the version into the name?

Actually! We potentially have a neat solution I thin...

#
sipsma
[dagger/dagger] New comment on discussion #12159: Return or receive another module's type

I ruled this idea out when we first started thinking about this problem for a few reasons:

One is what Justin mentioned about "for deeply nested modules, this produces names that would be fairly unreadable"; this approach might be okay for dependencies 1 level deep, but it really breaks down in terms of comprehensibility after that.

Imagine a dependency graph of modules that looks like:

Bar 
     \
      Foo - Blah - Qaz - Common 
     /
Baz

If your module has a dep o...

#
drmikecrowe
[dagger] New discussion #12160: 🐞 dagger init should not create a LICENSE file by default

What is the issue?

Yes, I know https://github.com/dagger/dagger/issues/7703 was debated. Not sure if the typescript SDK is not fixed yet or what.

Dagger is a build system for end-user projects. You should never arbitrarily add a LICENSE file if one does not exist.

@helderco -- your comments specify the exact behavior end users expect:

  • dagger init --sdk .... -- no license file created
  • `dagger init -...
vivid lintelBOT
ornate vigilBOT
#
IshaSri-17Speed
[dagger/dagger] New comment on discussion #12160: 🐞 dagger init should not create a LICENSE file by default

Yes , I agree with @helderco , got the same warning while using go SDK ( common for every SDK) .

12:46:58 WRN no LICENSE file found; generating one for you, feel free to change or remove license=Apache-2.0

As Dagger generated a default LICENSE file with the Apache-2.0 license , it's a common behavior when creating new projects or modules.
What I'm debating is , can I safely remove this LICENSE file without any issues ?

#
IshaSri-17Speed
[dagger/dagger] New comment on discussion #12160: 🐞 dagger init should not create a LICENSE file by default

Thankyou @drmikecrowe for the confirmation

Can we Introduce a command-line flag (e.g., --no-license) that users can specify when running dagger init to skip generating the LICENSE file.
Alternatively, moving the creation of the LICENSE file to the dagger develop command would ensure that it only appears when actual code is being generated as discussed in #6666.

Change would helps in **streamlining project initialization and reduce unnecessary clutter in reposit...

#
helderco
[dagger/dagger] New comment on discussion #12160: 🐞 dagger init should not create a LICENSE file by default

Can we Introduce a command-line flag (e.g., --no-license) that users can specify when running dagger init to skip generating the LICENSE file.

You can actually skip creating the license with an empty value to --license:

dagger init --sdk=go --license=

(--license= is that same as --license "")

But I realize that this is neither intuitive nor documented in --help. The reason for this is in:

#
drmikecrowe
[dagger/dagger] New comment on discussion #12160: 🐞 dagger init should not create a LICENSE file by default

These are my personal opinions as an end user:

  • Bottom line: NEVER add a license to an end user's production code unless the user adds the --license. Changing a proprietary project to open-source would be one of the worst things you could do (this is what happened to me).
  • The dagger init/develop --license is the correct approach
  • If you are trying to address Daggerverse modules, Daggerverse is broken beyond the LICENSE aspect and not very useful in the node/TypeScript world (sor...
#
drmikecrowe
[dagger/dagger] New comment on discussion #12160: 🐞 dagger init should not create a LICENSE file by default

I'm struggling with why you are trying to create a LICENSE file.

Here's my new-user view of my dagger journey. I'm your target demographic -- a devops experienced user that is looking for a better CI/CD solution

Scenario 1

dagger init inside the project directory

/my-project
  |- src/
  |- .git/
  |- README.md
  |- LICENSE (or lack thereof)
  |- dagger.json

Scenario 2

dagger init above the project directory

/dagger-folder
  |- my-project/...
#
helderco
[dagger/dagger] New comment on discussion #12160: 🐞 dagger init should not create a LICENSE file by default

@drmikecrowe, you're right if I understand you correctly, in that the LICENSE file is intended for the use case of standalone and reusable Dagger modules, that get indexed on the Daggerverse. However, we don't have a good way today to distinguish which case is which, and the discussion around how to integrate with existing projects is still in flux so things aren't set in stone.

#
helderco
[dagger] New discussion #12161: Broken SDK co-existence

This issue is a catch-all for a birds-eye view on the general subject of SDKs integrating with existing projects or monorepos in Dagger modules. Specific ideas or proposals benefit from a dedicated issue though.

Problem

SDKs have 2 conflicting requirements:

  1. Don't interfere with host project
  2. Make it easy to import code from the host project

This results in "wack-a-mole" situation where users encounter confusing errors when daggerizing, and require custom community suppo...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12162: Proposal: new variant of `With` helper for improved conditional chaining

Came up as an idea when discussing with @shykes. Currently, we can use this pattern:

ctr = dag.
	Container().
	From("alpine").
	With(func (ctr *dagger.Container) *dagger.Container {
		if !dev {
			return ctr
		}
		return ctr.WithExec([]string{"go", "env"})
	}).
	WithExec([]string{"go", "build", "-o", "binary", "./pkg"})

With can be used to easily insert little conditionals into the chain, without breaking it! However, it's a bit unwiedly. What would be really nice...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12164: Limitations in Platform API

Problem

The Dagger API lets you manage the platform (OS + Architecture) of containers. But there are limitations in the API that make some use cases awkward, or sometimes impossible.

For example, given a Container that may point to a multi-architecture image, I can't select a platform:

// Return the version of an image for a specific platform, or an error if the platform is not available
func GetPlatform(img *dagger.Container, platform dagger.Platform) (*dagger.Conta...
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12165: Import code from the host project

Sometimes a Dagger module needs to import code from the project it's embedded in. For example, in our own CI we use a distconsts package to share constants across the Dagger codebase, and the pipelines shipping it. Things like the default path of the engine's state directory; etc.

There's no officially recommended way to do this. What Dagger does for its own CI, is not mentioned in the docs, and we don't know whether it's a good pattern, because we haven't discussed it. So let's discuss ...

vivid lintelBOT
ornate vigilBOT
#
pjmagee
[dagger/dagger] New comment on discussion #12165: Import code from the host project

In a polyglot monorepo setup, how would this work? Should it be aligned with the Dagger module's SDK, meaning it can't import a pip package in a Dagger module built using the Go SDK? Overall, it's a great idea.

Multiple dagger sdks in one repo... is that an option? maybe that's not sane though. But, if a team is working in a polyglot way, if dagger does support that language, why not have multiple dagger sdks? - Would be cool to rollup into something like dagger functions - but it know...

#
shykes
[dagger/dagger] New comment on discussion #12165: Import code from the host project

@Excoriate @pjmagee yes, for polyglot monorepos, it's ideal to have different modules in different parts of the repo, each with its own SDK.

But sometimes one of those modules will need to import code from the application module within the same language silo. That's what this issue is about.

Using native imports as an alternative to multiple modules is definitely not recommended, edit in a polyglot repo.

#
helderco
[dagger/dagger] New comment on discussion #12165: Import code from the host project

But I think allowing include/exclude patterns in a dagger.json to reach the context directory, would provide more fine-grained control, and it would be easier to manage:

Correction, it's actually doable with include/exclude patterns to import code from outside the module, but could be better with:

#
rawkode
[dagger] New discussion #12166: ✨ daggerverse publishing - use package manager version info

What are you trying to do?

When we write a Dagger module, we need a package.json, go.mod, etc.

These files contain version information that can be consumed by the Daggerverse publishing process.

Currently, we're expected to use Git tags, which when used in a repository housing many modules is far from ideal; mostly because Git and GitHub just don't expect developers to work in this paradigm.

I'd like to see the package manager file used and mapped to a SHA rather than litter...

#
sagikazarmark
[dagger/dagger] New comment on discussion #12166: ✨ daggerverse publishing - use package manager version info

When we write a Dagger module, we need a package.json, go.mod, etc.

I don't think go.mod has a version in it. Or does it?

Currently, we're expected to use Git tags, which when used in a repository housing many modules is far from ideal; mostly because Git and GitHub just don't expect developers to work in this paradigm.

Go modules seem to have adopted this pattern and while it's not perfect, I've seen it work well enough.

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12167: Sandbox SDKs?

Problem

Dagger SDKs have access to the entire context of their host project. This allows for in-depth integration, but it also creates the risk of SDKs accidentally interfering with the project. This risk increases with the number of SDKs, and the fragmentation of host projects (different package managers, different conventions, etc).

Recently I have noticed an increase in the number of users reporting problems of this nature.

Solution

I propose sandboxing SDKs. This would p...

#
helderco
[dagger/dagger] New comment on discussion #12167: Sandbox SDKs?

For me, ideally, you'd sandbox by default (in any case), but allow users to explicitly specify what else from the repo should be included when the module loads.

Solutions may vary to allow for it, but when it's managed by the user explicitly, it still allows some advanced use cases, without the confusion in simple ones.

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger/dagger] New comment on discussion #12167: Sandbox SDKs?

To be clear, the Go SDK no longer modifies the parent go.mod during codegen:

But the Go SDK is still the only one that loads files from outside the root directory, in all cases:

If we stop overwriting existing files on init, it's a simple change that goes a long way towards this:

So then, the only thing remaining to completely sandbox, wo...

#
jedevc
[dagger/dagger] New comment on discussion #12167: Sandbox SDKs?

Agreed, this sounds like the right approach - if we do this, do we go for a slow deprecation with backwards-compat? Honestly, it feels unlikely to break reusable modules in the daggerverse, and more likely to effect the monorepo use case.

I think it's potentially worth considering a hard break here, since supporting both the code paths for this very tricky logic seems quite challening.

#
jpadams
[dagger] New discussion #12168: ✨ Info message if `_EXPERIMENTAL_DAGGER_RUNNER_HOST` is set. Error message if set but unable to connect to that engine.

What are you trying to do?

Users often use _EXPERIMENTAL_DAGGER_RUNNER_HOST during development with a dev engine or in corporate environments where they are using a custom engine with certs or proxies set. It's all too easy to leave that env var set in your shell (or in CI) when you intend to use a released Dagger engine via the dagger CLI, for example.

see also: https://github.com/dagger/dagger/issues/6599

We should give users informative messages if `_EXPERIMENTAL_DAGGER_EN...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12169: Catch missing embedded files at build?

Problem

I can access files "embedded" in my module source, with dag.CurrentModule().Source(), which is super practical. But if I make a mistake and the file is not there, I will not get an error when loading the module - only when calling the particular function that uses the file. I just got bit by this today.

Solution

It would be great if missing embedded files were detected at module load. This would give me more confidence that I am not shipping my module with a hidden fl...

vivid lintelBOT
ornate vigilBOT
#
rawkode
[dagger] New discussion #12170: ✨ Add environment variable support to arguments

What are you trying to do?

It would be fantastic if we could allow argument annotations on functions, this would allow pulling the environment from .envrc or .env files

Why is this important to you?

Each service in my mono repo configures itself with direnv. Currently I'm duplicating a lot to make workflows work locally and in Dagger.

How are you currently working around this?

Related

#
dragoonis
[dagger/dagger] New comment on discussion #12170: ✨ Add environment variable support to arguments

I'm curious @rawkode, is this what you're referring to?

I had to write this in PHP, as a workaround. It would be great not to have to do this

    private function loadEnvArgs(Container $container, Directory $source): Container
    {
        $envContents = $source->file('.env')->contents();
        $envVars = parse_ini_string($envContents);

        foreach($envVars as $envVarKey => $envVarValue) {
            $container = $container->withEnvVariable($envVarKey, $envVarVal...
vivid lintelBOT
ornate vigilBOT
#
bupd
[dagger/dagger] New comment on discussion #12170: ✨ Add environment variable support to arguments

I have written a shell script to workaround this in the meantime.

#!/bin/sh

envFile="/envFile"

# Check if the file exists
if [ ! -f "$envFile" ]; then
  echo "Environment file not found: $envFile"
  exit 1
fi

# Read the file and export each line as an environment variable
while IFS='=' read -r key value; do
  # Ignore empty lines or comments
  if [ -z "$key" ] || [ "${key:0:1}" = "#" ]; then
    continue
  fi

  # Export the environment variable
  export "$key"="$value"

  # print t...
#
thecooldrop
[dagger] New discussion #12171: Clarify that Dagger behaves more like `docker build` instead of like `docker run`

What is the issue?

I have recently started using Dagger and have experienced a lot of friction, due to not being able to understand how Dagger behaves. Here are two examples of things, which I found confusing as user:

  • When running with_exec on container, and starting a process in background such as socat, for example using .with_exec(["/bin/sh", "-c", "socat tcp-listen:2375,reuseaddr, fork tcp:dockerd:2375 &"]).with_exec(["ps", "aux"]), then ps aux does not display the socat p...
vivid lintelBOT
ornate vigilBOT
#
jpadams
[dagger/dagger] New comment on discussion #12171: Clarify that Dagger behaves more like `docker build` instead of like `docker run`

Thank you @thecooldrop. Agree that we can improve the docs in these areas.

  • Have you seen any docs that do a great job with setting that kind of context for new users?
  • Are there existing sections in the Dagger docs where you'd expect to find the information you were looking for? Which ones?
  • Or do we need a new section or sections? If so, what would you call them?
#
thecooldrop
[dagger/dagger] New comment on discussion #12171: Clarify that Dagger behaves more like `docker build` instead of like `docker run`

Regarding your questions:

  • I have no reference to an example from other projects, which could serve as a guide toward this. Gradle may be good, since it also starts out by demonstrating a simple usecase, and then iteratively deepens the discussion of specific areas in additional sections. For me usually structuring the documentation around basic usecase and then providing a deeper look, in one colocated area is usually very helpful.
  • There is currently no section which I would put this ...
#
levlaz
[dagger] New discussion #12172: 🐞 running service with `up` swallows up fatal errors

What is the issue?

I am using dagger as a part of my blog publishing workflow and one feature that includes is being able to preview my blog locally by wrapping hugo in a dagger function.

// build and serve local hugo site
func (m *Publish) ServeDev(ctx context.Context) *dagger.Service {
	path := "/src"
	return m.base().
		WithMountedDirectory(path, m.Src).
		WithWorkdir(path).
		WithExposedPort(1313).
		WithExec([]string{"hugo", "serve", "--bind", "0.0.0.0"}).
		AsSer...
vivid lintelBOT
ornate vigilBOT
#
liamjones-pw
[dagger] New discussion #12173: 🐞 Potential regression in dagger.ContainerWithFileOpts permissions

What is the issue?

Initially reported [in Discord](#github message), Dagger 0.15.1 running on GitHub Actions doesn't appear to respect ContainerWithFileOpts permissions as it did previously in 0.14.

The permissions are applied as defined in the dagger code in these cases:

  • Local
    • 0.14.x ✅
    • 0.15.x ✅
  • GitHub-hosted Runner
    • 0.14.x ✅
    • 0.15.x 🚫

I expected the permissions set in Dagger t...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12173: 🐞 Potential regression in dagger.ContainerWithFileOpts permissions

Okay, I've tried to reproduce what you're seeing, and kind of struggling.

So, I've setup a test repo https://github.com/jedevc/repro-dagger-9250 - here's the go code:

package main

import (
	"dagger/example/internal/dagger"
)

type Example struct{}

func (m *Example) Test(mydir *dagger.Directory) *dagger.Container {
	return dag.Container().From("alpine").
		WithFile("entrypoint.sh", mydir.File("entrypoint.sh"), dagger.ContainerWithFileOpts{Permissions: 0500})
}

and the github act...

#
jedevc
[dagger/dagger] New comment on discussion #12173: 🐞 Potential regression in dagger.ContainerWithFileOpts permissions

Investigating this now, will let you know if I need more info! Sorry for the delay!

One note... the dialing issue you see in the log output is likely caused by the engine startup taking longer than it used to. I suspect that this may be fixed by https://github.com/dagger/dagger/pull/9430 (which is where I was seeing such a slow-down). Regardless, I think this is probably unrelated to the permissions issue.

Were you using a shared /var/lib/dagger between different dagger jobs?

#
conorpp
[dagger] New discussion #12174: 🐞 Services are dropped in interactive shell

What is the issue?

I want to use interactive mode (dagger -i ...) to debug issues. But the services always get dropped. If they could remain up, it would be really helpful.

Dagger version

dagger v0.15.1 (registry.dagger.io/engine:v0.15.1) darwin/arm64

Steps to reproduce

// interactive shell looses the service dependency currently (0.15.1)
func (m *Treasury) ExperimentWithService(
	ctx context.Context,
) (string, error) {

	service := dag.Container()....
vivid lintelBOT
ornate vigilBOT
#
thecooldrop
[dagger] New discussion #12175: Service grace period is documented vaguely

What is the issue?

In the explanation of Services the grace period of Services is mentioned, but with no introduction of what that is, nor of what it does. I would like to have documented:

  • What does it mean that Services have grace period?
  • What is the effect of the grace period?
  • How can we avoid the effects of grace period ? From current documentation about Redis, it seems that the grace period can be wroked around.
  • Provide a clear ex...
vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12176: WithMountedCache uses the engine's default `user:group` combination permissions

Problem

It has been raised by a user in Discord (https://discord.com/channels/707636530424053791/1316771968762253312) that WithMountedCache seems to be honoring the engine's default user:group permissions. This causes issues as it breaks the pipeline portability depending on how the the engine is provisioned

Solution

How about if we enforce root:root for all WithCacheVolume calls unless the Owner field is set in the call?

Steps to reproduce

  1. run the eng...
vivid lintelBOT
ornate vigilBOT
#
gotcha
[dagger] New discussion #12177: Fuzzyness about rootless usage

https://github.com/dagger/dagger/blob/6a237d77d1277951d41f307b2505eee44a110747/docs/current_docs/integrations/nerdctl.mdx?plain=1#L15

The line above seems to imply that dagger can be used with rootless containers.

However, https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#can-i-run-the-dagger-engine-as-a-rootless-container states very explicitely that the Engine cannot be ran rootless.

Would it be useful to clarify that point on the nerdctl page ? Or did...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12177: Fuzzyness about rootless usage

Rootless support is eventually possible - however, it's not on the immediate roadmap. There's quite a lot of work and plumbing we'll need to make this work. We do a lot of custom networking to enable Services, and similar things.

That's not to say it's impossible (I'd definitely like to support it one day), it's that it's not currently a priority - we're a small team, we sadly don't have infinite capacity. Atm, a lot of our effort is working on improving performance, building out better ...

#
shykes
[dagger/dagger] New comment on discussion #12177: Fuzzyness about rootless usage

@gotcha the Dagger engine acts as a container runtime. As such it is normal for it to have system privileges required to run containers. Running dagger in a rootless container means running runc+containerd as a rootless container: not impossible but more of a science project; and not required for security.

Security-conscious teams should apply the same best practices to dagger as they do to buildkit/containerd/runc. In practice that means vm-level isolation of workloads.

#
gerhard
[dagger] New discussion #12178: 🐞 CLI continues to run AFTER the Engine stops running

What is the issue?

This is particularly problematic when the Engine exits / gets killed and the CLI continues to run. Without an external timeout (e.g. GitHub Actions job timeout), the CLI execution will not terminate.

Furthermore, pressing q once does not quit the CLI. In other words, the CLI is not able to terminate gracefully when it's no longer connected to the Engine.

Dagger version

dagger v0.14.0 (registry.dagger.io/engine:v0.14.0) darwin/amd64

Steps to ...

vivid lintelBOT
ornate vigilBOT
#
grouville
[dagger] New discussion #12179: core: login against registry mirrors requiring authentication does not seems to work

Follow-up of #python message

The Python SDK now relies on the ghcr.io/astral-sh/uv container image:

with the migration to uv (for python sdk) there is a new container image used (ghcr.io/astral-sh/uv). Now in enterprise environments that is not allowed, like in my case and you need to set container registry mirror. The problem with that is that our mirror requires credentials and I can not change than. I also can...

vivid lintelBOT
ornate vigilBOT
#
garbas
[dagger/dagger] New comment on discussion #12179: core: login against registry mirrors requiring authentication does not seems to work

Sorry for a delay from my side, I've been busy with other stuff.

I'm logged in into the mirror and I can do docker pull without any issues, uv images get pulled down via docker cli. Then running the dagger command fails because it can not pull from our mirror. This iswhat I'm doing.

engine.toml is:

debug = true
trace = false
insecure-entitlements = ["security.insecure"]
[registry."docker.io"]
  mirrors = ["OUR_MIRROR_DOMAIN"]
[registry."ghcr.io"]
  mirrors = ["OUR_MIRROR_DOM...
#
grouville
[dagger] New discussion #12180: docs: Using git credential helpers in CI examples

Some users have been encountering issues when trying out out Git credential helpers in CI

I am moving the conversation to its own issue @pspurlock, and cleaning up the other (everything was screenshotted)

After discussing with @nipuna-perera, they do manage to make it work ; It seems to be a configuration issue / patterns to use, which shall become a standard.

Next step on my end:

  • [x] Working repro for GitHub / GitLab / BitBucket
  • [ ] Isolate the pattern and documen...
vivid lintelBOT
ornate vigilBOT
#
grouville
[dagger/dagger] New comment on discussion #12180: docs: Using git credential helpers in CI examples

hey @pspurlock

I have a working implementation in a gitlab-ci.yml:

image: docker:latest

services:
  - name: docker:${DOCKER_VERSION}-dind
    alias: docker

variables:
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_VERIFY: '1'
  DOCKER_TLS_CERTDIR: '/certs'
  DOCKER_CERT_PATH: '/certs/client'
  DOCKER_DRIVER: overlay2
  DOCKER_VERSION: '24.0.6'
  GIT_USERNAME: "grouville"
  GIT_TOKEN: "${PAT}"

stages:
  - build

dagger_job:
  stage: build
  before_script...
#
pspurlock
[dagger/dagger] New comment on discussion #12180: docs: Using git credential helpers in CI examples

@grouville Thanks for doing this. I can't try out your example fully because dind requires privileged mode for the runner which we don't allow. However, I did try to mimic it as much as possible and still get the same access denied error.

For context, we have a gitlab runner deployed alongside a dagger engine/deamon in a cluster. When we run a CI Job, we then use _EXPERIMENTAL_DAGGER_RUNNER_HOST so that the CI Job will connect and use the daemon running in the cluster. So when `dagger -m...

#
grouville
[dagger/dagger] New comment on discussion #12180: docs: Using git credential helpers in CI examples

@grouville Thanks for doing this. I can't try out your example fully because dind requires privileged mode for the runner which we don't allow. However, I did try to mimic it as much as possible and still get the same access denied error.

For context, we have a gitlab runner deployed alongside a dagger engine/deamon in a cluster. When we run a CI Job, we then use _EXPERIMENTAL_DAGGER_RUNNER_HOST so that the CI Job will connect and use the daemon running in the cluster. So when `dag...

#
grouville
[dagger/dagger] New comment on discussion #12180: docs: Using git credential helpers in CI examples

I have tried to add credentials in both places and neither have worked, but it would help to clarify. I know it isn't a credential issue because I can run the same credential helper commands and then git clone successfully before the dagger command fails.

If you make the same behavior as I did above, basically setting your own credential helper, if you add a step that actually prints it:

- script:
    # this should print user and pat -- please ensure that 
    - /tmp/git-c...
#
grouville
[dagger/dagger] New comment on discussion #12180: docs: Using git credential helpers in CI examples

Hi @skycaptain 👋

Thank you for digging the root cause, I was falling short on time 🙏

  • We already retrieve the username, so totally doable -- it means that users will always have to specify the username, for every provider. If well documented in our docs, would that be ok ? x-access-token for all the other CI providers, gitlab-ci-token for gitlab

  • Regarding your pipeline example, I really like it ; smoother indeed.

  • Regarding the use of the cache `git config --global...

#
skycaptain
[dagger/dagger] New comment on discussion #12180: docs: Using git credential helpers in CI examples

We already retrieve the username, so totally doable -- it means that users will always have to specify the username, for every provider. If well documented in our docs, would that be ok ? x-access-token for all the other CI providers, gitlab-ci-token for gitlab

In my experience, users of private repositories are accustomed to specifying both username and token. If the following line from the documentation represents the intended goal, then I would expect Dagger to use whatever username t...

#
skycaptain
[dagger/dagger] New comment on discussion #12180: docs: Using git credential helpers in CI examples

I want to bring attention back to this issue, as it is currently blocking from effectively using modules on GitLab due to the incorrectly hard-coded username used for HTTPS authentication being incompatible with CI_JOB_TOKENs.

Should we split the "HTTPS with GitLab not working with CI_JOB_TOKEN" or "Dagger not respecting username for HTTPS auth" into a separate issue? The initial intent, as the title suggests, was to provide examples in the docs. However, the discussion revealed an issue wit...

vivid lintelBOT
ornate vigilBOT
#
aweris
[dagger] New discussion #12182: 🐞 Engine suddenly stop responding

What is the issue?

When the engine autoscaled due to increased load, it took approximately 4 to 5 minutes to start jobs because of the size of the caches. During 20 to 30% of these autoscaled job executions, the engine suddenly stopped responding and left the jobs hanging.

When I look to engine logs or k8s logs I can't see any error, issue or event related to this other than screen shot of the engine logs and job logs.

Dagger version

v0.13.7

Steps to reproduce

It is happe...

vivid lintelBOT
ornate vigilBOT
#
jiedxu
[dagger] New discussion #12183: 🐞 "failed to fetch oauth token" when get Python SDK using Dagger CLI

What is the issue?

I'm trying to run dagger develop --sdk=python after the initiation with a name.

Dagger version

dagger v0.13.6 (registry.dagger.io/engine:v0.13.6) darwin/arm64

Steps to reproduce

dagger init --name=blah

dagger develop --sdk=python

Log output

✔ connect 0.2s
✔ cache request: mkfile /schema.json 0.0s
✔ load cache: mkfile /schema.json 0.0s
✔ cache request: blob://sha256:9121d31121f4c30e725664dd17c07133b4cb49e375a465f39f740736d4e...

vivid lintelBOT
ornate vigilBOT
#
fcbry
[dagger/dagger] New comment on discussion #12183: 🐞 "failed to fetch oauth token" when get Python SDK using Dagger CLI

Same issue with dagger v0.14.0 (registry.dagger.io/engine:v0.14.0) darwin/arm64

Trying to run the quickstart in the docs here (w/o Dagger Cloud)
(Go SDK)
~trying to authenticate to azurecr.io (Microsoft Azure Container Registry).
Verified the credentials, if I do a docker login (locally) then it works, but it doesn't seem to be able to use the docker login credentials when run in a Github action.~

Works here, with azurecr.io...

#
ojroques
[dagger] New discussion #12184: 🖐️ Hide WithExec stdout in Dagger logs

What happened? What did you expect to happen?

Hello,

I'm running Dagger in GitLab CI. I'm using it to request certificate to Hashicorp Vault. However Dagger prints the output of the request in plaintext in our GitLab CI, which includes the certificate key. Is there an option to specifically disable the output of a command in Dagger UI, without silencing everything else (with -s)?

Thanks.

vivid lintelBOT
ornate vigilBOT
#
nipuna-perera
[dagger] New discussion #12185: ✨ CLI error message when using `default` instead of `defaultPath` is confusing

What are you trying to do?

I am creating a simple Dagger function that uses context directory. I am using defaultPath for the directory but when coding the function, I intuitively used default instead of defaultPath. The defaultPath is a special pragma for Directory however, the error I got in the CLI did not indicate this. It was cryptic.

Produce a better error message when the user accidentally uses default instead of defaultPath

Why is this important to you?

It...

vivid lintelBOT
ornate vigilBOT
#
kid
[dagger] New discussion #12186: 🐞 Inconsistent behavior of defaultPath with dagger -m <path> call and nested modules

What is the issue?

I have a daggerverse repository that contains multiple modules I want to use from other repositories using dagger -m ...

When I call dagger -m ../daggerverse/ci call , I expected defaultPath to load data from the current repository, but instead, I get the content of my daggerverse repository, which make defaultPath useless here

However, if my ci module call another module that also has a defaultPath="/" argument, then will get the content of th...

vivid lintelBOT
ornate vigilBOT
#
gtabrar
[dagger] New discussion #12187: 🐞 Dagger is generating malformed Go code

What is the issue?

This Go code in a dagger module:

func (m *Rack) Join(r *Rack) *Rack {
    m.Name = m.Name + "+" + r.Name
    m.Services = append(m.Services, r.Services...)
    return m
}

Gets generated as:

func (r *Rack) Join(r *Rack) *Rack {          
        assertNotNil("r", r)
        q := r.query.Select("join")           
        q = q.Arg("r", r)
                                             
        return &Rack{
                query: q,           ...
vivid lintelBOT
ornate vigilBOT
#
SteveKipping
[dagger] New discussion #12188: 🐞 Issue With Container Entrypoint when Dagger.json is missing engineVersion field

What is the issue?

Dagger container entrypoint issue when engineVersion is missing in the dagger.json (I had found this when I tidied up the remote private daggerverse JSON file based on a much older Dagger version example online)

In my particular case I was pulling a container, with a mounted directory and work dir and then running a simple with_exec(["ls","-ls"]) and that was failing when called with -m (with my "tidied up" dagger.json) but worked OK locally as that still engineV...

vivid lintelBOT
ornate vigilBOT
#
zaninime
[dagger] New discussion #12189: 🐞 `Directory` is incorrectly deserialized when passed across custom objects

What is the issue?

As asked on Discord, the code attached exhibits a weird behaviour depending on how it is invoked.

dagger call build something fails with Error: this.myDirectory.file is not a function
dagger call build env base with-exec --args cat,/x succeeds

Note how the build something is fundamentally doing the same as the second CLI command. It appears as if this.myDirectory magically becomes...

vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12190: `terminal` doesn't work if service contains `withHostname`

Problem

Can't call dagger call test terminal on this pipeline:

import { dag, Container, object, func } from "@dagger.io/dagger";

@object()
class Lala {
  /**
   * Returns a container that echoes whatever string argument is provided
   */
  @func()
  async test(): Promise {
    const s1 = await dag
      .container()
      .from("nginx:latest")
      .withExposedPort(80)
      .asService()
      .withHostname("web1")
      .start();

    return dag.container...
vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12191: Getting invalid "function $FN cannot return external type from dependency module $TYPE "

Problem

Getting the following error when trying to define a type with the same name a as a module type:

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to add object to module "program-backend": failed to validate type def: object "ProgramBackend" function "Cache" cannot return external type from dependency module "cache"

main module:

package main

type ProgramBackend struct{}

type Cache struct{}

// Returns a conta...
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12191: Getting invalid "function $FN cannot return external type from dependency module $TYPE "

Hm, this looks like a naming confusion error :thinking:

I suspect we're getting something wrong in the go codegen / engine here - this is where the error is from: https://github.com/dagger/dagger/blob/8b3b8af59328d341a0486eaf1463736ca376849b/core/module.go#L475-L484

(already validated, skip is not the right comment for this code :thinking:)

@marcosnils can you confirm, the main module has your dependent module installed? Also, what are the names of the modules? If the depend...

#
marcosnils
[dagger] New discussion #12192: `dagger develop` traces don't seem correctly grouped

Problem

dagger develop doesn't seem to be using proper grouping in the telemetry spans. Here's the output of an example call:

dagger develop
✔ connect 0.2s
✔ cache request: mkfile /schema.json 0.0s
✔ load cache: mkfile /schema.json 0.0s
✔ cache request: blob://sha256:3f96dc98cb3827315b7c4f8e21be0fec3eabd82f4c09031d83f87650b1b5519c 0.0s
✔ load cache: blob://sha256:3f96dc98cb3827315b7c4f8e21be0fec3eabd82f4c09031d83f87650b1b5519c 0.0s
✔ moduleSource(refString: "."): ModuleS...
vivid lintelBOT
ornate vigilBOT
#
cbochs
[dagger] New discussion #12193: 🐞 Fail to export cache with _EXPERIMENTAL_DAGGER_CACHE_CONFIG (`failed to compute blob by overlay differ (ok=false)`)

What is the issue?

Following some Discord discussions and buildkit comments (https://github.com/moby/buildkit/issues/1143#issuecomment-2327052682), I tried using the experimental dagger cache, pushing to a private docker registry. It appears to pull the cache (if it exists), but fails to export it.

I'm not sure if this is a Dagger issue or buildkit issue, but I've noticed a context canceled error cropping up when the cache is being exported. It seems odds that it's receiving a co...

vivid lintelBOT
ornate vigilBOT
#
drmikecrowe
[dagger] New discussion #12194: 🖐️ Monorepo support

What happened? What did you expect to happen?

Does anybody have any examples of a monorepo? For example, I'm trying to learn how to:

  • Creating the buildEnv at the top level and leveraging this for all apps
  • Caching of node_modules in each projects (some node tooling we use forces us to use individual node_modules at the moment)
  • Supporting 2 source folders (apps/ and packages/ -- both of which are used)
vivid lintelBOT
ornate vigilBOT
#
jim-hm
[dagger] New discussion #12195: Python SDK: Not all the decorators are working

What is the issue?

My function failed when I tried to add decorators on it. I can find workaround but not sure if it should add to docs.

Dagger version

dagger v0.13.3 (registry.dagger.io/engine:v0.13.3) darwin/arm64

Steps to reproduce

get following error when run dagger call load-context (module name is test)

def timer(func):
    """Print the runtime of the decorated function"""
    # @functools.wraps(func)
    async def wrapper_timer(*args, **kwar...
vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger/dagger] New comment on discussion #12195: Python SDK: Not all the decorators are working

At first glance, you should have @function closer to your function, like this:

@timer
@function
async def load_context(self) -> str:
    ...

The reason is that @function needs to correctly introspect the function, and your wrapper may hide the necessary information because it has a different signature. Notice that with Dagger you need to be careful about types because we need to report those to the API beforehand, not just be able to execute. Even during execution we ...

#
jim-hm
[dagger/dagger] New comment on discussion #12195: Python SDK: Not all the decorators are working

Thanks @helderco ,
move @function close to function did not help. Here's error,

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "test" to get functions: call constructor: process "/runtime" did not complete successfully: exit code: 1

Stderr:
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Function “Test.load_context” seems to be decorated in a class that is not    │
│ itself decorated with...
#
bdols
[dagger] New discussion #12196: 🐞 remote container doesn't load and success incorrectly reported

What is the issue?

I pass in a remote container and the CLI reports that it loads successfully (as indicated by checkmark) but doesn't perform any operations as defined in my dagger function.

dagger should at least report that the container failed to loed

Dagger version

dagger v0.13.5 (registry.dagger.io/engine:v0.13.5) linux/amd64

Steps to reproduce

I have a function that is just this:

func (m *InkyBuild) PythonSetup(ctx context.Context,
     ...
vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12197: Services WithHostname and WithServiceBinding don't play together across modules

We should find a way to consolidate the API between WithHostname and WithServiceBinding as they currently don't play along very well together in a multi-module setup.

Repro:

Module A:

// Returns a container that echoes whatever string argument is provided
func (m *Lala) Test(ctx context.Context) *dagger.Container {

    svc := dag.Container().From("nginx").
        WithExposedPort(80).AsService().WithHostname("web")

    svc.Start(ctx)

    return dag.B().Run(svc)...
#
vito
[dagger/dagger] New comment on discussion #12197: Services WithHostname and WithServiceBinding don't play together across modules

The catch is that we can only set a single Hostname with Buildkit for a service, so when WithHostname is used we set the configured one instead of the content-addressed one. That breaks in this case, since only one of the modules will have actually started it, and it'll only have that module's ID in the hostname.

Effectively, only one module can use a WithHostnamed service at a time within a session. You would have the same problem if you tried to reach the configured hostname within...

vivid lintelBOT
ornate vigilBOT
#
jpadams
[dagger] New discussion #12198: 🐞 Cannot use `dagger -s call ... terminal`

What is the issue?

I was running some functions repeatedly with dagger -s to get less verbose output to my terminal. I wanted concise output to share with colleagues as a repro. My dagger -s call commands were working fine until I added terminal to the end of a call that returned a Container type. I expected it to work as normal, but dagger threw an error.

An example:

Error: response from query: input: container.from.terminal resolve: failed to forward exit code: failed ...
vivid lintelBOT
ornate vigilBOT
#
rajatjindal
[dagger] New discussion #12199: 🐞 Changes in cache volume does not reflect as expected

What is the issue?

Hi Team,

I am running into an issue when trying to understand the current behavior of CacheVolume. I have the following simple module:


import (
	"context"
	"dagger/foo/internal/dagger"
	"fmt"
)

type Foo struct{}

func (f *Foo) TestCacheVolumePersistence(ctx context.Context, input string) (string, error) {
	_, err := f.PopulateCache(ctx, input)
	if err != nil {
		return "", err
	}

	output, err := f.ListCache(ctx)
	if err != nil ...
vivid lintelBOT
ornate vigilBOT
#
rajatjindal
[dagger/dagger] New comment on discussion #12199: 🐞 Changes in cache volume does not reflect as expected

not sure if this is related, but in same module (cloned from provided repo), I noticed if I try to create a file using "WithNewFile" inside the mounted cached directory, it errors out: /foo-cache/: cannot retrieve path from cache. (I am suspecting its looking for /foo-cache in the cache itself, while I am expecting /foo-cache to be the mounted dir)

dagger call with-new-file               
✔ connect 0.2s
✔ initialize 0.3s
✔ prepare 0.0s
✔ foo: Foo! 0.0s
✘ Foo.withNewFile: Cont...
#
MikaelElkiaer
[dagger] New discussion #12200: 🐞 Error with mounted ca-certificates in Kubernetes

What is the issue?

I see the error:

time="2024-10-04T08:35:43Z" level=warning msg="failed to rehash ca-certificates: ERROR: Access denied '/usr/local/share/ca-certificates'\n" error="exit status 2"

When I mount a configmap containing some-ca.crt into the folder.
It does not crash the dagger-engine, which starts up afterwards.

While it does not work following container restart, if I exec update-ca-certificates in a pod the CA certificate is applied and works as expected....

vivid lintelBOT
ornate vigilBOT
#
exlee
[dagger] New discussion #12201: 🐞 OTEL-related environment variables deteriorate overall experience

What is the issue?

Hi,

I've been evaluating Dagger and I found performance to be abysmal and TUI exposing buggy and erratic behavior, e.g. (runtimes on M2 Max machine).

Context

  • dagger functions took 20-25s (after aggressive exclude and filtering in module) to finish
  • both TUI and --progress=plain tended to hang indefinitely
  • TUI often showed weird traces, omitted results or heavily delayed result output
  • simple test flow took 6 minutes and didn't finish where r...
vivid lintelBOT
ornate vigilBOT
#
exlee
[dagger/dagger] New comment on discussion #12201: 🐞 OTEL-related environment variables deteriorate overall experience

OTEL_ configuration invalid, i.e. pointing to an unreachable service?

Using above definition yes, it's invalid for Dagger.

ENV variables are targetting developed application that has a network configuration allowing it to reach OTEL, Dagger Engine isn't configured to do so.

As mentioned above, for me the main issue is lack of information when running functions that Dagger not being able to reach OTEL_ configured hosts will impact on its performance.

#
helderco
[dagger] New discussion #12202: 📖 Document how to debug poorly filtered `Directory` objects

What is the issue?

This is related to the following issue, which makes some clarifications on the different mechanisms for filtering a Directory uploaded from the host to the engine:

However, even if it’s clear which is which in a trace, we can help users know how to debug if they’re actually filtered as expected.

See the following comment, which provides source material for this:

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12203: Python: document how to manage monorepo in IDE integration

What is the issue?

Question from user in [Discord](#python message) about how to manage multiple virtual environments in PyCharm, and realized we can help with more clear instructions here:

Try this, have PyCharm opened in your existing project, and File > Open the Dagger module’s directory. You should see a dialog with the option to attach to the opened project. Just select that and, assuming you alread...

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger/dagger] New comment on discussion #12203: Python: document how to manage monorepo in IDE integration

I know how workspaces work, the Python SDK uses it: 🙂

https://github.com/dagger/dagger/blob/d12a6f38adc24049287d50bf5d40ee85fd2d9666/sdk/python/pyproject.toml#L69-L70

Setting up the workspace won't be something that Dagger will manage for you. I'm just talking about documenting how this can be done for those interested in this solution, but it's up to the user to set it up.

#
ajaegle
[dagger/dagger] New comment on discussion #12203: Python: document how to manage monorepo in IDE integration

I did some tests with uv workspaces a week ago (outside of the dagger context) to figure out how they are supposed to be used. While dependencies are specified on the nested projects seem isolated, they are all installed in a single venv and thus must not have conflicting requirements with the other workspace packages. This could be problematic looking at what libraries are used for a minimal Dagger setup. And even if there are no conflicting requirements, I think it's still better to not pol...

#
danielgafni
[dagger/dagger] New comment on discussion #12203: Python: document how to manage monorepo in IDE integration

I'm new to dagger but ain't new to uv workspaces.

I currently have a monorepo with uv workspaces set up.

I'm kinda struggling to create dagger projects in this setup:

  • I'd like to have these projects as workspaces members. This means they won't get their own uv.lock files (all members share a single .venv a uv.lock file in the repo root). it's unclear how to override the default containerized dagger env build by using the repo root uv.lock file.
  • It's unclear why is `d...
#
drmikecrowe
[dagger] New discussion #12204: 🖐️ Caching Node PNPM monorepos (and yarn in pnpm mode)

What happened? What did you expect to happen?

So, at a high level, I have the following folders:

├── .yarn
│   └── berry
├── node_modules
│   └── .store
├── apps
│   ├── skills
│   │   └── node_modules
│   ├── stocks
│   │   ├── stocks-api
│   │   │   └── node_modules
│   │   └── stocks-ui
│   │       └── node_modules
│   └── landing
│       └── node_modules
├── packages
│   ├── pinnacle-layer
│   │   └── node_modules
│   ├── typescript-config
│   │   └── node_m...
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12205: 🐞 422 Unprocessable Entity

As mentioned in https://github.com/sagikazarmark/daggerverse/pull/177#discussion_r1771215596, this is caused by passing an invalid enum value directly to graphql.

This isn't correct, but we should make the error message clearer. Ideally, we should perform some sdk-side validation to ensure that the enum value is actually valid before attempting to marshal it into the query.

I think this is just a go issue, since other languages have stronger enum types that don't suffer from this partic...

vivid lintelBOT
ornate vigilBOT
#
verdverm
[dagger/dagger] New comment on discussion #12205: 🐞 422 Unprocessable Entity

I have a similar error

returned error 422: {"data":null,"errors":[{"message":"Expected Name, found :","locations":[{"line":1,"column":104}],"extensions":{"code":"GRAPHQL_PARSE_FAILED"}}]}
exit status 1

I have no idea what bad data I'm passing to the Go SDK. At least getting at the actual query text could help my narrow things down. A better error message would be appreciated greatly

I narrowed my issue down to

NetworkProtocolTcp NetworkProtocol = "TCP"

where I have...

#
sheldonhull
[dagger] New discussion #12206: 🐞 improve powershell installation experience by putting version requirement in

What is the issue?

For a windows user running the installation against a version incompatible with the latest language features (looking at you... the user who holds onto Windows Powershell 5.1 until their last breath and still tries to use ISE 🤣).... PowerShell has language feature to put required version so the error message is more useful in the installation script.

~[#Requires](https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_requires?v...

vivid lintelBOT
ornate vigilBOT
#
JCzz
[dagger] New discussion #12207: go: go.mod requires go >= 1.23.1 (running go 1.22.7; GOTOOLCHAIN=local)

What is the issue?

Hi

From this: https://docs.dagger.io/manuals/user/chaining

Running this func:
dagger -m github.com/kpenfound/dagger-modules/golang@v0.2.0 call build --source=https://github.com/dagger/dagger --args=./cmd/dagger entries 

Error: response from query: input: golang.build resolve: call function "Build": process "/runtime" did not complete successfully: exit code: 2

Stdout:
marshal: json: error calling MarshalJSON for type *dagger.Directory: input: contai...
vivid lintelBOT
ornate vigilBOT
#
DarkHeros09
[dagger] New discussion #12208: 🐞 go opentelemetry build errors in v0.13.0

What is the issue?

Hi

I'm getting this error during test

Error: ../../../go/pkg/mod/dagger.io/dagger@v0.13.0/telemetry/exporters.go:92:23: cannot use log (variable of type "go.opentelemetry.io/otel/sdk/log".Record) as *"go.opentelemetry.io/otel/sdk/log".Record value in argument to e.OnEmit

Dagger version

dagger v0.13.0

Steps to reproduce

No response

Log output

No response

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12208: 🐞 go opentelemetry build errors in v0.13.0

cc @vito

I think this happens because we pin the OTEL logging, but sometimes the base OTEL stack can upgraded (e.g. if using a parent go.mod, or dependabot decides to do it, etc) because we don't pin it.

Should we just pin all of OTEL for modules? Sucks, but given how unstable the upstream APIs are, I'm wondering if we should, as suggested here: https://github.com/dagger/dagger/pull/8234#discussion_r1732585400

#
vito
[dagger/dagger] New comment on discussion #12208: 🐞 go opentelemetry build errors in v0.13.0

@jedevc We already pin this dependency for modules, but the issue still remains for non-module use cases, which seems to be the situation here considering the error path (../../../go/pkg/mod/dagger.io/dagger@v0.13.0/telemetry/exporters.go).

For anyone hitting this with non-modules, you could copy-paste these replace rules - ugly solution, but it would prevent the dependency from drifting forward into brokenness.

https://github.com/dagger/dagger/blob/06839862e3437c2f16d95c3621e0d7b1f...

#
jedevc
[dagger] New discussion #12209: Building dagger/dagger in a git worktree fails

This is an issue we can easily encounter on the main branch of https://github.com/dagger/dagger.

Create a worktree:

$ git worktree add 
$ cd

Try and get some git metadata:

$ dagger call -m version git head
...
fatal: not a git repository: /home/jedevc/Documents/work/dagger/dagger/.git/worktrees/

This occurs because we look for where .git exists - there's a git file created at .//.git - it's not a directory, and instead contains contents like: `gitd...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12210: Create MIGRATION docs page for instruction on migrating

Similar to how we have a CHANGELOG.md which describes all of the changes, we should start writing a migration guide - when we release breaking changes, we should add entries to describe everything that users may want to know about upgrading.

This should include:

  • For API facing changes, codeblock diffs, with examples for each SDK
  • For SDK facing changes, codeblock diffs, with examples for the affected SDKs
  • For CLI facing changes, a before-and-after

In the first version of thi...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12211: Building dagger fails when alpinelinux.org is down

Overview

When alpinelinux.org is down, building dagger fails. This may also affect other dagger modules that use alpine in the same way we do.

What I did

dagger call -m github.com/dagger/dagger dev

What I expected

A successful build of the latest dagger engine

What happened

Intermittently I get:

[...]
│ ✔ alpine(branch: "3.20.2", packages: ["build-base", "go", "git"]): Alpine! 0.2s
│ ∅ .container: Container! 0.2s
│ ! failed to get alpine...
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12211: Building dagger fails when alpinelinux.org is down

Agreed, this is a little frustrating - however, this is the case for a whole variety of services:

  • Dockerhub (for our use of official images)
  • The go proxy, pythonhosted, npm (anything that our SDKs use)
  • Github (or anywhere our modules are hosted)

There's a few things we can do to improve this generally, but having better caching is really the best first step. Avoiding making as many network requests as possible will be a huge help here.

After that, I think having a concept of "p...

#
helderco
[dagger] New discussion #12212: 🐞 engine: modules can override core functions

You can’t create a module that conflicts with a core type:

❯ dagger init --sdk=go container
Initialized module container in container
❯ dagger functions -m container
Error: get module name: input: failed to get schema: failed to get schema for module "container": type "Container" is already defined by module "daggercore"

But a module doesn’t just create an object named after the module’s name, it also creates a constructor field under Query with the same name.

However, f...

vivid lintelBOT
ornate vigilBOT
#
kpenfound
[dagger] New discussion #12213: lazy blob error with s3 exporter and context directories 
Error: input: goreleaser failed to set call inputs: failed to load contextual arg "source": failed to load contextual directory "/": failed to get dir instance: failed to get blob descriptor: failed to solve blobsource: failed to load ref for blob snapshot: missing descriptor handlers for lazy blobs [sha256:391afdd16053cec967706502836ef1516804a1f0e7550d824c9b08b61eac13b9]

Currently in a broken state. @sipsma let me know what information will help to debug this

vivid lintelBOT
ornate vigilBOT
#
rajatjindal
[dagger] New discussion #12214: cache volume namespacing key

as a follow up of https://github.com/dagger/dagger/pull/8724#discussion_r1848219633, we want to possibly revisit later the key used for namespacing the cache volume.

originally the idea was to use module digest, but that would mean if change the module source, the new cache volume will be created, so as part of https://github.com/dagger/dagger/pull/8724 we are creating cache volume based on modName + modPath

This may bring up its own set of challenges, so keeping this issue to tra...

vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12215: Second to last step is not cached

Problem

Edit: check my last comment below: https://github.com/dagger/dagger/issues/8955. Seems like the original issue can't be reproduced anymore verbatim. However, there seems to be some other variations that still show a similar cache invalidation behavior.

I have the following pipeline:

func (m *Test) Test() *dagger.Container {
	return dag.Container().From("alpine").
		WithExec([]string{"sleep", "5"})
}

~~If I run that multiple times, I'll get a cached output as expec...

vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger/dagger] New comment on discussion #12215: Second to last step is not cached

👋 I can't seem to repro the above issue anymore. However, there's a new variation of this issue which manifested after a user reported a cache invalidation confusion here: #daggernauts message

Here's the new code to reproduce:

// Returns a container that echoes whatever string argument is provided
func (m *Second) Foo(context *dagger.Directory) *dagger.Container {
	return dag.Container().From("alpine:latest").
...
#
jedevc
[dagger/dagger] New comment on discussion #12215: Second to last step is not cached

Good call, that's definitely worth making sure of.

That said, there's still something else going on - the above shouldn't be going through the slow content cache. We should be able to look this up in the fast definition cache, so it seems likely that we're somehow generating subtly different buildkit operations between runs (and if we are, then why aren't we getting the same results)?

#
jedevc
[dagger/dagger] New comment on discussion #12215: Second to last step is not cached

Okay, the issue is here: https://github.com/dagger/dagger/blob/84dc5b3ba37238d957db1025c70dca797d35a055/engine/sources/local/source.go#L152-L152

Introduced in #8818 I think - when we stopped "freezing" blobs of the local source. The dagql IDs are all correct with the MakeDirectoryContentHashed function, but, this doesn't change the LLB. The session: prefix is magical - and is essentially breaking cache here.

The solution will be part of our dagop work - when we port the code out of buil...

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger] New discussion #12217: upstream buildkit bug: `missing providers for lazy blobs`

Creating an issue in our repo so I have somewhere to link to in our codebase pending creation of an upstream fix PR.

Noticed this while working on https://github.com/dagger/dagger/pull/8871

There's a bug upstream where:

  1. Two or more images are the same when uncompressed, but get referenced with different compression (i.e. gzip vs zstd)
  2. One of the images is lazy, but the other isn't
  3. The image is used as the "source" for a cache volume
    • Possible this happens in some other ...
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12219: 🐞 engine crash in telemetry stack

Originally reported by @helderco in https://github.com/dagger/dagger/pull/8812#issuecomment-2447985322

It's possible to crash the engine (v0.13.6) deep in the telemetry stack:

panic: runtime error: cannot convert slice with length 0 to array or pointer to array with length 16
│
│ github.com/dagger/dagger/engine/telemetry.ReexportLogsFromPB({0x287f500, 0xc0b038b3b0}, {0x2873b50, 0xc08ceef800}, 0xc0846a8a80)
│     /app/engine/telemetry/logs.go:35 +0x9df
│ github.com/dagger/dagger...
vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger/dagger] New comment on discussion #12219: 🐞 engine crash in telemetry stack

Definitely seems like this is caused by #8812 in part, but also we should guard against this happening in the engine - SDKs shouldn't be able to crash the engine.

Turns out, that PR was just triggering an issue that exists in main still. It has to do with a test in pytest triggering a log after another test used a connection on its own event loop, and when in a nested session (even without ExperimentalPrivilegedNesting).

I’ve fixed the PR to avoid the warning in the first place, b...

#
gmile
[dagger/dagger] New comment on discussion #12219: 🐞 engine crash in telemetry stack

What seems like a similar issue appears to still exist as of 0.16.2. I'm hitting it like this:

  1. create an empty dir and initialize elixir SDK in it,
  2. run dagger function command several times

Repeatedly running dagger function succeeds most of the times, but occasionally would fail with either of these two errors:

✘ load module 1.3s
! failed to get configured module: Post "http://dagger/query": unexpected EOF
│ ✘ finding module configuration 1.3s
│ ! Post "http://dagger/query": ...
#
jedevc
[dagger] New discussion #12220: Support backgrounding `plain` progress for interactive commands

Today, we don't support dagger shell or .Terminal when using --progress=plain: https://github.com/dagger/dagger/blob/d4d759140e4964437f9468d6537ff7a9fd313837/dagql/idtui/frontend_plain.go#L225-L227

Sometimes this is true - e.g. in many CI environments, the terminal isn't a proper TTY (or doesn't have stdin). But when running with --progress=plain locally, it's totally possible that I'm running this within the context of a full TTY.

We should handle backgrounding the `frontendP...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12221: Integration test consistency

Spinning out from https://github.com/dagger/dagger/pull/8557#discussion_r1808457164, and other similar conversations.

Our test suites have gradually become longer, more complicated, etc - there's a lot of repetition and complexity that can be hugely simplified. Some more concrete ideas:

  • Manual setup of test contain client as using c.Container().From(golangImage).WithMountedFile(testCLIBinPath, daggerCliFile(t, c)), in other places as daggerCliBase.
  • Manual dagger init steps in...
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12224: Improve the Python runtime module's performance

Summary

Look for and resolve cache misses in Python’s runtime module.

Motivation

We’ve been steadily improving Python’s performance, with the introduction of uv:

But some of the pipeline improvements have yet to be properly measured and fine tuned. They were rushed out mostly on intuition. This was flagged in the [first issue](https://gi...

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger/dagger] New comment on discussion #12224: Improve the Python runtime module's performance

I’ve been doing bits of progress on this over time so I should give an update.

The Python SDK runtime module uses a private field to store data while “introspecting” a module’s contents in order to support configuring the setup of the module, without the need for forking the SDK […] Could be affecting reliability and performance, so it might be a good time to refactor Python’s discovery mechanism to not store Dagger types in a // +private field in order to avoid any issues.

This has...

#
helderco
[dagger] New discussion #12225: ✨ core: add essential env vars for telemetry in runtime containers

What are you trying to do?

There's a few env vars that SDKs need to set manually, but they're essential to telemetry and should be provided by the engine automatically:

OTEL_LOGS_EXPORTER=otlp
OTEL_METRICS_EXPORTER=otlp
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_INSECURE=true
OTEL_EXPORTER_OTLP_METRICS_INSECURE=true
OTEL_EXPORTER_OTLP_LOGS_INSECURE=true
OTEL_EXPORTER_OTLP_TRACES_INSECURE=true

Why is this important to you?

I want to simplify SDKs as much ...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12226: Secret scrub OTEL logs

Currently we use the secret scrubber just for the results of withExec - this is neccessary so we don't end up accidentally caching secrets in the output.

However, this isn't entirely sufficient - secrets can still appear in the logs, for example by calling dagger call get-secret plaintext (where get-secret is a field that returns a Secret). Since these logs are uploaded to dagger cloud, we should censor all the logs.

This will additionally have the effect of scrubbing the cli ou...

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12226: Secret scrub OTEL logs

100% agree with scrubbing this from the OTEL logs and agree with scrubbing it from CLI output other than the one case where a user explicitly asks for the CLI to output the plaintext secret (e.g. dagger call get-secret plaintext).

  • I think we should have an exception for that case since it seems like a valid use case for dagger to support telling users the values of secrets created by modules. E.g. a module that generates certs for you would likely model private keys as Secrets, but it ...
#
helderco
[dagger] New discussion #12227: Resolve terminology ambiguities

[!note]
This is my own brainstorming epic, just putting it out there to allow me to reference it in discussions. It also provides an overview on the subject, allows for tracking and contributions to the conversation, but should create separate issues to facilitate discussion and tracking progress on specific items.

Motivation

We have a problem with some terminology. When Modules were being developed, we used the same names for things that have a more expanded meaning in that conte...

vivid lintelBOT
ornate vigilBOT
#
M-Pixel
[dagger/dagger] New comment on discussion #12227: Resolve terminology ambiguities

Regarding entrypoint vs entrypoint vs entrypoint:

ENTRYPOINT is decided for you. OCI defines this.

"The code in an SDK that’s used as the entrypoint for the “runtime container” (for registering and invoking Module Functions)" is an interesting case, as it does two things (registering and invoking). I think no matter which programming runtime is used, it makes sense for both things to be in the same "program", as this program must ask Dagger something to know if it should be resist...

#
M-Pixel
[dagger/dagger] New comment on discussion #12227: Resolve terminology ambiguities

So, for the most part I agree with the suggestions, but...

  • The "terminology to distinguish between “module vs non-module” support is "CLI". You can use the same exact library inside and outside of a module. The one that's outside might benefit from some added functionality for running the engine if it isn't already running, all of which is done via the CLI, so could call that additional functionality a Dagger CLI Client.
  • Instead of /sdk/lang/.dagger I would suggest `/sdk/lang/modu...
#
M-Pixel
[dagger/dagger] New comment on discussion #12227: Resolve terminology ambiguities

Regarding SDK disambiguation:

The Java SDK is something that Oracle publishes. It includes the Java runtime, and other tools (altogether a Kit) necessary for the Development of Software.

The Dagger Java SDK or Java Dagger SDK ("directory that has both things") is likewise a kit of multiple things for developing software, including (code that retrieves) the Java SDK.

What you refer to as the "runtime module" is not a runtime, but it is a module. It might retrieve a...

#
jdno
[dagger] New discussion #12228: 🐞 Rust SDK fails to export container and throws invalid type error

What is the issue?

I am trying to build a container with the Rust SDK and export it afterwards. Building the container works fine, but when trying to export it I get an error that seems to point to mismatched types somewhere in the API layer.

Unpack(Deserialize(Error("invalid type: boolean `true`, expected a string", line: 0, column: 0)))

Dagger version

dagger v0.13.0 (registry.dagger.io/engine:v0.13.0) darwin/arm64

Steps to reproduce

First, create a new Rust cra...

#
wingyplus
[dagger/dagger] New comment on discussion #12228: 🐞 Rust SDK fails to export container and throws invalid type error

The export api on 0.11.9 retrurns boolean while the generated api returns a string. It looks like that the sdk forgot to bump the engine version before publishing to crates registry, cause the api broken when use it together with auto provision engine. I believe that the issue is disappear if running it with dagger run.

The issue is disappear on the main branch, the engine version is up-to-date now.

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
#
kjuulh
[dagger/dagger] Pull request opened: #12229 feat: add rust module sdk

I've gone ahead and updated the rust sdk to now optionally include module support. Along with a few bug fixes (Void) etc.

It has been added as stable, and supports both being used, or using other functions. I did have some limited testing because it seems when modules are added externally they are pulled from git, and as I don't have a commit on main yet, it couldn't be loaded by the dev engine.

All in all, fairly minimal changes given the circumstances.

syntax is done using proc_mac...

ornate vigilBOT
#
marcosnils
[dagger] New discussion #12230: ✨ Enable ability for Dagger functions to return an error and a response

What are you trying to do?

Problem

Dagger functions can currently return an error OR any other value but they can't do both at the same time. This makes it impossible to implement pipelines steps that may fail (and ultimately should fail the pipeline), but always yields some report. Typical example: test or lint results.

ref: https://discord.com/channels/707636530424053791/1275216033862647890

https://github.com/dagger/dagger/issues/8275

Why is this important to you?...

vivid lintelBOT
ornate vigilBOT
#
vito
[dagger/dagger] New comment on discussion #12230: ✨ Enable ability for Dagger functions to return an error and a response

Half baked thought: we actually don't even have a nice way for functions to return errors at the moment.

If you return an error in the Go SDK that just bubbles up into a fmt.Println(os.Stderr, err.Error()) and os.Exit(1) in the runtime entrypoint. which just turns into an ugly error message in the trace, even if you returned a nicely formatted message.

If we changed that to something like function.ReturnError(…) similar to how we register function return values, that might be a bu...

#
jedevc
[dagger/dagger] New comment on discussion #12230: ✨ Enable ability for Dagger functions to return an error and a response

Getting back to your original use case ("I want to attach to an errored container with terminal") we might need some sort of syntax for "capture an error and continue the chain from it".

So if you ran this, and it failed, we could print the error out nicely

I kinda don't hate the ^ :laughing:

But, I do think that having some "type" of what the returned value is is important. It's not important for the error to have any specific type, agreed that that could be an interface...

#
vito
[dagger/dagger] New comment on discussion #12230: ✨ Enable ability for Dagger functions to return an error and a response

A bikeshedding opportunity emerges:

MVP API:

type Query {
  # Construct an Error value to be returned by a function.
  error(message: String!): Error!
}

type Error {
  id: ErrorID!
  # A brief description of the error.
  message: String!
}

type FunctionCall {
  # Return an error from the function.
  returnError(error: ErrorID!): Void
}

Quoting for convenience:

Broader questio...

#
jedevc
[dagger/dagger] New comment on discussion #12230: ✨ Enable ability for Dagger functions to return an error and a response

One issue I can forsee if we allow both errors and values returned - this pattern won't fit neatly into every language. For go, where v, err := doThing() is a very common pattern, it's fine, but if an SDK is using exceptions instead (like Python does), then it's not particularly easy to follow the control flow - the function throws an exception, and the value never gets assigned.

I think I prefer @vito's suggestion of rich error types - I really really like the idea of being able to ta...

#
vito
[dagger/dagger] New comment on discussion #12230: ✨ Enable ability for Dagger functions to return an error and a response

@TomChv I like the last option the most - assuming that's the closest to how errors would normally be used in TypeScript.

Same for Python I figure; each SDK runtime should catch errors/exceptions in the native manner and turn them into rich dagger.Error function return values.

@jedevc Hrm, I don't think we need to worry about conflicting return types - to me, it's not "return type V on success and type E on error", it's "return type V, or return a GraphQL error" - GraphQL error...

#
mgrinshpon
[dagger/dagger] New comment on discussion #12230: ✨ Enable ability for Dagger functions to return an error and a response

This significantly hampers adoption of dagger for teams using gitlab - especially since they're the other major game in town after github's most recent rugpull.

Specifically, in gitlab, regardless of errors or types, gitlab wants reports to push various specially formatted files to given artifact directories.

dagger.cloud, however, has these lovely red spans that ...

#
jedevc
[dagger] New discussion #12231: Global configuration for dependency modules

Let's pretend I've created a generic reusable module, golang - it can build, test, lint, generate docs, etc, everything I want a module to do.

I want to install this module as a dependency to my project. Because golang is a good module, it even supports setting the version in the constructor.

But there's an issue with this - if I want to call golang at multiple points in my module, and I need to pin the version everywhere, today I have a few options:

  • Specify the version e...
#
pjmagee
[dagger] New discussion #12232: 🐞 dagger install fails with local modules on Windows

What is the issue?

I spent quite some time today trying to replicate Solomons example of Gha module with submodule examples folder to have examples and show them on the daggerverse. I ran into a lot of issues and was confused with the output of the CLI tool, i was trying to follow as close as possible solomons Gha module. It turns out the missing piece of the puzzle was not doing git init in the main parent module!!

I managed to get this problem installing local modules but only o...

vivid lintelBOT
ornate vigilBOT
#
wingyplus
[dagger/dagger] New comment on discussion #12232: 🐞 dagger install fails with local modules on Windows

I can reproduced on Linux (with WSL) as well:

wingyplus@WINGYMOMO ~/s/g/w/i/r/e/example-case-1> dagger install ../..
Setup tracing at https://dagger.cloud/traces/setup. To hide: export SHUTUP=1

✔ connect 1.7s
✔ cache request: mkfile /schema.json 0.0s
✔ load cache: mkfile /schema.json 0.0s
✔ cache request: blob://sha256:bd16df95d81ec01ce4a9a69fb1278e02b80d395b6bac24e7223f51936d8ef3b9 0.0s
✔ load cache: blob://sha256:bd16df95d81ec01ce4a9a69fb1278e02b80d395b6bac24e7223f51936d8ef3...
#
pjmagee
[dagger] New discussion #12234: Private modules over SSH not supported on windows callers

What is the issue?

private-modules don't work with the Dagger Windows binary release, because Windows does not recognise export SSH_AUTH_SOCK. This is because on Windows, SSH is handled through a system wide OpenSSH pipe. \.\pipe\openssh-ssh-agent.

I think both documentation needs to reflect this and a discussion around features supported or unsupported with the different dagger binaries is needed.

Right now, I have not been able to confirm a working solution right now wit...

vivid lintelBOT
ornate vigilBOT
#
grouville
[dagger/dagger] New comment on discussion #12234: Private modules over SSH not supported on windows callers

Hey Patrick,

Took the time to dig a bit:

  1. Buildkit does support npipes, Dagger should be able to, no reason we wouldn't. The current primitives are not there yet though, on our end: https://github.com/moby/buildkit/pull/2127/files
  2. I'll dig a way to have a workaround with the WSL, to see if current implementation could be workaround'ed in the meantime

Then, it's a prioritization on our end:
I have seen that https://github.com/git-ecosystem/git-credential-manager is supported on a...

#
pjmagee
[dagger/dagger] New comment on discussion #12234: Private modules over SSH not supported on windows callers

FYI, I had some real pains trying to get something working in WSL. I do not store my SSH keys on my devices, I am always using 1Password where I am able to. 1Password integration into WSL is using some special interop to execute ssh.exe from Windows OS but this does not use the SSH_AUTH_SOCK. My attempt was to try and install Linux distribution of 1Password into WSL, however this turned out to be really painful and did not work properly for a few reasons, some to do with GUI issues and ws...

vivid lintelBOT
ornate vigilBOT
#
vrischmann
[dagger] New discussion #12237: 🐞 different behaviour regarding io_uring between docker run and dagger

What is the issue?

Hi,

I'm writing an application that uses the io_uring Linux feature.

It looks like io_uring is not supported by default because I'm running into permission issues executing the io_uring syscalls:

queue_init: Operation not permitted

However this can be solved by executing with InsecureRootCapabilities:

WithExec([]string{"io_uring-cp", "/foo", "/home/vincent/foo"}, dagger.ContainerWithExecOpts{
    InsecureRootCapabilities: true,
}).
...
vivid lintelBOT
ornate vigilBOT
#
vrischmann
[dagger/dagger] New comment on discussion #12237: 🐞 different behaviour regarding io_uring between docker run and dagger

Also, some additional information: I just got back on a Linux machine and tested the same docker run command and it doesn't work, I get the same Operation not permitted error; I have to use --privileged to make it work.

I think maybe something is different with Docker Desktop on macOS which make the container privileged by default or something. This might not be an issue with Dagger after all.

#
ielsharef
[dagger/dagger] New comment on discussion #12236: core(private-modules): ssh-agent forward seems to break when multiple keys are loaded

Hi there, I believe the issue I'm running into is related to this. I'm calling a private dagger module that takes as input a url to another private GitHub repo. The input type is *dagger.Directory since I'm looking to clone that second private repo for use by my private module.

Dagger version
dagger v0.16.1 (registry.dagger.io/engine:v0.16.1) linux/arm64
Go SDK

Steps to reproduce

  1. Create 2 private repos; one will contain the dagger module and the other will be the repo that's passed as an...
#
shykes
[dagger] New discussion #12238: One binary

Problem

Today Dagger is split in two components - a CLI and an OCI image - which are built and packaged separately, but should not be.

These components are tightly coupled: they are only usable together; must be upgraded together; the communication protocol between them is private and undocumented; and the functional boundary between them is not settled. The only reason for packaging them separately is that it's easier for us. But the price we (and our users) pay elsewhere may not be...

vivid lintelBOT
ornate vigilBOT
#
grouville
[dagger/dagger] New comment on discussion #12236: core(private-modules): ssh-agent forward seems to break when multiple keys are loaded

Hi there, I believe the issue I'm running into is related to this. I'm calling a private dagger module that takes as input a url to another private GitHub repo. The input type is *dagger.Directory since I'm looking to clone that second private repo for use by my private module.

Dagger version dagger v0.16.1 (registry.dagger.io/engine:v0.16.1) linux/arm64 Go SDK

Steps to reproduce

  1. Create 2 private repos; one will contain the dagger module and the other will be the repo that...
#
helderco
[dagger/dagger] New comment on discussion #12239: ✨ Support multi line `+ignore` directive

I asked for this here:

Feedback:

I need to check how I can parse that but that must be possible.

Hm, you want parsePragmaComment. This is tricky, currently it's all assumed in one line with a regexp.

Being able to do opening/closing braces means the language is no longer regular, so we won't be able to use regexp for it. I'm not confident there's a simple way to do it (but I'm happy to do it in a...

#
liamjones-pw
[dagger] New discussion #12241: 🐞 Service binding setting incorrect IP in `terminal` call

What is the issue?

The service binding is setting 0.0.0.0 in /etc/hosts instead of the correct service address.

Additional notes on the environment:

  • M3 MacBook Air
  • Docker via
$ colima version
colima version 0.6.10
git commit: 12c0c83

runtime: docker
arch: aarch64
client: v27.1.1
server: v27.0.3

$ colima status
INFO[0000] colima is running using QEMU                 
INFO[0000] arch: aarch64                                
INFO[0000] runtime: docker...
vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12243: Cache function calls by default

In the interest of being able to ship something quickly and then iterate, I think it's worth considering starting with TTL settings as being defined as properties of the function being invoked (i.e. by the author, rather than the caller).

Starting there is relatively straightforward implementation-wise. You could do it without having to go all the way into lock files or anything like that; could just re-use existing cache mechanisms (just need to scope cache to a time interval in the same ...

vivid lintelBOT
ornate vigilBOT
#
wingyplus
[dagger] New discussion #12248: ✨ elixir: add guard to required argument

What are you trying to do?

I would really helpful when guard every required arguments to make sure the user pass the correct data type.

Idea in my mind:

Example 1:

defmodule Dagger.Container do
  def with_env_variable(name, value) when is_binary(name) and is_binary(value) do 
    ...
  end
end

Example 2 - struct:

defmodule Dagger.Container do
  def with_secret_variable(name, %Secret{} = secret) when is_binary(name) do 
    ...
  end
end...
#
kminehart
[dagger/dagger] New comment on discussion #12247: How to scale Dagger in production?

At Grafana we use this tool we built to build a huge dagger pipeline.

If you want to see a "huge" dagger pipeline that runs in production feel free to try it out! code here: https://github.com/grafana/grafana-build

docker run --rm -it --entrypoint=/bin/dagger -v /var/run/docker.sock:/var/run/docker.sock grafana/grafana-build:latest \
  run --silent /src/grafana-build artifacts \
    --github-token=$(gh auth token) \
    --go-version=1.21.5 \
    --ubuntu-base=ubuntu:22.04 \
  ...
#
shykes
[dagger] New discussion #12251: Compute Drivers

This issue was previously named "engine drivers", but "compute drivers" is proving more clear.

Problem

The Dagger CLI has a builtin “compute driver”: a software interface which allows it to install, run and manage a the Dagger Engine as a container. This compute driver cannot be customized or swapped out: it always uses the docker CLI. Users who need to customize how or where the engine container is run, simply cannot use it. Instead they must manage the engine container themselv...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12251: Compute Drivers

Potentially it's this simple:

  • CLIs create dagger engines, with a timeout parameter
  • After the engine has exceeded a timeout without a new connection, it shuts itself down (this stops it's own container)

However, we probably want to allow each engine driver to set the "timeout". e.g. for local docker instances, there's no harm in having this be quite long, but for a cloud-compute backend, where each instance could be costing money, we'd probably want to have this set lower (minutes ins...

#
marcosnils
[dagger] New discussion #12255: 🐞 grpc connection closed when using podman + wsl2

What is the issue?

When using podman 5.1.2 + wsl2 and trying to do dagger init on an already existing codebase, Dagger fails with a grpc client connection closed error.

Discord thread: https://discord.com/channels/707636530424053791/1267572025791090759

note: I have the suspicion that this might be related to some podman client grpc timeout issue given that on every test that I saw failing, the error was always failing after around 30s the pipeline started consistently. I've ...

#
TomChv
[dagger/dagger] New comment on discussion #12253: Engine & CLI tests are reliable & fast

Thanks for this amazing issue @gerhard

I think https://github.com/dagger/dagger/pull/8185 should also be include to the effort and then spread globally to other test suite.
Moving TypeScript related test from TestModule into its own suites had multiple benefits:

  • Actually let me tests only what I need locally (because as we'll know, running the full test suite isn't possible)
  • Let me dig more into what happens with each tests and identify the slowest to do some tests.

I thin...

vivid lintelBOT
ornate vigilBOT
#
jiridanek
[dagger] New discussion #12258: /dev/.buildkit_qemu_emulator ... : did not complete successfully: failed to copy xattrs: failed to set xattr "security.selinux" on /tmp/buildkit-qe...

What is the issue?

18  : Container.stdout ERROR [6m19.4s]
18  : ! process "/dev/.buildkit_qemu_emulator /bin/sh -c /ods-ci/ods-ci/run_robot_test.sh" did not complete successfully: failed to copy xattrs: failed to set xattr "security.selinux" on /tmp/buildkit-qemu-emulator2534885165/dev/.buildkit_qemu_emulator: operation not supported

Dagger version

dagger v0.12.3 (registry.dagger.io/engine) darwin/arm64

Steps to reproduce

I have the following Python script

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12261: Traces: function error message not shown

Context

In Traces, function calls are shown with information about their status:

  • If the function returned an error, it's shown in red
  • The error message is displayed in a special box - at least in theory

Problem

The error message shown in the box is not the one actually returned by the function. Instead, it's always the same generic, and useless, message:

`call function "Foo": process "/runtime" did not complete successfully: exit ...

#
shykes
[dagger] New discussion #12263: dagger doc

Problem

Dagger has no equivalent to go doc: an easy way to get API documentation for a module's available types and functions.

There is dagger functions, and dagger call --help, but it's not as complete or useful.

Solution

Implement a command-line equivalent of Daggerverse module docs & core API reference docs. Possibly make them call Daggerverse to reuse existing docs generation capabilities?

#
solita-petterizit
[dagger] New discussion #12265: 🖐️ Quickstart issues with Go (`dagger init` + `dagger call`)

What happened? What did you expect to happen?

Hey! I'm testing out Dagger for the first time and followed the quickstart guide.

When I ran dagger init --sdk=go in the root of my existing repository (not hello-dagger), it generated a dagger Go module:

$ tree dagger
dagger
├── .gitattributes
├── .gitignore
├── dagger.gen.go
├── go.mod
├── go.sum
├── internal
│   ├── dagger
│   │   └── dagger.gen.go
│   ├── querybuild...
#
nipuna-perera
[dagger] New discussion #12266: ✨ Passing auth to Dagger commands can become tedious

What are you trying to do?

Many commands that I run especially in a corporate environment require authentication of some sort. Typically this is username/password but sometimes it could be a certain file or token too.

Why is this important to you?

When using Dagger modules it's typical to split certain functionality into it's own functions and each function can end up requiring to consumer to enter credentials. Examples:

  • For a Maven module I'd need compile, test, deploy...
vivid lintelBOT
ornate vigilBOT
#
solita-petterizit
[dagger/dagger] New comment on discussion #12265: 🖐️ Quickstart issues with Go (`dagger init` + `dagger call`)

I tried to start from scratch with version 0.11.9 and had the same exact issue, although a bit clearer logs:

$ dagger-0.11.9 call echo --stringArg=pong
✘ initialize 2.2s
! main object not found
  ✔ Module.initialize: Module! 0.9s
    ✔ exec go build -o /runtime . 0.8s # <-- This was not visible before

Using dagger query with examples from version 0.9 CLI docs do work fine.

After manually adding the r...

#
levlaz
[dagger] New discussion #12268: 🐞 TUI appears frozen when pulling large image and using `Terminal`

What is the issue?

This was reported by Nipuna in discord[1]

The terminal command seems to kick into "pre" terminal mode before the from image is pulled. So while the image is being pulled the TUI freezes on my screen.

I expected to have access to the TUI until at least the image for the corresponding container is pulled so I can watch that progress and know that it's being pulled. Pulling a large image can take time and it feels like everything is frozen while it does.

[1] ht...

#
pjmagee
[dagger] New discussion #12269: ✨ Confirm or Authorise a Dagger command with a Prompt

What are you trying to do?

Have better control in place for when a dagger pipeline should execute, similiar to a --dry-run of sorts, however the user must provide a input into a tty before dagger would execute the pipeline function.

Why is this important to you?

CI platforms usually have controls in place prior to pipeline stages being executed. This could be user groups, form values or manual stages that need pressing. With Dagger functions being seen as an alternative or ...

#
levlaz
[dagger] New discussion #12271: 🐞 docs: h1 and h2 have the same size

What is the issue?

While reviewing a PR I noticed that H1 and H2 appear to have the same font-size, looking in chrome dev tools I see some conflicting rules for h2, it looks like its being set to 2rem instead of using the variables.

h1  font-size: var(--ifm-h1-font-size);

.markdown>h2 {
    --ifm-h2-font-size: 2rem;
    margin-top: calc(var(--ifm-h2-vertical-rhythm-top)*var(--ifm-leading));
}
h2 {
    font-size: var(--ifm-h2-font-size);
}

It's not clea...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12275: Smarter export

Problem

In theory Dagger is perfect for generating code or docs. In practice, the logic for exporting files back to the client filesystem is simplistic and brittle, which makes the experience awkward for those use cases.

Limitations of export

  1. User must always specify the target path to write to. For example dagger call codegen -o .
  2. The target path is relative to the working directory, which means the command must be changed to match the current workdir - this makes ...
#
marcosnils
[dagger] New discussion #12279: 🐞 dagger init go SDK takes an excessive amount of time in codegen for large Go repositoryies

What is the issue?

coming from: https://discord.com/channels/707636530424053791/1248011149971292331

summary is that if you do a dagger init --sdk go in a large go codebase, the first cold codegen will take an excessive amount of time.

dagger could trace: https://dagger.cloud/marcos-test/traces/a996535073617e245051590c94cbb6e8

image

cc @jedevc @sipsma

note: this doesn't...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
sagikazarmark
[dagger/dagger] New comment on discussion #12284: Private modules

Yes, please!

Daggerverse should hide already visible modules.

I wonder if it would make sense to create a daggerverse specific section in dagger.json for future settings.

Private may also be a misleading name: my Open Source project CI isn't private, it's just not reusble. Same for my test modules.

So I'm thinking daggerverse.publish = false.

#
marcosnils
[dagger] New discussion #12285: 🐞 Dagger doesn't fail when using `kube-pod` scheme if `kubectl` is not present

What is the issue?

While using the kube-pod scheme in the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable, if kubectl is not present where the dagger CLI is running, the CLI will just keep trying to connect to the engine indefinitely.

ref: https://discord.com/channels/707636530424053791/1245072471481385051

My assumption is that we're not handling this specific error upon connection and we're probably retrying indefinitely as part of the backoff retry connection strategy .

D...

#
jedevc
[dagger] New discussion #12287: Services: stop policies

When starting a service with Service.start, it's expected that the user will call Service.stop to stop the service. The user can then choose to use SIGTERM/SIGKILL, etc.

The problem is - this model works poorly with modules - if a module function returns a Service, then how can the caller know the details of this? There's a real issue if this ends up called by the cli with Service.Up - when CTRL+C is send, the service is killed (the default if the user doesn't call SIGTERM)...

vivid lintelBOT
ornate vigilBOT
#
sagikazarmark
[dagger] New discussion #12288: 🐞 Inconsistent file mounting behavior when name isn't specified

What is the issue?

I'm supposed to be able to do this:

dag.Container().WithFile("", file)
 
dag.Directory().WithFile("", file)

When no path is specified, the file should be mounted in the current directory with it's current name.

Unfortunately, Dagger behaves differently in different scenarios which makes mounting files like this unreliable.

(The unfortunate consequence is every higher level API that allows mounting a file in a lower level container requires a second name p...

vivid lintelBOT
ornate vigilBOT
#
kpenfound
[dagger] New discussion #12291: DAG loses concurrency with module chaining

When chaining module functions from the CLI, it appears that the expected DAG concurrency is lost. Here is an example:

Reproduced with Dagger v0.11.3

package main

import (
	"context"
	"fmt"
	"math/rand"
)

type MyModule struct{}

type Job struct {
	*Container
	*Directory
	Key string
}

type Jobs struct {
	Jobs []Job
}

func (r *MyModule) JobGroup(
	ctx context.Context,
	// +optional
	sync bool,
) Jobs {
	jobs := Jobs{Jobs: []Job{
		r.echoAndSleep(5),
	...
#
sipsma
[dagger] New discussion #12293: TUI: hide `Void` from output

This came up while working with someone on an issue, just echoing what they brought up.

Showing Void as the return type in the TUI output is confusing. The word itself is kind of ambiguous from a end-user perspective (what does that mean? does that mean it did nothing? etc.), especially since it's just an implicit return type and hidden from code they write.

Seems like we could just drop it from the TUI and show no return type (which is what it means) and instead rely on the green che...

vivid lintelBOT
ornate vigilBOT
#
gerhard
[dagger] New discussion #12294: 🧐 `dagger call --source=.:default test all` is timing out locally - inconsistent behaviour

What is the issue?

This is an uber issue that we have been tackling in various places, including private Discord threads. It is meant to centralise & summarise the learnings. Expect heavy editing.

The command that we are running:

dagger call --debug --source=".:default" test all 2>&1 \
| ts \
| tee engine.test.all.txt

# ts is part of moreutils 👉 https://manpages.debian.org/testing/moreutils/ts.1.en.html

The root of the problem is inconsistent behaviour wh...

#
sipsma
[dagger] New discussion #12295: Storage Drivers

This has been discussed on various internal trackers and work has been started, converting to a public issue now.

Goals

Primary: Make the cache storage of the Dagger Engine pluggable and configurable, including both local and remote cache

  • Layer caching - Users should be able to get the same seamless remote caching enabled by Dagger Cloud but with their own storage locations (as opposed to the ones created by Dagger's infra).
    • The underlying goal being that users who can't a...
vivid lintelBOT
ornate vigilBOT
#
SKalt
[dagger] New discussion #12297: 🐞 Module names with single-dotted characters cause "main object not found"

What is the issue?

I have a go module with a weird name with single dotted characters in it (e.g. "m.a.s.h"). I was running through the setup script when I encountered a weird error, "unable to find main object". I determined this error only occurs when there are single-dotted characters in the name field of dagger.json. I suspect the culprit is a mismatch between the arguments informing the [generated go struct name](https://github.com/dagger/dagger/blob/1bc2cffabc80e614afea9dc0155a0e...

vivid lintelBOT
ornate vigilBOT
#
fredleger
[dagger/dagger] New comment on discussion #12297: 🐞 Module names with single-dotted characters cause "main object not found"

seems that is not only with dots

I think i faced this one with the name of my simple python script: hokk2mail
I add to adjust the dagger.json name entry to 'hook_2_mail' instead of the default generated hook2mail to make dagger call correctly the sample pipeline 'container-echo' with a newly dagger init project

running Python 3.13.0 in a virual env and dagger v0.15.3 on MacOS M1

Image

#
sagikazarmark
[dagger] New discussion #12300: Cannot write (or read?) from temp mounts

What is the issue?

After writing to a temp mount in a container, the mounted directory shows up as empty, but there is no error indicating that the command writing to the mount fails.

When opening up a terminal in the same container, it works.

Dagger version

dagger v0.11.2 (registry.dagger.io/engine) darwin/arm64

Steps to reproduce

 return dag.Container().From("alpine").
  WithUser("root").
  WithMountedTemp("/root/foo").
  WithExec(...
#
ghost
[dagger] New discussion #12301: ✨ An option to have a more verbose help output for a function

What are you trying to do?

This request was inspired by the addition of Enum support in 0.12.

@dagger.enum_type
class Severity(dagger.Enum):
    """Vulnerability severity levels"""

    UNKNOWN = "UNKNOWN", "Undetermined risk; analyze further"
    LOW = "LOW", "Minimal risk; routine fix"
    MEDIUM = "MEDIUM", "Moderate risk; timely fix"
    HIGH = "HIGH", "Serious risk; quick fix needed."
    CRITICAL = "CRITICAL", "Severe risk...
vivid lintelBOT
ornate vigilBOT
#
kjuulh
[dagger] New discussion #12303: 🐞 dagger-sdk in rust returns garbage on errors

What is the issue?

On errors the dagger-rust sdk doesn't understand how to parse the errors from the graphql endpoint

Dagger version

dagger v0.10.2

Steps to reproduce

Simply get a rust dagger program to crash

Log output

Error: failed to query dagger engine: http error: Failed to parse response: Error("invalid type: sequence, expected a string", line: 1, column: 211). The response body is: {"errors":[{"message":"resolve: container: withFile: withExec: withEntrypoint:...
#
shykes
[dagger] New discussion #12304: From `Service.up()` to web browser

Problem

When I use dagger call ... up to open a tunnel from a Dagger service to my local system, it's not easy to open one of the service endpoints in my web browser.

  • The URL is not reliably printed on the terminal
  • The CLI doesn't automatically open a browser for me
  • The TUI doesn't give me a clickable link or hotkey to open the browser either

These may not be practical, or a good idea. All I know is that I wish it were easier to go from dagger call [...] up to a browse...

vivid lintelBOT
ornate vigilBOT
#
nrs-status
[dagger] New discussion #12306: 🖐️ Best practices for local registries or passing containers from one module to another in the CLI

I'd like to pass a container from one module to another. I'm currently doing something like this:

TTL_SH_CONTAINER=$(dagger -m github.com/Dudesons/daggerverse/infrabox call terragrunt container \
...
publish --address ttl.sh/terragrunt-$RANDOM)

dagger -m github.com/Dudesons/daggerverse call terragrunt \          
with-container --ctr $TTL_SH_CONTAINER \
...

This definitely works at the moment, but I wish I could keep it local without needing to go through ttl.sh. What wou...

#
sipsma
[dagger] New discussion #12308: ci: workflows don't run on non-main release branches

Hit this while doing the v0.11.9 release, which was the first one we've done on a non-main branch.

Workflows ran fine in PRs and on tag pushes, but there are steps in RELEASING.md that say to wait for workflows on "main" to be green, which doesn't work now because we only run workflows on the main branch + PRs + tag pushes.

This wasn't the end of the world in the previous release, but there are a few workflows that run there which...

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger] New discussion #12311: ephemeral `context canceled` errors

This context canceled error has been reported by users and seen occasionally in CI. Most recent occurrence I saw was here https://github.com/dagger/dagger/actions/runs/9551005799/job/26324343803

That passed on many re-runs and also tried running SDK tests again in commits around it without it happening again.

I've been trying to repro what happens in that elixir SDK check failure locally since yesterday but have not replicated it.

  • To be clear, I don't think this has anything to do wi...
#
jpadams
[dagger/dagger] New comment on discussion #12309: 🐞 Dagger Publish Does Not Work with Tags

This was a result of a change that was made some time back to make Daggerverse URLs only really support full git shas -- with only two ways to get to a url with a full sha:

  1. Full sha: https://daggerverse.dev/mod/github.com/levlaz/daggerverse/novu@56559412d83814e93eb6444608119f03e5c1dec7
  2. No sha: https://daggerverse.dev/mod/github.com/levlaz/daggerverse/novu -> `https://daggerverse.dev/mod/github.com/levlaz/daggerverse/novu@56559412d83...
vivid lintelBOT
ornate vigilBOT
#
esafak
[dagger] New discussion #12312: 🐞 Failed run erases the cursor in a terminal

What is the issue?

I ran my CI script dagger run npx tsx ci.mts and it made my block cursor disappear.

Dagger version

dagger v0.11.0 (registry.dagger.io/engine) darwin/arm64

Steps to reproduce

dagger run npx tsx ci.mts with some Docker stuff that fails.

Log output

✘ Container.stderr: String! 47.2s
! failed to compute cache key: write /var/lib/dagger/runc-overlayfs/snapshots/snapshots/36/fs/usr/lib/aarch64-linux-gnu/libx265.so
...
#
sipsma
[dagger] New discussion #12314: Support releasing new SDK version to users without releasing the whole engine

Background: the root cause of https://github.com/dagger/dagger/issues/7583 was lack of pinning deps in the typescript sdk (one of the unpinned ones updated out of band and broke every ts module in all previous engine releases) but it was made much more painful by the fact that we needed to do an engine release in order to fix it rather than just patching the TS SDK.

If we could instead allow users to easily import all of the builtin SDKs as git refs if they want, that would improve this si...

#
dylanrhysscott
[dagger] New discussion #12316: 🖐️ Does Dagger support pipeline operation cache invalidation yet?

What happened? What did you expect to happen?

👋 Have been playing with dagger by building my own module for running Terraform and other scripts. Am running into a weird issue where previous deploy / destroy runs have been cached to the point where Dagger just shows the previous successful run rather than deploying infrastructure again even though it no longer exists. In essence making multiple runs quite tricky. While searching for a solution I found this in the archive as a workaround t...

#
shykes
[dagger] New discussion #12318: Can't return an interface

The interface documentation shows how to define an interface and use it in arguments. But it doesn't show how to use the interface in the return value. I tried, but it doesn't seem to be possible.

It would be very useful to make this possible.

Example:

type Check interface {
 Pass() bool
 Details() string
}
 
type MyCheck struct {}
 
func (c *MyCheck) Pass() bool {
  return true
}
 
func (c *MyCheck) Details() string {
  re...
#
marcosnils
[dagger] New discussion #12319: 🐞 Functions local context seems to be randomly uploaded.

What is the issue?

As a follow-up of this discord thread: https://discord.com/channels/707636530424053791/1227300888050139146 and this previous issue: https://github.com/dagger/dagger/issues/6462, seems like dagger call is randomly uploading the context in consecutive calls.

We confirmed this happens in all v0.10.{2,3} and v0.11.0 versions.

Dagger version

v0.11.0

Steps to reproduce

As described in: https://discord.com/channels/707636530424053791/1227300888050...

vivid lintelBOT
ornate vigilBOT
#
verdverm
[dagger] New discussion #12323: Dagger Function "initialize" times are prohibitively long (0.11.0)

What is the issue?

The startup time for Dagger Functions takes a long time, even on subsequent runs. It seems to be the step where the function is prepared, compiled, dependencies fetched. Watching the logs, once the function code is reached, it goes very quickly

Timings are very inconsistent

  • stark difference with TS (much longer)
  • subsequent runs can take longer than the first (PY)

Dagger version

0.11.0

Steps to reproduce

cd tmp 
➜  tmp git:(mai...
vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger] New discussion #12326: Enable nested exec by default

When nested execs (i.e. execs that can call back to the dagger API) were originally added, they weren't safe since they gave full blown access to the original caller's FS, but that's no longer the case. We can instead make this behavior the default for convenience and instead have a flag for opting out.

There's one last pre-req first, which is handled as part of https://github.com/dagger/dagger/pull/6916. That PR results in nested execs connecting back to the same server (among many other ...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12327: Go SDK: support embedding structs

It would be cool if the Go SDK supported embedded another type. For example:

type *SuperDirectory {
  *dagger.Directory
}

// Extend the core directory type with an extra function
func (dir *SuperDirectory) Hello() string {
 return "world"
}

Since SuperDirectory embeds dagger.Directory, it inherits all the methods of the embedded type. But this embedding does not carry into the corresponding Dagger module. So, even though I can call SuperDirectory.Entries() ...

#
jpadams
[dagger] New discussion #12329: 🐞 Codegen changing causing inconsistent func names across Go SDK module and dependency in case of protocol in name

What is the issue?

tl;dr I suspect that codegen is trying to make certain keywords or acronyms consistent and thus using TCP instead of Tcp, for example. We need to look into that, ensure we know and like the behavior, and document it. My suspicion seems to be confirmed when I use Foo instead of protocol names (see below). This has the effect that a Go SDK function dependency might have a TcpTest function, but to call it in the module that installed the dependency, you'd have to us...

#
jedevc
[dagger] New discussion #12328: Function args conflict with CLI args

Take the following go module:

package main

import (
	"context"
	"strings"
)

type Playground struct{}

func (m *Playground) Foo(ctx context.Context, progress string) (string, error) {
	return strings.ToUpper(progress), nil
}

On call:

$ dagger-dev call foo --progress=plain
Connected to engine ff7f91f8a66b (version 4b825dc642cb6eb9a060e54bf8d69288fbee4904)
✔ connect 0.2s
✘ initialize 3.5s
! flag already exists: progress
  ✔ ModuleSource.resolveFromCall...
#
helderco
[dagger/dagger] New comment on discussion #12329: 🐞 Codegen changing causing inconsistent func names across Go SDK module and dependency in case of protocol in name

I suspect this is due to the "Common Initialisms" used bygqlgen

You're right that it originally came from there, but we've managed our own fork since the very beginning of the Go SDK, in codegen. There's a couple Dagger additions as well:

https://github.com/dagger/dagger/blob/bdce95d0abfa9014a35f5e3962a64afce03d5fca/cmd/codegen/generator/go/templates/lint_name.go#L122-L124

This is the underlying problem of:

Difference is you're ...

#
jpadams
[dagger] New discussion #12330: ✨ Templates for Daggerizing a project from `dagger init`

What: Templates for Daggerizing different project types with a a way to inject them during dagger init.

Why: Daggerizing a project with modules is distinct from creating reusable standalone component modules, and we need an experience for that to speed up bootstrapping a project similar to https://github.com/actions/starter-workflows.

From @kpenfound :

What I propose is to have a built in scaffolder/template system to give you a starting module based on a specific template instead ...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
vikram-dagger
[dagger/dagger] New comment on discussion #12334: Using Services in multiple containers results in network issues

I'm also running into this issue when trying to build a Spring app which uses testcontainers.

Repro

git clone https://github.com/spring-projects/spring-petclinic.git
cd spring-petclinic
dagger init
dagger develop --sdk=go
dagger install github.com/levlaz/daggerverse/mariadb@v0.2.1
dagger install github.com/jcsirot/daggerverse/java@250eb3e0f49da321cca2241fe1e1d017b4143a91
dagger install github.com/shykes/daggerverse/docker@3338120927f8e291c4780de691ef63a7c9d825c0

Modul...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
MarcosRolando
[dagger] New discussion #12344: 🐞 Dagger panics when running in CircleCI pipeline

What is the issue?

I'm migrating my CI from GH actions to CircleCI in order to use ARM executors. Following is the pipeline for both GH Actions and CircleCI. I'm using the Dagger Python SDK. GH Actions pipeline runs fine, but CircleCI instantly panics when the dagger run command is executed.

Funnily enough, when SSHing into the CircleCI instance and running the same command there, the engine Dagger pipeline works just fine. I could not for the life of me find a related issue, and the ...

vivid lintelBOT
ornate vigilBOT
#
TomChv
[dagger/dagger] New comment on discussion #12345: ✨ [Typescript] eslint rule to avoid some common gotchas

Hey @Xiot, I was thinking about this issue recently, I think I could create some custom linter rules in the TS init template to catch some stuff.

I'm not sure how far I can get this done but here's some ideas I have:

Checking function

  • raise a warning if the function is exposed but not documented properly

Checking function argument

  • raise an error if we have an union type because we de not support it for now.
  • raise an error if there's a strict type as arg, we do not ...
vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12349: Go SDK issues with newer go.mod than packaged into engine

This silent failure

What's the failure? That Name is main rather than foobar? Just double checking.

In terms of handling this better, the fact that the go sdk is builtin to the engine is indeed a pretty big wrench here. We do that to avoid hard deps on the internet and to avoid problems in corp envs around need to use registry proxies. I can see some routes to do something like:

  1. use builtin sdk when at user module version or older
  2. fallback to dynamically pulling+building...
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12352: `test:race` job should enable race checker on the engine as well

At the moment, we're only checking for race conditions in the client - but ignoring race conditions in the engine.

https://github.com/dagger/dagger/blob/bf262aca916e91a69b2191582c306c1e3ec7b633/internal/mage/engine.go#L332-L375

This is useful, and it has caught a number of issues in the client, but there are also potentially race conditions in the engine! e.g. see https://github.com/vito/progrock/issues/12 (cc @vito)

Note: running with the race checker is likely going to have a *...

#
jpadams
[dagger] New discussion #12353: ✨ API for export of `Directory` and `File` from `Service` container

What are you trying to do?

Users have asked for the ability to get files out of Service containers so they can capture test reports, or logs, or other generated files. Currently there is no API.

integration test container cannot be a service if we want to extract files like tests reports at the end of a run. If it was possible to extract files from a service that would also work.

Discord references:

#1239817495309713428 message...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger] New discussion #12354: Closing a service should implicitly close all dependent tunnels *first*

This is a fun one. Let's suppose I'm running a dagger engine as engine:

// This works fine :tada:
tunnel.Stop(ctx)
engine.Stop(ctx)

If we swap the order, we block forever, yay!

// This blocks forever!
engine.Stop(ctx)  // we never return
tunnel.Stop(ctx)

This is kinda tricky, but essentially, there is still an open connection for the tunnel open to the server, so the server's Stop/GracefulStop can get stuck. We get stuck trying to do a handshake...

#
vito
[dagger] New discussion #12355: 🐞 Using a repository as both a module ref and a `git://` arg leads to a timeout

What is the issue?

Not sure why this is happening, made my best guess with the title.

When I call my github.com/vito/bass module and pass the repo as an arg, it successfully clones the repo for the module, but the subsequent clone for the arg hangs.

Dagger version

dagger v0.10.0

Steps to reproduce

dagger call -m github.com/vito/bass --src git://github.com/vito/bass unit --packages ./pkg/bass

Don't think the commit matters, but for reference HEAD is currently `...

vivid lintelBOT
ornate vigilBOT
#
sagikazarmark
[dagger] New discussion #12358: ✨ Make secret name optional

What are you trying to do?

I have a function that creates a secret: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/registry-config@b45dbd7448bb967aca4a538af9ce7f042abf0316#RegistryConfig.secret

The current secret API takes a name at the moment: dag.SetSecret("name", "value").

I'm not sure, but my guess is that name is supposed to be unique (ie. to make sure there is no interference between calls to the same function).

Why is this important to you?

Right no...

#
vito
[dagger] New discussion #12360: 🐞 Blobs referenced by private fields in a module object can disappear

What is the issue?

Right now linkDependencyBlobs walks over an object's map[string]any and uses the object's field definitions to convert each any to a HasPBDefinitions so it can find blobs to link to the current session.

The problem is private fields don't actually have a field definition, so we don't know how to convert it to a HasPBDefinitions-able value.

Maybe we need to also register private fields, and just mark them private so they don't show up in the schema?

###...

#
jedevc
[dagger/dagger] New comment on discussion #12358: ✨ Make secret name optional

@jedevc I was wondering about that, but don't we want to avoid caching secrets anyway?

This is debatable but yes! Generally, atm we just inherit buildkit's behavior - only the name of the secret is part of the cache key, the value isn't. So if you change the value, we keep caching, but if you change the name, then we don't.

I'd like to keep something close to this behavior, since otherwise, any step that uses a secret can never be cached (which would be a pretty sad regression a...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
vito
[dagger/dagger] New comment on discussion #12365: 🐞 `dagger up --ports` doesn't bind host address when multiple `WithExposedPort` are present in a container

If I tell port 4571 to skip the health-check it works. Looks like the service isn't binding on that port, and we're missing feedback for that.

package main

type MyMod struct{}

func (m *MyMod) Package() *Service {
	return dag.Container().
		From("localstack/localstack:latest").
		WithExposedPort(4566).
		WithExposedPort(4571, ContainerWithExposedPortOpts{
			ExperimentalSkipHealthcheck: true,
		}).
		AsService()
}
vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12367: 🐞 Exec shim doesn't allow background processes to be terminated successfully

What is the issue?

Trying to execute the following pipeline:

func main() {
    defer dag.Close()
    dag.Container().From("alpine").
        WithExec([]string{"apk", "add", "bash"}).
        WithNewFile("/entrypoint.sh", dagger.ContainerWithNewFileOpts{
            Contents: `#!/bin/bash
sleep 10 &
echo "foo"`,
            Permissions: 0755,
        }).
        WithEnvVariable("BUST", time.Now().String()).
        WithEntrypoint([]string{"/entrypoint.sh"}).
       ...
#
sipsma
[dagger] New discussion #12369: Integrate core types with interfaces

Follow up to https://github.com/dagger/dagger/pull/6254

You can't yet use types from the core api like Container/Directory/etc. to implement a given interface.

You also can't implement core types through the use of interface features.

  • This would be especially useful for use cases @shykes mentioned around e.g. implementing a virtual Directory type in a userspace module
  • This should also be totally backwards compatible. If a userspace module object implements e.g. the `Director...
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
nipuna-perera
[dagger/dagger] New comment on discussion #12370: Add SecretStdin to Container.WithExec options

I typically trust ENV vars for secrets. This is how I use them

secret := dag.SetSecret()

ctr.
WithSecretVariable("mysupersecret", secret).
WithExec(
    []string{"sh", "-c", "helm login --username foo --password $mysupersecret"}
)

You can also clean up the env var after the withExec if you don't care to keep it there beyond the exec. These are well masked in the logs in my experience.

Fix for docs here - #7232

#
marcosnils
[dagger] New discussion #12373: Improve modules quality by examples

Problem

In many cases it's not clear for module consumers how to use them once they find them in the Daggerverse. Even though https://daggerverse.dev has detailed docs about the module functions, their arguments and return types, sometimes modules are designed to be consumed in a particular way which is currently hard to express given the current module documentation features.

For example, looking at this module: https://daggerverse.dev/mod/github.com/jcsirot/daggerverse/java@c591...

#
jedevc
[dagger] New discussion #12374: dagger functions/call/etc should fail if none present

@kpenfound noticed this fun one while migrating to new the source fields:

❯ dagger init --sdk=go --name=playground --source .
Initialized module playground in .
❯ ls
dagger.gen.go  dagger.json  go.mod  go.sum  main.go  querybuilder
❯ rm main.go
❯ dagger functions
Name             Description
container-echo   example usage: "dagger call container-echo --string-arg yo stdout"
grep-dir         example usage: "dagger call grep-dir --directory-arg . --pattern GrepDir"

Eve...

#
sipsma
[dagger/dagger] New comment on discussion #12374: dagger functions/call/etc should fail if none present

I'll confirm this is the case to verify there's no deeper issue here, but yeah I believe this is happening because the engine gives the module's context to the go SDK, it sees no main.go and fills in the default one, then the engine asks that what objects+functions it has and this is the result.

I think a lawyer could probably make a compelling case that this is almost correct behavior 😄 But it is undeniably weird and surprising.

I'd need to think about what the fix would be too. Ri...

#
sipsma
[dagger] New discussion #12375: Cache efficient checkouts of subdirectories of large git repos

Right now when checking out git repos (whether users of our Git API in core or Dagger's internal use for e.g. loading modules), even if only one subdirectory of a git repo is needed, the full repo still needs a clone (to my knowledge).

We use buildkit's Git source implementation, which does jump through hoops to use sparse checkouts and performance optimizations, but it doesn't yet support features in newer versions of git that enable only pulling the files needed for a certain subdirec...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
jshbrntt
[dagger/dagger] New comment on discussion #12377: Support for SSL man-in-the-middle proxies

I'm a little confused as to the progress on this issue.

Is there a work around? Because I've tried provisioning a customized Dagger Engine with the required certificates mounted at /usr/local/share/ca-certificates.

Regardless I'm still getting a tls: failed to verify certificate: x509: certificate signed by unknown authority error.

✘ .withExec(args: ["codegen", "--output", "/src", "--module-source-path", "/src", "--module-name", "ty...
#
rsteube
[dagger] New discussion #12382: dagger functions: json output

What are you trying to do?

  • Expose functions as json in dagger functions.
[
  {
    "Name": "containerEcho",
    "Description": "example usage: \"dagger call container-echo --string-arg yo stdout\"",
    "ReturnType": {
      "Kind": "OBJECT_KIND",
      "Optional": false,
      "AsObject": {
        "Name": "Container",
        "Functions": null,
        "Fields": null,
        "Constructor": null,
        "SourceModuleName": ""
      },
      "AsInterfa...
#
marcosnils
[dagger] New discussion #12384: 🐞 Dockerfile pipelines grow the cache size indefinitely

What is the issue?

While using the support for Dockerfiles in Dagger, each time the pipeline is executed, the disk space grows indefinitely even if all operations are cached

Log output

N/A

Steps to reproduce

Use the following example pipeline

FROM alpine

package main

import (
	"context"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// initialize Dagger client
	client, err := dagger.Connect(ctx, dagge...
vivid lintelBOT
ornate vigilBOT
#
dariusj1
[dagger] New discussion #12385: ✨ <Feedback>

What are you trying to do?

I started looking into Dagger after I saw it on one DevOps Engineers's video in Youtube. I liked the idea: juggling containers, which are supposed to be the same everywhere: cloud, CI runners, localhost, server, etc. And so I started playing with it and I've been playing with it for a few days now. Here is my feedback

  • The Good
    • Everything is in containers. This allows me to have complete control over what's happening where
    • Programming language sup...
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger] New discussion #12393: Zenith: enable built-in SDKs to be usable without hard dependency on the internet

Some users hit roadblocks when the dagger engine has hard dependencies on pulling e.g. container images from dockerhub or other resources from the internet at runtime. This is typically due to stringent corporate firewall rules that require exceptions be made for any unapproved network calls to the internet.

Our builtin module SDKs (Go + Python currently, nodejs soon), currently have varying degrees of dependencies on base images + library dependencies that result in network calls to the i...

vivid lintelBOT
ornate vigilBOT
#
skycaptain
[dagger] New discussion #12394: 🐞 Zenith: Using secrets in SDK runtime modules results in error after re-run from cache

What is the issue?

Since SDKs are modules themselves, it is possible to use a custom SDK. In #6318, a custom SDK is used to install additional Python dependencies from a private repository, with the required credentials passed from a .env file. You can find an example at https://github.com/skycaptain/dagger/blob/python-sdk-runtime-with-env/sdk/python/runtime/main.go.

In this example, `d...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12397: ✨ Daggerverse: modules should display their engine version

Hm, I agree that this is currently an issue, but I don't think the solution is to encourage users to check the engine version in daggerverse.

IMO, if a dagger.json engine version is less than or equal to the engine version, we should always be able to run it - for old versions, we can introduce backwards compatibility to provide an old API for old modules. For newer modules... we could just refuse to run them for now? Something like #7031 but for modules?

That said, that's a ...

#
grouville
[dagger] New discussion #12399: 🐞 dates w/ unix time will break after 2038

Following @sipsma's comment:

Great catch, on a less urgent note we'll also run into problems representing dates w/ unix time after 2038 :-)

This follows a conversation with @JammUtkarsh, who wanted to contribute:

Root cause: "Unix Millennium Bug."

Unix time is the number of seconds that have elapsed since January 1, 1970, at 00:00:00 UTC...

#
sipsma
[dagger] New discussion #12401: Fix module name conflicts with core types

Related discord question: #1190052885686407179 message

Right now a module that has the same name as a core type (e.g. Module, Container, etc.) will eventually result in an error. This is not only due to conflicts in the SDK code (depending on the SDK language) but ultimately also on the graphql schema level.

At a bare minimum, we should error out in dagger mod init if a module has a name that will result in a conflict. T...

#
helderco
[dagger] New discussion #12402: Improve codegen reusability

Summary

Refactor the Go based codegen tooling to make it easier to maintain the codegen feature in a growing list of SDKs.

Background

The Go SDK launched with a custom implementation of GraphQL introspection to generate the Dagger client based on the API schema. This is similar to the GraphQL Code Generator project, but uses Go code rather than Javascript.

Python needed to be released shortly after but the Go introspection wasn’t read...

vivid lintelBOT
ornate vigilBOT
#
vbehar
[dagger] New discussion #12407: 🐞 dagger doesn't work with sysbox

What is the issue?

I've tried to run Dagger in docker-in-docker on top of sysbox, but it doesn't work - because it seems like sysbox doesn't allow Dagger to create a new network namespace.

Also note that sysbox has known and documented limitations: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md

Log output

dagger-engine container logs (in debug mode):

time="2023-12-12T09:19:13Z" level=info msg="detected mtu 1500 via interface eth0"
dn...
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12410: CLI: allow passing sub-directories of a git repo

Problem

When passing an argument of type Directory, the CLI accepts a git remote url (eg. https://github.com/dagger/dagger) which is awesome; but it does not support passing a sub-directory of that git repo (eg. https://github.com/dagger/dagger/docs) which is not awesome.

Solution

Support passing a sub-directory of a git remote as CLI argument to Dagger CLI.

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger] New discussion #12411: Remove need to run shim inside Exec container

We currently run the shim both outside the container (to reconfigure the runc bundle) and inside exec containers, which makes it straightforward to redirect stdout/stderr, serve nested sessions, etc. It's the pid 1 and spawns the actual user process as a subchild.

However, this can cause problems for programs that insist on being pid 1 such as systemd.

We can instead run the shim completely outside the container. This will require at least:

  1. Still handling stdout/stderr file redirect...
#
Scalahansolo
[dagger] New discussion #12413: ✨ Official Docker image for the CLI?

What are you trying to do?

The dagger docs for gitlab / github / etc have each job installing curl followed by installing the dagger CLI to use for a given CI run. It would be much nicer DX, and probably faster to just have an official docker image that can be used that has the CLI already installed?

Why is this important to you?

No response

How are you currently working around this?

No response

#
aweris
[dagger] New discussion #12412: 🐞 In Go SDK, can't return types defined in sub packages

What is the issue?

I'm organizing my code by moving non-Dagger definitions to a sub-package. But Dagger only accepts models from the main package as return types.

Define Everything in main

This example as working as expected

package main

type Foo struct{}

type Bar struct {
	BarID string
}

func NewBar() *Bar {
	return &Bar{
		BarID: "foo-bar-main-id",
	}
}

func (m *Foo) Test() *Bar {
	return NewBar()
}

Using sub package in module c...

#
helderco
[dagger] New discussion #12414: Proposal: Warning for orphan pipelines

Summary

Log a warning on pipelines that were never executed.

Background

This proposal is an alternative solution to Problem 3: Orphaned pipelines in:

Motivation

If a synchronous function is not explicitly called, the entire pipeline is orphaned:

lint()  # never executed

await test().sync()

To avoid making a mistake, we should make this situation more obvious to the user.

Specification

I...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12413: ✨ Official Docker image for the CLI?

@shykes could you elaborate a bit on point 2? I think the engine image probably won't have the CLI embedded, it'll just have the engine as we build it today - I don't necessarily see that changing dramatically, but I guess this also touches on #5583 (like everything else I guess lol).

For 3, for the CLI, I actually think we might be able to do something like distroless? I think we probably just need a little collection of static binaries. For the engine, I would like to unify the GPU/stand...

vivid lintelBOT
ornate vigilBOT
#
kpenfound
[dagger] New discussion #12418: ✨ Option to ignore hosts `.docker/config`

What are you trying to do?

The current behavior is to load the hosts .docker/config into the dagger session so that existing registry authentications can be used. While this is generally a convenience, there are cases where this is not desirable and there should be an option to opt out of this behavior

Why is this important to you?

No response

How are you currently working around this?

No response

#
PumpkinSeed
[dagger] New discussion #12419: GCR second private image pull throws 403 Forbidden

What is the issue?

I have two private images on gcr.io which necessary for run the e2e tests in our system. I created a service account which has permission to pull these images and I created a json key. We are using bitbucket pipelines at the moment and I can pull these images in a way like this:

 services:
    ....
    vault:
      image:
        name: gcr.io/company/vault:latest
        username: _json_key
        password: '$GCR_IMAGE_PULLER_KEY'
    couchbase:
      ...
#
PumpkinSeed
[dagger/dagger] New comment on discussion #12419: GCR second private image pull throws 403 Forbidden

I could resolve the issue by creating separate clients for the calls.

Originally I created one client and I reused it for those calls.

client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))

I modified the code so it creates separate clients and it works just fine.

clientForCouchbase, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stdout))
secretForCouchbase := clientForCouchbase.SetSecret("secretForCouchbase", key)
couchbase, err := clientForCouc...
#
sipsma
[dagger] New discussion #12421: CLI: support passing sensitive `Directories`

Discord ref: #daggernauts message

Previously, we suggested that users mount sensitive directories in "userspace" from their own code: https://docs.dagger.io/guides/457482/create-app-ci-module/#step-6-add-a-function-to-run-the-application-as-a-local-service

But that no longer is viable when just using dagger call.

Off the top of my head thoughts are to either:

  1. Create a new core type like SecretDirectory or some...
#
aweris
[dagger] New discussion #12422: 🐞 types.Map not supported at modules

What is the issue?

Currently, zenith do not support map types, resulting in compilation errors.

Log output

dagger call workflow string
✘ load call ERROR [3.73s]
├ [3.73s] loading module
┃ Error: failed to get loaded module ID: input:1: host.directory.asModule.s
┃ ve failed to load module types: failed to call module "gale" to get funct
┃ ns: failed to get function output directory: process "/usr/local/bin/code
┃ n --module . --propagate-logs=true --introspection-j...
vivid lintelBOT
ornate vigilBOT
#
stobias123
[dagger] New discussion #12424: ✨ Manually set Dagger Cloud run title

What are you trying to do?

I'd like to be able to set the title of my runs in dagger cloud. I execute multiple pipelines per commit, so it's a little hard to find what I'm looking for based on commit title.

Why is this important to you?

I run a (sequential) pipeline that

  1. Builds everything
  2. Runs terraform for each environment

Right now each of the above is a separate dagger call. It's hard to find say, the terraform dev run when it all looks like this

![image](ht...

#
sipsma
[dagger] New discussion #12425: Renaming a module is hard and results in unhelpful error

Right now changing the name of a module involves a manual write to dagger.json and then also updating your code to use the new name for the main module object, which is fairly involved.

We at minimum should give helpful error messages if someone tries to rename the module in dagger.json but doesn't rename it in their code.

It would be really great if it was possible to do the entire thing automatically, but the tricky part is that we'd need to rewrite parts of their module code automati...

vivid lintelBOT
ornate vigilBOT
#
xsxz
[dagger] New discussion #12428: 🖐️ Reading files from both containers and host as an array of bytes, not UTF-8 string

What happened? What did you expect to happen?

I'm trying to extract UNIX utilities from pre-built docker images, like kubectl and helm, using python script. As it is popular approach to cut some time while reusing already existing artifacts, instead of building and storing tar image.

As far as I understand: dagger engine, being written on Go, in which strings are always implicitly encoded in UTF-8, reads all system files as byte arrays by default. While using

host_file = awa...
#
gmile
[dagger/dagger] New comment on discussion #12428: 🖐️ Reading files from both containers and host as an array of bytes, not UTF-8 string

Just stumbled upon this while trying to read binary files using Elixir SDK. Today, on 0.15.1 below code will return an error:

<details>

Mix.install [:dagger]

client = Dagger.connect!()

File.write!("/tmp/test-file.bin", << 1, 2, 3 >>) # <- binary contents

{:ok, contents} =
  client
  |> Dagger.Client.host()
  |> Dagger.Host.file("/tmp/test-file.bin")
  |> Dagger.File.contents()

{:ok, stdout} =
  client
  |> Dagger.Client.container([])
  |> Dagger.Container.f...
#
sipsma
[dagger] New discussion #12431: `dagger call` is tedious with too many flags+args

Somewhat related to https://github.com/dagger/dagger/issues/6112 but a bit more general

Passing a large number of flags and args when using dagger call can become way too tedious to type out by hand. One example would be support for filtering dirs as mentioned here. But really any time a function accepts more than a few flags and/or the arguments to the flag end up being verbose are a problem.

This leaves users ...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger/dagger] New comment on discussion #12432: Make Dagger GHA pipelines more Dagger friendly

I think an actionable first step would be to document our own CI setup with comments and a README, to provide more context. Not just to explain what, but also why.

Especially for the "dagger sandwich" workflows like publish, where we still have lots of yaml spaghetti, but for pragmatic reasons that will resonate with experienced teams. One of them being: "this works well enough with an off-the-shelf github action, and is not worth the effort of porting yet. But that's ok since we can mix da...

#
jedevc
[dagger] New discussion #12434: `platformVariants` is unwieldy in modules

Using platformVariants is extremely unwieldy to use: https://github.com/dagger/dagger/blob/fca9e788f6d605ef9d256beace4656c11d7f6fe5/docs/docs-graphql/schema.graphqls#L196-L201

The idea is that you can construct multiple Container objects, and then compose them to together on Publish/Export to push a multi-platform image.

However, this feels like a weird abstraction - quite often, in our own CI in this repo, we end up calling a single function multiple times with different arch...

#
jedevc
[dagger/dagger] New comment on discussion #12434: `platformVariants` is unwieldy in modules

There was some discussion of this in discord: #maintainers message

Quoting @sipsma:

I don't think we can just have a single container be multiplatform. If you do a withExec then you'd be executing N containers (one for each platform, using qemu emulation when not matching the host platform), each of which have their own independent exitCode, stdout, stderr etc. which just doesn't make any sense at all with the Containe...

#
aweris
[dagger] New discussion #12436: 🐞 if log string contains a emoji TUI leaves an extra empty line after log

What is the issue?

When a log string contains an emoji, TUI logs an additional empty line.

Log output

Steps to reproduce

Print log messages containing emojis.

Example code to produce logs shared in the screenshot above.

package main

type Emoji struct{}

// Print returns a string with an emoji.
func (m *Emoji) Print() string {
	return "😀\n🐷🔑\nA normal string\n"
}

SDK version

DaggerCLI v0.9.2

OS version

macOS 14.0

vivid lintelBOT
ornate vigilBOT
#
rajatjindal
[dagger/dagger] New comment on discussion #12437: `dagger develop` support for re-naming/re-sdk'ing existing module

building on the suggestion from @helderco here, if we change code generator to generate the modules' main.go as follows:

Note: Foo is the name of the module and InbuiltName is a static name which will be constant across all modules.

(InbuiltName is a random name I selected, we will change it to something more meaningful before merging.)

type Foo struct {
	InbuiltName
}

type InbuiltName struct{}
...
vivid lintelBOT
ornate vigilBOT
#
jpadams
[dagger] New discussion #12439: Zenith: `dagger mod use` should work even if old/broken sha currently in `dagger.json` and should update to latest sha if none given

commentary: Though in theory it's possible to have completely different modules at different commits in a single git repo + subpath. That would imply calling dagger mod use github.com/jpadams/daggerverse/drupal could actually be expected to just append the newest commit to the list of dependencies rather than update the existing entry in the list of deps.

But that's probably such an obscure thing to want to do that maybe we should just ignore it and have the behavior be to replace the exi...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger] New discussion #12443: `dagger install` polish

Follow ups from https://github.com/dagger/dagger/pull/6533 around install (and managing dependencies generally):

  • [ ] add support for dagger update to update existing deps
    • Currently this implicitly happens if you dagger install an existing dep at a different version, but it would be better to have an explicit command for it.
    • We could then consider whether we want dagger install to fail if dep at different version already exists or to continue supporting the implicit upgr...
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12443: `dagger install` polish

Do we want to support updating just one dependency or everything in the dependencies list?

dagger upgrade by itself should upgrade all dependencies - but I guess it would be nice to upgrade a single named dependency (similar to how uninstall selects a dep).

I'm thinking that dagger upgrade should just re-fetch the source and potentially update the pin if it's changed since then - but it shouldn't try and modify the source.

So:

  • If a user installs @main, then we re-f...
#
marcosnils
[dagger] New discussion #12446: ✨ Raise a warning if /var/lib/dagger is an overlayfs mount inside the engine container

What are you trying to do?

Whenever the engine is provisioned within a container - default strategy by the SDK's - as highlighted in the operator's manual here (https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#execution-requirements), it's imperative to set /var/lib/dagger as a volume so it doesn't form part of the overlayfs of the container. Otherwise, this will have a severe performance penalty when running pipelines.

We should raise a warning if we...

#
marcosnils
[dagger/dagger] New comment on discussion #12446: ✨ Raise a warning if /var/lib/dagger is an overlayfs mount inside the engine container

A proposal: maybe we could have WithMountedEmpty or something - when run inside dagger using a WithExec we try and have an empty mount, so we could replace our weird cache volumes use:

I like this, maybe use WithMountedCacheEmpty so it's clear that this API is still part of mount cache / volunes? Otherwise, WithMountedEmpty could also be associated to a tmpfs or something else I assume.

#
olee
[dagger] New discussion #12448: Unable to use docker socket in windows

What happened? What did you expect to happen?

Referring to this example, I tried running a compose file through dagger like this:

        const compose = await client.container().from('docker')
            .withUnixSocket('/var/run/docker.sock', client.host().unixSocket('/var/run/docker.sock'))
            .withDirectory('/src', client.host().directory('.'), { exclude: [ 'node_modules', '/ci' ], })
            .withWorkdir('...
#
jedevc
[dagger] New discussion #12449: ✨ Access Service stdout/stderr and exit code

What are you trying to do?

Currently, once a service starts, there's not a ton of interaction points for it. We can stop it... and that's about it.

One key thing I'd like is access to the stdout/stderr of a service. At the moment, these are exclusively streamed to the progrock output, but it would be nice to be able to output them somewhere else.

[!NOTE]
In addition to this, I'd really like to have a way to suppress the default progrock service output. These in combina...

#
dolanor
[dagger] New discussion #12450: Fix Go codegen for required param documentation

Right now, if a required parameter has some documentation, it shows in NodeJS SDK, but not in the Go version.

In this next example, context *Directory doesn't get the documentation from the GraphQL Schema.

TS

export class Container extends BaseClient {
  /**
   * Initializes this container from a Dockerfile build, using the context, a dockerfile file path and some additional buildArgs.
   * @param context Directory context used by the Dockerfile.
   * @param op...
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
jpadams
[dagger] New discussion #12455: Dagger module best practices for naming, args, style, documenting

What are good names for

  • modules
  • functions
  • arguments

How should a module be commented?

  • for short overall description
  • for each function

Optional args and sane defaults preferred.

  • module name: noun
  • function name: verb
  • arg names: abbrev? not abbreviated? optionals where possible
  • version tags
  • return types: lean to core types for user flexibility? avoid engine eval where possible?

Secrets handling.

Interfaces?

Lintable, if possible to ensure style is c...

#
sagikazarmark
[dagger/dagger] New comment on discussion #12455: Dagger module best practices for naming, args, style, documenting

Proposal:

Agree with most of it.

  • abbrev (dir)? or not abbreviated (directory)?

My preference: pick one and call it idiomatic, so I don't have to waste brain power on deciding it. (It also allows others to shame me into doing "the right thing")

Some high level practices I'm interested in (although may not be lintable):

#
jpadams
[dagger] New discussion #12457: Fix help text for `dagger ` subcommands `-m`, `--mod` flags on `query`, `mod`, ... to use ampersands for joining params

today

-m, --mod string        Path to dagger.json config file for the module or a directory containing that file. Either local path (e.g. "/path/to/some/dir") or a git repo (e.g. "git://github.com/dagger/dagger?ref=branch?subpath=path/to/some/dir"). (default ".")

possible fix

-m, --mod string        Path to dagger.json config file for the module or a directory containing that file. Either local path (e.g. "/path/to/some/dir") or a git repo (e.g. "git://github.com/dagger/da...
vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12459: Python: Make library fully type correct

Todo

  • [ ] No mypy warnings[^1]
  • [ ] Add to lint step for CI (lint:typing)
  • [ ] Build with mypyc?
  • [x] Support Pyright (Microsoft's type checker, default in VS Code)
  • [ ] Make sure PyCharm is well supported as well

[^1]: The ones that are leftover represent some annoyances, and don't suggest any bug. I decided to move on to not waste time for now.

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
nicolas-palermo
[dagger] New discussion #12464: ✨ Is there any feature of dagger.Container to detach itself?

I want to run two container in the same python code.

The first container is an foundry anvil, this is a web3 endpoint and should be run indefinitely. The second container is a python image that interacts with the first container.

I've tried many ways to detach the command anvil: anvil &, (anvil &), nohup anvil &, etc

This is the code of the anvil container (container to detach):

  async def run_anvil(self, port:int=8545) -> dagger.Container:
      scripts_dir...
#
shykes
[dagger] New discussion #12465: Deprecate export functions?

Overview

I regularly see developers get confused (sometimes severely) by File.Export, Directory.Export, and Container.Export, to the point of failing to "get" Dagger, and sometimes even giving up entirely.

I think we should consider the possibility of deprecating these functions.

The timing matters, because we are currently discussing a new CLI experience for Zenith (dagger call etc) which might rely on these export functions. So it seems important to discuss the future of these e...

vivid lintelBOT
ornate vigilBOT
#
red8888
[dagger] New discussion #12468: Can I run dagger with docker in docker and use that to support CI environments that don't allow privileged access?

What happened? What did you expect to happen?

This might be a dumb question but can I run this in the official docker in docker image to get around my perms issues?

Bitbucket pipelines don't support enough privilege for dagger to work: https://github.com/dagger/dagger/discussions/1677#discussioncomment-2619008. And I'm not going to run them on-prem which is the workaround.

But could I have my dagger scripts built into the official docker in docker image (https://hub.docker.com/_/do...

#
sipsma
[dagger] New discussion #12469: Dagger `GetStarted` Interactive Shell Module

We could create a GetStarted module that opens an interactive shell and teaches you how to use Dagger.

You could take this really far, but as a baseline it could just be a container w/ the dagger CLI and some example module code. It then presents you with prompts for dagger CLI commands to run that show off the various things you can do.

More interesting stuff you could layer on top of that baseline:

  1. Per-SDK tutorials (i.e. specific ones for Go, Python, Node, etc.)
    • You could...
vivid lintelBOT
ornate vigilBOT
#
RonanQuigley
[dagger] New discussion #12470: Persist cache volumes into a container e.g. withCache

What are you trying to do?

Let's say I have a node app.

At present, I can use withMountedCache to reduce the npm install times i.e. it does not require us to re-fetch the dependencies over the internet.



```json
{
  "name": "test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.18.2"
...
#
purkhusid
[dagger] New discussion #12471: ✨ Support for stateful processes

What are you trying to do?

I've been looking into using Dagger for our CI pipeline in which one of the main tools we use is Bazel.

The issue with using Bazel in Dagger is that the Bazel startup time(Analysis phase) grows as your repository grows.
Most organizations using Bazel end up using persistent CI workers to lower this startup cost since Bazel keeps the repository state in memory and this makes incremental build very fast once the analysis phase is over.

This model is kind...

vivid lintelBOT
ornate vigilBOT
#
devops-matriz
[dagger] New discussion #12472: Pull docker images from private and insecure registry

Is there any way to pull docker images from a private and insecure registry instead of docker.io ??

This is my testing code:

import anyio
import dagger

async def main():
    config = dagger.Config(log_output=sys.stdout)

    async with dagger.Connection(config) as client:
        # use a node:16-slim container
        # mount the source code directory on the host
        # at /src in the container

        source = (
            client.container()            
          ...
vivid lintelBOT
ornate vigilBOT
#
sagikazarmark
[dagger] New discussion #12474: ✨ Default action for modules

What are you trying to do?

When grouping various actions in a "submodule", there is no way to assign an action to the parent at the moment.

Given the following module:

type Ci struct{}

func (m *Ci) Lint() *Lint {
	return &Lint{}
}

type Lint struct{}

func (m *Lint) All(ctx context.Context) error {
	var group errgroup.Group

	group.Go(func() error {
		_, err := m.Go().Sync(ctx)
		if err != nil {
			return err
		}

		return nil
	})

	group.Go(func() error...
#
gerhard
[dagger] New discussion #12475: 🐞 OCI labels no longer work by default in ghcr.io since v0.6.3

What is the issue?

When MIMEType defaults to application/vnd.oci.image.layer.v1.tar+gzip, ghcr.io no longer displays OCI labels correctly, i.e. https://github.com/users/gerhard/packages/container/sysadmin/114534202

This changed in v0.6.3 via:

When MIMEType is set to application/vnd.oci.image.layer.v1.tar.gzip, OCI labels work as expected, i.e. https://github.com/u...

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12475: 🐞 OCI labels no longer work by default in ghcr.io since v0.6.3

GHCR docs claim to support oci image spec annotations: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#labelling-container-images

So if I'm reading them right and they are to be believed, I think something else is probably breaking.

I know we have integ tests for publishing all the different mediatypes, and I think we have integ tests for labels, so it may be worth creating a mashup test and see what happens.

#
sipsma
[dagger] New discussion #12476: Make passing caller environment variables to modules more seamless

Right now callers can pass their environment variables to modules by just reading them and setting them to function arg values, e.g. w/ the cli:

dagger call my-function --some-arg $SOME_ARG_VALUE --other-arg env:OTHER_VALUE

This suffices for straightforward use cases but multiple users have found it to be tedious + clunky once you need to pass a lot of env vars. It also isn't ideal for some use cases where the caller may not know what they are even supposed to pass.

This issu...

#
dmarteau
[dagger] New discussion #12477: Dagger no resolving /etc/hosts

What happened? What did you expect to happen?

Using private registry with private CA (stored in /etc/docker/certs.d, we store the ip resolution in /etc/hosts.

It appears that Dagger do not take into account the /etc/hosts and fail to resolve the registry fqdn:

dagger.exceptions.QueryError: failed to do request: Head "...": dial tcp: lookup  on 10.87.0.1:53: no such host

Is there a way to tell Dagger to resolve the registry using the /etc/hosts and use the certificats...

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12476: Make passing caller environment variables to modules more seamless

The new default behavior, for an unspecified argument foo, is to look up the environment variable FOO, and use its value.

@shykes I like the general idea of your proposal in that we could make this a CLI-exclusive piece of functionality to start.

The auto-matching of arg name to env var would be the most magical part and I can definitely think of various corner cases or situations it might not handle super well. But that's obviously fine provided we at least improve the situation sig...

#
bitgandtter
[dagger/dagger] New comment on discussion #12476: Make passing caller environment variables to modules more seamless

@shykes, indeed, having it all in a sandbox makes the inner process more secure. It does not directly prevent secret exposure. But that said, getting back to the argument passing, if it's true that it's not a dagger issue itself, developers or ci/cd tools can make mistakes in exposing/printing those secrets to the logs/output example:

... --my-secret=$MY_SECRET --another-secret=$(gcloud auth print-access-token)

the second argument directly depends on the ci/cd tool handling/masking th...

#
grouville
[dagger/dagger] New comment on discussion #12479: 🐞 registry.dagger.io 503

Hi @bessey,

Thanks for your comment 🙏

Our registry piggybacks to ghcr, which is currently facing a major outage: https://www.githubstatus.com/
<img width="488" alt="image" src="https://user-images.githubusercontent.com/31691250/222166421-c739b28f-6b33-4085-91f2-013bf92a94b0.png">

We are taking that into consideration and have added in our backlog the implementation of a more resilient solution @gerhard

What you are experiencing is the fix we implemented a few weeks ago after a ...

#
marcosnils
[dagger] New discussion #12481: Dagger execution hangs when invoking func which requires docker socket

Steps to reproduce:

  1. Checkout the following branch https://github.com/marcosnils/kubeconeu-question-5/tree/dagger
  2. Run go run ./ci/main.go package and you'll notice that dagger execution never finishes after displaying something like #3 11.82 Layer 'buildpacksio/lifecycle:cache.sbom' SHA: sha256:a5cda63693b26825fec73e9367bbf216d1134e8c5d32f0459f568117dea3348b

Running the same pipeline with docker works:

`docker run -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd):/ap...

#
sipsma
[dagger/dagger] New comment on discussion #12481: Dagger execution hangs when invoking func which requires docker socket

@marcosnils thanks for the simpler reproducer, I poked at this for a little bit but am more confused than when I started now.

I was wondering if we just needed to propagate a close of the docker container conn to the buildkit client, so I made that change along with a few other println debugs here: https://github.com/sipsma/dagger/commit/0ee831ff343861718d4346951d6f302889c95b55

The first weird thing is that while most of the time I still get a hang when running with that branch, I occas...

#
Pymann
[dagger] New discussion #12483: 🖐️ When is image (re)pulled by dagger-engine

Dagger Cloud run URL

docker-container://dagger-engine

What happened? What did you expect to happen?

I tried some examples from dagger.io.

And for me is unclear in what cases the image is repulled by the dagger-engine.
Manually pulled images seem to be ignored?

The problem is even more critical if you imagine kubernetes pod running the dagger-engine. Because there will be no info after restart of pod. Mounted docker.sock should have wished effect... but If it will repull, dag...

#
grouville
[dagger/dagger] New comment on discussion #12483: 🖐️ When is image (re)pulled by dagger-engine

@Pymann, from what I understand, you've been:

  1. Pulling an image in Docker using the docker CLI
  2. Referencing the same image in a Dagger pipeline
  3. Realizing that the pipeline would download the Dagger image even though you have it in locally in Docker

If this is what you meant, it doesn't work that way: docker's cache and dagger's cache are not related, even when you mount the socket.

The problem is even more critical if you imagine kubernetes pod running the dagger-engine. Bec...

#
talios
[dagger] New discussion #12484: Support custom labels for service containers

What are you trying to do?

When using service containers, they just show up in the dagger TUI as "service nc512ita2341" with some randomly generated name, what I'd like to do is supply a user defined label to this, to make it more human-centered in its representation, so that I'd see something like

 Dagger Pipeline....                         
 │                                           
 ├──▶ service "Primary PostgreSQL Database"  
 │    ──▶ .....INSERT INTO xxxx            ...
#
sipsma
[dagger] New discussion #12485: WithWorkdir doesn't work as expected in existing dagger session

If you connect to an existing dagger session (e.g. via dagger-in-dagger or via dagger run) and specify a workdir it's ignored and instead the workdir will be the workdir of the session process.

I think the very minimum would be to throw an error if this happens (a workdir other than the one of the session is specified) just to at least replace the confusing behavior w/ annoying but comprehensible behavior.

It also might be possible to actually allow different clients to have different...

vivid lintelBOT
ornate vigilBOT
#
levlaz
[dagger] New discussion #12486: Make it more clear when to call dagger using CLI vs SDK directly

What is the issue?

It would be great to have a better high level overview of the differences between the CLI and the SDK to invoke dagger and when to use one or the other.

Right now we have this in the intro page but its a bit misleading because we say that you should use the CLI if you are a "fan of shell scripts", then the rest of the documentation (i.e. [go getting started](https://docs.dagger.io/sdk/go/959738/get-started#require...

#
red8888
[dagger] New discussion #12487: Examples for unit testing (or perhaps "integration" testing) pipeline code?

What is the issue?

I haven't found any examples of how this is suppose to work, but maybe its obvious and I'm not looking hard enough.

I know I can run the pipelines locally, but can I run pipelines locally and mock out all the actual commands? So I can test the logic without actually doing a deployment?

Is the solution to just factor your code in a way where you can unit test its components? Even if I do that I'd still want a way to "integration" test where I run the whole pipelin...

vivid lintelBOT
ornate vigilBOT
#
red8888
[dagger/dagger] New comment on discussion #12487: Examples for unit testing (or perhaps "integration" testing) pipeline code?

Just want to add more to this. This is not the testing I'm talking about: https://docs.dagger.io/sdk/python/628797/get-started#step-3-test-against-a-single-python-version

I don't want to actually execute commands I want to do something like mock them out. This is really important if you are capturing output of commands and parsing it in code.

Using python as an example there is a framework called pyinvoke designed to make it easy to execute shell commands programmatically. Its kind of a...

#
blaisep
[dagger] New discussion #12488: ✨Improve Ubuntu CLI packaging (aka Installer script for bash completion leads to yak shave)

What are you trying to do?

Installing dagger-cli from Homebrew is awesome.
less awesome is installing by

cd /usr/local
curl -L https://dl.dagger.io/dagger/install.sh | sh

I searched for "package" "install" and "completion" before opening this issue.
Labels: CLI, DX
(There is a thread describing the situation in detail at #1155601074393194658 message )

Why is this important to you?

I think that it will be e...

#
sipsma
[dagger] New discussion #12489: Support pushing to insecure http-only registry

User request to be able to push to http registry: #1046817210695417906 message

I can confirm that I get the same error but only when trying to push to a non-https registry that is not on localhost. There must be some logic buried somewhere in buildkit or its image push dependencies to not enfore https on localhost?

Either way, buildkit actually has a flag to support this during image export but for some reason I can't get it to ...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12490: Dagger Scripts: a more convenient way to call Dagger Functions from your project

TLDR: dagger call is very safe, but lacks the convenience of artisanal scripts. There is a way to get both: safety and convenience.

Problem

Now that Dagger Functions are generally available, the standard way to use Dagger from your project is to call functions. Today that is done with dagger call.

dagger call is a fantastic primitive: simple, reliably, and safe. If you understand dagger call, you understand how Dagge...

vivid lintelBOT
ornate vigilBOT
#
roboteng
[dagger] New discussion #12492: ✨ More idiomatic Rust examples

What are you trying to do?

This isn't quite a feature request, but a request for better examples.

I'm using the Rust sdk, as I want to use dagger to run my Rust project through CI. The examples do a good job of showing how to build go and js projects, but I still found it difficult to get a useable pipeline, written in Rust, for a Rust project.

I mad e Rust project, seperated into workspaces, one of which is ci. This is the main.rs file that I used to get Dagger to run with a r...

#
aluzzardi
[dagger] New discussion #12493: core: WithNewFIle doesn't follow relative paths

Test (see the Relative path):

func TestDirectoryWithNewFile(t *testing.T) {
	t.Parallel()

	c, ctx := connect(t)
	defer c.Close()

	entries, err := c.Directory().WithNewFile("some-file", dagger.DirectoryWithNewFileOpts{
		Contents: "some-content",
	}).Entries(ctx)
	require.NoError(t, err)
	require.Equal(t, []string{"some-file"}, entries)

	// Relative path
	dir := c.Directory().WithNewDirectory("a").WithNewFile("some-file", dagger.DirectoryWithNewFileOpts{
		Contents...
vivid lintelBOT
ornate vigilBOT
#
kpenfound
[dagger] New discussion #12494: ✨ Sign Dagger Release Artifacts

What are you trying to do?

The release artifacts (cli binary tars in the github releases and container in GHCR) should be signed

  • Detatched or clearsigned checksums.txt
  • Signed container in GHCR
  • ? SBOM for container in GHCR ?

Why is this important to you?

So that I know the artifact(s) are from Dagger

How are you currently working around this?

No response

vivid lintelBOT
ornate vigilBOT
#
kpenfound
[dagger/dagger] New comment on discussion #12495: Directory caching behavior

Based on a conversation in #help yesterday, I think this is true:

src := c.Host().Workdir().Directory("sdk/python")

This will send all of workdir to buildkit and changes outside of sdk/python will bust cache

src := c.Host().Directory("sdk/python")

this will just send sdk/python to buildkit and changes outside of sdk/python will not bust cache

We should make this more clear :)

vivid lintelBOT
ornate vigilBOT
#
nipuna-perera
[dagger] New discussion #12497: ✨ Better logging for cache mount conflicts

What are you trying to do?

I had a scenario while testing different builder images where I had to set different Owner for certain tool chain images.

...
WithMountedCache("/mydir/.m2", c.CacheVolume("m2"), dagger.ContainerWithMountedCacheOpts{
  Owner: "jenkins:jenkins",
}).
...

The current implementation of this silently ignores the new cache if a previous cache (with different owner) exists. https://pkg.go.dev/dagger.io/dagger#ContainerWithMountedCacheOpts

It ...

#
skycaptain
[dagger/dagger] New comment on discussion #12497: ✨ Better logging for cache mount conflicts

I believe I encountered the same problem. Here's the situation: I have two Dagger modules. The first builds my software as a non-root user and uses cache volumes. However, this module only handles the software's build. The second module uploads the cache to my Artifactory server. When I attempted to mount the Cache Volume using the same key, I received an empty directory. Through debugging, I discovered that I needed to mount the Cache Volume with the same owner for the files to even appear. ...

vivid lintelBOT
ornate vigilBOT
#
c-p-b
[dagger] New discussion #12498: ✨ Improved Cache visibility

What are you trying to do?

Part of the story around Dagger's appeal is its cache-first approach, backed by Buildkit.

Dagger Cloud, for those who have alpha access, offers some visibility features, namely a UI that shows whether something was cached or not. However, at Airbyte, where we leverage Dagger heavily, we have noticed limitations around the existing tooling that Dagger offers. Namely, we are missing some features that allow us to automatically analyze and qualify builds, as ...

vivid lintelBOT
ornate vigilBOT
#
d3rp3tt3
[dagger/dagger] New comment on discussion #12498: ✨ Improved Cache visibility

Thank you for your feedback, @cpdeethree!

It's on our roadmap for the UI to

  • Surface caching analytics across changes
  • Highlight caching analytics in a change
  • Make it easy to determine why something wasn't cached, debug it, fix it, and then verify it's now cached

For determining why something wasn't cached, there are two places in the UI we've considered adding more visibility: the tree view and the DAG view. You can see relations in the DAG view, but it's experimental a...

#
c-p-b
[dagger/dagger] New comment on discussion #12498: ✨ Improved Cache visibility

Thank you for your feedback, @cpdeethree!

It's on our roadmap for the UI to

  • Surface caching analytics across changes
  • Highlight caching analytics in a change
  • Make it easy to determine why something wasn't cached, debug it, fix it, and then verify it's now cached

For determining why something wasn't cached, there are two places in the UI we've considered adding more visibility: the tree view and the DAG view. You can see relations in the DAG view, but it's ex...

#
marcosnils
[dagger] New discussion #12499: Support --build-context [name] for Dockerfiles

Dagger currently has Dockerfile support through the DockerBuild and Build operations, but it only allows specifying a single build context at the time for that. Not so long ago, Docker (and Dockerfiles) added support for multiple named build contexts https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/ which allows to use multiple build contexts during a build.

I'd be nice if Dagger could add support for this use-case as well.

ref: https://discord.com/chann...

vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12500: Document caveats when using services due to the lack of an init process

What is the issue?

Differently than Docker which has an embedded tiny process (https://docs.docker.com/engine/reference/run/#specify-an-init-process), some Dagger users could potentially find differences while running service containers (i.e mongodb) as the service container will just exec as soon as the init process exists.

We should document this caveats and provide instructions for users finding themselves in this situations. Here's a discord thread which provides one possible ...

vivid lintelBOT
ornate vigilBOT
#
jpadams
[dagger] New discussion #12501: Document "magic" of how Dagger SDKs ensure that a Dagger Engine exists without an explicit install

What is the issue?

#1110935342577631253 message

From @ringods

you might want to highlight that the Dagger SDK will do some magic behind the scenes to ensure you have a Dagger Engine running in each of the CI integrations here: https://docs.dagger.io/7442989/cookbook#integrations
I was puzzled that I didn't need to install dagger as part of e.g. my Github Actions workflow.

vivid lintelBOT
ornate vigilBOT
#
nabsul
[dagger] New discussion #12502: ✨ Make Windows Binaries/Zip Trusted

What are you trying to do?

I'm downloading the Dagger binaries, but Edge browser is telling me that the zip archive is untrusted.

Why is this important to you?

It would be nice if zip/binary was signed so that Windows could trust it. I'm not sure how much effort this would involve.

How are you currently working around this?

I have to override and ignore the warnings that edge gives me.
![image](https://github.com/dagger/dagger/assets/11143071/ea1b01d9-6320-4b29-8a83-3795305f...

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12503: dagger query should reuse an existing session

What are you trying to do?

When running dagger query in a script by dagger run, the session from run isn’t being reused. dagger query should look for the port and token env vars to initiate a client connection.

The same goes for other commands since they share the same base for starting a session.

Why is this important to you?

Using setSecret with dagger query isn’t possible because the built-in secrets use an in-memory store which gets wiped on each new sess...

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger] New discussion #12504: Using images while offline fails even if they are cached

If you run a pipeline that references an image using a tag, the tag needs to be resolved to a SHA, which requires internet access (or, more specifically, network access to the registry).

This means that if you are offline you can't run those pipelines, even if the images are cached locally. This also results in the inability to run pretty much all of our test suite while offline, which is quite annoying if you are working on dagger and only have intermittent internet access.

Workarounds...

vivid lintelBOT
ornate vigilBOT
#
gerhard
[dagger/dagger] New comment on discussion #12504: Using images while offline fails even if they are cached

Yes, this makes a lot of sense.

We are already doing this for the Engine image:

failed to resolve image digest: Get "https://registry.dagger.io/v2/": dial tcp: lookup registry.dagger.io: no such host
falling back to leftover engine
Connected to engine fce67ef5b2b5

I just confirmed that sha256 image references uses the locally cached layers and works without an internet connection:

x ping -t 1 -c 1 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
pin...
#
bhmoto
[dagger/dagger] New comment on discussion #12476: Make passing caller environment variables to modules more seamless

Update: Build Fix Added — PR #1607

The deploy pipeline on main is broken due to a TypeScript error in the warehouse activity tab (from PR #1594, unrelated to our CA work). PR #1607 fixes it with a one-line type annotation.

Updated Merge Order

  1. PR #1607 — Build fix (unblocks deploys) — one-line, no behavior change
  2. PR #1604 — Root cause fix (remove attribute_not_exists(dc_id))
  3. PR #1598 — Post-push negative check (safety net)
  4. PR #1605 — Webhook receiver + t...
vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12506: Proposal: Helper for multiple pipeline synchronization

Summary

This proposal suggests adding an SDK helper for efficiently synchronizing multiple pipelines, in an easy to use way.

Background

This is based on one aspect of:

  • #4205

Motivation

Often times, you want to execute multiple pipelines in parallel (e.g. build & test). This is surprisingly tedious to do:

eg, gctx := errgroup.WithContext(ctx)

eg.Go(func() error {
    _, err := test.Sync(gctx)
    return err
})

eg.Go(func() error {
    _, err := li...
vivid lintelBOT
ornate vigilBOT
#
stobias123
[dagger] New discussion #12507: ✨ Observability - Find Opened/Read/Touched Files

What are you trying to do?

Would like to get a list of which files were read/opened during a pipeline run.

Context in discord [here](#general message)

Why is this important to you?

Lots of boilerplate code in our repos, significant amounts aren't actually used. Would be good cool to be able to say "xyz files can likely be deprecated"

How are you currently working around this?

If this isn't possible dire...

vivid lintelBOT
ornate vigilBOT
#
helderco
[dagger] New discussion #12508: Proposal: Codegen: Return simple objects

Summary

For simple object types where all their fields are non-parametrized scalars, make it more convenient to access the object’s properties in SDKs by returning the whole object at once as a leaf value.

Motivation

Our chainable API is pretty simple, but it also feels limiting when you want multiple related fields, especially if they’re a part of a simple object like EnvVariable, Port or Label.

For example, if I have an EnvVariable object, and I want both the `name...

vivid lintelBOT
ornate vigilBOT
#
vito
[dagger/dagger] New comment on discussion #12508: Proposal: Codegen: Return simple objects

For continuity from Discord conversations with @shykes:

We agree that there's a need for state and that there is effectively a fields vs. functions distinction internally, but @shykes 's point is that fields should just be an implementation detail of modules, not distinguished in any way through the API. The main take-away is that it should be easy for a module author to switch something from a field to a function internally without any impact on users.

So I think to address this we'll ...

#
helderco
[dagger] New discussion #12510: ✨ Python: make sure codegen types are hashable

What are you trying to do?

Making codegen types (from API) hashable would allow them to be used in some interesting use cases.

Why is this important to you?

As an example, one user could benefit from using an LRU cache in a function that receives a (lazy) pipeline, does something dynamic with it and returns it.

Assuming that such function is a dependency used more than once in the same project, and that the result will be the same if the input pipeline is the same, you ca...

vivid lintelBOT
ornate vigilBOT
#
d3rp3tt3
[dagger] New discussion #12511: Improvements for Jenkins integration documentation

What is the issue?

This issue will collect the various improvements we can make in the near term to the documentation for using Jenkins and Dagger together for CI.

Summary of issues

  • The Node and Go SDKs have examples of how to use CI with Dagger, but the CLI and Python SDKs do not
  • Users must know how to specify where the Dagger CLI gets installed. The docs do not detail that Dagger's uses the $XDG_CACHE_HOME env variable to find/download the CLI. If that variable is not...
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12512: Seamless ARM runners

Problem

Dagger supports multi-platform operations. The most common use of this feature is to run containers on both x86 and arm architectures. However in its standard, single-node configuration, the engine will use emulation under the hood, resulting in poor performance. In theory, this can be addressed by distributing execution across several machines, one for each architecture - the underlying buildkit runtime provides plumbing to do this. However in practice, configure buildkit i...

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12512: Seamless ARM runners

Rough immediate thoughts: we could support connecting to multiple runner backends from the session and then just route the requests based on the Platform field in the container/directory/file ID payloads.

I think the tricky part would be that currently separate runners don't know anything about each other, so when it came time to try to push a multiplatform image you'd somehow need them to coordinate between each other to push the different platforms for the image without stomping over e...

#
jpadams
[dagger] New discussion #12513: ✨ API for creating Dagger SDKs

What are you trying to do?

A consistent, more accessible way to generate new SDKs and add features to be implemented by existing SDKs.

Why is this important to you?

This will allow for the SDKs that users need to be produced by more community members and make them easier to maintain.

How are you currently working around this?

SDKs need deeper knowledge of and integration with Dagger core today.

cc @samalba

vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12513: ✨ API for creating Dagger SDKs

Passing thought, given the fact that:

  1. Quite a bit of the work here is pretty monotonous boilerplate (e.g. the codegen templates)
  2. We have a pre-existing set of examples (go, python, nodejs)

This feels like a potentially good use case for LLMs. It's too big an input+output to just plug into chatgpt, but I would not be surprised if a slightly more specialized setup of gpt or similar could take care of a lot of the boilerplate for something like this. Even if the result isn't perfect, ...

#
kjuulh
[dagger/dagger] New comment on discussion #12513: ✨ API for creating Dagger SDKs

I see a few issues in general. You kind of want to still hold the reins of what is in the sdks you don't want a super big sprawl because then adding features, and tweaking stuff will become a lot of work.

Each sdk has some core bits that needs to be added to a degree some of it could be solved using a shared library approach, most of the current languages technically could use a dll/so file approach for downloading the engine and creating a connection to it. This would reduce most the actu...

#
marcosnils
[dagger] New discussion #12514: Clarify the relationship between the docker and dagger images cache

When docker build is used with buildkit mode in the same machine, images present through docker pull are shared with buildkit automatically (not sure how it actually works under the hood). Some users (https://discord.com/channels/707636530424053791/1073696894380949545) which are currently trying out dagger pre-pull docker images as part of their node bootstrap and would like those images to be present in their Dagger pipelines as well so they don't have to be re-downloaded.

Probably ...

#
sipsma
[dagger/dagger] New comment on discussion #12514: Clarify the relationship between the docker and dagger images cache

When docker build is used with buildkit mode in the same machine, images present through docker pull are shared with buildkit automatically (not sure how it actually works under the hood)

This works because docker doesn't use vanilla buildkit, it imports buildkit as a library and then plugs in its own snapshotter and worker backend that integrates with docker's content stores: https://github.com/moby/moby/tree/master/builder

In docker's most recent release I believe support has been...

vivid lintelBOT
ornate vigilBOT
#
marcosnils
[dagger] New discussion #12515: Possible errors and inconsistent behavior if session binary and engine have different versions

Some advanced (#1073552228390031400 message) users are running the dagger engine separately and then using the _EXPERIMENTAL_DAGGER* flags to hint their SDK's accordingly.

The issue arises that if miss-matching session CLI's and engines are used, users could potentially get weird errors and/or behaviors.

Possible solutions

  • Print a warning message about these versions being different

cc @sipsma

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger/dagger] New comment on discussion #12518: Operator manual: Dagger on Gitlab CI self-hosted runners

User feedback [on discord](#general message):

In my customers organisation the DinD setup is (almost) forbidden on Gitlab runners so we are looking for ways to run Dagger without that. We are currently rolling out own runners for our team but maintaining them isn's something we are too keen about but we like Dagger that much that we are biting the bullet 🙂 Hopefully soon there are other options.

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
alixander
[dagger] New discussion #12523: d2oracle for dagger-graph

Hi @aluzzardi, I saw that you were using D2 for https://github.com/dagger/dagger/tree/main/cmd/dagger-graph, looks great!

I wanted to just bring up that we have an API that can make programmatic generation of D2 diagrams a lot easier/simpler to manage/debug than writing the script with fmt lines: https://pkg.go.dev/oss.terrastruct.com/d2@v0.1.5/d2oracle

Here's a blog post with runnable code on how it might be used: https://terrastruct.com/blog/post/generate-diagrams-programmatically/. A...

vivid lintelBOT
ornate vigilBOT
#
aluzzardi
[dagger/dagger] New comment on discussion #12523: d2oracle for dagger-graph

Neat, thanks @alixander! Indeed I missed it -- I went through the 1-d2lib example and entirely missed the 2-d2oracle one.

By the way, is there a way using d2oracle to set a label programmatically and skip "interpretation" altogether?

While generating using a string, I ran into escaping problems (single/double quotes,$ being interpreted etc). The graph "labels" are extracted from logs so they can contain anything.

#
davidwallacejackson
[dagger] New discussion #12524: ✨ Support preemptible execution environments

What are you trying to do?

I've recently been considering moving pieces of my existing (non-Dagger) CI pipeline to self-hosted infrastructure on Google's spot instances. Because these instances could be preempted at any time, it would be ideal if progress on my pipeline were saved incrementally, in as granular a way as possible. It occurred to me that Dagger's cache model (where each execution step can be cached, potentially to long-lived external storage) might make solving this problem...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger/dagger] New comment on discussion #12524: ✨ Support preemptible execution environments

That's very interesting. I wonder how much you could get out of the current version of Dagger on spot instances, with either a regular cache export at the end of the job, or persistent block storage to avoid the need for cache export entirely? Of course some jobs would be interrupted before the end, but depending on the average lifespan of a spot instance, it might happen infrequently enough to be worth it.

Regardless of caching, there are other aspects of this setup that would not be triv...

vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
RealHarshThakur
[dagger] New discussion #12529: Dagger build - resource requests/limits

What are you trying to do?

I'd like to set resource limits to build calls made to Dagger API.

Why is this important to you?

Having build resource limits will prevent abuse of the shared Dagger/Buildkit cluster we maintain

How are you currently working around this?

Not working around it currently.
Thanks to @sipsma pointing out that Buildkit already supports this so it would be something that Dagger API needs to support. https://discord.com/channels/707636530424053791/105084...

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger/dagger] New comment on discussion #12529: Dagger build - resource requests/limits

@sipsma can comment further, but I don't believe buildkit can be safely used in a multi-tenant way if tenants don't trust each other. It's reasonably sandboxed, but I wouldn't bet a customer's data on that. For example we run our API playground on a shared buildkit, but are very clear with users that they should not use it for anything critical, and they should not upload sensitive credentials or assets either.

This doesn't invalidate the need limits to avoid noisy ...

#
jedevc
[dagger/dagger] New comment on discussion #12529: Dagger build - resource requests/limits

@mircubed #8800 is just about allowing configuring existing options that today are configured through a different mechanism - it doesn't yet add any additional controls (but when we do add them, we would do so through the new engine.json mechanism it introduces).

Oh, can you please expand more on that? What would be the attack surface? Some sort of cache reuse attacks?

Yeah, buildkit's cache isn't separated per incoming user - it's shared across an entire instance. Essentially, if...

#
jedevc
[dagger/dagger] New comment on discussion #12529: Dagger build - resource requests/limits

In the original discord thread, the WithCgroupParent was mentioned - I think that's probably the right place to start?

Not entirely what level of granularity we'd want - I think it probably makes sense for the client to set session-wide cgroup limits (we probably don't want modules to have that level of machine-specific granular control over their WithExecs, but maybe there's a use case).

In terms of how to configure it: I think we'd want something like https://github.com/dagger/d...

#
marcosnils
[dagger] New discussion #12530: ✨ Make withEnvVariable return an error if variable is not set

What are you trying to do?

Currently there's no way to check if a host variable is unset or if it's value is empty since envVariable will always return empty regardless if the variable is set or not.

Given the following GraphQL queries:

{
  host {
    envVariable(name:"NON_EXISTING_VAR") {
      value
    }
  }
} 

{
  host {
    envVariable(name:"EMPTY_VAR") {
      value
    }
  }
}

dagger will always return "" for both cases. It'd be ni...

vivid lintelBOT
ornate vigilBOT
#
kminehart
[dagger] New discussion #12531: ✨ (Go) `container.Exec` with an anonymous function

What are you trying to do?

I would like to be able to, in a container, run an anonymous function instead of a shell script.

You can run a shell script using

container := client.Container().From("golang:1.19")

container = container.Exec(dagger.ContainerExecOpts{
    Args: []string{"some", "command},
})

but I would like to be able to run an anonymous function on the container using a function like container.Run

Ideally, it'd look like this:

vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12532: Save & load pipelines

Overview

I propose adding a new feature to the Dagger API: saving & loading of pipelines.

  • Save: snapshot the state of a pipeline, encode it as text, send it to the client
  • Load: restore the state of a pipeline, decoded from text received from the client

This would be a very powerful primitive that we could use in many ways:

  • Stitching. Allows clients to "stitch" pipelines together into more complex pipelines. This is necessary since some pipelines cannot be expressed a...
vivid lintelBOT
ornate vigilBOT
#
sipsma
[dagger/dagger] New comment on discussion #12532: Save & load pipelines

Many thoughts, but one immediate note is that this has significant overlap with some of the problems and possible solutions to session stickiness that @vito and I talked about previously. Specifically, see possible solution A) Pin sources in returned IDs in this issue: https://github.com/dagger/dagger/issues/3613

This issue and that one aren't the exact same (just overlapping venn diagrams), but that other issue may have some helpful context around the general idea of making IDs more re-...

#
vito
[dagger/dagger] New comment on discussion #12532: Save & load pipelines

Spent some time thinking about this and have a proposal to pick apart. There are still some loose ends and potential paths to explore, but there's enough to riff on async for now.

I started by defining a protocol format for IDs on the basis that they're closest thing we have to "save/loadable pipelines" and we've been kicking the can down the road with regard to their structure and format. We're going to want a canonical way to serialize/deserialize these, and [JSON ain't it][no-json], so ...

#
sipsma
[dagger] New discussion #12533: Improve engine garbage collection

The current approach for ensuring we don't leave a bunch of engine images+binaries around running forever is very primitive and doesn't allow multiple versions of an engine to coexist. More background in this comment and the one preceding it: https://github.com/dagger/dagger/pull/3647#issuecomment-1302721770

This means that we must always update engine dependencies in SDKs very close to each other and that users must update those SDKs in lockstep too if they use multiples from the same mac...

vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12533: Improve engine garbage collection

Agreed with point 1, whatever we do ends up needing some mechanism for that.

Point 2 though, does introduce a new point of friction for users - I'm a bit worried that this would be an annoying breaking change for most users.

Ideally, we'd have a way for engines to tear themselves down after not having been used for a period of time. But I can't see a way to convince the daemon to do this - maybe if we had a --restart=always or similar, then the engine could wipe it's own volume (even th...

#
spacedub
[dagger] New discussion #12534: Pipelines analytics

What are you trying to do?

One of the challenges with CI systems right now is the lack (or poor state) of analytics.

As a requirement for us to use dagger as a building block, we would love to have access to execution data so to build solid "pipeline observability", as a core feature to our product.

Why is this important to you?

Well, customers want it :)

Information about pipelines and "actions" execution time, failure trends, caching, are important to have for people in cha...

#
mgood
[dagger/dagger] New comment on discussion #12534: Pipelines analytics

Yes! I started to look into what's supported since I saw there is some tracing-related code, but I didn't see any documentation for it yet.

Something I'd love to see that I don't see mentioned here is a way to pass the tracing context all the way down so that I could add my own tracing to the jobs being run inside Dagger. For example, on a test run I'd love to have the ability to trace execution within the individual tests. This would be invaluable in pinpointing testing bottlenecks, or so...

vivid lintelBOT
ornate vigilBOT
#
benjaminstrasser
[dagger] New discussion #12535: ✨ Support for .gitignore

What are you trying to do?

Provide an easy way to clear a directory of any files ignored by git. The formal definition can be found here .

My suggestion would be the following:
func (r *Directory) WithoutGitignore() *Directory
This directory with all files removed that would be ignored by git

Side Note:
I don't currently need this but WithoutUntracked and other exclusions based on git might be useful as well.

Why is this impor...

#
marcosnils
[dagger/dagger] New comment on discussion #12535: ✨ Support for .gitignore

until it's decided if this will is a good candidate for the core API, you can currently do this by using the recently added include and exclude arguments in 0.4 https://github.com/dagger/dagger/pull/3583. In case you're wondering which is the pattern matching logic behind that include and exclude arguments, you can check it from here: https://github.com/moby/patternmatcher. It basically supports exact, prefix, suffix and regexp matching.

The .gitignore native feature wi...

vivid lintelBOT
ornate vigilBOT
#
benjaminstrasser
[dagger/dagger] New comment on discussion #12535: ✨ Support for .gitignore

I am not sure if simply parsing .gitignore files is sufficient as gitignored files are defined as followed:

A gitignore file specifies intentionally untracked files that Git should ignore. Files already tracked by Git are not affected

Using include and exclude might also cause problems, as tracked files, which match a .gitignore pattern will be removed. This file would have been present inside any CI framwork but now iw ignored.

To fully replicate this behavior one needs:

  • all .g...
#
marcosnils
[dagger/dagger] New comment on discussion #12535: ✨ Support for .gitignore

I am not sure if simply parsing .gitignore files is sufficient as gitignored files are defined as followed:

I agree, that's why I didn't suggest parsing the .gitignore files since that wasn't going to be enough. One thing you can do though is to get the output of git status --ignored or git check-ignore **/* and use that as an input for the exclude argument.

#
shykes
[dagger] New discussion #12536: Container.withError

Problem

When you have a beautiful chain of 20 function calls, no error checking end-to-end... except the one in the middle, then you're forced to break your chain, plumb error checking all the way up... So much beauty ruined.

Solution

Allow me to attach an error to the container, so that it is lazily returned to the caller, and I don't have to deal with error passing in my code. Example:

func Beautiful(ctx context.Context) *dagger.Container {
  checkTheThing := ...
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12536: Container.withError

Agreed, I previously considered whether we could instead have WithErr (naming tbd), which takes a func (*dagger.Container) (*dagger.Container, error). I'd rather we don't explicitly embed errors into containers, but I'm struggling to explain why I think that pattern is less desirable.

Regardless of the interface, the error would then get surfaced in the next function in the chain which returns an error, and if one is not present, it would be exposed at the top level of the module...

#
c-p-b
[dagger] New discussion #12537: 🐞 "Escapes Context" error confusing for end users

What is the issue?

The error message output by this should correctly indicate that two sibling modules being referenced via a local url must live within the same git repository. This will allow end users to much more easily understand why this is failing

Dagger version

dagger v0.10.0 (registry.dagger.io/engine) darwin/arm64

Steps to reproduce

☁  /tmp  rm -rf test     
☁  /tmp  mkdir test
☁  /tmp  cd test
☁  test  mkdir mod1
☁  test  mkdir mod2
☁  test ...
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
vivid lintelBOT
ornate vigilBOT
#
vikram-dagger
[dagger/dagger] New comment on discussion #12538: Module documentation is hard to find

Right now this is buried underneath Dagger API > Working With Modules.
...
I'd like to propose that we consolidate these documentation pages into a new top-level section - something like Developing Modules that combines these pages into a single section.

We iterated quite a few times on the docs structure and even at the time, modules were one of the topics we had a hard time fitting into the overall navigation. This is because modules a...

#
jpadams
[dagger] New discussion #12540: Continuous benchmark enhancements. Running real-world pipeline example across all SDKs.

Today we have benchmarking for running very basic benchmarks for
dagger init (to create the Dagger module with the default layout and example functions) and then dagger functions (to list the available functions) a few times (first uncached, then cached, then after modifying source code), that show latency of these commands and improvement areas for the SDKs around "getting things ready to run".

https://github.com/dagger/dagger/actions/workflows/benchmark.yml

We should also be running ...

#
gerhard
[dagger] New discussion #12539: Running the Engine in kata-containers fails with `failed to create hash for : operation not supported`

Given the following Talos v1.9.1 cluster:

kind: Cluster
name: dagger-ci-2025-01-02
talos:
  version: v1.9.1
kubernetes:
  version: v1.32.0
patches:
  # Single-node cluster FTW! 💪
  - name: allow-scheduling-on-controlplanes
    file: ../patches/allow-scheduling-on-controlplanes.yaml
  # https://www.talos.dev/v1.9/talos-guides/network/kubespan
  - name: kubespan
    file: ../patches/kubespan.yaml
  # https://github.com/siderolabs/omni-feedback/issues/41
  # https://s...
vivid lintelBOT
ornate vigilBOT
#
shykes
[dagger] New discussion #12541: NFS mounts

Problem

Dagger doesn't support mounting NFS directories directly into containers. Instead, the user must mount them into the client machine, and copy them from there into the engine, like any other host directory. This works reliably, but is less efficient than a direct bind-mount.

Solution

At this time there is no design for bypassing the copy into the engine. Nikola Jokic has [proposed implementing support for direct bind mounts](https://discord.com/channels/70763653042405379...

#
gerhard
[dagger/dagger] New comment on discussion #12539: Running the Engine in kata-containers fails with `failed to create hash for : operation not supported`

Having followed https://github.com/kata-containers/kata-containers/pull/3137 from @t3hmrman, I was able to get this Dagger StatefulSet to work:

<details>
<summary><code>kata-emptydir-memory.statefulset.yaml</code></summary>

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/name: dagger-kata-emptydir-memory
  name: dagger-kata-emptydir-memory
  namespace: dagger-2025-01-03
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kube...
#
jedevc
[dagger] New discussion #12542: Tests flake in `TestTelemetry/TestGolden`

See https://v3.dagger.cloud/dagger/traces/7ac8bb11c84ec5925eab8c0c8528c9d7#c8b39737d86fb4f8:EL46:

assertion failed: 
--- expected
+++ actual
@@ -31,4 +31,5 @@
 ✘ .stdout: String! X.Xs
 ! process "/bin/sh -c echo im failing && false" did not complete successfully: exit code: 1
+│ ✔ remotes.docker.resolver.HTTPRequest X.Xs
 
 Error logs:


You can run 'go test . -update' to automatically update testdata/TestTelemetry/TestGolden/docker-build-fail to the new expected value.'

Most of the...

#
jpadams
[dagger] New discussion #12543: 🐞 `WithoutEntrypoint` seems to be position-dependent wrt `WithDefaultArgs`

What is the issue?

In main it seems that these two pipelines should be equivalent:

dagger core container from --address postgres without-entrypoint with-default-args --args docker-entrypoint.sh,postgres,-c,log_min_messages=notice with-exposed-port --port 5432 with-env-variable --name POSTGRES_PASSWORD --value postgres up

dagger core container from --address postgres with-default-args --args docker-entrypoint.sh,postgres,-c,log_min_messages=notice without-entrypoint...
vivid lintelBOT
ornate vigilBOT
#
jedevc
[dagger/dagger] New comment on discussion #12541: NFS mounts

I'm kind of interested into diving into what the NFS mounts are going to be used for :eyes:

  • is it for avoiding the expensive context transfer of files?
  • is it to enable a faster speed between an engine-fileserver while the cli is remote and slow?
  • is it to allow a writeback to the NFS server?

The use case determines what we'd want to build here I think - whether it can be entirely transparent, or whether there's a dag.NFS equivalent like we have dag.Git, etc.

I've also...

#
sipsma
[dagger/dagger] New comment on discussion #12541: NFS mounts

I'm curious to know the specifics of the use case here, especially around how the data in the NFS mount would be used. The two categories of data we support currently are "immutable recipe-defined layers" and "mutable cache mounts". I'm guessing the use case here probably follows the "mutable cache mounts" pattern, but want to confirm. I think this actually impacts the design pretty significantly.

I can't tell yet how much this relates back to the engine/cli split. Theoretically you coul...

#
IljaKroonen
[dagger/dagger] New comment on discussion #12541: NFS mounts

We run Dagger on hosts with mounted fileshares used for file uploads/downloads. Programs may read configuration files from these shares, upload files for external API processing, or download files generated by APIs.

Currently, this workflow isn't supported in Dagger, restricting its use to a subset of CI workflows.

is it for avoiding the expensive context transfer of files?
is it to allow a writeback to the NFS server?

Yes to both. These fileshares are too large to copy, and we ...